Merge pull request #3: Release 0.9.0-beta.1

Release 0.9.0-beta.1

CHANGELOG.md

@@ -6,6 +6,51 @@ All notable changes to NG Backup are documented here. The format follows
 own `info.xsd` doesn't allow a `-alpha`/`-beta`/`-rc` suffix there); the stability stage is instead
 carried by the git tag, the release, and this file.
 
+## [0.9.0-beta.1] - 2026-10-07
+
+First beta. Tested thoroughly and usable **alongside** your current backup, but **not yet a
+drop-in replacement** for it. We test what we can ourselves (Nextcloud 32–35, SQLite, MariaDB and
+PostgreSQL, S3, SFTP, FTP, SMB and NFS), but we need field experience from other setups: please
+report what works and what doesn't, especially on large instances and on storage providers not
+listed in the README.
+
+### Added
+- Disaster recovery: `occ backup:key:import` and `occ backup:restore:full` restore the database,
+  data directory and config onto a fresh installation of the same major version.
+- Per-user backup and restore via user_migration: `occ backup:user:backup` and
+  `occ backup:user:restore` (replace the account after a safety export, or restore as
+  `<uid>-bak`), also onto a server where the user does not exist yet. The commands ask before
+  installing user_migration (or take `--install-user-migration`).
+- Checksum audit: `occ backup:verify`, shallow or `--deep`, for snapshots and per-user exports.
+- Retention for per-user exports: `occ backup:retention --user-last=N`.
+- FTP and SMB covered by the integration tests; NFS through the `local` backend on a mounted share.
+- `docs/HOWTO.md`: step by step from setup to a tested restore.
+- Warnings when a backup finishes with unreadable files or folders, or with files that kept
+  changing while being read (log, `occ`, notification, snapshot).
+
+### Changed
+- `backup:restore:full` refuses on an installation with more than one user and asks first;
+  `backup:user:restore --mode=replace` asks before deleting an existing account (`--force` for
+  scripts in both cases).
+- A failed scheduled backup is retried at most three times a day, an hour apart.
+- The recovery kit carries the deletion delay; the kit file is created owner-only.
+- Adding a location through the web API needs password confirmation.
+
+### Fixed
+- Exports of users whose id contains `.part-` could be deleted by cleanup on remote locations.
+- Cleanup no longer frees the data of a per-user export whose manifest is missing.
+- Crash safety of moving snapshots to and from the trash, of catalog writes by concurrent
+  backups and per-user exports, and of short uploads.
+- A file that keeps changing no longer makes a backup restart forever.
+- Long restores and exports keep their lock on the location alive.
+- Floating-point database columns are dumped exactly.
+
+### Known limitations
+See "Known limitations" in the README: memory use on very large instances, never two servers
+writing to one location, removing a passphrase slot does not revoke access, no rollback detection
+before a disaster recovery, and a few smaller points. None of them block the road to 1.0.0; they
+are on the backlog in `PLAN.md`.
+
 ## [0.1.0-alpha.1] - 2026-10-05
 
 First public alpha. Feature-complete for phase 1: scheduled, encrypted, incremental backups, and
@@ -43,4 +88,5 @@ and not implemented yet.
 - See the "Storage backend experiences" section of the README for provider-specific quirks found
   so far (e.g. Hetzner Storage Box sub-accounts).
 
+[0.9.0-beta.1]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.0-beta.1
 [0.1.0-alpha.1]: https://github.com/dillardblom/ng_backup/releases/tag/v0.1.0-alpha.1

PLAN.md

@@ -359,6 +359,24 @@ minimale volledige restore (`occ backup:restore:full` op een verse installatie)
     Backlog: (a) de originele weergavenaam herstellen (het label uit de export is niet getest als
     weergavenaam); (b) de interne `migrated-`-naam hernoemen (vereist directe databasewijziging, niet voor de beta).
 
+**Backlog uit de pre-beta review (2026-10-07)** — geen belemmering voor 0.9.0-beta of 1.0.0, staat
+in README "Known limitations":
+1. Geheugen bij grote instanties (>1 TB, miljoenen bestanden): blob-index, prune-set, browse van een
+   snapshot in de web-UI, `trashExtras` bij restore en het manifest van een user-export staan volledig
+   in het geheugen; streaming/paginering nodig.
+2. Twee servers op één locatie: catalogus kan breken (generatie overschreven, of anchor >200 generaties
+   achter). Docs-waarschuwing staat er; mogelijk later een echte put-if-absent per backend.
+3. Sleutelrotatie: een slot verwijderen trekt geen toegang in (master-key blijft gelijk).
+4. Rollback vóór DR niet detecteerbaar: referentie-generatie en -hash in de kit opnemen of tonen.
+5. Kit-velden buiten `wrapped_key` (o.a. `delete_delay_days`) zijn niet geauthenticeerd; nu alleen getoond.
+6. Repository-id in de AD van objecten (formaatwijziging).
+7. Wachtwoorden van locaties via `--option-file` of omgevingsvariabele i.p.v. de commandoregel.
+8. Tijdelijke restore-bestanden niet via de Files-API verwijderen (komen nu in de prullenbak).
+9. DB-restore in foreign-key-volgorde (cascades).
+10. Packs zonder index (index-upload mislukt) opruimen.
+11. `backup:restore:full` in onderhoudsmodus en hervatbaar maken; nu: opnieuw draaien met `--force`.
+12. Config/rename op FTP/SMB-servers die geen rename over een bestaand bestand toestaan.
+
 Doorgeschoven naar **v1.3.0** (fase 3, niet meer fase 2): in-place volledige restore
 (eigen restore-modus i.p.v. onderhoudsmodus) en zelfbediening voor gebruikers (eigen bestanden
 terugzetten vanuit de Files-app).

README.md

@@ -7,8 +7,9 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
 see "Storage backend experiences" below — that requirement comes from files_external, not from
 ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
 
-**Status: pre-release. Not yet tested on production.** See `PLAN.md` for the design and `docs/HOWTO.md` for a
-step-by-step guide from setup to a tested restore.
+**Status: beta (0.9.0).** Tested thoroughly, usable alongside your current backup, but not yet a
+drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and
+`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore.
 
 License: AGPL-3.0-or-later.
 
@@ -147,6 +148,29 @@ Contributions welcome via PR — add what you ran into with your own provider.
   mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS
   can't be driven from PHP directly.
 
+## Known limitations
+
+On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it:
+
+- **Very large instances** (well over 1 TB, millions of files): the blob index, cleanup, browsing a
+  snapshot in the web interface and per-user export manifests are held in memory. Expect high
+  memory use there; reports from such instances are very welcome.
+- **One location, one server.** Never let two Nextcloud servers write to the same location (for
+  example a disaster recovery test server pointed at the production location): they can break each
+  other's catalog, and the location is then refused as possibly tampered with. Test a disaster
+  recovery against a copy of the location, or only read from it.
+- **Removing a passphrase slot does not revoke access.** The master key stays the same, so someone
+  with an old recovery kit and that slot's passphrase can still decrypt the location if they can
+  read it. Key rotation is not available yet.
+- **Rollback before a disaster recovery is not detected.** A fresh server has no record of the
+  newest catalog generation, so a location rolled back to an older state looks valid there.
+- Location passwords passed with `-o password=...` are visible in the process list and shell
+  history while the command runs.
+- Temporary files of a restore into a user's files end up in that user's trash.
+- Restoring the database does not order tables by foreign keys; apps with cascading foreign keys
+  could lose rows (Nextcloud core does not use them).
+- A pack whose index upload failed stays on the location until removed by hand.
+
 ## Commands
 
 | Command | Purpose |

appinfo/info.xml

@@ -4,12 +4,12 @@
     <id>ng_backup</id>
     <name>NG Backup</name>
     <summary>Encrypted, incremental backups of your Nextcloud with restore from the web interface</summary>
-    <description><![CDATA[Alpha: feature-complete for scheduled, encrypted, incremental backups to
-S3/SFTP/WebDAV/local, with restore of files and folders from the web interface and occ. Meant for
-testing on a copy of production (dev, test, staging) -- not yet for production use, and full
-disaster recovery is not implemented yet. See the recovery kit section of the README before
-relying on this for anything.]]></description>
-    <version>0.1.0</version>
+    <description><![CDATA[Beta: scheduled, encrypted, incremental backups of the database, config and data
+directory to S3, SFTP, WebDAV, FTP, SMB or a local path, with restore of files and folders, of
+single users (via user_migration) and of the whole server onto a fresh installation. Tested
+thoroughly and usable alongside your current backup, but not yet a drop-in replacement for it.
+See the README for known limitations and the recovery kit before relying on it.]]></description>
+    <version>0.9.0</version>
     <licence>agpl</licence>
     <author>Dillard Blom</author>
     <namespace>NgBackup</namespace>

docs/HOWTO.md

@@ -14,7 +14,8 @@ occ backup:key:confirm -u <your-admin-uid> --code=<code shown> --confirm="Yes, I
 
 - The passphrase protects the backup. Keep it somewhere other than this server.
 - Optional: extra passphrases for other people with `occ backup:key:slot:add --label="..."`.
-  Any one passphrase is enough to restore.
+  Any one passphrase is enough to restore. Removing a slot later does not revoke access for someone
+  who kept an old kit and that passphrase (see "Known limitations" in the README).
 - The recovery kit (`/var/tmp/ng_backup-kit.json` above) is also required for disaster recovery. The web
   user writes it there; then move it off the server as root, next to the passphrase, and delete the
   copy in `/var/tmp`. Without the kit and a passphrase, the backup cannot be read.
@@ -117,7 +118,8 @@ occ backup:user:restore offsite users/<uid>/<exportId> --mode=as-backup
 ## 7. Disaster recovery: restore the whole server
 
 Use a fresh Nextcloud installation of the same major version, with NG Backup installed and
-the same location available:
+the same location available. For a test, use a **copy** of the location: two servers writing to one
+location break each other's catalog.
 
 ```
 occ backup:key:import /path/ng_backup-kit.json --passphrase-file=/path/passphrase

lib/Backend/StorageBackend.php

@@ -155,12 +155,16 @@ final class StorageBackend implements IBackend {
 		closedir($dh);
 	}
 
-	/** Bytes left to read in a seekable $stream, or null when it cannot tell. @param resource $stream */
+	/**
+	 * Bytes left to read in a local $stream (php://temp, php://memory or a plain file), or null:
+	 * stream wrappers of remote storages do not report a reliable size.
+	 */
 	public static function remaining(mixed $stream): ?int {
 		if (!is_resource($stream)) {
 			return null;
 		}
-		if (!(stream_get_meta_data($stream)['seekable'] ?? false)) {
+		$meta = stream_get_meta_data($stream);
+		if (!($meta['seekable'] ?? false) || !in_array($meta['wrapper_type'] ?? '', ['PHP', 'plainfile'], true)) {
 			return null;
 		}
 		$stat = fstat($stream);

lib/BackgroundJob/BackupJob.php

@@ -46,11 +46,11 @@ final class BackupJob extends TimedJob {
 		$this->startScheduled();
 		foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) {
 			while (microtime(true) < $deadline && $run->getStatus() === 'running') {
-				$before = $run->getUpdatedAt();
-				$run = $this->backups->step($run, min($deadline, microtime(true) + $budget));
-				if ($run->getUpdatedAt() === $before && $run->getStatus() === 'running') {
-					break; // another process holds this run's lease: leave it to that process
+				$next = $this->backups->step($run, min($deadline, microtime(true) + $budget));
+				if ($next === $run) {
+					break; // step() hands back the same object when another process holds the run's lease
 				}
+				$run = $next;
 			}
 		}
 		$this->failStaleRestores();

lib/Command/KeyKit.php

@@ -47,8 +47,14 @@ final class KeyKit extends Command {
 		} finally {
 			umask($old);
 		}
-		if ($fh === false || fwrite($fh, $json) !== strlen($json) || !fclose($fh)) {
-			$output->writeln("<error>Cannot write $file (it must not exist yet)</error>");
+		if ($fh === false) {
+			$output->writeln("<error>Cannot create $file (it must not exist yet)</error>");
+			return 1;
+		}
+		$ok = fwrite($fh, $json) === strlen($json);
+		if (!fclose($fh) || !$ok) {
+			@unlink($file);
+			$output->writeln("<error>Cannot write $file</error>");
 			return 1;
 		}
 		$output->writeln("Recovery kit written to <info>$file</info>. Copy it off this server, then run <info>occ backup:key:confirm</info>.");

lib/Repository/Repository.php

@@ -385,7 +385,7 @@ final class Repository {
 		// either the old state or the new one plus unreferenced leftovers, never a catalog entry
 		// pointing at a missing object. The snapshot object is bound to its id (not its path),
 		// so it can be copied as raw bytes.
-		$this->backend->put('trash/snapshots/' . $snapshotId, $this->backend->get('snapshots/' . $snapshotId));
+		$this->copyRaw('snapshots/' . $snapshotId, 'trash/snapshots/' . $snapshotId);
 		$this->putObject('trash/info/' . $snapshotId, json_encode(['forgottenAt' => time(), 'by' => $by, 'db' => $meta['db'] ?? null,
 			'dbInTrash' => $dbInTrash, 'time' => $meta['time'] ?? null, 'label' => $meta['label'] ?? ''], JSON_THROW_ON_ERROR));
 		unset($this->catalog->snapshots[$snapshotId]);
@@ -405,7 +405,7 @@ final class Repository {
 			$this->putObject($info['db'], $this->getObject($info['dbInTrash']));
 		}
 		// Same order as forget(): copy, switch the catalog, then delete the trash copies.
-		$this->backend->put('snapshots/' . $snapshotId, $this->backend->get('trash/snapshots/' . $snapshotId));
+		$this->copyRaw('trash/snapshots/' . $snapshotId, 'snapshots/' . $snapshotId);
 		$this->catalog->snapshots[$snapshotId] = true;
 		unset($this->catalog->trash[$snapshotId]);
 		$this->catalog->write();
@@ -416,6 +416,25 @@ final class Repository {
 		$this->backend->delete('trash/info/' . $snapshotId);
 	}
 
+	/** Copy an object as raw bytes, buffered locally (remote read streams cannot be uploaded directly everywhere). */
+	private function copyRaw(string $from, string $to): void {
+		$in = $this->backend->get($from);
+		$buffer = fopen('php://temp/maxmemory:' . (4 * 1048576), 'w+b');
+		if ($buffer === false) {
+			fclose($in);
+			throw new RepositoryException("Cannot buffer $from");
+		}
+		try {
+			if (stream_copy_to_stream($in, $buffer) === false) {
+				throw new RepositoryException("Cannot read $from");
+			}
+		} finally {
+			fclose($in);
+		}
+		rewind($buffer);
+		$this->backend->put($to, $buffer);
+	}
+
 	/** Record a user_migration export's manifest in the catalog (rollback-protected like a snapshot). */
 	public function recordUserExport(string $manifestPath): void {
 		$this->catalog->userExports[$manifestPath] = true;

tests/integration/full-restore-roundtrip.sh

@@ -68,7 +68,7 @@ run_on ngb-frt-restore backup:target:add offsite local -a null::null -o datadir=
 run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location"
 ok "recovery kit imported, existing repository adopted, snapshot visible"
 
-run_on ngb-frt-restore backup:restore:full offsite "$SNAP" >/dev/null || fail "restore:full"
+run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction >/dev/null || fail "restore:full"
 ok "backup:restore:full completed"
 
 # Everything below must work in a FRESH occ process (a new PHP process per docker exec), since