Merge pull request #18 from dillardblom/rc5

Release 1.0.0-rc.5

.github/workflows/ci.yml

@@ -114,3 +114,13 @@ jobs:
         run: tests/integration/full-restore-roundtrip.sh 35-apache mysql legacy
       - name: Full restore over SFTP on PostgreSQL, reading packs ahead on several connections
         run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql sftp
+      - name: Full restore of a MariaDB backup onto PostgreSQL (ID counters after the data)
+        run: tests/integration/full-restore-roundtrip.sh 35-apache mysql to-pgsql
+      - name: Full restore of a PostgreSQL backup onto MariaDB
+        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql to-mysql
+      - name: Full restore of a PostgreSQL backup onto SQLite
+        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql to-sqlite
+      - name: Full restore of a MariaDB backup onto SQLite
+        run: tests/integration/full-restore-roundtrip.sh 35-apache mysql to-sqlite
+      - name: Full restore of a SQLite backup onto PostgreSQL
+        run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite to-pgsql

CHANGELOG.md

@@ -7,6 +7,39 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
 beta/daily update channel, which would hide the app from most admins. The stability stage is
 instead carried by the app description, the git tag, the release, and this file.
 
+## [1.0.0-rc.5] - 2026-10-22
+
+Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
+stopped after 13 hours with "Blob not in index"; the backup itself was fine.
+
+### Changed
+- README and app store description: NG Backup needs background jobs on system cron, with how to
+  set that up (issues #13, #17).
+
+### Fixed
+- A full restore no longer stops with "Blob not in index" after a pack could not be read for a
+  moment. Reading the index again then got an empty listing (from a connection the location had
+  dropped) and emptied the index. Reading the index again now only adds, an empty listing of a
+  location that has an index is an error instead of an empty index, and a pack that could not be
+  read is read once more before the restore gives up.
+- After a full restore of a MySQL/MariaDB or SQLite backup onto PostgreSQL, new files, users or
+  activities could not be added ("duplicate key", "File entry could not be inserted"): such
+  backups record no ID counters, and they stayed at 1. Every counter of a restored table now goes
+  after its data. Found in a disaster recovery simulation of 138 GB from MariaDB to PostgreSQL,
+  where all 15,772 files and every count were identical.
+- A full restore now runs in maintenance mode: a background job, a page visit or a user writing
+  into the instance meanwhile could collide with restored rows ("duplicate key") or be lost. It
+  is switched off when the restore completes; after an error it stays on, and the message gives
+  the one line to switch it off and run the restore again. Maintenance mode that was on before
+  the restore stays on.
+- Adding a location whose repository config could not be read, while backup data is there,
+  no longer creates a new repository over it: it stops with an explanation. A snapshot that
+  cannot be taken over when a location is added is now logged instead of left out silently.
+- An SFTP connection that a Hetzner Storage Box dropped without phpseclib noticing is now
+  recognised and replaced: the check asks the server and counts its answer.
+- The extra connection a full restore reads ahead on is replaced when it breaks (up to 20 times),
+  instead of reading the rest of the restore the slow way.
+
 ## [1.0.0-rc.4] - 2026-10-19
 
 Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not

README.md

@@ -33,6 +33,26 @@ Need help restoring a backup after a crash? Ensembia offers paid restore assista
 - Restores go through Nextcloud, so overwritten files keep their old content as a version and
   removed files go to the trash.
 
+## Requirements
+
+- **Background jobs on system cron.** NG Backup runs backups in steps from Nextcloud's
+  background job, so a backup only moves forward when that job runs. Choosing "Cron" under
+  **Administration settings → Basic settings → Background jobs** is not enough on its own: it
+  tells Nextcloud that the system will call it, so you also need a cron line that does. Without
+  it a backup stays at "running (db)", and so does one started with "Back up now".
+  `occ backup:run` in a terminal works without cron (it runs in the foreground).
+  - Regular server: `crontab -u www-data -e` and add
+    `*/5 * * * * php -f /var/www/nextcloud/cron.php` (your Nextcloud path).
+  - Docker (the official `nextcloud` image): on the host, `crontab -e` as root and add
+    `*/5 * * * * docker exec -u www-data <container> php -f /var/www/html/cron.php`, or add a
+    service with the same image and `entrypoint: /cron.sh` to your `docker-compose.yml`.
+  - Nextcloud All-in-One already runs cron.
+
+  After a few minutes, the Background jobs section shows that the last job ran just now. See the
+  [Nextcloud documentation on background jobs](https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/background_jobs_configuration.html).
+- A storage location: S3 or compatible, SFTP, WebDAV, SMB (needs the PHP `smbclient` extension,
+  see below), FTP or a local path.
+
 ## Quick start
 
 ```

appinfo/info.xml

@@ -36,12 +36,13 @@ All-in-One and hosted Nextcloud.
 - One user (account, files, calendars, contacts) via user_migration.
 - The whole server onto a fresh installation (disaster recovery).
 
+NG Backup needs Nextcloud's background jobs on system cron (a real cron line, not only the setting).
 Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very
 welcome: use the "Share your experience" link on the admin page.
 
 Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
 you need help with restoring a backup after a crash.]]></description>
-    <version>0.9.10</version>
+    <version>0.9.11</version>
     <licence>agpl</licence>
     <author mail="dillard.blom@ensembia.com">Dillard Blom</author>
     <namespace>NgBackup</namespace>
@@ -52,9 +53,9 @@ you need help with restoring a backup after a crash.]]></description>
     <website>https://github.com/dillardblom/ng_backup</website>
     <bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
     <repository type="git">https://github.com/dillardblom/ng_backup</repository>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/1-backups.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/2-restore.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/3-settings.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/1-backups.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/2-restore.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/3-settings.png</screenshot>
     <dependencies>
         <php min-version="8.1"/>
         <lib>sodium</lib>

docs/DISASTER-RECOVERY.md

@@ -94,7 +94,16 @@ tables that do not exist here, install the apps they belong to and run it again,
 out with `--skip-missing-tables`.
 
 **8. Let the restore run.** It restores the database, then the data directory, and takes hours for
-a large backup. If it stops halfway, fix the cause and run the same command again with `--force`.
+a large backup. Nextcloud is in maintenance mode meanwhile, so nobody and no background job
+writes into it; the restore switches it off when it completes. (If you had switched maintenance
+mode on yourself before, it stays on: switch it off with `occ maintenance:mode --off` when ready.)
+
+If it stops halfway, maintenance mode stays on (the instance may be partly restored). Fix the cause,
+then switch maintenance mode off and run the restore again with `--force`, in one line (the message
+prints it): `occ maintenance:mode --off && occ backup:restore:full <location> <snapshot> --force`.
+Leave out `--install-apps` this time, so that the instance is open only for the seconds until the
+restore has switched maintenance mode on again; apps the first run could not install are listed at
+the end of the restore, install those afterwards.
 If occ then says that Nextcloud or one of the apps requires an upgrade, the database is already
 the old server's: run `occ upgrade` first, then the same command with `--force`.
 

lib/Backend/SftpRangeFetcher.php

@@ -54,6 +54,9 @@ final class SftpRangeFetcher {
 	private const FXF_READ = 1;
 	private const FX_OK = 0;
 
+	/** Connections that broke and are replaced, at most, in one restore. */
+	public const MAX_RECONNECTS = 20;
+
 	private const OPEN = 0;
 	private const READ = 1;
 	private const CLOSE = 2;
@@ -65,6 +68,8 @@ final class SftpRangeFetcher {
 	/** @var array<int, resource|null> ranges read (or failed) and not yet given to the caller */
 	private array $done = [];
 	private int $requestId = 0x4e520000;
+	/** Connections replaced so far (see MAX_RECONNECTS). */
+	public int $reconnects = 0;
 
 	/**
 	 * @param array<int, object> $connections phpseclib 2 connections with their SFTP channel open
@@ -75,6 +80,7 @@ final class SftpRangeFetcher {
 		private string $root,
 		private array $ranges,
 		private string $spool,
+		private ?\Closure $connect = null,
 	) {
 		foreach (array_keys($connections) as $c) {
 			$this->chunks[$c] = self::CHUNK;
@@ -86,10 +92,12 @@ final class SftpRangeFetcher {
 	 *
 	 * @param list<object> $connections
 	 * @param list<array{0:string, 1:int, 2:int}> $ranges
+	 * @param (\Closure():?object)|null $connect a new connection in place of one that broke
+	 *                                         (null when it cannot be made)
 	 * @return \Generator<int, resource|null>
 	 */
-	public static function fetch(array $connections, string $root, array $ranges, string $spool): \Generator {
-		$fetcher = new self($connections, rtrim($root, '/'), $ranges, $spool);
+	public static function fetch(array $connections, string $root, array $ranges, string $spool, ?\Closure $connect = null): \Generator {
+		$fetcher = new self($connections, rtrim($root, '/'), $ranges, $spool, $connect);
 		try {
 			yield from $fetcher->run();
 		} finally {
@@ -319,6 +327,27 @@ final class SftpRangeFetcher {
 		}
 		self::disconnect($this->connections[$c]);
 		unset($this->connections[$c]);
+		// A long restore outlives connections (a Storage Box drops one now and then): without a
+		// new one, the rest would be read the slow way.
+		if ($this->connect === null || $this->reconnects >= self::MAX_RECONNECTS) {
+			return;
+		}
+		$this->reconnects++;
+		try {
+			$new = ($this->connect)();
+		} catch (\Throwable) {
+			$new = null;
+		}
+		if (function_exists('OCP\Log\logger')) {
+			\OCP\Log\logger('ng_backup')->info($new === null
+				? 'Reading ahead: an SFTP connection broke and no new one could be made'
+				: 'Reading ahead: an SFTP connection broke; reading on with a new one');
+		}
+		if ($new !== null) {
+			$key = max(array_merge(array_keys($this->chunks), [-1])) + 1;
+			$this->connections[$key] = $new;
+			$this->chunks[$key] = self::CHUNK;
+		}
 	}
 
 	/** @return resource|null a new file in the spool, already unlinked: nothing is left behind if the process dies */

lib/Backend/StorageBackend.php

@@ -282,10 +282,19 @@ final class StorageBackend implements IBackend, IReadsAhead {
 		if ($connections === []) {
 			return null;
 		}
-		return (static function () use ($storages, $connections, $root, $abs, $spool): \Generator {
+		// in place of a connection that broke: a new storage, kept like the others
+		$connect = static function () use ($reconnect, &$storages): ?object {
+			$storage = $reconnect();
+			$sftp = SftpReadStream::connection($storage);
+			if ($sftp !== null) {
+				$storages[] = $storage;
+			}
+			return $sftp;
+		};
+		return (static function () use (&$storages, $connections, $root, $abs, $spool, $connect): \Generator {
 			// the storages hold their connections: kept until the fetcher is done
 			try {
-				yield from SftpRangeFetcher::fetch($connections, $root, $abs, $spool);
+				yield from SftpRangeFetcher::fetch($connections, $root, $abs, $spool, $connect);
 			} finally {
 				unset($storages);
 			}
@@ -478,8 +487,9 @@ final class StorageBackend implements IBackend, IReadsAhead {
 		$this->lastUsed = microtime(true);
 	}
 
+	/** After a failed request: ask the server (one round trip), as phpseclib may not know it is gone. */
 	private function reopenIfDead(): bool {
-		if ($this->reconnect === null || !$this->sftpReads || $this->alive(false) !== false) {
+		if ($this->reconnect === null || !$this->sftpReads || $this->alive(true) !== false) {
 			return false;
 		}
 		$this->reopen();
@@ -508,8 +518,9 @@ final class StorageBackend implements IBackend, IReadsAhead {
 			if (is_callable([$sftp, 'clearStatCache'])) {
 				$sftp->clearStatCache(); // or stat() answers from memory
 			}
-			$sftp->stat('.');
-			return (bool)$sftp->isConnected();
+			// A Storage Box drops a quiet connection without phpseclib noticing: it still says
+			// connected, and every request fails. So the answer counts, not what it says after.
+			return $sftp->stat('.') !== false && (bool)$sftp->isConnected();
 		} catch (\Throwable) {
 			return false;
 		}

lib/Command/RestoreFull.php

@@ -88,10 +88,20 @@ HELP);
 					$this->installApp($app, $output);
 				}
 			}
+			$output->writeln('Nextcloud is in maintenance mode while the restore runs; it is switched off when the restore completes.');
 			$result = $this->restoreFull->restoreFull($target, $snapshot, $skipMissingTables);
 		} catch (\Throwable $e) {
 			$output->writeln('<error>' . $e->getMessage() . '</error>');
-			$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
+			if (\OCP\Server::get(\OCP\IConfig::class)->getSystemValueBool('maintenance', false)) {
+				$output->writeln('The restore had started: the instance may be partly restored, and maintenance mode stays on so that nobody uses it like that.');
+				$output->writeln('Fix the cause, then switch maintenance mode off and run the same command again with --force, in one line:');
+				$again = 'occ backup:restore:full ' . escapeshellarg((string)$input->getArgument('target')) . ' ' . escapeshellarg((string)$input->getArgument('snapshot'))
+					. ($input->getOption('skip-missing-tables') ? ' --skip-missing-tables' : '') . ' --force --no-interaction';
+				$output->writeln('  occ maintenance:mode --off && ' . $again);
+				$output->writeln('(Without --install-apps, so the instance is open only for the seconds until the restore starts again. Apps the first run could not install are listed at the end: install those afterwards.)');
+			} else {
+				$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
+			}
 			$output->writeln('If occ then says that Nextcloud needs an upgrade (the database is already the old server\'s), run occ upgrade first.');
 			return 1;
 		}
@@ -121,6 +131,9 @@ HELP);
 		if (!$result['dbConsistent']) {
 			$output->writeln('<comment>Warning: this snapshot\'s database dump was made over several steps (web cron), so it is not one consistent point in time. Check the instance carefully; prefer a snapshot made with system cron.</comment>');
 		}
+		if ($result['maintenanceLeftOn']) {
+			$output->writeln('<comment>Maintenance mode was on before the restore and stays on: switch it off with occ maintenance:mode --off when the instance is ready.</comment>');
+		}
 		$output->writeln('Next: check trusted_domains and overwritehost/overwriteprotocol match this server, then occ maintenance:repair.');
 		return 0;
 	}

lib/Db/DbDumper.php

@@ -216,7 +216,8 @@ final class DbDumper {
 				}
 				$report['restored'][] = $table;
 			}
-			$this->restoreSequences($manifest['sequences'] ?? [], $report['missing']);
+			// unchanged tables too: a restore that stopped earlier may have left their counters at 1
+			$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
 			$this->resumeForeignKeys($foreignKeys);
 			$this->db->commit();
 			// Foreign keys of the tables created above, now their rows are in. Not inside the
@@ -573,16 +574,21 @@ final class DbDumper {
 		return $name;
 	}
 
-	/** @param list<string> $leftOut tables not restored (--skip-missing-tables): they do not exist here */
-	private function restoreSequences(array $sequences, array $leftOut = []): void {
+	/**
+	 * @param list<string> $leftOut tables not restored (--skip-missing-tables): they do not exist here
+	 * @param list<string> $restored tables restored or unchanged: their other counters go after their data
+	 */
+	private function restoreSequences(array $sequences, array $leftOut = [], array $restored = []): void {
 		if (!$this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\PostgreSQLPlatform) {
 			return;
 		}
 		$leftOut = array_flip($leftOut);
+		$set = [];
 		foreach ($sequences as $info) {
 			if (isset($leftOut[$info['table']])) {
 				continue;
 			}
+			$set[$info['table'] . '.' . $info['column']] = true;
 			$seq = $this->db->fetchOne('SELECT pg_get_serial_sequence(quote_ident(?), ?)', [$info['table'], $info['column']]);
 			if ($seq === false || $seq === null) {
 				continue;
@@ -597,5 +603,26 @@ final class DbDumper {
 					[\Doctrine\DBAL\ParameterType::STRING, \Doctrine\DBAL\ParameterType::INTEGER, \Doctrine\DBAL\ParameterType::BOOLEAN]);
 			}
 		}
+		// A backup from MySQL/MariaDB or SQLite records no counters (those derive them from the
+		// data): every other counter of a restored table goes after its data, or new rows (a new
+		// file, an activity) would collide with restored ones.
+		$restored = array_flip($restored);
+		$rows = $this->db->fetchAllAssociative(
+			"SELECT c.table_name, c.column_name, pg_get_serial_sequence(quote_ident(c.table_name), c.column_name) AS seq
+			 FROM information_schema.columns c
+			 WHERE c.table_schema = current_schema() AND pg_get_serial_sequence(quote_ident(c.table_name), c.column_name) IS NOT NULL");
+		foreach ($rows as $r) {
+			if (!isset($restored[$r['table_name']]) || isset($set[$r['table_name'] . '.' . $r['column_name']])) {
+				continue;
+			}
+			$max = (int)$this->db->fetchOne(sprintf('SELECT COALESCE(MAX(%s), 0) FROM %s',
+				$this->db->quoteIdentifier($r['column_name']), $this->db->quoteIdentifier($r['table_name'])));
+			// never backwards: a counter already past the data stays where it is
+			$state = $this->db->fetchAssociative('SELECT last_value, is_called FROM ' . self::sequenceIdentifier($r['seq']));
+			$next = $state === false ? 1 : (int)$state['last_value'] + ($state['is_called'] ? 1 : 0);
+			if ($next <= $max) {
+				$this->db->executeQuery('SELECT setval(CAST(? AS regclass), ?, false)', [$r['seq'], $max + 1]);
+			}
+		}
 	}
 }

lib/Repository/BlobIndex.php

@@ -7,6 +7,7 @@ declare(strict_types=1);
 
 namespace OCA\NgBackup\Repository;
 
+use OCA\NgBackup\Backend\BackendException;
 use OCA\NgBackup\Backend\IBackend;
 use OCA\NgBackup\Crypto\StreamCipher;
 
@@ -32,6 +33,8 @@ final class BlobIndex {
 	private const CACHE_LOT = 256;
 	/** @var int index files downloaded by the last load() (for tests and diagnostics) */
 	public int $downloaded = 0;
+	/** @var array<string, true> packs whose index file is loaded (refresh() fetches only others) */
+	private array $loaded = [];
 
 	public function __construct(
 		private IBackend $backend,
@@ -90,13 +93,14 @@ final class BlobIndex {
 	 * longer exist on the backend are dropped from the cache.
 	 */
 	public function load(bool $verify = false, ?callable $progress = null): void {
-		$this->blobs = [];
+		$hadIndex = $this->blobs !== [];
 		$cached = $verify ? [] : ($this->cache?->all($this->repositoryId) ?? []);
 		$present = [];
 		$this->downloaded = 0;
 		$order = []; // pack ids in listing order: where a blob is in two packs, the last listed wins
 		$fetch = [];
-		foreach ($this->backend->listFiles('index') as $path) {
+		$known = array_keys($this->loaded + $cached + ($verify ? ($this->cache?->all($this->repositoryId) ?? []) : []));
+		foreach ($this->listIndex($known) as $path) {
 			$packId = basename($path);
 			$present[$packId] = true;
 			$order[] = $packId;
@@ -104,6 +108,9 @@ final class BlobIndex {
 				$fetch[] = $path;
 			}
 		}
+		// only now: an empty listing that listIndex() refused leaves the index as it was
+		$this->blobs = [];
+		$this->loaded = $present;
 		// The rest is downloaded several at a time where the location allows it: on a large
 		// repository these are tens of thousands of small files. getMany() keeps their order, so
 		// the cached packs listed before each one are added first.
@@ -185,12 +192,80 @@ final class BlobIndex {
 		rewind($stream);
 		$this->backend->put('index/' . $packId, $stream);
 		$this->cachePut($packId, $entries);
+		$this->loaded[$packId] = true;
 		foreach ($entries as $e) {
 			$this->blobs[$e['id']] = [$packId, $e['offset'], $e['length'], $e['raw'], $e['flags']];
 			unset($this->pending[$e['id']]);
 		}
 	}
 
+	/**
+	 * Add the index files that appeared on the location since load(), such as those of packs
+	 * another server's cleanup wrote while a restore runs. Only adds and updates: a blob is
+	 * never dropped, as a listing that misses files (a connection that failed) must not make a
+	 * restore lose blobs it knows. A blob in a new pack takes that pack's place.
+	 */
+	public function refresh(): void {
+		$fetch = [];
+		$listed = 0;
+		foreach ($this->listIndex(array_keys($this->loaded)) as $path) {
+			$listed++;
+			if (!isset($this->loaded[basename($path)])) {
+				$fetch[] = $path;
+			}
+		}
+		$toCache = [];
+		foreach ($this->backend->getMany($fetch) as $path => $data) {
+			$packId = basename($path);
+			/** @var list<array{id:string, offset:int, length:int, raw:int, flags:int}> $entries authenticated: written by NG Backup */
+			$entries = json_decode($this->cipher->decryptString($data, 'index:' . $packId), true, 512, JSON_THROW_ON_ERROR);
+			$this->downloaded++;
+			$this->add($packId, $entries);
+			$this->loaded[$packId] = true;
+			$toCache[$packId] = $entries;
+		}
+		if ($toCache !== []) {
+			$this->cache?->putMany($this->repositoryId, $toCache);
+		}
+	}
+
+	/**
+	 * The index files on the location. An empty listing while index files are known ($known,
+	 * pack ids) is usually a connection that failed quietly; taken as is, it would empty the
+	 * index and the cache, make a check report every blob missing, or make a cleanup wipe the
+	 * cache. So the connection is checked (the repository's config must be there) and the folder
+	 * listed once more. Empty again, three of the known index files are looked up one by one:
+	 * only when they are gone too is the repository taken as empty (another server removed its
+	 * last snapshot and cleaned up). Even when that is wrong, nothing on the location is lost:
+	 * only this index and the cache are empty until the next load.
+	 *
+	 * @param list<string> $known
+	 * @return list<string>
+	 * @throws BackendException when the location does not answer as it should
+	 */
+	private function listIndex(array $known): array {
+		$paths = $this->backend->listFiles('index');
+		if ($paths !== [] || $known === []) {
+			return $paths;
+		}
+		if (!$this->backend->exists('config')) {
+			throw new BackendException('The location listed no index files and has no repository config: not trusting that listing');
+		}
+		$paths = $this->backend->listFiles('index');
+		if ($paths !== []) {
+			return $paths;
+		}
+		// spread over the list: first, middle and last (another server's cleanup removes packs
+		// here and there, not in this order)
+		$n = count($known);
+		foreach (array_unique([$known[0], $known[intdiv($n, 2)], $known[$n - 1]]) as $packId) {
+			if ($this->backend->exists('index/' . $packId)) {
+				throw new BackendException('The location listed no index files, while index/' . $packId . ' is there: not trusting that listing');
+			}
+		}
+		return [];
+	}
+
 	/** @return array<string, list<array{id:string, offset:int, length:int}>> packId => blobs currently located in that pack */
 	public function packs(): array {
 		$packs = [];
@@ -204,6 +279,7 @@ final class BlobIndex {
 	public function removePack(string $packId): void {
 		$this->backend->delete('index/' . $packId);
 		$this->cacheRemove([$packId]);
+		unset($this->loaded[$packId]);
 		foreach ($this->blobs as $id => $loc) {
 			if ($loc[0] === $packId) {
 				unset($this->blobs[$id]);

lib/Repository/Repository.php

@@ -72,6 +72,14 @@ final class Repository {
 		if ($backend->exists('config')) {
 			throw new RepositoryException('A repository already exists at this location');
 		}
+		// No config, but backup data: the config is missing (or could not be seen just now, a
+		// connection that failed). A new repository here would write a config over the one that
+		// opens this data; refuse instead.
+		foreach (['index', 'snapshots', 'trash/snapshots', 'catalog'] as $dir) {
+			if ($backend->listFiles($dir) !== []) {
+				throw new RepositoryException("This location holds backup data ($dir/) but no repository config could be read: not creating a new repository over it. Check the connection and the folder; a missing config can be restored from the recovery kit.");
+			}
+		}
 		$id = bin2hex(random_bytes(16));
 		$config = ['format' => self::FORMAT, 'id' => $id, 'created' => gmdate('c'), 'key' => $wrappedKey,
 			'check' => base64_encode((new StreamCipher($keys))->encryptString($id, 'config-check'))];
@@ -841,9 +849,9 @@ final class Repository {
 					$done++;
 				}
 			}
-		} catch (BackendException $e) {
-			// A pack moved (see loadBlob()): the rest one blob at a time, from the fresh index.
-			$this->refreshIndex($e);
+		} catch (BackendException) {
+			// A pack moved, or the read failed for a while (see loadBlob()): the rest one blob at a
+			// time, each read once more after a failure.
 			foreach (array_slice($ids, $done) as $id) {
 				yield $this->loadBlob($id);
 			}
@@ -872,27 +880,25 @@ final class Repository {
 			$encrypted = $this->packReader->read($pack, $offset, $length);
 		} catch (BackendException $e) {
 			// Another server's cleanup may have repacked this blob into a new pack since the index
-			// was loaded (a disaster recovery test while the original server still runs). Reload
-			// the index from the location; read again only if the blob has moved.
-			$this->refreshIndex($e);
+			// was loaded (a disaster recovery test while the original server still runs): add the
+			// new index files, and read the blob where it is now. Or the read failed for a while
+			// (a connection that died; the backend has replaced it by now): then read it once
+			// more where it was. A second failure is the real one.
+			try {
+				$this->index->refresh();
+			} catch (\Throwable) {
+				// the location could not be listed either: read again where the blob was
+			}
 			[$moved, $offset, $length] = $this->index->get($id);
-			if ($moved === $pack) {
+			try {
+				$encrypted = $this->packReader->read($moved, $offset, $length);
+			} catch (BackendException) {
 				throw $e;
 			}
-			$encrypted = $this->packReader->read($moved, $offset, $length);
 		}
 		return $this->decodeBlob($id, $encrypted);
 	}
 
-	/** Reload the index from the location after a failed pack read; rethrows that failure if it cannot. */
-	private function refreshIndex(BackendException $cause): void {
-		try {
-			$this->index->load();
-		} catch (\Throwable) {
-			throw $cause;
-		}
-	}
-
 	/** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */
 	private function decodeBlob(string $id, string $encrypted): string {
 		[, , , $raw, $flags] = $this->index->get($id);

lib/Service/RestoreFullService.php

@@ -27,9 +27,14 @@ final class RestoreFullService {
 	 * Config keys that must keep pointing at THIS installation's actual, already-working
 	 * database and data directory (set by occ maintenance:install), never at the old server's.
 	 */
+	/** App setting: when a full restore started that has not completed (see restoreFull()). */
+	private const INTERRUPTED_KEY = 'restore_full_started';
+
 	private const CONNECTION_KEYS = [
 		'dbtype', 'dbhost', 'dbport', 'dbname', 'dbuser', 'dbpassword', 'dbpersistent',
 		'dbtableprefix', 'dbdriveroptions', 'datadirectory',
+		// set by restoreFull() itself for the length of the restore
+		'maintenance',
 	];
 
 	public function __construct(
@@ -85,12 +90,50 @@ final class RestoreFullService {
 		];
 	}
 
-	/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>} */
+	/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>, maintenanceLeftOn:bool} */
 	public function restoreFull(Target $target, string $snapshotId, bool $skipMissingTables = false): array {
-		$check = $this->checkCompatibility($target, $snapshotId, $skipMissingTables);
+		// Maintenance mode for the whole restore: background jobs (cron.php stops in maintenance
+		// mode), page visits (AJAX cron) and users must not write into tables or the data
+		// directory while they are being replaced; a row written in between would collide with a
+		// restored one or be silently lost. Switched off after a restore that completed. After an
+		// error it stays on: the instance may be partly restored, and must not be used like that.
+		// occ only offers app commands outside maintenance mode, so the admin switches it off
+		// right before running the restore again (see RestoreFull). Already on before: left on.
+		// On before the checks below, so that a run again after an error leaves the partly
+		// restored instance open for seconds at most; off again when they refuse (nothing changed).
+		// A marker in ng_backup's own app settings (which the restore keeps) says a restore started
+		// and did not complete: then the instance may be partly restored, and a check that refuses
+		// a run again must not open it.
+		$appConfig = \OCP\Server::get(\OCP\IAppConfig::class);
+		$interrupted = $appConfig->getValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, 0) > 0;
+		$wasInMaintenance = $this->config->getSystemValueBool('maintenance', false);
+		$this->config->setSystemValue('maintenance', true);
+		$reopen = function () use ($wasInMaintenance, $interrupted): void {
+			if (!$wasInMaintenance && !$interrupted) {
+				$this->config->setSystemValue('maintenance', false);
+			}
+		};
+		try {
+			$check = $this->checkCompatibility($target, $snapshotId, $skipMissingTables);
+		} catch (\Throwable $e) {
+			$reopen();
+			throw $e;
+		}
 		if (!$check['ok']) {
+			$reopen();
 			throw new \RuntimeException($this->describeBlockers($check));
 		}
+		$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
+		$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check);
+		$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
+		if (!$wasInMaintenance) {
+			$this->config->setSystemValue('maintenance', false);
+		}
+		return $result + ['maintenanceLeftOn' => $wasInMaintenance];
+	}
+
+	/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>} */
+	private function restoreInMaintenance(Target $target, string $snapshotId, bool $skipMissingTables, array $check): array {
 		return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $skipMissingTables, $check) {
 			$beat = LeaseService::throttled($refresh);
 			$repo = $this->targets->repository($target);

lib/Service/TargetService.php

@@ -228,7 +228,8 @@ final class TargetService {
 				$s->setFiles((int)($meta['stats']['files'] ?? 0));
 				$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
 				$this->snapshots->insert($s);
-			} catch (\Throwable) {
+			} catch (\Throwable $e) {
+				Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]);
 			}
 		}
 	}

tests/Unit/Backend/SftpRangeFetcherTest.php

@@ -184,6 +184,38 @@ class SftpRangeFetcherTest extends TestCase {
 		$this->assertSame(array_fill(0, 10, null), $result);
 	}
 
+	public function testABrokenConnectionIsReplacedAndTheRestIsReadAhead(): void {
+		$connections = $this->connections(1, [['loseAt' => 1048576]]);
+		$made = [];
+		$connect = function () use (&$made): object {
+			$made[] = $c = new FakeSftpConnection($this->files);
+			return SftpReadStream::connection(new FakeSftpStorage($c));
+		};
+		$ranges = $this->ranges();
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir(), $connect));
+
+		$expected = $this->expected($ranges);
+		$expected[0] = null; // the range it was reading when it broke: read the usual way
+		$this->assertSame($expected, $result);
+		$this->assertCount(1, $made);
+		$this->assertTrue($made[0]->disconnected, 'the new one ends with the fetcher too');
+	}
+
+	public function testConnectionsThatKeepBreakingAreReplacedOnlySoOften(): void {
+		$made = 0;
+		$connect = function () use (&$made): object {
+			$made++;
+			return SftpReadStream::connection(new FakeSftpStorage(new FakeSftpConnection($this->files, loseAt: 0)));
+		};
+		$ranges = array_merge($this->ranges(), $this->ranges(), $this->ranges());
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($this->connections(1, [['loseAt' => 0]])), '/home/', $ranges, $this->tempDir(), $connect));
+
+		$this->assertSame(array_fill(0, count($ranges), null), $result);
+		$this->assertSame(SftpRangeFetcher::MAX_RECONNECTS, $made);
+	}
+
 	public function testStoppingHalfwayEndsTheConnectionsAndLeavesNothingBehind(): void {
 		$connections = $this->connections(2);
 		$spool = $this->tempDir();

tests/Unit/Backend/StorageBackendReconnectTest.php

@@ -77,6 +77,24 @@ class StorageBackendReconnectTest extends TestCase {
 		$this->assertSame(1, $backend->reconnects);
 	}
 
+	public function testASilentlyDroppedConnectionIsReplacedAfterAPause(): void {
+		$backend = $this->backend();
+		$this->connections[0]->dropped = true;
+		$this->connections[0]->silent = true; // phpseclib still says connected
+		self::idle($backend);
+		$this->assertTrue($backend->exists('config'));
+		$this->assertSame(1, $backend->reconnects, 'the check failed, so the connection was replaced');
+	}
+
+	public function testAFailureOnASilentlyDroppedConnectionIsRepeated(): void {
+		$backend = $this->backend();
+		$this->connections[0]->dropped = true;
+		$this->connections[0]->silent = true;
+		$fh = $backend->get('config');
+		$this->assertSame('config', stream_get_contents($fh));
+		$this->assertSame(1, $backend->reconnects);
+	}
+
 	public function testAMissingFileOnALiveConnectionIsJustMissing(): void {
 		$backend = $this->backend();
 		$this->assertFalse($backend->exists('nothing'));
@@ -121,6 +139,8 @@ interface SftpLikeStorage extends IStorage {
 /** The public phpseclib calls the backend uses to tell a dead connection. */
 final class ReconnectFakeSftp {
 	public bool $dropped = false;
+	/** Dropped without phpseclib noticing (as a Hetzner Storage Box does): it still says connected. */
+	public bool $silent = false;
 	private bool $connected = true;
 	public int $requests = 0;
 
@@ -139,6 +159,9 @@ final class ReconnectFakeSftp {
 	public function request(): bool {
 		$this->requests++;
 		if ($this->dropped) {
+			if ($this->silent) {
+				return false;
+			}
 			$this->connected = false;
 		}
 		return $this->connected;

tests/Unit/Repository/IndexListingTest.php

@@ -0,0 +1,239 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Repository;
+
+use OCA\NgBackup\Backend\BackendException;
+use OCA\NgBackup\Backend\IBackend;
+use OCA\NgBackup\Backend\LocalBackend;
+use OCA\NgBackup\Crypto\KeyRing;
+use OCA\NgBackup\Repository\IIndexCache;
+use OCA\NgBackup\Repository\Repository;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use PHPUnit\Framework\TestCase;
+
+/**
+ * A listing of the location that comes back empty because a connection failed quietly (an SFTP
+ * connection a Storage Box dropped) must not empty the index. In a 1 TB disaster recovery test a
+ * pack read failed after 13 hours, the reload of the index got an empty listing, and the restore
+ * stopped with "Blob not in index".
+ */
+class IndexListingTest extends TestCase {
+	use TempDirTrait;
+
+	private string $root;
+	private KeyRing $keys;
+	private string $src;
+	private string $snapshot;
+
+	protected function setUp(): void {
+		$this->root = $this->tempDir();
+		$this->keys = KeyRing::generate();
+		$this->src = $this->tempDir();
+		for ($i = 0; $i < 12; $i++) {
+			file_put_contents("$this->src/f$i", random_bytes(3 * 1048576));
+		}
+		$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
+		$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
+	}
+
+	private static function cache(): IIndexCache {
+		return indexCache();
+	}
+
+	public function testARestoreGoesOnAfterAPackReadFailedAndTheListingCameBackEmpty(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$backend->failPackReads = 1;      // the read of a pack fails once
+		$backend->emptyIndexListings = 1; // and listing the index right after comes back empty
+		$target = $this->tempDir();
+
+		$this->assertSame(12, $repo->restore($this->snapshot, $target));
+
+		$this->assertSame(self::hashTree($this->src), self::hashTree($target));
+		$this->assertSame(0, $backend->failPackReads, 'the failure happened');
+		$this->assertSame(0, $backend->emptyIndexListings, 'the empty listing was seen');
+	}
+
+	public function testAPackThatStaysUnreadableIsStillReportedAsSuch(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$backend->failPackReads = 1000;
+
+		try {
+			$repo->restore($this->snapshot, $this->tempDir());
+			$this->fail('the packs cannot be read');
+		} catch (BackendException $e) {
+			$this->assertStringStartsWith('Not found: packs/', $e->getMessage());
+		}
+	}
+
+	public function testACheckWithAnEmptyListingIsAnErrorNotEveryBlobMissing(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$backend->dead = true;
+
+		$this->expectException(BackendException::class);
+		$this->expectExceptionMessage('listed no index files');
+		$repo->verify($this->snapshot);
+	}
+
+	public function testRefreshingFetchesOnlyIndexFilesItHasNotLoaded(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$index = (new \ReflectionProperty($repo, 'index'))->getValue($repo);
+		$index->downloaded = 0;
+
+		$index->refresh();
+
+		$this->assertSame(0, $index->downloaded, 'every index file is loaded already, also those whose blobs are elsewhere too');
+	}
+
+	public function testARestoreGoesOnWhenTheLocationCannotBeListedAfterAFailedRead(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$backend->failPackReads = 1;
+		$backend->dead = true; // listing and config look gone; reading packs works again
+		$target = $this->tempDir();
+
+		$this->assertSame(12, $repo->restore($this->snapshot, $target));
+
+		$this->assertSame(self::hashTree($this->src), self::hashTree($target));
+	}
+
+	public function testAnEmptyListingThatWasAHiccupIsListedAgain(): void {
+		$cache = self::cache();
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		Repository::openWithKey($backend, $this->keys, $cache);
+		$cached = $cache->data;
+		$backend->emptyIndexListings = 1;
+
+		$again = Repository::openWithKey($backend, $this->keys, $cache);
+
+		$this->assertSame($cached, $cache->data);
+		$this->assertSame(12, $again->restore($this->snapshot, $this->tempDir()));
+	}
+
+	public function testAListingThatStaysEmptyWhileIndexFilesAreThereIsNotTaken(): void {
+		$cache = self::cache();
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		Repository::openWithKey($backend, $this->keys, $cache);
+		$cached = $cache->data;
+		$backend->emptyIndexListings = 5; // the config answers, the listing keeps lying
+
+		try {
+			Repository::openWithKey($backend, $this->keys, $cache);
+			$this->fail('index files are there');
+		} catch (BackendException $e) {
+			$this->assertStringContainsString('is there', $e->getMessage());
+		}
+		$this->assertSame($cached, $cache->data);
+	}
+
+	public function testARepositoryThatReallyBecameEmptyIsTakenAsEmpty(): void {
+		$cache = self::cache();
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		Repository::openWithKey($backend, $this->keys, $cache);
+		$this->assertNotSame([], array_filter($cache->data));
+		// another server forgot the last snapshot and cleaned up: no packs and index files left
+		foreach (array_merge($backend->list('index'), $backend->list('packs')) as $path) {
+			$backend->delete($path);
+		}
+
+		Repository::openWithKey($backend, $this->keys, $cache);
+
+		$this->assertSame([], array_filter($cache->data), 'the stale cache rows are gone');
+	}
+
+	public function testOpeningWithAnEmptyListingKeepsTheCachedIndex(): void {
+		$cache = self::cache();
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		Repository::openWithKey($backend, $this->keys, $cache);
+		$cached = $cache->data;
+		$this->assertNotSame([], $cached);
+
+		$backend->dead = true;
+		try {
+			Repository::openWithKey($backend, $this->keys, $cache);
+			$this->fail('an empty listing of a repository with an index must not be taken as is');
+		} catch (BackendException $e) {
+			$this->assertStringContainsString('listed no index files', $e->getMessage());
+		}
+		$this->assertSame($cached, $cache->data, 'the cache is untouched');
+	}
+}
+
+/** @internal */
+function indexCache(): IIndexCache {
+	return new class implements IIndexCache {
+		public array $data = [];
+		public function all(string $r): array { return $this->data[$r] ?? []; }
+		public function put(string $r, string $p, array $e): void { $this->data[$r][$p] = $e; }
+		public function remove(string $r, array $ps): void { foreach ($ps as $p) { unset($this->data[$r][$p]); } }
+		public function putMany(string $r, array $packs): void { foreach ($packs as $p => $e) { $this->put($r, (string)$p, $e); } }
+		public function replaceAll(string $r, array $packs): void { $this->data[$r] = []; $this->putMany($r, $packs); }
+	};
+}
+
+/** A location whose pack reads fail, and whose index listings come back empty, a set number of times. */
+final class FlakyBackend implements IBackend {
+	use \OCA\NgBackup\Backend\GetsOneByOne;
+
+	public int $failPackReads = 0;
+	public int $emptyIndexListings = 0;
+	/** The connection is gone: also the repository config looks missing. */
+	public bool $dead = false;
+
+	public function __construct(
+		private IBackend $inner,
+	) {
+	}
+
+	public function put(string $path, $stream): void {
+		$this->inner->put($path, $stream);
+	}
+
+	public function get(string $path) {
+		if (str_starts_with($path, 'packs/') && $this->failPackReads > 0) {
+			$this->failPackReads--;
+			throw new BackendException('Not found: ' . $path);
+		}
+		return $this->inner->get($path);
+	}
+
+	public function getRange(string $path, int $offset, int $length): string {
+		if (str_starts_with($path, 'packs/') && $this->failPackReads > 0) {
+			$this->failPackReads--;
+			throw new BackendException('Not found: ' . $path);
+		}
+		return $this->inner->getRange($path, $offset, $length);
+	}
+
+	public function exists(string $path): bool {
+		return !$this->dead && $this->inner->exists($path);
+	}
+
+	public function move(string $from, string $to): void {
+		$this->inner->move($from, $to);
+	}
+
+	public function list(string $prefix): array {
+		return $this->inner->list($prefix);
+	}
+
+	public function listFiles(string $dir): array {
+		if ($dir === 'index' && $this->dead) {
+			return [];
+		}
+		if ($dir === 'index' && $this->emptyIndexListings > 0) {
+			$this->emptyIndexListings--;
+			return [];
+		}
+		return $this->inner->listFiles($dir);
+	}
+
+	public function delete(string $path): void {
+		$this->inner->delete($path);
+	}
+}

tests/Unit/Repository/RepositoryTest.php

@@ -99,6 +99,20 @@ class RepositoryTest extends TestCase {
 		Repository::init($this->backend, 'pw');
 	}
 
+	public function testInitRefusesALocationWithBackupDataButNoConfig(): void {
+		$repo = Repository::init($this->backend, 'pw');
+		$repo->backupDirectory($this->src);
+		$this->backend->delete('config'); // missing, or not seen for a moment
+
+		try {
+			Repository::init($this->backend, 'pw');
+			$this->fail('a new repository over existing backup data');
+		} catch (RepositoryException $e) {
+			$this->assertStringContainsString('holds backup data', $e->getMessage());
+		}
+		$this->assertFalse($this->backend->exists('config'), 'no new config written');
+	}
+
 	public function testTamperedPackIsDetected(): void {
 		$repo = Repository::init($this->backend, 'pw');
 		$r = $repo->backupDirectory($this->src);

tests/integration/full-restore-roundtrip.sh

@@ -2,10 +2,13 @@
 # Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a
 # SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup
 # location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7.
-# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp]
+# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp|to-pgsql|to-mysql|to-sqlite]
 #   e.g. full-restore-roundtrip.sh 35-apache pgsql
 # With "sftp" the location is an SFTP server instead of a shared directory, and the origin has a
 # file spread over several packs: the restore reads them ahead on connections of its own.
+# With "to-<db>" the restore instance runs that database whatever the origin runs (a MySQL or
+# SQLite backup records no ID counters; after the restore new rows must not collide with restored
+# ones, whatever the database on either side).
 # With "legacy" the origin runs NG Backup 1.0.0-rc.1 (0.9.7), whose backups do not record the
 # structure of their tables: the restore must then leave the tables of the disabled app out
 # with --skip-missing-tables, as for a real backup made before 1.0.0-rc.2.
@@ -127,6 +130,8 @@ ok "test data backed up, snapshot $SNAP"
 
 docker cp ngb-frt-origin:/tmp/kit.json /tmp/ngb-frt-kit.json
 docker exec ngb-frt-origin sh -c "cat /tmp/pass" > /tmp/ngb-frt-pass
+ORIGIN_DBTYPE=$DB
+case "$MODE" in to-*) DB=${MODE#to-} ;; esac
 RESTORE_DB=$(db_env restore)
 # shellcheck disable=SC2086
 docker run -d --name ngb-frt-restore --network "$NET" $RESTORE_DB \
@@ -141,8 +146,14 @@ docker cp /tmp/ngb-frt-pass ngb-frt-restore:/tmp/pass
 ok "fresh restore instance up"
 
 run_on ngb-frt-restore backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass >/dev/null || fail "key import"
-run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo >/dev/null || fail "target add on the restore instance"
-run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location"
+ADD_OUT=$(run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo 2>&1) || { printf '%s\n' "$ADD_OUT"; fail "target add on the restore instance"; }
+LIST_OUT=$(run_on ngb-frt-restore backup:list offsite 2>&1)
+printf '%s\n' "$LIST_OUT" | grep -q "$SNAP" || {
+	printf 'target:add said:\n%s\nbackup:list said:\n%s\n' "$ADD_OUT" "$LIST_OUT"
+	docker exec ngb-frt-restore sh -c 'grep -h ng_backup /var/www/html/data/nextcloud.log | tail -5' || true
+	docker exec ngb-frt-restore sh -c 'ls -la /mnt/shared/repo /mnt/shared/repo/snapshots' || true
+	fail "the origin's snapshot is not listed after connecting the location"
+}
 ok "recovery kit imported, existing repository adopted, snapshot visible"
 
 RESTORE_OPTS=(--no-interaction --install-apps)
@@ -161,7 +172,10 @@ OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS
 	docker exec ngb-frt-restore sh -c 'tail -c 6000 /var/www/html/data/nextcloud.log' || true
 	fail "restore:full"
 }
-ok "backup:restore:full completed"
+printf '%s\n' "$OUT" | grep -q "maintenance mode while the restore runs" || fail "the restore did not say it runs in maintenance mode"
+run_on ngb-frt-restore status 2>&1 | grep -q "maintenance: false" || fail "maintenance mode is still on after the restore"
+run_on ngb-frt-restore config:app:get ng_backup restore_full_started >/dev/null 2>&1 && fail "the restore left its 'started' marker behind"
+ok "backup:restore:full completed, in maintenance mode and out of it again"
 
 # Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
 # earlier bugs here only showed up after config.php's secret was rewritten mid-restore.
@@ -170,13 +184,23 @@ run_on ngb-frt-restore user:list | grep -q frtest || fail "the backed-up user is
 docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
 [ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
 run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
+# New rows after the restore: a new file (file cache) and a new user must not collide with
+# restored rows (ID counters after the data).
+docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"
+run_on ngb-frt-restore files:scan frtest >/dev/null || fail "a new file cannot be added after the restore"
+if [ "$DB" != sqlite ]; then
+	[ "$(sql restore "select count(*) from oc_filecache where path = 'files/Docs/after-restore.txt'")" = 1 ] \
+		|| fail "a new file did not make it into the file cache after the restore (ID counters?)"
+fi
+docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-restore php occ user:add --password-from-env frnew >/dev/null || fail "a new user cannot be added after the restore"
 ok "database, files and ng_backup's own state are all intact and usable after the restore"
 
-if [ "$DB" != sqlite ] && [ "$LEGACY" = legacy ]; then
+# Mail and Deck were only set up on an origin with a database server (see above)
+if [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ] && [ "$LEGACY" = legacy ]; then
 	[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 0 ] || fail "a table of unknown structure was created"
 	[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
 	ok "older backup: tables of the disabled app left out, Mail restored"
-elif [ "$DB" != sqlite ]; then
+elif [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ]; then
 	[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created"
 	[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
 	[ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \
@@ -197,4 +221,4 @@ fi
 run_on ngb-frt-restore backup:run offsite -l post-restore-check >/dev/null || fail "a new backup after the restore fails"
 ok "the restored instance can run a new backup of its own"
 
-echo "FULL RESTORE ROUNDTRIP PASS ($DB${MODE:+, $MODE})"
+echo "FULL RESTORE ROUNDTRIP PASS ($ORIGIN_DBTYPE${MODE:+, $MODE})"