Merge pull request #18 from dillardblom/rc5

Release 1.0.0-rc.5
Full restore: a run again after an interrupted restore keeps maintenance mode on when its check refuses; retry text about apps
2026-10-22 02:28:05 +02:00 · 2026-10-22 01:59:01 +02:00 · 2026-10-22 01:03:52 +02:00 · 2026-10-22 00:53:08 +02:00 · 2026-10-22 00:36:58 +02:00 · 2026-10-22 00:31:34 +02:00
27 changed files with 2098 additions and 53 deletions
@@ -108,3 +108,19 @@ jobs:
        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql
      - name: Full restore round trip on MariaDB, with Mail (foreign keys) and a disabled app
        run: tests/integration/full-restore-roundtrip.sh 35-apache mysql
+      - name: Full restore of a backup made by 1.0.0-rc.1 (no table structure) on PostgreSQL
+        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql legacy
+      - name: Full restore of a backup made by 1.0.0-rc.1 (no table structure) on MariaDB
+        run: tests/integration/full-restore-roundtrip.sh 35-apache mysql legacy
+      - name: Full restore over SFTP on PostgreSQL, reading packs ahead on several connections
+        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql sftp
+      - name: Full restore of a MariaDB backup onto PostgreSQL (ID counters after the data)
+        run: tests/integration/full-restore-roundtrip.sh 35-apache mysql to-pgsql
+      - name: Full restore of a PostgreSQL backup onto MariaDB
+        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql to-mysql
+      - name: Full restore of a PostgreSQL backup onto SQLite
+        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql to-sqlite
+      - name: Full restore of a MariaDB backup onto SQLite
+        run: tests/integration/full-restore-roundtrip.sh 35-apache mysql to-sqlite
+      - name: Full restore of a SQLite backup onto PostgreSQL
+        run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite to-pgsql
@@ -7,7 +7,62 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
 beta/daily update channel, which would hide the app from most admins. The stability stage is
 instead carried by the app description, the git tag, the release, and this file.

-## [Unreleased]
+## [1.0.0-rc.5] - 2026-10-22
+
+Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
+stopped after 13 hours with "Blob not in index"; the backup itself was fine.
+
+### Changed
+- README and app store description: NG Backup needs background jobs on system cron, with how to
+  set that up (issues #13, #17).
+
+### Fixed
+- A full restore no longer stops with "Blob not in index" after a pack could not be read for a
+  moment. Reading the index again then got an empty listing (from a connection the location had
+  dropped) and emptied the index. Reading the index again now only adds, an empty listing of a
+  location that has an index is an error instead of an empty index, and a pack that could not be
+  read is read once more before the restore gives up.
+- After a full restore of a MySQL/MariaDB or SQLite backup onto PostgreSQL, new files, users or
+  activities could not be added ("duplicate key", "File entry could not be inserted"): such
+  backups record no ID counters, and they stayed at 1. Every counter of a restored table now goes
+  after its data. Found in a disaster recovery simulation of 138 GB from MariaDB to PostgreSQL,
+  where all 15,772 files and every count were identical.
+- A full restore now runs in maintenance mode: a background job, a page visit or a user writing
+  into the instance meanwhile could collide with restored rows ("duplicate key") or be lost. It
+  is switched off when the restore completes; after an error it stays on, and the message gives
+  the one line to switch it off and run the restore again. Maintenance mode that was on before
+  the restore stays on.
+- Adding a location whose repository config could not be read, while backup data is there,
+  no longer creates a new repository over it: it stops with an explanation. A snapshot that
+  cannot be taken over when a location is added is now logged instead of left out silently.
+- An SFTP connection that a Hetzner Storage Box dropped without phpseclib noticing is now
+  recognised and replaced: the check asks the server and counts its answer.
+- The extra connection a full restore reads ahead on is replaced when it breaks (up to 20 times),
+  instead of reading the rest of the restore the slow way.
+
+## [1.0.0-rc.4] - 2026-10-19
+
+Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not
+released: its fixes are in here.
+
+### Changed
+- A full restore over SFTP reads the packs ahead on a connection of its own, 32 requests in
+  flight, into temporary files in the restore folder. From a Hetzner Storage Box to a server in
+  Canada (122 ms) packs now come in at about 14 MiB/s instead of 2.8. More connections can be set
+  with `restore_sftp_connections` (README). A part that cannot be read this way is read as
+  before. Restores from other locations work as before.
+- App store description: the line about Ensembia.
+
+### Fixed
+- After a full restore stopped halfway, running it again with `--force` was refused when the
+  database was already restored: Nextcloud then first wants `occ upgrade`. The guide and the
+  error message now say so.
+- A full restore of a backup made before 1.0.0-rc.2 with `--skip-missing-tables` failed on
+  PostgreSQL with "relation ... does not exist": the left-out tables were still looked up to set
+  their ID counters. It now skips them. A new test restores a backup made by 1.0.0-rc.1 on
+  PostgreSQL and MariaDB.
+- The app store showed old screenshots: it keeps a copy per URL, and the URLs always pointed at
+  `main`. They now point at the release's tag.

 ## [1.0.0-rc.2] - 2026-10-18

@@ -383,6 +383,23 @@ in README "Known limitations":
  de gebruiker, wachtwoordzin min. 16 tekens). Geen nieuwe functies meer.
 - **Vóór 1.0.0 (verplicht):** volledige restore met kit op een tweede server (PVE02), alleen lezend
  of op een kopie van de locatie; plus een user-restore. Zonder geslaagde test geen 1.0.0.
+- **1.0.0 (besluit 2026-10-18):** alleen betrouwbaarheid. rc2 uitgebracht; slaagt de grote
+  DR-test op VM 980 (snapshot van 12-10, met `--skip-missing-tables`), dan 1.0.0. Zo niet: fix,
+  rc3, en rc3 pas uitbrengen na een geslaagde volledige DR-test.
+- **rc4 (besluit 2026-10-19):** rc3 is niet uitgebracht. De DR-test met rc3 liep ~2,8 MiB/s
+  (~3,5 dag); parallel lezen over SFTP haalde vanaf VM 980 ~7,6 MiB/s. Daarom rc4 = rc3 +
+  vooruitlezen over SFTP (4 eigen verbindingen). Volgorde: volledige DR-test op VM 980 volgens de
+  gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een
+  productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0.
+  Problemen of meldingen van gebruikers: rc5.
+- **1.0.1:**
+  1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
+     restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
+  2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron
+     staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue
+     #13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles.
+  3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later),
+     zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest.
 - **1.1.0:**
  1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
     als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
@@ -33,6 +33,26 @@ Need help restoring a backup after a crash? Ensembia offers paid restore assista
 - Restores go through Nextcloud, so overwritten files keep their old content as a version and
  removed files go to the trash.

+## Requirements
+
+- **Background jobs on system cron.** NG Backup runs backups in steps from Nextcloud's
+  background job, so a backup only moves forward when that job runs. Choosing "Cron" under
+  **Administration settings → Basic settings → Background jobs** is not enough on its own: it
+  tells Nextcloud that the system will call it, so you also need a cron line that does. Without
+  it a backup stays at "running (db)", and so does one started with "Back up now".
+  `occ backup:run` in a terminal works without cron (it runs in the foreground).
+  - Regular server: `crontab -u www-data -e` and add
+    `*/5 * * * * php -f /var/www/nextcloud/cron.php` (your Nextcloud path).
+  - Docker (the official `nextcloud` image): on the host, `crontab -e` as root and add
+    `*/5 * * * * docker exec -u www-data <container> php -f /var/www/html/cron.php`, or add a
+    service with the same image and `entrypoint: /cron.sh` to your `docker-compose.yml`.
+  - Nextcloud All-in-One already runs cron.
+
+  After a few minutes, the Background jobs section shows that the last job ran just now. See the
+  [Nextcloud documentation on background jobs](https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/background_jobs_configuration.html).
+- A storage location: S3 or compatible, SFTP, WebDAV, SMB (needs the PHP `smbclient` extension,
+  see below), FTP or a local path.
+
 ## Quick start

 ```
@@ -181,6 +201,10 @@ Contributions welcome via PR — add what you ran into with your own provider.
  server (or the VM/container host where TCP ends) to BBR:
  `sysctl -w net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr`
  (`modprobe tcp_bbr` first; make it permanent in `/etc/sysctl.d/`).
+  A full restore over SFTP reads the packs ahead on a second connection, with many requests in
+  flight. A server that handles parallel reads well (not a Hetzner Storage Box, which gets slower)
+  can be read on more connections: `occ config:app:set ng_backup restore_sftp_connections
+  --value=4` (at most 8; the server must allow that many plus one at once for the user).

 - **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself
  (e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the
@@ -36,12 +36,13 @@ All-in-One and hosted Nextcloud.
 - One user (account, files, calendars, contacts) via user_migration.
 - The whole server onto a fresh installation (disaster recovery).

+NG Backup needs Nextcloud's background jobs on system cron (a real cron line, not only the setting).
 Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very
 welcome: use the "Share your experience" link on the admin page.

-Developed and sponsored by [Ensembia](https://www.ensembia.com), which also offers paid help with
-restoring a backup after a crash.]]></description>
-    <version>0.9.8</version>
+Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
+you need help with restoring a backup after a crash.]]></description>
+    <version>0.9.11</version>
    <licence>agpl</licence>
    <author mail="dillard.blom@ensembia.com">Dillard Blom</author>
    <namespace>NgBackup</namespace>
@@ -52,9 +53,9 @@ restoring a backup after a crash.]]></description>
    <website>https://github.com/dillardblom/ng_backup</website>
    <bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
    <repository type="git">https://github.com/dillardblom/ng_backup</repository>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/1-backups.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/2-restore.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/3-settings.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/1-backups.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/2-restore.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/3-settings.png</screenshot>
    <dependencies>
        <php min-version="8.1"/>
        <lib>sodium</lib>
@@ -94,7 +94,18 @@ tables that do not exist here, install the apps they belong to and run it again,
 out with `--skip-missing-tables`.

 **8. Let the restore run.** It restores the database, then the data directory, and takes hours for
-a large backup. If it stops halfway, fix the cause and run the same command again with `--force`.
+a large backup. Nextcloud is in maintenance mode meanwhile, so nobody and no background job
+writes into it; the restore switches it off when it completes. (If you had switched maintenance
+mode on yourself before, it stays on: switch it off with `occ maintenance:mode --off` when ready.)
+
+If it stops halfway, maintenance mode stays on (the instance may be partly restored). Fix the cause,
+then switch maintenance mode off and run the restore again with `--force`, in one line (the message
+prints it): `occ maintenance:mode --off && occ backup:restore:full <location> <snapshot> --force`.
+Leave out `--install-apps` this time, so that the instance is open only for the seconds until the
+restore has switched maintenance mode on again; apps the first run could not install are listed at
+the end of the restore, install those afterwards.
+If occ then says that Nextcloud or one of the apps requires an upgrade, the database is already
+the old server's: run `occ upgrade` first, then the same command with `--force`.

 **9. After the restore.** These worked in our tests:

@@ -0,0 +1,30 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Backend;
+
+/**
+ * A backend that can read many byte ranges ahead of use, several at a time over connections of
+ * its own (a full restore, where one connection at a time is bound by the round trip).
+ */
+interface IReadsAhead {
+	/** Whether readAhead() is worth planning for: false when it would return null anyway. */
+	public function canReadAhead(): bool;
+
+	/**
+	 * The ranges in the order given, each as a local stream at offset 0 holding its bytes, read a
+	 * few ranges ahead of the one the caller is at. A stream is closed when the generator is
+	 * resumed. Null for a range that could not be read this way (a missing object, a server or
+	 * connection error): the caller reads it the usual way, which reports why.
+	 *
+	 * @param list<array{0:string, 1:int, 2:int}> $ranges [path, offset, length] each
+	 * @param string $spool an existing local directory for the files that hold the ranges
+	 * @return \Generator<int, resource|null>|null by position in $ranges; null when this backend
+	 *                                             cannot read ahead now
+	 */
+	public function readAhead(array $ranges, string $spool): ?\Generator;
+}
@@ -0,0 +1,395 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Backend;
+
+/**
+ * Reads byte ranges of files on an SFTP location ahead of use, over several connections at once,
+ * into local files (StorageBackend::readAhead(), for a full restore).
+ *
+ * A restore that reads its packs as it goes is bound by the round trip: from a Hetzner Storage
+ * Box to Canada (RTT 122 ms) it reached about 2.8 MiB/s, with the PHP process mostly waiting.
+ * Here every connection reads its own range with DEPTH requests in flight, ahead of the restore. PHP runs one thing at a time: while it waits for an answer on one connection, the
+ * answers of the others arrive in the kernel's buffers, so the connections still transfer at the
+ * same time.
+ *
+ * Uses the same internal phpseclib 2 methods as SftpReadStream (see SftpReadStream::connection()).
+ * The connections are this class's own, never the storage's: no other request goes over them.
+ * A range that fails (missing file, server error, a connection that breaks) comes out as null,
+ * for the caller to read the usual way; a broken connection is not used again.
+ */
+final class SftpRangeFetcher {
+	/**
+	 * Connections a full restore uses, unless the app setting restore_sftp_connections says
+	 * otherwise. One: a Hetzner Storage Box gave about 14 MiB/s on one connection with DEPTH
+	 * requests in flight, and 4-6 MiB/s on 2 or 4 (it reads from disks, and parallel readers make
+	 * them seek). On a line with packet loss to a server that handles parallel reads well, 4
+	 * connections did better (21 against 14 MiB/s).
+	 */
+	public const DEFAULT_CONNECTIONS = 1;
+	public const MAX_CONNECTIONS = 8;
+	/** READ requests in flight per connection. */
+	public const DEPTH = 32;
+	/** Ranges read ahead of the one the caller is at, in flight or waiting in the spool. */
+	public const AHEAD = 8;
+	/**
+	 * Seconds ready() waits for any connection to answer before it waits for each in turn: short,
+	 * in case a connection holds an answer where ready() does not look.
+	 */
+	private const SELECT_TIMEOUT = 1;
+	/** See SftpReadStream::CHUNK and MIN_CHUNK. */
+	private const CHUNK = 261120;
+	private const MIN_CHUNK = 32768;
+
+	private const FXP_OPEN = 3;
+	private const FXP_CLOSE = 4;
+	private const FXP_READ = 5;
+	private const FXP_STATUS = 101;
+	private const FXP_HANDLE = 102;
+	private const FXP_DATA = 103;
+	private const FXF_READ = 1;
+	private const FX_OK = 0;
+
+	/** Connections that broke and are replaced, at most, in one restore. */
+	public const MAX_RECONNECTS = 20;
+
+	private const OPEN = 0;
+	private const READ = 1;
+	private const CLOSE = 2;
+
+	/** @var array<int, int> READ size per connection, lowered when a server answers less than asked */
+	private array $chunks = [];
+	/** @var array<int, SftpRangeJob> the range each busy connection reads */
+	private array $jobs = [];
+	/** @var array<int, resource|null> ranges read (or failed) and not yet given to the caller */
+	private array $done = [];
+	private int $requestId = 0x4e520000;
+	/** Connections replaced so far (see MAX_RECONNECTS). */
+	public int $reconnects = 0;
+
+	/**
+	 * @param array<int, object> $connections phpseclib 2 connections with their SFTP channel open
+	 * @param list<array{0:string, 1:int, 2:int}> $ranges paths below $root
+	 */
+	private function __construct(
+		private array $connections,
+		private string $root,
+		private array $ranges,
+		private string $spool,
+		private ?\Closure $connect = null,
+	) {
+		foreach (array_keys($connections) as $c) {
+			$this->chunks[$c] = self::CHUNK;
+		}
+	}
+
+	/**
+	 * See IReadsAhead::readAhead(). The connections are ended when the generator ends.
+	 *
+	 * @param list<object> $connections
+	 * @param list<array{0:string, 1:int, 2:int}> $ranges
+	 * @param (\Closure():?object)|null $connect a new connection in place of one that broke
+	 *                                         (null when it cannot be made)
+	 * @return \Generator<int, resource|null>
+	 */
+	public static function fetch(array $connections, string $root, array $ranges, string $spool, ?\Closure $connect = null): \Generator {
+		$fetcher = new self($connections, rtrim($root, '/'), $ranges, $spool, $connect);
+		try {
+			yield from $fetcher->run();
+		} finally {
+			$fetcher->end();
+		}
+	}
+
+	/** @return \Generator<int, resource|null> */
+	private function run(): \Generator {
+		$count = count($this->ranges);
+		$next = 0;
+		for ($want = 0; $want < $count; $want++) {
+			while (!array_key_exists($want, $this->done)) {
+				foreach (array_keys($this->connections) as $c) {
+					if (!isset($this->jobs[$c]) && $next < $count && $next < $want + self::AHEAD) {
+						$this->start($c, $next++);
+					}
+				}
+				if ($this->jobs === [] && !array_key_exists($want, $this->done)) {
+					// no connection left: the rest the usual way
+					$this->done[$want] = null;
+					$next = max($next, $want + 1);
+					break;
+				}
+				foreach ($this->ready() as $c) {
+					$this->step($c);
+				}
+			}
+			$fh = $this->done[$want];
+			unset($this->done[$want]);
+			try {
+				yield $want => $fh;
+			} finally {
+				if (is_resource($fh)) {
+					fclose($fh);
+				}
+			}
+		}
+	}
+
+	/**
+	 * The busy connections that have an answer waiting, so that a connection that waits for a
+	 * lost packet to be sent again does not hold up the others; all of them when the connections
+	 * do not tell (then they are taken in turn).
+	 *
+	 * @return list<int>
+	 */
+	private function ready(): array {
+		$sockets = [];
+		$ready = [];
+		foreach ($this->jobs as $c => $job) {
+			$sftp = $this->connections[$c];
+			$vars = get_object_vars($sftp); // only what is public: phpseclib 2's are
+			// Answers phpseclib has read already: the socket does not show them. Its request
+			// buffer holds whole answers read while it waited for another; its packet buffer and
+			// channel buffers hold what came in with the last packet it read (small answers, such
+			// as a CLOSE and the next OPEN, often arrive together).
+			$buffered = $vars['requestBuffer'] ?? null;
+			$channels = $vars['channel_buffers'] ?? null;
+			if ((is_array($buffered) && isset($job->queue[0]) && array_key_exists($job->queue[0][0], $buffered))
+				|| (is_string($vars['packet_buffer'] ?? null) && self::wholePacket($vars['packet_buffer']))
+				|| (is_array($channels) && array_filter($channels) !== [])) {
+				$ready[] = $c;
+				continue;
+			}
+			$socket = $vars['fsock'] ?? null;
+			if (!is_resource($socket)) {
+				return array_keys($this->jobs);
+			}
+			$sockets[$c] = $socket;
+		}
+		if ($ready !== [] || $sockets === []) {
+			return $ready;
+		}
+		$read = $sockets;
+		$write = null;
+		$except = null;
+		// Data in PHP's own buffer of a socket counts as readable. Nothing within the time:
+		// each in turn, where phpseclib's own timeout applies.
+		$n = @stream_select($read, $write, $except, self::SELECT_TIMEOUT);
+		return $n === false || $n === 0 ? array_keys($this->jobs) : array_keys($read);
+	}
+
+	/** Whether $buffer starts with a whole SFTP packet (length, then that many bytes). */
+	private static function wholePacket(string $buffer): bool {
+		$length = strlen($buffer) >= 4 ? unpack('N', $buffer) : false;
+		return is_array($length) && strlen($buffer) >= 4 + $length[1];
+	}
+
+	private function start(int $c, int $i): void {
+		[$path, $offset, $length] = $this->ranges[$i];
+		$fh = $this->spoolFile();
+		if ($fh === null) {
+			$this->done[$i] = null;
+			return;
+		}
+		if ($length <= 0) {
+			$this->done[$i] = $fh;
+			return;
+		}
+		$this->jobs[$c] = new SftpRangeJob($i, $fh, $offset, $offset + $length);
+		try {
+			$remote = $this->connections[$c]->_realpath($this->root . '/' . ltrim($path, '/'));
+			if (!is_string($remote)) {
+				$this->finish($c);
+				return;
+			}
+			$this->send($c, self::FXP_OPEN, pack('Na*N2', strlen($remote), $remote, self::FXF_READ, 0), self::OPEN);
+		} catch (\Throwable) {
+			$this->drop($c);
+		}
+	}
+
+	/** Take the oldest answer on connection $c, then keep its requests in flight. */
+	private function step(int $c): void {
+		$job = $this->jobs[$c];
+		[$id, $kind, $offset, $length] = array_shift($job->queue) ?? throw new \LogicException('no request in flight');
+		$sftp = $this->connections[$c];
+		try {
+			$response = $sftp->_get_sftp_packet($id);
+			if (!is_string($response)) {
+				throw new BackendException('SFTP connection lost');
+			}
+			$type = $sftp->packet_type;
+			if ($kind === self::OPEN) {
+				if ($type === self::FXP_HANDLE) {
+					$job->handle = substr($response, 4);
+				} else {
+					$job->failed = true; // missing, or no permission
+				}
+			} elseif ($kind === self::READ) {
+				if (!$job->failed) {
+					$this->received($c, $type, $response, $offset, $length);
+				}
+			} else {
+				$status = $type === self::FXP_STATUS && strlen($response) >= 4 ? unpack('N', $response) : false;
+				$closed = is_array($status) && $status[1] === self::FX_OK;
+				$this->finish($c);
+				if (!$closed) {
+					// handles might pile up on the server: not used again
+					$this->drop($c);
+				}
+				return;
+			}
+			$this->refill($c);
+		} catch (\Throwable) {
+			$this->drop($c);
+		}
+	}
+
+	private function received(int $c, int $type, string $response, int $offset, int $length): void {
+		$job = $this->jobs[$c];
+		$declared = strlen($response) >= 4 ? unpack('N', $response) : false;
+		$n = is_array($declared) ? $declared[1] : 0;
+		if ($type !== self::FXP_DATA || $n <= 0 || $n !== strlen($response) - 4 || $n > $length) {
+			// end of file before the end of the range, a server error or a malformed answer
+			$job->failed = true;
+			return;
+		}
+		if (fseek($job->fh, $offset - $job->start) !== 0 || fwrite($job->fh, substr($response, 4)) !== $n) {
+			$job->failed = true;
+			return;
+		}
+		$job->got += $n;
+		if ($n < $length) {
+			// a server that caps a READ: ask for the rest, and no more than it gives from now on
+			$job->redo[] = [$offset + $n, $length - $n];
+			$this->chunks[$c] = max(self::MIN_CHUNK, min($this->chunks[$c], $n));
+		}
+	}
+
+	/** Send READs until DEPTH are in flight; when nothing is left to ask for, close the file. */
+	private function refill(int $c): void {
+		$job = $this->jobs[$c];
+		if ($job->handle !== '' && !$job->failed) {
+			while (count($job->queue) < self::DEPTH) {
+				if ($job->redo !== []) {
+					[$offset, $length] = array_shift($job->redo);
+				} elseif ($job->next < $job->end) {
+					$offset = $job->next;
+					$length = min($this->chunks[$c], $job->end - $offset);
+					$job->next += $length;
+				} else {
+					break;
+				}
+				$packet = pack('Na*N3', strlen($job->handle), $job->handle, intdiv($offset, 4294967296), $offset % 4294967296, $length);
+				$this->send($c, self::FXP_READ, $packet, self::READ, $offset, $length);
+			}
+		}
+		if ($job->queue !== []) {
+			return;
+		}
+		if ($job->handle === '') {
+			$this->finish($c); // not opened: nothing to close
+			return;
+		}
+		$this->send($c, self::FXP_CLOSE, pack('Na*', strlen($job->handle), $job->handle), self::CLOSE);
+	}
+
+	private function send(int $c, int $type, string $packet, int $kind, int $offset = 0, int $length = 0): void {
+		$this->requestId = $this->requestId >= 0x7fffffff ? 0x4e520000 : $this->requestId + 1;
+		if (!$this->connections[$c]->_send_sftp_packet($type, $packet, $this->requestId)) {
+			throw new BackendException('SFTP connection failed while sending a request');
+		}
+		$this->jobs[$c]->queue[] = [$this->requestId, $kind, $offset, $length];
+	}
+
+	/** The range of connection $c is done: its stream for the caller, or null when it failed. */
+	private function finish(int $c): void {
+		$job = $this->jobs[$c];
+		unset($this->jobs[$c]);
+		if ($job->failed || $job->got !== $job->end - $job->start || !rewind($job->fh)) {
+			self::closeFile($job->fh);
+			$this->done[$job->i] = null;
+			return;
+		}
+		$this->done[$job->i] = $job->fh;
+	}
+
+	/** A connection in an unknown state: its range fails, and it is not used again. */
+	private function drop(int $c): void {
+		if (isset($this->jobs[$c])) {
+			$job = $this->jobs[$c];
+			unset($this->jobs[$c]);
+			self::closeFile($job->fh);
+			$this->done[$job->i] = null;
+		}
+		self::disconnect($this->connections[$c]);
+		unset($this->connections[$c]);
+		// A long restore outlives connections (a Storage Box drops one now and then): without a
+		// new one, the rest would be read the slow way.
+		if ($this->connect === null || $this->reconnects >= self::MAX_RECONNECTS) {
+			return;
+		}
+		$this->reconnects++;
+		try {
+			$new = ($this->connect)();
+		} catch (\Throwable) {
+			$new = null;
+		}
+		if (function_exists('OCP\Log\logger')) {
+			\OCP\Log\logger('ng_backup')->info($new === null
+				? 'Reading ahead: an SFTP connection broke and no new one could be made'
+				: 'Reading ahead: an SFTP connection broke; reading on with a new one');
+		}
+		if ($new !== null) {
+			$key = max(array_merge(array_keys($this->chunks), [-1])) + 1;
+			$this->connections[$key] = $new;
+			$this->chunks[$key] = self::CHUNK;
+		}
+	}
+
+	/** @return resource|null a new file in the spool, already unlinked: nothing is left behind if the process dies */
+	private function spoolFile() {
+		$path = $this->spool . '/' . bin2hex(random_bytes(8));
+		$fh = @fopen($path, 'x+b');
+		if ($fh === false) {
+			return null;
+		}
+		@unlink($path);
+		return $fh;
+	}
+
+	private function end(): void {
+		foreach ($this->jobs as $job) {
+			self::closeFile($job->fh);
+		}
+		foreach ($this->done as $fh) {
+			if (is_resource($fh)) {
+				fclose($fh);
+			}
+		}
+		$this->jobs = [];
+		$this->done = [];
+		foreach ($this->connections as $sftp) {
+			self::disconnect($sftp);
+		}
+		$this->connections = [];
+	}
+
+	private static function closeFile(mixed $fh): void {
+		if (is_resource($fh)) {
+			fclose($fh);
+		}
+	}
+
+	private static function disconnect(object $sftp): void {
+		try {
+			if (is_callable([$sftp, 'disconnect'])) {
+				$sftp->disconnect();
+			}
+		} catch (\Throwable) {
+		}
+	}
+}
@@ -0,0 +1,40 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Backend;
+
+/**
+ * One range that SftpRangeFetcher reads on one connection.
+ *
+ * @internal
+ */
+final class SftpRangeJob {
+	public string $handle = '';
+	/** Offset of the next READ to send. */
+	public int $next;
+	/** @var list<array{int, int}> offset and length of parts to ask for again (after a short answer) */
+	public array $redo = [];
+	/** @var list<array{int, int, int, int}> request id, kind, offset and length, in the order sent */
+	public array $queue = [];
+	/** Bytes received. */
+	public int $got = 0;
+	public bool $failed = false;
+
+	/** @var resource the spool file that holds the range */
+	public $fh;
+
+	/** @param resource $fh */
+	public function __construct(
+		public int $i,
+		$fh,
+		public int $start,
+		public int $end,
+	) {
+		$this->fh = $fh;
+		$this->next = $start;
+	}
+}
@@ -144,9 +144,9 @@ final class SftpReadStream {

 	/**
 	 * The storage's phpseclib connection, with its SFTP channel open, when it offers what these
-	 * streams need; null otherwise.
+	 * streams (and SftpRangeFetcher) need; null otherwise.
 	 */
-	private static function connection(object $storage): ?object {
+	public static function connection(object $storage): ?object {
 		if (!is_callable([$storage, 'getConnection']) || !is_callable([$storage, 'getRoot'])) {
 			return null;
 		}
@@ -14,7 +14,7 @@ use OCP\Files\Storage\IWriteStreamStorage;
 * IBackend on top of a Nextcloud storage object (from files_external or core), below a base
 * directory on that storage.
 */
-final class StorageBackend implements IBackend {
+final class StorageBackend implements IBackend, IReadsAhead {
 	/** Object stores write whole objects atomically; other storages get a temp name + rename. */
 	private bool $atomicPut;
 	/** SFTP: read through SftpReadStream, which keeps several requests in flight. */
@@ -29,6 +29,8 @@ final class StorageBackend implements IBackend {
 	/** Seconds without a request after which an SFTP connection is checked before it is used. */
 	public const IDLE_CHECK = 60;
 	private float $lastUsed;
+	/** Connections readAhead() opens (see SftpRangeFetcher::DEFAULT_CONNECTIONS). */
+	public int $readAheadConnections = SftpRangeFetcher::DEFAULT_CONNECTIONS;
 	/** Times the storage was built anew after its connection died (for tests and diagnostics). */
 	public int $reconnects = 0;

@@ -243,6 +245,62 @@ final class StorageBackend implements IBackend {
 		}
 	}

+	/**
+	 * Over SFTP: SftpRangeFetcher, on readAheadConnections new connections of its own (built with the
+	 * reconnect function), so the storage's connection stays free for other requests. Elsewhere
+	 * not (yet): null.
+	 */
+	public function canReadAhead(): bool {
+		return $this->sftpReads && $this->reconnect !== null;
+	}
+
+	public function readAhead(array $ranges, string $spool): ?\Generator {
+		$reconnect = $this->reconnect;
+		if (!$this->sftpReads || $reconnect === null || $ranges === []) {
+			return null;
+		}
+		// before any connection is opened: an invalid path throws
+		$abs = array_map(fn (array $r): array => [$this->abs($r[0]), $r[1], $r[2]], $ranges);
+		$storages = [];
+		$connections = [];
+		$root = '';
+		$wanted = max(1, min(SftpRangeFetcher::MAX_CONNECTIONS, $this->readAheadConnections));
+		for ($n = 0; $n < $wanted; $n++) {
+			try {
+				$storage = $reconnect();
+				$sftp = SftpReadStream::connection($storage);
+				if ($sftp === null || !is_callable([$storage, 'getRoot'])) {
+					break;
+				}
+				$root = (string)$storage->getRoot();
+			} catch (\Throwable) {
+				break; // the server may allow fewer connections: go on with those that work
+			}
+			$storages[] = $storage;
+			$connections[] = $sftp;
+		}
+		if ($connections === []) {
+			return null;
+		}
+		// in place of a connection that broke: a new storage, kept like the others
+		$connect = static function () use ($reconnect, &$storages): ?object {
+			$storage = $reconnect();
+			$sftp = SftpReadStream::connection($storage);
+			if ($sftp !== null) {
+				$storages[] = $storage;
+			}
+			return $sftp;
+		};
+		return (static function () use (&$storages, $connections, $root, $abs, $spool, $connect): \Generator {
+			// the storages hold their connections: kept until the fetcher is done
+			try {
+				yield from SftpRangeFetcher::fetch($connections, $root, $abs, $spool, $connect);
+			} finally {
+				unset($storages);
+			}
+		})();
+	}
+
 	public function getRange(string $path, int $offset, int $length): string {
 		return $this->call(fn () => $this->readRange($path, $offset, $length));
 	}
@@ -429,8 +487,9 @@ final class StorageBackend implements IBackend {
 		$this->lastUsed = microtime(true);
 	}

+	/** After a failed request: ask the server (one round trip), as phpseclib may not know it is gone. */
 	private function reopenIfDead(): bool {
-		if ($this->reconnect === null || !$this->sftpReads || $this->alive(false) !== false) {
+		if ($this->reconnect === null || !$this->sftpReads || $this->alive(true) !== false) {
 			return false;
 		}
 		$this->reopen();
@@ -459,8 +518,9 @@ final class StorageBackend implements IBackend {
 			if (is_callable([$sftp, 'clearStatCache'])) {
 				$sftp->clearStatCache(); // or stat() answers from memory
 			}
-			$sftp->stat('.');
-			return (bool)$sftp->isConnected();
+			// A Storage Box drops a quiet connection without phpseclib noticing: it still says
+			// connected, and every request fails. So the answer counts, not what it says after.
+			return $sftp->stat('.') !== false && (bool)$sftp->isConnected();
 		} catch (\Throwable) {
 			return false;
 		}
@@ -88,10 +88,21 @@ HELP);
 					$this->installApp($app, $output);
 				}
 			}
+			$output->writeln('Nextcloud is in maintenance mode while the restore runs; it is switched off when the restore completes.');
 			$result = $this->restoreFull->restoreFull($target, $snapshot, $skipMissingTables);
 		} catch (\Throwable $e) {
 			$output->writeln('<error>' . $e->getMessage() . '</error>');
-			$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
+			if (\OCP\Server::get(\OCP\IConfig::class)->getSystemValueBool('maintenance', false)) {
+				$output->writeln('The restore had started: the instance may be partly restored, and maintenance mode stays on so that nobody uses it like that.');
+				$output->writeln('Fix the cause, then switch maintenance mode off and run the same command again with --force, in one line:');
+				$again = 'occ backup:restore:full ' . escapeshellarg((string)$input->getArgument('target')) . ' ' . escapeshellarg((string)$input->getArgument('snapshot'))
+					. ($input->getOption('skip-missing-tables') ? ' --skip-missing-tables' : '') . ' --force --no-interaction';
+				$output->writeln('  occ maintenance:mode --off && ' . $again);
+				$output->writeln('(Without --install-apps, so the instance is open only for the seconds until the restore starts again. Apps the first run could not install are listed at the end: install those afterwards.)');
+			} else {
+				$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
+			}
+			$output->writeln('If occ then says that Nextcloud needs an upgrade (the database is already the old server\'s), run occ upgrade first.');
 			return 1;
 		}
 		$output->writeln(sprintf(
@@ -120,6 +131,9 @@ HELP);
 		if (!$result['dbConsistent']) {
 			$output->writeln('<comment>Warning: this snapshot\'s database dump was made over several steps (web cron), so it is not one consistent point in time. Check the instance carefully; prefer a snapshot made with system cron.</comment>');
 		}
+		if ($result['maintenanceLeftOn']) {
+			$output->writeln('<comment>Maintenance mode was on before the restore and stays on: switch it off with occ maintenance:mode --off when the instance is ready.</comment>');
+		}
 		$output->writeln('Next: check trusted_domains and overwritehost/overwriteprotocol match this server, then occ maintenance:repair.');
 		return 0;
 	}
@@ -216,7 +216,8 @@ final class DbDumper {
 				}
 				$report['restored'][] = $table;
 			}
-			$this->restoreSequences($manifest['sequences'] ?? []);
+			// unchanged tables too: a restore that stopped earlier may have left their counters at 1
+			$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
 			$this->resumeForeignKeys($foreignKeys);
 			$this->db->commit();
 			// Foreign keys of the tables created above, now their rows are in. Not inside the
@@ -573,11 +574,21 @@ final class DbDumper {
 		return $name;
 	}

-	private function restoreSequences(array $sequences): void {
+	/**
+	 * @param list<string> $leftOut tables not restored (--skip-missing-tables): they do not exist here
+	 * @param list<string> $restored tables restored or unchanged: their other counters go after their data
+	 */
+	private function restoreSequences(array $sequences, array $leftOut = [], array $restored = []): void {
 		if (!$this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\PostgreSQLPlatform) {
 			return;
 		}
+		$leftOut = array_flip($leftOut);
+		$set = [];
 		foreach ($sequences as $info) {
+			if (isset($leftOut[$info['table']])) {
+				continue;
+			}
+			$set[$info['table'] . '.' . $info['column']] = true;
 			$seq = $this->db->fetchOne('SELECT pg_get_serial_sequence(quote_ident(?), ?)', [$info['table'], $info['column']]);
 			if ($seq === false || $seq === null) {
 				continue;
@@ -592,5 +603,26 @@ final class DbDumper {
 					[\Doctrine\DBAL\ParameterType::STRING, \Doctrine\DBAL\ParameterType::INTEGER, \Doctrine\DBAL\ParameterType::BOOLEAN]);
 			}
 		}
+		// A backup from MySQL/MariaDB or SQLite records no counters (those derive them from the
+		// data): every other counter of a restored table goes after its data, or new rows (a new
+		// file, an activity) would collide with restored ones.
+		$restored = array_flip($restored);
+		$rows = $this->db->fetchAllAssociative(
+			"SELECT c.table_name, c.column_name, pg_get_serial_sequence(quote_ident(c.table_name), c.column_name) AS seq
+			 FROM information_schema.columns c
+			 WHERE c.table_schema = current_schema() AND pg_get_serial_sequence(quote_ident(c.table_name), c.column_name) IS NOT NULL");
+		foreach ($rows as $r) {
+			if (!isset($restored[$r['table_name']]) || isset($set[$r['table_name'] . '.' . $r['column_name']])) {
+				continue;
+			}
+			$max = (int)$this->db->fetchOne(sprintf('SELECT COALESCE(MAX(%s), 0) FROM %s',
+				$this->db->quoteIdentifier($r['column_name']), $this->db->quoteIdentifier($r['table_name'])));
+			// never backwards: a counter already past the data stays where it is
+			$state = $this->db->fetchAssociative('SELECT last_value, is_called FROM ' . self::sequenceIdentifier($r['seq']));
+			$next = $state === false ? 1 : (int)$state['last_value'] + ($state['is_called'] ? 1 : 0);
+			if ($next <= $max) {
+				$this->db->executeQuery('SELECT setval(CAST(? AS regclass), ?, false)', [$r['seq'], $max + 1]);
+			}
+		}
 	}
 }
@@ -7,6 +7,7 @@ declare(strict_types=1);

 namespace OCA\NgBackup\Repository;

+use OCA\NgBackup\Backend\BackendException;
 use OCA\NgBackup\Backend\IBackend;
 use OCA\NgBackup\Crypto\StreamCipher;

@@ -32,6 +33,8 @@ final class BlobIndex {
 	private const CACHE_LOT = 256;
 	/** @var int index files downloaded by the last load() (for tests and diagnostics) */
 	public int $downloaded = 0;
+	/** @var array<string, true> packs whose index file is loaded (refresh() fetches only others) */
+	private array $loaded = [];

 	public function __construct(
 		private IBackend $backend,
@@ -90,13 +93,14 @@ final class BlobIndex {
 	 * longer exist on the backend are dropped from the cache.
 	 */
 	public function load(bool $verify = false, ?callable $progress = null): void {
-		$this->blobs = [];
+		$hadIndex = $this->blobs !== [];
 		$cached = $verify ? [] : ($this->cache?->all($this->repositoryId) ?? []);
 		$present = [];
 		$this->downloaded = 0;
 		$order = []; // pack ids in listing order: where a blob is in two packs, the last listed wins
 		$fetch = [];
-		foreach ($this->backend->listFiles('index') as $path) {
+		$known = array_keys($this->loaded + $cached + ($verify ? ($this->cache?->all($this->repositoryId) ?? []) : []));
+		foreach ($this->listIndex($known) as $path) {
 			$packId = basename($path);
 			$present[$packId] = true;
 			$order[] = $packId;
@@ -104,6 +108,9 @@ final class BlobIndex {
 				$fetch[] = $path;
 			}
 		}
+		// only now: an empty listing that listIndex() refused leaves the index as it was
+		$this->blobs = [];
+		$this->loaded = $present;
 		// The rest is downloaded several at a time where the location allows it: on a large
 		// repository these are tens of thousands of small files. getMany() keeps their order, so
 		// the cached packs listed before each one are added first.
@@ -185,12 +192,80 @@ final class BlobIndex {
 		rewind($stream);
 		$this->backend->put('index/' . $packId, $stream);
 		$this->cachePut($packId, $entries);
+		$this->loaded[$packId] = true;
 		foreach ($entries as $e) {
 			$this->blobs[$e['id']] = [$packId, $e['offset'], $e['length'], $e['raw'], $e['flags']];
 			unset($this->pending[$e['id']]);
 		}
 	}

+	/**
+	 * Add the index files that appeared on the location since load(), such as those of packs
+	 * another server's cleanup wrote while a restore runs. Only adds and updates: a blob is
+	 * never dropped, as a listing that misses files (a connection that failed) must not make a
+	 * restore lose blobs it knows. A blob in a new pack takes that pack's place.
+	 */
+	public function refresh(): void {
+		$fetch = [];
+		$listed = 0;
+		foreach ($this->listIndex(array_keys($this->loaded)) as $path) {
+			$listed++;
+			if (!isset($this->loaded[basename($path)])) {
+				$fetch[] = $path;
+			}
+		}
+		$toCache = [];
+		foreach ($this->backend->getMany($fetch) as $path => $data) {
+			$packId = basename($path);
+			/** @var list<array{id:string, offset:int, length:int, raw:int, flags:int}> $entries authenticated: written by NG Backup */
+			$entries = json_decode($this->cipher->decryptString($data, 'index:' . $packId), true, 512, JSON_THROW_ON_ERROR);
+			$this->downloaded++;
+			$this->add($packId, $entries);
+			$this->loaded[$packId] = true;
+			$toCache[$packId] = $entries;
+		}
+		if ($toCache !== []) {
+			$this->cache?->putMany($this->repositoryId, $toCache);
+		}
+	}
+
+	/**
+	 * The index files on the location. An empty listing while index files are known ($known,
+	 * pack ids) is usually a connection that failed quietly; taken as is, it would empty the
+	 * index and the cache, make a check report every blob missing, or make a cleanup wipe the
+	 * cache. So the connection is checked (the repository's config must be there) and the folder
+	 * listed once more. Empty again, three of the known index files are looked up one by one:
+	 * only when they are gone too is the repository taken as empty (another server removed its
+	 * last snapshot and cleaned up). Even when that is wrong, nothing on the location is lost:
+	 * only this index and the cache are empty until the next load.
+	 *
+	 * @param list<string> $known
+	 * @return list<string>
+	 * @throws BackendException when the location does not answer as it should
+	 */
+	private function listIndex(array $known): array {
+		$paths = $this->backend->listFiles('index');
+		if ($paths !== [] || $known === []) {
+			return $paths;
+		}
+		if (!$this->backend->exists('config')) {
+			throw new BackendException('The location listed no index files and has no repository config: not trusting that listing');
+		}
+		$paths = $this->backend->listFiles('index');
+		if ($paths !== []) {
+			return $paths;
+		}
+		// spread over the list: first, middle and last (another server's cleanup removes packs
+		// here and there, not in this order)
+		$n = count($known);
+		foreach (array_unique([$known[0], $known[intdiv($n, 2)], $known[$n - 1]]) as $packId) {
+			if ($this->backend->exists('index/' . $packId)) {
+				throw new BackendException('The location listed no index files, while index/' . $packId . ' is there: not trusting that listing');
+			}
+		}
+		return [];
+	}
+
 	/** @return array<string, list<array{id:string, offset:int, length:int}>> packId => blobs currently located in that pack */
 	public function packs(): array {
 		$packs = [];
@@ -204,6 +279,7 @@ final class BlobIndex {
 	public function removePack(string $packId): void {
 		$this->backend->delete('index/' . $packId);
 		$this->cacheRemove([$packId]);
+		unset($this->loaded[$packId]);
 		foreach ($this->blobs as $id => $loc) {
 			if ($loc[0] === $packId) {
 				unset($this->blobs[$id]);
@@ -8,9 +8,10 @@ declare(strict_types=1);
 namespace OCA\NgBackup\Repository;

 use OCA\NgBackup\Backend\IBackend;
+use OCA\NgBackup\Backend\IReadsAhead;

 /** The location as Repository sees it: closes the PackReader's open stream before every other request. */
-final class PackReaderGuard implements IBackend {
+final class PackReaderGuard implements IBackend, IReadsAhead {
 	public function __construct(
 		private IBackend $inner,
 		private PackReader $reader,
@@ -43,6 +44,15 @@ final class PackReaderGuard implements IBackend {
 		return $this->inner->getRanges($ranges);
 	}

+	public function canReadAhead(): bool {
+		return $this->inner instanceof IReadsAhead && $this->inner->canReadAhead();
+	}
+
+	/** Over connections of the location's own: the open stream may stay open. */
+	public function readAhead(array $ranges, string $spool): ?\Generator {
+		return $this->inner instanceof IReadsAhead ? $this->inner->readAhead($ranges, $spool) : null;
+	}
+
 	public function exists(string $path): bool {
 		$this->reader->close();
 		return $this->inner->exists($path);
@@ -0,0 +1,123 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Repository;
+
+/**
+ * The ranges of packs a restore reads, in order, for IReadsAhead::readAhead(), and the reading of
+ * blobs from them as the restore goes.
+ *
+ * add() takes the blobs in the order the restore needs them. A blob in the same pack as the range
+ * before it, not before that range and at most GAP after its end, extends that range; anything
+ * else starts a new one. read() takes the same blobs in the same order: a blob is in the current
+ * range or in the next one, which then becomes current; a blob not in the plan is read elsewhere.
+ */
+final class ReadAheadPlan {
+	/** Unused bytes between two blobs of a pack that are read along rather than asked for apart. */
+	public const GAP = 4 * 1048576;
+	/** Longest range: packs are 32 MiB, repacked ones too. */
+	public const MAX_RANGE = 64 * 1048576;
+	/** Ranges after the current one that read() looks in for a blob. */
+	private const SEARCH = 16;
+
+	/** @var list<array{0:string, 1:int, 2:int}> [path, offset, length] */
+	private array $ranges = [];
+	/** @var \Generator<int, resource|null>|null */
+	private ?\Generator $streams = null;
+	private int $current = -1;
+	/** @var resource|null */
+	private $stream = null;
+
+	/** @param string $spool local folder for the ranges read ahead (see IReadsAhead::readAhead()) */
+	public function __construct(
+		public readonly string $spool,
+	) {
+	}
+
+	public function add(string $path, int $offset, int $length): void {
+		$last = array_pop($this->ranges);
+		if ($last !== null) {
+			[$p, $start, $len] = $last;
+			if ($p === $path && $offset >= $start && $offset <= $start + $len + self::GAP
+				&& $offset + $length - $start <= self::MAX_RANGE) {
+				$this->ranges[] = [$p, $start, max($len, $offset + $length - $start)];
+				return;
+			}
+			$this->ranges[] = $last;
+		}
+		$this->ranges[] = [$path, $offset, $length];
+	}
+
+	/** @return list<array{0:string, 1:int, 2:int}> */
+	public function ranges(): array {
+		return $this->ranges;
+	}
+
+	/** @param \Generator<int, resource|null> $streams what IReadsAhead::readAhead() gave for ranges() */
+	public function readFrom(\Generator $streams): void {
+		$this->streams = $streams;
+	}
+
+	/** The bytes of a blob, or null when its range could not be read ahead (read it the usual way). */
+	public function read(string $path, int $offset, int $length): ?string {
+		if ($this->streams === null) {
+			return null;
+		}
+		if (!$this->contains($this->current, $path, $offset, $length)) {
+			// Normally the next range. Further on when blobs before it were read elsewhere: the
+			// index was reloaded after another server's cleanup moved them to new packs.
+			$to = null;
+			for ($i = $this->current + 1; $i <= $this->current + self::SEARCH && $to === null; $i++) {
+				if ($this->contains($i, $path, $offset, $length)) {
+					$to = $i;
+				}
+			}
+			if ($to === null) {
+				return null; // not where the plan has it
+			}
+			if ($this->current < 0) {
+				$this->streams->current();
+				$this->current = 0;
+			}
+			while ($this->current < $to) {
+				$this->streams->next();
+				$this->current++;
+			}
+			$stream = $this->streams->valid() ? $this->streams->current() : null;
+			$this->stream = is_resource($stream) ? $stream : null;
+		}
+		$fh = $this->stream;
+		if ($fh === null || fseek($fh, $offset - $this->ranges[$this->current][1]) !== 0) {
+			return null;
+		}
+		$data = '';
+		while (strlen($data) < $length) {
+			$chunk = fread($fh, $length - strlen($data));
+			if ($chunk === false || $chunk === '') {
+				return null;
+			}
+			$data .= $chunk;
+		}
+		return $data;
+	}
+
+	/** Ends the reads ahead, also halfway: closes their streams and connections. */
+	public function close(): void {
+		$this->stream = null;
+		$streams = $this->streams;
+		$this->streams = null;
+		unset($streams); // the generator's finally blocks run when it is destroyed
+	}
+
+	private function contains(int $i, string $path, int $offset, int $length): bool {
+		if ($i < 0 || !isset($this->ranges[$i])) {
+			return false;
+		}
+		[$p, $start, $len] = $this->ranges[$i];
+		return $p === $path && $offset >= $start && $offset + $length <= $start + $len;
+	}
+}
@@ -9,6 +9,7 @@ namespace OCA\NgBackup\Repository;

 use OCA\NgBackup\Backend\BackendException;
 use OCA\NgBackup\Backend\IBackend;
+use OCA\NgBackup\Backend\IReadsAhead;
 use OCA\NgBackup\Crypto\KeyRing;
 use OCA\NgBackup\Crypto\StreamCipher;

@@ -71,6 +72,14 @@ final class Repository {
 		if ($backend->exists('config')) {
 			throw new RepositoryException('A repository already exists at this location');
 		}
+		// No config, but backup data: the config is missing (or could not be seen just now, a
+		// connection that failed). A new repository here would write a config over the one that
+		// opens this data; refuse instead.
+		foreach (['index', 'snapshots', 'trash/snapshots', 'catalog'] as $dir) {
+			if ($backend->listFiles($dir) !== []) {
+				throw new RepositoryException("This location holds backup data ($dir/) but no repository config could be read: not creating a new repository over it. Check the connection and the folder; a missing config can be restored from the recovery kit.");
+			}
+		}
 		$id = bin2hex(random_bytes(16));
 		$config = ['format' => self::FORMAT, 'id' => $id, 'created' => gmdate('c'), 'key' => $wrappedKey,
 			'check' => base64_encode((new StreamCipher($keys))->encryptString($id, 'config-check'))];
@@ -216,6 +225,66 @@ final class Repository {
 		if ($prefix !== '' && !self::isSafePath(rtrim($prefix, '/'))) {
 			throw new RepositoryException('Invalid path prefix');
 		}
+		$plan = $this->readAhead($snapshotId, $prefix, $target);
+		try {
+			return $this->restoreEntries($snapshotId, $target, $prefix, $stripPrefix, $heartbeat, $plan);
+		} finally {
+			$plan?->close();
+			if ($plan !== null) {
+				@rmdir($plan->spool);
+			}
+		}
+	}
+
+	/**
+	 * Start reading the packs of a restore ahead, when the location can (IReadsAhead): a first
+	 * pass over the snapshot lists the ranges of packs it needs, in order. The spool is a
+	 * folder in the target, which has room for the restore anyway.
+	 */
+	private function readAhead(string $snapshotId, string $prefix, string $target): ?ReadAheadPlan {
+		if (!$this->backend instanceof IReadsAhead || !$this->backend->canReadAhead()) {
+			return null;
+		}
+		// what a restore that was stopped left behind: empty, its files were unlinked when opened
+		$stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR);
+		foreach ($stale === false ? [] : $stale as $dir) {
+			@rmdir($dir);
+		}
+		$plan = new ReadAheadPlan(rtrim($target, '/') . '/.ngb-restore-spool-' . bin2hex(random_bytes(4)));
+		try {
+			foreach ($this->streamEntries($snapshotId) as $entry) {
+				if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
+					continue;
+				}
+				foreach ($entry['b'] as $id) {
+					try {
+						[$pack, $offset, $length] = $this->index->get($id);
+					} catch (RepositoryException) {
+						continue; // the restore reports it when it gets there
+					}
+					$plan->add('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
+				}
+			}
+		} catch (\Throwable) {
+			return null; // reading ahead is only faster: the restore itself reads the snapshot again and reports what is wrong
+		}
+		if (count($plan->ranges()) < 2 || (!is_dir($plan->spool) && !@mkdir($plan->spool, 0700, true))) {
+			return null;
+		}
+		try {
+			$streams = $this->backend->readAhead($plan->ranges(), $plan->spool);
+		} catch (\Throwable) {
+			$streams = null; // the usual reads report what is wrong
+		}
+		if ($streams === null) {
+			@rmdir($plan->spool);
+			return null;
+		}
+		$plan->readFrom($streams);
+		return $plan;
+	}
+
+	private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan): int {
 		$count = 0;
 		foreach ($this->streamEntries($snapshotId) as $entry) {
 			if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
@@ -240,7 +309,7 @@ final class Repository {
 				throw new RepositoryException('Cannot create a temporary file for ' . $entry['p']);
 			}
 			foreach ($entry['b'] as $id) {
-				fwrite($out, $this->loadBlob($id));
+				fwrite($out, $this->restoreBlob($id, $plan));
 				if ($heartbeat !== null) {
 					$heartbeat();
 				}
@@ -780,42 +849,56 @@ final class Repository {
 					$done++;
 				}
 			}
-		} catch (BackendException $e) {
-			// A pack moved (see loadBlob()): the rest one blob at a time, from the fresh index.
-			$this->refreshIndex($e);
+		} catch (BackendException) {
+			// A pack moved, or the read failed for a while (see loadBlob()): the rest one blob at a
+			// time, each read once more after a failure.
 			foreach (array_slice($ids, $done) as $id) {
 				yield $this->loadBlob($id);
 			}
 		}
 	}

+	/** A blob for restore(): read ahead when it could be, otherwise (or when that copy is bad) the usual way. */
+	private function restoreBlob(string $id, ?ReadAheadPlan $plan): string {
+		if ($plan !== null) {
+			[$pack, $offset, $length] = $this->index->get($id);
+			$encrypted = $plan->read('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
+			if ($encrypted !== null) {
+				try {
+					return $this->decodeBlob($id, $encrypted);
+				} catch (\Exception) {
+					// read again from the location below, which reports what is wrong
+				}
+			}
+		}
+		return $this->loadBlob($id);
+	}
+
 	private function loadBlob(string $id): string {
 		[$pack, $offset, $length] = $this->index->get($id);
 		try {
 			$encrypted = $this->packReader->read($pack, $offset, $length);
 		} catch (BackendException $e) {
 			// Another server's cleanup may have repacked this blob into a new pack since the index
-			// was loaded (a disaster recovery test while the original server still runs). Reload
-			// the index from the location; read again only if the blob has moved.
-			$this->refreshIndex($e);
+			// was loaded (a disaster recovery test while the original server still runs): add the
+			// new index files, and read the blob where it is now. Or the read failed for a while
+			// (a connection that died; the backend has replaced it by now): then read it once
+			// more where it was. A second failure is the real one.
+			try {
+				$this->index->refresh();
+			} catch (\Throwable) {
+				// the location could not be listed either: read again where the blob was
+			}
 			[$moved, $offset, $length] = $this->index->get($id);
-			if ($moved === $pack) {
+			try {
+				$encrypted = $this->packReader->read($moved, $offset, $length);
+			} catch (BackendException) {
 				throw $e;
 			}
-			$encrypted = $this->packReader->read($moved, $offset, $length);
 		}
 		return $this->decodeBlob($id, $encrypted);
 	}

-	/** Reload the index from the location after a failed pack read; rethrows that failure if it cannot. */
-	private function refreshIndex(BackendException $cause): void {
-		try {
-			$this->index->load();
-		} catch (\Throwable) {
-			throw $cause;
-		}
-	}
-
 	/** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */
 	private function decodeBlob(string $id, string $encrypted): string {
 		[, , , $raw, $flags] = $this->index->get($id);
@@ -27,9 +27,14 @@ final class RestoreFullService {
 	 * Config keys that must keep pointing at THIS installation's actual, already-working
 	 * database and data directory (set by occ maintenance:install), never at the old server's.
 	 */
+	/** App setting: when a full restore started that has not completed (see restoreFull()). */
+	private const INTERRUPTED_KEY = 'restore_full_started';
+
 	private const CONNECTION_KEYS = [
 		'dbtype', 'dbhost', 'dbport', 'dbname', 'dbuser', 'dbpassword', 'dbpersistent',
 		'dbtableprefix', 'dbdriveroptions', 'datadirectory',
+		// set by restoreFull() itself for the length of the restore
+		'maintenance',
 	];

 	public function __construct(
@@ -85,12 +90,50 @@ final class RestoreFullService {
 		];
 	}

-	/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>} */
+	/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>, maintenanceLeftOn:bool} */
 	public function restoreFull(Target $target, string $snapshotId, bool $skipMissingTables = false): array {
-		$check = $this->checkCompatibility($target, $snapshotId, $skipMissingTables);
+		// Maintenance mode for the whole restore: background jobs (cron.php stops in maintenance
+		// mode), page visits (AJAX cron) and users must not write into tables or the data
+		// directory while they are being replaced; a row written in between would collide with a
+		// restored one or be silently lost. Switched off after a restore that completed. After an
+		// error it stays on: the instance may be partly restored, and must not be used like that.
+		// occ only offers app commands outside maintenance mode, so the admin switches it off
+		// right before running the restore again (see RestoreFull). Already on before: left on.
+		// On before the checks below, so that a run again after an error leaves the partly
+		// restored instance open for seconds at most; off again when they refuse (nothing changed).
+		// A marker in ng_backup's own app settings (which the restore keeps) says a restore started
+		// and did not complete: then the instance may be partly restored, and a check that refuses
+		// a run again must not open it.
+		$appConfig = \OCP\Server::get(\OCP\IAppConfig::class);
+		$interrupted = $appConfig->getValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, 0) > 0;
+		$wasInMaintenance = $this->config->getSystemValueBool('maintenance', false);
+		$this->config->setSystemValue('maintenance', true);
+		$reopen = function () use ($wasInMaintenance, $interrupted): void {
+			if (!$wasInMaintenance && !$interrupted) {
+				$this->config->setSystemValue('maintenance', false);
+			}
+		};
+		try {
+			$check = $this->checkCompatibility($target, $snapshotId, $skipMissingTables);
+		} catch (\Throwable $e) {
+			$reopen();
+			throw $e;
+		}
 		if (!$check['ok']) {
+			$reopen();
 			throw new \RuntimeException($this->describeBlockers($check));
 		}
+		$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
+		$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check);
+		$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
+		if (!$wasInMaintenance) {
+			$this->config->setSystemValue('maintenance', false);
+		}
+		return $result + ['maintenanceLeftOn' => $wasInMaintenance];
+	}
+
+	/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>} */
+	private function restoreInMaintenance(Target $target, string $snapshotId, bool $skipMissingTables, array $check): array {
 		return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $skipMissingTables, $check) {
 			$beat = LeaseService::throttled($refresh);
 			$repo = $this->targets->repository($target);
@@ -228,7 +228,8 @@ final class TargetService {
 				$s->setFiles((int)($meta['stats']['files'] ?? 0));
 				$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
 				$this->snapshots->insert($s);
-			} catch (\Throwable) {
+			} catch (\Throwable $e) {
+				Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]);
 			}
 		}
 	}
@@ -287,7 +288,10 @@ final class TargetService {

 	private function buildBackend(string $backend, string $auth, array $options, string $basePath, bool $createBase = true): StorageBackend {
 		$create = fn (): IStorage => $this->factory()->create($backend, $auth, $options);
-		return new StorageBackend($create(), $basePath, $createBase, $create);
+		$storage = new StorageBackend($create(), $basePath, $createBase, $create);
+		$storage->readAheadConnections = Server::get(\OCP\IAppConfig::class)
+			->getValueInt('ng_backup', 'restore_sftp_connections', \OCA\NgBackup\Backend\SftpRangeFetcher::DEFAULT_CONNECTIONS);
+		return $storage;
 	}

 	private ?ExternalStorageFactory $factory = null;
@@ -0,0 +1,240 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Backend;
+
+use OCA\NgBackup\Backend\SftpRangeFetcher;
+use OCA\NgBackup\Backend\SftpReadStream;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use PHPUnit\Framework\TestCase;
+
+require_once __DIR__ . '/SftpReadStreamTest.php'; // FakeSftpConnection, FakeSftpStorage
+
+/** SftpRangeFetcher against the fake SFTP server of SftpReadStreamTest, several connections at once. */
+class SftpRangeFetcherTest extends TestCase {
+	use TempDirTrait;
+
+	/** @var array<string, string> */
+	private array $files = [];
+
+	protected function setUp(): void {
+		SftpReadStream::reset();
+		for ($i = 0; $i < 10; $i++) {
+			$this->files["/home/repo/packs/p$i"] = random_bytes(3 * 1048576 + $i * 1000);
+		}
+	}
+
+	/** @return list<FakeSftpConnection> */
+	private function connections(int $n, array $options = []): array {
+		$result = [];
+		for ($c = 0; $c < $n; $c++) {
+			$result[] = new FakeSftpConnection($this->files, ...($options[$c] ?? []));
+		}
+		return $result;
+	}
+
+	/** @return list<object> what StorageBackend passes on: each with its SFTP channel open */
+	private static function opened(array $connections): array {
+		return array_map(fn (FakeSftpConnection $c): object => SftpReadStream::connection(new FakeSftpStorage($c)), $connections);
+	}
+
+	/** @return list<array{0:string, 1:int, 2:int}> */
+	private function ranges(): array {
+		$ranges = [];
+		for ($i = 0; $i < 10; $i++) {
+			$ranges[] = ["repo/packs/p$i", 1000 * $i, 2 * 1048576 + 333];
+		}
+		return $ranges;
+	}
+
+	/** @return array<int, string|null> sha1 of the contents by position (a failing comparison of MiBs would not fit in memory) */
+	private static function collect(\Generator $streams): array {
+		$result = [];
+		foreach ($streams as $i => $fh) {
+			$result[$i] = $fh === null ? null : sha1(stream_get_contents($fh));
+		}
+		return $result;
+	}
+
+	private function expected(array $ranges): array {
+		return array_map(fn (array $r): string => sha1(substr($this->files['/home/' . $r[0]], $r[1], $r[2])), $ranges);
+	}
+
+	public function testReadsEveryRangeInOrderOverAllConnections(): void {
+		$connections = $this->connections(4);
+		$ranges = $this->ranges();
+		$spool = $this->tempDir();
+
+		$this->assertSame($this->expected($ranges), self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $spool)));
+
+		foreach ($connections as $c) {
+			$this->assertGreaterThan(0, count($c->reads), 'every connection reads');
+			$this->assertLessThanOrEqual(SftpRangeFetcher::DEPTH, $c->maxInFlight);
+			$this->assertSame([], $c->openHandles);
+			$this->assertSame(0, $c->pending());
+			$this->assertTrue($c->disconnected, 'its own connections end with it');
+		}
+		$this->assertSame([], glob("$spool/*"), 'nothing left in the spool');
+	}
+
+	public function testKeepsSeveralRequestsInFlightOnEachConnection(): void {
+		$this->files['/home/repo/packs/big'] = random_bytes(10 * 1048576);
+		$connections = $this->connections(2);
+		self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', [['repo/packs/big', 0, 10 * 1048576], ['repo/packs/big', 5, 10 * 1048576 - 5]], $this->tempDir()));
+
+		$this->assertSame(SftpRangeFetcher::DEPTH, $connections[0]->maxInFlight);
+		$this->assertSame(SftpRangeFetcher::DEPTH, $connections[1]->maxInFlight);
+	}
+
+	public function testAnAnswerPhpseclibHasReadAlreadyIsTakenWithoutWaitingOnTheSocket(): void {
+		// The fake's answers are all "read already": its socket never becomes readable, and
+		// phpseclib's packet buffer holds data. Waiting on the socket would take a second a step.
+		$connections = $this->connections(2);
+		$sockets = [];
+		foreach ($connections as $c) {
+			$pair = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, STREAM_IPPROTO_IP);
+			$sockets[] = $pair;
+			$c->fsock = $pair[0];
+			$c->packet_buffer = pack('N', 4) . 'more'; // a whole packet
+		}
+		$ranges = array_slice($this->ranges(), 0, 3);
+		$start = microtime(true);
+
+		$this->assertSame($this->expected($ranges), self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir())));
+
+		$this->assertLessThan(1.0, microtime(true) - $start);
+	}
+
+	public function testAServerThatCapsAReadStillGivesEveryByte(): void {
+		$connections = $this->connections(3, array_fill(0, 3, ['cap' => 100000]));
+		$ranges = $this->ranges();
+
+		$this->assertSame($this->expected($ranges), self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir())));
+		// learned: after the first short answers (and asking for their rest) it asks for no more than the server gives
+		foreach ($connections as $c) {
+			$this->assertLessThanOrEqual(2 * SftpRangeFetcher::DEPTH, count(array_filter($c->reads, fn (int $n): bool => $n > 100000)));
+		}
+	}
+
+	public function testAMissingFileIsLeftToTheCaller(): void {
+		$connections = $this->connections(2);
+		$ranges = $this->ranges();
+		$ranges[3][0] = 'repo/packs/gone';
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir()));
+
+		$expected = $this->expected($this->ranges());
+		$expected[3] = null;
+		$this->assertSame($expected, $result);
+		$this->assertSame([], $connections[0]->openHandles);
+	}
+
+	public function testARangePastTheEndOfTheFileIsLeftToTheCaller(): void {
+		$ranges = $this->ranges();
+		$ranges[5][2] = 10 * 1048576;
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($this->connections(2)), '/home/', $ranges, $this->tempDir()));
+
+		$this->assertNull($result[5]);
+		$this->assertSame($this->expected($this->ranges())[6], $result[6]);
+	}
+
+	public function testAServerErrorFailsOnlyThatRange(): void {
+		// every read from 1 MiB on fails on connection 0: its ranges, not those of the others
+		$connections = $this->connections(2, [['failAt' => 1048576]]);
+		$ranges = $this->ranges();
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir()));
+
+		$expected = $this->expected($ranges);
+		$failed = array_keys(array_filter($result, fn (?string $r): bool => $r === null));
+		$this->assertNotSame([], $failed);
+		foreach ($result as $i => $data) {
+			if ($data !== null) {
+				$this->assertSame($expected[$i], $data);
+			}
+		}
+		$this->assertLessThan(10, count($failed), 'the other connection goes on');
+		$this->assertSame([], $connections[0]->openHandles, 'a failed range is closed too');
+	}
+
+	public function testALostConnectionIsNotUsedAgainAndTheOthersGoOn(): void {
+		$connections = $this->connections(3, [['loseAt' => 1048576]]);
+		$ranges = $this->ranges();
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir()));
+
+		$expected = $this->expected($ranges);
+		$this->assertSame(1, count(array_filter($result, fn (?string $r): bool => $r === null)), 'only the range it was reading');
+		foreach ($result as $i => $data) {
+			if ($data !== null) {
+				$this->assertSame($expected[$i], $data);
+			}
+		}
+		$this->assertTrue($connections[0]->disconnected);
+		$this->assertLessThanOrEqual(1048576 / 261120 + SftpRangeFetcher::DEPTH + 1, count($connections[0]->reads), 'not used after it broke');
+	}
+
+	public function testWhenEveryConnectionIsLostTheRestIsLeftToTheCaller(): void {
+		$connections = $this->connections(2, [['throwAt' => 1048576], ['loseAt' => 1048576]]);
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $this->ranges(), $this->tempDir()));
+
+		$this->assertSame(array_fill(0, 10, null), $result);
+	}
+
+	public function testABrokenConnectionIsReplacedAndTheRestIsReadAhead(): void {
+		$connections = $this->connections(1, [['loseAt' => 1048576]]);
+		$made = [];
+		$connect = function () use (&$made): object {
+			$made[] = $c = new FakeSftpConnection($this->files);
+			return SftpReadStream::connection(new FakeSftpStorage($c));
+		};
+		$ranges = $this->ranges();
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir(), $connect));
+
+		$expected = $this->expected($ranges);
+		$expected[0] = null; // the range it was reading when it broke: read the usual way
+		$this->assertSame($expected, $result);
+		$this->assertCount(1, $made);
+		$this->assertTrue($made[0]->disconnected, 'the new one ends with the fetcher too');
+	}
+
+	public function testConnectionsThatKeepBreakingAreReplacedOnlySoOften(): void {
+		$made = 0;
+		$connect = function () use (&$made): object {
+			$made++;
+			return SftpReadStream::connection(new FakeSftpStorage(new FakeSftpConnection($this->files, loseAt: 0)));
+		};
+		$ranges = array_merge($this->ranges(), $this->ranges(), $this->ranges());
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($this->connections(1, [['loseAt' => 0]])), '/home/', $ranges, $this->tempDir(), $connect));
+
+		$this->assertSame(array_fill(0, count($ranges), null), $result);
+		$this->assertSame(SftpRangeFetcher::MAX_RECONNECTS, $made);
+	}
+
+	public function testStoppingHalfwayEndsTheConnectionsAndLeavesNothingBehind(): void {
+		$connections = $this->connections(2);
+		$spool = $this->tempDir();
+		$streams = SftpRangeFetcher::fetch(self::opened($connections), '/home/', $this->ranges(), $spool);
+		$this->assertIsResource($streams->current());
+
+		unset($streams);
+
+		$this->assertTrue($connections[0]->disconnected);
+		$this->assertTrue($connections[1]->disconnected);
+		$this->assertSame([], glob("$spool/*"));
+	}
+
+	public function testReadsNoMoreThanAheadOfTheCaller(): void {
+		$connections = $this->connections(4);
+		$streams = SftpRangeFetcher::fetch(self::opened($connections), '/home/', $this->ranges(), $this->tempDir());
+		$streams->current();
+
+		$opened = array_sum(array_map(fn (FakeSftpConnection $c): int => count(array_unique(array_values($c->opened))), $connections));
+		$this->assertLessThanOrEqual(SftpRangeFetcher::AHEAD, $opened);
+	}
+}
@@ -466,6 +466,12 @@ final class FakeSftpConnection {
 	public array $reads = [];
 	/** @var array<string, string> */
 	public array $openHandles = [];
+	/** @var list<string> paths of every OPEN request */
+	public array $opened = [];
+	/** @var resource|null like phpseclib's socket (SftpRangeFetcher waits on it) */
+	public $fsock = null;
+	/** like phpseclib's: what came in with the last packet and is not read yet */
+	public string $packet_buffer = '';
 	/** Whether the SFTP channel is open: phpseclib 2 opens it on the first SFTP call. */
 	public bool $initialised = false;
 	/** Packets sent */
@@ -532,6 +538,7 @@ final class FakeSftpConnection {
 					return false;
 				}
 				$path = substr($data, 4, unpack('N', $data)[1]);
+				$this->opened[] = $path;
 				if (!isset($this->files[$path])) {
 					$this->answers[$id] = [101, pack('N', 2)];
 					break;
@@ -0,0 +1,115 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Backend;
+
+use OCA\NgBackup\Backend\SftpRangeFetcher;
+use OCA\NgBackup\Backend\SftpReadStream;
+use OCA\NgBackup\Backend\StorageBackend;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use OCP\Files\Storage\IStorage;
+use PHPUnit\Framework\TestCase;
+
+require_once __DIR__ . '/SftpReadStreamTest.php'; // FakeSftpConnection
+require_once __DIR__ . '/StorageBackendReconnectTest.php'; // SftpLikeStorage
+
+/** StorageBackend::readAhead(): over SFTP on connections of its own, built with the reconnect function. */
+class StorageBackendReadAheadTest extends TestCase {
+	use TempDirTrait;
+
+	/** @var list<FakeSftpConnection> */
+	private array $connections = [];
+	/** @var array<string, string> */
+	private array $files = [];
+	/** New storages after this many fail to connect. */
+	private ?int $connectLimit = null;
+
+	protected function setUp(): void {
+		SftpReadStream::reset();
+		for ($i = 0; $i < 6; $i++) {
+			$this->files["/home/base/repo/packs/p$i"] = random_bytes(1048576 + $i);
+		}
+	}
+
+	private function storage(): IStorage {
+		if ($this->connectLimit !== null && count($this->connections) >= $this->connectLimit) {
+			throw new \RuntimeException('Too many connections');
+		}
+		$sftp = new FakeSftpConnection($this->files);
+		$this->connections[] = $sftp;
+		$storage = $this->createMock(SftpLikeRootStorage::class);
+		$storage->method('instanceOfStorage')->willReturnCallback(fn (string $class): bool => $class === \OCA\Files_External\Lib\Storage\SFTP::class);
+		$storage->method('getConnection')->willReturn($sftp);
+		$storage->method('getRoot')->willReturn('/home/');
+		$storage->method('is_dir')->willReturn(true);
+		return $storage;
+	}
+
+	/** @return list<array{0:string, 1:int, 2:int}> */
+	private static function ranges(): array {
+		return array_map(fn (int $i): array => ["packs/p$i", 7, 1048576 - 7], range(0, 5));
+	}
+
+	public function testReadsAheadOnNewConnectionsOnly(): void {
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, fn (): IStorage => $this->storage());
+		$backend->readAheadConnections = 4;
+		$this->assertTrue($backend->canReadAhead());
+
+		$result = [];
+		foreach ($backend->readAhead(self::ranges(), $this->tempDir()) as $i => $fh) {
+			$result[$i] = sha1(stream_get_contents($fh));
+		}
+
+		$expected = array_map(fn (int $i): string => sha1(substr($this->files["/home/base/repo/packs/p$i"], 7, 1048576 - 7)), range(0, 5));
+		$this->assertSame($expected, $result);
+		$this->assertCount(1 + 4, $this->connections);
+		$this->assertSame(0, $this->connections[0]->sent, 'the storage\'s own connection stays free');
+		foreach (array_slice($this->connections, 1) as $c) {
+			$this->assertTrue($c->disconnected);
+		}
+	}
+
+	public function testGoesOnWithTheConnectionsTheServerAllows(): void {
+		$this->connectLimit = 3;
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, fn (): IStorage => $this->storage());
+		$backend->readAheadConnections = 4;
+
+		$count = 0;
+		foreach ($backend->readAhead(self::ranges(), $this->tempDir()) as $fh) {
+			$this->assertIsResource($fh);
+			$count++;
+		}
+
+		$this->assertSame(6, $count);
+		$this->assertCount(3, $this->connections);
+	}
+
+	public function testByDefaultOneConnectionNextToTheStoragesOwn(): void {
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, fn (): IStorage => $this->storage());
+		foreach ($backend->readAhead(self::ranges(), $this->tempDir()) as $fh) {
+			$this->assertIsResource($fh);
+		}
+
+		$this->assertSame(1, SftpRangeFetcher::DEFAULT_CONNECTIONS);
+		$this->assertCount(2, $this->connections);
+	}
+
+	public function testNoNewConnectionMeansNoReadingAhead(): void {
+		$this->connectLimit = 1;
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, fn (): IStorage => $this->storage());
+
+		$this->assertNull($backend->readAhead(self::ranges(), $this->tempDir()));
+	}
+
+	public function testWithoutAWayToConnectAgainItDoesNotReadAhead(): void {
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, null);
+
+		$this->assertFalse($backend->canReadAhead());
+		$this->assertNull($backend->readAhead(self::ranges(), $this->tempDir()));
+	}
+}
+
+interface SftpLikeRootStorage extends SftpLikeStorage {
+	public function getRoot(): string;
+}
@@ -77,6 +77,24 @@ class StorageBackendReconnectTest extends TestCase {
 		$this->assertSame(1, $backend->reconnects);
 	}

+	public function testASilentlyDroppedConnectionIsReplacedAfterAPause(): void {
+		$backend = $this->backend();
+		$this->connections[0]->dropped = true;
+		$this->connections[0]->silent = true; // phpseclib still says connected
+		self::idle($backend);
+		$this->assertTrue($backend->exists('config'));
+		$this->assertSame(1, $backend->reconnects, 'the check failed, so the connection was replaced');
+	}
+
+	public function testAFailureOnASilentlyDroppedConnectionIsRepeated(): void {
+		$backend = $this->backend();
+		$this->connections[0]->dropped = true;
+		$this->connections[0]->silent = true;
+		$fh = $backend->get('config');
+		$this->assertSame('config', stream_get_contents($fh));
+		$this->assertSame(1, $backend->reconnects);
+	}
+
 	public function testAMissingFileOnALiveConnectionIsJustMissing(): void {
 		$backend = $this->backend();
 		$this->assertFalse($backend->exists('nothing'));
@@ -121,6 +139,8 @@ interface SftpLikeStorage extends IStorage {
 /** The public phpseclib calls the backend uses to tell a dead connection. */
 final class ReconnectFakeSftp {
 	public bool $dropped = false;
+	/** Dropped without phpseclib noticing (as a Hetzner Storage Box does): it still says connected. */
+	public bool $silent = false;
 	private bool $connected = true;
 	public int $requests = 0;

@@ -139,6 +159,9 @@ final class ReconnectFakeSftp {
 	public function request(): bool {
 		$this->requests++;
 		if ($this->dropped) {
+			if ($this->silent) {
+				return false;
+			}
 			$this->connected = false;
 		}
 		return $this->connected;
@@ -0,0 +1,239 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Repository;
+
+use OCA\NgBackup\Backend\BackendException;
+use OCA\NgBackup\Backend\IBackend;
+use OCA\NgBackup\Backend\LocalBackend;
+use OCA\NgBackup\Crypto\KeyRing;
+use OCA\NgBackup\Repository\IIndexCache;
+use OCA\NgBackup\Repository\Repository;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use PHPUnit\Framework\TestCase;
+
+/**
+ * A listing of the location that comes back empty because a connection failed quietly (an SFTP
+ * connection a Storage Box dropped) must not empty the index. In a 1 TB disaster recovery test a
+ * pack read failed after 13 hours, the reload of the index got an empty listing, and the restore
+ * stopped with "Blob not in index".
+ */
+class IndexListingTest extends TestCase {
+	use TempDirTrait;
+
+	private string $root;
+	private KeyRing $keys;
+	private string $src;
+	private string $snapshot;
+
+	protected function setUp(): void {
+		$this->root = $this->tempDir();
+		$this->keys = KeyRing::generate();
+		$this->src = $this->tempDir();
+		for ($i = 0; $i < 12; $i++) {
+			file_put_contents("$this->src/f$i", random_bytes(3 * 1048576));
+		}
+		$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
+		$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
+	}
+
+	private static function cache(): IIndexCache {
+		return indexCache();
+	}
+
+	public function testARestoreGoesOnAfterAPackReadFailedAndTheListingCameBackEmpty(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$backend->failPackReads = 1;      // the read of a pack fails once
+		$backend->emptyIndexListings = 1; // and listing the index right after comes back empty
+		$target = $this->tempDir();
+
+		$this->assertSame(12, $repo->restore($this->snapshot, $target));
+
+		$this->assertSame(self::hashTree($this->src), self::hashTree($target));
+		$this->assertSame(0, $backend->failPackReads, 'the failure happened');
+		$this->assertSame(0, $backend->emptyIndexListings, 'the empty listing was seen');
+	}
+
+	public function testAPackThatStaysUnreadableIsStillReportedAsSuch(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$backend->failPackReads = 1000;
+
+		try {
+			$repo->restore($this->snapshot, $this->tempDir());
+			$this->fail('the packs cannot be read');
+		} catch (BackendException $e) {
+			$this->assertStringStartsWith('Not found: packs/', $e->getMessage());
+		}
+	}
+
+	public function testACheckWithAnEmptyListingIsAnErrorNotEveryBlobMissing(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$backend->dead = true;
+
+		$this->expectException(BackendException::class);
+		$this->expectExceptionMessage('listed no index files');
+		$repo->verify($this->snapshot);
+	}
+
+	public function testRefreshingFetchesOnlyIndexFilesItHasNotLoaded(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$index = (new \ReflectionProperty($repo, 'index'))->getValue($repo);
+		$index->downloaded = 0;
+
+		$index->refresh();
+
+		$this->assertSame(0, $index->downloaded, 'every index file is loaded already, also those whose blobs are elsewhere too');
+	}
+
+	public function testARestoreGoesOnWhenTheLocationCannotBeListedAfterAFailedRead(): void {
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		$repo = Repository::openWithKey($backend, $this->keys);
+		$backend->failPackReads = 1;
+		$backend->dead = true; // listing and config look gone; reading packs works again
+		$target = $this->tempDir();
+
+		$this->assertSame(12, $repo->restore($this->snapshot, $target));
+
+		$this->assertSame(self::hashTree($this->src), self::hashTree($target));
+	}
+
+	public function testAnEmptyListingThatWasAHiccupIsListedAgain(): void {
+		$cache = self::cache();
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		Repository::openWithKey($backend, $this->keys, $cache);
+		$cached = $cache->data;
+		$backend->emptyIndexListings = 1;
+
+		$again = Repository::openWithKey($backend, $this->keys, $cache);
+
+		$this->assertSame($cached, $cache->data);
+		$this->assertSame(12, $again->restore($this->snapshot, $this->tempDir()));
+	}
+
+	public function testAListingThatStaysEmptyWhileIndexFilesAreThereIsNotTaken(): void {
+		$cache = self::cache();
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		Repository::openWithKey($backend, $this->keys, $cache);
+		$cached = $cache->data;
+		$backend->emptyIndexListings = 5; // the config answers, the listing keeps lying
+
+		try {
+			Repository::openWithKey($backend, $this->keys, $cache);
+			$this->fail('index files are there');
+		} catch (BackendException $e) {
+			$this->assertStringContainsString('is there', $e->getMessage());
+		}
+		$this->assertSame($cached, $cache->data);
+	}
+
+	public function testARepositoryThatReallyBecameEmptyIsTakenAsEmpty(): void {
+		$cache = self::cache();
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		Repository::openWithKey($backend, $this->keys, $cache);
+		$this->assertNotSame([], array_filter($cache->data));
+		// another server forgot the last snapshot and cleaned up: no packs and index files left
+		foreach (array_merge($backend->list('index'), $backend->list('packs')) as $path) {
+			$backend->delete($path);
+		}
+
+		Repository::openWithKey($backend, $this->keys, $cache);
+
+		$this->assertSame([], array_filter($cache->data), 'the stale cache rows are gone');
+	}
+
+	public function testOpeningWithAnEmptyListingKeepsTheCachedIndex(): void {
+		$cache = self::cache();
+		$backend = new FlakyBackend(new LocalBackend($this->root));
+		Repository::openWithKey($backend, $this->keys, $cache);
+		$cached = $cache->data;
+		$this->assertNotSame([], $cached);
+
+		$backend->dead = true;
+		try {
+			Repository::openWithKey($backend, $this->keys, $cache);
+			$this->fail('an empty listing of a repository with an index must not be taken as is');
+		} catch (BackendException $e) {
+			$this->assertStringContainsString('listed no index files', $e->getMessage());
+		}
+		$this->assertSame($cached, $cache->data, 'the cache is untouched');
+	}
+}
+
+/** @internal */
+function indexCache(): IIndexCache {
+	return new class implements IIndexCache {
+		public array $data = [];
+		public function all(string $r): array { return $this->data[$r] ?? []; }
+		public function put(string $r, string $p, array $e): void { $this->data[$r][$p] = $e; }
+		public function remove(string $r, array $ps): void { foreach ($ps as $p) { unset($this->data[$r][$p]); } }
+		public function putMany(string $r, array $packs): void { foreach ($packs as $p => $e) { $this->put($r, (string)$p, $e); } }
+		public function replaceAll(string $r, array $packs): void { $this->data[$r] = []; $this->putMany($r, $packs); }
+	};
+}
+
+/** A location whose pack reads fail, and whose index listings come back empty, a set number of times. */
+final class FlakyBackend implements IBackend {
+	use \OCA\NgBackup\Backend\GetsOneByOne;
+
+	public int $failPackReads = 0;
+	public int $emptyIndexListings = 0;
+	/** The connection is gone: also the repository config looks missing. */
+	public bool $dead = false;
+
+	public function __construct(
+		private IBackend $inner,
+	) {
+	}
+
+	public function put(string $path, $stream): void {
+		$this->inner->put($path, $stream);
+	}
+
+	public function get(string $path) {
+		if (str_starts_with($path, 'packs/') && $this->failPackReads > 0) {
+			$this->failPackReads--;
+			throw new BackendException('Not found: ' . $path);
+		}
+		return $this->inner->get($path);
+	}
+
+	public function getRange(string $path, int $offset, int $length): string {
+		if (str_starts_with($path, 'packs/') && $this->failPackReads > 0) {
+			$this->failPackReads--;
+			throw new BackendException('Not found: ' . $path);
+		}
+		return $this->inner->getRange($path, $offset, $length);
+	}
+
+	public function exists(string $path): bool {
+		return !$this->dead && $this->inner->exists($path);
+	}
+
+	public function move(string $from, string $to): void {
+		$this->inner->move($from, $to);
+	}
+
+	public function list(string $prefix): array {
+		return $this->inner->list($prefix);
+	}
+
+	public function listFiles(string $dir): array {
+		if ($dir === 'index' && $this->dead) {
+			return [];
+		}
+		if ($dir === 'index' && $this->emptyIndexListings > 0) {
+			$this->emptyIndexListings--;
+			return [];
+		}
+		return $this->inner->listFiles($dir);
+	}
+
+	public function delete(string $path): void {
+		$this->inner->delete($path);
+	}
+}
@@ -0,0 +1,298 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Repository;
+
+use OCA\NgBackup\Backend\BackendException;
+use OCA\NgBackup\Backend\IBackend;
+use OCA\NgBackup\Backend\IReadsAhead;
+use OCA\NgBackup\Backend\LocalBackend;
+use OCA\NgBackup\Crypto\KeyRing;
+use OCA\NgBackup\Repository\ReadAheadPlan;
+use OCA\NgBackup\Repository\Repository;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use PHPUnit\Framework\TestCase;
+
+require_once __DIR__ . '/PackReadTest.php'; // PackOpenCountingBackend
+
+/** A full restore reads the packs ahead (IReadsAhead) and falls back to the usual reads per range. */
+class ReadAheadTest extends TestCase {
+	use TempDirTrait;
+
+	private string $root;
+	private KeyRing $keys;
+	private string $src;
+	private string $snapshot;
+
+	protected function setUp(): void {
+		$this->root = $this->tempDir();
+		$this->keys = KeyRing::generate();
+		$this->src = $this->tempDir();
+		// three packs of 32 MiB, and small files in between
+		for ($i = 0; $i < 4; $i++) {
+			file_put_contents(sprintf('%s/big%d', $this->src, $i), random_bytes(20 * 1048576));
+			for ($j = 0; $j < 20; $j++) {
+				file_put_contents(sprintf('%s/small%d-%02d', $this->src, $i, $j), random_bytes(30000));
+			}
+		}
+		file_put_contents("$this->src/same", file_get_contents("$this->src/small0-00")); // the same blob again, later
+		$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
+		$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
+	}
+
+	private function restore(ReadAheadBackend $backend, string $prefix = '', ?string $target = null): string {
+		$target ??= $this->tempDir();
+		$repo = Repository::openWithKey($backend, $this->keys, null, null);
+		$repo->restore($this->snapshot, $target, $prefix);
+		return $target;
+	}
+
+	private function assertRestored(string $target, string $prefix = ''): void {
+		$expected = array_filter(self::hashTree($this->src), fn (string $p): bool => str_starts_with($p, $prefix), ARRAY_FILTER_USE_KEY);
+		$this->assertSame($expected, self::hashTree($target));
+		$this->assertSame([], glob("$target/.ngb-restore-spool-*"), 'the spool is removed');
+	}
+
+	public function testARestoreReadsEveryPackAhead(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root));
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertSame(0, $backend->counter->packOpens(), 'nothing read the usual way');
+		$this->assertGreaterThanOrEqual(3, count($backend->asked));
+		// in the order of the files, one range per pack the restore goes through (the same blob
+		// again later is a range of its own)
+		$this->assertLessThanOrEqual(count($backend->inner->list('packs')) + 1, count($backend->asked));
+	}
+
+	public function testAPlanThatCannotBeMadeLeavesTheRestoreToTheUsualReads(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), failSnapshotReads: 1);
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertSame([], $backend->asked, 'no reading ahead without a plan');
+		$this->assertSame(2, $backend->snapshotReads, 'the restore itself read the snapshot again');
+	}
+
+	public function testTheEmptySpoolOfAStoppedRestoreIsRemoved(): void {
+		$target = $this->tempDir();
+		mkdir("$target/.ngb-restore-spool-0badc0de", 0700); // a restore that was killed
+
+		$this->assertRestored($this->restore(new ReadAheadBackend(new LocalBackend($this->root)), '', $target));
+	}
+
+	public function testARangeThatCouldNotBeReadAheadIsReadTheUsualWay(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), fail: [1]);
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertGreaterThan(0, $backend->counter->packOpens());
+	}
+
+	public function testADamagedCopyIsReadAgainTheUsualWay(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), damage: [0, 2]);
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertGreaterThan(0, $backend->counter->packOpens());
+	}
+
+	public function testAPartialRestoreReadsOnlyWhatItNeeds(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root));
+
+		$this->assertRestored($this->restore($backend, 'big1'), 'big1');
+
+		$this->assertSame(0, $backend->counter->packOpens());
+		$this->assertCount(2, $backend->asked, 'the end of one pack and the start of the next');
+		$total = array_sum(array_map(fn (array $r): int => $r[2], $backend->asked));
+		$this->assertLessThan(21 * 1048576, $total, 'not the whole packs');
+	}
+
+	public function testABackendThatCannotReadAheadNowRestoresAsBefore(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), unavailable: true);
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertGreaterThan(0, $backend->counter->packOpens());
+	}
+
+	public function testAFailingRestoreStillEndsTheReadsAheadAndRemovesTheSpool(): void {
+		// the second range fails, and so does reading it the usual way
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), fail: [1], packsGone: true);
+		$target = $this->tempDir();
+		$repo = Repository::openWithKey($backend, $this->keys, null, null);
+
+		try {
+			$repo->restore($this->snapshot, $target);
+			$this->fail('restore should fail');
+		} catch (BackendException $e) {
+			$this->assertStringStartsWith('Not found', $e->getMessage());
+		}
+
+		$this->assertTrue($backend->ended, 'the generator was ended');
+		$this->assertSame([], glob("$target/.ngb-restore-spool-*"));
+	}
+
+	public function testPlanJoinsBlobsThatFollowEachOtherInAPack(): void {
+		$plan = new ReadAheadPlan('/nowhere');
+		$plan->add('packs/aa/a', 0, 100);
+		$plan->add('packs/aa/a', 100, 100);
+		$plan->add('packs/aa/a', 50, 10); // the same blob again: inside the range
+		$plan->add('packs/aa/a', 200 + ReadAheadPlan::GAP, 100); // a gap it reads along
+		$plan->add('packs/aa/a', 300 + 3 * ReadAheadPlan::GAP, 100); // too far: a new range
+		$plan->add('packs/bb/b', 0, 100);
+		$plan->add('packs/bb/b', 0, 100); // backwards is fine as long as it is inside
+		$plan->add('packs/aa/a', 0, 100); // back to an earlier pack: a new range
+
+		$this->assertSame([
+			['packs/aa/a', 0, 300 + ReadAheadPlan::GAP],
+			['packs/aa/a', 300 + 3 * ReadAheadPlan::GAP, 100],
+			['packs/bb/b', 0, 100],
+			['packs/aa/a', 0, 100],
+		], $plan->ranges());
+	}
+
+	public function testPlanSkipsRangesWhoseBlobsWereReadElsewhere(): void {
+		$plan = new ReadAheadPlan('/nowhere');
+		foreach (range(0, 4) as $p) {
+			$plan->add("packs/0$p/p$p", 0, 10);
+		}
+		$plan->readFrom((function (): \Generator {
+			foreach (range(0, 4) as $p) {
+				$fh = fopen('php://temp', 'w+b');
+				fwrite($fh, str_repeat((string)$p, 10));
+				rewind($fh);
+				yield $p => $fh;
+			}
+		})());
+
+		$this->assertSame('00000', $plan->read('packs/00/p0', 0, 5));
+		$this->assertNull($plan->read('packs/77/moved', 0, 5), 'moved: not in the plan');
+		$this->assertSame('333', $plan->read('packs/03/p3', 2, 3), 'the ranges in between were read elsewhere');
+		$this->assertNull($plan->read('packs/01/p1', 0, 5), 'never backwards');
+		$this->assertSame('44', $plan->read('packs/04/p4', 0, 2));
+	}
+
+	public function testPlanKeepsARangeShorterThanTheMaximum(): void {
+		$plan = new ReadAheadPlan('/nowhere');
+		for ($o = 0; $o < 3 * ReadAheadPlan::MAX_RANGE; $o += 4 * 1048576) {
+			$plan->add('packs/aa/a', $o, 4 * 1048576);
+		}
+		$this->assertCount(3, $plan->ranges());
+		foreach ($plan->ranges() as $r) {
+			$this->assertLessThanOrEqual(ReadAheadPlan::MAX_RANGE, $r[2]);
+		}
+	}
+}
+
+/** Reads ahead from a local repository, one range after the other; can fail or damage ranges. */
+final class ReadAheadBackend implements IBackend, IReadsAhead {
+	public PackOpenCountingBackend $counter;
+	/** @var list<array{0:string, 1:int, 2:int}> */
+	public array $asked = [];
+	public bool $ended = false;
+
+	/**
+	 * @param list<int> $fail ranges given as null
+	 * @param list<int> $damage ranges given with a flipped byte in every blob
+	 */
+	public function __construct(
+		public LocalBackend $inner,
+		private array $fail = [],
+		private array $damage = [],
+		private bool $unavailable = false,
+		/** reading a pack the usual way fails */
+		private bool $packsGone = false,
+		/** this many reads of a snapshot fail, then they work */
+		private int $failSnapshotReads = 0,
+	) {
+		$this->counter = new PackOpenCountingBackend($inner);
+	}
+
+	public function canReadAhead(): bool {
+		return true;
+	}
+
+	public function readAhead(array $ranges, string $spool): ?\Generator {
+		if ($this->unavailable) {
+			return null;
+		}
+		\PHPUnit\Framework\Assert::assertDirectoryExists($spool);
+		$this->asked = $ranges;
+		return (function () use ($ranges): \Generator {
+			try {
+				foreach ($ranges as $i => [$path, $offset, $length]) {
+					if (in_array($i, $this->fail, true)) {
+						yield $i => null;
+						continue;
+					}
+					$data = $this->inner->getRange($path, $offset, $length);
+					if (in_array($i, $this->damage, true)) {
+						for ($k = 0; $k < strlen($data); $k += 4096) {
+							$data[$k] = chr(ord($data[$k]) ^ 1);
+						}
+					}
+					$fh = fopen('php://temp', 'w+b');
+					fwrite($fh, $data);
+					rewind($fh);
+					yield $i => $fh;
+					fclose($fh);
+				}
+			} finally {
+				$this->ended = true;
+			}
+		})();
+	}
+
+	public function put(string $path, $stream): void {
+		$this->counter->put($path, $stream);
+	}
+
+	public int $snapshotReads = 0;
+
+	public function get(string $path) {
+		if (str_starts_with($path, 'snapshots/')) {
+			$this->snapshotReads++;
+			if ($this->snapshotReads <= $this->failSnapshotReads) {
+				throw new BackendException('Connection lost reading ' . $path);
+			}
+		}
+		if ($this->packsGone && str_starts_with($path, 'packs/')) {
+			throw new BackendException('Not found: ' . $path);
+		}
+		return $this->counter->get($path);
+	}
+
+	public function getMany(array $paths): \Generator {
+		return $this->counter->getMany($paths);
+	}
+
+	public function getRange(string $path, int $offset, int $length): string {
+		return $this->counter->getRange($path, $offset, $length);
+	}
+
+	public function getRanges(array $ranges): \Generator {
+		return $this->counter->getRanges($ranges);
+	}
+
+	public function exists(string $path): bool {
+		return $this->counter->exists($path);
+	}
+
+	public function move(string $from, string $to): void {
+		$this->counter->move($from, $to);
+	}
+
+	public function list(string $prefix): array {
+		return $this->counter->list($prefix);
+	}
+
+	public function listFiles(string $dir): array {
+		return $this->counter->listFiles($dir);
+	}
+
+	public function delete(string $path): void {
+		$this->counter->delete($path);
+	}
+}
@@ -99,6 +99,20 @@ class RepositoryTest extends TestCase {
 		Repository::init($this->backend, 'pw');
 	}

+	public function testInitRefusesALocationWithBackupDataButNoConfig(): void {
+		$repo = Repository::init($this->backend, 'pw');
+		$repo->backupDirectory($this->src);
+		$this->backend->delete('config'); // missing, or not seen for a moment
+
+		try {
+			Repository::init($this->backend, 'pw');
+			$this->fail('a new repository over existing backup data');
+		} catch (RepositoryException $e) {
+			$this->assertStringContainsString('holds backup data', $e->getMessage());
+		}
+		$this->assertFalse($this->backend->exists('config'), 'no new config written');
+	}
+
 	public function testTamperedPackIsDetected(): void {
 		$repo = Repository::init($this->backend, 'pw');
 		$r = $repo->backupDirectory($this->src);
@@ -2,15 +2,27 @@
 # Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a
 # SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup
 # location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7.
-# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql]
+# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp|to-pgsql|to-mysql|to-sqlite]
 #   e.g. full-restore-roundtrip.sh 35-apache pgsql
+# With "sftp" the location is an SFTP server instead of a shared directory, and the origin has a
+# file spread over several packs: the restore reads them ahead on connections of its own.
+# With "to-<db>" the restore instance runs that database whatever the origin runs (a MySQL or
+# SQLite backup records no ID counters; after the restore new rows must not collide with restored
+# ones, whatever the database on either side).
+# With "legacy" the origin runs NG Backup 1.0.0-rc.1 (0.9.7), whose backups do not record the
+# structure of their tables: the restore must then leave the tables of the disabled app out
+# with --skip-missing-tables, as for a real backup made before 1.0.0-rc.2.
 # With pgsql or mysql the origin also has what broke real disaster recoveries: the Mail app with a synced
 # account (foreign keys between its tables) and Deck installed and then switched off (tables of
 # an app that is not enabled). The restore instance has neither; --install-apps adds Mail.
 set -euo pipefail
 IMAGE="nextcloud:${1:-35-apache}"
 DB="${2:-sqlite}"
+MODE="${3:-}"
+LEGACY=""
+[ "$MODE" = legacy ] && LEGACY=legacy
 APP=$(cd "$(dirname "$0")/../.." && pwd)
+ORIGIN_APP="$APP"
 NET=ngb-frt-net
 SHARED=$(mktemp -d)
 chmod 777 "$SHARED"
@@ -18,14 +30,22 @@ fail() { echo "FAIL: $*"; exit 1; }
 ok() { echo "ok   $*"; }

 cleanup() {
-	docker rm -fv ngb-frt-origin ngb-frt-restore ngb-frt-db-origin ngb-frt-db-restore ngb-frt-imap >/dev/null 2>&1 || true
+	docker rm -fv ngb-frt-origin ngb-frt-restore ngb-frt-db-origin ngb-frt-db-restore ngb-frt-imap ngb-frt-sftp >/dev/null 2>&1 || true
 	docker network rm "$NET" >/dev/null 2>&1 || true
 	docker run --rm -v "$SHARED:/target" alpine sh -c 'rm -rf /target/*' >/dev/null 2>&1 || true
 	rmdir "$SHARED" 2>/dev/null || true
 	rm -f /tmp/ngb-frt-kit.json /tmp/ngb-frt-pass
+	[ -n "${LEGACY_DIR:-}" ] && rm -rf "$LEGACY_DIR"
+	return 0
 }
 trap cleanup EXIT

+if [ "$LEGACY" = legacy ]; then
+	LEGACY_DIR=$(mktemp -d)
+	curl -fsSL https://github.com/dillardblom/ng_backup/releases/download/v1.0.0-rc.1/ng_backup-0.9.7.tar.gz | tar -xz -C "$LEGACY_DIR"
+	ORIGIN_APP="$LEGACY_DIR/ng_backup"
+fi
+
 run_on() { docker exec -u www-data "$1" php occ "${@:2}"; }

 # Database settings for an instance: SQLite inside the container, or its own PostgreSQL container
@@ -46,27 +66,37 @@ db_env() {
 }

 docker network create "$NET" >/dev/null
+LOCATION=(local -a null::null -o datadir=/mnt/shared)
+if [ "$MODE" = sftp ]; then
+	docker run -d --name ngb-frt-sftp --network "$NET" atmoz/sftp:alpine ngbackup:ngbackuppw:::upload >/dev/null
+	LOCATION=(sftp -a password::password -o host=ngb-frt-sftp -o root=/upload -o user=ngbackup -o password=ngbackuppw)
+fi
 ORIGIN_DB=$(db_env origin)
 # shellcheck disable=SC2086
 docker run -d --name ngb-frt-origin --network "$NET" $ORIGIN_DB \
 	-e NEXTCLOUD_ADMIN_USER=admin -e NEXTCLOUD_ADMIN_PASSWORD=admin \
-	-v "$APP:/var/www/html/custom_apps/ng_backup:ro" -v "$SHARED:/mnt/shared" "$IMAGE" >/dev/null
+	-v "$ORIGIN_APP:/var/www/html/custom_apps/ng_backup:ro" -v "$SHARED:/mnt/shared" "$IMAGE" >/dev/null
 for i in $(seq 1 90); do run_on ngb-frt-origin status 2>/dev/null | grep -q "installed: true" && break; sleep 3; done
 docker exec ngb-frt-origin chown www-data:www-data /var/www/html/custom_apps
 run_on ngb-frt-origin app:enable ng_backup >/dev/null
-ok "origin instance up ($DB)"
+ok "origin instance up ($DB${LEGACY:+, NG Backup 0.9.7})"

 docker exec ngb-frt-origin sh -c "printf 'full restore roundtrip passphrase 2026\n' > /tmp/pass"
 run_on ngb-frt-origin backup:key:init --passphrase-file=/tmp/pass --label=CI >/dev/null || fail "key init"
 run_on ngb-frt-origin backup:key:kit --output=/tmp/kit.json >/dev/null
 CODE=$(run_on ngb-frt-origin backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
 run_on ngb-frt-origin backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "confirm"
-run_on ngb-frt-origin backup:target:add offsite local -a null::null -o datadir=/mnt/shared --path=repo >/dev/null || fail "target add"
+run_on ngb-frt-origin backup:target:add offsite "${LOCATION[@]}" --path=repo >/dev/null || fail "target add"
 ok "key, kit and location on the origin"

 NC_PASS=$(openssl rand -hex 16)
 docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
 docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
+if [ "$MODE" = sftp ]; then
+	# about three packs of 32 MiB
+	docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
+	LARGE_SUM=$(docker exec ngb-frt-origin sha256sum /var/www/html/data/frtest/files/Docs/large.bin | cut -d' ' -f1)
+fi
 run_on ngb-frt-origin files:scan frtest >/dev/null

 # SQL on an instance's own database (pgsql/mysql only)
@@ -100,6 +130,8 @@ ok "test data backed up, snapshot $SNAP"

 docker cp ngb-frt-origin:/tmp/kit.json /tmp/ngb-frt-kit.json
 docker exec ngb-frt-origin sh -c "cat /tmp/pass" > /tmp/ngb-frt-pass
+ORIGIN_DBTYPE=$DB
+case "$MODE" in to-*) DB=${MODE#to-} ;; esac
 RESTORE_DB=$(db_env restore)
 # shellcheck disable=SC2086
 docker run -d --name ngb-frt-restore --network "$NET" $RESTORE_DB \
@@ -114,16 +146,36 @@ docker cp /tmp/ngb-frt-pass ngb-frt-restore:/tmp/pass
 ok "fresh restore instance up"

 run_on ngb-frt-restore backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass >/dev/null || fail "key import"
-run_on ngb-frt-restore backup:target:add offsite local -a null::null -o datadir=/mnt/shared --path=repo >/dev/null || fail "target add on the restore instance"
-run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location"
+ADD_OUT=$(run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo 2>&1) || { printf '%s\n' "$ADD_OUT"; fail "target add on the restore instance"; }
+LIST_OUT=$(run_on ngb-frt-restore backup:list offsite 2>&1)
+printf '%s\n' "$LIST_OUT" | grep -q "$SNAP" || {
+	printf 'target:add said:\n%s\nbackup:list said:\n%s\n' "$ADD_OUT" "$LIST_OUT"
+	docker exec ngb-frt-restore sh -c 'grep -h ng_backup /var/www/html/data/nextcloud.log | tail -5' || true
+	docker exec ngb-frt-restore sh -c 'ls -la /mnt/shared/repo /mnt/shared/repo/snapshots' || true
+	fail "the origin's snapshot is not listed after connecting the location"
+}
 ok "recovery kit imported, existing repository adopted, snapshot visible"

-OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction --install-apps 2>&1) || {
+RESTORE_OPTS=(--no-interaction --install-apps)
+if [ "$LEGACY" = legacy ] && [ "$DB" != sqlite ]; then
+	# Without --skip-missing-tables an older backup must be refused, naming the app's tables
+	REFUSED=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS[@]}" 2>&1) \
+		&& fail "an older backup with tables of unknown structure was restored without --skip-missing-tables"
+	printf '%s\n' "$REFUSED" | grep -q "oc_deck_boards" || { printf '%s\n' "$REFUSED" | tail -20; fail "the refusal does not name the missing tables"; }
+	ok "older backup refused without --skip-missing-tables, missing tables named"
+	RESTORE_OPTS+=(--skip-missing-tables)
+fi
+sftp_logins() { docker logs ngb-frt-sftp 2>&1 | grep -c "Accepted password" || true; }
+[ "$MODE" = sftp ] && LOGINS_BEFORE=$(sftp_logins)
+OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS[@]}" 2>&1) || {
 	printf '%s\n' "$OUT" | tail -40
 	docker exec ngb-frt-restore sh -c 'tail -c 6000 /var/www/html/data/nextcloud.log' || true
 	fail "restore:full"
 }
-ok "backup:restore:full completed"
+printf '%s\n' "$OUT" | grep -q "maintenance mode while the restore runs" || fail "the restore did not say it runs in maintenance mode"
+run_on ngb-frt-restore status 2>&1 | grep -q "maintenance: false" || fail "maintenance mode is still on after the restore"
+run_on ngb-frt-restore config:app:get ng_backup restore_full_started >/dev/null 2>&1 && fail "the restore left its 'started' marker behind"
+ok "backup:restore:full completed, in maintenance mode and out of it again"

 # Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
 # earlier bugs here only showed up after config.php's secret was rewritten mid-restore.
@@ -132,9 +184,23 @@ run_on ngb-frt-restore user:list | grep -q frtest || fail "the backed-up user is
 docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
 [ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
 run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
+# New rows after the restore: a new file (file cache) and a new user must not collide with
+# restored rows (ID counters after the data).
+docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"
+run_on ngb-frt-restore files:scan frtest >/dev/null || fail "a new file cannot be added after the restore"
+if [ "$DB" != sqlite ]; then
+	[ "$(sql restore "select count(*) from oc_filecache where path = 'files/Docs/after-restore.txt'")" = 1 ] \
+		|| fail "a new file did not make it into the file cache after the restore (ID counters?)"
+fi
+docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-restore php occ user:add --password-from-env frnew >/dev/null || fail "a new user cannot be added after the restore"
 ok "database, files and ng_backup's own state are all intact and usable after the restore"

-if [ "$DB" != sqlite ]; then
+# Mail and Deck were only set up on an origin with a database server (see above)
+if [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ] && [ "$LEGACY" = legacy ]; then
+	[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 0 ] || fail "a table of unknown structure was created"
+	[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
+	ok "older backup: tables of the disabled app left out, Mail restored"
+elif [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ]; then
 	[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created"
 	[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
 	[ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \
@@ -143,7 +209,16 @@ if [ "$DB" != sqlite ]; then
 	ok "tables of a disabled app created, Mail restored with its foreign keys"
 fi

+if [ "$MODE" = sftp ]; then
+	[ "$(docker exec ngb-frt-restore sha256sum /var/www/html/data/frtest/files/Docs/large.bin | cut -d' ' -f1)" = "$LARGE_SUM" ] \
+		|| fail "the large file differs after the restore"
+	LOGINS=$(( $(sftp_logins) - LOGINS_BEFORE ))
+	[ "$LOGINS" -gt 1 ] || fail "the restore did not read ahead on connections of its own ($LOGINS SFTP logins)"
+	docker exec ngb-frt-restore sh -c 'ls -a /var/www/html/data' | grep -q ngb-restore-spool && fail "the spool folder was left behind"
+	ok "over SFTP: read ahead on connections of its own ($LOGINS SFTP logins during the restore), large file identical, no spool left"
+fi
+
 run_on ngb-frt-restore backup:run offsite -l post-restore-check >/dev/null || fail "a new backup after the restore fails"
 ok "the restored instance can run a new backup of its own"

-echo "FULL RESTORE ROUNDTRIP PASS ($DB)"
+echo "FULL RESTORE ROUNDTRIP PASS ($ORIGIN_DBTYPE${MODE:+, $MODE})"
Author	SHA1	Message	Date
Dillard Blom	f0537f0f2c	Merge pull request #18 from dillardblom/rc5 CI / unit (8.2) (push) Has been cancelled Details CI / unit (8.3) (push) Has been cancelled Details CI / unit (8.4) (push) Has been cancelled Details CI / psalm (push) Has been cancelled Details CI / integration (mariadb, 32) (push) Has been cancelled Details CI / integration (mariadb, 33) (push) Has been cancelled Details CI / integration (mariadb, 34) (push) Has been cancelled Details CI / integration (mariadb, 35) (push) Has been cancelled Details CI / integration (pgsql, 32) (push) Has been cancelled Details CI / integration (pgsql, 33) (push) Has been cancelled Details CI / integration (pgsql, 34) (push) Has been cancelled Details CI / integration (pgsql, 35) (push) Has been cancelled Details CI / integration (sqlite, 32) (push) Has been cancelled Details CI / integration (sqlite, 33) (push) Has been cancelled Details CI / integration (sqlite, 34) (push) Has been cancelled Details CI / integration (sqlite, 35) (push) Has been cancelled Details CI / integration-targets (push) Has been cancelled Details CI / integration-user (32) (push) Has been cancelled Details CI / integration-user (33) (push) Has been cancelled Details CI / integration-user (34) (push) Has been cancelled Details CI / integration-user (35) (push) Has been cancelled Details CI / integration-full-restore (push) Has been cancelled Details Release 1.0.0-rc.5	2026-10-22 02:28:05 +02:00
dillard	fd0f786f5f	Full restore: a run again after an interrupted restore keeps maintenance mode on when its check refuses; retry text about apps	2026-10-22 01:59:01 +02:00
dillard	2eabdf44d5	README and store description: background jobs must run on system cron, with how to set it up; rc5 date	2026-10-22 01:03:52 +02:00
dillard	61e2a28b67	Full restore: maintenance mode on before the checks (off again when they refuse); the printed retry command is complete	2026-10-22 00:53:08 +02:00
dillard	7ed53ad0eb	Full restore: after an error maintenance mode stays on, with the one line to switch it off and run again; maintenance that was on before stays on	2026-10-22 00:36:58 +02:00
dillard	bdcb4f0cae	Full restore runs in maintenance mode, switched off again at the end (also after an error)	2026-10-22 00:31:34 +02:00
dillard	2616b6e543	Init guard also counts snapshots in the trash and the catalog as backup data	2026-10-21 22:42:43 +02:00
dillard	9bc4e4a4ab	Never create a repository where backup data lies but no config could be read	2026-10-21 22:25:48 +02:00
dillard	827e46381e	Log a snapshot that cannot be adopted when a location is added; full-restore test prints what target:add and backup:list said when the snapshot is missing	2026-10-21 22:08:14 +02:00
dillard	8eb6c4c033	Index: check known index files spread over the list	2026-10-21 20:56:45 +02:00
dillard	d623416306	Index: an empty listing is taken only when known index files are gone too	2026-10-21 20:49:32 +02:00
dillard	95184dadc7	Index: an empty listing is checked (config present, listed once more) before it is taken, so a repository another server emptied is accepted and a hiccup recovers	2026-10-21 19:51:46 +02:00
dillard	9b6bd7d2ab	Release 1.0.0-rc.5 (app version 0.9.11)	2026-10-21 13:10:03 +02:00
dillard	2fcd38f478	Full-restore test: Mail and Deck checks only when the origin had them (not on a SQLite origin)	2026-10-21 13:09:54 +02:00
dillard	177361acae	Index load keeps the index when it refuses an empty listing; counters of unchanged tables are repaired too, never moved backwards	2026-10-21 09:08:50 +02:00
dillard	85ea0acc50	Full-restore test: restore onto another database (to-pgsql, to-mysql, to-sqlite), every crossing in CI	2026-10-20 23:10:25 +02:00
dillard	ac57e31c93	Full restore onto PostgreSQL: counters of restored tables go after their data when the backup records none (MySQL, SQLite); test MariaDB onto PostgreSQL	2026-10-20 23:04:53 +02:00
dillard	e461c48056	CI: back to Nextcloud 35 for the full-restore test (the failures were a slow app store, not the version)	2026-10-19 22:18:33 +02:00
dillard	d504067cd8	Index: an empty listing is an error in checks and cleanups too; refresh() fetches only index files it has not loaded; full-restore CI on Nextcloud 35.0.0	2026-10-19 22:13:26 +02:00
dillard	c37b26ea38	Full restore: an empty listing never empties the index; read a failed pack once more; recognise and replace SFTP connections a Storage Box dropped quietly; version 0.9.11	2026-10-19 21:53:05 +02:00
Dillard Blom	0accd5a3f2	Merge pull request #16 from dillardblom/rc4 CI / unit (8.2) (push) Has been cancelled Details CI / unit (8.3) (push) Has been cancelled Details CI / unit (8.4) (push) Has been cancelled Details CI / psalm (push) Has been cancelled Details CI / integration (mariadb, 32) (push) Has been cancelled Details CI / integration (mariadb, 33) (push) Has been cancelled Details CI / integration (mariadb, 34) (push) Has been cancelled Details CI / integration (mariadb, 35) (push) Has been cancelled Details CI / integration (pgsql, 32) (push) Has been cancelled Details CI / integration (pgsql, 33) (push) Has been cancelled Details CI / integration (pgsql, 34) (push) Has been cancelled Details CI / integration (pgsql, 35) (push) Has been cancelled Details CI / integration (sqlite, 32) (push) Has been cancelled Details CI / integration (sqlite, 33) (push) Has been cancelled Details CI / integration (sqlite, 34) (push) Has been cancelled Details CI / integration (sqlite, 35) (push) Has been cancelled Details CI / integration-targets (push) Has been cancelled Details CI / integration-user (32) (push) Has been cancelled Details CI / integration-user (33) (push) Has been cancelled Details CI / integration-user (34) (push) Has been cancelled Details CI / integration-user (35) (push) Has been cancelled Details CI / integration-full-restore (push) Has been cancelled Details Release 1.0.0-rc.4	2026-10-19 11:04:15 +02:00
dillard	583592f69b	Release 1.0.0-rc.4 (app version 0.9.10)	2026-10-19 10:34:01 +02:00
dillard	a384ab41bb	App store description: Ensembia line	2026-10-19 10:31:21 +02:00
dillard	9b85221e50	Read ahead: a plan that cannot be made leaves the restore to the usual reads; only a whole buffered SFTP packet counts as an answer waiting	2026-10-19 08:54:13 +02:00
dillard	c95f552cc6	Read ahead over SFTP: one connection with 32 requests in flight by default, more with restore_sftp_connections; a Storage Box gets slower with parallel readers	2026-10-19 07:46:12 +02:00
dillard	6121f684e9	Full restore: say to run occ upgrade before rerunning a restore that stopped after the database; remove empty spool folders of a stopped restore	2026-10-19 05:34:54 +02:00
dillard	43a71684f9	Read ahead over SFTP: take answers phpseclib has read already instead of waiting up to 10 s on the socket	2026-10-19 05:28:26 +02:00
dillard	88b719879d	Release candidate 4 (app version 0.9.10): full restore over SFTP reads ahead; SFTP leg in the full-restore test	2026-10-19 03:51:21 +02:00
dillard	1d7f7e6423	Read ahead: resolve paths before opening connections; a failing readAhead() leaves no spool folder	2026-10-19 01:52:58 +02:00
dillard	ccbbf01c54	Full restore over SFTP: read packs ahead on four connections at once A restore read one pack at a time over one SFTP connection, which over a long line is bound by the round trip and by what one TCP connection gets: about 2-3 MiB/s from a Hetzner Storage Box to Canada, with PHP mostly waiting. Repository::restore() now first lists the ranges of packs the restore needs, in order (ReadAheadPlan), and the location reads them ahead into unlinked files in a spool folder in the target (IReadsAhead). Over SFTP, SftpRangeFetcher does that on four connections of its own, 16 requests in flight each, taking the answers of whichever connection has one. A range that fails, a damaged copy, or a blob that moved since the plan is read the usual way, which reports what is wrong. Other locations restore as before. On a test line with 120 ms RTT and 0.2% loss, a 1.6 GiB restore went from 5.1 to 17.8 MiB/s, all files identical.	2026-10-19 01:40:49 +02:00
dillard	b789d17636	Screenshot URLs per release tag, so the app store fetches new images	2026-10-18 23:47:32 +02:00
dillard	8e5c8ccda4	Full restore: skip the ID counters of tables left out with --skip-missing-tables; test a backup made by rc.1; version 0.9.9	2026-10-18 23:44:00 +02:00
dillard	e0b79939f9	PLAN: 1.0.0 only reliability; 1.0.1 restore speed, cron warning, DR test with table structure	2026-10-18 23:19:18 +02:00