Merge pull request #16 from dillardblom/rc4

Release 1.0.0-rc.4

.github/workflows/ci.yml

@@ -108,3 +108,9 @@ jobs:
         run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql
       - name: Full restore round trip on MariaDB, with Mail (foreign keys) and a disabled app
         run: tests/integration/full-restore-roundtrip.sh 35-apache mysql
+      - name: Full restore of a backup made by 1.0.0-rc.1 (no table structure) on PostgreSQL
+        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql legacy
+      - name: Full restore of a backup made by 1.0.0-rc.1 (no table structure) on MariaDB
+        run: tests/integration/full-restore-roundtrip.sh 35-apache mysql legacy
+      - name: Full restore over SFTP on PostgreSQL, reading packs ahead on several connections
+        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql sftp

CHANGELOG.md

@@ -7,7 +7,29 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
 beta/daily update channel, which would hide the app from most admins. The stability stage is
 instead carried by the app description, the git tag, the release, and this file.
 
-## [Unreleased]
+## [1.0.0-rc.4] - 2026-10-19
+
+Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not
+released: its fixes are in here.
+
+### Changed
+- A full restore over SFTP reads the packs ahead on a connection of its own, 32 requests in
+  flight, into temporary files in the restore folder. From a Hetzner Storage Box to a server in
+  Canada (122 ms) packs now come in at about 14 MiB/s instead of 2.8. More connections can be set
+  with `restore_sftp_connections` (README). A part that cannot be read this way is read as
+  before. Restores from other locations work as before.
+- App store description: the line about Ensembia.
+
+### Fixed
+- After a full restore stopped halfway, running it again with `--force` was refused when the
+  database was already restored: Nextcloud then first wants `occ upgrade`. The guide and the
+  error message now say so.
+- A full restore of a backup made before 1.0.0-rc.2 with `--skip-missing-tables` failed on
+  PostgreSQL with "relation ... does not exist": the left-out tables were still looked up to set
+  their ID counters. It now skips them. A new test restores a backup made by 1.0.0-rc.1 on
+  PostgreSQL and MariaDB.
+- The app store showed old screenshots: it keeps a copy per URL, and the URLs always pointed at
+  `main`. They now point at the release's tag.
 
 ## [1.0.0-rc.2] - 2026-10-18
 

PLAN.md

@@ -383,6 +383,23 @@ in README "Known limitations":
   de gebruiker, wachtwoordzin min. 16 tekens). Geen nieuwe functies meer.
 - **Vóór 1.0.0 (verplicht):** volledige restore met kit op een tweede server (PVE02), alleen lezend
   of op een kopie van de locatie; plus een user-restore. Zonder geslaagde test geen 1.0.0.
+- **1.0.0 (besluit 2026-10-18):** alleen betrouwbaarheid. rc2 uitgebracht; slaagt de grote
+  DR-test op VM 980 (snapshot van 12-10, met `--skip-missing-tables`), dan 1.0.0. Zo niet: fix,
+  rc3, en rc3 pas uitbrengen na een geslaagde volledige DR-test.
+- **rc4 (besluit 2026-10-19):** rc3 is niet uitgebracht. De DR-test met rc3 liep ~2,8 MiB/s
+  (~3,5 dag); parallel lezen over SFTP haalde vanaf VM 980 ~7,6 MiB/s. Daarom rc4 = rc3 +
+  vooruitlezen over SFTP (4 eigen verbindingen). Volgorde: volledige DR-test op VM 980 volgens de
+  gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een
+  productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0.
+  Problemen of meldingen van gebruikers: rc5.
+- **1.0.1:**
+  1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
+     restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
+  2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron
+     staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue
+     #13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles.
+  3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later),
+     zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest.
 - **1.1.0:**
   1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
      als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van

README.md

@@ -181,6 +181,10 @@ Contributions welcome via PR — add what you ran into with your own provider.
   server (or the VM/container host where TCP ends) to BBR:
   `sysctl -w net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr`
   (`modprobe tcp_bbr` first; make it permanent in `/etc/sysctl.d/`).
+  A full restore over SFTP reads the packs ahead on a second connection, with many requests in
+  flight. A server that handles parallel reads well (not a Hetzner Storage Box, which gets slower)
+  can be read on more connections: `occ config:app:set ng_backup restore_sftp_connections
+  --value=4` (at most 8; the server must allow that many plus one at once for the user).
 
 - **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself
   (e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the

appinfo/info.xml

@@ -39,9 +39,9 @@ All-in-One and hosted Nextcloud.
 Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very
 welcome: use the "Share your experience" link on the admin page.
 
-Developed and sponsored by [Ensembia](https://www.ensembia.com), which also offers paid help with
+Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
-restoring a backup after a crash.]]></description>
+you need help with restoring a backup after a crash.]]></description>
-    <version>0.9.8</version>
+    <version>0.9.10</version>
     <licence>agpl</licence>
     <author mail="dillard.blom@ensembia.com">Dillard Blom</author>
     <namespace>NgBackup</namespace>
@@ -52,9 +52,9 @@ restoring a backup after a crash.]]></description>
     <website>https://github.com/dillardblom/ng_backup</website>
     <bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
     <repository type="git">https://github.com/dillardblom/ng_backup</repository>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/1-backups.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/1-backups.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/2-restore.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/2-restore.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/3-settings.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/3-settings.png</screenshot>
     <dependencies>
         <php min-version="8.1"/>
         <lib>sodium</lib>

docs/DISASTER-RECOVERY.md

@@ -95,6 +95,8 @@ out with `--skip-missing-tables`.
 
 **8. Let the restore run.** It restores the database, then the data directory, and takes hours for
 a large backup. If it stops halfway, fix the cause and run the same command again with `--force`.
+If occ then says that Nextcloud or one of the apps requires an upgrade, the database is already
+the old server's: run `occ upgrade` first, then the same command with `--force`.
 
 **9. After the restore.** These worked in our tests:
 

lib/Backend/IReadsAhead.php

@@ -0,0 +1,30 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Backend;
+
+/**
+ * A backend that can read many byte ranges ahead of use, several at a time over connections of
+ * its own (a full restore, where one connection at a time is bound by the round trip).
+ */
+interface IReadsAhead {
+	/** Whether readAhead() is worth planning for: false when it would return null anyway. */
+	public function canReadAhead(): bool;
+
+	/**
+	 * The ranges in the order given, each as a local stream at offset 0 holding its bytes, read a
+	 * few ranges ahead of the one the caller is at. A stream is closed when the generator is
+	 * resumed. Null for a range that could not be read this way (a missing object, a server or
+	 * connection error): the caller reads it the usual way, which reports why.
+	 *
+	 * @param list<array{0:string, 1:int, 2:int}> $ranges [path, offset, length] each
+	 * @param string $spool an existing local directory for the files that hold the ranges
+	 * @return \Generator<int, resource|null>|null by position in $ranges; null when this backend
+	 *                                             cannot read ahead now
+	 */
+	public function readAhead(array $ranges, string $spool): ?\Generator;
+}

lib/Backend/SftpRangeFetcher.php

@@ -0,0 +1,366 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Backend;
+
+/**
+ * Reads byte ranges of files on an SFTP location ahead of use, over several connections at once,
+ * into local files (StorageBackend::readAhead(), for a full restore).
+ *
+ * A restore that reads its packs as it goes is bound by the round trip: from a Hetzner Storage
+ * Box to Canada (RTT 122 ms) it reached about 2.8 MiB/s, with the PHP process mostly waiting.
+ * Here every connection reads its own range with DEPTH requests in flight, ahead of the restore. PHP runs one thing at a time: while it waits for an answer on one connection, the
+ * answers of the others arrive in the kernel's buffers, so the connections still transfer at the
+ * same time.
+ *
+ * Uses the same internal phpseclib 2 methods as SftpReadStream (see SftpReadStream::connection()).
+ * The connections are this class's own, never the storage's: no other request goes over them.
+ * A range that fails (missing file, server error, a connection that breaks) comes out as null,
+ * for the caller to read the usual way; a broken connection is not used again.
+ */
+final class SftpRangeFetcher {
+	/**
+	 * Connections a full restore uses, unless the app setting restore_sftp_connections says
+	 * otherwise. One: a Hetzner Storage Box gave about 14 MiB/s on one connection with DEPTH
+	 * requests in flight, and 4-6 MiB/s on 2 or 4 (it reads from disks, and parallel readers make
+	 * them seek). On a line with packet loss to a server that handles parallel reads well, 4
+	 * connections did better (21 against 14 MiB/s).
+	 */
+	public const DEFAULT_CONNECTIONS = 1;
+	public const MAX_CONNECTIONS = 8;
+	/** READ requests in flight per connection. */
+	public const DEPTH = 32;
+	/** Ranges read ahead of the one the caller is at, in flight or waiting in the spool. */
+	public const AHEAD = 8;
+	/**
+	 * Seconds ready() waits for any connection to answer before it waits for each in turn: short,
+	 * in case a connection holds an answer where ready() does not look.
+	 */
+	private const SELECT_TIMEOUT = 1;
+	/** See SftpReadStream::CHUNK and MIN_CHUNK. */
+	private const CHUNK = 261120;
+	private const MIN_CHUNK = 32768;
+
+	private const FXP_OPEN = 3;
+	private const FXP_CLOSE = 4;
+	private const FXP_READ = 5;
+	private const FXP_STATUS = 101;
+	private const FXP_HANDLE = 102;
+	private const FXP_DATA = 103;
+	private const FXF_READ = 1;
+	private const FX_OK = 0;
+
+	private const OPEN = 0;
+	private const READ = 1;
+	private const CLOSE = 2;
+
+	/** @var array<int, int> READ size per connection, lowered when a server answers less than asked */
+	private array $chunks = [];
+	/** @var array<int, SftpRangeJob> the range each busy connection reads */
+	private array $jobs = [];
+	/** @var array<int, resource|null> ranges read (or failed) and not yet given to the caller */
+	private array $done = [];
+	private int $requestId = 0x4e520000;
+
+	/**
+	 * @param array<int, object> $connections phpseclib 2 connections with their SFTP channel open
+	 * @param list<array{0:string, 1:int, 2:int}> $ranges paths below $root
+	 */
+	private function __construct(
+		private array $connections,
+		private string $root,
+		private array $ranges,
+		private string $spool,
+	) {
+		foreach (array_keys($connections) as $c) {
+			$this->chunks[$c] = self::CHUNK;
+		}
+	}
+
+	/**
+	 * See IReadsAhead::readAhead(). The connections are ended when the generator ends.
+	 *
+	 * @param list<object> $connections
+	 * @param list<array{0:string, 1:int, 2:int}> $ranges
+	 * @return \Generator<int, resource|null>
+	 */
+	public static function fetch(array $connections, string $root, array $ranges, string $spool): \Generator {
+		$fetcher = new self($connections, rtrim($root, '/'), $ranges, $spool);
+		try {
+			yield from $fetcher->run();
+		} finally {
+			$fetcher->end();
+		}
+	}
+
+	/** @return \Generator<int, resource|null> */
+	private function run(): \Generator {
+		$count = count($this->ranges);
+		$next = 0;
+		for ($want = 0; $want < $count; $want++) {
+			while (!array_key_exists($want, $this->done)) {
+				foreach (array_keys($this->connections) as $c) {
+					if (!isset($this->jobs[$c]) && $next < $count && $next < $want + self::AHEAD) {
+						$this->start($c, $next++);
+					}
+				}
+				if ($this->jobs === [] && !array_key_exists($want, $this->done)) {
+					// no connection left: the rest the usual way
+					$this->done[$want] = null;
+					$next = max($next, $want + 1);
+					break;
+				}
+				foreach ($this->ready() as $c) {
+					$this->step($c);
+				}
+			}
+			$fh = $this->done[$want];
+			unset($this->done[$want]);
+			try {
+				yield $want => $fh;
+			} finally {
+				if (is_resource($fh)) {
+					fclose($fh);
+				}
+			}
+		}
+	}
+
+	/**
+	 * The busy connections that have an answer waiting, so that a connection that waits for a
+	 * lost packet to be sent again does not hold up the others; all of them when the connections
+	 * do not tell (then they are taken in turn).
+	 *
+	 * @return list<int>
+	 */
+	private function ready(): array {
+		$sockets = [];
+		$ready = [];
+		foreach ($this->jobs as $c => $job) {
+			$sftp = $this->connections[$c];
+			$vars = get_object_vars($sftp); // only what is public: phpseclib 2's are
+			// Answers phpseclib has read already: the socket does not show them. Its request
+			// buffer holds whole answers read while it waited for another; its packet buffer and
+			// channel buffers hold what came in with the last packet it read (small answers, such
+			// as a CLOSE and the next OPEN, often arrive together).
+			$buffered = $vars['requestBuffer'] ?? null;
+			$channels = $vars['channel_buffers'] ?? null;
+			if ((is_array($buffered) && isset($job->queue[0]) && array_key_exists($job->queue[0][0], $buffered))
+				|| (is_string($vars['packet_buffer'] ?? null) && self::wholePacket($vars['packet_buffer']))
+				|| (is_array($channels) && array_filter($channels) !== [])) {
+				$ready[] = $c;
+				continue;
+			}
+			$socket = $vars['fsock'] ?? null;
+			if (!is_resource($socket)) {
+				return array_keys($this->jobs);
+			}
+			$sockets[$c] = $socket;
+		}
+		if ($ready !== [] || $sockets === []) {
+			return $ready;
+		}
+		$read = $sockets;
+		$write = null;
+		$except = null;
+		// Data in PHP's own buffer of a socket counts as readable. Nothing within the time:
+		// each in turn, where phpseclib's own timeout applies.
+		$n = @stream_select($read, $write, $except, self::SELECT_TIMEOUT);
+		return $n === false || $n === 0 ? array_keys($this->jobs) : array_keys($read);
+	}
+
+	/** Whether $buffer starts with a whole SFTP packet (length, then that many bytes). */
+	private static function wholePacket(string $buffer): bool {
+		$length = strlen($buffer) >= 4 ? unpack('N', $buffer) : false;
+		return is_array($length) && strlen($buffer) >= 4 + $length[1];
+	}
+
+	private function start(int $c, int $i): void {
+		[$path, $offset, $length] = $this->ranges[$i];
+		$fh = $this->spoolFile();
+		if ($fh === null) {
+			$this->done[$i] = null;
+			return;
+		}
+		if ($length <= 0) {
+			$this->done[$i] = $fh;
+			return;
+		}
+		$this->jobs[$c] = new SftpRangeJob($i, $fh, $offset, $offset + $length);
+		try {
+			$remote = $this->connections[$c]->_realpath($this->root . '/' . ltrim($path, '/'));
+			if (!is_string($remote)) {
+				$this->finish($c);
+				return;
+			}
+			$this->send($c, self::FXP_OPEN, pack('Na*N2', strlen($remote), $remote, self::FXF_READ, 0), self::OPEN);
+		} catch (\Throwable) {
+			$this->drop($c);
+		}
+	}
+
+	/** Take the oldest answer on connection $c, then keep its requests in flight. */
+	private function step(int $c): void {
+		$job = $this->jobs[$c];
+		[$id, $kind, $offset, $length] = array_shift($job->queue) ?? throw new \LogicException('no request in flight');
+		$sftp = $this->connections[$c];
+		try {
+			$response = $sftp->_get_sftp_packet($id);
+			if (!is_string($response)) {
+				throw new BackendException('SFTP connection lost');
+			}
+			$type = $sftp->packet_type;
+			if ($kind === self::OPEN) {
+				if ($type === self::FXP_HANDLE) {
+					$job->handle = substr($response, 4);
+				} else {
+					$job->failed = true; // missing, or no permission
+				}
+			} elseif ($kind === self::READ) {
+				if (!$job->failed) {
+					$this->received($c, $type, $response, $offset, $length);
+				}
+			} else {
+				$status = $type === self::FXP_STATUS && strlen($response) >= 4 ? unpack('N', $response) : false;
+				$closed = is_array($status) && $status[1] === self::FX_OK;
+				$this->finish($c);
+				if (!$closed) {
+					// handles might pile up on the server: not used again
+					$this->drop($c);
+				}
+				return;
+			}
+			$this->refill($c);
+		} catch (\Throwable) {
+			$this->drop($c);
+		}
+	}
+
+	private function received(int $c, int $type, string $response, int $offset, int $length): void {
+		$job = $this->jobs[$c];
+		$declared = strlen($response) >= 4 ? unpack('N', $response) : false;
+		$n = is_array($declared) ? $declared[1] : 0;
+		if ($type !== self::FXP_DATA || $n <= 0 || $n !== strlen($response) - 4 || $n > $length) {
+			// end of file before the end of the range, a server error or a malformed answer
+			$job->failed = true;
+			return;
+		}
+		if (fseek($job->fh, $offset - $job->start) !== 0 || fwrite($job->fh, substr($response, 4)) !== $n) {
+			$job->failed = true;
+			return;
+		}
+		$job->got += $n;
+		if ($n < $length) {
+			// a server that caps a READ: ask for the rest, and no more than it gives from now on
+			$job->redo[] = [$offset + $n, $length - $n];
+			$this->chunks[$c] = max(self::MIN_CHUNK, min($this->chunks[$c], $n));
+		}
+	}
+
+	/** Send READs until DEPTH are in flight; when nothing is left to ask for, close the file. */
+	private function refill(int $c): void {
+		$job = $this->jobs[$c];
+		if ($job->handle !== '' && !$job->failed) {
+			while (count($job->queue) < self::DEPTH) {
+				if ($job->redo !== []) {
+					[$offset, $length] = array_shift($job->redo);
+				} elseif ($job->next < $job->end) {
+					$offset = $job->next;
+					$length = min($this->chunks[$c], $job->end - $offset);
+					$job->next += $length;
+				} else {
+					break;
+				}
+				$packet = pack('Na*N3', strlen($job->handle), $job->handle, intdiv($offset, 4294967296), $offset % 4294967296, $length);
+				$this->send($c, self::FXP_READ, $packet, self::READ, $offset, $length);
+			}
+		}
+		if ($job->queue !== []) {
+			return;
+		}
+		if ($job->handle === '') {
+			$this->finish($c); // not opened: nothing to close
+			return;
+		}
+		$this->send($c, self::FXP_CLOSE, pack('Na*', strlen($job->handle), $job->handle), self::CLOSE);
+	}
+
+	private function send(int $c, int $type, string $packet, int $kind, int $offset = 0, int $length = 0): void {
+		$this->requestId = $this->requestId >= 0x7fffffff ? 0x4e520000 : $this->requestId + 1;
+		if (!$this->connections[$c]->_send_sftp_packet($type, $packet, $this->requestId)) {
+			throw new BackendException('SFTP connection failed while sending a request');
+		}
+		$this->jobs[$c]->queue[] = [$this->requestId, $kind, $offset, $length];
+	}
+
+	/** The range of connection $c is done: its stream for the caller, or null when it failed. */
+	private function finish(int $c): void {
+		$job = $this->jobs[$c];
+		unset($this->jobs[$c]);
+		if ($job->failed || $job->got !== $job->end - $job->start || !rewind($job->fh)) {
+			self::closeFile($job->fh);
+			$this->done[$job->i] = null;
+			return;
+		}
+		$this->done[$job->i] = $job->fh;
+	}
+
+	/** A connection in an unknown state: its range fails, and it is not used again. */
+	private function drop(int $c): void {
+		if (isset($this->jobs[$c])) {
+			$job = $this->jobs[$c];
+			unset($this->jobs[$c]);
+			self::closeFile($job->fh);
+			$this->done[$job->i] = null;
+		}
+		self::disconnect($this->connections[$c]);
+		unset($this->connections[$c]);
+	}
+
+	/** @return resource|null a new file in the spool, already unlinked: nothing is left behind if the process dies */
+	private function spoolFile() {
+		$path = $this->spool . '/' . bin2hex(random_bytes(8));
+		$fh = @fopen($path, 'x+b');
+		if ($fh === false) {
+			return null;
+		}
+		@unlink($path);
+		return $fh;
+	}
+
+	private function end(): void {
+		foreach ($this->jobs as $job) {
+			self::closeFile($job->fh);
+		}
+		foreach ($this->done as $fh) {
+			if (is_resource($fh)) {
+				fclose($fh);
+			}
+		}
+		$this->jobs = [];
+		$this->done = [];
+		foreach ($this->connections as $sftp) {
+			self::disconnect($sftp);
+		}
+		$this->connections = [];
+	}
+
+	private static function closeFile(mixed $fh): void {
+		if (is_resource($fh)) {
+			fclose($fh);
+		}
+	}
+
+	private static function disconnect(object $sftp): void {
+		try {
+			if (is_callable([$sftp, 'disconnect'])) {
+				$sftp->disconnect();
+			}
+		} catch (\Throwable) {
+		}
+	}
+}

lib/Backend/SftpRangeJob.php

@@ -0,0 +1,40 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Backend;
+
+/**
+ * One range that SftpRangeFetcher reads on one connection.
+ *
+ * @internal
+ */
+final class SftpRangeJob {
+	public string $handle = '';
+	/** Offset of the next READ to send. */
+	public int $next;
+	/** @var list<array{int, int}> offset and length of parts to ask for again (after a short answer) */
+	public array $redo = [];
+	/** @var list<array{int, int, int, int}> request id, kind, offset and length, in the order sent */
+	public array $queue = [];
+	/** Bytes received. */
+	public int $got = 0;
+	public bool $failed = false;
+
+	/** @var resource the spool file that holds the range */
+	public $fh;
+
+	/** @param resource $fh */
+	public function __construct(
+		public int $i,
+		$fh,
+		public int $start,
+		public int $end,
+	) {
+		$this->fh = $fh;
+		$this->next = $start;
+	}
+}

lib/Backend/SftpReadStream.php

@@ -144,9 +144,9 @@ final class SftpReadStream {
 
 	/**
 	 * The storage's phpseclib connection, with its SFTP channel open, when it offers what these
-	 * streams need; null otherwise.
+	 * streams (and SftpRangeFetcher) need; null otherwise.
 	 */
-	private static function connection(object $storage): ?object {
+	public static function connection(object $storage): ?object {
 		if (!is_callable([$storage, 'getConnection']) || !is_callable([$storage, 'getRoot'])) {
 			return null;
 		}

lib/Backend/StorageBackend.php

@@ -14,7 +14,7 @@ use OCP\Files\Storage\IWriteStreamStorage;
  * IBackend on top of a Nextcloud storage object (from files_external or core), below a base
  * directory on that storage.
  */
-final class StorageBackend implements IBackend {
+final class StorageBackend implements IBackend, IReadsAhead {
 	/** Object stores write whole objects atomically; other storages get a temp name + rename. */
 	private bool $atomicPut;
 	/** SFTP: read through SftpReadStream, which keeps several requests in flight. */
@@ -29,6 +29,8 @@ final class StorageBackend implements IBackend {
 	/** Seconds without a request after which an SFTP connection is checked before it is used. */
 	public const IDLE_CHECK = 60;
 	private float $lastUsed;
+	/** Connections readAhead() opens (see SftpRangeFetcher::DEFAULT_CONNECTIONS). */
+	public int $readAheadConnections = SftpRangeFetcher::DEFAULT_CONNECTIONS;
 	/** Times the storage was built anew after its connection died (for tests and diagnostics). */
 	public int $reconnects = 0;
 
@@ -243,6 +245,53 @@ final class StorageBackend implements IBackend {
 		}
 	}
 
+	/**
+	 * Over SFTP: SftpRangeFetcher, on readAheadConnections new connections of its own (built with the
+	 * reconnect function), so the storage's connection stays free for other requests. Elsewhere
+	 * not (yet): null.
+	 */
+	public function canReadAhead(): bool {
+		return $this->sftpReads && $this->reconnect !== null;
+	}
+
+	public function readAhead(array $ranges, string $spool): ?\Generator {
+		$reconnect = $this->reconnect;
+		if (!$this->sftpReads || $reconnect === null || $ranges === []) {
+			return null;
+		}
+		// before any connection is opened: an invalid path throws
+		$abs = array_map(fn (array $r): array => [$this->abs($r[0]), $r[1], $r[2]], $ranges);
+		$storages = [];
+		$connections = [];
+		$root = '';
+		$wanted = max(1, min(SftpRangeFetcher::MAX_CONNECTIONS, $this->readAheadConnections));
+		for ($n = 0; $n < $wanted; $n++) {
+			try {
+				$storage = $reconnect();
+				$sftp = SftpReadStream::connection($storage);
+				if ($sftp === null || !is_callable([$storage, 'getRoot'])) {
+					break;
+				}
+				$root = (string)$storage->getRoot();
+			} catch (\Throwable) {
+				break; // the server may allow fewer connections: go on with those that work
+			}
+			$storages[] = $storage;
+			$connections[] = $sftp;
+		}
+		if ($connections === []) {
+			return null;
+		}
+		return (static function () use ($storages, $connections, $root, $abs, $spool): \Generator {
+			// the storages hold their connections: kept until the fetcher is done
+			try {
+				yield from SftpRangeFetcher::fetch($connections, $root, $abs, $spool);
+			} finally {
+				unset($storages);
+			}
+		})();
+	}
+
 	public function getRange(string $path, int $offset, int $length): string {
 		return $this->call(fn () => $this->readRange($path, $offset, $length));
 	}

lib/Command/RestoreFull.php

@@ -92,6 +92,7 @@ HELP);
 		} catch (\Throwable $e) {
 			$output->writeln('<error>' . $e->getMessage() . '</error>');
 			$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
+			$output->writeln('If occ then says that Nextcloud needs an upgrade (the database is already the old server\'s), run occ upgrade first.');
 			return 1;
 		}
 		$output->writeln(sprintf(

lib/Db/DbDumper.php

@@ -216,7 +216,7 @@ final class DbDumper {
 				}
 				$report['restored'][] = $table;
 			}
-			$this->restoreSequences($manifest['sequences'] ?? []);
+			$this->restoreSequences($manifest['sequences'] ?? [], $report['missing']);
 			$this->resumeForeignKeys($foreignKeys);
 			$this->db->commit();
 			// Foreign keys of the tables created above, now their rows are in. Not inside the
@@ -573,11 +573,16 @@ final class DbDumper {
 		return $name;
 	}
 
-	private function restoreSequences(array $sequences): void {
+	/** @param list<string> $leftOut tables not restored (--skip-missing-tables): they do not exist here */
+	private function restoreSequences(array $sequences, array $leftOut = []): void {
 		if (!$this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\PostgreSQLPlatform) {
 			return;
 		}
+		$leftOut = array_flip($leftOut);
 		foreach ($sequences as $info) {
+			if (isset($leftOut[$info['table']])) {
+				continue;
+			}
 			$seq = $this->db->fetchOne('SELECT pg_get_serial_sequence(quote_ident(?), ?)', [$info['table'], $info['column']]);
 			if ($seq === false || $seq === null) {
 				continue;

lib/Repository/PackReaderGuard.php

@@ -8,9 +8,10 @@ declare(strict_types=1);
 namespace OCA\NgBackup\Repository;
 
 use OCA\NgBackup\Backend\IBackend;
+use OCA\NgBackup\Backend\IReadsAhead;
 
 /** The location as Repository sees it: closes the PackReader's open stream before every other request. */
-final class PackReaderGuard implements IBackend {
+final class PackReaderGuard implements IBackend, IReadsAhead {
 	public function __construct(
 		private IBackend $inner,
 		private PackReader $reader,
@@ -43,6 +44,15 @@ final class PackReaderGuard implements IBackend {
 		return $this->inner->getRanges($ranges);
 	}
 
+	public function canReadAhead(): bool {
+		return $this->inner instanceof IReadsAhead && $this->inner->canReadAhead();
+	}
+
+	/** Over connections of the location's own: the open stream may stay open. */
+	public function readAhead(array $ranges, string $spool): ?\Generator {
+		return $this->inner instanceof IReadsAhead ? $this->inner->readAhead($ranges, $spool) : null;
+	}
+
 	public function exists(string $path): bool {
 		$this->reader->close();
 		return $this->inner->exists($path);

lib/Repository/ReadAheadPlan.php

@@ -0,0 +1,123 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Repository;
+
+/**
+ * The ranges of packs a restore reads, in order, for IReadsAhead::readAhead(), and the reading of
+ * blobs from them as the restore goes.
+ *
+ * add() takes the blobs in the order the restore needs them. A blob in the same pack as the range
+ * before it, not before that range and at most GAP after its end, extends that range; anything
+ * else starts a new one. read() takes the same blobs in the same order: a blob is in the current
+ * range or in the next one, which then becomes current; a blob not in the plan is read elsewhere.
+ */
+final class ReadAheadPlan {
+	/** Unused bytes between two blobs of a pack that are read along rather than asked for apart. */
+	public const GAP = 4 * 1048576;
+	/** Longest range: packs are 32 MiB, repacked ones too. */
+	public const MAX_RANGE = 64 * 1048576;
+	/** Ranges after the current one that read() looks in for a blob. */
+	private const SEARCH = 16;
+
+	/** @var list<array{0:string, 1:int, 2:int}> [path, offset, length] */
+	private array $ranges = [];
+	/** @var \Generator<int, resource|null>|null */
+	private ?\Generator $streams = null;
+	private int $current = -1;
+	/** @var resource|null */
+	private $stream = null;
+
+	/** @param string $spool local folder for the ranges read ahead (see IReadsAhead::readAhead()) */
+	public function __construct(
+		public readonly string $spool,
+	) {
+	}
+
+	public function add(string $path, int $offset, int $length): void {
+		$last = array_pop($this->ranges);
+		if ($last !== null) {
+			[$p, $start, $len] = $last;
+			if ($p === $path && $offset >= $start && $offset <= $start + $len + self::GAP
+				&& $offset + $length - $start <= self::MAX_RANGE) {
+				$this->ranges[] = [$p, $start, max($len, $offset + $length - $start)];
+				return;
+			}
+			$this->ranges[] = $last;
+		}
+		$this->ranges[] = [$path, $offset, $length];
+	}
+
+	/** @return list<array{0:string, 1:int, 2:int}> */
+	public function ranges(): array {
+		return $this->ranges;
+	}
+
+	/** @param \Generator<int, resource|null> $streams what IReadsAhead::readAhead() gave for ranges() */
+	public function readFrom(\Generator $streams): void {
+		$this->streams = $streams;
+	}
+
+	/** The bytes of a blob, or null when its range could not be read ahead (read it the usual way). */
+	public function read(string $path, int $offset, int $length): ?string {
+		if ($this->streams === null) {
+			return null;
+		}
+		if (!$this->contains($this->current, $path, $offset, $length)) {
+			// Normally the next range. Further on when blobs before it were read elsewhere: the
+			// index was reloaded after another server's cleanup moved them to new packs.
+			$to = null;
+			for ($i = $this->current + 1; $i <= $this->current + self::SEARCH && $to === null; $i++) {
+				if ($this->contains($i, $path, $offset, $length)) {
+					$to = $i;
+				}
+			}
+			if ($to === null) {
+				return null; // not where the plan has it
+			}
+			if ($this->current < 0) {
+				$this->streams->current();
+				$this->current = 0;
+			}
+			while ($this->current < $to) {
+				$this->streams->next();
+				$this->current++;
+			}
+			$stream = $this->streams->valid() ? $this->streams->current() : null;
+			$this->stream = is_resource($stream) ? $stream : null;
+		}
+		$fh = $this->stream;
+		if ($fh === null || fseek($fh, $offset - $this->ranges[$this->current][1]) !== 0) {
+			return null;
+		}
+		$data = '';
+		while (strlen($data) < $length) {
+			$chunk = fread($fh, $length - strlen($data));
+			if ($chunk === false || $chunk === '') {
+				return null;
+			}
+			$data .= $chunk;
+		}
+		return $data;
+	}
+
+	/** Ends the reads ahead, also halfway: closes their streams and connections. */
+	public function close(): void {
+		$this->stream = null;
+		$streams = $this->streams;
+		$this->streams = null;
+		unset($streams); // the generator's finally blocks run when it is destroyed
+	}
+
+	private function contains(int $i, string $path, int $offset, int $length): bool {
+		if ($i < 0 || !isset($this->ranges[$i])) {
+			return false;
+		}
+		[$p, $start, $len] = $this->ranges[$i];
+		return $p === $path && $offset >= $start && $offset + $length <= $start + $len;
+	}
+}

lib/Repository/Repository.php

@@ -9,6 +9,7 @@ namespace OCA\NgBackup\Repository;
 
 use OCA\NgBackup\Backend\BackendException;
 use OCA\NgBackup\Backend\IBackend;
+use OCA\NgBackup\Backend\IReadsAhead;
 use OCA\NgBackup\Crypto\KeyRing;
 use OCA\NgBackup\Crypto\StreamCipher;
 
@@ -216,6 +217,66 @@ final class Repository {
 		if ($prefix !== '' && !self::isSafePath(rtrim($prefix, '/'))) {
 			throw new RepositoryException('Invalid path prefix');
 		}
+		$plan = $this->readAhead($snapshotId, $prefix, $target);
+		try {
+			return $this->restoreEntries($snapshotId, $target, $prefix, $stripPrefix, $heartbeat, $plan);
+		} finally {
+			$plan?->close();
+			if ($plan !== null) {
+				@rmdir($plan->spool);
+			}
+		}
+	}
+
+	/**
+	 * Start reading the packs of a restore ahead, when the location can (IReadsAhead): a first
+	 * pass over the snapshot lists the ranges of packs it needs, in order. The spool is a
+	 * folder in the target, which has room for the restore anyway.
+	 */
+	private function readAhead(string $snapshotId, string $prefix, string $target): ?ReadAheadPlan {
+		if (!$this->backend instanceof IReadsAhead || !$this->backend->canReadAhead()) {
+			return null;
+		}
+		// what a restore that was stopped left behind: empty, its files were unlinked when opened
+		$stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR);
+		foreach ($stale === false ? [] : $stale as $dir) {
+			@rmdir($dir);
+		}
+		$plan = new ReadAheadPlan(rtrim($target, '/') . '/.ngb-restore-spool-' . bin2hex(random_bytes(4)));
+		try {
+			foreach ($this->streamEntries($snapshotId) as $entry) {
+				if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
+					continue;
+				}
+				foreach ($entry['b'] as $id) {
+					try {
+						[$pack, $offset, $length] = $this->index->get($id);
+					} catch (RepositoryException) {
+						continue; // the restore reports it when it gets there
+					}
+					$plan->add('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
+				}
+			}
+		} catch (\Throwable) {
+			return null; // reading ahead is only faster: the restore itself reads the snapshot again and reports what is wrong
+		}
+		if (count($plan->ranges()) < 2 || (!is_dir($plan->spool) && !@mkdir($plan->spool, 0700, true))) {
+			return null;
+		}
+		try {
+			$streams = $this->backend->readAhead($plan->ranges(), $plan->spool);
+		} catch (\Throwable) {
+			$streams = null; // the usual reads report what is wrong
+		}
+		if ($streams === null) {
+			@rmdir($plan->spool);
+			return null;
+		}
+		$plan->readFrom($streams);
+		return $plan;
+	}
+
+	private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan): int {
 		$count = 0;
 		foreach ($this->streamEntries($snapshotId) as $entry) {
 			if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
@@ -240,7 +301,7 @@ final class Repository {
 				throw new RepositoryException('Cannot create a temporary file for ' . $entry['p']);
 			}
 			foreach ($entry['b'] as $id) {
-				fwrite($out, $this->loadBlob($id));
+				fwrite($out, $this->restoreBlob($id, $plan));
 				if ($heartbeat !== null) {
 					$heartbeat();
 				}
@@ -789,6 +850,22 @@ final class Repository {
 		}
 	}
 
+	/** A blob for restore(): read ahead when it could be, otherwise (or when that copy is bad) the usual way. */
+	private function restoreBlob(string $id, ?ReadAheadPlan $plan): string {
+		if ($plan !== null) {
+			[$pack, $offset, $length] = $this->index->get($id);
+			$encrypted = $plan->read('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
+			if ($encrypted !== null) {
+				try {
+					return $this->decodeBlob($id, $encrypted);
+				} catch (\Exception) {
+					// read again from the location below, which reports what is wrong
+				}
+			}
+		}
+		return $this->loadBlob($id);
+	}
+
 	private function loadBlob(string $id): string {
 		[$pack, $offset, $length] = $this->index->get($id);
 		try {

lib/Service/TargetService.php

@@ -287,7 +287,10 @@ final class TargetService {
 
 	private function buildBackend(string $backend, string $auth, array $options, string $basePath, bool $createBase = true): StorageBackend {
 		$create = fn (): IStorage => $this->factory()->create($backend, $auth, $options);
-		return new StorageBackend($create(), $basePath, $createBase, $create);
+		$storage = new StorageBackend($create(), $basePath, $createBase, $create);
+		$storage->readAheadConnections = Server::get(\OCP\IAppConfig::class)
+			->getValueInt('ng_backup', 'restore_sftp_connections', \OCA\NgBackup\Backend\SftpRangeFetcher::DEFAULT_CONNECTIONS);
+		return $storage;
 	}
 
 	private ?ExternalStorageFactory $factory = null;

tests/Unit/Backend/SftpRangeFetcherTest.php

@@ -0,0 +1,208 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Backend;
+
+use OCA\NgBackup\Backend\SftpRangeFetcher;
+use OCA\NgBackup\Backend\SftpReadStream;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use PHPUnit\Framework\TestCase;
+
+require_once __DIR__ . '/SftpReadStreamTest.php'; // FakeSftpConnection, FakeSftpStorage
+
+/** SftpRangeFetcher against the fake SFTP server of SftpReadStreamTest, several connections at once. */
+class SftpRangeFetcherTest extends TestCase {
+	use TempDirTrait;
+
+	/** @var array<string, string> */
+	private array $files = [];
+
+	protected function setUp(): void {
+		SftpReadStream::reset();
+		for ($i = 0; $i < 10; $i++) {
+			$this->files["/home/repo/packs/p$i"] = random_bytes(3 * 1048576 + $i * 1000);
+		}
+	}
+
+	/** @return list<FakeSftpConnection> */
+	private function connections(int $n, array $options = []): array {
+		$result = [];
+		for ($c = 0; $c < $n; $c++) {
+			$result[] = new FakeSftpConnection($this->files, ...($options[$c] ?? []));
+		}
+		return $result;
+	}
+
+	/** @return list<object> what StorageBackend passes on: each with its SFTP channel open */
+	private static function opened(array $connections): array {
+		return array_map(fn (FakeSftpConnection $c): object => SftpReadStream::connection(new FakeSftpStorage($c)), $connections);
+	}
+
+	/** @return list<array{0:string, 1:int, 2:int}> */
+	private function ranges(): array {
+		$ranges = [];
+		for ($i = 0; $i < 10; $i++) {
+			$ranges[] = ["repo/packs/p$i", 1000 * $i, 2 * 1048576 + 333];
+		}
+		return $ranges;
+	}
+
+	/** @return array<int, string|null> sha1 of the contents by position (a failing comparison of MiBs would not fit in memory) */
+	private static function collect(\Generator $streams): array {
+		$result = [];
+		foreach ($streams as $i => $fh) {
+			$result[$i] = $fh === null ? null : sha1(stream_get_contents($fh));
+		}
+		return $result;
+	}
+
+	private function expected(array $ranges): array {
+		return array_map(fn (array $r): string => sha1(substr($this->files['/home/' . $r[0]], $r[1], $r[2])), $ranges);
+	}
+
+	public function testReadsEveryRangeInOrderOverAllConnections(): void {
+		$connections = $this->connections(4);
+		$ranges = $this->ranges();
+		$spool = $this->tempDir();
+
+		$this->assertSame($this->expected($ranges), self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $spool)));
+
+		foreach ($connections as $c) {
+			$this->assertGreaterThan(0, count($c->reads), 'every connection reads');
+			$this->assertLessThanOrEqual(SftpRangeFetcher::DEPTH, $c->maxInFlight);
+			$this->assertSame([], $c->openHandles);
+			$this->assertSame(0, $c->pending());
+			$this->assertTrue($c->disconnected, 'its own connections end with it');
+		}
+		$this->assertSame([], glob("$spool/*"), 'nothing left in the spool');
+	}
+
+	public function testKeepsSeveralRequestsInFlightOnEachConnection(): void {
+		$this->files['/home/repo/packs/big'] = random_bytes(10 * 1048576);
+		$connections = $this->connections(2);
+		self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', [['repo/packs/big', 0, 10 * 1048576], ['repo/packs/big', 5, 10 * 1048576 - 5]], $this->tempDir()));
+
+		$this->assertSame(SftpRangeFetcher::DEPTH, $connections[0]->maxInFlight);
+		$this->assertSame(SftpRangeFetcher::DEPTH, $connections[1]->maxInFlight);
+	}
+
+	public function testAnAnswerPhpseclibHasReadAlreadyIsTakenWithoutWaitingOnTheSocket(): void {
+		// The fake's answers are all "read already": its socket never becomes readable, and
+		// phpseclib's packet buffer holds data. Waiting on the socket would take a second a step.
+		$connections = $this->connections(2);
+		$sockets = [];
+		foreach ($connections as $c) {
+			$pair = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, STREAM_IPPROTO_IP);
+			$sockets[] = $pair;
+			$c->fsock = $pair[0];
+			$c->packet_buffer = pack('N', 4) . 'more'; // a whole packet
+		}
+		$ranges = array_slice($this->ranges(), 0, 3);
+		$start = microtime(true);
+
+		$this->assertSame($this->expected($ranges), self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir())));
+
+		$this->assertLessThan(1.0, microtime(true) - $start);
+	}
+
+	public function testAServerThatCapsAReadStillGivesEveryByte(): void {
+		$connections = $this->connections(3, array_fill(0, 3, ['cap' => 100000]));
+		$ranges = $this->ranges();
+
+		$this->assertSame($this->expected($ranges), self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir())));
+		// learned: after the first short answers (and asking for their rest) it asks for no more than the server gives
+		foreach ($connections as $c) {
+			$this->assertLessThanOrEqual(2 * SftpRangeFetcher::DEPTH, count(array_filter($c->reads, fn (int $n): bool => $n > 100000)));
+		}
+	}
+
+	public function testAMissingFileIsLeftToTheCaller(): void {
+		$connections = $this->connections(2);
+		$ranges = $this->ranges();
+		$ranges[3][0] = 'repo/packs/gone';
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir()));
+
+		$expected = $this->expected($this->ranges());
+		$expected[3] = null;
+		$this->assertSame($expected, $result);
+		$this->assertSame([], $connections[0]->openHandles);
+	}
+
+	public function testARangePastTheEndOfTheFileIsLeftToTheCaller(): void {
+		$ranges = $this->ranges();
+		$ranges[5][2] = 10 * 1048576;
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($this->connections(2)), '/home/', $ranges, $this->tempDir()));
+
+		$this->assertNull($result[5]);
+		$this->assertSame($this->expected($this->ranges())[6], $result[6]);
+	}
+
+	public function testAServerErrorFailsOnlyThatRange(): void {
+		// every read from 1 MiB on fails on connection 0: its ranges, not those of the others
+		$connections = $this->connections(2, [['failAt' => 1048576]]);
+		$ranges = $this->ranges();
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir()));
+
+		$expected = $this->expected($ranges);
+		$failed = array_keys(array_filter($result, fn (?string $r): bool => $r === null));
+		$this->assertNotSame([], $failed);
+		foreach ($result as $i => $data) {
+			if ($data !== null) {
+				$this->assertSame($expected[$i], $data);
+			}
+		}
+		$this->assertLessThan(10, count($failed), 'the other connection goes on');
+		$this->assertSame([], $connections[0]->openHandles, 'a failed range is closed too');
+	}
+
+	public function testALostConnectionIsNotUsedAgainAndTheOthersGoOn(): void {
+		$connections = $this->connections(3, [['loseAt' => 1048576]]);
+		$ranges = $this->ranges();
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir()));
+
+		$expected = $this->expected($ranges);
+		$this->assertSame(1, count(array_filter($result, fn (?string $r): bool => $r === null)), 'only the range it was reading');
+		foreach ($result as $i => $data) {
+			if ($data !== null) {
+				$this->assertSame($expected[$i], $data);
+			}
+		}
+		$this->assertTrue($connections[0]->disconnected);
+		$this->assertLessThanOrEqual(1048576 / 261120 + SftpRangeFetcher::DEPTH + 1, count($connections[0]->reads), 'not used after it broke');
+	}
+
+	public function testWhenEveryConnectionIsLostTheRestIsLeftToTheCaller(): void {
+		$connections = $this->connections(2, [['throwAt' => 1048576], ['loseAt' => 1048576]]);
+
+		$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $this->ranges(), $this->tempDir()));
+
+		$this->assertSame(array_fill(0, 10, null), $result);
+	}
+
+	public function testStoppingHalfwayEndsTheConnectionsAndLeavesNothingBehind(): void {
+		$connections = $this->connections(2);
+		$spool = $this->tempDir();
+		$streams = SftpRangeFetcher::fetch(self::opened($connections), '/home/', $this->ranges(), $spool);
+		$this->assertIsResource($streams->current());
+
+		unset($streams);
+
+		$this->assertTrue($connections[0]->disconnected);
+		$this->assertTrue($connections[1]->disconnected);
+		$this->assertSame([], glob("$spool/*"));
+	}
+
+	public function testReadsNoMoreThanAheadOfTheCaller(): void {
+		$connections = $this->connections(4);
+		$streams = SftpRangeFetcher::fetch(self::opened($connections), '/home/', $this->ranges(), $this->tempDir());
+		$streams->current();
+
+		$opened = array_sum(array_map(fn (FakeSftpConnection $c): int => count(array_unique(array_values($c->opened))), $connections));
+		$this->assertLessThanOrEqual(SftpRangeFetcher::AHEAD, $opened);
+	}
+}

tests/Unit/Backend/SftpReadStreamTest.php

@@ -466,6 +466,12 @@ final class FakeSftpConnection {
 	public array $reads = [];
 	/** @var array<string, string> */
 	public array $openHandles = [];
+	/** @var list<string> paths of every OPEN request */
+	public array $opened = [];
+	/** @var resource|null like phpseclib's socket (SftpRangeFetcher waits on it) */
+	public $fsock = null;
+	/** like phpseclib's: what came in with the last packet and is not read yet */
+	public string $packet_buffer = '';
 	/** Whether the SFTP channel is open: phpseclib 2 opens it on the first SFTP call. */
 	public bool $initialised = false;
 	/** Packets sent */
@@ -532,6 +538,7 @@ final class FakeSftpConnection {
 					return false;
 				}
 				$path = substr($data, 4, unpack('N', $data)[1]);
+				$this->opened[] = $path;
 				if (!isset($this->files[$path])) {
 					$this->answers[$id] = [101, pack('N', 2)];
 					break;

tests/Unit/Backend/StorageBackendReadAheadTest.php

@@ -0,0 +1,115 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Backend;
+
+use OCA\NgBackup\Backend\SftpRangeFetcher;
+use OCA\NgBackup\Backend\SftpReadStream;
+use OCA\NgBackup\Backend\StorageBackend;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use OCP\Files\Storage\IStorage;
+use PHPUnit\Framework\TestCase;
+
+require_once __DIR__ . '/SftpReadStreamTest.php'; // FakeSftpConnection
+require_once __DIR__ . '/StorageBackendReconnectTest.php'; // SftpLikeStorage
+
+/** StorageBackend::readAhead(): over SFTP on connections of its own, built with the reconnect function. */
+class StorageBackendReadAheadTest extends TestCase {
+	use TempDirTrait;
+
+	/** @var list<FakeSftpConnection> */
+	private array $connections = [];
+	/** @var array<string, string> */
+	private array $files = [];
+	/** New storages after this many fail to connect. */
+	private ?int $connectLimit = null;
+
+	protected function setUp(): void {
+		SftpReadStream::reset();
+		for ($i = 0; $i < 6; $i++) {
+			$this->files["/home/base/repo/packs/p$i"] = random_bytes(1048576 + $i);
+		}
+	}
+
+	private function storage(): IStorage {
+		if ($this->connectLimit !== null && count($this->connections) >= $this->connectLimit) {
+			throw new \RuntimeException('Too many connections');
+		}
+		$sftp = new FakeSftpConnection($this->files);
+		$this->connections[] = $sftp;
+		$storage = $this->createMock(SftpLikeRootStorage::class);
+		$storage->method('instanceOfStorage')->willReturnCallback(fn (string $class): bool => $class === \OCA\Files_External\Lib\Storage\SFTP::class);
+		$storage->method('getConnection')->willReturn($sftp);
+		$storage->method('getRoot')->willReturn('/home/');
+		$storage->method('is_dir')->willReturn(true);
+		return $storage;
+	}
+
+	/** @return list<array{0:string, 1:int, 2:int}> */
+	private static function ranges(): array {
+		return array_map(fn (int $i): array => ["packs/p$i", 7, 1048576 - 7], range(0, 5));
+	}
+
+	public function testReadsAheadOnNewConnectionsOnly(): void {
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, fn (): IStorage => $this->storage());
+		$backend->readAheadConnections = 4;
+		$this->assertTrue($backend->canReadAhead());
+
+		$result = [];
+		foreach ($backend->readAhead(self::ranges(), $this->tempDir()) as $i => $fh) {
+			$result[$i] = sha1(stream_get_contents($fh));
+		}
+
+		$expected = array_map(fn (int $i): string => sha1(substr($this->files["/home/base/repo/packs/p$i"], 7, 1048576 - 7)), range(0, 5));
+		$this->assertSame($expected, $result);
+		$this->assertCount(1 + 4, $this->connections);
+		$this->assertSame(0, $this->connections[0]->sent, 'the storage\'s own connection stays free');
+		foreach (array_slice($this->connections, 1) as $c) {
+			$this->assertTrue($c->disconnected);
+		}
+	}
+
+	public function testGoesOnWithTheConnectionsTheServerAllows(): void {
+		$this->connectLimit = 3;
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, fn (): IStorage => $this->storage());
+		$backend->readAheadConnections = 4;
+
+		$count = 0;
+		foreach ($backend->readAhead(self::ranges(), $this->tempDir()) as $fh) {
+			$this->assertIsResource($fh);
+			$count++;
+		}
+
+		$this->assertSame(6, $count);
+		$this->assertCount(3, $this->connections);
+	}
+
+	public function testByDefaultOneConnectionNextToTheStoragesOwn(): void {
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, fn (): IStorage => $this->storage());
+		foreach ($backend->readAhead(self::ranges(), $this->tempDir()) as $fh) {
+			$this->assertIsResource($fh);
+		}
+
+		$this->assertSame(1, SftpRangeFetcher::DEFAULT_CONNECTIONS);
+		$this->assertCount(2, $this->connections);
+	}
+
+	public function testNoNewConnectionMeansNoReadingAhead(): void {
+		$this->connectLimit = 1;
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, fn (): IStorage => $this->storage());
+
+		$this->assertNull($backend->readAhead(self::ranges(), $this->tempDir()));
+	}
+
+	public function testWithoutAWayToConnectAgainItDoesNotReadAhead(): void {
+		$backend = new StorageBackend($this->storage(), 'base/repo', false, null);
+
+		$this->assertFalse($backend->canReadAhead());
+		$this->assertNull($backend->readAhead(self::ranges(), $this->tempDir()));
+	}
+}
+
+interface SftpLikeRootStorage extends SftpLikeStorage {
+	public function getRoot(): string;
+}

tests/Unit/Repository/ReadAheadTest.php

@@ -0,0 +1,298 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Repository;
+
+use OCA\NgBackup\Backend\BackendException;
+use OCA\NgBackup\Backend\IBackend;
+use OCA\NgBackup\Backend\IReadsAhead;
+use OCA\NgBackup\Backend\LocalBackend;
+use OCA\NgBackup\Crypto\KeyRing;
+use OCA\NgBackup\Repository\ReadAheadPlan;
+use OCA\NgBackup\Repository\Repository;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use PHPUnit\Framework\TestCase;
+
+require_once __DIR__ . '/PackReadTest.php'; // PackOpenCountingBackend
+
+/** A full restore reads the packs ahead (IReadsAhead) and falls back to the usual reads per range. */
+class ReadAheadTest extends TestCase {
+	use TempDirTrait;
+
+	private string $root;
+	private KeyRing $keys;
+	private string $src;
+	private string $snapshot;
+
+	protected function setUp(): void {
+		$this->root = $this->tempDir();
+		$this->keys = KeyRing::generate();
+		$this->src = $this->tempDir();
+		// three packs of 32 MiB, and small files in between
+		for ($i = 0; $i < 4; $i++) {
+			file_put_contents(sprintf('%s/big%d', $this->src, $i), random_bytes(20 * 1048576));
+			for ($j = 0; $j < 20; $j++) {
+				file_put_contents(sprintf('%s/small%d-%02d', $this->src, $i, $j), random_bytes(30000));
+			}
+		}
+		file_put_contents("$this->src/same", file_get_contents("$this->src/small0-00")); // the same blob again, later
+		$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
+		$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
+	}
+
+	private function restore(ReadAheadBackend $backend, string $prefix = '', ?string $target = null): string {
+		$target ??= $this->tempDir();
+		$repo = Repository::openWithKey($backend, $this->keys, null, null);
+		$repo->restore($this->snapshot, $target, $prefix);
+		return $target;
+	}
+
+	private function assertRestored(string $target, string $prefix = ''): void {
+		$expected = array_filter(self::hashTree($this->src), fn (string $p): bool => str_starts_with($p, $prefix), ARRAY_FILTER_USE_KEY);
+		$this->assertSame($expected, self::hashTree($target));
+		$this->assertSame([], glob("$target/.ngb-restore-spool-*"), 'the spool is removed');
+	}
+
+	public function testARestoreReadsEveryPackAhead(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root));
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertSame(0, $backend->counter->packOpens(), 'nothing read the usual way');
+		$this->assertGreaterThanOrEqual(3, count($backend->asked));
+		// in the order of the files, one range per pack the restore goes through (the same blob
+		// again later is a range of its own)
+		$this->assertLessThanOrEqual(count($backend->inner->list('packs')) + 1, count($backend->asked));
+	}
+
+	public function testAPlanThatCannotBeMadeLeavesTheRestoreToTheUsualReads(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), failSnapshotReads: 1);
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertSame([], $backend->asked, 'no reading ahead without a plan');
+		$this->assertSame(2, $backend->snapshotReads, 'the restore itself read the snapshot again');
+	}
+
+	public function testTheEmptySpoolOfAStoppedRestoreIsRemoved(): void {
+		$target = $this->tempDir();
+		mkdir("$target/.ngb-restore-spool-0badc0de", 0700); // a restore that was killed
+
+		$this->assertRestored($this->restore(new ReadAheadBackend(new LocalBackend($this->root)), '', $target));
+	}
+
+	public function testARangeThatCouldNotBeReadAheadIsReadTheUsualWay(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), fail: [1]);
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertGreaterThan(0, $backend->counter->packOpens());
+	}
+
+	public function testADamagedCopyIsReadAgainTheUsualWay(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), damage: [0, 2]);
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertGreaterThan(0, $backend->counter->packOpens());
+	}
+
+	public function testAPartialRestoreReadsOnlyWhatItNeeds(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root));
+
+		$this->assertRestored($this->restore($backend, 'big1'), 'big1');
+
+		$this->assertSame(0, $backend->counter->packOpens());
+		$this->assertCount(2, $backend->asked, 'the end of one pack and the start of the next');
+		$total = array_sum(array_map(fn (array $r): int => $r[2], $backend->asked));
+		$this->assertLessThan(21 * 1048576, $total, 'not the whole packs');
+	}
+
+	public function testABackendThatCannotReadAheadNowRestoresAsBefore(): void {
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), unavailable: true);
+
+		$this->assertRestored($this->restore($backend));
+
+		$this->assertGreaterThan(0, $backend->counter->packOpens());
+	}
+
+	public function testAFailingRestoreStillEndsTheReadsAheadAndRemovesTheSpool(): void {
+		// the second range fails, and so does reading it the usual way
+		$backend = new ReadAheadBackend(new LocalBackend($this->root), fail: [1], packsGone: true);
+		$target = $this->tempDir();
+		$repo = Repository::openWithKey($backend, $this->keys, null, null);
+
+		try {
+			$repo->restore($this->snapshot, $target);
+			$this->fail('restore should fail');
+		} catch (BackendException $e) {
+			$this->assertStringStartsWith('Not found', $e->getMessage());
+		}
+
+		$this->assertTrue($backend->ended, 'the generator was ended');
+		$this->assertSame([], glob("$target/.ngb-restore-spool-*"));
+	}
+
+	public function testPlanJoinsBlobsThatFollowEachOtherInAPack(): void {
+		$plan = new ReadAheadPlan('/nowhere');
+		$plan->add('packs/aa/a', 0, 100);
+		$plan->add('packs/aa/a', 100, 100);
+		$plan->add('packs/aa/a', 50, 10); // the same blob again: inside the range
+		$plan->add('packs/aa/a', 200 + ReadAheadPlan::GAP, 100); // a gap it reads along
+		$plan->add('packs/aa/a', 300 + 3 * ReadAheadPlan::GAP, 100); // too far: a new range
+		$plan->add('packs/bb/b', 0, 100);
+		$plan->add('packs/bb/b', 0, 100); // backwards is fine as long as it is inside
+		$plan->add('packs/aa/a', 0, 100); // back to an earlier pack: a new range
+
+		$this->assertSame([
+			['packs/aa/a', 0, 300 + ReadAheadPlan::GAP],
+			['packs/aa/a', 300 + 3 * ReadAheadPlan::GAP, 100],
+			['packs/bb/b', 0, 100],
+			['packs/aa/a', 0, 100],
+		], $plan->ranges());
+	}
+
+	public function testPlanSkipsRangesWhoseBlobsWereReadElsewhere(): void {
+		$plan = new ReadAheadPlan('/nowhere');
+		foreach (range(0, 4) as $p) {
+			$plan->add("packs/0$p/p$p", 0, 10);
+		}
+		$plan->readFrom((function (): \Generator {
+			foreach (range(0, 4) as $p) {
+				$fh = fopen('php://temp', 'w+b');
+				fwrite($fh, str_repeat((string)$p, 10));
+				rewind($fh);
+				yield $p => $fh;
+			}
+		})());
+
+		$this->assertSame('00000', $plan->read('packs/00/p0', 0, 5));
+		$this->assertNull($plan->read('packs/77/moved', 0, 5), 'moved: not in the plan');
+		$this->assertSame('333', $plan->read('packs/03/p3', 2, 3), 'the ranges in between were read elsewhere');
+		$this->assertNull($plan->read('packs/01/p1', 0, 5), 'never backwards');
+		$this->assertSame('44', $plan->read('packs/04/p4', 0, 2));
+	}
+
+	public function testPlanKeepsARangeShorterThanTheMaximum(): void {
+		$plan = new ReadAheadPlan('/nowhere');
+		for ($o = 0; $o < 3 * ReadAheadPlan::MAX_RANGE; $o += 4 * 1048576) {
+			$plan->add('packs/aa/a', $o, 4 * 1048576);
+		}
+		$this->assertCount(3, $plan->ranges());
+		foreach ($plan->ranges() as $r) {
+			$this->assertLessThanOrEqual(ReadAheadPlan::MAX_RANGE, $r[2]);
+		}
+	}
+}
+
+/** Reads ahead from a local repository, one range after the other; can fail or damage ranges. */
+final class ReadAheadBackend implements IBackend, IReadsAhead {
+	public PackOpenCountingBackend $counter;
+	/** @var list<array{0:string, 1:int, 2:int}> */
+	public array $asked = [];
+	public bool $ended = false;
+
+	/**
+	 * @param list<int> $fail ranges given as null
+	 * @param list<int> $damage ranges given with a flipped byte in every blob
+	 */
+	public function __construct(
+		public LocalBackend $inner,
+		private array $fail = [],
+		private array $damage = [],
+		private bool $unavailable = false,
+		/** reading a pack the usual way fails */
+		private bool $packsGone = false,
+		/** this many reads of a snapshot fail, then they work */
+		private int $failSnapshotReads = 0,
+	) {
+		$this->counter = new PackOpenCountingBackend($inner);
+	}
+
+	public function canReadAhead(): bool {
+		return true;
+	}
+
+	public function readAhead(array $ranges, string $spool): ?\Generator {
+		if ($this->unavailable) {
+			return null;
+		}
+		\PHPUnit\Framework\Assert::assertDirectoryExists($spool);
+		$this->asked = $ranges;
+		return (function () use ($ranges): \Generator {
+			try {
+				foreach ($ranges as $i => [$path, $offset, $length]) {
+					if (in_array($i, $this->fail, true)) {
+						yield $i => null;
+						continue;
+					}
+					$data = $this->inner->getRange($path, $offset, $length);
+					if (in_array($i, $this->damage, true)) {
+						for ($k = 0; $k < strlen($data); $k += 4096) {
+							$data[$k] = chr(ord($data[$k]) ^ 1);
+						}
+					}
+					$fh = fopen('php://temp', 'w+b');
+					fwrite($fh, $data);
+					rewind($fh);
+					yield $i => $fh;
+					fclose($fh);
+				}
+			} finally {
+				$this->ended = true;
+			}
+		})();
+	}
+
+	public function put(string $path, $stream): void {
+		$this->counter->put($path, $stream);
+	}
+
+	public int $snapshotReads = 0;
+
+	public function get(string $path) {
+		if (str_starts_with($path, 'snapshots/')) {
+			$this->snapshotReads++;
+			if ($this->snapshotReads <= $this->failSnapshotReads) {
+				throw new BackendException('Connection lost reading ' . $path);
+			}
+		}
+		if ($this->packsGone && str_starts_with($path, 'packs/')) {
+			throw new BackendException('Not found: ' . $path);
+		}
+		return $this->counter->get($path);
+	}
+
+	public function getMany(array $paths): \Generator {
+		return $this->counter->getMany($paths);
+	}
+
+	public function getRange(string $path, int $offset, int $length): string {
+		return $this->counter->getRange($path, $offset, $length);
+	}
+
+	public function getRanges(array $ranges): \Generator {
+		return $this->counter->getRanges($ranges);
+	}
+
+	public function exists(string $path): bool {
+		return $this->counter->exists($path);
+	}
+
+	public function move(string $from, string $to): void {
+		$this->counter->move($from, $to);
+	}
+
+	public function list(string $prefix): array {
+		return $this->counter->list($prefix);
+	}
+
+	public function listFiles(string $dir): array {
+		return $this->counter->listFiles($dir);
+	}
+
+	public function delete(string $path): void {
+		$this->counter->delete($path);
+	}
+}

tests/integration/full-restore-roundtrip.sh

@@ -2,15 +2,24 @@
 # Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a
 # SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup
 # location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7.
-# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql]
+# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp]
 #   e.g. full-restore-roundtrip.sh 35-apache pgsql
+# With "sftp" the location is an SFTP server instead of a shared directory, and the origin has a
+# file spread over several packs: the restore reads them ahead on connections of its own.
+# With "legacy" the origin runs NG Backup 1.0.0-rc.1 (0.9.7), whose backups do not record the
+# structure of their tables: the restore must then leave the tables of the disabled app out
+# with --skip-missing-tables, as for a real backup made before 1.0.0-rc.2.
 # With pgsql or mysql the origin also has what broke real disaster recoveries: the Mail app with a synced
 # account (foreign keys between its tables) and Deck installed and then switched off (tables of
 # an app that is not enabled). The restore instance has neither; --install-apps adds Mail.
 set -euo pipefail
 IMAGE="nextcloud:${1:-35-apache}"
 DB="${2:-sqlite}"
+MODE="${3:-}"
+LEGACY=""
+[ "$MODE" = legacy ] && LEGACY=legacy
 APP=$(cd "$(dirname "$0")/../.." && pwd)
+ORIGIN_APP="$APP"
 NET=ngb-frt-net
 SHARED=$(mktemp -d)
 chmod 777 "$SHARED"
@@ -18,14 +27,22 @@ fail() { echo "FAIL: $*"; exit 1; }
 ok() { echo "ok   $*"; }
 
 cleanup() {
-	docker rm -fv ngb-frt-origin ngb-frt-restore ngb-frt-db-origin ngb-frt-db-restore ngb-frt-imap >/dev/null 2>&1 || true
+	docker rm -fv ngb-frt-origin ngb-frt-restore ngb-frt-db-origin ngb-frt-db-restore ngb-frt-imap ngb-frt-sftp >/dev/null 2>&1 || true
 	docker network rm "$NET" >/dev/null 2>&1 || true
 	docker run --rm -v "$SHARED:/target" alpine sh -c 'rm -rf /target/*' >/dev/null 2>&1 || true
 	rmdir "$SHARED" 2>/dev/null || true
 	rm -f /tmp/ngb-frt-kit.json /tmp/ngb-frt-pass
+	[ -n "${LEGACY_DIR:-}" ] && rm -rf "$LEGACY_DIR"
+	return 0
 }
 trap cleanup EXIT
 
+if [ "$LEGACY" = legacy ]; then
+	LEGACY_DIR=$(mktemp -d)
+	curl -fsSL https://github.com/dillardblom/ng_backup/releases/download/v1.0.0-rc.1/ng_backup-0.9.7.tar.gz | tar -xz -C "$LEGACY_DIR"
+	ORIGIN_APP="$LEGACY_DIR/ng_backup"
+fi
+
 run_on() { docker exec -u www-data "$1" php occ "${@:2}"; }
 
 # Database settings for an instance: SQLite inside the container, or its own PostgreSQL container
@@ -46,27 +63,37 @@ db_env() {
 }
 
 docker network create "$NET" >/dev/null
+LOCATION=(local -a null::null -o datadir=/mnt/shared)
+if [ "$MODE" = sftp ]; then
+	docker run -d --name ngb-frt-sftp --network "$NET" atmoz/sftp:alpine ngbackup:ngbackuppw:::upload >/dev/null
+	LOCATION=(sftp -a password::password -o host=ngb-frt-sftp -o root=/upload -o user=ngbackup -o password=ngbackuppw)
+fi
 ORIGIN_DB=$(db_env origin)
 # shellcheck disable=SC2086
 docker run -d --name ngb-frt-origin --network "$NET" $ORIGIN_DB \
 	-e NEXTCLOUD_ADMIN_USER=admin -e NEXTCLOUD_ADMIN_PASSWORD=admin \
-	-v "$APP:/var/www/html/custom_apps/ng_backup:ro" -v "$SHARED:/mnt/shared" "$IMAGE" >/dev/null
+	-v "$ORIGIN_APP:/var/www/html/custom_apps/ng_backup:ro" -v "$SHARED:/mnt/shared" "$IMAGE" >/dev/null
 for i in $(seq 1 90); do run_on ngb-frt-origin status 2>/dev/null | grep -q "installed: true" && break; sleep 3; done
 docker exec ngb-frt-origin chown www-data:www-data /var/www/html/custom_apps
 run_on ngb-frt-origin app:enable ng_backup >/dev/null
-ok "origin instance up ($DB)"
+ok "origin instance up ($DB${LEGACY:+, NG Backup 0.9.7})"
 
 docker exec ngb-frt-origin sh -c "printf 'full restore roundtrip passphrase 2026\n' > /tmp/pass"
 run_on ngb-frt-origin backup:key:init --passphrase-file=/tmp/pass --label=CI >/dev/null || fail "key init"
 run_on ngb-frt-origin backup:key:kit --output=/tmp/kit.json >/dev/null
 CODE=$(run_on ngb-frt-origin backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
 run_on ngb-frt-origin backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "confirm"
-run_on ngb-frt-origin backup:target:add offsite local -a null::null -o datadir=/mnt/shared --path=repo >/dev/null || fail "target add"
+run_on ngb-frt-origin backup:target:add offsite "${LOCATION[@]}" --path=repo >/dev/null || fail "target add"
 ok "key, kit and location on the origin"
 
 NC_PASS=$(openssl rand -hex 16)
 docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
 docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
+if [ "$MODE" = sftp ]; then
+	# about three packs of 32 MiB
+	docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
+	LARGE_SUM=$(docker exec ngb-frt-origin sha256sum /var/www/html/data/frtest/files/Docs/large.bin | cut -d' ' -f1)
+fi
 run_on ngb-frt-origin files:scan frtest >/dev/null
 
 # SQL on an instance's own database (pgsql/mysql only)
@@ -114,11 +141,22 @@ docker cp /tmp/ngb-frt-pass ngb-frt-restore:/tmp/pass
 ok "fresh restore instance up"
 
 run_on ngb-frt-restore backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass >/dev/null || fail "key import"
-run_on ngb-frt-restore backup:target:add offsite local -a null::null -o datadir=/mnt/shared --path=repo >/dev/null || fail "target add on the restore instance"
+run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo >/dev/null || fail "target add on the restore instance"
 run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location"
 ok "recovery kit imported, existing repository adopted, snapshot visible"
 
-OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction --install-apps 2>&1) || {
+RESTORE_OPTS=(--no-interaction --install-apps)
+if [ "$LEGACY" = legacy ] && [ "$DB" != sqlite ]; then
+	# Without --skip-missing-tables an older backup must be refused, naming the app's tables
+	REFUSED=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS[@]}" 2>&1) \
+		&& fail "an older backup with tables of unknown structure was restored without --skip-missing-tables"
+	printf '%s\n' "$REFUSED" | grep -q "oc_deck_boards" || { printf '%s\n' "$REFUSED" | tail -20; fail "the refusal does not name the missing tables"; }
+	ok "older backup refused without --skip-missing-tables, missing tables named"
+	RESTORE_OPTS+=(--skip-missing-tables)
+fi
+sftp_logins() { docker logs ngb-frt-sftp 2>&1 | grep -c "Accepted password" || true; }
+[ "$MODE" = sftp ] && LOGINS_BEFORE=$(sftp_logins)
+OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS[@]}" 2>&1) || {
 	printf '%s\n' "$OUT" | tail -40
 	docker exec ngb-frt-restore sh -c 'tail -c 6000 /var/www/html/data/nextcloud.log' || true
 	fail "restore:full"
@@ -134,7 +172,11 @@ docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.tx
 run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
 ok "database, files and ng_backup's own state are all intact and usable after the restore"
 
-if [ "$DB" != sqlite ]; then
+if [ "$DB" != sqlite ] && [ "$LEGACY" = legacy ]; then
+	[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 0 ] || fail "a table of unknown structure was created"
+	[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
+	ok "older backup: tables of the disabled app left out, Mail restored"
+elif [ "$DB" != sqlite ]; then
 	[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created"
 	[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
 	[ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \
@@ -143,7 +185,16 @@ if [ "$DB" != sqlite ]; then
 	ok "tables of a disabled app created, Mail restored with its foreign keys"
 fi
 
+if [ "$MODE" = sftp ]; then
+	[ "$(docker exec ngb-frt-restore sha256sum /var/www/html/data/frtest/files/Docs/large.bin | cut -d' ' -f1)" = "$LARGE_SUM" ] \
+		|| fail "the large file differs after the restore"
+	LOGINS=$(( $(sftp_logins) - LOGINS_BEFORE ))
+	[ "$LOGINS" -gt 1 ] || fail "the restore did not read ahead on connections of its own ($LOGINS SFTP logins)"
+	docker exec ngb-frt-restore sh -c 'ls -a /var/www/html/data' | grep -q ngb-restore-spool && fail "the spool folder was left behind"
+	ok "over SFTP: read ahead on connections of its own ($LOGINS SFTP logins during the restore), large file identical, no spool left"
+fi
+
 run_on ngb-frt-restore backup:run offsite -l post-restore-check >/dev/null || fail "a new backup after the restore fails"
 ok "the restored instance can run a new backup of its own"
 
-echo "FULL RESTORE ROUNDTRIP PASS ($DB)"
+echo "FULL RESTORE ROUNDTRIP PASS ($DB${MODE:+, $MODE})"