Release candidate 4 (app version 0.9.10): full restore over SFTP reads ahead; SFTP leg in the full-restore test

.github/workflows/ci.yml

@@ -112,3 +112,5 @@ jobs:
         run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql legacy
       - name: Full restore of a backup made by 1.0.0-rc.1 (no table structure) on MariaDB
         run: tests/integration/full-restore-roundtrip.sh 35-apache mysql legacy
+      - name: Full restore over SFTP on PostgreSQL, reading packs ahead on several connections
+        run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql sftp

CHANGELOG.md

@@ -9,6 +9,13 @@ instead carried by the app description, the git tag, the release, and this file.
 
 ## [Unreleased]
 
+### Changed
+- A full restore over SFTP reads the packs ahead on 4 connections of its own, many requests in
+  flight on each, into temporary files in the restore folder. From a Hetzner Storage Box to a
+  server in Canada (122 ms) packs now come in at about 7.6 MiB/s instead of 2.8; on a test line
+  with 120 ms and some packet loss, a 1.6 GiB restore went from 5.1 to 17.8 MiB/s. A part that
+  cannot be read this way is read as before. Restores from other locations work as before.
+
 ### Fixed
 - A full restore of a backup made before 1.0.0-rc.2 with `--skip-missing-tables` failed on
   PostgreSQL with "relation ... does not exist": the left-out tables were still looked up to set

PLAN.md

@@ -386,9 +386,15 @@ in README "Known limitations":
 - **1.0.0 (besluit 2026-10-18):** alleen betrouwbaarheid. rc2 uitgebracht; slaagt de grote
   DR-test op VM 980 (snapshot van 12-10, met `--skip-missing-tables`), dan 1.0.0. Zo niet: fix,
   rc3, en rc3 pas uitbrengen na een geslaagde volledige DR-test.
+- **rc4 (besluit 2026-10-19):** rc3 is niet uitgebracht. De DR-test met rc3 liep ~2,8 MiB/s
+  (~3,5 dag); parallel lezen over SFTP haalde vanaf VM 980 ~7,6 MiB/s. Daarom rc4 = rc3 +
+  vooruitlezen over SFTP (4 eigen verbindingen). Volgorde: volledige DR-test op VM 980 volgens de
+  gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een
+  productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0.
+  Problemen of meldingen van gebruikers: rc5.
 - **1.0.1:**
-  1. Herstelsnelheid: bij een restore meerdere pakketten tegelijk lezen (SFTP: meerdere
+  1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
-     verbindingen; S3 heeft al 16 tegelijk). Raakt het leespad, daarom niet in 1.0.0.
+     restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
   2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron
      staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue
      #13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles.

README.md

@@ -181,6 +181,9 @@ Contributions welcome via PR — add what you ran into with your own provider.
   server (or the VM/container host where TCP ends) to BBR:
   `sysctl -w net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr`
   (`modprobe tcp_bbr` first; make it permanent in `/etc/sysctl.d/`).
+  A full restore over SFTP reads the packs ahead on 4 connections of its own, next to the one
+  NG Backup already has open: the SFTP server must allow 5 connections at once for that user (a
+  Hetzner Storage Box allows 10). With fewer it uses the connections it gets.
 
 - **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself
   (e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the

appinfo/info.xml

@@ -41,7 +41,7 @@ welcome: use the "Share your experience" link on the admin page.
 
 Developed and sponsored by [Ensembia](https://www.ensembia.com), which also offers paid help with
 restoring a backup after a crash.]]></description>
-    <version>0.9.9</version>
+    <version>0.9.10</version>
     <licence>agpl</licence>
     <author mail="dillard.blom@ensembia.com">Dillard Blom</author>
     <namespace>NgBackup</namespace>
@@ -52,9 +52,9 @@ restoring a backup after a crash.]]></description>
     <website>https://github.com/dillardblom/ng_backup</website>
     <bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
     <repository type="git">https://github.com/dillardblom/ng_backup</repository>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.3/screenshots/1-backups.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/1-backups.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.3/screenshots/2-restore.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/2-restore.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.3/screenshots/3-settings.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/3-settings.png</screenshot>
     <dependencies>
         <php min-version="8.1"/>
         <lib>sodium</lib>

tests/integration/full-restore-roundtrip.sh

@@ -2,8 +2,10 @@
 # Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a
 # SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup
 # location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7.
-# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy]
+# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp]
 #   e.g. full-restore-roundtrip.sh 35-apache pgsql
+# With "sftp" the location is an SFTP server instead of a shared directory, and the origin has a
+# file spread over several packs: the restore reads them ahead on connections of its own.
 # With "legacy" the origin runs NG Backup 1.0.0-rc.1 (0.9.7), whose backups do not record the
 # structure of their tables: the restore must then leave the tables of the disabled app out
 # with --skip-missing-tables, as for a real backup made before 1.0.0-rc.2.
@@ -13,7 +15,9 @@
 set -euo pipefail
 IMAGE="nextcloud:${1:-35-apache}"
 DB="${2:-sqlite}"
-LEGACY="${3:-}"
+MODE="${3:-}"
+LEGACY=""
+[ "$MODE" = legacy ] && LEGACY=legacy
 APP=$(cd "$(dirname "$0")/../.." && pwd)
 ORIGIN_APP="$APP"
 NET=ngb-frt-net
@@ -23,7 +27,7 @@ fail() { echo "FAIL: $*"; exit 1; }
 ok() { echo "ok   $*"; }
 
 cleanup() {
-	docker rm -fv ngb-frt-origin ngb-frt-restore ngb-frt-db-origin ngb-frt-db-restore ngb-frt-imap >/dev/null 2>&1 || true
+	docker rm -fv ngb-frt-origin ngb-frt-restore ngb-frt-db-origin ngb-frt-db-restore ngb-frt-imap ngb-frt-sftp >/dev/null 2>&1 || true
 	docker network rm "$NET" >/dev/null 2>&1 || true
 	docker run --rm -v "$SHARED:/target" alpine sh -c 'rm -rf /target/*' >/dev/null 2>&1 || true
 	rmdir "$SHARED" 2>/dev/null || true
@@ -59,6 +63,11 @@ db_env() {
 }
 
 docker network create "$NET" >/dev/null
+LOCATION=(local -a null::null -o datadir=/mnt/shared)
+if [ "$MODE" = sftp ]; then
+	docker run -d --name ngb-frt-sftp --network "$NET" atmoz/sftp:alpine ngbackup:ngbackuppw:::upload >/dev/null
+	LOCATION=(sftp -a password::password -o host=ngb-frt-sftp -o root=/upload -o user=ngbackup -o password=ngbackuppw)
+fi
 ORIGIN_DB=$(db_env origin)
 # shellcheck disable=SC2086
 docker run -d --name ngb-frt-origin --network "$NET" $ORIGIN_DB \
@@ -74,12 +83,17 @@ run_on ngb-frt-origin backup:key:init --passphrase-file=/tmp/pass --label=CI >/d
 run_on ngb-frt-origin backup:key:kit --output=/tmp/kit.json >/dev/null
 CODE=$(run_on ngb-frt-origin backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
 run_on ngb-frt-origin backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "confirm"
-run_on ngb-frt-origin backup:target:add offsite local -a null::null -o datadir=/mnt/shared --path=repo >/dev/null || fail "target add"
+run_on ngb-frt-origin backup:target:add offsite "${LOCATION[@]}" --path=repo >/dev/null || fail "target add"
 ok "key, kit and location on the origin"
 
 NC_PASS=$(openssl rand -hex 16)
 docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
 docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
+if [ "$MODE" = sftp ]; then
+	# about three packs of 32 MiB
+	docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
+	LARGE_SUM=$(docker exec ngb-frt-origin sha256sum /var/www/html/data/frtest/files/Docs/large.bin | cut -d' ' -f1)
+fi
 run_on ngb-frt-origin files:scan frtest >/dev/null
 
 # SQL on an instance's own database (pgsql/mysql only)
@@ -127,7 +141,7 @@ docker cp /tmp/ngb-frt-pass ngb-frt-restore:/tmp/pass
 ok "fresh restore instance up"
 
 run_on ngb-frt-restore backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass >/dev/null || fail "key import"
-run_on ngb-frt-restore backup:target:add offsite local -a null::null -o datadir=/mnt/shared --path=repo >/dev/null || fail "target add on the restore instance"
+run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo >/dev/null || fail "target add on the restore instance"
 run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location"
 ok "recovery kit imported, existing repository adopted, snapshot visible"
 
@@ -140,6 +154,8 @@ if [ "$LEGACY" = legacy ] && [ "$DB" != sqlite ]; then
 	ok "older backup refused without --skip-missing-tables, missing tables named"
 	RESTORE_OPTS+=(--skip-missing-tables)
 fi
+sftp_logins() { docker logs ngb-frt-sftp 2>&1 | grep -c "Accepted password" || true; }
+[ "$MODE" = sftp ] && LOGINS_BEFORE=$(sftp_logins)
 OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS[@]}" 2>&1) || {
 	printf '%s\n' "$OUT" | tail -40
 	docker exec ngb-frt-restore sh -c 'tail -c 6000 /var/www/html/data/nextcloud.log' || true
@@ -169,7 +185,16 @@ elif [ "$DB" != sqlite ]; then
 	ok "tables of a disabled app created, Mail restored with its foreign keys"
 fi
 
+if [ "$MODE" = sftp ]; then
+	[ "$(docker exec ngb-frt-restore sha256sum /var/www/html/data/frtest/files/Docs/large.bin | cut -d' ' -f1)" = "$LARGE_SUM" ] \
+		|| fail "the large file differs after the restore"
+	LOGINS=$(( $(sftp_logins) - LOGINS_BEFORE ))
+	[ "$LOGINS" -gt 1 ] || fail "the restore did not read ahead on connections of its own ($LOGINS SFTP logins)"
+	docker exec ngb-frt-restore sh -c 'ls -a /var/www/html/data' | grep -q ngb-restore-spool && fail "the spool folder was left behind"
+	ok "over SFTP: read ahead on $LOGINS connections, large file identical, no spool left"
+fi
+
 run_on ngb-frt-restore backup:run offsite -l post-restore-check >/dev/null || fail "a new backup after the restore fails"
 ok "the restored instance can run a new backup of its own"
 
-echo "FULL RESTORE ROUNDTRIP PASS ($DB${LEGACY:+, legacy})"
+echo "FULL RESTORE ROUNDTRIP PASS ($DB${MODE:+, $MODE})"