Merge pull request #30 from dillardblom/1.0.3

Release 1.0.3

.github/workflows/ci.yml

@@ -140,3 +140,5 @@ jobs:
         run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite to-pgsql
       - name: Full restore with a damaged database dump (--salvage)
         run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite salvage
+      - name: Full restore with a damaged and a missing pack (--salvage)
+        run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite salvage-files

CHANGELOG.md

@@ -11,7 +11,37 @@ Each heading starts with that `<version>`, once: the app store shows a release t
 whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
 shown under 1.0.0. The git tag of a release candidate goes after the date.
 
-## [1.0.2] - unreleased
+## [1.0.3] - 2026-11-06
+
+### Fixed
+- Empty folders are backed up and restored (#27), in `occ backup:restore`, `occ backup:restore:full`
+  and the restore page, where they are shown too. They come back from the first backup made with
+  this version; older restore points do not contain them. An older version of NG Backup can still
+  read new restore points, and then only leaves the empty folders out.
+- A full disk during a restore no longer looks like a damaged backup. The restore now stops with a
+  message about the disk, also with `--salvage`, and does not point at `--salvage` any more. The same
+  goes for other trouble writing to the restore directory, such as permissions.
+- A file or folder whose name a snapshot cannot hold (a backslash, a control character, not UTF-8) is
+  left out and reported as unreadable. Before, a backslash or control character could make the whole
+  restore point unreadable, and a name that is not UTF-8 made the backup fail.
+- A restore that goes on after a stop also removes the temporary files in folders where every file
+  is already in place, and checks that those folders are inside the restore directory before it
+  leaves their files alone.
+- After a restore that was killed and then ran again, a backup no longer has to wait up to two hours
+  for the locks of the killed run.
+- A restore only removes read-ahead folders of its own (`.ngb-restore-spool-` with 8 hex digits).
+- A restore keeps its lock on the location during a long row of files that are already in place.
+- A database restore that fails no longer leaves behind tables it created, which could stop the
+  installation of that app later. When only NG Backup's own settings cannot be put back after the
+  database was restored, `--salvage` no longer says the database is still the fresh one.
+- A connection that breaks while opening a pack on the location (seen with SMB) is retried once and
+  is handled by `--salvage`, instead of stopping the restore with the error of the storage.
+- Without a system cron, several running backups take turns while the page is open.
+- The Bin tab shows how far it is while it reads the removed restore points, which can take a while.
+  It also shows when it cannot read a location, instead of an empty bin, and refreshes when a run
+  ends.
+
+## [1.0.2] - 2026-11-06
 
 ### Added
 - A warning on the admin page and in `occ backup:status` when background jobs do not run on a
@@ -34,9 +64,30 @@ shown under 1.0.0. The git tag of a release candidate goes after the date.
 - Full restores from S3 read the packs ahead, eight ranges at a time, as restores over SFTP do
   since 1.0.0-rc.4. Over SMB the packs are read in 8 MiB reads instead of 1 MiB: over a line with
   a 130 ms round trip, reading was four times faster in a test (18 instead of 4.5 MiB/s).
+- Over SMB the index files, which a new server reads in `backup:target:add`, take fewer round
+  trips: from Canada to a Hetzner Storage Box (130 ms), 0.66 instead of 1.05 s per file, so about
+  5 instead of 8 hours for the 27,000 index files of a 1 TB repository.
+- `occ backup:restore:full` that runs again after it stopped (`--force`) goes on where it was: data
+  files already in place with the size and modification time of the backup are left, and only
+  the rest is read from the location. Before, a stopped restore of 1 TB started over.
+
+### Fixed
+- A full restore over SMB could hang at 100% CPU when the connection to the server dropped: a
+  failed read was counted as data. It is now read again over a new connection. Found in a
+  disaster recovery test of 1 TB from Canada to a Hetzner Storage Box, after 392 GiB.
+- The read-ahead folder of a restore that was killed while it wrote is removed by the next run;
+  it kept one file of up to 8 MiB.
 - `occ backup:target:add` only estimates the time left for reading the index once enough of it
   is in. The first file also carries the time to connect, which made it say "about 642 min left"
   for something done within a minute.
+- A restore that stops because a file is gone from the location now points at `--salvage`, as it
+  already did for damaged data.
+
+### Known issues
+- Empty folders are not backed up or restored; to be fixed in 1.0.3 (#27).
+- A full disk during a restore looks like a damaged backup ("Size mismatch"), and with `--salvage`
+  the restore goes on and lists the remaining files. The backup itself is fine: make room and run
+  the restore again. To be fixed in 1.0.3.
 
 ## [1.0.1] - 2026-11-01
 

PLAN.md

@@ -407,6 +407,29 @@ in README "Known limitations":
     `--skip-missing-tables`; daarna `--salvage` op een deel van de bestanden (moet hetzelfde
     resultaat geven, exitcode 0). Uitvoeren met Dillard, die VM 980 klaarzet; deze week, vóór
     de release. Screenshots akkoord (2026-11-04).
+  **DR-test 1.0.2 over SMB (VM 980, snapshot a86271ed van nc01, 983 GiB, gestart 2026-11-04):**
+  index lezen 8 uur (~1 bestand/s; daarna versneld naar 0,66 s/bestand, gemeten vanaf VM 980);
+  restore zonder opties (geen `--install-apps`, geen `--skip-missing-tables`) liep ~12 MiB/s
+  (SFTP eerder 2,8-7,6; SMB vóór 1.0.2 2,6). Na 392 GiB viel de SMB-verbinding weg en bleef de
+  restore op 100% CPU hangen: de smbclient-extensie geeft bij een mislukte read een string met
+  lengte -1. Opgelost (controle + nieuwe verbinding), plus hervatten: `--force` laat bestanden met
+  dezelfde grootte en mtime staan. Hervat 2026-11-05 07:21 UTC na `maintenance:mode --off`
+  volgens de gids; binnen 10 min voorbij de 392 GiB. SMB op sub5 na de test weer uitzetten.
+  **Resultaat (klaar 2026-11-06 02:11 UTC, nazorg 02:27):** hervatte run 18 u 50 min, exit 0, 150.755
+  bestanden; 7-13 MiB/s (langzamer vanaf ~18 uur Duitse tijd, oorzaak niet vastgesteld). Vergeleken
+  met nc01: 150.467 van 150.758 bestanden gelijk in grootte en mtime, 397 op hash zonder verschil;
+  gebruikers, agenda's, contacten, shares en Deck gelijk; bestanden, agenda en contacten via WebDAV
+  getest als gebruiker. `backup:restore --salvage` op 16.904 bestanden: exit 0, geen verschil. Niet
+  vastgesteld: de 50.080 "updated" bij `files:scan`. Zonder `--install-apps` 51 van de 81 apps aan.
+  VM's 980-982 en de sim-backups op OVH en sub6 opgeruimd.
+  **Naar 1.0.3 (besluit 2026-11-06):** lege mappen komen niet terug (3.982 op nc01, 2.076 daarvan in
+  `files`, verklaart vrijwel zeker de 2.370 "removed" bij `files:scan`). De backup legt alleen
+  bestanden vast (`walk()`), dus oudere snapshots hebben ze niet. Voor een DR acceptabel, wel
+  fixen; issue #27. Daarna kleine volledige simulaties; een DR van 1 TB is hiervoor niet nodig.
+  **Salvage vóór de release getest (besluit 2026-11-06):** nieuwe modus `salvage-files` in
+  `full-restore-roundtrip.sh`, ook in CI: één pack beschadigd en één weg. Eerst stopt de restore,
+  daarna `--salvage --force` volgens de gids. Vond één fout: bij een ontbrekend bestand wees de
+  melding niet naar `--salvage` (opgelost).
   Stand 2026-11-04 (`polish`, gepusht): cron-waarschuwing + tick vanaf de pagina +
   voortgang van de db-fase; `--salvage` voor `backup:restore` en `backup:restore:full`; S3
   vooruitlezen (8 ranges tegelijk, met `SaveAs` naar de spool); SMB in leesacties van 8 MiB
@@ -414,7 +437,44 @@ in README "Known limitations":
   met vertraging is nog niet gemeten); Bin-tabblad; schatting `target:add`; nieuwe screenshots
   (1–3 vervangen, 4-bin erbij; in `info.xml` bij de release naar tag v1.0.2 laten wijzen en
   4-bin toevoegen). Getest: unit (208), psalm, integratie NC35 sqlite/pgsql, S3/SFTP/FTP/SMB,
-  volledige restore met `--salvage`. Nog te doen: DR-test, release.
+  volledige restore met `--salvage`. **1.0.2 uitgebracht 2026-11-06** (tag `v1.0.2`, PR #28).
+- **Naar 1.0.3, uit de reviews van 1.0.2 (achteraf gedraaid, 2026-11-06):**
+  1. Lege mappen (#27).
+  2. Een volle schijf onder `--salvage` telt als beschadigde backup en de restore gaat door; zonder
+     `--salvage` wijst de melding er ten onrechte naar. `fwrite` controleren, schrijffouten stoppen de
+     restore (`Repository::restoreEntries`; ook `RestoreService::writeFile` nakijken). Beide reviews.
+  3. Hervatten: controle op de doelmap vóór het overslaan, en tijdelijke bestanden ook opruimen in
+     een map waar alles al klopt.
+  4. Alleen `.ngb-restore-spool-<hex>` opruimen.
+  5. Salvage-test: tijdelijke bestanden recursief zoeken; `admin` en `frtest` apart controleren.
+  6. Nog nakijken: lease verversen bij overgeslagen bestanden; fout ná de database-commit
+     (`restoreOwnAppConfig`) en tabellen die na een rollback blijven staan; backups zonder cron
+     via de pagina (eerlijke beurten, stappen die de webtimeout halen); Bin verversen en fouten
+     tonen; SMB-uitzondering bij het openen van een pack onder `--salvage`.
+  Niet doen: bij hervatten de inhoud vergelijken (grootte en mtime is ook het criterium van de backup).
+  **Stand op branch `1.0.3` (2026-11-06):** punten 1-6 gebouwd. Lege mappen staan in het metarecord
+  van de snapshot, niet tussen de bestanden: 1.0.2 en ouder lezen zo'n snapshot gewoon en slaan alleen
+  de lege mappen over. Geen webtimeout-wijziging zonder cron: alleen in de code nagelopen, niet getest
+  (Nextcloud zet de PHP-tijdslimiet ruim, een gestopte stap wordt overgedaan); wel eerlijke beurten.
+  Bin leest in delen met teller en balk (besluit 2026-11-06, ook met oog op meerdere backups per
+  dag). Getest: unit (volle schijf op echte tmpfs), integratie NC35 sqlite/pgsql/mysql, sftp, salvage,
+  salvage-files; Bin in een echte browser met vertraagde verzoeken (1 van 7 t/m 7). Nog: reviews
+  vóór de PR, README-regels bij de versiecommit.
+  **Simulaties (2026-11-06, NC 34.0.4, 2 GB, lokale SFTP, `~/dev-local/ngb-sim/sim.sh`):** normaal
+  (pgsql), stop en hervat (pgsql → MariaDB, restore met kill -9 gestopt), volle schijf met `--salvage`:
+  alle drie geslaagd, alle 3.101 gebruikersbestanden en 7 lege mappen gelijk, share, agenda en WebDAV
+  werken. Vond één fout, opgelost: na een kill bleven vergrendelingen tot 2 uur staan, waardoor de
+  backup van stap 11 werd geweigerd. `files:scan` "Updated" zijn in de gemeten run alleen mappen
+  (mtime van mappen wordt niet teruggezet, dus nieuwe mtime en etag; bestanden ongemoeid). "Removed":
+  o.a. `cache` van een gebruiker, die de backup uitsluit. Waarschijnlijk verklaart dit ook de 50.080
+  "updated" op nc01, niet op nc01 zelf vastgesteld.
+  **Naar 1.1.0 of 1.2.0 (besluit 2026-11-06):** alle mappen krijgen bij een restore hun oorspronkelijke
+  tijd terug. Hoort zo, vooral bij oudere verzamelingen: de maptijd wordt gebruikt om in te schatten
+  wanneer een map is aangemaakt. De backup moet dan de tijd van alle mappen vastleggen en de restore
+  die aan het eind terugzetten; werkt pas vanaf de eerste backup met die versie. Op de roadmap onder 1.1.0.
+- **Branches (besluit 2026-11-06):** werk voor een versie op een branch met die naam (`1.0.3`);
+  `polish` alleen voor kleine cosmetische zaken, per geval bekijken of en wanneer die gemerged
+  worden. SMB staat uit op sub5 en sub6; sub6 is het testaccount, eerst aanzetten voor een SMB-test.
 - **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
   1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
   niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij

README.md

@@ -7,7 +7,7 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
 see "Storage backend experiences" below — that requirement comes from files_external, not from
 ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
 
-**Status: 1.0.1, stable.** Disaster recovery has been tested on real installations (see "Tested
+**Status: 1.0.3, stable.** Disaster recovery has been tested on real installations (see "Tested
 disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
 `docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
 `docs/DISASTER-RECOVERY.md`.
@@ -168,6 +168,8 @@ Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECO
 | Backup | Location | Database | Result |
 |---|---|---|---|
 | Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
+| The same, 1.0.2 | Hetzner Storage Box (SMB), Germany → Canada | PostgreSQL → PostgreSQL | 150,755 files restored in about 29 h (7-13 MiB/s), going on after a lost connection; sample identical, empty folders not restored ([#27](https://github.com/dillardblom/ng_backup/issues/27)) |
+| 1.0.3, small simulations of 2 GB: as planned, with the restore killed halfway, and with a disk that runs full | Local SFTP server | PostgreSQL → PostgreSQL and MariaDB | all 3,101 user files and all empty folders identical; users, a share and a calendar back |
 | 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
 | The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
 | The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
@@ -264,22 +266,12 @@ What is planned, without dates. Ideas and votes are welcome in the
 [issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
 a 👍 on its issue helps decide what comes first.
 
-**1.0.2**
-- A warning on the admin page and in `occ backup:status` when background jobs do not run on a
-  system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on
-  while the NG Backup page is open, and the database phase of a backup shows its progress
-  ([#13](https://github.com/dillardblom/ng_backup/issues/13)).
-- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4.
-- `--salvage` for a restore: restore what can be read, skip a file that cannot, and list those at
-  the end instead of stopping.
-- The trash of restore points (most of them moved there by the retention policy) on a tab of its
-  own, "Bin", instead of below the list of restore points.
-
 **1.1.0**
 - Restore without the recovery kit: the backup key from the location itself, opened with one of
   your passphrases.
 - A schedule per location, also "every N hours".
 - Locations backed up one after the other instead of at the same time.
+- All folders get their original modification time back on a restore, not only the empty ones.
 - A note on a restore point that you can change afterwards.
 - Pinning a restore point, so the retention policy never removes it (for example the one from
   before an upgrade).
@@ -299,6 +291,10 @@ a 👍 on its issue helps decide what comes first.
 
 On the backlog; know them before you rely on NG Backup:
 
+- **Empty folders only come back from backups made with 1.0.3 or newer.** Older restore points do
+  not contain them.
+- **Folders with files in them show the time of the restore**, not their original modification time.
+  The files keep theirs. On the roadmap.
 - **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
   `config.php`) is not supported: a backup refuses to start there. Folders mounted through the
   External storage app are not in the data directory and are **not included**, without a warning.

appinfo/info.xml

@@ -40,12 +40,12 @@ NG Backup needs Nextcloud's background jobs on system cron (a real cron line, no
 Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very
 welcome: use the "Share your experience" link on the admin page.
 
-If you like NG Backup, don't forget to give the app a star in the app store and the
+**If you like NG Backup, don't forget to give the app a star in the app store and the
-[repository](https://github.com/dillardblom/ng_backup) a star on GitHub!
+[repository](https://github.com/dillardblom/ng_backup) a star on GitHub!**
 
 Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
 you need help with restoring a backup after a crash.]]></description>
-    <version>1.0.1</version>
+    <version>1.0.3</version>
     <licence>agpl</licence>
     <author mail="dillard.blom@ensembia.com">Dillard Blom</author>
     <namespace>NgBackup</namespace>
@@ -56,9 +56,10 @@ you need help with restoring a backup after a crash.]]></description>
     <website>https://github.com/dillardblom/ng_backup</website>
     <bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
     <repository type="git">https://github.com/dillardblom/ng_backup</repository>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/1-backups.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/1-backups.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/2-restore.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/2-restore.png</screenshot>
-    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/3-settings.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/3-settings.png</screenshot>
+    <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/4-bin.png</screenshot>
     <dependencies>
         <php min-version="8.1"/>
         <lib>sodium</lib>

docs/DISASTER-RECOVERY.md

@@ -103,7 +103,9 @@ mode on yourself before, it stays on: switch it off with `occ maintenance:mode -
 If it stops halfway, maintenance mode stays on (the instance may be partly restored). Fix the cause,
 then switch maintenance mode off and run the restore again with `--force`, in one line (the message
 prints it): `occ maintenance:mode --off && occ backup:restore:full <location> <snapshot> --force`.
-Leave out `--install-apps` this time, so that the instance is open only for the seconds until the
+It goes on where it stopped: data files that are already in place with the size and modification
+time of the backup are left, so only the rest comes from the location again (since 1.0.2; before,
+it started over). The database is restored again in full, which takes minutes. Leave out `--install-apps` this time, so that the instance is open only for the seconds until the
 restore has switched maintenance mode on again; apps the first run could not install are listed at
 the end of the restore, install those afterwards.
 If occ then says that Nextcloud or one of the apps requires an upgrade, the database is already
@@ -115,7 +117,18 @@ first. If none is better, `--salvage` restores whatever can be read:
 
     occ backup:restore:full offsite <snapshot> --install-apps --salvage
 
+If a restore already stopped halfway, run it as in step 8: maintenance mode off, with `--force` and
+without `--install-apps`:
+
+    occ maintenance:mode --off && occ backup:restore:full offsite <snapshot> --salvage --force
+
 - Files that cannot be read are left out and listed at the end; everything else is restored.
+- A full disk is not damage: the restore stops, also with `--salvage`, and says so. Make room and
+  run the restore again as in step 8.
+- **On version 1.0.2, check the disk first if the reason is "Size mismatch".** There a full disk
+  looks like damaged data: the message points at `--salvage`, and with `--salvage` the restore goes
+  on and lists the remaining files. The backup itself is fine. Update NG Backup, make room and run
+  the restore again as in step 8.
 - If the database cannot be restored (its dump is damaged, or it does not fit this
   installation), the database stays as the fresh installation made it, and the data directory
   and config of the backup go to a folder of their own in the data directory,
@@ -138,6 +151,12 @@ The command then ends with exit code 2 instead of 0. The same option exists for
 - From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`,
   `redis`), which point at AIO's containers.
 - Log in, open a few files, a calendar and a contact.
+- `occ files:scan` can report many folders as "Updated". That is expected: a restore does not yet put
+  back the modification time of folders with files in them, so they show the time of the restore.
+  The files themselves keep their time and content.
+- Empty folders only come back from backups made with version 1.0.3 or newer. With an older
+  restore point they are gone: `occ files:scan` then removes them from the file cache, so it reports
+  them as removed.
 - Start background jobs again (cron), as in step 2 in reverse.
 
 **10. Point your domain at the new server** and check that clients reconnect.

lib/Backend/StorageBackend.php

@@ -124,7 +124,13 @@ final class StorageBackend implements IBackend, IReadsAhead {
 			// null: not supported or failing for now; false: not opened, which need not mean
 			// missing. Either way the storage's own stream decides.
 		}
-		$fh = $this->storage->fopen($this->abs($path), 'r');
+		try {
+			$fh = $this->storage->fopen($this->abs($path), 'r');
+		} catch (\Exception $e) {
+			// Some storages throw for what is not a missing file (SMB: a connection that broke).
+			// As a BackendException, the caller reads once more and --salvage can go on.
+			throw new BackendException('Cannot read ' . $path . ': ' . $e->getMessage(), 0, $e);
+		}
 		if ($fh === false) {
 			throw new BackendException('Not found: ' . $path);
 		}
@@ -133,7 +139,7 @@ final class StorageBackend implements IBackend, IReadsAhead {
 
 	/**
 	 * Over SFTP and S3 several at a time (SftpReadStream::readMany(), s3Fetch()); elsewhere one
-	 * after the other.
+	 * after the other, over SMB in fewer round trips (smbWhole()).
 	 */
 	public function getMany(array $paths): \Generator {
 		$size = $this->s3Reads ? 16 * self::S3_CONCURRENCY : 8 * SftpReadStream::BATCH;
@@ -149,7 +155,8 @@ final class StorageBackend implements IBackend, IReadsAhead {
 				$fast = $fast === null ? null : array_combine($chunk, array_map(fn (string $a): ?string => $fast[$a] ?? null, $abs));
 			}
 			foreach ($chunk as $path) {
-				$data = $fast[$path] ?? null;
+				// over SMB in fewer round trips, one after the other (smbWhole())
+				$data = $fast[$path] ?? $this->smbWhole($this->abs($path));
 				if ($data === null) {
 					// not read in the batch (or no batches here): the usual way, which also
 					// reports a missing file as such
@@ -411,7 +418,8 @@ final class StorageBackend implements IBackend, IReadsAhead {
 				}
 			}
 		} catch (\Throwable) {
-			// read the usual way below, which reports what is wrong
+			// read the usual way below, which reports what is wrong, over a new connection
+			$this->smbReconnect();
 		}
 		$fh = $this->get($path);
 		try {
@@ -543,6 +551,126 @@ final class StorageBackend implements IBackend, IReadsAhead {
 	 * @psalm-suppress UndefinedClass icewind/smb comes with files_external, not with this app
 	 */
 	private function smbChunks(string $abs, int $offset, int $length): ?\Generator {
+		$handle = $this->smbHandle($abs);
+		if ($handle === null) {
+			return null;
+		}
+		[$state, $url] = $handle;
+		/** @psalm-suppress MixedMethodCall the state is icewind's NativeState */
+		return (static function () use ($state, $url, $offset, $length): \Generator {
+			$fh = $state->open($url, 'r');
+			try {
+				$state->lseek($fh, $offset, SEEK_SET, $url);
+				for ($got = 0; $got < $length;) {
+					$want = min(self::SMB_READ, $length - $got);
+					$chunk = self::smbRead($state, $fh, $want, $url);
+					if ($chunk === '') {
+						throw new BackendException('Short read in ' . $url);
+					}
+					$got += strlen($chunk);
+					yield $chunk;
+				}
+			} finally {
+				$state->close($fh, $url);
+			}
+		})();
+	}
+
+	/**
+	 * A whole (small) file over SMB in four round trips: open, read, read up to the end, close.
+	 * files_external's fopen() first checks that the file exists (a stat with its attributes) and
+	 * icewind's stream reads 1 MiB at a time: about 7 round trips, 1 s per index file from Canada
+	 * to the Storage Box (130 ms), so ~8 h for the 27,000 index files of a 1 TB repository. Null
+	 * when the storage is not SMB through the extension or anything fails: the usual way then
+	 * reads the file and reports a missing one as such.
+	 *
+	 * @psalm-suppress UndefinedClass icewind/smb comes with files_external, not with this app
+	 */
+	private function smbWhole(string $abs): ?string {
+		$handle = $this->smbHandle($abs);
+		if ($handle === null) {
+			return null;
+		}
+		[$state, $url] = $handle;
+		try {
+			/** @psalm-suppress MixedMethodCall the state is icewind's NativeState */
+			$fh = $state->open($url, 'r');
+		} catch (\Throwable) {
+			return null;
+		}
+		$data = '';
+		try {
+			do {
+				$chunk = self::smbRead($state, $fh, self::SMB_READ, $url);
+				$data .= $chunk;
+			} while ($chunk !== '');
+		} catch (\Throwable) {
+			$data = null;
+		}
+		try {
+			/** @psalm-suppress MixedMethodCall */
+			$state->close($fh, $url);
+		} catch (\Throwable) {
+		}
+		if ($data === null) {
+			$this->smbReconnect(); // a read failed: the usual way reads it over a new connection
+		}
+		return $data;
+	}
+
+	/**
+	 * One read of at most $bytes through the native state, checked. When libsmbclient fails a read
+	 * (seen when the connection to the server dropped during a long restore), the smbclient
+	 * extension does not return false but a string whose length is -1, because it stores
+	 * smbc_read()'s -1 in an unsigned length; icewind passes it on. Counting that as data made
+	 * the read loop go backwards forever at 100% CPU (DR test 2026-11-05, after 392 GiB).
+	 *
+	 * @param resource $fh
+	 * @psalm-suppress MixedMethodCall, MixedArgument the state is icewind's NativeState
+	 */
+	private static function smbRead(object $state, $fh, int $bytes, string $url): string {
+		$chunk = $state->read($fh, $bytes, $url);
+		$n = is_string($chunk) ? strlen($chunk) : -1;
+		if ($n < 0 || $n > $bytes) {
+			throw new BackendException('Read failed in ' . $url);
+		}
+		return $chunk;
+	}
+
+	/**
+	 * After a failed read: drop icewind's connection state, so the next read (the usual way, or
+	 * the next range) connects again instead of failing on a connection that is gone.
+	 *
+	 * @psalm-suppress UndefinedClass icewind/smb comes with files_external, not with this app
+	 */
+	private function smbReconnect(): void {
+		if (!$this->storage instanceof \OCA\Files_External\Lib\Storage\SMB) {
+			return;
+		}
+		try {
+			(function (): void {
+				/** @psalm-suppress UndefinedThisPropertyFetch */
+				$share = $this->share;
+				if ($share instanceof \Icewind\SMB\Native\NativeShare) {
+					(function (): void {
+						/** @psalm-suppress UndefinedThisPropertyAssignment */
+						$this->state = null;
+					})->call($share);
+				}
+			})->call($this->storage);
+		} catch (\Throwable) {
+		}
+	}
+
+	/**
+	 * icewind's NativeState and the smb:// URL of $abs behind files_external's SMB storage, reached
+	 * through bound closures; null when the storage is not SMB through the native extension or its
+	 * internals are not as expected.
+	 *
+	 * @return array{0: object, 1: string}|null
+	 * @psalm-suppress UndefinedClass icewind/smb comes with files_external, not with this app
+	 */
+	private function smbHandle(string $abs): ?array {
 		if (!$this->storage instanceof \OCA\Files_External\Lib\Storage\SMB || !extension_loaded('smbclient')) {
 			return null;
 		}
@@ -562,27 +690,7 @@ final class StorageBackend implements IBackend, IReadsAhead {
 		} catch (\Throwable) {
 			return null;
 		}
-		if ($handle === null) {
+		return $handle;
-			return null;
-		}
-		[$state, $url] = $handle;
-		/** @psalm-suppress MixedMethodCall the state is icewind's NativeState */
-		return (static function () use ($state, $url, $offset, $length): \Generator {
-			$fh = $state->open($url, 'r');
-			try {
-				$state->lseek($fh, $offset, SEEK_SET, $url);
-				for ($got = 0; $got < $length;) {
-					$chunk = (string)$state->read($fh, min(self::SMB_READ, $length - $got), $url);
-					if ($chunk === '') {
-						throw new BackendException('Short read in ' . $url);
-					}
-					$got += strlen($chunk);
-					yield $chunk;
-				}
-			} finally {
-				$state->close($fh, $url);
-			}
-		})();
 	}
 
 	/**
@@ -602,12 +710,15 @@ final class StorageBackend implements IBackend, IReadsAhead {
 			try {
 				foreach ($abs as $i => [$path, $offset, $length]) {
 					$fh = null;
+					$out = false;
 					try {
 						$chunks = $this->smbChunks($path, $offset, $length);
 						$out = $chunks === null ? false : fopen($file, 'w+b');
 						if ($chunks !== null && $out !== false) {
 							foreach ($chunks as $chunk) {
-								fwrite($out, $chunk);
+								if (fwrite($out, $chunk) !== strlen($chunk)) {
+									throw new BackendException('Cannot write to the spool');
+								}
 							}
 							$fh = ftell($out) === $length && rewind($out) ? $out : null;
 							if ($fh === null) {
@@ -615,7 +726,11 @@ final class StorageBackend implements IBackend, IReadsAhead {
 							}
 						}
 					} catch (\Throwable) {
-						$fh = null; // read the usual way, which reports why
+						if (is_resource($out)) {
+							fclose($out);
+						}
+						$fh = null; // read the usual way, which reports why, over a new connection
+						$this->smbReconnect();
 					} finally {
 						@unlink($file); // an open stream keeps its data
 					}

lib/Command/Restore.php

@@ -67,7 +67,7 @@ HELP);
 				}, $skipped);
 		} catch (\Throwable $e) {
 			$output->writeln('<error>' . $e->getMessage() . '</error>');
-			if ($skipped === null && ($e instanceof \OCA\NgBackup\Repository\RepositoryException || $e instanceof \OCA\NgBackup\Crypto\CryptoException)) {
+			if ($skipped === null && self::backupLooksDamaged($e)) {
 				$output->writeln('The data of a file on the location is damaged or missing. Run the same command with --salvage to restore everything else and list what could not be restored.');
 			}
 			return 1;
@@ -76,6 +76,13 @@ HELP);
 		return self::reportSkipped($skipped, $output);
 	}
 
+	/** Damaged data, or a file that is gone from the location: what --salvage is for. */
+	public static function backupLooksDamaged(\Throwable $e): bool {
+		return $e instanceof \OCA\NgBackup\Repository\RepositoryException
+			|| $e instanceof \OCA\NgBackup\Crypto\CryptoException
+			|| ($e instanceof \OCA\NgBackup\Backend\BackendException && str_starts_with($e->getMessage(), 'Not found'));
+	}
+
 	/**
 	 * The files --salvage left out; exit code 2 when there are any.
 	 *

lib/Command/RestoreFull.php

@@ -111,7 +111,7 @@ HELP);
 				$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
 			}
 			$output->writeln('If occ then says that Nextcloud needs an upgrade (the database is already the old server\'s), run occ upgrade first.');
-			if (!$input->getOption('salvage') && ($e instanceof \OCA\NgBackup\Repository\RepositoryException || $e instanceof \OCA\NgBackup\Crypto\CryptoException)) {
+			if (!$input->getOption('salvage') && Restore::backupLooksDamaged($e)) {
 				$output->writeln('The backup itself looks damaged here. --salvage restores everything that can be read and lists the rest.');
 			}
 			return 1;

lib/Controller/AdminApiController.php

@@ -82,6 +82,8 @@ final class AdminApiController extends Controller {
 
 	/** Seconds of backup work one tick() does: well within a normal web request. */
 	public const TICK_SECONDS = 10.0;
+	/** Seconds one request for the Bin reads, before the page shows how far it is and asks for more. */
+	private const TRASH_PAGE_SECONDS = 5.0;
 
 	/**
 	 * Move running backups forward from the open admin page, when background jobs do not run on a
@@ -95,7 +97,10 @@ final class AdminApiController extends Controller {
 		}
 		$deadline = microtime(true) + self::TICK_SECONDS;
 		$stepped = 0;
-		foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) {
+		$running = $this->runs->findRunning(null, BackupService::KIND_FULL);
+		// Longest wait first: a run that takes all of a tick does not keep another from its turn.
+		usort($running, static fn ($a, $b) => $a->getUpdatedAt() <=> $b->getUpdatedAt());
+		foreach ($running as $run) {
 			if (microtime(true) >= $deadline) {
 				break;
 			}
@@ -174,15 +179,18 @@ final class AdminApiController extends Controller {
 		}
 	}
 
-	public function trash(int $targetId): JSONResponse {
+	public function trash(int $targetId, string $after = ''): JSONResponse {
 		try {
 			$delay = $this->keys->deleteDelayDays() * 86400;
+			$repo = $this->targets->repository($this->targets->get((string)$targetId));
+			// In parts, so that the page can show how far it is (a read per removed restore point).
+			$page = $repo->trashPage($after, microtime(true) + self::TRASH_PAGE_SECONDS);
 			$out = [];
-			foreach ($this->targets->repository($this->targets->get((string)$targetId))->trash() as $id => $info) {
+			foreach ($page['items'] as $id => $info) {
 				$out[] = ['id' => $id, 'label' => $info['label'], 'time' => $info['time'], 'removedAt' => $info['forgottenAt'],
 					'by' => $info['by'], 'deleteAfter' => $info['forgottenAt'] + $delay];
 			}
-			return new JSONResponse($out);
+			return new JSONResponse(['total' => $page['total'], 'items' => $out]);
 		} catch (\Throwable $e) {
 			return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
 		}

lib/Db/DbDumper.php

@@ -157,17 +157,27 @@ final class DbDumper {
 		}
 		// Tables the backup knows the structure of are created first (DDL is not transactional
 		// everywhere); tables it does not know are left out, as asked.
-		foreach ($missing as $table) {
+		try {
-			if (isset($manifest['tables'][$table]['schema'])) {
+			foreach ($missing as $table) {
-				$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
+				if (isset($manifest['tables'][$table]['schema'])) {
-				$report['created'][] = $table;
+					$report['created'][] = $table; // first: creating it can stop halfway (its indexes)
-			} else {
+					$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
-				$report['missing'][] = $table;
+				} else {
+					$report['missing'][] = $table;
+				}
 			}
+		} catch (\Throwable $e) {
+			$this->dropCreatedTables($report);
+			throw $e;
 		}
 		$total = 0;
 		$foreignKeys = [];
-		$this->db->beginTransaction();
+		try {
+			$this->db->beginTransaction();
+		} catch (\Throwable $e) {
+			$this->dropCreatedTables($report);
+			throw $e;
+		}
 		try {
 			$foreignKeys = $this->suspendForeignKeys(); // first: on SQLite it also takes the write lock
 			foreach ($manifest['tables'] as $table => $info) {
@@ -220,21 +230,6 @@ final class DbDumper {
 			$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
 			$this->resumeForeignKeys($foreignKeys);
 			$this->db->commit();
-			// Foreign keys of the tables created above, now their rows are in. Not inside the
-			// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is
-			// restored either way.
-			foreach ($report['created'] as $table) {
-				$structure = $manifest['tables'][$table]['schema'] ?? [];
-				foreach ($structure['foreignKeys'] ?? [] as $fk) {
-					try {
-						$this->db->createSchemaManager()->createForeignKey(
-							new \Doctrine\DBAL\Schema\ForeignKeyConstraint($fk['columns'], $fk['table'], $fk['foreign'], $fk['name'], $fk['options'] ?? []),
-							$table);
-					} catch (\Throwable $e) {
-						$report['foreignKeyErrors'][] = $table . '.' . $fk['name'] . ': ' . $e->getMessage();
-					}
-				}
-			}
 		} catch (\Throwable $e) {
 			$this->db->rollBack();
 			// PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session
@@ -242,11 +237,44 @@ final class DbDumper {
 			if (in_array('SET FOREIGN_KEY_CHECKS = 1', $foreignKeys, true)) {
 				$this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 1');
 			}
+			$this->dropCreatedTables($report);
 			throw $e;
 		}
+		// Committed. Foreign keys of the tables created above, now their rows are in. Not inside the
+		// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is restored
+		// either way, and the created tables stay.
+		foreach ($report['created'] as $table) {
+			$structure = $manifest['tables'][$table]['schema'] ?? [];
+			foreach ($structure['foreignKeys'] ?? [] as $fk) {
+				try {
+					$this->db->createSchemaManager()->createForeignKey(
+						new \Doctrine\DBAL\Schema\ForeignKeyConstraint($fk['columns'], $fk['table'], $fk['foreign'], $fk['name'], $fk['options'] ?? []),
+						$table);
+				} catch (\Throwable $e) {
+					$report['foreignKeyErrors'][] = $table . '.' . $fk['name'] . ': ' . $e->getMessage();
+				}
+			}
+		}
 		return $total;
 	}
 
+	/**
+	 * The restore did not go through: a rollback does not undo CREATE TABLE everywhere. Left behind,
+	 * the empty tables of an app would stop that app's installation later ("table already exists").
+	 *
+	 * @param array{created: list<string>, ...} $report
+	 */
+	private function dropCreatedTables(array &$report): void {
+		foreach ($report['created'] as $table) {
+			try {
+				$this->db->createSchemaManager()->dropTable($table);
+			} catch (\Throwable) {
+				// the error that brought us here is the one to report; a run again uses the table
+			}
+		}
+		$report['created'] = [];
+	}
+
 	/**
 	 * The table prefix of the instance the dump was made on (from its appconfig table), or null
 	 * when the dump has none.

lib/Job/BackupRun.php

@@ -55,9 +55,32 @@ final class BackupRun {
 			});
 			$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
 			$lines = [];
+			$emptyDirs = [];
 			$finishedWalk = true;
 
 			foreach ($walker->walk($state['after']) as $logical => $path) {
+				if (!mb_check_encoding($logical, 'UTF-8') || !Repository::isSafePath($logical)) {
+					// A name a snapshot cannot hold (a backslash, a control character, not UTF-8). One
+					// such path must not make a backup fail, or the whole snapshot unreadable later.
+					// The cursor stays where it is: the state is JSON, which cannot hold such a name. A
+					// next step meets it again, and it is reported once.
+					self::recordUnreadable($state, $stats, mb_scrub($logical, 'UTF-8'));
+					continue;
+				}
+				if (is_dir($path)) {
+					$mtime = @filemtime($path);
+					if ($mtime === false) {
+						continue; // removed while walking
+					}
+					$emptyDirs[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR);
+					$stats['emptyDirs'] = ($stats['emptyDirs'] ?? 0) + 1;
+					$state['after'] = $logical;
+					if (microtime(true) >= $deadline) {
+						$finishedWalk = false;
+						break;
+					}
+					continue;
+				}
 				$version = Repository::fileVersion($path);
 				if ($version === 'missing') {
 					continue; // deleted while walking
@@ -136,6 +159,11 @@ final class BackupRun {
 				$writer->write(implode("\n", $lines) . "\n");
 				$state['segments'][] = $writer->finish();
 			}
+			if ($emptyDirs !== []) {
+				$writer = $repo->blobWriter($stats);
+				$writer->write(implode("\n", $emptyDirs) . "\n");
+				$state['dirSegments'][] = $writer->finish();
+			}
 			$repo->flushPacks();
 			$state['parentPos'] = $parent->position();
 			$state['stats'] = $stats;
@@ -150,10 +178,18 @@ final class BackupRun {
 		if ($state['phase'] === 'finish') {
 			// Every segment ends with a newline, so they read as one stream: fetched together,
 			// several at a time where the location allows it.
+			// Empty folders go in the meta record, not between the files: a version that does not
+			// know them reads the snapshot as before, and only restores no empty folders.
+			$emptyDirs = [];
+			foreach (isset($state['dirSegments']) ? $repo->readLines(array_values(array_merge(...$state['dirSegments']))) : [] as $line) {
+				if ($line !== '') {
+					$emptyDirs[] = json_decode($line, true, 8, JSON_THROW_ON_ERROR);
+				}
+			}
 			$repo->putSnapshot($state['snapshot'],
 				['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
 					'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
-					'warnings' => self::warnings($state)] + $state['meta'],
+					'warnings' => self::warnings($state)] + ($emptyDirs === [] ? [] : ['emptyDirs' => $emptyDirs]) + $state['meta'],
 				$repo->readLines(array_values(array_merge(...$state['segments']))));
 			$state['phase'] = 'commit';
 		}

lib/Job/TreeWalker.php

@@ -9,7 +9,9 @@ namespace OCA\NgBackup\Job;
 
 /**
  * Walks one or more source roots in a stable depth-first order (entries of a directory sorted
- * by name, children right after their directory), yielding "<root>/<relative path>" for files.
+ * by name, children right after their directory), yielding "<root>/<relative path>" for files
+ * and for directories with nothing in them to walk (the caller tells them apart with is_dir()).
+ * A root itself without a name is never yielded.
  *
  * The same order is used for comparing paths (compare()), so a walk can resume after a cursor
  * and be merge-joined with the previous snapshot without holding any list in memory.
@@ -66,6 +68,7 @@ final class TreeWalker {
 		}
 		$names = array_values(array_diff($names, ['.', '..']));
 		sort($names, SORT_STRING);
+		$empty = true; // no file or directory below it that is walked, whatever the cursor skips
 		foreach ($names as $name) {
 			$childLogical = $logical === '' ? $name : $logical . '/' . $name;
 			$childAbs = $abs . '/' . $name;
@@ -73,20 +76,26 @@ final class TreeWalker {
 				continue; // never follow symlinks out of the data directory
 			}
 			if (is_dir($childAbs)) {
+				$empty = $empty && $this->isExcluded($childLogical, true);
 				// Skip a directory entirely if everything in it sorts before the cursor.
 				if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) {
 					continue;
 				}
 				yield from $this->walkDir($childAbs, $childLogical, $after);
 			} elseif (is_file($childAbs)) {
+				$excluded = $this->isExcluded($childLogical, false);
+				$empty = $empty && $excluded;
 				if ($after !== null && self::compare($childLogical, $after) <= 0) {
 					continue;
 				}
-				if (!$this->isExcluded($childLogical, false)) {
+				if (!$excluded) {
 					yield $childLogical => $childAbs;
 				}
 			}
 		}
+		if ($empty && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) {
+			yield $logical => $abs;
+		}
 	}
 
 	private function isExcluded(string $logical, bool $isDir): bool {

lib/Repository/Repository.php

@@ -44,6 +44,8 @@ final class Repository {
 	private ?int $maxBytes = null;
 	private Catalog $catalog;
 	private PackReader $packReader;
+	/** @var array<string, array<string, mixed>> meta record of the snapshot read last */
+	private array $lastMeta = [];
 	/** verify() lists the pack folders from this many packs on (see presentPacks()); tests lower it. */
 	public int $listPacksAbove = 256;
 
@@ -224,14 +226,18 @@ final class Repository {
 	 * @param array<string, string>|null $skipped salvage: an array to collect the files that could
 	 *             not be read or decrypted (path => reason), which are then left out instead of
 	 *             stopping the restore; null (default) stops at the first such file
+	 * @param bool $keepUnchanged leave a file that is already in place with the size and
+	 *             modification time of the snapshot, as a backup takes such a file as unchanged:
+	 *             a restore that was stopped and runs again goes on where it was, without reading
+	 *             what it already restored from the location again
 	 */
-	public function restore(string $snapshotId, string $target, string $prefix = '', bool $stripPrefix = false, ?callable $heartbeat = null, ?array &$skipped = null): int {
+	public function restore(string $snapshotId, string $target, string $prefix = '', bool $stripPrefix = false, ?callable $heartbeat = null, ?array &$skipped = null, bool $keepUnchanged = false): int {
 		if ($prefix !== '' && !self::isSafePath(rtrim($prefix, '/'))) {
 			throw new RepositoryException('Invalid path prefix');
 		}
-		$plan = $this->readAhead($snapshotId, $prefix, $target);
+		$plan = $this->readAhead($snapshotId, $prefix, $target, $stripPrefix, $keepUnchanged);
 		try {
-			return $this->restoreEntries($snapshotId, $target, $prefix, $stripPrefix, $heartbeat, $plan, $skipped);
+			return $this->restoreEntries($snapshotId, $target, $prefix, $stripPrefix, $heartbeat, $plan, $skipped, $keepUnchanged);
 		} finally {
 			$plan?->close();
 			if ($plan !== null) {
@@ -245,13 +251,23 @@ final class Repository {
 	 * pass over the snapshot lists the ranges of packs it needs, in order. The spool is a
 	 * folder in the target, which has room for the restore anyway.
 	 */
-	private function readAhead(string $snapshotId, string $prefix, string $target): ?ReadAheadPlan {
+	private function readAhead(string $snapshotId, string $prefix, string $target, bool $stripPrefix, bool $keepUnchanged): ?ReadAheadPlan {
 		if (!$this->backend instanceof IReadsAhead || !$this->backend->canReadAhead()) {
 			return null;
 		}
-		// what a restore that was stopped left behind: empty, its files were unlinked when opened
+		// what a restore that was stopped left behind: its files are unlinked once opened, but one
+		// that was being written when the process was killed stays (8 MiB, DR test 2026-11-05)
 		$stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR);
 		foreach ($stale === false ? [] : $stale as $dir) {
+			if (is_link($dir) || preg_match('/^\.ngb-restore-spool-[0-9a-f]{8}$/', basename($dir)) !== 1) {
+				continue; // not a spool of ours, whatever it is called
+			}
+			$files = glob($dir . '/*');
+			foreach ($files === false ? [] : $files as $file) {
+				if (is_file($file) && !is_link($file)) {
+					@unlink($file);
+				}
+			}
 			@rmdir($dir);
 		}
 		$plan = new ReadAheadPlan(rtrim($target, '/') . '/.ngb-restore-spool-' . bin2hex(random_bytes(4)));
@@ -260,6 +276,9 @@ final class Repository {
 				if (!self::underPrefix($entry['p'], $prefix)) {
 					continue;
 				}
+				if ($keepUnchanged && self::inPlace(self::destination($target, $entry['p'], $prefix, $stripPrefix), $entry)) {
+					continue;
+				}
 				foreach ($entry['b'] as $id) {
 					try {
 						[$pack, $offset, $length] = $this->index->get($id);
@@ -289,46 +308,46 @@ final class Repository {
 	}
 
 	/** @param array<string, string>|null $skipped see restore() */
-	private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan, ?array &$skipped): int {
+	private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan, ?array &$skipped, bool $keepUnchanged): int {
 		$count = 0;
 		$cleaned = [];
 		foreach ($this->streamEntries($snapshotId) as $entry) {
 			if (!self::underPrefix($entry['p'], $prefix)) {
 				continue;
 			}
-			$relPath = $entry['p'];
+			if ($heartbeat !== null) {
-			if ($stripPrefix && $prefix !== '') {
+				$heartbeat(); // also for a long row of files that are skipped or have no data
-				$relPath = $entry['p'] === rtrim($prefix, '/') ? basename($entry['p']) : substr($entry['p'], strlen($prefix));
 			}
-			$dest = rtrim($target, '/') . '/' . $relPath;
+			$dest = self::destination($target, $entry['p'], $prefix, $stripPrefix);
-			if (!is_dir(dirname($dest))) {
+			$dir = self::directoryInTarget($target, dirname($dest), $entry['p']);
-				mkdir(dirname($dest), 0700, true);
+			if (is_link($dest)) {
-			}
+				throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $entry['p']);
-			// Defence in depth: the resolved directory must stay below the target (no symlink escapes).
-			$base = realpath($target);
-			$dir = realpath(dirname($dest));
-			if ($base === false || $dir === false || ($dir !== $base && !str_starts_with($dir, $base . '/')) || is_link($dest)) {
-				throw new RepositoryException('Refusing to write outside the restore directory: ' . $entry['p']);
 			}
 			if (!isset($cleaned[$dir])) {
 				self::removeStaleTemporaries($dir);
 				$cleaned[$dir] = true;
 			}
+			if ($keepUnchanged && self::inPlace($dest, $entry)) {
+				$count++;
+				continue;
+			}
 			// Write next to the destination and rename after all blobs verified: at most one file
 			// is ever present twice.
 			$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
 			$out = @fopen($tmp, 'xb'); // fails if anything (including a symlink) already exists there
 			if ($out === false) {
-				throw new RepositoryException('Cannot create a temporary file for ' . $entry['p']);
+				throw new RestoreTargetException('Cannot create a temporary file for ' . $entry['p'] . ' (disk full, or no permission?)');
 			}
 			try {
 				foreach ($entry['b'] as $id) {
-					fwrite($out, $this->restoreBlob($id, $plan));
+					self::writeAll($out, $this->restoreBlob($id, $plan), $entry['p']);
 					if ($heartbeat !== null) {
 						$heartbeat();
 					}
 				}
-				fclose($out);
+				if (!@fclose($out)) {
+					throw new RestoreTargetException('Cannot write ' . $entry['p'] . ' to the restore directory (disk full?)');
+				}
 				clearstatcache(true, $tmp);
 				if (filesize($tmp) !== $entry['s']) {
 					throw new RepositoryException('Size mismatch restoring ' . $entry['p']);
@@ -345,19 +364,132 @@ final class Repository {
 				}
 				$skipped[$entry['p']] = $e->getMessage();
 				continue;
+			} catch (\Throwable $e) {
+				if (is_resource($out)) {
+					@fclose($out);
+				}
+				@unlink($tmp);
+				throw $e;
 			}
 			if (!rename($tmp, $dest)) {
 				@unlink($tmp);
-				throw new RepositoryException('Cannot put the restored file in place: ' . $entry['p']);
+				throw new RestoreTargetException('Cannot put the restored file in place: ' . $entry['p']);
 			}
 			if (!touch($dest, $entry['m'])) {
-				throw new RepositoryException('Cannot set the modification time of ' . $entry['p']);
+				throw new RestoreTargetException('Cannot set the modification time of ' . $entry['p']);
 			}
 			$count++;
 		}
+		// Last: a file can only be restored into a folder that exists, and the meta record was
+		// read along with the files, so this asks nothing more of the location.
+		foreach ($this->emptyDirs($snapshotId) as [$path, $mtime]) {
+			if (self::underPrefix($path, $prefix)) {
+				if ($heartbeat !== null) {
+					$heartbeat();
+				}
+				// The prefix itself, stripped, is the target (for a file it is the file's name there).
+				$dest = $stripPrefix && $path === rtrim($prefix, '/') ? rtrim($target, '/') : self::destination($target, $path, $prefix, $stripPrefix);
+				self::restoreEmptyDir($target, $dest, $path, $mtime);
+			}
+		}
 		return $count;
 	}
 
+	/**
+	 * The folders without anything in them that a snapshot recorded (none before 1.0.3).
+	 *
+	 * @return list<array{0: string, 1: int}> path, modification time
+	 */
+	public function emptyDirs(string $snapshotId): array {
+		$meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId);
+		$dirs = [];
+		foreach (is_array($meta['emptyDirs'] ?? null) ? $meta['emptyDirs'] : [] as $dir) {
+			// One folder that makes no sense is no reason to give up on the files of a restore.
+			if (is_array($dir) && is_string($dir[0] ?? null) && self::isSafePath($dir[0]) && is_int($dir[1] ?? null)) {
+				$dirs[] = [$dir[0], $dir[1]];
+			}
+		}
+		return $dirs;
+	}
+
+	private static function restoreEmptyDir(string $target, string $dest, string $path, int $mtime): void {
+		if (is_link($dest)) {
+			throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
+		}
+		$existed = is_dir($dest);
+		self::directoryInTarget($target, $dest, $path);
+		if (!$existed && !@touch($dest, $mtime)) {
+			throw new RestoreTargetException('Cannot set the modification time of ' . $path);
+		}
+	}
+
+	/**
+	 * Make sure a folder exists below the target, without following a link out of it: what exists
+	 * of the path is checked before anything is created.
+	 *
+	 * @return string the folder's real path
+	 */
+	private static function directoryInTarget(string $target, string $dir, string $path): string {
+		if (!is_dir($target) && !@mkdir($target, 0700, true) && !is_dir($target)) {
+			throw new RestoreTargetException('Cannot create the restore directory ' . $target);
+		}
+		$base = realpath($target);
+		$existing = $dir;
+		while (!file_exists($existing) && !is_link($existing) && dirname($existing) !== $existing) {
+			$existing = dirname($existing);
+		}
+		$real = realpath($existing);
+		if ($base === false || $real === false || ($real !== $base && !str_starts_with($real, $base . '/'))) {
+			throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
+		}
+		if ($existing === $dir) {
+			if (!is_dir($real)) {
+				throw new RestoreTargetException('Cannot create the folder for ' . $path . ': a file is in the way');
+			}
+			return $real;
+		}
+		if (!@mkdir($dir, 0700, true) && !is_dir($dir)) {
+			throw new RestoreTargetException('Cannot create the folder for ' . $path . ' (disk full, no permission, or a file in the way?)');
+		}
+		return (string)realpath($dir);
+	}
+
+	/**
+	 * Write all of $data, or stop: a short write leaves a file that looks damaged only by its size.
+	 * A write may take only part of the data and still be fine; one that takes nothing is not.
+	 *
+	 * @param resource $out
+	 */
+	public static function writeAll($out, string $data, string $path): void {
+		while ($data !== '') {
+			$written = @fwrite($out, $data);
+			if ($written === false || $written === 0) {
+				throw new RestoreTargetException('Cannot write ' . $path . ' to the restore directory (disk full?)');
+			}
+			$data = substr($data, $written);
+		}
+	}
+
+	/** Where restore() writes the snapshot path $path. */
+	private static function destination(string $target, string $path, string $prefix, bool $stripPrefix): string {
+		$relPath = $path;
+		if ($stripPrefix && $prefix !== '') {
+			$relPath = $path === rtrim($prefix, '/') ? basename($path) : substr($path, strlen($prefix));
+		}
+		return rtrim($target, '/') . '/' . $relPath;
+	}
+
+	/**
+	 * A regular file (no link) at $dest with the size and modification time of the snapshot
+	 * entry: what a backup takes as unchanged, and what a finished restore of it leaves.
+	 *
+	 * @param array{s: int, m: int, ...} $entry
+	 */
+	private static function inPlace(string $dest, array $entry): bool {
+		clearstatcache(true, $dest);
+		return !is_link($dest) && is_file($dest) && filesize($dest) === $entry['s'] && filemtime($dest) === $entry['m'];
+	}
+
 	/** $prefix is "" (everything) or ends with "/"; the path without that "/" is a file to restore too. */
 	private static function underPrefix(string $path, string $prefix): bool {
 		return $prefix === '' || str_starts_with($path, $prefix) || $path === rtrim($prefix, '/');
@@ -684,6 +816,34 @@ final class Repository {
 		return array_keys($doomed);
 	}
 
+	/**
+	 * Part of trash(), for a page that shows how far it is: each entry is a read of its own on the
+	 * location. Reads the entries with an id after $after, in order, until $deadline
+	 * (microtime(true)), at least one. The id, not a position: a cleanup that changes the trash
+	 * between two parts then makes no entry come twice or be missed.
+	 *
+	 * @return array{total: int, items: array<string, array<string, mixed>>}
+	 */
+	public function trashPage(string $after, float $deadline): array {
+		$ids = array_map('strval', array_keys($this->catalog->trash));
+		sort($ids, SORT_STRING);
+		$items = [];
+		foreach ($ids as $id) {
+			if (strcmp($id, $after) <= 0) {
+				continue;
+			}
+			try {
+				$items[$id] = $this->trashInfo($id);
+			} catch (\Throwable) {
+				throw new RepositoryException("Trash entry $id listed in the catalog is missing on the location (possible tampering)");
+			}
+			if (microtime(true) >= $deadline) {
+				break;
+			}
+		}
+		return ['total' => count($ids), 'items' => $items];
+	}
+
 	private function trashInfo(string $snapshotId): array {
 		return json_decode($this->getObject('trash/info/' . $snapshotId), true, 512, JSON_THROW_ON_ERROR);
 	}
@@ -1243,7 +1403,10 @@ final class Repository {
 		$this->cipher->decrypt($fh, $plain, 'snapshot:' . $snapshotId);
 		fclose($fh);
 		rewind($plain);
-		fgets($plain); // meta line
+		$meta = json_decode((string)fgets($plain), true);
+		if (is_array($meta)) {
+			$this->lastMeta = [$snapshotId => $meta]; // a snapshot never changes
+		}
 		while (($line = fgets($plain)) !== false) {
 			$entry = json_decode($line, true, 512, JSON_THROW_ON_ERROR);
 			if (!is_array($entry) || !is_string($entry['p'] ?? null) || !self::isSafePath($entry['p'])

lib/Repository/RestoreTargetException.php

@@ -0,0 +1,15 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Repository;
+
+/**
+ * A restore cannot write to where it restores to (a full disk, permissions, something in the way).
+ * The backup is fine, so this never counts as damaged data: the restore stops, also with --salvage.
+ */
+final class RestoreTargetException extends \RuntimeException {
+}

lib/Service/BackupService.php

@@ -79,7 +79,7 @@ final class BackupService {
 		try {
 			return $this->leases->with(PruneService::lockKey($target), LeaseService::EXCLUSIVE, 60, fn () => $this->createRun($target, $label));
 		} catch (LockedException) {
-			throw new \RuntimeException('A cleanup or another start for ' . $target->getName() . ' is running; try again later');
+			throw new \RuntimeException('A cleanup, a restore or another start for ' . $target->getName() . ' is running, or one was stopped by force (that clears by itself within two hours); try again later');
 		}
 	}
 

lib/Service/DatabaseAlreadyRestoredException.php

@@ -0,0 +1,12 @@
+<?php
+
+declare(strict_types=1);
+
+// SPDX-FileCopyrightText: 2026 Dillard Blom
+// SPDX-License-Identifier: AGPL-3.0-or-later
+
+namespace OCA\NgBackup\Service;
+
+/** A step after the database was committed failed: the database is no longer the fresh one. */
+final class DatabaseAlreadyRestoredException extends \RuntimeException {
+}

lib/Service/LeaseService.php

@@ -76,6 +76,15 @@ final class LeaseService {
 		$qb->executeStatement();
 	}
 
+	/** Remove the shared leases on $name, for a caller that knows their holders were stopped. */
+	public function dropShared(string $name): void {
+		$qb = $this->db->getQueryBuilder();
+		$qb->delete('ngb_leases')
+			->where($qb->expr()->eq('name', $qb->createNamedParameter($name)))
+			->andWhere($qb->expr()->eq('mode', $qb->createNamedParameter(self::SHARED)));
+		$qb->executeStatement();
+	}
+
 	/** Wraps $refresh so calling it often (per file, per row batch) only touches the database every $seconds. */
 	public static function throttled(callable $refresh, int $seconds = 60): \Closure {
 		$last = time();

lib/Service/RestoreFullService.php

@@ -151,6 +151,12 @@ final class RestoreFullService {
 			// The database cannot go into this installation; the files can.
 			$check['dbUnreadable'] = 'the database cannot be restored here: ' . $this->describeBlockers($check);
 		}
+		if ($interrupted) {
+			// A restore that was killed leaves its leases, for up to two hours: they would keep
+			// the first backup of the recovered server waiting after this restore completes.
+			// In maintenance mode, on a partly restored instance, nothing else reads the location.
+			$this->leases->dropShared(PruneService::lockKey($target));
+		}
 		$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
 		$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check, $salvage);
 		$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
@@ -175,6 +181,8 @@ final class RestoreFullService {
 			if ($dbError === null) {
 				try {
 					$this->restoreDatabase($repo, $meta, $manifest, $dbReport, $beat, $skipMissingTables);
+				} catch (DatabaseAlreadyRestoredException $e) {
+					throw $e;
 				} catch (\Throwable $e) {
 					if (!$salvage) {
 						throw $e;
@@ -186,7 +194,7 @@ final class RestoreFullService {
 			}
 			if ($dbError !== null) {
 				$salvageDir = $dataDir . '/ngb-salvage-' . substr($snapshotId, 0, 12);
-				$dataFiles = $this->restore->restoreToDirectory($target, $snapshotId, 'data', $salvageDir . '/data', stripPrefix: true, heartbeat: $beat, skipped: $skipped);
+				$dataFiles = $this->restore->restoreToDirectory($target, $snapshotId, 'data', $salvageDir . '/data', stripPrefix: true, heartbeat: $beat, skipped: $skipped, keepUnchanged: true);
 				$this->restore->restoreToDirectory($target, $snapshotId, 'config', $salvageDir . '/config', stripPrefix: true, heartbeat: $beat, skipped: $skipped);
 				return ['dbReport' => ['skipped' => [], 'restored' => [], 'created' => [], 'missing' => []], 'dataFiles' => $dataFiles, 'configKeys' => 0,
 					'dbConsistent' => true, 'missingApps' => $check['missingApps'], 'skipped' => $skipped ?? [], 'dbError' => $dbError,
@@ -201,6 +209,8 @@ final class RestoreFullService {
 			try {
 				/** @var array<string, mixed> $backupConfig */
 				$backupConfig = $this->fetchBackupConfig($target, $snapshotId, $salvage);
+			} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
+				throw $e; // nowhere to put it: not a config.php that cannot be read
 			} catch (\Throwable $e) {
 				if (!$salvage) {
 					throw $e;
@@ -209,7 +219,7 @@ final class RestoreFullService {
 				$backupConfig = [];
 			}
 
-			$dataFiles = $this->restore->restoreToDirectory($target, $snapshotId, 'data', $dataDir, stripPrefix: true, heartbeat: $beat, skipped: $skipped);
+			$dataFiles = $this->restore->restoreToDirectory($target, $snapshotId, 'data', $dataDir, stripPrefix: true, heartbeat: $beat, skipped: $skipped, keepUnchanged: true);
 
 			// Last step: nothing after this may need $this->targets->repository($target) again.
 			if ($backupConfig !== []) {
@@ -259,8 +269,20 @@ final class RestoreFullService {
 		$dumper = new DbDumper($this->db->getInner(), $prefix);
 		try {
 			$dumper->restore($repo, $manifest, $dbReport, $beat, $skipMissingTables);
-		} finally {
+		} catch (\Throwable $e) {
+			try {
+				$this->restoreOwnAppConfig($ownAppConfig);
+			} catch (\Throwable) {
+				// rolled back, so they are still there: the error that brought us here is the one to report
+			}
+			throw $e;
+		}
+		try {
 			$this->restoreOwnAppConfig($ownAppConfig);
+		} catch (\Throwable $e) {
+			// The old server's database is in place by now: not a case for "the database stays as
+			// the fresh installation made it" under --salvage. A run again restores these rows.
+			throw new DatabaseAlreadyRestoredException('The database was restored, but NG Backup could not put back its own settings: ' . $e->getMessage(), 0, $e);
 		}
 	}
 

lib/Service/RestoreService.php

@@ -61,7 +61,8 @@ final class RestoreService {
 		$prefix = trim($path, '/');
 		$prefix = $prefix === '' ? '' : $prefix . '/';
 		$children = [];
-		foreach ($this->targets->repository($target)->entries($snapshotId) as $e) {
+		$repo = $this->targets->repository($target);
+		foreach ($repo->entries($snapshotId) as $e) {
 			if ($prefix !== '' && !str_starts_with($e['p'], $prefix)) {
 				continue;
 			}
@@ -75,6 +76,12 @@ final class RestoreService {
 			}
 			$children[$name] = $c;
 		}
+		foreach ($repo->emptyDirs($snapshotId) as [$dir]) { // read along with the entries
+			if ($prefix === '' || str_starts_with($dir, $prefix)) {
+				$name = explode('/', substr($dir, strlen($prefix)), 2)[0];
+				$children[$name] ??= ['name' => $name, 'type' => 'dir', 'size' => 0, 'mtime' => null, 'files' => 0];
+			}
+		}
 		ksort($children, SORT_STRING);
 		return array_values($children);
 	}
@@ -123,7 +130,17 @@ final class RestoreService {
 				$entries[substr($e['p'], strlen($source) + 1)] = $e;
 			}
 		}
-		if ($entries === []) {
+		// Folders without anything in them, relative to $source ('' when $source itself is one).
+		/** @var array<array-key, int> $emptyDirs modification times; a folder called "2026" is an integer key */
+		$emptyDirs = [];
+		foreach ($single ? [] : $repo->emptyDirs($snapshotId) as [$dir, $mtime]) {
+			if ($dir === $source) {
+				$emptyDirs[''] = $mtime;
+			} elseif (str_starts_with($dir, $source . '/')) {
+				$emptyDirs[substr($dir, strlen($source) + 1)] = $mtime;
+			}
+		}
+		if ($entries === [] && $emptyDirs === []) {
 			throw new NotFoundException("Nothing at $source in snapshot $snapshotId");
 		}
 
@@ -134,6 +151,7 @@ final class RestoreService {
 		} elseif ($mode !== self::MODE_MERGE && $mode !== self::MODE_REPLACE) {
 			throw new \InvalidArgumentException("Unknown mode $mode");
 		}
+		$destExisted = $parentRel === '' || $userFolder->nodeExists($parentRel);
 		$dest = $this->ensureFolder($userFolder, $parentRel);
 		$names = $single ? [basename($rel) => $entries['']] : $entries;
 
@@ -156,9 +174,21 @@ final class RestoreService {
 			}
 		}
 
+		foreach ($emptyDirs as $dir => $mtime) {
+			$dir = (string)$dir;
+			$existed = $dir === '' ? $destExisted : $dest->nodeExists($dir); // one that is there keeps its own time
+			$folder = $this->ensureFolder($dest, $dir);
+			if (!$existed) {
+				$folder->touch($mtime);
+			}
+			if ($progress !== null) {
+				$progress($restored, count($names), $dir); // keeps the lease alive too
+			}
+		}
+
 		$trashed = 0;
 		if ($mode === self::MODE_REPLACE && !$single) {
-			$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)));
+			$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)) + $emptyDirs);
 		}
 		return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())];
 	}
@@ -169,15 +199,18 @@ final class RestoreService {
 	 * @param bool $stripPrefix write into $directory itself instead of $directory/$prefix (see
 	 *             Repository::restore()); used to restore "data/" or "config/" straight into the
 	 *             real data/config directory for occ backup:restore:full.
+	 * @param bool $keepUnchanged leave files already in place with the snapshot's size and
+	 *             modification time (see Repository::restore()): a full restore that runs again
+	 *             goes on where it stopped
 	 */
-	public function restoreToDirectory(Target $target, string $snapshotId, string $prefix, string $directory, bool $stripPrefix = false, ?callable $heartbeat = null, ?array &$skipped = null): int {
+	public function restoreToDirectory(Target $target, string $snapshotId, string $prefix, string $directory, bool $stripPrefix = false, ?callable $heartbeat = null, ?array &$skipped = null, bool $keepUnchanged = false): int {
 		if ($prefix !== '' && !Repository::isSafePath($prefix)) {
 			throw new \InvalidArgumentException('Invalid path: ' . $prefix);
 		}
 		if (!is_dir($directory) && !mkdir($directory, 0750, true) && !is_dir($directory)) {
-			throw new \RuntimeException("Cannot create $directory");
+			throw new \OCA\NgBackup\Repository\RestoreTargetException("Cannot create $directory (disk full, or no permission?)");
 		}
-		return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $prefix, $directory, $stripPrefix, $heartbeat, &$skipped): int {
+		return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $prefix, $directory, $stripPrefix, $heartbeat, &$skipped, $keepUnchanged): int {
 			$own = LeaseService::throttled($refresh);
 			$beat = static function () use ($own, $heartbeat): void {
 				$own();
@@ -185,7 +218,7 @@ final class RestoreService {
 					$heartbeat();
 				}
 			};
-			return $this->targets->repository($target)->restore($snapshotId, $directory, trim($prefix, '/') === '' ? '' : trim($prefix, '/') . '/', $stripPrefix, $beat, $skipped);
+			return $this->targets->repository($target)->restore($snapshotId, $directory, trim($prefix, '/') === '' ? '' : trim($prefix, '/') . '/', $stripPrefix, $beat, $skipped, $keepUnchanged);
 		}, 3600);
 	}
 
@@ -259,7 +292,7 @@ final class RestoreService {
 		foreach ($folder->getDirectoryListing() as $node) {
 			$rel = $prefix === '' ? $node->getName() : $prefix . '/' . $node->getName();
 			if ($node instanceof Folder) {
-				$hasKept = false;
+				$hasKept = isset($keep[$rel]); // a folder that was empty in the snapshot
 				foreach (array_keys($keep) as $k) {
 					if (str_starts_with((string)$k, $rel . '/')) {
 						$hasKept = true;

psalm-baseline.xml

@@ -186,6 +186,7 @@
       <code><![CDATA[$this->db]]></code>
       <code><![CDATA[$this->db]]></code>
       <code><![CDATA[$this->db]]></code>
+      <code><![CDATA[$this->db]]></code>
       <code><![CDATA[TransactionIsolationLevel]]></code>
       <code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
       <code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>

src/api.js

@@ -21,7 +21,7 @@ export default {
 	addSlot: (passphrase, label) => call('post', '/key/slots', { passphrase, label }),
 	replaceSlot: (slot, passphrase, label) => call('put', `/key/slots/${slot}`, { passphrase, label }),
 	removeSlot: (slot) => call('delete', `/key/slots/${slot}`),
-	trash: (targetId) => call('get', `/targets/${targetId}/trash`),
+	trash: (targetId, after = '') => call('get', `/targets/${targetId}/trash`, { after }),
 	untrash: (targetId, snapshotId) => call('post', `/targets/${targetId}/trash/${snapshotId}`),
 	kitUrl: () => url('/key/kit'),
 	confirmKit: (accepted, phrase, code) => call('post', '/key/confirm', { accepted, phrase, code }),

src/components/Bin.vue

@@ -6,8 +6,18 @@
 	<NcSettingsSection :name="t('ng_backup', 'Bin')"
 		:description="t('ng_backup', 'Restore points that were removed, most of them by the retention policy. They can be brought back until they are deleted permanently.')">
 		<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
-		<NcLoadingIcon v-if="loading" :size="32" />
+		<div v-if="reading" class="ngb-reading" role="status">
-		<NcNoteCard v-else-if="empty" type="info">{{ t('ng_backup', 'The bin is empty.') }}</NcNoteCard>
+			<template v-if="reading.total === null">
+				<NcLoadingIcon :size="20" />
+				<span>{{ t('ng_backup', 'Connecting to {location} …', { location: reading.name }) }}</span>
+			</template>
+			<template v-else>
+				<span>{{ t('ng_backup', 'Reading removed restore points from {location}: {done} of {total}', { location: reading.name, done: reading.done, total: reading.total }) }}</span>
+				<NcProgressBar :value="reading.total ? Math.round(100 * reading.done / reading.total) : 0" />
+			</template>
+			<p class="ngb-muted">{{ t('ng_backup', 'This can take a while: every removed restore point is read from the location.') }}</p>
+		</div>
+		<NcNoteCard v-else-if="empty && !error" type="info">{{ t('ng_backup', 'The bin is empty.') }}</NcNoteCard>
 
 		<template v-for="target in status.targets" :key="'trash' + target.id">
 			<div v-if="trash[target.id]?.length" class="ngb-trash">
@@ -39,12 +49,13 @@
 </template>
 
 <script setup>
-import { computed, reactive, ref, watch } from 'vue'
+import { computed, onBeforeUnmount, reactive, ref, watch } from 'vue'
 import { n, t } from '@nextcloud/l10n'
 import { confirmPassword } from '@nextcloud/password-confirmation'
 import NcButton from '@nextcloud/vue/components/NcButton'
 import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
 import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
+import NcProgressBar from '@nextcloud/vue/components/NcProgressBar'
 import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
 import api from '../api.js'
 import { formatDate, formatDateTime } from '../format.js'
@@ -56,21 +67,49 @@ const emit = defineEmits(['changed'])
 const RETENTION = 'retention policy'
 const retentionInfo = ref(false)
 const trash = reactive({})
-const loading = ref(true)
+const reading = ref({ name: '', total: null, done: 0 }) // the location being read; null when done
 const error = ref('')
 const empty = computed(() => props.status.targets.every(target => !trash[target.id]?.length))
 
-async function loadTrash() {
+let current = 0 // the newest load; an older one stops (another took over, or the tab was left)
-	for (const target of props.status.targets) {
+onBeforeUnmount(() => { current++ })
-		try {
+
-			trash[target.id] = await api.trash(target.id)
+async function loadTarget(target, load) {
-		} catch (e) {
+	reading.value = { name: target.name, total: null, done: 0 }
-			trash[target.id] = []
+	const items = []
+	for (;;) {
+		const page = await api.trash(target.id, items.length ? items[items.length - 1].id : '')
+		if (load !== current) {
+			return
+		}
+		items.push(...page.items)
+		reading.value = { name: target.name, total: page.total, done: Math.min(items.length, page.total) }
+		if (!page.items.length || items.length >= page.total) {
+			break
 		}
 	}
-	loading.value = false
+	trash[target.id] = items.sort((a, b) => String(b.time ?? '').localeCompare(String(a.time ?? '')))
 }
-watch(() => props.status.targets.map(tg => tg.id).join(','), loadTrash, { immediate: true })
+
+async function loadTrash() {
+	const load = ++current
+	const failed = []
+	for (const target of props.status.targets) {
+		try {
+			await loadTarget(target, load)
+		} catch (e) {
+			// keep what was shown: a location that cannot be reached does not mean its bin is empty
+			failed.push(target.name + (e?.message ? ': ' + e.message : ''))
+		}
+		if (load !== current) {
+			return
+		}
+	}
+	error.value = failed.length ? t('ng_backup', 'The bin could not be read from {locations}', { locations: failed.join('; ') }) : ''
+	reading.value = null
+}
+// Again when a run ends: a cleanup moves restore points here.
+watch(() => props.status.targets.map(tg => tg.id).join(',') + '|' + (props.status.runs ?? []).filter(run => run.status !== 'running').map(run => run.id).join(','), loadTrash, { immediate: true })
 
 async function untrash(target, item) {
 	error.value = ''
@@ -88,6 +127,8 @@ async function untrash(target, item) {
 <style scoped>
 .ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); }
 .ngb-muted { color: var(--color-text-maxcontrast); }
+.ngb-reading { margin: 12px 0; display: flex; flex-direction: column; gap: 6px; max-width: 600px; }
+.ngb-reading p { margin: 0; }
 .ngb-trash { margin: 12px 0; }
 .ngb-trash h3 { font-weight: bold; }
 </style>

tests/Unit/Job/TreeWalkerTest.php

@@ -28,6 +28,28 @@ class TreeWalkerTest extends TestCase {
 		$this->assertSame(['Z', 'a', 'a-b', 'a.txt', 'b', 'skip'], array_values(array_unique(array_map(fn ($p) => explode('/', $p)[0], $paths))));
 	}
 
+	public function testEmptyDirectoriesAreYieldedInOrderAndOnResume(): void {
+		$d = $this->tree();
+		mkdir("$d/a/empty");
+		mkdir("$d/a-hollow/inner", 0700, true);
+		symlink("$d/a.txt", "$d/skip/link");
+		mkdir("$d/only-link");
+		symlink("$d/a.txt", "$d/only-link/link");
+		$w = new TreeWalker(['' => $d]);
+		$all = array_keys(iterator_to_array($w->walk()));
+		$this->assertContains('a/empty', $all);
+		$this->assertContains('a-hollow/inner', $all);
+		$this->assertNotContains('a-hollow', $all, 'only the deepest');
+		$this->assertContains('only-link', $all, 'a link is not walked');
+		$this->assertNotContains('skip', $all);
+		$sorted = $all;
+		usort($sorted, [TreeWalker::class, 'compare']);
+		$this->assertSame($sorted, $all);
+		foreach ($all as $i => $cursor) {
+			$this->assertSame(array_slice($all, $i + 1), array_keys(iterator_to_array($w->walk($cursor))), "after $cursor");
+		}
+	}
+
 	public function testResumeAfterCursorYieldsTheRest(): void {
 		$w = new TreeWalker(['' => $this->tree()]);
 		$all = array_keys(iterator_to_array($w->walk()));

tests/Unit/Repository/CatalogTest.php

@@ -50,6 +50,40 @@ class CatalogTest extends TestCase {
 		$this->assertSame(4, $again->catalogGeneration());
 	}
 
+	public function testTheTrashIsReadInPartsThatTogetherAreAll(): void {
+		$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
+		for ($i = 0; $i < 3; $i++) {
+			$repo->forget($repo->backupDirectory($this->src)['snapshot']);
+		}
+		$again = $this->open();
+		$seen = [];
+		// A deadline already past: one entry per request, as on a slow location
+		$after = '';
+		for ($request = 0; count($seen) < 3 && $request < 5; $request++) {
+			$page = $again->trashPage($after, 0.0);
+			$this->assertSame(3, $page['total']);
+			$this->assertCount(1, $page['items']);
+			$seen += $page['items'];
+			$after = (string)array_key_last($page['items']);
+		}
+		$this->assertEqualsCanonicalizing(array_keys($again->trash()), array_keys($seen));
+		$this->assertSame(['total' => 3, 'items' => []], $again->trashPage($after, 0.0));
+		$this->assertCount(3, $again->trashPage('', microtime(true) + 60)['items']);
+	}
+
+	public function testAnEntryThatLeavesTheTrashBetweenPartsMakesNoneComeTwiceOrBeMissed(): void {
+		$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
+		for ($i = 0; $i < 3; $i++) {
+			$repo->forget($repo->backupDirectory($this->src)['snapshot']);
+		}
+		$reader = $this->open();
+		$ids = array_keys($reader->trashPage('', microtime(true) + 60)['items']);
+		$first = $reader->trashPage('', 0.0)['items'];
+		$this->assertSame([$ids[0]], array_keys($first));
+		$this->open()->untrash($ids[0]); // another administrator, between two parts
+		$this->assertSame([$ids[1], $ids[2]], array_keys($this->open()->trashPage($ids[0], microtime(true) + 60)['items']));
+	}
+
 	public function testRollbackIsDetected(): void {
 		$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
 		$repo->backupDirectory($this->src);

tests/Unit/Repository/EmptyDirsTest.php

@@ -0,0 +1,154 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Repository;
+
+use OCA\NgBackup\Backend\LocalBackend;
+use OCA\NgBackup\Crypto\KeyRing;
+use OCA\NgBackup\Job\BackupRun;
+use OCA\NgBackup\Repository\Repository;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use PHPUnit\Framework\TestCase;
+
+/** Folders without files are recorded in the snapshot's meta record and come back on a restore. */
+class EmptyDirsTest extends TestCase {
+	use TempDirTrait;
+
+	private LocalBackend $backend;
+	private KeyRing $keys;
+	private string $src;
+
+	protected function setUp(): void {
+		$this->backend = new LocalBackend($this->tempDir());
+		$this->keys = KeyRing::generate();
+		Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
+		$this->src = $this->tempDir();
+		mkdir("$this->src/data/alice/files/Docs", 0700, true);
+		mkdir("$this->src/data/alice/files/Hollow/Nested", 0700, true);
+		mkdir("$this->src/data/skipped/inner", 0700, true);
+		file_put_contents("$this->src/data/alice/files/Docs/note.txt", "hello\n");
+		touch("$this->src/data/alice/files/Hollow/Nested", 1700000000);
+	}
+
+	private function repo(): Repository {
+		return Repository::openWithKey($this->backend, $this->keys);
+	}
+
+	/** @param list<string> $exclude */
+	private function backup(array $exclude = [], float $seconds = 30.0): string {
+		$state = BackupRun::start(['data' => "$this->src/data"], null, '', $exclude);
+		for ($i = 0; $i < 500 && $state['phase'] !== 'commit'; $i++) {
+			$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, microtime(true) + $seconds)), true);
+		}
+		$this->assertSame('commit', $state['phase']);
+		$this->repo()->recordSnapshot($state['snapshot']);
+		return $state['snapshot'];
+	}
+
+	public function testOnlyTheDeepestEmptyFolderIsRecordedAndFilesStayAsTheyAre(): void {
+		$snapshot = $this->backup();
+		$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
+		$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], $paths);
+		$this->assertSame(['data/alice/files/Docs/note.txt'], array_column(iterator_to_array($this->repo()->entries($snapshot), false), 'p'));
+		$this->assertSame(2, $this->repo()->snapshotMeta($snapshot)['stats']['emptyDirs']);
+	}
+
+	public function testAFolderWithOnlyExcludedContentCountsAsEmpty(): void {
+		$snapshot = $this->backup(['data/skipped/inner']);
+		$this->assertContains('data/skipped', array_column($this->repo()->emptyDirs($snapshot), 0));
+		$this->assertNotContains('data/skipped/inner', array_column($this->repo()->emptyDirs($snapshot), 0));
+	}
+
+	public function testAResumedWalkRecordsEveryFolderOnce(): void {
+		for ($i = 0; $i < 30; $i++) {
+			mkdir("$this->src/data/many/d$i", 0700, true);
+			file_put_contents("$this->src/data/many/f$i", 'x');
+		}
+		$snapshot = $this->backup([], 0.0); // a step per entry
+		$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
+		$this->assertCount(32, $paths);
+		$this->assertSame(array_unique($paths), $paths);
+	}
+
+	public function testANameASnapshotCannotHoldIsReportedAndTheSnapshotStaysReadable(): void {
+		mkdir("$this->src/data/odd\\folder", 0700, true);
+		mkdir("$this->src/data/alice/files/line\nbreak", 0700, true);
+		file_put_contents("$this->src/data/alice/files/back\\slash.txt", 'x');
+		file_put_contents("$this->src/data/alice/files/\xff.txt", 'x');
+		$state = BackupRun::start(['data' => "$this->src/data"]);
+		for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
+			// A step per entry, state through JSON: as separate processes would
+			$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
+		}
+		$this->assertSame('commit', $state['phase']);
+		$this->assertSame(4, $state['stats']['unreadable'], 'each reported once, over several steps');
+		$this->repo()->recordSnapshot($state['snapshot']);
+
+		$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], array_column($this->repo()->emptyDirs($state['snapshot']), 0));
+		$target = $this->tempDir();
+		$this->assertSame(1, $this->repo()->restore($state['snapshot'], $target));
+		$this->assertFileExists("$target/data/alice/files/Docs/note.txt");
+	}
+
+	public function testAnEmptyFolderThatIsThePrefixItselfIsTheTarget(): void {
+		$snapshot = $this->backup();
+		$target = $this->tempDir();
+		$this->repo()->restore($snapshot, $target, 'data/skipped/inner/', true);
+		$this->assertSame([], glob("$target/*"), 'not a folder "inner" inside the target');
+	}
+
+	public function testRestoreBringsThemBackWithTheirTime(): void {
+		$snapshot = $this->backup();
+		$target = $this->tempDir();
+		$this->assertSame(1, $this->repo()->restore($snapshot, $target));
+		$this->assertDirectoryExists("$target/data/alice/files/Hollow/Nested");
+		clearstatcache();
+		$this->assertSame(1700000000, filemtime("$target/data/alice/files/Hollow/Nested"));
+		$this->assertDirectoryExists("$target/data/skipped/inner");
+	}
+
+	public function testRestoreOfAPrefixStripsAndLeavesOthersOut(): void {
+		$snapshot = $this->backup();
+		$target = $this->tempDir();
+		$this->repo()->restore($snapshot, $target, 'data/alice/', true);
+		$this->assertDirectoryExists("$target/files/Hollow/Nested");
+		$this->assertDirectoryDoesNotExist("$target/skipped");
+		$this->assertDirectoryDoesNotExist("$target/inner");
+	}
+
+	public function testRunningAgainLeavesAFolderThatIsThereAlone(): void {
+		$snapshot = $this->backup();
+		$target = $this->tempDir();
+		$this->repo()->restore($snapshot, $target);
+		touch("$target/data/alice/files/Hollow/Nested", 1800000000);
+		file_put_contents("$target/data/alice/files/Hollow/Nested/added", 'kept');
+		touch("$target/data/alice/files/Hollow/Nested", 1800000000);
+		$this->repo()->restore($snapshot, $target, keepUnchanged: true);
+		clearstatcache();
+		$this->assertSame(1800000000, filemtime("$target/data/alice/files/Hollow/Nested"));
+		$this->assertFileExists("$target/data/alice/files/Hollow/Nested/added");
+	}
+
+	public function testALinkOutOfTheTargetIsRefused(): void {
+		$snapshot = $this->backup();
+		$target = $this->tempDir();
+		$outside = $this->tempDir();
+		mkdir("$target/data/alice/files", 0700, true);
+		symlink($outside, "$target/data/alice/files/Hollow");
+		try {
+			$this->repo()->restore($snapshot, $target);
+			$this->fail('restored through a link');
+		} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
+			$this->assertStringContainsString('outside', $e->getMessage());
+		}
+		$this->assertSame([], glob("$outside/*"));
+	}
+
+	public function testASnapshotWithoutThemHasNone(): void {
+		$src = $this->tempDir();
+		file_put_contents("$src/a", 'x');
+		$snapshot = $this->repo()->backupDirectory($src)['snapshot'];
+		$this->assertSame([], $this->repo()->emptyDirs($snapshot));
+	}
+}

tests/Unit/Repository/ReadAheadTest.php

@@ -75,11 +75,14 @@ class ReadAheadTest extends TestCase {
 		$this->assertSame(2, $backend->snapshotReads, 'the restore itself read the snapshot again');
 	}
 
-	public function testTheEmptySpoolOfAStoppedRestoreIsRemoved(): void {
+	public function testTheSpoolOfAStoppedRestoreIsRemoved(): void {
 		$target = $this->tempDir();
 		mkdir("$target/.ngb-restore-spool-0badc0de", 0700); // a restore that was killed
+		mkdir("$target/.ngb-restore-spool-8badf00d", 0700); // ... while it wrote a range
+		file_put_contents("$target/.ngb-restore-spool-8badf00d/smb-01234567", str_repeat('x', 4096));
 
 		$this->assertRestored($this->restore(new ReadAheadBackend(new LocalBackend($this->root)), '', $target));
+		$this->assertSame([], glob("$target/.ngb-restore-spool-*"));
 	}
 
 	public function testARangeThatCouldNotBeReadAheadIsReadTheUsualWay(): void {

tests/Unit/Repository/RepositoryTest.php

@@ -101,6 +101,34 @@ class RepositoryTest extends TestCase {
 		$this->assertFileExists($stripped . '/note-1.txt');
 	}
 
+	public function testARestoreThatRunsAgainKeepsWhatIsInPlaceAndRestoresTheRest(): void {
+		$location = $this->tempDir();
+		$backend = new LocalBackend($location);
+		$repo = Repository::init($backend, 'pw');
+		$r = $repo->backupDirectory($this->src);
+		$target = $this->tempDir();
+		$repo->restore($r['snapshot'], $target);
+
+		// as a stopped restore leaves it: one file not there yet, one with other content
+		unlink($target . '/docs/d1/note-1.txt');
+		file_put_contents($target . '/docs/d0/note-0.txt', 'something else');
+		$this->assertSame(34, $repo->restore($r['snapshot'], $target, keepUnchanged: true));
+		$this->assertSame(self::hashTree($this->src), self::hashTree($target));
+
+		// all in place: nothing is read from the location, which could not give it anyway now
+		self::removeTree($location . '/packs');
+		$this->assertSame(34, Repository::open($backend, 'pw')->restore($r['snapshot'], $target, keepUnchanged: true));
+		$this->expectException(\Throwable::class);
+		Repository::open($backend, 'pw')->restore($r['snapshot'], $target);
+	}
+
+	private static function removeTree(string $dir): void {
+		foreach (new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator($dir, \FilesystemIterator::SKIP_DOTS), \RecursiveIteratorIterator::CHILD_FIRST) as $f) {
+			$f->isDir() ? rmdir($f->getPathname()) : unlink($f->getPathname());
+		}
+		rmdir($dir);
+	}
+
 	public function testRestoreRemovesTemporariesOfAStoppedRestore(): void {
 		$repo = Repository::init($this->backend, 'pw');
 		$r = $repo->backupDirectory($this->src);

tests/Unit/Repository/RestoreTargetTest.php

@@ -0,0 +1,149 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\NgBackup\Tests\Unit\Repository;
+
+use OCA\NgBackup\Backend\LocalBackend;
+use OCA\NgBackup\Command\Restore;
+use OCA\NgBackup\Crypto\KeyRing;
+use OCA\NgBackup\Repository\Repository;
+use OCA\NgBackup\Repository\RestoreTargetException;
+use OCA\NgBackup\Tests\Unit\TempDirTrait;
+use PHPUnit\Framework\TestCase;
+
+/** A stream that takes at most 3 bytes per write. */
+class PartialWriteStream {
+	public static string $written = '';
+	/** @var resource|null set by PHP */
+	public $context;
+
+	public function stream_open(string $path, string $mode, int $options, ?string &$opened): bool {
+		self::$written = '';
+		return true;
+	}
+
+	public function stream_write(string $data): int {
+		$part = substr($data, 0, 3);
+		self::$written .= $part;
+		return strlen($part);
+	}
+}
+
+/**
+ * What goes wrong where a restore writes to is not damage in the backup: it stops the restore, also
+ * with --salvage, and the message does not point at --salvage.
+ */
+class RestoreTargetTest extends TestCase {
+	use TempDirTrait;
+
+	private LocalBackend $backend;
+	private KeyRing $keys;
+	private string $snapshot;
+
+	protected function setUp(): void {
+		$this->backend = new LocalBackend($this->tempDir());
+		$this->keys = KeyRing::generate();
+		$src = $this->tempDir();
+		mkdir("$src/dir");
+		file_put_contents("$src/dir/a.bin", random_bytes(6 * 1048576));
+		file_put_contents("$src/dir/b.bin", random_bytes(6 * 1048576));
+		file_put_contents("$src/dir/c.txt", 'small');
+		$this->snapshot = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'))->backupDirectory($src)['snapshot'];
+	}
+
+	private function repo(): Repository {
+		return Repository::openWithKey($this->backend, $this->keys);
+	}
+
+	public function testAShortWriteIsATargetError(): void {
+		if (!is_writable('/dev/full')) {
+			$this->markTestSkipped('needs /dev/full');
+		}
+		$out = fopen('/dev/full', 'wb');
+		stream_set_write_buffer($out, 0);
+		$this->expectException(RestoreTargetException::class);
+		Repository::writeAll($out, 'data', 'dir/a.bin');
+	}
+
+	public function testAWriteThatTakesPartOfTheDataGoesOn(): void {
+		stream_wrapper_register('ngbpartial', PartialWriteStream::class);
+		try {
+			$out = fopen('ngbpartial://x', 'wb');
+			Repository::writeAll($out, 'abcdefgh', 'dir/a.bin');
+			fclose($out);
+			$this->assertSame('abcdefgh', PartialWriteStream::$written);
+		} finally {
+			stream_wrapper_unregister('ngbpartial');
+		}
+	}
+
+	public function testATargetErrorDoesNotLookLikeDamage(): void {
+		$this->assertFalse(Restore::backupLooksDamaged(new RestoreTargetException('Cannot write')));
+	}
+
+	/**
+	 * A real full disk: run with NGB_TEST_SMALL_DISK set to an empty, writable tmpfs of a few MiB
+	 * (docker run --tmpfs /small:size=8m,uid=...), with and without --salvage.
+	 *
+	 * @dataProvider salvageOrNot
+	 */
+	public function testAFullDiskStopsTheRestoreAndLeavesNoTemporary(bool $salvage): void {
+		$disk = getenv('NGB_TEST_SMALL_DISK');
+		if ($disk === false || $disk === '') {
+			$this->markTestSkipped('needs NGB_TEST_SMALL_DISK');
+		}
+		$target = $disk . '/' . bin2hex(random_bytes(4));
+		$skipped = $salvage ? [] : null;
+		try {
+			$this->repo()->restore($this->snapshot, $target, skipped: $skipped);
+			$this->fail('12 MiB fitted on the small disk');
+		} catch (RestoreTargetException $e) {
+			$this->assertStringContainsString('disk full', $e->getMessage());
+		} finally {
+			$left = glob("$target/dir/*.ngb-restore-*");
+			foreach (glob("$target/dir/*") ?: [] as $f) {
+				unlink($f);
+			}
+			@rmdir("$target/dir");
+			@rmdir($target);
+		}
+		$this->assertSame([], $left);
+		if ($salvage) {
+			$this->assertSame([], $skipped, 'nothing is listed as damaged');
+		}
+	}
+
+	/** @return array<string, array{bool}> */
+	public static function salvageOrNot(): array {
+		return ['without --salvage' => [false], 'with --salvage' => [true]];
+	}
+
+	public function testResumeChecksTheFolderBeforeSkipping(): void {
+		$target = $this->tempDir();
+		$this->repo()->restore($this->snapshot, $target);
+		$outside = $this->tempDir();
+		rename("$target/dir", "$outside/dir");
+		symlink("$outside/dir", "$target/dir");
+		$this->expectException(RestoreTargetException::class);
+		$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
+	}
+
+	public function testResumeRemovesTemporariesWhereEverythingIsInPlace(): void {
+		$target = $this->tempDir();
+		$this->repo()->restore($this->snapshot, $target);
+		file_put_contents("$target/dir/a.bin.ngb-restore-0123456789ab", 'left by a stopped restore');
+		$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
+		$this->assertFileDoesNotExist("$target/dir/a.bin.ngb-restore-0123456789ab");
+	}
+
+	public function testHeartbeatRunsForFilesThatAreSkipped(): void {
+		$target = $this->tempDir();
+		$this->repo()->restore($this->snapshot, $target);
+		$beats = 0;
+		$this->repo()->restore($this->snapshot, $target, '', false, function () use (&$beats): void {
+			$beats++;
+		}, keepUnchanged: true);
+		$this->assertSame(3, $beats);
+	}
+}

tests/integration/full-restore-roundtrip.sh

@@ -2,8 +2,11 @@
 # Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a
 # SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup
 # location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7.
-# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp|salvage|to-pgsql|to-mysql|to-sqlite]
+# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp|salvage|salvage-files|to-pgsql|to-mysql|to-sqlite]
 #   e.g. full-restore-roundtrip.sh 35-apache pgsql
+# With "salvage-files" one pack is damaged and another is gone from the location. The restore must
+# stop and point at --salvage; run again as the guide says, it restores the database and every file
+# that can be read, and lists the rest (exit 2). NGB_FRT_DATA_MB sets the amount of data (default 200).
 # With "sftp" the location is an SFTP server instead of a shared directory, and the origin has a
 # file spread over several packs: the restore reads them ahead on connections of its own.
 # With "to-<db>" the restore instance runs that database whatever the origin runs (a MySQL or
@@ -36,7 +39,7 @@ cleanup() {
 	docker network rm "$NET" >/dev/null 2>&1 || true
 	docker run --rm -v "$SHARED:/target" alpine sh -c 'rm -rf /target/*' >/dev/null 2>&1 || true
 	rmdir "$SHARED" 2>/dev/null || true
-	rm -f /tmp/ngb-frt-kit.json /tmp/ngb-frt-pass
+	rm -f /tmp/ngb-frt-kit.json /tmp/ngb-frt-pass /tmp/ngb-frt-sums
 	[ -n "${LEGACY_DIR:-}" ] && rm -rf "$LEGACY_DIR"
 	return 0
 }
@@ -99,12 +102,23 @@ ok "key, kit and location on the origin"
 
 NC_PASS=$(openssl rand -hex 16)
 docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
-docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
+docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs /var/www/html/data/frtest/files/Empty/Nested && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
 if [ "$MODE" = sftp ]; then
 	# about three packs of 32 MiB
 	docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
 	LARGE_SUM=$(docker exec ngb-frt-origin sha256sum /var/www/html/data/frtest/files/Docs/large.bin | cut -d' ' -f1)
 fi
+if [ "$MODE" = salvage-files ]; then
+	# Half in large files that fill packs of their own, half in files of 1 MiB; incompressible
+	DATA_MB=${NGB_FRT_DATA_MB:-200}
+	docker exec -e DATA_MB="$DATA_MB" ngb-frt-origin sh -c '
+		cd /var/www/html/data/frtest/files && mkdir -p Large Small/a Small/b
+		i=0; while [ $((i * 50)) -lt $((DATA_MB / 2)) ]; do head -c 52428800 /dev/urandom > Large/f$i.bin; i=$((i + 1)); done
+		i=0; while [ $i -lt $((DATA_MB / 2)) ]; do head -c 1048576 /dev/urandom > Small/$([ $((i % 2)) = 0 ] && echo a || echo b)/f$i.bin; i=$((i + 1)); done
+		chown -R www-data:www-data . && find . -type f -print0 | xargs -0 sha256sum > /tmp/frt-sums'
+	docker cp ngb-frt-origin:/tmp/frt-sums /tmp/ngb-frt-sums
+	ok "origin has $DATA_MB MiB of test data, $(wc -l < /tmp/ngb-frt-sums) files"
+fi
 run_on ngb-frt-origin files:scan frtest >/dev/null
 
 # SQL on an instance's own database (pgsql/mysql only)
@@ -143,6 +157,18 @@ if [ "$MODE" = salvage ]; then
 		|| fail "damage the database dump"
 	ok "database dump of the snapshot damaged on the location"
 fi
+if [ "$MODE" = salvage-files ]; then
+	# The two largest packs hold user files for sure: bytes flipped in one, the other gone
+	docker exec -u www-data ngb-frt-origin php -r '
+		$packs = glob("/mnt/shared/repo/packs/*/*");
+		usort($packs, fn ($a, $b) => filesize($b) <=> filesize($a));
+		count($packs) >= 3 or exit(1);
+		$h = fopen($packs[0], "r+b");
+		for ($o = 4096; $o < filesize($packs[0]); $o += 1048576) { fseek($h, $o); $b = fread($h, 16); fseek($h, $o); fwrite($h, ~$b); }
+		fclose($h);
+		unlink($packs[1]);' || fail "damage the packs"
+	ok "one pack damaged and one pack gone on the location"
+fi
 
 docker cp ngb-frt-origin:/tmp/kit.json /tmp/ngb-frt-kit.json
 docker exec ngb-frt-origin sh -c "cat /tmp/pass" > /tmp/ngb-frt-pass
@@ -188,6 +214,7 @@ if [ "$MODE" = salvage ]; then
 	printf '%s\n' "$OUT" | grep "$SALVAGE" >/dev/null || { printf '%s\n' "$OUT" | tail -30; fail "the output does not name the salvage folder"; }
 	[ "$(docker exec ngb-frt-restore cat "$SALVAGE/data/frtest/files/Docs/note.txt")" = "hello disaster recovery" ] || fail "the user's file is not in the salvage folder"
 	docker exec ngb-frt-restore test -f "$SALVAGE/config/config.php" || fail "config.php is not in the salvage folder"
+	docker exec ngb-frt-restore test -d "$SALVAGE/data/frtest/files/Empty/Nested" || fail "the empty folder is not in the salvage folder"
 	run_on ngb-frt-restore user:list | grep frtest >/dev/null && fail "the fresh database was changed (the backed-up user is there)"
 	run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
 	run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
@@ -196,6 +223,60 @@ if [ "$MODE" = salvage ]; then
 	exit 0
 fi
 
+if [ "$MODE" = salvage-files ]; then
+	set +e
+	OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction 2>&1); CODE=$?
+	set -e
+	[ "$CODE" = 1 ] || { printf '%s\n' "$OUT" | tail -20; fail "a restore with damaged data did not stop (exit $CODE)"; }
+	printf '%s\n' "$OUT" | grep -- "--salvage" >/dev/null || { printf '%s\n' "$OUT" | tail -20; fail "the stopped restore does not point at --salvage"; }
+	run_on ngb-frt-restore status 2>&1 | grep "maintenance: true" >/dev/null || fail "maintenance mode is off after a restore that stopped halfway"
+	ok "damaged data: the restore stops, keeps maintenance mode on and points at --salvage"
+
+	# As the guide and the message say: maintenance mode off and again, now with --salvage
+	run_on ngb-frt-restore maintenance:mode --off >/dev/null || fail "maintenance mode off"
+	set +e
+	OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction --salvage --force 2>&1); CODE=$?
+	set -e
+	[ "$CODE" = 2 ] || { printf '%s\n' "$OUT" | tail -30; fail "restore:full --salvage exited with $CODE, not 2"; }
+	printf '%s\n' "$OUT" | grep "could not be restored" >/dev/null || { printf '%s\n' "$OUT" | tail -30; fail "the damaged files are not listed"; }
+	printf '%s\n' "$OUT" | grep "ngb-salvage-" >/dev/null && { printf '%s\n' "$OUT" | tail -30; fail "the database was given up, only files are damaged"; }
+	docker cp /tmp/ngb-frt-sums ngb-frt-restore:/tmp/frt-sums
+	printf '%s\n' "$OUT" | docker exec -i ngb-frt-restore sh -c 'cat > /tmp/frt-out'
+	# Every file: listed and not there, or there and bit-identical
+	CHECK=$(docker exec ngb-frt-restore php -r '
+		$root = "/var/www/html/data/frtest/files/"; $listed = []; $reasons = [];
+		foreach (file("/tmp/frt-out", FILE_IGNORE_NEW_LINES) as $l) {
+			if (preg_match("#^  (?:data/)?frtest/files/(.+?): (.*)$#", $l, $m)) { $listed[$m[1]] = true; $reasons[] = $m[2]; }
+		}
+		$same = 0; $gone = 0; $bad = [];
+		foreach (file("/tmp/frt-sums", FILE_IGNORE_NEW_LINES) as $l) {
+			[$sum, $p] = preg_split("/\s+/", $l, 2); $p = substr($p, 2);
+			if (isset($listed[$p])) { file_exists($root . $p) ? $bad[] = "listed but there: $p" : $gone++; continue; }
+			if (!file_exists($root . $p)) { $bad[] = "not listed and not there: $p"; continue; }
+			hash_file("sha256", $root . $p) === $sum ? $same++ : $bad[] = "different: $p";
+		}
+		$walk = new RecursiveIteratorIterator(new RecursiveDirectoryIterator("/var/www/html/data", FilesystemIterator::SKIP_DOTS), RecursiveIteratorIterator::SELF_FIRST);
+		foreach ($walk as $f) { if (str_contains($f->getFilename(), ".ngb-restore-")) { $bad[] = "temporary left: " . $f->getPathname(); } }
+		echo "same=$same gone=$gone bad=", count($bad), "\n", implode("\n", array_slice($bad, 0, 10)), "\n";
+		echo "reasons: ", implode(" | ", array_unique(array_map(fn ($r) => substr(preg_replace("#[0-9a-f]{16,}#", "<id>", $r), 0, 70), $reasons))), "\n";')
+	printf '%s\n' "$CHECK"
+	printf '%s\n' "$CHECK" | grep -q " bad=0" || fail "files after the salvage are not as listed"
+	printf '%s\n' "$CHECK" | grep -qE "same=[1-9]" || fail "nothing was restored"
+	printf '%s\n' "$CHECK" | grep -qE "gone=[1-9]" || fail "no user file was hit by the damage"
+	ok "with --salvage: every file listed and absent, or there and identical"
+	run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
+	run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing: the database was not restored"
+	run_on ngb-frt-restore user:list | grep admin >/dev/null || fail "the admin is missing: the database was not restored"
+	docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back with --salvage"
+	[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt 2>/dev/null)" = "hello disaster recovery" ] \
+		|| printf '%s\n' "$OUT" | grep "Docs/note.txt" >/dev/null || fail "note.txt is neither restored nor listed"
+	run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
+	run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after the salvage"
+	ok "the database and config are the old server's, the instance works"
+	echo "FULL RESTORE ROUNDTRIP PASS ($DB, salvage-files)"
+	exit 0
+fi
+
 RESTORE_OPTS=(--no-interaction --install-apps)
 if [ "$LEGACY" = legacy ] && [ "$DB" != sqlite ]; then
 	# Without --skip-missing-tables an older backup must be refused, naming the app's tables
@@ -245,7 +326,16 @@ run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails aft
 run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database"
 docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
 [ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
-run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
+if [ "$LEGACY" = legacy ]; then
+	# Backups made before 1.0.3 do not record empty folders
+	docker exec ngb-frt-restore test -e /var/www/html/data/frtest/files/Empty/Nested && fail "an empty folder came back from a backup that does not record them"
+	run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
+else
+	docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back"
+	run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
+	docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "files:scan removed the empty folder"
+	ok "the empty folder came back and stays after files:scan"
+fi
 # New rows after the restore: a new file (file cache) and a new user must not collide with
 # restored rows (ID counters after the data).
 docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"

tests/integration/roundtrip.sh

@@ -26,7 +26,7 @@ ok "location added"
 # Test data: a user with files.
 $X sh -c "OC_PASS='Roundtrip-pw-2026' php occ user:add --password-from-env $U >/dev/null"
 # large.bin (40 MB) fills more than one pack: a restore to a directory reads ahead over several ranges
-$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin"
+$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs/Empty/Nested $DATA/$U/files/Hollow && touch -d @1700000000 $DATA/$U/files/Hollow && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin"
 occ files:scan "$U" >/dev/null
 SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
 
@@ -39,11 +39,18 @@ SNAP=$(occ backup:list ci | awk -F'|' '!f && / second /{gsub(/ /,"",$3); print $
 [ -n "$SNAP" ] || fail "snapshot not listed"
 
 # Damage, then restore in each mode.
-$X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt"
+$X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt && rm -r Docs/Empty && rmdir Hollow"
 occ files:scan "$U" >/dev/null
 occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m new-folder >/dev/null || fail "restore new-folder"
 $X sh -c "ls $DATA/$U/files | grep -q '^Restored '" || fail "no Restored folder"
+$X sh -c "test -d $DATA/$U/files/Restored\ */Docs/Empty/Nested" || fail "the empty folder is not in the Restored folder"
+occ backup:browse ci "$SNAP" "data/$U/files" | grep Hollow >/dev/null || fail "an empty folder is not shown when browsing the snapshot"
+occ backup:restore ci "$SNAP" "data/$U/files/Hollow" -m merge >/dev/null || fail "restore of a folder that was empty"
+$X sh -c "test -d $DATA/$U/files/Hollow" || fail "the empty folder did not come back by itself"
+[ "$($X stat -c %Y "$DATA/$U/files/Hollow")" = 1700000000 ] || fail "the empty folder did not get its time back"
 occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m replace >/dev/null || fail "restore replace"
+$X sh -c "test -d $DATA/$U/files/Docs/Empty/Nested" || fail "replace did not bring back the empty folder, or moved it to the trash"
+ok "empty folders: restored in each mode, by themselves and shown when browsing"
 $X sh -c "rm -rf '$DATA/$U/files/Restored '*" ; occ files:scan "$U" >/dev/null
 SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
 [ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore"
@@ -63,6 +70,7 @@ occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-dir >/dev/null || f
 SUM_DIR=$($X sh -c "cd /tmp/ngb-rt-dir/data/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
 [ "$SUM_BEFORE" = "$SUM_DIR" ] || fail "the files restored to a directory differ"
 $X sh -c "ls -A /tmp/ngb-rt-dir | grep -q ngb-restore-spool" && fail "the read-ahead spool was left in the directory"
+$X sh -c "test -d /tmp/ngb-rt-dir/data/$U/files/Docs/Empty/Nested" || fail "the empty folder is not in the directory"
 $X sh -c "rm -rf /tmp/ngb-rt-dir"
 ok "restore of all the user's files to a directory: identical, no spool left"
 # not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
@@ -90,6 +98,9 @@ done
 [ "$RS" = done ] || fail "the backup started through the API ended as '$RS' after $i ticks"
 occ backup:list ci | grep " page " >/dev/null || fail "the snapshot of the ticked backup is not listed"
 ok "without system cron: warning in backup:status and the API, backup finished by $i ticks of the open page"
+# The Bin reads in parts, as the page does (what is in the parts: CatalogTest)
+api "$API/targets/$TID/trash" | php -r '$p = json_decode(stream_get_contents(STDIN), true); exit(is_int($p["total"] ?? null) && is_array($p["items"] ?? null) && count($p["items"]) <= $p["total"] ? 0 : 1);' || fail "the Bin API does not give a total and items"
+ok "the Bin API reads in parts, with a total"
 occ background:cron >/dev/null
 $X php -f /var/www/html/cron.php >/dev/null 2>&1 || true
 occ backup:status | grep "system cron" >/dev/null && fail "backup:status still warns after cron ran"