@@ -11,6 +11,36 @@ Each heading starts with that `<version>`, once: the app store shows a release t
|
||||
whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
|
||||
shown under 1.0.0. The git tag of a release candidate goes after the date.
|
||||
|
||||
## [1.0.3] - 2026-11-06
|
||||
|
||||
### Fixed
|
||||
- Empty folders are backed up and restored (#27), in `occ backup:restore`, `occ backup:restore:full`
|
||||
and the restore page, where they are shown too. They come back from the first backup made with
|
||||
this version; older restore points do not contain them. An older version of NG Backup can still
|
||||
read new restore points, and then only leaves the empty folders out.
|
||||
- A full disk during a restore no longer looks like a damaged backup. The restore now stops with a
|
||||
message about the disk, also with `--salvage`, and does not point at `--salvage` any more. The same
|
||||
goes for other trouble writing to the restore directory, such as permissions.
|
||||
- A file or folder whose name a snapshot cannot hold (a backslash, a control character, not UTF-8) is
|
||||
left out and reported as unreadable. Before, a backslash or control character could make the whole
|
||||
restore point unreadable, and a name that is not UTF-8 made the backup fail.
|
||||
- A restore that goes on after a stop also removes the temporary files in folders where every file
|
||||
is already in place, and checks that those folders are inside the restore directory before it
|
||||
leaves their files alone.
|
||||
- After a restore that was killed and then ran again, a backup no longer has to wait up to two hours
|
||||
for the locks of the killed run.
|
||||
- A restore only removes read-ahead folders of its own (`.ngb-restore-spool-` with 8 hex digits).
|
||||
- A restore keeps its lock on the location during a long row of files that are already in place.
|
||||
- A database restore that fails no longer leaves behind tables it created, which could stop the
|
||||
installation of that app later. When only NG Backup's own settings cannot be put back after the
|
||||
database was restored, `--salvage` no longer says the database is still the fresh one.
|
||||
- A connection that breaks while opening a pack on the location (seen with SMB) is retried once and
|
||||
is handled by `--salvage`, instead of stopping the restore with the error of the storage.
|
||||
- Without a system cron, several running backups take turns while the page is open.
|
||||
- The Bin tab shows how far it is while it reads the removed restore points, which can take a while.
|
||||
It also shows when it cannot read a location, instead of an empty bin, and refreshes when a run
|
||||
ends.
|
||||
|
||||
## [1.0.2] - 2026-11-06
|
||||
|
||||
### Added
|
||||
|
||||
@@ -437,7 +437,44 @@ in README "Known limitations":
|
||||
met vertraging is nog niet gemeten); Bin-tabblad; schatting `target:add`; nieuwe screenshots
|
||||
(1–3 vervangen, 4-bin erbij; in `info.xml` bij de release naar tag v1.0.2 laten wijzen en
|
||||
4-bin toevoegen). Getest: unit (208), psalm, integratie NC35 sqlite/pgsql, S3/SFTP/FTP/SMB,
|
||||
volledige restore met `--salvage`. Nog te doen: DR-test, release.
|
||||
volledige restore met `--salvage`. **1.0.2 uitgebracht 2026-11-06** (tag `v1.0.2`, PR #28).
|
||||
- **Naar 1.0.3, uit de reviews van 1.0.2 (achteraf gedraaid, 2026-11-06):**
|
||||
1. Lege mappen (#27).
|
||||
2. Een volle schijf onder `--salvage` telt als beschadigde backup en de restore gaat door; zonder
|
||||
`--salvage` wijst de melding er ten onrechte naar. `fwrite` controleren, schrijffouten stoppen de
|
||||
restore (`Repository::restoreEntries`; ook `RestoreService::writeFile` nakijken). Beide reviews.
|
||||
3. Hervatten: controle op de doelmap vóór het overslaan, en tijdelijke bestanden ook opruimen in
|
||||
een map waar alles al klopt.
|
||||
4. Alleen `.ngb-restore-spool-<hex>` opruimen.
|
||||
5. Salvage-test: tijdelijke bestanden recursief zoeken; `admin` en `frtest` apart controleren.
|
||||
6. Nog nakijken: lease verversen bij overgeslagen bestanden; fout ná de database-commit
|
||||
(`restoreOwnAppConfig`) en tabellen die na een rollback blijven staan; backups zonder cron
|
||||
via de pagina (eerlijke beurten, stappen die de webtimeout halen); Bin verversen en fouten
|
||||
tonen; SMB-uitzondering bij het openen van een pack onder `--salvage`.
|
||||
Niet doen: bij hervatten de inhoud vergelijken (grootte en mtime is ook het criterium van de backup).
|
||||
**Stand op branch `1.0.3` (2026-11-06):** punten 1-6 gebouwd. Lege mappen staan in het metarecord
|
||||
van de snapshot, niet tussen de bestanden: 1.0.2 en ouder lezen zo'n snapshot gewoon en slaan alleen
|
||||
de lege mappen over. Geen webtimeout-wijziging zonder cron: alleen in de code nagelopen, niet getest
|
||||
(Nextcloud zet de PHP-tijdslimiet ruim, een gestopte stap wordt overgedaan); wel eerlijke beurten.
|
||||
Bin leest in delen met teller en balk (besluit 2026-11-06, ook met oog op meerdere backups per
|
||||
dag). Getest: unit (volle schijf op echte tmpfs), integratie NC35 sqlite/pgsql/mysql, sftp, salvage,
|
||||
salvage-files; Bin in een echte browser met vertraagde verzoeken (1 van 7 t/m 7). Nog: reviews
|
||||
vóór de PR, README-regels bij de versiecommit.
|
||||
**Simulaties (2026-11-06, NC 34.0.4, 2 GB, lokale SFTP, `~/dev-local/ngb-sim/sim.sh`):** normaal
|
||||
(pgsql), stop en hervat (pgsql → MariaDB, restore met kill -9 gestopt), volle schijf met `--salvage`:
|
||||
alle drie geslaagd, alle 3.101 gebruikersbestanden en 7 lege mappen gelijk, share, agenda en WebDAV
|
||||
werken. Vond één fout, opgelost: na een kill bleven vergrendelingen tot 2 uur staan, waardoor de
|
||||
backup van stap 11 werd geweigerd. `files:scan` "Updated" zijn in de gemeten run alleen mappen
|
||||
(mtime van mappen wordt niet teruggezet, dus nieuwe mtime en etag; bestanden ongemoeid). "Removed":
|
||||
o.a. `cache` van een gebruiker, die de backup uitsluit. Waarschijnlijk verklaart dit ook de 50.080
|
||||
"updated" op nc01, niet op nc01 zelf vastgesteld.
|
||||
**Naar 1.1.0 of 1.2.0 (besluit 2026-11-06):** alle mappen krijgen bij een restore hun oorspronkelijke
|
||||
tijd terug. Hoort zo, vooral bij oudere verzamelingen: de maptijd wordt gebruikt om in te schatten
|
||||
wanneer een map is aangemaakt. De backup moet dan de tijd van alle mappen vastleggen en de restore
|
||||
die aan het eind terugzetten; werkt pas vanaf de eerste backup met die versie. Op de roadmap onder 1.1.0.
|
||||
- **Branches (besluit 2026-11-06):** werk voor een versie op een branch met die naam (`1.0.3`);
|
||||
`polish` alleen voor kleine cosmetische zaken, per geval bekijken of en wanneer die gemerged
|
||||
worden. SMB staat uit op sub5 en sub6; sub6 is het testaccount, eerst aanzetten voor een SMB-test.
|
||||
- **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
|
||||
1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
|
||||
niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij
|
||||
|
||||
@@ -7,7 +7,7 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
|
||||
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
||||
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
||||
|
||||
**Status: 1.0.2, stable.** Disaster recovery has been tested on real installations (see "Tested
|
||||
**Status: 1.0.3, stable.** Disaster recovery has been tested on real installations (see "Tested
|
||||
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
|
||||
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
|
||||
`docs/DISASTER-RECOVERY.md`.
|
||||
@@ -169,6 +169,7 @@ Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECO
|
||||
|---|---|---|---|
|
||||
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
|
||||
| The same, 1.0.2 | Hetzner Storage Box (SMB), Germany → Canada | PostgreSQL → PostgreSQL | 150,755 files restored in about 29 h (7-13 MiB/s), going on after a lost connection; sample identical, empty folders not restored ([#27](https://github.com/dillardblom/ng_backup/issues/27)) |
|
||||
| 1.0.3, small simulations of 2 GB: as planned, with the restore killed halfway, and with a disk that runs full | Local SFTP server | PostgreSQL → PostgreSQL and MariaDB | all 3,101 user files and all empty folders identical; users, a share and a calendar back |
|
||||
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
|
||||
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
|
||||
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
|
||||
@@ -265,15 +266,12 @@ What is planned, without dates. Ideas and votes are welcome in the
|
||||
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
|
||||
a 👍 on its issue helps decide what comes first.
|
||||
|
||||
**1.0.3**
|
||||
- Empty folders in the backup and the restore
|
||||
([#27](https://github.com/dillardblom/ng_backup/issues/27)).
|
||||
|
||||
**1.1.0**
|
||||
- Restore without the recovery kit: the backup key from the location itself, opened with one of
|
||||
your passphrases.
|
||||
- A schedule per location, also "every N hours".
|
||||
- Locations backed up one after the other instead of at the same time.
|
||||
- All folders get their original modification time back on a restore, not only the empty ones.
|
||||
- A note on a restore point that you can change afterwards.
|
||||
- Pinning a restore point, so the retention policy never removes it (for example the one from
|
||||
before an upgrade).
|
||||
@@ -293,12 +291,10 @@ a 👍 on its issue helps decide what comes first.
|
||||
|
||||
On the backlog; know them before you rely on NG Backup:
|
||||
|
||||
- **Empty folders do not come back.** A backup only records files, so a folder without files is
|
||||
not restored. To be fixed in 1.0.3
|
||||
([#27](https://github.com/dillardblom/ng_backup/issues/27)).
|
||||
- **A full disk during a restore looks like a damaged backup.** The reason shown is "Size mismatch",
|
||||
and with `--salvage` the restore goes on and lists the remaining files. The backup itself is fine:
|
||||
make room and run the restore again. To be fixed in 1.0.3.
|
||||
- **Empty folders only come back from backups made with 1.0.3 or newer.** Older restore points do
|
||||
not contain them.
|
||||
- **Folders with files in them show the time of the restore**, not their original modification time.
|
||||
The files keep theirs. On the roadmap.
|
||||
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
|
||||
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
|
||||
External storage app are not in the data directory and are **not included**, without a warning.
|
||||
|
||||
+5
-5
@@ -45,7 +45,7 @@ welcome: use the "Share your experience" link on the admin page.
|
||||
|
||||
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
|
||||
you need help with restoring a backup after a crash.]]></description>
|
||||
<version>1.0.2</version>
|
||||
<version>1.0.3</version>
|
||||
<licence>agpl</licence>
|
||||
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
|
||||
<namespace>NgBackup</namespace>
|
||||
@@ -56,10 +56,10 @@ you need help with restoring a backup after a crash.]]></description>
|
||||
<website>https://github.com/dillardblom/ng_backup</website>
|
||||
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
|
||||
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/1-backups.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/2-restore.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/3-settings.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/4-bin.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/1-backups.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/2-restore.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/3-settings.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/4-bin.png</screenshot>
|
||||
<dependencies>
|
||||
<php min-version="8.1"/>
|
||||
<lib>sodium</lib>
|
||||
|
||||
@@ -123,10 +123,12 @@ without `--install-apps`:
|
||||
occ maintenance:mode --off && occ backup:restore:full offsite <snapshot> --salvage --force
|
||||
|
||||
- Files that cannot be read are left out and listed at the end; everything else is restored.
|
||||
- **Check the disk first if the reason is "Size mismatch".** In this version a full disk looks like
|
||||
damaged data: the message then points at `--salvage`, and with `--salvage` the restore goes on and
|
||||
lists the remaining files. The backup itself is fine: make room and run the restore again as in
|
||||
step 8, without `--salvage` unless the backup is damaged as well. To be fixed in 1.0.3.
|
||||
- A full disk is not damage: the restore stops, also with `--salvage`, and says so. Make room and
|
||||
run the restore again as in step 8.
|
||||
- **On version 1.0.2, check the disk first if the reason is "Size mismatch".** There a full disk
|
||||
looks like damaged data: the message points at `--salvage`, and with `--salvage` the restore goes
|
||||
on and lists the remaining files. The backup itself is fine. Update NG Backup, make room and run
|
||||
the restore again as in step 8.
|
||||
- If the database cannot be restored (its dump is damaged, or it does not fit this
|
||||
installation), the database stays as the fresh installation made it, and the data directory
|
||||
and config of the backup go to a folder of their own in the data directory,
|
||||
@@ -149,9 +151,12 @@ The command then ends with exit code 2 instead of 0. The same option exists for
|
||||
- From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`,
|
||||
`redis`), which point at AIO's containers.
|
||||
- Log in, open a few files, a calendar and a contact.
|
||||
- Empty folders do not come back: a backup only records files (to be fixed in 1.0.3, see
|
||||
[issue #27](https://github.com/dillardblom/ng_backup/issues/27)). `occ files:scan` then removes
|
||||
them from the file cache, so it reports them as removed.
|
||||
- `occ files:scan` can report many folders as "Updated". That is expected: a restore does not yet put
|
||||
back the modification time of folders with files in them, so they show the time of the restore.
|
||||
The files themselves keep their time and content.
|
||||
- Empty folders only come back from backups made with version 1.0.3 or newer. With an older
|
||||
restore point they are gone: `occ files:scan` then removes them from the file cache, so it reports
|
||||
them as removed.
|
||||
- Start background jobs again (cron), as in step 2 in reverse.
|
||||
|
||||
**10. Point your domain at the new server** and check that clients reconnect.
|
||||
|
||||
@@ -124,7 +124,13 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
||||
// null: not supported or failing for now; false: not opened, which need not mean
|
||||
// missing. Either way the storage's own stream decides.
|
||||
}
|
||||
$fh = $this->storage->fopen($this->abs($path), 'r');
|
||||
try {
|
||||
$fh = $this->storage->fopen($this->abs($path), 'r');
|
||||
} catch (\Exception $e) {
|
||||
// Some storages throw for what is not a missing file (SMB: a connection that broke).
|
||||
// As a BackendException, the caller reads once more and --salvage can go on.
|
||||
throw new BackendException('Cannot read ' . $path . ': ' . $e->getMessage(), 0, $e);
|
||||
}
|
||||
if ($fh === false) {
|
||||
throw new BackendException('Not found: ' . $path);
|
||||
}
|
||||
|
||||
@@ -82,6 +82,8 @@ final class AdminApiController extends Controller {
|
||||
|
||||
/** Seconds of backup work one tick() does: well within a normal web request. */
|
||||
public const TICK_SECONDS = 10.0;
|
||||
/** Seconds one request for the Bin reads, before the page shows how far it is and asks for more. */
|
||||
private const TRASH_PAGE_SECONDS = 5.0;
|
||||
|
||||
/**
|
||||
* Move running backups forward from the open admin page, when background jobs do not run on a
|
||||
@@ -95,7 +97,10 @@ final class AdminApiController extends Controller {
|
||||
}
|
||||
$deadline = microtime(true) + self::TICK_SECONDS;
|
||||
$stepped = 0;
|
||||
foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) {
|
||||
$running = $this->runs->findRunning(null, BackupService::KIND_FULL);
|
||||
// Longest wait first: a run that takes all of a tick does not keep another from its turn.
|
||||
usort($running, static fn ($a, $b) => $a->getUpdatedAt() <=> $b->getUpdatedAt());
|
||||
foreach ($running as $run) {
|
||||
if (microtime(true) >= $deadline) {
|
||||
break;
|
||||
}
|
||||
@@ -174,15 +179,18 @@ final class AdminApiController extends Controller {
|
||||
}
|
||||
}
|
||||
|
||||
public function trash(int $targetId): JSONResponse {
|
||||
public function trash(int $targetId, string $after = ''): JSONResponse {
|
||||
try {
|
||||
$delay = $this->keys->deleteDelayDays() * 86400;
|
||||
$repo = $this->targets->repository($this->targets->get((string)$targetId));
|
||||
// In parts, so that the page can show how far it is (a read per removed restore point).
|
||||
$page = $repo->trashPage($after, microtime(true) + self::TRASH_PAGE_SECONDS);
|
||||
$out = [];
|
||||
foreach ($this->targets->repository($this->targets->get((string)$targetId))->trash() as $id => $info) {
|
||||
foreach ($page['items'] as $id => $info) {
|
||||
$out[] = ['id' => $id, 'label' => $info['label'], 'time' => $info['time'], 'removedAt' => $info['forgottenAt'],
|
||||
'by' => $info['by'], 'deleteAfter' => $info['forgottenAt'] + $delay];
|
||||
}
|
||||
return new JSONResponse($out);
|
||||
return new JSONResponse(['total' => $page['total'], 'items' => $out]);
|
||||
} catch (\Throwable $e) {
|
||||
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||
}
|
||||
|
||||
+50
-22
@@ -157,17 +157,27 @@ final class DbDumper {
|
||||
}
|
||||
// Tables the backup knows the structure of are created first (DDL is not transactional
|
||||
// everywhere); tables it does not know are left out, as asked.
|
||||
foreach ($missing as $table) {
|
||||
if (isset($manifest['tables'][$table]['schema'])) {
|
||||
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
|
||||
$report['created'][] = $table;
|
||||
} else {
|
||||
$report['missing'][] = $table;
|
||||
try {
|
||||
foreach ($missing as $table) {
|
||||
if (isset($manifest['tables'][$table]['schema'])) {
|
||||
$report['created'][] = $table; // first: creating it can stop halfway (its indexes)
|
||||
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
|
||||
} else {
|
||||
$report['missing'][] = $table;
|
||||
}
|
||||
}
|
||||
} catch (\Throwable $e) {
|
||||
$this->dropCreatedTables($report);
|
||||
throw $e;
|
||||
}
|
||||
$total = 0;
|
||||
$foreignKeys = [];
|
||||
$this->db->beginTransaction();
|
||||
try {
|
||||
$this->db->beginTransaction();
|
||||
} catch (\Throwable $e) {
|
||||
$this->dropCreatedTables($report);
|
||||
throw $e;
|
||||
}
|
||||
try {
|
||||
$foreignKeys = $this->suspendForeignKeys(); // first: on SQLite it also takes the write lock
|
||||
foreach ($manifest['tables'] as $table => $info) {
|
||||
@@ -220,21 +230,6 @@ final class DbDumper {
|
||||
$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
|
||||
$this->resumeForeignKeys($foreignKeys);
|
||||
$this->db->commit();
|
||||
// Foreign keys of the tables created above, now their rows are in. Not inside the
|
||||
// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is
|
||||
// restored either way.
|
||||
foreach ($report['created'] as $table) {
|
||||
$structure = $manifest['tables'][$table]['schema'] ?? [];
|
||||
foreach ($structure['foreignKeys'] ?? [] as $fk) {
|
||||
try {
|
||||
$this->db->createSchemaManager()->createForeignKey(
|
||||
new \Doctrine\DBAL\Schema\ForeignKeyConstraint($fk['columns'], $fk['table'], $fk['foreign'], $fk['name'], $fk['options'] ?? []),
|
||||
$table);
|
||||
} catch (\Throwable $e) {
|
||||
$report['foreignKeyErrors'][] = $table . '.' . $fk['name'] . ': ' . $e->getMessage();
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (\Throwable $e) {
|
||||
$this->db->rollBack();
|
||||
// PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session
|
||||
@@ -242,11 +237,44 @@ final class DbDumper {
|
||||
if (in_array('SET FOREIGN_KEY_CHECKS = 1', $foreignKeys, true)) {
|
||||
$this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 1');
|
||||
}
|
||||
$this->dropCreatedTables($report);
|
||||
throw $e;
|
||||
}
|
||||
// Committed. Foreign keys of the tables created above, now their rows are in. Not inside the
|
||||
// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is restored
|
||||
// either way, and the created tables stay.
|
||||
foreach ($report['created'] as $table) {
|
||||
$structure = $manifest['tables'][$table]['schema'] ?? [];
|
||||
foreach ($structure['foreignKeys'] ?? [] as $fk) {
|
||||
try {
|
||||
$this->db->createSchemaManager()->createForeignKey(
|
||||
new \Doctrine\DBAL\Schema\ForeignKeyConstraint($fk['columns'], $fk['table'], $fk['foreign'], $fk['name'], $fk['options'] ?? []),
|
||||
$table);
|
||||
} catch (\Throwable $e) {
|
||||
$report['foreignKeyErrors'][] = $table . '.' . $fk['name'] . ': ' . $e->getMessage();
|
||||
}
|
||||
}
|
||||
}
|
||||
return $total;
|
||||
}
|
||||
|
||||
/**
|
||||
* The restore did not go through: a rollback does not undo CREATE TABLE everywhere. Left behind,
|
||||
* the empty tables of an app would stop that app's installation later ("table already exists").
|
||||
*
|
||||
* @param array{created: list<string>, ...} $report
|
||||
*/
|
||||
private function dropCreatedTables(array &$report): void {
|
||||
foreach ($report['created'] as $table) {
|
||||
try {
|
||||
$this->db->createSchemaManager()->dropTable($table);
|
||||
} catch (\Throwable) {
|
||||
// the error that brought us here is the one to report; a run again uses the table
|
||||
}
|
||||
}
|
||||
$report['created'] = [];
|
||||
}
|
||||
|
||||
/**
|
||||
* The table prefix of the instance the dump was made on (from its appconfig table), or null
|
||||
* when the dump has none.
|
||||
|
||||
+37
-1
@@ -55,9 +55,32 @@ final class BackupRun {
|
||||
});
|
||||
$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
|
||||
$lines = [];
|
||||
$emptyDirs = [];
|
||||
$finishedWalk = true;
|
||||
|
||||
foreach ($walker->walk($state['after']) as $logical => $path) {
|
||||
if (!mb_check_encoding($logical, 'UTF-8') || !Repository::isSafePath($logical)) {
|
||||
// A name a snapshot cannot hold (a backslash, a control character, not UTF-8). One
|
||||
// such path must not make a backup fail, or the whole snapshot unreadable later.
|
||||
// The cursor stays where it is: the state is JSON, which cannot hold such a name. A
|
||||
// next step meets it again, and it is reported once.
|
||||
self::recordUnreadable($state, $stats, mb_scrub($logical, 'UTF-8'));
|
||||
continue;
|
||||
}
|
||||
if (is_dir($path)) {
|
||||
$mtime = @filemtime($path);
|
||||
if ($mtime === false) {
|
||||
continue; // removed while walking
|
||||
}
|
||||
$emptyDirs[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR);
|
||||
$stats['emptyDirs'] = ($stats['emptyDirs'] ?? 0) + 1;
|
||||
$state['after'] = $logical;
|
||||
if (microtime(true) >= $deadline) {
|
||||
$finishedWalk = false;
|
||||
break;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
$version = Repository::fileVersion($path);
|
||||
if ($version === 'missing') {
|
||||
continue; // deleted while walking
|
||||
@@ -136,6 +159,11 @@ final class BackupRun {
|
||||
$writer->write(implode("\n", $lines) . "\n");
|
||||
$state['segments'][] = $writer->finish();
|
||||
}
|
||||
if ($emptyDirs !== []) {
|
||||
$writer = $repo->blobWriter($stats);
|
||||
$writer->write(implode("\n", $emptyDirs) . "\n");
|
||||
$state['dirSegments'][] = $writer->finish();
|
||||
}
|
||||
$repo->flushPacks();
|
||||
$state['parentPos'] = $parent->position();
|
||||
$state['stats'] = $stats;
|
||||
@@ -150,10 +178,18 @@ final class BackupRun {
|
||||
if ($state['phase'] === 'finish') {
|
||||
// Every segment ends with a newline, so they read as one stream: fetched together,
|
||||
// several at a time where the location allows it.
|
||||
// Empty folders go in the meta record, not between the files: a version that does not
|
||||
// know them reads the snapshot as before, and only restores no empty folders.
|
||||
$emptyDirs = [];
|
||||
foreach (isset($state['dirSegments']) ? $repo->readLines(array_values(array_merge(...$state['dirSegments']))) : [] as $line) {
|
||||
if ($line !== '') {
|
||||
$emptyDirs[] = json_decode($line, true, 8, JSON_THROW_ON_ERROR);
|
||||
}
|
||||
}
|
||||
$repo->putSnapshot($state['snapshot'],
|
||||
['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
|
||||
'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
|
||||
'warnings' => self::warnings($state)] + $state['meta'],
|
||||
'warnings' => self::warnings($state)] + ($emptyDirs === [] ? [] : ['emptyDirs' => $emptyDirs]) + $state['meta'],
|
||||
$repo->readLines(array_values(array_merge(...$state['segments']))));
|
||||
$state['phase'] = 'commit';
|
||||
}
|
||||
|
||||
+11
-2
@@ -9,7 +9,9 @@ namespace OCA\NgBackup\Job;
|
||||
|
||||
/**
|
||||
* Walks one or more source roots in a stable depth-first order (entries of a directory sorted
|
||||
* by name, children right after their directory), yielding "<root>/<relative path>" for files.
|
||||
* by name, children right after their directory), yielding "<root>/<relative path>" for files
|
||||
* and for directories with nothing in them to walk (the caller tells them apart with is_dir()).
|
||||
* A root itself without a name is never yielded.
|
||||
*
|
||||
* The same order is used for comparing paths (compare()), so a walk can resume after a cursor
|
||||
* and be merge-joined with the previous snapshot without holding any list in memory.
|
||||
@@ -66,6 +68,7 @@ final class TreeWalker {
|
||||
}
|
||||
$names = array_values(array_diff($names, ['.', '..']));
|
||||
sort($names, SORT_STRING);
|
||||
$empty = true; // no file or directory below it that is walked, whatever the cursor skips
|
||||
foreach ($names as $name) {
|
||||
$childLogical = $logical === '' ? $name : $logical . '/' . $name;
|
||||
$childAbs = $abs . '/' . $name;
|
||||
@@ -73,20 +76,26 @@ final class TreeWalker {
|
||||
continue; // never follow symlinks out of the data directory
|
||||
}
|
||||
if (is_dir($childAbs)) {
|
||||
$empty = $empty && $this->isExcluded($childLogical, true);
|
||||
// Skip a directory entirely if everything in it sorts before the cursor.
|
||||
if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) {
|
||||
continue;
|
||||
}
|
||||
yield from $this->walkDir($childAbs, $childLogical, $after);
|
||||
} elseif (is_file($childAbs)) {
|
||||
$excluded = $this->isExcluded($childLogical, false);
|
||||
$empty = $empty && $excluded;
|
||||
if ($after !== null && self::compare($childLogical, $after) <= 0) {
|
||||
continue;
|
||||
}
|
||||
if (!$this->isExcluded($childLogical, false)) {
|
||||
if (!$excluded) {
|
||||
yield $childLogical => $childAbs;
|
||||
}
|
||||
}
|
||||
}
|
||||
if ($empty && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) {
|
||||
yield $logical => $abs;
|
||||
}
|
||||
}
|
||||
|
||||
private function isExcluded(string $logical, bool $isDir): bool {
|
||||
|
||||
+146
-21
@@ -44,6 +44,8 @@ final class Repository {
|
||||
private ?int $maxBytes = null;
|
||||
private Catalog $catalog;
|
||||
private PackReader $packReader;
|
||||
/** @var array<string, array<string, mixed>> meta record of the snapshot read last */
|
||||
private array $lastMeta = [];
|
||||
/** verify() lists the pack folders from this many packs on (see presentPacks()); tests lower it. */
|
||||
public int $listPacksAbove = 256;
|
||||
|
||||
@@ -257,8 +259,8 @@ final class Repository {
|
||||
// that was being written when the process was killed stays (8 MiB, DR test 2026-11-05)
|
||||
$stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR);
|
||||
foreach ($stale === false ? [] : $stale as $dir) {
|
||||
if (is_link($dir)) {
|
||||
continue;
|
||||
if (is_link($dir) || preg_match('/^\.ngb-restore-spool-[0-9a-f]{8}$/', basename($dir)) !== 1) {
|
||||
continue; // not a spool of ours, whatever it is called
|
||||
}
|
||||
$files = glob($dir . '/*');
|
||||
foreach ($files === false ? [] : $files as $file) {
|
||||
@@ -313,40 +315,39 @@ final class Repository {
|
||||
if (!self::underPrefix($entry['p'], $prefix)) {
|
||||
continue;
|
||||
}
|
||||
if ($heartbeat !== null) {
|
||||
$heartbeat(); // also for a long row of files that are skipped or have no data
|
||||
}
|
||||
$dest = self::destination($target, $entry['p'], $prefix, $stripPrefix);
|
||||
if ($keepUnchanged && self::inPlace($dest, $entry)) {
|
||||
// below the target as written by an earlier run of this restore (checked then)
|
||||
$count++;
|
||||
continue;
|
||||
}
|
||||
if (!is_dir(dirname($dest))) {
|
||||
mkdir(dirname($dest), 0700, true);
|
||||
}
|
||||
// Defence in depth: the resolved directory must stay below the target (no symlink escapes).
|
||||
$base = realpath($target);
|
||||
$dir = realpath(dirname($dest));
|
||||
if ($base === false || $dir === false || ($dir !== $base && !str_starts_with($dir, $base . '/')) || is_link($dest)) {
|
||||
throw new RepositoryException('Refusing to write outside the restore directory: ' . $entry['p']);
|
||||
$dir = self::directoryInTarget($target, dirname($dest), $entry['p']);
|
||||
if (is_link($dest)) {
|
||||
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $entry['p']);
|
||||
}
|
||||
if (!isset($cleaned[$dir])) {
|
||||
self::removeStaleTemporaries($dir);
|
||||
$cleaned[$dir] = true;
|
||||
}
|
||||
if ($keepUnchanged && self::inPlace($dest, $entry)) {
|
||||
$count++;
|
||||
continue;
|
||||
}
|
||||
// Write next to the destination and rename after all blobs verified: at most one file
|
||||
// is ever present twice.
|
||||
$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
|
||||
$out = @fopen($tmp, 'xb'); // fails if anything (including a symlink) already exists there
|
||||
if ($out === false) {
|
||||
throw new RepositoryException('Cannot create a temporary file for ' . $entry['p']);
|
||||
throw new RestoreTargetException('Cannot create a temporary file for ' . $entry['p'] . ' (disk full, or no permission?)');
|
||||
}
|
||||
try {
|
||||
foreach ($entry['b'] as $id) {
|
||||
fwrite($out, $this->restoreBlob($id, $plan));
|
||||
self::writeAll($out, $this->restoreBlob($id, $plan), $entry['p']);
|
||||
if ($heartbeat !== null) {
|
||||
$heartbeat();
|
||||
}
|
||||
}
|
||||
fclose($out);
|
||||
if (!@fclose($out)) {
|
||||
throw new RestoreTargetException('Cannot write ' . $entry['p'] . ' to the restore directory (disk full?)');
|
||||
}
|
||||
clearstatcache(true, $tmp);
|
||||
if (filesize($tmp) !== $entry['s']) {
|
||||
throw new RepositoryException('Size mismatch restoring ' . $entry['p']);
|
||||
@@ -363,19 +364,112 @@ final class Repository {
|
||||
}
|
||||
$skipped[$entry['p']] = $e->getMessage();
|
||||
continue;
|
||||
} catch (\Throwable $e) {
|
||||
if (is_resource($out)) {
|
||||
@fclose($out);
|
||||
}
|
||||
@unlink($tmp);
|
||||
throw $e;
|
||||
}
|
||||
if (!rename($tmp, $dest)) {
|
||||
@unlink($tmp);
|
||||
throw new RepositoryException('Cannot put the restored file in place: ' . $entry['p']);
|
||||
throw new RestoreTargetException('Cannot put the restored file in place: ' . $entry['p']);
|
||||
}
|
||||
if (!touch($dest, $entry['m'])) {
|
||||
throw new RepositoryException('Cannot set the modification time of ' . $entry['p']);
|
||||
throw new RestoreTargetException('Cannot set the modification time of ' . $entry['p']);
|
||||
}
|
||||
$count++;
|
||||
}
|
||||
// Last: a file can only be restored into a folder that exists, and the meta record was
|
||||
// read along with the files, so this asks nothing more of the location.
|
||||
foreach ($this->emptyDirs($snapshotId) as [$path, $mtime]) {
|
||||
if (self::underPrefix($path, $prefix)) {
|
||||
if ($heartbeat !== null) {
|
||||
$heartbeat();
|
||||
}
|
||||
// The prefix itself, stripped, is the target (for a file it is the file's name there).
|
||||
$dest = $stripPrefix && $path === rtrim($prefix, '/') ? rtrim($target, '/') : self::destination($target, $path, $prefix, $stripPrefix);
|
||||
self::restoreEmptyDir($target, $dest, $path, $mtime);
|
||||
}
|
||||
}
|
||||
return $count;
|
||||
}
|
||||
|
||||
/**
|
||||
* The folders without anything in them that a snapshot recorded (none before 1.0.3).
|
||||
*
|
||||
* @return list<array{0: string, 1: int}> path, modification time
|
||||
*/
|
||||
public function emptyDirs(string $snapshotId): array {
|
||||
$meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId);
|
||||
$dirs = [];
|
||||
foreach (is_array($meta['emptyDirs'] ?? null) ? $meta['emptyDirs'] : [] as $dir) {
|
||||
// One folder that makes no sense is no reason to give up on the files of a restore.
|
||||
if (is_array($dir) && is_string($dir[0] ?? null) && self::isSafePath($dir[0]) && is_int($dir[1] ?? null)) {
|
||||
$dirs[] = [$dir[0], $dir[1]];
|
||||
}
|
||||
}
|
||||
return $dirs;
|
||||
}
|
||||
|
||||
private static function restoreEmptyDir(string $target, string $dest, string $path, int $mtime): void {
|
||||
if (is_link($dest)) {
|
||||
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
|
||||
}
|
||||
$existed = is_dir($dest);
|
||||
self::directoryInTarget($target, $dest, $path);
|
||||
if (!$existed && !@touch($dest, $mtime)) {
|
||||
throw new RestoreTargetException('Cannot set the modification time of ' . $path);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Make sure a folder exists below the target, without following a link out of it: what exists
|
||||
* of the path is checked before anything is created.
|
||||
*
|
||||
* @return string the folder's real path
|
||||
*/
|
||||
private static function directoryInTarget(string $target, string $dir, string $path): string {
|
||||
if (!is_dir($target) && !@mkdir($target, 0700, true) && !is_dir($target)) {
|
||||
throw new RestoreTargetException('Cannot create the restore directory ' . $target);
|
||||
}
|
||||
$base = realpath($target);
|
||||
$existing = $dir;
|
||||
while (!file_exists($existing) && !is_link($existing) && dirname($existing) !== $existing) {
|
||||
$existing = dirname($existing);
|
||||
}
|
||||
$real = realpath($existing);
|
||||
if ($base === false || $real === false || ($real !== $base && !str_starts_with($real, $base . '/'))) {
|
||||
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
|
||||
}
|
||||
if ($existing === $dir) {
|
||||
if (!is_dir($real)) {
|
||||
throw new RestoreTargetException('Cannot create the folder for ' . $path . ': a file is in the way');
|
||||
}
|
||||
return $real;
|
||||
}
|
||||
if (!@mkdir($dir, 0700, true) && !is_dir($dir)) {
|
||||
throw new RestoreTargetException('Cannot create the folder for ' . $path . ' (disk full, no permission, or a file in the way?)');
|
||||
}
|
||||
return (string)realpath($dir);
|
||||
}
|
||||
|
||||
/**
|
||||
* Write all of $data, or stop: a short write leaves a file that looks damaged only by its size.
|
||||
* A write may take only part of the data and still be fine; one that takes nothing is not.
|
||||
*
|
||||
* @param resource $out
|
||||
*/
|
||||
public static function writeAll($out, string $data, string $path): void {
|
||||
while ($data !== '') {
|
||||
$written = @fwrite($out, $data);
|
||||
if ($written === false || $written === 0) {
|
||||
throw new RestoreTargetException('Cannot write ' . $path . ' to the restore directory (disk full?)');
|
||||
}
|
||||
$data = substr($data, $written);
|
||||
}
|
||||
}
|
||||
|
||||
/** Where restore() writes the snapshot path $path. */
|
||||
private static function destination(string $target, string $path, string $prefix, bool $stripPrefix): string {
|
||||
$relPath = $path;
|
||||
@@ -722,6 +816,34 @@ final class Repository {
|
||||
return array_keys($doomed);
|
||||
}
|
||||
|
||||
/**
|
||||
* Part of trash(), for a page that shows how far it is: each entry is a read of its own on the
|
||||
* location. Reads the entries with an id after $after, in order, until $deadline
|
||||
* (microtime(true)), at least one. The id, not a position: a cleanup that changes the trash
|
||||
* between two parts then makes no entry come twice or be missed.
|
||||
*
|
||||
* @return array{total: int, items: array<string, array<string, mixed>>}
|
||||
*/
|
||||
public function trashPage(string $after, float $deadline): array {
|
||||
$ids = array_map('strval', array_keys($this->catalog->trash));
|
||||
sort($ids, SORT_STRING);
|
||||
$items = [];
|
||||
foreach ($ids as $id) {
|
||||
if (strcmp($id, $after) <= 0) {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
$items[$id] = $this->trashInfo($id);
|
||||
} catch (\Throwable) {
|
||||
throw new RepositoryException("Trash entry $id listed in the catalog is missing on the location (possible tampering)");
|
||||
}
|
||||
if (microtime(true) >= $deadline) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return ['total' => count($ids), 'items' => $items];
|
||||
}
|
||||
|
||||
private function trashInfo(string $snapshotId): array {
|
||||
return json_decode($this->getObject('trash/info/' . $snapshotId), true, 512, JSON_THROW_ON_ERROR);
|
||||
}
|
||||
@@ -1281,7 +1403,10 @@ final class Repository {
|
||||
$this->cipher->decrypt($fh, $plain, 'snapshot:' . $snapshotId);
|
||||
fclose($fh);
|
||||
rewind($plain);
|
||||
fgets($plain); // meta line
|
||||
$meta = json_decode((string)fgets($plain), true);
|
||||
if (is_array($meta)) {
|
||||
$this->lastMeta = [$snapshotId => $meta]; // a snapshot never changes
|
||||
}
|
||||
while (($line = fgets($plain)) !== false) {
|
||||
$entry = json_decode($line, true, 512, JSON_THROW_ON_ERROR);
|
||||
if (!is_array($entry) || !is_string($entry['p'] ?? null) || !self::isSafePath($entry['p'])
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
namespace OCA\NgBackup\Repository;
|
||||
|
||||
/**
|
||||
* A restore cannot write to where it restores to (a full disk, permissions, something in the way).
|
||||
* The backup is fine, so this never counts as damaged data: the restore stops, also with --salvage.
|
||||
*/
|
||||
final class RestoreTargetException extends \RuntimeException {
|
||||
}
|
||||
@@ -79,7 +79,7 @@ final class BackupService {
|
||||
try {
|
||||
return $this->leases->with(PruneService::lockKey($target), LeaseService::EXCLUSIVE, 60, fn () => $this->createRun($target, $label));
|
||||
} catch (LockedException) {
|
||||
throw new \RuntimeException('A cleanup or another start for ' . $target->getName() . ' is running; try again later');
|
||||
throw new \RuntimeException('A cleanup, a restore or another start for ' . $target->getName() . ' is running, or one was stopped by force (that clears by itself within two hours); try again later');
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
namespace OCA\NgBackup\Service;
|
||||
|
||||
/** A step after the database was committed failed: the database is no longer the fresh one. */
|
||||
final class DatabaseAlreadyRestoredException extends \RuntimeException {
|
||||
}
|
||||
@@ -76,6 +76,15 @@ final class LeaseService {
|
||||
$qb->executeStatement();
|
||||
}
|
||||
|
||||
/** Remove the shared leases on $name, for a caller that knows their holders were stopped. */
|
||||
public function dropShared(string $name): void {
|
||||
$qb = $this->db->getQueryBuilder();
|
||||
$qb->delete('ngb_leases')
|
||||
->where($qb->expr()->eq('name', $qb->createNamedParameter($name)))
|
||||
->andWhere($qb->expr()->eq('mode', $qb->createNamedParameter(self::SHARED)));
|
||||
$qb->executeStatement();
|
||||
}
|
||||
|
||||
/** Wraps $refresh so calling it often (per file, per row batch) only touches the database every $seconds. */
|
||||
public static function throttled(callable $refresh, int $seconds = 60): \Closure {
|
||||
$last = time();
|
||||
|
||||
@@ -151,6 +151,12 @@ final class RestoreFullService {
|
||||
// The database cannot go into this installation; the files can.
|
||||
$check['dbUnreadable'] = 'the database cannot be restored here: ' . $this->describeBlockers($check);
|
||||
}
|
||||
if ($interrupted) {
|
||||
// A restore that was killed leaves its leases, for up to two hours: they would keep
|
||||
// the first backup of the recovered server waiting after this restore completes.
|
||||
// In maintenance mode, on a partly restored instance, nothing else reads the location.
|
||||
$this->leases->dropShared(PruneService::lockKey($target));
|
||||
}
|
||||
$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
|
||||
$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check, $salvage);
|
||||
$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
|
||||
@@ -175,6 +181,8 @@ final class RestoreFullService {
|
||||
if ($dbError === null) {
|
||||
try {
|
||||
$this->restoreDatabase($repo, $meta, $manifest, $dbReport, $beat, $skipMissingTables);
|
||||
} catch (DatabaseAlreadyRestoredException $e) {
|
||||
throw $e;
|
||||
} catch (\Throwable $e) {
|
||||
if (!$salvage) {
|
||||
throw $e;
|
||||
@@ -201,6 +209,8 @@ final class RestoreFullService {
|
||||
try {
|
||||
/** @var array<string, mixed> $backupConfig */
|
||||
$backupConfig = $this->fetchBackupConfig($target, $snapshotId, $salvage);
|
||||
} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
|
||||
throw $e; // nowhere to put it: not a config.php that cannot be read
|
||||
} catch (\Throwable $e) {
|
||||
if (!$salvage) {
|
||||
throw $e;
|
||||
@@ -259,8 +269,20 @@ final class RestoreFullService {
|
||||
$dumper = new DbDumper($this->db->getInner(), $prefix);
|
||||
try {
|
||||
$dumper->restore($repo, $manifest, $dbReport, $beat, $skipMissingTables);
|
||||
} finally {
|
||||
} catch (\Throwable $e) {
|
||||
try {
|
||||
$this->restoreOwnAppConfig($ownAppConfig);
|
||||
} catch (\Throwable) {
|
||||
// rolled back, so they are still there: the error that brought us here is the one to report
|
||||
}
|
||||
throw $e;
|
||||
}
|
||||
try {
|
||||
$this->restoreOwnAppConfig($ownAppConfig);
|
||||
} catch (\Throwable $e) {
|
||||
// The old server's database is in place by now: not a case for "the database stays as
|
||||
// the fresh installation made it" under --salvage. A run again restores these rows.
|
||||
throw new DatabaseAlreadyRestoredException('The database was restored, but NG Backup could not put back its own settings: ' . $e->getMessage(), 0, $e);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -61,7 +61,8 @@ final class RestoreService {
|
||||
$prefix = trim($path, '/');
|
||||
$prefix = $prefix === '' ? '' : $prefix . '/';
|
||||
$children = [];
|
||||
foreach ($this->targets->repository($target)->entries($snapshotId) as $e) {
|
||||
$repo = $this->targets->repository($target);
|
||||
foreach ($repo->entries($snapshotId) as $e) {
|
||||
if ($prefix !== '' && !str_starts_with($e['p'], $prefix)) {
|
||||
continue;
|
||||
}
|
||||
@@ -75,6 +76,12 @@ final class RestoreService {
|
||||
}
|
||||
$children[$name] = $c;
|
||||
}
|
||||
foreach ($repo->emptyDirs($snapshotId) as [$dir]) { // read along with the entries
|
||||
if ($prefix === '' || str_starts_with($dir, $prefix)) {
|
||||
$name = explode('/', substr($dir, strlen($prefix)), 2)[0];
|
||||
$children[$name] ??= ['name' => $name, 'type' => 'dir', 'size' => 0, 'mtime' => null, 'files' => 0];
|
||||
}
|
||||
}
|
||||
ksort($children, SORT_STRING);
|
||||
return array_values($children);
|
||||
}
|
||||
@@ -123,7 +130,17 @@ final class RestoreService {
|
||||
$entries[substr($e['p'], strlen($source) + 1)] = $e;
|
||||
}
|
||||
}
|
||||
if ($entries === []) {
|
||||
// Folders without anything in them, relative to $source ('' when $source itself is one).
|
||||
/** @var array<array-key, int> $emptyDirs modification times; a folder called "2026" is an integer key */
|
||||
$emptyDirs = [];
|
||||
foreach ($single ? [] : $repo->emptyDirs($snapshotId) as [$dir, $mtime]) {
|
||||
if ($dir === $source) {
|
||||
$emptyDirs[''] = $mtime;
|
||||
} elseif (str_starts_with($dir, $source . '/')) {
|
||||
$emptyDirs[substr($dir, strlen($source) + 1)] = $mtime;
|
||||
}
|
||||
}
|
||||
if ($entries === [] && $emptyDirs === []) {
|
||||
throw new NotFoundException("Nothing at $source in snapshot $snapshotId");
|
||||
}
|
||||
|
||||
@@ -134,6 +151,7 @@ final class RestoreService {
|
||||
} elseif ($mode !== self::MODE_MERGE && $mode !== self::MODE_REPLACE) {
|
||||
throw new \InvalidArgumentException("Unknown mode $mode");
|
||||
}
|
||||
$destExisted = $parentRel === '' || $userFolder->nodeExists($parentRel);
|
||||
$dest = $this->ensureFolder($userFolder, $parentRel);
|
||||
$names = $single ? [basename($rel) => $entries['']] : $entries;
|
||||
|
||||
@@ -156,9 +174,21 @@ final class RestoreService {
|
||||
}
|
||||
}
|
||||
|
||||
foreach ($emptyDirs as $dir => $mtime) {
|
||||
$dir = (string)$dir;
|
||||
$existed = $dir === '' ? $destExisted : $dest->nodeExists($dir); // one that is there keeps its own time
|
||||
$folder = $this->ensureFolder($dest, $dir);
|
||||
if (!$existed) {
|
||||
$folder->touch($mtime);
|
||||
}
|
||||
if ($progress !== null) {
|
||||
$progress($restored, count($names), $dir); // keeps the lease alive too
|
||||
}
|
||||
}
|
||||
|
||||
$trashed = 0;
|
||||
if ($mode === self::MODE_REPLACE && !$single) {
|
||||
$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)));
|
||||
$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)) + $emptyDirs);
|
||||
}
|
||||
return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())];
|
||||
}
|
||||
@@ -178,7 +208,7 @@ final class RestoreService {
|
||||
throw new \InvalidArgumentException('Invalid path: ' . $prefix);
|
||||
}
|
||||
if (!is_dir($directory) && !mkdir($directory, 0750, true) && !is_dir($directory)) {
|
||||
throw new \RuntimeException("Cannot create $directory");
|
||||
throw new \OCA\NgBackup\Repository\RestoreTargetException("Cannot create $directory (disk full, or no permission?)");
|
||||
}
|
||||
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $prefix, $directory, $stripPrefix, $heartbeat, &$skipped, $keepUnchanged): int {
|
||||
$own = LeaseService::throttled($refresh);
|
||||
@@ -262,7 +292,7 @@ final class RestoreService {
|
||||
foreach ($folder->getDirectoryListing() as $node) {
|
||||
$rel = $prefix === '' ? $node->getName() : $prefix . '/' . $node->getName();
|
||||
if ($node instanceof Folder) {
|
||||
$hasKept = false;
|
||||
$hasKept = isset($keep[$rel]); // a folder that was empty in the snapshot
|
||||
foreach (array_keys($keep) as $k) {
|
||||
if (str_starts_with((string)$k, $rel . '/')) {
|
||||
$hasKept = true;
|
||||
|
||||
@@ -186,6 +186,7 @@
|
||||
<code><![CDATA[$this->db]]></code>
|
||||
<code><![CDATA[$this->db]]></code>
|
||||
<code><![CDATA[$this->db]]></code>
|
||||
<code><![CDATA[$this->db]]></code>
|
||||
<code><![CDATA[TransactionIsolationLevel]]></code>
|
||||
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
||||
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
||||
|
||||
+1
-1
@@ -21,7 +21,7 @@ export default {
|
||||
addSlot: (passphrase, label) => call('post', '/key/slots', { passphrase, label }),
|
||||
replaceSlot: (slot, passphrase, label) => call('put', `/key/slots/${slot}`, { passphrase, label }),
|
||||
removeSlot: (slot) => call('delete', `/key/slots/${slot}`),
|
||||
trash: (targetId) => call('get', `/targets/${targetId}/trash`),
|
||||
trash: (targetId, after = '') => call('get', `/targets/${targetId}/trash`, { after }),
|
||||
untrash: (targetId, snapshotId) => call('post', `/targets/${targetId}/trash/${snapshotId}`),
|
||||
kitUrl: () => url('/key/kit'),
|
||||
confirmKit: (accepted, phrase, code) => call('post', '/key/confirm', { accepted, phrase, code }),
|
||||
|
||||
+53
-12
@@ -6,8 +6,18 @@
|
||||
<NcSettingsSection :name="t('ng_backup', 'Bin')"
|
||||
:description="t('ng_backup', 'Restore points that were removed, most of them by the retention policy. They can be brought back until they are deleted permanently.')">
|
||||
<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
|
||||
<NcLoadingIcon v-if="loading" :size="32" />
|
||||
<NcNoteCard v-else-if="empty" type="info">{{ t('ng_backup', 'The bin is empty.') }}</NcNoteCard>
|
||||
<div v-if="reading" class="ngb-reading" role="status">
|
||||
<template v-if="reading.total === null">
|
||||
<NcLoadingIcon :size="20" />
|
||||
<span>{{ t('ng_backup', 'Connecting to {location} …', { location: reading.name }) }}</span>
|
||||
</template>
|
||||
<template v-else>
|
||||
<span>{{ t('ng_backup', 'Reading removed restore points from {location}: {done} of {total}', { location: reading.name, done: reading.done, total: reading.total }) }}</span>
|
||||
<NcProgressBar :value="reading.total ? Math.round(100 * reading.done / reading.total) : 0" />
|
||||
</template>
|
||||
<p class="ngb-muted">{{ t('ng_backup', 'This can take a while: every removed restore point is read from the location.') }}</p>
|
||||
</div>
|
||||
<NcNoteCard v-else-if="empty && !error" type="info">{{ t('ng_backup', 'The bin is empty.') }}</NcNoteCard>
|
||||
|
||||
<template v-for="target in status.targets" :key="'trash' + target.id">
|
||||
<div v-if="trash[target.id]?.length" class="ngb-trash">
|
||||
@@ -39,12 +49,13 @@
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { computed, reactive, ref, watch } from 'vue'
|
||||
import { computed, onBeforeUnmount, reactive, ref, watch } from 'vue'
|
||||
import { n, t } from '@nextcloud/l10n'
|
||||
import { confirmPassword } from '@nextcloud/password-confirmation'
|
||||
import NcButton from '@nextcloud/vue/components/NcButton'
|
||||
import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
|
||||
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
||||
import NcProgressBar from '@nextcloud/vue/components/NcProgressBar'
|
||||
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||
import api from '../api.js'
|
||||
import { formatDate, formatDateTime } from '../format.js'
|
||||
@@ -56,21 +67,49 @@ const emit = defineEmits(['changed'])
|
||||
const RETENTION = 'retention policy'
|
||||
const retentionInfo = ref(false)
|
||||
const trash = reactive({})
|
||||
const loading = ref(true)
|
||||
const reading = ref({ name: '', total: null, done: 0 }) // the location being read; null when done
|
||||
const error = ref('')
|
||||
const empty = computed(() => props.status.targets.every(target => !trash[target.id]?.length))
|
||||
|
||||
async function loadTrash() {
|
||||
for (const target of props.status.targets) {
|
||||
try {
|
||||
trash[target.id] = await api.trash(target.id)
|
||||
} catch (e) {
|
||||
trash[target.id] = []
|
||||
let current = 0 // the newest load; an older one stops (another took over, or the tab was left)
|
||||
onBeforeUnmount(() => { current++ })
|
||||
|
||||
async function loadTarget(target, load) {
|
||||
reading.value = { name: target.name, total: null, done: 0 }
|
||||
const items = []
|
||||
for (;;) {
|
||||
const page = await api.trash(target.id, items.length ? items[items.length - 1].id : '')
|
||||
if (load !== current) {
|
||||
return
|
||||
}
|
||||
items.push(...page.items)
|
||||
reading.value = { name: target.name, total: page.total, done: Math.min(items.length, page.total) }
|
||||
if (!page.items.length || items.length >= page.total) {
|
||||
break
|
||||
}
|
||||
}
|
||||
loading.value = false
|
||||
trash[target.id] = items.sort((a, b) => String(b.time ?? '').localeCompare(String(a.time ?? '')))
|
||||
}
|
||||
watch(() => props.status.targets.map(tg => tg.id).join(','), loadTrash, { immediate: true })
|
||||
|
||||
async function loadTrash() {
|
||||
const load = ++current
|
||||
const failed = []
|
||||
for (const target of props.status.targets) {
|
||||
try {
|
||||
await loadTarget(target, load)
|
||||
} catch (e) {
|
||||
// keep what was shown: a location that cannot be reached does not mean its bin is empty
|
||||
failed.push(target.name + (e?.message ? ': ' + e.message : ''))
|
||||
}
|
||||
if (load !== current) {
|
||||
return
|
||||
}
|
||||
}
|
||||
error.value = failed.length ? t('ng_backup', 'The bin could not be read from {locations}', { locations: failed.join('; ') }) : ''
|
||||
reading.value = null
|
||||
}
|
||||
// Again when a run ends: a cleanup moves restore points here.
|
||||
watch(() => props.status.targets.map(tg => tg.id).join(',') + '|' + (props.status.runs ?? []).filter(run => run.status !== 'running').map(run => run.id).join(','), loadTrash, { immediate: true })
|
||||
|
||||
async function untrash(target, item) {
|
||||
error.value = ''
|
||||
@@ -88,6 +127,8 @@ async function untrash(target, item) {
|
||||
<style scoped>
|
||||
.ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); }
|
||||
.ngb-muted { color: var(--color-text-maxcontrast); }
|
||||
.ngb-reading { margin: 12px 0; display: flex; flex-direction: column; gap: 6px; max-width: 600px; }
|
||||
.ngb-reading p { margin: 0; }
|
||||
.ngb-trash { margin: 12px 0; }
|
||||
.ngb-trash h3 { font-weight: bold; }
|
||||
</style>
|
||||
|
||||
@@ -28,6 +28,28 @@ class TreeWalkerTest extends TestCase {
|
||||
$this->assertSame(['Z', 'a', 'a-b', 'a.txt', 'b', 'skip'], array_values(array_unique(array_map(fn ($p) => explode('/', $p)[0], $paths))));
|
||||
}
|
||||
|
||||
public function testEmptyDirectoriesAreYieldedInOrderAndOnResume(): void {
|
||||
$d = $this->tree();
|
||||
mkdir("$d/a/empty");
|
||||
mkdir("$d/a-hollow/inner", 0700, true);
|
||||
symlink("$d/a.txt", "$d/skip/link");
|
||||
mkdir("$d/only-link");
|
||||
symlink("$d/a.txt", "$d/only-link/link");
|
||||
$w = new TreeWalker(['' => $d]);
|
||||
$all = array_keys(iterator_to_array($w->walk()));
|
||||
$this->assertContains('a/empty', $all);
|
||||
$this->assertContains('a-hollow/inner', $all);
|
||||
$this->assertNotContains('a-hollow', $all, 'only the deepest');
|
||||
$this->assertContains('only-link', $all, 'a link is not walked');
|
||||
$this->assertNotContains('skip', $all);
|
||||
$sorted = $all;
|
||||
usort($sorted, [TreeWalker::class, 'compare']);
|
||||
$this->assertSame($sorted, $all);
|
||||
foreach ($all as $i => $cursor) {
|
||||
$this->assertSame(array_slice($all, $i + 1), array_keys(iterator_to_array($w->walk($cursor))), "after $cursor");
|
||||
}
|
||||
}
|
||||
|
||||
public function testResumeAfterCursorYieldsTheRest(): void {
|
||||
$w = new TreeWalker(['' => $this->tree()]);
|
||||
$all = array_keys(iterator_to_array($w->walk()));
|
||||
|
||||
@@ -50,6 +50,40 @@ class CatalogTest extends TestCase {
|
||||
$this->assertSame(4, $again->catalogGeneration());
|
||||
}
|
||||
|
||||
public function testTheTrashIsReadInPartsThatTogetherAreAll(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
for ($i = 0; $i < 3; $i++) {
|
||||
$repo->forget($repo->backupDirectory($this->src)['snapshot']);
|
||||
}
|
||||
$again = $this->open();
|
||||
$seen = [];
|
||||
// A deadline already past: one entry per request, as on a slow location
|
||||
$after = '';
|
||||
for ($request = 0; count($seen) < 3 && $request < 5; $request++) {
|
||||
$page = $again->trashPage($after, 0.0);
|
||||
$this->assertSame(3, $page['total']);
|
||||
$this->assertCount(1, $page['items']);
|
||||
$seen += $page['items'];
|
||||
$after = (string)array_key_last($page['items']);
|
||||
}
|
||||
$this->assertEqualsCanonicalizing(array_keys($again->trash()), array_keys($seen));
|
||||
$this->assertSame(['total' => 3, 'items' => []], $again->trashPage($after, 0.0));
|
||||
$this->assertCount(3, $again->trashPage('', microtime(true) + 60)['items']);
|
||||
}
|
||||
|
||||
public function testAnEntryThatLeavesTheTrashBetweenPartsMakesNoneComeTwiceOrBeMissed(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
for ($i = 0; $i < 3; $i++) {
|
||||
$repo->forget($repo->backupDirectory($this->src)['snapshot']);
|
||||
}
|
||||
$reader = $this->open();
|
||||
$ids = array_keys($reader->trashPage('', microtime(true) + 60)['items']);
|
||||
$first = $reader->trashPage('', 0.0)['items'];
|
||||
$this->assertSame([$ids[0]], array_keys($first));
|
||||
$this->open()->untrash($ids[0]); // another administrator, between two parts
|
||||
$this->assertSame([$ids[1], $ids[2]], array_keys($this->open()->trashPage($ids[0], microtime(true) + 60)['items']));
|
||||
}
|
||||
|
||||
public function testRollbackIsDetected(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$repo->backupDirectory($this->src);
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||
|
||||
use OCA\NgBackup\Backend\LocalBackend;
|
||||
use OCA\NgBackup\Crypto\KeyRing;
|
||||
use OCA\NgBackup\Job\BackupRun;
|
||||
use OCA\NgBackup\Repository\Repository;
|
||||
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
|
||||
/** Folders without files are recorded in the snapshot's meta record and come back on a restore. */
|
||||
class EmptyDirsTest extends TestCase {
|
||||
use TempDirTrait;
|
||||
|
||||
private LocalBackend $backend;
|
||||
private KeyRing $keys;
|
||||
private string $src;
|
||||
|
||||
protected function setUp(): void {
|
||||
$this->backend = new LocalBackend($this->tempDir());
|
||||
$this->keys = KeyRing::generate();
|
||||
Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
|
||||
$this->src = $this->tempDir();
|
||||
mkdir("$this->src/data/alice/files/Docs", 0700, true);
|
||||
mkdir("$this->src/data/alice/files/Hollow/Nested", 0700, true);
|
||||
mkdir("$this->src/data/skipped/inner", 0700, true);
|
||||
file_put_contents("$this->src/data/alice/files/Docs/note.txt", "hello\n");
|
||||
touch("$this->src/data/alice/files/Hollow/Nested", 1700000000);
|
||||
}
|
||||
|
||||
private function repo(): Repository {
|
||||
return Repository::openWithKey($this->backend, $this->keys);
|
||||
}
|
||||
|
||||
/** @param list<string> $exclude */
|
||||
private function backup(array $exclude = [], float $seconds = 30.0): string {
|
||||
$state = BackupRun::start(['data' => "$this->src/data"], null, '', $exclude);
|
||||
for ($i = 0; $i < 500 && $state['phase'] !== 'commit'; $i++) {
|
||||
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, microtime(true) + $seconds)), true);
|
||||
}
|
||||
$this->assertSame('commit', $state['phase']);
|
||||
$this->repo()->recordSnapshot($state['snapshot']);
|
||||
return $state['snapshot'];
|
||||
}
|
||||
|
||||
public function testOnlyTheDeepestEmptyFolderIsRecordedAndFilesStayAsTheyAre(): void {
|
||||
$snapshot = $this->backup();
|
||||
$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
|
||||
$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], $paths);
|
||||
$this->assertSame(['data/alice/files/Docs/note.txt'], array_column(iterator_to_array($this->repo()->entries($snapshot), false), 'p'));
|
||||
$this->assertSame(2, $this->repo()->snapshotMeta($snapshot)['stats']['emptyDirs']);
|
||||
}
|
||||
|
||||
public function testAFolderWithOnlyExcludedContentCountsAsEmpty(): void {
|
||||
$snapshot = $this->backup(['data/skipped/inner']);
|
||||
$this->assertContains('data/skipped', array_column($this->repo()->emptyDirs($snapshot), 0));
|
||||
$this->assertNotContains('data/skipped/inner', array_column($this->repo()->emptyDirs($snapshot), 0));
|
||||
}
|
||||
|
||||
public function testAResumedWalkRecordsEveryFolderOnce(): void {
|
||||
for ($i = 0; $i < 30; $i++) {
|
||||
mkdir("$this->src/data/many/d$i", 0700, true);
|
||||
file_put_contents("$this->src/data/many/f$i", 'x');
|
||||
}
|
||||
$snapshot = $this->backup([], 0.0); // a step per entry
|
||||
$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
|
||||
$this->assertCount(32, $paths);
|
||||
$this->assertSame(array_unique($paths), $paths);
|
||||
}
|
||||
|
||||
public function testANameASnapshotCannotHoldIsReportedAndTheSnapshotStaysReadable(): void {
|
||||
mkdir("$this->src/data/odd\\folder", 0700, true);
|
||||
mkdir("$this->src/data/alice/files/line\nbreak", 0700, true);
|
||||
file_put_contents("$this->src/data/alice/files/back\\slash.txt", 'x');
|
||||
file_put_contents("$this->src/data/alice/files/\xff.txt", 'x');
|
||||
$state = BackupRun::start(['data' => "$this->src/data"]);
|
||||
for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
|
||||
// A step per entry, state through JSON: as separate processes would
|
||||
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
|
||||
}
|
||||
$this->assertSame('commit', $state['phase']);
|
||||
$this->assertSame(4, $state['stats']['unreadable'], 'each reported once, over several steps');
|
||||
$this->repo()->recordSnapshot($state['snapshot']);
|
||||
|
||||
$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], array_column($this->repo()->emptyDirs($state['snapshot']), 0));
|
||||
$target = $this->tempDir();
|
||||
$this->assertSame(1, $this->repo()->restore($state['snapshot'], $target));
|
||||
$this->assertFileExists("$target/data/alice/files/Docs/note.txt");
|
||||
}
|
||||
|
||||
public function testAnEmptyFolderThatIsThePrefixItselfIsTheTarget(): void {
|
||||
$snapshot = $this->backup();
|
||||
$target = $this->tempDir();
|
||||
$this->repo()->restore($snapshot, $target, 'data/skipped/inner/', true);
|
||||
$this->assertSame([], glob("$target/*"), 'not a folder "inner" inside the target');
|
||||
}
|
||||
|
||||
public function testRestoreBringsThemBackWithTheirTime(): void {
|
||||
$snapshot = $this->backup();
|
||||
$target = $this->tempDir();
|
||||
$this->assertSame(1, $this->repo()->restore($snapshot, $target));
|
||||
$this->assertDirectoryExists("$target/data/alice/files/Hollow/Nested");
|
||||
clearstatcache();
|
||||
$this->assertSame(1700000000, filemtime("$target/data/alice/files/Hollow/Nested"));
|
||||
$this->assertDirectoryExists("$target/data/skipped/inner");
|
||||
}
|
||||
|
||||
public function testRestoreOfAPrefixStripsAndLeavesOthersOut(): void {
|
||||
$snapshot = $this->backup();
|
||||
$target = $this->tempDir();
|
||||
$this->repo()->restore($snapshot, $target, 'data/alice/', true);
|
||||
$this->assertDirectoryExists("$target/files/Hollow/Nested");
|
||||
$this->assertDirectoryDoesNotExist("$target/skipped");
|
||||
$this->assertDirectoryDoesNotExist("$target/inner");
|
||||
}
|
||||
|
||||
public function testRunningAgainLeavesAFolderThatIsThereAlone(): void {
|
||||
$snapshot = $this->backup();
|
||||
$target = $this->tempDir();
|
||||
$this->repo()->restore($snapshot, $target);
|
||||
touch("$target/data/alice/files/Hollow/Nested", 1800000000);
|
||||
file_put_contents("$target/data/alice/files/Hollow/Nested/added", 'kept');
|
||||
touch("$target/data/alice/files/Hollow/Nested", 1800000000);
|
||||
$this->repo()->restore($snapshot, $target, keepUnchanged: true);
|
||||
clearstatcache();
|
||||
$this->assertSame(1800000000, filemtime("$target/data/alice/files/Hollow/Nested"));
|
||||
$this->assertFileExists("$target/data/alice/files/Hollow/Nested/added");
|
||||
}
|
||||
|
||||
public function testALinkOutOfTheTargetIsRefused(): void {
|
||||
$snapshot = $this->backup();
|
||||
$target = $this->tempDir();
|
||||
$outside = $this->tempDir();
|
||||
mkdir("$target/data/alice/files", 0700, true);
|
||||
symlink($outside, "$target/data/alice/files/Hollow");
|
||||
try {
|
||||
$this->repo()->restore($snapshot, $target);
|
||||
$this->fail('restored through a link');
|
||||
} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
|
||||
$this->assertStringContainsString('outside', $e->getMessage());
|
||||
}
|
||||
$this->assertSame([], glob("$outside/*"));
|
||||
}
|
||||
|
||||
public function testASnapshotWithoutThemHasNone(): void {
|
||||
$src = $this->tempDir();
|
||||
file_put_contents("$src/a", 'x');
|
||||
$snapshot = $this->repo()->backupDirectory($src)['snapshot'];
|
||||
$this->assertSame([], $this->repo()->emptyDirs($snapshot));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||
|
||||
use OCA\NgBackup\Backend\LocalBackend;
|
||||
use OCA\NgBackup\Command\Restore;
|
||||
use OCA\NgBackup\Crypto\KeyRing;
|
||||
use OCA\NgBackup\Repository\Repository;
|
||||
use OCA\NgBackup\Repository\RestoreTargetException;
|
||||
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
|
||||
/** A stream that takes at most 3 bytes per write. */
|
||||
class PartialWriteStream {
|
||||
public static string $written = '';
|
||||
/** @var resource|null set by PHP */
|
||||
public $context;
|
||||
|
||||
public function stream_open(string $path, string $mode, int $options, ?string &$opened): bool {
|
||||
self::$written = '';
|
||||
return true;
|
||||
}
|
||||
|
||||
public function stream_write(string $data): int {
|
||||
$part = substr($data, 0, 3);
|
||||
self::$written .= $part;
|
||||
return strlen($part);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* What goes wrong where a restore writes to is not damage in the backup: it stops the restore, also
|
||||
* with --salvage, and the message does not point at --salvage.
|
||||
*/
|
||||
class RestoreTargetTest extends TestCase {
|
||||
use TempDirTrait;
|
||||
|
||||
private LocalBackend $backend;
|
||||
private KeyRing $keys;
|
||||
private string $snapshot;
|
||||
|
||||
protected function setUp(): void {
|
||||
$this->backend = new LocalBackend($this->tempDir());
|
||||
$this->keys = KeyRing::generate();
|
||||
$src = $this->tempDir();
|
||||
mkdir("$src/dir");
|
||||
file_put_contents("$src/dir/a.bin", random_bytes(6 * 1048576));
|
||||
file_put_contents("$src/dir/b.bin", random_bytes(6 * 1048576));
|
||||
file_put_contents("$src/dir/c.txt", 'small');
|
||||
$this->snapshot = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'))->backupDirectory($src)['snapshot'];
|
||||
}
|
||||
|
||||
private function repo(): Repository {
|
||||
return Repository::openWithKey($this->backend, $this->keys);
|
||||
}
|
||||
|
||||
public function testAShortWriteIsATargetError(): void {
|
||||
if (!is_writable('/dev/full')) {
|
||||
$this->markTestSkipped('needs /dev/full');
|
||||
}
|
||||
$out = fopen('/dev/full', 'wb');
|
||||
stream_set_write_buffer($out, 0);
|
||||
$this->expectException(RestoreTargetException::class);
|
||||
Repository::writeAll($out, 'data', 'dir/a.bin');
|
||||
}
|
||||
|
||||
public function testAWriteThatTakesPartOfTheDataGoesOn(): void {
|
||||
stream_wrapper_register('ngbpartial', PartialWriteStream::class);
|
||||
try {
|
||||
$out = fopen('ngbpartial://x', 'wb');
|
||||
Repository::writeAll($out, 'abcdefgh', 'dir/a.bin');
|
||||
fclose($out);
|
||||
$this->assertSame('abcdefgh', PartialWriteStream::$written);
|
||||
} finally {
|
||||
stream_wrapper_unregister('ngbpartial');
|
||||
}
|
||||
}
|
||||
|
||||
public function testATargetErrorDoesNotLookLikeDamage(): void {
|
||||
$this->assertFalse(Restore::backupLooksDamaged(new RestoreTargetException('Cannot write')));
|
||||
}
|
||||
|
||||
/**
|
||||
* A real full disk: run with NGB_TEST_SMALL_DISK set to an empty, writable tmpfs of a few MiB
|
||||
* (docker run --tmpfs /small:size=8m,uid=...), with and without --salvage.
|
||||
*
|
||||
* @dataProvider salvageOrNot
|
||||
*/
|
||||
public function testAFullDiskStopsTheRestoreAndLeavesNoTemporary(bool $salvage): void {
|
||||
$disk = getenv('NGB_TEST_SMALL_DISK');
|
||||
if ($disk === false || $disk === '') {
|
||||
$this->markTestSkipped('needs NGB_TEST_SMALL_DISK');
|
||||
}
|
||||
$target = $disk . '/' . bin2hex(random_bytes(4));
|
||||
$skipped = $salvage ? [] : null;
|
||||
try {
|
||||
$this->repo()->restore($this->snapshot, $target, skipped: $skipped);
|
||||
$this->fail('12 MiB fitted on the small disk');
|
||||
} catch (RestoreTargetException $e) {
|
||||
$this->assertStringContainsString('disk full', $e->getMessage());
|
||||
} finally {
|
||||
$left = glob("$target/dir/*.ngb-restore-*");
|
||||
foreach (glob("$target/dir/*") ?: [] as $f) {
|
||||
unlink($f);
|
||||
}
|
||||
@rmdir("$target/dir");
|
||||
@rmdir($target);
|
||||
}
|
||||
$this->assertSame([], $left);
|
||||
if ($salvage) {
|
||||
$this->assertSame([], $skipped, 'nothing is listed as damaged');
|
||||
}
|
||||
}
|
||||
|
||||
/** @return array<string, array{bool}> */
|
||||
public static function salvageOrNot(): array {
|
||||
return ['without --salvage' => [false], 'with --salvage' => [true]];
|
||||
}
|
||||
|
||||
public function testResumeChecksTheFolderBeforeSkipping(): void {
|
||||
$target = $this->tempDir();
|
||||
$this->repo()->restore($this->snapshot, $target);
|
||||
$outside = $this->tempDir();
|
||||
rename("$target/dir", "$outside/dir");
|
||||
symlink("$outside/dir", "$target/dir");
|
||||
$this->expectException(RestoreTargetException::class);
|
||||
$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
|
||||
}
|
||||
|
||||
public function testResumeRemovesTemporariesWhereEverythingIsInPlace(): void {
|
||||
$target = $this->tempDir();
|
||||
$this->repo()->restore($this->snapshot, $target);
|
||||
file_put_contents("$target/dir/a.bin.ngb-restore-0123456789ab", 'left by a stopped restore');
|
||||
$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
|
||||
$this->assertFileDoesNotExist("$target/dir/a.bin.ngb-restore-0123456789ab");
|
||||
}
|
||||
|
||||
public function testHeartbeatRunsForFilesThatAreSkipped(): void {
|
||||
$target = $this->tempDir();
|
||||
$this->repo()->restore($this->snapshot, $target);
|
||||
$beats = 0;
|
||||
$this->repo()->restore($this->snapshot, $target, '', false, function () use (&$beats): void {
|
||||
$beats++;
|
||||
}, keepUnchanged: true);
|
||||
$this->assertSame(3, $beats);
|
||||
}
|
||||
}
|
||||
@@ -102,7 +102,7 @@ ok "key, kit and location on the origin"
|
||||
|
||||
NC_PASS=$(openssl rand -hex 16)
|
||||
docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
|
||||
docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
|
||||
docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs /var/www/html/data/frtest/files/Empty/Nested && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
|
||||
if [ "$MODE" = sftp ]; then
|
||||
# about three packs of 32 MiB
|
||||
docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
|
||||
@@ -214,6 +214,7 @@ if [ "$MODE" = salvage ]; then
|
||||
printf '%s\n' "$OUT" | grep "$SALVAGE" >/dev/null || { printf '%s\n' "$OUT" | tail -30; fail "the output does not name the salvage folder"; }
|
||||
[ "$(docker exec ngb-frt-restore cat "$SALVAGE/data/frtest/files/Docs/note.txt")" = "hello disaster recovery" ] || fail "the user's file is not in the salvage folder"
|
||||
docker exec ngb-frt-restore test -f "$SALVAGE/config/config.php" || fail "config.php is not in the salvage folder"
|
||||
docker exec ngb-frt-restore test -d "$SALVAGE/data/frtest/files/Empty/Nested" || fail "the empty folder is not in the salvage folder"
|
||||
run_on ngb-frt-restore user:list | grep frtest >/dev/null && fail "the fresh database was changed (the backed-up user is there)"
|
||||
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
|
||||
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
|
||||
@@ -254,8 +255,8 @@ if [ "$MODE" = salvage-files ]; then
|
||||
if (!file_exists($root . $p)) { $bad[] = "not listed and not there: $p"; continue; }
|
||||
hash_file("sha256", $root . $p) === $sum ? $same++ : $bad[] = "different: $p";
|
||||
}
|
||||
$temps = glob($root . "*/*.ngb-restore-*") ?: [];
|
||||
foreach ($temps as $t) { $bad[] = "temporary left: $t"; }
|
||||
$walk = new RecursiveIteratorIterator(new RecursiveDirectoryIterator("/var/www/html/data", FilesystemIterator::SKIP_DOTS), RecursiveIteratorIterator::SELF_FIRST);
|
||||
foreach ($walk as $f) { if (str_contains($f->getFilename(), ".ngb-restore-")) { $bad[] = "temporary left: " . $f->getPathname(); } }
|
||||
echo "same=$same gone=$gone bad=", count($bad), "\n", implode("\n", array_slice($bad, 0, 10)), "\n";
|
||||
echo "reasons: ", implode(" | ", array_unique(array_map(fn ($r) => substr(preg_replace("#[0-9a-f]{16,}#", "<id>", $r), 0, 70), $reasons))), "\n";')
|
||||
printf '%s\n' "$CHECK"
|
||||
@@ -265,6 +266,8 @@ if [ "$MODE" = salvage-files ]; then
|
||||
ok "with --salvage: every file listed and absent, or there and identical"
|
||||
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
|
||||
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing: the database was not restored"
|
||||
run_on ngb-frt-restore user:list | grep admin >/dev/null || fail "the admin is missing: the database was not restored"
|
||||
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back with --salvage"
|
||||
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt 2>/dev/null)" = "hello disaster recovery" ] \
|
||||
|| printf '%s\n' "$OUT" | grep "Docs/note.txt" >/dev/null || fail "note.txt is neither restored nor listed"
|
||||
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
|
||||
@@ -323,7 +326,16 @@ run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails aft
|
||||
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database"
|
||||
docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
|
||||
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
|
||||
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
||||
if [ "$LEGACY" = legacy ]; then
|
||||
# Backups made before 1.0.3 do not record empty folders
|
||||
docker exec ngb-frt-restore test -e /var/www/html/data/frtest/files/Empty/Nested && fail "an empty folder came back from a backup that does not record them"
|
||||
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
||||
else
|
||||
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back"
|
||||
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
||||
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "files:scan removed the empty folder"
|
||||
ok "the empty folder came back and stays after files:scan"
|
||||
fi
|
||||
# New rows after the restore: a new file (file cache) and a new user must not collide with
|
||||
# restored rows (ID counters after the data).
|
||||
docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"
|
||||
|
||||
@@ -26,7 +26,7 @@ ok "location added"
|
||||
# Test data: a user with files.
|
||||
$X sh -c "OC_PASS='Roundtrip-pw-2026' php occ user:add --password-from-env $U >/dev/null"
|
||||
# large.bin (40 MB) fills more than one pack: a restore to a directory reads ahead over several ranges
|
||||
$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin"
|
||||
$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs/Empty/Nested $DATA/$U/files/Hollow && touch -d @1700000000 $DATA/$U/files/Hollow && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin"
|
||||
occ files:scan "$U" >/dev/null
|
||||
SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
||||
|
||||
@@ -39,11 +39,18 @@ SNAP=$(occ backup:list ci | awk -F'|' '!f && / second /{gsub(/ /,"",$3); print $
|
||||
[ -n "$SNAP" ] || fail "snapshot not listed"
|
||||
|
||||
# Damage, then restore in each mode.
|
||||
$X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt"
|
||||
$X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt && rm -r Docs/Empty && rmdir Hollow"
|
||||
occ files:scan "$U" >/dev/null
|
||||
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m new-folder >/dev/null || fail "restore new-folder"
|
||||
$X sh -c "ls $DATA/$U/files | grep -q '^Restored '" || fail "no Restored folder"
|
||||
$X sh -c "test -d $DATA/$U/files/Restored\ */Docs/Empty/Nested" || fail "the empty folder is not in the Restored folder"
|
||||
occ backup:browse ci "$SNAP" "data/$U/files" | grep Hollow >/dev/null || fail "an empty folder is not shown when browsing the snapshot"
|
||||
occ backup:restore ci "$SNAP" "data/$U/files/Hollow" -m merge >/dev/null || fail "restore of a folder that was empty"
|
||||
$X sh -c "test -d $DATA/$U/files/Hollow" || fail "the empty folder did not come back by itself"
|
||||
[ "$($X stat -c %Y "$DATA/$U/files/Hollow")" = 1700000000 ] || fail "the empty folder did not get its time back"
|
||||
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m replace >/dev/null || fail "restore replace"
|
||||
$X sh -c "test -d $DATA/$U/files/Docs/Empty/Nested" || fail "replace did not bring back the empty folder, or moved it to the trash"
|
||||
ok "empty folders: restored in each mode, by themselves and shown when browsing"
|
||||
$X sh -c "rm -rf '$DATA/$U/files/Restored '*" ; occ files:scan "$U" >/dev/null
|
||||
SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
||||
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore"
|
||||
@@ -63,6 +70,7 @@ occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-dir >/dev/null || f
|
||||
SUM_DIR=$($X sh -c "cd /tmp/ngb-rt-dir/data/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
||||
[ "$SUM_BEFORE" = "$SUM_DIR" ] || fail "the files restored to a directory differ"
|
||||
$X sh -c "ls -A /tmp/ngb-rt-dir | grep -q ngb-restore-spool" && fail "the read-ahead spool was left in the directory"
|
||||
$X sh -c "test -d /tmp/ngb-rt-dir/data/$U/files/Docs/Empty/Nested" || fail "the empty folder is not in the directory"
|
||||
$X sh -c "rm -rf /tmp/ngb-rt-dir"
|
||||
ok "restore of all the user's files to a directory: identical, no spool left"
|
||||
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
|
||||
@@ -90,6 +98,9 @@ done
|
||||
[ "$RS" = done ] || fail "the backup started through the API ended as '$RS' after $i ticks"
|
||||
occ backup:list ci | grep " page " >/dev/null || fail "the snapshot of the ticked backup is not listed"
|
||||
ok "without system cron: warning in backup:status and the API, backup finished by $i ticks of the open page"
|
||||
# The Bin reads in parts, as the page does (what is in the parts: CatalogTest)
|
||||
api "$API/targets/$TID/trash" | php -r '$p = json_decode(stream_get_contents(STDIN), true); exit(is_int($p["total"] ?? null) && is_array($p["items"] ?? null) && count($p["items"]) <= $p["total"] ? 0 : 1);' || fail "the Bin API does not give a total and items"
|
||||
ok "the Bin API reads in parts, with a total"
|
||||
occ background:cron >/dev/null
|
||||
$X php -f /var/www/html/cron.php >/dev/null 2>&1 || true
|
||||
occ backup:status | grep "system cron" >/dev/null && fail "backup:status still warns after cron ran"
|
||||
|
||||
Reference in New Issue
Block a user