Compare commits

..

21 Commits

Author SHA1 Message Date
dillard a8b5e85c9b All folders get their original modification time back on a restore
The backup records the time of the folders with something in them, packed in the meta record,
so an older version still reads the snapshot. A restore through Nextcloud leaves a folder that
was already there alone. The round trip checks what Nextcloud shows, not only the disk.
2026-11-07 01:07:09 +01:00
dillard 36f2427e9a Tell administrators about a location without a schedule
A notification when a schedule is turned off, then a reminder: once a week while another
location is backed up, every day when none is. The round trip now also checks the text of
these notifications and of the one about names, and that one location never gets two runs.
2026-11-07 00:23:28 +01:00
dillard e58b86fc48 A schedule per location, and locations backed up one after the other
Off, once a day or every N hours from a time, each in its own time zone. Existing locations
keep the time that held for all of them. A backup that has to wait shows as queued, and its
restore point has the time it really began.
2026-11-07 00:04:00 +01:00
dillard 193f65940b Pin a restore point and keep a note on it
Both live in the catalog on the location, so a new server sees them; the database holds
a local copy for the lists. The retention policy leaves a pinned restore point alone and
does not count it. Available on the restore page and as occ backup:pin and backup:note.
2026-11-06 23:49:50 +01:00
dillard 6540a9314a A name a snapshot cannot hold gets a notification of its own
Reported apart from unreadable files, with example paths and the advice to rename
them, instead of advice about permissions. Start of the 1.1.0 plan in PLAN.md.
2026-11-06 23:38:40 +01:00
Dillard Blom f81c5e13e9 Merge pull request #30 from dillardblom/1.0.3
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.3
2026-11-06 20:29:09 +01:00
dillard 1b621b7149 Backup state stays valid JSON with a name that is not UTF-8; Bin reads on from an id, not a position 2026-11-06 20:02:09 +01:00
dillard 1736005648 Review of 1.0.3: names a snapshot cannot hold, restore-directory errors under salvage, browse speed
- Leave out and report a file or folder whose name a snapshot cannot hold
- A staging write failure stops a full restore under --salvage
- Browse reuses its repository; empty folders read along with the entries
- Keep leases alive while empty folders are created; exact-prefix empty folder is the target
- Clean up tables that were partly created or before the transaction started
- Bin: no empty message after a read error; longer pages
2026-11-06 19:44:52 +01:00
dillard c7a651f269 Version 1.0.3: changelog, README status and roadmap 2026-11-06 19:05:54 +01:00
dillard 8897d8074f Restore through Nextcloud: also a restored folder that is itself empty gets its time back 2026-11-06 18:40:10 +01:00
dillard 0b9fa1ae92 Restore through Nextcloud: empty folders get their time back 2026-11-06 18:36:15 +01:00
dillard ee406c481e Restore: go on after a partial write; created tables stay once the database is committed 2026-11-06 18:27:35 +01:00
dillard dd2e1af997 Roadmap: original modification time for all folders; guide: folders reported as updated 2026-11-06 18:08:34 +01:00
dillard d96229f370 Release the locks of a killed restore when it runs again; legacy test without empty folders
A killed restore left its leases for up to two hours, so the first backup of the recovered
server was refused. A backup from before 1.0.3 records no empty folders: the test now expects that.
2026-11-06 17:34:33 +01:00
dillard b14b16c66c Empty folders in backup and restore, a full disk stops a restore, progress in the Bin
- Record folders without files in the meta record of a snapshot and restore them (#27)
- Errors writing to the restore directory stop the restore, also with --salvage
- Resume: check the folder before skipping, clean temporaries in folders that are complete
- Clean only read-ahead folders of our own; refresh the lease on skipped files
- Drop tables a failed database restore created; keep --salvage from calling a restored database fresh
- Retry a pack that fails to open on the storage (SMB)
- Without cron, running backups take turns from the open page
- Bin: read in parts with a counter and progress bar, show read errors, refresh when a run ends
2026-11-06 15:50:02 +01:00
dillard ce40053cec Merge remote-tracking branch 'github/main' into 1.0.3 2026-11-06 12:23:09 +01:00
Dillard Blom 46e0186f35 Merge pull request #29 from dillardblom/polish
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Docs: --salvage after a stopped restore; a full disk during a restore
2026-11-06 12:22:58 +01:00
dillard d5eb7abbff Guide: after a full disk, run the restore again as in step 8
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-11-06 12:02:38 +01:00
dillard 4c973d5cd8 Docs: a full disk during a restore looks like a damaged backup, to be fixed in 1.0.3 2026-11-06 12:00:52 +01:00
dillard e9592cb6c6 PLAN: findings from the reviews of 1.0.2, for 1.0.3 2026-11-06 11:57:08 +01:00
dillard b86b07a49c Disaster recovery guide: --salvage after a restore that stopped halfway 2026-11-06 11:56:10 +01:00
57 changed files with 2236 additions and 208 deletions
+59
View File
@@ -11,6 +11,62 @@ Each heading starts with that `<version>`, once: the app store shows a release t
whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
shown under 1.0.0. The git tag of a release candidate goes after the date. shown under 1.0.0. The git tag of a release candidate goes after the date.
## [Unreleased]
### Added
- Pin a restore point, on the restore page or with `occ backup:pin`: the retention policy leaves it
alone, for example the one from before an upgrade. Unpinning notifies all administrators.
- A note on a restore point that you can change afterwards, on the restore page or with
`occ backup:note`. Pins and notes are stored encrypted on the location, so a new server sees them.
An older version of NG Backup can still read the location, but drops them when it writes to it.
- A schedule per location: off, once a day, or every N hours from a given time, on the page under
Backups or with `occ backup:target:schedule`. After the update every location keeps the time that
held for all of them; a new location starts with the schedule of the first one.
- All folders get their original modification time back on a restore, not only the empty ones, from
the first backup made with this version. An older version of NG Backup can still read new restore
points. A restore through Nextcloud into a folder that is already there leaves its time alone.
### Changed
- Locations are backed up one after the other: a backup that has to wait shows as queued. A restore
point has the time its backup really began, not the time it was queued.
- Choosing a time no longer starts a backup right away when that time has already passed today.
- A location without a schedule no longer gets the notification that it has had no recent backup.
Instead, all administrators are told when a schedule is turned off, and reminded once a week while
another location is still backed up, or every day when none is.
- A file or folder that is left out because of its name now gets a notification of its own, in the
log and in `occ backup:run`. It says the item is missing from the backup, gives example paths and
asks to rename it; before, it was reported as unreadable, with advice about permissions.
## [1.0.3] - 2026-11-06
### Fixed
- Empty folders are backed up and restored (#27), in `occ backup:restore`, `occ backup:restore:full`
and the restore page, where they are shown too. They come back from the first backup made with
this version; older restore points do not contain them. An older version of NG Backup can still
read new restore points, and then only leaves the empty folders out.
- A full disk during a restore no longer looks like a damaged backup. The restore now stops with a
message about the disk, also with `--salvage`, and does not point at `--salvage` any more. The same
goes for other trouble writing to the restore directory, such as permissions.
- A file or folder whose name a snapshot cannot hold (a backslash, a control character, not UTF-8) is
left out and reported as unreadable. Before, a backslash or control character could make the whole
restore point unreadable, and a name that is not UTF-8 made the backup fail.
- A restore that goes on after a stop also removes the temporary files in folders where every file
is already in place, and checks that those folders are inside the restore directory before it
leaves their files alone.
- After a restore that was killed and then ran again, a backup no longer has to wait up to two hours
for the locks of the killed run.
- A restore only removes read-ahead folders of its own (`.ngb-restore-spool-` with 8 hex digits).
- A restore keeps its lock on the location during a long row of files that are already in place.
- A database restore that fails no longer leaves behind tables it created, which could stop the
installation of that app later. When only NG Backup's own settings cannot be put back after the
database was restored, `--salvage` no longer says the database is still the fresh one.
- A connection that breaks while opening a pack on the location (seen with SMB) is retried once and
is handled by `--salvage`, instead of stopping the restore with the error of the storage.
- Without a system cron, several running backups take turns while the page is open.
- The Bin tab shows how far it is while it reads the removed restore points, which can take a while.
It also shows when it cannot read a location, instead of an empty bin, and refreshes when a run
ends.
## [1.0.2] - 2026-11-06 ## [1.0.2] - 2026-11-06
### Added ### Added
@@ -55,6 +111,9 @@ shown under 1.0.0. The git tag of a release candidate goes after the date.
### Known issues ### Known issues
- Empty folders are not backed up or restored; to be fixed in 1.0.3 (#27). - Empty folders are not backed up or restored; to be fixed in 1.0.3 (#27).
- A full disk during a restore looks like a damaged backup ("Size mismatch"), and with `--salvage`
the restore goes on and lists the remaining files. The backup itself is fine: make room and run
the restore again. To be fixed in 1.0.3.
## [1.0.1] - 2026-11-01 ## [1.0.1] - 2026-11-01
+52 -5
View File
@@ -438,9 +438,46 @@ in README "Known limitations":
(1–3 vervangen, 4-bin erbij; in `info.xml` bij de release naar tag v1.0.2 laten wijzen en (1–3 vervangen, 4-bin erbij; in `info.xml` bij de release naar tag v1.0.2 laten wijzen en
4-bin toevoegen). Getest: unit (208), psalm, integratie NC35 sqlite/pgsql, S3/SFTP/FTP/SMB, 4-bin toevoegen). Getest: unit (208), psalm, integratie NC35 sqlite/pgsql, S3/SFTP/FTP/SMB,
volledige restore met `--salvage`. **1.0.2 uitgebracht 2026-11-06** (tag `v1.0.2`, PR #28). volledige restore met `--salvage`. **1.0.2 uitgebracht 2026-11-06** (tag `v1.0.2`, PR #28).
- **Naar 1.0.3, uit de reviews van 1.0.2 (achteraf gedraaid, 2026-11-06):**
1. Lege mappen (#27).
2. Een volle schijf onder `--salvage` telt als beschadigde backup en de restore gaat door; zonder
`--salvage` wijst de melding er ten onrechte naar. `fwrite` controleren, schrijffouten stoppen de
restore (`Repository::restoreEntries`; ook `RestoreService::writeFile` nakijken). Beide reviews.
3. Hervatten: controle op de doelmap vóór het overslaan, en tijdelijke bestanden ook opruimen in
een map waar alles al klopt.
4. Alleen `.ngb-restore-spool-<hex>` opruimen.
5. Salvage-test: tijdelijke bestanden recursief zoeken; `admin` en `frtest` apart controleren.
6. Nog nakijken: lease verversen bij overgeslagen bestanden; fout ná de database-commit
(`restoreOwnAppConfig`) en tabellen die na een rollback blijven staan; backups zonder cron
via de pagina (eerlijke beurten, stappen die de webtimeout halen); Bin verversen en fouten
tonen; SMB-uitzondering bij het openen van een pack onder `--salvage`.
Niet doen: bij hervatten de inhoud vergelijken (grootte en mtime is ook het criterium van de backup).
**Stand op branch `1.0.3` (2026-11-06):** punten 1-6 gebouwd. Lege mappen staan in het metarecord
van de snapshot, niet tussen de bestanden: 1.0.2 en ouder lezen zo'n snapshot gewoon en slaan alleen
de lege mappen over. Geen webtimeout-wijziging zonder cron: alleen in de code nagelopen, niet getest
(Nextcloud zet de PHP-tijdslimiet ruim, een gestopte stap wordt overgedaan); wel eerlijke beurten.
Bin leest in delen met teller en balk (besluit 2026-11-06, ook met oog op meerdere backups per
dag). Getest: unit (volle schijf op echte tmpfs), integratie NC35 sqlite/pgsql/mysql, sftp, salvage,
salvage-files; Bin in een echte browser met vertraagde verzoeken (1 van 7 t/m 7). Nog: reviews
vóór de PR, README-regels bij de versiecommit.
**Simulaties (2026-11-06, NC 34.0.4, 2 GB, lokale SFTP, `~/dev-local/ngb-sim/sim.sh`):** normaal
(pgsql), stop en hervat (pgsql → MariaDB, restore met kill -9 gestopt), volle schijf met `--salvage`:
alle drie geslaagd, alle 3.101 gebruikersbestanden en 7 lege mappen gelijk, share, agenda en WebDAV
werken. Vond één fout, opgelost: na een kill bleven vergrendelingen tot 2 uur staan, waardoor de
backup van stap 11 werd geweigerd. `files:scan` "Updated" zijn in de gemeten run alleen mappen
(mtime van mappen wordt niet teruggezet, dus nieuwe mtime en etag; bestanden ongemoeid). "Removed":
o.a. `cache` van een gebruiker, die de backup uitsluit. Waarschijnlijk verklaart dit ook de 50.080
"updated" op nc01, niet op nc01 zelf vastgesteld.
**Naar 1.1.0 of 1.2.0 (besluit 2026-11-06):** alle mappen krijgen bij een restore hun oorspronkelijke
tijd terug. Hoort zo, vooral bij oudere verzamelingen: de maptijd wordt gebruikt om in te schatten
wanneer een map is aangemaakt. De backup moet dan de tijd van alle mappen vastleggen en de restore
die aan het eind terugzetten; werkt pas vanaf de eerste backup met die versie. Op de roadmap onder 1.1.0.
**1.0.3 uitgebracht (2026-11-06)**, ook op nc01 geïnstalleerd.
- **Branches (besluit 2026-11-06):** werk voor een versie op een branch met die naam (`1.0.3`); - **Branches (besluit 2026-11-06):** werk voor een versie op een branch met die naam (`1.0.3`);
`polish` alleen voor kleine cosmetische zaken, per geval bekijken of en wanneer die gemerged `polish` alleen voor kleine cosmetische zaken, per geval bekijken of en wanneer die gemerged
worden. SMB staat uit op sub5 en sub6; sub6 is het testaccount, eerst aanzetten voor een SMB-test. worden. SMB staat uit op sub5 en sub6; sub6 is het testaccount, eerst aanzetten voor een SMB-test.
Na 1.0.3 gaat het werk naar `1.1.0`. Een spoedfix die niet kan wachten gaat op `1.0.4` vanaf `main`,
krijgt een eigen merge en release; daarna `main` in `1.1.0` mergen, zodat daar met de fix getest wordt.
- **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar - **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november, 1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij
@@ -535,7 +572,18 @@ in README "Known limitations":
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3, - **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan 4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
een paar bestanden. een paar bestanden.
- **1.1.0:** - **1.1.0** (geen harde datum: een week was een voorzet, langer mag; de releasefrequentie mag omlaag,
besluit 2026-11-06):
**Betere melding voor namen die een snapshot niet kan bevatten** (backslash, stuurteken, geen UTF-8):
een eigen melding in plaats van "unreadable" met het advies over rechten. Zegt dat deze bestanden of
mappen in de backup ontbreken, geeft voorbeelden van de paden en vraagt ze te hernoemen; de volgende
backup neemt ze mee. Alleen als Nextcloud-melding: de app mailt niet zelf, of meldingen gemaild worden
is de keuze van de beheerder (besluit 2026-11-06).
**Stand op branch `1.1.0` (2026-11-07):** klaar: betere melding, punten 4, 5, 6 en 8. Pin en notitie staan
in de catalogus, met een kopie in de database voor de lijsten. Een moment van vóór het instellen van
een schema start niets. Geen schema: geen melding "geen recente backup", wel een melding aan alle
beheerders bij het uitzetten en een herinnering, wekelijks zolang een andere locatie een schema heeft,
anders dagelijks (besluit 2026-11-07). Nog niet getest: de update vanaf 1.0.3, de pagina in een browser.
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit 1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
`target:add` en sleutelimport (het gevoeligste pad), daarom niet in rc1. Daarna de `target:add` en sleutelimport (het gevoeligste pad), daarom niet in rc1. Daarna de
@@ -567,11 +615,10 @@ in README "Known limitations":
nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met
`occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de `occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de
opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly. opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly.
9. **Feedbackvraag weghalen** in 1.1.0 of 1.2.0 (besluit 2026-10-27): de regel "We would like 9. **Feedbackvraag weghalen in 1.2.0** (besluit 2026-11-07, niet in 1.1.0): de regel "We would like
to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot
dan staan (de vermelding "in beta" is er in 1.0.1 al af). Zijn er tegen die dan staan. Er komt weinig feedback bij zo'n 150 downloads per versie; geen bericht is goed
tijd geen noemenswaardige meldingen, dan werkt het: mensen melden eerder wat niet werkt dan bericht. Het formulier (`experience.yml`) in de issues blijft.
wat gewoon werkt. Het formulier (`experience.yml`) mag blijven voor wie het zelf opzoekt.
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met - **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
PostgreSQL 18)** — voor **1.0.0-rc.2**: PostgreSQL 18)** — voor **1.0.0-rc.2**:
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE, 1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
+10 -8
View File
@@ -7,7 +7,7 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
see "Storage backend experiences" below — that requirement comes from files_external, not from see "Storage backend experiences" below — that requirement comes from files_external, not from
ng_backup, and applies to any Nextcloud app that writes to an SMB share.) ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
**Status: 1.0.2, stable.** Disaster recovery has been tested on real installations (see "Tested **Status: 1.0.3, stable.** Disaster recovery has been tested on real installations (see "Tested
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See `docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
`docs/DISASTER-RECOVERY.md`. `docs/DISASTER-RECOVERY.md`.
@@ -169,6 +169,7 @@ Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECO
|---|---|---|---| |---|---|---|---|
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical | | Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
| The same, 1.0.2 | Hetzner Storage Box (SMB), Germany → Canada | PostgreSQL → PostgreSQL | 150,755 files restored in about 29 h (7-13 MiB/s), going on after a lost connection; sample identical, empty folders not restored ([#27](https://github.com/dillardblom/ng_backup/issues/27)) | | The same, 1.0.2 | Hetzner Storage Box (SMB), Germany → Canada | PostgreSQL → PostgreSQL | 150,755 files restored in about 29 h (7-13 MiB/s), going on after a lost connection; sample identical, empty folders not restored ([#27](https://github.com/dillardblom/ng_backup/issues/27)) |
| 1.0.3, small simulations of 2 GB: as planned, with the restore killed halfway, and with a disk that runs full | Local SFTP server | PostgreSQL → PostgreSQL and MariaDB | all 3,101 user files and all empty folders identical; users, a share and a calendar back |
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical | | 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical | | The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) | | The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
@@ -265,15 +266,12 @@ What is planned, without dates. Ideas and votes are welcome in the
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here, [issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
a 👍 on its issue helps decide what comes first. a 👍 on its issue helps decide what comes first.
**1.0.3**
- Empty folders in the backup and the restore
([#27](https://github.com/dillardblom/ng_backup/issues/27)).
**1.1.0** **1.1.0**
- Restore without the recovery kit: the backup key from the location itself, opened with one of - Restore without the recovery kit: the backup key from the location itself, opened with one of
your passphrases. your passphrases.
- A schedule per location, also "every N hours". - A schedule per location, also "every N hours".
- Locations backed up one after the other instead of at the same time. - Locations backed up one after the other instead of at the same time.
- All folders get their original modification time back on a restore, not only the empty ones.
- A note on a restore point that you can change afterwards. - A note on a restore point that you can change afterwards.
- Pinning a restore point, so the retention policy never removes it (for example the one from - Pinning a restore point, so the retention policy never removes it (for example the one from
before an upgrade). before an upgrade).
@@ -293,9 +291,11 @@ a 👍 on its issue helps decide what comes first.
On the backlog; know them before you rely on NG Backup: On the backlog; know them before you rely on NG Backup:
- **Empty folders do not come back.** A backup only records files, so a folder without files is - **Empty folders only come back from backups made with 1.0.3 or newer.** Older restore points do
not restored. To be fixed in 1.0.3 not contain them.
([#27](https://github.com/dillardblom/ng_backup/issues/27)). - **Folders with files in them only get their original modification time back from backups made with
1.1.0 or newer.** From older restore points they show the time of the restore; the files keep theirs.
A restore through Nextcloud into a folder that is already there leaves that folder's time alone.
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in - **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the `config.php`) is not supported: a backup refuses to start there. Folders mounted through the
External storage app are not in the data directory and are **not included**, without a warning. External storage app are not in the data directory and are **not included**, without a warning.
@@ -330,10 +330,12 @@ On the backlog; know them before you rely on NG Backup:
| `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation | | `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation |
| `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots | | `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots |
| `backup:target:backends/add/edit/list/remove/append-only` | backup locations | | `backup:target:backends/add/edit/list/remove/append-only` | backup locations |
| `backup:target:schedule` | when a location is backed up: off, daily, or every N hours |
| `backup:run`, `backup:status`, `backup:list` | back up, status (runs of the last 7 days; `--all` for all), restore points | | `backup:run`, `backup:status`, `backup:list` | back up, status (runs of the last 7 days; `--all` for all), restore points |
| `backup:cancel <run>` | stop a running backup | | `backup:cancel <run>` | stop a running backup |
| `backup:browse`, `backup:restore` | browse and restore | | `backup:browse`, `backup:restore` | browse and restore |
| `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash | | `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash |
| `backup:pin`, `backup:note` | keep a restore point, a note you can change later |
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) | | `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) | | `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
| `backup:key:import` | import a recovery kit on a fresh installation | | `backup:key:import` | import a recovery kit on a fresh installation |
+8 -5
View File
@@ -45,7 +45,7 @@ welcome: use the "Share your experience" link on the admin page.
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
you need help with restoring a backup after a crash.]]></description> you need help with restoring a backup after a crash.]]></description>
<version>1.0.2</version> <version>1.0.3</version>
<licence>agpl</licence> <licence>agpl</licence>
<author mail="dillard.blom@ensembia.com">Dillard Blom</author> <author mail="dillard.blom@ensembia.com">Dillard Blom</author>
<namespace>NgBackup</namespace> <namespace>NgBackup</namespace>
@@ -56,10 +56,10 @@ you need help with restoring a backup after a crash.]]></description>
<website>https://github.com/dillardblom/ng_backup</website> <website>https://github.com/dillardblom/ng_backup</website>
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs> <bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
<repository type="git">https://github.com/dillardblom/ng_backup</repository> <repository type="git">https://github.com/dillardblom/ng_backup</repository>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/1-backups.png</screenshot> <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/2-restore.png</screenshot> <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/3-settings.png</screenshot> <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/3-settings.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/4-bin.png</screenshot> <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/4-bin.png</screenshot>
<dependencies> <dependencies>
<php min-version="8.1"/> <php min-version="8.1"/>
<lib>sodium</lib> <lib>sodium</lib>
@@ -83,11 +83,14 @@ you need help with restoring a backup after a crash.]]></description>
<command>OCA\NgBackup\Command\ListSnapshots</command> <command>OCA\NgBackup\Command\ListSnapshots</command>
<command>OCA\NgBackup\Command\Browse</command> <command>OCA\NgBackup\Command\Browse</command>
<command>OCA\NgBackup\Command\Restore</command> <command>OCA\NgBackup\Command\Restore</command>
<command>OCA\NgBackup\Command\Pin</command>
<command>OCA\NgBackup\Command\Note</command>
<command>OCA\NgBackup\Command\Prune</command> <command>OCA\NgBackup\Command\Prune</command>
<command>OCA\NgBackup\Command\Retention</command> <command>OCA\NgBackup\Command\Retention</command>
<command>OCA\NgBackup\Command\TargetAppendOnly</command> <command>OCA\NgBackup\Command\TargetAppendOnly</command>
<command>OCA\NgBackup\Command\TargetTrustCurrent</command> <command>OCA\NgBackup\Command\TargetTrustCurrent</command>
<command>OCA\NgBackup\Command\TargetLimit</command> <command>OCA\NgBackup\Command\TargetLimit</command>
<command>OCA\NgBackup\Command\TargetSchedule</command>
<command>OCA\NgBackup\Command\Trash</command> <command>OCA\NgBackup\Command\Trash</command>
<command>OCA\NgBackup\Command\TrashRestore</command> <command>OCA\NgBackup\Command\TrashRestore</command>
<command>OCA\NgBackup\Command\KeySlots</command> <command>OCA\NgBackup\Command\KeySlots</command>
+3 -1
View File
@@ -25,8 +25,10 @@ return [
['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'], ['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'],
['name' => 'AdminApi#startBackup', 'url' => '/api/backups', 'verb' => 'POST'], ['name' => 'AdminApi#startBackup', 'url' => '/api/backups', 'verb' => 'POST'],
['name' => 'AdminApi#tick', 'url' => '/api/tick', 'verb' => 'POST'], ['name' => 'AdminApi#tick', 'url' => '/api/tick', 'verb' => 'POST'],
['name' => 'AdminApi#setSchedule', 'url' => '/api/schedule', 'verb' => 'PUT'], ['name' => 'AdminApi#setTargetSchedule', 'url' => '/api/targets/{id}/schedule', 'verb' => 'PUT'],
['name' => 'AdminApi#snapshots', 'url' => '/api/targets/{targetId}/snapshots', 'verb' => 'GET'], ['name' => 'AdminApi#snapshots', 'url' => '/api/targets/{targetId}/snapshots', 'verb' => 'GET'],
['name' => 'AdminApi#setSnapshotNote', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/note', 'verb' => 'PUT'],
['name' => 'AdminApi#setSnapshotPinned', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/pin', 'verb' => 'PUT'],
['name' => 'AdminApi#browse', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/browse', 'verb' => 'GET'], ['name' => 'AdminApi#browse', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/browse', 'verb' => 'GET'],
['name' => 'AdminApi#startRestore', 'url' => '/api/restores', 'verb' => 'POST'], ['name' => 'AdminApi#startRestore', 'url' => '/api/restores', 'verb' => 'POST'],
], ],
+17 -3
View File
@@ -117,7 +117,18 @@ first. If none is better, `--salvage` restores whatever can be read:
occ backup:restore:full offsite <snapshot> --install-apps --salvage occ backup:restore:full offsite <snapshot> --install-apps --salvage
If a restore already stopped halfway, run it as in step 8: maintenance mode off, with `--force` and
without `--install-apps`:
occ maintenance:mode --off && occ backup:restore:full offsite <snapshot> --salvage --force
- Files that cannot be read are left out and listed at the end; everything else is restored. - Files that cannot be read are left out and listed at the end; everything else is restored.
- A full disk is not damage: the restore stops, also with `--salvage`, and says so. Make room and
run the restore again as in step 8.
- **On version 1.0.2, check the disk first if the reason is "Size mismatch".** There a full disk
looks like damaged data: the message points at `--salvage`, and with `--salvage` the restore goes
on and lists the remaining files. The backup itself is fine. Update NG Backup, make room and run
the restore again as in step 8.
- If the database cannot be restored (its dump is damaged, or it does not fit this - If the database cannot be restored (its dump is damaged, or it does not fit this
installation), the database stays as the fresh installation made it, and the data directory installation), the database stays as the fresh installation made it, and the data directory
and config of the backup go to a folder of their own in the data directory, and config of the backup go to a folder of their own in the data directory,
@@ -140,9 +151,12 @@ The command then ends with exit code 2 instead of 0. The same option exists for
- From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`, - From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`,
`redis`), which point at AIO's containers. `redis`), which point at AIO's containers.
- Log in, open a few files, a calendar and a contact. - Log in, open a few files, a calendar and a contact.
- Empty folders do not come back: a backup only records files (to be fixed in 1.0.3, see - `occ files:scan` can report many folders as "Updated". That is expected: a restore does not yet put
[issue #27](https://github.com/dillardblom/ng_backup/issues/27)). `occ files:scan` then removes back the modification time of folders with files in them, so they show the time of the restore.
them from the file cache, so it reports them as removed. The files themselves keep their time and content.
- Empty folders only come back from backups made with version 1.0.3 or newer. With an older
restore point they are gone: `occ files:scan` then removes them from the file cache, so it reports
them as removed.
- Start background jobs again (cron), as in step 2 in reverse. - Start background jobs again (cron), as in step 2 in reverse.
**10. Point your domain at the new server** and check that clients reconnect. **10. Point your domain at the new server** and check that clients reconnect.
+8 -1
View File
@@ -79,7 +79,9 @@ occ backup:list
Later runs upload only changed data. An unchanged run uploads (almost) nothing; the output shows Later runs upload only changed data. An unchanged run uploads (almost) nothing; the output shows
the number of uploaded bytes, so check that value when you test incremental behaviour. the number of uploaded bytes, so check that value when you test incremental behaviour.
Schedule `occ backup:run` from system cron (or use the Administration settings, if you prefer). Set when each location is backed up in the Administration settings, or with
`occ backup:target:schedule offsite 02:00` (once a day), `02:00/6` (every 6 hours from 02:00) or `off`.
Locations are backed up one after the other.
## 4. Check the backup regularly ## 4. Check the backup regularly
@@ -173,8 +175,13 @@ occ backup:prune offsite --dry-run # show first
occ backup:prune offsite # apply the policy; forgotten snapshots go to the trash first occ backup:prune offsite # apply the policy; forgotten snapshots go to the trash first
occ backup:trash # list the trash occ backup:trash # list the trash
occ backup:trash:restore offsite <id> occ backup:trash:restore offsite <id>
occ backup:pin offsite <id> # keep this restore point; --unpin to undo
occ backup:note offsite <id> "text" # a note you can change later; without text it is removed
``` ```
The retention policy leaves a pinned restore point alone, and a pinned one does not take the place of
a daily, weekly or monthly one. Pin and note are stored on the location, so a new server sees them too.
Snapshots are kept according to the policy, and their data is deleted only after the deletion delay Snapshots are kept according to the policy, and their data is deleted only after the deletion delay
(set with `backup:key:init --delete-delay-days`). `occ backup:prune offsite --dry-run` shows what would be (set with `backup:key:init --delete-delay-days`). `occ backup:prune offsite --dry-run` shows what would be
removed without changing anything. removed without changing anything.
+6
View File
@@ -124,7 +124,13 @@ final class StorageBackend implements IBackend, IReadsAhead {
// null: not supported or failing for now; false: not opened, which need not mean // null: not supported or failing for now; false: not opened, which need not mean
// missing. Either way the storage's own stream decides. // missing. Either way the storage's own stream decides.
} }
try {
$fh = $this->storage->fopen($this->abs($path), 'r'); $fh = $this->storage->fopen($this->abs($path), 'r');
} catch (\Exception $e) {
// Some storages throw for what is not a missing file (SMB: a connection that broke).
// As a BackendException, the caller reads once more and --salvage can go on.
throw new BackendException('Cannot read ' . $path . ': ' . $e->getMessage(), 0, $e);
}
if ($fh === false) { if ($fh === false) {
throw new BackendException('Not found: ' . $path); throw new BackendException('Not found: ' . $path);
} }
+15 -16
View File
@@ -43,25 +43,27 @@ final class BackupJob extends TimedJob {
$this->setTimeSensitivity(IJob::TIME_SENSITIVE); $this->setTimeSensitivity(IJob::TIME_SENSITIVE);
} }
private const MAX_DAILY_ATTEMPTS = 3;
protected function run($argument): void { protected function run($argument): void {
$budget = PHP_SAPI === 'cli' ? 240.0 : 20.0; $budget = PHP_SAPI === 'cli' ? 240.0 : 20.0;
$deadline = microtime(true) + $budget; $deadline = microtime(true) + $budget;
$this->startScheduled(); $this->startScheduled();
foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) { // One after the other: runs that read the same disks at the same time are not faster together.
foreach ($this->backups->runsInTurn() as $run) {
while (microtime(true) < $deadline && $run->getStatus() === 'running') { while (microtime(true) < $deadline && $run->getStatus() === 'running') {
$next = $this->backups->step($run, min($deadline, microtime(true) + $budget)); $next = $this->backups->step($run, min($deadline, microtime(true) + $budget));
if ($next === $run) { if ($next === $run) {
break; // step() hands back the same object when another process holds the run's lease break 2; // step() hands back the same object when another process holds the run's lease
} }
$run = $next; $run = $next;
} }
} }
$this->failStaleRestores(); $this->failStaleRestores();
$this->weeklyPrune(); $this->weeklyPrune();
foreach ($this->targets->list() as $t) { $targets = $this->targets->list();
foreach ($targets as $t) {
$this->alerts->checkStale($t); $this->alerts->checkStale($t);
$othersActive = array_filter($targets, static fn ($o) => $o->getId() !== $t->getId() && (string)$o->getSchedule() !== '') !== [];
$this->alerts->checkOff($t, $othersActive);
} }
} }
@@ -102,21 +104,18 @@ final class BackupJob extends TimedJob {
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due); $this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
} }
/** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */ /** Start the backups the schedules of the locations ask for; they run in turn (see LocationSchedule). */
private function startScheduled(): void { private function startScheduled(): void {
$todayAt = $this->schedule->todayAt(); $now = time();
if ($todayAt === null || time() < $todayAt) {
return;
}
foreach ($this->targets->list() as $t) { foreach ($this->targets->list() as $t) {
$last = $this->runs->lastFinished($t->getId(), BackupService::KIND_FULL); $moment = \OCA\NgBackup\Service\LocationSchedule::lastMoment((string)$t->getSchedule(), $this->schedule->zoneOr($t->getScheduleZone()), $now);
if ($this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== [] || ($last !== null && $last->getStartedAt() >= $todayAt)) { if ($moment === null || $this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== []) {
continue; continue;
} }
// After a failure: retry at most MAX_DAILY_ATTEMPTS times a day, an hour apart, instead $attempts = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $moment);
// of starting (and notifying about) a new full run every five minutes all day. if (!\OCA\NgBackup\Service\LocationSchedule::isDue($moment, (int)$t->getScheduleSince(),
$today = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $todayAt); $this->runs->lastFinished($t->getId(), BackupService::KIND_FULL)?->getStartedAt(),
if (count($today) >= self::MAX_DAILY_ATTEMPTS || ($today !== [] && time() - (int)$today[0]->getFinishedAt() < 3600)) { count($attempts), ($attempts[0] ?? null)?->getFinishedAt(), $now)) {
continue; continue;
} }
try { try {
+2 -2
View File
@@ -40,14 +40,14 @@ final class ListSnapshots extends Command {
foreach ($targets as $t) { foreach ($targets as $t) {
foreach ($this->snapshots->findForTarget($t->getId()) as $s) { foreach ($this->snapshots->findForTarget($t->getId()) as $s) {
$rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(), $rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(),
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getLabel() ?? '']; $s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getPinned() ? 'yes' : '', $s->getLabel() ?? '', $s->getNote() ?? ''];
} }
} }
if ($rows === []) { if ($rows === []) {
$output->writeln('No snapshots yet (occ backup:run).'); $output->writeln('No snapshots yet (occ backup:run).');
return 0; return 0;
} }
(new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'label'])->setRows($rows)->render(); (new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'pinned', 'label', 'note'])->setRows($rows)->render();
return 0; return 0;
} }
} }
+44
View File
@@ -0,0 +1,44 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\PruneService;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
final class Note extends Command {
public function __construct(
private TargetService $targets,
private PruneService $prune,
) {
parent::__construct();
}
protected function configure(): void {
$this->setName('backup:note')
->setDescription('Set or remove the note on a snapshot')
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)')
->addArgument('note', InputArgument::OPTIONAL, 'The note, at most ' . PruneService::NOTE_MAX . ' characters; without it the note is removed', '');
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$note = (string)$input->getArgument('note');
try {
$this->prune->setNote($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), $note);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
$output->writeln(trim($note) === '' ? 'Note removed.' : 'Note saved.');
return 0;
}
}
+45
View File
@@ -0,0 +1,45 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\PruneService;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
final class Pin extends Command {
public function __construct(
private TargetService $targets,
private PruneService $prune,
) {
parent::__construct();
}
protected function configure(): void {
$this->setName('backup:pin')
->setDescription('Keep a snapshot: the retention policy does not remove a pinned one')
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)')
->addOption('unpin', null, InputOption::VALUE_NONE, 'Leave the snapshot to the retention policy again');
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$unpin = (bool)$input->getOption('unpin');
try {
$this->prune->setPinned($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), !$unpin);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
$output->writeln($unpin ? 'Snapshot unpinned; the retention policy applies to it again.' : 'Snapshot pinned; the retention policy leaves it alone.');
return 0;
}
}
+3
View File
@@ -79,6 +79,9 @@ HELP);
if (($f['unreadable'] ?? 0) > 0) { if (($f['unreadable'] ?? 0) > 0) {
$output->writeln(sprintf('<comment> warning: %d unreadable files or folders are missing from this backup (see the server log)</comment>', $f['unreadable'])); $output->writeln(sprintf('<comment> warning: %d unreadable files or folders are missing from this backup (see the server log)</comment>', $f['unreadable']));
} }
if (($f['badNames'] ?? 0) > 0) {
$output->writeln(sprintf('<comment> warning: %d files or folders are missing from this backup because their name is not supported; rename them and the next backup includes them (see the server log)</comment>', $f['badNames']));
}
if (($f['changing'] ?? 0) > 0) { if (($f['changing'] ?? 0) > 0) {
$output->writeln(sprintf('<comment> warning: %d files kept changing and were saved as read (see the server log)</comment>', $f['changing'])); $output->writeln(sprintf('<comment> warning: %d files kept changing and were saved as read (see the server log)</comment>', $f['changing']));
} }
+4 -1
View File
@@ -31,6 +31,7 @@ final class Status extends Command {
private VerifyService $verify, private VerifyService $verify,
private Schedule $schedule, private Schedule $schedule,
private CronHealth $cron, private CronHealth $cron,
private \OCA\NgBackup\Service\BackupService $backups,
) { ) {
parent::__construct(); parent::__construct();
} }
@@ -61,9 +62,11 @@ final class Status extends Command {
$rows = []; $rows = [];
$all = (bool)$input->getOption('all'); $all = (bool)$input->getOption('all');
$shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400); $shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400);
$queued = array_map(static fn (\OCA\NgBackup\Db\Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1));
foreach ($shown as $r) { foreach ($shown as $r) {
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? []; $s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus(), $rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(),
in_array($r->getId(), $queued, true) ? 'queued' : ($r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus()),
Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '', Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '', $s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')]; isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
+2 -1
View File
@@ -31,7 +31,8 @@ final class TargetList extends Base {
$rows = []; $rows = [];
foreach ($this->targets->list() as $t) { foreach ($this->targets->list() as $t) {
$row = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(), $row = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt())]; 'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt()),
'schedule' => (string)$t->getSchedule() !== '' ? $t->getSchedule() : 'off'];
if ($input->getOption('test')) { if ($input->getOption('test')) {
try { try {
$repo = $this->targets->repository($t); $repo = $this->targets->repository($t);
+50
View File
@@ -0,0 +1,50 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
final class TargetSchedule extends Command {
public function __construct(
private TargetService $targets,
private Schedule $schedule,
) {
parent::__construct();
}
protected function configure(): void {
$this->setName('backup:target:schedule')
->setDescription('Show or set when a location is backed up')
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('schedule', InputArgument::OPTIONAL, '"off", HH:MM for once a day, or HH:MM/N for every N hours from HH:MM; omit to show the current one')
->addOption('timezone', null, InputOption::VALUE_REQUIRED, 'Time zone of the time, e.g. Europe/Amsterdam');
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$t = $this->targets->get($input->getArgument('target'));
$schedule = $input->getArgument('schedule');
$zone = $input->getOption('timezone') ?? $this->schedule->zoneOr($t->getScheduleZone());
if ($schedule !== null) {
try {
$t = $this->targets->setSchedule($t, $schedule, (string)$zone);
} catch (\InvalidArgumentException $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
}
$current = (string)$t->getSchedule();
$output->writeln($t->getName() . ': ' . ($current === '' ? 'off' : $current . ' (' . $zone . ')'));
return 0;
}
}
+42 -16
View File
@@ -59,10 +59,12 @@ final class AdminApiController extends Controller {
foreach ($this->targets->list() as $t) { foreach ($this->targets->list() as $t) {
$targets[] = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(), $targets[] = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
'repositoryId' => $t->getRepositoryId(), 'createdAt' => $t->getCreatedAt(), 'appendOnly' => (bool)$t->getAppendOnly(), 'repositoryId' => $t->getRepositoryId(), 'createdAt' => $t->getCreatedAt(), 'appendOnly' => (bool)$t->getAppendOnly(),
'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t)]; 'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t),
'schedule' => (string)$t->getSchedule(), 'scheduleZone' => $this->schedule->zoneOr($t->getScheduleZone())];
} }
$shown = $this->runs->findShown($allRuns ? null : time() - \OCA\NgBackup\Command\Status::RECENT_DAYS * 86400); $shown = $this->runs->findShown($allRuns ? null : time() - \OCA\NgBackup\Command\Status::RECENT_DAYS * 86400);
$runs = array_map(fn (Run $r) => $this->runJson($r), $shown); $queued = array_flip(array_map(static fn (Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1)));
$runs = array_map(fn (Run $r) => $this->runJson($r) + ['queued' => isset($queued[$r->getId()])], $shown);
return new JSONResponse([ return new JSONResponse([
'key' => ['initialized' => $this->keys->isInitialized(), 'fingerprint' => $this->keys->fingerprint(), 'key' => ['initialized' => $this->keys->isInitialized(), 'fingerprint' => $this->keys->fingerprint(),
'confirmation' => $this->keys->recoveryKitConfirmation(), 'everConfirmed' => $this->keys->isInitialized() && $this->keys->everConfirmed(), 'confirmation' => $this->keys->recoveryKitConfirmation(), 'everConfirmed' => $this->keys->isInitialized() && $this->keys->everConfirmed(),
@@ -74,14 +76,14 @@ final class AdminApiController extends Controller {
'runs' => $runs, 'runs' => $runs,
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown), 'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS, 'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
'schedule' => $this->schedule->get()['time'],
'scheduleTimezone' => $this->schedule->get()['timezone'],
'cron' => $this->cron->check(), 'cron' => $this->cron->check(),
]); ]);
} }
/** Seconds of backup work one tick() does: well within a normal web request. */ /** Seconds of backup work one tick() does: well within a normal web request. */
public const TICK_SECONDS = 10.0; public const TICK_SECONDS = 10.0;
/** Seconds one request for the Bin reads, before the page shows how far it is and asks for more. */
private const TRASH_PAGE_SECONDS = 5.0;
/** /**
* Move running backups forward from the open admin page, when background jobs do not run on a * Move running backups forward from the open admin page, when background jobs do not run on a
@@ -95,13 +97,16 @@ final class AdminApiController extends Controller {
} }
$deadline = microtime(true) + self::TICK_SECONDS; $deadline = microtime(true) + self::TICK_SECONDS;
$stepped = 0; $stepped = 0;
foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) { foreach ($this->backups->runsInTurn() as $run) {
if (microtime(true) >= $deadline) { if (microtime(true) >= $deadline) {
break; break;
} }
// step() returns at once when the background job (or another tab) holds the run. // step() returns at once when the background job (or another tab) holds the run.
$this->backups->step($run, $deadline); $next = $this->backups->step($run, $deadline);
$stepped++; $stepped++;
if ($next->getStatus() === Run::RUNNING) {
break; // not finished: the next one waits for its turn
}
} }
return new JSONResponse(['stepped' => $stepped]); return new JSONResponse(['stepped' => $stepped]);
} }
@@ -174,15 +179,18 @@ final class AdminApiController extends Controller {
} }
} }
public function trash(int $targetId): JSONResponse { public function trash(int $targetId, string $after = ''): JSONResponse {
try { try {
$delay = $this->keys->deleteDelayDays() * 86400; $delay = $this->keys->deleteDelayDays() * 86400;
$repo = $this->targets->repository($this->targets->get((string)$targetId));
// In parts, so that the page can show how far it is (a read per removed restore point).
$page = $repo->trashPage($after, microtime(true) + self::TRASH_PAGE_SECONDS);
$out = []; $out = [];
foreach ($this->targets->repository($this->targets->get((string)$targetId))->trash() as $id => $info) { foreach ($page['items'] as $id => $info) {
$out[] = ['id' => $id, 'label' => $info['label'], 'time' => $info['time'], 'removedAt' => $info['forgottenAt'], $out[] = ['id' => $id, 'label' => $info['label'], 'time' => $info['time'], 'removedAt' => $info['forgottenAt'],
'by' => $info['by'], 'deleteAfter' => $info['forgottenAt'] + $delay]; 'by' => $info['by'], 'deleteAfter' => $info['forgottenAt'] + $delay];
} }
return new JSONResponse($out); return new JSONResponse(['total' => $page['total'], 'items' => $out]);
} catch (\Throwable $e) { } catch (\Throwable $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST); return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
} }
@@ -270,25 +278,43 @@ final class AdminApiController extends Controller {
return new JSONResponse($this->runJson($run)); return new JSONResponse($this->runJson($run));
} }
/** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */ /** @param string $timezone the administrator's zone (the browser's); empty: the default one */
public function setSchedule(string $time, string $timezone = ''): JSONResponse { public function setTargetSchedule(int $id, string $schedule, string $timezone = ''): JSONResponse {
try { try {
$this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']); $t = $this->targets->setSchedule($this->targets->get((string)$id), $schedule, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
} catch (\InvalidArgumentException $e) { } catch (\Throwable $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST); return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
} }
return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]); return new JSONResponse(['schedule' => $t->getSchedule(), 'scheduleZone' => $t->getScheduleZone()]);
} }
public function snapshots(int $targetId): JSONResponse { public function snapshots(int $targetId): JSONResponse {
$out = []; $out = [];
foreach ($this->snapshots->findForTarget($targetId) as $s) { foreach ($this->snapshots->findForTarget($targetId) as $s) {
$out[] = ['id' => $s->getSnapshotId(), 'kind' => $s->getKind(), 'label' => $s->getLabel(), 'createdAt' => $s->getCreatedAt(), $out[] = ['id' => $s->getSnapshotId(), 'kind' => $s->getKind(), 'label' => $s->getLabel(), 'createdAt' => $s->getCreatedAt(),
'files' => $s->getFiles(), 'bytes' => $s->getBytes()]; 'files' => $s->getFiles(), 'bytes' => $s->getBytes(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()];
} }
return new JSONResponse($out); return new JSONResponse($out);
} }
public function setSnapshotNote(int $targetId, string $snapshotId, string $note = ''): JSONResponse {
try {
$this->prune->setNote($this->targets->get((string)$targetId), $snapshotId, $note);
} catch (\Throwable $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
}
return new JSONResponse(['ok' => true]);
}
public function setSnapshotPinned(int $targetId, string $snapshotId, bool $pinned = true): JSONResponse {
try {
$this->prune->setPinned($this->targets->get((string)$targetId), $snapshotId, $pinned);
} catch (\Throwable $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
}
return new JSONResponse(['ok' => true]);
}
public function browse(int $targetId, string $snapshotId, string $path = ''): JSONResponse { public function browse(int $targetId, string $snapshotId, string $path = ''): JSONResponse {
try { try {
return new JSONResponse($this->restore->browse($this->targets->get((string)$targetId), $snapshotId, $path)); return new JSONResponse($this->restore->browse($this->targets->get((string)$targetId), $snapshotId, $path));
@@ -333,7 +359,7 @@ final class AdminApiController extends Controller {
if ($r->getSnapshotId() !== null) { if ($r->getSnapshotId() !== null) {
try { try {
$s = $this->snapshots->findOne($r->getTargetId(), $r->getSnapshotId()); $s = $this->snapshots->findOne($r->getTargetId(), $r->getSnapshotId());
$snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel()]; $snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()];
} catch (\Throwable) { } catch (\Throwable) {
// Snapshot already pruned or trashed: the run keeps its id, just not the date. // Snapshot already pruned or trashed: the run keeps its id, just not the date.
} }
+41 -13
View File
@@ -157,17 +157,27 @@ final class DbDumper {
} }
// Tables the backup knows the structure of are created first (DDL is not transactional // Tables the backup knows the structure of are created first (DDL is not transactional
// everywhere); tables it does not know are left out, as asked. // everywhere); tables it does not know are left out, as asked.
try {
foreach ($missing as $table) { foreach ($missing as $table) {
if (isset($manifest['tables'][$table]['schema'])) { if (isset($manifest['tables'][$table]['schema'])) {
$report['created'][] = $table; // first: creating it can stop halfway (its indexes)
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions())); $this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
$report['created'][] = $table;
} else { } else {
$report['missing'][] = $table; $report['missing'][] = $table;
} }
} }
} catch (\Throwable $e) {
$this->dropCreatedTables($report);
throw $e;
}
$total = 0; $total = 0;
$foreignKeys = []; $foreignKeys = [];
try {
$this->db->beginTransaction(); $this->db->beginTransaction();
} catch (\Throwable $e) {
$this->dropCreatedTables($report);
throw $e;
}
try { try {
$foreignKeys = $this->suspendForeignKeys(); // first: on SQLite it also takes the write lock $foreignKeys = $this->suspendForeignKeys(); // first: on SQLite it also takes the write lock
foreach ($manifest['tables'] as $table => $info) { foreach ($manifest['tables'] as $table => $info) {
@@ -220,9 +230,19 @@ final class DbDumper {
$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped'])); $this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
$this->resumeForeignKeys($foreignKeys); $this->resumeForeignKeys($foreignKeys);
$this->db->commit(); $this->db->commit();
// Foreign keys of the tables created above, now their rows are in. Not inside the } catch (\Throwable $e) {
// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is $this->db->rollBack();
// restored either way. // PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session
// switch must be turned on again by hand.
if (in_array('SET FOREIGN_KEY_CHECKS = 1', $foreignKeys, true)) {
$this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 1');
}
$this->dropCreatedTables($report);
throw $e;
}
// Committed. Foreign keys of the tables created above, now their rows are in. Not inside the
// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is restored
// either way, and the created tables stay.
foreach ($report['created'] as $table) { foreach ($report['created'] as $table) {
$structure = $manifest['tables'][$table]['schema'] ?? []; $structure = $manifest['tables'][$table]['schema'] ?? [];
foreach ($structure['foreignKeys'] ?? [] as $fk) { foreach ($structure['foreignKeys'] ?? [] as $fk) {
@@ -235,18 +255,26 @@ final class DbDumper {
} }
} }
} }
} catch (\Throwable $e) {
$this->db->rollBack();
// PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session
// switch must be turned on again by hand.
if (in_array('SET FOREIGN_KEY_CHECKS = 1', $foreignKeys, true)) {
$this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 1');
}
throw $e;
}
return $total; return $total;
} }
/**
* The restore did not go through: a rollback does not undo CREATE TABLE everywhere. Left behind,
* the empty tables of an app would stop that app's installation later ("table already exists").
*
* @param array{created: list<string>, ...} $report
*/
private function dropCreatedTables(array &$report): void {
foreach ($report['created'] as $table) {
try {
$this->db->createSchemaManager()->dropTable($table);
} catch (\Throwable) {
// the error that brought us here is the one to report; a run again uses the table
}
}
$report['created'] = [];
}
/** /**
* The table prefix of the instance the dump was made on (from its appconfig table), or null * The table prefix of the instance the dump was made on (from its appconfig table), or null
* when the dump has none. * when the dump has none.
+7
View File
@@ -24,6 +24,10 @@ use OCP\AppFramework\Db\Entity;
* @method void setFiles(?int $v) * @method void setFiles(?int $v)
* @method int|null getBytes() * @method int|null getBytes()
* @method void setBytes(?int $v) * @method void setBytes(?int $v)
* @method string|null getNote()
* @method void setNote(?string $v)
* @method bool|null getPinned()
* @method void setPinned(?bool $v)
*/ */
final class Snapshot extends Entity { final class Snapshot extends Entity {
protected int $targetId = 0; protected int $targetId = 0;
@@ -33,8 +37,11 @@ final class Snapshot extends Entity {
protected int $createdAt = 0; protected int $createdAt = 0;
protected ?int $files = null; protected ?int $files = null;
protected ?int $bytes = null; protected ?int $bytes = null;
protected ?string $note = null;
protected ?bool $pinned = null;
public function __construct() { public function __construct() {
$this->addType('pinned', 'boolean');
$this->addType('targetId', 'integer'); $this->addType('targetId', 'integer');
$this->addType('createdAt', 'integer'); $this->addType('createdAt', 'integer');
$this->addType('files', 'integer'); $this->addType('files', 'integer');
+10
View File
@@ -28,6 +28,12 @@ use OCP\AppFramework\Db\Entity;
* @method void setMaxBytes(?int $v) * @method void setMaxBytes(?int $v)
* @method int getCreatedAt() * @method int getCreatedAt()
* @method void setCreatedAt(int $v) * @method void setCreatedAt(int $v)
* @method string|null getSchedule()
* @method void setSchedule(?string $v)
* @method string|null getScheduleZone()
* @method void setScheduleZone(?string $v)
* @method int|null getScheduleSince()
* @method void setScheduleSince(?int $v)
*/ */
final class Target extends Entity { final class Target extends Entity {
protected string $name = ''; protected string $name = '';
@@ -39,8 +45,12 @@ final class Target extends Entity {
protected int $createdAt = 0; protected int $createdAt = 0;
protected ?bool $appendOnly = null; protected ?bool $appendOnly = null;
protected ?int $maxBytes = null; protected ?int $maxBytes = null;
protected ?string $schedule = null;
protected ?string $scheduleZone = null;
protected ?int $scheduleSince = null;
public function __construct() { public function __construct() {
$this->addType('scheduleSince', 'integer');
$this->addType('createdAt', 'integer'); $this->addType('createdAt', 'integer');
$this->addType('appendOnly', 'boolean'); $this->addType('appendOnly', 'boolean');
$this->addType('maxBytes', 'integer'); $this->addType('maxBytes', 'integer');
+80 -3
View File
@@ -52,12 +52,51 @@ final class BackupRun {
$stats = $state['stats']; $stats = $state['stats'];
$walker = new TreeWalker($state['roots'], $state['exclude'], function (string $logical) use (&$state, &$stats): void { $walker = new TreeWalker($state['roots'], $state['exclude'], function (string $logical) use (&$state, &$stats): void {
self::recordUnreadable($state, $stats, $logical); self::recordUnreadable($state, $stats, $logical);
}); }, true);
$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']); $parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
$lines = []; $lines = [];
$emptyDirs = [];
$folders = [];
$finishedWalk = true; $finishedWalk = true;
foreach ($walker->walk($state['after']) as $logical => $path) { foreach ($walker->walk($state['after']) as $logical => $path) {
$safe = mb_check_encoding($logical, 'UTF-8') && Repository::isSafePath($logical);
if ($walker->atFolderWithContent()) {
// Only its time: what is in it is reported by itself, and a bad name in the path of
// each of those is what gets reported. The cursor stays for such a name, see below.
$mtime = $safe ? @filemtime($path) : false;
if ($safe && $mtime !== false) {
$folders[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR);
$state['after'] = $logical;
if (microtime(true) >= $deadline) {
$finishedWalk = false;
break;
}
}
continue;
}
if (!$safe) {
// A name a snapshot cannot hold (a backslash, a control character, not UTF-8). One
// such path must not make a backup fail, or the whole snapshot unreadable later.
// The cursor stays where it is: the state is JSON, which cannot hold such a name. A
// next step meets it again, and it is reported once.
self::recordBadName($state, $stats, $logical);
continue;
}
if (is_dir($path)) {
$mtime = @filemtime($path);
if ($mtime === false) {
continue; // removed while walking
}
$emptyDirs[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR);
$stats['emptyDirs'] = ($stats['emptyDirs'] ?? 0) + 1;
$state['after'] = $logical;
if (microtime(true) >= $deadline) {
$finishedWalk = false;
break;
}
continue;
}
$version = Repository::fileVersion($path); $version = Repository::fileVersion($path);
if ($version === 'missing') { if ($version === 'missing') {
continue; // deleted while walking continue; // deleted while walking
@@ -136,6 +175,16 @@ final class BackupRun {
$writer->write(implode("\n", $lines) . "\n"); $writer->write(implode("\n", $lines) . "\n");
$state['segments'][] = $writer->finish(); $state['segments'][] = $writer->finish();
} }
if ($emptyDirs !== []) {
$writer = $repo->blobWriter($stats);
$writer->write(implode("\n", $emptyDirs) . "\n");
$state['dirSegments'][] = $writer->finish();
}
if ($folders !== []) {
$writer = $repo->blobWriter($stats);
$writer->write(implode("\n", $folders) . "\n");
$state['folderSegments'][] = $writer->finish();
}
$repo->flushPacks(); $repo->flushPacks();
$state['parentPos'] = $parent->position(); $state['parentPos'] = $parent->position();
$state['stats'] = $stats; $state['stats'] = $stats;
@@ -150,10 +199,22 @@ final class BackupRun {
if ($state['phase'] === 'finish') { if ($state['phase'] === 'finish') {
// Every segment ends with a newline, so they read as one stream: fetched together, // Every segment ends with a newline, so they read as one stream: fetched together,
// several at a time where the location allows it. // several at a time where the location allows it.
// Empty folders go in the meta record, not between the files: a version that does not
// know them reads the snapshot as before, and only restores no empty folders.
$emptyDirs = [];
foreach (isset($state['dirSegments']) ? $repo->readLines(array_values(array_merge(...$state['dirSegments']))) : [] as $line) {
if ($line !== '') {
$emptyDirs[] = json_decode($line, true, 8, JSON_THROW_ON_ERROR);
}
}
// The times of the other folders too, packed: every look at a snapshot's meta record reads
// this, and a string of a few hundred KiB costs that far less than tens of thousands of arrays.
$folders = isset($state['folderSegments']) ? implode("\n", iterator_to_array($repo->readLines(array_values(array_merge(...$state['folderSegments']))), false)) : '';
$repo->putSnapshot($state['snapshot'], $repo->putSnapshot($state['snapshot'],
['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'], ['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
'roots' => array_keys($state['roots']), 'stats' => $state['stats'], 'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
'warnings' => self::warnings($state)] + $state['meta'], 'warnings' => self::warnings($state)] + ($emptyDirs === [] ? [] : ['emptyDirs' => $emptyDirs])
+ ($folders === '' ? [] : ['folderTimes' => base64_encode((string)gzdeflate($folders))]) + $state['meta'],
$repo->readLines(array_values(array_merge(...$state['segments'])))); $repo->readLines(array_values(array_merge(...$state['segments']))));
$state['phase'] = 'commit'; $state['phase'] = 'commit';
} }
@@ -214,10 +275,26 @@ final class BackupRun {
$stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1; $stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1;
} }
/** @return array{unreadable: list<string>, changing: list<string>} examples for the report (first 50 each) */ /**
* A file or directory left out because of its name. Remembered like an unreadable one, in a form
* that fits the JSON state and is safe to show: control characters and invalid bytes replaced.
*/
private static function recordBadName(array &$state, array &$stats, string $logical): void {
$shown = (string)preg_replace('/[\x00-\x1f\x7f]/', '?', mb_scrub($logical, 'UTF-8'));
if (isset($state['badNames'][$shown])) {
return;
}
if (count($state['badNames'] ?? []) < 1000) {
$state['badNames'][$shown] = true;
}
$stats['badNames'] = ($stats['badNames'] ?? 0) + 1;
}
/** @return array{unreadable: list<string>, badNames: list<string>, changing: list<string>} examples for the report (first 50 each) */
public static function warnings(array $state): array { public static function warnings(array $state): array {
return [ return [
'unreadable' => array_slice(array_map('strval', array_keys($state['unreadable'] ?? [])), 0, 50), 'unreadable' => array_slice(array_map('strval', array_keys($state['unreadable'] ?? [])), 0, 50),
'badNames' => array_slice(array_map('strval', array_keys($state['badNames'] ?? [])), 0, 50),
'changing' => array_slice(array_map('strval', array_values($state['changing'] ?? [])), 0, 50), 'changing' => array_slice(array_map('strval', array_values($state['changing'] ?? [])), 0, 50),
]; ];
} }
+28 -3
View File
@@ -9,12 +9,16 @@ namespace OCA\NgBackup\Job;
/** /**
* Walks one or more source roots in a stable depth-first order (entries of a directory sorted * Walks one or more source roots in a stable depth-first order (entries of a directory sorted
* by name, children right after their directory), yielding "<root>/<relative path>" for files. * by name, children right after their directory), yielding "<root>/<relative path>" for files
* and for directories with nothing in them to walk (the caller tells them apart with is_dir()).
* A root itself without a name is never yielded.
* *
* The same order is used for comparing paths (compare()), so a walk can resume after a cursor * The same order is used for comparing paths (compare()), so a walk can resume after a cursor
* and be merge-joined with the previous snapshot without holding any list in memory. * and be merge-joined with the previous snapshot without holding any list in memory.
*/ */
final class TreeWalker { final class TreeWalker {
private bool $atFolder = false;
/** /**
* @param array<string, string> $roots name => absolute directory, walked in this order * @param array<string, string> $roots name => absolute directory, walked in this order
* @param list<string> $exclude path prefixes ("<root>/<relative>") that are skipped entirely; * @param list<string> $exclude path prefixes ("<root>/<relative>") that are skipped entirely;
@@ -25,9 +29,16 @@ final class TreeWalker {
private array $exclude = [], private array $exclude = [],
/** Called with the logical path of a directory that cannot be listed. */ /** Called with the logical path of a directory that cannot be listed. */
private ?\Closure $onUnreadable = null, private ?\Closure $onUnreadable = null,
/** Also yield directories with something in them, right before the first thing in them. */
private bool $withFolders = false,
) { ) {
} }
/** Whether what was just yielded is a directory with something in it (see $withFolders). */
public function atFolderWithContent(): bool {
return $this->atFolder;
}
/** @return \Generator<string, string> logical path => absolute path, strictly after $after */ /** @return \Generator<string, string> logical path => absolute path, strictly after $after */
public function walk(?string $after = null): \Generator { public function walk(?string $after = null): \Generator {
foreach ($this->roots as $name => $dir) { foreach ($this->roots as $name => $dir) {
@@ -66,27 +77,41 @@ final class TreeWalker {
} }
$names = array_values(array_diff($names, ['.', '..'])); $names = array_values(array_diff($names, ['.', '..']));
sort($names, SORT_STRING); sort($names, SORT_STRING);
$empty = true; // no file or directory below it that is walked, whatever the cursor skips
foreach ($names as $name) { foreach ($names as $name) {
$childLogical = $logical === '' ? $name : $logical . '/' . $name; $childLogical = $logical === '' ? $name : $logical . '/' . $name;
$childAbs = $abs . '/' . $name; $childAbs = $abs . '/' . $name;
if (is_link($childAbs)) { if (is_link($childAbs)) {
continue; // never follow symlinks out of the data directory continue; // never follow symlinks out of the data directory
} }
if (is_dir($childAbs)) { $childIsDir = is_dir($childAbs);
if (($childIsDir || is_file($childAbs)) && $empty && !$this->isExcluded($childLogical, $childIsDir)
&& $this->withFolders && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) {
$this->atFolder = true;
yield $logical => $abs;
$this->atFolder = false;
}
if ($childIsDir) {
$empty = $empty && $this->isExcluded($childLogical, true);
// Skip a directory entirely if everything in it sorts before the cursor. // Skip a directory entirely if everything in it sorts before the cursor.
if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) { if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) {
continue; continue;
} }
yield from $this->walkDir($childAbs, $childLogical, $after); yield from $this->walkDir($childAbs, $childLogical, $after);
} elseif (is_file($childAbs)) { } elseif (is_file($childAbs)) {
$excluded = $this->isExcluded($childLogical, false);
$empty = $empty && $excluded;
if ($after !== null && self::compare($childLogical, $after) <= 0) { if ($after !== null && self::compare($childLogical, $after) <= 0) {
continue; continue;
} }
if (!$this->isExcluded($childLogical, false)) { if (!$excluded) {
yield $childLogical => $childAbs; yield $childLogical => $childAbs;
} }
} }
} }
if ($empty && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) {
yield $logical => $abs;
}
} }
private function isExcluded(string $logical, bool $isDir): bool { private function isExcluded(string $logical, bool $isDir): bool {
@@ -0,0 +1,33 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Migration;
use Closure;
use OCP\DB\ISchemaWrapper;
use OCP\DB\Types;
use OCP\Migration\IOutput;
use OCP\Migration\SimpleMigrationStep;
/** Local copy of what the catalog on the location says about a snapshot: its note and pin. */
final class Version000007Date20261107000000 extends SimpleMigrationStep {
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
/** @var ISchemaWrapper $schema */
$schema = $schemaClosure();
$t = $schema->getTable('ngb_snapshots');
$changed = false;
if (!$t->hasColumn('note')) {
$t->addColumn('note', Types::STRING, ['notnull' => false, 'length' => 1000]);
$changed = true;
}
if (!$t->hasColumn('pinned')) {
$t->addColumn('pinned', Types::BOOLEAN, ['notnull' => false, 'default' => false]);
$changed = true;
}
return $changed ? $schema : null;
}
}
@@ -0,0 +1,49 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Migration;
use Closure;
use OCA\NgBackup\Service\LocationSchedule;
use OCA\NgBackup\Service\Schedule;
use OCP\DB\ISchemaWrapper;
use OCP\DB\Types;
use OCP\IDBConnection;
use OCP\Migration\IOutput;
use OCP\Migration\SimpleMigrationStep;
/** A schedule per location. Each existing location gets the daily time that held for all of them. */
final class Version000008Date20261107010000 extends SimpleMigrationStep {
public function __construct(
private IDBConnection $db,
private Schedule $schedule,
) {
}
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
/** @var ISchemaWrapper $schema */
$schema = $schemaClosure();
$t = $schema->getTable('ngb_targets');
if ($t->hasColumn('schedule')) {
return null;
}
$t->addColumn('schedule', Types::STRING, ['notnull' => false, 'length' => 16]);
$t->addColumn('schedule_zone', Types::STRING, ['notnull' => false, 'length' => 64]);
$t->addColumn('schedule_since', Types::BIGINT, ['notnull' => false, 'unsigned' => true]);
return $schema;
}
public function postSchemaChange(IOutput $output, Closure $schemaClosure, array $options): void {
['time' => $time, 'timezone' => $zone] = $this->schedule->get();
$qb = $this->db->getQueryBuilder();
$qb->update('ngb_targets')
->set('schedule', $qb->createNamedParameter(Schedule::validTime($time) ? LocationSchedule::normalize($time) : ''))
->set('schedule_zone', $qb->createNamedParameter($zone))
->where($qb->expr()->isNull('schedule'));
$qb->executeStatement();
}
}
+12 -1
View File
@@ -46,6 +46,10 @@ final class Notifier implements INotifier {
$notification->setParsedSubject($l->t('Backup to %1$s finished with warnings: %2$s unreadable, %3$s changed while being read', [$p['target'], $p['unreadable'], $p['changing']])); $notification->setParsedSubject($l->t('Backup to %1$s finished with warnings: %2$s unreadable, %3$s changed while being read', [$p['target'], $p['unreadable'], $p['changing']]));
$notification->setParsedMessage($l->t('Unreadable files and folders are missing from this backup (check their permissions). Files that kept changing were saved as they were while being read. Examples: %s', [$p['examples'] ?? ''])); $notification->setParsedMessage($l->t('Unreadable files and folders are missing from this backup (check their permissions). Files that kept changing were saved as they were while being read. Examples: %s', [$p['examples'] ?? '']));
break; break;
case 'run_bad_names':
$notification->setParsedSubject($l->n('Backup to %s finished, but %n item is missing because of its name', 'Backup to %s finished, but %n items are missing because of their names', (int)$p['count'], [$p['target']]));
$notification->setParsedMessage($l->t('Names with a backslash or a control character, or that are not valid UTF-8, are not supported, so these files and folders are not in this backup. Rename them; the next backup includes them. Examples: %s', [$p['examples'] ?? '']));
break;
case 'quota_warning': case 'quota_warning':
$notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']])); $notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']]));
$notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.')); $notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.'));
@@ -57,13 +61,20 @@ final class Notifier implements INotifier {
: $l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']])); : $l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']]));
$notification->setParsedMessage($l->t('Check the backup settings and the server log.')); $notification->setParsedMessage($l->t('Check the backup settings and the server log.'));
break; break;
case 'schedule_off':
$notification->setParsedSubject($l->t('Backup location %s has no schedule', [$p['target']]));
$notification->setParsedMessage(($p['others'] ?? '') === '1'
? $l->t('No backups run to this location until it gets a schedule or someone starts one. If that is what you want, you can ignore this reminder.')
: $l->t('No location has a schedule, so no backups run at all unless someone starts one. Set a schedule under Backups.'));
break;
case 'security_event': case 'security_event':
$who = (string)($p['actor'] ?? ''); $who = (string)($p['actor'] ?? '');
$notification->setParsedSubject(match ($p['event'] ?? '') { $notification->setParsedSubject(match ($p['event'] ?? '') {
'kit_downloaded' => $l->t('%s downloaded the backup recovery kit', [$who]), 'kit_downloaded' => $l->t('%s downloaded the backup recovery kit', [$who]),
'target_removed' => $l->t('%1$s removed backup location %2$s', [$who, $p['target'] ?? '']), 'target_removed' => $l->t('%1$s removed backup location %2$s', [$who, $p['target'] ?? '']),
'target_changed' => $l->t('%1$s changed the connection settings of backup location %2$s', [$who, $p['target'] ?? '']), 'target_changed' => $l->t('%1$s changed the connection settings of backup location %2$s', [$who, $p['target'] ?? '']),
'append_only_off' => $l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']), 'schedule_off' => $l->t('%1$s turned off the schedule of backup location %2$s', [$who, $p['target'] ?? '']),
'append_only_off' =>$l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']),
'retention_changed' => $l->t('%1$s changed the backup retention to %2$s', [$who, $p['policy'] ?? '']), 'retention_changed' => $l->t('%1$s changed the backup retention to %2$s', [$who, $p['policy'] ?? '']),
'snapshots_forgotten' => $l->t('%1$s moved %2$s snapshots of %3$s to the trash; they are deleted permanently after %4$s unless restored', [$who, $p['count'] ?? '', $p['target'] ?? '', $p['until'] ?? '']), 'snapshots_forgotten' => $l->t('%1$s moved %2$s snapshots of %3$s to the trash; they are deleted permanently after %4$s unless restored', [$who, $p['count'] ?? '', $p['target'] ?? '', $p['until'] ?? '']),
'snapshot_untrashed' => $l->t('%1$s restored snapshot %2$s of %3$s from the trash', [$who, $p['snapshot'] ?? '', $p['target'] ?? '']), 'snapshot_untrashed' => $l->t('%1$s restored snapshot %2$s of %3$s from the trash', [$who, $p['snapshot'] ?? '', $p['target'] ?? '']),
+10 -1
View File
@@ -30,6 +30,10 @@ final class Catalog {
public array $trash = []; public array $trash = [];
/** @var array<string, true> manifest path => true, for a per-user export (users/<uid>/<exportId>) */ /** @var array<string, true> manifest path => true, for a per-user export (users/<uid>/<exportId>) */
public array $userExports = []; public array $userExports = [];
/** @var array<string, true> snapshots the retention policy leaves alone */
public array $pinned = [];
/** @var array<string, string> snapshot id => note, also while the snapshot is in the trash */
public array $notes = [];
public string $hash = ''; public string $hash = '';
public function __construct( public function __construct(
@@ -82,6 +86,8 @@ final class Catalog {
// Catalogs written before user exports existed have no such key: default to empty rather // Catalogs written before user exports existed have no such key: default to empty rather
// than adopting the location's current users/ listing (that would skip verification). // than adopting the location's current users/ listing (that would skip verification).
$this->userExports = array_fill_keys($data['userExports'] ?? [], true); $this->userExports = array_fill_keys($data['userExports'] ?? [], true);
$this->pinned = array_fill_keys($data['pinned'] ?? [], true);
$this->notes = $data['notes'] ?? [];
$this->anchor?->set($this->repositoryId, $this->gen, $this->hash); $this->anchor?->set($this->repositoryId, $this->gen, $this->hash);
} }
@@ -93,9 +99,12 @@ final class Catalog {
if ($this->backend->exists($this->path($next))) { if ($this->backend->exists($this->path($next))) {
throw new RepositoryException("Catalog generation $next was written by another process at the same time; try again"); throw new RepositoryException("Catalog generation $next was written by another process at the same time; try again");
} }
// A pin ends when its snapshot leaves the list, a note when the snapshot is gone for good.
$this->pinned = array_intersect_key($this->pinned, $this->snapshots);
$this->notes = array_intersect_key($this->notes, $this->snapshots + $this->trash);
$plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash, $plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash,
'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash), 'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash),
'userExports' => array_keys($this->userExports)], JSON_THROW_ON_ERROR); 'userExports' => array_keys($this->userExports), 'pinned' => array_keys($this->pinned), 'notes' => $this->notes], JSON_THROW_ON_ERROR);
$encrypted = $this->cipher->encryptString($plain, $this->ad($next)); $encrypted = $this->cipher->encryptString($plain, $this->ad($next));
$stream = fopen('php://memory', 'w+b'); $stream = fopen('php://memory', 'w+b');
fwrite($stream, $encrypted); fwrite($stream, $encrypted);
+221 -21
View File
@@ -44,6 +44,8 @@ final class Repository {
private ?int $maxBytes = null; private ?int $maxBytes = null;
private Catalog $catalog; private Catalog $catalog;
private PackReader $packReader; private PackReader $packReader;
/** @var array<string, array<string, mixed>> meta record of the snapshot read last */
private array $lastMeta = [];
/** verify() lists the pack folders from this many packs on (see presentPacks()); tests lower it. */ /** verify() lists the pack folders from this many packs on (see presentPacks()); tests lower it. */
public int $listPacksAbove = 256; public int $listPacksAbove = 256;
@@ -132,6 +134,49 @@ final class Repository {
return $this->catalog->gen; return $this->catalog->gen;
} }
/** @return list<string> snapshots the retention policy leaves alone */
public function pinnedSnapshots(): array {
return array_map('strval', array_keys($this->catalog->pinned));
}
/** @return array<string, string> snapshot id => note */
public function snapshotNotes(): array {
return $this->catalog->notes;
}
public function setPinned(string $snapshotId, bool $pinned): void {
$this->requireSnapshot($snapshotId);
if (isset($this->catalog->pinned[$snapshotId]) === $pinned) {
return;
}
if ($pinned) {
$this->catalog->pinned[$snapshotId] = true;
} else {
unset($this->catalog->pinned[$snapshotId]);
}
$this->catalog->write();
}
/** A note kept in the catalog: unlike the label inside the snapshot, it can change afterwards. */
public function setSnapshotNote(string $snapshotId, string $note): void {
$this->requireSnapshot($snapshotId);
if (($this->catalog->notes[$snapshotId] ?? '') === $note) {
return;
}
if ($note === '') {
unset($this->catalog->notes[$snapshotId]);
} else {
$this->catalog->notes[$snapshotId] = $note;
}
$this->catalog->write();
}
private function requireSnapshot(string $snapshotId): void {
if (!isset($this->catalog->snapshots[$snapshotId])) {
throw new \InvalidArgumentException('There is no snapshot ' . $snapshotId . ' on this location');
}
}
/** Index files downloaded when this repository was opened (0 when the local cache was complete). */ /** Index files downloaded when this repository was opened (0 when the local cache was complete). */
public function indexDownloads(): int { public function indexDownloads(): int {
return $this->index->downloaded; return $this->index->downloaded;
@@ -257,8 +302,8 @@ final class Repository {
// that was being written when the process was killed stays (8 MiB, DR test 2026-11-05) // that was being written when the process was killed stays (8 MiB, DR test 2026-11-05)
$stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR); $stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR);
foreach ($stale === false ? [] : $stale as $dir) { foreach ($stale === false ? [] : $stale as $dir) {
if (is_link($dir)) { if (is_link($dir) || preg_match('/^\.ngb-restore-spool-[0-9a-f]{8}$/', basename($dir)) !== 1) {
continue; continue; // not a spool of ours, whatever it is called
} }
$files = glob($dir . '/*'); $files = glob($dir . '/*');
foreach ($files === false ? [] : $files as $file) { foreach ($files === false ? [] : $files as $file) {
@@ -313,40 +358,39 @@ final class Repository {
if (!self::underPrefix($entry['p'], $prefix)) { if (!self::underPrefix($entry['p'], $prefix)) {
continue; continue;
} }
if ($heartbeat !== null) {
$heartbeat(); // also for a long row of files that are skipped or have no data
}
$dest = self::destination($target, $entry['p'], $prefix, $stripPrefix); $dest = self::destination($target, $entry['p'], $prefix, $stripPrefix);
if ($keepUnchanged && self::inPlace($dest, $entry)) { $dir = self::directoryInTarget($target, dirname($dest), $entry['p']);
// below the target as written by an earlier run of this restore (checked then) if (is_link($dest)) {
$count++; throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $entry['p']);
continue;
}
if (!is_dir(dirname($dest))) {
mkdir(dirname($dest), 0700, true);
}
// Defence in depth: the resolved directory must stay below the target (no symlink escapes).
$base = realpath($target);
$dir = realpath(dirname($dest));
if ($base === false || $dir === false || ($dir !== $base && !str_starts_with($dir, $base . '/')) || is_link($dest)) {
throw new RepositoryException('Refusing to write outside the restore directory: ' . $entry['p']);
} }
if (!isset($cleaned[$dir])) { if (!isset($cleaned[$dir])) {
self::removeStaleTemporaries($dir); self::removeStaleTemporaries($dir);
$cleaned[$dir] = true; $cleaned[$dir] = true;
} }
if ($keepUnchanged && self::inPlace($dest, $entry)) {
$count++;
continue;
}
// Write next to the destination and rename after all blobs verified: at most one file // Write next to the destination and rename after all blobs verified: at most one file
// is ever present twice. // is ever present twice.
$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6)); $tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
$out = @fopen($tmp, 'xb'); // fails if anything (including a symlink) already exists there $out = @fopen($tmp, 'xb'); // fails if anything (including a symlink) already exists there
if ($out === false) { if ($out === false) {
throw new RepositoryException('Cannot create a temporary file for ' . $entry['p']); throw new RestoreTargetException('Cannot create a temporary file for ' . $entry['p'] . ' (disk full, or no permission?)');
} }
try { try {
foreach ($entry['b'] as $id) { foreach ($entry['b'] as $id) {
fwrite($out, $this->restoreBlob($id, $plan)); self::writeAll($out, $this->restoreBlob($id, $plan), $entry['p']);
if ($heartbeat !== null) { if ($heartbeat !== null) {
$heartbeat(); $heartbeat();
} }
} }
fclose($out); if (!@fclose($out)) {
throw new RestoreTargetException('Cannot write ' . $entry['p'] . ' to the restore directory (disk full?)');
}
clearstatcache(true, $tmp); clearstatcache(true, $tmp);
if (filesize($tmp) !== $entry['s']) { if (filesize($tmp) !== $entry['s']) {
throw new RepositoryException('Size mismatch restoring ' . $entry['p']); throw new RepositoryException('Size mismatch restoring ' . $entry['p']);
@@ -363,19 +407,144 @@ final class Repository {
} }
$skipped[$entry['p']] = $e->getMessage(); $skipped[$entry['p']] = $e->getMessage();
continue; continue;
} catch (\Throwable $e) {
if (is_resource($out)) {
@fclose($out);
}
@unlink($tmp);
throw $e;
} }
if (!rename($tmp, $dest)) { if (!rename($tmp, $dest)) {
@unlink($tmp); @unlink($tmp);
throw new RepositoryException('Cannot put the restored file in place: ' . $entry['p']); throw new RestoreTargetException('Cannot put the restored file in place: ' . $entry['p']);
} }
if (!touch($dest, $entry['m'])) { if (!touch($dest, $entry['m'])) {
throw new RepositoryException('Cannot set the modification time of ' . $entry['p']); throw new RestoreTargetException('Cannot set the modification time of ' . $entry['p']);
} }
$count++; $count++;
} }
// Last: a file can only be restored into a folder that exists, and the meta record was
// read along with the files, so this asks nothing more of the location.
foreach ($this->emptyDirs($snapshotId) as [$path, $mtime]) {
if (self::underPrefix($path, $prefix)) {
if ($heartbeat !== null) {
$heartbeat();
}
// The prefix itself, stripped, is the target (for a file it is the file's name there).
$dest = $stripPrefix && $path === rtrim($prefix, '/') ? rtrim($target, '/') : self::destination($target, $path, $prefix, $stripPrefix);
self::restoreEmptyDir($target, $dest, $path, $mtime);
}
}
// Very last, nothing is written in them any more. Also on a run that goes on after a stop, and
// the deepest first. The time is not worth stopping for: all the files are there by now.
foreach (array_reverse($this->folderTimes($snapshotId)) as [$path, $mtime]) {
if (!self::underPrefix($path, $prefix) || ($stripPrefix && $path === rtrim($prefix, '/'))) {
continue; // the target itself is not ours to change
}
if ($heartbeat !== null) {
$heartbeat();
}
$dest = self::destination($target, $path, $prefix, $stripPrefix);
@touch(self::directoryInTarget($target, $dest, $path), $mtime);
}
return $count; return $count;
} }
/**
* The folders with something in them that a snapshot recorded, parents before children (none
* before 1.1.0). Damage here is no reason to give up on the files of a restore.
*
* @return list<array{0: string, 1: int}> path, modification time
*/
public function folderTimes(string $snapshotId): array {
$meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId);
$packed = is_string($meta['folderTimes'] ?? null) ? base64_decode($meta['folderTimes'], true) : false;
$plain = $packed === false ? false : @gzinflate($packed);
$folders = [];
foreach ($plain === false ? [] : explode("\n", $plain) as $line) {
$folder = json_decode($line, true);
if (is_array($folder) && is_string($folder[0] ?? null) && self::isSafePath($folder[0]) && is_int($folder[1] ?? null)) {
$folders[] = [$folder[0], $folder[1]];
}
}
return $folders;
}
/**
* The folders without anything in them that a snapshot recorded (none before 1.0.3).
*
* @return list<array{0: string, 1: int}> path, modification time
*/
public function emptyDirs(string $snapshotId): array {
$meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId);
$dirs = [];
foreach (is_array($meta['emptyDirs'] ?? null) ? $meta['emptyDirs'] : [] as $dir) {
// One folder that makes no sense is no reason to give up on the files of a restore.
if (is_array($dir) && is_string($dir[0] ?? null) && self::isSafePath($dir[0]) && is_int($dir[1] ?? null)) {
$dirs[] = [$dir[0], $dir[1]];
}
}
return $dirs;
}
private static function restoreEmptyDir(string $target, string $dest, string $path, int $mtime): void {
if (is_link($dest)) {
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
}
$existed = is_dir($dest);
self::directoryInTarget($target, $dest, $path);
if (!$existed && !@touch($dest, $mtime)) {
throw new RestoreTargetException('Cannot set the modification time of ' . $path);
}
}
/**
* Make sure a folder exists below the target, without following a link out of it: what exists
* of the path is checked before anything is created.
*
* @return string the folder's real path
*/
private static function directoryInTarget(string $target, string $dir, string $path): string {
if (!is_dir($target) && !@mkdir($target, 0700, true) && !is_dir($target)) {
throw new RestoreTargetException('Cannot create the restore directory ' . $target);
}
$base = realpath($target);
$existing = $dir;
while (!file_exists($existing) && !is_link($existing) && dirname($existing) !== $existing) {
$existing = dirname($existing);
}
$real = realpath($existing);
if ($base === false || $real === false || ($real !== $base && !str_starts_with($real, $base . '/'))) {
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
}
if ($existing === $dir) {
if (!is_dir($real)) {
throw new RestoreTargetException('Cannot create the folder for ' . $path . ': a file is in the way');
}
return $real;
}
if (!@mkdir($dir, 0700, true) && !is_dir($dir)) {
throw new RestoreTargetException('Cannot create the folder for ' . $path . ' (disk full, no permission, or a file in the way?)');
}
return (string)realpath($dir);
}
/**
* Write all of $data, or stop: a short write leaves a file that looks damaged only by its size.
* A write may take only part of the data and still be fine; one that takes nothing is not.
*
* @param resource $out
*/
public static function writeAll($out, string $data, string $path): void {
while ($data !== '') {
$written = @fwrite($out, $data);
if ($written === false || $written === 0) {
throw new RestoreTargetException('Cannot write ' . $path . ' to the restore directory (disk full?)');
}
$data = substr($data, $written);
}
}
/** Where restore() writes the snapshot path $path. */ /** Where restore() writes the snapshot path $path. */
private static function destination(string $target, string $path, string $prefix, bool $stripPrefix): string { private static function destination(string $target, string $path, string $prefix, bool $stripPrefix): string {
$relPath = $path; $relPath = $path;
@@ -722,6 +891,34 @@ final class Repository {
return array_keys($doomed); return array_keys($doomed);
} }
/**
* Part of trash(), for a page that shows how far it is: each entry is a read of its own on the
* location. Reads the entries with an id after $after, in order, until $deadline
* (microtime(true)), at least one. The id, not a position: a cleanup that changes the trash
* between two parts then makes no entry come twice or be missed.
*
* @return array{total: int, items: array<string, array<string, mixed>>}
*/
public function trashPage(string $after, float $deadline): array {
$ids = array_map('strval', array_keys($this->catalog->trash));
sort($ids, SORT_STRING);
$items = [];
foreach ($ids as $id) {
if (strcmp($id, $after) <= 0) {
continue;
}
try {
$items[$id] = $this->trashInfo($id);
} catch (\Throwable) {
throw new RepositoryException("Trash entry $id listed in the catalog is missing on the location (possible tampering)");
}
if (microtime(true) >= $deadline) {
break;
}
}
return ['total' => count($ids), 'items' => $items];
}
private function trashInfo(string $snapshotId): array { private function trashInfo(string $snapshotId): array {
return json_decode($this->getObject('trash/info/' . $snapshotId), true, 512, JSON_THROW_ON_ERROR); return json_decode($this->getObject('trash/info/' . $snapshotId), true, 512, JSON_THROW_ON_ERROR);
} }
@@ -1281,7 +1478,10 @@ final class Repository {
$this->cipher->decrypt($fh, $plain, 'snapshot:' . $snapshotId); $this->cipher->decrypt($fh, $plain, 'snapshot:' . $snapshotId);
fclose($fh); fclose($fh);
rewind($plain); rewind($plain);
fgets($plain); // meta line $meta = json_decode((string)fgets($plain), true);
if (is_array($meta)) {
$this->lastMeta = [$snapshotId => $meta]; // a snapshot never changes
}
while (($line = fgets($plain)) !== false) { while (($line = fgets($plain)) !== false) {
$entry = json_decode($line, true, 512, JSON_THROW_ON_ERROR); $entry = json_decode($line, true, 512, JSON_THROW_ON_ERROR);
if (!is_array($entry) || !is_string($entry['p'] ?? null) || !self::isSafePath($entry['p']) if (!is_array($entry) || !is_string($entry['p'] ?? null) || !self::isSafePath($entry['p'])
+15
View File
@@ -0,0 +1,15 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Repository;
/**
* A restore cannot write to where it restores to (a full disk, permissions, something in the way).
* The backup is fine, so this never counts as damaged data: the restore stops, also with --salvage.
*/
final class RestoreTargetException extends \RuntimeException {
}
+38 -1
View File
@@ -61,6 +61,16 @@ final class AlertService {
'examples' => mb_strimwidth(implode(', ', array_merge($examples['unreadable'], $examples['changing'])), 0, 300, '…')]); 'examples' => mb_strimwidth(implode(', ', array_merge($examples['unreadable'], $examples['changing'])), 0, 300, '…')]);
} }
/**
* A backup that finished but left out files or folders because of their name.
*
* @param list<string> $examples
*/
public function runBadNames(Run $run, string $targetName, int $count, array $examples): void {
$this->notifyAdmins('run_bad_names', 'run-' . $run->getId(), ['target' => $targetName, 'count' => (string)$count,
'examples' => mb_strimwidth(implode(', ', $examples), 0, 300, '…')]);
}
/** /**
* Remember how much is stored on a location (for the settings page) and warn administrators * Remember how much is stored on a location (for the settings page) and warn administrators
* once a day when it is at 80% of its limit or more. * once a day when it is at 80% of its limit or more.
@@ -91,9 +101,36 @@ final class AlertService {
return $v === '' ? null : (int)$v; return $v === '' ? null : (int)$v;
} }
/**
* Whether it is time to remind administrators of a location without a schedule: once a week
* while another location is still backed up, every day when none is. An hour of margin, so
* that the moment of the cron run does not push a reminder to the next day.
*/
public static function offReminderDue(int $last, int $now, bool $othersActive): bool {
return $now - $last >= ($othersActive ? 7 * 86400 : 86400) - 3600;
}
public function checkOff(Target $target, bool $othersActive): void {
if ((string)$target->getSchedule() !== '') {
return;
}
$key = 'off_notified_' . $target->getId();
$now = time();
$last = max((int)$target->getScheduleSince(), $this->appConfig->getValueInt(Application::APP_ID, $key, 0));
if (!self::offReminderDue($last, $now, $othersActive)) {
return;
}
$this->appConfig->setValueInt(Application::APP_ID, $key, $now);
$this->notifyAdmins('schedule_off', 'off-' . $target->getId() . '-' . date('Y-m-d', $now),
['target' => $target->getName(), 'others' => $othersActive ? '1' : '0']);
}
/** At most one notification per location per day. */ /** At most one notification per location per day. */
public function checkStale(Target $target): void { public function checkStale(Target $target): void {
$last = $this->runs->lastFinished($target->getId(), BackupService::KIND_FULL); if ((string)$target->getSchedule() === '') {
return; // no schedule is a choice, as for a location that is only connected now and then
}
$last =$this->runs->lastFinished($target->getId(), BackupService::KIND_FULL);
$lastTime = $last?->getFinishedAt() ?? $target->getCreatedAt(); $lastTime = $last?->getFinishedAt() ?? $target->getCreatedAt();
if (time() - $lastTime < self::STALE_AFTER || $this->runs->findRunning($target->getId(), BackupService::KIND_FULL) !== []) { if (time() - $lastTime < self::STALE_AFTER || $this->runs->findRunning($target->getId(), BackupService::KIND_FULL) !== []) {
return; return;
+23 -2
View File
@@ -58,6 +58,18 @@ final class BackupService {
$this->openedWith = $opened; $this->openedWith = $opened;
} }
/**
* The running backups in the order they get their turn: first come, first served. The first one
* runs, the others are queued; only one at a time reads the disks.
*
* @return list<Run>
*/
public function runsInTurn(): array {
$running = $this->runs->findRunning(null, self::KIND_FULL);
usort($running, static fn (Run $a, Run $b): int => [$a->getStartedAt(), $a->getId()] <=> [$b->getStartedAt(), $b->getId()]);
return $running;
}
public function start(Target $target, string $label = ''): Run { public function start(Target $target, string $label = ''): Run {
$label = trim($label); $label = trim($label);
if (mb_strlen($label) > self::LABEL_MAX) { if (mb_strlen($label) > self::LABEL_MAX) {
@@ -79,7 +91,7 @@ final class BackupService {
try { try {
return $this->leases->with(PruneService::lockKey($target), LeaseService::EXCLUSIVE, 60, fn () => $this->createRun($target, $label)); return $this->leases->with(PruneService::lockKey($target), LeaseService::EXCLUSIVE, 60, fn () => $this->createRun($target, $label));
} catch (LockedException) { } catch (LockedException) {
throw new \RuntimeException('A cleanup or another start for ' . $target->getName() . ' is running; try again later'); throw new \RuntimeException('A cleanup, a restore or another start for ' . $target->getName() . ' is running, or one was stopped by force (that clears by itself within two hours); try again later');
} }
} }
@@ -210,6 +222,10 @@ final class BackupService {
$state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR); $state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR);
if ($run->getPhase() === 'db') { if ($run->getPhase() === 'db') {
if (!isset($state['dump'])) {
// It may have been queued for hours: the restore point has the time of its data.
$state['files']['started'] = gmdate('c');
}
$dumper = new DbDumper($this->db->getInner(), $this->config->getSystemValueString('dbtableprefix', 'oc_')); $dumper = new DbDumper($this->db->getInner(), $this->config->getSystemValueString('dbtableprefix', 'oc_'));
$dbStats = $state['dbStats'] ?? []; $dbStats = $state['dbStats'] ?? [];
// With system cron the whole dump usually fits in one step (one transaction, fully // With system cron the whole dump usually fits in one step (one transaction, fully
@@ -253,8 +269,13 @@ final class BackupService {
try { try {
$this->alerts->storedBytes($target, $repo->storedBytes()); $this->alerts->storedBytes($target, $repo->storedBytes());
$stats = $state['files']['stats']; $stats = $state['files']['stats'];
if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) {
$warnings = BackupRun::warnings($state['files']); $warnings = BackupRun::warnings($state['files']);
if (($stats['badNames'] ?? 0) > 0) {
$this->logger->warning(sprintf('NG Backup run %d to %s: %d left out because their name is not supported (rename them): %s',
$run->getId(), $target->getName(), $stats['badNames'], implode(', ', $warnings['badNames'])));
$this->alerts->runBadNames($run, $target->getName(), (int)$stats['badNames'], $warnings['badNames']);
}
if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) {
$this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s', $this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s',
$run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']), $run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']),
$stats['changing'] ?? 0, implode(', ', $warnings['changing']))); $stats['changing'] ?? 0, implode(', ', $warnings['changing'])));
@@ -0,0 +1,12 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Service;
/** A step after the database was committed failed: the database is no longer the fresh one. */
final class DatabaseAlreadyRestoredException extends \RuntimeException {
}
+9
View File
@@ -76,6 +76,15 @@ final class LeaseService {
$qb->executeStatement(); $qb->executeStatement();
} }
/** Remove the shared leases on $name, for a caller that knows their holders were stopped. */
public function dropShared(string $name): void {
$qb = $this->db->getQueryBuilder();
$qb->delete('ngb_leases')
->where($qb->expr()->eq('name', $qb->createNamedParameter($name)))
->andWhere($qb->expr()->eq('mode', $qb->createNamedParameter(self::SHARED)));
$qb->executeStatement();
}
/** Wraps $refresh so calling it often (per file, per row batch) only touches the database every $seconds. */ /** Wraps $refresh so calling it often (per file, per row batch) only touches the database every $seconds. */
public static function throttled(callable $refresh, int $seconds = 60): \Closure { public static function throttled(callable $refresh, int $seconds = 60): \Closure {
$last = time(); $last = time();
+78
View File
@@ -0,0 +1,78 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Service;
/**
* When a location is backed up: "" (off), "HH:MM" (daily) or "HH:MM/N" (from HH:MM every N hours,
* starting over at HH:MM the next day), in the location's time zone.
*/
final class LocationSchedule {
/** Attempts per moment after a failure, an hour apart, instead of one every few minutes. */
public const MAX_ATTEMPTS = 3;
public const RETRY_AFTER = 3600;
/** @throws \InvalidArgumentException */
public static function normalize(string $schedule): string {
$schedule = trim($schedule);
if ($schedule === '' || strtolower($schedule) === 'off') {
return '';
}
if (preg_match('/^(([01]\d|2[0-3]):[0-5]\d)(?:\/(\d{1,2}))?$/', $schedule, $m) !== 1) {
throw new \InvalidArgumentException('Use "off", HH:MM for once a day, or HH:MM/N for every N hours from HH:MM');
}
$hours = (int)($m[3] ?? 24);
if ($hours < 1 || $hours > 24) {
throw new \InvalidArgumentException('Every 1 to 24 hours');
}
return $hours === 24 ? $m[1] : $m[1] . '/' . $hours;
}
/**
* The most recent scheduled moment at or before $now; null without a schedule.
*
* @param non-empty-string $zone
*/
public static function lastMoment(string $schedule, string $zone, int $now): ?int {
if ($schedule === '') {
return null;
}
$tz = new \DateTimeZone($zone);
$today = (new \DateTimeImmutable('@' . $now))->setTimezone($tz);
$moments = array_merge(self::moments($schedule, $today->modify('-1 day'), $tz), self::moments($schedule, $today, $tz));
$last = null;
foreach ($moments as $moment) {
if ($moment <= $now && ($last === null || $moment > $last)) {
$last = $moment;
}
}
return $last;
}
/** @return list<int> the moments of one day */
private static function moments(string $schedule, \DateTimeImmutable $day, \DateTimeZone $tz): array {
[$time, $every] = array_pad(explode('/', $schedule), 2, '24');
[$hour, $minute] = array_map('intval', explode(':', $time));
$moments = [];
for ($h = $hour; $h < 24; $h += (int)$every) {
$moments[] = (new \DateTimeImmutable($day->format('Y-m-d') . sprintf(' %02d:%02d', $h, $minute), $tz))->getTimestamp();
}
return $moments;
}
/**
* Whether a backup has to start for $moment. One that was made since then is enough, also when
* several moments went by during a long backup. A moment from before the schedule was set does
* not count: choosing a time must not start a backup right away.
*/
public static function isDue(int $moment, int $since, ?int $lastDoneStarted, int $attempts, ?int $lastAttemptFinished, int $now): bool {
if ($moment < $since || ($lastDoneStarted !== null && $lastDoneStarted >= $moment) || $attempts >= self::MAX_ATTEMPTS) {
return false;
}
return $attempts === 0 || $now - (int)$lastAttemptFinished >= self::RETRY_AFTER;
}
}
+51 -1
View File
@@ -11,6 +11,7 @@ use OCA\NgBackup\AppInfo\Application;
use OCA\NgBackup\Db\RunMapper; use OCA\NgBackup\Db\RunMapper;
use OCA\NgBackup\Db\SnapshotMapper; use OCA\NgBackup\Db\SnapshotMapper;
use OCA\NgBackup\Db\Target; use OCA\NgBackup\Db\Target;
use OCA\NgBackup\Repository\Repository;
use OCP\IAppConfig; use OCP\IAppConfig;
use OCP\Lock\ILockingProvider; use OCP\Lock\ILockingProvider;
use OCP\Lock\LockedException; use OCP\Lock\LockedException;
@@ -33,6 +34,54 @@ final class PruneService {
) { ) {
} }
/** Characters of a note on a snapshot (the column holds 1000) */
public const NOTE_MAX = 500;
/** Pin or unpin a snapshot: the retention policy leaves a pinned one alone. */
public function setPinned(Target $target, string $snapshotId, bool $pinned): void {
$changed = $this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $pinned): bool {
$was = in_array($snapshotId, $repo->pinnedSnapshots(), true);
$repo->setPinned($snapshotId, $pinned);
return $was !== $pinned;
});
$this->targets->syncSnapshot($target, $snapshotId, pinned: $pinned);
if ($changed) {
// Unpinning opens the way to removing it, so every administrator hears about that.
$this->alerts->securityEvent($pinned ? 'snapshot_pinned' : 'snapshot_unpinned', ['target' => $target->getName(), 'snapshot' => $snapshotId], !$pinned);
}
}
public function setNote(Target $target, string $snapshotId, string $note): void {
$note = trim((string)preg_replace('/[\x00-\x1f\x7f]+/', ' ', $note));
if (!mb_check_encoding($note, 'UTF-8')) {
throw new \InvalidArgumentException('A note must be valid UTF-8');
}
if (mb_strlen($note) > self::NOTE_MAX) {
throw new \InvalidArgumentException('A note has at most ' . self::NOTE_MAX . ' characters');
}
$this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $note): bool {
$repo->setSnapshotNote($snapshotId, $note);
return true;
});
$this->targets->syncSnapshot($target, $snapshotId, note: $note);
}
/**
* Write to the catalog next to a running backup or restore, which only takes it for a moment to
* commit. Not during a cleanup, which holds a repository opened earlier and writes from that.
*
* @param callable(Repository): bool $change
*/
private function changeCatalog(Target $target, callable $change): bool {
try {
return $this->leases->with(self::lockKey($target), LeaseService::SHARED, 300,
fn (): bool => $this->leases->with(self::catalogLockKey($target), LeaseService::EXCLUSIVE, 300,
fn (): bool => $change($this->targets->repository($target))));
} catch (LockedException) {
throw new \RuntimeException('A cleanup of ' . $target->getName() . ' or another change to its restore points is in progress; try again later');
}
}
public function untrash(Target $target, string $snapshotId): void { public function untrash(Target $target, string $snapshotId): void {
$lease = $this->lockExclusive($target, 300); $lease = $this->lockExclusive($target, 300);
try { try {
@@ -87,6 +136,7 @@ final class PruneService {
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now'))); $s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
$s->setFiles((int)($meta['stats']['files'] ?? 0)); $s->setFiles((int)($meta['stats']['files'] ?? 0));
$s->setBytes((int)($meta['stats']['bytes'] ?? 0)); $s->setBytes((int)($meta['stats']['bytes'] ?? 0));
$s->setNote($repo->snapshotNotes()[$snapshotId] ?? null);
try { try {
$this->snapshots->insert($s); $this->snapshots->insert($s);
} catch (\Throwable) { } catch (\Throwable) {
@@ -208,7 +258,7 @@ final class PruneService {
$meta = $repo->snapshotMeta($id); $meta = $repo->snapshotMeta($id);
$times[$id] = (int)strtotime((string)($meta['time'] ?? '@0')); $times[$id] = (int)strtotime((string)($meta['time'] ?? '@0'));
} }
$keep = $this->policy()->keep($times); $keep = $this->policy()->keepWithPinned($times, $repo->pinnedSnapshots());
$forget = array_values(array_diff(array_keys($times), array_keys($keep))); $forget = array_values(array_diff(array_keys($times), array_keys($keep)));
$actor = $scheduled ? 'retention policy' : $this->alerts->actor(); $actor = $scheduled ? 'retention policy' : $this->alerts->actor();
$purged = []; $purged = [];
+23 -1
View File
@@ -151,6 +151,12 @@ final class RestoreFullService {
// The database cannot go into this installation; the files can. // The database cannot go into this installation; the files can.
$check['dbUnreadable'] = 'the database cannot be restored here: ' . $this->describeBlockers($check); $check['dbUnreadable'] = 'the database cannot be restored here: ' . $this->describeBlockers($check);
} }
if ($interrupted) {
// A restore that was killed leaves its leases, for up to two hours: they would keep
// the first backup of the recovered server waiting after this restore completes.
// In maintenance mode, on a partly restored instance, nothing else reads the location.
$this->leases->dropShared(PruneService::lockKey($target));
}
$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time()); $appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check, $salvage); $result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check, $salvage);
$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY); $appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
@@ -175,6 +181,8 @@ final class RestoreFullService {
if ($dbError === null) { if ($dbError === null) {
try { try {
$this->restoreDatabase($repo, $meta, $manifest, $dbReport, $beat, $skipMissingTables); $this->restoreDatabase($repo, $meta, $manifest, $dbReport, $beat, $skipMissingTables);
} catch (DatabaseAlreadyRestoredException $e) {
throw $e;
} catch (\Throwable $e) { } catch (\Throwable $e) {
if (!$salvage) { if (!$salvage) {
throw $e; throw $e;
@@ -201,6 +209,8 @@ final class RestoreFullService {
try { try {
/** @var array<string, mixed> $backupConfig */ /** @var array<string, mixed> $backupConfig */
$backupConfig = $this->fetchBackupConfig($target, $snapshotId, $salvage); $backupConfig = $this->fetchBackupConfig($target, $snapshotId, $salvage);
} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
throw $e; // nowhere to put it: not a config.php that cannot be read
} catch (\Throwable $e) { } catch (\Throwable $e) {
if (!$salvage) { if (!$salvage) {
throw $e; throw $e;
@@ -259,8 +269,20 @@ final class RestoreFullService {
$dumper = new DbDumper($this->db->getInner(), $prefix); $dumper = new DbDumper($this->db->getInner(), $prefix);
try { try {
$dumper->restore($repo, $manifest, $dbReport, $beat, $skipMissingTables); $dumper->restore($repo, $manifest, $dbReport, $beat, $skipMissingTables);
} finally { } catch (\Throwable $e) {
try {
$this->restoreOwnAppConfig($ownAppConfig); $this->restoreOwnAppConfig($ownAppConfig);
} catch (\Throwable) {
// rolled back, so they are still there: the error that brought us here is the one to report
}
throw $e;
}
try {
$this->restoreOwnAppConfig($ownAppConfig);
} catch (\Throwable $e) {
// The old server's database is in place by now: not a case for "the database stays as
// the fresh installation made it" under --salvage. A run again restores these rows.
throw new DatabaseAlreadyRestoredException('The database was restored, but NG Backup could not put back its own settings: ' . $e->getMessage(), 0, $e);
} }
} }
+53 -5
View File
@@ -61,7 +61,8 @@ final class RestoreService {
$prefix = trim($path, '/'); $prefix = trim($path, '/');
$prefix = $prefix === '' ? '' : $prefix . '/'; $prefix = $prefix === '' ? '' : $prefix . '/';
$children = []; $children = [];
foreach ($this->targets->repository($target)->entries($snapshotId) as $e) { $repo = $this->targets->repository($target);
foreach ($repo->entries($snapshotId) as $e) {
if ($prefix !== '' && !str_starts_with($e['p'], $prefix)) { if ($prefix !== '' && !str_starts_with($e['p'], $prefix)) {
continue; continue;
} }
@@ -75,6 +76,12 @@ final class RestoreService {
} }
$children[$name] = $c; $children[$name] = $c;
} }
foreach ($repo->emptyDirs($snapshotId) as [$dir]) { // read along with the entries
if ($prefix === '' || str_starts_with($dir, $prefix)) {
$name = explode('/', substr($dir, strlen($prefix)), 2)[0];
$children[$name] ??= ['name' => $name, 'type' => 'dir', 'size' => 0, 'mtime' => null, 'files' => 0];
}
}
ksort($children, SORT_STRING); ksort($children, SORT_STRING);
return array_values($children); return array_values($children);
} }
@@ -123,7 +130,17 @@ final class RestoreService {
$entries[substr($e['p'], strlen($source) + 1)] = $e; $entries[substr($e['p'], strlen($source) + 1)] = $e;
} }
} }
if ($entries === []) { // Folders without anything in them, relative to $source ('' when $source itself is one).
/** @var array<array-key, int> $emptyDirs modification times; a folder called "2026" is an integer key */
$emptyDirs = [];
foreach ($single ? [] : $repo->emptyDirs($snapshotId) as [$dir, $mtime]) {
if ($dir === $source) {
$emptyDirs[''] = $mtime;
} elseif (str_starts_with($dir, $source . '/')) {
$emptyDirs[substr($dir, strlen($source) + 1)] = $mtime;
}
}
if ($entries === [] && $emptyDirs === []) {
throw new NotFoundException("Nothing at $source in snapshot $snapshotId"); throw new NotFoundException("Nothing at $source in snapshot $snapshotId");
} }
@@ -134,9 +151,21 @@ final class RestoreService {
} elseif ($mode !== self::MODE_MERGE && $mode !== self::MODE_REPLACE) { } elseif ($mode !== self::MODE_MERGE && $mode !== self::MODE_REPLACE) {
throw new \InvalidArgumentException("Unknown mode $mode"); throw new \InvalidArgumentException("Unknown mode $mode");
} }
$destExisted = $parentRel === '' || $userFolder->nodeExists($parentRel);
$dest = $this->ensureFolder($userFolder, $parentRel); $dest = $this->ensureFolder($userFolder, $parentRel);
$names = $single ? [basename($rel) => $entries['']] : $entries; $names = $single ? [basename($rel) => $entries['']] : $entries;
// The folders with something in them that this restore creates get their time back; one that
// was already there keeps its own, as its content changes now. Noted before anything is written.
/** @var array<array-key, int> $newFolders a folder called "2026" is an integer key */
$newFolders = [];
foreach ($single ? [] : $repo->folderTimes($snapshotId) as [$dir, $mtime]) {
$relative = $dir === $source ? '' : (str_starts_with($dir, $source . '/') ? substr($dir, strlen($source) + 1) : null);
if ($relative !== null && ($relative === '' ? !$destExisted : !$dest->nodeExists($relative))) {
$newFolders[$relative] = $mtime;
}
}
$this->cleaned = []; $this->cleaned = [];
$restored = 0; $restored = 0;
foreach ($names as $relPath => $entry) { foreach ($names as $relPath => $entry) {
@@ -156,9 +185,28 @@ final class RestoreService {
} }
} }
foreach ($emptyDirs as $dir => $mtime) {
$dir = (string)$dir;
$existed = $dir === '' ? $destExisted : $dest->nodeExists($dir); // one that is there keeps its own time
$folder = $this->ensureFolder($dest, $dir);
if (!$existed) {
$folder->touch($mtime);
}
if ($progress !== null) {
$progress($restored, count($names), $dir); // keeps the lease alive too
}
}
$trashed = 0; $trashed = 0;
if ($mode === self::MODE_REPLACE && !$single) { if ($mode === self::MODE_REPLACE && !$single) {
$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names))); $trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)) + $emptyDirs);
}
// Last and deepest first: a change in a folder also touches the folders above it.
foreach (array_reverse($newFolders, true) as $dir => $mtime) {
$this->ensureFolder($dest, (string)$dir)->touch($mtime);
if ($progress !== null) {
$progress($restored, count($names), (string)$dir); // keeps the lease alive too
}
} }
return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())]; return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())];
} }
@@ -178,7 +226,7 @@ final class RestoreService {
throw new \InvalidArgumentException('Invalid path: ' . $prefix); throw new \InvalidArgumentException('Invalid path: ' . $prefix);
} }
if (!is_dir($directory) && !mkdir($directory, 0750, true) && !is_dir($directory)) { if (!is_dir($directory) && !mkdir($directory, 0750, true) && !is_dir($directory)) {
throw new \RuntimeException("Cannot create $directory"); throw new \OCA\NgBackup\Repository\RestoreTargetException("Cannot create $directory (disk full, or no permission?)");
} }
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $prefix, $directory, $stripPrefix, $heartbeat, &$skipped, $keepUnchanged): int { return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $prefix, $directory, $stripPrefix, $heartbeat, &$skipped, $keepUnchanged): int {
$own = LeaseService::throttled($refresh); $own = LeaseService::throttled($refresh);
@@ -262,7 +310,7 @@ final class RestoreService {
foreach ($folder->getDirectoryListing() as $node) { foreach ($folder->getDirectoryListing() as $node) {
$rel = $prefix === '' ? $node->getName() : $prefix . '/' . $node->getName(); $rel = $prefix === '' ? $node->getName() : $prefix . '/' . $node->getName();
if ($node instanceof Folder) { if ($node instanceof Folder) {
$hasKept = false; $hasKept = isset($keep[$rel]); // a folder that was empty in the snapshot
foreach (array_keys($keep) as $k) { foreach (array_keys($keep) as $k) {
if (str_starts_with((string)$k, $rel . '/')) { if (str_starts_with((string)$k, $rel . '/')) {
$hasKept = true; $hasKept = true;
+13
View File
@@ -28,6 +28,19 @@ final class RetentionPolicy {
return ['last' => $this->last, 'daily' => $this->daily, 'weekly' => $this->weekly, 'monthly' => $this->monthly]; return ['last' => $this->last, 'daily' => $this->daily, 'weekly' => $this->weekly, 'monthly' => $this->monthly];
} }
/**
* keep() for the snapshots that are not pinned, plus the pinned ones. A pinned snapshot does not
* take the place of a daily, weekly or monthly one.
*
* @param array<string, int> $snapshots id => unix time
* @param list<string> $pinned
* @return array<string, list<string>>
*/
public function keepWithPinned(array $snapshots, array $pinned): array {
$pins = array_intersect_key($snapshots, array_flip($pinned));
return $this->keep(array_diff_key($snapshots, $pins)) + array_fill_keys(array_keys($pins), ['pinned']);
}
/** /**
* @param array<string, int> $snapshots id => unix time * @param array<string, int> $snapshots id => unix time
* @return array<string, list<string>> id => reasons it is kept (ids not in the result are removed) * @return array<string, list<string>> id => reasons it is kept (ids not in the result are removed)
+8 -25
View File
@@ -13,10 +13,11 @@ use OCP\IConfig;
use OCP\IGroupManager; use OCP\IGroupManager;
/** /**
* The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a * The time zone times are shown in, and the daily time from before each location had its own
* zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone * schedule (see LocationSchedule). Nextcloud runs PHP in UTC, so a time without a zone meant UTC
* is converted once to the zone of the server or of an administrator, at today's offset: it then * (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone is converted once
* shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer). * to the zone of the server or of an administrator, at today's offset: it then shows as it
* effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
*/ */
final class Schedule { final class Schedule {
public const TIME_KEY = 'schedule_time'; public const TIME_KEY = 'schedule_time';
@@ -44,27 +45,9 @@ final class Schedule {
return ['time' => $time, 'timezone' => $zone]; return ['time' => $time, 'timezone' => $zone];
} }
/** @throws \InvalidArgumentException for a malformed time or an unknown zone */ /** @return non-empty-string the zone of a location's schedule, or else this one */
public function set(string $time, string $zone): void { public function zoneOr(?string $zone): string {
if ($time !== '' && !self::validTime($time)) { return $zone !== null && self::validZone($zone) ? $zone : $this->get()['timezone'];
throw new \InvalidArgumentException('Use HH:MM or empty');
}
if (!self::validZone($zone)) {
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
}
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
}
/** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */
public function todayAt(?int $now = null): ?int {
['time' => $time, 'timezone' => $zone] = $this->get();
if ($time === '' || !self::validTime($time)) {
return null;
}
$tz = new \DateTimeZone($zone);
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d');
return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp();
} }
/** /**
+46
View File
@@ -35,6 +35,7 @@ final class TargetService {
private \OCA\NgBackup\Db\DbIndexCache $indexCache, private \OCA\NgBackup\Db\DbIndexCache $indexCache,
private \OCA\NgBackup\Db\AppConfigCatalogAnchor $anchor, private \OCA\NgBackup\Db\AppConfigCatalogAnchor $anchor,
private SnapshotMapper $snapshots, private SnapshotMapper $snapshots,
private AlertService $alerts,
) { ) {
} }
@@ -78,6 +79,11 @@ final class TargetService {
$target->setBasePath(trim($basePath, '/')); $target->setBasePath(trim($basePath, '/'));
$target->setRepositoryId($repo->id()); $target->setRepositoryId($repo->id());
$target->setCreatedAt(time()); $target->setCreatedAt(time());
// Like the location before it: a backup scheme is usually one scheme.
$first = $this->mapper->findAll()[0] ?? null;
$target->setSchedule($first?->getSchedule() ?? '');
$target->setScheduleZone($first?->getScheduleZone());
$target->setScheduleSince(time());
$target = $this->mapper->insert($target); $target = $this->mapper->insert($target);
if (!$created) { if (!$created) {
// Disaster recovery: occ backup:list reads this local cache, not the repository // Disaster recovery: occ backup:list reads this local cache, not the repository
@@ -208,7 +214,25 @@ final class TargetService {
return $result; return $result;
} }
/** Update the local copy of a snapshot's note or pin, if the snapshot is listed here at all. */
public function syncSnapshot(Target $target, string $snapshotId, ?string $note = null, ?bool $pinned = null): void {
try {
$s = $this->snapshots->findOne($target->getId(), $snapshotId);
} catch (DoesNotExistException) {
return;
}
if ($note !== null) {
$s->setNote($note !== '' ? $note : null);
}
if ($pinned !== null) {
$s->setPinned($pinned);
}
$this->snapshots->update($s);
}
private function adoptSnapshots(Target $target, Repository $repo): void { private function adoptSnapshots(Target $target, Repository $repo): void {
$notes = $repo->snapshotNotes();
$pinned = array_flip($repo->pinnedSnapshots());
foreach ($repo->listSnapshots() as $snapshotId) { foreach ($repo->listSnapshots() as $snapshotId) {
try { try {
$this->snapshots->findOne($target->getId(), $snapshotId); $this->snapshots->findOne($target->getId(), $snapshotId);
@@ -227,6 +251,8 @@ final class TargetService {
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now'))); $s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
$s->setFiles((int)($meta['stats']['files'] ?? 0)); $s->setFiles((int)($meta['stats']['files'] ?? 0));
$s->setBytes((int)($meta['stats']['bytes'] ?? 0)); $s->setBytes((int)($meta['stats']['bytes'] ?? 0));
$s->setNote($notes[$snapshotId] ?? null);
$s->setPinned(isset($pinned[$snapshotId]));
$this->snapshots->insert($s); $this->snapshots->insert($s);
} catch (\Throwable $e) { } catch (\Throwable $e) {
Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]); Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]);
@@ -248,6 +274,26 @@ final class TargetService {
} }
/** Size limit for the data stored on this location (null = none). */ /** Size limit for the data stored on this location (null = none). */
/**
* @param string $schedule see LocationSchedule
* @throws \InvalidArgumentException
*/
public function setSchedule(Target $target, string $schedule, string $zone): Target {
if (!Schedule::validZone($zone)) {
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
}
$new = LocationSchedule::normalize($schedule);
$turnedOff = $new === '' && (string)$target->getSchedule() !== '';
$target->setSchedule($new);
$target->setScheduleZone($zone);
$target->setScheduleSince(time());
$target = $this->mapper->update($target);
if ($turnedOff) {
$this->alerts->securityEvent('schedule_off', ['target' => $target->getName()]);
}
return $target;
}
public function setMaxBytes(Target $target, ?int $maxBytes): Target { public function setMaxBytes(Target $target, ?int $maxBytes): Target {
$target->setMaxBytes($maxBytes); $target->setMaxBytes($maxBytes);
return $this->mapper->update($target); return $this->mapper->update($target);
+17
View File
@@ -186,6 +186,7 @@
<code><![CDATA[$this->db]]></code> <code><![CDATA[$this->db]]></code>
<code><![CDATA[$this->db]]></code> <code><![CDATA[$this->db]]></code>
<code><![CDATA[$this->db]]></code> <code><![CDATA[$this->db]]></code>
<code><![CDATA[$this->db]]></code>
<code><![CDATA[TransactionIsolationLevel]]></code> <code><![CDATA[TransactionIsolationLevel]]></code>
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code> <code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code> <code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
@@ -299,6 +300,22 @@
<code><![CDATA[$t]]></code> <code><![CDATA[$t]]></code>
</UndefinedDocblockClass> </UndefinedDocblockClass>
</file> </file>
<file src="lib/Migration/Version000008Date20261107010000.php">
<UndefinedDocblockClass>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
</UndefinedDocblockClass>
</file>
<file src="lib/Migration/Version000007Date20261107000000.php">
<UndefinedDocblockClass>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
</UndefinedDocblockClass>
</file>
<file src="lib/Migration/Version000005Date20261002040000.php"> <file src="lib/Migration/Version000005Date20261002040000.php">
<UndefinedDocblockClass> <UndefinedDocblockClass>
<code><![CDATA[$t]]></code> <code><![CDATA[$t]]></code>
+4 -2
View File
@@ -21,7 +21,7 @@ export default {
addSlot: (passphrase, label) => call('post', '/key/slots', { passphrase, label }), addSlot: (passphrase, label) => call('post', '/key/slots', { passphrase, label }),
replaceSlot: (slot, passphrase, label) => call('put', `/key/slots/${slot}`, { passphrase, label }), replaceSlot: (slot, passphrase, label) => call('put', `/key/slots/${slot}`, { passphrase, label }),
removeSlot: (slot) => call('delete', `/key/slots/${slot}`), removeSlot: (slot) => call('delete', `/key/slots/${slot}`),
trash: (targetId) => call('get', `/targets/${targetId}/trash`), trash: (targetId, after = '') => call('get', `/targets/${targetId}/trash`, { after }),
untrash: (targetId, snapshotId) => call('post', `/targets/${targetId}/trash/${snapshotId}`), untrash: (targetId, snapshotId) => call('post', `/targets/${targetId}/trash/${snapshotId}`),
kitUrl: () => url('/key/kit'), kitUrl: () => url('/key/kit'),
confirmKit: (accepted, phrase, code) => call('post', '/key/confirm', { accepted, phrase, code }), confirmKit: (accepted, phrase, code) => call('post', '/key/confirm', { accepted, phrase, code }),
@@ -34,8 +34,10 @@ export default {
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }), setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }), startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
tick: () => call('post', '/tick'), tick: () => call('post', '/tick'),
setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }), setTargetSchedule: (id, schedule, timezone) => call('put', `/targets/${id}/schedule`, { schedule, timezone }),
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`), snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
setSnapshotNote: (targetId, snapshotId, note) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/note`, { note }),
setSnapshotPinned: (targetId, snapshotId, pinned) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/pin`, { pinned }),
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }), browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }), startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
} }
+8 -24
View File
@@ -5,10 +5,7 @@
<template> <template>
<NcSettingsSection :name="t('ng_backup', 'Backups')" <NcSettingsSection :name="t('ng_backup', 'Backups')"
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')"> :description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
<div class="ngb-row"> <LocationScheduleRow v-for="target in status.targets" :key="target.id" :target="target" @changed="emit('changed')" @error="error = $event" />
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, {zone}, empty = off)', { zone: browserZone })" />
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
</div>
<div class="ngb-row"> <div class="ngb-row">
<NcTextField v-model="label" class="ngb-label" :maxlength="100" :label="t('ng_backup', 'Label (optional)')" <NcTextField v-model="label" class="ngb-label" :maxlength="100" :label="t('ng_backup', 'Label (optional)')"
:placeholder="t('ng_backup', 'e.g. before the upgrade to Nextcloud 35')" /> :placeholder="t('ng_backup', 'e.g. before the upgrade to Nextcloud 35')" />
@@ -37,7 +34,7 @@
<td>{{ formatDateTime(run.startedAt) }}</td> <td>{{ formatDateTime(run.startedAt) }}</td>
<td>{{ targetName(run.targetId) }}</td> <td>{{ targetName(run.targetId) }}</td>
<td>{{ run.kind }}</td> <td>{{ run.kind }}</td>
<td>{{ run.snapshot ? formatDateTime(run.snapshot.createdAt) + (run.snapshot.label ? ' – ' + run.snapshot.label : '') : '' }}</td> <td>{{ run.snapshot ? (run.snapshot.pinned ? '📌 ' : '') + formatDateTime(run.snapshot.createdAt) + (run.snapshot.label ? ' – ' + run.snapshot.label : '') + (run.snapshot.note ? ' – ' + run.snapshot.note : '') : '' }}</td>
<td :class="'ngb-' + run.status">{{ runStatus(run) }}{{ run.error ? ': ' + run.error : '' }}</td> <td :class="'ngb-' + run.status">{{ runStatus(run) }}{{ run.error ? ': ' + run.error : '' }}</td>
<td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td> <td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
<td>{{ run.stats?.files ? formatSize(run.stats.files.bytes) : '' }}</td> <td>{{ run.stats?.files ? formatSize(run.stats.files.bytes) : '' }}</td>
@@ -60,24 +57,19 @@
</template> </template>
<script setup> <script setup>
import { ref, watch } from 'vue' import { ref } from 'vue'
import { n, t } from '@nextcloud/l10n' import { n, t } from '@nextcloud/l10n'
import NcButton from '@nextcloud/vue/components/NcButton' import NcButton from '@nextcloud/vue/components/NcButton'
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard' import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection' import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
import NcTextField from '@nextcloud/vue/components/NcTextField' import NcTextField from '@nextcloud/vue/components/NcTextField'
import LocationScheduleRow from './LocationScheduleRow.vue'
import api from '../api.js' import api from '../api.js'
import { formatDateTime, formatSize, toZone } from '../format.js' import { formatDateTime, formatSize } from '../format.js'
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } }) const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
const emit = defineEmits(['changed', 'show-all-runs']) const emit = defineEmits(['changed', 'show-all-runs'])
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
// zone is shown in this one, so it reads as the moment it runs here.
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s
const schedule = ref(shown(props.status.schedule))
const error = ref('') const error = ref('')
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
const label = ref('') const label = ref('')
@@ -97,6 +89,9 @@ const runStatus = (run) => {
if (run.status !== 'running') { if (run.status !== 'running') {
return run.status return run.status
} }
if (run.queued) {
return t('ng_backup', 'queued, after the backup that is running now')
}
const detail = run.phase + percent(run) const detail = run.phase + percent(run)
if (run.waitingFor) { if (run.waitingFor) {
return t('ng_backup', 'waiting for background jobs ({detail}, last step {minutes} min ago)', { detail, minutes: Math.floor(run.waitingFor / 60) }) return t('ng_backup', 'waiting for background jobs ({detail}, last step {minutes} min ago)', { detail, minutes: Math.floor(run.waitingFor / 60) })
@@ -116,21 +111,10 @@ async function start(target) {
error.value = e.message error.value = e.message
} }
} }
async function saveSchedule() {
error.value = ''
try {
await api.setSchedule(schedule.value.trim(), browserZone)
emit('changed')
} catch (e) {
error.value = e.message
}
}
</script> </script>
<style scoped> <style scoped>
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; } .ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-time { max-width: 260px; }
.ngb-label { max-width: 360px; } .ngb-label { max-width: 360px; }
.ngb-table-wrap { overflow-x: auto; } .ngb-table-wrap { overflow-x: auto; }
.ngb-table { width: 100%; border-collapse: collapse; } .ngb-table { width: 100%; border-collapse: collapse; }
+49 -8
View File
@@ -6,8 +6,18 @@
<NcSettingsSection :name="t('ng_backup', 'Bin')" <NcSettingsSection :name="t('ng_backup', 'Bin')"
:description="t('ng_backup', 'Restore points that were removed, most of them by the retention policy. They can be brought back until they are deleted permanently.')"> :description="t('ng_backup', 'Restore points that were removed, most of them by the retention policy. They can be brought back until they are deleted permanently.')">
<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard> <NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
<NcLoadingIcon v-if="loading" :size="32" /> <div v-if="reading" class="ngb-reading" role="status">
<NcNoteCard v-else-if="empty" type="info">{{ t('ng_backup', 'The bin is empty.') }}</NcNoteCard> <template v-if="reading.total === null">
<NcLoadingIcon :size="20" />
<span>{{ t('ng_backup', 'Connecting to {location} …', { location: reading.name }) }}</span>
</template>
<template v-else>
<span>{{ t('ng_backup', 'Reading removed restore points from {location}: {done} of {total}', { location: reading.name, done: reading.done, total: reading.total }) }}</span>
<NcProgressBar :value="reading.total ? Math.round(100 * reading.done / reading.total) : 0" />
</template>
<p class="ngb-muted">{{ t('ng_backup', 'This can take a while: every removed restore point is read from the location.') }}</p>
</div>
<NcNoteCard v-else-if="empty && !error" type="info">{{ t('ng_backup', 'The bin is empty.') }}</NcNoteCard>
<template v-for="target in status.targets" :key="'trash' + target.id"> <template v-for="target in status.targets" :key="'trash' + target.id">
<div v-if="trash[target.id]?.length" class="ngb-trash"> <div v-if="trash[target.id]?.length" class="ngb-trash">
@@ -39,12 +49,13 @@
</template> </template>
<script setup> <script setup>
import { computed, reactive, ref, watch } from 'vue' import { computed, onBeforeUnmount, reactive, ref, watch } from 'vue'
import { n, t } from '@nextcloud/l10n' import { n, t } from '@nextcloud/l10n'
import { confirmPassword } from '@nextcloud/password-confirmation' import { confirmPassword } from '@nextcloud/password-confirmation'
import NcButton from '@nextcloud/vue/components/NcButton' import NcButton from '@nextcloud/vue/components/NcButton'
import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon' import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard' import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
import NcProgressBar from '@nextcloud/vue/components/NcProgressBar'
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection' import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
import api from '../api.js' import api from '../api.js'
import { formatDate, formatDateTime } from '../format.js' import { formatDate, formatDateTime } from '../format.js'
@@ -56,21 +67,49 @@ const emit = defineEmits(['changed'])
const RETENTION = 'retention policy' const RETENTION = 'retention policy'
const retentionInfo = ref(false) const retentionInfo = ref(false)
const trash = reactive({}) const trash = reactive({})
const loading = ref(true) const reading = ref({ name: '', total: null, done: 0 }) // the location being read; null when done
const error = ref('') const error = ref('')
const empty = computed(() => props.status.targets.every(target => !trash[target.id]?.length)) const empty = computed(() => props.status.targets.every(target => !trash[target.id]?.length))
let current = 0 // the newest load; an older one stops (another took over, or the tab was left)
onBeforeUnmount(() => { current++ })
async function loadTarget(target, load) {
reading.value = { name: target.name, total: null, done: 0 }
const items = []
for (;;) {
const page = await api.trash(target.id, items.length ? items[items.length - 1].id : '')
if (load !== current) {
return
}
items.push(...page.items)
reading.value = { name: target.name, total: page.total, done: Math.min(items.length, page.total) }
if (!page.items.length || items.length >= page.total) {
break
}
}
trash[target.id] = items.sort((a, b) => String(b.time ?? '').localeCompare(String(a.time ?? '')))
}
async function loadTrash() { async function loadTrash() {
const load = ++current
const failed = []
for (const target of props.status.targets) { for (const target of props.status.targets) {
try { try {
trash[target.id] = await api.trash(target.id) await loadTarget(target, load)
} catch (e) { } catch (e) {
trash[target.id] = [] // keep what was shown: a location that cannot be reached does not mean its bin is empty
failed.push(target.name + (e?.message ? ': ' + e.message : ''))
}
if (load !== current) {
return
} }
} }
loading.value = false error.value = failed.length ? t('ng_backup', 'The bin could not be read from {locations}', { locations: failed.join('; ') }) : ''
reading.value = null
} }
watch(() => props.status.targets.map(tg => tg.id).join(','), loadTrash, { immediate: true }) // Again when a run ends: a cleanup moves restore points here.
watch(() => props.status.targets.map(tg => tg.id).join(',') + '|' + (props.status.runs ?? []).filter(run => run.status !== 'running').map(run => run.id).join(','), loadTrash, { immediate: true })
async function untrash(target, item) { async function untrash(target, item) {
error.value = '' error.value = ''
@@ -88,6 +127,8 @@ async function untrash(target, item) {
<style scoped> <style scoped>
.ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); } .ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); }
.ngb-muted { color: var(--color-text-maxcontrast); } .ngb-muted { color: var(--color-text-maxcontrast); }
.ngb-reading { margin: 12px 0; display: flex; flex-direction: column; gap: 6px; max-width: 600px; }
.ngb-reading p { margin: 0; }
.ngb-trash { margin: 12px 0; } .ngb-trash { margin: 12px 0; }
.ngb-trash h3 { font-weight: bold; } .ngb-trash h3 { font-weight: bold; }
</style> </style>
+77
View File
@@ -0,0 +1,77 @@
<!--
- SPDX-FileCopyrightText: 2026 Dillard Blom
- SPDX-License-Identifier: AGPL-3.0-or-later
-->
<template>
<div class="ngb-row">
<strong class="ngb-name">{{ target.name }}</strong>
<NcSelect v-model="kind" :options="kinds" label="label" :input-label="t('ng_backup', 'Schedule')" :clearable="false" />
<NcTextField v-if="kind.id !== 'off'" v-model="time" class="ngb-time" :label="kind.id === 'daily' ? t('ng_backup', 'At (HH:MM, {zone})', { zone: browserZone }) : t('ng_backup', 'From (HH:MM, {zone})', { zone: browserZone })" />
<NcTextField v-if="kind.id === 'hours'" v-model="hours" class="ngb-hours" type="number" min="1" max="23" :label="t('ng_backup', 'Every … hours')" />
<NcButton :disabled="busy || wanted === current" @click="save">{{ t('ng_backup', 'Save') }}</NcButton>
</div>
</template>
<script setup>
import { computed, ref, watch } from 'vue'
import { t } from '@nextcloud/l10n'
import NcButton from '@nextcloud/vue/components/NcButton'
import NcSelect from '@nextcloud/vue/components/NcSelect'
import NcTextField from '@nextcloud/vue/components/NcTextField'
import api from '../api.js'
import { toZone } from '../format.js'
const props = defineProps({ target: { type: Object, required: true } })
const emit = defineEmits(['changed', 'error'])
const kinds = [
{ id: 'off', label: t('ng_backup', 'Off') },
{ id: 'daily', label: t('ng_backup', 'Once a day') },
{ id: 'hours', label: t('ng_backup', 'Every few hours') },
]
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
// zone is shown in this one, so it reads as the moment it runs here.
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
const kind = ref(kinds[0])
const time = ref('02:00')
const hours = ref('6')
const busy = ref(false)
const current = computed(() => {
const [at, every] = props.target.schedule.split('/')
return at === '' ? '' : (props.target.scheduleZone !== browserZone ? toZone(at, props.target.scheduleZone, browserZone) : at) + (every ? '/' + every : '')
})
const wanted = computed(() => {
if (kind.value.id === 'off') {
return ''
}
return time.value.trim() + (kind.value.id === 'hours' ? '/' + String(hours.value).trim() : '')
})
watch(current, (schedule) => {
const [at, every] = schedule.split('/')
kind.value = kinds[at === '' ? 0 : (every ? 2 : 1)]
time.value = at === '' ? time.value : at
hours.value = every ?? hours.value
}, { immediate: true })
async function save() {
busy.value = true
emit('error', '')
try {
await api.setTargetSchedule(props.target.id, wanted.value, browserZone)
emit('changed')
} catch (e) {
emit('error', e.message)
} finally {
busy.value = false
}
}
</script>
<style scoped>
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-name { min-width: 120px; align-self: center; }
.ngb-time { max-width: 220px; }
.ngb-hours { max-width: 120px; }
</style>
+37 -2
View File
@@ -11,6 +11,13 @@
</div> </div>
<template v-if="snapshot"> <template v-if="snapshot">
<div class="ngb-row ngb-marks">
<NcTextField v-model="note" class="ngb-note" :maxlength="500" :label="t('ng_backup', 'Note (can be changed later)')" />
<NcButton :disabled="marking || note.trim() === (snapshot.note ?? '')" @click="saveNote">{{ t('ng_backup', 'Save note') }}</NcButton>
<NcCheckboxRadioSwitch :model-value="snapshot.pinned" :disabled="marking" @update:model-value="savePinned">
{{ t('ng_backup', 'Pin: the retention policy does not remove this restore point') }}
</NcCheckboxRadioSwitch>
</div>
<div class="ngb-crumbs"> <div class="ngb-crumbs">
<NcButton variant="tertiary" @click="open('data')">data</NcButton> <NcButton variant="tertiary" @click="open('data')">data</NcButton>
<template v-for="(crumb, i) in crumbs" :key="crumb.path"> <template v-for="(crumb, i) in crumbs" :key="crumb.path">
@@ -49,6 +56,7 @@ import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard' import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
import NcSelect from '@nextcloud/vue/components/NcSelect' import NcSelect from '@nextcloud/vue/components/NcSelect'
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection' import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
import NcTextField from '@nextcloud/vue/components/NcTextField'
import { confirmPassword } from '@nextcloud/password-confirmation' import { confirmPassword } from '@nextcloud/password-confirmation'
import '@nextcloud/password-confirmation/style.css' import '@nextcloud/password-confirmation/style.css'
import api from '../api.js' import api from '../api.js'
@@ -74,7 +82,10 @@ const modes = [
{ id: 'replace', label: t('ng_backup', 'Into the original place, overwrite and move files that are not in the backup to the trash') }, { id: 'replace', label: t('ng_backup', 'Into the original place, overwrite and move files that are not in the backup to the trash') },
] ]
const snapshotLabel = (s) => formatDateTime(s.createdAt) + (s.label ? ' – ' + s.label : '') const note = ref('')
const marking = ref(false)
const snapshotLabel = (s) => (s.pinned ? '📌 ' : '') + formatDateTime(s.createdAt) + (s.label ? ' – ' + s.label : '') + (s.note ? ' – ' + s.note : '')
const join = (name) => path.value + '/' + name const join = (name) => path.value + '/' + name
const restorable = (p) => /^data\/[^/]+\/files(\/.*)?$/.test(p) const restorable = (p) => /^data\/[^/]+\/files(\/.*)?$/.test(p)
const crumbs = computed(() => { const crumbs = computed(() => {
@@ -88,7 +99,29 @@ watch(target, async (tg) => {
snapshots.value = list.map(s => ({ ...s, displayLabel: snapshotLabel(s) })) snapshots.value = list.map(s => ({ ...s, displayLabel: snapshotLabel(s) }))
snapshot.value = snapshots.value[0] ?? null snapshot.value = snapshots.value[0] ?? null
}, { immediate: true }) }, { immediate: true })
watch(snapshot, () => { selected.value = ''; open('data') }) watch(snapshot, (s) => { selected.value = ''; note.value = s?.note ?? ''; open('data') })
// Changes the snapshot in place: choosing another one, which opens its folders again, is not needed
async function mark(call, change) {
const s = snapshot.value
marking.value = true
error.value = message.value = ''
try {
await call(target.value.id, s.id)
change(s)
s.displayLabel = snapshotLabel(s)
emit('changed')
} catch (e) {
error.value = e.message
} finally {
marking.value = false
}
}
const saveNote = () => {
const text = note.value.trim()
return mark((targetId, id) => api.setSnapshotNote(targetId, id, text), (s) => { s.note = text })
}
const savePinned = (pinned) => mark((targetId, id) => api.setSnapshotPinned(targetId, id, pinned), (s) => { s.pinned = pinned })
async function open(p) { async function open(p) {
if (!snapshot.value) { if (!snapshot.value) {
@@ -125,6 +158,8 @@ async function restore() {
<style scoped> <style scoped>
.ngb-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 12px; } .ngb-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-row :deep(.ngb-snapshot) { min-width: min(360px, 100%); } .ngb-row :deep(.ngb-snapshot) { min-width: min(360px, 100%); }
.ngb-marks { align-items: flex-end; }
.ngb-note { max-width: 360px; }
.ngb-crumbs { display: flex; align-items: center; flex-wrap: wrap; } .ngb-crumbs { display: flex; align-items: center; flex-wrap: wrap; }
.ngb-entries li { display: flex; align-items: center; gap: 12px; min-height: 36px; } .ngb-entries li { display: flex; align-items: center; gap: 12px; min-height: 36px; }
.ngb-entries li.selected { background: var(--color-primary-element-light); border-radius: var(--border-radius); } .ngb-entries li.selected { background: var(--color-primary-element-light); border-radius: var(--border-radius); }
+22
View File
@@ -28,6 +28,28 @@ class TreeWalkerTest extends TestCase {
$this->assertSame(['Z', 'a', 'a-b', 'a.txt', 'b', 'skip'], array_values(array_unique(array_map(fn ($p) => explode('/', $p)[0], $paths)))); $this->assertSame(['Z', 'a', 'a-b', 'a.txt', 'b', 'skip'], array_values(array_unique(array_map(fn ($p) => explode('/', $p)[0], $paths))));
} }
public function testEmptyDirectoriesAreYieldedInOrderAndOnResume(): void {
$d = $this->tree();
mkdir("$d/a/empty");
mkdir("$d/a-hollow/inner", 0700, true);
symlink("$d/a.txt", "$d/skip/link");
mkdir("$d/only-link");
symlink("$d/a.txt", "$d/only-link/link");
$w = new TreeWalker(['' => $d]);
$all = array_keys(iterator_to_array($w->walk()));
$this->assertContains('a/empty', $all);
$this->assertContains('a-hollow/inner', $all);
$this->assertNotContains('a-hollow', $all, 'only the deepest');
$this->assertContains('only-link', $all, 'a link is not walked');
$this->assertNotContains('skip', $all);
$sorted = $all;
usort($sorted, [TreeWalker::class, 'compare']);
$this->assertSame($sorted, $all);
foreach ($all as $i => $cursor) {
$this->assertSame(array_slice($all, $i + 1), array_keys(iterator_to_array($w->walk($cursor))), "after $cursor");
}
}
public function testResumeAfterCursorYieldsTheRest(): void { public function testResumeAfterCursorYieldsTheRest(): void {
$w = new TreeWalker(['' => $this->tree()]); $w = new TreeWalker(['' => $this->tree()]);
$all = array_keys(iterator_to_array($w->walk())); $all = array_keys(iterator_to_array($w->walk()));
+34
View File
@@ -50,6 +50,40 @@ class CatalogTest extends TestCase {
$this->assertSame(4, $again->catalogGeneration()); $this->assertSame(4, $again->catalogGeneration());
} }
public function testTheTrashIsReadInPartsThatTogetherAreAll(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
for ($i = 0; $i < 3; $i++) {
$repo->forget($repo->backupDirectory($this->src)['snapshot']);
}
$again = $this->open();
$seen = [];
// A deadline already past: one entry per request, as on a slow location
$after = '';
for ($request = 0; count($seen) < 3 && $request < 5; $request++) {
$page = $again->trashPage($after, 0.0);
$this->assertSame(3, $page['total']);
$this->assertCount(1, $page['items']);
$seen += $page['items'];
$after = (string)array_key_last($page['items']);
}
$this->assertEqualsCanonicalizing(array_keys($again->trash()), array_keys($seen));
$this->assertSame(['total' => 3, 'items' => []], $again->trashPage($after, 0.0));
$this->assertCount(3, $again->trashPage('', microtime(true) + 60)['items']);
}
public function testAnEntryThatLeavesTheTrashBetweenPartsMakesNoneComeTwiceOrBeMissed(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
for ($i = 0; $i < 3; $i++) {
$repo->forget($repo->backupDirectory($this->src)['snapshot']);
}
$reader = $this->open();
$ids = array_keys($reader->trashPage('', microtime(true) + 60)['items']);
$first = $reader->trashPage('', 0.0)['items'];
$this->assertSame([$ids[0]], array_keys($first));
$this->open()->untrash($ids[0]); // another administrator, between two parts
$this->assertSame([$ids[1], $ids[2]], array_keys($this->open()->trashPage($ids[0], microtime(true) + 60)['items']));
}
public function testRollbackIsDetected(): void { public function testRollbackIsDetected(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor); $repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$repo->backupDirectory($this->src); $repo->backupDirectory($this->src);
+158
View File
@@ -0,0 +1,158 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Job\BackupRun;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/** Folders without files are recorded in the snapshot's meta record and come back on a restore. */
class EmptyDirsTest extends TestCase {
use TempDirTrait;
private LocalBackend $backend;
private KeyRing $keys;
private string $src;
protected function setUp(): void {
$this->backend = new LocalBackend($this->tempDir());
$this->keys = KeyRing::generate();
Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
$this->src = $this->tempDir();
mkdir("$this->src/data/alice/files/Docs", 0700, true);
mkdir("$this->src/data/alice/files/Hollow/Nested", 0700, true);
mkdir("$this->src/data/skipped/inner", 0700, true);
file_put_contents("$this->src/data/alice/files/Docs/note.txt", "hello\n");
touch("$this->src/data/alice/files/Hollow/Nested", 1700000000);
}
private function repo(): Repository {
return Repository::openWithKey($this->backend, $this->keys);
}
/** @param list<string> $exclude */
private function backup(array $exclude = [], float $seconds = 30.0): string {
$state = BackupRun::start(['data' => "$this->src/data"], null, '', $exclude);
for ($i = 0; $i < 500 && $state['phase'] !== 'commit'; $i++) {
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, microtime(true) + $seconds)), true);
}
$this->assertSame('commit', $state['phase']);
$this->repo()->recordSnapshot($state['snapshot']);
return $state['snapshot'];
}
public function testOnlyTheDeepestEmptyFolderIsRecordedAndFilesStayAsTheyAre(): void {
$snapshot = $this->backup();
$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], $paths);
$this->assertSame(['data/alice/files/Docs/note.txt'], array_column(iterator_to_array($this->repo()->entries($snapshot), false), 'p'));
$this->assertSame(2, $this->repo()->snapshotMeta($snapshot)['stats']['emptyDirs']);
}
public function testAFolderWithOnlyExcludedContentCountsAsEmpty(): void {
$snapshot = $this->backup(['data/skipped/inner']);
$this->assertContains('data/skipped', array_column($this->repo()->emptyDirs($snapshot), 0));
$this->assertNotContains('data/skipped/inner', array_column($this->repo()->emptyDirs($snapshot), 0));
}
public function testAResumedWalkRecordsEveryFolderOnce(): void {
for ($i = 0; $i < 30; $i++) {
mkdir("$this->src/data/many/d$i", 0700, true);
file_put_contents("$this->src/data/many/f$i", 'x');
}
$snapshot = $this->backup([], 0.0); // a step per entry
$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
$this->assertCount(32, $paths);
$this->assertSame(array_unique($paths), $paths);
}
public function testANameASnapshotCannotHoldIsReportedAndTheSnapshotStaysReadable(): void {
mkdir("$this->src/data/odd\\folder", 0700, true);
mkdir("$this->src/data/alice/files/line\nbreak", 0700, true);
file_put_contents("$this->src/data/alice/files/back\\slash.txt", 'x');
file_put_contents("$this->src/data/alice/files/\xff.txt", 'x');
$state = BackupRun::start(['data' => "$this->src/data"]);
for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
// A step per entry, state through JSON: as separate processes would
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
}
$this->assertSame('commit', $state['phase']);
$this->assertSame(4, $state['stats']['badNames'], 'each reported once, over several steps');
$this->assertArrayNotHasKey('unreadable', $state['stats'], 'not a problem with permissions');
$shown = BackupRun::warnings($state)['badNames'];
$this->assertContains('data/alice/files/line?break', $shown, 'no control character in a message');
$this->assertSame($shown, array_filter($shown, fn (string $name): bool => mb_check_encoding($name, 'UTF-8')));
$this->repo()->recordSnapshot($state['snapshot']);
$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], array_column($this->repo()->emptyDirs($state['snapshot']), 0));
$target = $this->tempDir();
$this->assertSame(1, $this->repo()->restore($state['snapshot'], $target));
$this->assertFileExists("$target/data/alice/files/Docs/note.txt");
}
public function testAnEmptyFolderThatIsThePrefixItselfIsTheTarget(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->repo()->restore($snapshot, $target, 'data/skipped/inner/', true);
$this->assertSame([], glob("$target/*"), 'not a folder "inner" inside the target');
}
public function testRestoreBringsThemBackWithTheirTime(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->assertSame(1, $this->repo()->restore($snapshot, $target));
$this->assertDirectoryExists("$target/data/alice/files/Hollow/Nested");
clearstatcache();
$this->assertSame(1700000000, filemtime("$target/data/alice/files/Hollow/Nested"));
$this->assertDirectoryExists("$target/data/skipped/inner");
}
public function testRestoreOfAPrefixStripsAndLeavesOthersOut(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->repo()->restore($snapshot, $target, 'data/alice/', true);
$this->assertDirectoryExists("$target/files/Hollow/Nested");
$this->assertDirectoryDoesNotExist("$target/skipped");
$this->assertDirectoryDoesNotExist("$target/inner");
}
public function testRunningAgainLeavesAFolderThatIsThereAlone(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->repo()->restore($snapshot, $target);
touch("$target/data/alice/files/Hollow/Nested", 1800000000);
file_put_contents("$target/data/alice/files/Hollow/Nested/added", 'kept');
touch("$target/data/alice/files/Hollow/Nested", 1800000000);
$this->repo()->restore($snapshot, $target, keepUnchanged: true);
clearstatcache();
$this->assertSame(1800000000, filemtime("$target/data/alice/files/Hollow/Nested"));
$this->assertFileExists("$target/data/alice/files/Hollow/Nested/added");
}
public function testALinkOutOfTheTargetIsRefused(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$outside = $this->tempDir();
mkdir("$target/data/alice/files", 0700, true);
symlink($outside, "$target/data/alice/files/Hollow");
try {
$this->repo()->restore($snapshot, $target);
$this->fail('restored through a link');
} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
$this->assertStringContainsString('outside', $e->getMessage());
}
$this->assertSame([], glob("$outside/*"));
}
public function testASnapshotWithoutThemHasNone(): void {
$src = $this->tempDir();
file_put_contents("$src/a", 'x');
$snapshot = $this->repo()->backupDirectory($src)['snapshot'];
$this->assertSame([], $this->repo()->emptyDirs($snapshot));
}
}
+127
View File
@@ -0,0 +1,127 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Job\BackupRun;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/** Folders with something in them get their original modification time back on a restore. */
class FolderTimesTest extends TestCase {
use TempDirTrait;
private const OLD = 1500000000;
private const OLDER = 1400000000;
private LocalBackend $backend;
private KeyRing $keys;
private string $src;
protected function setUp(): void {
$this->backend = new LocalBackend($this->tempDir());
$this->keys = KeyRing::generate();
Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
$this->src = $this->tempDir();
mkdir("$this->src/data/alice/files/Old/Deep", 0700, true);
mkdir("$this->src/data/alice/files/2026", 0700, true);
file_put_contents("$this->src/data/alice/files/Old/Deep/a.txt", 'a');
file_put_contents("$this->src/data/alice/files/Old/b.txt", 'b');
file_put_contents("$this->src/data/alice/files/2026/c.txt", 'c');
touch("$this->src/data/alice/files/Old/Deep", self::OLDER);
touch("$this->src/data/alice/files/Old", self::OLD);
touch("$this->src/data/alice/files/2026", self::OLD);
}
private function repo(): Repository {
return Repository::openWithKey($this->backend, $this->keys);
}
private function backup(float $seconds = 30.0): string {
$state = BackupRun::start(['data' => "$this->src/data"]);
for ($i = 0; $i < 500 && $state['phase'] !== 'commit'; $i++) {
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, microtime(true) + $seconds)), true);
}
$this->assertSame('commit', $state['phase']);
$this->repo()->recordSnapshot($state['snapshot']);
return $state['snapshot'];
}
private function time(string $path): int {
clearstatcache();
return (int)filemtime($path);
}
public function testEachFolderIsRecordedOnceAlsoInSmallSteps(): void {
$expected = ['data', 'data/alice', 'data/alice/files', 'data/alice/files/2026', 'data/alice/files/Old', 'data/alice/files/Old/Deep'];
$this->assertSame($expected, array_column($this->repo()->folderTimes($this->backup()), 0));
$this->assertSame($expected, array_column($this->repo()->folderTimes($this->backup(0.0)), 0), 'a step per entry');
}
public function testNoEmptyFolderIsAmongThem(): void {
mkdir("$this->src/data/alice/files/Hollow");
$snapshot = $this->backup();
$this->assertNotContains('data/alice/files/Hollow', array_column($this->repo()->folderTimes($snapshot), 0));
$this->assertContains('data/alice/files/Hollow', array_column($this->repo()->emptyDirs($snapshot), 0));
}
public function testARestoreBringsThemBack(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->repo()->restore($snapshot, $target);
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/Old"));
$this->assertSame(self::OLDER, $this->time("$target/data/alice/files/Old/Deep"));
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/2026"));
}
public function testARestoreThatGoesOnAfterAStopToo(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->repo()->restore($snapshot, $target);
touch("$target/data/alice/files/Old", 1800000000);
$this->repo()->restore($snapshot, $target, keepUnchanged: true);
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/Old"));
}
public function testAPrefixLeavesTheTargetAndOthersAlone(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->repo()->restore($snapshot, $target, 'data/alice/files/Old/', true);
$this->assertSame(self::OLDER, $this->time("$target/Deep"));
$this->assertNotSame(self::OLD, $this->time($target), 'the directory that was chosen is not ours to change');
$this->assertDirectoryDoesNotExist("$target/2026");
}
public function testALinkOutOfTheTargetIsRefused(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$outside = $this->tempDir();
touch($outside, 1800000000);
mkdir("$target/data/alice/files", 0700, true);
symlink($outside, "$target/data/alice/files/Old");
$this->expectException(\OCA\NgBackup\Repository\RestoreTargetException::class);
$this->repo()->restore($snapshot, $target);
}
public function testAFolderWithABadNameIsLeftOutAndWhatIsInItReported(): void {
mkdir("$this->src/data/odd\\folder");
file_put_contents("$this->src/data/odd\\folder/x", 'x');
$state = BackupRun::start(['data' => "$this->src/data"]);
for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
}
$this->assertSame('commit', $state['phase']);
$this->assertSame(1, $state['stats']['badNames']);
$this->repo()->recordSnapshot($state['snapshot']);
$this->assertNotContains("data/odd\\folder", array_column($this->repo()->folderTimes($state['snapshot']), 0));
}
public function testASnapshotWithoutThemHasNone(): void {
$snapshot = $this->repo()->backupDirectory($this->src)['snapshot'];
$this->assertSame([], $this->repo()->folderTimes($snapshot));
}
}
+149
View File
@@ -0,0 +1,149 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Command\Restore;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Repository\RestoreTargetException;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/** A stream that takes at most 3 bytes per write. */
class PartialWriteStream {
public static string $written = '';
/** @var resource|null set by PHP */
public $context;
public function stream_open(string $path, string $mode, int $options, ?string &$opened): bool {
self::$written = '';
return true;
}
public function stream_write(string $data): int {
$part = substr($data, 0, 3);
self::$written .= $part;
return strlen($part);
}
}
/**
* What goes wrong where a restore writes to is not damage in the backup: it stops the restore, also
* with --salvage, and the message does not point at --salvage.
*/
class RestoreTargetTest extends TestCase {
use TempDirTrait;
private LocalBackend $backend;
private KeyRing $keys;
private string $snapshot;
protected function setUp(): void {
$this->backend = new LocalBackend($this->tempDir());
$this->keys = KeyRing::generate();
$src = $this->tempDir();
mkdir("$src/dir");
file_put_contents("$src/dir/a.bin", random_bytes(6 * 1048576));
file_put_contents("$src/dir/b.bin", random_bytes(6 * 1048576));
file_put_contents("$src/dir/c.txt", 'small');
$this->snapshot = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'))->backupDirectory($src)['snapshot'];
}
private function repo(): Repository {
return Repository::openWithKey($this->backend, $this->keys);
}
public function testAShortWriteIsATargetError(): void {
if (!is_writable('/dev/full')) {
$this->markTestSkipped('needs /dev/full');
}
$out = fopen('/dev/full', 'wb');
stream_set_write_buffer($out, 0);
$this->expectException(RestoreTargetException::class);
Repository::writeAll($out, 'data', 'dir/a.bin');
}
public function testAWriteThatTakesPartOfTheDataGoesOn(): void {
stream_wrapper_register('ngbpartial', PartialWriteStream::class);
try {
$out = fopen('ngbpartial://x', 'wb');
Repository::writeAll($out, 'abcdefgh', 'dir/a.bin');
fclose($out);
$this->assertSame('abcdefgh', PartialWriteStream::$written);
} finally {
stream_wrapper_unregister('ngbpartial');
}
}
public function testATargetErrorDoesNotLookLikeDamage(): void {
$this->assertFalse(Restore::backupLooksDamaged(new RestoreTargetException('Cannot write')));
}
/**
* A real full disk: run with NGB_TEST_SMALL_DISK set to an empty, writable tmpfs of a few MiB
* (docker run --tmpfs /small:size=8m,uid=...), with and without --salvage.
*
* @dataProvider salvageOrNot
*/
public function testAFullDiskStopsTheRestoreAndLeavesNoTemporary(bool $salvage): void {
$disk = getenv('NGB_TEST_SMALL_DISK');
if ($disk === false || $disk === '') {
$this->markTestSkipped('needs NGB_TEST_SMALL_DISK');
}
$target = $disk . '/' . bin2hex(random_bytes(4));
$skipped = $salvage ? [] : null;
try {
$this->repo()->restore($this->snapshot, $target, skipped: $skipped);
$this->fail('12 MiB fitted on the small disk');
} catch (RestoreTargetException $e) {
$this->assertStringContainsString('disk full', $e->getMessage());
} finally {
$left = glob("$target/dir/*.ngb-restore-*");
foreach (glob("$target/dir/*") ?: [] as $f) {
unlink($f);
}
@rmdir("$target/dir");
@rmdir($target);
}
$this->assertSame([], $left);
if ($salvage) {
$this->assertSame([], $skipped, 'nothing is listed as damaged');
}
}
/** @return array<string, array{bool}> */
public static function salvageOrNot(): array {
return ['without --salvage' => [false], 'with --salvage' => [true]];
}
public function testResumeChecksTheFolderBeforeSkipping(): void {
$target = $this->tempDir();
$this->repo()->restore($this->snapshot, $target);
$outside = $this->tempDir();
rename("$target/dir", "$outside/dir");
symlink("$outside/dir", "$target/dir");
$this->expectException(RestoreTargetException::class);
$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
}
public function testResumeRemovesTemporariesWhereEverythingIsInPlace(): void {
$target = $this->tempDir();
$this->repo()->restore($this->snapshot, $target);
file_put_contents("$target/dir/a.bin.ngb-restore-0123456789ab", 'left by a stopped restore');
$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
$this->assertFileDoesNotExist("$target/dir/a.bin.ngb-restore-0123456789ab");
}
public function testHeartbeatRunsForFilesThatAreSkipped(): void {
$target = $this->tempDir();
$this->repo()->restore($this->snapshot, $target);
$beats = 0;
$this->repo()->restore($this->snapshot, $target, '', false, function () use (&$beats): void {
$beats++;
}, keepUnchanged: true);
$this->assertSame(3, $beats);
}
}
@@ -0,0 +1,89 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/** A pin and a note on a snapshot live in the catalog on the location, so a new server sees them too. */
class SnapshotMarksTest extends TestCase {
use TempDirTrait;
private LocalBackend $backend;
private KeyRing $keys;
private string $snapshot;
protected function setUp(): void {
$this->backend = new LocalBackend($this->tempDir());
$this->keys = KeyRing::generate();
$src = $this->tempDir();
file_put_contents("$src/a", 'x');
$this->snapshot = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'))->backupDirectory($src)['snapshot'];
}
private function repo(): Repository {
return Repository::openWithKey($this->backend, $this->keys);
}
public function testTheyAreThereAfterOpeningAgain(): void {
$this->repo()->setPinned($this->snapshot, true);
$this->repo()->setSnapshotNote($this->snapshot, 'before the upgrade');
$this->assertSame([$this->snapshot], $this->repo()->pinnedSnapshots());
$this->assertSame([$this->snapshot => 'before the upgrade'], $this->repo()->snapshotNotes());
}
public function testTheyCanBeTakenOffAgain(): void {
$this->repo()->setPinned($this->snapshot, true);
$this->repo()->setSnapshotNote($this->snapshot, 'x');
$this->repo()->setPinned($this->snapshot, false);
$this->repo()->setSnapshotNote($this->snapshot, '');
$this->assertSame([], $this->repo()->pinnedSnapshots());
$this->assertSame([], $this->repo()->snapshotNotes());
}
public function testNoChangeWritesNoCatalog(): void {
$repo = $this->repo();
$repo->setPinned($this->snapshot, true);
$generation = $repo->catalogGeneration();
$repo->setPinned($this->snapshot, true);
$repo->setSnapshotNote($this->snapshot, '');
$this->assertSame($generation, $repo->catalogGeneration());
}
public function testASnapshotThatIsNotThereIsRefused(): void {
$this->expectException(\InvalidArgumentException::class);
$this->repo()->setPinned('0123456789abcdef0123456789abcdef', true);
}
public function testATrashedSnapshotLosesItsPinAndKeepsItsNote(): void {
$repo = $this->repo();
$repo->setPinned($this->snapshot, true);
$repo->setSnapshotNote($this->snapshot, 'kept');
$repo->forget($this->snapshot);
$this->assertSame([], $this->repo()->pinnedSnapshots());
$this->assertSame([$this->snapshot => 'kept'], $this->repo()->snapshotNotes());
$repo = $this->repo();
$repo->untrash($this->snapshot);
$this->assertSame([], $this->repo()->pinnedSnapshots(), 'back in the list, as a normal one');
$this->assertSame([$this->snapshot => 'kept'], $this->repo()->snapshotNotes());
}
public function testThePurgeTakesTheNoteAlong(): void {
$repo = $this->repo();
$repo->setSnapshotNote($this->snapshot, 'kept');
$repo->forget($this->snapshot);
$repo->purgeTrash(0);
$this->assertSame([], $this->repo()->snapshotNotes());
}
public function testACatalogFromBeforeThisVersionReadsAsNone(): void {
$this->assertSame([], $this->repo()->pinnedSnapshots());
$this->assertSame([], $this->repo()->snapshotNotes());
}
}
@@ -0,0 +1,74 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Service;
use OCA\NgBackup\Service\LocationSchedule as S;
use PHPUnit\Framework\TestCase;
class LocationScheduleTest extends TestCase {
private const BERLIN = 'Europe/Berlin';
private static function at(string $time): int {
return (int)strtotime($time);
}
public function testWhatCanBeSaved(): void {
$this->assertSame('', S::normalize('off'));
$this->assertSame('', S::normalize(' '));
$this->assertSame('02:30', S::normalize('02:30'));
$this->assertSame('02:30', S::normalize('02:30/24'));
$this->assertSame('00:00/6', S::normalize('00:00/6'));
foreach (['2:30', '24:00', '02:30/0', '02:30/25', 'daily', '02:30/x'] as $bad) {
try {
S::normalize($bad);
$this->fail("$bad accepted");
} catch (\InvalidArgumentException) {
$this->addToAssertionCount(1);
}
}
}
public function testOffHasNoMoment(): void {
$this->assertNull(S::lastMoment('', self::BERLIN, self::at('2026-11-10 12:00 UTC')));
}
public function testDailyIsInTheZoneOfTheLocation(): void {
$this->assertSame(self::at('2026-11-10 02:00 UTC'), S::lastMoment('03:00', self::BERLIN, self::at('2026-11-10 12:00 UTC')));
$this->assertSame(self::at('2026-11-09 02:00 UTC'), S::lastMoment('03:00', self::BERLIN, self::at('2026-11-10 01:00 UTC')), 'not yet today');
}
public function testEveryNHoursStartsOverTheNextDay(): void {
$zone = 'UTC';
$this->assertSame(self::at('2026-11-10 12:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 13:00 UTC')));
$this->assertSame(self::at('2026-11-09 22:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 01:59 UTC')), '02, 07, 12, 17, 22, then 02 again');
$this->assertSame(self::at('2026-11-10 02:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 02:00 UTC')), 'at the moment itself');
}
public function testAClockChangeInTheNightStillGivesOneMoment(): void {
$this->assertSame(self::at('2026-03-29 01:30 UTC'), S::lastMoment('02:30', self::BERLIN, self::at('2026-03-29 12:00 UTC')), '02:30 does not exist that night');
$this->assertNotNull(S::lastMoment('02:30', self::BERLIN, self::at('2026-10-25 12:00 UTC')));
}
public function testChoosingATimeDoesNotStartABackup(): void {
$moment = self::at('2026-11-10 02:00 UTC');
$this->assertFalse(S::isDue($moment, $moment + 1, null, 0, null, $moment + 3600));
$this->assertTrue(S::isDue($moment, $moment - 1, null, 0, null, $moment + 3600));
$this->assertTrue(S::isDue($moment, 0, null, 0, null, $moment), 'a location from before schedules per location');
}
public function testABackupSinceTheMomentIsEnough(): void {
$moment = 1000000;
$this->assertFalse(S::isDue($moment, 0, $moment, 1, $moment + 10, $moment + 20));
$this->assertTrue(S::isDue($moment, 0, $moment - 1, 0, null, $moment + 20), 'it started before this moment: one more');
}
public function testAfterAFailureOnlyAFewAttemptsAnHourApart(): void {
$moment = 1000000;
$finished = $moment + 60;
$this->assertFalse(S::isDue($moment, 0, null, 1, $finished, $finished + S::RETRY_AFTER - 1));
$this->assertTrue(S::isDue($moment, 0, null, 1, $finished, $finished + S::RETRY_AFTER));
$this->assertFalse(S::isDue($moment, 0, null, S::MAX_ATTEMPTS, $finished, $finished + 86400));
}
}
@@ -25,6 +25,17 @@ class RetentionPolicyTest extends TestCase {
$this->assertCount(3, $months); $this->assertCount(3, $months);
} }
public function testAPinnedSnapshotStaysAndDoesNotCount(): void {
$policy = new RetentionPolicy(1, 0, 0, 0);
$snaps = ['old' => 100, 'pinned' => 300, 'new' => 200];
$this->assertSame(['pinned'], array_keys($policy->keep($snaps)), 'without the pin it would take the place of "last"');
$keep = $policy->keepWithPinned($snaps, ['pinned', 'gone-already']);
$this->assertSame(['pinned'], $keep['pinned']);
$this->assertSame(['last'], $keep['new'], 'the newest one that is not pinned');
$this->assertArrayNotHasKey('old', $keep);
$this->assertArrayNotHasKey('gone-already', $keep);
}
public function testKeepsEverythingWhenFewSnapshots(): void { public function testKeepsEverythingWhenFewSnapshots(): void {
$keep = (new RetentionPolicy())->keep(['a' => 100, 'b' => 200]); $keep = (new RetentionPolicy())->keep(['a' => 100, 'b' => 200]);
$this->assertSame(['a', 'b'], array_values(array_intersect(['a', 'b'], array_keys($keep)))); $this->assertSame(['a', 'b'], array_values(array_intersect(['a', 'b'], array_keys($keep))));
@@ -0,0 +1,22 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Service;
use OCA\NgBackup\Service\AlertService;
use PHPUnit\Framework\TestCase;
/** A location without a schedule: a weekly reminder while another is backed up, else a daily one. */
class ScheduleOffReminderTest extends TestCase {
public function testOnceAWeekWhileAnotherLocationIsActive(): void {
$this->assertFalse(AlertService::offReminderDue(0, 6 * 86400, true));
$this->assertTrue(AlertService::offReminderDue(0, 7 * 86400, true));
}
public function testEveryDayWhenNoLocationIsActive(): void {
$this->assertFalse(AlertService::offReminderDue(0, 3600, false));
$this->assertTrue(AlertService::offReminderDue(0, 86400, false));
$this->assertTrue(AlertService::offReminderDue(0, 86400 - 60, false), 'a cron run a minute earlier than yesterday');
}
}
-11
View File
@@ -57,15 +57,4 @@ class ScheduleTest extends TestCase {
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get()); $this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
} }
public function testTodayAtIsTheMomentInTheSchedulesZone(): void {
$this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin'];
$at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC'));
$this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC');
}
public function testAnUnknownZoneOrABadTimeIsRefused(): void {
$s = $this->schedule();
$this->expectException(\InvalidArgumentException::class);
$s->set('03:00', 'Mars/Olympus');
}
} }
+16 -4
View File
@@ -102,7 +102,7 @@ ok "key, kit and location on the origin"
NC_PASS=$(openssl rand -hex 16) NC_PASS=$(openssl rand -hex 16)
docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest" docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs /var/www/html/data/frtest/files/Empty/Nested && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
if [ "$MODE" = sftp ]; then if [ "$MODE" = sftp ]; then
# about three packs of 32 MiB # about three packs of 32 MiB
docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin" docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
@@ -214,6 +214,7 @@ if [ "$MODE" = salvage ]; then
printf '%s\n' "$OUT" | grep "$SALVAGE" >/dev/null || { printf '%s\n' "$OUT" | tail -30; fail "the output does not name the salvage folder"; } printf '%s\n' "$OUT" | grep "$SALVAGE" >/dev/null || { printf '%s\n' "$OUT" | tail -30; fail "the output does not name the salvage folder"; }
[ "$(docker exec ngb-frt-restore cat "$SALVAGE/data/frtest/files/Docs/note.txt")" = "hello disaster recovery" ] || fail "the user's file is not in the salvage folder" [ "$(docker exec ngb-frt-restore cat "$SALVAGE/data/frtest/files/Docs/note.txt")" = "hello disaster recovery" ] || fail "the user's file is not in the salvage folder"
docker exec ngb-frt-restore test -f "$SALVAGE/config/config.php" || fail "config.php is not in the salvage folder" docker exec ngb-frt-restore test -f "$SALVAGE/config/config.php" || fail "config.php is not in the salvage folder"
docker exec ngb-frt-restore test -d "$SALVAGE/data/frtest/files/Empty/Nested" || fail "the empty folder is not in the salvage folder"
run_on ngb-frt-restore user:list | grep frtest >/dev/null && fail "the fresh database was changed (the backed-up user is there)" run_on ngb-frt-restore user:list | grep frtest >/dev/null && fail "the fresh database was changed (the backed-up user is there)"
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage" run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage" run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
@@ -254,8 +255,8 @@ if [ "$MODE" = salvage-files ]; then
if (!file_exists($root . $p)) { $bad[] = "not listed and not there: $p"; continue; } if (!file_exists($root . $p)) { $bad[] = "not listed and not there: $p"; continue; }
hash_file("sha256", $root . $p) === $sum ? $same++ : $bad[] = "different: $p"; hash_file("sha256", $root . $p) === $sum ? $same++ : $bad[] = "different: $p";
} }
$temps = glob($root . "*/*.ngb-restore-*") ?: []; $walk = new RecursiveIteratorIterator(new RecursiveDirectoryIterator("/var/www/html/data", FilesystemIterator::SKIP_DOTS), RecursiveIteratorIterator::SELF_FIRST);
foreach ($temps as $t) { $bad[] = "temporary left: $t"; } foreach ($walk as $f) { if (str_contains($f->getFilename(), ".ngb-restore-")) { $bad[] = "temporary left: " . $f->getPathname(); } }
echo "same=$same gone=$gone bad=", count($bad), "\n", implode("\n", array_slice($bad, 0, 10)), "\n"; echo "same=$same gone=$gone bad=", count($bad), "\n", implode("\n", array_slice($bad, 0, 10)), "\n";
echo "reasons: ", implode(" | ", array_unique(array_map(fn ($r) => substr(preg_replace("#[0-9a-f]{16,}#", "<id>", $r), 0, 70), $reasons))), "\n";') echo "reasons: ", implode(" | ", array_unique(array_map(fn ($r) => substr(preg_replace("#[0-9a-f]{16,}#", "<id>", $r), 0, 70), $reasons))), "\n";')
printf '%s\n' "$CHECK" printf '%s\n' "$CHECK"
@@ -265,6 +266,8 @@ if [ "$MODE" = salvage-files ]; then
ok "with --salvage: every file listed and absent, or there and identical" ok "with --salvage: every file listed and absent, or there and identical"
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage" run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing: the database was not restored" run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing: the database was not restored"
run_on ngb-frt-restore user:list | grep admin >/dev/null || fail "the admin is missing: the database was not restored"
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back with --salvage"
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt 2>/dev/null)" = "hello disaster recovery" ] \ [ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt 2>/dev/null)" = "hello disaster recovery" ] \
|| printf '%s\n' "$OUT" | grep "Docs/note.txt" >/dev/null || fail "note.txt is neither restored nor listed" || printf '%s\n' "$OUT" | grep "Docs/note.txt" >/dev/null || fail "note.txt is neither restored nor listed"
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage" run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
@@ -323,7 +326,16 @@ run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails aft
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database" run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database"
docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing" docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs" [ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore" if [ "$LEGACY" = legacy ]; then
# Backups made before 1.0.3 do not record empty folders
docker exec ngb-frt-restore test -e /var/www/html/data/frtest/files/Empty/Nested && fail "an empty folder came back from a backup that does not record them"
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
else
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back"
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "files:scan removed the empty folder"
ok "the empty folder came back and stays after files:scan"
fi
# New rows after the restore: a new file (file cache) and a new user must not collide with # New rows after the restore: a new file (file cache) and a new user must not collide with
# restored rows (ID counters after the data). # restored rows (ID counters after the data).
docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt" docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"
+98 -7
View File
@@ -23,12 +23,15 @@ ok "key, kit code and confirmation"
occ backup:target:add ci $TARGET_ARGS --path=repo >/dev/null || fail "target add" occ backup:target:add ci $TARGET_ARGS --path=repo >/dev/null || fail "target add"
ok "location added" ok "location added"
# Test data: a user with files. # Test data: a user with files. No skeleton files on the first login, below, in the middle of the checks.
occ config:system:set skeletondirectory --value='' >/dev/null
occ config:system:set templatedirectory --value='' >/dev/null
$X sh -c "OC_PASS='Roundtrip-pw-2026' php occ user:add --password-from-env $U >/dev/null" $X sh -c "OC_PASS='Roundtrip-pw-2026' php occ user:add --password-from-env $U >/dev/null"
# large.bin (40 MB) fills more than one pack: a restore to a directory reads ahead over several ranges # large.bin (40 MB) fills more than one pack: a restore to a directory reads ahead over several ranges
$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin" $X sh -c "mkdir -p $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs/Empty/Nested $DATA/$U/files/Hollow && touch -d @1700000000 $DATA/$U/files/Hollow && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin"
$X sh -c "touch -d @1600000000 $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs"
occ files:scan "$U" >/dev/null occ files:scan "$U" >/dev/null
SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum") SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
occ backup:run ci -l first >/dev/null || fail "first backup" occ backup:run ci -l first >/dev/null || fail "first backup"
SECOND=$(occ backup:run ci -l second) SECOND=$(occ backup:run ci -l second)
@@ -39,14 +42,29 @@ SNAP=$(occ backup:list ci | awk -F'|' '!f && / second /{gsub(/ /,"",$3); print $
[ -n "$SNAP" ] || fail "snapshot not listed" [ -n "$SNAP" ] || fail "snapshot not listed"
# Damage, then restore in each mode. # Damage, then restore in each mode.
$X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt" $X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt && rm -r Docs/Empty && rmdir Hollow"
occ files:scan "$U" >/dev/null occ files:scan "$U" >/dev/null
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m new-folder >/dev/null || fail "restore new-folder" occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m new-folder >/dev/null || fail "restore new-folder"
$X sh -c "ls $DATA/$U/files | grep -q '^Restored '" || fail "no Restored folder" $X sh -c "ls $DATA/$U/files | grep -q '^Restored '" || fail "no Restored folder"
$X sh -c "test -d $DATA/$U/files/Restored\ */Docs/Empty/Nested" || fail "the empty folder is not in the Restored folder"
# Folders with files: on disk and in what Nextcloud shows, the parent as well as the child.
RDIR=$($X sh -c "cd $DATA/$U/files && ls -d Restored\ *")
[ "$($X sh -c "stat -c %Y $DATA/$U/files/Restored\ */Docs/Sub")" = 1600000000 ] || fail "a folder with files did not get its time back"
[ "$($X sh -c "stat -c %Y $DATA/$U/files/Restored\ */Docs")" = 1600000000 ] || fail "the parent folder did not get its time back"
DAV=$($X curl -sf -u "$U:Roundtrip-pw-2026" -X PROPFIND -H 'Depth: 1' "http://localhost/remote.php/dav/files/$U/$(php -r 'echo rawurlencode($argv[1]);' "$RDIR")/Docs/") || fail "PROPFIND"
[ "$(echo "$DAV" | grep -o 'Sun, 13 Sep 2020 12:26:40 GMT' | wc -l)" = 2 ] || fail "Nextcloud does not show the time of the folders: $(echo "$DAV" | grep -o '<d:getlastmodified>[^<]*' | sort | uniq -c)"
occ backup:restore ci "$SNAP" "data/$U/files/Docs/Sub" -m merge >/dev/null || fail "restore into a folder that is there"
[ "$($X stat -c %Y "$DATA/$U/files/Docs/Sub")" != 1600000000 ] || fail "a folder that was already there got the old time"
occ backup:browse ci "$SNAP" "data/$U/files" | grep Hollow >/dev/null || fail "an empty folder is not shown when browsing the snapshot"
occ backup:restore ci "$SNAP" "data/$U/files/Hollow" -m merge >/dev/null || fail "restore of a folder that was empty"
$X sh -c "test -d $DATA/$U/files/Hollow" || fail "the empty folder did not come back by itself"
[ "$($X stat -c %Y "$DATA/$U/files/Hollow")" = 1700000000 ] || fail "the empty folder did not get its time back"
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m replace >/dev/null || fail "restore replace" occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m replace >/dev/null || fail "restore replace"
$X sh -c "test -d $DATA/$U/files/Docs/Empty/Nested" || fail "replace did not bring back the empty folder, or moved it to the trash"
ok "empty folders: restored in each mode, by themselves and shown when browsing"
$X sh -c "rm -rf '$DATA/$U/files/Restored '*" ; occ files:scan "$U" >/dev/null $X sh -c "rm -rf '$DATA/$U/files/Restored '*" ; occ files:scan "$U" >/dev/null
SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum") SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore" [ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore: $(diff <(echo "$SUM_BEFORE") <(echo "$SUM_AFTER") | cut -c1-120)"
$X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash" $X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash"
ok "restore new-folder and replace: files identical, extra file in the trash" ok "restore new-folder and replace: files identical, extra file in the trash"
@@ -60,9 +78,10 @@ ok "restore of one file to a directory; the temporary of a stopped restore remov
# All of the user's files to a directory: over S3 and SFTP the packs are read ahead # All of the user's files to a directory: over S3 and SFTP the packs are read ahead
occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-dir >/dev/null || fail "restore of a folder to a directory" occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-dir >/dev/null || fail "restore of a folder to a directory"
SUM_DIR=$($X sh -c "cd /tmp/ngb-rt-dir/data/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum") SUM_DIR=$($X sh -c "cd /tmp/ngb-rt-dir/data/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
[ "$SUM_BEFORE" = "$SUM_DIR" ] || fail "the files restored to a directory differ" [ "$SUM_BEFORE" = "$SUM_DIR" ] || fail "the files restored to a directory differ"
$X sh -c "ls -A /tmp/ngb-rt-dir | grep -q ngb-restore-spool" && fail "the read-ahead spool was left in the directory" $X sh -c "ls -A /tmp/ngb-rt-dir | grep -q ngb-restore-spool" && fail "the read-ahead spool was left in the directory"
$X sh -c "test -d /tmp/ngb-rt-dir/data/$U/files/Docs/Empty/Nested" || fail "the empty folder is not in the directory"
$X sh -c "rm -rf /tmp/ngb-rt-dir" $X sh -c "rm -rf /tmp/ngb-rt-dir"
ok "restore of all the user's files to a directory: identical, no spool left" ok "restore of all the user's files to a directory: identical, no spool left"
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail) # not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
@@ -90,13 +109,74 @@ done
[ "$RS" = done ] || fail "the backup started through the API ended as '$RS' after $i ticks" [ "$RS" = done ] || fail "the backup started through the API ended as '$RS' after $i ticks"
occ backup:list ci | grep " page " >/dev/null || fail "the snapshot of the ticked backup is not listed" occ backup:list ci | grep " page " >/dev/null || fail "the snapshot of the ticked backup is not listed"
ok "without system cron: warning in backup:status and the API, backup finished by $i ticks of the open page" ok "without system cron: warning in backup:status and the API, backup finished by $i ticks of the open page"
# The Bin reads in parts, as the page does (what is in the parts: CatalogTest)
api "$API/targets/$TID/trash" | php -r '$p = json_decode(stream_get_contents(STDIN), true); exit(is_int($p["total"] ?? null) && is_array($p["items"] ?? null) && count($p["items"]) <= $p["total"] ? 0 : 1);' || fail "the Bin API does not give a total and items"
ok "the Bin API reads in parts, with a total"
# Locations are backed up one after the other: the second one started is queued until the first is done.
for L in 2 3; do
$X sh -c "mkdir -p /tmp/ngb-rt-target$L"
occ backup:target:add ci$L local -a null::null -o datadir=/tmp/ngb-rt-target$L --path=repo >/dev/null || fail "target add ci$L"
done
IDS=$(api "$API/status" | php -r 'foreach (json_decode(stream_get_contents(STDIN), true)["targets"] as $t) { if ($t["name"] === "ci2" || $t["name"] === "ci3") { echo $t["id"], " "; } }')
for ID in $IDS; do api -X POST -d "{\"targetId\": $ID, \"label\": \"turn\"}" "$API/backups" >/dev/null || fail "start a backup to location $ID"; done
# Never two runs to one location: a second start gets the run that is there.
ID=${IDS%% *}
RUNS=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
api -X POST -d "{\"targetId\": $ID, \"label\": \"again\"}" "$API/backups" >/dev/null || fail "second start"
AGAIN=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
[ "$RUNS" = 2 ] && [ "$AGAIN" = 2 ] || fail "a second start to one location: $RUNS, then $AGAIN running"
occ backup:status | grep "queued" >/dev/null || fail "the second backup is not queued"
QUEUED=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["queued"]));')
[ "$QUEUED" = 1 ] || fail "$QUEUED backups queued instead of 1"
for i in $(seq 1 60); do
api -X POST "$API/tick" >/dev/null || fail "tick"
RUNNING=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
[ "$RUNNING" != 0 ] || break
done
[ "$RUNNING" = 0 ] || fail "the queued backups did not finish"
occ backup:list ci3 | grep " turn " >/dev/null || fail "the queued backup made no snapshot"
ok "locations in turn: the second backup queued, and done after the first"
occ background:cron >/dev/null occ background:cron >/dev/null
$X php -f /var/www/html/cron.php >/dev/null 2>&1 || true $X php -f /var/www/html/cron.php >/dev/null 2>&1 || true
occ backup:status | grep "system cron" >/dev/null && fail "backup:status still warns after cron ran" occ backup:status | grep "system cron" >/dev/null && fail "backup:status still warns after cron ran"
sleep 8 # APCu app config cache of the web server sleep 8 # APCu app config cache of the web server
api "$API/status" | php -r 'exit(json_decode(stream_get_contents(STDIN), true)["cron"]["ok"] ? 0 : 1);' || fail "the status API still reports a missing cron after cron ran" api "$API/status" | php -r 'exit(json_decode(stream_get_contents(STDIN), true)["cron"]["ok"] ? 0 : 1);' || fail "the status API still reports a missing cron after cron ran"
ok "with a system cron that ran: no warning in backup:status or the API" ok "with a system cron that ran: no warning in backup:status or the API"
# A schedule per location. Choosing a time starts nothing; the moment itself does, and only there.
JOB=$(occ background-job:list --class='OCA\NgBackup\BackgroundJob\BackupJob' --output=json | php -r 'echo json_decode(stream_get_contents(STDIN), true)[0]["id"];')
job() { occ background-job:execute "$JOB" --force-execute >/dev/null; }
occ backup:target:schedule ci off >/dev/null
occ backup:target:schedule ci3 off >/dev/null
occ backup:target:schedule ci2 00:00 --timezone=UTC >/dev/null || fail "set a schedule"
occ backup:target:schedule ci2 | grep "00:00 (UTC)" >/dev/null || fail "schedule not shown"
job
[ "$(occ backup:list ci2 | grep -c ' scheduled ')" = 0 ] || fail "choosing a time started a backup"
occ backup:target:schedule ci2 "$($X date -u -d '+1 minute' +%H:%M)/6" --timezone=UTC >/dev/null || fail "set every 6 hours"
sleep 65
job
occ backup:list ci2 | grep " scheduled " >/dev/null || fail "the scheduled moment started no backup"
[ "$(occ backup:list ci3 | grep -c ' scheduled ')" = 0 ] || fail "a location that is off was backed up"
job
[ "$(occ backup:list ci2 | grep -c ' scheduled ')" = 1 ] || fail "one moment started more than one backup"
occ backup:target:schedule ci2 nonsense >/dev/null 2>&1 && fail "a bad schedule was accepted"
ok "schedule per location: nothing on choosing a time, one backup at the moment, none when off"
# What administrators are told, as Nextcloud shows it.
notes() { api "http://localhost/ocs/v2.php/apps/notifications/api/v2/notifications?format=json"; }
occ backup:target:schedule ci2 off >/dev/null
$X sh -c "touch '$DATA/ngb-rt-bad\\name.txt'"
occ backup:run ci3 -l names | grep "their name is not supported" >/dev/null || fail "occ does not report the name"
$X sh -c "rm '$DATA/ngb-rt-bad\\name.txt'"
NOTES=$(notes) || fail "read the notifications"
echo "$NOTES" | grep "turned off the schedule of backup location ci2" >/dev/null || fail "no notification about the schedule that was turned off"
echo "$NOTES" | grep "is missing because of its name" >/dev/null || fail "no notification about the name"
echo "$NOTES" | grep "Rename them" >/dev/null || fail "the notification about the name does not ask to rename"
job
notes | grep "has no schedule" >/dev/null && fail "a reminder right after turning it off"
for L in 2 3; do occ backup:target:remove ci$L >/dev/null || fail "target remove ci$L"; done
occ user:delete "$ADM" >/dev/null occ user:delete "$ADM" >/dev/null
ok "notifications: schedule turned off, name not supported, no early reminder"
# Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob). # Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob).
occ backup:verify ci "$SNAP" | grep "^OK$" >/dev/null || fail "shallow verify" occ backup:verify ci "$SNAP" | grep "^OK$" >/dev/null || fail "shallow verify"
@@ -109,6 +189,17 @@ ok "database dump and restore"
# Retention, trash and catalog. # Retention, trash and catalog.
occ backup:retention --last=1 --daily=0 --weekly=0 --monthly=0 >/dev/null occ backup:retention --last=1 --daily=0 --weekly=0 --monthly=0 >/dev/null
# A pinned restore point stays, and a note can change afterwards.
FIRST=$(occ backup:list ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')
occ backup:pin ci "$FIRST" >/dev/null || fail "pin"
occ backup:note ci "$FIRST" "a note made later" >/dev/null || fail "note"
occ backup:list ci | grep " first " | grep " yes " | grep "a note made later" >/dev/null || fail "pin and note not listed"
occ backup:prune ci --dry-run | grep "would forget 1 " >/dev/null || fail "a pinned snapshot would be removed"
occ backup:pin ci "$FIRST" --unpin >/dev/null || fail "unpin"
occ backup:prune ci --dry-run | grep "would forget 2 " >/dev/null || fail "an unpinned snapshot would stay"
occ backup:note ci "$FIRST" >/dev/null || fail "remove note"
occ backup:list ci | grep -q "a note made later" && fail "note not removed"
ok "pin and note on a restore point"
occ backup:prune ci | grep "forgot 2" >/dev/null || fail "prune" # first and second; "page" (above) is the last occ backup:prune ci | grep "forgot 2" >/dev/null || fail "prune" # first and second; "page" (above) is the last
occ backup:trash ci | grep " first " >/dev/null || fail "trash" occ backup:trash ci | grep " first " >/dev/null || fail "trash"
TID=$(occ backup:trash ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}') TID=$(occ backup:trash ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')