Compare commits

..

22 Commits

Author SHA1 Message Date
dillard 8e06d05cf4 Roadmap 1.2.0: restore only the database from a restore point, with occ
CI / unit (8.2) (push) Waiting to run
CI / unit (8.3) (push) Waiting to run
CI / unit (8.4) (push) Waiting to run
CI / psalm (push) Waiting to run
CI / frontend (push) Waiting to run
CI / integration (mariadb, 32) (push) Waiting to run
CI / integration (mariadb, 33) (push) Waiting to run
CI / integration (mariadb, 34) (push) Waiting to run
CI / integration (mariadb, 35) (push) Waiting to run
CI / integration (pgsql, 32) (push) Waiting to run
CI / integration (pgsql, 33) (push) Waiting to run
CI / integration (pgsql, 34) (push) Waiting to run
CI / integration (pgsql, 35) (push) Waiting to run
CI / integration (sqlite, 32) (push) Waiting to run
CI / integration (sqlite, 33) (push) Waiting to run
CI / integration (sqlite, 34) (push) Waiting to run
CI / integration (sqlite, 35) (push) Waiting to run
CI / integration-targets (push) Waiting to run
CI / integration-user (32) (push) Waiting to run
CI / integration-user (33) (push) Waiting to run
CI / integration-user (34) (push) Waiting to run
CI / integration-user (35) (push) Waiting to run
CI / integration-full-restore (push) Waiting to run
2026-11-10 09:39:12 +01:00
Dillard Blom c6fdaad487 Merge pull request #31 from dillardblom/1.1.0
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.1.0
2026-11-09 17:44:23 +01:00
dillard 6ae1b2277f Version 1.1.0: release date
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-11-09 17:02:30 +01:00
dillard 2a72acb70b Review fixes: the derived key is wiped also when a key is damaged, each tidy-up step after a removal on its own
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- The key derived from the passphrase is wiped on every path, also when the key or nonce turns out damaged
- After a removal the pending flag and the cached index are tidied up independently; a failing log cannot stop the removal event
- Tests: a short salt or nonce gives the damaged-key error
2026-11-09 02:33:50 +01:00
dillard fb8e8d0694 Review fixes: a malformed key reads as damaged, removing a location always reports it, a key from before 1.1.0 in the recovery guide
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- Invalid base64 or a wrong salt/nonce length in a key gives the damaged-key error, so the recovery options are shown
- Tidying up after a removed location cannot fail the removal anymore, so the removal event is always sent
- Recovery guide: a key made before 1.1.0 counts as used and cannot make way
2026-11-09 00:55:20 +01:00
dillard 042fdae686 Review fixes: the longest deletion delay when the location's settings cannot be read, a damaged passphrase slot does not stop the next one
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- Unreadable settings or an invalid delay on the location: 365 days, so nothing leaves the Bin sooner than on the old server
- A passphrase change that still cannot reach a location is logged; removing a location clears its flag
- A damaged slot (missing parts, invalid base64) is skipped and the next slot is tried
- Tests: damaged settings scenario, damaged slot, upgrade test checks the migration and the time zone
2026-11-08 18:29:31 +01:00
dillard 99f1655d04 Review fixes: a missed passphrase change follows at the next backup, a key from before 1.1.0 counts as used, schedules edited in their own time zone
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- A location that a passphrase change could not reach gets it at its next backup
- A key from before 1.1.0 is never replaced as an unused one
- An unreadable settings file on a location is told apart from a missing one
- A schedule is shown and saved in the zone it was saved in
- Saving an unchanged schedule keeps its moment; --timezone needs a schedule
- Schedule coverage uses the time the backup began reading its data
- Tests: catch-up scenario, upgrade test restores the new file, autumn clock change
2026-11-08 17:18:13 +01:00
dillard 373d7e0424 Screenshots of 1.1.0: a schedule per location, a pinned restore point with a note, passphrase slots, the Bin
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-11-08 16:12:07 +01:00
dillard c5024d3ddc Version 1.1.0: changelog, README status and roadmap 2026-11-08 13:02:28 +01:00
dillard 345b1d545e Keep the list of backups within the page: long text wraps instead of scrolling 2026-11-08 12:28:23 +01:00
dillard 98067f7746 Roadmap: what 1.1.0 brings, parallel uploads in 1.2.0 2026-11-08 12:11:15 +01:00
dillard cd951af1a1 Test the update from a released version on a server that was backing up 2026-11-08 11:58:36 +01:00
dillard cf9cf50286 Replace a backup key that was never used by the key of existing backups
A key made by mistake on a server that is being restored left no way back: key:import and the
passphrase route refused because a key existed, and nothing removed it. As long as the key never
protected anything (no location created or adopted with it, no location or run in the database),
occ backup:key:import --replace-unused-key and occ backup:target:add --replace-unused-key replace
it, after the new key was opened. A key that was used is never replaced, also after its location
was removed. The refusal names the option, and all administrators are notified of a replacement.
2026-11-08 11:53:55 +01:00
dillard 2b21596b3a Plan: speed measurements, parallel uploads planned for 1.2.0 2026-11-08 11:37:13 +01:00
dillard 953312bf44 Read files for a backup in pieces of 4 MiB instead of 8 KiB
PHP reads a file 8 KiB at a time by default. On a disk where every
request waits, such as a network or cloud volume, that made reading five
times slower: on the test server 5 MiB/s instead of 27 MiB/s. Without
PHP's read buffer every read of a blob is one request to the disk. The
same for packs on a local location.
2026-11-07 12:44:57 +01:00
dillard 569077fa69 Restore without the recovery kit: the key from the location, with a passphrase
On a server without a backup key, backup:target:add finds the backups at
the location and takes their key from its config with one of the old
server's passphrases. Backups then wait until a new recovery kit is
confirmed. The deletion delay and the repository id are now stored on the
location, sealed with the key, so the delay survives a restore without a
kit and --repair-config can write a lost config again from the kit.

The key page warns against creating a new key when restoring a server.
New test: tests/integration/recovery-scenarios.sh, and a no-kit mode in
full-restore-roundtrip.sh.
2026-11-07 12:22:38 +01:00
dillard 51a4a67bc4 Heavy background jobs of other apps pause while a backup runs
Reserved in Nextcloud's job list, so no app is turned off and a job that is running goes on.
At most 6 hours in a row, then they get an hour; Nextcloud ends a reservation by itself after
12 hours, should the end of a pause never come. On by default, with a setting to turn it off.
2026-11-07 01:56:44 +01:00
dillard a8b5e85c9b All folders get their original modification time back on a restore
The backup records the time of the folders with something in them, packed in the meta record,
so an older version still reads the snapshot. A restore through Nextcloud leaves a folder that
was already there alone. The round trip checks what Nextcloud shows, not only the disk.
2026-11-07 01:07:09 +01:00
dillard 36f2427e9a Tell administrators about a location without a schedule
A notification when a schedule is turned off, then a reminder: once a week while another
location is backed up, every day when none is. The round trip now also checks the text of
these notifications and of the one about names, and that one location never gets two runs.
2026-11-07 00:23:28 +01:00
dillard e58b86fc48 A schedule per location, and locations backed up one after the other
Off, once a day or every N hours from a time, each in its own time zone. Existing locations
keep the time that held for all of them. A backup that has to wait shows as queued, and its
restore point has the time it really began.
2026-11-07 00:04:00 +01:00
dillard 193f65940b Pin a restore point and keep a note on it
Both live in the catalog on the location, so a new server sees them; the database holds
a local copy for the lists. The retention policy leaves a pinned restore point alone and
does not count it. Available on the restore page and as occ backup:pin and backup:note.
2026-11-06 23:49:50 +01:00
dillard 6540a9314a A name a snapshot cannot hold gets a notification of its own
Reported apart from unreadable files, with example paths and the advice to rename
them, instead of advice about permissions. Start of the 1.1.0 plan in PLAN.md.
2026-11-06 23:38:40 +01:00
67 changed files with 2982 additions and 243 deletions
+54
View File
@@ -11,6 +11,60 @@ Each heading starts with that `<version>`, once: the app store shows a release t
whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
shown under 1.0.0. The git tag of a release candidate goes after the date.
## [1.1.0] - 2026-11-09
### Added
- Pin a restore point, on the restore page or with `occ backup:pin`: the retention policy leaves it
alone, for example the one from before an upgrade. Unpinning notifies all administrators.
- A note on a restore point that you can change afterwards, on the restore page or with
`occ backup:note`. Pins and notes are stored encrypted on the location, so a new server sees them.
An older version of NG Backup can still read the location, but drops them when it writes to it.
- A schedule per location: off, once a day, or every N hours from a given time, on the page under
Backups or with `occ backup:target:schedule`. After the update every location keeps the time that
held for all of them; a new location starts with the schedule of the first one. A schedule is
shown and changed in the time zone it was saved in, which the page names next to the time.
- All folders get their original modification time back on a restore, not only the empty ones, from
the first backup made with this version. An older version of NG Backup can still read new restore
points. A restore through Nextcloud into a folder that is already there leaves its time alone.
- Heavy background jobs (Memories, Recognize, Preview Generator, full text search, the antivirus
background scan) pause while a backup runs, at most 6 hours in a row, then they get an hour. No app
is turned off, and a job that is running goes on. On by default; turn it off under Backups or with
`occ backup:pause-jobs off`.
- Disaster recovery without the recovery kit: on a new server without a backup key,
`occ backup:target:add` finds the backups on the location and takes their key from there with one
of the old server's passphrases. Backups then wait until a new recovery kit is confirmed. The
deletion delay is now also stored on the location, sealed with the key, from its first backup
with this version; a location backed up only by an older version gives the default of 7 days.
When the settings on the location cannot be read, the longest delay (365 days) is taken, so
nothing leaves the Bin sooner than on the old server; the command says so.
- A passphrase change that cannot reach a location (offline at the time) now reaches it at the
location's next backup, so a new server can open it with the new passphrase.
- `occ backup:target:add --repair-config`: when the config file of a location is gone, the recovery
kit writes it again (for locations backed up by this version or newer). Before, the message said
this was possible, but it was not.
- The key page warns against creating a new key when you are restoring a server. A key made there
by mistake can still make way for the key of the backups, as long as it was never used (no
location, no backup): `occ backup:key:import --replace-unused-key` with the kit, or
`occ backup:target:add --replace-unused-key` with a passphrase of the old server. A key that
protected a location is never replaced, and neither is a key from before this version. All
administrators are notified of a replacement.
### Changed
- The list of backups stays within the page: a long label, note or error wraps onto more lines
instead of making the table scroll sideways.
- Reading files for a backup is much faster on disks where every request waits, such as network and
cloud volumes: files are read in pieces of 4 MiB instead of 8 KiB. On a test server the disk
delivered 27 MiB/s instead of 5 MiB/s.
- Locations are backed up one after the other: a backup that has to wait shows as queued. A restore
point has the time its backup really began, not the time it was queued.
- Choosing a time no longer starts a backup right away when that time has already passed today.
- A location without a schedule no longer gets the notification that it has had no recent backup.
Instead, all administrators are told when a schedule is turned off, and reminded once a week while
another location is still backed up, or every day when none is.
- A file or folder that is left out because of its name now gets a notification of its own, in the
log and in `occ backup:run`. It says the item is missing from the backup, gives example paths and
asks to rename it; before, it was reported as unreadable, with advice about permissions.
## [1.0.3] - 2026-11-06
### Fixed
+48 -5
View File
@@ -472,9 +472,12 @@ in README "Known limitations":
tijd terug. Hoort zo, vooral bij oudere verzamelingen: de maptijd wordt gebruikt om in te schatten
wanneer een map is aangemaakt. De backup moet dan de tijd van alle mappen vastleggen en de restore
die aan het eind terugzetten; werkt pas vanaf de eerste backup met die versie. Op de roadmap onder 1.1.0.
**1.0.3 uitgebracht (2026-11-06)**, ook op nc01 geïnstalleerd.
- **Branches (besluit 2026-11-06):** werk voor een versie op een branch met die naam (`1.0.3`);
`polish` alleen voor kleine cosmetische zaken, per geval bekijken of en wanneer die gemerged
worden. SMB staat uit op sub5 en sub6; sub6 is het testaccount, eerst aanzetten voor een SMB-test.
Na 1.0.3 gaat het werk naar `1.1.0`. Een spoedfix die niet kan wachten gaat op `1.0.4` vanaf `main`,
krijgt een eigen merge en release; daarna `main` in `1.1.0` mergen, zodat daar met de fix getest wordt.
- **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij
@@ -569,11 +572,33 @@ in README "Known limitations":
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
een paar bestanden.
- **1.1.0:**
- **1.1.0** (geen harde datum: een week was een voorzet, langer mag; de releasefrequentie mag omlaag,
besluit 2026-11-06):
**Betere melding voor namen die een snapshot niet kan bevatten** (backslash, stuurteken, geen UTF-8):
een eigen melding in plaats van "unreadable" met het advies over rechten. Zegt dat deze bestanden of
mappen in de backup ontbreken, geeft voorbeelden van de paden en vraagt ze te hernoemen; de volgende
backup neemt ze mee. Alleen als Nextcloud-melding: de app mailt niet zelf, of meldingen gemaild worden
is de keuze van de beheerder (besluit 2026-11-06).
**Stand op branch `1.1.0` (2026-11-07):** klaar: betere melding, punten 2, 4, 5, 6 en 8, en de maptijden
(`files:scan` na een volledige restore: 0 "Updated" in de simulatie; opslag gepakt in het metarecord).
Punt 2 reserveert de jobs in `oc_jobs` met één tijdstempel, alleen wat Nextcloud op dat moment zou kunnen
starten; 6 uur aan, dan 1 uur vrij; bij een crash geeft Nextcloud de reservering na 12 uur zelf vrij;
de logregels staan op niveau info, dus standaard niet zichtbaar. Pin en notitie staan
in de catalogus, met een kopie in de database voor de lijsten. Een moment van vóór het instellen van
een schema start niets. Geen schema: geen melding "geen recente backup", wel een melding aan alle
beheerders bij het uitzetten en een herinnering, wekelijks zolang een andere locatie een schema heeft,
anders dagelijks (besluit 2026-11-07). Nog niet getest: de update vanaf 1.0.3, de pagina in een browser.
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
`target:add` en sleutelimport (het gevoeligste pad), daarom niet in rc1. Daarna de
DR-test op PVE02 herhalen zonder kit.
**Gebouwd (2026-11-07):** `backup:target:add` op een server zonder sleutel kijkt eerst of de
locatie een repository heeft, vraagt dan een wachtwoordzin (of `--passphrase-file`), opent de
slots uit `config`, controleert de sleutel tegen de `check` en slaat hem pas dan op; geen
kitbevestiging, dus backups wachten op een nieuwe kit. Nieuw bestand `settings` op de locatie,
versleuteld met de hoofdsleutel en eenmalig geschreven (bij `target:add` en bij het begin van
een backup): de verwijdertermijn, anders de standaard 7 dagen. Getest: unit-tests en
`full-restore-roundtrip.sh 35-apache pgsql no-kit`. Nog open: de DR-test op PVE02 zonder kit.
2. Instelling "zware achtergrondjobs pauzeren tijdens een backup" (standaard aan, uit te zetten):
Memories, files_antivirus BackgroundScanner, Recognize, Preview Generator, fulltextsearch.
Via de reservering/last_run van die jobs (geen apps uitzetten), crashbestendig, maximaal
@@ -581,6 +606,14 @@ in README "Known limitations":
3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel
een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf,
33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1.
**Gemeten 2026-11-07 op nc01 (alleen lezen):** de schijf (sdb, 2 TB, ROTA=1) haalt met `dd`
21 MiB/s met 1 stroom, 37 met 4, 55 met 8. PHP las 8 KiB per verzoek: 5,2 MiB/s. Met
`stream_set_read_buffer(0)` en blokken van 4 MiB: 26,8 MiB/s. Hulpprocessen die vooruitlezen
gaven daarbovenop niets (20 MiB/s, ze concurreren). Gebouwd: grote leesverzoeken (commit
"Read files ... 4 MiB"). Niet gebouwd: meerdere uploads tegelijk; SMB bundelt al per 1 MB
(icewind/smb), dus winst over een lange lijn vraagt meerdere packs tegelijk.
**Besluit 2026-11-08: meerdere uploads tegelijk in 1.2.0**, met een instelling (standaard 2)
zodat de beheerder het afstemt op zijn server en de omvang van zijn backup.
4. **Schema per locatie** (besluit 2026-10-19): in de instellingen van een locatie "uit",
"dagelijks om HH:MM" of "elke N uur (vanaf HH:MM)". Nu is er één `schedule_time` voor alle
locaties, één keer per dag. Bij de update wordt de huidige tijd het schema van elke
@@ -601,11 +634,21 @@ in README "Known limitations":
nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met
`occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de
opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly.
9. **Feedbackvraag weghalen** in 1.1.0 of 1.2.0 (besluit 2026-10-27): de regel "We would like
9. **Feedbackvraag weghalen in 1.2.0** (besluit 2026-11-07, niet in 1.1.0): de regel "We would like
to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot
dan staan (de vermelding "in beta" is er in 1.0.1 al af). Zijn er tegen die
tijd geen noemenswaardige meldingen, dan werkt het: mensen melden eerder wat niet werkt dan
wat gewoon werkt. Het formulier (`experience.yml`) mag blijven voor wie het zelf opzoekt.
dan staan. Er komt weinig feedback bij zo'n 150 downloads per versie; geen bericht is goed
bericht. Het formulier (`experience.yml`) in de issues blijft.
10. **Alleen de database terughalen, CLI-only** (besluit 2026-11-10). Aanleiding: op nc01 liep
de automatische AIO-update 34 → 35 vast op een migratie van de (uitgeschakelde) app
files_archive; AIO had alle apps uitgezet en hun status alleen in het geheugen bewaard, dus
wat aan stond (en voor welke groepen) was weg. Uit de database in het restorepunt van 02:45
is `oc_appconfig` (rijen `enabled`) met een eenmalig script gelezen. Te bouwen:
`occ backup:restore:db <locatie> <snapshot>` met `--tables=` (één of meer tabellen) en
`--to-directory=DIR` (per tabel een leesbaar bestand, zonder iets aan de database te doen),
of zonder `--to-directory` de gekozen tabellen terugzetten in de draaiende instantie, in
onderhoudsmodus, met dezelfde controles als `restore:full` (schema, eigen ngb-rijen blijven).
Eventueel `--where` voor alleen sommige rijen (zoals `configkey = 'enabled'`). Geen web-UI:
daar kan een beheerder weinig mee.
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
PostgreSQL 18)** — voor **1.0.0-rc.2**:
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
+17 -17
View File
@@ -7,7 +7,7 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
see "Storage backend experiences" below — that requirement comes from files_external, not from
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
**Status: 1.0.3, stable.** Disaster recovery has been tested on real installations (see "Tested
**Status: 1.1.0, stable.** Disaster recovery has been tested on real installations (see "Tested
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
`docs/DISASTER-RECOVERY.md`.
@@ -87,8 +87,9 @@ All backups of an installation are encrypted with one master key.
- The **recovery kit** is your own copy of that encrypted key. Still keep it off the server and
off the location: an attacker who gets write access to the location can damage or delete its
`config` file on purpose, and without the kit that makes every backup there unreadable.
- For a **restore on a new server** (disaster recovery) you currently need the **recovery kit**
and **one passphrase**.
- For a **restore on a new server** (disaster recovery) **one passphrase** is enough: the new
server takes the key from the location. With the **recovery kit** you can restore even when the
location's `config` file is damaged or gone.
- Planned: an option to not store the key on the location at all. Then the kit is the only copy,
and losing it means losing every backup; that option will come with an explicit warning.
- Up to **three passphrases** ("slots") can open the key, so one person leaving or dying does not
@@ -170,6 +171,7 @@ Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECO
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
| The same, 1.0.2 | Hetzner Storage Box (SMB), Germany → Canada | PostgreSQL → PostgreSQL | 150,755 files restored in about 29 h (7-13 MiB/s), going on after a lost connection; sample identical, empty folders not restored ([#27](https://github.com/dillardblom/ng_backup/issues/27)) |
| 1.0.3, small simulations of 2 GB: as planned, with the restore killed halfway, and with a disk that runs full | Local SFTP server | PostgreSQL → PostgreSQL and MariaDB | all 3,101 user files and all empty folders identical; users, a share and a calendar back |
| 1.1.0, the same three simulations | Local SFTP server | PostgreSQL → PostgreSQL and MariaDB | all 3,101 user files identical in content, size and time, all empty folders back; users, a share and a calendar back |
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
@@ -266,18 +268,12 @@ What is planned, without dates. Ideas and votes are welcome in the
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
a 👍 on its issue helps decide what comes first.
**1.1.0**
- Restore without the recovery kit: the backup key from the location itself, opened with one of
your passphrases.
- A schedule per location, also "every N hours".
- Locations backed up one after the other instead of at the same time.
- All folders get their original modification time back on a restore, not only the empty ones.
- A note on a restore point that you can change afterwards.
- Pinning a restore point, so the retention policy never removes it (for example the one from
before an upgrade).
- An option to pause heavy background jobs (such as Memories, Recognize or the Preview
Generator) while a backup runs.
- Faster backups: reading and uploading at the same time.
**1.2.0**
- Faster backups over a long line: several uploads at the same time, with a setting
(2 by default) to suit the server and the size of the backup.
- Only the database from a restore point, with `occ` (no web interface): export one or more
tables to a local directory to look something up, or put chosen tables back into the running
instance in maintenance mode. For example the app settings after an update that went wrong.
**Later**
- A full restore onto the existing installation, without first installing a fresh Nextcloud, in a
@@ -293,8 +289,9 @@ On the backlog; know them before you rely on NG Backup:
- **Empty folders only come back from backups made with 1.0.3 or newer.** Older restore points do
not contain them.
- **Folders with files in them show the time of the restore**, not their original modification time.
The files keep theirs. On the roadmap.
- **Folders with files in them only get their original modification time back from backups made with
1.1.0 or newer.** From older restore points they show the time of the restore; the files keep theirs.
A restore through Nextcloud into a folder that is already there leaves that folder's time alone.
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
External storage app are not in the data directory and are **not included**, without a warning.
@@ -329,10 +326,13 @@ On the backlog; know them before you rely on NG Backup:
| `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation |
| `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots |
| `backup:target:backends/add/edit/list/remove/append-only` | backup locations |
| `backup:target:schedule` | when a location is backed up: off, daily, or every N hours |
| `backup:run`, `backup:status`, `backup:list` | back up, status (runs of the last 7 days; `--all` for all), restore points |
| `backup:cancel <run>` | stop a running backup |
| `backup:browse`, `backup:restore` | browse and restore |
| `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash |
| `backup:pin`, `backup:note` | keep a restore point, a note you can change later |
| `backup:pause-jobs` | whether heavy background jobs of other apps pause during a backup |
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
| `backup:key:import` | import a recovery kit on a fresh installation |
+9 -5
View File
@@ -45,7 +45,7 @@ welcome: use the "Share your experience" link on the admin page.
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
you need help with restoring a backup after a crash.]]></description>
<version>1.0.3</version>
<version>1.1.0</version>
<licence>agpl</licence>
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
<namespace>NgBackup</namespace>
@@ -56,10 +56,10 @@ you need help with restoring a backup after a crash.]]></description>
<website>https://github.com/dillardblom/ng_backup</website>
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/3-settings.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/4-bin.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.1.0/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.1.0/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.1.0/screenshots/3-settings.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.1.0/screenshots/4-bin.png</screenshot>
<dependencies>
<php min-version="8.1"/>
<lib>sodium</lib>
@@ -83,11 +83,15 @@ you need help with restoring a backup after a crash.]]></description>
<command>OCA\NgBackup\Command\ListSnapshots</command>
<command>OCA\NgBackup\Command\Browse</command>
<command>OCA\NgBackup\Command\Restore</command>
<command>OCA\NgBackup\Command\PauseJobs</command>
<command>OCA\NgBackup\Command\Pin</command>
<command>OCA\NgBackup\Command\Note</command>
<command>OCA\NgBackup\Command\Prune</command>
<command>OCA\NgBackup\Command\Retention</command>
<command>OCA\NgBackup\Command\TargetAppendOnly</command>
<command>OCA\NgBackup\Command\TargetTrustCurrent</command>
<command>OCA\NgBackup\Command\TargetLimit</command>
<command>OCA\NgBackup\Command\TargetSchedule</command>
<command>OCA\NgBackup\Command\Trash</command>
<command>OCA\NgBackup\Command\TrashRestore</command>
<command>OCA\NgBackup\Command\KeySlots</command>
+4 -1
View File
@@ -25,8 +25,11 @@ return [
['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'],
['name' => 'AdminApi#startBackup', 'url' => '/api/backups', 'verb' => 'POST'],
['name' => 'AdminApi#tick', 'url' => '/api/tick', 'verb' => 'POST'],
['name' => 'AdminApi#setSchedule', 'url' => '/api/schedule', 'verb' => 'PUT'],
['name' => 'AdminApi#setPauseHeavyJobs', 'url' => '/api/settings/pause-heavy-jobs', 'verb' => 'PUT'],
['name' => 'AdminApi#setTargetSchedule', 'url' => '/api/targets/{id}/schedule', 'verb' => 'PUT'],
['name' => 'AdminApi#snapshots', 'url' => '/api/targets/{targetId}/snapshots', 'verb' => 'GET'],
['name' => 'AdminApi#setSnapshotNote', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/note', 'verb' => 'PUT'],
['name' => 'AdminApi#setSnapshotPinned', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/pin', 'verb' => 'PUT'],
['name' => 'AdminApi#browse', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/browse', 'verb' => 'GET'],
['name' => 'AdminApi#startRestore', 'url' => '/api/restores', 'verb' => 'POST'],
],
+33 -9
View File
@@ -5,7 +5,8 @@ Nextcloud back from an NG Backup location. Do them in this order.
## What you need
- [ ] The **recovery kit** file (`ng_backup-kit.json`) and **one passphrase** of it.
- [ ] **One passphrase** of the old server, and if you have it, the **recovery kit** file
(`ng_backup-kit.json`).
- [ ] The **connection details of the backup location**: host, user, password or keys, bucket, folder.
- [ ] A new server with the **same Nextcloud major version** as the old one (for example 34),
a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for
@@ -15,8 +16,10 @@ Nextcloud back from an NG Backup location. Do them in this order.
- [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by
hand). Keep their packages somewhere safe beforehand.
No kit? If the location still holds its `config` file, the key is there too, but this version
can only import it from a kit. Keep the kit off the server, next to the passphrase.
No kit? Since version 1.1.0 the passphrase is enough: the key is also stored on the location (in
its `config` file), locked with the same passphrases. Skip step 4. The kit is still the safer
copy: it helps when that file on the location is damaged. Keep it off the server, next to the
passphrase.
## Steps
@@ -40,7 +43,8 @@ This keeps the new server quiet (no mail syncs, no notifications, no backups) un
**3. Install NG Backup** (the same version or newer) and enable it.
**4. Import the backup key.** Copy the kit to the new server so the web server user can read it.
**4. Import the backup key** (with a kit; without one, go to step 5). Copy the kit to the new
server so the web server user can read it.
From your own computer:
@@ -67,13 +71,29 @@ the index and shows how far it is, for example
occ backup:target:add offsite sftp -a password::password -o host=... -o user=... -o password=... -o root=... --path=...
If it says "No backup key yet", the key is not imported. Run step 4:
Without a kit (step 4 skipped), it finds the backups and asks for a passphrase of the old server;
you type it yourself. It then shows the key fingerprint and the deletion delay of the old server.
A wrong passphrase changes nothing: run the command again. Backups made before version 1.1.0 did
not store the deletion delay on the location; it is then set to 7 days.
occ backup:key:import /tmp/ng_backup-kit.json
If it says that the location's **config file is gone**, the key is only in the recovery kit:
import it (step 4), then add the location with `--repair-config` at the end of the same command.
That writes the config again from the kit; it works for locations backed up by version 1.1.0 or
newer.
Do **not** run `backup:key:init`, even though the message suggests it: that makes a brand new key,
and your backups were encrypted with the old one. NG Backup then refuses the location ("created
with a different key").
If it says "No backup key yet, and no backups at this location", the connection details or
`--path` differ from the old server's. Check them; do **not** run `backup:key:init`: that makes a
brand new key, and your backups were encrypted with the old one. NG Backup then refuses the
location ("created with a different key").
**Made a new key by mistake** (with `backup:key:init` or on the page) before adding the location?
As long as that key was never used (no location added with it, no backup), it can make way for the
key of your backups. NG Backup says so when it refuses the location. Add `--replace-unused-key` to
`occ backup:key:import` (with the kit) or to `occ backup:target:add` (without the kit; it asks for
a passphrase of the old server). The passphrases and kit of the unused key then stop working. A key
that was used for a location is never replaced: its own backups depend on it. A key made with a
version before 1.1.0 always counts as used (those versions did not keep track), also when it never
protected a location; recover onto a fresh Nextcloud installation instead.
**6. List the restore points** and pick one, normally the newest:
@@ -164,6 +184,10 @@ The command then ends with exit code 2 instead of 0. The same option exists for
**11. Make a backup of the new server** to the same location (`occ backup:run offsite`). The old
server must stay off: two servers must never write to the same location.
Restored without a kit? Then backups wait until you have a new one: download it on the backup
page (or `occ backup:key:kit --output=...`), store it off the server and confirm it
(`occ backup:key:confirm`).
## Never
- Never let the old server write to the location again if it comes back. Switch its backups off
+12 -1
View File
@@ -79,7 +79,9 @@ occ backup:list
Later runs upload only changed data. An unchanged run uploads (almost) nothing; the output shows
the number of uploaded bytes, so check that value when you test incremental behaviour.
Schedule `occ backup:run` from system cron (or use the Administration settings, if you prefer).
Set when each location is backed up in the Administration settings, or with
`occ backup:target:schedule offsite 02:00` (once a day), `02:00/6` (every 6 hours from 02:00) or `off`.
Locations are backed up one after the other.
## 4. Check the backup regularly
@@ -155,6 +157,10 @@ occ backup:list offsite # the snapshots must be lis
occ backup:restore:full offsite <snapshot>
```
Without the kit, leave out `backup:key:import`: `backup:target:add` finds the backups and takes
the key from the location with a passphrase (`--passphrase-file=/path/passphrase`, or it asks).
Backups then wait until a new kit is downloaded and confirmed.
The command refuses on an installation that already has more than one user, and asks before it
starts; `--force` skips both (also needed to rerun it after it stopped halfway).
@@ -173,8 +179,13 @@ occ backup:prune offsite --dry-run # show first
occ backup:prune offsite # apply the policy; forgotten snapshots go to the trash first
occ backup:trash # list the trash
occ backup:trash:restore offsite <id>
occ backup:pin offsite <id> # keep this restore point; --unpin to undo
occ backup:note offsite <id> "text" # a note you can change later; without text it is removed
```
The retention policy leaves a pinned restore point alone, and a pinned one does not take the place of
a daily, weekly or monthly one. Pin and note are stored on the location, so a new server sees them too.
Snapshots are kept according to the policy, and their data is deleted only after the deletion delay
(set with `backup:key:init --delete-delay-days`). `occ backup:prune offsite --dry-run` shows what would be
removed without changing anything.
+3
View File
@@ -67,6 +67,9 @@ final class LocalBackend implements IBackend {
if ($fh === false) {
throw new BackendException('Not found: ' . $path);
}
// Packs are read in large pieces: one request each, not 8 KiB at a time (see Repository::openForReading()).
stream_set_read_buffer($fh, 0);
stream_set_chunk_size($fh, 4 * 1048576);
return $fh;
}
+28 -16
View File
@@ -33,6 +33,7 @@ final class BackupJob extends TimedJob {
private IAppConfig $appConfig,
private LoggerInterface $logger,
private \OCA\NgBackup\Service\Schedule $schedule,
private \OCA\NgBackup\Service\HeavyJobPause $heavyJobs,
) {
parent::__construct($time);
// Well below the usual 5-minute cron interval: the last run is stamped a moment after cron
@@ -43,25 +44,28 @@ final class BackupJob extends TimedJob {
$this->setTimeSensitivity(IJob::TIME_SENSITIVE);
}
private const MAX_DAILY_ATTEMPTS = 3;
protected function run($argument): void {
$budget = PHP_SAPI === 'cli' ? 240.0 : 20.0;
$deadline = microtime(true) + $budget;
$this->startScheduled();
foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) {
$this->heavyJobs->sync($this->backups->runsInTurn() !== []);
// One after the other: runs that read the same disks at the same time are not faster together.
foreach ($this->backups->runsInTurn() as $run) {
while (microtime(true) < $deadline && $run->getStatus() === 'running') {
$next = $this->backups->step($run, min($deadline, microtime(true) + $budget));
if ($next === $run) {
break; // step() hands back the same object when another process holds the run's lease
break 2; // step() hands back the same object when another process holds the run's lease
}
$run = $next;
}
}
$this->failStaleRestores();
$this->weeklyPrune();
foreach ($this->targets->list() as $t) {
$targets = $this->targets->list();
foreach ($targets as $t) {
$this->alerts->checkStale($t);
$othersActive = array_filter($targets, static fn ($o) => $o->getId() !== $t->getId() && (string)$o->getSchedule() !== '') !== [];
$this->alerts->checkOff($t, $othersActive);
}
}
@@ -102,21 +106,18 @@ final class BackupJob extends TimedJob {
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
}
/** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */
/** Start the backups the schedules of the locations ask for; they run in turn (see LocationSchedule). */
private function startScheduled(): void {
$todayAt = $this->schedule->todayAt();
if ($todayAt === null || time() < $todayAt) {
return;
}
$now = time();
foreach ($this->targets->list() as $t) {
$last = $this->runs->lastFinished($t->getId(), BackupService::KIND_FULL);
if ($this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== [] || ($last !== null && $last->getStartedAt() >= $todayAt)) {
$moment = \OCA\NgBackup\Service\LocationSchedule::lastMoment((string)$t->getSchedule(), $this->schedule->zoneOr($t->getScheduleZone()), $now);
if ($moment === null || $this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== []) {
continue;
}
// After a failure: retry at most MAX_DAILY_ATTEMPTS times a day, an hour apart, instead
// of starting (and notifying about) a new full run every five minutes all day.
$today = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $todayAt);
if (count($today) >= self::MAX_DAILY_ATTEMPTS || ($today !== [] && time() - (int)$today[0]->getFinishedAt() < 3600)) {
$attempts = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $moment);
if (!\OCA\NgBackup\Service\LocationSchedule::isDue($moment, (int)$t->getScheduleSince(),
$this->dataStarted($this->runs->lastFinished($t->getId(), BackupService::KIND_FULL)),
count($attempts), ($attempts[0] ?? null)?->getFinishedAt(), $now)) {
continue;
}
try {
@@ -126,4 +127,15 @@ final class BackupJob extends TimedJob {
}
}
}
/** When a backup began reading its data: a queued one can start well after it was asked for. */
private function dataStarted(?\OCA\NgBackup\Db\Run $run): ?int {
if ($run === null) {
return null;
}
$state = json_decode($run->getState(), true);
$started = is_array($state) ? ($state['files']['started'] ?? null) : null;
$at = is_string($started) ? strtotime($started) : false;
return $at !== false ? $at : $run->getStartedAt();
}
}
+32 -4
View File
@@ -8,6 +8,7 @@ declare(strict_types=1);
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\KeyService;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
@@ -18,6 +19,7 @@ use Symfony\Component\Console\Question\Question;
final class KeyImport extends Command {
public function __construct(
private KeyService $keys,
private TargetService $targets,
) {
parent::__construct();
}
@@ -27,11 +29,19 @@ final class KeyImport extends Command {
->setDescription('Disaster recovery: adopt the backup key from a recovery kit on a fresh installation')
->addArgument('kit-file', InputArgument::REQUIRED, 'The recovery kit file (occ backup:key:kit on the original installation)')
->addOption('passphrase-file', null, InputOption::VALUE_REQUIRED, 'Read the passphrase from this file instead of asking')
->addOption('replace-unused-key', null, InputOption::VALUE_NONE, 'A new key was made here by mistake: replace it by the kit\'s key, if it was never used (no location, no backup)')
->setHelp(<<<'HELP'
For a fresh installation only (occ backup:key:init must not have been run here yet). Reads one
of the kit's passphrase slots to recover the original master key, so occ backup:target:add opens
the existing repository on the location instead of creating a new, empty one.
A new key was made here by mistake before the location was added? As long as it was never used
(no location, no backup), --replace-unused-key replaces it by the kit's key. A key that was used
is never replaced.
No kit? Skip this command: occ backup:target:add then takes the key from the location with one
of the old server's passphrases. The kit still helps when the location's config file is damaged.
Example:
occ backup:key:import /root/ng_backup-recovery-kit.json
occ backup:target:add offsite sftp -a password::password -o host=... -o user=... -o password=...
@@ -40,10 +50,24 @@ HELP);
}
protected function execute(InputInterface $input, OutputInterface $output): int {
if ($this->keys->isInitialized()) {
$output->writeln('<error>A backup key already exists here (fingerprint ' . $this->keys->fingerprint() . '). occ backup:key:import is only for a fresh installation.</error>');
$replace = (bool)$input->getOption('replace-unused-key');
if ($replace && !$this->keys->isInitialized()) {
$output->writeln('<error>There is no backup key here to replace: leave out --replace-unused-key.</error>');
return 1;
}
if ($this->keys->isInitialized()) {
if (!$this->targets->keyIsUnused()) {
$output->writeln('<error>A backup key already exists here (fingerprint ' . $this->keys->fingerprint() . ') and it has been used for a backup location: it is never replaced.</error>');
$output->writeln('occ backup:key:import is for a server that is being restored.');
return 1;
}
if (!$replace) {
$output->writeln('<error>A backup key already exists here (fingerprint ' . $this->keys->fingerprint() . '), but it was never used: no location, no backup.</error>');
$output->writeln('Made by mistake on a server that is being restored? Run this command again with --replace-unused-key:');
$output->writeln('the kit\'s key then takes its place, and the passphrases and recovery kit of the unused key stop working.');
return 1;
}
}
$kitFile = (string)$input->getArgument('kit-file');
$raw = @file_get_contents($kitFile);
if ($raw === false) {
@@ -70,12 +94,16 @@ HELP);
$pass = (string)$helper->ask($input, $output, $q);
}
try {
$this->keys->importRecoveryKit($kit, $pass);
if ($replace) {
$this->targets->importKitReplacingUnusedKey($kit, $pass);
} else {
$this->keys->importRecoveryKit($kit, $pass);
}
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
$output->writeln('Backup key imported, fingerprint <info>' . $this->keys->fingerprint() . '</info>.');
$output->writeln(($replace ? 'The unused key was replaced. ' : '') . 'Backup key imported, fingerprint <info>' . $this->keys->fingerprint() . '</info>.');
// Read from the kit file, which is not authenticated beyond the wrapped key: show it so a
// wrong or altered value stands out.
$output->writeln('Deletion delay for forgotten snapshots: <info>' . $this->keys->deleteDelayDays() . '</info> days (from the kit).');
+2 -2
View File
@@ -40,14 +40,14 @@ final class ListSnapshots extends Command {
foreach ($targets as $t) {
foreach ($this->snapshots->findForTarget($t->getId()) as $s) {
$rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(),
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getLabel() ?? ''];
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getPinned() ? 'yes' : '', $s->getLabel() ?? '', $s->getNote() ?? ''];
}
}
if ($rows === []) {
$output->writeln('No snapshots yet (occ backup:run).');
return 0;
}
(new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'label'])->setRows($rows)->render();
(new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'pinned', 'label', 'note'])->setRows($rows)->render();
return 0;
}
}
+44
View File
@@ -0,0 +1,44 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\PruneService;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
final class Note extends Command {
public function __construct(
private TargetService $targets,
private PruneService $prune,
) {
parent::__construct();
}
protected function configure(): void {
$this->setName('backup:note')
->setDescription('Set or remove the note on a snapshot')
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)')
->addArgument('note', InputArgument::OPTIONAL, 'The note, at most ' . PruneService::NOTE_MAX . ' characters; without it the note is removed', '');
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$note = (string)$input->getArgument('note');
try {
$this->prune->setNote($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), $note);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
$output->writeln(trim($note) === '' ? 'Note removed.' : 'Note saved.');
return 0;
}
}
+45
View File
@@ -0,0 +1,45 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\HeavyJobPause;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
final class PauseJobs extends Command {
public function __construct(
private HeavyJobPause $pause,
) {
parent::__construct();
}
protected function configure(): void {
$this->setName('backup:pause-jobs')
->setDescription('Show or set whether heavy background jobs (Memories, Recognize, Preview Generator, ...) pause while a backup runs')
->addArgument('setting', InputArgument::OPTIONAL, '"on" or "off"; omit to show the current one');
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$setting = $input->getArgument('setting');
if ($setting !== null) {
if ($setting !== 'on' && $setting !== 'off') {
$output->writeln('<error>Use "on" or "off".</error>');
return 1;
}
$this->pause->setEnabled($setting === 'on');
}
$status = $this->pause->status();
$output->writeln('Pausing heavy background jobs during a backup: ' . ($status['enabled'] ? 'on' : 'off'));
if ($status['jobs'] !== []) {
$output->writeln('Paused now: ' . implode(', ', $status['jobs']));
}
return 0;
}
}
+45
View File
@@ -0,0 +1,45 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\PruneService;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
final class Pin extends Command {
public function __construct(
private TargetService $targets,
private PruneService $prune,
) {
parent::__construct();
}
protected function configure(): void {
$this->setName('backup:pin')
->setDescription('Keep a snapshot: the retention policy does not remove a pinned one')
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)')
->addOption('unpin', null, InputOption::VALUE_NONE, 'Leave the snapshot to the retention policy again');
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$unpin = (bool)$input->getOption('unpin');
try {
$this->prune->setPinned($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), !$unpin);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
$output->writeln($unpin ? 'Snapshot unpinned; the retention policy applies to it again.' : 'Snapshot pinned; the retention policy leaves it alone.');
return 0;
}
}
+3
View File
@@ -79,6 +79,9 @@ HELP);
if (($f['unreadable'] ?? 0) > 0) {
$output->writeln(sprintf('<comment> warning: %d unreadable files or folders are missing from this backup (see the server log)</comment>', $f['unreadable']));
}
if (($f['badNames'] ?? 0) > 0) {
$output->writeln(sprintf('<comment> warning: %d files or folders are missing from this backup because their name is not supported; rename them and the next backup includes them (see the server log)</comment>', $f['badNames']));
}
if (($f['changing'] ?? 0) > 0) {
$output->writeln(sprintf('<comment> warning: %d files kept changing and were saved as read (see the server log)</comment>', $f['changing']));
}
+4 -1
View File
@@ -31,6 +31,7 @@ final class Status extends Command {
private VerifyService $verify,
private Schedule $schedule,
private CronHealth $cron,
private \OCA\NgBackup\Service\BackupService $backups,
) {
parent::__construct();
}
@@ -61,9 +62,11 @@ final class Status extends Command {
$rows = [];
$all = (bool)$input->getOption('all');
$shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400);
$queued = array_map(static fn (\OCA\NgBackup\Db\Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1));
foreach ($shown as $r) {
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus(),
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(),
in_array($r->getId(), $queued, true) ? 'queued' : ($r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus()),
Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
+120 -9
View File
@@ -13,7 +13,9 @@ use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Helper\QuestionHelper;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Question\Question;
final class TargetAdd extends Command {
public function __construct(
@@ -31,10 +33,25 @@ final class TargetAdd extends Command {
->addOption('auth', 'a', InputOption::VALUE_REQUIRED, 'Auth mechanism id (see backup:target:backends)')
->addOption('option', 'o', InputOption::VALUE_REQUIRED | InputOption::VALUE_IS_ARRAY, 'Backend/auth option key=value (repeatable)')
->addOption('path', 'p', InputOption::VALUE_REQUIRED, 'Folder on the location for this repository', 'ng_backup')
->addOption('passphrase-file', null, InputOption::VALUE_REQUIRED, 'Disaster recovery without the recovery kit: read the passphrase from this file instead of asking')
->addOption('repair-config', null, InputOption::VALUE_NONE, 'The location\'s config file is gone: write it again with the key of this server or of the imported recovery kit (locations backed up by 1.1.0 or newer)')
->addOption('replace-unused-key', null, InputOption::VALUE_NONE, 'Disaster recovery after a new key was made here by mistake: replace that key, if it was never used, by the key of the backups at this location (asks for a passphrase of the old server)')
->setHelp(<<<'HELP'
The location is reached with files_external's own connection code, but it is never added as an
external storage: it does not appear in Files for anyone. Credentials are stored encrypted.
Disaster recovery without the recovery kit: on a server without a backup key, a location that
holds backups gives its key with one of the old server's passphrases. Run this command first
(no occ backup:key:import), then occ backup:list and occ backup:restore:full.
A new key was made here by mistake (occ backup:key:init or the web page) before the location
was added? As long as that key was never used (no location, no backup), --replace-unused-key
replaces it by the key of the backups. A key that was used is never replaced.
With the recovery kit: occ backup:key:import first; this command then asks for nothing more.
If the location's config file is gone, only the kit has the key: import it, then add the
location with --repair-config.
Examples:
occ backup:target:add offsite amazons3 -a amazons3::accesskey -o bucket=backups -o hostname=s3.example.com -o region=eu-central-1 -o use_ssl=true -o use_path_style=true -o key=AKIA... -o secret=... --path=nextcloud
@@ -43,13 +60,6 @@ HELP);
}
protected function execute(InputInterface $input, OutputInterface $output): int {
if (!$this->keys->isInitialized()) {
$output->writeln('<error>No backup key yet.</error>');
$output->writeln('Restoring from a location that already has backups (disaster recovery)? Import the key from the recovery kit first:');
$output->writeln(' occ backup:key:import /path/to/ng_backup-kit.json');
$output->writeln('Only for a new location without backups: create a new key with occ backup:key:init. A new key cannot read existing backups.');
return 1;
}
$options = [];
foreach ($input->getOption('option') as $pair) {
if (!str_contains($pair, '=')) {
@@ -63,24 +73,125 @@ HELP);
$output->writeln('<error>--auth is required (see occ backup:target:backends)</error>');
return 1;
}
$name = (string)$input->getArgument('name');
$backend = (string)$input->getArgument('backend');
$auth = (string)$input->getOption('auth');
$path = (string)$input->getOption('path');
$passphrase = null;
$repair = (bool)$input->getOption('repair-config');
$replace = (bool)$input->getOption('replace-unused-key');
if ($replace) {
if ($repair) {
$output->writeln('<error>--replace-unused-key needs the config at the location; with the config gone only the kit helps: occ backup:key:import <kit file> --replace-unused-key, then --repair-config.</error>');
return 1;
}
if (!$this->keys->isInitialized()) {
$output->writeln('<error>There is no backup key here to replace: leave out --replace-unused-key.</error>');
return 1;
}
if (!$this->targets->keyIsUnused()) {
$output->writeln('<error>The backup key of this server (fingerprint ' . $this->keys->fingerprint() . ') has been used for a backup location: it is never replaced.</error>');
return 1;
}
$output->writeln('The backup key of this server (fingerprint ' . $this->keys->fingerprint() . ') was never used. It is replaced by the key of the backups at');
$output->writeln('this location, once a passphrase of the old server opens it. Its own passphrases and recovery kit then stop working.');
$passphrase = $this->passphrase($input, $output);
if ($passphrase === null) {
return 1;
}
} elseif (!$this->keys->isInitialized()) {
try {
$state = $this->targets->locationState($backend, $auth, $options, $path);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
if ($state === TargetService::LOCATION_EMPTY) {
$output->writeln('<error>No backup key yet, and no backups at this location.</error>');
$output->writeln('Restoring a server (disaster recovery)? Check the connection details and --path: they must be the same as on the old server.');
$output->writeln('Only for a new location: create a new key first with occ backup:key:init. A new key cannot read existing backups.');
return 1;
}
if ($state === TargetService::LOCATION_DATA_WITHOUT_CONFIG) {
$output->writeln('<error>This location holds backups, but its config file is gone, and the key was stored in it.</error>');
$output->writeln('Now only the recovery kit has the key. Import it, then run this command again with --repair-config:');
$output->writeln(' occ backup:key:import /path/to/ng_backup-kit.json');
$output->writeln('Do not run occ backup:key:init: a new key cannot read these backups.');
return 1;
}
if ($repair) {
$output->writeln('<error>The config at this location is there: leave out --repair-config.</error>');
return 1;
}
$output->writeln('No backup key on this server yet, and this location holds backups. Their key is stored there too,');
$output->writeln('locked with the passphrases of the old server: one of them unlocks it (the recovery kit is not needed).');
$passphrase = $this->passphrase($input, $output);
if ($passphrase === null) {
return 1;
}
}
// Adopting an existing repository reads its whole index first, which takes a while on a
// large backup or a slow line; show how far it is instead of looking stuck.
$output->writeln('Connecting to the location; with existing backups this reads their index first ...');
try {
$r = $this->targets->add($input->getArgument('name'), $input->getArgument('backend'), $input->getOption('auth'), $options,
(string)$input->getOption('path'), self::progress($output));
$r = $passphrase === null
? $this->targets->add($name, $backend, $auth, $options, $path, self::progress($output), $repair)
: $this->targets->addWithPassphrase($name, $backend, $auth, $options, $path, $passphrase, self::progress($output), $replace);
} catch (\OCA\NgBackup\Crypto\CryptoException $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
if (!$this->keys->isInitialized() || $replace) {
$output->writeln('Nothing was changed. Run the command again to try another passphrase, or use the recovery kit (occ backup:key:import).');
}
return 1;
} catch (\Throwable $e) {
if ($output->isDecorated()) {
$output->write("\r\033[K"); // clear an unfinished progress line
}
$output->writeln('<error>' . $e->getMessage() . '</error>');
if (str_contains($e->getMessage(), 'different key') && !$replace) {
$output->writeln('The backup key here (fingerprint ' . $this->keys->fingerprint() . ') is not the one these backups were made with.');
$output->writeln('Restoring a server? The recovery kit must be the old server\'s: its fingerprint is in the kit file. Nothing was changed.');
}
return 1;
}
if ($repair) {
$output->writeln('The config of the location was written again from the recovery kit.');
}
$output->writeln(sprintf('Location <info>%s</info> added (id %d), repository %s %s.', $r['target']->getName(), $r['target']->getId(),
$r['target']->getRepositoryId(), $r['created'] ? 'created' : 'found and adopted'));
if ($passphrase !== null) {
$output->writeln(($replace ? 'The unused key was replaced. ' : '') . 'Backup key taken from the location, fingerprint <info>' . $this->keys->fingerprint() . '</info>.');
$days = 'Deletion delay for forgotten snapshots: <info>' . $this->keys->deleteDelayDays() . '</info> days';
$output->writeln(match (true) {
($r['deleteDelayDays'] ?? null) !== null => $days . ' (from the location).',
$r['settingsUnreadable'] ?? false => '<comment>' . $days . ' (the longest: the settings on the location could not be read or hold no valid delay, so nothing leaves the Bin sooner than on the old server; removed restore points take space for that long).</comment>',
default => $days . ' (the default: the location has none stored, its backups were made before NG Backup 1.1.0).',
});
$output->writeln('Next: <info>occ backup:list</info> and <info>occ backup:restore:full</info>. Before backups run again, download a new recovery kit and confirm it (<info>occ backup:key:kit</info>, <info>occ backup:key:confirm</info>).');
}
return 0;
}
private function passphrase(InputInterface $input, OutputInterface $output): ?string {
$file = $input->getOption('passphrase-file');
if ($file !== null) {
$raw = @file_get_contents((string)$file);
if ($raw === false) {
$output->writeln('<error>Cannot read the passphrase file</error>');
return null;
}
return rtrim($raw, "\r\n");
}
if (!$input->isInteractive()) {
$output->writeln('<error>Give the passphrase with --passphrase-file, or run the command without --no-interaction to type it.</error>');
return null;
}
$q = (new Question('Passphrase of the old server (one of its slots): '))->setHidden(true)->setHiddenFallback(false);
/** @var QuestionHelper $helper */
$helper = $this->getHelper('question');
return (string)$helper->ask($input, $output, $q);
}
/**
* Progress of reading the index: on a terminal one line that is updated in place, otherwise
* (a log, a pipe) a line every 30 seconds. The time left is estimated from the rate so far.
+2 -1
View File
@@ -31,7 +31,8 @@ final class TargetList extends Base {
$rows = [];
foreach ($this->targets->list() as $t) {
$row = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt())];
'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt()),
'schedule' => (string)$t->getSchedule() !== '' ? $t->getSchedule() : 'off'];
if ($input->getOption('test')) {
try {
$repo = $this->targets->repository($t);
+54
View File
@@ -0,0 +1,54 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
final class TargetSchedule extends Command {
public function __construct(
private TargetService $targets,
private Schedule $schedule,
) {
parent::__construct();
}
protected function configure(): void {
$this->setName('backup:target:schedule')
->setDescription('Show or set when a location is backed up')
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('schedule', InputArgument::OPTIONAL, '"off", HH:MM for once a day, or HH:MM/N for every N hours from HH:MM; omit to show the current one')
->addOption('timezone', null, InputOption::VALUE_REQUIRED, 'Time zone of the time, e.g. Europe/Amsterdam');
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$t = $this->targets->get($input->getArgument('target'));
$schedule = $input->getArgument('schedule');
if ($schedule === null && $input->getOption('timezone') !== null) {
$output->writeln('<error>--timezone goes with a schedule, e.g. backup:target:schedule ' . $t->getName() . ' 02:30 --timezone=Europe/Amsterdam</error>');
return 1;
}
$zone = $input->getOption('timezone') ?? $this->schedule->zoneOr($t->getScheduleZone());
if ($schedule !== null) {
try {
$t = $this->targets->setSchedule($t, $schedule, (string)$zone);
} catch (\InvalidArgumentException $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
}
$current = (string)$t->getSchedule();
$output->writeln($t->getName() . ': ' . ($current === '' ? 'off' : $current . ' (' . $zone . ')'));
return 0;
}
}
+42 -16
View File
@@ -48,6 +48,7 @@ final class AdminApiController extends Controller {
private \OCA\NgBackup\Service\KeySlotService $slots,
private \OCA\NgBackup\Service\PruneService $prune,
private \OCA\NgBackup\Service\Schedule $schedule,
private \OCA\NgBackup\Service\HeavyJobPause $heavyJobs,
private \OCA\NgBackup\Service\CronHealth $cron,
) {
parent::__construct(Application::APP_ID, $request);
@@ -59,10 +60,12 @@ final class AdminApiController extends Controller {
foreach ($this->targets->list() as $t) {
$targets[] = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
'repositoryId' => $t->getRepositoryId(), 'createdAt' => $t->getCreatedAt(), 'appendOnly' => (bool)$t->getAppendOnly(),
'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t)];
'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t),
'schedule' => (string)$t->getSchedule(), 'scheduleZone' => $this->schedule->zoneOr($t->getScheduleZone())];
}
$shown = $this->runs->findShown($allRuns ? null : time() - \OCA\NgBackup\Command\Status::RECENT_DAYS * 86400);
$runs = array_map(fn (Run $r) => $this->runJson($r), $shown);
$queued = array_flip(array_map(static fn (Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1)));
$runs = array_map(fn (Run $r) => $this->runJson($r) + ['queued' => isset($queued[$r->getId()])], $shown);
return new JSONResponse([
'key' => ['initialized' => $this->keys->isInitialized(), 'fingerprint' => $this->keys->fingerprint(),
'confirmation' => $this->keys->recoveryKitConfirmation(), 'everConfirmed' => $this->keys->isInitialized() && $this->keys->everConfirmed(),
@@ -74,12 +77,16 @@ final class AdminApiController extends Controller {
'runs' => $runs,
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
'schedule' => $this->schedule->get()['time'],
'scheduleTimezone' => $this->schedule->get()['timezone'],
'cron' => $this->cron->check(),
'heavyJobs' => $this->heavyJobs->status(),
]);
}
public function setPauseHeavyJobs(bool $enabled): JSONResponse {
$this->heavyJobs->setEnabled($enabled);
return new JSONResponse(['enabled' => $enabled]);
}
/** Seconds of backup work one tick() does: well within a normal web request. */
public const TICK_SECONDS = 10.0;
/** Seconds one request for the Bin reads, before the page shows how far it is and asks for more. */
@@ -97,16 +104,17 @@ final class AdminApiController extends Controller {
}
$deadline = microtime(true) + self::TICK_SECONDS;
$stepped = 0;
$running = $this->runs->findRunning(null, BackupService::KIND_FULL);
// Longest wait first: a run that takes all of a tick does not keep another from its turn.
usort($running, static fn ($a, $b) => $a->getUpdatedAt() <=> $b->getUpdatedAt());
foreach ($running as $run) {
$this->heavyJobs->sync($this->backups->runsInTurn() !== []);
foreach ($this->backups->runsInTurn() as $run) {
if (microtime(true) >= $deadline) {
break;
}
// step() returns at once when the background job (or another tab) holds the run.
$this->backups->step($run, $deadline);
$next = $this->backups->step($run, $deadline);
$stepped++;
if ($next->getStatus() === Run::RUNNING) {
break; // not finished: the next one waits for its turn
}
}
return new JSONResponse(['stepped' => $stepped]);
}
@@ -278,25 +286,43 @@ final class AdminApiController extends Controller {
return new JSONResponse($this->runJson($run));
}
/** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */
public function setSchedule(string $time, string $timezone = ''): JSONResponse {
/** @param string $timezone the administrator's zone (the browser's); empty: the default one */
public function setTargetSchedule(int $id, string $schedule, string $timezone = ''): JSONResponse {
try {
$this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
} catch (\InvalidArgumentException $e) {
$t = $this->targets->setSchedule($this->targets->get((string)$id), $schedule, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
} catch (\Throwable $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
}
return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]);
return new JSONResponse(['schedule' => $t->getSchedule(), 'scheduleZone' => $t->getScheduleZone()]);
}
public function snapshots(int $targetId): JSONResponse {
$out = [];
foreach ($this->snapshots->findForTarget($targetId) as $s) {
$out[] = ['id' => $s->getSnapshotId(), 'kind' => $s->getKind(), 'label' => $s->getLabel(), 'createdAt' => $s->getCreatedAt(),
'files' => $s->getFiles(), 'bytes' => $s->getBytes()];
'files' => $s->getFiles(), 'bytes' => $s->getBytes(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()];
}
return new JSONResponse($out);
}
public function setSnapshotNote(int $targetId, string $snapshotId, string $note = ''): JSONResponse {
try {
$this->prune->setNote($this->targets->get((string)$targetId), $snapshotId, $note);
} catch (\Throwable $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
}
return new JSONResponse(['ok' => true]);
}
public function setSnapshotPinned(int $targetId, string $snapshotId, bool $pinned = true): JSONResponse {
try {
$this->prune->setPinned($this->targets->get((string)$targetId), $snapshotId, $pinned);
} catch (\Throwable $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
}
return new JSONResponse(['ok' => true]);
}
public function browse(int $targetId, string $snapshotId, string $path = ''): JSONResponse {
try {
return new JSONResponse($this->restore->browse($this->targets->get((string)$targetId), $snapshotId, $path));
@@ -341,7 +367,7 @@ final class AdminApiController extends Controller {
if ($r->getSnapshotId() !== null) {
try {
$s = $this->snapshots->findOne($r->getTargetId(), $r->getSnapshotId());
$snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel()];
$snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()];
} catch (\Throwable) {
// Snapshot already pruned or trashed: the run keeps its id, just not the date.
}
+24 -6
View File
@@ -87,14 +87,20 @@ final class KeyRing {
foreach ($field['slots'] as $slot) {
try {
return self::unwrap($slot['key'], $passphrase);
} catch (CryptoException) {
} catch (CryptoException|\SodiumException|\TypeError|\ValueError) {
// a wrong passphrase for this slot, or a damaged slot: the next one may still open
}
}
throw new CryptoException('Wrong passphrase or damaged key');
}
/** @param array{v:int, salt:string, ops:int, mem:int, nonce:string, key:string} $wrapped */
/** @param array<string, mixed> $wrapped from a kit file or a location's config: checked here, not trusted */
public static function unwrap(array $wrapped, #[\SensitiveParameter] string $passphrase): self {
if (!isset($wrapped['salt'], $wrapped['nonce'], $wrapped['key'], $wrapped['ops'], $wrapped['mem'])
|| !is_string($wrapped['salt']) || !is_string($wrapped['nonce']) || !is_string($wrapped['key'])
|| !is_int($wrapped['ops']) || !is_int($wrapped['mem'])) {
throw new CryptoException('Damaged key file: a part of the key is missing');
}
$b64 = static fn (string $s): string => sodium_base642bin($s, SODIUM_BASE64_VARIANT_ORIGINAL);
// ops/mem come from unauthenticated JSON (kit file, repository config): bound them, so a
// damaged or crafted file cannot make the KDF use unbounded memory or time.
@@ -102,10 +108,22 @@ final class KeyRing {
|| $wrapped['mem'] < SODIUM_CRYPTO_PWHASH_MEMLIMIT_INTERACTIVE || $wrapped['mem'] > SODIUM_CRYPTO_PWHASH_MEMLIMIT_SENSITIVE) {
throw new CryptoException('Unsupported key derivation parameters (damaged or foreign key file)');
}
$kek = sodium_crypto_pwhash(SODIUM_CRYPTO_SECRETBOX_KEYBYTES, $passphrase, $b64($wrapped['salt']),
$wrapped['ops'], $wrapped['mem'], SODIUM_CRYPTO_PWHASH_ALG_ARGON2ID13);
$master = sodium_crypto_secretbox_open($b64($wrapped['key']), $b64($wrapped['nonce']), $kek);
sodium_memzero($kek);
$kek = null;
try {
$salt = $b64($wrapped['salt']);
$sealed = $b64($wrapped['key']);
$nonce = $b64($wrapped['nonce']);
$kek = sodium_crypto_pwhash(SODIUM_CRYPTO_SECRETBOX_KEYBYTES, $passphrase, $salt,
$wrapped['ops'], $wrapped['mem'], SODIUM_CRYPTO_PWHASH_ALG_ARGON2ID13);
$master = sodium_crypto_secretbox_open($sealed, $nonce, $kek);
} catch (\SodiumException|\ValueError $e) {
// invalid base64 or a salt/nonce of the wrong length: damaged, so the caller shows its recovery options
throw new CryptoException('Damaged key file: ' . $e->getMessage(), 0, $e);
} finally {
if ($kek !== null) {
sodium_memzero($kek);
}
}
if ($master === false) {
throw new CryptoException('Wrong passphrase or damaged key');
}
+7
View File
@@ -24,6 +24,10 @@ use OCP\AppFramework\Db\Entity;
* @method void setFiles(?int $v)
* @method int|null getBytes()
* @method void setBytes(?int $v)
* @method string|null getNote()
* @method void setNote(?string $v)
* @method bool|null getPinned()
* @method void setPinned(?bool $v)
*/
final class Snapshot extends Entity {
protected int $targetId = 0;
@@ -33,8 +37,11 @@ final class Snapshot extends Entity {
protected int $createdAt = 0;
protected ?int $files = null;
protected ?int $bytes = null;
protected ?string $note = null;
protected ?bool $pinned = null;
public function __construct() {
$this->addType('pinned', 'boolean');
$this->addType('targetId', 'integer');
$this->addType('createdAt', 'integer');
$this->addType('files', 'integer');
+10
View File
@@ -28,6 +28,12 @@ use OCP\AppFramework\Db\Entity;
* @method void setMaxBytes(?int $v)
* @method int getCreatedAt()
* @method void setCreatedAt(int $v)
* @method string|null getSchedule()
* @method void setSchedule(?string $v)
* @method string|null getScheduleZone()
* @method void setScheduleZone(?string $v)
* @method int|null getScheduleSince()
* @method void setScheduleSince(?int $v)
*/
final class Target extends Entity {
protected string $name = '';
@@ -39,8 +45,12 @@ final class Target extends Entity {
protected int $createdAt = 0;
protected ?bool $appendOnly = null;
protected ?int $maxBytes = null;
protected ?string $schedule = null;
protected ?string $scheduleZone = null;
protected ?int $scheduleSince = null;
public function __construct() {
$this->addType('scheduleSince', 'integer');
$this->addType('createdAt', 'integer');
$this->addType('appendOnly', 'boolean');
$this->addType('maxBytes', 'integer');
+47 -6
View File
@@ -52,19 +52,35 @@ final class BackupRun {
$stats = $state['stats'];
$walker = new TreeWalker($state['roots'], $state['exclude'], function (string $logical) use (&$state, &$stats): void {
self::recordUnreadable($state, $stats, $logical);
});
}, true);
$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
$lines = [];
$emptyDirs = [];
$folders = [];
$finishedWalk = true;
foreach ($walker->walk($state['after']) as $logical => $path) {
if (!mb_check_encoding($logical, 'UTF-8') || !Repository::isSafePath($logical)) {
$safe = mb_check_encoding($logical, 'UTF-8') && Repository::isSafePath($logical);
if ($walker->atFolderWithContent()) {
// Only its time: what is in it is reported by itself, and a bad name in the path of
// each of those is what gets reported. The cursor stays for such a name, see below.
$mtime = $safe ? @filemtime($path) : false;
if ($safe && $mtime !== false) {
$folders[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR);
$state['after'] = $logical;
if (microtime(true) >= $deadline) {
$finishedWalk = false;
break;
}
}
continue;
}
if (!$safe) {
// A name a snapshot cannot hold (a backslash, a control character, not UTF-8). One
// such path must not make a backup fail, or the whole snapshot unreadable later.
// The cursor stays where it is: the state is JSON, which cannot hold such a name. A
// next step meets it again, and it is reported once.
self::recordUnreadable($state, $stats, mb_scrub($logical, 'UTF-8'));
self::recordBadName($state, $stats, $logical);
continue;
}
if (is_dir($path)) {
@@ -104,7 +120,7 @@ final class BackupRun {
if ($cur === null || $cur['path'] !== $logical) {
$cur = ['path' => $logical, 'offset' => 0, 'blobs' => [], 'version' => $version];
}
$fh = @fopen($path, 'rb');
$fh = Repository::openForReading($path);
if ($fh === false) {
self::recordUnreadable($state, $stats, $logical);
$state['cur'] = null;
@@ -164,6 +180,11 @@ final class BackupRun {
$writer->write(implode("\n", $emptyDirs) . "\n");
$state['dirSegments'][] = $writer->finish();
}
if ($folders !== []) {
$writer = $repo->blobWriter($stats);
$writer->write(implode("\n", $folders) . "\n");
$state['folderSegments'][] = $writer->finish();
}
$repo->flushPacks();
$state['parentPos'] = $parent->position();
$state['stats'] = $stats;
@@ -186,10 +207,14 @@ final class BackupRun {
$emptyDirs[] = json_decode($line, true, 8, JSON_THROW_ON_ERROR);
}
}
// The times of the other folders too, packed: every look at a snapshot's meta record reads
// this, and a string of a few hundred KiB costs that far less than tens of thousands of arrays.
$folders = isset($state['folderSegments']) ? implode("\n", iterator_to_array($repo->readLines(array_values(array_merge(...$state['folderSegments']))), false)) : '';
$repo->putSnapshot($state['snapshot'],
['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
'warnings' => self::warnings($state)] + ($emptyDirs === [] ? [] : ['emptyDirs' => $emptyDirs]) + $state['meta'],
'warnings' => self::warnings($state)] + ($emptyDirs === [] ? [] : ['emptyDirs' => $emptyDirs])
+ ($folders === '' ? [] : ['folderTimes' => base64_encode((string)gzdeflate($folders))]) + $state['meta'],
$repo->readLines(array_values(array_merge(...$state['segments']))));
$state['phase'] = 'commit';
}
@@ -250,10 +275,26 @@ final class BackupRun {
$stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1;
}
/** @return array{unreadable: list<string>, changing: list<string>} examples for the report (first 50 each) */
/**
* A file or directory left out because of its name. Remembered like an unreadable one, in a form
* that fits the JSON state and is safe to show: control characters and invalid bytes replaced.
*/
private static function recordBadName(array &$state, array &$stats, string $logical): void {
$shown = (string)preg_replace('/[\x00-\x1f\x7f]/', '?', mb_scrub($logical, 'UTF-8'));
if (isset($state['badNames'][$shown])) {
return;
}
if (count($state['badNames'] ?? []) < 1000) {
$state['badNames'][$shown] = true;
}
$stats['badNames'] = ($stats['badNames'] ?? 0) + 1;
}
/** @return array{unreadable: list<string>, badNames: list<string>, changing: list<string>} examples for the report (first 50 each) */
public static function warnings(array $state): array {
return [
'unreadable' => array_slice(array_map('strval', array_keys($state['unreadable'] ?? [])), 0, 50),
'badNames' => array_slice(array_map('strval', array_keys($state['badNames'] ?? [])), 0, 50),
'changing' => array_slice(array_map('strval', array_values($state['changing'] ?? [])), 0, 50),
];
}
+17 -1
View File
@@ -17,6 +17,8 @@ namespace OCA\NgBackup\Job;
* and be merge-joined with the previous snapshot without holding any list in memory.
*/
final class TreeWalker {
private bool $atFolder = false;
/**
* @param array<string, string> $roots name => absolute directory, walked in this order
* @param list<string> $exclude path prefixes ("<root>/<relative>") that are skipped entirely;
@@ -27,9 +29,16 @@ final class TreeWalker {
private array $exclude = [],
/** Called with the logical path of a directory that cannot be listed. */
private ?\Closure $onUnreadable = null,
/** Also yield directories with something in them, right before the first thing in them. */
private bool $withFolders = false,
) {
}
/** Whether what was just yielded is a directory with something in it (see $withFolders). */
public function atFolderWithContent(): bool {
return $this->atFolder;
}
/** @return \Generator<string, string> logical path => absolute path, strictly after $after */
public function walk(?string $after = null): \Generator {
foreach ($this->roots as $name => $dir) {
@@ -75,7 +84,14 @@ final class TreeWalker {
if (is_link($childAbs)) {
continue; // never follow symlinks out of the data directory
}
if (is_dir($childAbs)) {
$childIsDir = is_dir($childAbs);
if (($childIsDir || is_file($childAbs)) && $empty && !$this->isExcluded($childLogical, $childIsDir)
&& $this->withFolders && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) {
$this->atFolder = true;
yield $logical => $abs;
$this->atFolder = false;
}
if ($childIsDir) {
$empty = $empty && $this->isExcluded($childLogical, true);
// Skip a directory entirely if everything in it sorts before the cursor.
if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) {
@@ -0,0 +1,33 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Migration;
use Closure;
use OCP\DB\ISchemaWrapper;
use OCP\DB\Types;
use OCP\Migration\IOutput;
use OCP\Migration\SimpleMigrationStep;
/** Local copy of what the catalog on the location says about a snapshot: its note and pin. */
final class Version000007Date20261107000000 extends SimpleMigrationStep {
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
/** @var ISchemaWrapper $schema */
$schema = $schemaClosure();
$t = $schema->getTable('ngb_snapshots');
$changed = false;
if (!$t->hasColumn('note')) {
$t->addColumn('note', Types::STRING, ['notnull' => false, 'length' => 1000]);
$changed = true;
}
if (!$t->hasColumn('pinned')) {
$t->addColumn('pinned', Types::BOOLEAN, ['notnull' => false, 'default' => false]);
$changed = true;
}
return $changed ? $schema : null;
}
}
@@ -0,0 +1,49 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Migration;
use Closure;
use OCA\NgBackup\Service\LocationSchedule;
use OCA\NgBackup\Service\Schedule;
use OCP\DB\ISchemaWrapper;
use OCP\DB\Types;
use OCP\IDBConnection;
use OCP\Migration\IOutput;
use OCP\Migration\SimpleMigrationStep;
/** A schedule per location. Each existing location gets the daily time that held for all of them. */
final class Version000008Date20261107010000 extends SimpleMigrationStep {
public function __construct(
private IDBConnection $db,
private Schedule $schedule,
) {
}
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
/** @var ISchemaWrapper $schema */
$schema = $schemaClosure();
$t = $schema->getTable('ngb_targets');
if ($t->hasColumn('schedule')) {
return null;
}
$t->addColumn('schedule', Types::STRING, ['notnull' => false, 'length' => 16]);
$t->addColumn('schedule_zone', Types::STRING, ['notnull' => false, 'length' => 64]);
$t->addColumn('schedule_since', Types::BIGINT, ['notnull' => false, 'unsigned' => true]);
return $schema;
}
public function postSchemaChange(IOutput $output, Closure $schemaClosure, array $options): void {
['time' => $time, 'timezone' => $zone] = $this->schedule->get();
$qb = $this->db->getQueryBuilder();
$qb->update('ngb_targets')
->set('schedule', $qb->createNamedParameter(Schedule::validTime($time) ? LocationSchedule::normalize($time) : ''))
->set('schedule_zone', $qb->createNamedParameter($zone))
->where($qb->expr()->isNull('schedule'));
$qb->executeStatement();
}
}
@@ -0,0 +1,31 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Migration;
use Closure;
use OCA\NgBackup\Service\KeyService;
use OCP\Migration\IOutput;
use OCP\Migration\SimpleMigrationStep;
/**
* A key from before 1.1.0 counts as used: those versions did not keep track, and it may protect
* backups on a location that was removed from this server later (user exports included). Only a
* key made with 1.1.0 or newer can be replaced as an unused one.
*/
final class Version000009Date20261108000000 extends SimpleMigrationStep {
public function __construct(
private KeyService $keys,
) {
}
public function postSchemaChange(IOutput $output, Closure $schemaClosure, array $options): void {
if ($this->keys->isInitialized()) {
$this->keys->markUsed();
}
}
}
+13 -1
View File
@@ -46,6 +46,10 @@ final class Notifier implements INotifier {
$notification->setParsedSubject($l->t('Backup to %1$s finished with warnings: %2$s unreadable, %3$s changed while being read', [$p['target'], $p['unreadable'], $p['changing']]));
$notification->setParsedMessage($l->t('Unreadable files and folders are missing from this backup (check their permissions). Files that kept changing were saved as they were while being read. Examples: %s', [$p['examples'] ?? '']));
break;
case 'run_bad_names':
$notification->setParsedSubject($l->n('Backup to %s finished, but %n item is missing because of its name', 'Backup to %s finished, but %n items are missing because of their names', (int)$p['count'], [$p['target']]));
$notification->setParsedMessage($l->t('Names with a backslash or a control character, or that are not valid UTF-8, are not supported, so these files and folders are not in this backup. Rename them; the next backup includes them. Examples: %s', [$p['examples'] ?? '']));
break;
case 'quota_warning':
$notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']]));
$notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.'));
@@ -57,13 +61,21 @@ final class Notifier implements INotifier {
: $l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']]));
$notification->setParsedMessage($l->t('Check the backup settings and the server log.'));
break;
case 'schedule_off':
$notification->setParsedSubject($l->t('Backup location %s has no schedule', [$p['target']]));
$notification->setParsedMessage(($p['others'] ?? '') === '1'
? $l->t('No backups run to this location until it gets a schedule or someone starts one. If that is what you want, you can ignore this reminder.')
: $l->t('No location has a schedule, so no backups run at all unless someone starts one. Set a schedule under Backups.'));
break;
case 'security_event':
$who = (string)($p['actor'] ?? '');
$notification->setParsedSubject(match ($p['event'] ?? '') {
'kit_downloaded' => $l->t('%s downloaded the backup recovery kit', [$who]),
'key_replaced' => $l->t('%1$s replaced the unused backup key %2$s by the key of existing backups, %3$s; download and confirm the new recovery kit', [$who, $p['old'] ?? '', $p['new'] ?? '']),
'target_removed' => $l->t('%1$s removed backup location %2$s', [$who, $p['target'] ?? '']),
'target_changed' => $l->t('%1$s changed the connection settings of backup location %2$s', [$who, $p['target'] ?? '']),
'append_only_off' => $l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']),
'schedule_off' => $l->t('%1$s turned off the schedule of backup location %2$s', [$who, $p['target'] ?? '']),
'append_only_off' =>$l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']),
'retention_changed' => $l->t('%1$s changed the backup retention to %2$s', [$who, $p['policy'] ?? '']),
'snapshots_forgotten' => $l->t('%1$s moved %2$s snapshots of %3$s to the trash; they are deleted permanently after %4$s unless restored', [$who, $p['count'] ?? '', $p['target'] ?? '', $p['until'] ?? '']),
'snapshot_untrashed' => $l->t('%1$s restored snapshot %2$s of %3$s from the trash', [$who, $p['snapshot'] ?? '', $p['target'] ?? '']),
+10 -1
View File
@@ -30,6 +30,10 @@ final class Catalog {
public array $trash = [];
/** @var array<string, true> manifest path => true, for a per-user export (users/<uid>/<exportId>) */
public array $userExports = [];
/** @var array<string, true> snapshots the retention policy leaves alone */
public array $pinned = [];
/** @var array<string, string> snapshot id => note, also while the snapshot is in the trash */
public array $notes = [];
public string $hash = '';
public function __construct(
@@ -82,6 +86,8 @@ final class Catalog {
// Catalogs written before user exports existed have no such key: default to empty rather
// than adopting the location's current users/ listing (that would skip verification).
$this->userExports = array_fill_keys($data['userExports'] ?? [], true);
$this->pinned = array_fill_keys($data['pinned'] ?? [], true);
$this->notes = $data['notes'] ?? [];
$this->anchor?->set($this->repositoryId, $this->gen, $this->hash);
}
@@ -93,9 +99,12 @@ final class Catalog {
if ($this->backend->exists($this->path($next))) {
throw new RepositoryException("Catalog generation $next was written by another process at the same time; try again");
}
// A pin ends when its snapshot leaves the list, a note when the snapshot is gone for good.
$this->pinned = array_intersect_key($this->pinned, $this->snapshots);
$this->notes = array_intersect_key($this->notes, $this->snapshots + $this->trash);
$plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash,
'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash),
'userExports' => array_keys($this->userExports)], JSON_THROW_ON_ERROR);
'userExports' => array_keys($this->userExports), 'pinned' => array_keys($this->pinned), 'notes' => $this->notes], JSON_THROW_ON_ERROR);
$encrypted = $this->cipher->encryptString($plain, $this->ad($next));
$stream = fopen('php://memory', 'w+b');
fwrite($stream, $encrypted);
+191 -8
View File
@@ -18,6 +18,7 @@ use OCA\NgBackup\Crypto\StreamCipher;
*
* Layout on the backend:
* config plaintext JSON: format version, repository id, wrapped master key
* settings encrypted JSON: settings fixed at installation (deletion delay)
* packs/xx/<packId> encrypted blobs + encrypted pack header
* index/<packId> encrypted list of the blobs in that pack
* snapshots/<snapId> encrypted JSON lines: one meta line, then one line per file
@@ -28,6 +29,7 @@ use OCA\NgBackup\Crypto\StreamCipher;
final class Repository {
public const FORMAT = 1;
public const BLOB_SIZE = 4 * 1048576;
private const SETTINGS = 'settings';
private const FLAG_DEFLATE = 1;
/** Blobs from this size on get a quick compression test before the full compression. */
private const DEFLATE_PROBE_FROM = 256 * 1024;
@@ -79,7 +81,7 @@ final class Repository {
// opens this data; refuse instead.
foreach (['index', 'snapshots', 'trash/snapshots', 'catalog'] as $dir) {
if ($backend->listFiles($dir) !== []) {
throw new RepositoryException("This location holds backup data ($dir/) but no repository config could be read: not creating a new repository over it. Check the connection and the folder; a missing config can be restored from the recovery kit.");
throw new RepositoryException("This location holds backup data ($dir/) but no repository config could be read: not creating a new repository over it. Check the connection and the folder. If the config is really gone, occ backup:target:add with --repair-config writes it again with this server's key (on a new server: import the recovery kit first, occ backup:key:import).");
}
}
$id = bin2hex(random_bytes(16));
@@ -95,12 +97,7 @@ final class Repository {
/** @param ?callable(int, int): void $progress index files downloaded so far, and in all (see BlobIndex::load()) */
public static function openWithKey(IBackend $backend, KeyRing $keys, ?IIndexCache $cache = null, ?ICatalogAnchor $anchor = null, ?callable $progress = null): self {
$config = self::readConfig($backend);
try {
$ok = isset($config['check']) && (new StreamCipher($keys))->decryptString((string)base64_decode($config['check'], true), 'config-check') === $config['id'];
} catch (\OCA\NgBackup\Crypto\CryptoException) {
$ok = false;
}
if (!$ok) {
if (!self::configMatches($config, $keys)) {
throw new RepositoryException('This repository was created with a different key');
}
$repo = new self($backend, $keys, $config['id']);
@@ -134,6 +131,49 @@ final class Repository {
return $this->catalog->gen;
}
/** @return list<string> snapshots the retention policy leaves alone */
public function pinnedSnapshots(): array {
return array_map('strval', array_keys($this->catalog->pinned));
}
/** @return array<string, string> snapshot id => note */
public function snapshotNotes(): array {
return $this->catalog->notes;
}
public function setPinned(string $snapshotId, bool $pinned): void {
$this->requireSnapshot($snapshotId);
if (isset($this->catalog->pinned[$snapshotId]) === $pinned) {
return;
}
if ($pinned) {
$this->catalog->pinned[$snapshotId] = true;
} else {
unset($this->catalog->pinned[$snapshotId]);
}
$this->catalog->write();
}
/** A note kept in the catalog: unlike the label inside the snapshot, it can change afterwards. */
public function setSnapshotNote(string $snapshotId, string $note): void {
$this->requireSnapshot($snapshotId);
if (($this->catalog->notes[$snapshotId] ?? '') === $note) {
return;
}
if ($note === '') {
unset($this->catalog->notes[$snapshotId]);
} else {
$this->catalog->notes[$snapshotId] = $note;
}
$this->catalog->write();
}
private function requireSnapshot(string $snapshotId): void {
if (!isset($this->catalog->snapshots[$snapshotId])) {
throw new \InvalidArgumentException('There is no snapshot ' . $snapshotId . ' on this location');
}
}
/** Index files downloaded when this repository was opened (0 when the local cache was complete). */
public function indexDownloads(): int {
return $this->index->downloaded;
@@ -153,6 +193,114 @@ final class Repository {
return $config;
}
/**
* Open a local file to read it in whole blobs. PHP reads a file in pieces of 8 KiB by default;
* on a disk where each request waits (a network or cloud volume) that made reading five times
* slower: 5 MiB/s instead of 27 MiB/s on nc01. Without PHP's read buffer every fread() of a
* blob is one request to the disk.
*
* @return resource|false
*/
public static function openForReading(string $path) {
$fh = @fopen($path, 'rb');
if ($fh !== false) {
stream_set_read_buffer($fh, 0);
stream_set_chunk_size($fh, self::BLOB_SIZE);
}
return $fh;
}
private static function configMatches(array $config, KeyRing $keys): bool {
try {
return isset($config['check'], $config['id'])
&& (new StreamCipher($keys))->decryptString((string)base64_decode((string)$config['check'], true), 'config-check') === $config['id'];
} catch (\OCA\NgBackup\Crypto\CryptoException) {
return false;
}
}
/**
* The passphrase-wrapped key slots in the config at this location: disaster recovery without
* the recovery kit, where any one of their passphrases gives back the master key.
*/
public static function wrappedKeyAt(IBackend $backend): array {
$config = self::readConfig($backend);
if (!isset($config['key']) || !is_array($config['key'])) {
throw new RepositoryException('The repository config at this location holds no key');
}
return $config['key'];
}
/** Whether the repository at this location was created with this key. Does not read the index. */
public static function madeWithKey(IBackend $backend, KeyRing $keys): bool {
return self::configMatches(self::readConfig($backend), $keys);
}
/**
* Settings of the installation that never change (the deletion delay), sealed with the
* master key so that, unlike the config and the recovery kit, nobody can alter them unseen.
* Empty when the location has none yet (made before 1.1.0, until its next backup); only
* ['unreadable' => true] when they are there but cannot be read.
*
* @return array<string, mixed>
*/
public static function settingsAt(IBackend $backend, KeyRing $keys): array {
try {
if (!$backend->exists(self::SETTINGS)) {
return [];
}
$fh = $backend->get(self::SETTINGS);
$raw = (string)stream_get_contents($fh);
fclose($fh);
$settings = json_decode((new StreamCipher($keys))->decryptString($raw, 'settings'), true, 16, JSON_THROW_ON_ERROR);
return is_array($settings) ? $settings : ['unreadable' => true];
} catch (BackendException|\OCA\NgBackup\Crypto\CryptoException|\JsonException) {
return ['unreadable' => true];
}
}
/**
* Write the settings once; a location that has them keeps them. The repository id goes with
* them: with it and the key from the recovery kit, a lost config can be written again.
*/
public function recordSettings(array $settings): void {
if ($this->backend->exists(self::SETTINGS)) {
return;
}
$settings['repository'] = $this->repositoryId;
self::putString($this->backend, self::SETTINGS, $this->cipher->encryptString(json_encode($settings, JSON_THROW_ON_ERROR), 'settings'));
}
/** Whether the location holds backup data, with or without a config. */
public static function holdsData(IBackend $backend): bool {
foreach (['index', 'snapshots', 'catalog'] as $dir) {
if ($backend->listFiles($dir) !== []) {
return true;
}
}
return false;
}
/**
* Write a lost config again, from the key (recovery kit) and the repository id in the
* sealed settings. Only where there is no config and the settings open with this key, so
* nothing that works is ever overwritten and a wrong key writes nothing.
*/
public static function repairConfig(IBackend $backend, KeyRing $keys, array $wrappedKey): void {
if ($backend->exists('config')) {
throw new RepositoryException('This location has a config: nothing to repair, leave out --repair-config');
}
$id = self::settingsAt($backend, $keys)['repository'] ?? null;
if (!is_string($id) || !preg_match('/^[0-9a-f]{32}$/', $id)) {
throw new RepositoryException($backend->exists(self::SETTINGS)
? 'The settings at this location do not open with this key: the recovery kit is of another server, or the settings are damaged. The config cannot be written again.'
: 'This location has no settings file (it was backed up only by NG Backup before 1.1.0), so its config cannot be written again.');
}
$config = ['format' => self::FORMAT, 'id' => $id, 'created' => gmdate('c'), 'key' => $wrappedKey,
'check' => base64_encode((new StreamCipher($keys))->encryptString($id, 'config-check')), 'repaired' => true];
self::putString($backend, 'config', json_encode($config, JSON_PRETTY_PRINT | JSON_THROW_ON_ERROR));
}
/** Replace the passphrase-wrapped key slots in this repository's config (after a slot change). */
public function updateWrappedKey(array $wrappedKey): void {
$config = self::readConfig($this->backend);
@@ -392,9 +540,41 @@ final class Repository {
self::restoreEmptyDir($target, $dest, $path, $mtime);
}
}
// Very last, nothing is written in them any more. Also on a run that goes on after a stop, and
// the deepest first. The time is not worth stopping for: all the files are there by now.
foreach (array_reverse($this->folderTimes($snapshotId)) as [$path, $mtime]) {
if (!self::underPrefix($path, $prefix) || ($stripPrefix && $path === rtrim($prefix, '/'))) {
continue; // the target itself is not ours to change
}
if ($heartbeat !== null) {
$heartbeat();
}
$dest = self::destination($target, $path, $prefix, $stripPrefix);
@touch(self::directoryInTarget($target, $dest, $path), $mtime);
}
return $count;
}
/**
* The folders with something in them that a snapshot recorded, parents before children (none
* before 1.1.0). Damage here is no reason to give up on the files of a restore.
*
* @return list<array{0: string, 1: int}> path, modification time
*/
public function folderTimes(string $snapshotId): array {
$meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId);
$packed = is_string($meta['folderTimes'] ?? null) ? base64_decode($meta['folderTimes'], true) : false;
$plain = $packed === false ? false : @gzinflate($packed);
$folders = [];
foreach ($plain === false ? [] : explode("\n", $plain) as $line) {
$folder = json_decode($line, true);
if (is_array($folder) && is_string($folder[0] ?? null) && self::isSafePath($folder[0]) && is_int($folder[1] ?? null)) {
$folders[] = [$folder[0], $folder[1]];
}
}
return $folders;
}
/**
* The folders without anything in them that a snapshot recorded (none before 1.0.3).
*
@@ -985,7 +1165,10 @@ final class Repository {
for ($attempt = 1; $attempt <= 3; $attempt++) {
$before = self::fileVersion($path);
$blobs = [];
$fh = fopen($path, 'rb');
$fh = self::openForReading($path);
if ($fh === false) {
throw new RepositoryException('Cannot read ' . $path);
}
while (($data = self::readFull($fh, self::BLOB_SIZE)) !== '') {
$blobs[] = $this->storeBlob($data, $packs, $stats);
}
+38 -1
View File
@@ -61,6 +61,16 @@ final class AlertService {
'examples' => mb_strimwidth(implode(', ', array_merge($examples['unreadable'], $examples['changing'])), 0, 300, '…')]);
}
/**
* A backup that finished but left out files or folders because of their name.
*
* @param list<string> $examples
*/
public function runBadNames(Run $run, string $targetName, int $count, array $examples): void {
$this->notifyAdmins('run_bad_names', 'run-' . $run->getId(), ['target' => $targetName, 'count' => (string)$count,
'examples' => mb_strimwidth(implode(', ', $examples), 0, 300, '…')]);
}
/**
* Remember how much is stored on a location (for the settings page) and warn administrators
* once a day when it is at 80% of its limit or more.
@@ -91,9 +101,36 @@ final class AlertService {
return $v === '' ? null : (int)$v;
}
/**
* Whether it is time to remind administrators of a location without a schedule: once a week
* while another location is still backed up, every day when none is. An hour of margin, so
* that the moment of the cron run does not push a reminder to the next day.
*/
public static function offReminderDue(int $last, int $now, bool $othersActive): bool {
return $now - $last >= ($othersActive ? 7 * 86400 : 86400) - 3600;
}
public function checkOff(Target $target, bool $othersActive): void {
if ((string)$target->getSchedule() !== '') {
return;
}
$key = 'off_notified_' . $target->getId();
$now = time();
$last = max((int)$target->getScheduleSince(), $this->appConfig->getValueInt(Application::APP_ID, $key, 0));
if (!self::offReminderDue($last, $now, $othersActive)) {
return;
}
$this->appConfig->setValueInt(Application::APP_ID, $key, $now);
$this->notifyAdmins('schedule_off', 'off-' . $target->getId() . '-' . date('Y-m-d', $now),
['target' => $target->getName(), 'others' => $othersActive ? '1' : '0']);
}
/** At most one notification per location per day. */
public function checkStale(Target $target): void {
$last = $this->runs->lastFinished($target->getId(), BackupService::KIND_FULL);
if ((string)$target->getSchedule() === '') {
return; // no schedule is a choice, as for a location that is only connected now and then
}
$last =$this->runs->lastFinished($target->getId(), BackupService::KIND_FULL);
$lastTime = $last?->getFinishedAt() ?? $target->getCreatedAt();
if (time() - $lastTime < self::STALE_AFTER || $this->runs->findRunning($target->getId(), BackupService::KIND_FULL) !== []) {
return;
+25 -1
View File
@@ -58,6 +58,18 @@ final class BackupService {
$this->openedWith = $opened;
}
/**
* The running backups in the order they get their turn: first come, first served. The first one
* runs, the others are queued; only one at a time reads the disks.
*
* @return list<Run>
*/
public function runsInTurn(): array {
$running = $this->runs->findRunning(null, self::KIND_FULL);
usort($running, static fn (Run $a, Run $b): int => [$a->getStartedAt(), $a->getId()] <=> [$b->getStartedAt(), $b->getId()]);
return $running;
}
public function start(Target $target, string $label = ''): Run {
$label = trim($label);
if (mb_strlen($label) > self::LABEL_MAX) {
@@ -210,6 +222,13 @@ final class BackupService {
$state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR);
if ($run->getPhase() === 'db') {
if (!isset($state['dump'])) {
// It may have been queued for hours: the restore point has the time of its data.
$state['files']['started'] = gmdate('c');
// Locations from before 1.1.0 get their sealed settings at their next backup.
$this->targets->recordSettings($repo);
$this->targets->catchUpWrappedKey($target, $repo);
}
$dumper = new DbDumper($this->db->getInner(), $this->config->getSystemValueString('dbtableprefix', 'oc_'));
$dbStats = $state['dbStats'] ?? [];
// With system cron the whole dump usually fits in one step (one transaction, fully
@@ -253,8 +272,13 @@ final class BackupService {
try {
$this->alerts->storedBytes($target, $repo->storedBytes());
$stats = $state['files']['stats'];
$warnings = BackupRun::warnings($state['files']);
if (($stats['badNames'] ?? 0) > 0) {
$this->logger->warning(sprintf('NG Backup run %d to %s: %d left out because their name is not supported (rename them): %s',
$run->getId(), $target->getName(), $stats['badNames'], implode(', ', $warnings['badNames'])));
$this->alerts->runBadNames($run, $target->getName(), (int)$stats['badNames'], $warnings['badNames']);
}
if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) {
$warnings = BackupRun::warnings($state['files']);
$this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s',
$run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']),
$stats['changing'] ?? 0, implode(', ', $warnings['changing'])));
+156
View File
@@ -0,0 +1,156 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Service;
use OCA\NgBackup\AppInfo\Application;
use OCP\DB\QueryBuilder\IQueryBuilder;
use OCP\IAppConfig;
use OCP\IDBConnection;
use Psr\Log\LoggerInterface;
/**
* Keeps heavy background jobs of other apps from running while a backup reads the same disks, by
* reserving them in Nextcloud's job list: no app is turned off, and a job that is running goes on.
* Nextcloud itself gives up on a reservation after 12 hours, so a pause that is never ended, because
* cron or this app is gone, ends by itself. It lasts at most MAX_SECONDS, then the jobs get REST_SECONDS.
*/
final class HeavyJobPause {
public const ENABLED_KEY = 'pause_heavy_jobs';
private const SINCE_KEY = 'heavy_jobs_paused_since';
private const REST_KEY = 'heavy_jobs_rest_until';
public const MAX_SECONDS = 6 * 3600;
public const REST_SECONDS = 3600;
/** Nextcloud runs a job again this long after it was reserved. */
private const RESERVATION_SECONDS = 12 * 3600;
/** Class, or the start of a class name ending in a backslash => what to call it */
private const JOBS = [
'OCA\\Memories\\' => 'Memories',
'OCA\\Files_Antivirus\\BackgroundJob\\BackgroundScanner' => 'Antivirus background scan',
'OCA\\Recognize\\' => 'Recognize',
'OCA\\PreviewGenerator\\' => 'Preview Generator',
'OCA\\FullTextSearch\\' => 'Full text search',
];
public const NOTHING = 'nothing';
public const PAUSE = 'pause';
public const RELEASE = 'release';
public const REST = 'rest';
public function __construct(
private IDBConnection $db,
private IAppConfig $appConfig,
private LoggerInterface $logger,
) {
}
/** @return self::NOTHING|self::PAUSE|self::RELEASE|self::REST */
public static function decide(bool $enabled, bool $backupRunning, int $since, int $restUntil, int $now): string {
if ($since > 0 && $now - $since >= self::MAX_SECONDS) {
return self::REST;
}
if ($enabled && $backupRunning && $now >= $restUntil) {
return self::PAUSE;
}
return $since > 0 ? self::RELEASE : self::NOTHING;
}
/** Call regularly: pauses or ends the pause as needed. A problem here never stops a backup. */
public function sync(bool $backupRunning): void {
try {
$this->apply($backupRunning, time());
} catch (\Throwable $e) {
$this->logger->warning('NG Backup: pausing heavy background jobs failed: ' . $e->getMessage(), ['exception' => $e]);
}
}
private function apply(bool $backupRunning, int $now): void {
$since = $this->appConfig->getValueInt(Application::APP_ID, self::SINCE_KEY, 0);
$restUntil = $this->appConfig->getValueInt(Application::APP_ID, self::REST_KEY, 0);
switch (self::decide($this->enabled(), $backupRunning, $since, $restUntil, $now)) {
case self::PAUSE:
if ($since === 0) {
$since = $now;
$this->appConfig->setValueInt(Application::APP_ID, self::SINCE_KEY, $since);
$this->logger->info('NG Backup: heavy background jobs paused while a backup runs');
}
$this->reserve($since, $now); // also the jobs that were added or running until now
break;
case self::REST:
$this->appConfig->setValueInt(Application::APP_ID, self::REST_KEY, $now + self::REST_SECONDS);
$this->release($since);
$this->logger->info('NG Backup: heavy background jobs get their turn for an hour during a long backup');
break;
case self::RELEASE:
$this->release($since);
$this->logger->info('NG Backup: heavy background jobs run again');
break;
}
}
private function reserve(int $stamp, int $now): void {
$qb = $this->db->getQueryBuilder();
$qb->update('jobs')
->set('reserved_at', $qb->createNamedParameter($stamp, IQueryBuilder::PARAM_INT))
->where($qb->expr()->lte('reserved_at', $qb->createNamedParameter($now - self::RESERVATION_SECONDS, IQueryBuilder::PARAM_INT)))
->andWhere($this->isHeavy($qb));
$qb->executeStatement();
}
/** Only our own reservations: a job that Nextcloud reserved at another moment is running. */
private function release(int $stamp): void {
$qb = $this->db->getQueryBuilder();
$qb->update('jobs')
->set('reserved_at', $qb->createNamedParameter(0, IQueryBuilder::PARAM_INT))
->where($qb->expr()->eq('reserved_at', $qb->createNamedParameter($stamp, IQueryBuilder::PARAM_INT)))
->andWhere($this->isHeavy($qb));
$qb->executeStatement();
$this->appConfig->deleteKey(Application::APP_ID, self::SINCE_KEY);
}
/** @return \OCP\DB\QueryBuilder\ICompositeExpression */
private function isHeavy(IQueryBuilder $qb) {
$conditions = [];
foreach (array_keys(self::JOBS) as $class) {
$conditions[] = str_ends_with($class, '\\')
? $qb->expr()->like('class', $qb->createNamedParameter($this->db->escapeLikeParameter($class) . '%'))
: $qb->expr()->eq('class', $qb->createNamedParameter($class));
}
return $qb->expr()->orX(...$conditions);
}
public function enabled(): bool {
return $this->appConfig->getValueString(Application::APP_ID, self::ENABLED_KEY, '1') !== '0';
}
public function setEnabled(bool $enabled): void {
$this->appConfig->setValueString(Application::APP_ID, self::ENABLED_KEY, $enabled ? '1' : '0');
}
/** @return array{enabled: bool, since: int, jobs: list<string>} what is paused now */
public function status(): array {
$since = $this->appConfig->getValueInt(Application::APP_ID, self::SINCE_KEY, 0);
$jobs = [];
if ($since > 0) {
$qb = $this->db->getQueryBuilder();
$qb->selectDistinct('class')->from('jobs')
->where($qb->expr()->eq('reserved_at', $qb->createNamedParameter($since, IQueryBuilder::PARAM_INT)))
->andWhere($this->isHeavy($qb));
$result = $qb->executeQuery();
while (($class = $result->fetchOne()) !== false) {
foreach (self::JOBS as $start => $name) {
if (str_starts_with((string)$class, $start)) {
$jobs[$name] = true;
}
}
}
$result->closeCursor();
}
return ['enabled' => $this->enabled(), 'since' => $since, 'jobs' => array_map('strval', array_keys($jobs))];
}
}
+85 -10
View File
@@ -30,6 +30,9 @@ final class KeyService {
private const KEY_DELETE_DELAY = 'delete_delay_days';
private const KEY_KIT_VERSION = 'recovery_kit_version';
private const KEY_KIT_CODE = 'recovery_kit_code';
/** Set once a location was created or adopted with this key: from then on it is never replaced. */
private const KEY_USED = 'master_key_used';
private const KEY_WRAPPED_PENDING = 'master_key_wrapped_pending';
public const DEFAULT_DELETE_DELAY = 7;
/**
* The location holds the passphrase-wrapped key too, so whoever can read it can try
@@ -83,28 +86,100 @@ final class KeyService {
* @param array<string, mixed> $kit as produced by recoveryKit()
* @throws \OCA\NgBackup\Crypto\CryptoException wrong passphrase, or a damaged/foreign kit
*/
public function importRecoveryKit(array $kit, #[\SensitiveParameter] string $passphrase): void {
if ($this->isInitialized()) {
public function importRecoveryKit(array $kit, #[\SensitiveParameter] string $passphrase, bool $replaceUnused = false): void {
if ($this->isInitialized() && !$replaceUnused) {
throw new \RuntimeException('A backup key already exists');
}
if (!isset($kit['wrapped_key'])) {
throw new \InvalidArgumentException('Not a NG Backup recovery kit file');
}
$keys = KeyRing::unwrapAny($kit['wrapped_key'], $passphrase);
$this->appConfig->setValueString(Application::APP_ID, self::KEY_MASTER,
$this->crypto->encrypt(base64_encode($keys->exportMasterKey())), true, true);
$this->cached = $keys;
$this->appConfig->setValueString(Application::APP_ID, self::KEY_WRAPPED, json_encode($kit['wrapped_key'], JSON_THROW_ON_ERROR), true, true);
$this->appConfig->setValueString(Application::APP_ID, self::KEY_FINGERPRINT, $this->fingerprintOf($keys), true);
$this->discardUnused($keys, $replaceUnused);
// Kits from before this field existed fall back to the default.
$delay = (int)($kit['delete_delay_days'] ?? self::DEFAULT_DELETE_DELAY);
$this->appConfig->setValueInt(Application::APP_ID, self::KEY_DELETE_DELAY, ($delay >= 1 && $delay <= 365) ? $delay : self::DEFAULT_DELETE_DELAY, true);
$this->appConfig->setValueInt(Application::APP_ID, self::KEY_KIT_VERSION, (int)($kit['kit_version'] ?? 1), true);
$this->store($keys, $kit['wrapped_key'], (int)($kit['delete_delay_days'] ?? self::DEFAULT_DELETE_DELAY), (int)($kit['kit_version'] ?? 1));
// Having and successfully using a real kit file is stronger proof than the normal
// download-and-confirm flow it replaces.
$this->confirmRecoveryKit('cli-import');
}
/**
* Disaster recovery without the recovery kit: adopt the master key that a passphrase opened
* from the key slots in a location's config (TargetService::addWithPassphrase). There is no
* kit, so none is confirmed: backups wait until a new kit is downloaded and confirmed.
*
* @param ?int $deleteDelayDays from the location's sealed settings, null when it has none
*/
public function adoptKey(KeyRing $keys, array $wrappedKey, ?int $deleteDelayDays, bool $replaceUnused = false): void {
if ($this->isInitialized() && !$replaceUnused) {
throw new \RuntimeException('A backup key already exists');
}
$this->discardUnused($keys, $replaceUnused);
$this->store($keys, $wrappedKey, $deleteDelayDays ?? self::DEFAULT_DELETE_DELAY, 1);
}
/** Whether a location was ever created or adopted with this key (TargetService::keyIsUnused() also looks at the database). */
public function wasUsed(): bool {
return $this->appConfig->getValueBool(Application::APP_ID, self::KEY_USED, false, true);
}
public function markUsed(): void {
if (!$this->wasUsed()) {
$this->appConfig->setValueBool(Application::APP_ID, self::KEY_USED, true, true);
}
}
/**
* Locations that missed the last passphrase change (offline at the time): they still hold
* the previous slots until the change reaches them, which is tried again at each backup.
*/
public function wrappedKeyPending(int $targetId): bool {
return $this->appConfig->getValueBool(Application::APP_ID, self::KEY_WRAPPED_PENDING . '_' . $targetId);
}
/** One flag per location, so that two processes never overwrite each other's. */
public function setWrappedKeyPending(int $targetId, bool $pending): void {
$key = self::KEY_WRAPPED_PENDING . '_' . $targetId;
if ($pending) {
$this->appConfig->setValueBool(Application::APP_ID, $key, true);
} elseif ($this->wrappedKeyPending($targetId)) {
$this->appConfig->deleteKey(Application::APP_ID, $key);
}
}
/**
* A key made by mistake on a server that is being restored: remove it, so the key of the
* backups can take its place. Only after the new key was opened with its passphrase, and
* only when the caller checked that the old key never protected anything
* (TargetService::keyIsUnused()). Its passphrases and recovery kit become worthless.
*/
private function discardUnused(KeyRing $replacement, bool $replaceUnused): void {
if (!$replaceUnused || !$this->isInitialized()) {
return;
}
if ($this->wasUsed()) {
throw new \RuntimeException('The backup key of this server has been used for a backup location: it is never replaced');
}
if ($this->fingerprintOf($replacement) === $this->fingerprint()) {
throw new \RuntimeException('This server already has this backup key: nothing to replace');
}
foreach ([self::KEY_MASTER, self::KEY_WRAPPED, self::KEY_CONFIRMED, self::KEY_FINGERPRINT, self::KEY_DELETE_DELAY,
self::KEY_KIT_VERSION, self::KEY_KIT_CODE, self::KEY_USED] as $key) {
$this->appConfig->deleteKey(Application::APP_ID, $key);
}
$this->cached = null;
}
private function store(KeyRing $keys, array $wrappedKey, int $deleteDelayDays, int $kitVersion): void {
$this->appConfig->setValueString(Application::APP_ID, self::KEY_MASTER,
$this->crypto->encrypt(base64_encode($keys->exportMasterKey())), true, true);
$this->cached = $keys;
$this->appConfig->setValueString(Application::APP_ID, self::KEY_WRAPPED, json_encode($wrappedKey, JSON_THROW_ON_ERROR), true, true);
$this->appConfig->setValueString(Application::APP_ID, self::KEY_FINGERPRINT, $this->fingerprintOf($keys), true);
$this->appConfig->setValueInt(Application::APP_ID, self::KEY_DELETE_DELAY,
($deleteDelayDays >= 1 && $deleteDelayDays <= 365) ? $deleteDelayDays : self::DEFAULT_DELETE_DELAY, true);
$this->appConfig->setValueInt(Application::APP_ID, self::KEY_KIT_VERSION, max(1, $kitVersion), true);
}
/**
* The master key and the recovery kit code are stored encrypted with Nextcloud's own
* instance secret (ICrypto), not with themselves; the wrapped key is not (it is sealed with
+3 -1
View File
@@ -43,8 +43,10 @@ final class KeySlotService {
foreach ($this->targets->list() as $t) {
try {
$this->targets->repository($t)->updateWrappedKey($wrapped);
$this->keys->setWrappedKeyPending($t->getId(), false);
} catch (\Throwable $e) {
$failed[] = $t->getName();
$this->keys->setWrappedKeyPending($t->getId(), true);
}
}
$label = '';
@@ -55,7 +57,7 @@ final class KeySlotService {
}
$this->alerts->securityEvent($event, ['slot' => (string)$slot, 'label' => $label]);
if ($failed !== []) {
throw new \RuntimeException('Saved, but these locations could not be updated (they still accept the previous passphrases): ' . implode(', ', $failed));
throw new \RuntimeException('Saved, but these locations could not be updated (they still accept the previous passphrases; the change is tried again at their next backup): ' . implode(', ', $failed));
}
}
}
+78
View File
@@ -0,0 +1,78 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Service;
/**
* When a location is backed up: "" (off), "HH:MM" (daily) or "HH:MM/N" (from HH:MM every N hours,
* starting over at HH:MM the next day), in the location's time zone.
*/
final class LocationSchedule {
/** Attempts per moment after a failure, an hour apart, instead of one every few minutes. */
public const MAX_ATTEMPTS = 3;
public const RETRY_AFTER = 3600;
/** @throws \InvalidArgumentException */
public static function normalize(string $schedule): string {
$schedule = trim($schedule);
if ($schedule === '' || strtolower($schedule) === 'off') {
return '';
}
if (preg_match('/^(([01]\d|2[0-3]):[0-5]\d)(?:\/(\d{1,2}))?$/', $schedule, $m) !== 1) {
throw new \InvalidArgumentException('Use "off", HH:MM for once a day, or HH:MM/N for every N hours from HH:MM');
}
$hours = (int)($m[3] ?? 24);
if ($hours < 1 || $hours > 24) {
throw new \InvalidArgumentException('Every 1 to 24 hours');
}
return $hours === 24 ? $m[1] : $m[1] . '/' . $hours;
}
/**
* The most recent scheduled moment at or before $now; null without a schedule.
*
* @param non-empty-string $zone
*/
public static function lastMoment(string $schedule, string $zone, int $now): ?int {
if ($schedule === '') {
return null;
}
$tz = new \DateTimeZone($zone);
$today = (new \DateTimeImmutable('@' . $now))->setTimezone($tz);
$moments = array_merge(self::moments($schedule, $today->modify('-1 day'), $tz), self::moments($schedule, $today, $tz));
$last = null;
foreach ($moments as $moment) {
if ($moment <= $now && ($last === null || $moment > $last)) {
$last = $moment;
}
}
return $last;
}
/** @return list<int> the moments of one day */
private static function moments(string $schedule, \DateTimeImmutable $day, \DateTimeZone $tz): array {
[$time, $every] = array_pad(explode('/', $schedule), 2, '24');
[$hour, $minute] = array_map('intval', explode(':', $time));
$moments = [];
for ($h = $hour; $h < 24; $h += (int)$every) {
$moments[] = (new \DateTimeImmutable($day->format('Y-m-d') . sprintf(' %02d:%02d', $h, $minute), $tz))->getTimestamp();
}
return $moments;
}
/**
* Whether a backup has to start for $moment. One that was made since then is enough, also when
* several moments went by during a long backup. A moment from before the schedule was set does
* not count: choosing a time must not start a backup right away.
*/
public static function isDue(int $moment, int $since, ?int $lastDoneStarted, int $attempts, ?int $lastAttemptFinished, int $now): bool {
if ($moment < $since || ($lastDoneStarted !== null && $lastDoneStarted >= $moment) || $attempts >= self::MAX_ATTEMPTS) {
return false;
}
return $attempts === 0 || $now - (int)$lastAttemptFinished >= self::RETRY_AFTER;
}
}
+51 -1
View File
@@ -11,6 +11,7 @@ use OCA\NgBackup\AppInfo\Application;
use OCA\NgBackup\Db\RunMapper;
use OCA\NgBackup\Db\SnapshotMapper;
use OCA\NgBackup\Db\Target;
use OCA\NgBackup\Repository\Repository;
use OCP\IAppConfig;
use OCP\Lock\ILockingProvider;
use OCP\Lock\LockedException;
@@ -33,6 +34,54 @@ final class PruneService {
) {
}
/** Characters of a note on a snapshot (the column holds 1000) */
public const NOTE_MAX = 500;
/** Pin or unpin a snapshot: the retention policy leaves a pinned one alone. */
public function setPinned(Target $target, string $snapshotId, bool $pinned): void {
$changed = $this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $pinned): bool {
$was = in_array($snapshotId, $repo->pinnedSnapshots(), true);
$repo->setPinned($snapshotId, $pinned);
return $was !== $pinned;
});
$this->targets->syncSnapshot($target, $snapshotId, pinned: $pinned);
if ($changed) {
// Unpinning opens the way to removing it, so every administrator hears about that.
$this->alerts->securityEvent($pinned ? 'snapshot_pinned' : 'snapshot_unpinned', ['target' => $target->getName(), 'snapshot' => $snapshotId], !$pinned);
}
}
public function setNote(Target $target, string $snapshotId, string $note): void {
$note = trim((string)preg_replace('/[\x00-\x1f\x7f]+/', ' ', $note));
if (!mb_check_encoding($note, 'UTF-8')) {
throw new \InvalidArgumentException('A note must be valid UTF-8');
}
if (mb_strlen($note) > self::NOTE_MAX) {
throw new \InvalidArgumentException('A note has at most ' . self::NOTE_MAX . ' characters');
}
$this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $note): bool {
$repo->setSnapshotNote($snapshotId, $note);
return true;
});
$this->targets->syncSnapshot($target, $snapshotId, note: $note);
}
/**
* Write to the catalog next to a running backup or restore, which only takes it for a moment to
* commit. Not during a cleanup, which holds a repository opened earlier and writes from that.
*
* @param callable(Repository): bool $change
*/
private function changeCatalog(Target $target, callable $change): bool {
try {
return $this->leases->with(self::lockKey($target), LeaseService::SHARED, 300,
fn (): bool => $this->leases->with(self::catalogLockKey($target), LeaseService::EXCLUSIVE, 300,
fn (): bool => $change($this->targets->repository($target))));
} catch (LockedException) {
throw new \RuntimeException('A cleanup of ' . $target->getName() . ' or another change to its restore points is in progress; try again later');
}
}
public function untrash(Target $target, string $snapshotId): void {
$lease = $this->lockExclusive($target, 300);
try {
@@ -87,6 +136,7 @@ final class PruneService {
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
$s->setFiles((int)($meta['stats']['files'] ?? 0));
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
$s->setNote($repo->snapshotNotes()[$snapshotId] ?? null);
try {
$this->snapshots->insert($s);
} catch (\Throwable) {
@@ -208,7 +258,7 @@ final class PruneService {
$meta = $repo->snapshotMeta($id);
$times[$id] = (int)strtotime((string)($meta['time'] ?? '@0'));
}
$keep = $this->policy()->keep($times);
$keep = $this->policy()->keepWithPinned($times, $repo->pinnedSnapshots());
$forget = array_values(array_diff(array_keys($times), array_keys($keep)));
$actor = $scheduled ? 'retention policy' : $this->alerts->actor();
$purged = [];
+18
View File
@@ -155,6 +155,17 @@ final class RestoreService {
$dest = $this->ensureFolder($userFolder, $parentRel);
$names = $single ? [basename($rel) => $entries['']] : $entries;
// The folders with something in them that this restore creates get their time back; one that
// was already there keeps its own, as its content changes now. Noted before anything is written.
/** @var array<array-key, int> $newFolders a folder called "2026" is an integer key */
$newFolders = [];
foreach ($single ? [] : $repo->folderTimes($snapshotId) as [$dir, $mtime]) {
$relative = $dir === $source ? '' : (str_starts_with($dir, $source . '/') ? substr($dir, strlen($source) + 1) : null);
if ($relative !== null && ($relative === '' ? !$destExisted : !$dest->nodeExists($relative))) {
$newFolders[$relative] = $mtime;
}
}
$this->cleaned = [];
$restored = 0;
foreach ($names as $relPath => $entry) {
@@ -190,6 +201,13 @@ final class RestoreService {
if ($mode === self::MODE_REPLACE && !$single) {
$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)) + $emptyDirs);
}
// Last and deepest first: a change in a folder also touches the folders above it.
foreach (array_reverse($newFolders, true) as $dir => $mtime) {
$this->ensureFolder($dest, (string)$dir)->touch($mtime);
if ($progress !== null) {
$progress($restored, count($names), (string)$dir); // keeps the lease alive too
}
}
return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())];
}
+13
View File
@@ -28,6 +28,19 @@ final class RetentionPolicy {
return ['last' => $this->last, 'daily' => $this->daily, 'weekly' => $this->weekly, 'monthly' => $this->monthly];
}
/**
* keep() for the snapshots that are not pinned, plus the pinned ones. A pinned snapshot does not
* take the place of a daily, weekly or monthly one.
*
* @param array<string, int> $snapshots id => unix time
* @param list<string> $pinned
* @return array<string, list<string>>
*/
public function keepWithPinned(array $snapshots, array $pinned): array {
$pins = array_intersect_key($snapshots, array_flip($pinned));
return $this->keep(array_diff_key($snapshots, $pins)) + array_fill_keys(array_keys($pins), ['pinned']);
}
/**
* @param array<string, int> $snapshots id => unix time
* @return array<string, list<string>> id => reasons it is kept (ids not in the result are removed)
+8 -25
View File
@@ -13,10 +13,11 @@ use OCP\IConfig;
use OCP\IGroupManager;
/**
* The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a
* zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone
* is converted once to the zone of the server or of an administrator, at today's offset: it then
* shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
* The time zone times are shown in, and the daily time from before each location had its own
* schedule (see LocationSchedule). Nextcloud runs PHP in UTC, so a time without a zone meant UTC
* (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone is converted once
* to the zone of the server or of an administrator, at today's offset: it then shows as it
* effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
*/
final class Schedule {
public const TIME_KEY = 'schedule_time';
@@ -44,27 +45,9 @@ final class Schedule {
return ['time' => $time, 'timezone' => $zone];
}
/** @throws \InvalidArgumentException for a malformed time or an unknown zone */
public function set(string $time, string $zone): void {
if ($time !== '' && !self::validTime($time)) {
throw new \InvalidArgumentException('Use HH:MM or empty');
}
if (!self::validZone($zone)) {
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
}
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
}
/** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */
public function todayAt(?int $now = null): ?int {
['time' => $time, 'timezone' => $zone] = $this->get();
if ($time === '' || !self::validTime($time)) {
return null;
}
$tz = new \DateTimeZone($zone);
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d');
return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp();
/** @return non-empty-string the zone of a location's schedule, or else this one */
public function zoneOr(?string $zone): string {
return $zone !== null && self::validZone($zone) ? $zone : $this->get()['timezone'];
}
/**
+219 -11
View File
@@ -27,6 +27,9 @@ use OCP\Server;
* classes but its enabled/disabled state is left as the admin set it.
*/
final class TargetService {
/** Deletion delay taken when a location's own cannot be read: nothing leaves the Bin sooner than on the old server. */
public const LONGEST_DELETE_DELAY = 365;
public function __construct(
private TargetMapper $mapper,
private KeyService $keys,
@@ -35,6 +38,8 @@ final class TargetService {
private \OCA\NgBackup\Db\DbIndexCache $indexCache,
private \OCA\NgBackup\Db\AppConfigCatalogAnchor $anchor,
private SnapshotMapper $snapshots,
private AlertService $alerts,
private \OCA\NgBackup\Db\RunMapper $runs,
) {
}
@@ -51,18 +56,31 @@ final class TargetService {
* @param ?callable(int, int): void $progress reading an existing repository's index: files done, and in all
* @return array{target: Target, created: bool}
*/
public function add(string $name, string $backend, string $auth, array $options, string $basePath, ?callable $progress = null): array {
public function add(string $name, string $backend, string $auth, array $options, string $basePath, ?callable $progress = null, bool $repairConfig = false): array {
$this->validateName($name);
$this->requireUsableBackend($backend);
$keys = $this->keys->keyRing();
$storageBackend = $this->buildBackend($backend, $auth, $options, $basePath);
if ($storageBackend->exists('config')) {
$repo = Repository::openWithKey($storageBackend, $keys, $this->indexCache, $this->anchor, $progress);
$created = false;
} else {
$repo = Repository::initWithKey($storageBackend, $keys, $this->keys->wrappedKey(), $this->anchor);
$created = true;
$storageBackend = $this->buildBackend($backend, $auth, $options, $basePath, !$repairConfig);
try {
if ($repairConfig) {
Repository::repairConfig($storageBackend, $keys, $this->keys->wrappedKey());
}
if ($storageBackend->exists('config')) {
$repo = Repository::openWithKey($storageBackend, $keys, $this->indexCache, $this->anchor, $progress);
$created = false;
} else {
$repo = Repository::initWithKey($storageBackend, $keys, $this->keys->wrappedKey(), $this->anchor);
$created = true;
}
} catch (\OCA\NgBackup\Repository\RepositoryException $e) {
$otherKey = str_contains($e->getMessage(), 'different key') || str_contains($e->getMessage(), 'do not open with this key');
if ($otherKey && $this->keyIsUnused()) {
throw new \OCA\NgBackup\Repository\RepositoryException(rtrim($e->getMessage(), '.') . '. ' . self::UNUSED_KEY_HINT, 0, $e);
}
throw $e;
}
$this->keys->markUsed();
$this->recordSettings($repo);
foreach ($this->mapper->findAll() as $existing) {
if ($existing->getRepositoryId() === $repo->id()) {
@@ -78,6 +96,11 @@ final class TargetService {
$target->setBasePath(trim($basePath, '/'));
$target->setRepositoryId($repo->id());
$target->setCreatedAt(time());
// Like the location before it: a backup scheme is usually one scheme.
$first = $this->mapper->findAll()[0] ?? null;
$target->setSchedule($first?->getSchedule() ?? '');
$target->setScheduleZone($first?->getScheduleZone());
$target->setScheduleSince(time());
$target = $this->mapper->insert($target);
if (!$created) {
// Disaster recovery: occ backup:list reads this local cache, not the repository
@@ -87,6 +110,128 @@ final class TargetService {
return ['target' => $target, 'created' => $created];
}
public const UNUSED_KEY_HINT = 'The backup key of this server was never used (no location, no backup), so it can make way for the key of these backups: '
. 'with the old server\'s recovery kit, occ backup:key:import <kit file> --replace-unused-key; without the kit, '
. 'occ backup:target:add with the same details and --replace-unused-key (asks for a passphrase of the old server)';
/**
* A key that never protected anything: no location was created or adopted with it, and the
* database has no location and no run (installations from before the flag existed). Only such
* a key may make way for the key of existing backups, e.g. after a new key was made by
* mistake on a server that is being restored.
*/
public function keyIsUnused(): bool {
return $this->keys->isInitialized() && !$this->keys->wasUsed()
&& $this->mapper->findAll() === [] && $this->runs->countAll() === 0;
}
/**
* occ backup:key:import on a server whose own key was never used: the kit's key takes its place.
*
* @param array<string, mixed> $kit
* @throws \OCA\NgBackup\Crypto\CryptoException wrong passphrase: nothing changed
*/
public function importKitReplacingUnusedKey(array $kit, #[\SensitiveParameter] string $passphrase): void {
$old = $this->requireUnusedKey();
$this->keys->importRecoveryKit($kit, $passphrase, true);
$this->alerts->securityEvent('key_replaced', ['old' => $old, 'new' => $this->keys->fingerprint()]);
}
/** @return string the fingerprint of the unused key */
private function requireUnusedKey(): string {
if (!$this->keys->isInitialized()) {
throw new \RuntimeException('There is no backup key here to replace');
}
if (!$this->keyIsUnused()) {
throw new \RuntimeException('The backup key of this server (fingerprint ' . $this->keys->fingerprint() . ') has been used for a backup location: it is never replaced');
}
return $this->keys->fingerprint();
}
public const LOCATION_REPOSITORY = 'repository';
public const LOCATION_DATA_WITHOUT_CONFIG = 'data-without-config';
public const LOCATION_EMPTY = 'empty';
/**
* What these settings lead to: a repository, backup data whose config is gone, or nothing
* of NG Backup. Creates nothing.
*
* @return self::LOCATION_*
*/
public function locationState(string $backend, string $auth, array $options, string $basePath): string {
$this->requireUsableBackend($backend);
$storage = $this->buildBackend($backend, $auth, $options, $basePath, false);
if ($storage->exists('config')) {
return self::LOCATION_REPOSITORY;
}
return Repository::holdsData($storage) ? self::LOCATION_DATA_WITHOUT_CONFIG : self::LOCATION_EMPTY;
}
/**
* Disaster recovery without the recovery kit, on an installation without a backup key: take
* the key from the location's config with one of its passphrases, then add the location as
* usual. The key is only kept when it really is the one the repository was made with.
* With $replaceUnusedKey, a key this server has but never used makes way (keyIsUnused()).
*
* @param array<string, mixed> $options
* @param ?callable(int, int): void $progress
* @return array{target: Target, created: bool, deleteDelayDays: ?int, settingsUnreadable: bool} the deletion delay the location had stored, if any, and whether its settings could not be read
* @throws \OCA\NgBackup\Crypto\CryptoException wrong passphrase
*/
public function addWithPassphrase(string $name, string $backend, string $auth, array $options, string $basePath,
#[\SensitiveParameter] string $passphrase, ?callable $progress = null, bool $replaceUnusedKey = false): array {
$old = null;
if ($replaceUnusedKey) {
$old = $this->requireUnusedKey();
} elseif ($this->keys->isInitialized()) {
throw new \RuntimeException('A backup key already exists here');
}
$this->validateName($name);
$this->requireUsableBackend($backend);
$storageBackend = $this->buildBackend($backend, $auth, $options, $basePath, false);
if (!$storageBackend->exists('config')) {
throw new \InvalidArgumentException("No NG Backup repository found in folder '" . trim($basePath, '/') . "'");
}
$wrapped = Repository::wrappedKeyAt($storageBackend);
$keys = \OCA\NgBackup\Crypto\KeyRing::unwrapAny($wrapped, $passphrase);
if (!Repository::madeWithKey($storageBackend, $keys)) {
throw new \RuntimeException('The key in the config at this location does not open its backups: the config is damaged or was replaced. Only the recovery kit can help now: occ backup:key:import <kit file>, then this command again.');
}
$settings = Repository::settingsAt($storageBackend, $keys);
$stored = $settings['delete_delay_days'] ?? null;
$delay = is_int($stored) && $stored >= 1 && $stored <= 365 ? $stored : null;
// Settings that are there but cannot be read, or hold no valid delay (also none at all): the old server's delay
// is unknown, so take the longest one; a shorter guess could empty the Bin too soon.
$unknown = ($settings['unreadable'] ?? false) === true || ($settings !== [] && $delay === null);
$this->keys->adoptKey($keys, $wrapped, $delay ?? ($unknown ? self::LONGEST_DELETE_DELAY : null), $replaceUnusedKey);
if ($old !== null) {
$this->alerts->securityEvent('key_replaced', ['old' => $old, 'new' => $this->keys->fingerprint()]);
}
return $this->add($name, $backend, $auth, $options, $basePath, $progress) + ['deleteDelayDays' => $delay, 'settingsUnreadable' => $unknown];
}
/** A passphrase change that missed this location (offline at the time) reaches it now. A failure is tried again at the next backup. */
public function catchUpWrappedKey(Target $target, Repository $repo): void {
if (!$this->keys->wrappedKeyPending($target->getId())) {
return;
}
try {
$repo->updateWrappedKey($this->keys->wrappedKey());
$this->keys->setWrappedKeyPending($target->getId(), false);
} catch (\Throwable $e) {
Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: the last passphrase change still does not reach ' . $target->getName()
. ' (it keeps the previous passphrases); tried again at its next backup: ' . $e->getMessage(), ['exception' => $e]);
}
}
/** Seal the settings fixed at installation on the location, once. A failure is tried again at the next backup. */
public function recordSettings(Repository $repo): void {
try {
$repo->recordSettings(['delete_delay_days' => $this->keys->deleteDelayDays()]);
} catch (\Throwable) {
}
}
/**
* Change how an existing location is reached (credentials, host, folder, even the connection
* type) or its name. The new settings must lead to the same repository: the history stays
@@ -208,7 +353,25 @@ final class TargetService {
return $result;
}
/** Update the local copy of a snapshot's note or pin, if the snapshot is listed here at all. */
public function syncSnapshot(Target $target, string $snapshotId, ?string $note = null, ?bool $pinned = null): void {
try {
$s = $this->snapshots->findOne($target->getId(), $snapshotId);
} catch (DoesNotExistException) {
return;
}
if ($note !== null) {
$s->setNote($note !== '' ? $note : null);
}
if ($pinned !== null) {
$s->setPinned($pinned);
}
$this->snapshots->update($s);
}
private function adoptSnapshots(Target $target, Repository $repo): void {
$notes = $repo->snapshotNotes();
$pinned = array_flip($repo->pinnedSnapshots());
foreach ($repo->listSnapshots() as $snapshotId) {
try {
$this->snapshots->findOne($target->getId(), $snapshotId);
@@ -227,6 +390,8 @@ final class TargetService {
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
$s->setFiles((int)($meta['stats']['files'] ?? 0));
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
$s->setNote($notes[$snapshotId] ?? null);
$s->setPinned(isset($pinned[$snapshotId]));
$this->snapshots->insert($s);
} catch (\Throwable $e) {
Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]);
@@ -248,6 +413,30 @@ final class TargetService {
}
/** Size limit for the data stored on this location (null = none). */
/**
* @param string $schedule see LocationSchedule
* @throws \InvalidArgumentException
*/
public function setSchedule(Target $target, string $schedule, string $zone): Target {
if (!Schedule::validZone($zone)) {
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
}
$new = LocationSchedule::normalize($schedule);
$turnedOff = $new === '' && (string)$target->getSchedule() !== '';
if ($new === (string)$target->getSchedule() && $zone === $target->getScheduleZone()) {
// Saved again unchanged: a moment that already passed still gets its backup
return $target;
}
$target->setSchedule($new);
$target->setScheduleZone($zone);
$target->setScheduleSince(time());
$target = $this->mapper->update($target);
if ($turnedOff) {
$this->alerts->securityEvent('schedule_off', ['target' => $target->getName()]);
}
return $target;
}
public function setMaxBytes(Target $target, ?int $maxBytes): Target {
$target->setMaxBytes($maxBytes);
return $this->mapper->update($target);
@@ -261,10 +450,29 @@ final class TargetService {
/** Remove the target from NG Backup. The backups on the target itself are not touched. */
public function remove(Target $target): void {
$this->mapper->delete($target);
// The location is gone now: tidying up must not fail the removal, or the caller would skip
// the removal event and a retry would no longer find the location
$this->tidyUp($target, fn () => $this->keys->setWrappedKeyPending($target->getId(), false));
// The cached index of its repository is only of use while a location points at it
$repositoryId = $target->getRepositoryId();
if ($repositoryId !== null && !array_filter($this->list(), fn (Target $t) => $t->getRepositoryId() === $repositoryId)) {
$this->indexCache->forget($repositoryId);
$this->tidyUp($target, function () use ($target): void {
$repositoryId = $target->getRepositoryId();
if ($repositoryId !== null && !array_filter($this->list(), fn (Target $t) => $t->getRepositoryId() === $repositoryId)) {
$this->indexCache->forget($repositoryId);
}
});
}
/** One tidy-up step after a removal: a failure is logged (if even that works) and never thrown. */
private function tidyUp(Target $target, callable $step): void {
try {
$step();
} catch (\Throwable $e) {
try {
Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: location ' . $target->getName()
. ' removed, tidying up after it failed: ' . $e->getMessage(), ['exception' => $e]);
} catch (\Throwable) {
// nothing left to tell it to
}
}
}
+16 -1
View File
@@ -300,6 +300,22 @@
<code><![CDATA[$t]]></code>
</UndefinedDocblockClass>
</file>
<file src="lib/Migration/Version000008Date20261107010000.php">
<UndefinedDocblockClass>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
</UndefinedDocblockClass>
</file>
<file src="lib/Migration/Version000007Date20261107000000.php">
<UndefinedDocblockClass>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
<code><![CDATA[$t]]></code>
</UndefinedDocblockClass>
</file>
<file src="lib/Migration/Version000005Date20261002040000.php">
<UndefinedDocblockClass>
<code><![CDATA[$t]]></code>
@@ -351,7 +367,6 @@
<code><![CDATA[$encrypted]]></code>
<code><![CDATA[$encrypted]]></code>
<code><![CDATA[$encrypted]]></code>
<code><![CDATA[$fh]]></code>
<code><![CDATA[$plain]]></code>
<code><![CDATA[$plain]]></code>
<code><![CDATA[$plain]]></code>
Binary file not shown.

Before

Width:  |  Height:  |  Size: 259 KiB

After

Width:  |  Height:  |  Size: 244 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 268 KiB

After

Width:  |  Height:  |  Size: 228 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 252 KiB

After

Width:  |  Height:  |  Size: 222 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 189 KiB

After

Width:  |  Height:  |  Size: 172 KiB

+4 -1
View File
@@ -34,8 +34,11 @@ export default {
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
tick: () => call('post', '/tick'),
setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }),
setPauseHeavyJobs: (enabled) => call('put', '/settings/pause-heavy-jobs', { enabled }),
setTargetSchedule: (id, schedule, timezone) => call('put', `/targets/${id}/schedule`, { schedule, timezone }),
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
setSnapshotNote: (targetId, snapshotId, note) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/note`, { note }),
setSnapshotPinned: (targetId, snapshotId, pinned) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/pin`, { pinned }),
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
}
+31 -29
View File
@@ -5,10 +5,11 @@
<template>
<NcSettingsSection :name="t('ng_backup', 'Backups')"
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
<div class="ngb-row">
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, {zone}, empty = off)', { zone: browserZone })" />
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
</div>
<LocationScheduleRow v-for="target in status.targets" :key="target.id" :target="target" @changed="emit('changed')" @error="error = $event" />
<NcCheckboxRadioSwitch :model-value="status.heavyJobs.enabled" @update:model-value="savePause">
{{ t('ng_backup', 'Pause heavy background jobs (Memories, Recognize, Preview Generator, full text search, antivirus scan) while a backup runs, at most 6 hours in a row') }}
</NcCheckboxRadioSwitch>
<p v-if="status.heavyJobs.jobs.length" class="ngb-muted">{{ t('ng_backup', 'Paused now: {jobs}', { jobs: status.heavyJobs.jobs.join(', ') }) }}</p>
<div class="ngb-row">
<NcTextField v-model="label" class="ngb-label" :maxlength="100" :label="t('ng_backup', 'Label (optional)')"
:placeholder="t('ng_backup', 'e.g. before the upgrade to Nextcloud 35')" />
@@ -37,11 +38,11 @@
<td>{{ formatDateTime(run.startedAt) }}</td>
<td>{{ targetName(run.targetId) }}</td>
<td>{{ run.kind }}</td>
<td>{{ run.snapshot ? formatDateTime(run.snapshot.createdAt) + (run.snapshot.label ? ' – ' + run.snapshot.label : '') : '' }}</td>
<td class="ngb-point">{{ run.snapshot ? (run.snapshot.pinned ? '📌 ' : '') + formatDateTime(run.snapshot.createdAt) + (run.snapshot.label ? ' – ' + run.snapshot.label : '') + (run.snapshot.note ? ' – ' + run.snapshot.note : '') : '' }}</td>
<td :class="'ngb-' + run.status">{{ runStatus(run) }}{{ run.error ? ': ' + run.error : '' }}</td>
<td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
<td>{{ run.stats?.files ? formatSize(run.stats.files.bytes) : '' }}</td>
<td>{{ run.stats?.files ? formatSize(run.stats.files.uploaded) : '' }}</td>
<td class="ngb-num">{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
<td class="ngb-num">{{ run.stats?.files ? formatSize(run.stats.files.bytes) : '' }}</td>
<td class="ngb-num">{{ run.stats?.files ? formatSize(run.stats.files.uploaded) : '' }}</td>
</tr>
</tbody>
</table>
@@ -60,24 +61,20 @@
</template>
<script setup>
import { ref, watch } from 'vue'
import { ref } from 'vue'
import { n, t } from '@nextcloud/l10n'
import NcButton from '@nextcloud/vue/components/NcButton'
import NcCheckboxRadioSwitch from '@nextcloud/vue/components/NcCheckboxRadioSwitch'
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
import NcTextField from '@nextcloud/vue/components/NcTextField'
import LocationScheduleRow from './LocationScheduleRow.vue'
import api from '../api.js'
import { formatDateTime, formatSize, toZone } from '../format.js'
import { formatDateTime, formatSize } from '../format.js'
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
const emit = defineEmits(['changed', 'show-all-runs'])
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
// zone is shown in this one, so it reads as the moment it runs here.
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s
const schedule = ref(shown(props.status.schedule))
const error = ref('')
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
const label = ref('')
@@ -97,6 +94,9 @@ const runStatus = (run) => {
if (run.status !== 'running') {
return run.status
}
if (run.queued) {
return t('ng_backup', 'queued, after the backup that is running now')
}
const detail = run.phase + percent(run)
if (run.waitingFor) {
return t('ng_backup', 'waiting for background jobs ({detail}, last step {minutes} min ago)', { detail, minutes: Math.floor(run.waitingFor / 60) })
@@ -106,6 +106,16 @@ const runStatus = (run) => {
const isRunning = (targetId) => props.status.runs.some(r => r.targetId === targetId && r.kind === 'full' && r.status === 'running')
const targetName = (id) => props.status.targets.find(tg => tg.id === id)?.name ?? '#' + id
async function savePause(enabled) {
error.value = ''
try {
await api.setPauseHeavyJobs(enabled)
emit('changed')
} catch (e) {
error.value = e.message
}
}
async function start(target) {
error.value = ''
try {
@@ -116,25 +126,17 @@ async function start(target) {
error.value = e.message
}
}
async function saveSchedule() {
error.value = ''
try {
await api.setSchedule(schedule.value.trim(), browserZone)
emit('changed')
} catch (e) {
error.value = e.message
}
}
</script>
<style scoped>
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-time { max-width: 260px; }
.ngb-label { max-width: 360px; }
.ngb-table-wrap { overflow-x: auto; }
.ngb-table-wrap { max-width: 90%; overflow-x: auto; } /* scrolls only on a really narrow screen */
.ngb-table { width: 100%; border-collapse: collapse; }
.ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); }
.ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); white-space: normal; overflow-wrap: break-word; }
/* Text wraps; numbers stay on one line, the restore point keeps room for its label and note */
.ngb-table td.ngb-num { white-space: nowrap; }
.ngb-table td.ngb-point { min-width: 9em; }
.ngb-failed { color: var(--color-error-text); }
.ngb-muted { color: var(--color-text-maxcontrast); }
.ngb-done { color: var(--color-success-text); }
+4 -1
View File
@@ -4,8 +4,11 @@
-->
<template>
<NcSettingsSection :name="t('ng_backup', 'Backup key and recovery kit')"
:description="t('ng_backup', 'All backups are encrypted with one key. The server keeps it for scheduled backups. To restore on a new server you need the recovery kit and one of the passphrases.')">
:description="t('ng_backup', 'All backups are encrypted with one key. The server keeps it for scheduled backups. To restore on a new server you need one of the passphrases; the recovery kit also helps when the key on the backup location is damaged.')">
<!-- 1. Create the key -->
<NcNoteCard v-if="!status.key.initialized" type="warning">
{{ t('ng_backup', 'Restoring a server after a crash? Do not create a key here: a new key cannot read your backups. Follow the disaster recovery guide instead (occ backup:target:add asks for one of your passphrases, or import the recovery kit with occ backup:key:import).') }}
</NcNoteCard>
<form v-if="!status.key.initialized" class="ngb-form" @submit.prevent="createKey">
<NcTextField v-model="label" :label="t('ng_backup', 'Who keeps this passphrase (e.g. Safe, CTO)')" />
<NcPasswordField v-model="pass1" :label="t('ng_backup', 'Passphrase (at least 16 characters)')" autocomplete="new-password" />
+78
View File
@@ -0,0 +1,78 @@
<!--
- SPDX-FileCopyrightText: 2026 Dillard Blom
- SPDX-License-Identifier: AGPL-3.0-or-later
-->
<template>
<div class="ngb-row">
<strong class="ngb-name">{{ target.name }}</strong>
<NcSelect v-model="kind" :options="kinds" label="label" :input-label="t('ng_backup', 'Schedule')" :clearable="false" />
<NcTextField v-if="kind.id !== 'off'" v-model="time" class="ngb-time" :label="kind.id === 'daily' ? t('ng_backup', 'At (HH:MM, {zone})', { zone }) : t('ng_backup', 'From (HH:MM, {zone})', { zone })" />
<NcTextField v-if="kind.id === 'hours'" v-model="hours" class="ngb-hours" type="number" min="1" max="23" :label="t('ng_backup', 'Every … hours')" />
<NcButton :disabled="busy || wanted === current" @click="save">{{ t('ng_backup', 'Save') }}</NcButton>
</div>
</template>
<script setup>
import { computed, ref, watch } from 'vue'
import { t } from '@nextcloud/l10n'
import NcButton from '@nextcloud/vue/components/NcButton'
import NcSelect from '@nextcloud/vue/components/NcSelect'
import NcTextField from '@nextcloud/vue/components/NcTextField'
import api from '../api.js'
const props = defineProps({ target: { type: Object, required: true } })
const emit = defineEmits(['changed', 'error'])
const kinds = [
{ id: 'off', label: t('ng_backup', 'Off') },
{ id: 'daily', label: t('ng_backup', 'Once a day') },
{ id: 'hours', label: t('ng_backup', 'Every few hours') },
]
// A schedule is shown and edited in the zone it was saved in, named next to the time: "every 6
// hours from 02:00" runs until midnight of that zone, so moving it to another zone would change
// how often it runs. A location without a schedule gets the administrator's own zone (the browser's).
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
const zone = computed(() => props.target.schedule !== '' && props.target.scheduleZone ? props.target.scheduleZone : browserZone)
const kind = ref(kinds[0])
const time = ref('02:00')
const hours = ref('6')
const busy = ref(false)
const current = computed(() => {
const [at, every] = props.target.schedule.split('/')
return at === '' ? '' : at + (every ? '/' + every : '')
})
const wanted = computed(() => {
if (kind.value.id === 'off') {
return ''
}
return time.value.trim() + (kind.value.id === 'hours' ? '/' + String(hours.value).trim() : '')
})
watch(current, (schedule) => {
const [at, every] = schedule.split('/')
kind.value = kinds[at === '' ? 0 : (every ? 2 : 1)]
time.value = at === '' ? time.value : at
hours.value = every ?? hours.value
}, { immediate: true })
async function save() {
busy.value = true
emit('error', '')
try {
await api.setTargetSchedule(props.target.id, wanted.value, zone.value)
emit('changed')
} catch (e) {
emit('error', e.message)
} finally {
busy.value = false
}
}
</script>
<style scoped>
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-name { min-width: 120px; align-self: center; }
.ngb-time { max-width: 220px; }
.ngb-hours { max-width: 120px; }
</style>
+37 -2
View File
@@ -11,6 +11,13 @@
</div>
<template v-if="snapshot">
<div class="ngb-row ngb-marks">
<NcTextField v-model="note" class="ngb-note" :maxlength="500" :label="t('ng_backup', 'Note (can be changed later)')" />
<NcButton :disabled="marking || note.trim() === (snapshot.note ?? '')" @click="saveNote">{{ t('ng_backup', 'Save note') }}</NcButton>
<NcCheckboxRadioSwitch :model-value="snapshot.pinned" :disabled="marking" @update:model-value="savePinned">
{{ t('ng_backup', 'Pin: the retention policy does not remove this restore point') }}
</NcCheckboxRadioSwitch>
</div>
<div class="ngb-crumbs">
<NcButton variant="tertiary" @click="open('data')">data</NcButton>
<template v-for="(crumb, i) in crumbs" :key="crumb.path">
@@ -49,6 +56,7 @@ import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
import NcSelect from '@nextcloud/vue/components/NcSelect'
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
import NcTextField from '@nextcloud/vue/components/NcTextField'
import { confirmPassword } from '@nextcloud/password-confirmation'
import '@nextcloud/password-confirmation/style.css'
import api from '../api.js'
@@ -74,7 +82,10 @@ const modes = [
{ id: 'replace', label: t('ng_backup', 'Into the original place, overwrite and move files that are not in the backup to the trash') },
]
const snapshotLabel = (s) => formatDateTime(s.createdAt) + (s.label ? ' – ' + s.label : '')
const note = ref('')
const marking = ref(false)
const snapshotLabel = (s) => (s.pinned ? '📌 ' : '') + formatDateTime(s.createdAt) + (s.label ? ' – ' + s.label : '') + (s.note ? ' – ' + s.note : '')
const join = (name) => path.value + '/' + name
const restorable = (p) => /^data\/[^/]+\/files(\/.*)?$/.test(p)
const crumbs = computed(() => {
@@ -88,7 +99,29 @@ watch(target, async (tg) => {
snapshots.value = list.map(s => ({ ...s, displayLabel: snapshotLabel(s) }))
snapshot.value = snapshots.value[0] ?? null
}, { immediate: true })
watch(snapshot, () => { selected.value = ''; open('data') })
watch(snapshot, (s) => { selected.value = ''; note.value = s?.note ?? ''; open('data') })
// Changes the snapshot in place: choosing another one, which opens its folders again, is not needed
async function mark(call, change) {
const s = snapshot.value
marking.value = true
error.value = message.value = ''
try {
await call(target.value.id, s.id)
change(s)
s.displayLabel = snapshotLabel(s)
emit('changed')
} catch (e) {
error.value = e.message
} finally {
marking.value = false
}
}
const saveNote = () => {
const text = note.value.trim()
return mark((targetId, id) => api.setSnapshotNote(targetId, id, text), (s) => { s.note = text })
}
const savePinned = (pinned) => mark((targetId, id) => api.setSnapshotPinned(targetId, id, pinned), (s) => { s.pinned = pinned })
async function open(p) {
if (!snapshot.value) {
@@ -125,6 +158,8 @@ async function restore() {
<style scoped>
.ngb-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-row :deep(.ngb-snapshot) { min-width: min(360px, 100%); }
.ngb-marks { align-items: flex-end; }
.ngb-note { max-width: 360px; }
.ngb-crumbs { display: flex; align-items: center; flex-wrap: wrap; }
.ngb-entries li { display: flex; align-items: center; gap: 12px; min-height: 36px; }
.ngb-entries li.selected { background: var(--color-primary-element-light); border-radius: var(--border-radius); }
-27
View File
@@ -26,30 +26,3 @@ export function formatSize(bytes) {
}
return i === 0 ? `${b} B` : `${b.toFixed(1)} ${units[i]}`
}
/**
* A time of day "HH:MM" in zone `from`, as "HH:MM" in zone `to`, at today's offsets.
*
* @param {string} hhmm the time
* @param {string} from IANA zone it is in
* @param {string} to IANA zone to show it in
* @param {Date} now the day to convert on (today)
* @return {string}
*/
export function toZone(hhmm, from, to, now = new Date()) {
const [h, m] = hhmm.split(':').map(Number)
const parts = (zone, date) => Object.fromEntries(new Intl.DateTimeFormat('en-GB', {
timeZone: zone, hourCycle: 'h23', year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit',
}).formatToParts(date).map((p) => [p.type, p.value]))
const offsetAt = (at) => {
const seen = parts(from, new Date(at))
return Date.UTC(+seen.year, +seen.month - 1, +seen.day, +seen.hour, +seen.minute) - at
}
const today = parts(from, now)
const wall = Date.UTC(+today.year, +today.month - 1, +today.day, h, m)
// the offset at the wall time itself: on a day the clocks change it differs from the one at
// midnight, so it is looked up a second time at the moment found first
const at = wall - offsetAt(wall - offsetAt(wall))
const there = parts(to, new Date(at))
return `${there.hour}:${there.minute}`
}
+35
View File
@@ -22,6 +22,27 @@ class KeyRingTest extends TestCase {
KeyRing::unwrap($wrapped, 'wrong');
}
public function testInvalidBase64IsADamagedKey(): void {
$wrapped = KeyRing::generate()->wrap('right');
$wrapped['salt'] = '***not base64***';
$this->expectException(CryptoException::class);
$this->expectExceptionMessage('Damaged key file');
KeyRing::unwrapAny($wrapped, 'right');
}
public function testAShortSaltOrNonceIsADamagedKey(): void {
foreach (['salt', 'nonce'] as $part) {
$wrapped = KeyRing::generate()->wrap('right');
$wrapped[$part] = base64_encode('short');
try {
KeyRing::unwrapAny($wrapped, 'right');
$this->fail("a short $part was accepted");
} catch (CryptoException $e) {
$this->assertStringContainsString('Damaged key file', $e->getMessage(), $part);
}
}
}
public function testBlobIdsAreKeyed(): void {
$this->assertNotSame(KeyRing::generate()->blobId('same'), KeyRing::generate()->blobId('same'));
$k = KeyRing::generate();
@@ -34,6 +55,20 @@ class KeyRingTest extends TestCase {
new KeyRing('short');
}
public function testADamagedSlotDoesNotStopTheNextOne(): void {
$keys = KeyRing::generate();
$broken = $keys->wrap('passphrase in the safe');
$broken['salt'] = '*** not base64 ***';
$field = ['slots' => [
['slot' => 1, 'label' => 'Safe', 'key' => $broken],
['slot' => 2, 'label' => 'Missing parts', 'key' => ['v' => 1]],
['slot' => 3, 'label' => 'CTO', 'key' => $keys->wrap('passphrase of the cto')],
]];
$this->assertSame($keys->blobId('x'), KeyRing::unwrapAny($field, 'passphrase of the cto')->blobId('x'));
$this->expectException(CryptoException::class);
KeyRing::unwrapAny($field, 'passphrase in the safe');
}
public function testAnySlotOpensTheSameKey(): void {
$keys = KeyRing::generate();
$field = ['slots' => [
+5 -1
View File
@@ -82,7 +82,11 @@ class EmptyDirsTest extends TestCase {
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
}
$this->assertSame('commit', $state['phase']);
$this->assertSame(4, $state['stats']['unreadable'], 'each reported once, over several steps');
$this->assertSame(4, $state['stats']['badNames'], 'each reported once, over several steps');
$this->assertArrayNotHasKey('unreadable', $state['stats'], 'not a problem with permissions');
$shown = BackupRun::warnings($state)['badNames'];
$this->assertContains('data/alice/files/line?break', $shown, 'no control character in a message');
$this->assertSame($shown, array_filter($shown, fn (string $name): bool => mb_check_encoding($name, 'UTF-8')));
$this->repo()->recordSnapshot($state['snapshot']);
$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], array_column($this->repo()->emptyDirs($state['snapshot']), 0));
+127
View File
@@ -0,0 +1,127 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Job\BackupRun;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/** Folders with something in them get their original modification time back on a restore. */
class FolderTimesTest extends TestCase {
use TempDirTrait;
private const OLD = 1500000000;
private const OLDER = 1400000000;
private LocalBackend $backend;
private KeyRing $keys;
private string $src;
protected function setUp(): void {
$this->backend = new LocalBackend($this->tempDir());
$this->keys = KeyRing::generate();
Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
$this->src = $this->tempDir();
mkdir("$this->src/data/alice/files/Old/Deep", 0700, true);
mkdir("$this->src/data/alice/files/2026", 0700, true);
file_put_contents("$this->src/data/alice/files/Old/Deep/a.txt", 'a');
file_put_contents("$this->src/data/alice/files/Old/b.txt", 'b');
file_put_contents("$this->src/data/alice/files/2026/c.txt", 'c');
touch("$this->src/data/alice/files/Old/Deep", self::OLDER);
touch("$this->src/data/alice/files/Old", self::OLD);
touch("$this->src/data/alice/files/2026", self::OLD);
}
private function repo(): Repository {
return Repository::openWithKey($this->backend, $this->keys);
}
private function backup(float $seconds = 30.0): string {
$state = BackupRun::start(['data' => "$this->src/data"]);
for ($i = 0; $i < 500 && $state['phase'] !== 'commit'; $i++) {
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, microtime(true) + $seconds)), true);
}
$this->assertSame('commit', $state['phase']);
$this->repo()->recordSnapshot($state['snapshot']);
return $state['snapshot'];
}
private function time(string $path): int {
clearstatcache();
return (int)filemtime($path);
}
public function testEachFolderIsRecordedOnceAlsoInSmallSteps(): void {
$expected = ['data', 'data/alice', 'data/alice/files', 'data/alice/files/2026', 'data/alice/files/Old', 'data/alice/files/Old/Deep'];
$this->assertSame($expected, array_column($this->repo()->folderTimes($this->backup()), 0));
$this->assertSame($expected, array_column($this->repo()->folderTimes($this->backup(0.0)), 0), 'a step per entry');
}
public function testNoEmptyFolderIsAmongThem(): void {
mkdir("$this->src/data/alice/files/Hollow");
$snapshot = $this->backup();
$this->assertNotContains('data/alice/files/Hollow', array_column($this->repo()->folderTimes($snapshot), 0));
$this->assertContains('data/alice/files/Hollow', array_column($this->repo()->emptyDirs($snapshot), 0));
}
public function testARestoreBringsThemBack(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->repo()->restore($snapshot, $target);
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/Old"));
$this->assertSame(self::OLDER, $this->time("$target/data/alice/files/Old/Deep"));
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/2026"));
}
public function testARestoreThatGoesOnAfterAStopToo(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->repo()->restore($snapshot, $target);
touch("$target/data/alice/files/Old", 1800000000);
$this->repo()->restore($snapshot, $target, keepUnchanged: true);
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/Old"));
}
public function testAPrefixLeavesTheTargetAndOthersAlone(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$this->repo()->restore($snapshot, $target, 'data/alice/files/Old/', true);
$this->assertSame(self::OLDER, $this->time("$target/Deep"));
$this->assertNotSame(self::OLD, $this->time($target), 'the directory that was chosen is not ours to change');
$this->assertDirectoryDoesNotExist("$target/2026");
}
public function testALinkOutOfTheTargetIsRefused(): void {
$snapshot = $this->backup();
$target = $this->tempDir();
$outside = $this->tempDir();
touch($outside, 1800000000);
mkdir("$target/data/alice/files", 0700, true);
symlink($outside, "$target/data/alice/files/Old");
$this->expectException(\OCA\NgBackup\Repository\RestoreTargetException::class);
$this->repo()->restore($snapshot, $target);
}
public function testAFolderWithABadNameIsLeftOutAndWhatIsInItReported(): void {
mkdir("$this->src/data/odd\\folder");
file_put_contents("$this->src/data/odd\\folder/x", 'x');
$state = BackupRun::start(['data' => "$this->src/data"]);
for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
}
$this->assertSame('commit', $state['phase']);
$this->assertSame(1, $state['stats']['badNames']);
$this->repo()->recordSnapshot($state['snapshot']);
$this->assertNotContains("data/odd\\folder", array_column($this->repo()->folderTimes($state['snapshot']), 0));
}
public function testASnapshotWithoutThemHasNone(): void {
$snapshot = $this->repo()->backupDirectory($this->src)['snapshot'];
$this->assertSame([], $this->repo()->folderTimes($snapshot));
}
}
@@ -0,0 +1,113 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\CryptoException;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/** Disaster recovery without the recovery kit: the location gives its key with a passphrase. */
class KeyFromLocationTest extends TestCase {
use TempDirTrait;
private string $dir;
private LocalBackend $backend;
private KeyRing $keys;
protected function setUp(): void {
$this->dir = $this->tempDir();
$this->backend = new LocalBackend($this->dir);
$this->keys = KeyRing::generate();
$slots = ['slots' => [
['slot' => 1, 'label' => 'A', 'created' => 0, 'key' => $this->keys->wrap('first passphrase')],
['slot' => 2, 'label' => 'B', 'created' => 0, 'key' => $this->keys->wrap('second passphrase')],
]];
Repository::initWithKey($this->backend, $this->keys, $slots)->recordSettings(['delete_delay_days' => 30]);
}
public function testEachPassphraseGivesTheKeyThatOpensTheBackups(): void {
foreach (['first passphrase', 'second passphrase'] as $passphrase) {
$keys = KeyRing::unwrapAny(Repository::wrappedKeyAt($this->backend), $passphrase);
$this->assertTrue(Repository::madeWithKey($this->backend, $keys));
$this->assertSame($this->keys->exportMasterKey(), $keys->exportMasterKey());
}
}
public function testAWrongPassphraseGivesNothing(): void {
$this->expectException(CryptoException::class);
KeyRing::unwrapAny(Repository::wrappedKeyAt($this->backend), 'not the passphrase');
}
public function testAnotherKeyIsNotTakenForTheRightOne(): void {
$this->assertFalse(Repository::madeWithKey($this->backend, KeyRing::generate()));
}
public function testTheSettingsAreSealedAndWrittenOnce(): void {
$this->assertSame(30, Repository::settingsAt($this->backend, $this->keys)['delete_delay_days']);
Repository::openWithKey($this->backend, $this->keys)->recordSettings(['delete_delay_days' => 1]);
$this->assertSame(30, Repository::settingsAt($this->backend, $this->keys)['delete_delay_days']);
$this->assertSame(['unreadable' => true], Repository::settingsAt($this->backend, KeyRing::generate()), 'another key: there, but not readable');
unlink($this->dir . '/settings');
$this->assertSame([], Repository::settingsAt($this->backend, $this->keys), 'none at all: a location from before 1.1.0');
}
public function testAlteredSettingsAreNotBelieved(): void {
$file = $this->dir . '/settings';
$raw = file_get_contents($file);
$raw[strlen($raw) - 1] = $raw[strlen($raw) - 1] ^ "\x01";
file_put_contents($file, $raw);
$this->assertSame(['unreadable' => true], Repository::settingsAt($this->backend, $this->keys));
}
public function testTheKitWritesALostConfigAgainAndTheBackupsOpen(): void {
$src = $this->tempDir();
file_put_contents("$src/a", 'still here');
$repo = Repository::openWithKey($this->backend, $this->keys);
$id = $repo->id();
$snapshot = $repo->backupDirectory($src)['snapshot'];
unlink($this->dir . '/config');
$this->assertTrue(Repository::holdsData($this->backend));
Repository::repairConfig($this->backend, $this->keys, $this->keys->wrap('kit passphrase'));
$repo = Repository::openWithKey($this->backend, $this->keys);
$this->assertSame($id, $repo->id());
$this->assertSame([$snapshot], $repo->listSnapshots());
$out = $this->tempDir();
$repo->restore($snapshot, $out);
$this->assertSame('still here', file_get_contents("$out/a"));
$this->assertSame($this->keys->exportMasterKey(), KeyRing::unwrapAny(Repository::wrappedKeyAt($this->backend), 'kit passphrase')->exportMasterKey());
}
public function testAnotherKeyWritesNoConfig(): void {
unlink($this->dir . '/config');
$other = KeyRing::generate();
try {
Repository::repairConfig($this->backend, $other, $other->wrap('x'));
$this->fail('a config was written with another key');
} catch (\OCA\NgBackup\Repository\RepositoryException $e) {
$this->assertStringContainsString('another server', $e->getMessage());
}
$this->assertFileDoesNotExist($this->dir . '/config');
}
public function testAConfigThatIsThereIsNeverRepaired(): void {
$before = file_get_contents($this->dir . '/config');
try {
Repository::repairConfig($this->backend, $this->keys, $this->keys->wrap('x'));
$this->fail('an existing config was replaced');
} catch (\OCA\NgBackup\Repository\RepositoryException) {
}
$this->assertSame($before, file_get_contents($this->dir . '/config'));
}
public function testALocationWithoutSettingsHasNone(): void {
$other = new LocalBackend($this->tempDir());
Repository::initWithKey($other, $this->keys, $this->keys->wrap('x'));
$this->assertSame([], Repository::settingsAt($other, $this->keys));
}
}
@@ -0,0 +1,89 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/** A pin and a note on a snapshot live in the catalog on the location, so a new server sees them too. */
class SnapshotMarksTest extends TestCase {
use TempDirTrait;
private LocalBackend $backend;
private KeyRing $keys;
private string $snapshot;
protected function setUp(): void {
$this->backend = new LocalBackend($this->tempDir());
$this->keys = KeyRing::generate();
$src = $this->tempDir();
file_put_contents("$src/a", 'x');
$this->snapshot = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'))->backupDirectory($src)['snapshot'];
}
private function repo(): Repository {
return Repository::openWithKey($this->backend, $this->keys);
}
public function testTheyAreThereAfterOpeningAgain(): void {
$this->repo()->setPinned($this->snapshot, true);
$this->repo()->setSnapshotNote($this->snapshot, 'before the upgrade');
$this->assertSame([$this->snapshot], $this->repo()->pinnedSnapshots());
$this->assertSame([$this->snapshot => 'before the upgrade'], $this->repo()->snapshotNotes());
}
public function testTheyCanBeTakenOffAgain(): void {
$this->repo()->setPinned($this->snapshot, true);
$this->repo()->setSnapshotNote($this->snapshot, 'x');
$this->repo()->setPinned($this->snapshot, false);
$this->repo()->setSnapshotNote($this->snapshot, '');
$this->assertSame([], $this->repo()->pinnedSnapshots());
$this->assertSame([], $this->repo()->snapshotNotes());
}
public function testNoChangeWritesNoCatalog(): void {
$repo = $this->repo();
$repo->setPinned($this->snapshot, true);
$generation = $repo->catalogGeneration();
$repo->setPinned($this->snapshot, true);
$repo->setSnapshotNote($this->snapshot, '');
$this->assertSame($generation, $repo->catalogGeneration());
}
public function testASnapshotThatIsNotThereIsRefused(): void {
$this->expectException(\InvalidArgumentException::class);
$this->repo()->setPinned('0123456789abcdef0123456789abcdef', true);
}
public function testATrashedSnapshotLosesItsPinAndKeepsItsNote(): void {
$repo = $this->repo();
$repo->setPinned($this->snapshot, true);
$repo->setSnapshotNote($this->snapshot, 'kept');
$repo->forget($this->snapshot);
$this->assertSame([], $this->repo()->pinnedSnapshots());
$this->assertSame([$this->snapshot => 'kept'], $this->repo()->snapshotNotes());
$repo = $this->repo();
$repo->untrash($this->snapshot);
$this->assertSame([], $this->repo()->pinnedSnapshots(), 'back in the list, as a normal one');
$this->assertSame([$this->snapshot => 'kept'], $this->repo()->snapshotNotes());
}
public function testThePurgeTakesTheNoteAlong(): void {
$repo = $this->repo();
$repo->setSnapshotNote($this->snapshot, 'kept');
$repo->forget($this->snapshot);
$repo->purgeTrash(0);
$this->assertSame([], $this->repo()->snapshotNotes());
}
public function testACatalogFromBeforeThisVersionReadsAsNone(): void {
$this->assertSame([], $this->repo()->pinnedSnapshots());
$this->assertSame([], $this->repo()->snapshotNotes());
}
}
+35
View File
@@ -0,0 +1,35 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Service;
use OCA\NgBackup\Service\HeavyJobPause as P;
use PHPUnit\Framework\TestCase;
class HeavyJobPauseTest extends TestCase {
private const NOW = 1000000;
public function testDuringABackupTheyArePaused(): void {
$this->assertSame(P::PAUSE, P::decide(true, true, 0, 0, self::NOW));
$this->assertSame(P::PAUSE, P::decide(true, true, self::NOW - 60, 0, self::NOW), 'jobs added since then too');
}
public function testWithoutABackupNothingHappensOrThePauseEnds(): void {
$this->assertSame(P::NOTHING, P::decide(true, false, 0, 0, self::NOW));
$this->assertSame(P::RELEASE, P::decide(true, false, self::NOW - 60, 0, self::NOW));
}
public function testTurningItOffEndsThePause(): void {
$this->assertSame(P::NOTHING, P::decide(false, true, 0, 0, self::NOW));
$this->assertSame(P::RELEASE, P::decide(false, true, self::NOW - 60, 0, self::NOW));
}
public function testAfterSixHoursTheyGetTheirTurnAndThenPauseAgain(): void {
$this->assertSame(P::PAUSE, P::decide(true, true, self::NOW - P::MAX_SECONDS + 1, 0, self::NOW));
$this->assertSame(P::REST, P::decide(true, true, self::NOW - P::MAX_SECONDS, 0, self::NOW));
$rest = self::NOW + P::REST_SECONDS;
$this->assertSame(P::NOTHING, P::decide(true, true, 0, $rest, $rest - 1));
$this->assertSame(P::PAUSE, P::decide(true, true, 0, $rest, $rest));
}
}
@@ -0,0 +1,77 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Service;
use OCA\NgBackup\Service\LocationSchedule as S;
use PHPUnit\Framework\TestCase;
class LocationScheduleTest extends TestCase {
private const BERLIN = 'Europe/Berlin';
private static function at(string $time): int {
return (int)strtotime($time);
}
public function testWhatCanBeSaved(): void {
$this->assertSame('', S::normalize('off'));
$this->assertSame('', S::normalize(' '));
$this->assertSame('02:30', S::normalize('02:30'));
$this->assertSame('02:30', S::normalize('02:30/24'));
$this->assertSame('00:00/6', S::normalize('00:00/6'));
foreach (['2:30', '24:00', '02:30/0', '02:30/25', 'daily', '02:30/x'] as $bad) {
try {
S::normalize($bad);
$this->fail("$bad accepted");
} catch (\InvalidArgumentException) {
$this->addToAssertionCount(1);
}
}
}
public function testOffHasNoMoment(): void {
$this->assertNull(S::lastMoment('', self::BERLIN, self::at('2026-11-10 12:00 UTC')));
}
public function testDailyIsInTheZoneOfTheLocation(): void {
$this->assertSame(self::at('2026-11-10 02:00 UTC'), S::lastMoment('03:00', self::BERLIN, self::at('2026-11-10 12:00 UTC')));
$this->assertSame(self::at('2026-11-09 02:00 UTC'), S::lastMoment('03:00', self::BERLIN, self::at('2026-11-10 01:00 UTC')), 'not yet today');
}
public function testEveryNHoursStartsOverTheNextDay(): void {
$zone = 'UTC';
$this->assertSame(self::at('2026-11-10 12:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 13:00 UTC')));
$this->assertSame(self::at('2026-11-09 22:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 01:59 UTC')), '02, 07, 12, 17, 22, then 02 again');
$this->assertSame(self::at('2026-11-10 02:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 02:00 UTC')), 'at the moment itself');
}
public function testAClockChangeInTheNightStillGivesOneMoment(): void {
$this->assertSame(self::at('2026-03-29 01:30 UTC'), S::lastMoment('02:30', self::BERLIN, self::at('2026-03-29 12:00 UTC')), '02:30 does not exist that night');
$autumn = S::lastMoment('02:30', self::BERLIN, self::at('2026-10-25 12:00 UTC'));
$this->assertNotNull($autumn);
$local = (new \DateTimeImmutable('@' . $autumn))->setTimezone(new \DateTimeZone(self::BERLIN));
$this->assertSame('2026-10-25 02:30', $local->format('Y-m-d H:i'), '02:30 happens twice that night: one of them, on that day');
}
public function testChoosingATimeDoesNotStartABackup(): void {
$moment = self::at('2026-11-10 02:00 UTC');
$this->assertFalse(S::isDue($moment, $moment + 1, null, 0, null, $moment + 3600));
$this->assertTrue(S::isDue($moment, $moment - 1, null, 0, null, $moment + 3600));
$this->assertTrue(S::isDue($moment, 0, null, 0, null, $moment), 'a location from before schedules per location');
}
public function testABackupSinceTheMomentIsEnough(): void {
$moment = 1000000;
$this->assertFalse(S::isDue($moment, 0, $moment, 1, $moment + 10, $moment + 20));
$this->assertTrue(S::isDue($moment, 0, $moment - 1, 0, null, $moment + 20), 'it started before this moment: one more');
}
public function testAfterAFailureOnlyAFewAttemptsAnHourApart(): void {
$moment = 1000000;
$finished = $moment + 60;
$this->assertFalse(S::isDue($moment, 0, null, 1, $finished, $finished + S::RETRY_AFTER - 1));
$this->assertTrue(S::isDue($moment, 0, null, 1, $finished, $finished + S::RETRY_AFTER));
$this->assertFalse(S::isDue($moment, 0, null, S::MAX_ATTEMPTS, $finished, $finished + 86400));
}
}
@@ -25,6 +25,17 @@ class RetentionPolicyTest extends TestCase {
$this->assertCount(3, $months);
}
public function testAPinnedSnapshotStaysAndDoesNotCount(): void {
$policy = new RetentionPolicy(1, 0, 0, 0);
$snaps = ['old' => 100, 'pinned' => 300, 'new' => 200];
$this->assertSame(['pinned'], array_keys($policy->keep($snaps)), 'without the pin it would take the place of "last"');
$keep = $policy->keepWithPinned($snaps, ['pinned', 'gone-already']);
$this->assertSame(['pinned'], $keep['pinned']);
$this->assertSame(['last'], $keep['new'], 'the newest one that is not pinned');
$this->assertArrayNotHasKey('old', $keep);
$this->assertArrayNotHasKey('gone-already', $keep);
}
public function testKeepsEverythingWhenFewSnapshots(): void {
$keep = (new RetentionPolicy())->keep(['a' => 100, 'b' => 200]);
$this->assertSame(['a', 'b'], array_values(array_intersect(['a', 'b'], array_keys($keep))));
@@ -0,0 +1,22 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Service;
use OCA\NgBackup\Service\AlertService;
use PHPUnit\Framework\TestCase;
/** A location without a schedule: a weekly reminder while another is backed up, else a daily one. */
class ScheduleOffReminderTest extends TestCase {
public function testOnceAWeekWhileAnotherLocationIsActive(): void {
$this->assertFalse(AlertService::offReminderDue(0, 6 * 86400, true));
$this->assertTrue(AlertService::offReminderDue(0, 7 * 86400, true));
}
public function testEveryDayWhenNoLocationIsActive(): void {
$this->assertFalse(AlertService::offReminderDue(0, 3600, false));
$this->assertTrue(AlertService::offReminderDue(0, 86400, false));
$this->assertTrue(AlertService::offReminderDue(0, 86400 - 60, false), 'a cron run a minute earlier than yesterday');
}
}
-11
View File
@@ -57,15 +57,4 @@ class ScheduleTest extends TestCase {
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
}
public function testTodayAtIsTheMomentInTheSchedulesZone(): void {
$this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin'];
$at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC'));
$this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC');
}
public function testAnUnknownZoneOrABadTimeIsRefused(): void {
$s = $this->schedule();
$this->expectException(\InvalidArgumentException::class);
$s->set('03:00', 'Mars/Olympus');
}
}
+39 -5
View File
@@ -2,8 +2,10 @@
# Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a
# SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup
# location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7.
# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp|salvage|salvage-files|to-pgsql|to-mysql|to-sqlite]
# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp|salvage|salvage-files|to-pgsql|to-mysql|to-sqlite|no-kit]
# e.g. full-restore-roundtrip.sh 35-apache pgsql
# With "no-kit" the recovery kit is gone: backup:target:add takes the key from the location with
# the passphrase, and backups wait until a new kit is confirmed.
# With "salvage-files" one pack is damaged and another is gone from the location. The restore must
# stop and point at --salvage; run again as the guide says, it restores the database and every file
# that can be read, and lists the rest (exit 2). NGB_FRT_DATA_MB sets the amount of data (default 200).
@@ -93,7 +95,10 @@ run_on ngb-frt-origin app:enable ng_backup >/dev/null
ok "origin instance up ($DB${LEGACY:+, NG Backup 0.9.7})"
docker exec ngb-frt-origin sh -c "printf 'full restore roundtrip passphrase 2026\n' > /tmp/pass"
run_on ngb-frt-origin backup:key:init --passphrase-file=/tmp/pass --label=CI >/dev/null || fail "key init"
DELAY_OPT=()
[ "$MODE" = no-kit ] && DELAY_OPT=(--delete-delay-days=21)
run_on ngb-frt-origin backup:key:init --passphrase-file=/tmp/pass --label=CI "${DELAY_OPT[@]}" >/dev/null || fail "key init"
ORIGIN_FP=$(run_on ngb-frt-origin backup:status | grep -o 'Key: [0-9a-f-]*' | cut -d' ' -f2)
run_on ngb-frt-origin backup:key:kit --output=/tmp/kit.json >/dev/null
CODE=$(run_on ngb-frt-origin backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
run_on ngb-frt-origin backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "confirm"
@@ -187,8 +192,29 @@ docker exec ngb-frt-restore chmod 644 /tmp/kit.json
docker cp /tmp/ngb-frt-pass ngb-frt-restore:/tmp/pass
ok "fresh restore instance up"
run_on ngb-frt-restore backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass >/dev/null || fail "key import"
ADD_OUT=$(run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo 2>&1) || { printf '%s\n' "$ADD_OUT"; fail "target add on the restore instance"; }
if [ "$MODE" = no-kit ]; then
# No recovery kit: backup:target:add takes the key from the location with a passphrase
docker exec ngb-frt-restore rm -f /tmp/kit.json
docker exec ngb-frt-restore sh -c "printf 'not the passphrase of the old server\n' > /tmp/wrongpass && chmod 644 /tmp/wrongpass"
OUT=$(run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo --passphrase-file=/tmp/wrongpass 2>&1) \
&& { printf '%s\n' "$OUT"; fail "a wrong passphrase was accepted"; }
printf '%s\n' "$OUT" | grep -q "Nothing was changed" || { printf '%s\n' "$OUT"; fail "a wrong passphrase did not say that nothing was changed"; }
STATUS_OUT=$(run_on ngb-frt-restore backup:status 2>&1 || true)
printf '%s\n' "$STATUS_OUT" | grep -q "Key: not created" || { printf '%s\n' "$STATUS_OUT"; fail "a wrong passphrase left a key behind"; }
[ -z "$(run_on ngb-frt-restore backup:target:list 2>&1 | grep offsite)" ] || fail "a wrong passphrase left a location behind"
OUT=$(run_on ngb-frt-restore backup:target:add other "${LOCATION[@]}" --path=nothing-here --passphrase-file=/tmp/pass 2>&1) \
&& { printf '%s\n' "$OUT"; fail "a folder without backups was added without a key"; }
printf '%s\n' "$OUT" | grep -q "no backups at this location" || { printf '%s\n' "$OUT"; fail "a folder without backups did not say so"; }
ok "without a kit: a wrong passphrase and a folder without backups change nothing"
ADD_OUT=$(run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo --passphrase-file=/tmp/pass 2>&1) \
|| { printf '%s\n' "$ADD_OUT"; fail "target add with the passphrase on the restore instance"; }
printf '%s\n' "$ADD_OUT" | grep -q "fingerprint $ORIGIN_FP" || { printf '%s\n' "$ADD_OUT"; fail "the key from the location is not the origin's ($ORIGIN_FP)"; }
printf '%s\n' "$ADD_OUT" | grep -q "21 days (from the location)" || { printf '%s\n' "$ADD_OUT"; fail "the deletion delay did not come from the location"; }
ok "without a kit: the key and the deletion delay came from the location"
else
run_on ngb-frt-restore backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass >/dev/null || fail "key import"
ADD_OUT=$(run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo 2>&1) || { printf '%s\n' "$ADD_OUT"; fail "target add on the restore instance"; }
fi
LIST_OUT=$(run_on ngb-frt-restore backup:list offsite 2>&1)
printf '%s\n' "$LIST_OUT" | grep -q "$SNAP" || {
printf 'target:add said:\n%s\nbackup:list said:\n%s\n' "$ADD_OUT" "$LIST_OUT"
@@ -196,7 +222,8 @@ printf '%s\n' "$LIST_OUT" | grep -q "$SNAP" || {
docker exec ngb-frt-restore sh -c 'ls -la /mnt/shared/repo /mnt/shared/repo/snapshots' || true
fail "the origin's snapshot is not listed after connecting the location"
}
ok "recovery kit imported, existing repository adopted, snapshot visible"
if [ "$MODE" = no-kit ]; then KEY_FROM="key from the location"; else KEY_FROM="recovery kit imported"; fi
ok "$KEY_FROM, existing repository adopted, snapshot visible"
if [ "$MODE" = salvage ]; then
set +e
@@ -371,6 +398,13 @@ if [ "$MODE" = sftp ]; then
ok "over SFTP: read ahead on connections of its own ($LOGINS SFTP logins during the restore), large file identical, no spool left"
fi
if [ "$MODE" = no-kit ]; then
OUT=$(run_on ngb-frt-restore backup:run offsite -l post-restore-check 2>&1) && { printf '%s\n' "$OUT"; fail "a backup ran without a confirmed recovery kit"; }
printf '%s\n' "$OUT" | grep -q "Confirm the recovery kit" || { printf '%s\n' "$OUT"; fail "the refused backup does not point at the recovery kit"; }
CODE=$(run_on ngb-frt-restore backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
run_on ngb-frt-restore backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "confirm a new kit after the restore"
ok "without a kit: backups wait for a new recovery kit to be confirmed"
fi
run_on ngb-frt-restore backup:run offsite -l post-restore-check >/dev/null || fail "a new backup after the restore fails"
ok "the restored instance can run a new backup of its own"
+311
View File
@@ -0,0 +1,311 @@
#!/bin/bash
# Getting the backup key onto a new server, with and without the recovery kit, and what the
# administrator is told in each case. Every scenario starts on a fresh Nextcloud (SQLite) with a
# copy of the origin's location; nothing is restored (full-restore-roundtrip.sh does that).
# Usage: recovery-scenarios.sh [nextcloud-image-tag] e.g. recovery-scenarios.sh 35-apache
#
# kit kit + passphrase: backup:key:import, then backup:target:add asks nothing
# kit-wrong kit with a wrong passphrase: refused, no key left behind
# no-kit passphrase only: target:add asks for it (or --passphrase-file), a wrong one
# changes nothing; backups wait for a new kit
# no-kit-old as no-kit, on a location backed up only before 1.1.0 (no settings file):
# the deletion delay is the default, and the admin is told so
# no-kit-damaged as no-kit, with a damaged settings file: the longest deletion delay, and why
# no-config-no-kit the location's config is gone and there is no kit: told that only the kit helps
# no-config-kit config gone, kit there: --repair-config writes it again, the backups open
# other-kit the kit of another server: refused, told to use the old server's kit
# empty nothing at the location: told to check the details, not to make a new key
# unused-key-kit a new key was made by mistake: key:import refuses and names
# --replace-unused-key; with it the kit's key takes its place
# unused-key-no-kit the same without a kit: target:add with --replace-unused-key and a passphrase
# of the old server; a wrong one keeps the unused key
# used-key a key that protected a location is never replaced, also after the location was removed
# slot-catch-up a passphrase added while a location was unreachable reaches it at its next
# backup; a new server then gets the key from it with that passphrase
set -euo pipefail
IMAGE="nextcloud:${1:-35-apache}"
APP=$(cd "$(dirname "$0")/../.." && pwd)
SHARED=$(mktemp -d)
chmod 777 "$SHARED"
PASS='recovery scenarios passphrase 2026'
fail() { echo "FAIL: $*"; exit 1; }
ok() { echo "ok $*"; }
cleanup() {
docker rm -fv ngb-rs-origin ngb-rs-other ngb-rs-new >/dev/null 2>&1 || true
docker run --rm -v "$SHARED:/target" alpine sh -c 'rm -rf /target/*' >/dev/null 2>&1 || true
rmdir "$SHARED" 2>/dev/null || true
return 0
}
trap cleanup EXIT
run_on() { docker exec -u www-data "$1" php occ "${@:2}"; }
# Output of a command that must fail; the scenario fails when it succeeds.
must_fail() { # container what args...
local out
if out=$(run_on "$1" "${@:3}" 2>&1); then printf '%s\n' "$out"; fail "$2: the command succeeded"; fi
printf '%s\n' "$out"
}
expect() { # output text what
printf '%s\n' "$1" | grep -qF -- "$2" || { printf '%s\n' "$1"; fail "$3: the output does not say \"$2\""; }
}
not_expect() {
printf '%s\n' "$1" | grep -qF -- "$2" && { printf '%s\n' "$1"; fail "$3: the output says \"$2\""; }
return 0
}
start() { # name
docker rm -fv "$1" >/dev/null 2>&1 || true
docker run -d --name "$1" -e SQLITE_DATABASE=nextcloud -e NEXTCLOUD_ADMIN_USER=admin -e NEXTCLOUD_ADMIN_PASSWORD=admin \
-v "$APP:/var/www/html/custom_apps/ng_backup:ro" -v "$SHARED:/mnt/shared" "$IMAGE" >/dev/null
for i in $(seq 1 90); do run_on "$1" status 2>/dev/null | grep -q "installed: true" && break; sleep 3; done
docker exec "$1" chown www-data:www-data /var/www/html/custom_apps
run_on "$1" app:enable ng_backup >/dev/null
docker exec "$1" sh -c "printf '%s\n' '$PASS' > /tmp/pass; printf 'not the passphrase at all\n' > /tmp/wrongpass; chmod 644 /tmp/pass /tmp/wrongpass"
}
key_of() { run_on "$1" backup:status 2>&1 | sed -n 's/^Key: \([0-9a-f-]*\),.*/\1/p'; }
no_key() { # container what
local s
s=$(run_on "$1" backup:status 2>&1 || true)
expect "$s" "Key: not created" "$2: a key was left behind"
[ -z "$(run_on "$1" backup:target:list 2>&1 | grep offsite || true)" ] || fail "$2: a location was left behind"
}
location() { echo "local -a null::null -o datadir=/mnt/shared/$1 --path=repo"; }
# A fresh server with its own copy of the origin's location
scenario() { # name
docker exec ngb-rs-origin sh -c "rm -rf /mnt/shared/$1 && mkdir -p /mnt/shared/$1 && cp -a /mnt/shared/origin/repo /mnt/shared/$1/ && chown -R www-data /mnt/shared/$1"
start ngb-rs-new
docker exec -i ngb-rs-new sh -c 'cat > /tmp/kit.json && chmod 644 /tmp/kit.json' < "$SHARED/kit.json"
}
# The origin: a key with a deletion delay of its own, a kit, one backup
start ngb-rs-origin
run_on ngb-rs-origin backup:key:init --passphrase-file=/tmp/pass --label=Safe --delete-delay-days=21 >/dev/null || fail "key init"
run_on ngb-rs-origin backup:key:kit --output=/tmp/kit.json >/dev/null
CODE=$(run_on ngb-rs-origin backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
run_on ngb-rs-origin backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "confirm"
docker exec ngb-rs-origin sh -c 'mkdir -p /mnt/shared/origin && chown www-data /mnt/shared/origin'
# shellcheck disable=SC2046
run_on ngb-rs-origin backup:target:add offsite $(location origin) >/dev/null || fail "target add on the origin"
run_on ngb-rs-origin backup:run offsite >/dev/null || fail "backup on the origin"
SNAP=$(run_on ngb-rs-origin backup:list offsite | grep -o '[0-9a-f]\{32\}' | head -1)
FP=$(key_of ngb-rs-origin)
docker exec ngb-rs-origin cat /tmp/kit.json > "$SHARED/kit.json"
docker exec ngb-rs-origin test -f /mnt/shared/origin/repo/settings || fail "the origin's location has no settings file"
ok "origin: key $FP, kit, snapshot $SNAP, settings on the location"
# kit: the passphrase once, for the kit; target:add asks nothing more
scenario kit
OUT=$(run_on ngb-rs-new backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass 2>&1) || { printf '%s\n' "$OUT"; fail "kit: key import"; }
expect "$OUT" "fingerprint $FP" "kit"
expect "$OUT" "21 days (from the kit)" "kit"
expect "$OUT" "backup:target:add" "kit: the next step"
# shellcheck disable=SC2046
OUT=$(run_on ngb-rs-new backup:target:add offsite $(location kit) --no-interaction 2>&1) || { printf '%s\n' "$OUT"; fail "kit: target add"; }
not_expect "$OUT" "Passphrase" "kit: target:add asked for a passphrase"
expect "$OUT" "found and adopted" "kit"
run_on ngb-rs-new backup:list offsite | grep -q "$SNAP" || fail "kit: the snapshot is not listed"
run_on ngb-rs-new backup:run offsite >/dev/null || fail "kit: a backup after the kit import"
ok "kit: imported with the passphrase, location added without asking again, backups run"
# kit-wrong
scenario kit-wrong
OUT=$(must_fail ngb-rs-new "kit-wrong" backup:key:import /tmp/kit.json --passphrase-file=/tmp/wrongpass)
expect "$OUT" "Wrong passphrase" "kit-wrong"
no_key ngb-rs-new "kit-wrong"
ok "kit-wrong: refused, nothing left behind"
# no-kit
scenario no-kit
docker exec ngb-rs-new rm /tmp/kit.json
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "no-kit without a passphrase" backup:target:add offsite $(location no-kit) --no-interaction)
expect "$OUT" "this location holds backups" "no-kit: explains where the key comes from"
expect "$OUT" "--passphrase-file" "no-kit: says how to give the passphrase"
no_key ngb-rs-new "no-kit without a passphrase"
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "no-kit wrong passphrase" backup:target:add offsite $(location no-kit) --passphrase-file=/tmp/wrongpass)
expect "$OUT" "Wrong passphrase" "no-kit wrong passphrase"
expect "$OUT" "Nothing was changed" "no-kit wrong passphrase"
no_key ngb-rs-new "no-kit wrong passphrase"
# shellcheck disable=SC2046
OUT=$(run_on ngb-rs-new backup:target:add offsite $(location no-kit) --passphrase-file=/tmp/pass 2>&1) || { printf '%s\n' "$OUT"; fail "no-kit: target add"; }
expect "$OUT" "fingerprint $FP" "no-kit"
expect "$OUT" "21 days (from the location)" "no-kit"
expect "$OUT" "download a new recovery kit" "no-kit: asks for a new kit"
run_on ngb-rs-new backup:list offsite | grep -q "$SNAP" || fail "no-kit: the snapshot is not listed"
OUT=$(must_fail ngb-rs-new "no-kit backup before a new kit" backup:run offsite)
expect "$OUT" "Confirm the recovery kit" "no-kit: a backup before a new kit"
CODE=$(run_on ngb-rs-new backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
run_on ngb-rs-new backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "no-kit: confirm the new kit"
run_on ngb-rs-new backup:run offsite >/dev/null || fail "no-kit: a backup after the new kit"
ok "no-kit: passphrase asked, wrong one changes nothing, key and delay from the location, backups wait for a new kit"
# no-kit-old: a location from before 1.1.0 has no settings file
scenario no-kit-old
docker exec ngb-rs-new rm /tmp/kit.json /mnt/shared/no-kit-old/repo/settings
# shellcheck disable=SC2046
OUT=$(run_on ngb-rs-new backup:target:add offsite $(location no-kit-old) --passphrase-file=/tmp/pass 2>&1) || { printf '%s\n' "$OUT"; fail "no-kit-old: target add"; }
expect "$OUT" "fingerprint $FP" "no-kit-old"
expect "$OUT" "7 days (the default" "no-kit-old: the default delay and why"
docker exec ngb-rs-new test -f /mnt/shared/no-kit-old/repo/settings || fail "no-kit-old: the settings were not written when the location was added"
ok "no-kit-old: key from the location, default delay with the reason, settings written now"
# no-kit-damaged: the settings file is there but damaged: the longest delay, and the admin is told
scenario no-kit-damaged
docker exec ngb-rs-new sh -c 'rm /tmp/kit.json && printf garbage >> /mnt/shared/no-kit-damaged/repo/settings'
# shellcheck disable=SC2046
OUT=$(run_on ngb-rs-new backup:target:add offsite $(location no-kit-damaged) --passphrase-file=/tmp/pass 2>&1) || { printf '%s\n' "$OUT"; fail "no-kit-damaged: target add"; }
expect "$OUT" "fingerprint $FP" "no-kit-damaged"
expect "$OUT" "365 days (the longest" "no-kit-damaged: the longest delay and why"
ok "no-kit-damaged: key from the location, the longest delay with the reason"
# no-config-no-kit
scenario no-config-no-kit
docker exec ngb-rs-new rm /tmp/kit.json /mnt/shared/no-config-no-kit/repo/config
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "no-config-no-kit" backup:target:add offsite $(location no-config-no-kit) --passphrase-file=/tmp/pass)
expect "$OUT" "its config file is gone" "no-config-no-kit"
expect "$OUT" "only the recovery kit has the key" "no-config-no-kit"
expect "$OUT" "--repair-config" "no-config-no-kit"
no_key ngb-rs-new "no-config-no-kit"
docker exec ngb-rs-new test ! -f /mnt/shared/no-config-no-kit/repo/config || fail "no-config-no-kit: a config was written"
ok "no-config-no-kit: told the kit is needed, nothing written"
# no-config-kit
scenario no-config-kit
docker exec ngb-rs-new rm /mnt/shared/no-config-kit/repo/config
run_on ngb-rs-new backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass >/dev/null || fail "no-config-kit: key import"
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "no-config-kit without --repair-config" backup:target:add offsite $(location no-config-kit))
expect "$OUT" "--repair-config" "no-config-kit: points at --repair-config"
docker exec ngb-rs-new test ! -f /mnt/shared/no-config-kit/repo/config || fail "no-config-kit: a config was written without --repair-config"
# shellcheck disable=SC2046
OUT=$(run_on ngb-rs-new backup:target:add offsite $(location no-config-kit) --repair-config 2>&1) || { printf '%s\n' "$OUT"; fail "no-config-kit: --repair-config"; }
expect "$OUT" "written again" "no-config-kit"
expect "$OUT" "found and adopted" "no-config-kit"
run_on ngb-rs-new backup:list offsite | grep -q "$SNAP" || fail "no-config-kit: the snapshot is not listed after the repair"
run_on ngb-rs-new backup:browse offsite "$SNAP" >/dev/null 2>&1 || fail "no-config-kit: the snapshot cannot be read after the repair"
ok "no-config-kit: refused without --repair-config, config written again with it, the backups open"
# other-kit: the kit of another server
start ngb-rs-other
run_on ngb-rs-other backup:key:init --passphrase-file=/tmp/pass >/dev/null || fail "other: key init"
run_on ngb-rs-other backup:key:kit --output=/tmp/other.json >/dev/null
docker exec ngb-rs-other cat /tmp/other.json > "$SHARED/other.json"
docker rm -fv ngb-rs-other >/dev/null
scenario other-kit
docker exec ngb-rs-new rm /tmp/kit.json
docker exec -i ngb-rs-new sh -c 'cat > /tmp/other.json && chmod 644 /tmp/other.json' < "$SHARED/other.json"
run_on ngb-rs-new backup:key:import /tmp/other.json --passphrase-file=/tmp/pass >/dev/null || fail "other-kit: key import"
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "other-kit" backup:target:add offsite $(location other-kit))
expect "$OUT" "different key" "other-kit"
expect "$OUT" "must be the old server's" "other-kit: points at the right kit"
[ -z "$(run_on ngb-rs-new backup:target:list 2>&1 | grep offsite || true)" ] || fail "other-kit: a location was left behind"
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "other-kit repair" backup:target:add offsite $(location other-kit) --repair-config)
expect "$OUT" "leave out --repair-config" "other-kit: a config that is there is not repaired"
ok "other-kit: refused, told to use the old server's kit, the config untouched"
# empty
scenario empty
docker exec ngb-rs-new sh -c 'mkdir -p /mnt/shared/nothing && chown www-data /mnt/shared/nothing'
OUT=$(must_fail ngb-rs-new "empty" backup:target:add offsite local -a null::null -o datadir=/mnt/shared/nothing --path=repo --passphrase-file=/tmp/pass)
expect "$OUT" "no backups at this location" "empty"
expect "$OUT" "Check the connection details and --path" "empty"
no_key ngb-rs-new "empty"
ok "empty: told to check the details, no key made"
# unused-key-kit: a key made by mistake, then the kit
scenario unused-key-kit
run_on ngb-rs-new backup:key:init --passphrase-file=/tmp/pass >/dev/null || fail "unused-key-kit: key init"
MISTAKE=$(key_of ngb-rs-new)
OUT=$(must_fail ngb-rs-new "unused-key-kit without the option" backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass)
expect "$OUT" "never used" "unused-key-kit"
expect "$OUT" "--replace-unused-key" "unused-key-kit: names the option"
[ "$(key_of ngb-rs-new)" = "$MISTAKE" ] || fail "unused-key-kit: the key changed without the option"
OUT=$(must_fail ngb-rs-new "unused-key-kit wrong passphrase" backup:key:import /tmp/kit.json --passphrase-file=/tmp/wrongpass --replace-unused-key)
expect "$OUT" "Wrong passphrase" "unused-key-kit wrong passphrase"
[ "$(key_of ngb-rs-new)" = "$MISTAKE" ] || fail "unused-key-kit: a wrong passphrase changed the key"
OUT=$(run_on ngb-rs-new backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass --replace-unused-key 2>&1) || { printf '%s\n' "$OUT"; fail "unused-key-kit: replace"; }
expect "$OUT" "The unused key was replaced" "unused-key-kit"
expect "$OUT" "fingerprint $FP" "unused-key-kit"
expect "$OUT" "21 days (from the kit)" "unused-key-kit"
# shellcheck disable=SC2046
OUT=$(run_on ngb-rs-new backup:target:add offsite $(location unused-key-kit) --no-interaction 2>&1) || { printf '%s\n' "$OUT"; fail "unused-key-kit: target add"; }
expect "$OUT" "found and adopted" "unused-key-kit"
run_on ngb-rs-new backup:run offsite >/dev/null || fail "unused-key-kit: a backup after the replacement"
ok "unused-key-kit: refused without the option, a wrong passphrase changes nothing, replaced with it, backups run"
# unused-key-no-kit: a key made by mistake, no kit
scenario unused-key-no-kit
docker exec ngb-rs-new rm /tmp/kit.json
run_on ngb-rs-new backup:key:init --passphrase-file=/tmp/pass >/dev/null || fail "unused-key-no-kit: key init"
MISTAKE=$(key_of ngb-rs-new)
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "unused-key-no-kit without the option" backup:target:add offsite $(location unused-key-no-kit) --no-interaction)
expect "$OUT" "different key" "unused-key-no-kit"
expect "$OUT" "never used" "unused-key-no-kit: says the key can make way"
expect "$OUT" "--replace-unused-key" "unused-key-no-kit: names the option"
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "unused-key-no-kit wrong passphrase" backup:target:add offsite $(location unused-key-no-kit) --replace-unused-key --passphrase-file=/tmp/wrongpass)
expect "$OUT" "Nothing was changed" "unused-key-no-kit wrong passphrase"
[ "$(key_of ngb-rs-new)" = "$MISTAKE" ] || fail "unused-key-no-kit: a wrong passphrase changed the key"
[ -z "$(run_on ngb-rs-new backup:target:list 2>&1 | grep offsite || true)" ] || fail "unused-key-no-kit: a location was left behind"
# shellcheck disable=SC2046
OUT=$(run_on ngb-rs-new backup:target:add offsite $(location unused-key-no-kit) --replace-unused-key --passphrase-file=/tmp/pass 2>&1) || { printf '%s\n' "$OUT"; fail "unused-key-no-kit: replace"; }
expect "$OUT" "The unused key was replaced" "unused-key-no-kit"
expect "$OUT" "fingerprint $FP" "unused-key-no-kit"
expect "$OUT" "download a new recovery kit" "unused-key-no-kit: asks for a new kit"
run_on ngb-rs-new backup:list offsite | grep -q "$SNAP" || fail "unused-key-no-kit: the snapshot is not listed"
OUT=$(must_fail ngb-rs-new "unused-key-no-kit backup before a new kit" backup:run offsite)
expect "$OUT" "Confirm the recovery kit" "unused-key-no-kit: the old key's kit confirmation does not count"
ok "unused-key-no-kit: hint on the refusal, a wrong passphrase changes nothing, replaced with a passphrase, a new kit is needed"
# used-key: a key that protected a location stays, also after that location is removed
scenario used-key
docker exec ngb-rs-new sh -c 'mkdir -p /mnt/shared/own && chown www-data /mnt/shared/own'
run_on ngb-rs-new backup:key:init --passphrase-file=/tmp/pass >/dev/null || fail "used-key: key init"
OWN=$(key_of ngb-rs-new)
run_on ngb-rs-new backup:target:add own local -a null::null -o datadir=/mnt/shared/own --path=repo >/dev/null || fail "used-key: own location"
run_on ngb-rs-new backup:target:remove own >/dev/null || fail "used-key: remove the own location"
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "used-key target:add" backup:target:add offsite $(location used-key) --replace-unused-key --passphrase-file=/tmp/pass)
expect "$OUT" "never replaced" "used-key target:add"
# shellcheck disable=SC2046
OUT=$(must_fail ngb-rs-new "used-key target:add without the option" backup:target:add offsite $(location used-key) --no-interaction)
not_expect "$OUT" "--replace-unused-key" "used-key: offers a replacement for a used key"
OUT=$(must_fail ngb-rs-new "used-key key:import" backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass --replace-unused-key)
expect "$OUT" "never replaced" "used-key key:import"
[ "$(key_of ngb-rs-new)" = "$OWN" ] || fail "used-key: the key changed"
ok "used-key: never replaced, by kit or passphrase, also after its location was removed"
# slot-catch-up: a passphrase change that missed a location reaches it at the next backup
start ngb-rs-new
docker exec ngb-rs-new sh -c "printf '%s\n' 'catch up passphrase of the cto' > /tmp/pass2 && chmod 644 /tmp/pass2 && rm -rf /mnt/shared/catchup && mkdir -p /mnt/shared/catchup && chown www-data /mnt/shared/catchup"
run_on ngb-rs-new backup:key:init --passphrase-file=/tmp/pass --label=Safe >/dev/null || fail "slot-catch-up: key init"
CODE=$(run_on ngb-rs-new backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
run_on ngb-rs-new backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "slot-catch-up: confirm"
# shellcheck disable=SC2046
run_on ngb-rs-new backup:target:add offsite $(location catchup) >/dev/null || fail "slot-catch-up: target add"
run_on ngb-rs-new backup:run offsite >/dev/null || fail "slot-catch-up: first backup"
BEFORE=$(docker exec ngb-rs-new md5sum /mnt/shared/catchup/repo/config | cut -d' ' -f1)
docker exec ngb-rs-new chmod 000 /mnt/shared/catchup/repo
OUT=$(must_fail ngb-rs-new "slot-catch-up: slot add with the location unreachable" backup:key:slot:add --label=CTO --passphrase-file=/tmp/pass2)
expect "$OUT" "tried again at their next backup" "slot-catch-up: says the change follows"
docker exec ngb-rs-new chmod 755 /mnt/shared/catchup/repo
[ "$(docker exec ngb-rs-new md5sum /mnt/shared/catchup/repo/config | cut -d' ' -f1)" = "$BEFORE" ] || fail "slot-catch-up: the location changed while unreachable"
CODE=$(run_on ngb-rs-new backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
run_on ngb-rs-new backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "slot-catch-up: confirm the new kit"
run_on ngb-rs-new backup:run offsite >/dev/null || fail "slot-catch-up: backup after the location came back"
[ "$(docker exec ngb-rs-new md5sum /mnt/shared/catchup/repo/config | cut -d' ' -f1)" != "$BEFORE" ] || fail "slot-catch-up: the backup did not bring the new passphrase to the location"
CATCHFP=$(key_of ngb-rs-new)
start ngb-rs-other
docker exec ngb-rs-other sh -c "printf '%s\n' 'catch up passphrase of the cto' > /tmp/pass2 && chmod 644 /tmp/pass2"
# shellcheck disable=SC2046
OUT=$(run_on ngb-rs-other backup:target:add offsite $(location catchup) --passphrase-file=/tmp/pass2 2>&1) || { printf '%s\n' "$OUT"; fail "slot-catch-up: a new server with the new passphrase"; }
expect "$OUT" "fingerprint $CATCHFP" "slot-catch-up: the new server"
ok "slot-catch-up: a passphrase that missed a location reaches it at the next backup and opens it on a new server"
echo "RECOVERY SCENARIOS PASS"
+112 -5
View File
@@ -23,12 +23,15 @@ ok "key, kit code and confirmation"
occ backup:target:add ci $TARGET_ARGS --path=repo >/dev/null || fail "target add"
ok "location added"
# Test data: a user with files.
# Test data: a user with files. No skeleton files on the first login, below, in the middle of the checks.
occ config:system:set skeletondirectory --value='' >/dev/null
occ config:system:set templatedirectory --value='' >/dev/null
$X sh -c "OC_PASS='Roundtrip-pw-2026' php occ user:add --password-from-env $U >/dev/null"
# large.bin (40 MB) fills more than one pack: a restore to a directory reads ahead over several ranges
$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs/Empty/Nested $DATA/$U/files/Hollow && touch -d @1700000000 $DATA/$U/files/Hollow && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin"
$X sh -c "touch -d @1600000000 $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs"
occ files:scan "$U" >/dev/null
SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
occ backup:run ci -l first >/dev/null || fail "first backup"
SECOND=$(occ backup:run ci -l second)
@@ -44,6 +47,14 @@ occ files:scan "$U" >/dev/null
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m new-folder >/dev/null || fail "restore new-folder"
$X sh -c "ls $DATA/$U/files | grep -q '^Restored '" || fail "no Restored folder"
$X sh -c "test -d $DATA/$U/files/Restored\ */Docs/Empty/Nested" || fail "the empty folder is not in the Restored folder"
# Folders with files: on disk and in what Nextcloud shows, the parent as well as the child.
RDIR=$($X sh -c "cd $DATA/$U/files && ls -d Restored\ *")
[ "$($X sh -c "stat -c %Y $DATA/$U/files/Restored\ */Docs/Sub")" = 1600000000 ] || fail "a folder with files did not get its time back"
[ "$($X sh -c "stat -c %Y $DATA/$U/files/Restored\ */Docs")" = 1600000000 ] || fail "the parent folder did not get its time back"
DAV=$($X curl -sf -u "$U:Roundtrip-pw-2026" -X PROPFIND -H 'Depth: 1' "http://localhost/remote.php/dav/files/$U/$(php -r 'echo rawurlencode($argv[1]);' "$RDIR")/Docs/") || fail "PROPFIND"
[ "$(echo "$DAV" | grep -o 'Sun, 13 Sep 2020 12:26:40 GMT' | wc -l)" = 2 ] || fail "Nextcloud does not show the time of the folders: $(echo "$DAV" | grep -o '<d:getlastmodified>[^<]*' | sort | uniq -c)"
occ backup:restore ci "$SNAP" "data/$U/files/Docs/Sub" -m merge >/dev/null || fail "restore into a folder that is there"
[ "$($X stat -c %Y "$DATA/$U/files/Docs/Sub")" != 1600000000 ] || fail "a folder that was already there got the old time"
occ backup:browse ci "$SNAP" "data/$U/files" | grep Hollow >/dev/null || fail "an empty folder is not shown when browsing the snapshot"
occ backup:restore ci "$SNAP" "data/$U/files/Hollow" -m merge >/dev/null || fail "restore of a folder that was empty"
$X sh -c "test -d $DATA/$U/files/Hollow" || fail "the empty folder did not come back by itself"
@@ -52,8 +63,8 @@ occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m replace >/dev/null || fail
$X sh -c "test -d $DATA/$U/files/Docs/Empty/Nested" || fail "replace did not bring back the empty folder, or moved it to the trash"
ok "empty folders: restored in each mode, by themselves and shown when browsing"
$X sh -c "rm -rf '$DATA/$U/files/Restored '*" ; occ files:scan "$U" >/dev/null
SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore"
SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore: $(diff <(echo "$SUM_BEFORE") <(echo "$SUM_AFTER") | cut -c1-120)"
$X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash"
ok "restore new-folder and replace: files identical, extra file in the trash"
@@ -67,7 +78,7 @@ ok "restore of one file to a directory; the temporary of a stopped restore remov
# All of the user's files to a directory: over S3 and SFTP the packs are read ahead
occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-dir >/dev/null || fail "restore of a folder to a directory"
SUM_DIR=$($X sh -c "cd /tmp/ngb-rt-dir/data/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
SUM_DIR=$($X sh -c "cd /tmp/ngb-rt-dir/data/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
[ "$SUM_BEFORE" = "$SUM_DIR" ] || fail "the files restored to a directory differ"
$X sh -c "ls -A /tmp/ngb-rt-dir | grep -q ngb-restore-spool" && fail "the read-ahead spool was left in the directory"
$X sh -c "test -d /tmp/ngb-rt-dir/data/$U/files/Docs/Empty/Nested" || fail "the empty folder is not in the directory"
@@ -101,13 +112,98 @@ ok "without system cron: warning in backup:status and the API, backup finished b
# The Bin reads in parts, as the page does (what is in the parts: CatalogTest)
api "$API/targets/$TID/trash" | php -r '$p = json_decode(stream_get_contents(STDIN), true); exit(is_int($p["total"] ?? null) && is_array($p["items"] ?? null) && count($p["items"]) <= $p["total"] ? 0 : 1);' || fail "the Bin API does not give a total and items"
ok "the Bin API reads in parts, with a total"
# Locations are backed up one after the other: the second one started is queued until the first is done.
for L in 2 3; do
$X sh -c "mkdir -p /tmp/ngb-rt-target$L"
occ backup:target:add ci$L local -a null::null -o datadir=/tmp/ngb-rt-target$L --path=repo >/dev/null || fail "target add ci$L"
done
IDS=$(api "$API/status" | php -r 'foreach (json_decode(stream_get_contents(STDIN), true)["targets"] as $t) { if ($t["name"] === "ci2" || $t["name"] === "ci3") { echo $t["id"], " "; } }')
for ID in $IDS; do api -X POST -d "{\"targetId\": $ID, \"label\": \"turn\"}" "$API/backups" >/dev/null || fail "start a backup to location $ID"; done
# Never two runs to one location: a second start gets the run that is there.
ID=${IDS%% *}
RUNS=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
api -X POST -d "{\"targetId\": $ID, \"label\": \"again\"}" "$API/backups" >/dev/null || fail "second start"
AGAIN=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
[ "$RUNS" = 2 ] && [ "$AGAIN" = 2 ] || fail "a second start to one location: $RUNS, then $AGAIN running"
occ backup:status | grep "queued" >/dev/null || fail "the second backup is not queued"
QUEUED=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["queued"]));')
[ "$QUEUED" = 1 ] || fail "$QUEUED backups queued instead of 1"
for i in $(seq 1 60); do
api -X POST "$API/tick" >/dev/null || fail "tick"
RUNNING=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
[ "$RUNNING" != 0 ] || break
done
[ "$RUNNING" = 0 ] || fail "the queued backups did not finish"
occ backup:list ci3 | grep " turn " >/dev/null || fail "the queued backup made no snapshot"
ok "locations in turn: the second backup queued, and done after the first"
# Heavy background jobs of other apps wait during a backup, and only then. Ticks keep this one moving.
HEAVY='OCA\Recognize\BackgroundJobs\NgbTestJob'
reserved() { $X php -r 'require "lib/base.php"; $q = \OC::$server->get(\OCP\IDBConnection::class)->getQueryBuilder(); echo (int)$q->select("reserved_at")->from("jobs")->where($q->expr()->eq("class", $q->createNamedParameter($argv[1])))->executeQuery()->fetchOne();' "$HEAVY"; }
tick_until_done() {
for i in $(seq 1 60); do
api -X POST "$API/tick" >/dev/null || fail "tick"
RUNNING=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
[ "$RUNNING" != 0 ] || return 0
done
fail "the backup did not finish"
}
$X php -r 'require "lib/base.php"; \OC::$server->get(\OCP\BackgroundJob\IJobList::class)->add($argv[1]);' "$HEAVY" || fail "add a job"
[ "$(reserved)" = 0 ] || fail "a new job is reserved"
L3=$(echo "$IDS" | awk '{print $2}')
api -X POST -d "{\"targetId\": $L3, \"label\": \"heavy\"}" "$API/backups" >/dev/null || fail "start a backup"
tick_until_done
[ "$(reserved)" != 0 ] || fail "the job was not paused during the backup"
api -X POST "$API/tick" >/dev/null
[ "$(reserved)" = 0 ] || fail "the job stays paused after the backup"
occ backup:pause-jobs off >/dev/null || fail "turn off"
sleep 8 # APCu app config cache of the web server
api -X POST -d "{\"targetId\": $L3, \"label\": \"heavy\"}" "$API/backups" >/dev/null || fail "start a backup"
tick_until_done
[ "$(reserved)" = 0 ] || fail "the job was paused with the setting off"
occ backup:pause-jobs on >/dev/null
$X php -r 'require "lib/base.php"; \OC::$server->get(\OCP\BackgroundJob\IJobList::class)->remove($argv[1]);' "$HEAVY"
ok "heavy background jobs: paused during a backup, back after it, left alone with the setting off"
occ background:cron >/dev/null
$X php -f /var/www/html/cron.php >/dev/null 2>&1 || true
occ backup:status | grep "system cron" >/dev/null && fail "backup:status still warns after cron ran"
sleep 8 # APCu app config cache of the web server
api "$API/status" | php -r 'exit(json_decode(stream_get_contents(STDIN), true)["cron"]["ok"] ? 0 : 1);' || fail "the status API still reports a missing cron after cron ran"
ok "with a system cron that ran: no warning in backup:status or the API"
# A schedule per location. Choosing a time starts nothing; the moment itself does, and only there.
JOB=$(occ background-job:list --class='OCA\NgBackup\BackgroundJob\BackupJob' --output=json | php -r 'echo json_decode(stream_get_contents(STDIN), true)[0]["id"];')
job() { occ background-job:execute "$JOB" --force-execute >/dev/null; }
occ backup:target:schedule ci off >/dev/null
occ backup:target:schedule ci3 off >/dev/null
occ backup:target:schedule ci2 00:00 --timezone=UTC >/dev/null || fail "set a schedule"
occ backup:target:schedule ci2 | grep "00:00 (UTC)" >/dev/null || fail "schedule not shown"
job
[ "$(occ backup:list ci2 | grep -c ' scheduled ')" = 0 ] || fail "choosing a time started a backup"
occ backup:target:schedule ci2 "$($X date -u -d '+1 minute' +%H:%M)/6" --timezone=UTC >/dev/null || fail "set every 6 hours"
sleep 65
job
occ backup:list ci2 | grep " scheduled " >/dev/null || fail "the scheduled moment started no backup"
[ "$(occ backup:list ci3 | grep -c ' scheduled ')" = 0 ] || fail "a location that is off was backed up"
job
[ "$(occ backup:list ci2 | grep -c ' scheduled ')" = 1 ] || fail "one moment started more than one backup"
occ backup:target:schedule ci2 nonsense >/dev/null 2>&1 && fail "a bad schedule was accepted"
ok "schedule per location: nothing on choosing a time, one backup at the moment, none when off"
# What administrators are told, as Nextcloud shows it.
notes() { api "http://localhost/ocs/v2.php/apps/notifications/api/v2/notifications?format=json"; }
occ backup:target:schedule ci2 off >/dev/null
$X sh -c "touch '$DATA/ngb-rt-bad\\name.txt'"
occ backup:run ci3 -l names | grep "their name is not supported" >/dev/null || fail "occ does not report the name"
$X sh -c "rm '$DATA/ngb-rt-bad\\name.txt'"
NOTES=$(notes) || fail "read the notifications"
echo "$NOTES" | grep "turned off the schedule of backup location ci2" >/dev/null || fail "no notification about the schedule that was turned off"
echo "$NOTES" | grep "is missing because of its name" >/dev/null || fail "no notification about the name"
echo "$NOTES" | grep "Rename them" >/dev/null || fail "the notification about the name does not ask to rename"
job
notes | grep "has no schedule" >/dev/null && fail "a reminder right after turning it off"
for L in 2 3; do occ backup:target:remove ci$L >/dev/null || fail "target remove ci$L"; done
occ user:delete "$ADM" >/dev/null
ok "notifications: schedule turned off, name not supported, no early reminder"
# Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob).
occ backup:verify ci "$SNAP" | grep "^OK$" >/dev/null || fail "shallow verify"
@@ -120,6 +216,17 @@ ok "database dump and restore"
# Retention, trash and catalog.
occ backup:retention --last=1 --daily=0 --weekly=0 --monthly=0 >/dev/null
# A pinned restore point stays, and a note can change afterwards.
FIRST=$(occ backup:list ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')
occ backup:pin ci "$FIRST" >/dev/null || fail "pin"
occ backup:note ci "$FIRST" "a note made later" >/dev/null || fail "note"
occ backup:list ci | grep " first " | grep " yes " | grep "a note made later" >/dev/null || fail "pin and note not listed"
occ backup:prune ci --dry-run | grep "would forget 1 " >/dev/null || fail "a pinned snapshot would be removed"
occ backup:pin ci "$FIRST" --unpin >/dev/null || fail "unpin"
occ backup:prune ci --dry-run | grep "would forget 2 " >/dev/null || fail "an unpinned snapshot would stay"
occ backup:note ci "$FIRST" >/dev/null || fail "remove note"
occ backup:list ci | grep -q "a note made later" && fail "note not removed"
ok "pin and note on a restore point"
occ backup:prune ci | grep "forgot 2" >/dev/null || fail "prune" # first and second; "page" (above) is the last
occ backup:trash ci | grep " first " >/dev/null || fail "trash"
TID=$(occ backup:trash ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')
+109
View File
@@ -0,0 +1,109 @@
#!/bin/bash
# The update from a released version to this working tree, on a server that was backing up.
# Usage: upgrade-from-release.sh <release package .tar.gz> [nextcloud-image-tag] [sqlite|pgsql] [new-version]
# e.g. upgrade-from-release.sh ~/dev-local/release/ng_backup-1.0.3/ng_backup-1.0.3.tar.gz 34-apache pgsql 1.1.0
#
# The old version gets a key, a confirmed kit, a location with a daily time and one backup. Then the
# app files are replaced by this working tree (version raised to new-version when given) and
# occ upgrade runs. After it: the location keeps its time as its own schedule, the old restore point
# is listed, verifies and restores a file byte for byte, a new backup runs, the location gets its
# settings file, and the key counts as used.
set -euo pipefail
PKG=${1:?release package}
IMAGE="nextcloud:${2:-34-apache}"
DB=${3:-sqlite}
NEWVER=${4:-}
APP=$(cd "$(dirname "$0")/../.." && pwd)
N=ngb-up; NET=ngb-up-net
SHARED=$(mktemp -d)
NEWAPP=$(mktemp -d)
chmod 777 "$SHARED"
PASS='upgrade test passphrase 2026'
fail() { echo "FAIL: $*"; exit 1; }
ok() { echo "ok $*"; }
cleanup() {
docker rm -fv $N $N-db >/dev/null 2>&1 || true
docker network rm $NET >/dev/null 2>&1 || true
docker run --rm -v "$SHARED:/target" alpine sh -c 'rm -rf /target/*' >/dev/null 2>&1 || true
rm -rf "$NEWAPP"
rmdir "$SHARED" 2>/dev/null || true
return 0
}
trap cleanup EXIT
occ() { docker exec -u www-data $N php occ "$@"; }
docker network create $NET >/dev/null
env=(-e NEXTCLOUD_ADMIN_USER=admin -e NEXTCLOUD_ADMIN_PASSWORD=admin-upgrade-pw)
case $DB in
sqlite) env+=(-e SQLITE_DATABASE=nextcloud) ;;
pgsql) docker run -d --name $N-db --network $NET -e POSTGRES_DB=nextcloud -e POSTGRES_USER=nc -e POSTGRES_PASSWORD=nc postgres:16-alpine >/dev/null
env+=(-e POSTGRES_HOST=$N-db -e POSTGRES_DB=nextcloud -e POSTGRES_USER=nc -e POSTGRES_PASSWORD=nc)
sleep 8 ;;
*) fail "unknown database $DB" ;;
esac
docker run -d --name $N --network $NET "${env[@]}" -v "$SHARED:/mnt/shared" "$IMAGE" >/dev/null
for i in $(seq 1 90); do occ status 2>/dev/null | grep -q "installed: true" && break; sleep 3; done
docker exec $N chown www-data:www-data /var/www/html/custom_apps
docker exec -i -u www-data $N tar xz -C /var/www/html/custom_apps < "$PKG"
OLDVER=$(docker exec $N sed -n 's#.*<version>\(.*\)</version>.*#\1#p' /var/www/html/custom_apps/ng_backup/appinfo/info.xml)
occ app:enable ng_backup >/dev/null || fail "enable $OLDVER"
ok "NG Backup $OLDVER on $(occ -V) / $DB"
# The old version at work: key, kit, a location with a daily time, a user file, one backup
docker exec $N sh -c "printf '%s\n' '$PASS' > /tmp/pass; chmod 644 /tmp/pass; mkdir -p /mnt/shared/loc; chown www-data /mnt/shared/loc"
occ backup:key:init --passphrase-file=/tmp/pass >/dev/null || fail "key init"
occ backup:key:kit --output=/tmp/kit.json >/dev/null
CODE=$(occ backup:key:kit | php -r 'echo json_decode(stream_get_contents(STDIN), true)["confirmation_code"];')
occ backup:key:confirm -u admin --code="$CODE" --confirm="Yes, I confirm" >/dev/null || fail "confirm"
occ config:app:set ng_backup schedule_time --value=03:15 >/dev/null
occ config:app:set ng_backup schedule_timezone --value=America/Toronto >/dev/null
occ backup:target:add offsite local -a null::null -o datadir=/mnt/shared/loc --path=repo >/dev/null || fail "target add"
docker exec -u www-data $N sh -c 'mkdir -p data/admin/files/Docs && head -c 3000000 /dev/urandom > data/admin/files/Docs/before.bin'
occ files:scan admin >/dev/null
SUM=$(docker exec $N sha256sum /var/www/html/data/admin/files/Docs/before.bin | cut -d' ' -f1)
occ backup:run offsite >/dev/null || fail "backup with $OLDVER"
OLDSNAP=$(occ backup:list offsite | grep -o '[0-9a-f]\{32\}' | head -1)
[ -n "$OLDSNAP" ] || fail "no restore point with $OLDVER"
ok "$OLDVER: key, kit, location at 03:15, restore point $OLDSNAP"
# The update: this working tree's runtime files in place of the old ones
mkdir -p "$NEWAPP/ng_backup/js"
cp -r "$APP/appinfo" "$APP/lib" "$APP/templates" "$APP/img" "$APP/css" "$NEWAPP/ng_backup/"
cp "$APP"/js/*.mjs "$NEWAPP/ng_backup/js/"
[ -n "$NEWVER" ] && sed -i "s#<version>.*</version>#<version>$NEWVER</version>#" "$NEWAPP/ng_backup/appinfo/info.xml"
docker exec -u www-data $N rm -rf /var/www/html/custom_apps/ng_backup
tar c -C "$NEWAPP" ng_backup | docker exec -i -u www-data $N tar x -C /var/www/html/custom_apps
OUT=$(occ upgrade 2>&1) || { printf '%s\n' "$OUT"; fail "occ upgrade"; }
VER=$(occ app:list --output=json | php -r '$a = json_decode(stream_get_contents(STDIN), true); echo $a["enabled"]["ng_backup"] ?? "";')
[ -z "$NEWVER" ] || [ "$VER" = "$NEWVER" ] || fail "after the upgrade the app has version '$VER'"
ok "occ upgrade to $VER"
# Before anything else touches the key: only the migration can have marked it as used
[ "$(occ config:app:get ng_backup master_key_used 2>/dev/null)" = 1 ] || [ "$(occ config:app:get ng_backup master_key_used 2>/dev/null)" = true ] || fail "the migration did not mark the key from before the upgrade as used"
ok "the key from before the upgrade counts as used"
SCHED=$(occ backup:target:schedule offsite)
printf '%s\n' "$SCHED" | grep -qx "offsite: 03:15 (America/Toronto)" || { printf '%s\n' "$SCHED"; fail "the location did not keep 03:15 as its schedule"; }
occ backup:list offsite | grep -q "$OLDSNAP" || fail "the old restore point is not listed"
occ backup:verify offsite "$OLDSNAP" >/dev/null || fail "the old restore point does not verify"
occ backup:status >/dev/null || fail "backup:status"
docker exec $N sh -c 'rm -rf /tmp/rst && mkdir /tmp/rst && chown www-data /tmp/rst'
occ backup:restore offsite "$OLDSNAP" data/admin/files/Docs/before.bin --to-directory=/tmp/rst >/dev/null || fail "restore from the old restore point"
GOT=$(docker exec $N sh -c 'find /tmp/rst -name before.bin -exec sha256sum {} \;' | cut -d' ' -f1)
[ "$GOT" = "$SUM" ] || fail "the restored file differs"
ok "after the upgrade: schedule kept, old restore point listed, verified, a file restored byte for byte"
docker exec -u www-data $N sh -c 'head -c 2000000 /dev/urandom > data/admin/files/Docs/after.bin'
occ files:scan admin >/dev/null
occ backup:run offsite >/dev/null || fail "backup after the upgrade"
NEWSNAP=$(occ backup:list offsite | grep -o '[0-9a-f]\{32\}' | grep -v "$OLDSNAP" | head -1)
[ -n "$NEWSNAP" ] || fail "no new restore point after the upgrade"
SUM2=$(docker exec $N sha256sum /var/www/html/data/admin/files/Docs/after.bin | cut -d' ' -f1)
docker exec $N sh -c 'rm -rf /tmp/rst && mkdir /tmp/rst && chown www-data /tmp/rst'
occ backup:restore offsite "$NEWSNAP" data/admin/files/Docs/after.bin --to-directory=/tmp/rst >/dev/null || fail "restore from the new restore point"
GOT=$(docker exec $N sh -c 'find /tmp/rst -name after.bin -exec sha256sum {} \;' | cut -d' ' -f1)
[ "$GOT" = "$SUM2" ] || fail "the file restored from the new restore point differs"
docker exec $N test -f /mnt/shared/loc/repo/settings || fail "the location got no settings file"
OUT=$(occ backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass --replace-unused-key 2>&1) && fail "a used key from before the upgrade was offered for replacement"
printf '%s\n' "$OUT" | grep -q "never replaced" || { printf '%s\n' "$OUT"; fail "the refusal does not say the key is never replaced"; }
ok "a new backup runs and restores, the location got its settings, the key counts as used"
echo "UPGRADE $OLDVER -> ${VER:-working tree} PASS"