Compare commits

..

6 Commits

Author SHA1 Message Date
Dillard Blom be3746e1c3 Merge pull request #26 from dillardblom/release-1.0.1
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.1
2026-11-01 12:58:34 +01:00
dillard f5473495ce App description and README: ask for a star in the app store and on GitHub
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-11-01 12:08:01 +01:00
dillard 3694184674 Version 1.0.1: changelog, README status and roadmap
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-11-01 12:03:19 +01:00
dillard a0fc91a700 Integration tests: grep the whole occ output instead of grep -q, which stopped reading and failed the pipe under pipefail
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-31 23:13:27 +01:00
dillard a11e6c7254 1.0.1: label for "Back up now", retention explained in the trash, occ times in the schedule's zone, single-file restore to a directory, leftovers of a stopped restore removed
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- Admin page: an optional label (at most 100 characters) for a backup started with "Back up now";
  an (i) by "removed by retention policy" in the trash that explains what it means and until when
  the restore point can be brought back. backup:trash says the same below its list.
- backup:status, backup:list, backup:trash and backup:browse show times in the zone of the daily
  schedule and name it, instead of the server's UTC.
- backup:restore --to-directory of a file restored nothing: the path got a "/" appended.
- A restore removes "<name>.ngb-restore-<hex>" temporaries that a stopped restore left in the
  folders it writes to.
- Roadmap: the cron warning (#13), reading ahead from S3/SMB and --salvage move to 1.0.2.
2026-10-30 19:32:53 +01:00
dillard 6f1a37f4f4 Changelog: release candidate headings carry their app version, so the app store no longer shows all of them under 1.0.0 2026-10-29 13:07:25 +01:00
19 changed files with 261 additions and 52 deletions
+39 -7
View File
@@ -7,14 +7,33 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
beta/daily update channel, which would hide the app from most admins. The stability stage is beta/daily update channel, which would hide the app from most admins. The stability stage is
instead carried by the app description, the git tag, the release, and this file. instead carried by the app description, the git tag, the release, and this file.
## [Unreleased] Each heading starts with that `<version>`, once: the app store shows a release the section(s)
whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
shown under 1.0.0. The git tag of a release candidate goes after the date.
## [1.0.1] - 2026-11-01
Fixes for what was reported after 1.0.0 (#22, #24) and small improvements. Nothing changes in
your backups or settings.
### Added ### Added
- `occ backup:user:list [location] [user]` shows the user exports on a location, with the manifest - `occ backup:user:list [location] [user]` shows the user exports on a location, with the manifest
path `backup:user:restore` expects. Before, that path was only printed once by path `backup:user:restore` expects. Before, that path was only printed once by
`backup:user:backup` (#22). `backup:user:backup` (#22).
- A "Docs" tab on the admin page, with links to the how-to, the disaster recovery guide and the
roadmap. It also shows before the backup key is set up.
- A label for a backup started with "Back up now" (optional; `occ backup:run -l` already had one).
It shows with the restore point.
- An explanation in the trash of restore points of what "removed by retention policy" means: the
policy keeps the newest restore points and the last one of each recent day, week and month, and
moves the others to the trash, where they can be brought back until they are deleted
permanently. `occ backup:trash` says the same below its list.
### Changed ### Changed
- `occ backup:status`, `backup:list`, `backup:trash` and `backup:browse` show times in the time
zone of the daily schedule and name it, as the admin page shows them in the administrator's.
Before, they showed the server's UTC, which read as a wrong start time.
- The admin page no longer says NG Backup is in beta.
- `occ backup:user:restore` also takes a user id instead of the manifest path, when that user has - `occ backup:user:restore` also takes a user id instead of the manifest path, when that user has
exactly one export on the location. With more, and for a manifest path that is not on the exactly one export on the location. With more, and for a manifest path that is not on the
location, the error lists that user's exports; for a user without exports it names the location, the error lists that user's exports; for a user without exports it names the
@@ -32,6 +51,13 @@ instead carried by the app description, the git tag, the release, and this file.
wrote to the database just as the restore began (a background job that was already running). wrote to the database just as the restore began (a background job that was already running).
The restore now takes the write lock first and waits for the other process instead. The The restore now takes the write lock first and waits for the other process instead. The
full-restore test on SQLite runs with a second process writing alongside. full-restore test on SQLite runs with a second process writing alongside.
- `occ backup:restore ... --to-directory` of a single file restored nothing ("Restored 0
files"); only a folder worked. A path to a file now restores that file.
- A restore that was stopped (a crash, a timeout, a restart) left its half-written temporary file
(`<name>.ngb-restore-<id>`) next to the file it was restoring, and a `files:scan` then showed it
in Files. A restore now removes such leftovers in each folder it writes to.
- The app store showed the notes of every release candidate under 1.0.0, because their headings
started with "1.0.0-rc". Each heading now starts with the app version of that release.
## [1.0.0] - 2026-10-29 ## [1.0.0] - 2026-10-29
@@ -43,7 +69,7 @@ from the web interface or with occ. Disaster recovery has been tested on real in
The code is the same as 1.0.0-rc.6 (app version 0.9.12): updating from it changes nothing in your The code is the same as 1.0.0-rc.6 (app version 0.9.12): updating from it changes nothing in your
backups or settings. backups or settings.
## [1.0.0-rc.6] - 2026-10-24 ## [0.9.12] - 2026-10-24 (1.0.0-rc.6)
Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery
simulations and a user report. simulations and a user report.
@@ -71,7 +97,7 @@ simulations and a user report.
- The full-restore integration test checks a 4-byte character in a table the restore creates, and - The full-restore integration test checks a 4-byte character in a table the restore creates, and
retries app installs from the app store. retries app installs from the app store.
## [1.0.0-rc.5] - 2026-10-22 ## [0.9.11] - 2026-10-22 (1.0.0-rc.5)
Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
stopped after 13 hours with "Blob not in index"; the backup itself was fine. stopped after 13 hours with "Blob not in index"; the backup itself was fine.
@@ -104,7 +130,7 @@ stopped after 13 hours with "Blob not in index"; the backup itself was fine.
- The extra connection a full restore reads ahead on is replaced when it breaks (up to 20 times), - The extra connection a full restore reads ahead on is replaced when it breaks (up to 20 times),
instead of reading the rest of the restore the slow way. instead of reading the rest of the restore the slow way.
## [1.0.0-rc.4] - 2026-10-19 ## [0.9.10] - 2026-10-19 (1.0.0-rc.4)
Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not
released: its fixes are in here. released: its fixes are in here.
@@ -128,7 +154,7 @@ released: its fixes are in here.
- The app store showed old screenshots: it keeps a copy per URL, and the URLs always pointed at - The app store showed old screenshots: it keeps a copy per URL, and the URLs always pointed at
`main`. They now point at the release's tag. `main`. They now point at the release's tag.
## [1.0.0-rc.2] - 2026-10-18 ## [0.9.8] - 2026-10-18 (1.0.0-rc.2)
Second release candidate for 1.0.0; the app version is 0.9.8. Fixes from a full disaster recovery Second release candidate for 1.0.0; the app version is 0.9.8. Fixes from a full disaster recovery
test of a production instance (about 1 TB, Nextcloud All-in-One onto a plain Nextcloud with test of a production instance (about 1 TB, Nextcloud All-in-One onto a plain Nextcloud with
@@ -169,7 +195,7 @@ PostgreSQL).
- Removing a location left its cached index in the database (tens of thousands of rows for a large - Removing a location left its cached index in the database (tens of thousands of rows for a large
backup). It is removed with the location now, and once for locations that are already gone. backup). It is removed with the location now, and once for locations that are already gone.
## [1.0.0-rc.1] - 2026-10-17 ## [0.9.7] - 2026-10-17 (1.0.0-rc.1)
Release candidate for 1.0.0; the app version is 0.9.7. Only bug fixes until 1.0.0. Tested on a Release candidate for 1.0.0; the app version is 0.9.7. Only bug fixes until 1.0.0. Tested on a
production instance (about 1 TB, 150,000 files): full and incremental backups and a verify on production instance (about 1 TB, 150,000 files): full and incremental backups and a verify on
@@ -448,7 +474,13 @@ and not implemented yet.
- See the "Storage backend experiences" section of the README for provider-specific quirks found - See the "Storage backend experiences" section of the README for provider-specific quirks found
so far (e.g. Hetzner Storage Box sub-accounts). so far (e.g. Hetzner Storage Box sub-accounts).
[1.0.0-rc.1]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.1 [1.0.1]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.1
[1.0.0]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0
[0.9.12]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.6
[0.9.11]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.5
[0.9.10]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.4
[0.9.8]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.2
[0.9.7]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.1
[0.9.6-beta.7]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.6-beta.7 [0.9.6-beta.7]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.6-beta.7
[0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6 [0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6
[0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5 [0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5
+15 -9
View File
@@ -392,10 +392,12 @@ in README "Known limitations":
gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een
productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0. productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0.
Problemen of meldingen van gebruikers: rc5. Problemen of meldingen van gebruikers: rc5.
- **1.0.1:** - **1.0.1** (klaar voor release 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een 1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
niet aangekondigd en los van de release.
1. **Naar 1.0.2.** (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
restore leest de packs nog één voor één: ook vooruitlezen via de S3-client. restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
2. Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit 2. **Naar 1.0.2.** Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit
oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21): oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21):
a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode` a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode`
(`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de (`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de
@@ -424,25 +426,29 @@ in README "Known limitations":
4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de 4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de
tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop
stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een
bestaande parameter. bestaande parameter. Ten hoogste 100 tekens — **klaar** (2026-10-30).
5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen 5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen
wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind
opsommen, in plaats van stoppen. opsommen, in plaats van stoppen. **Naar 1.0.2** (besluit 2026-10-30).
6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van 6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van
rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-<id>` staan; de rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-<id>` staan; de
geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files. geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files.
Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op. Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op (alleen bestanden met
precies dat achtervoegsel en 6 of 12 hextekens) — **klaar** (2026-10-30).
7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden 7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden
terug (er komt een `/` achter); nu werkt alleen een map. terug (er komt een `/` achter); nu werkt alleen een map. Gold alleen voor `--to-directory`;
naar de bestanden van een gebruiker werkte een los bestand al — **klaar** (2026-10-30).
8. **Uitleg (i) bij de prullenbak van herstelpunten** (besluit 2026-10-25): bij "removed by 8. **Uitleg (i) bij de prullenbak van herstelpunten** (besluit 2026-10-25): bij "removed by
retention policy" een infoknop met wat het betekent: het bewaarbeleid houdt voor oudere dagen retention policy" een infoknop met wat het betekent: het bewaarbeleid houdt voor oudere dagen
één herstelpunt per dag (het laatste), weken en maanden; de rest gaat naar de prullenbak en één herstelpunt per dag (het laatste), weken en maanden; de rest gaat naar de prullenbak en
wordt pas na de ingestelde wachttijd (`delete_delay_days`, standaard 30) echt verwijderd, en is wordt pas na de ingestelde wachttijd (`delete_delay_days`, standaard 30) echt verwijderd, en is
tot die datum terug te zetten. Aanleiding: de beheerder zag "19-10 13:12 – after rc4 upgrade tot die datum terug te zetten. Aanleiding: de beheerder zag "19-10 13:12 – after rc4 upgrade
removed by retention policy" en dacht dat er iets was weggegooid. removed by retention policy" en dacht dat er iets was weggegooid. `occ backup:trash` zegt
hetzelfde onder de lijst — **klaar** (2026-10-30).
9. **Tijdzone in `occ backup:status`**: de tabel toont de servertijd (UTC), de webinterface de 9. **Tijdzone in `occ backup:status`**: de tabel toont de servertijd (UTC), de webinterface de
tijd van de beheerder; dat leest als een verkeerde starttijd. Zet de zone in de kolomkop tijd van de beheerder; dat leest als een verkeerde starttijd. Zet de zone in de kolomkop
("started (UTC)") of toon de tijden in de zone van het schema. ("started (UTC)") of toon de tijden in de zone van het schema. Gedaan: beide, en ook in
`backup:list`, `backup:trash` en `backup:browse` — **klaar** (2026-10-30).
10. **User-restore op gebruikersnaam** (issue #22, 2026-10-27): `backup:user:restore <locatie> 10. **User-restore op gebruikersnaam** (issue #22, 2026-10-27): `backup:user:restore <locatie>
New_User` gaf alleen "is not a recorded user export"; het argument moet het manifestpad New_User` gaf alleen "is not a recorded user export"; het argument moet het manifestpad
zijn, dat alleen `backup:user:backup` eenmalig toonde. Nu: `backup:user:list` toont de zijn, dat alleen `backup:user:backup` eenmalig toonde. Nu: `backup:user:list` toont de
+9 -8
View File
@@ -7,11 +7,14 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
see "Storage backend experiences" below — that requirement comes from files_external, not from see "Storage backend experiences" below — that requirement comes from files_external, not from
ng_backup, and applies to any Nextcloud app that writes to an SMB share.) ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
**Status: 1.0.0, stable.** Disaster recovery has been tested on real installations (see "Tested **Status: 1.0.1, stable.** Disaster recovery has been tested on real installations (see "Tested
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See `docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
`docs/DISASTER-RECOVERY.md`. `docs/DISASTER-RECOVERY.md`.
If you like NG Backup, don't forget to give the app a star in the
[app store](https://apps.nextcloud.com/apps/ng_backup) and this repository a star on GitHub!
License: AGPL-3.0-or-later. Developed and sponsored by [Ensembia](https://www.ensembia.com). License: AGPL-3.0-or-later. Developed and sponsored by [Ensembia](https://www.ensembia.com).
Need help restoring a backup after a crash? Ensembia offers paid restore assistance: Need help restoring a backup after a crash? Ensembia offers paid restore assistance:
[www.ensembia.com](https://www.ensembia.com). [www.ensembia.com](https://www.ensembia.com).
@@ -260,16 +263,14 @@ What is planned, without dates. Ideas and votes are welcome in the
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here, [issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
a 👍 on its issue helps decide what comes first. a 👍 on its issue helps decide what comes first.
**1.0.1** **1.0.2**
- A warning on the admin page and in `occ backup:status` when background jobs do not run on a - A warning on the admin page and in `occ backup:status` when background jobs do not run on a
system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on
while the NG Backup page is open ([#13](https://github.com/dillardblom/ng_backup/issues/13)). while the NG Backup page is open, and the database phase of a backup shows its progress
- An optional label for a backup started with "Back up now". ([#13](https://github.com/dillardblom/ng_backup/issues/13)).
- An explanation in the trash of restore points of what "removed by retention policy" means and
until when a restore point can be brought back.
- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4. - Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4.
- A "Docs" tab on the admin page with links to the how-to, the disaster recovery guide and this - `--salvage` for a restore: restore what can be read, skip a file that cannot, and list those at
roadmap. the end instead of stopping.
**1.1.0** **1.1.0**
- Restore without the recovery kit: the backup key from the location itself, opened with one of - Restore without the recovery kit: the backup key from the location itself, opened with one of
+7 -4
View File
@@ -40,9 +40,12 @@ NG Backup needs Nextcloud's background jobs on system cron (a real cron line, no
Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very
welcome: use the "Share your experience" link on the admin page. welcome: use the "Share your experience" link on the admin page.
If you like NG Backup, don't forget to give the app a star in the app store and the
[repository](https://github.com/dillardblom/ng_backup) a star on GitHub!
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
you need help with restoring a backup after a crash.]]></description> you need help with restoring a backup after a crash.]]></description>
<version>1.0.0</version> <version>1.0.1</version>
<licence>agpl</licence> <licence>agpl</licence>
<author mail="dillard.blom@ensembia.com">Dillard Blom</author> <author mail="dillard.blom@ensembia.com">Dillard Blom</author>
<namespace>NgBackup</namespace> <namespace>NgBackup</namespace>
@@ -53,9 +56,9 @@ you need help with restoring a backup after a crash.]]></description>
<website>https://github.com/dillardblom/ng_backup</website> <website>https://github.com/dillardblom/ng_backup</website>
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs> <bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
<repository type="git">https://github.com/dillardblom/ng_backup</repository> <repository type="git">https://github.com/dillardblom/ng_backup</repository>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0/screenshots/1-backups.png</screenshot> <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0/screenshots/2-restore.png</screenshot> <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0/screenshots/3-settings.png</screenshot> <screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/3-settings.png</screenshot>
<dependencies> <dependencies>
<php min-version="8.1"/> <php min-version="8.1"/>
<lib>sodium</lib> <lib>sodium</lib>
+6 -2
View File
@@ -7,7 +7,9 @@ declare(strict_types=1);
namespace OCA\NgBackup\Command; namespace OCA\NgBackup\Command;
use OCA\NgBackup\Format;
use OCA\NgBackup\Service\RestoreService; use OCA\NgBackup\Service\RestoreService;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService; use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\Table; use Symfony\Component\Console\Helper\Table;
@@ -19,6 +21,7 @@ final class Browse extends Command {
public function __construct( public function __construct(
private TargetService $targets, private TargetService $targets,
private RestoreService $restore, private RestoreService $restore,
private Schedule $schedule,
) { ) {
parent::__construct(); parent::__construct();
} }
@@ -43,9 +46,10 @@ final class Browse extends Command {
$output->writeln('Nothing here.'); $output->writeln('Nothing here.');
return 0; return 0;
} }
$zone = $this->schedule->get()['timezone'];
$rows = array_map(fn ($i) => [$i['type'] === 'dir' ? $i['name'] . '/' : $i['name'], $this->size($i['size']), $rows = array_map(fn ($i) => [$i['type'] === 'dir' ? $i['name'] . '/' : $i['name'], $this->size($i['size']),
$i['type'] === 'dir' ? $i['files'] . ' files' : date('Y-m-d H:i', (int)$i['mtime'])], $items); $i['type'] === 'dir' ? $i['files'] . ' files' : Format::time((int)$i['mtime'], $zone)], $items);
(new Table($output))->setHeaders(['name', 'size', 'files / modified'])->setRows($rows)->render(); (new Table($output))->setHeaders(['name', 'size', "files / modified ($zone)"])->setRows($rows)->render();
return 0; return 0;
} }
+6 -2
View File
@@ -8,6 +8,8 @@ declare(strict_types=1);
namespace OCA\NgBackup\Command; namespace OCA\NgBackup\Command;
use OCA\NgBackup\Db\SnapshotMapper; use OCA\NgBackup\Db\SnapshotMapper;
use OCA\NgBackup\Format;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService; use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\Table; use Symfony\Component\Console\Helper\Table;
@@ -19,6 +21,7 @@ final class ListSnapshots extends Command {
public function __construct( public function __construct(
private TargetService $targets, private TargetService $targets,
private SnapshotMapper $snapshots, private SnapshotMapper $snapshots,
private Schedule $schedule,
) { ) {
parent::__construct(); parent::__construct();
} }
@@ -32,10 +35,11 @@ final class ListSnapshots extends Command {
protected function execute(InputInterface $input, OutputInterface $output): int { protected function execute(InputInterface $input, OutputInterface $output): int {
$name = $input->getArgument('target'); $name = $input->getArgument('target');
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)]; $targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
$zone = $this->schedule->get()['timezone'];
$rows = []; $rows = [];
foreach ($targets as $t) { foreach ($targets as $t) {
foreach ($this->snapshots->findForTarget($t->getId()) as $s) { foreach ($this->snapshots->findForTarget($t->getId()) as $s) {
$rows[] = [$t->getName(), $s->getSnapshotId(), date('Y-m-d H:i', $s->getCreatedAt()), $s->getKind(), $rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(),
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getLabel() ?? '']; $s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getLabel() ?? ''];
} }
} }
@@ -43,7 +47,7 @@ final class ListSnapshots extends Command {
$output->writeln('No snapshots yet (occ backup:run).'); $output->writeln('No snapshots yet (occ backup:run).');
return 0; return 0;
} }
(new Table($output))->setHeaders(['location', 'snapshot', 'date', 'kind', 'files', 'size', 'label'])->setRows($rows)->render(); (new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'label'])->setRows($rows)->render();
return 0; return 0;
} }
} }
+7 -3
View File
@@ -8,7 +8,9 @@ declare(strict_types=1);
namespace OCA\NgBackup\Command; namespace OCA\NgBackup\Command;
use OCA\NgBackup\Db\RunMapper; use OCA\NgBackup\Db\RunMapper;
use OCA\NgBackup\Format;
use OCA\NgBackup\Service\KeyService; use OCA\NgBackup\Service\KeyService;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService; use OCA\NgBackup\Service\TargetService;
use OCA\NgBackup\Service\VerifyService; use OCA\NgBackup\Service\VerifyService;
use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Command\Command;
@@ -26,6 +28,7 @@ final class Status extends Command {
private TargetService $targets, private TargetService $targets,
private RunMapper $runs, private RunMapper $runs,
private VerifyService $verify, private VerifyService $verify,
private Schedule $schedule,
) { ) {
parent::__construct(); parent::__construct();
} }
@@ -36,12 +39,13 @@ final class Status extends Command {
} }
protected function execute(InputInterface $input, OutputInterface $output): int { protected function execute(InputInterface $input, OutputInterface $output): int {
$zone = $this->schedule->get()['timezone'];
if (!$this->keys->isInitialized()) { if (!$this->keys->isInitialized()) {
$output->writeln('Key: <error>not created</error> (occ backup:key:init)'); $output->writeln('Key: <error>not created</error> (occ backup:key:init)');
} else { } else {
$c = $this->keys->recoveryKitConfirmation(); $c = $this->keys->recoveryKitConfirmation();
$output->writeln('Key: ' . $this->keys->fingerprint() . ', recovery kit ' $output->writeln('Key: ' . $this->keys->fingerprint() . ', recovery kit '
. ($c ? 'confirmed by ' . $c['uid'] . ' on ' . date('Y-m-d H:i', $c['time']) : '<error>not confirmed</error> (occ backup:key:confirm)')); . ($c ? 'confirmed by ' . $c['uid'] . ' on ' . Format::time($c['time'], $zone) . " ($zone)" : '<error>not confirmed</error> (occ backup:key:confirm)'));
} }
$names = []; $names = [];
foreach ($this->targets->list() as $t) { foreach ($this->targets->list() as $t) {
@@ -54,12 +58,12 @@ final class Status extends Command {
foreach ($shown as $r) { foreach ($shown as $r) {
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? []; $s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''), $rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''),
date('Y-m-d H:i', $r->getStartedAt()), $r->getFinishedAt() ? \OCA\NgBackup\Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '', Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '', $s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')]; isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
} }
if ($rows !== []) { if ($rows !== []) {
(new Table($output))->setHeaders(['run', 'location', 'kind', 'status', 'started', 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render(); (new Table($output))->setHeaders(['run', 'location', 'kind', 'status', "started ($zone)", 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render();
} }
$older = $all ? 0 : $this->runs->countAll() - count($shown); $older = $all ? 0 : $this->runs->countAll() - count($shown);
if ($older > 0) { if ($older > 0) {
+9 -2
View File
@@ -7,7 +7,9 @@ declare(strict_types=1);
namespace OCA\NgBackup\Command; namespace OCA\NgBackup\Command;
use OCA\NgBackup\Format;
use OCA\NgBackup\Service\KeyService; use OCA\NgBackup\Service\KeyService;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService; use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\Table; use Symfony\Component\Console\Helper\Table;
@@ -19,6 +21,7 @@ final class Trash extends Command {
public function __construct( public function __construct(
private TargetService $targets, private TargetService $targets,
private KeyService $keys, private KeyService $keys,
private Schedule $schedule,
) { ) {
parent::__construct(); parent::__construct();
} }
@@ -32,11 +35,14 @@ final class Trash extends Command {
protected function execute(InputInterface $input, OutputInterface $output): int { protected function execute(InputInterface $input, OutputInterface $output): int {
$name = $input->getArgument('target'); $name = $input->getArgument('target');
$delay = $this->keys->deleteDelayDays() * 86400; $delay = $this->keys->deleteDelayDays() * 86400;
$zone = $this->schedule->get()['timezone'];
$rows = []; $rows = [];
foreach ($name === null ? $this->targets->list() : [$this->targets->get($name)] as $t) { foreach ($name === null ? $this->targets->list() : [$this->targets->get($name)] as $t) {
foreach ($this->targets->repository($t)->trash() as $id => $info) { foreach ($this->targets->repository($t)->trash() as $id => $info) {
$rows[] = [$t->getName(), $id, (string)($info['time'] ?? ''), $info['label'], date('Y-m-d H:i', $info['forgottenAt']), $info['by'], $time = $info['time'] ?? null;
date('Y-m-d H:i', $info['forgottenAt'] + $delay)]; $made = is_string($time) ? strtotime($time) : false;
$rows[] = [$t->getName(), $id, $made !== false ? Format::time($made, $zone) : '', $info['label'], Format::time($info['forgottenAt'], $zone), $info['by'],
Format::time($info['forgottenAt'] + $delay, $zone)];
} }
} }
if ($rows === []) { if ($rows === []) {
@@ -44,6 +50,7 @@ final class Trash extends Command {
return 0; return 0;
} }
(new Table($output))->setHeaders(['location', 'snapshot', 'made', 'label', 'removed', 'by', 'deleted after'])->setRows($rows)->render(); (new Table($output))->setHeaders(['location', 'snapshot', 'made', 'label', 'removed', 'by', 'deleted after'])->setRows($rows)->render();
$output->writeln("Times in $zone. The retention policy keeps the newest snapshots and the last of each recent day, week and month; it moves the others here, and they are deleted permanently after " . $this->keys->deleteDelayDays() . ' days.');
return 0; return 0;
} }
} }
+10
View File
@@ -15,4 +15,14 @@ final class Format {
$time = gmdate('H:i:s', $seconds % 86400); $time = gmdate('H:i:s', $seconds % 86400);
return $days > 0 ? $days . 'd ' . $time : $time; return $days > 0 ? $days . 'd ' . $time : $time;
} }
/**
* "2026-10-29 04:30" in $zone. Nextcloud runs PHP in UTC; occ shows times in the zone of the
* schedule, the administrator's, as the web interface does.
*
* @param non-empty-string $zone
*/
public static function time(int $timestamp, string $zone): string {
return (new \DateTimeImmutable('@' . $timestamp))->setTimezone(new \DateTimeZone($zone))->format('Y-m-d H:i');
}
} }
+37 -4
View File
@@ -214,7 +214,8 @@ final class Repository {
} }
/** /**
* Restore a snapshot (or only paths starting with $prefix) into a local directory. * Restore a snapshot (or only paths starting with $prefix) into a local directory. A $prefix
* "dir/file/" also restores the file "dir/file" itself.
* *
* @param bool $stripPrefix write "$target/<path without $prefix>" instead of * @param bool $stripPrefix write "$target/<path without $prefix>" instead of
* "$target/<full logical path>" (disaster recovery: restoring e.g. "data/" or * "$target/<full logical path>" (disaster recovery: restoring e.g. "data/" or
@@ -253,7 +254,7 @@ final class Repository {
$plan = new ReadAheadPlan(rtrim($target, '/') . '/.ngb-restore-spool-' . bin2hex(random_bytes(4))); $plan = new ReadAheadPlan(rtrim($target, '/') . '/.ngb-restore-spool-' . bin2hex(random_bytes(4)));
try { try {
foreach ($this->streamEntries($snapshotId) as $entry) { foreach ($this->streamEntries($snapshotId) as $entry) {
if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) { if (!self::underPrefix($entry['p'], $prefix)) {
continue; continue;
} }
foreach ($entry['b'] as $id) { foreach ($entry['b'] as $id) {
@@ -286,11 +287,15 @@ final class Repository {
private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan): int { private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan): int {
$count = 0; $count = 0;
$cleaned = [];
foreach ($this->streamEntries($snapshotId) as $entry) { foreach ($this->streamEntries($snapshotId) as $entry) {
if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) { if (!self::underPrefix($entry['p'], $prefix)) {
continue; continue;
} }
$relPath = ($stripPrefix && $prefix !== '') ? substr($entry['p'], strlen(rtrim($prefix, '/')) + 1) : $entry['p']; $relPath = $entry['p'];
if ($stripPrefix && $prefix !== '') {
$relPath = $entry['p'] === rtrim($prefix, '/') ? basename($entry['p']) : substr($entry['p'], strlen($prefix));
}
$dest = rtrim($target, '/') . '/' . $relPath; $dest = rtrim($target, '/') . '/' . $relPath;
if (!is_dir(dirname($dest))) { if (!is_dir(dirname($dest))) {
mkdir(dirname($dest), 0700, true); mkdir(dirname($dest), 0700, true);
@@ -301,6 +306,10 @@ final class Repository {
if ($base === false || $dir === false || ($dir !== $base && !str_starts_with($dir, $base . '/')) || is_link($dest)) { if ($base === false || $dir === false || ($dir !== $base && !str_starts_with($dir, $base . '/')) || is_link($dest)) {
throw new RepositoryException('Refusing to write outside the restore directory: ' . $entry['p']); throw new RepositoryException('Refusing to write outside the restore directory: ' . $entry['p']);
} }
if (!isset($cleaned[$dir])) {
self::removeStaleTemporaries($dir);
$cleaned[$dir] = true;
}
// Write next to the destination and rename after all blobs verified: at most one file // Write next to the destination and rename after all blobs verified: at most one file
// is ever present twice. // is ever present twice.
$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6)); $tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
@@ -331,6 +340,30 @@ final class Repository {
return $count; return $count;
} }
/** $prefix is "" (everything) or ends with "/"; the path without that "/" is a file to restore too. */
private static function underPrefix(string $path, string $prefix): bool {
return $prefix === '' || str_starts_with($path, $prefix) || $path === rtrim($prefix, '/');
}
/**
* Remove what a restore that was stopped left in a directory this restore writes to:
* "<name>.ngb-restore-<hex>" from here, ".<name>.ngb-restore-<hex>" from a restore through
* Nextcloud. A files:scan would otherwise make them show up in Files.
*/
public static function removeStaleTemporaries(string $dir): void {
$names = scandir($dir);
foreach ($names === false ? [] : $names as $name) {
$path = $dir . '/' . $name;
if (self::isRestoreTemporary($name) && is_file($path) && !is_link($path)) {
@unlink($path);
}
}
}
public static function isRestoreTemporary(string $name): bool {
return preg_match('/.\.ngb-restore-[0-9a-f]{6}(?:[0-9a-f]{6})?$/', $name) === 1;
}
/** /**
* Start writing a generated stream (e.g. a table dump) as blobs. Call flushPacks() when the * Start writing a generated stream (e.g. a table dump) as blobs. Call flushPacks() when the
* run is done, so the last pack is uploaded before anything references it. * run is done, so the last pack is uploaded before anything references it.
+6
View File
@@ -34,6 +34,8 @@ use Psr\Log\LoggerInterface;
*/ */
final class BackupService { final class BackupService {
public const KIND_FULL = 'full'; public const KIND_FULL = 'full';
/** Characters of a snapshot label (the column holds 255) */
public const LABEL_MAX = 100;
/** @var \WeakMap<Repository, string> the location settings each kept repository was opened with */ /** @var \WeakMap<Repository, string> the location settings each kept repository was opened with */
private \WeakMap $openedWith; private \WeakMap $openedWith;
@@ -57,6 +59,10 @@ final class BackupService {
} }
public function start(Target $target, string $label = ''): Run { public function start(Target $target, string $label = ''): Run {
$label = trim($label);
if (mb_strlen($label) > self::LABEL_MAX) {
throw new \InvalidArgumentException('A label has at most ' . self::LABEL_MAX . ' characters');
}
if (!$this->keys->everConfirmed()) { if (!$this->keys->everConfirmed()) {
throw new \RuntimeException('Confirm the recovery kit first (occ backup:key:confirm)'); throw new \RuntimeException('Confirm the recovery kit first (occ backup:key:confirm)');
} }
+21
View File
@@ -33,6 +33,9 @@ final class RestoreService {
public const MODE_MERGE = 'merge'; public const MODE_MERGE = 'merge';
public const MODE_REPLACE = 'replace'; public const MODE_REPLACE = 'replace';
/** @var array<string, true> folders of the current restore already cleared of stale temporaries */
private array $cleaned = [];
public function __construct( public function __construct(
private TargetService $targets, private TargetService $targets,
private IRootFolder $root, private IRootFolder $root,
@@ -133,6 +136,7 @@ final class RestoreService {
$dest = $this->ensureFolder($userFolder, $parentRel); $dest = $this->ensureFolder($userFolder, $parentRel);
$names = $single ? [basename($rel) => $entries['']] : $entries; $names = $single ? [basename($rel) => $entries['']] : $entries;
$this->cleaned = [];
$restored = 0; $restored = 0;
foreach ($names as $relPath => $entry) { foreach ($names as $relPath => $entry) {
$this->writeFile($repo, $dest, (string)$relPath, $entry); $this->writeFile($repo, $dest, (string)$relPath, $entry);
@@ -178,6 +182,10 @@ final class RestoreService {
private function writeFile(Repository $repo, Folder $dest, string $relPath, array $entry): void { private function writeFile(Repository $repo, Folder $dest, string $relPath, array $entry): void {
$dir = dirname($relPath); $dir = dirname($relPath);
$folder = $dir === '.' ? $dest : $this->ensureFolder($dest, $dir); $folder = $dir === '.' ? $dest : $this->ensureFolder($dest, $dir);
if (!isset($this->cleaned[$folder->getPath()])) {
$this->removeStaleTemporaries($folder);
$this->cleaned[$folder->getPath()] = true;
}
$name = basename($relPath); $name = basename($relPath);
// Stage under a temporary name; only rename into place after every blob was verified. // Stage under a temporary name; only rename into place after every blob was verified.
$tmpName = '.' . $name . '.ngb-restore-' . bin2hex(random_bytes(3)); $tmpName = '.' . $name . '.ngb-restore-' . bin2hex(random_bytes(3));
@@ -223,6 +231,19 @@ final class RestoreService {
} }
} }
/** See Repository::removeStaleTemporaries(): what a restore that was stopped left in this folder. */
private function removeStaleTemporaries(Folder $folder): void {
foreach ($folder->getDirectoryListing() as $node) {
if ($node instanceof File && Repository::isRestoreTemporary($node->getName())) {
try {
$node->delete();
} catch (\Throwable) {
// in the way of nothing: the new temporary gets a name of its own
}
}
}
}
private function trashExtras(Folder $folder, string $prefix, array $keep): int { private function trashExtras(Folder $folder, string $prefix, array $keep): int {
$count = 0; $count = 0;
foreach ($folder->getDirectoryListing() as $node) { foreach ($folder->getDirectoryListing() as $node) {
-5
View File
@@ -110,11 +110,6 @@
<code><![CDATA[ask]]></code> <code><![CDATA[ask]]></code>
</UndefinedInterfaceMethod> </UndefinedInterfaceMethod>
</file> </file>
<file src="lib/Command/Trash.php">
<RedundantCastGivenDocblockType>
<code><![CDATA[(string)($info['time'] ?? '')]]></code>
</RedundantCastGivenDocblockType>
</file>
<file src="lib/Controller/AdminApiController.php"> <file src="lib/Controller/AdminApiController.php">
<InvalidOperand> <InvalidOperand>
<code><![CDATA[$maxGb * 1073741824]]></code> <code><![CDATA[$maxGb * 1073741824]]></code>
+24 -1
View File
@@ -10,6 +10,8 @@
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton> <NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
</div> </div>
<div class="ngb-row"> <div class="ngb-row">
<NcTextField v-model="label" class="ngb-label" :maxlength="100" :label="t('ng_backup', 'Label (optional)')"
:placeholder="t('ng_backup', 'e.g. before the upgrade to Nextcloud 35')" />
<NcButton v-for="target in status.targets" :key="target.id" :disabled="isRunning(target.id)" @click="start(target)"> <NcButton v-for="target in status.targets" :key="target.id" :disabled="isRunning(target.id)" @click="start(target)">
{{ isRunning(target.id) ? t('ng_backup', 'Backup to {name} running …', { name: target.name }) : t('ng_backup', 'Back up now to {name}', { name: target.name }) }} {{ isRunning(target.id) ? t('ng_backup', 'Backup to {name} running …', { name: target.name }) : t('ng_backup', 'Back up now to {name}', { name: target.name }) }}
</NcButton> </NcButton>
@@ -24,9 +26,23 @@
<li v-for="item in trash[target.id]" :key="item.id"> <li v-for="item in trash[target.id]" :key="item.id">
{{ item.time ? formatDateTime(item.time) : item.id }}{{ item.label ? ' – ' + item.label : '' }} {{ item.time ? formatDateTime(item.time) : item.id }}{{ item.label ? ' – ' + item.label : '' }}
<span class="ngb-muted">{{ t('ng_backup', 'removed by {by}, deleted after {date}', { by: item.by, date: formatDate(item.deleteAfter) }) }}</span> <span class="ngb-muted">{{ t('ng_backup', 'removed by {by}, deleted after {date}', { by: item.by, date: formatDate(item.deleteAfter) }) }}</span>
<button v-if="item.by === RETENTION"
class="ngb-info"
:aria-expanded="retentionInfo ? 'true' : 'false'"
:aria-label="t('ng_backup', 'What does this mean?')"
:title="t('ng_backup', 'What does this mean?')"
@click="retentionInfo = !retentionInfo">
ⓘ
</button>
<NcButton variant="tertiary" @click="untrash(target, item)">{{ t('ng_backup', 'Restore') }}</NcButton> <NcButton variant="tertiary" @click="untrash(target, item)">{{ t('ng_backup', 'Restore') }}</NcButton>
</li> </li>
</ul> </ul>
<NcNoteCard v-if="retentionInfo && trash[target.id].some(item => item.by === RETENTION)" type="info">
{{ n('ng_backup',
'The retention policy keeps the newest restore points, and for older days, weeks and months only the last restore point of each. It moves the others to this trash, where they stay for %n day before they are deleted permanently. Until then you can bring one back with "Restore". Nothing was lost by accident.',
'The retention policy keeps the newest restore points, and for older days, weeks and months only the last restore point of each. It moves the others to this trash, where they stay for %n days before they are deleted permanently. Until then you can bring one back with "Restore". Nothing was lost by accident.',
status.key.deleteDelayDays) }}
</NcNoteCard>
</div> </div>
</template> </template>
@@ -93,6 +109,10 @@ const schedule = ref(shown(props.status.schedule))
const error = ref('') const error = ref('')
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) }) watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
const label = ref('')
// "by" of a restore point the weekly cleanup removed (PruneService)
const RETENTION = 'retention policy'
const retentionInfo = ref(false)
const trash = reactive({}) const trash = reactive({})
async function loadTrash() { async function loadTrash() {
for (const target of props.status.targets) { for (const target of props.status.targets) {
@@ -125,7 +145,8 @@ const targetName = (id) => props.status.targets.find(tg => tg.id === id)?.name ?
async function start(target) { async function start(target) {
error.value = '' error.value = ''
try { try {
await api.startBackup(target.id, '') await api.startBackup(target.id, label.value.trim())
label.value = ''
emit('changed') emit('changed')
} catch (e) { } catch (e) {
error.value = e.message error.value = e.message
@@ -146,6 +167,8 @@ async function saveSchedule() {
<style scoped> <style scoped>
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; } .ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-time { max-width: 260px; } .ngb-time { max-width: 260px; }
.ngb-label { max-width: 360px; }
.ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); }
.ngb-table-wrap { overflow-x: auto; } .ngb-table-wrap { overflow-x: auto; }
.ngb-table { width: 100%; border-collapse: collapse; } .ngb-table { width: 100%; border-collapse: collapse; }
.ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); } .ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); }
+7
View File
@@ -14,4 +14,11 @@ class FormatTest extends TestCase {
$this->assertSame('1d 00:00:00', Format::duration(86400)); $this->assertSame('1d 00:00:00', Format::duration(86400));
$this->assertSame('1d 12:10:13', Format::duration(86400 + 12 * 3600 + 10 * 60 + 13)); $this->assertSame('1d 12:10:13', Format::duration(86400 + 12 * 3600 + 10 * 60 + 13));
} }
public function testTimeInZone(): void {
$t = (int)strtotime('2026-10-19 13:12:00 UTC');
$this->assertSame('2026-10-19 13:12', Format::time($t, 'UTC'));
$this->assertSame('2026-10-19 15:12', Format::time($t, 'Europe/Berlin')); // summer time
$this->assertSame('2026-12-01 01:00', Format::time((int)strtotime('2026-12-01 00:00:00 UTC'), 'Europe/Amsterdam'));
}
} }
+40
View File
@@ -87,6 +87,46 @@ class RepositoryTest extends TestCase {
$this->assertSame(10, $repo->restore($r['snapshot'], $target, 'docs/d1/')); $this->assertSame(10, $repo->restore($r['snapshot'], $target, 'docs/d1/'));
} }
public function testRestoreOfOneFile(): void {
$repo = Repository::init($this->backend, 'pw');
$r = $repo->backupDirectory($this->src);
$target = $this->tempDir();
// occ backup:restore --to-directory passes the path with a "/" appended
$this->assertSame(1, $repo->restore($r['snapshot'], $target, 'docs/d1/note-1.txt/'));
$this->assertSame(file_get_contents($this->src . '/docs/d1/note-1.txt'), file_get_contents($target . '/docs/d1/note-1.txt'));
$this->assertSame(0, $repo->restore($r['snapshot'], $this->tempDir(), 'docs/d1/note-1/'), 'a prefix of a name is not that file');
$stripped = $this->tempDir();
$this->assertSame(1, $repo->restore($r['snapshot'], $stripped, 'docs/d1/note-1.txt/', true));
$this->assertFileExists($stripped . '/note-1.txt');
}
public function testRestoreRemovesTemporariesOfAStoppedRestore(): void {
$repo = Repository::init($this->backend, 'pw');
$r = $repo->backupDirectory($this->src);
$target = $this->tempDir();
mkdir($target . '/docs/d1', 0700, true);
mkdir($target . '/elsewhere', 0700, true);
$stale = [
$target . '/docs/d1/note-1.txt.ngb-restore-0123456789ab', // from a restore to a directory
$target . '/docs/d1/.note-4.txt.ngb-restore-a1b2c3', // from a restore through Nextcloud
];
$kept = [
$target . '/docs/d1/notes.ngb-restore-draft.txt', // not a temporary of ours
$target . '/elsewhere/x.ngb-restore-0123456789ab', // not a directory this restore writes to
];
foreach ([...$stale, ...$kept] as $f) {
file_put_contents($f, 'partial');
}
$repo->restore($r['snapshot'], $target, 'docs/');
foreach ($stale as $f) {
$this->assertFileDoesNotExist($f);
}
foreach ($kept as $f) {
$this->assertFileExists($f);
}
}
public function testWrongPassphrase(): void { public function testWrongPassphrase(): void {
Repository::init($this->backend, 'pw'); Repository::init($this->backend, 'pw');
$this->expectException(CryptoException::class); $this->expectException(CryptoException::class);
+3 -2
View File
@@ -195,7 +195,7 @@ OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS
fail "restore:full" fail "restore:full"
} }
printf '%s\n' "$OUT" | grep -q "maintenance mode while the restore runs" || fail "the restore did not say it runs in maintenance mode" printf '%s\n' "$OUT" | grep -q "maintenance mode while the restore runs" || fail "the restore did not say it runs in maintenance mode"
run_on ngb-frt-restore status 2>&1 | grep -q "maintenance: false" || fail "maintenance mode is still on after the restore" run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the restore"
run_on ngb-frt-restore config:app:get ng_backup restore_full_started >/dev/null 2>&1 && fail "the restore left its 'started' marker behind" run_on ngb-frt-restore config:app:get ng_backup restore_full_started >/dev/null 2>&1 && fail "the restore left its 'started' marker behind"
ok "backup:restore:full completed, in maintenance mode and out of it again" ok "backup:restore:full completed, in maintenance mode and out of it again"
if [ "$DB" = sqlite ]; then if [ "$DB" = sqlite ]; then
@@ -209,7 +209,8 @@ fi
# Everything below must work in a FRESH occ process (a new PHP process per docker exec), since # Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
# earlier bugs here only showed up after config.php's secret was rewritten mid-restore. # earlier bugs here only showed up after config.php's secret was rewritten mid-restore.
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after restore (own key/target undecryptable?)" run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after restore (own key/target undecryptable?)"
run_on ngb-frt-restore user:list | grep -q frtest || fail "the backed-up user is missing from the restored database" # not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database"
docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing" docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs" [ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore" run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
+13 -2
View File
@@ -49,9 +49,20 @@ SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xa
$X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash" $X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash"
ok "restore new-folder and replace: files identical, extra file in the trash" ok "restore new-folder and replace: files identical, extra file in the trash"
# One file to a local directory, where a restore that was stopped left its temporary.
ONE=/tmp/ngb-rt-one/data/$U/files/Docs
$X sh -c "mkdir -p $ONE && echo partial > $ONE/f2.bin.ngb-restore-0123456789ab"
occ backup:restore ci "$SNAP" "data/$U/files/Docs/f2.bin" -d /tmp/ngb-rt-one | grep "Restored 1 files" >/dev/null || fail "restore of one file to a directory"
$X sh -c "cmp -s $ONE/f2.bin $DATA/$U/files/Docs/f2.bin" || fail "the file restored to a directory differs"
$X sh -c "ls -A $ONE | grep -q ngb-restore" && fail "the temporary of a stopped restore was left"
ok "restore of one file to a directory; the temporary of a stopped restore removed"
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
occ backup:status | grep "started (" >/dev/null || fail "backup:status does not name the time zone"
ok "backup:status names the time zone of its times"
# Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob). # Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob).
occ backup:verify ci "$SNAP" | grep -q "^OK$" || fail "shallow verify" occ backup:verify ci "$SNAP" | grep "^OK$" >/dev/null || fail "shallow verify"
occ backup:verify ci "$SNAP" --deep | grep -q "^OK$" || fail "deep verify" occ backup:verify ci "$SNAP" --deep | grep "^OK$" >/dev/null || fail "deep verify"
ok "shallow and deep checksum audit" ok "shallow and deep checksum audit"
# Database dump/restore round trip (all tables, checksums, sequences on PostgreSQL). # Database dump/restore round trip (all tables, checksums, sequences on PostgreSQL).
+2 -1
View File
@@ -36,7 +36,8 @@ MANIFEST=$(occ backup:user:backup ci "$U" | sed -n 's/.* to \(.*\)\.$/\1/p')
[ -n "$MANIFEST" ] || fail "backup:user:backup printed no manifest path" [ -n "$MANIFEST" ] || fail "backup:user:backup printed no manifest path"
ok "backed up $U to $MANIFEST" ok "backed up $U to $MANIFEST"
occ backup:user:list ci "$U" | grep -qF "$MANIFEST" || fail "backup:user:list does not show $MANIFEST" # not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
occ backup:user:list ci "$U" | grep -F "$MANIFEST" >/dev/null || fail "backup:user:list does not show $MANIFEST"
ok "backup:user:list shows the export" ok "backup:user:list shows the export"
# as-backup: restore into <uid>-bak; the original account must stay untouched. $U has one export, # as-backup: restore into <uid>-bak; the original account must stay untouched. $U has one export,