Compare commits

..

23 Commits

Author SHA1 Message Date
Dillard Blom bfd54447b0 Merge pull request #6: Release 0.9.2-beta.3
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-08 01:28:52 +02:00
dillard 0b50bb6b81 Verify: retry storage exceptions only, let programming errors surface
Catch \Exception (storage adapters throw their own exception types) instead
of \Throwable, so an \Error such as a TypeError is not retried and reported
as an unreadable blob.
2026-10-08 01:21:15 +02:00
dillard f415bab519 Verify, README: address review notes
An unreadable blob may still be a truncated pack, so don't rule out damage
in the verify message; README: Hetzner needs WebDAV/SMB and external
reachability enabled for the sub-account.
2026-10-08 01:11:21 +02:00
dillard 27eb0b32fc Release 0.9.2-beta.3 2026-10-08 01:00:52 +02:00
dillard b1d4f6a469 README: Hetzner Storage Box over WebDAV and SMB
Tested both on sub-accounts: works end to end; WebDAV needs an existing
root, SMB needs port 445 and the smbclient extension; fixed cost per run
and slow deep verify over remote storage.
2026-10-08 01:00:46 +02:00
dillard fe7ad9b4b5 Verify: report read errors as unreadable, not as corrupted
verifyBlob() caught every exception from loading a blob and counted it as
a failed (corrupted or tampered) blob, so a transient network error during
a deep verify over WebDAV looked like data corruption. Read the range
separately, retry it, and report a read that keeps failing as unreadable
with its error; only a blob that was read but does not decrypt or match
its id counts as failed.
2026-10-08 00:53:31 +02:00
dillard 695d972785 Backups: let the runs table scroll on narrow screens
The table could not shrink below its content width and ran off the page,
cutting off the last columns; wrap it in an overflow-x container.
2026-10-08 00:47:47 +02:00
dillard 972a52e665 Settings: align the tabs with the sections
The tabs had no inline margin, so they sat against the edge while every
NcSettingsSection is inset by 7 grid baselines.
2026-10-08 00:47:47 +02:00
dillard 7c08a10621 Locations: ask for password confirmation before adding
addTarget requires a recent password confirmation, but the form never
asked for one. Within 30 minutes of logging in that went unnoticed; after
that, adding a location failed with a bare 403.
2026-10-08 00:40:48 +02:00
Dillard Blom 209520165e Merge pull request #5: Release 0.9.1-beta.2
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-07 18:09:23 +02:00
dillard 9554e774e5 CHANGELOG: correct why info.xml carries no -beta suffix 2026-10-07 18:01:20 +02:00
dillard 8b2dca28e3 Release 0.9.1-beta.2 2026-10-07 17:47:40 +02:00
dillard 5742f77253 App store listing: screenshots, links and UI fixes
Add three screenshots (backups, restore, settings) taken on a demo
instance, plus website, repository and the HOWTO as admin documentation
in info.xml.

Fixes found while taking them:
- the "new folder" restore option lost "<date>" (l10n sanitizes it as
  a tag); show the folder name pattern via a placeholder instead
- the restore-point dropdown cut off the time
- passphrase actions stacked vertically in the slot table
- HOWTO: SFTP example used "sftp::password", the auth mechanism is
  "password::password"
2026-10-07 17:43:41 +02:00
Dillard Blom 550249570b Merge pull request #4: README key rotation note
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
README: key rotation for revoking old kits is on the backlog
2026-10-07 02:26:19 +02:00
dillard 0eb23f56e7 README: note that key rotation for revoking old kits is on the backlog 2026-10-07 02:18:45 +02:00
Dillard Blom a0c1f1cd58 Merge pull request #3: Release 0.9.0-beta.1
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 0.9.0-beta.1
2026-10-07 01:15:19 +02:00
dillard 26cc9a897e Release 0.9.0-beta.1
Version 0.9.0 in info.xml, the app description for the beta, and the
CHANGELOG entry for 0.9.0-beta.1.
2026-10-07 01:07:55 +02:00
Dillard Blom fe0385ad8c Merge pull request #2: Pre-beta review
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Pre-beta review: data safety, safer commands, known limitations
2026-10-07 01:06:40 +02:00
dillard b37a953cfd Detect a busy run by identity, and clean up a failed kit write
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
step() hands back the same Run object when another process holds the lease;
comparing update times per second could stop the scheduled job early after a
quick step. A failed write of the recovery kit file now closes the handle and
removes the partial file.
2026-10-07 00:48:11 +02:00
dillard b89b5b5fa6 Document known limitations for the beta and record the review backlog
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
The open points from the pre-beta review are on the backlog in PLAN.md and
listed as known limitations in the README; none block 0.9.0-beta or 1.0.0.
The how-to warns against pointing a test server at a live location and
explains that removing a slot does not revoke access.
2026-10-07 00:19:46 +02:00
dillard 16e09f4fde Fix trash on SFTP and the disaster recovery test after the review fixes
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
forget() and untrash() uploaded a remote read stream directly, which the
new size check could not judge on SFTP, so cleanup failed there. They now
copy through a local buffer, and the size check only trusts local streams
(php://temp, php://memory, plain files).

The disaster recovery test calls backup:restore:full with --no-interaction:
docker exec without a terminal still counts as interactive, so the new
confirmation question read an empty answer and cancelled.
2026-10-06 23:59:17 +02:00
dillard 45069994e9 Pre-beta review: safer destructive commands and hardening
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- backup:restore:full refuses on an installation with more than one user
  and asks before it starts (--force skips both, also to rerun after a
  failure halfway), and warns when the snapshot's database dump was not
  one consistent transaction.
- backup:user:restore --mode=replace asks before deleting an existing
  account; non-interactive use needs --force.
- The recovery kit file is created owner-only and never over an existing
  file or symlink; backup:key:import shows the deletion delay it read from
  the kit.
- Adding a location through the web API needs password confirmation, like
  removing one.
- Key derivation parameters from a kit or repository config are bounded,
  and a stalled network stream no longer spins at full CPU.
- Floating-point columns are dumped exactly instead of with 14 digits.
2026-10-06 23:48:41 +02:00
dillard 0f4a238414 Pre-beta review: data safety in storage, catalog and backup runs
Storage and repository:
- Listing skipped every name containing ".part-", directories too, so the
  exports of a user like "john.part-time" were invisible to prune on S3,
  SFTP, FTP and SMB and their data was deleted. Only real temporary uploads
  ("<name>.part-<8 hex>") are skipped now.
- prune refuses to run while a recorded per-user export's manifest is
  missing, instead of freeing that export's data.
- forget() and untrash() copy first, switch the catalog, and delete the
  originals last, so a crash can no longer leave the catalog pointing at a
  missing snapshot or database dump.
- Writes are checked against the expected size, so a short upload is an
  error instead of a truncated pack; raw restores check rename() and touch().
- A catalog write refuses to overwrite a generation another process just
  wrote, and the catalog lock lasts 300 s, so opening a large repository
  inside it no longer lets the lease run out.

Backup runs and scheduling:
- A file that keeps changing is read again at most three times; then the
  last read is kept and reported, instead of restarting forever and
  blocking cleanup.
- Unreadable files and folders are recorded, logged, shown by occ
  backup:run, stored in the snapshot and notified to admins, instead of
  being skipped silently.
- After a failed scheduled backup, at most three attempts a day, an hour
  apart, instead of a new run and notification every five minutes.
- The job no longer spins on a run whose lease another process holds, and
  the weekly cleanup catches up after a Sunday spent backing up.
- Notifications for the per-user export retention events.
2026-10-06 23:48:40 +02:00
43 changed files with 757 additions and 103 deletions
+85 -3
View File
@@ -2,9 +2,88 @@
All notable changes to NG Backup are documented here. The format follows All notable changes to NG Backup are documented here. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning follows the stages in [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning follows the stages in
`PLAN.md` section 9. The `<version>` in `appinfo/info.xml` is plain SemVer-numeric (Nextcloud's `PLAN.md` section 9. The `<version>` in `appinfo/info.xml` is plain SemVer-numeric on purpose:
own `info.xsd` doesn't allow a `-alpha`/`-beta`/`-rc` suffix there); the stability stage is instead the app store accepts a `-beta` suffix, but Nextcloud servers then only offer the release on the
carried by the git tag, the release, and this file. beta/daily update channel, which would hide the app from most admins. The stability stage is
instead carried by the app description, the git tag, the release, and this file.
## [0.9.2-beta.3] - 2026-10-08
Bugfix release.
### Added
- README: notes on using a Hetzner Storage Box over WebDAV and SMB.
### Fixed
- Adding a backup location in the web interface failed with "Request failed with status code 403"
when the login was more than 30 minutes old: the form did not ask for password confirmation.
- Settings page: the "Backup and restore" / "Settings" tabs line up with the sections below them.
- Backups: the runs table scrolls horizontally on narrow screens instead of running off the page.
- `occ backup:verify --deep`: a blob that could not be read (network or storage error) was reported
as "corrupted or tampered". Reads are now retried, and a read that keeps failing is reported
separately as unreadable, with the error (run the check again; if the same blobs stay unreadable,
their pack may be damaged or truncated). Only a blob that was read but fails decryption or its
hash check counts as corrupted. Found on a Hetzner Storage Box over WebDAV, where the blob
turned out to be intact.
## [0.9.1-beta.2] - 2026-10-07
Second beta, the first release in the Nextcloud app store (signed).
### Added
- App store listing: screenshots, website, repository and documentation link in `appinfo/info.xml`.
### Fixed
- Restore: the "new folder" option showed `"Restored "` without the date pattern.
- Restore: the restore-point dropdown cut off the time.
- Settings: the passphrase actions are on one line instead of stacked.
- `docs/HOWTO.md`: the SFTP example used a non-existent auth mechanism (`sftp::password`
instead of `password::password`).
## [0.9.0-beta.1] - 2026-10-07
First beta. Tested thoroughly and usable **alongside** your current backup, but **not yet a
drop-in replacement** for it. We test what we can ourselves (Nextcloud 32–35, SQLite, MariaDB and
PostgreSQL, S3, SFTP, FTP, SMB and NFS), but we need field experience from other setups: please
report what works and what doesn't, especially on large instances and on storage providers not
listed in the README.
### Added
- Disaster recovery: `occ backup:key:import` and `occ backup:restore:full` restore the database,
data directory and config onto a fresh installation of the same major version.
- Per-user backup and restore via user_migration: `occ backup:user:backup` and
`occ backup:user:restore` (replace the account after a safety export, or restore as
`<uid>-bak`), also onto a server where the user does not exist yet. The commands ask before
installing user_migration (or take `--install-user-migration`).
- Checksum audit: `occ backup:verify`, shallow or `--deep`, for snapshots and per-user exports.
- Retention for per-user exports: `occ backup:retention --user-last=N`.
- FTP and SMB covered by the integration tests; NFS through the `local` backend on a mounted share.
- `docs/HOWTO.md`: step by step from setup to a tested restore.
- Warnings when a backup finishes with unreadable files or folders, or with files that kept
changing while being read (log, `occ`, notification, snapshot).
### Changed
- `backup:restore:full` refuses on an installation with more than one user and asks first;
`backup:user:restore --mode=replace` asks before deleting an existing account (`--force` for
scripts in both cases).
- A failed scheduled backup is retried at most three times a day, an hour apart.
- The recovery kit carries the deletion delay; the kit file is created owner-only.
- Adding a location through the web API needs password confirmation.
### Fixed
- Exports of users whose id contains `.part-` could be deleted by cleanup on remote locations.
- Cleanup no longer frees the data of a per-user export whose manifest is missing.
- Crash safety of moving snapshots to and from the trash, of catalog writes by concurrent
backups and per-user exports, and of short uploads.
- A file that keeps changing no longer makes a backup restart forever.
- Long restores and exports keep their lock on the location alive.
- Floating-point database columns are dumped exactly.
### Known limitations
See "Known limitations" in the README: memory use on very large instances, never two servers
writing to one location, removing a passphrase slot does not revoke access, no rollback detection
before a disaster recovery, and a few smaller points. None of them block the road to 1.0.0; they
are on the backlog in `PLAN.md`.
## [0.1.0-alpha.1] - 2026-10-05 ## [0.1.0-alpha.1] - 2026-10-05
@@ -43,4 +122,7 @@ and not implemented yet.
- See the "Storage backend experiences" section of the README for provider-specific quirks found - See the "Storage backend experiences" section of the README for provider-specific quirks found
so far (e.g. Hetzner Storage Box sub-accounts). so far (e.g. Hetzner Storage Box sub-accounts).
[0.9.2-beta.3]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.2-beta.3
[0.9.1-beta.2]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.1-beta.2
[0.9.0-beta.1]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.0-beta.1
[0.1.0-alpha.1]: https://github.com/dillardblom/ng_backup/releases/tag/v0.1.0-alpha.1 [0.1.0-alpha.1]: https://github.com/dillardblom/ng_backup/releases/tag/v0.1.0-alpha.1
+18
View File
@@ -359,6 +359,24 @@ minimale volledige restore (`occ backup:restore:full` op een verse installatie)
Backlog: (a) de originele weergavenaam herstellen (het label uit de export is niet getest als Backlog: (a) de originele weergavenaam herstellen (het label uit de export is niet getest als
weergavenaam); (b) de interne `migrated-`-naam hernoemen (vereist directe databasewijziging, niet voor de beta). weergavenaam); (b) de interne `migrated-`-naam hernoemen (vereist directe databasewijziging, niet voor de beta).
**Backlog uit de pre-beta review (2026-10-07)** — geen belemmering voor 0.9.0-beta of 1.0.0, staat
in README "Known limitations":
1. Geheugen bij grote instanties (>1 TB, miljoenen bestanden): blob-index, prune-set, browse van een
snapshot in de web-UI, `trashExtras` bij restore en het manifest van een user-export staan volledig
in het geheugen; streaming/paginering nodig.
2. Twee servers op één locatie: catalogus kan breken (generatie overschreven, of anchor >200 generaties
achter). Docs-waarschuwing staat er; mogelijk later een echte put-if-absent per backend.
3. Sleutelrotatie: een slot verwijderen trekt geen toegang in (master-key blijft gelijk).
4. Rollback vóór DR niet detecteerbaar: referentie-generatie en -hash in de kit opnemen of tonen.
5. Kit-velden buiten `wrapped_key` (o.a. `delete_delay_days`) zijn niet geauthenticeerd; nu alleen getoond.
6. Repository-id in de AD van objecten (formaatwijziging).
7. Wachtwoorden van locaties via `--option-file` of omgevingsvariabele i.p.v. de commandoregel.
8. Tijdelijke restore-bestanden niet via de Files-API verwijderen (komen nu in de prullenbak).
9. DB-restore in foreign-key-volgorde (cascades).
10. Packs zonder index (index-upload mislukt) opruimen.
11. `backup:restore:full` in onderhoudsmodus en hervatbaar maken; nu: opnieuw draaien met `--force`.
12. Config/rename op FTP/SMB-servers die geen rename over een bestaand bestand toestaan.
Doorgeschoven naar **v1.3.0** (fase 3, niet meer fase 2): in-place volledige restore Doorgeschoven naar **v1.3.0** (fase 3, niet meer fase 2): in-place volledige restore
(eigen restore-modus i.p.v. onderhoudsmodus) en zelfbediening voor gebruikers (eigen bestanden (eigen restore-modus i.p.v. onderhoudsmodus) en zelfbediening voor gebruikers (eigen bestanden
terugzetten vanuit de Files-app). terugzetten vanuit de Files-app).
+42 -3
View File
@@ -7,8 +7,9 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
see "Storage backend experiences" below — that requirement comes from files_external, not from see "Storage backend experiences" below — that requirement comes from files_external, not from
ng_backup, and applies to any Nextcloud app that writes to an SMB share.) ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
**Status: pre-release. Not yet tested on production.** See `PLAN.md` for the design and `docs/HOWTO.md` for a **Status: beta (0.9.2).** Tested thoroughly, usable alongside your current backup, but not yet a
step-by-step guide from setup to a tested restore. drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore.
License: AGPL-3.0-or-later. License: AGPL-3.0-or-later.
@@ -62,7 +63,8 @@ All backups of an installation are encrypted with one master key.
a single slot is possible, but then that one passphrase is a single point of failure. a single slot is possible, but then that one passphrase is a single point of failure.
- When a holder leaves, give their slot a new passphrase (`occ backup:key:slot:replace`), download - When a holder leaves, give their slot a new passphrase (`occ backup:key:slot:replace`), download
the new kit and confirm it. Note: an old kit together with the old passphrase still opens the the new kit and confirm it. Note: an old kit together with the old passphrase still opens the
same master key; to revoke it completely, start new locations with a new key. same master key; to revoke it completely, start new locations with a new key. This is noted on
the backlog and will be addressed in a future version (key rotation).
- Downloading the kit, and every change to the slots, requires the administrator's password, is - Downloading the kit, and every change to the slots, requires the administrator's password, is
written to the audit log and is announced to all administrators. written to the audit log and is announced to all administrators.
@@ -124,6 +126,20 @@ Contributions welcome via PR — add what you ran into with your own provider.
verified whether this also applies to a Hetzner Storage Box **main** account (only tested with a verified whether this also applies to a Hetzner Storage Box **main** account (only tested with a
sub-account so far) — if you've tried a main account, please say so in a PR. sub-account so far) — if you've tried a main account, please say so in a PR.
- **Hetzner Storage Box over WebDAV and SMB (sub-accounts, 2026-10-07):** both work end to end
(backup, incremental run, restore of a folder, verify), tested from outside the Hetzner network
with about 66 MiB of data. In the Hetzner Console, enable WebDAV or SMB support for the
sub-account, and "External reachability" when your server is not in the Hetzner network.
WebDAV: host `https://uXXXXX-subN.your-storagebox.de`, and the `root`
must be a folder that already exists (or empty): WebDAV does not create missing parent folders,
so a root that doesn't exist yet fails `backup:target:add` with `Sabre\HTTP\ClientHttpException:
Conflict`. SMB: share = the sub-account name, port 445 must be reachable from your server (it was
from a home connection here, but some providers block it), and the `smbclient` PECL extension is
required (see the SMB entry below). Both have a noticeable fixed cost per run: an incremental run
that uploads nothing took about 20 s over WebDAV and 50 s over SMB. A deep verify is slow over
remote storage, since each blob is fetched with its own request (19 minutes for 66 MiB over
WebDAV); run it occasionally, not after every backup.
- **FTP, server-dependent absolute-path handling:** files_external's FTP backend always builds - **FTP, server-dependent absolute-path handling:** files_external's FTP backend always builds
absolute paths (e.g. `MKD /repo`) for the configured root, never relative ones. `atmoz/sftp`-style absolute paths (e.g. `MKD /repo`) for the configured root, never relative ones. `atmoz/sftp`-style
`vsftpd` test servers (chrooted) can reject that with a generic "create directory operation `vsftpd` test servers (chrooted) can reject that with a generic "create directory operation
@@ -147,6 +163,29 @@ Contributions welcome via PR — add what you ran into with your own provider.
mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS
can't be driven from PHP directly. can't be driven from PHP directly.
## Known limitations
On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it:
- **Very large instances** (well over 1 TB, millions of files): the blob index, cleanup, browsing a
snapshot in the web interface and per-user export manifests are held in memory. Expect high
memory use there; reports from such instances are very welcome.
- **One location, one server.** Never let two Nextcloud servers write to the same location (for
example a disaster recovery test server pointed at the production location): they can break each
other's catalog, and the location is then refused as possibly tampered with. Test a disaster
recovery against a copy of the location, or only read from it.
- **Removing a passphrase slot does not revoke access.** The master key stays the same, so someone
with an old recovery kit and that slot's passphrase can still decrypt the location if they can
read it. Key rotation is not available yet.
- **Rollback before a disaster recovery is not detected.** A fresh server has no record of the
newest catalog generation, so a location rolled back to an older state looks valid there.
- Location passwords passed with `-o password=...` are visible in the process list and shell
history while the command runs.
- Temporary files of a restore into a user's files end up in that user's trash.
- Restoring the database does not order tables by foreign keys; apps with cascading foreign keys
could lose rows (Nextcloud core does not use them).
- A pack whose index upload failed stays on the location until removed by hand.
## Commands ## Commands
| Command | Purpose | | Command | Purpose |
+14 -6
View File
@@ -4,17 +4,25 @@
<id>ng_backup</id> <id>ng_backup</id>
<name>NG Backup</name> <name>NG Backup</name>
<summary>Encrypted, incremental backups of your Nextcloud with restore from the web interface</summary> <summary>Encrypted, incremental backups of your Nextcloud with restore from the web interface</summary>
<description><![CDATA[Alpha: feature-complete for scheduled, encrypted, incremental backups to <description><![CDATA[Beta: scheduled, encrypted, incremental backups of the database, config and data
S3/SFTP/WebDAV/local, with restore of files and folders from the web interface and occ. Meant for directory to S3, SFTP, WebDAV, FTP, SMB or a local path, with restore of files and folders, of
testing on a copy of production (dev, test, staging) -- not yet for production use, and full single users (via user_migration) and of the whole server onto a fresh installation. Tested
disaster recovery is not implemented yet. See the recovery kit section of the README before thoroughly and usable alongside your current backup, but not yet a drop-in replacement for it.
relying on this for anything.]]></description> See the README for known limitations and the recovery kit before relying on it.]]></description>
<version>0.1.0</version> <version>0.9.2</version>
<licence>agpl</licence> <licence>agpl</licence>
<author>Dillard Blom</author> <author>Dillard Blom</author>
<namespace>NgBackup</namespace> <namespace>NgBackup</namespace>
<documentation>
<admin>https://github.com/dillardblom/ng_backup/blob/main/docs/HOWTO.md</admin>
</documentation>
<category>tools</category> <category>tools</category>
<website>https://apps.nextcloud.com/apps/ng_backup</website>
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs> <bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/3-settings.png</screenshot>
<dependencies> <dependencies>
<php min-version="8.1"/> <php min-version="8.1"/>
<lib>sodium</lib> <lib>sodium</lib>
+11 -4
View File
@@ -14,7 +14,8 @@ occ backup:key:confirm -u <your-admin-uid> --code=<code shown> --confirm="Yes, I
- The passphrase protects the backup. Keep it somewhere other than this server. - The passphrase protects the backup. Keep it somewhere other than this server.
- Optional: extra passphrases for other people with `occ backup:key:slot:add --label="..."`. - Optional: extra passphrases for other people with `occ backup:key:slot:add --label="..."`.
Any one passphrase is enough to restore. Any one passphrase is enough to restore. Removing a slot later does not revoke access for someone
who kept an old kit and that passphrase (see "Known limitations" in the README).
- The recovery kit (`/var/tmp/ng_backup-kit.json` above) is also required for disaster recovery. The web - The recovery kit (`/var/tmp/ng_backup-kit.json` above) is also required for disaster recovery. The web
user writes it there; then move it off the server as root, next to the passphrase, and delete the user writes it there; then move it off the server as root, next to the passphrase, and delete the
copy in `/var/tmp`. Without the kit and a passphrase, the backup cannot be read. copy in `/var/tmp`. Without the kit and a passphrase, the backup cannot be read.
@@ -31,7 +32,7 @@ Examples:
``` ```
occ backup:target:add offsite amazons3 -a amazons3::accesskey -o bucket=<bucket> -o key=<key> -o secret=<secret> occ backup:target:add offsite amazons3 -a amazons3::accesskey -o bucket=<bucket> -o key=<key> -o secret=<secret>
occ backup:target:add nas sftp -a sftp::password -o host=<host> -o user=<user> -o password=<pw> -o root=home occ backup:target:add nas sftp -a password::password -o host=<host> -o user=<user> -o password=<pw> -o root=home
occ backup:target:add local local -a null::null -o datadir=/mnt/backup occ backup:target:add local local -a null::null -o datadir=/mnt/backup
``` ```
@@ -102,7 +103,9 @@ occ backup:user:restore offsite users/<uid>/<exportId> --mode=as-backup
use, add `--install-user-migration` to approve that explicitly. The command stops after the use, add `--install-user-migration` to approve that explicitly. The command stops after the
installation; run it once more. installation; run it once more.
- `replace` first makes a safety export of the current account, then recreates the user under the - `replace` first makes a safety export of the current account, then recreates the user under the
same id. `as-backup` restores into `<uid>-bak` and leaves the original untouched. same id. When the account exists, the command asks first; in scripts, add `--force`. Shares to
others and data of apps without a user_migration migrator are not restored, so prefer
`as-backup` when in doubt: it restores into `<uid>-bak` and leaves the original untouched.
- A restore onto a server where the user does not exist works. Calendars and address books get an - A restore onto a server where the user does not exist works. Calendars and address books get an
internal name with the prefix `migrated-` (for example `migrated-personal`). That prefix comes from internal name with the prefix `migrated-` (for example `migrated-personal`). That prefix comes from
user_migration and cannot be turned off. The user can change the display name of these calendars and user_migration and cannot be turned off. The user can change the display name of these calendars and
@@ -115,7 +118,8 @@ occ backup:user:restore offsite users/<uid>/<exportId> --mode=as-backup
## 7. Disaster recovery: restore the whole server ## 7. Disaster recovery: restore the whole server
Use a fresh Nextcloud installation of the same major version, with NG Backup installed and Use a fresh Nextcloud installation of the same major version, with NG Backup installed and
the same location available: the same location available. For a test, use a **copy** of the location: two servers writing to one
location break each other's catalog.
``` ```
occ backup:key:import /path/ng_backup-kit.json --passphrase-file=/path/passphrase occ backup:key:import /path/ng_backup-kit.json --passphrase-file=/path/passphrase
@@ -124,6 +128,9 @@ occ backup:list offsite # the snapshots must be lis
occ backup:restore:full offsite <snapshot> occ backup:restore:full offsite <snapshot>
``` ```
The command refuses on an installation that already has more than one user, and asks before it
starts; `--force` skips both (also needed to rerun it after it stopped halfway).
Afterwards, check that users, files and the database are complete, and run a backup of the restored Afterwards, check that users, files and the database are complete, and run a backup of the restored
server. The restored installation keeps its own app configuration and secret. server. The restored installation keeps its own app configuration and secret.
+6 -1
View File
@@ -33,11 +33,16 @@ final class LocalBackend implements IBackend {
if ($out === false) { if ($out === false) {
throw new BackendException('Cannot write ' . $path); throw new BackendException('Cannot write ' . $path);
} }
$expected = StorageBackend::remaining($stream);
if (stream_copy_to_stream($stream, $out) === false || !fflush($out)) { if (stream_copy_to_stream($stream, $out) === false || !fflush($out)) {
throw new BackendException('Write failed for ' . $path); throw new BackendException('Write failed for ' . $path);
} }
fclose($out); fclose($out);
$out = false; $out = false;
clearstatcache(true, $tmp);
if ($expected !== null && filesize($tmp) !== $expected) {
throw new BackendException('Incomplete write for ' . $path);
}
if (!rename($tmp, $target)) { if (!rename($tmp, $target)) {
throw new BackendException('Cannot finalise ' . $path); throw new BackendException('Cannot finalise ' . $path);
} }
@@ -94,7 +99,7 @@ final class LocalBackend implements IBackend {
$result = []; $result = [];
$it = new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator($base, \FilesystemIterator::SKIP_DOTS)); $it = new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator($base, \FilesystemIterator::SKIP_DOTS));
foreach ($it as $file) { foreach ($it as $file) {
if ($file->isFile() && !str_contains($file->getFilename(), '.part-')) { if ($file->isFile() && !StorageBackend::isTemporary($file->getFilename())) {
$result[] = ltrim(substr($file->getPathname(), strlen($this->root)), '/'); $result[] = ltrim(substr($file->getPathname(), strlen($this->root)), '/');
} }
} }
+32 -2
View File
@@ -30,7 +30,13 @@ final class StorageBackend implements IBackend {
} }
} }
/** A temporary upload of put(): "<name>.part-<8 hex>". */
public static function isTemporary(string $name): bool {
return preg_match('/\.part-[0-9a-f]{8}$/', $name) === 1;
}
public function put(string $path, $stream): void { public function put(string $path, $stream): void {
$expected = self::remaining($stream);
$target = $this->abs($path); $target = $this->abs($path);
$this->mkdirs(dirname($target)); $this->mkdirs(dirname($target));
$writeTo = $this->atomicPut ? $target : $target . '.part-' . bin2hex(random_bytes(4)); $writeTo = $this->atomicPut ? $target : $target . '.part-' . bin2hex(random_bytes(4));
@@ -54,6 +60,10 @@ final class StorageBackend implements IBackend {
if (is_resource($stream)) { if (is_resource($stream)) {
fclose($stream); fclose($stream);
} }
// Some adapters report success after a short write or a failed flush: check the size.
if ($expected !== null && $this->storage->filesize($writeTo) !== $expected) {
throw new BackendException('Incomplete write for ' . $path);
}
if ($writeTo !== $target && !$this->storage->rename($writeTo, $target)) { if ($writeTo !== $target && !$this->storage->rename($writeTo, $target)) {
throw new BackendException('Cannot finalise ' . $path); throw new BackendException('Cannot finalise ' . $path);
} }
@@ -132,19 +142,39 @@ final class StorageBackend implements IBackend {
return; return;
} }
while (($name = readdir($dh)) !== false) { while (($name = readdir($dh)) !== false) {
if ($name === '.' || $name === '..' || str_contains($name, '.part-')) { if ($name === '.' || $name === '..') {
continue; continue;
} }
$child = $dir . '/' . $name; $child = $dir . '/' . $name;
if ($this->storage->is_dir($child)) { if ($this->storage->is_dir($child)) {
$this->listInto($child, $result); $this->listInto($child, $result);
} else { } elseif (!self::isTemporary($name)) {
$result[] = $child; $result[] = $child;
} }
} }
closedir($dh); closedir($dh);
} }
/**
* Bytes left to read in a local $stream (php://temp, php://memory or a plain file), or null:
* stream wrappers of remote storages do not report a reliable size.
*/
public static function remaining(mixed $stream): ?int {
if (!is_resource($stream)) {
return null;
}
$meta = stream_get_meta_data($stream);
if (!($meta['seekable'] ?? false) || !in_array($meta['wrapper_type'] ?? '', ['PHP', 'plainfile'], true)) {
return null;
}
$stat = fstat($stream);
$pos = ftell($stream);
if ($stat === false || $pos === false) {
return null;
}
return max(0, $stat['size'] - $pos);
}
private function mkdirs(string $dir): void { private function mkdirs(string $dir): void {
if ($dir === '' || $dir === '.' || $this->storage->is_dir($dir)) { if ($dir === '' || $dir === '.' || $this->storage->is_dir($dir)) {
return; return;
+22 -5
View File
@@ -38,13 +38,19 @@ final class BackupJob extends TimedJob {
$this->setTimeSensitivity(IJob::TIME_SENSITIVE); $this->setTimeSensitivity(IJob::TIME_SENSITIVE);
} }
private const MAX_DAILY_ATTEMPTS = 3;
protected function run($argument): void { protected function run($argument): void {
$budget = PHP_SAPI === 'cli' ? 240.0 : 20.0; $budget = PHP_SAPI === 'cli' ? 240.0 : 20.0;
$deadline = microtime(true) + $budget; $deadline = microtime(true) + $budget;
$this->startScheduled(); $this->startScheduled();
foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) { foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) {
while (microtime(true) < $deadline && $run->getStatus() === 'running') { while (microtime(true) < $deadline && $run->getStatus() === 'running') {
$run = $this->backups->step($run, min($deadline, microtime(true) + $budget)); $next = $this->backups->step($run, min($deadline, microtime(true) + $budget));
if ($next === $run) {
break; // step() hands back the same object when another process holds the run's lease
}
$run = $next;
} }
} }
$this->failStaleRestores(); $this->failStaleRestores();
@@ -67,10 +73,15 @@ final class BackupJob extends TimedJob {
} }
} }
/** Once a week (after Sunday's backup), apply the retention policy to non-append-only locations. */ /**
* Once a week, apply the retention policy to non-append-only locations: from Sunday on, at the
* first moment no backup is running, so a long Sunday backup only delays it instead of skipping
* the week.
*/
private function weeklyPrune(): void { private function weeklyPrune(): void {
$lastWeek = $this->appConfig->getValueString(Application::APP_ID, 'last_prune_week', ''); $lastWeek = $this->appConfig->getValueString(Application::APP_ID, 'last_prune_week', '');
if (date('N') !== '7' || $lastWeek === date('o-W') || $this->runs->findRunning(null, BackupService::KIND_FULL) !== []) { $due = date('N') === '7' ? date('o-W') : date('o-W', (int)strtotime('last sunday'));
if (strcmp($lastWeek, $due) >= 0 || $this->runs->findRunning(null, BackupService::KIND_FULL) !== []) {
return; return;
} }
foreach ($this->targets->list() as $t) { foreach ($this->targets->list() as $t) {
@@ -83,7 +94,7 @@ final class BackupJob extends TimedJob {
$this->logger->warning('NG Backup: cleanup of ' . $t->getName() . ' failed: ' . $e->getMessage()); $this->logger->warning('NG Backup: cleanup of ' . $t->getName() . ' failed: ' . $e->getMessage());
} }
} }
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', date('o-W')); $this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
} }
/** Daily schedule: appconfig ng_backup/schedule_time "HH:MM" (server time); empty = no schedule. */ /** Daily schedule: appconfig ng_backup/schedule_time "HH:MM" (server time); empty = no schedule. */
@@ -92,7 +103,7 @@ final class BackupJob extends TimedJob {
if (!preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $at)) { if (!preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $at)) {
return; return;
} }
$todayAt = strtotime(date('Y-m-d') . ' ' . $at); $todayAt = (int)strtotime(date('Y-m-d') . ' ' . $at);
if (time() < $todayAt) { if (time() < $todayAt) {
return; return;
} }
@@ -101,6 +112,12 @@ final class BackupJob extends TimedJob {
if ($this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== [] || ($last !== null && $last->getStartedAt() >= $todayAt)) { if ($this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== [] || ($last !== null && $last->getStartedAt() >= $todayAt)) {
continue; continue;
} }
// After a failure: retry at most MAX_DAILY_ATTEMPTS times a day, an hour apart, instead
// of starting (and notifying about) a new full run every five minutes all day.
$today = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $todayAt);
if (count($today) >= self::MAX_DAILY_ATTEMPTS || ($today !== [] && time() - (int)$today[0]->getFinishedAt() < 3600)) {
continue;
}
try { try {
$this->backups->start($t, 'scheduled'); $this->backups->start($t, 'scheduled');
} catch (\Throwable $e) { } catch (\Throwable $e) {
+3
View File
@@ -76,6 +76,9 @@ HELP);
return 1; return 1;
} }
$output->writeln('Backup key imported, fingerprint <info>' . $this->keys->fingerprint() . '</info>.'); $output->writeln('Backup key imported, fingerprint <info>' . $this->keys->fingerprint() . '</info>.');
// Read from the kit file, which is not authenticated beyond the wrapped key: show it so a
// wrong or altered value stands out.
$output->writeln('Deletion delay for forgotten snapshots: <info>' . $this->keys->deleteDelayDays() . '</info> days (from the kit).');
$output->writeln('Next: connect the backup location with the same details as before (<info>occ backup:target:add</info>), then <info>occ backup:list</info> and <info>occ backup:restore:full</info>.'); $output->writeln('Next: connect the backup location with the same details as before (<info>occ backup:target:add</info>), then <info>occ backup:list</info> and <info>occ backup:restore:full</info>.');
return 0; return 0;
} }
+15 -2
View File
@@ -39,11 +39,24 @@ final class KeyKit extends Command {
$output->write($json, false, OutputInterface::OUTPUT_RAW); $output->write($json, false, OutputInterface::OUTPUT_RAW);
return 0; return 0;
} }
if (file_put_contents($file, $json) === false) { // Created owner-only from the start (no window where it is world-readable), never over an
// existing file or symlink someone else may have placed at a predictable path.
$old = umask(0077);
try {
$fh = @fopen($file, 'xb');
} finally {
umask($old);
}
if ($fh === false) {
$output->writeln("<error>Cannot create $file (it must not exist yet)</error>");
return 1;
}
$ok = fwrite($fh, $json) === strlen($json);
if (!fclose($fh) || !$ok) {
@unlink($file);
$output->writeln("<error>Cannot write $file</error>"); $output->writeln("<error>Cannot write $file</error>");
return 1; return 1;
} }
@chmod($file, 0600);
$output->writeln("Recovery kit written to <info>$file</info>. Copy it off this server, then run <info>occ backup:key:confirm</info>."); $output->writeln("Recovery kit written to <info>$file</info>. Copy it off this server, then run <info>occ backup:key:confirm</info>.");
return 0; return 0;
} }
+26
View File
@@ -9,7 +9,11 @@ namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\RestoreFullService; use OCA\NgBackup\Service\RestoreFullService;
use OCA\NgBackup\Service\TargetService; use OCA\NgBackup\Service\TargetService;
use OCP\IUserManager;
use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\QuestionHelper;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Question\ConfirmationQuestion;
use Symfony\Component\Console\Input\InputArgument; use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface; use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface; use Symfony\Component\Console\Output\OutputInterface;
@@ -18,6 +22,7 @@ final class RestoreFull extends Command {
public function __construct( public function __construct(
private TargetService $targets, private TargetService $targets,
private RestoreFullService $restoreFull, private RestoreFullService $restoreFull,
private IUserManager $userManager,
) { ) {
parent::__construct(); parent::__construct();
} }
@@ -27,6 +32,7 @@ final class RestoreFull extends Command {
->setDescription('Disaster recovery: restore the database, data directory and config on a fresh installation') ->setDescription('Disaster recovery: restore the database, data directory and config on a fresh installation')
->addArgument('target', InputArgument::REQUIRED, 'Location name or id') ->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)') ->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)')
->addOption('force', null, InputOption::VALUE_NONE, 'Also on an installation that already has more than one user, and without asking')
->setHelp(<<<'HELP' ->setHelp(<<<'HELP'
For a FRESH Nextcloud installation only, same major version as the backup, right after For a FRESH Nextcloud installation only, same major version as the backup, right after
occ maintenance:install, occ backup:key:import and occ backup:target:add -- before anyone else occ maintenance:install, occ backup:key:import and occ backup:target:add -- before anyone else
@@ -50,12 +56,29 @@ HELP);
} }
protected function execute(InputInterface $input, OutputInterface $output): int { protected function execute(InputInterface $input, OutputInterface $output): int {
$force = (bool)$input->getOption('force');
// A whole-server restore replaces the database and data directory of THIS instance. On
// anything but a fresh install (only the admin from occ maintenance:install) that is
// almost certainly a mistake, so it needs --force.
$users = $this->userManager->countUsersTotal(2);
if (!$force && ($users === false || $users > 1)) {
$output->writeln('<error>This installation already has more than one user: backup:restore:full would replace its whole database and data directory. It is meant for a fresh installation. Use --force if this really is what you want.</error>');
return 1;
}
if (!$force && $input->isInteractive()) {
$q = new ConfirmationQuestion('This replaces the database, the data directory and config.php of this installation. Continue? [y/N] ', false);
if (!(new QuestionHelper())->ask($input, $output, $q)) {
$output->writeln('Cancelled.');
return 1;
}
}
try { try {
$target = $this->targets->get($input->getArgument('target')); $target = $this->targets->get($input->getArgument('target'));
$snapshot = (string)$input->getArgument('snapshot'); $snapshot = (string)$input->getArgument('snapshot');
$result = $this->restoreFull->restoreFull($target, $snapshot); $result = $this->restoreFull->restoreFull($target, $snapshot);
} catch (\Throwable $e) { } catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>'); $output->writeln('<error>' . $e->getMessage() . '</error>');
$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
return 1; return 1;
} }
$output->writeln(sprintf( $output->writeln(sprintf(
@@ -65,6 +88,9 @@ HELP);
$result['configKeys'], $result['configKeys'],
$result['dataFiles'], $result['dataFiles'],
)); ));
if (!$result['dbConsistent']) {
$output->writeln('<comment>Warning: this snapshot\'s database dump was made over several steps (web cron), so it is not one consistent point in time. Check the instance carefully; prefer a snapshot made with system cron.</comment>');
}
$output->writeln('Next: check trusted_domains and overwritehost/overwriteprotocol match this server, then occ maintenance:repair.'); $output->writeln('Next: check trusted_domains and overwritehost/overwriteprotocol match this server, then occ maintenance:repair.');
return 0; return 0;
} }
+21 -1
View File
@@ -10,7 +10,10 @@ namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\TargetService; use OCA\NgBackup\Service\TargetService;
use OCA\NgBackup\Service\UserRestoreService; use OCA\NgBackup\Service\UserRestoreService;
use OCA\NgBackup\UserMigration\UserMigrationInstaller; use OCA\NgBackup\UserMigration\UserMigrationInstaller;
use OCP\IUserManager;
use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\QuestionHelper;
use Symfony\Component\Console\Question\ConfirmationQuestion;
use Symfony\Component\Console\Input\InputArgument; use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface; use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption; use Symfony\Component\Console\Input\InputOption;
@@ -21,6 +24,7 @@ final class RestoreUser extends Command {
private TargetService $targets, private TargetService $targets,
private UserRestoreService $userRestore, private UserRestoreService $userRestore,
private UserMigrationInstaller $installer, private UserMigrationInstaller $installer,
private IUserManager $userManager,
) { ) {
parent::__construct(); parent::__construct();
} }
@@ -31,6 +35,7 @@ final class RestoreUser extends Command {
->addArgument('target', InputArgument::REQUIRED, 'Location name or id') ->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>)') ->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>)')
->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'replace or as-backup', UserRestoreService::MODE_REPLACE) ->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'replace or as-backup', UserRestoreService::MODE_REPLACE)
->addOption('force', null, InputOption::VALUE_NONE, 'Replace an existing account without asking (required when not interactive)')
->addOption('install-user-migration', null, InputOption::VALUE_NONE, 'Install and enable user_migration if it is missing (non-interactive consent)') ->addOption('install-user-migration', null, InputOption::VALUE_NONE, 'Install and enable user_migration if it is missing (non-interactive consent)')
->setHelp(<<<'HELP' ->setHelp(<<<'HELP'
Never imports over a live account. Modes: Never imports over a live account. Modes:
@@ -47,11 +52,26 @@ HELP);
} }
protected function execute(InputInterface $input, OutputInterface $output): int { protected function execute(InputInterface $input, OutputInterface $output): int {
$mode = (string)$input->getOption('mode');
$uid = explode('/', (string)$input->getArgument('manifest'))[1] ?? '';
if ($mode === UserRestoreService::MODE_REPLACE && $uid !== '' && $this->userManager->userExists($uid) && !$input->getOption('force')) {
// Replace deletes the live account (after a safety export). Shares to others and data of
// apps without a user_migration migrator do not come back, so never do it unasked.
if (!$input->isInteractive()) {
$output->writeln("<error>$uid exists: --mode=replace deletes and recreates it. Add --force, or use --mode=as-backup.</error>");
return 1;
}
$q = new ConfirmationQuestion("$uid exists and will be deleted and recreated from the backup (a safety export is made first; its shares to others and data of apps without a migrator are not restored). Continue? [y/N] ", false);
if (!(new QuestionHelper())->ask($input, $output, $q)) {
$output->writeln('Cancelled.');
return 1;
}
}
try { try {
$output->writeln('<comment>Note: on restore, user_migration only imports the app settings on its own allowlist (e.g. calendar view and reminder settings); other app settings of this user are skipped. Reason: NG Backup uses user_migration\'s importer; widening the list would mean building a settings importer of our own.</comment>'); $output->writeln('<comment>Note: on restore, user_migration only imports the app settings on its own allowlist (e.g. calendar view and reminder settings); other app settings of this user are skipped. Reason: NG Backup uses user_migration\'s importer; widening the list would mean building a settings importer of our own.</comment>');
$target = $this->targets->get($input->getArgument('target')); $target = $this->targets->get($input->getArgument('target'));
$this->installer->ensureEnabled($input, $output); $this->installer->ensureEnabled($input, $output);
$user = $this->userRestore->restoreUser($target, (string)$input->getArgument('manifest'), (string)$input->getOption('mode')); $user = $this->userRestore->restoreUser($target, (string)$input->getArgument('manifest'), $mode);
} catch (\Throwable $e) { } catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>'); $output->writeln('<error>' . $e->getMessage() . '</error>');
return 1; return 1;
+7
View File
@@ -75,6 +75,13 @@ HELP);
}); });
if ($run->getStatus() === Run::DONE) { if ($run->getStatus() === Run::DONE) {
$output->writeln(sprintf(' done in %.0fs', microtime(true) - $start)); $output->writeln(sprintf(' done in %.0fs', microtime(true) - $start));
$f = json_decode($run->getStats() ?? '{}', true)['files'] ?? [];
if (($f['unreadable'] ?? 0) > 0) {
$output->writeln(sprintf('<comment> warning: %d unreadable files or folders are missing from this backup (see the server log)</comment>', $f['unreadable']));
}
if (($f['changing'] ?? 0) > 0) {
$output->writeln(sprintf('<comment> warning: %d files kept changing and were saved as read (see the server log)</comment>', $f['changing']));
}
} else { } else {
$output->writeln('<error> ' . $run->getStatus() . ': ' . $run->getError() . '</error>'); $output->writeln('<error> ' . $run->getStatus() . ': ' . $run->getError() . '</error>');
$failed++; $failed++;
+7 -1
View File
@@ -66,7 +66,13 @@ HELP);
if ($result['failed'] !== []) { if ($result['failed'] !== []) {
$output->writeln('<error>' . count($result['failed']) . ' blob(s) failed verification (corrupted or tampered):</error> ' . implode(', ', $result['failed'])); $output->writeln('<error>' . count($result['failed']) . ' blob(s) failed verification (corrupted or tampered):</error> ' . implode(', ', $result['failed']));
} }
if ($result['missing'] === [] && $result['failed'] === []) { if ($result['unreadable'] !== []) {
$output->writeln('<error>' . count($result['unreadable']) . ' blob(s) could not be read from the location</error> (often a temporary storage '
. 'or network error: run the check again; if the same blobs stay unreadable, their pack may be damaged or truncated): '
. implode(', ', $result['unreadable']));
$output->writeln('Last read error: ' . $result['readError']);
}
if ($result['missing'] === [] && $result['failed'] === [] && $result['unreadable'] === []) {
$output->writeln('<info>OK</info>'); $output->writeln('<info>OK</info>');
return 0; return 0;
} }
+1
View File
@@ -168,6 +168,7 @@ final class AdminApiController extends Controller {
return new JSONResponse($this->targets->availableBackends()); return new JSONResponse($this->targets->availableBackends());
} }
#[PasswordConfirmationRequired]
public function addTarget(string $name, string $backend, string $auth, array $options = [], string $path = 'ng_backup', ?float $maxGb = null): JSONResponse { public function addTarget(string $name, string $backend, string $auth, array $options = [], string $path = 'ng_backup', ?float $maxGb = null): JSONResponse {
try { try {
$r = $this->targets->add($name, $backend, $auth, $options, $path); $r = $this->targets->add($name, $backend, $auth, $options, $path);
+6
View File
@@ -96,6 +96,12 @@ final class KeyRing {
/** @param array{v:int, salt:string, ops:int, mem:int, nonce:string, key:string} $wrapped */ /** @param array{v:int, salt:string, ops:int, mem:int, nonce:string, key:string} $wrapped */
public static function unwrap(array $wrapped, #[\SensitiveParameter] string $passphrase): self { public static function unwrap(array $wrapped, #[\SensitiveParameter] string $passphrase): self {
$b64 = static fn (string $s): string => sodium_base642bin($s, SODIUM_BASE64_VARIANT_ORIGINAL); $b64 = static fn (string $s): string => sodium_base642bin($s, SODIUM_BASE64_VARIANT_ORIGINAL);
// ops/mem come from unauthenticated JSON (kit file, repository config): bound them, so a
// damaged or crafted file cannot make the KDF use unbounded memory or time.
if ($wrapped['ops'] < SODIUM_CRYPTO_PWHASH_OPSLIMIT_INTERACTIVE || $wrapped['ops'] > SODIUM_CRYPTO_PWHASH_OPSLIMIT_SENSITIVE
|| $wrapped['mem'] < SODIUM_CRYPTO_PWHASH_MEMLIMIT_INTERACTIVE || $wrapped['mem'] > SODIUM_CRYPTO_PWHASH_MEMLIMIT_SENSITIVE) {
throw new CryptoException('Unsupported key derivation parameters (damaged or foreign key file)');
}
$kek = sodium_crypto_pwhash(SODIUM_CRYPTO_SECRETBOX_KEYBYTES, $passphrase, $b64($wrapped['salt']), $kek = sodium_crypto_pwhash(SODIUM_CRYPTO_SECRETBOX_KEYBYTES, $passphrase, $b64($wrapped['salt']),
$wrapped['ops'], $wrapped['mem'], SODIUM_CRYPTO_PWHASH_ALG_ARGON2ID13); $wrapped['ops'], $wrapped['mem'], SODIUM_CRYPTO_PWHASH_ALG_ARGON2ID13);
$master = sodium_crypto_secretbox_open($b64($wrapped['key']), $b64($wrapped['nonce']), $kek); $master = sodium_crypto_secretbox_open($b64($wrapped['key']), $b64($wrapped['nonce']), $kek);
+8
View File
@@ -122,6 +122,7 @@ final class StreamCipher {
/** fread() may return short reads on network streams; keep reading until $length or EOF. */ /** fread() may return short reads on network streams; keep reading until $length or EOF. */
private static function readFull($in, int $length): string { private static function readFull($in, int $length): string {
$buffer = ''; $buffer = '';
$empty = 0;
while (strlen($buffer) < $length && !feof($in)) { while (strlen($buffer) < $length && !feof($in)) {
$chunk = fread($in, $length - strlen($buffer)); $chunk = fread($in, $length - strlen($buffer));
if ($chunk === false) { if ($chunk === false) {
@@ -132,8 +133,15 @@ final class StreamCipher {
if (feof($in)) { if (feof($in)) {
break; break;
} }
// A stalled network stream returns '' without EOF forever: back off, then give up
// (about a minute) instead of spinning at full CPU until the lease expires.
if (++$empty > 600) {
throw new CryptoException('Read stalled');
}
usleep(100000);
continue; continue;
} }
$empty = 0;
$buffer .= $chunk; $buffer .= $chunk;
} }
return $buffer; return $buffer;
+5 -1
View File
@@ -256,7 +256,11 @@ final class DbDumper {
} elseif (is_string($row[$i]) && !mb_check_encoding($row[$i], 'UTF-8')) { } elseif (is_string($row[$i]) && !mb_check_encoding($row[$i], 'UTF-8')) {
// Keep invalid UTF-8 byte-exact instead of substituting characters. // Keep invalid UTF-8 byte-exact instead of substituting characters.
$row[$i] = ['$b' => base64_encode($row[$i])]; $row[$i] = ['$b' => base64_encode($row[$i])];
} elseif (is_int($row[$i]) || is_float($row[$i]) || is_bool($row[$i])) { } elseif (is_float($row[$i])) {
// var_export uses serialize_precision (shortest exact form), not the lossy
// 14-digit 'precision' of a (string) cast.
$row[$i] = var_export($row[$i], true);
} elseif (is_int($row[$i]) || is_bool($row[$i])) {
// Normalise: drivers differ in returning numbers as int or string. // Normalise: drivers differ in returning numbers as int or string.
$row[$i] = (string)(is_bool($row[$i]) ? (int)$row[$i] : $row[$i]); $row[$i] = (string)(is_bool($row[$i]) ? (int)$row[$i] : $row[$i]);
} }
+11
View File
@@ -43,6 +43,17 @@ final class RunMapper extends QBMapper {
return $this->findEntities($qb); return $this->findEntities($qb);
} }
/** @return list<Run> runs of $kind started at or after $since, newest first */
public function startedSince(int $targetId, string $kind, int $since): array {
$qb = $this->db->getQueryBuilder();
$qb->select('*')->from($this->getTableName())
->where($qb->expr()->eq('target_id', $qb->createNamedParameter($targetId, IQueryBuilder::PARAM_INT)))
->andWhere($qb->expr()->eq('kind', $qb->createNamedParameter($kind)))
->andWhere($qb->expr()->gte('started_at', $qb->createNamedParameter($since, IQueryBuilder::PARAM_INT)))
->orderBy('id', 'DESC');
return $this->findEntities($qb);
}
public function lastFinished(int $targetId, string $kind): ?Run { public function lastFinished(int $targetId, string $kind): ?Run {
$qb = $this->db->getQueryBuilder(); $qb = $this->db->getQueryBuilder();
$qb->select('*')->from($this->getTableName()) $qb->select('*')->from($this->getTableName())
+74 -14
View File
@@ -49,9 +49,11 @@ final class BackupRun {
public static function step(Repository $repo, array $state, float $deadline): array { public static function step(Repository $repo, array $state, float $deadline): array {
$state['steps']++; $state['steps']++;
if ($state['phase'] === 'files') { if ($state['phase'] === 'files') {
$walker = new TreeWalker($state['roots'], $state['exclude']);
$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
$stats = $state['stats']; $stats = $state['stats'];
$walker = new TreeWalker($state['roots'], $state['exclude'], function (string $logical) use (&$state, &$stats): void {
self::recordUnreadable($state, $stats, $logical);
});
$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
$lines = []; $lines = [];
$finishedWalk = true; $finishedWalk = true;
@@ -68,15 +70,20 @@ final class BackupRun {
$stats['reused']++; $stats['reused']++;
} else { } else {
$cur = $state['cur']; $cur = $state['cur'];
if ($cur === null || $cur['path'] !== $logical || $cur['version'] !== $version) { if ($cur !== null && $cur['path'] === $logical && $cur['version'] !== $version) {
if ($cur !== null && $cur['path'] === $logical) { // Changed since the previous step: start over, unless it keeps changing.
$stats['retried'] = ($stats['retried'] ?? 0) + 1; // changed since the previous step if (self::countRetry($state, $stats, $logical)) {
$cur['version'] = $version;
} else {
$cur = null;
} }
}
if ($cur === null || $cur['path'] !== $logical) {
$cur = ['path' => $logical, 'offset' => 0, 'blobs' => [], 'version' => $version]; $cur = ['path' => $logical, 'offset' => 0, 'blobs' => [], 'version' => $version];
} }
$fh = fopen($path, 'rb'); $fh = @fopen($path, 'rb');
if ($fh === false) { if ($fh === false) {
$stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1; self::recordUnreadable($state, $stats, $logical);
$state['cur'] = null; $state['cur'] = null;
continue; continue;
} }
@@ -91,12 +98,17 @@ final class BackupRun {
} }
} }
fclose($fh); fclose($fh);
if (Repository::fileVersion($path) !== $cur['version']) { $now = Repository::fileVersion($path);
// Changed while reading: read it again from the start (now or next step). if ($now !== $cur['version']) {
$stats['retried'] = ($stats['retried'] ?? 0) + 1; if (!self::countRetry($state, $stats, $logical)) {
$state['cur'] = null; // Changed while reading: read it again from the start (now or next step).
$finishedWalk = false; $state['cur'] = null;
break; $finishedWalk = false;
break;
}
// Keeps changing (a live database, VM image or log): keep what was read,
// like any copy of a file in use, and report it instead of retrying forever.
$cur['version'] = $now;
} }
if (!$complete) { if (!$complete) {
$state['cur'] = $cur; $state['cur'] = $cur;
@@ -104,6 +116,7 @@ final class BackupRun {
break; break;
} }
$blobs = $cur['blobs']; $blobs = $cur['blobs'];
$size = $cur['offset'];
$state['cur'] = null; $state['cur'] = null;
$stats['read']++; $stats['read']++;
} }
@@ -135,7 +148,8 @@ final class BackupRun {
$segments = $state['segments']; $segments = $state['segments'];
$repo->putSnapshot($state['snapshot'], $repo->putSnapshot($state['snapshot'],
['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'], ['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
'roots' => array_keys($state['roots']), 'stats' => $state['stats']] + $state['meta'], 'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
'warnings' => self::warnings($state)] + $state['meta'],
(function () use ($repo, $segments) { (function () use ($repo, $segments) {
foreach ($segments as $blobs) { foreach ($segments as $blobs) {
yield from $repo->readLines($blobs); yield from $repo->readLines($blobs);
@@ -146,6 +160,52 @@ final class BackupRun {
return $state; return $state;
} }
/**
* A file or directory that could not be read: it is missing from this snapshot. Remembered (first
* 1000, deduplicated across steps) so the run can report it instead of silently succeeding.
*/
private static function recordUnreadable(array &$state, array &$stats, string $logical): void {
if (isset($state['unreadable'][$logical])) {
return;
}
if (count($state['unreadable'] ?? []) < 1000) {
$state['unreadable'][$logical] = true;
}
$stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1;
}
/** @return array{unreadable: list<string>, changing: list<string>} examples for the report (first 50 each) */
public static function warnings(array $state): array {
return [
'unreadable' => array_slice(array_map('strval', array_keys($state['unreadable'] ?? [])), 0, 50),
'changing' => array_slice(array_map('strval', array_values($state['changing'] ?? [])), 0, 50),
];
}
/** Retries of one file that changes while it is read, before its last read is kept anyway. */
public const MAX_RETRIES = 3;
/**
* Count a retry of $logical. Returns true once it has changed MAX_RETRIES times: the caller then
* keeps the data it read, and the file is listed in $state['changing'] (first 50) for the report.
*/
private static function countRetry(array &$state, array &$stats, string $logical): bool {
$stats['retried'] = ($stats['retried'] ?? 0) + 1;
$n = (($state['retryPath'] ?? null) === $logical ? (int)($state['retryCount'] ?? 0) : 0) + 1;
$state['retryPath'] = $logical;
$state['retryCount'] = $n;
if ($n < self::MAX_RETRIES) {
return false;
}
if ($n === self::MAX_RETRIES) {
$stats['changing'] = ($stats['changing'] ?? 0) + 1;
if (count($state['changing'] ?? []) < 50) {
$state['changing'][] = $logical;
}
}
return true;
}
private static function readFull($fh, int $length): string { private static function readFull($fh, int $length): string {
$buffer = ''; $buffer = '';
while (strlen($buffer) < $length && !feof($fh)) { while (strlen($buffer) < $length && !feof($fh)) {
+5
View File
@@ -23,6 +23,8 @@ final class TreeWalker {
public function __construct( public function __construct(
private array $roots, private array $roots,
private array $exclude = [], private array $exclude = [],
/** Called with the logical path of a directory that cannot be listed. */
private ?\Closure $onUnreadable = null,
) { ) {
} }
@@ -57,6 +59,9 @@ final class TreeWalker {
} }
$names = @scandir($abs); $names = @scandir($abs);
if ($names === false) { if ($names === false) {
if ($this->onUnreadable !== null) {
($this->onUnreadable)($logical === '' ? '/' : $logical);
}
return; return;
} }
$names = array_values(array_diff($names, ['.', '..'])); $names = array_values(array_diff($names, ['.', '..']));
+6
View File
@@ -42,6 +42,10 @@ final class Notifier implements INotifier {
: $l->t('Backup to %s failed', [$p['target']])); : $l->t('Backup to %s failed', [$p['target']]));
$notification->setParsedMessage((string)($p['error'] ?? '')); $notification->setParsedMessage((string)($p['error'] ?? ''));
break; break;
case 'run_warnings':
$notification->setParsedSubject($l->t('Backup to %1$s finished with warnings: %2$s unreadable, %3$s changed while being read', [$p['target'], $p['unreadable'], $p['changing']]));
$notification->setParsedMessage($l->t('Unreadable files and folders are missing from this backup (check their permissions). Files that kept changing were saved as they were while being read. Examples: %s', [$p['examples'] ?? '']));
break;
case 'quota_warning': case 'quota_warning':
$notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']])); $notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']]));
$notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.')); $notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.'));
@@ -62,6 +66,8 @@ final class Notifier implements INotifier {
'catalog_trusted' => $l->t('%1$s accepted an older state of backup location %2$s (catalog generation %3$s)', [$who, $p['target'] ?? '', $p['generation'] ?? '']), 'catalog_trusted' => $l->t('%1$s accepted an older state of backup location %2$s (catalog generation %3$s)', [$who, $p['target'] ?? '', $p['generation'] ?? '']),
'slot_added' => $l->t('%1$s added backup passphrase slot %2$s (%3$s); download and confirm the new recovery kit', [$who, $p['slot'] ?? '', $p['label'] ?? '']), 'slot_added' => $l->t('%1$s added backup passphrase slot %2$s (%3$s); download and confirm the new recovery kit', [$who, $p['slot'] ?? '', $p['label'] ?? '']),
'slot_replaced' => $l->t('%1$s set a new passphrase for backup key slot %2$s (%3$s); download and confirm the new recovery kit', [$who, $p['slot'] ?? '', $p['label'] ?? '']), 'slot_replaced' => $l->t('%1$s set a new passphrase for backup key slot %2$s (%3$s); download and confirm the new recovery kit', [$who, $p['slot'] ?? '', $p['label'] ?? '']),
'user_export_retention_changed' => $l->t('%1$s changed how many exports per user are kept to %2$s', [$who, $p['keep'] ?? '']),
'user_exports_forgotten' => $l->t('%1$s removed %2$s per-user exports from %3$s', [$who, $p['count'] ?? '', $p['target'] ?? '']),
'slot_removed' => $l->t('%1$s removed backup passphrase slot %2$s; download and confirm the new recovery kit', [$who, $p['slot'] ?? '']), 'slot_removed' => $l->t('%1$s removed backup passphrase slot %2$s; download and confirm the new recovery kit', [$who, $p['slot'] ?? '']),
default => $l->t('NG Backup security event'), default => $l->t('NG Backup security event'),
}); });
+5
View File
@@ -88,6 +88,11 @@ final class Catalog {
/** Write the next generation after the in-memory state was changed. */ /** Write the next generation after the in-memory state was changed. */
public function write(): void { public function write(): void {
$next = $this->gen + 1; $next = $this->gen + 1;
// Two writers starting from the same generation would otherwise overwrite each other
// silently and break the chain for whichever server anchored the lost write.
if ($this->backend->exists($this->path($next))) {
throw new RepositoryException("Catalog generation $next was written by another process at the same time; try again");
}
$plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash, $plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash,
'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash), 'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash),
'userExports' => array_keys($this->userExports)], JSON_THROW_ON_ERROR); 'userExports' => array_keys($this->userExports)], JSON_THROW_ON_ERROR);
+78 -16
View File
@@ -28,6 +28,8 @@ final class Repository {
public const FORMAT = 1; public const FORMAT = 1;
public const BLOB_SIZE = 4 * 1048576; public const BLOB_SIZE = 4 * 1048576;
private const FLAG_DEFLATE = 1; private const FLAG_DEFLATE = 1;
/** Reads per blob in a deep verify before it is reported as unreadable (not corrupted). */
private const VERIFY_READ_ATTEMPTS = 3;
private StreamCipher $cipher; private StreamCipher $cipher;
private BlobIndex $index; private BlobIndex $index;
@@ -236,8 +238,13 @@ final class Repository {
unlink($tmp); unlink($tmp);
throw new RepositoryException('Size mismatch restoring ' . $entry['p']); throw new RepositoryException('Size mismatch restoring ' . $entry['p']);
} }
rename($tmp, $dest); if (!rename($tmp, $dest)) {
touch($dest, $entry['m']); @unlink($tmp);
throw new RepositoryException('Cannot put the restored file in place: ' . $entry['p']);
}
if (!touch($dest, $entry['m'])) {
throw new RepositoryException('Cannot set the modification time of ' . $entry['p']);
}
$count++; $count++;
} }
return $count; return $count;
@@ -375,31 +382,61 @@ final class Repository {
// must be decrypted and re-encrypted under the new path, not moved as raw bytes. // must be decrypted and re-encrypted under the new path, not moved as raw bytes.
$dbInTrash = 'trash/' . $meta['db']; $dbInTrash = 'trash/' . $meta['db'];
$this->putObject($dbInTrash, $this->getObject($meta['db'])); $this->putObject($dbInTrash, $this->getObject($meta['db']));
$this->backend->delete($meta['db']);
} }
// The snapshot object is bound to its id (not its path), so it can be moved as raw bytes. // Copy first, switch the catalog, delete the originals last: a crash at any point leaves
$this->backend->move('snapshots/' . $snapshotId, 'trash/snapshots/' . $snapshotId); // either the old state or the new one plus unreferenced leftovers, never a catalog entry
// pointing at a missing object. The snapshot object is bound to its id (not its path),
// so it can be copied as raw bytes.
$this->copyRaw('snapshots/' . $snapshotId, 'trash/snapshots/' . $snapshotId);
$this->putObject('trash/info/' . $snapshotId, json_encode(['forgottenAt' => time(), 'by' => $by, 'db' => $meta['db'] ?? null, $this->putObject('trash/info/' . $snapshotId, json_encode(['forgottenAt' => time(), 'by' => $by, 'db' => $meta['db'] ?? null,
'dbInTrash' => $dbInTrash, 'time' => $meta['time'] ?? null, 'label' => $meta['label'] ?? ''], JSON_THROW_ON_ERROR)); 'dbInTrash' => $dbInTrash, 'time' => $meta['time'] ?? null, 'label' => $meta['label'] ?? ''], JSON_THROW_ON_ERROR));
unset($this->catalog->snapshots[$snapshotId]); unset($this->catalog->snapshots[$snapshotId]);
$this->catalog->trash[$snapshotId] = true; $this->catalog->trash[$snapshotId] = true;
$this->catalog->write(); $this->catalog->write();
if ($dbInTrash !== null) {
$this->backend->delete($meta['db']);
}
$this->backend->delete('snapshots/' . $snapshotId);
} }
/** Undo forget(): move a snapshot back from the trash. */ /** Undo forget(): move a snapshot back from the trash. */
public function untrash(string $snapshotId): void { public function untrash(string $snapshotId): void {
$info = $this->trashInfo($snapshotId); $info = $this->trashInfo($snapshotId);
if ($info['dbInTrash'] !== null && $this->backend->exists($info['dbInTrash'])) { $restoreDb = $info['dbInTrash'] !== null && $this->backend->exists($info['dbInTrash']);
if ($restoreDb) {
$this->putObject($info['db'], $this->getObject($info['dbInTrash'])); $this->putObject($info['db'], $this->getObject($info['dbInTrash']));
$this->backend->delete($info['dbInTrash']);
} }
$this->backend->move('trash/snapshots/' . $snapshotId, 'snapshots/' . $snapshotId); // Same order as forget(): copy, switch the catalog, then delete the trash copies.
$this->copyRaw('trash/snapshots/' . $snapshotId, 'snapshots/' . $snapshotId);
$this->catalog->snapshots[$snapshotId] = true; $this->catalog->snapshots[$snapshotId] = true;
unset($this->catalog->trash[$snapshotId]); unset($this->catalog->trash[$snapshotId]);
$this->catalog->write(); $this->catalog->write();
if ($restoreDb) {
$this->backend->delete($info['dbInTrash']);
}
$this->backend->delete('trash/snapshots/' . $snapshotId);
$this->backend->delete('trash/info/' . $snapshotId); $this->backend->delete('trash/info/' . $snapshotId);
} }
/** Copy an object as raw bytes, buffered locally (remote read streams cannot be uploaded directly everywhere). */
private function copyRaw(string $from, string $to): void {
$in = $this->backend->get($from);
$buffer = fopen('php://temp/maxmemory:' . (4 * 1048576), 'w+b');
if ($buffer === false) {
fclose($in);
throw new RepositoryException("Cannot buffer $from");
}
try {
if (stream_copy_to_stream($in, $buffer) === false) {
throw new RepositoryException("Cannot read $from");
}
} finally {
fclose($in);
}
rewind($buffer);
$this->backend->put($to, $buffer);
}
/** Record a user_migration export's manifest in the catalog (rollback-protected like a snapshot). */ /** Record a user_migration export's manifest in the catalog (rollback-protected like a snapshot). */
public function recordUserExport(string $manifestPath): void { public function recordUserExport(string $manifestPath): void {
$this->catalog->userExports[$manifestPath] = true; $this->catalog->userExports[$manifestPath] = true;
@@ -530,7 +567,10 @@ final class Repository {
} }
} }
} }
foreach ($this->backend->list('users') as $path) { // listUserExports() throws when a recorded export's manifest is missing: then stop, since
// freeing its blobs would turn a tampering or storage alarm into irreversible deletion.
$exports = array_unique(array_merge($this->listUserExports(), $this->backend->list('users')));
foreach ($exports as $path) {
foreach (json_decode($this->getObject($path), true, 512, JSON_THROW_ON_ERROR)['entries'] as $e) { foreach (json_decode($this->getObject($path), true, 512, JSON_THROW_ON_ERROR)['entries'] as $e) {
foreach ($e['b'] ?? [] as $b) { foreach ($e['b'] ?? [] as $b) {
$used[$b] = true; $used[$b] = true;
@@ -658,8 +698,13 @@ final class Repository {
} }
private function loadBlob(string $id): string { private function loadBlob(string $id): string {
[$pack, $offset, $length, $raw, $flags] = $this->index->get($id); [$pack, $offset, $length] = $this->index->get($id);
$encrypted = $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length); return $this->decodeBlob($id, $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length));
}
/** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */
private function decodeBlob(string $id, string $encrypted): string {
[, , , $raw, $flags] = $this->index->get($id);
$payload = $this->cipher->decryptString($encrypted, 'blob:' . $id); $payload = $this->cipher->decryptString($encrypted, 'blob:' . $id);
$data = ($flags & self::FLAG_DEFLATE) ? gzinflate($payload) : $payload; $data = ($flags & self::FLAG_DEFLATE) ? gzinflate($payload) : $payload;
if ($data === false || strlen($data) !== $raw || $this->keys->blobId($data) !== $id) { if ($data === false || strlen($data) !== $raw || $this->keys->blobId($data) !== $id) {
@@ -695,13 +740,13 @@ final class Repository {
* *
* @param ?callable $heartbeat called after every blob (shallow) or every blob read (deep); * @param ?callable $heartbeat called after every blob (shallow) or every blob read (deep);
* callers use it to keep a time-boxed lease alive during a large audit * callers use it to keep a time-boxed lease alive during a large audit
* @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list<string>, failed:list<string>} * @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list<string>, failed:list<string>, unreadable:list<string>, readError:string}
*/ */
public function verify(string $snapshotId, bool $deep = false, ?callable $heartbeat = null): array { public function verify(string $snapshotId, bool $deep = false, ?callable $heartbeat = null): array {
$heartbeat ??= static function (): void { $heartbeat ??= static function (): void {
}; };
$this->index->load(true); $this->index->load(true);
$stats = ['filesChecked' => 0, 'blobsChecked' => 0, 'bytesChecked' => 0, 'missing' => [], 'failed' => []]; $stats = ['filesChecked' => 0, 'blobsChecked' => 0, 'bytesChecked' => 0, 'missing' => [], 'failed' => [], 'unreadable' => [], 'readError' => ''];
$seen = []; $seen = [];
$packSeen = []; // pack id => exists on the backend; avoids one exists() call per blob $packSeen = []; // pack id => exists on the backend; avoids one exists() call per blob
foreach ($this->streamEntries($snapshotId) as $entry) { foreach ($this->streamEntries($snapshotId) as $entry) {
@@ -740,7 +785,7 @@ final class Repository {
* Same checks as verify(), for a user export: every blob its manifest references must be in * Same checks as verify(), for a user export: every blob its manifest references must be in
* the freshly re-authenticated index with its pack present; $deep also decrypts each. * the freshly re-authenticated index with its pack present; $deep also decrypts each.
* *
* @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list<string>, failed:list<string>} * @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list<string>, failed:list<string>, unreadable:list<string>, readError:string}
*/ */
public function verifyUserExport(string $manifestPath, bool $deep = false, ?callable $heartbeat = null): array { public function verifyUserExport(string $manifestPath, bool $deep = false, ?callable $heartbeat = null): array {
if (!$this->isRecordedUserExport($manifestPath)) { if (!$this->isRecordedUserExport($manifestPath)) {
@@ -749,7 +794,7 @@ final class Repository {
$heartbeat ??= static function (): void { $heartbeat ??= static function (): void {
}; };
$this->index->load(true); $this->index->load(true);
$stats = ['filesChecked' => 0, 'blobsChecked' => 0, 'bytesChecked' => 0, 'missing' => [], 'failed' => []]; $stats = ['filesChecked' => 0, 'blobsChecked' => 0, 'bytesChecked' => 0, 'missing' => [], 'failed' => [], 'unreadable' => [], 'readError' => ''];
$seen = []; $seen = [];
$packSeen = []; $packSeen = [];
$manifest = json_decode($this->getObject($manifestPath), true, 512, JSON_THROW_ON_ERROR); $manifest = json_decode($this->getObject($manifestPath), true, 512, JSON_THROW_ON_ERROR);
@@ -786,8 +831,25 @@ final class Repository {
if (!$deep) { if (!$deep) {
return; return;
} }
// Reading and checking are kept apart: a read that fails is usually a network or
// storage error, so it is retried and reported as unreadable (a pack that stays
// unreadable may still be truncated). Only a blob that was read but does not
// decrypt or match its id counts as failed: that is proof of damage.
[, $offset, $length] = $this->index->get($id);
$encrypted = null;
for ($attempt = 1; $attempt <= self::VERIFY_READ_ATTEMPTS && $encrypted === null; $attempt++) {
try {
$encrypted = $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
} catch (\Exception $e) { // storage adapters throw their own exceptions; \Error (a bug) is not retried
$stats['readError'] = $e->getMessage();
}
}
if ($encrypted === null) {
$stats['unreadable'][] = $id;
return;
}
try { try {
$stats['bytesChecked'] += strlen($this->loadBlob($id)); $stats['bytesChecked'] += strlen($this->decodeBlob($id, $encrypted));
} catch (\Throwable) { } catch (\Throwable) {
$stats['failed'][] = $id; $stats['failed'][] = $id;
} }
+7
View File
@@ -54,6 +54,13 @@ final class AlertService {
['kind' => $run->getKind(), 'target' => $targetName, 'error' => mb_strimwidth((string)$run->getError(), 0, 300, '…')]); ['kind' => $run->getKind(), 'target' => $targetName, 'error' => mb_strimwidth((string)$run->getError(), 0, 300, '…')]);
} }
/** A backup that finished but is missing unreadable files, or holds files that changed while read. */
public function runWarnings(Run $run, string $targetName, int $unreadable, int $changing, array $examples): void {
$this->notifyAdmins('run_warnings', 'run-' . $run->getId(), ['target' => $targetName,
'unreadable' => (string)$unreadable, 'changing' => (string)$changing,
'examples' => mb_strimwidth(implode(', ', array_merge($examples['unreadable'], $examples['changing'])), 0, 300, '…')]);
}
/** /**
* Remember how much is stored on a location (for the settings page) and warn administrators * Remember how much is stored on a location (for the settings page) and warn administrators
* once a day when it is at 80% of its limit or more. * once a day when it is at 80% of its limit or more.
+13 -3
View File
@@ -14,6 +14,7 @@ use OCA\NgBackup\Db\Snapshot;
use OCA\NgBackup\Db\SnapshotMapper; use OCA\NgBackup\Db\SnapshotMapper;
use OCA\NgBackup\Db\Target; use OCA\NgBackup\Db\Target;
use OCA\NgBackup\Job\BackupRun; use OCA\NgBackup\Job\BackupRun;
use OCA\NgBackup\Repository\RepositoryException;
use OCP\App\IAppManager; use OCP\App\IAppManager;
use OCP\IConfig; use OCP\IConfig;
use OCP\IDBConnection; use OCP\IDBConnection;
@@ -151,16 +152,25 @@ final class BackupService {
// Catalog write under the catalog lock, from a repository opened inside it: a per-user // Catalog write under the catalog lock, from a repository opened inside it: a per-user
// export or restore may have written the catalog since this step opened $repo. // export or restore may have written the catalog since this step opened $repo.
try { try {
$this->leases->with(PruneService::catalogLockKey($target), LeaseService::EXCLUSIVE, 30, $this->leases->with(PruneService::catalogLockKey($target), LeaseService::EXCLUSIVE, 300,
fn () => $this->targets->repository($target)->recordSnapshot((string)$state['files']['snapshot'])); fn () => $this->targets->repository($target)->recordSnapshot((string)$state['files']['snapshot']));
$state['files']['phase'] = 'done'; $state['files']['phase'] = 'done';
} catch (LockedException) { } catch (LockedException|RepositoryException) {
// Another catalog write is in progress: the next step retries this commit. // Another catalog write is in progress (or just raced us to the next
// generation): the next step retries this commit on the fresh catalog.
} }
} }
if ($state['files']['phase'] === 'done') { if ($state['files']['phase'] === 'done') {
$this->recordSnapshot($run, $state['files']); $this->recordSnapshot($run, $state['files']);
$this->alerts->storedBytes($target, $repo->storedBytes()); $this->alerts->storedBytes($target, $repo->storedBytes());
$stats = $state['files']['stats'];
if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) {
$warnings = BackupRun::warnings($state['files']);
$this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s',
$run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']),
$stats['changing'] ?? 0, implode(', ', $warnings['changing'])));
$this->alerts->runWarnings($run, $target->getName(), (int)($stats['unreadable'] ?? 0), (int)($stats['changing'] ?? 0), $warnings);
}
$run->setStatus(Run::DONE); $run->setStatus(Run::DONE);
$run->setPhase('done'); $run->setPhase('done');
$run->setFinishedAt(time()); $run->setFinishedAt(time());
+4 -2
View File
@@ -64,7 +64,7 @@ final class RestoreFullService {
]; ];
} }
/** @return array{dbReport:array, dataFiles:int, configKeys:int} */ /** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool} */
public function restoreFull(Target $target, string $snapshotId): array { public function restoreFull(Target $target, string $snapshotId): array {
$check = $this->checkCompatibility($target, $snapshotId); $check = $this->checkCompatibility($target, $snapshotId);
if (!$check['ok']) { if (!$check['ok']) {
@@ -76,6 +76,7 @@ final class RestoreFullService {
$meta = $repo->snapshotMeta($snapshotId); $meta = $repo->snapshotMeta($snapshotId);
$dbReport = null; $dbReport = null;
$manifest = [];
if (isset($meta['db'])) { if (isset($meta['db'])) {
$manifest = json_decode($repo->getObject($meta['db']), true, 512, JSON_THROW_ON_ERROR); $manifest = json_decode($repo->getObject($meta['db']), true, 512, JSON_THROW_ON_ERROR);
$prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_'); $prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_');
@@ -124,7 +125,8 @@ final class RestoreFullService {
); );
} }
return ['dbReport' => $dbReport ?? ['skipped' => [], 'restored' => []], 'dataFiles' => $dataFiles, 'configKeys' => count($backupConfig)]; return ['dbReport' => $dbReport ?? ['skipped' => [], 'restored' => []], 'dataFiles' => $dataFiles, 'configKeys' => count($backupConfig),
'dbConsistent' => (bool)($manifest['consistent'] ?? true)];
}); });
} }
+1 -1
View File
@@ -58,7 +58,7 @@ final class UserBackupService {
private function withCatalogLock(Target $target, callable $fn): mixed { private function withCatalogLock(Target $target, callable $fn): mixed {
try { try {
return $this->leases->with(PruneService::catalogLockKey($target), LeaseService::EXCLUSIVE, 30, $fn); return $this->leases->with(PruneService::catalogLockKey($target), LeaseService::EXCLUSIVE, 300, $fn);
} catch (LockedException) { } catch (LockedException) {
throw new \RuntimeException('Another catalog write for ' . $target->getName() . ' is in progress; try again'); throw new \RuntimeException('Another catalog write for ' . $target->getName() . ' is in progress; try again');
} }
+1 -1
View File
@@ -129,7 +129,7 @@ final class UserRestoreService {
private function withCatalogLock(Target $target, callable $fn): mixed { private function withCatalogLock(Target $target, callable $fn): mixed {
try { try {
return $this->leases->with(PruneService::catalogLockKey($target), LeaseService::EXCLUSIVE, 30, $fn); return $this->leases->with(PruneService::catalogLockKey($target), LeaseService::EXCLUSIVE, 300, $fn);
} catch (LockedException) { } catch (LockedException) {
throw new \RuntimeException('Another catalog write for ' . $target->getName() . ' is in progress; try again'); throw new \RuntimeException('Another catalog write for ' . $target->getName() . ' is in progress; try again');
} }
+2 -2
View File
@@ -18,12 +18,12 @@ final class VerifyService {
) { ) {
} }
/** @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list<string>, failed:list<string>} */ /** @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list<string>, failed:list<string>, unreadable:list<string>, readError:string} */
public function verify(Target $target, string $snapshotId, bool $deep = false): array { public function verify(Target $target, string $snapshotId, bool $deep = false): array {
return $this->run($target, $deep, fn (\OCA\NgBackup\Repository\Repository $repo, callable $heartbeat): array => $repo->verify($snapshotId, $deep, $heartbeat)); return $this->run($target, $deep, fn (\OCA\NgBackup\Repository\Repository $repo, callable $heartbeat): array => $repo->verify($snapshotId, $deep, $heartbeat));
} }
/** @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list<string>, failed:list<string>} */ /** @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list<string>, failed:list<string>, unreadable:list<string>, readError:string} */
public function verifyUserExport(Target $target, string $manifestPath, bool $deep = false): array { public function verifyUserExport(Target $target, string $manifestPath, bool $deep = false): array {
return $this->run($target, $deep, fn (\OCA\NgBackup\Repository\Repository $repo, callable $heartbeat): array => $repo->verifyUserExport($manifestPath, $deep, $heartbeat)); return $this->run($target, $deep, fn (\OCA\NgBackup\Repository\Repository $repo, callable $heartbeat): array => $repo->verifyUserExport($manifestPath, $deep, $heartbeat));
} }
-5
View File
@@ -224,14 +224,9 @@
<MissingParamType> <MissingParamType>
<code><![CDATA[$fh]]></code> <code><![CDATA[$fh]]></code>
</MissingParamType> </MissingParamType>
<PossiblyNullIterator>
<code><![CDATA[$segments]]></code>
</PossiblyNullIterator>
<PossiblyUndefinedArrayOffset> <PossiblyUndefinedArrayOffset>
<code><![CDATA[$mtime]]></code> <code><![CDATA[$mtime]]></code>
<code><![CDATA[$state['cur']]]></code>
<code><![CDATA[$state['phase']]]></code> <code><![CDATA[$state['phase']]]></code>
<code><![CDATA[$state['segments']]]></code>
<code><![CDATA[$state['stats']]]></code> <code><![CDATA[$state['stats']]]></code>
</PossiblyUndefinedArrayOffset> </PossiblyUndefinedArrayOffset>
</file> </file>
Binary file not shown.

After

Width:  |  Height:  |  Size: 221 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 238 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 226 KiB

+2 -1
View File
@@ -82,6 +82,7 @@ onBeforeUnmount(() => clearTimeout(timer))
.ngb-tabs { .ngb-tabs {
display: flex; display: flex;
gap: 8px; gap: 8px;
margin: 0 0 12px; /* Same inline margin as NcSettingsSection, so the tabs line up with the section content */
margin: 0 calc(var(--default-grid-baseline) * 7) 12px;
} }
</style> </style>
+27 -24
View File
@@ -30,30 +30,32 @@
</div> </div>
</template> </template>
<table v-if="status.runs.length" class="ngb-table"> <div v-if="status.runs.length" class="ngb-table-wrap">
<thead> <table class="ngb-table">
<tr> <thead>
<th>{{ t('ng_backup', 'Started') }}</th> <tr>
<th>{{ t('ng_backup', 'Location') }}</th> <th>{{ t('ng_backup', 'Started') }}</th>
<th>{{ t('ng_backup', 'Kind') }}</th> <th>{{ t('ng_backup', 'Location') }}</th>
<th>{{ t('ng_backup', 'Restore point') }}</th> <th>{{ t('ng_backup', 'Kind') }}</th>
<th>{{ t('ng_backup', 'Status') }}</th> <th>{{ t('ng_backup', 'Restore point') }}</th>
<th>{{ t('ng_backup', 'Files') }}</th> <th>{{ t('ng_backup', 'Status') }}</th>
<th>{{ t('ng_backup', 'Uploaded') }}</th> <th>{{ t('ng_backup', 'Files') }}</th>
</tr> <th>{{ t('ng_backup', 'Uploaded') }}</th>
</thead> </tr>
<tbody> </thead>
<tr v-for="run in status.runs" :key="run.id" :title="run.error || ''"> <tbody>
<td>{{ new Date(run.startedAt * 1000).toLocaleString() }}</td> <tr v-for="run in status.runs" :key="run.id" :title="run.error || ''">
<td>{{ targetName(run.targetId) }}</td> <td>{{ new Date(run.startedAt * 1000).toLocaleString() }}</td>
<td>{{ run.kind }}</td> <td>{{ targetName(run.targetId) }}</td>
<td>{{ run.snapshot ? new Date(run.snapshot.createdAt * 1000).toLocaleString() + (run.snapshot.label ? ' – ' + run.snapshot.label : '') : '' }}</td> <td>{{ run.kind }}</td>
<td :class="'ngb-' + run.status">{{ run.status }}{{ run.status === 'running' ? ' (' + run.phase + ')' : '' }}{{ run.error ? ': ' + run.error : '' }}</td> <td>{{ run.snapshot ? new Date(run.snapshot.createdAt * 1000).toLocaleString() + (run.snapshot.label ? ' – ' + run.snapshot.label : '') : '' }}</td>
<td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td> <td :class="'ngb-' + run.status">{{ run.status }}{{ run.status === 'running' ? ' (' + run.phase + ')' : '' }}{{ run.error ? ': ' + run.error : '' }}</td>
<td>{{ run.stats?.files ? formatSize(run.stats.files.uploaded) : '' }}</td> <td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
</tr> <td>{{ run.stats?.files ? formatSize(run.stats.files.uploaded) : '' }}</td>
</tbody> </tr>
</table> </tbody>
</table>
</div>
</NcSettingsSection> </NcSettingsSection>
</template> </template>
@@ -126,6 +128,7 @@ async function saveSchedule() {
<style scoped> <style scoped>
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; } .ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-time { max-width: 260px; } .ngb-time { max-width: 260px; }
.ngb-table-wrap { overflow-x: auto; }
.ngb-table { width: 100%; border-collapse: collapse; } .ngb-table { width: 100%; border-collapse: collapse; }
.ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); } .ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); }
.ngb-failed { color: var(--color-error-text); } .ngb-failed { color: var(--color-error-text); }
+2 -1
View File
@@ -36,7 +36,7 @@
<td>{{ t('ng_backup', 'Slot {n}', { n: slot.slot }) }}</td> <td>{{ t('ng_backup', 'Slot {n}', { n: slot.slot }) }}</td>
<td><strong>{{ slot.label }}</strong></td> <td><strong>{{ slot.label }}</strong></td>
<td class="ngb-muted">{{ slot.created ? new Date(slot.created * 1000).toLocaleDateString() : '' }}</td> <td class="ngb-muted">{{ slot.created ? new Date(slot.created * 1000).toLocaleDateString() : '' }}</td>
<td> <td class="ngb-actions">
<NcButton variant="tertiary" @click="openSlotForm(slot)">{{ t('ng_backup', 'New passphrase') }}</NcButton> <NcButton variant="tertiary" @click="openSlotForm(slot)">{{ t('ng_backup', 'New passphrase') }}</NcButton>
<NcButton v-if="status.key.slots.length > 1" variant="tertiary" @click="removeSlot(slot)">{{ t('ng_backup', 'Remove') }}</NcButton> <NcButton v-if="status.key.slots.length > 1" variant="tertiary" @click="removeSlot(slot)">{{ t('ng_backup', 'Remove') }}</NcButton>
</td> </td>
@@ -162,5 +162,6 @@ const confirm = () => guarded(async () => {
.ngb-form, .ngb-confirm { display: flex; flex-direction: column; gap: 12px; max-width: 600px; margin: 8px 0; } .ngb-form, .ngb-confirm { display: flex; flex-direction: column; gap: 12px; max-width: 600px; margin: 8px 0; }
.ngb-row { display: flex; gap: 8px; } .ngb-row { display: flex; gap: 8px; }
.ngb-table td { padding: 2px 12px 2px 0; } .ngb-table td { padding: 2px 12px 2px 0; }
.ngb-actions { display: flex; gap: 4px; }
h3 { margin-top: 16px; font-weight: bold; } h3 { margin-top: 16px; font-weight: bold; }
</style> </style>
+1
View File
@@ -104,6 +104,7 @@ async function add() {
busy.value = true busy.value = true
error.value = '' error.value = ''
try { try {
await confirmPassword()
await api.addTarget({ name: form.name, backend: backend.value.id, auth: auth.value.id, options: form.options, path: form.path, await api.addTarget({ name: form.name, backend: backend.value.id, auth: auth.value.id, options: form.options, path: form.path,
maxGb: form.maxGb === '' ? null : Number(form.maxGb) }) maxGb: form.maxGb === '' ? null : Number(form.maxGb) })
adding.value = false adding.value = false
+3 -2
View File
@@ -7,7 +7,7 @@
:description="t('ng_backup', 'Pick a restore point and a user folder or file. Restores run in the background and go through Nextcloud, so overwritten files keep their old content as a version and removed files go to the trash.')"> :description="t('ng_backup', 'Pick a restore point and a user folder or file. Restores run in the background and go through Nextcloud, so overwritten files keep their old content as a version and removed files go to the trash.')">
<div class="ngb-row"> <div class="ngb-row">
<NcSelect v-model="target" :options="status.targets" label="name" :input-label="t('ng_backup', 'Location')" :clearable="false" /> <NcSelect v-model="target" :options="status.targets" label="name" :input-label="t('ng_backup', 'Location')" :clearable="false" />
<NcSelect v-if="target" v-model="snapshot" :options="snapshots" label="displayLabel" :input-label="t('ng_backup', 'Restore point')" :clearable="false" /> <NcSelect v-if="target" v-model="snapshot" :options="snapshots" label="displayLabel" class="ngb-snapshot" :input-label="t('ng_backup', 'Restore point')" :clearable="false" />
</div> </div>
<template v-if="snapshot"> <template v-if="snapshot">
@@ -69,7 +69,7 @@ const busy = ref(false)
const error = ref('') const error = ref('')
const message = ref('') const message = ref('')
const modes = [ const modes = [
{ id: 'new-folder', label: t('ng_backup', 'Into a new folder "Restored <date>" (nothing existing is changed)') }, { id: 'new-folder', label: t('ng_backup', 'Into a new folder "Restored {date}" (nothing existing is changed)', { date: 'YYYY-MM-DD HH.MM' }) },
{ id: 'merge', label: t('ng_backup', 'Into the original place, overwrite (old content kept as a version)') }, { id: 'merge', label: t('ng_backup', 'Into the original place, overwrite (old content kept as a version)') },
{ id: 'replace', label: t('ng_backup', 'Into the original place, overwrite and move files that are not in the backup to the trash') }, { id: 'replace', label: t('ng_backup', 'Into the original place, overwrite and move files that are not in the backup to the trash') },
] ]
@@ -124,6 +124,7 @@ async function restore() {
<style scoped> <style scoped>
.ngb-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 12px; } .ngb-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-row :deep(.ngb-snapshot) { min-width: min(360px, 100%); }
.ngb-crumbs { display: flex; align-items: center; flex-wrap: wrap; } .ngb-crumbs { display: flex; align-items: center; flex-wrap: wrap; }
.ngb-entries li { display: flex; align-items: center; gap: 12px; min-height: 36px; } .ngb-entries li { display: flex; align-items: center; gap: 12px; min-height: 36px; }
.ngb-entries li.selected { background: var(--color-primary-element-light); border-radius: var(--border-radius); } .ngb-entries li.selected { background: var(--color-primary-element-light); border-radius: var(--border-radius); }
+106
View File
@@ -0,0 +1,106 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Job\BackupRun;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Repository\RepositoryException;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/** Data-safety cases found in the pre-beta review. */
class CrashSafetyTest extends TestCase {
use TempDirTrait;
public function testListingKeepsNamesThatOnlyLookLikeTemporaryFiles(): void {
$root = $this->tempDir();
$backend = new LocalBackend($root);
mkdir("$root/users/john.part-time", 0700, true);
file_put_contents("$root/users/john.part-time/20261006T120000Z-aaaa0001", 'x');
file_put_contents("$root/users/upload.part-deadbeef", 'partial');
$this->assertSame(['users/john.part-time/20261006T120000Z-aaaa0001'], $backend->list('users'));
}
public function testPruneStopsWhenARecordedExportManifestIsMissing(): void {
$root = $this->tempDir();
$backend = new LocalBackend($root);
$keys = KeyRing::generate();
$repo = Repository::initWithKey($backend, $keys, $keys->wrap('pw'));
$stats = [];
$blob = $repo->storeData(random_bytes(1000), $stats);
$repo->flushPacks();
$path = 'users/alice/20261006T120000Z-aaaa0001';
$repo->putObject($path, json_encode(['uid' => 'alice', 'entries' => ['f.txt' => ['t' => 'f', 'b' => [$blob], 's' => 1000]]]));
$repo->recordUserExport($path);
unlink("$root/$path");
$packsBefore = $backend->list('packs');
try {
Repository::openWithKey($backend, $keys)->prune();
$this->fail('prune must refuse while a recorded export is missing');
} catch (RepositoryException $e) {
$this->assertStringContainsString('missing on the location', $e->getMessage());
}
$this->assertSame($packsBefore, $backend->list('packs'), 'no pack was deleted');
}
public function testForgetAndUntrashKeepTheSnapshotRestorable(): void {
$src = $this->tempDir();
file_put_contents("$src/a.txt", 'hello');
$root = $this->tempDir();
$backend = new LocalBackend($root);
$keys = KeyRing::generate();
Repository::initWithKey($backend, $keys, $keys->wrap('pw'));
$state = BackupRun::start($src);
while ($state['phase'] !== 'commit') {
$state = BackupRun::step(Repository::openWithKey($backend, $keys), $state, microtime(true) + 5);
}
Repository::openWithKey($backend, $keys)->recordSnapshot($state['snapshot']);
$id = $state['snapshot'];
Repository::openWithKey($backend, $keys)->forget($id, 'test');
$repo = Repository::openWithKey($backend, $keys);
$this->assertSame([], $repo->listSnapshots());
$this->assertArrayHasKey($id, $repo->trash());
$this->assertFalse($backend->exists('snapshots/' . $id), 'original removed after the catalog switch');
$repo->untrash($id);
$repo = Repository::openWithKey($backend, $keys);
$this->assertSame([$id], $repo->listSnapshots());
$this->assertSame([], $repo->trash());
$target = $this->tempDir();
$repo->restore($id, $target);
$this->assertSame('hello', file_get_contents("$target/a.txt"));
}
public function testFileThatKeepsChangingDoesNotStallTheRun(): void {
$src = $this->tempDir();
file_put_contents("$src/live.db", random_bytes(3 * Repository::BLOB_SIZE));
$root = $this->tempDir();
$backend = new LocalBackend($root);
$keys = KeyRing::generate();
Repository::initWithKey($backend, $keys, $keys->wrap('pw'));
$state = BackupRun::start($src);
$n = 0;
for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
$state = BackupRun::step(Repository::openWithKey($backend, $keys), $state, microtime(true) + 0.001);
// Rewrite the file after every step: it never stays the same between two steps.
$fh = fopen("$src/live.db", 'r+b');
fwrite($fh, random_bytes(16));
fclose($fh);
touch("$src/live.db", time() + ++$n);
clearstatcache();
}
$this->assertSame('commit', $state['phase'], 'the run finishes instead of retrying forever');
$this->assertSame(1, $state['stats']['changing'] ?? 0);
$this->assertSame(['live.db'], BackupRun::warnings($state)['changing']);
}
}
+77
View File
@@ -4,6 +4,8 @@ declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository; namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\BackendException;
use OCA\NgBackup\Backend\IBackend;
use OCA\NgBackup\Backend\LocalBackend; use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing; use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Repository\Repository; use OCA\NgBackup\Repository\Repository;
@@ -92,4 +94,79 @@ class VerifyTest extends TestCase {
$this->assertNotSame([], $result['failed']); $this->assertNotSame([], $result['failed']);
} }
public function testDeepVerifyRetriesAReadThatFailsOnce(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
$s = $repo->backupDirectory($this->src);
$flaky = new FlakyRangeBackend($this->backend, failures: 1);
$result = Repository::openWithKey($flaky, $this->keys, null, null)->verify($s['snapshot'], deep: true);
$this->assertSame(200000, $result['bytesChecked']);
$this->assertSame([], $result['failed']);
$this->assertSame([], $result['unreadable']);
}
public function testDeepVerifyReportsAPersistentReadErrorAsUnreadableNotCorrupted(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
$s = $repo->backupDirectory($this->src);
$broken = new FlakyRangeBackend($this->backend, failures: PHP_INT_MAX);
$result = Repository::openWithKey($broken, $this->keys, null, null)->verify($s['snapshot'], deep: true);
$this->assertSame([], $result['failed']);
$this->assertNotSame([], $result['unreadable']);
$this->assertSame('connection reset', $result['readError']);
}
public function testDeepVerifyDoesNotHideAProgrammingError(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
$s = $repo->backupDirectory($this->src);
$buggy = new FlakyRangeBackend($this->backend, failures: PHP_INT_MAX, error: new \TypeError('bug'));
$this->expectException(\TypeError::class);
Repository::openWithKey($buggy, $this->keys, null, null)->verify($s['snapshot'], deep: true);
}
}
/** Delegates to a real backend, but getRange() throws for the first $failures calls. */
final class FlakyRangeBackend implements IBackend {
public function __construct(
private IBackend $inner,
private int $failures,
private ?\Throwable $error = null,
) {
}
public function put(string $path, $stream): void {
$this->inner->put($path, $stream);
}
public function get(string $path) {
return $this->inner->get($path);
}
public function getRange(string $path, int $offset, int $length): string {
if ($this->failures > 0) {
$this->failures--;
throw $this->error ?? new BackendException('connection reset');
}
return $this->inner->getRange($path, $offset, $length);
}
public function exists(string $path): bool {
return $this->inner->exists($path);
}
public function move(string $from, string $to): void {
$this->inner->move($from, $to);
}
public function list(string $prefix): array {
return $this->inner->list($prefix);
}
public function delete(string $path): void {
$this->inner->delete($path);
}
} }
+1 -1
View File
@@ -68,7 +68,7 @@ run_on ngb-frt-restore backup:target:add offsite local -a null::null -o datadir=
run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location" run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location"
ok "recovery kit imported, existing repository adopted, snapshot visible" ok "recovery kit imported, existing repository adopted, snapshot visible"
run_on ngb-frt-restore backup:restore:full offsite "$SNAP" >/dev/null || fail "restore:full" run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction >/dev/null || fail "restore:full"
ok "backup:restore:full completed" ok "backup:restore:full completed"
# Everything below must work in a FRESH occ process (a new PHP process per docker exec), since # Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
+2 -1
View File
@@ -48,7 +48,8 @@ ok "as-backup restores into <uid>-bak, byte-identical, original account untouche
# replace: damage the account, then restore it in place under the same uid. # replace: damage the account, then restore it in place under the same uid.
$X sh -c "rm '$DATA/$U/files/Docs/f1.bin' && echo damaged > '$DATA/$U/files/Docs/f1.bin'" $X sh -c "rm '$DATA/$U/files/Docs/f1.bin' && echo damaged > '$DATA/$U/files/Docs/f1.bin'"
occ files:scan "$U" >/dev/null occ files:scan "$U" >/dev/null
occ backup:user:restore ci "$MANIFEST" --mode=replace >/dev/null || fail "restore replace" if occ backup:user:restore ci "$MANIFEST" --mode=replace --no-interaction >/dev/null 2>&1; then fail "replace of an existing account ran without --force"; fi
occ backup:user:restore ci "$MANIFEST" --mode=replace --force >/dev/null || fail "restore replace"
occ files:scan "$U" >/dev/null occ files:scan "$U" >/dev/null
SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum") SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after a replace restore" [ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after a replace restore"