Compare commits

...

65 Commits

Author SHA1 Message Date
Dillard Blom be3746e1c3 Merge pull request #26 from dillardblom/release-1.0.1
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.1
2026-11-01 12:58:34 +01:00
dillard f5473495ce App description and README: ask for a star in the app store and on GitHub
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-11-01 12:08:01 +01:00
dillard 3694184674 Version 1.0.1: changelog, README status and roadmap
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-11-01 12:03:19 +01:00
dillard a0fc91a700 Integration tests: grep the whole occ output instead of grep -q, which stopped reading and failed the pipe under pipefail
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-31 23:13:27 +01:00
dillard a11e6c7254 1.0.1: label for "Back up now", retention explained in the trash, occ times in the schedule's zone, single-file restore to a directory, leftovers of a stopped restore removed
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- Admin page: an optional label (at most 100 characters) for a backup started with "Back up now";
  an (i) by "removed by retention policy" in the trash that explains what it means and until when
  the restore point can be brought back. backup:trash says the same below its list.
- backup:status, backup:list, backup:trash and backup:browse show times in the zone of the daily
  schedule and name it, instead of the server's UTC.
- backup:restore --to-directory of a file restored nothing: the path got a "/" appended.
- A restore removes "<name>.ngb-restore-<hex>" temporaries that a stopped restore left in the
  folders it writes to.
- Roadmap: the cron warning (#13), reading ahead from S3/SMB and --salvage move to 1.0.2.
2026-10-30 19:32:53 +01:00
dillard 6f1a37f4f4 Changelog: release candidate headings carry their app version, so the app store no longer shows all of them under 1.0.0 2026-10-29 13:07:25 +01:00
dillard 5cde57bc1b Changelog: 1.0.0 says what the app is instead of release bookkeeping; rc.6 drops its outdated outlook
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-29 12:42:44 +01:00
dillard 3eec2e9554 Full restore on SQLite: take the write lock first, so another writer is waited for instead of "database is locked"
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
SQLite in WAL mode starts a transaction as a reader; a write from another
process between our first read and our first write makes our first write
fail at once, whatever the busy timeout. The SQLite full-restore test now
runs a second writer alongside the restore.
2026-10-29 03:06:05 +01:00
dillard cdf0b70e25 Changelog: the admin page's single-line key field as the cause of #24; PLAN 1.0.1: #24, database locked at the start of a full restore on SQLite
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-29 02:40:03 +01:00
dillard 66615c2a88 Merge branch 'main' into polish
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
# Conflicts:
#	CHANGELOG.md
2026-10-29 02:27:01 +01:00
Dillard Blom cc75c7128c Merge pull request #25 from dillardblom/release-1.0.0
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0
2026-10-29 02:26:04 +01:00
dillard a0c4abb60a Version 1.0.0: same code as 1.0.0-rc.6; changelog, README status
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-29 01:32:12 +01:00
dillard 4847ce293d SFTP: restore the line breaks of an OpenSSH private key before use (#24)
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Nextcloud 35 reads keys with phpseclib 3, which refuses an OpenSSH-format
key whose line breaks were lost ("Unable to read key"); phpseclib 2 on
Nextcloud 34 read it. The key is fixed when used, so stored locations work
again without re-entering it.
2026-10-28 01:41:58 +01:00
dillard 90dd97977a CI: build the admin page on every push 2026-10-27 11:34:53 +01:00
dillard 08534975d7 Admin page: Docs tab with links to the how-to, disaster recovery guide and roadmap; drop "in beta" from the feedback line
PLAN: 1.0.1 item 11 (Docs tab, done); 1.1.0 item 9: remove the feedback request in 1.1.0 or 1.2.0.
2026-10-27 11:23:34 +01:00
dillard 95e3dfcf6d User restore by user id, backup:user:list (#22)
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
backup:user:restore <location> <uid> only said "is not a recorded user export": the argument
is the manifest path, which only backup:user:backup printed once. backup:user:list shows the
exports again; a bare user id works when that user has exactly one export, otherwise the error
lists them. The export is resolved before the replace confirmation.

Fixes #22
2026-10-27 00:10:59 +01:00
dillard 7f4cae0e42 Roadmap: pin a restore point (1.1.0); PLAN 1.0.1: explain retention in the trash, time zone in backup:status
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-25 02:40:13 +02:00
Dillard Blom 965440d1f7 Merge pull request #20 from dillardblom/rc6
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.6
2026-10-24 01:04:23 +02:00
dillard 3a0f882a38 1.0.0-rc.6: changelog, SMB disaster recovery result in the README, 1.0.1 items (salvage mode, leftover restore files, single-file restore)
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-24 00:40:51 +02:00
dillard 632d2a2838 Admin page: widen the sections through their width; NcSettingsSection sets width, not max-width
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-23 18:10:48 +02:00
dillard 9e9234e2a8 Admin page: sections up to 1400px wide, so the runs table fits without scrolling
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-23 13:29:47 +02:00
dillard 3ac3298313 Database restore: close the pack stream of a blob read again before the next range 2026-10-22 20:30:18 +02:00
dillard 384b2d2792 Full-restore test: the emoji deck board sets archived as a boolean (PostgreSQL refuses 0) 2026-10-22 20:09:58 +02:00
dillard 8032411789 Restore: the warning for a blob read again says how the first read differed (zeros, shifted, other bytes) 2026-10-22 19:50:27 +02:00
dillard b4f20cb070 Version 0.9.12 (1.0.0-rc.6) 2026-10-22 19:48:57 +02:00
dillard 343389b98b Changelog: the second read covers the database phase too 2026-10-22 19:48:22 +02:00
dillard 133f5f9d34 Schedule: say why a server zone of UTC is passed over 2026-10-22 19:47:30 +02:00
dillard cf4bd9db7f Review fixes: the database restore also reads a damaged blob once more; schedule conversion right on days the clocks change; the field follows a changed schedule time zone 2026-10-22 19:45:49 +02:00
dillard 67cc350b8c Daily backup time in the administrator's time zone (#19); a time saved before is converted once so it shows as it ran 2026-10-22 16:38:54 +02:00
dillard fb5b234e8a Restore: a blob that does not decrypt as first read is read once more on a request of its own before the restore gives up 2026-10-22 14:43:48 +02:00
dillard fd12c586b6 Roadmap: restore mode for hosted Nextcloud, read-ahead for S3 and SMB; changelog for rc6 2026-10-22 13:07:09 +02:00
dillard 6ffe979ea7 Merge branch 'polish' into rc6 2026-10-22 13:06:59 +02:00
dillard a03b7354b3 README: tested disaster recoveries, backend measurements, restores without shell access; guide lists shell access; changelog for rc6 2026-10-22 13:06:49 +02:00
dillard be1c628f07 Full-restore test: install Mail and Deck with up to three tries (the app store is slow at times) 2026-10-22 03:38:30 +02:00
dillard a8355df12e Full restore: tables the restore creates get this installation's charset and collation (utf8mb4 on MySQL/MariaDB); test with an emoji 2026-10-22 03:36:46 +02:00
Dillard Blom f0537f0f2c Merge pull request #18 from dillardblom/rc5
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.5
2026-10-22 02:28:05 +02:00
dillard fd0f786f5f Full restore: a run again after an interrupted restore keeps maintenance mode on when its check refuses; retry text about apps 2026-10-22 01:59:01 +02:00
dillard 2eabdf44d5 README and store description: background jobs must run on system cron, with how to set it up; rc5 date 2026-10-22 01:03:52 +02:00
dillard 61e2a28b67 Full restore: maintenance mode on before the checks (off again when they refuse); the printed retry command is complete 2026-10-22 00:53:08 +02:00
dillard 7ed53ad0eb Full restore: after an error maintenance mode stays on, with the one line to switch it off and run again; maintenance that was on before stays on 2026-10-22 00:36:58 +02:00
dillard bdcb4f0cae Full restore runs in maintenance mode, switched off again at the end (also after an error) 2026-10-22 00:31:34 +02:00
dillard 2616b6e543 Init guard also counts snapshots in the trash and the catalog as backup data 2026-10-21 22:42:43 +02:00
dillard 9bc4e4a4ab Never create a repository where backup data lies but no config could be read 2026-10-21 22:25:48 +02:00
dillard 827e46381e Log a snapshot that cannot be adopted when a location is added; full-restore test prints what target:add and backup:list said when the snapshot is missing 2026-10-21 22:08:14 +02:00
dillard 8eb6c4c033 Index: check known index files spread over the list 2026-10-21 20:56:45 +02:00
dillard d623416306 Index: an empty listing is taken only when known index files are gone too 2026-10-21 20:49:32 +02:00
dillard cd251cffb3 PLAN: SMB index reads one at a time (1.1.0) 2026-10-21 20:02:43 +02:00
dillard 0b4872599d PLAN: the 1.0.1 cron PR closes #13 2026-10-21 19:51:57 +02:00
dillard 95184dadc7 Index: an empty listing is checked (config present, listed once more) before it is taken, so a repository another server emptied is accepted and a hiccup recovers 2026-10-21 19:51:46 +02:00
dillard 9b6bd7d2ab Release 1.0.0-rc.5 (app version 0.9.11) 2026-10-21 13:10:03 +02:00
dillard 2fcd38f478 Full-restore test: Mail and Deck checks only when the origin had them (not on a SQLite origin) 2026-10-21 13:09:54 +02:00
dillard b7d09c9784 PLAN: 1.1.0 faster SMB (local and long-line measurements) 2026-10-21 09:27:15 +02:00
dillard 177361acae Index load keeps the index when it refuses an empty listing; counters of unchanged tables are repaired too, never moved backwards 2026-10-21 09:08:50 +02:00
dillard 24686897c3 PLAN: DR tests across databases (integration for every crossing; S3 sim onto MariaDB, SMB sim onto SQLite before 1.0.0; nc01 onto MariaDB after) 2026-10-20 23:12:19 +02:00
dillard 85ea0acc50 Full-restore test: restore onto another database (to-pgsql, to-mysql, to-sqlite), every crossing in CI 2026-10-20 23:10:25 +02:00
dillard ac57e31c93 Full restore onto PostgreSQL: counters of restored tables go after their data when the backup records none (MySQL, SQLite); test MariaDB onto PostgreSQL 2026-10-20 23:04:53 +02:00
dillard 315e83b847 PLAN: 1.0.1 progress in the database phase and a waiting notice, so a backup does not look stuck 2026-10-20 20:49:44 +02:00
dillard e461c48056 CI: back to Nextcloud 35 for the full-restore test (the failures were a slow app store, not the version) 2026-10-19 22:18:33 +02:00
dillard d504067cd8 Index: an empty listing is an error in checks and cleanups too; refresh() fetches only index files it has not loaded; full-restore CI on Nextcloud 35.0.0 2026-10-19 22:13:26 +02:00
dillard c37b26ea38 Full restore: an empty listing never empties the index; read a failed pack once more; recognise and replace SFTP connections a Storage Box dropped quietly; version 0.9.11 2026-10-19 21:53:05 +02:00
dillard b728db36c7 README: roadmap 2026-10-19 21:35:23 +02:00
dillard 6ed0a555cf PLAN: label field next to Back up now (1.0.1); per-location schedules, every N hours, locations one after the other, editable notes (1.1.0) 2026-10-19 21:26:20 +02:00
dillard 221ed9dc17 PLAN: the cron warning says cron is needed for all backups, links to how to set it up, and that without it a backup started with the button only moves while the page is open 2026-10-19 21:18:23 +02:00
dillard 285153a968 PLAN: 1.0.1 cron warning says to keep the page open under AJAX, with a link to the Nextcloud docs 2026-10-19 21:16:23 +02:00
dillard 6c8a78f6bb PLAN: 1.0.1 background jobs without system cron: warn, and let the open admin page move a backup 2026-10-19 20:41:11 +02:00
45 changed files with 1917 additions and 118 deletions
+24
View File
@@ -41,6 +41,20 @@ jobs:
- run: composer install --no-interaction --no-progress
- run: composer psalm
frontend:
# Builds the admin page, so a broken Vue component fails CI instead of showing up after a release.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: npm
- run: npm ci --no-audit --no-fund
- run: npm run build
integration:
# Full round trip (key, kit, location, backup, incremental, restore modes, database dump and
# restore, retention, trash) on a fresh Nextcloud per version and database.
@@ -114,3 +128,13 @@ jobs:
run: tests/integration/full-restore-roundtrip.sh 35-apache mysql legacy
- name: Full restore over SFTP on PostgreSQL, reading packs ahead on several connections
run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql sftp
- name: Full restore of a MariaDB backup onto PostgreSQL (ID counters after the data)
run: tests/integration/full-restore-roundtrip.sh 35-apache mysql to-pgsql
- name: Full restore of a PostgreSQL backup onto MariaDB
run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql to-mysql
- name: Full restore of a PostgreSQL backup onto SQLite
run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql to-sqlite
- name: Full restore of a MariaDB backup onto SQLite
run: tests/integration/full-restore-roundtrip.sh 35-apache mysql to-sqlite
- name: Full restore of a SQLite backup onto PostgreSQL
run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite to-pgsql
+133 -4
View File
@@ -7,7 +7,130 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
beta/daily update channel, which would hide the app from most admins. The stability stage is
instead carried by the app description, the git tag, the release, and this file.
## [1.0.0-rc.4] - 2026-10-19
Each heading starts with that `<version>`, once: the app store shows a release the section(s)
whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
shown under 1.0.0. The git tag of a release candidate goes after the date.
## [1.0.1] - 2026-11-01
Fixes for what was reported after 1.0.0 (#22, #24) and small improvements. Nothing changes in
your backups or settings.
### Added
- `occ backup:user:list [location] [user]` shows the user exports on a location, with the manifest
path `backup:user:restore` expects. Before, that path was only printed once by
`backup:user:backup` (#22).
- A "Docs" tab on the admin page, with links to the how-to, the disaster recovery guide and the
roadmap. It also shows before the backup key is set up.
- A label for a backup started with "Back up now" (optional; `occ backup:run -l` already had one).
It shows with the restore point.
- An explanation in the trash of restore points of what "removed by retention policy" means: the
policy keeps the newest restore points and the last one of each recent day, week and month, and
moves the others to the trash, where they can be brought back until they are deleted
permanently. `occ backup:trash` says the same below its list.
### Changed
- `occ backup:status`, `backup:list`, `backup:trash` and `backup:browse` show times in the time
zone of the daily schedule and name it, as the admin page shows them in the administrator's.
Before, they showed the server's UTC, which read as a wrong start time.
- The admin page no longer says NG Backup is in beta.
- `occ backup:user:restore` also takes a user id instead of the manifest path, when that user has
exactly one export on the location. With more, and for a manifest path that is not on the
location, the error lists that user's exports; for a user without exports it names the
expected form (#22). Before, `backup:user:restore <location> <uid>` only said "is not a
recorded user export for this location".
### Fixed
- An SFTP location with an RSA key in OpenSSH format ("BEGIN OPENSSH PRIVATE KEY") stopped
working after an upgrade to Nextcloud 35 with "Unable to read key", and such a key could not be
added there at all (#24). The "Private key" field on the admin page is a single line, so a
pasted key lost its line breaks; Nextcloud 34 still read it, Nextcloud 35 reads keys with a
stricter library. The line breaks are now restored before the key is used, so a stored location
works again without re-entering it.
- A full restore onto SQLite could stop at once with "database is locked" when another process
wrote to the database just as the restore began (a background job that was already running).
The restore now takes the write lock first and waits for the other process instead. The
full-restore test on SQLite runs with a second process writing alongside.
- `occ backup:restore ... --to-directory` of a single file restored nothing ("Restored 0
files"); only a folder worked. A path to a file now restores that file.
- A restore that was stopped (a crash, a timeout, a restart) left its half-written temporary file
(`<name>.ngb-restore-<id>`) next to the file it was restoring, and a `files:scan` then showed it
in Files. A restore now removes such leftovers in each folder it writes to.
- The app store showed the notes of every release candidate under 1.0.0, because their headings
started with "1.0.0-rc". Each heading now starts with the app version of that release.
## [1.0.0] - 2026-10-29
First stable release. NG Backup makes encrypted, incremental backups of your whole Nextcloud
(files, database and configuration) to S3, SFTP, WebDAV, SMB and other storage, and restores them
from the web interface or with occ. Disaster recovery has been tested on real installations; see
"Tested disaster recoveries" in the README, and read "Known limitations" before you rely on it.
The code is the same as 1.0.0-rc.6 (app version 0.9.12): updating from it changes nothing in your
backups or settings.
## [0.9.12] - 2026-10-24 (1.0.0-rc.6)
Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery
simulations and a user report.
### Fixed
- The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in
Berlin (#19). The time is now saved with the administrator's time zone and shown with it. A time
saved before is converted once to the zone of the server or an administrator, so it shows as it
effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer) and nothing moves.
- A full restore over SMB stopped after 17 hours with "Decryption failed": a read over a long
line handed out wrong bytes (the backup itself was fine, the same files restored identically
afterwards). A blob that does not decrypt as first read, of a file or of the database, is now
read once more on a request of its own, and only a second failure stops the restore.
- A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table
(of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the
table got 3-byte utf8. Tables the restore creates now get this installation's character set and
collation (utf8mb4). Found in a disaster recovery simulation onto MariaDB.
### Changed
- The admin page uses more of the screen (sections up to 1400 pixels instead of 900), so the
list of backup runs fits without scrolling sideways.
- README: a roadmap, the tested disaster recoveries, measurements for OVH S3, the Storage Box over
SFTP and SMB, how to build the SMB extension when PECL is down, and what works without shell
access. The guide lists shell access among what you need.
- The full-restore integration test checks a 4-byte character in a table the restore creates, and
retries app installs from the app store.
## [0.9.11] - 2026-10-22 (1.0.0-rc.5)
Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
stopped after 13 hours with "Blob not in index"; the backup itself was fine.
### Changed
- README and app store description: NG Backup needs background jobs on system cron, with how to
set that up (issues #13, #17).
### Fixed
- A full restore no longer stops with "Blob not in index" after a pack could not be read for a
moment. Reading the index again then got an empty listing (from a connection the location had
dropped) and emptied the index. Reading the index again now only adds, an empty listing of a
location that has an index is an error instead of an empty index, and a pack that could not be
read is read once more before the restore gives up.
- After a full restore of a MySQL/MariaDB or SQLite backup onto PostgreSQL, new files, users or
activities could not be added ("duplicate key", "File entry could not be inserted"): such
backups record no ID counters, and they stayed at 1. Every counter of a restored table now goes
after its data. Found in a disaster recovery simulation of 138 GB from MariaDB to PostgreSQL,
where all 15,772 files and every count were identical.
- A full restore now runs in maintenance mode: a background job, a page visit or a user writing
into the instance meanwhile could collide with restored rows ("duplicate key") or be lost. It
is switched off when the restore completes; after an error it stays on, and the message gives
the one line to switch it off and run the restore again. Maintenance mode that was on before
the restore stays on.
- Adding a location whose repository config could not be read, while backup data is there,
no longer creates a new repository over it: it stops with an explanation. A snapshot that
cannot be taken over when a location is added is now logged instead of left out silently.
- An SFTP connection that a Hetzner Storage Box dropped without phpseclib noticing is now
recognised and replaced: the check asks the server and counts its answer.
- The extra connection a full restore reads ahead on is replaced when it breaks (up to 20 times),
instead of reading the rest of the restore the slow way.
## [0.9.10] - 2026-10-19 (1.0.0-rc.4)
Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not
released: its fixes are in here.
@@ -31,7 +154,7 @@ released: its fixes are in here.
- The app store showed old screenshots: it keeps a copy per URL, and the URLs always pointed at
`main`. They now point at the release's tag.
## [1.0.0-rc.2] - 2026-10-18
## [0.9.8] - 2026-10-18 (1.0.0-rc.2)
Second release candidate for 1.0.0; the app version is 0.9.8. Fixes from a full disaster recovery
test of a production instance (about 1 TB, Nextcloud All-in-One onto a plain Nextcloud with
@@ -72,7 +195,7 @@ PostgreSQL).
- Removing a location left its cached index in the database (tens of thousands of rows for a large
backup). It is removed with the location now, and once for locations that are already gone.
## [1.0.0-rc.1] - 2026-10-17
## [0.9.7] - 2026-10-17 (1.0.0-rc.1)
Release candidate for 1.0.0; the app version is 0.9.7. Only bug fixes until 1.0.0. Tested on a
production instance (about 1 TB, 150,000 files): full and incremental backups and a verify on
@@ -351,7 +474,13 @@ and not implemented yet.
- See the "Storage backend experiences" section of the README for provider-specific quirks found
so far (e.g. Hetzner Storage Box sub-accounts).
[1.0.0-rc.1]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.1
[1.0.1]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.1
[1.0.0]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0
[0.9.12]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.6
[0.9.11]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.5
[0.9.10]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.4
[0.9.8]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.2
[0.9.7]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.1
[0.9.6-beta.7]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.6-beta.7
[0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6
[0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5
+112 -5
View File
@@ -392,14 +392,96 @@ in README "Known limitations":
gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een
productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0.
Problemen of meldingen van gebruikers: rc5.
- **1.0.1:**
1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
- **1.0.1** (klaar voor release 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
niet aangekondigd en los van de release.
1. **Naar 1.0.2.** (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron
staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue
#13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles.
2. **Naar 1.0.2.** Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit
oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21):
a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode`
(`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de
laatste run te lang geleden: melding met de cron-regel erbij ("Backups only move while
someone has Nextcloud open"), en "waiting for background jobs" als een lopende backup lang
geen stap zette.
b. **De pagina laat de backup lopen** zolang ze openstaat: in `ajax`/`webcron` voert de
statusaanroep die de pagina toch al herhaalt ook een korte, begrensde stap uit (zoals de
korte stappen die AJAX-cron nu al krijgt). "Back up now" werkt dan ook zonder cron zolang
het tabblad openstaat; de melding zegt dat geplande backups echte cron nodig hebben.
De melding zegt (zonder werkende systeem-cron): backups hebben systeem-cron nodig, ook de
gewone nachtelijke; zo stel je cron in (link naar de Nextcloud-documentatie over
achtergrondjobs voor de versie van de server, nu
https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html,
versienummer uit de server, niet vast); zolang er geen cron is, loopt een backup die je
met de knop start alleen door terwijl de NG Backup-pagina openstaat.
c. **Voortgang in de databasefase** (issue #13): nu staat er alleen "running (db)", wat er
hetzelfde uitziet als vastzitten. Toon "running (db: 47/183 tables)" zoals de bestandsfase
een percentage toont, en "waiting for background jobs (last ran 12 min ago)" als een run
een tijd geen stap zette, ook bij een cron die is blijven staan.
Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en
`cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop
(vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen).
3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later),
zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest.
4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de
tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop
stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een
bestaande parameter. Ten hoogste 100 tekens — **klaar** (2026-10-30).
5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen
wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind
opsommen, in plaats van stoppen. **Naar 1.0.2** (besluit 2026-10-30).
6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van
rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-<id>` staan; de
geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files.
Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op (alleen bestanden met
precies dat achtervoegsel en 6 of 12 hextekens) — **klaar** (2026-10-30).
7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden
terug (er komt een `/` achter); nu werkt alleen een map. Gold alleen voor `--to-directory`;
naar de bestanden van een gebruiker werkte een los bestand al — **klaar** (2026-10-30).
8. **Uitleg (i) bij de prullenbak van herstelpunten** (besluit 2026-10-25): bij "removed by
retention policy" een infoknop met wat het betekent: het bewaarbeleid houdt voor oudere dagen
één herstelpunt per dag (het laatste), weken en maanden; de rest gaat naar de prullenbak en
wordt pas na de ingestelde wachttijd (`delete_delay_days`, standaard 30) echt verwijderd, en is
tot die datum terug te zetten. Aanleiding: de beheerder zag "19-10 13:12 – after rc4 upgrade
removed by retention policy" en dacht dat er iets was weggegooid. `occ backup:trash` zegt
hetzelfde onder de lijst — **klaar** (2026-10-30).
9. **Tijdzone in `occ backup:status`**: de tabel toont de servertijd (UTC), de webinterface de
tijd van de beheerder; dat leest als een verkeerde starttijd. Zet de zone in de kolomkop
("started (UTC)") of toon de tijden in de zone van het schema. Gedaan: beide, en ook in
`backup:list`, `backup:trash` en `backup:browse` — **klaar** (2026-10-30).
10. **User-restore op gebruikersnaam** (issue #22, 2026-10-27): `backup:user:restore <locatie>
New_User` gaf alleen "is not a recorded user export"; het argument moet het manifestpad
zijn, dat alleen `backup:user:backup` eenmalig toonde. Nu: `backup:user:list` toont de
exports, een kale uid werkt bij precies één export, anders een lijst om uit te kiezen —
**klaar** (2026-10-27).
11. **Tabblad "Docs" in de beheerpagina** (besluit 2026-10-27): een derde knop naast "Backup and
restore" en "Settings", met links naar de HOWTO (`docs/HOWTO.md`), de DR-gids
(`docs/DISASTER-RECOVERY.md`) en de roadmap (README, `#roadmap`) op GitHub, elk met een
regel wat erin staat. Nu zijn ze alleen via de README te vinden. Zichtbaar ook zonder
sleutel, zodat de gids vindbaar is vóór de eerste installatie-stap. "NG Backup is in beta"
gaat uit de feedbackregel — **klaar** (2026-10-27).
12. **SFTP-sleutel zonder regeleinden** (issue #24, 2026-10-28): het veld "Private key" op de
beheerpagina is één regel, dus een geplakte sleutel verliest zijn regeleinden. NC34
(phpseclib 2) las dat nog, NC35 (phpseclib 3) weigert een OpenSSH-sleutel dan met "Unable
to read key". De regeleinden worden nu teruggezet vlak voor gebruik — **klaar** (2026-10-28).
13. **"database is locked" bij het begin van `backup:restore:full` op SQLite** (CI, 2026-10-26
bij #22 en 2026-10-29 op `main` na de merge van 1.0.0): de restore stopt meteen na het
aanzetten van de onderhoudsmodus. Niet altijd, dus een tweede proces dat op hetzelfde moment
de database gebruikt (cron, een webverzoek of de opstart van de container). Kan ook bij een
echte restore naar SQLite gebeuren. Oorzaak (2026-10-29): SQLite (WAL) begint een transactie
als lezer; schrijft een ander proces tussen ons eerste lezen en ons eerste schrijven, dan
faalt dat schrijven direct (geen wachttijd helpt). In CI was dat de installer, die
"installed" zet en daarna nog seconden doorschrijft. Nu neemt de restore eerst het
schrijfslot en wacht hij op de ander; de SQLite-test draait met een tweede schrijver
ernaast — **klaar** (2026-10-29).
- **DR-tests over databasegrenzen** (besluit 2026-10-20; productie nc01 draait PostgreSQL): na
de MariaDB→PostgreSQL-fout (tellers, rc5) elke overstap in de integratietest (`to-pgsql`,
`to-mysql`, `to-sqlite`, ook in CI). Volle DR-tests met echte data: vóór 1.0.0 de S3-simulatie
(MariaDB-bron, 138 GB) nogmaals naar een MariaDB-doel en de SMB-simulatie naar een SQLite-doel;
**na 1.0.0** de 1 TB van nc01 (PostgreSQL) naar een MariaDB-doel.
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
een paar bestanden.
- **1.1.0:**
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
@@ -412,6 +494,31 @@ in README "Known limitations":
3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel
een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf,
33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1.
4. **Schema per locatie** (besluit 2026-10-19): in de instellingen van een locatie "uit",
"dagelijks om HH:MM" of "elke N uur (vanaf HH:MM)". Nu is er één `schedule_time` voor alle
locaties, één keer per dag. Bij de update wordt de huidige tijd het schema van elke
bestaande locatie (voor bestaande gebruikers verandert er niets).
5. **Locaties na elkaar laten starten** in plaats van tegelijk: twee runs die dezelfde schijven
lezen zijn samen niet sneller (gezien op nc01); wachtende runs zichtbaar als "queued".
7. **SMB sneller** (gemeten 2026-10-21): lokaal netwerk zonder vertraging (Samba-container op de
laptop, 1,8 GB) back-up 24,9 MB/s, restore 27,6 MiB/s; vanaf Canada naar de Storage Box (130 ms)
2,6 MiB/s, omdat SMB per blok op het antwoord wacht. Grotere schrijfblokken, en vooruitlezen
bij een restore zoals SFTP (IReadsAhead) of meerdere packs tegelijk. Een NAS met 1 Gbit haalt
~110 MB/s: daar is winst.
Ook de index: `target:add` over SMB vanaf Canada haalt ~1 indexbestand per seconde (4.927
bestanden ~85 min), want SMB krijgt geen `getMany()` met verzoeken tegelijk zoals S3 en SFTP.
6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot
is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus),
zodat een nieuwe server na een DR hem ook ziet.
8. **Herstelpunt vastpinnen** (besluit 2026-10-25): een herstelpunt dat het bewaarbeleid
nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met
`occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de
opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly.
9. **Feedbackvraag weghalen** in 1.1.0 of 1.2.0 (besluit 2026-10-27): de regel "We would like
to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot
dan staan (de vermelding "in beta" is er in 1.0.1 al af). Zijn er tegen die
tijd geen noemenswaardige meldingen, dan werkt het: mensen melden eerder wat niet werkt dan
wat gewoon werkt. Het formulier (`experience.yml`) mag blijven voor wie het zelf opzoekt.
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
PostgreSQL 18)** — voor **1.0.0-rc.2**:
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
+110 -7
View File
@@ -7,11 +7,14 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
see "Storage backend experiences" below — that requirement comes from files_external, not from
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
**Status: release candidate for 1.0.0 (1.0.0-rc.1, app version 0.9.7).** Tested thoroughly, usable
alongside your current backup, but not yet a drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and
**Status: 1.0.1, stable.** Disaster recovery has been tested on real installations (see "Tested
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
`docs/DISASTER-RECOVERY.md`.
If you like NG Backup, don't forget to give the app a star in the
[app store](https://apps.nextcloud.com/apps/ng_backup) and this repository a star on GitHub!
License: AGPL-3.0-or-later. Developed and sponsored by [Ensembia](https://www.ensembia.com).
Need help restoring a backup after a crash? Ensembia offers paid restore assistance:
[www.ensembia.com](https://www.ensembia.com).
@@ -33,6 +36,26 @@ Need help restoring a backup after a crash? Ensembia offers paid restore assista
- Restores go through Nextcloud, so overwritten files keep their old content as a version and
removed files go to the trash.
## Requirements
- **Background jobs on system cron.** NG Backup runs backups in steps from Nextcloud's
background job, so a backup only moves forward when that job runs. Choosing "Cron" under
**Administration settings → Basic settings → Background jobs** is not enough on its own: it
tells Nextcloud that the system will call it, so you also need a cron line that does. Without
it a backup stays at "running (db)", and so does one started with "Back up now".
`occ backup:run` in a terminal works without cron (it runs in the foreground).
- Regular server: `crontab -u www-data -e` and add
`*/5 * * * * php -f /var/www/nextcloud/cron.php` (your Nextcloud path).
- Docker (the official `nextcloud` image): on the host, `crontab -e` as root and add
`*/5 * * * * docker exec -u www-data <container> php -f /var/www/html/cron.php`, or add a
service with the same image and `entrypoint: /cron.sh` to your `docker-compose.yml`.
- Nextcloud All-in-One already runs cron.
After a few minutes, the Background jobs section shows that the last job ran just now. See the
[Nextcloud documentation on background jobs](https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/background_jobs_configuration.html).
- A storage location: S3 or compatible, SFTP, WebDAV, SMB (needs the PHP `smbclient` extension,
see below), FTP or a local path.
## Quick start
```
@@ -126,10 +149,31 @@ What you should do on the location (strongly recommended):
- **Files and folders** of a user: in the web interface or `occ backup:restore`, into a new folder
("Restored <date>"), merged into the original place, or replacing it.
- **Raw restore** of any path (for example `config/`) to a local directory: `occ backup:restore ... --to-directory=DIR`.
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see the how-to.
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see
[docs/DISASTER-RECOVERY.md](docs/DISASTER-RECOVERY.md). It runs in maintenance mode, so that
nobody and no background job writes into the instance while it is being replaced.
- **One user** (account, settings, files, calendars, contacts via user_migration): `occ backup:user:backup` and
`occ backup:user:restore`, replacing the account or restoring it as `<uid>-bak`. See the how-to.
**Without shell access** (a hosted Nextcloud where you cannot run `occ`): files and folders can be
restored in the web interface. Restoring a whole user or the whole server needs `occ`, so on such
hosting it is something your provider (or whoever has shell access) runs. A restore mode that
works without maintenance mode, and so from the web interface, is on the roadmap.
## Tested disaster recoveries
Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECOVERY.md):
| Backup | Location | Database | Result |
|---|---|---|---|
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
| 51 GB test instance | OVH S3 | PostgreSQL → PostgreSQL | 23,519/23,519 checksums identical |
The integration tests restore across every pair of PostgreSQL, MariaDB and SQLite.
## Storage backend experiences
Notes from actually running a backend, beyond what's in files_external's own documentation.
@@ -186,15 +230,71 @@ Contributions welcome via PR — add what you ran into with your own provider.
can be read on more connections: `occ config:app:set ng_backup restore_sftp_connections
--value=4` (at most 8; the server must allow that many plus one at once for the user).
- **OVH Object Storage (S3, Warsaw, 2026-10):** works with the `amazons3` backend (path style,
region `waw`). A full backup of 1 TB from Germany ran at about 7 MiB/s, from Canada about
6 MiB/s with BBR; a check of a 1 TB backup took 3.5 minutes (index files are fetched 16 at a
time); a restore of 138 GB from Warsaw to Canada took about 6 hours. Turn on versioning and
Object Lock on the bucket (see above); OVH charges no egress.
- **Hetzner Storage Box over SFTP, full restore:** about 13-14 MiB/s from Germany to Canada
(122 ms) with the read-ahead of 1.0.0-rc.4; without it about 2.8 MiB/s. A Storage Box gets
slower with several connections reading at once (4-6 MiB/s for 2-4 connections, it reads from
disks), so one extra connection is the default. A Storage Box drops an SFTP connection that
stayed quiet for a while without telling the client; NG Backup checks and replaces it.
- **SMB speed:** on a local network (Samba, no delay) a backup ran at about 25 MB/s and a
restore at about 28 MiB/s. Over a long line SMB waits for an answer to each request: from
Canada to a Storage Box in Germany (130 ms) about 2.6 MiB/s for a backup and 1.6 MiB/s for a
restore, and connecting a location with an existing backup reads its index at about one file a
second. SMB is meant for a local network; use SFTP or S3 for a location far away. When
`pecl install smbclient` fails (pecl.php.net was unreachable for hours on 2026-10-21), the
extension builds from its source: `libsmbclient-php` on GitHub, `phpize && ./configure && make
&& make install`, then `docker-php-ext-enable smbclient`.
- **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself
(e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the
mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping
mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS
can't be driven from PHP directly.
## Roadmap
What is planned, without dates. Ideas and votes are welcome in the
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
a 👍 on its issue helps decide what comes first.
**1.0.2**
- A warning on the admin page and in `occ backup:status` when background jobs do not run on a
system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on
while the NG Backup page is open, and the database phase of a backup shows its progress
([#13](https://github.com/dillardblom/ng_backup/issues/13)).
- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4.
- `--salvage` for a restore: restore what can be read, skip a file that cannot, and list those at
the end instead of stopping.
**1.1.0**
- Restore without the recovery kit: the backup key from the location itself, opened with one of
your passphrases.
- A schedule per location, also "every N hours".
- Locations backed up one after the other instead of at the same time.
- A note on a restore point that you can change afterwards.
- Pinning a restore point, so the retention policy never removes it (for example the one from
before an upgrade).
- An option to pause heavy background jobs (such as Memories, Recognize or the Preview
Generator) while a backup runs.
- Faster backups: reading and uploading at the same time.
**Later**
- A full restore onto the existing installation, without first installing a fresh Nextcloud, in a
restore mode of its own instead of maintenance mode, so that it can also run from the web
interface (for hosted Nextcloud without shell access).
- Users restoring their own files from the Files app.
- Primary object storage and External Storage mounts (see Known limitations).
- An option not to keep the backup key on the location (the recovery kit is then the only copy).
## Known limitations
On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it:
On the backlog; know them before you rely on NG Backup:
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
@@ -216,8 +316,11 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
- Location passwords passed with `-o password=...` are visible in the process list and shell
history while the command runs.
- Temporary files of a restore into a user's files end up in that user's trash.
- Restoring the database does not order tables by foreign keys; apps with cascading foreign keys
could lose rows (Nextcloud core does not use them).
- **Restoring a whole user or the whole server needs `occ`** (shell access); see "Without shell
access" above.
- **`--install-apps` depends on the app store.** Nextcloud downloads the whole app list (about
38 MB) and gives up after 120 seconds; on a slow day apps are not installed. The restore then
lists them: install them afterwards with `occ app:install`, their data is restored already.
- A pack whose index upload failed stays on the location until removed by hand.
## Commands
@@ -234,4 +337,4 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
| `backup:key:import` | import a recovery kit on a fresh installation |
| `backup:user:backup`, `backup:user:restore` | one user via user_migration (asks before installing it) |
| `backup:user:backup`, `backup:user:list`, `backup:user:restore` | one user via user_migration (asks before installing it); list the exports |
+9 -4
View File
@@ -36,12 +36,16 @@ All-in-One and hosted Nextcloud.
- One user (account, files, calendars, contacts) via user_migration.
- The whole server onto a fresh installation (disaster recovery).
NG Backup needs Nextcloud's background jobs on system cron (a real cron line, not only the setting).
Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very
welcome: use the "Share your experience" link on the admin page.
If you like NG Backup, don't forget to give the app a star in the app store and the
[repository](https://github.com/dillardblom/ng_backup) a star on GitHub!
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
you need help with restoring a backup after a crash.]]></description>
<version>0.9.10</version>
<version>1.0.1</version>
<licence>agpl</licence>
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
<namespace>NgBackup</namespace>
@@ -52,9 +56,9 @@ you need help with restoring a backup after a crash.]]></description>
<website>https://github.com/dillardblom/ng_backup</website>
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.4/screenshots/3-settings.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/3-settings.png</screenshot>
<dependencies>
<php min-version="8.1"/>
<lib>sodium</lib>
@@ -91,6 +95,7 @@ you need help with restoring a backup after a crash.]]></description>
<command>OCA\NgBackup\Command\KeySlotRemove</command>
<command>OCA\NgBackup\Command\BackupUser</command>
<command>OCA\NgBackup\Command\RestoreUser</command>
<command>OCA\NgBackup\Command\ListUserExports</command>
<command>OCA\NgBackup\Command\KeyImport</command>
<command>OCA\NgBackup\Command\RestoreFull</command>
<command>OCA\NgBackup\Command\Verify</command>
+12 -1
View File
@@ -10,6 +10,8 @@ Nextcloud back from an NG Backup location. Do them in this order.
- [ ] A new server with the **same Nextcloud major version** as the old one (for example 34),
a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for
the whole backup plus a margin.
- [ ] **Shell access** to the new server, to run `occ` (on a hosted Nextcloud without it, your
provider runs these steps).
- [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by
hand). Keep their packages somewhere safe beforehand.
@@ -94,7 +96,16 @@ tables that do not exist here, install the apps they belong to and run it again,
out with `--skip-missing-tables`.
**8. Let the restore run.** It restores the database, then the data directory, and takes hours for
a large backup. If it stops halfway, fix the cause and run the same command again with `--force`.
a large backup. Nextcloud is in maintenance mode meanwhile, so nobody and no background job
writes into it; the restore switches it off when it completes. (If you had switched maintenance
mode on yourself before, it stays on: switch it off with `occ maintenance:mode --off` when ready.)
If it stops halfway, maintenance mode stays on (the instance may be partly restored). Fix the cause,
then switch maintenance mode off and run the restore again with `--force`, in one line (the message
prints it): `occ maintenance:mode --off && occ backup:restore:full <location> <snapshot> --force`.
Leave out `--install-apps` this time, so that the instance is open only for the seconds until the
restore has switched maintenance mode on again; apps the first run could not install are listed at
the end of the restore, install those afterwards.
If occ then says that Nextcloud or one of the apps requires an upgrade, the database is already
the old server's: run `occ upgrade` first, then the same command with `--force`.
+6
View File
@@ -114,10 +114,16 @@ Needs the `user_migration` app. NG Backup asks before installing it:
```
occ backup:user:backup offsite <uid>
occ backup:user:list offsite [<uid>] # the exports and their manifest
occ backup:user:restore offsite users/<uid>/<exportId> # replace (default)
occ backup:user:restore offsite users/<uid>/<exportId> --mode=as-backup
```
- `backup:user:restore` needs the manifest path that `backup:user:backup` prints at the end
(`users/<uid>/<exportId>`); `backup:user:list` shows it again. A user id alone also works when
that user has exactly one export on the location. With more (a `replace` restore adds a safety
export), the command lists them and you pass one.
- If `user_migration` is missing, the command asks to install and enable it. For non-interactive
use, add `--install-user-migration` to approve that explicitly. The command stops after the
installation; run it once more.
+22
View File
@@ -72,6 +72,10 @@ final class ExternalStorageFactory {
throw new BackendException("Auth mechanism $authId does not fit backend $backendId");
}
if (is_string($options['private_key'] ?? null)) {
$options['private_key'] = self::normalizePrivateKey($options['private_key']);
}
$config = new StorageConfig();
$config->setBackend($backend);
$config->setAuthMechanism($auth);
@@ -86,4 +90,22 @@ final class ExternalStorageFactory {
$storage = $backend->wrapStorage($storage);
return $auth->wrapStorage($storage);
}
/**
* An OpenSSH-format private key with its line breaks restored. Up to Nextcloud 34
* (phpseclib 2) a key whose newlines were lost or turned into spaces, e.g. on its way through
* a shell or a form, still loaded; Nextcloud 35 (phpseclib 3) refuses it with "Unable to read
* key" (#24). The key itself is not changed, so it is fixed when it is used, not when stored.
* Other formats (PEM, PKCS#8) still load as they are and are returned unchanged.
*/
public static function normalizePrivateKey(string $key): string {
if (!preg_match('/-----BEGIN OPENSSH PRIVATE KEY-----(.*?)-----END OPENSSH PRIVATE KEY-----/s', $key, $m)) {
return $key;
}
$body = (string)preg_replace('/\s+/', '', $m[1]);
if (!preg_match('#^[A-Za-z0-9+/]+=*$#', $body)) {
return $key;
}
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
}
}
+31 -2
View File
@@ -54,6 +54,9 @@ final class SftpRangeFetcher {
private const FXF_READ = 1;
private const FX_OK = 0;
/** Connections that broke and are replaced, at most, in one restore. */
public const MAX_RECONNECTS = 20;
private const OPEN = 0;
private const READ = 1;
private const CLOSE = 2;
@@ -65,6 +68,8 @@ final class SftpRangeFetcher {
/** @var array<int, resource|null> ranges read (or failed) and not yet given to the caller */
private array $done = [];
private int $requestId = 0x4e520000;
/** Connections replaced so far (see MAX_RECONNECTS). */
public int $reconnects = 0;
/**
* @param array<int, object> $connections phpseclib 2 connections with their SFTP channel open
@@ -75,6 +80,7 @@ final class SftpRangeFetcher {
private string $root,
private array $ranges,
private string $spool,
private ?\Closure $connect = null,
) {
foreach (array_keys($connections) as $c) {
$this->chunks[$c] = self::CHUNK;
@@ -86,10 +92,12 @@ final class SftpRangeFetcher {
*
* @param list<object> $connections
* @param list<array{0:string, 1:int, 2:int}> $ranges
* @param (\Closure():?object)|null $connect a new connection in place of one that broke
* (null when it cannot be made)
* @return \Generator<int, resource|null>
*/
public static function fetch(array $connections, string $root, array $ranges, string $spool): \Generator {
$fetcher = new self($connections, rtrim($root, '/'), $ranges, $spool);
public static function fetch(array $connections, string $root, array $ranges, string $spool, ?\Closure $connect = null): \Generator {
$fetcher = new self($connections, rtrim($root, '/'), $ranges, $spool, $connect);
try {
yield from $fetcher->run();
} finally {
@@ -319,6 +327,27 @@ final class SftpRangeFetcher {
}
self::disconnect($this->connections[$c]);
unset($this->connections[$c]);
// A long restore outlives connections (a Storage Box drops one now and then): without a
// new one, the rest would be read the slow way.
if ($this->connect === null || $this->reconnects >= self::MAX_RECONNECTS) {
return;
}
$this->reconnects++;
try {
$new = ($this->connect)();
} catch (\Throwable) {
$new = null;
}
if (function_exists('OCP\Log\logger')) {
\OCP\Log\logger('ng_backup')->info($new === null
? 'Reading ahead: an SFTP connection broke and no new one could be made'
: 'Reading ahead: an SFTP connection broke; reading on with a new one');
}
if ($new !== null) {
$key = max(array_merge(array_keys($this->chunks), [-1])) + 1;
$this->connections[$key] = $new;
$this->chunks[$key] = self::CHUNK;
}
}
/** @return resource|null a new file in the spool, already unlinked: nothing is left behind if the process dies */
+16 -5
View File
@@ -282,10 +282,19 @@ final class StorageBackend implements IBackend, IReadsAhead {
if ($connections === []) {
return null;
}
return (static function () use ($storages, $connections, $root, $abs, $spool): \Generator {
// in place of a connection that broke: a new storage, kept like the others
$connect = static function () use ($reconnect, &$storages): ?object {
$storage = $reconnect();
$sftp = SftpReadStream::connection($storage);
if ($sftp !== null) {
$storages[] = $storage;
}
return $sftp;
};
return (static function () use (&$storages, $connections, $root, $abs, $spool, $connect): \Generator {
// the storages hold their connections: kept until the fetcher is done
try {
yield from SftpRangeFetcher::fetch($connections, $root, $abs, $spool);
yield from SftpRangeFetcher::fetch($connections, $root, $abs, $spool, $connect);
} finally {
unset($storages);
}
@@ -478,8 +487,9 @@ final class StorageBackend implements IBackend, IReadsAhead {
$this->lastUsed = microtime(true);
}
/** After a failed request: ask the server (one round trip), as phpseclib may not know it is gone. */
private function reopenIfDead(): bool {
if ($this->reconnect === null || !$this->sftpReads || $this->alive(false) !== false) {
if ($this->reconnect === null || !$this->sftpReads || $this->alive(true) !== false) {
return false;
}
$this->reopen();
@@ -508,8 +518,9 @@ final class StorageBackend implements IBackend, IReadsAhead {
if (is_callable([$sftp, 'clearStatCache'])) {
$sftp->clearStatCache(); // or stat() answers from memory
}
$sftp->stat('.');
return (bool)$sftp->isConnected();
// A Storage Box drops a quiet connection without phpseclib noticing: it still says
// connected, and every request fails. So the answer counts, not what it says after.
return $sftp->stat('.') !== false && (bool)$sftp->isConnected();
} catch (\Throwable) {
return false;
}
+4 -7
View File
@@ -32,6 +32,7 @@ final class BackupJob extends TimedJob {
private RunMapper $runs,
private IAppConfig $appConfig,
private LoggerInterface $logger,
private \OCA\NgBackup\Service\Schedule $schedule,
) {
parent::__construct($time);
// Well below the usual 5-minute cron interval: the last run is stamped a moment after cron
@@ -101,14 +102,10 @@ final class BackupJob extends TimedJob {
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
}
/** Daily schedule: appconfig ng_backup/schedule_time "HH:MM" (server time); empty = no schedule. */
/** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */
private function startScheduled(): void {
$at = $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', '');
if (!preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $at)) {
return;
}
$todayAt = (int)strtotime(date('Y-m-d') . ' ' . $at);
if (time() < $todayAt) {
$todayAt = $this->schedule->todayAt();
if ($todayAt === null || time() < $todayAt) {
return;
}
foreach ($this->targets->list() as $t) {
+6 -2
View File
@@ -7,7 +7,9 @@ declare(strict_types=1);
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Format;
use OCA\NgBackup\Service\RestoreService;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\Table;
@@ -19,6 +21,7 @@ final class Browse extends Command {
public function __construct(
private TargetService $targets,
private RestoreService $restore,
private Schedule $schedule,
) {
parent::__construct();
}
@@ -43,9 +46,10 @@ final class Browse extends Command {
$output->writeln('Nothing here.');
return 0;
}
$zone = $this->schedule->get()['timezone'];
$rows = array_map(fn ($i) => [$i['type'] === 'dir' ? $i['name'] . '/' : $i['name'], $this->size($i['size']),
$i['type'] === 'dir' ? $i['files'] . ' files' : date('Y-m-d H:i', (int)$i['mtime'])], $items);
(new Table($output))->setHeaders(['name', 'size', 'files / modified'])->setRows($rows)->render();
$i['type'] === 'dir' ? $i['files'] . ' files' : Format::time((int)$i['mtime'], $zone)], $items);
(new Table($output))->setHeaders(['name', 'size', "files / modified ($zone)"])->setRows($rows)->render();
return 0;
}
+6 -2
View File
@@ -8,6 +8,8 @@ declare(strict_types=1);
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Db\SnapshotMapper;
use OCA\NgBackup\Format;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\Table;
@@ -19,6 +21,7 @@ final class ListSnapshots extends Command {
public function __construct(
private TargetService $targets,
private SnapshotMapper $snapshots,
private Schedule $schedule,
) {
parent::__construct();
}
@@ -32,10 +35,11 @@ final class ListSnapshots extends Command {
protected function execute(InputInterface $input, OutputInterface $output): int {
$name = $input->getArgument('target');
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
$zone = $this->schedule->get()['timezone'];
$rows = [];
foreach ($targets as $t) {
foreach ($this->snapshots->findForTarget($t->getId()) as $s) {
$rows[] = [$t->getName(), $s->getSnapshotId(), date('Y-m-d H:i', $s->getCreatedAt()), $s->getKind(),
$rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(),
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getLabel() ?? ''];
}
}
@@ -43,7 +47,7 @@ final class ListSnapshots extends Command {
$output->writeln('No snapshots yet (occ backup:run).');
return 0;
}
(new Table($output))->setHeaders(['location', 'snapshot', 'date', 'kind', 'files', 'size', 'label'])->setRows($rows)->render();
(new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'label'])->setRows($rows)->render();
return 0;
}
}
+64
View File
@@ -0,0 +1,64 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\Table;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
final class ListUserExports extends Command {
public function __construct(
private TargetService $targets,
) {
parent::__construct();
}
protected function configure(): void {
$this->setName('backup:user:list')
->setDescription('List the user exports (backup:user:backup) on a location')
->addArgument('target', InputArgument::OPTIONAL, 'Location name or id (default: all)')
->addArgument('user', InputArgument::OPTIONAL, 'Only the exports of this user id')
->setHelp(<<<'HELP'
The manifest column is what occ backup:user:restore expects. A safety export made by a restore
with --mode=replace is listed here as well.
Example:
occ backup:user:list offsite alice
HELP);
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$name = $input->getArgument('target');
$uid = $input->getArgument('user');
$rows = [];
try {
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
foreach ($targets as $t) {
foreach ($this->targets->repository($t)->listUserExports($uid) as $path) {
$parts = explode('/', $path, 3);
$user = $parts[1] ?? '';
$exportId = $parts[2] ?? '';
$time = \DateTimeImmutable::createFromFormat('Ymd\THis\Z', substr($exportId, 0, 16), new \DateTimeZone('UTC'));
$rows[] = [$t->getName(), $user, $time !== false ? $time->format('Y-m-d H:i') : '', $path];
}
}
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
if ($rows === []) {
$output->writeln('No user exports yet (occ backup:user:backup).');
return 0;
}
(new Table($output))->setHeaders(['location', 'user', 'date (UTC)', 'manifest'])->setRows($rows)->render();
return 0;
}
}
+14 -1
View File
@@ -88,10 +88,20 @@ HELP);
$this->installApp($app, $output);
}
}
$output->writeln('Nextcloud is in maintenance mode while the restore runs; it is switched off when the restore completes.');
$result = $this->restoreFull->restoreFull($target, $snapshot, $skipMissingTables);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
if (\OCP\Server::get(\OCP\IConfig::class)->getSystemValueBool('maintenance', false)) {
$output->writeln('The restore had started: the instance may be partly restored, and maintenance mode stays on so that nobody uses it like that.');
$output->writeln('Fix the cause, then switch maintenance mode off and run the same command again with --force, in one line:');
$again = 'occ backup:restore:full ' . escapeshellarg((string)$input->getArgument('target')) . ' ' . escapeshellarg((string)$input->getArgument('snapshot'))
. ($input->getOption('skip-missing-tables') ? ' --skip-missing-tables' : '') . ' --force --no-interaction';
$output->writeln(' occ maintenance:mode --off && ' . $again);
$output->writeln('(Without --install-apps, so the instance is open only for the seconds until the restore starts again. Apps the first run could not install are listed at the end: install those afterwards.)');
} else {
$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
}
$output->writeln('If occ then says that Nextcloud needs an upgrade (the database is already the old server\'s), run occ upgrade first.');
return 1;
}
@@ -121,6 +131,9 @@ HELP);
if (!$result['dbConsistent']) {
$output->writeln('<comment>Warning: this snapshot\'s database dump was made over several steps (web cron), so it is not one consistent point in time. Check the instance carefully; prefer a snapshot made with system cron.</comment>');
}
if ($result['maintenanceLeftOn']) {
$output->writeln('<comment>Maintenance mode was on before the restore and stays on: switch it off with occ maintenance:mode --off when the instance is ready.</comment>');
}
$output->writeln('Next: check trusted_domains and overwritehost/overwriteprotocol match this server, then occ maintenance:repair.');
return 0;
}
+19 -5
View File
@@ -33,7 +33,7 @@ final class RestoreUser extends Command {
$this->setName('backup:user:restore')
->setDescription('Restore a user from a user_migration export (see backup:user:backup)')
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>)')
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>), or a user id with exactly one export')
->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'replace or as-backup', UserRestoreService::MODE_REPLACE)
->addOption('force', null, InputOption::VALUE_NONE, 'Replace an existing account without asking (required when not interactive)')
->addOption('install-user-migration', null, InputOption::VALUE_NONE, 'Install and enable user_migration if it is missing (non-interactive consent)')
@@ -45,16 +45,31 @@ Never imports over a live account. Modes:
compare and transfer by hand; Nextcloud cannot rename a user id, so the backup gets
the new name, never the existing account
occ backup:user:list shows the exports on a location. A user id alone works when that user
has exactly one export there; with more, the command lists them and you pass one.
Example:
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34
occ backup:user:restore offsite alice
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34 --mode=as-backup
HELP);
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$mode = (string)$input->getOption('mode');
$uid = explode('/', (string)$input->getArgument('manifest'))[1] ?? '';
if ($mode === UserRestoreService::MODE_REPLACE && $uid !== '' && $this->userManager->userExists($uid) && !$input->getOption('force')) {
try {
$target = $this->targets->get($input->getArgument('target'));
$ref = (string)$input->getArgument('manifest');
$manifest = $this->targets->repository($target)->resolveUserExport($ref);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
if ($manifest !== $ref) {
$output->writeln("Using $manifest.");
}
$uid = explode('/', $manifest)[1];
if ($mode === UserRestoreService::MODE_REPLACE && $this->userManager->userExists($uid) && !$input->getOption('force')) {
// Replace deletes the live account (after a safety export). Shares to others and data of
// apps without a user_migration migrator do not come back, so never do it unasked.
if (!$input->isInteractive()) {
@@ -69,9 +84,8 @@ HELP);
}
try {
$output->writeln('<comment>Note: on restore, user_migration only imports the app settings on its own allowlist (e.g. calendar view and reminder settings); other app settings of this user are skipped. Reason: NG Backup uses user_migration\'s importer; widening the list would mean building a settings importer of our own.</comment>');
$target = $this->targets->get($input->getArgument('target'));
$this->installer->ensureEnabled($input, $output);
$user = $this->userRestore->restoreUser($target, (string)$input->getArgument('manifest'), $mode);
$user = $this->userRestore->restoreUser($target, $manifest, $mode);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
+7 -3
View File
@@ -8,7 +8,9 @@ declare(strict_types=1);
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Db\RunMapper;
use OCA\NgBackup\Format;
use OCA\NgBackup\Service\KeyService;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService;
use OCA\NgBackup\Service\VerifyService;
use Symfony\Component\Console\Command\Command;
@@ -26,6 +28,7 @@ final class Status extends Command {
private TargetService $targets,
private RunMapper $runs,
private VerifyService $verify,
private Schedule $schedule,
) {
parent::__construct();
}
@@ -36,12 +39,13 @@ final class Status extends Command {
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$zone = $this->schedule->get()['timezone'];
if (!$this->keys->isInitialized()) {
$output->writeln('Key: <error>not created</error> (occ backup:key:init)');
} else {
$c = $this->keys->recoveryKitConfirmation();
$output->writeln('Key: ' . $this->keys->fingerprint() . ', recovery kit '
. ($c ? 'confirmed by ' . $c['uid'] . ' on ' . date('Y-m-d H:i', $c['time']) : '<error>not confirmed</error> (occ backup:key:confirm)'));
. ($c ? 'confirmed by ' . $c['uid'] . ' on ' . Format::time($c['time'], $zone) . " ($zone)" : '<error>not confirmed</error> (occ backup:key:confirm)'));
}
$names = [];
foreach ($this->targets->list() as $t) {
@@ -54,12 +58,12 @@ final class Status extends Command {
foreach ($shown as $r) {
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''),
date('Y-m-d H:i', $r->getStartedAt()), $r->getFinishedAt() ? \OCA\NgBackup\Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
}
if ($rows !== []) {
(new Table($output))->setHeaders(['run', 'location', 'kind', 'status', 'started', 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render();
(new Table($output))->setHeaders(['run', 'location', 'kind', 'status', "started ($zone)", 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render();
}
$older = $all ? 0 : $this->runs->countAll() - count($shown);
if ($older > 0) {
+9 -2
View File
@@ -7,7 +7,9 @@ declare(strict_types=1);
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Format;
use OCA\NgBackup\Service\KeyService;
use OCA\NgBackup\Service\Schedule;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\Table;
@@ -19,6 +21,7 @@ final class Trash extends Command {
public function __construct(
private TargetService $targets,
private KeyService $keys,
private Schedule $schedule,
) {
parent::__construct();
}
@@ -32,11 +35,14 @@ final class Trash extends Command {
protected function execute(InputInterface $input, OutputInterface $output): int {
$name = $input->getArgument('target');
$delay = $this->keys->deleteDelayDays() * 86400;
$zone = $this->schedule->get()['timezone'];
$rows = [];
foreach ($name === null ? $this->targets->list() : [$this->targets->get($name)] as $t) {
foreach ($this->targets->repository($t)->trash() as $id => $info) {
$rows[] = [$t->getName(), $id, (string)($info['time'] ?? ''), $info['label'], date('Y-m-d H:i', $info['forgottenAt']), $info['by'],
date('Y-m-d H:i', $info['forgottenAt'] + $delay)];
$time = $info['time'] ?? null;
$made = is_string($time) ? strtotime($time) : false;
$rows[] = [$t->getName(), $id, $made !== false ? Format::time($made, $zone) : '', $info['label'], Format::time($info['forgottenAt'], $zone), $info['by'],
Format::time($info['forgottenAt'] + $delay, $zone)];
}
}
if ($rows === []) {
@@ -44,6 +50,7 @@ final class Trash extends Command {
return 0;
}
(new Table($output))->setHeaders(['location', 'snapshot', 'made', 'label', 'removed', 'by', 'deleted after'])->setRows($rows)->render();
$output->writeln("Times in $zone. The retention policy keeps the newest snapshots and the last of each recent day, week and month; it moves the others here, and they are deleted permanently after " . $this->keys->deleteDelayDays() . ' days.');
return 0;
}
}
+10 -6
View File
@@ -47,6 +47,7 @@ final class AdminApiController extends Controller {
private \OCA\NgBackup\Service\AlertService $alerts,
private \OCA\NgBackup\Service\KeySlotService $slots,
private \OCA\NgBackup\Service\PruneService $prune,
private \OCA\NgBackup\Service\Schedule $schedule,
) {
parent::__construct(Application::APP_ID, $request);
}
@@ -72,7 +73,8 @@ final class AdminApiController extends Controller {
'runs' => $runs,
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
'schedule' => $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''),
'schedule' => $this->schedule->get()['time'],
'scheduleTimezone' => $this->schedule->get()['timezone'],
]);
}
@@ -240,12 +242,14 @@ final class AdminApiController extends Controller {
return new JSONResponse($this->runJson($run));
}
public function setSchedule(string $time): JSONResponse {
if ($time !== '' && !preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time)) {
return new JSONResponse(['error' => 'Use HH:MM or empty'], Http::STATUS_BAD_REQUEST);
/** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */
public function setSchedule(string $time, string $timezone = ''): JSONResponse {
try {
$this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
} catch (\InvalidArgumentException $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
}
$this->appConfig->setValueString(Application::APP_ID, 'schedule_time', $time);
return new JSONResponse(['schedule' => $time]);
return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]);
}
public function snapshots(int $targetId): JSONResponse {
+63 -7
View File
@@ -159,7 +159,7 @@ final class DbDumper {
// everywhere); tables it does not know are left out, as asked.
foreach ($missing as $table) {
if (isset($manifest['tables'][$table]['schema'])) {
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema']));
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
$report['created'][] = $table;
} else {
$report['missing'][] = $table;
@@ -169,7 +169,7 @@ final class DbDumper {
$foreignKeys = [];
$this->db->beginTransaction();
try {
$foreignKeys = $this->suspendForeignKeys();
$foreignKeys = $this->suspendForeignKeys(); // first: on SQLite it also takes the write lock
foreach ($manifest['tables'] as $table => $info) {
if (in_array($table, $report['missing'], true)) {
continue;
@@ -216,7 +216,8 @@ final class DbDumper {
}
$report['restored'][] = $table;
}
$this->restoreSequences($manifest['sequences'] ?? [], $report['missing']);
// unchanged tables too: a restore that stopped earlier may have left their counters at 1
$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
$this->resumeForeignKeys($foreignKeys);
$this->db->commit();
// Foreign keys of the tables created above, now their rows are in. Not inside the
@@ -318,8 +319,31 @@ final class DbDumper {
'foreignKeys' => $foreignKeys];
}
private static function tableFromStructure(string $name, array $structure): \Doctrine\DBAL\Schema\Table {
$table = new \Doctrine\DBAL\Schema\Table($name);
/**
* Table options for a table the restore creates: those of this installation's own tables.
* On MySQL/MariaDB that is the character set and collation, as Nextcloud's tables have them
* (utf8mb4, so emoji and other 4-byte characters fit); DBAL alone would use 3-byte utf8.
*
* @return array<string, mixed>
* @psalm-suppress UndefinedClass the DBAL classes come with Nextcloud, not with this app
*/
private function tableOptions(): array {
$options = $this->db->createSchemaManager()->createSchemaConfig()->getDefaultTableOptions();
if ($this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\AbstractMySQLPlatform) {
$collation = $this->db->fetchOne('SELECT TABLE_COLLATION FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ?',
[$this->prefix . 'appconfig']);
if (is_string($collation) && $collation !== '') {
$options['collation'] = $collation;
$options['charset'] = explode('_', $collation, 2)[0];
unset($options['collate']);
}
}
return $options;
}
/** @param array<string, mixed> $options table options (see tableOptions()) */
private static function tableFromStructure(string $name, array $structure, array $options = []): \Doctrine\DBAL\Schema\Table {
$table = new \Doctrine\DBAL\Schema\Table($name, [], [], [], [], $options);
foreach ($structure['columns'] as $c) {
$table->addColumn($c['name'], $c['type'], array_filter([
'length' => $c['length'], 'precision' => $c['precision'], 'scale' => $c['scale'],
@@ -388,6 +412,12 @@ final class DbDumper {
return $checks;
}
if ($platform instanceof \Doctrine\DBAL\Platforms\SqlitePlatform) {
// SQLite (WAL) starts a transaction as a reader. When another process (a cron job, the
// installer still finishing) writes between our first read and our first write, that
// write fails at once with "database is locked": no waiting helps a reader whose view
// is out of date. Take the write lock first, so we wait for the other writer instead
// and it waits for us after that.
$this->db->executeStatement('DELETE FROM ' . $this->db->quoteIdentifier($this->prefix . 'appconfig') . ' WHERE 0 = 1');
$this->db->executeStatement('PRAGMA defer_foreign_keys = ON'); // checked at commit
}
return [];
@@ -573,16 +603,21 @@ final class DbDumper {
return $name;
}
/** @param list<string> $leftOut tables not restored (--skip-missing-tables): they do not exist here */
private function restoreSequences(array $sequences, array $leftOut = []): void {
/**
* @param list<string> $leftOut tables not restored (--skip-missing-tables): they do not exist here
* @param list<string> $restored tables restored or unchanged: their other counters go after their data
*/
private function restoreSequences(array $sequences, array $leftOut = [], array $restored = []): void {
if (!$this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\PostgreSQLPlatform) {
return;
}
$leftOut = array_flip($leftOut);
$set = [];
foreach ($sequences as $info) {
if (isset($leftOut[$info['table']])) {
continue;
}
$set[$info['table'] . '.' . $info['column']] = true;
$seq = $this->db->fetchOne('SELECT pg_get_serial_sequence(quote_ident(?), ?)', [$info['table'], $info['column']]);
if ($seq === false || $seq === null) {
continue;
@@ -597,5 +632,26 @@ final class DbDumper {
[\Doctrine\DBAL\ParameterType::STRING, \Doctrine\DBAL\ParameterType::INTEGER, \Doctrine\DBAL\ParameterType::BOOLEAN]);
}
}
// A backup from MySQL/MariaDB or SQLite records no counters (those derive them from the
// data): every other counter of a restored table goes after its data, or new rows (a new
// file, an activity) would collide with restored ones.
$restored = array_flip($restored);
$rows = $this->db->fetchAllAssociative(
"SELECT c.table_name, c.column_name, pg_get_serial_sequence(quote_ident(c.table_name), c.column_name) AS seq
FROM information_schema.columns c
WHERE c.table_schema = current_schema() AND pg_get_serial_sequence(quote_ident(c.table_name), c.column_name) IS NOT NULL");
foreach ($rows as $r) {
if (!isset($restored[$r['table_name']]) || isset($set[$r['table_name'] . '.' . $r['column_name']])) {
continue;
}
$max = (int)$this->db->fetchOne(sprintf('SELECT COALESCE(MAX(%s), 0) FROM %s',
$this->db->quoteIdentifier($r['column_name']), $this->db->quoteIdentifier($r['table_name'])));
// never backwards: a counter already past the data stays where it is
$state = $this->db->fetchAssociative('SELECT last_value, is_called FROM ' . self::sequenceIdentifier($r['seq']));
$next = $state === false ? 1 : (int)$state['last_value'] + ($state['is_called'] ? 1 : 0);
if ($next <= $max) {
$this->db->executeQuery('SELECT setval(CAST(? AS regclass), ?, false)', [$r['seq'], $max + 1]);
}
}
}
}
+10
View File
@@ -15,4 +15,14 @@ final class Format {
$time = gmdate('H:i:s', $seconds % 86400);
return $days > 0 ? $days . 'd ' . $time : $time;
}
/**
* "2026-10-29 04:30" in $zone. Nextcloud runs PHP in UTC; occ shows times in the zone of the
* schedule, the administrator's, as the web interface does.
*
* @param non-empty-string $zone
*/
public static function time(int $timestamp, string $zone): string {
return (new \DateTimeImmutable('@' . $timestamp))->setTimezone(new \DateTimeZone($zone))->format('Y-m-d H:i');
}
}
+78 -2
View File
@@ -7,6 +7,7 @@ declare(strict_types=1);
namespace OCA\NgBackup\Repository;
use OCA\NgBackup\Backend\BackendException;
use OCA\NgBackup\Backend\IBackend;
use OCA\NgBackup\Crypto\StreamCipher;
@@ -32,6 +33,8 @@ final class BlobIndex {
private const CACHE_LOT = 256;
/** @var int index files downloaded by the last load() (for tests and diagnostics) */
public int $downloaded = 0;
/** @var array<string, true> packs whose index file is loaded (refresh() fetches only others) */
private array $loaded = [];
public function __construct(
private IBackend $backend,
@@ -90,13 +93,14 @@ final class BlobIndex {
* longer exist on the backend are dropped from the cache.
*/
public function load(bool $verify = false, ?callable $progress = null): void {
$this->blobs = [];
$hadIndex = $this->blobs !== [];
$cached = $verify ? [] : ($this->cache?->all($this->repositoryId) ?? []);
$present = [];
$this->downloaded = 0;
$order = []; // pack ids in listing order: where a blob is in two packs, the last listed wins
$fetch = [];
foreach ($this->backend->listFiles('index') as $path) {
$known = array_keys($this->loaded + $cached + ($verify ? ($this->cache?->all($this->repositoryId) ?? []) : []));
foreach ($this->listIndex($known) as $path) {
$packId = basename($path);
$present[$packId] = true;
$order[] = $packId;
@@ -104,6 +108,9 @@ final class BlobIndex {
$fetch[] = $path;
}
}
// only now: an empty listing that listIndex() refused leaves the index as it was
$this->blobs = [];
$this->loaded = $present;
// The rest is downloaded several at a time where the location allows it: on a large
// repository these are tens of thousands of small files. getMany() keeps their order, so
// the cached packs listed before each one are added first.
@@ -185,12 +192,80 @@ final class BlobIndex {
rewind($stream);
$this->backend->put('index/' . $packId, $stream);
$this->cachePut($packId, $entries);
$this->loaded[$packId] = true;
foreach ($entries as $e) {
$this->blobs[$e['id']] = [$packId, $e['offset'], $e['length'], $e['raw'], $e['flags']];
unset($this->pending[$e['id']]);
}
}
/**
* Add the index files that appeared on the location since load(), such as those of packs
* another server's cleanup wrote while a restore runs. Only adds and updates: a blob is
* never dropped, as a listing that misses files (a connection that failed) must not make a
* restore lose blobs it knows. A blob in a new pack takes that pack's place.
*/
public function refresh(): void {
$fetch = [];
$listed = 0;
foreach ($this->listIndex(array_keys($this->loaded)) as $path) {
$listed++;
if (!isset($this->loaded[basename($path)])) {
$fetch[] = $path;
}
}
$toCache = [];
foreach ($this->backend->getMany($fetch) as $path => $data) {
$packId = basename($path);
/** @var list<array{id:string, offset:int, length:int, raw:int, flags:int}> $entries authenticated: written by NG Backup */
$entries = json_decode($this->cipher->decryptString($data, 'index:' . $packId), true, 512, JSON_THROW_ON_ERROR);
$this->downloaded++;
$this->add($packId, $entries);
$this->loaded[$packId] = true;
$toCache[$packId] = $entries;
}
if ($toCache !== []) {
$this->cache?->putMany($this->repositoryId, $toCache);
}
}
/**
* The index files on the location. An empty listing while index files are known ($known,
* pack ids) is usually a connection that failed quietly; taken as is, it would empty the
* index and the cache, make a check report every blob missing, or make a cleanup wipe the
* cache. So the connection is checked (the repository's config must be there) and the folder
* listed once more. Empty again, three of the known index files are looked up one by one:
* only when they are gone too is the repository taken as empty (another server removed its
* last snapshot and cleaned up). Even when that is wrong, nothing on the location is lost:
* only this index and the cache are empty until the next load.
*
* @param list<string> $known
* @return list<string>
* @throws BackendException when the location does not answer as it should
*/
private function listIndex(array $known): array {
$paths = $this->backend->listFiles('index');
if ($paths !== [] || $known === []) {
return $paths;
}
if (!$this->backend->exists('config')) {
throw new BackendException('The location listed no index files and has no repository config: not trusting that listing');
}
$paths = $this->backend->listFiles('index');
if ($paths !== []) {
return $paths;
}
// spread over the list: first, middle and last (another server's cleanup removes packs
// here and there, not in this order)
$n = count($known);
foreach (array_unique([$known[0], $known[intdiv($n, 2)], $known[$n - 1]]) as $packId) {
if ($this->backend->exists('index/' . $packId)) {
throw new BackendException('The location listed no index files, while index/' . $packId . ' is there: not trusting that listing');
}
}
return [];
}
/** @return array<string, list<array{id:string, offset:int, length:int}>> packId => blobs currently located in that pack */
public function packs(): array {
$packs = [];
@@ -204,6 +279,7 @@ final class BlobIndex {
public function removePack(string $packId): void {
$this->backend->delete('index/' . $packId);
$this->cacheRemove([$packId]);
unset($this->loaded[$packId]);
foreach ($this->blobs as $id => $loc) {
if ($loc[0] === $packId) {
unset($this->blobs[$id]);
+130 -20
View File
@@ -72,6 +72,14 @@ final class Repository {
if ($backend->exists('config')) {
throw new RepositoryException('A repository already exists at this location');
}
// No config, but backup data: the config is missing (or could not be seen just now, a
// connection that failed). A new repository here would write a config over the one that
// opens this data; refuse instead.
foreach (['index', 'snapshots', 'trash/snapshots', 'catalog'] as $dir) {
if ($backend->listFiles($dir) !== []) {
throw new RepositoryException("This location holds backup data ($dir/) but no repository config could be read: not creating a new repository over it. Check the connection and the folder; a missing config can be restored from the recovery kit.");
}
}
$id = bin2hex(random_bytes(16));
$config = ['format' => self::FORMAT, 'id' => $id, 'created' => gmdate('c'), 'key' => $wrappedKey,
'check' => base64_encode((new StreamCipher($keys))->encryptString($id, 'config-check'))];
@@ -206,7 +214,8 @@ final class Repository {
}
/**
* Restore a snapshot (or only paths starting with $prefix) into a local directory.
* Restore a snapshot (or only paths starting with $prefix) into a local directory. A $prefix
* "dir/file/" also restores the file "dir/file" itself.
*
* @param bool $stripPrefix write "$target/<path without $prefix>" instead of
* "$target/<full logical path>" (disaster recovery: restoring e.g. "data/" or
@@ -245,7 +254,7 @@ final class Repository {
$plan = new ReadAheadPlan(rtrim($target, '/') . '/.ngb-restore-spool-' . bin2hex(random_bytes(4)));
try {
foreach ($this->streamEntries($snapshotId) as $entry) {
if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
if (!self::underPrefix($entry['p'], $prefix)) {
continue;
}
foreach ($entry['b'] as $id) {
@@ -278,11 +287,15 @@ final class Repository {
private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan): int {
$count = 0;
$cleaned = [];
foreach ($this->streamEntries($snapshotId) as $entry) {
if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
if (!self::underPrefix($entry['p'], $prefix)) {
continue;
}
$relPath = ($stripPrefix && $prefix !== '') ? substr($entry['p'], strlen(rtrim($prefix, '/')) + 1) : $entry['p'];
$relPath = $entry['p'];
if ($stripPrefix && $prefix !== '') {
$relPath = $entry['p'] === rtrim($prefix, '/') ? basename($entry['p']) : substr($entry['p'], strlen($prefix));
}
$dest = rtrim($target, '/') . '/' . $relPath;
if (!is_dir(dirname($dest))) {
mkdir(dirname($dest), 0700, true);
@@ -293,6 +306,10 @@ final class Repository {
if ($base === false || $dir === false || ($dir !== $base && !str_starts_with($dir, $base . '/')) || is_link($dest)) {
throw new RepositoryException('Refusing to write outside the restore directory: ' . $entry['p']);
}
if (!isset($cleaned[$dir])) {
self::removeStaleTemporaries($dir);
$cleaned[$dir] = true;
}
// Write next to the destination and rename after all blobs verified: at most one file
// is ever present twice.
$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
@@ -323,6 +340,30 @@ final class Repository {
return $count;
}
/** $prefix is "" (everything) or ends with "/"; the path without that "/" is a file to restore too. */
private static function underPrefix(string $path, string $prefix): bool {
return $prefix === '' || str_starts_with($path, $prefix) || $path === rtrim($prefix, '/');
}
/**
* Remove what a restore that was stopped left in a directory this restore writes to:
* "<name>.ngb-restore-<hex>" from here, ".<name>.ngb-restore-<hex>" from a restore through
* Nextcloud. A files:scan would otherwise make them show up in Files.
*/
public static function removeStaleTemporaries(string $dir): void {
$names = scandir($dir);
foreach ($names === false ? [] : $names as $name) {
$path = $dir . '/' . $name;
if (self::isRestoreTemporary($name) && is_file($path) && !is_link($path)) {
@unlink($path);
}
}
}
public static function isRestoreTemporary(string $name): bool {
return preg_match('/.\.ngb-restore-[0-9a-f]{6}(?:[0-9a-f]{6})?$/', $name) === 1;
}
/**
* Start writing a generated stream (e.g. a table dump) as blobs. Call flushPacks() when the
* run is done, so the last pack is uploaded before anything references it.
@@ -553,6 +594,31 @@ final class Repository {
return isset($this->catalog->userExports[$manifestPath]);
}
/**
* The recorded export $ref names: a manifest path (users/<uid>/<exportId>) as is, or a bare
* user id (also "users/<uid>") when that user has exactly one export here. Anything else
* throws, with the exports to choose from in the message.
*/
public function resolveUserExport(string $ref): string {
if ($this->isRecordedUserExport($ref)) {
return $ref;
}
$rest = trim($ref, '/');
if (str_starts_with($rest, 'users/')) {
$rest = substr($rest, strlen('users/'));
}
$uid = explode('/', $rest)[0];
$exports = $uid !== '' ? $this->listUserExports($uid) : [];
if (!str_contains($rest, '/') && count($exports) === 1) {
return $exports[0];
}
if ($exports === []) {
throw new RepositoryException("$ref is not a recorded user export for this location, and $uid has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports");
}
$which = str_contains($rest, '/') ? "$ref is not a recorded user export for this location" : "$uid has " . count($exports) . ' exports here';
throw new RepositoryException("$which; pass one of these (newest first):\n " . implode("\n ", $exports));
}
/** Permanently delete a user export. Its blobs are freed by the next prune() like any other. */
public function forgetUserExport(string $manifestPath): void {
if (!isset($this->catalog->userExports[$manifestPath])) {
@@ -837,13 +903,21 @@ final class Repository {
try {
foreach ($this->backend->getRanges($ranges) as $i => $data) {
foreach ($parts[$i] as [$id, $at, $length]) {
yield $this->decodeBlob($id, substr($data, $at, $length));
try {
$blob = $this->decodeBlob($id, substr($data, $at, $length));
} catch (\Exception) {
// a read that went wrong on the way (see loadBlob()): this blob on its own
$blob = $this->loadBlob($id);
// its pack stream is closed before the next range is read on the same connection
$this->packReader->close();
}
yield $blob;
$done++;
}
}
} catch (BackendException $e) {
// A pack moved (see loadBlob()): the rest one blob at a time, from the fresh index.
$this->refreshIndex($e);
} catch (BackendException) {
// A pack moved, or the read failed for a while (see loadBlob()): the rest one blob at a
// time, each read once more after a failure.
foreach (array_slice($ids, $done) as $id) {
yield $this->loadBlob($id);
}
@@ -872,25 +946,61 @@ final class Repository {
$encrypted = $this->packReader->read($pack, $offset, $length);
} catch (BackendException $e) {
// Another server's cleanup may have repacked this blob into a new pack since the index
// was loaded (a disaster recovery test while the original server still runs). Reload
// the index from the location; read again only if the blob has moved.
$this->refreshIndex($e);
// was loaded (a disaster recovery test while the original server still runs): add the
// new index files, and read the blob where it is now. Or the read failed for a while
// (a connection that died; the backend has replaced it by now): then read it once
// more where it was. A second failure is the real one.
try {
$this->index->refresh();
} catch (\Throwable) {
// the location could not be listed either: read again where the blob was
}
[$moved, $offset, $length] = $this->index->get($id);
if ($moved === $pack) {
try {
$encrypted = $this->packReader->read($moved, $offset, $length);
} catch (BackendException) {
throw $e;
}
$encrypted = $this->packReader->read($moved, $offset, $length);
$pack = $moved;
}
try {
return $this->decodeBlob($id, $encrypted);
} catch (\Exception $damaged) {
// Bytes that do not decrypt: damaged on the location, or a read that went wrong on the
// way (a stream that lost its place: an SMB restore stopped after 17 hours like this).
// Read exactly this blob once more, on a request of its own; only a second failure
// means the data itself is damaged.
$this->packReader->close();
try {
$again = $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
} catch (\Exception) {
throw $damaged;
}
$data = $this->decodeBlob($id, $again);
if (function_exists('OCP\Log\logger')) {
\OCP\Log\logger('ng_backup')->warning('NG Backup: blob ' . $id . ' did not decrypt as first read from pack ' . $pack . ' at ' . $offset . '; read again, it is fine. First read: ' . self::howItDiffered($encrypted, $again));
}
return $data;
}
return $this->decodeBlob($id, $encrypted);
}
/** Reload the index from the location after a failed pack read; rethrows that failure if it cannot. */
private function refreshIndex(BackendException $cause): void {
try {
$this->index->load();
} catch (\Throwable) {
throw $cause;
/**
* What a read that went wrong looked like next to the right bytes, for the log: where it starts
* to differ, and whether the rest is zeros or the right bytes from another place (a stream that
* lost its place).
*/
private static function howItDiffered(string $wrong, string $right): string {
$at = strspn($wrong ^ $right, "\0");
$text = strlen($wrong) . ' bytes, differ from byte ' . $at . ' of ' . strlen($right);
$tail = substr($wrong, $at, 64);
if ($tail === '') {
return $text . ' (too short)';
}
if (trim($tail, "\0") === '') {
return $text . ' (zeros)';
}
$found = strpos($right, $tail);
return $text . ($found !== false ? ' (the right bytes from ' . ($found - $at) . ' bytes further on)' : ' (other bytes)');
}
/** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */
+6
View File
@@ -34,6 +34,8 @@ use Psr\Log\LoggerInterface;
*/
final class BackupService {
public const KIND_FULL = 'full';
/** Characters of a snapshot label (the column holds 255) */
public const LABEL_MAX = 100;
/** @var \WeakMap<Repository, string> the location settings each kept repository was opened with */
private \WeakMap $openedWith;
@@ -57,6 +59,10 @@ final class BackupService {
}
public function start(Target $target, string $label = ''): Run {
$label = trim($label);
if (mb_strlen($label) > self::LABEL_MAX) {
throw new \InvalidArgumentException('A label has at most ' . self::LABEL_MAX . ' characters');
}
if (!$this->keys->everConfirmed()) {
throw new \RuntimeException('Confirm the recovery kit first (occ backup:key:confirm)');
}
+45 -2
View File
@@ -27,9 +27,14 @@ final class RestoreFullService {
* Config keys that must keep pointing at THIS installation's actual, already-working
* database and data directory (set by occ maintenance:install), never at the old server's.
*/
/** App setting: when a full restore started that has not completed (see restoreFull()). */
private const INTERRUPTED_KEY = 'restore_full_started';
private const CONNECTION_KEYS = [
'dbtype', 'dbhost', 'dbport', 'dbname', 'dbuser', 'dbpassword', 'dbpersistent',
'dbtableprefix', 'dbdriveroptions', 'datadirectory',
// set by restoreFull() itself for the length of the restore
'maintenance',
];
public function __construct(
@@ -85,12 +90,50 @@ final class RestoreFullService {
];
}
/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>} */
/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>, maintenanceLeftOn:bool} */
public function restoreFull(Target $target, string $snapshotId, bool $skipMissingTables = false): array {
$check = $this->checkCompatibility($target, $snapshotId, $skipMissingTables);
// Maintenance mode for the whole restore: background jobs (cron.php stops in maintenance
// mode), page visits (AJAX cron) and users must not write into tables or the data
// directory while they are being replaced; a row written in between would collide with a
// restored one or be silently lost. Switched off after a restore that completed. After an
// error it stays on: the instance may be partly restored, and must not be used like that.
// occ only offers app commands outside maintenance mode, so the admin switches it off
// right before running the restore again (see RestoreFull). Already on before: left on.
// On before the checks below, so that a run again after an error leaves the partly
// restored instance open for seconds at most; off again when they refuse (nothing changed).
// A marker in ng_backup's own app settings (which the restore keeps) says a restore started
// and did not complete: then the instance may be partly restored, and a check that refuses
// a run again must not open it.
$appConfig = \OCP\Server::get(\OCP\IAppConfig::class);
$interrupted = $appConfig->getValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, 0) > 0;
$wasInMaintenance = $this->config->getSystemValueBool('maintenance', false);
$this->config->setSystemValue('maintenance', true);
$reopen = function () use ($wasInMaintenance, $interrupted): void {
if (!$wasInMaintenance && !$interrupted) {
$this->config->setSystemValue('maintenance', false);
}
};
try {
$check = $this->checkCompatibility($target, $snapshotId, $skipMissingTables);
} catch (\Throwable $e) {
$reopen();
throw $e;
}
if (!$check['ok']) {
$reopen();
throw new \RuntimeException($this->describeBlockers($check));
}
$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check);
$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
if (!$wasInMaintenance) {
$this->config->setSystemValue('maintenance', false);
}
return $result + ['maintenanceLeftOn' => $wasInMaintenance];
}
/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>} */
private function restoreInMaintenance(Target $target, string $snapshotId, bool $skipMissingTables, array $check): array {
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $skipMissingTables, $check) {
$beat = LeaseService::throttled($refresh);
$repo = $this->targets->repository($target);
+21
View File
@@ -33,6 +33,9 @@ final class RestoreService {
public const MODE_MERGE = 'merge';
public const MODE_REPLACE = 'replace';
/** @var array<string, true> folders of the current restore already cleared of stale temporaries */
private array $cleaned = [];
public function __construct(
private TargetService $targets,
private IRootFolder $root,
@@ -133,6 +136,7 @@ final class RestoreService {
$dest = $this->ensureFolder($userFolder, $parentRel);
$names = $single ? [basename($rel) => $entries['']] : $entries;
$this->cleaned = [];
$restored = 0;
foreach ($names as $relPath => $entry) {
$this->writeFile($repo, $dest, (string)$relPath, $entry);
@@ -178,6 +182,10 @@ final class RestoreService {
private function writeFile(Repository $repo, Folder $dest, string $relPath, array $entry): void {
$dir = dirname($relPath);
$folder = $dir === '.' ? $dest : $this->ensureFolder($dest, $dir);
if (!isset($this->cleaned[$folder->getPath()])) {
$this->removeStaleTemporaries($folder);
$this->cleaned[$folder->getPath()] = true;
}
$name = basename($relPath);
// Stage under a temporary name; only rename into place after every blob was verified.
$tmpName = '.' . $name . '.ngb-restore-' . bin2hex(random_bytes(3));
@@ -223,6 +231,19 @@ final class RestoreService {
}
}
/** See Repository::removeStaleTemporaries(): what a restore that was stopped left in this folder. */
private function removeStaleTemporaries(Folder $folder): void {
foreach ($folder->getDirectoryListing() as $node) {
if ($node instanceof File && Repository::isRestoreTemporary($node->getName())) {
try {
$node->delete();
} catch (\Throwable) {
// in the way of nothing: the new temporary gets a name of its own
}
}
}
}
private function trashExtras(Folder $folder, string $prefix, array $keep): int {
$count = 0;
foreach ($folder->getDirectoryListing() as $node) {
+110
View File
@@ -0,0 +1,110 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Service;
use OCA\NgBackup\AppInfo\Application;
use OCP\IAppConfig;
use OCP\IConfig;
use OCP\IGroupManager;
/**
* The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a
* zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone
* is converted once to the zone of the server or of an administrator, at today's offset: it then
* shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
*/
final class Schedule {
public const TIME_KEY = 'schedule_time';
public const ZONE_KEY = 'schedule_timezone';
public function __construct(
private IAppConfig $appConfig,
private IConfig $config,
private IGroupManager $groups,
) {
}
/** @return array{time: string, timezone: non-empty-string} time "" = no schedule */
public function get(): array {
$time = $this->appConfig->getValueString(Application::APP_ID, self::TIME_KEY, '');
$zone = $this->appConfig->getValueString(Application::APP_ID, self::ZONE_KEY, '');
if ($zone === '' || !self::validZone($zone)) {
$zone = $this->guessZone();
if ($time !== '' && self::validTime($time)) {
$time = self::convert($time, 'UTC', $zone);
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
}
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
}
return ['time' => $time, 'timezone' => $zone];
}
/** @throws \InvalidArgumentException for a malformed time or an unknown zone */
public function set(string $time, string $zone): void {
if ($time !== '' && !self::validTime($time)) {
throw new \InvalidArgumentException('Use HH:MM or empty');
}
if (!self::validZone($zone)) {
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
}
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
}
/** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */
public function todayAt(?int $now = null): ?int {
['time' => $time, 'timezone' => $zone] = $this->get();
if ($time === '' || !self::validTime($time)) {
return null;
}
$tz = new \DateTimeZone($zone);
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d');
return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp();
}
/**
* "HH:MM" in zone $from as "HH:MM" in zone $to, at today's offsets.
*
* @param non-empty-string $from
* @param non-empty-string $to
*/
public static function convert(string $time, string $from, string $to, ?int $now = null): string {
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone(new \DateTimeZone($from))->format('Y-m-d');
return (new \DateTimeImmutable($day . ' ' . $time, new \DateTimeZone($from)))->setTimezone(new \DateTimeZone($to))->format('H:i');
}
public static function validTime(string $time): bool {
return preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time) === 1;
}
/** @psalm-assert-if-true non-empty-string $zone */
public static function validZone(string $zone): bool {
return in_array($zone, \DateTimeZone::listIdentifiers(), true);
}
/**
* The server's default_timezone when set, else the first administrator's, else UTC. A server
* zone of UTC is passed over on purpose: it is mostly what an image or installer left there,
* while the administrator who set the time read it as their own (#19).
*
* @return non-empty-string
*/
private function guessZone(): string {
$zone = $this->config->getSystemValueString('default_timezone', '');
if ($zone !== '' && $zone !== 'UTC' && self::validZone($zone)) {
return $zone;
}
foreach ($this->groups->get('admin')?->getUsers() ?? [] as $user) {
$zone = $this->config->getUserValue($user->getUID(), 'core', 'timezone', '');
if ($zone !== '' && self::validZone($zone)) {
return $zone;
}
}
return 'UTC';
}
}
+2 -1
View File
@@ -228,7 +228,8 @@ final class TargetService {
$s->setFiles((int)($meta['stats']['files'] ?? 0));
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
$this->snapshots->insert($s);
} catch (\Throwable) {
} catch (\Throwable $e) {
Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]);
}
}
}
-5
View File
@@ -110,11 +110,6 @@
<code><![CDATA[ask]]></code>
</UndefinedInterfaceMethod>
</file>
<file src="lib/Command/Trash.php">
<RedundantCastGivenDocblockType>
<code><![CDATA[(string)($info['time'] ?? '')]]></code>
</RedundantCastGivenDocblockType>
</file>
<file src="lib/Controller/AdminApiController.php">
<InvalidOperand>
<code><![CDATA[$maxGb * 1073741824]]></code>
+44 -3
View File
@@ -7,7 +7,7 @@
<NcSettingsSection :name="t('ng_backup', 'NG Backup')"
:description="t('ng_backup', 'Encrypted, incremental backups of this Nextcloud (database, configuration and data). Backup locations are never mounted and are not visible in Files.')">
<p class="ngb-feedback">
{{ t('ng_backup', 'NG Backup is in beta. We would like to hear how it works for you, also when everything went fine.') }}
{{ t('ng_backup', 'We would like to hear how NG Backup works for you, also when everything went fine.') }}
<a :href="feedbackUrl" target="_blank" rel="noopener noreferrer">{{ t('ng_backup', 'Share your experience ↗') }}</a>
</p>
<NcLoadingIcon v-if="!status" :size="32" />
@@ -18,9 +18,18 @@
<div class="ngb-tabs">
<NcButton :variant="tab === 'backup' ? 'primary' : 'secondary'" @click="tab = 'backup'">{{ t('ng_backup', 'Backup and restore') }}</NcButton>
<NcButton :variant="tab === 'settings' ? 'primary' : 'secondary'" @click="tab = 'settings'">{{ t('ng_backup', 'Settings') }}</NcButton>
<NcButton :variant="tab === 'docs' ? 'primary' : 'secondary'" @click="tab = 'docs'">{{ t('ng_backup', 'Docs') }}</NcButton>
</div>
<template v-if="tab === 'settings'">
<NcSettingsSection v-if="tab === 'docs'" :name="t('ng_backup', 'Documentation')">
<ul class="ngb-docs">
<li v-for="doc in docs" :key="doc.url">
<a :href="doc.url" target="_blank" rel="noopener noreferrer">{{ doc.name }} ↗</a>
<span>{{ doc.description }}</span>
</li>
</ul>
</NcSettingsSection>
<template v-else-if="tab === 'settings'">
<KeySetup :status="status" @changed="refresh" />
<Locations v-if="status.key.everConfirmed" :status="status" @changed="refresh" />
</template>
@@ -52,6 +61,25 @@ import RestoreBrowser from './components/RestoreBrowser.vue'
// Opens the experience report form in .github/ISSUE_TEMPLATE
const feedbackUrl = 'https://github.com/dillardblom/ng_backup/issues/new?template=experience.yml'
const repoUrl = 'https://github.com/dillardblom/ng_backup'
const docs = [
{
name: t('ng_backup', 'How-to'),
description: t('ng_backup', 'Step by step: from first install to a tested restore.'),
url: repoUrl + '/blob/main/docs/HOWTO.md',
},
{
name: t('ng_backup', 'Disaster recovery'),
description: t('ng_backup', 'Your server is gone: get Nextcloud back on a new server from a backup location.'),
url: repoUrl + '/blob/main/docs/DISASTER-RECOVERY.md',
},
{
name: t('ng_backup', 'Roadmap'),
description: t('ng_backup', 'What is planned for the next versions.'),
url: repoUrl + '#roadmap',
},
]
const status = ref(null)
const allRuns = ref(false) // runs older than the last days too
const error = ref('')
@@ -89,8 +117,10 @@ onBeforeUnmount(() => clearTimeout(timer))
</script>
<style scoped>
/* NcSettingsSection sets its own width (at most 900px): the runs table needs more */
.ng-backup :deep(.settings-section) {
max-width: 900px;
width: min(1400px, 100% - var(--default-grid-baseline) * 7 * 2);
max-width: none;
}
.ngb-feedback {
color: var(--color-text-maxcontrast);
@@ -100,6 +130,17 @@ onBeforeUnmount(() => clearTimeout(timer))
color: var(--color-primary-element);
text-decoration: underline;
}
.ngb-docs li {
margin-bottom: 8px;
}
.ngb-docs a {
color: var(--color-primary-element);
text-decoration: underline;
margin-right: 8px;
}
.ngb-docs span {
color: var(--color-text-maxcontrast);
}
.ngb-tabs {
display: flex;
gap: 8px;
+1 -1
View File
@@ -33,7 +33,7 @@ export default {
testTarget: (id) => call('post', `/targets/${id}/test`),
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
setSchedule: (time) => call('put', '/schedule', { time }),
setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }),
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
+34 -7
View File
@@ -6,10 +6,12 @@
<NcSettingsSection :name="t('ng_backup', 'Backups')"
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
<div class="ngb-row">
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, empty = off)')" />
<NcButton :disabled="schedule === status.schedule" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, {zone}, empty = off)', { zone: browserZone })" />
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
</div>
<div class="ngb-row">
<NcTextField v-model="label" class="ngb-label" :maxlength="100" :label="t('ng_backup', 'Label (optional)')"
:placeholder="t('ng_backup', 'e.g. before the upgrade to Nextcloud 35')" />
<NcButton v-for="target in status.targets" :key="target.id" :disabled="isRunning(target.id)" @click="start(target)">
{{ isRunning(target.id) ? t('ng_backup', 'Backup to {name} running …', { name: target.name }) : t('ng_backup', 'Back up now to {name}', { name: target.name }) }}
</NcButton>
@@ -24,9 +26,23 @@
<li v-for="item in trash[target.id]" :key="item.id">
{{ item.time ? formatDateTime(item.time) : item.id }}{{ item.label ? ' – ' + item.label : '' }}
<span class="ngb-muted">{{ t('ng_backup', 'removed by {by}, deleted after {date}', { by: item.by, date: formatDate(item.deleteAfter) }) }}</span>
<button v-if="item.by === RETENTION"
class="ngb-info"
:aria-expanded="retentionInfo ? 'true' : 'false'"
:aria-label="t('ng_backup', 'What does this mean?')"
:title="t('ng_backup', 'What does this mean?')"
@click="retentionInfo = !retentionInfo">
ⓘ
</button>
<NcButton variant="tertiary" @click="untrash(target, item)">{{ t('ng_backup', 'Restore') }}</NcButton>
</li>
</ul>
<NcNoteCard v-if="retentionInfo && trash[target.id].some(item => item.by === RETENTION)" type="info">
{{ n('ng_backup',
'The retention policy keeps the newest restore points, and for older days, weeks and months only the last restore point of each. It moves the others to this trash, where they stay for %n day before they are deleted permanently. Until then you can bring one back with "Restore". Nothing was lost by accident.',
'The retention policy keeps the newest restore points, and for older days, weeks and months only the last restore point of each. It moves the others to this trash, where they stay for %n days before they are deleted permanently. Until then you can bring one back with "Restore". Nothing was lost by accident.',
status.key.deleteDelayDays) }}
</NcNoteCard>
</div>
</template>
@@ -79,16 +95,24 @@ import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
import NcTextField from '@nextcloud/vue/components/NcTextField'
import api from '../api.js'
import { formatDateTime, formatDate, formatSize } from '../format.js'
import { formatDateTime, formatDate, formatSize, toZone } from '../format.js'
import { confirmPassword } from '@nextcloud/password-confirmation'
import { reactive } from 'vue'
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
const emit = defineEmits(['changed', 'show-all-runs'])
const schedule = ref(props.status.schedule)
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
// zone is shown in this one, so it reads as the moment it runs here.
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s
const schedule = ref(shown(props.status.schedule))
const error = ref('')
watch(() => props.status.schedule, (s) => { schedule.value = s })
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
const label = ref('')
// "by" of a restore point the weekly cleanup removed (PruneService)
const RETENTION = 'retention policy'
const retentionInfo = ref(false)
const trash = reactive({})
async function loadTrash() {
for (const target of props.status.targets) {
@@ -121,7 +145,8 @@ const targetName = (id) => props.status.targets.find(tg => tg.id === id)?.name ?
async function start(target) {
error.value = ''
try {
await api.startBackup(target.id, '')
await api.startBackup(target.id, label.value.trim())
label.value = ''
emit('changed')
} catch (e) {
error.value = e.message
@@ -131,7 +156,7 @@ async function start(target) {
async function saveSchedule() {
error.value = ''
try {
await api.setSchedule(schedule.value.trim())
await api.setSchedule(schedule.value.trim(), browserZone)
emit('changed')
} catch (e) {
error.value = e.message
@@ -142,6 +167,8 @@ async function saveSchedule() {
<style scoped>
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
.ngb-time { max-width: 260px; }
.ngb-label { max-width: 360px; }
.ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); }
.ngb-table-wrap { overflow-x: auto; }
.ngb-table { width: 100%; border-collapse: collapse; }
.ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); }
+27
View File
@@ -26,3 +26,30 @@ export function formatSize(bytes) {
}
return i === 0 ? `${b} B` : `${b.toFixed(1)} ${units[i]}`
}
/**
* A time of day "HH:MM" in zone `from`, as "HH:MM" in zone `to`, at today's offsets.
*
* @param {string} hhmm the time
* @param {string} from IANA zone it is in
* @param {string} to IANA zone to show it in
* @param {Date} now the day to convert on (today)
* @return {string}
*/
export function toZone(hhmm, from, to, now = new Date()) {
const [h, m] = hhmm.split(':').map(Number)
const parts = (zone, date) => Object.fromEntries(new Intl.DateTimeFormat('en-GB', {
timeZone: zone, hourCycle: 'h23', year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit',
}).formatToParts(date).map((p) => [p.type, p.value]))
const offsetAt = (at) => {
const seen = parts(from, new Date(at))
return Date.UTC(+seen.year, +seen.month - 1, +seen.day, +seen.hour, +seen.minute) - at
}
const today = parts(from, now)
const wall = Date.UTC(+today.year, +today.month - 1, +today.day, h, m)
// the offset at the wall time itself: on a day the clocks change it differs from the one at
// midnight, so it is looked up a second time at the moment found first
const at = wall - offsetAt(wall - offsetAt(wall))
const there = parts(to, new Date(at))
return `${there.hour}:${there.minute}`
}
@@ -0,0 +1,35 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Backend;
use OCA\NgBackup\Backend\ExternalStorageFactory;
use PHPUnit\Framework\TestCase;
/** ExternalStorageFactory::normalizePrivateKey(): keys that phpseclib 2 read, for phpseclib 3 (#24). */
class ExternalStorageFactoryTest extends TestCase {
/** An OpenSSH-format key laid out as ssh-keygen writes it; the body is random, not a real key. */
private static function key(): string {
$body = base64_encode(random_bytes(1000));
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
}
public function testRestoresLostLineBreaks(): void {
$key = self::key();
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey($key));
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", ' ', $key)), 'newlines as spaces');
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", '', $key)), 'no newlines');
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", "\r\n", $key)), 'CRLF');
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(rtrim($key)), 'no trailing newline');
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(preg_replace('/^/m', ' ', $key)), 'indented');
}
public function testLeavesOtherFormatsAlone(): void {
$pem = "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC,FF44E0A94036766C\n\nAAAA\n-----END RSA PRIVATE KEY-----\n";
$this->assertSame($pem, ExternalStorageFactory::normalizePrivateKey($pem));
$this->assertSame('not a key', ExternalStorageFactory::normalizePrivateKey('not a key'));
$odd = "-----BEGIN OPENSSH PRIVATE KEY-----\\nAAAA\\n-----END OPENSSH PRIVATE KEY-----";
$this->assertSame($odd, ExternalStorageFactory::normalizePrivateKey($odd), 'not base64 inside: unchanged');
}
}
@@ -184,6 +184,38 @@ class SftpRangeFetcherTest extends TestCase {
$this->assertSame(array_fill(0, 10, null), $result);
}
public function testABrokenConnectionIsReplacedAndTheRestIsReadAhead(): void {
$connections = $this->connections(1, [['loseAt' => 1048576]]);
$made = [];
$connect = function () use (&$made): object {
$made[] = $c = new FakeSftpConnection($this->files);
return SftpReadStream::connection(new FakeSftpStorage($c));
};
$ranges = $this->ranges();
$result = self::collect(SftpRangeFetcher::fetch(self::opened($connections), '/home/', $ranges, $this->tempDir(), $connect));
$expected = $this->expected($ranges);
$expected[0] = null; // the range it was reading when it broke: read the usual way
$this->assertSame($expected, $result);
$this->assertCount(1, $made);
$this->assertTrue($made[0]->disconnected, 'the new one ends with the fetcher too');
}
public function testConnectionsThatKeepBreakingAreReplacedOnlySoOften(): void {
$made = 0;
$connect = function () use (&$made): object {
$made++;
return SftpReadStream::connection(new FakeSftpStorage(new FakeSftpConnection($this->files, loseAt: 0)));
};
$ranges = array_merge($this->ranges(), $this->ranges(), $this->ranges());
$result = self::collect(SftpRangeFetcher::fetch(self::opened($this->connections(1, [['loseAt' => 0]])), '/home/', $ranges, $this->tempDir(), $connect));
$this->assertSame(array_fill(0, count($ranges), null), $result);
$this->assertSame(SftpRangeFetcher::MAX_RECONNECTS, $made);
}
public function testStoppingHalfwayEndsTheConnectionsAndLeavesNothingBehind(): void {
$connections = $this->connections(2);
$spool = $this->tempDir();
@@ -77,6 +77,24 @@ class StorageBackendReconnectTest extends TestCase {
$this->assertSame(1, $backend->reconnects);
}
public function testASilentlyDroppedConnectionIsReplacedAfterAPause(): void {
$backend = $this->backend();
$this->connections[0]->dropped = true;
$this->connections[0]->silent = true; // phpseclib still says connected
self::idle($backend);
$this->assertTrue($backend->exists('config'));
$this->assertSame(1, $backend->reconnects, 'the check failed, so the connection was replaced');
}
public function testAFailureOnASilentlyDroppedConnectionIsRepeated(): void {
$backend = $this->backend();
$this->connections[0]->dropped = true;
$this->connections[0]->silent = true;
$fh = $backend->get('config');
$this->assertSame('config', stream_get_contents($fh));
$this->assertSame(1, $backend->reconnects);
}
public function testAMissingFileOnALiveConnectionIsJustMissing(): void {
$backend = $this->backend();
$this->assertFalse($backend->exists('nothing'));
@@ -121,6 +139,8 @@ interface SftpLikeStorage extends IStorage {
/** The public phpseclib calls the backend uses to tell a dead connection. */
final class ReconnectFakeSftp {
public bool $dropped = false;
/** Dropped without phpseclib noticing (as a Hetzner Storage Box does): it still says connected. */
public bool $silent = false;
private bool $connected = true;
public int $requests = 0;
@@ -139,6 +159,9 @@ final class ReconnectFakeSftp {
public function request(): bool {
$this->requests++;
if ($this->dropped) {
if ($this->silent) {
return false;
}
$this->connected = false;
}
return $this->connected;
+7
View File
@@ -14,4 +14,11 @@ class FormatTest extends TestCase {
$this->assertSame('1d 00:00:00', Format::duration(86400));
$this->assertSame('1d 12:10:13', Format::duration(86400 + 12 * 3600 + 10 * 60 + 13));
}
public function testTimeInZone(): void {
$t = (int)strtotime('2026-10-19 13:12:00 UTC');
$this->assertSame('2026-10-19 13:12', Format::time($t, 'UTC'));
$this->assertSame('2026-10-19 15:12', Format::time($t, 'Europe/Berlin')); // summer time
$this->assertSame('2026-12-01 01:00', Format::time((int)strtotime('2026-12-01 00:00:00 UTC'), 'Europe/Amsterdam'));
}
}
+133
View File
@@ -0,0 +1,133 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\IBackend;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/**
* A pack stream that hands out wrong bytes once (an SMB restore stopped after 17 hours with
* "Decryption failed"): the blob is read once more on a request of its own; only data that is
* damaged on the location stops the restore.
*/
class DamagedReadTest extends TestCase {
use TempDirTrait;
private string $root;
private KeyRing $keys;
private string $src;
private string $snapshot;
protected function setUp(): void {
$this->root = $this->tempDir();
$this->keys = KeyRing::generate();
$this->src = $this->tempDir();
for ($i = 0; $i < 6; $i++) {
file_put_contents("$this->src/f$i", random_bytes(3 * 1048576));
}
$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
}
public function testABlobThatCameInDamagedOnceIsReadAgain(): void {
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1);
$target = $this->tempDir();
$this->assertSame(6, Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $target));
$this->assertSame(self::hashTree($this->src), self::hashTree($target));
$this->assertSame(0, $backend->streams, 'the damaged read happened');
}
public function testATableDumpThatCameInDamagedOnceIsReadAgain(): void {
$repo = Repository::openWithKey(new LocalBackend($this->root), $this->keys);
$stats = [];
$writer = $repo->blobWriter($stats);
$lines = [];
for ($i = 0; $i < 20000; $i++) {
$lines[] = json_encode(['i' => $i, 'pad' => str_repeat('x', $i % 400)]);
}
$writer->write(implode("\n", $lines) . "\n");
$blobs = $writer->finish();
$repo->flushPacks();
$backend = new GarblingBackend(new LocalBackend($this->root), ranges: 1);
$this->assertSame($lines, iterator_to_array(Repository::openWithKey($backend, $this->keys)->readLines($blobs), false));
$this->assertSame(0, $backend->ranges, 'the damaged read happened');
}
public function testDataDamagedOnTheLocationStillStopsTheRestore(): void {
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1000, ranges: 1000);
$this->expectException(\Exception::class);
Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $this->tempDir());
}
}
/** Pack reads that flip a byte: the first $streams pack streams and $ranges range reads. */
final class GarblingBackend implements IBackend {
use \OCA\NgBackup\Backend\GetsOneByOne;
public function __construct(
private IBackend $inner,
public int $streams = 0,
public int $ranges = 0,
) {
}
public function put(string $path, $stream): void {
$this->inner->put($path, $stream);
}
public function get(string $path) {
$fh = $this->inner->get($path);
if (!str_starts_with($path, 'packs/') || $this->streams <= 0) {
return $fh;
}
$this->streams--;
$data = stream_get_contents($fh);
fclose($fh);
for ($k = 100; $k < strlen($data); $k += 65536) {
$data[$k] = chr(ord($data[$k]) ^ 0x55);
}
$out = fopen('php://temp', 'w+b');
fwrite($out, $data);
rewind($out);
return $out;
}
public function getRange(string $path, int $offset, int $length): string {
$data = $this->inner->getRange($path, $offset, $length);
if (str_starts_with($path, 'packs/') && $this->ranges > 0) {
$this->ranges--;
$data[0] = chr(ord($data[0]) ^ 0x55);
}
return $data;
}
public function exists(string $path): bool {
return $this->inner->exists($path);
}
public function move(string $from, string $to): void {
$this->inner->move($from, $to);
}
public function list(string $prefix): array {
return $this->inner->list($prefix);
}
public function listFiles(string $dir): array {
return $this->inner->listFiles($dir);
}
public function delete(string $path): void {
$this->inner->delete($path);
}
}
+239
View File
@@ -0,0 +1,239 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\BackendException;
use OCA\NgBackup\Backend\IBackend;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Repository\IIndexCache;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/**
* A listing of the location that comes back empty because a connection failed quietly (an SFTP
* connection a Storage Box dropped) must not empty the index. In a 1 TB disaster recovery test a
* pack read failed after 13 hours, the reload of the index got an empty listing, and the restore
* stopped with "Blob not in index".
*/
class IndexListingTest extends TestCase {
use TempDirTrait;
private string $root;
private KeyRing $keys;
private string $src;
private string $snapshot;
protected function setUp(): void {
$this->root = $this->tempDir();
$this->keys = KeyRing::generate();
$this->src = $this->tempDir();
for ($i = 0; $i < 12; $i++) {
file_put_contents("$this->src/f$i", random_bytes(3 * 1048576));
}
$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
}
private static function cache(): IIndexCache {
return indexCache();
}
public function testARestoreGoesOnAfterAPackReadFailedAndTheListingCameBackEmpty(): void {
$backend = new FlakyBackend(new LocalBackend($this->root));
$repo = Repository::openWithKey($backend, $this->keys);
$backend->failPackReads = 1; // the read of a pack fails once
$backend->emptyIndexListings = 1; // and listing the index right after comes back empty
$target = $this->tempDir();
$this->assertSame(12, $repo->restore($this->snapshot, $target));
$this->assertSame(self::hashTree($this->src), self::hashTree($target));
$this->assertSame(0, $backend->failPackReads, 'the failure happened');
$this->assertSame(0, $backend->emptyIndexListings, 'the empty listing was seen');
}
public function testAPackThatStaysUnreadableIsStillReportedAsSuch(): void {
$backend = new FlakyBackend(new LocalBackend($this->root));
$repo = Repository::openWithKey($backend, $this->keys);
$backend->failPackReads = 1000;
try {
$repo->restore($this->snapshot, $this->tempDir());
$this->fail('the packs cannot be read');
} catch (BackendException $e) {
$this->assertStringStartsWith('Not found: packs/', $e->getMessage());
}
}
public function testACheckWithAnEmptyListingIsAnErrorNotEveryBlobMissing(): void {
$backend = new FlakyBackend(new LocalBackend($this->root));
$repo = Repository::openWithKey($backend, $this->keys);
$backend->dead = true;
$this->expectException(BackendException::class);
$this->expectExceptionMessage('listed no index files');
$repo->verify($this->snapshot);
}
public function testRefreshingFetchesOnlyIndexFilesItHasNotLoaded(): void {
$backend = new FlakyBackend(new LocalBackend($this->root));
$repo = Repository::openWithKey($backend, $this->keys);
$index = (new \ReflectionProperty($repo, 'index'))->getValue($repo);
$index->downloaded = 0;
$index->refresh();
$this->assertSame(0, $index->downloaded, 'every index file is loaded already, also those whose blobs are elsewhere too');
}
public function testARestoreGoesOnWhenTheLocationCannotBeListedAfterAFailedRead(): void {
$backend = new FlakyBackend(new LocalBackend($this->root));
$repo = Repository::openWithKey($backend, $this->keys);
$backend->failPackReads = 1;
$backend->dead = true; // listing and config look gone; reading packs works again
$target = $this->tempDir();
$this->assertSame(12, $repo->restore($this->snapshot, $target));
$this->assertSame(self::hashTree($this->src), self::hashTree($target));
}
public function testAnEmptyListingThatWasAHiccupIsListedAgain(): void {
$cache = self::cache();
$backend = new FlakyBackend(new LocalBackend($this->root));
Repository::openWithKey($backend, $this->keys, $cache);
$cached = $cache->data;
$backend->emptyIndexListings = 1;
$again = Repository::openWithKey($backend, $this->keys, $cache);
$this->assertSame($cached, $cache->data);
$this->assertSame(12, $again->restore($this->snapshot, $this->tempDir()));
}
public function testAListingThatStaysEmptyWhileIndexFilesAreThereIsNotTaken(): void {
$cache = self::cache();
$backend = new FlakyBackend(new LocalBackend($this->root));
Repository::openWithKey($backend, $this->keys, $cache);
$cached = $cache->data;
$backend->emptyIndexListings = 5; // the config answers, the listing keeps lying
try {
Repository::openWithKey($backend, $this->keys, $cache);
$this->fail('index files are there');
} catch (BackendException $e) {
$this->assertStringContainsString('is there', $e->getMessage());
}
$this->assertSame($cached, $cache->data);
}
public function testARepositoryThatReallyBecameEmptyIsTakenAsEmpty(): void {
$cache = self::cache();
$backend = new FlakyBackend(new LocalBackend($this->root));
Repository::openWithKey($backend, $this->keys, $cache);
$this->assertNotSame([], array_filter($cache->data));
// another server forgot the last snapshot and cleaned up: no packs and index files left
foreach (array_merge($backend->list('index'), $backend->list('packs')) as $path) {
$backend->delete($path);
}
Repository::openWithKey($backend, $this->keys, $cache);
$this->assertSame([], array_filter($cache->data), 'the stale cache rows are gone');
}
public function testOpeningWithAnEmptyListingKeepsTheCachedIndex(): void {
$cache = self::cache();
$backend = new FlakyBackend(new LocalBackend($this->root));
Repository::openWithKey($backend, $this->keys, $cache);
$cached = $cache->data;
$this->assertNotSame([], $cached);
$backend->dead = true;
try {
Repository::openWithKey($backend, $this->keys, $cache);
$this->fail('an empty listing of a repository with an index must not be taken as is');
} catch (BackendException $e) {
$this->assertStringContainsString('listed no index files', $e->getMessage());
}
$this->assertSame($cached, $cache->data, 'the cache is untouched');
}
}
/** @internal */
function indexCache(): IIndexCache {
return new class implements IIndexCache {
public array $data = [];
public function all(string $r): array { return $this->data[$r] ?? []; }
public function put(string $r, string $p, array $e): void { $this->data[$r][$p] = $e; }
public function remove(string $r, array $ps): void { foreach ($ps as $p) { unset($this->data[$r][$p]); } }
public function putMany(string $r, array $packs): void { foreach ($packs as $p => $e) { $this->put($r, (string)$p, $e); } }
public function replaceAll(string $r, array $packs): void { $this->data[$r] = []; $this->putMany($r, $packs); }
};
}
/** A location whose pack reads fail, and whose index listings come back empty, a set number of times. */
final class FlakyBackend implements IBackend {
use \OCA\NgBackup\Backend\GetsOneByOne;
public int $failPackReads = 0;
public int $emptyIndexListings = 0;
/** The connection is gone: also the repository config looks missing. */
public bool $dead = false;
public function __construct(
private IBackend $inner,
) {
}
public function put(string $path, $stream): void {
$this->inner->put($path, $stream);
}
public function get(string $path) {
if (str_starts_with($path, 'packs/') && $this->failPackReads > 0) {
$this->failPackReads--;
throw new BackendException('Not found: ' . $path);
}
return $this->inner->get($path);
}
public function getRange(string $path, int $offset, int $length): string {
if (str_starts_with($path, 'packs/') && $this->failPackReads > 0) {
$this->failPackReads--;
throw new BackendException('Not found: ' . $path);
}
return $this->inner->getRange($path, $offset, $length);
}
public function exists(string $path): bool {
return !$this->dead && $this->inner->exists($path);
}
public function move(string $from, string $to): void {
$this->inner->move($from, $to);
}
public function list(string $prefix): array {
return $this->inner->list($prefix);
}
public function listFiles(string $dir): array {
if ($dir === 'index' && $this->dead) {
return [];
}
if ($dir === 'index' && $this->emptyIndexListings > 0) {
$this->emptyIndexListings--;
return [];
}
return $this->inner->listFiles($dir);
}
public function delete(string $path): void {
$this->inner->delete($path);
}
}
+54
View File
@@ -87,6 +87,46 @@ class RepositoryTest extends TestCase {
$this->assertSame(10, $repo->restore($r['snapshot'], $target, 'docs/d1/'));
}
public function testRestoreOfOneFile(): void {
$repo = Repository::init($this->backend, 'pw');
$r = $repo->backupDirectory($this->src);
$target = $this->tempDir();
// occ backup:restore --to-directory passes the path with a "/" appended
$this->assertSame(1, $repo->restore($r['snapshot'], $target, 'docs/d1/note-1.txt/'));
$this->assertSame(file_get_contents($this->src . '/docs/d1/note-1.txt'), file_get_contents($target . '/docs/d1/note-1.txt'));
$this->assertSame(0, $repo->restore($r['snapshot'], $this->tempDir(), 'docs/d1/note-1/'), 'a prefix of a name is not that file');
$stripped = $this->tempDir();
$this->assertSame(1, $repo->restore($r['snapshot'], $stripped, 'docs/d1/note-1.txt/', true));
$this->assertFileExists($stripped . '/note-1.txt');
}
public function testRestoreRemovesTemporariesOfAStoppedRestore(): void {
$repo = Repository::init($this->backend, 'pw');
$r = $repo->backupDirectory($this->src);
$target = $this->tempDir();
mkdir($target . '/docs/d1', 0700, true);
mkdir($target . '/elsewhere', 0700, true);
$stale = [
$target . '/docs/d1/note-1.txt.ngb-restore-0123456789ab', // from a restore to a directory
$target . '/docs/d1/.note-4.txt.ngb-restore-a1b2c3', // from a restore through Nextcloud
];
$kept = [
$target . '/docs/d1/notes.ngb-restore-draft.txt', // not a temporary of ours
$target . '/elsewhere/x.ngb-restore-0123456789ab', // not a directory this restore writes to
];
foreach ([...$stale, ...$kept] as $f) {
file_put_contents($f, 'partial');
}
$repo->restore($r['snapshot'], $target, 'docs/');
foreach ($stale as $f) {
$this->assertFileDoesNotExist($f);
}
foreach ($kept as $f) {
$this->assertFileExists($f);
}
}
public function testWrongPassphrase(): void {
Repository::init($this->backend, 'pw');
$this->expectException(CryptoException::class);
@@ -99,6 +139,20 @@ class RepositoryTest extends TestCase {
Repository::init($this->backend, 'pw');
}
public function testInitRefusesALocationWithBackupDataButNoConfig(): void {
$repo = Repository::init($this->backend, 'pw');
$repo->backupDirectory($this->src);
$this->backend->delete('config'); // missing, or not seen for a moment
try {
Repository::init($this->backend, 'pw');
$this->fail('a new repository over existing backup data');
} catch (RepositoryException $e) {
$this->assertStringContainsString('holds backup data', $e->getMessage());
}
$this->assertFalse($this->backend->exists('config'), 'no new config written');
}
public function testTamperedPackIsDetected(): void {
$repo = Repository::init($this->backend, 'pw');
$r = $repo->backupDirectory($this->src);
+47
View File
@@ -106,4 +106,51 @@ class UserExportTest extends TestCase {
$this->expectExceptionMessage('missing on the location');
$repo->listUserExports();
}
public function testResolveTakesARecordedPathAsIs(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$path = 'users/alice/20261005T120000Z-aaaa0001';
$this->putManifest($repo, $path);
$this->assertSame($path, $repo->resolveUserExport($path));
}
public function testResolveTakesAUserIdWithOneExport(): void {
// Issue #22: "backup:user:restore <location> New_User" was rejected as not a recorded export.
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$path = 'users/alice/20261005T120000Z-aaaa0001';
$this->putManifest($repo, $path);
$this->putManifest($repo, 'users/alice2/20261005T120000Z-aaaa0002');
$this->assertSame($path, $repo->resolveUserExport('alice'));
$this->assertSame($path, $repo->resolveUserExport('users/alice/'));
}
public function testResolveListsTheExportsOfAUserWithMore(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
$this->putManifest($repo, 'users/alice/20261006T120000Z-aaaa0002');
$this->expectException(RepositoryException::class);
$this->expectExceptionMessage("alice has 2 exports here; pass one of these (newest first):\n users/alice/20261006T120000Z-aaaa0002\n users/alice/20261005T120000Z-aaaa0001");
$repo->resolveUserExport('alice');
}
public function testResolveListsTheExportsOfTheUserOfAWrongPath(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
$this->expectException(RepositoryException::class);
$this->expectExceptionMessage("users/alice/typo is not a recorded user export for this location; pass one of these (newest first):\n users/alice/20261005T120000Z-aaaa0001");
$repo->resolveUserExport('users/alice/typo');
}
public function testResolveNamesTheExpectedFormForAUserWithoutExports(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
$this->expectException(RepositoryException::class);
$this->expectExceptionMessage('bob has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports');
$repo->resolveUserExport('bob');
}
}
+71
View File
@@ -0,0 +1,71 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Service;
use OCA\NgBackup\Service\Schedule;
use OCP\IAppConfig;
use OCP\IConfig;
use OCP\IGroup;
use OCP\IGroupManager;
use OCP\IUser;
use PHPUnit\Framework\TestCase;
/** The daily time in a time zone (issue #19), and the conversion of a time saved without one. */
class ScheduleTest extends TestCase {
private array $values = [];
private function schedule(string $serverZone = '', string $adminZone = ''): Schedule {
$app = $this->createMock(IAppConfig::class);
$app->method('getValueString')->willReturnCallback(fn (string $a, string $k, string $d = '') => $this->values[$k] ?? $d);
$app->method('setValueString')->willReturnCallback(function (string $a, string $k, string $v): bool {
$this->values[$k] = $v;
return true;
});
$config = $this->createMock(IConfig::class);
$config->method('getSystemValueString')->willReturn($serverZone);
$config->method('getUserValue')->willReturn($adminZone);
$user = $this->createMock(IUser::class);
$user->method('getUID')->willReturn('admin');
$group = $this->createMock(IGroup::class);
$group->method('getUsers')->willReturn([$user]);
$groups = $this->createMock(IGroupManager::class);
$groups->method('get')->willReturn($group);
return new Schedule($app, $config, $groups);
}
public function testATimeSavedWithoutAZoneShowsAsItRanInTheAdministratorsZone(): void {
$this->values = [Schedule::TIME_KEY => '02:30'];
$s = $this->schedule(adminZone: 'Europe/Berlin')->get();
// it ran at 02:30 UTC: in summer that is 04:30 in Berlin, in winter 03:30
$summer = Schedule::convert('02:30', 'UTC', 'Europe/Berlin', strtotime('2026-07-01 12:00 UTC'));
$this->assertSame('04:30', $summer);
$this->assertSame('Europe/Berlin', $s['timezone']);
$this->assertSame(Schedule::convert('02:30', 'UTC', 'Europe/Berlin'), $s['time']);
}
public function testTheServersDefaultZoneComesFirst(): void {
$this->values = [Schedule::TIME_KEY => '01:00'];
$this->assertSame('America/Toronto', $this->schedule('America/Toronto', 'Europe/Berlin')->get()['timezone']);
}
public function testWithoutAnyZoneItStaysUtc(): void {
$this->values = [Schedule::TIME_KEY => '02:30'];
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
}
public function testTodayAtIsTheMomentInTheSchedulesZone(): void {
$this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin'];
$at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC'));
$this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC');
}
public function testAnUnknownZoneOrABadTimeIsRefused(): void {
$s = $this->schedule();
$this->expectException(\InvalidArgumentException::class);
$s->set('03:00', 'Mars/Olympus');
}
}
+65 -10
View File
@@ -2,10 +2,13 @@
# Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a
# SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup
# location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7.
# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp]
# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp|to-pgsql|to-mysql|to-sqlite]
# e.g. full-restore-roundtrip.sh 35-apache pgsql
# With "sftp" the location is an SFTP server instead of a shared directory, and the origin has a
# file spread over several packs: the restore reads them ahead on connections of its own.
# With "to-<db>" the restore instance runs that database whatever the origin runs (a MySQL or
# SQLite backup records no ID counters; after the restore new rows must not collide with restored
# ones, whatever the database on either side).
# With "legacy" the origin runs NG Backup 1.0.0-rc.1 (0.9.7), whose backups do not record the
# structure of their tables: the restore must then leave the tables of the disabled app out
# with --skip-missing-tables, as for a real backup made before 1.0.0-rc.2.
@@ -44,6 +47,12 @@ if [ "$LEGACY" = legacy ]; then
fi
run_on() { docker exec -u www-data "$1" php occ "${@:2}"; }
# The app store sends the whole app list (~38 MB) and Nextcloud gives up after 120 s: on a slow day
# an install fails for reasons outside this test. Up to three tries.
install_app() { # container app
for i in 1 2 3; do run_on "$1" app:install "$2" >/dev/null 2>&1 && return 0; sleep 20; done
return 1
}
# Database settings for an instance: SQLite inside the container, or its own PostgreSQL container
db_env() {
@@ -106,7 +115,7 @@ if [ "$DB" != sqlite ]; then
# while the restore loads tables in name order, accounts before mailboxes.
docker run -d --name ngb-frt-imap --network "$NET" -e MAILNAME=mail.domain.tld -e MAIL_ADDRESS=user@domain.tld \
-e MAIL_PASS=mypassword ghcr.io/christophwurst/docker-imap-devel:latest >/dev/null
run_on ngb-frt-origin app:install mail >/dev/null || fail "install mail on the origin"
install_app ngb-frt-origin mail || fail "install mail on the origin (app store unreachable?)"
run_on ngb-frt-origin config:system:set app.mail.verify-tls-peer --type=boolean --value=false >/dev/null
sleep 10
run_on ngb-frt-origin mail:account:create frtest Test user@domain.tld ngb-frt-imap 993 ssl user@domain.tld mypassword \
@@ -115,7 +124,9 @@ if [ "$DB" != sqlite ]; then
[ "$(sql origin 'select count(*) from oc_mail_accounts where drafts_mailbox_id is not null')" = 1 ] \
|| fail "the mail account does not point at its mailboxes"
# Deck installed and then switched off: its tables stay in the database and in the backup
run_on ngb-frt-origin app:install deck >/dev/null || fail "install deck on the origin"
install_app ngb-frt-origin deck || fail "install deck on the origin (app store unreachable?)"
# a 4-byte character (emoji) in a table the restore has to create: utf8mb4 on MySQL/MariaDB
sql origin "insert into oc_deck_boards (title, owner, color, archived, deleted_at, last_modified) values ('Plan 📌', 'frtest', '0087C5', false, 0, 0)" >/dev/null
run_on ngb-frt-origin app:disable deck >/dev/null
ok "origin has a synced Mail account and the tables of a disabled app"
fi
@@ -127,6 +138,8 @@ ok "test data backed up, snapshot $SNAP"
docker cp ngb-frt-origin:/tmp/kit.json /tmp/ngb-frt-kit.json
docker exec ngb-frt-origin sh -c "cat /tmp/pass" > /tmp/ngb-frt-pass
ORIGIN_DBTYPE=$DB
case "$MODE" in to-*) DB=${MODE#to-} ;; esac
RESTORE_DB=$(db_env restore)
# shellcheck disable=SC2086
docker run -d --name ngb-frt-restore --network "$NET" $RESTORE_DB \
@@ -141,8 +154,14 @@ docker cp /tmp/ngb-frt-pass ngb-frt-restore:/tmp/pass
ok "fresh restore instance up"
run_on ngb-frt-restore backup:key:import /tmp/kit.json --passphrase-file=/tmp/pass >/dev/null || fail "key import"
run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo >/dev/null || fail "target add on the restore instance"
run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location"
ADD_OUT=$(run_on ngb-frt-restore backup:target:add offsite "${LOCATION[@]}" --path=repo 2>&1) || { printf '%s\n' "$ADD_OUT"; fail "target add on the restore instance"; }
LIST_OUT=$(run_on ngb-frt-restore backup:list offsite 2>&1)
printf '%s\n' "$LIST_OUT" | grep -q "$SNAP" || {
printf 'target:add said:\n%s\nbackup:list said:\n%s\n' "$ADD_OUT" "$LIST_OUT"
docker exec ngb-frt-restore sh -c 'grep -h ng_backup /var/www/html/data/nextcloud.log | tail -5' || true
docker exec ngb-frt-restore sh -c 'ls -la /mnt/shared/repo /mnt/shared/repo/snapshots' || true
fail "the origin's snapshot is not listed after connecting the location"
}
ok "recovery kit imported, existing repository adopted, snapshot visible"
RESTORE_OPTS=(--no-interaction --install-apps)
@@ -154,6 +173,20 @@ if [ "$LEGACY" = legacy ] && [ "$DB" != sqlite ]; then
ok "older backup refused without --skip-missing-tables, missing tables named"
RESTORE_OPTS+=(--skip-missing-tables)
fi
if [ "$DB" = sqlite ]; then
# Another process writing while the restore starts (a cron job that was already running, or
# the installer still finishing after it says "installed"): the restore must wait for it, not
# stop with "database is locked". Short write transactions on a table of its own, until the
# restore is done (/tmp/frt-side-stop; Nextcloud's own commands after it are not under test).
docker exec -d -u www-data ngb-frt-restore php -r '
$p = new PDO("sqlite:/var/www/html/data/nextcloud.db"); $p->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$p->exec("CREATE TABLE IF NOT EXISTS frt_side_writer (t REAL)");
for ($end = microtime(true) + 300; microtime(true) < $end && !file_exists("/tmp/frt-side-stop");) {
$p->beginTransaction(); $p->exec("INSERT INTO frt_side_writer VALUES (" . microtime(true) . ")"); usleep(300000); $p->commit(); usleep(20000);
}
touch("/tmp/frt-side-done");'
sleep 1
fi
sftp_logins() { docker logs ngb-frt-sftp 2>&1 | grep -c "Accepted password" || true; }
[ "$MODE" = sftp ] && LOGINS_BEFORE=$(sftp_logins)
OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS[@]}" 2>&1) || {
@@ -161,23 +194,45 @@ OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS
docker exec ngb-frt-restore sh -c 'tail -c 6000 /var/www/html/data/nextcloud.log' || true
fail "restore:full"
}
ok "backup:restore:full completed"
printf '%s\n' "$OUT" | grep -q "maintenance mode while the restore runs" || fail "the restore did not say it runs in maintenance mode"
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the restore"
run_on ngb-frt-restore config:app:get ng_backup restore_full_started >/dev/null 2>&1 && fail "the restore left its 'started' marker behind"
ok "backup:restore:full completed, in maintenance mode and out of it again"
if [ "$DB" = sqlite ]; then
docker exec ngb-frt-restore touch /tmp/frt-side-stop
for i in $(seq 1 20); do docker exec ngb-frt-restore test -f /tmp/frt-side-done && break; sleep 0.5; done
WRITES=$(docker exec -u www-data ngb-frt-restore php -r 'echo (new PDO("sqlite:/var/www/html/data/nextcloud.db"))->query("SELECT COUNT(*) FROM frt_side_writer")->fetchColumn();')
[ "${WRITES:-0}" -gt 3 ] || fail "the other writer did not run alongside the restore ($WRITES writes)"
ok "the restore waited for another process writing to SQLite ($WRITES writes alongside)"
fi
# Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
# earlier bugs here only showed up after config.php's secret was rewritten mid-restore.
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after restore (own key/target undecryptable?)"
run_on ngb-frt-restore user:list | grep -q frtest || fail "the backed-up user is missing from the restored database"
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database"
docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
# New rows after the restore: a new file (file cache) and a new user must not collide with
# restored rows (ID counters after the data).
docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "a new file cannot be added after the restore"
if [ "$DB" != sqlite ]; then
[ "$(sql restore "select count(*) from oc_filecache where path = 'files/Docs/after-restore.txt'")" = 1 ] \
|| fail "a new file did not make it into the file cache after the restore (ID counters?)"
fi
docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-restore php occ user:add --password-from-env frnew >/dev/null || fail "a new user cannot be added after the restore"
ok "database, files and ng_backup's own state are all intact and usable after the restore"
if [ "$DB" != sqlite ] && [ "$LEGACY" = legacy ]; then
# Mail and Deck were only set up on an origin with a database server (see above)
if [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ] && [ "$LEGACY" = legacy ]; then
[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 0 ] || fail "a table of unknown structure was created"
[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
ok "older backup: tables of the disabled app left out, Mail restored"
elif [ "$DB" != sqlite ]; then
elif [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ]; then
[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created"
[ "$(sql restore "select title from oc_deck_boards")" = "Plan 📌" ] || fail "a 4-byte character did not survive in a created table"
[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
[ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \
|| fail "the foreign keys of Mail are missing after the restore"
@@ -197,4 +252,4 @@ fi
run_on ngb-frt-restore backup:run offsite -l post-restore-check >/dev/null || fail "a new backup after the restore fails"
ok "the restored instance can run a new backup of its own"
echo "FULL RESTORE ROUNDTRIP PASS ($DB${MODE:+, $MODE})"
echo "FULL RESTORE ROUNDTRIP PASS ($ORIGIN_DBTYPE${MODE:+, $MODE})"
+13 -2
View File
@@ -49,9 +49,20 @@ SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xa
$X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash"
ok "restore new-folder and replace: files identical, extra file in the trash"
# One file to a local directory, where a restore that was stopped left its temporary.
ONE=/tmp/ngb-rt-one/data/$U/files/Docs
$X sh -c "mkdir -p $ONE && echo partial > $ONE/f2.bin.ngb-restore-0123456789ab"
occ backup:restore ci "$SNAP" "data/$U/files/Docs/f2.bin" -d /tmp/ngb-rt-one | grep "Restored 1 files" >/dev/null || fail "restore of one file to a directory"
$X sh -c "cmp -s $ONE/f2.bin $DATA/$U/files/Docs/f2.bin" || fail "the file restored to a directory differs"
$X sh -c "ls -A $ONE | grep -q ngb-restore" && fail "the temporary of a stopped restore was left"
ok "restore of one file to a directory; the temporary of a stopped restore removed"
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
occ backup:status | grep "started (" >/dev/null || fail "backup:status does not name the time zone"
ok "backup:status names the time zone of its times"
# Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob).
occ backup:verify ci "$SNAP" | grep -q "^OK$" || fail "shallow verify"
occ backup:verify ci "$SNAP" --deep | grep -q "^OK$" || fail "deep verify"
occ backup:verify ci "$SNAP" | grep "^OK$" >/dev/null || fail "shallow verify"
occ backup:verify ci "$SNAP" --deep | grep "^OK$" >/dev/null || fail "deep verify"
ok "shallow and deep checksum audit"
# Database dump/restore round trip (all tables, checksums, sequences on PostgreSQL).
+13 -2
View File
@@ -36,8 +36,14 @@ MANIFEST=$(occ backup:user:backup ci "$U" | sed -n 's/.* to \(.*\)\.$/\1/p')
[ -n "$MANIFEST" ] || fail "backup:user:backup printed no manifest path"
ok "backed up $U to $MANIFEST"
# as-backup: restore into <uid>-bak; the original account must stay untouched.
occ backup:user:restore ci "$MANIFEST" --mode=as-backup >/dev/null || fail "restore as-backup"
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
occ backup:user:list ci "$U" | grep -F "$MANIFEST" >/dev/null || fail "backup:user:list does not show $MANIFEST"
ok "backup:user:list shows the export"
# as-backup: restore into <uid>-bak; the original account must stay untouched. $U has one export,
# so the bare user id names it (#22).
OUT=$(occ backup:user:restore ci "$U" --mode=as-backup) || fail "restore as-backup by user id"
echo "$OUT" | grep -qF "Using $MANIFEST." || fail "restore by user id did not say which export it used"
occ files:scan "$U-bak" >/dev/null
SUM_BAK=$($X sh -c "cd $DATA/$U-bak/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
[ "$SUM_BEFORE" = "$SUM_BAK" ] || fail "the <uid>-bak account's files differ from the original"
@@ -55,6 +61,11 @@ SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xa
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after a replace restore"
ok "replace restores the original content under the same uid, byte-identical"
# The replace above added a safety export: a bare user id is now ambiguous and must list both.
if OUT=$(occ backup:user:restore ci "$U" --mode=as-backup 2>&1); then fail "restore by an ambiguous user id ran"; fi
echo "$OUT" | grep -qF "$U has 2 exports here" && echo "$OUT" | grep -qF "$MANIFEST" || fail "an ambiguous user id did not list the exports"
ok "an ambiguous user id lists the exports instead of picking one"
# Retention: $U now has the manual export above plus the automatic safety-export the replace
# restore made of it just before deleting it (4 total after these two more), then --user-last=2
# must forget exactly the 2 oldest.