- tests/integration/roundtrip.sh: key, kit code and confirmation,
location, full and unchanged incremental backup, restore new-folder and
replace (files identical, extra file in the trash), database dump and
restore, retention, trash and restore from trash, no files_external
errors in the log
- tests/integration/matrix.sh: fresh Nextcloud per version and database
(SQLite, MariaDB, PostgreSQL), removes only its own containers and
their volumes; locally green on NC32 and NC35 with all three databases
- GitHub Actions: unit tests on PHP 8.2-8.4, round trip on NC32-35 x
SQLite/MariaDB/PostgreSQL
Fixes found by the matrix:
- files_external is no longer booted when it is disabled: only its
classes are loaded and a private BackendService gets its backends from
its Application class (booting registered its mount provider, which
failed on installations where its tables were never created)
- SQLite: the database file is excluded from the file backup (it is
covered by the logical dump and copying the live file is inconsistent)
- occ backup:key:confirm accepts --code and --confirm for scripts
- DbDumper::dumpStep(): tables are dumped until the step's deadline in one
REPEATABLE READ transaction and resumed at the last primary key in the
next step; the running table checksum is carried over. A dump that fits
in one step is fully consistent, otherwise the snapshot records
dbConsistent=false. BackupService uses it, so the database phase no
longer needs one long request (hosted Nextcloud, web cron)
- restore inserts 100 rows per statement and leaves tables whose checksum
already matches the dump untouched
- spike C: 30-step dump gives the same checksums as the one-shot dump;
restore rewrote only the 3 damaged tables of 147; 300k-row table
restored in 14.6 s instead of 46 s
- one locking protocol per location: backup steps and restores hold a
shared lock, prune and untrash an exclusive one and refuse while any
run (backup, restore, export) exists; starting a backup checks and
creates the run under the exclusive lock, closing the race with prune
- snapshot entries are validated when read (relative path without '.',
'..', NUL, backslash or control characters); raw restore also checks
that the resolved directory stays below the target; the restore API
and service accept canonical paths only
- recovery kit confirmation requires a recent password confirmation and
the confirmation code printed in the current kit file (new code per
kit version), proving the kit was actually downloaded
- prune decides on the authenticated index files on the location and
refreshes the local index cache from them
- restore job marks the run failed on any error; restore runs without
progress for 6 hours are failed so they cannot block cleanup forever
- tests for safe paths and a malicious snapshot path
- removed snapshots go to a trash on the location and their data is only
freed after the deletion delay, which is chosen when the key is created
(default 7 days, 1-365) and cannot be changed afterwards; any admin can
bring them back (occ backup:trash, backup:trash:restore, web UI)
- up to three passphrase slots open the same master key (like LUKS), so
one person leaving or dying does not lock the company out; slot changes
are written to every location, announced to all admins and require a
new recovery kit confirmation, while backups keep running
- occ backup:key:slots, backup:key:slot:add|replace|remove; key init takes
--label and --delete-delay-days
- web UI: deletion delay and first slot at key creation, slot management,
trash per location with restore; all behind password confirmation
- README: key, slots (safe / CTO / head of IT), and protection against
ransomware: object lock in compliance mode (e.g. 90 days), credentials
without delete rights, provider snapshots, off-site copy
- notifications to administrators for failed backup/restore runs and
locations without a successful backup for two days (once a day)
- setup check in the admin overview: error when the key/kit is not set
up, warning for missing, old or failed backups
- security events (recovery kit downloaded, location removed,
append-only turned off, retention changed, snapshots forgotten) go to
the audit log and are announced to all administrators, from the web
and from occ; the scheduled weekly cleanup is logged only
- fix: the recovery-kit download link failed the CSRF check (412); it
now requires a recent password confirmation instead, like creating the
key, removing a location and restoring
- RetentionPolicy (last/daily/weekly/monthly) and PruneService: forget
snapshots the policy drops, delete packs nothing uses and repack mostly
unused ones by copying encrypted blobs as-is; new packs are indexed
before old ones are removed
- a location lock keeps prune and backups apart; append-only locations
are never pruned (cleanup via the location's lifecycle rules)
- weekly prune from the background job; occ backup:prune [--dry-run],
backup:retention, backup:target:append-only
- tests: forget+prune frees space and the remaining snapshot restores
identically, repack of mixed packs, dry-run preview, retention buckets
- AdminApiController (admin only, CSRF-protected): status, create key,
download recovery kit, confirm (statement accepted + exact phrase),
backends with their usable auth mechanisms, add/test/remove locations,
start backups, schedule, snapshots, browse, queue restores
- RestoreJob runs web-initiated restores in the background; backup and
restore runs are kept apart in the run table
- admin section 'NG Backup' with a Vue 3 / @nextcloud/vue 9 page: key and
recovery-kit flow (download, tick statement, 'Yes, I confirm'), location
form built from files_external's parameter definitions, schedule, run
list with polling, snapshot browser and restore with the three modes
- auth mechanisms that depend on a user session or on files_external's
own stored credentials are not offered
API tested with curl on NC34 (status, backends, schedule validation,
confirmation refusal, browse, queued restore and backup completed by
cron); non-admins get 403, anonymous 401.
BackupService starts a run per location (database dump in one
transaction, then config + data directory as a resumable file run),
stores the state in ngb_runs after every step, locks a run while a step
executes and records the snapshot. Previews, logs, updater backups and
per-user cache/uploads are excluded (TreeWalker now supports patterns).
BackupJob (every 5 min) starts the daily schedule and steps running runs
with a short budget under web cron and a long one under system cron.
occ: backup:run, backup:status, backup:list.
Tested on NC34/PostgreSQL: first run 128 files / 1 GiB in 64 s (443 MiB
uploaded to S3), unchanged second run 2 s / 0 bytes; a background run to
SFTP completed through cron.php.
dillard