- RetentionPolicy (last/daily/weekly/monthly) and PruneService: forget
snapshots the policy drops, delete packs nothing uses and repack mostly
unused ones by copying encrypted blobs as-is; new packs are indexed
before old ones are removed
- a location lock keeps prune and backups apart; append-only locations
are never pruned (cleanup via the location's lifecycle rules)
- weekly prune from the background job; occ backup:prune [--dry-run],
backup:retention, backup:target:append-only
- tests: forget+prune frees space and the remaining snapshot restores
identically, repack of mixed packs, dry-run preview, retention buckets
- AdminApiController (admin only, CSRF-protected): status, create key,
download recovery kit, confirm (statement accepted + exact phrase),
backends with their usable auth mechanisms, add/test/remove locations,
start backups, schedule, snapshots, browse, queue restores
- RestoreJob runs web-initiated restores in the background; backup and
restore runs are kept apart in the run table
- admin section 'NG Backup' with a Vue 3 / @nextcloud/vue 9 page: key and
recovery-kit flow (download, tick statement, 'Yes, I confirm'), location
form built from files_external's parameter definitions, schedule, run
list with polling, snapshot browser and restore with the three modes
- auth mechanisms that depend on a user session or on files_external's
own stored credentials are not offered
API tested with curl on NC34 (status, backends, schedule validation,
confirmation refusal, browse, queued restore and backup completed by
cron); non-admins get 403, anonymous 401.
Opening a repository lists index/ once and downloads only index files
that are not cached; packs written by this server are cached on write,
stale cache rows are dropped. Each resumable step therefore no longer
downloads every index file from the location.
BackupService starts a run per location (database dump in one
transaction, then config + data directory as a resumable file run),
stores the state in ngb_runs after every step, locks a run while a step
executes and records the snapshot. Previews, logs, updater backups and
per-user cache/uploads are excluded (TreeWalker now supports patterns).
BackupJob (every 5 min) starts the daily schedule and steps running runs
with a short budget under web cron and a long one under system cron.
occ: backup:run, backup:status, backup:list.
Tested on NC34/PostgreSQL: first run 128 files / 1 GiB in 64 s (443 MiB
uploaded to S3), unchanged second run 2 s / 0 bytes; a background run to
SFTP completed through cron.php.
- migration: ngb_targets, ngb_runs, ngb_snapshots
- KeyService: master key stored encrypted with the instance secret
(unattended backups), passphrase-wrapped copy in every repository and
in the recovery kit, recorded admin confirmation per key fingerprint
- TargetService: locations built from files_external backends without a
mount; files_external is loaded through IAppManager::loadApp() and its
enabled state is left alone; credentials encrypted with ICrypto;
existing repositories are adopted only if made with this key, and a
repository cannot be registered twice
- Repository: initWithKey/openWithKey with an encrypted key check
- occ: backup:key:init|kit|confirm, backup:target:backends|add|list|remove
- detect files changing during reading or between resumable steps and
re-read them (size/mtime/ctime/inode), spike F overwrites mid-run
- drop pending blob ids when a pack or index upload fails
- dump and restore PostgreSQL sequence state exactly, bound parameters,
validated catalog names; spike C checks all sequences exactly
- keep invalid UTF-8 byte-exact (tagged base64)
- refuse large tables without a primary key
- StorageBackend: check write/flush/close results, validate base path
- user_migration import/export: root paths, intermediate folders,
partial writes, read errors
- spikes: never touch an existing account, own fixtures, 512M note
DbDumper reads every prefixed table through the Nextcloud connection in
one REPEATABLE READ (read-only) transaction, with keyset pagination on
the primary key, and streams each table as JSON lines into its own blob
stream (BlobWriter), so unchanged tables are deduplicated. Restore
replaces table contents in one transaction and resets PostgreSQL
sequences.
Spike C (NC34, PostgreSQL, standard image without pg_dump/pgsql ext):
300k-row table with bytea: dump 6.3s / 50 MiB peak, 94 MiB JSON -> 20 MiB
stored; unchanged second dump stores nothing; restore makes every table
checksum-identical and all 110 sequences continue above the maximum.
dillard