Opening a repository lists index/ once and downloads only index files
that are not cached; packs written by this server are cached on write,
stale cache rows are dropped. Each resumable step therefore no longer
downloads every index file from the location.
BackupService starts a run per location (database dump in one
transaction, then config + data directory as a resumable file run),
stores the state in ngb_runs after every step, locks a run while a step
executes and records the snapshot. Previews, logs, updater backups and
per-user cache/uploads are excluded (TreeWalker now supports patterns).
BackupJob (every 5 min) starts the daily schedule and steps running runs
with a short budget under web cron and a long one under system cron.
occ: backup:run, backup:status, backup:list.
Tested on NC34/PostgreSQL: first run 128 files / 1 GiB in 64 s (443 MiB
uploaded to S3), unchanged second run 2 s / 0 bytes; a background run to
SFTP completed through cron.php.
- migration: ngb_targets, ngb_runs, ngb_snapshots
- KeyService: master key stored encrypted with the instance secret
(unattended backups), passphrase-wrapped copy in every repository and
in the recovery kit, recorded admin confirmation per key fingerprint
- TargetService: locations built from files_external backends without a
mount; files_external is loaded through IAppManager::loadApp() and its
enabled state is left alone; credentials encrypted with ICrypto;
existing repositories are adopted only if made with this key, and a
repository cannot be registered twice
- Repository: initWithKey/openWithKey with an encrypted key check
- occ: backup:key:init|kit|confirm, backup:target:backends|add|list|remove
- detect files changing during reading or between resumable steps and
re-read them (size/mtime/ctime/inode), spike F overwrites mid-run
- drop pending blob ids when a pack or index upload fails
- dump and restore PostgreSQL sequence state exactly, bound parameters,
validated catalog names; spike C checks all sequences exactly
- keep invalid UTF-8 byte-exact (tagged base64)
- refuse large tables without a primary key
- StorageBackend: check write/flush/close results, validate base path
- user_migration import/export: root paths, intermediate folders,
partial writes, read errors
- spikes: never touch an existing account, own fixtures, 512M note
DbDumper reads every prefixed table through the Nextcloud connection in
one REPEATABLE READ (read-only) transaction, with keyset pagination on
the primary key, and streams each table as JSON lines into its own blob
stream (BlobWriter), so unchanged tables are deduplicated. Restore
replaces table contents in one transaction and resets PostgreSQL
sequences.
Spike C (NC34, PostgreSQL, standard image without pg_dump/pgsql ext):
300k-row table with bytea: dump 6.3s / 50 MiB peak, 94 MiB JSON -> 20 MiB
stored; unchanged second dump stores nothing; restore makes every table
checksum-identical and all 110 sequences continue above the maximum.
dillard