- RetentionPolicy (last/daily/weekly/monthly) and PruneService: forget
snapshots the policy drops, delete packs nothing uses and repack mostly
unused ones by copying encrypted blobs as-is; new packs are indexed
before old ones are removed
- a location lock keeps prune and backups apart; append-only locations
are never pruned (cleanup via the location's lifecycle rules)
- weekly prune from the background job; occ backup:prune [--dry-run],
backup:retention, backup:target:append-only
- tests: forget+prune frees space and the remaining snapshot restores
identically, repack of mixed packs, dry-run preview, retention buckets
- AdminApiController (admin only, CSRF-protected): status, create key,
download recovery kit, confirm (statement accepted + exact phrase),
backends with their usable auth mechanisms, add/test/remove locations,
start backups, schedule, snapshots, browse, queue restores
- RestoreJob runs web-initiated restores in the background; backup and
restore runs are kept apart in the run table
- admin section 'NG Backup' with a Vue 3 / @nextcloud/vue 9 page: key and
recovery-kit flow (download, tick statement, 'Yes, I confirm'), location
form built from files_external's parameter definitions, schedule, run
list with polling, snapshot browser and restore with the three modes
- auth mechanisms that depend on a user session or on files_external's
own stored credentials are not offered
API tested with curl on NC34 (status, backends, schedule validation,
confirmation refusal, browse, queued restore and backup completed by
cron); non-admins get 403, anonymous 401.
User files are restored through the Files API so the file cache,
versions and trash stay consistent: modes new-folder (default, into
'Restored <date>/<path>'), merge (overwritten files keep their old
content as a version) and replace (extra files go to the trash). Each
file is staged under a temporary name and moved into place only after
all its blobs were verified. Any snapshot path can also be restored raw
to a local directory (disaster recovery).
Tested on NC34: all three modes restore checksum-identical files, the
overwritten content is kept as a version, the extra file is in the
trash, no temporary files remain.
Opening a repository lists index/ once and downloads only index files
that are not cached; packs written by this server are cached on write,
stale cache rows are dropped. Each resumable step therefore no longer
downloads every index file from the location.
BackupService starts a run per location (database dump in one
transaction, then config + data directory as a resumable file run),
stores the state in ngb_runs after every step, locks a run while a step
executes and records the snapshot. Previews, logs, updater backups and
per-user cache/uploads are excluded (TreeWalker now supports patterns).
BackupJob (every 5 min) starts the daily schedule and steps running runs
with a short budget under web cron and a long one under system cron.
occ: backup:run, backup:status, backup:list.
Tested on NC34/PostgreSQL: first run 128 files / 1 GiB in 64 s (443 MiB
uploaded to S3), unchanged second run 2 s / 0 bytes; a background run to
SFTP completed through cron.php.
TreeWalker walks one or more roots in a stable segment-wise order with
exclusions and resumes after a cursor; ParentCursor merge-joins the
previous snapshot in the same order. BackupRun no longer keeps a file
list or the parent's entries in memory or in its state.
- migration: ngb_targets, ngb_runs, ngb_snapshots
- KeyService: master key stored encrypted with the instance secret
(unattended backups), passphrase-wrapped copy in every repository and
in the recovery kit, recorded admin confirmation per key fingerprint
- TargetService: locations built from files_external backends without a
mount; files_external is loaded through IAppManager::loadApp() and its
enabled state is left alone; credentials encrypted with ICrypto;
existing repositories are adopted only if made with this key, and a
repository cannot be registered twice
- Repository: initWithKey/openWithKey with an encrypted key check
- occ: backup:key:init|kit|confirm, backup:target:backends|add|list|remove
Dev tooling via composer (phpunit, psalm, nextcloud/ocp); vendor/ is
mounted from outside the repository by a helper script. 28 unit tests
ported from the phase-0 spikes (round trips, corruption, wrong keys,
dedup, incremental runs, partial restore, tampering, object binding,
time-boxed steps, file changed between steps).
- detect files changing during reading or between resumable steps and
re-read them (size/mtime/ctime/inode), spike F overwrites mid-run
- drop pending blob ids when a pack or index upload fails
- dump and restore PostgreSQL sequence state exactly, bound parameters,
validated catalog names; spike C checks all sequences exactly
- keep invalid UTF-8 byte-exact (tagged base64)
- refuse large tables without a primary key
- StorageBackend: check write/flush/close results, validate base path
- user_migration import/export: root paths, intermediate folders,
partial writes, read errors
- spikes: never touch an existing account, own fixtures, 512M note
BackupRun is a resumable state machine: each step works until a deadline,
uploads its pack and tree segment, and returns the state; large files are
resumed at the next blob. Snapshot is written last.
Spike F: 200 MiB file resumed across steps in separate processes; with 6
random kill -9 the run completes and restores bit-identically.
spikes/RESULTS.md summarises phase 0 and the findings for the plan.
RepositoryExportDestination / RepositoryImportSource implement the
public OCP\UserMigration interfaces on top of the repository, so
user_migration's migrators (calendar, contacts, trashbin, account,
files) export straight into the encrypted repository.
Spike E (NC34): export 0.8s, unchanged second export stores nothing;
after deleting the user, import restores account, files (bit-identical),
calendar event and contact. Findings: calendars/address books come back
as 'migrated-*' with new object URIs; importing over an existing user
duplicates contacts and adds a migrated birthday calendar; creating the
user on import needs >128 MB (password_policy word list).
ExternalStorageFactory builds storages with files_external's own
backends and auth mechanisms from an unsaved StorageConfig (as its
ConfigAdapter does), and StorageBackend adapts any IStorage to IBackend
(writeStream, seekable range reads, temp name + rename except on S3).
IBackend::put() now owns and always closes the stream, because
files_external's S3 writeStream closes it.
Spike D (NC34): S3 (SeaweedFS) and SFTP: 120 MiB backup ~6s, unchanged
run uploads nothing, restore bit-identical, no files_external mount
created, no temp files (max 12 KiB in /tmp). Works with files_external
disabled when its classes are loaded. Backup peak memory 85-89 MiB on
these targets (to investigate).
DbDumper reads every prefixed table through the Nextcloud connection in
one REPEATABLE READ (read-only) transaction, with keyset pagination on
the primary key, and streams each table as JSON lines into its own blob
stream (BlobWriter), so unchanged tables are deduplicated. Restore
replaces table contents in one transaction and resets PostgreSQL
sequences.
Spike C (NC34, PostgreSQL, standard image without pg_dump/pgsql ext):
300k-row table with bytea: dump 6.3s / 50 MiB peak, 94 MiB JSON -> 20 MiB
stored; unchanged second dump stores nothing; restore makes every table
checksum-identical and all 110 sequences continue above the maximum.
Content-addressed repository on an IBackend (local implementation):
4 MiB blobs, deflate when it helps, each blob encrypted with its id as
associated data, ~32 MiB packs with an encrypted header, one index file
per pack, snapshots as encrypted JSON lines written last.
Spike B: 390 MiB tree, peak 54 MiB; unchanged run reads nothing; a 1 MiB
change in a 300 MiB file uploads one blob; full and partial restore are
bit-identical; tampering is detected.
dillard