files_external's SFTP read stream asks for the next 256 KiB only after
the previous answer has arrived, so reading costs one round trip per
request whatever the line speed: 4.9 MB/s at 47 ms to a Hetzner Storage
Box. SftpReadStream keeps up to eight READ requests in flight over the
same phpseclib connection (15 MB/s on that link), starting with one so
a small read does not fetch megabytes ahead.
A server that answers less than asked is handled by asking again from
where the answer ended; when more data follows, that size is used for
later reads over the same connection, never below the 32 KiB every
server must support. Without the needed connection methods, or when the
open fails, the storage's own stream is used as before.
Review notes on the open pack stream: it stayed open while a backup run went on to upload
(e.g. the snapshot after reading the file list back), which SFTP cannot do on one connection.
Reading moves to PackReader, and Repository reaches the location through PackReaderGuard,
which closes the stream before every other request. A pack stream that cannot seek is read up
to the offset. Deep verify no longer reports an \Error during decoding as damage.
Restores and deep verifies requested every blob separately: a new file handle and a seek on
the location each time. They read the blobs of a snapshot in the order they were written, so
the stream of the pack read last now stays open and is read on; a jump backwards, to another
pack, or more than 8 MiB ahead opens the pack again at the offset, as before. On a Hetzner
Storage Box over SMB, restoring 86 MB in 202 files went from 0.78 MB/s to 4.2 MB/s.
Opening a repository lists index/, which holds one file per pack.
StorageBackend::list() checked every entry with is_dir(), and on a
remote storage each check is a round trip. Over SFTP to a Hetzner Storage
Box (33 ms) with a 60 GiB repository, occ backup:run (30 s steps) spent
nearly every step on these requests and uploaded 45 KiB/s; with this
change it uploads 5.8 MB/s. The cost grows with the number of packs.
index/, snapshots/, catalog/ and trash/info/ only ever hold objects:
list them with the new IBackend::listFiles(), one listing without
per-entry checks.
A run only showed "running (files)", which says nothing during a backup
of a terabyte. The files phase now records an expected size on its first
step: the previous snapshot's size, or for a first backup the users'
files, versions and trash from Nextcloud's file cache (approximate, shown
with a ~). The web interface and occ backup:status show the share of it
done so far, including a large file that is still being read; it stays
below 100% until the run finishes.
A database dump that fills a whole pack (32 MiB) uploads that pack while
the dump transaction is still open, and the pack's index cache row was
inserted on the same connection. PostgreSQL refuses that (25006, read-only
transaction) and the run failed; on other databases the row was rolled
back with the dump transaction. The index now holds those cache writes
back until the transaction is over.
Catch \Exception (storage adapters throw their own exception types) instead
of \Throwable, so an \Error such as a TypeError is not retried and reported
as an unreadable blob.
verifyBlob() caught every exception from loading a blob and counted it as
a failed (corrupted or tampered) blob, so a transient network error during
a deep verify over WebDAV looked like data corruption. Read the range
separately, retry it, and report a read that keeps failing as unreadable
with its error; only a blob that was read but does not decrypt or match
its id counts as failed.
Storage and repository:
- Listing skipped every name containing ".part-", directories too, so the
exports of a user like "john.part-time" were invisible to prune on S3,
SFTP, FTP and SMB and their data was deleted. Only real temporary uploads
("<name>.part-<8 hex>") are skipped now.
- prune refuses to run while a recorded per-user export's manifest is
missing, instead of freeing that export's data.
- forget() and untrash() copy first, switch the catalog, and delete the
originals last, so a crash can no longer leave the catalog pointing at a
missing snapshot or database dump.
- Writes are checked against the expected size, so a short upload is an
error instead of a truncated pack; raw restores check rename() and touch().
- A catalog write refuses to overwrite a generation another process just
wrote, and the catalog lock lasts 300 s, so opening a large repository
inside it no longer lets the lease run out.
Backup runs and scheduling:
- A file that keeps changing is read again at most three times; then the
last read is kept and reported, instead of restarting forever and
blocking cleanup.
- Unreadable files and folders are recorded, logged, shown by occ
backup:run, stored in the snapshot and notified to admins, instead of
being skipped silently.
- After a failed scheduled backup, at most three attempts a day, an hour
apart, instead of a new run and notification every five minutes.
- The job no longer spins on a run whose lease another process holds, and
the weekly cleanup catches up after a Sunday spent backing up.
- Notifications for the per-user export retention events.
Full restore, raw directory restore, per-user export and the safety export
held a fixed-TTL lease on the location that nothing refreshed. A restore or
export running longer than that TTL let a cleanup take over and delete data
still in use. They now refresh the lease while they work (per blob, per row
batch, per written chunk), throttled to once a minute.
Also:
- backup:prune still applies snapshot retention when user-export retention
fails for a location.
- The recovery kit carries the deletion delay, so a disaster recovery keeps
the original protection instead of falling back to 7 days. Older kits
still import with the default.
- CI checkouts no longer persist the token in the git config.
- recoveryKit() has its real return type; the stale baseline entry is gone.
A full backup opened its repository at the start of a step and wrote the
catalog at the end of it, without the catalog lock. A per-user export or
restore that wrote the catalog in between could lose its entry, or the
snapshot could lose its own.
The final step now only stores the snapshot object and moves the run to a
new 'commit' phase. BackupService then records the snapshot under the
catalog lock, from a repository opened inside that lock; if the lock is
busy, the next step retries the commit. Repository::writeSnapshot() keeps
its old behaviour as putSnapshot() plus recordSnapshot().
Repository::verify(): every blob a snapshot's files and database dump reference must be in a
freshly re-downloaded, re-authenticated index (never trusted from the local cache, same reasoning
as prune()) and have its pack still present on the location -- shallow, cheap, no blob content
read, catches a deleted pack, a tampered/rolled-back index, or a database manifest that went
missing (reported as a failure, not silently skipped). With $deep, each blob is also downloaded
and decrypted, same as loadBlob() already does on every real read (AEAD authentication tag,
content hash against its blob id), done here deliberately for everything in the snapshot rather
than waiting for it to come up during an actual restore. A blob referenced by more than one file
is only checked once; pack presence is cached per pack, not re-checked per blob.
occ backup:verify <target> <snapshot> [--deep], meant to run periodically (cron, a sample of
recent snapshots), per PLAN.md's "Verificatie: periodiek checksums controleren, steekproef-restore".
The lease covering the audit is refreshed roughly every half of its TTL while blobs are checked,
rather than relying on a single long-lived lease like the other new phase-2 commands do.
Verified against a real S3-backed 1.4 GiB / 603-blob snapshot (shallow: instant; --deep: ~28s,
fully decrypted) in addition to the new unit tests and the roundtrip.sh addition (shallow and deep
on a snapshot with real, varied file sizes, now covered on every CI matrix combination).
Local review (Codex) on the first pass found three real issues, all fixed: a missing database
manifest was silently skipped instead of reported; shallow verification trusted the (possibly
stale or unauthenticated) local index cache instead of re-checking the authenticated index on the
location; and the lease had no refresh, so a --deep audit of a very large snapshot could outrun
its TTL and run unprotected against a concurrent prune.
Wires the existing RepositoryExportDestination/RepositoryImportSource (see
lib/UserMigration/) into user_migration's own UserMigrationService, resolved on demand
through a new UserMigrationServiceLocator since the app is an optional dependency
(PLAN.md 3.8) -- without it, only this feature is unavailable, the full backup still
works.
New: UserBackupService::backupUser() (occ backup:backup:user) streams a user_migration
export straight into the repository, under the same shared location lock as a regular
backup. UserRestoreService::restoreUser() (occ backup:restore:user) never imports over a
live account: --mode=replace safety-exports the current state first, then deletes and
recreates the account from the backup under the same uid; --mode=as-backup restores into
a new "<uid>-bak" account instead, since Nextcloud cannot rename a uid.
The catalog now also tracks user export manifests (Repository::recordUserExport/
listUserExports/forgetUserExport/isRecordedUserExport), rollback-protected like a
snapshot: listUserExports() verifies each one is still present on the location, and
restore only ever reads a manifest the verified catalog actually recorded, never an
arbitrary caller-supplied path. prune()'s existing raw users/ scan for blob retention is
unchanged. A new short exclusive "catalog" lock (PruneService::catalogLockKey())
serializes the catalog write itself, so two concurrent per-user backups (or one
alongside a full backup) can no longer race to write the same next generation and
silently drop one of them -- the broader per-target shared lock only ever protected
against a concurrent cleanup, not against other shared holders.
Two real bugs found by testing this against a live Nextcloud instance, not by unit
tests (RepositoryExportDestination/RepositoryImportSource implement OCP interfaces that
plain PHPUnit can't load, see tests/Unit/Repository/UserExportTest.php's docblock; the
new tests/integration/user-roundtrip.sh covers the real round trip instead):
- RepositoryImportSource::copyToFolder() double-closed the stream it handed to
putContent()/newFile() (both already close it themselves), which throws a TypeError on
PHP 8 and broke every restored file with content.
- A replace-mode restore failed on NotPermittedException: IRootFolder::getUserFolder()
caches a Folder per uid for the process lifetime, so after delete+recreate of the same
uid the import still got the deleted account's (now storage-less) folder. Fixed by
dispatching FilesystemTornDownEvent, which Root already listens for to clear exactly
that cache -- no internal OC\ classes needed.
Local review (CodeRabbit + Codex) against this diff found three more real issues, all
fixed: the catalog write race above, a failed as-backup import leaving a broken stub
account behind (now deleted on failure), and an export stream leak on a read/backend
failure in RepositoryExportDestination::copyFolder() (now closed in a finally).
Deliberately left out: wiring per-user export into the automatic backup schedule
(on-demand via occ only, for now) and retention/pruning of old user exports -- both open
design questions, not yet decided. The lease covering a single export/import still has a
fixed TTL (3600s) with no mid-call refresh, since UserMigrationService exposes no
progress; a resumable, step-based per-user backup (like the main BackupService run)
would remove that cap but is a bigger change than this pass.
S3 has no bucket quota, so NG Backup enforces one: before a pack is
uploaded, the stored size plus the pack must stay under the location's
limit, otherwise the backup stops with an error (and administrators are
notified through the failed run). Repacking during prune is exempt, as it
is how space is freed. occ backup:target:limit NAME [SIZE|none].
Also ignore credential files in the repository.
Every change to the set of snapshots or trash entries writes an
encrypted catalog generation bound to the repository and its position,
with the hash of the previous generation. The server anchors the newest
generation it has seen in its app config; on open, the location must
show at least that generation and chain back to it, and every snapshot
listed must exist. A rolled-back or tampered location is refused (the
run fails and administrators are notified). Opening with the passphrase
on a new server accepts the newest valid chain.
occ backup:target:trust-current accepts a legitimately older state
(e.g. restored from the provider's snapshot) after typing the location
name; audited and announced to all administrators.
Tests: chain advances, rollback, hidden snapshot, swapped generation,
disaster recovery without anchor. Tested on NC34 by deleting the newest
catalog generation on the SFTP location.
- one locking protocol per location: backup steps and restores hold a
shared lock, prune and untrash an exclusive one and refuse while any
run (backup, restore, export) exists; starting a backup checks and
creates the run under the exclusive lock, closing the race with prune
- snapshot entries are validated when read (relative path without '.',
'..', NUL, backslash or control characters); raw restore also checks
that the resolved directory stays below the target; the restore API
and service accept canonical paths only
- recovery kit confirmation requires a recent password confirmation and
the confirmation code printed in the current kit file (new code per
kit version), proving the kit was actually downloaded
- prune decides on the authenticated index files on the location and
refreshes the local index cache from them
- restore job marks the run failed on any error; restore runs without
progress for 6 hours are failed so they cannot block cleanup forever
- tests for safe paths and a malicious snapshot path
- removed snapshots go to a trash on the location and their data is only
freed after the deletion delay, which is chosen when the key is created
(default 7 days, 1-365) and cannot be changed afterwards; any admin can
bring them back (occ backup:trash, backup:trash:restore, web UI)
- up to three passphrase slots open the same master key (like LUKS), so
one person leaving or dying does not lock the company out; slot changes
are written to every location, announced to all admins and require a
new recovery kit confirmation, while backups keep running
- occ backup:key:slots, backup:key:slot:add|replace|remove; key init takes
--label and --delete-delay-days
- web UI: deletion delay and first slot at key creation, slot management,
trash per location with restore; all behind password confirmation
- README: key, slots (safe / CTO / head of IT), and protection against
ransomware: object lock in compliance mode (e.g. 90 days), credentials
without delete rights, provider snapshots, off-site copy
- RetentionPolicy (last/daily/weekly/monthly) and PruneService: forget
snapshots the policy drops, delete packs nothing uses and repack mostly
unused ones by copying encrypted blobs as-is; new packs are indexed
before old ones are removed
- a location lock keeps prune and backups apart; append-only locations
are never pruned (cleanup via the location's lifecycle rules)
- weekly prune from the background job; occ backup:prune [--dry-run],
backup:retention, backup:target:append-only
- tests: forget+prune frees space and the remaining snapshot restores
identically, repack of mixed packs, dry-run preview, retention buckets
Opening a repository lists index/ once and downloads only index files
that are not cached; packs written by this server are cached on write,
stale cache rows are dropped. Each resumable step therefore no longer
downloads every index file from the location.
TreeWalker walks one or more roots in a stable segment-wise order with
exclusions and resumes after a cursor; ParentCursor merge-joins the
previous snapshot in the same order. BackupRun no longer keeps a file
list or the parent's entries in memory or in its state.
Dev tooling via composer (phpunit, psalm, nextcloud/ocp); vendor/ is
mounted from outside the repository by a helper script. 28 unit tests
ported from the phase-0 spikes (round trips, corruption, wrong keys,
dedup, incremental runs, partial restore, tampering, object binding,
time-boxed steps, file changed between steps).
dillard