Off, once a day or every N hours from a time, each in its own time zone. Existing locations
keep the time that held for all of them. A backup that has to wait shows as queued, and its
restore point has the time it really began.
Both live in the catalog on the location, so a new server sees them; the database holds
a local copy for the lists. The retention policy leaves a pinned restore point alone and
does not count it. Available on the restore page and as occ backup:pin and backup:note.
Reported apart from unreadable files, with example paths and the advice to rename
them, instead of advice about permissions. Start of the 1.1.0 plan in PLAN.md.
- Leave out and report a file or folder whose name a snapshot cannot hold
- A staging write failure stops a full restore under --salvage
- Browse reuses its repository; empty folders read along with the entries
- Keep leases alive while empty folders are created; exact-prefix empty folder is the target
- Clean up tables that were partly created or before the transaction started
- Bin: no empty message after a read error; longer pages
A killed restore left its leases for up to two hours, so the first backup of the recovered
server was refused. A backup from before 1.0.3 records no empty folders: the test now expects that.
- Record folders without files in the meta record of a snapshot and restore them (#27)
- Errors writing to the restore directory stop the restore, also with --salvage
- Resume: check the folder before skipping, clean temporaries in folders that are complete
- Clean only read-ahead folders of our own; refresh the lease on skipped files
- Drop tables a failed database restore created; keep --salvage from calling a restored database fresh
- Retry a pack that fails to open on the storage (SMB)
- Without cron, running backups take turns from the open page
- Bin: read in parts with a counter and progress bar, show read errors, refresh when a run ends
The DR test of 1.0.2 over SMB stopped after 392 GiB of 983: the connection to the Storage Box
dropped, and the smbclient extension then returns a string of length -1 for a failed read (it
stores smbc_read()'s -1 in an unsigned length). The read loop counted that as data and went
backwards forever at 100% CPU. Reads over SMB are now checked; a failed one is read again the
usual way over a new connection.
backup:restore:full that runs again (--force) leaves the data files already in place with the
snapshot's size and modification time, as a backup takes them as unchanged, and reads ahead only
what is still missing: after 392 GiB it goes on instead of starting over.
- The admin page and backup:status warn when background jobs do not run on a
system cron, or the cron stopped, with a link to how to set one up.
- Without a working system cron, the open admin page moves a running backup
forward through /api/tick, a short step per poll.
- The database phase shows the tables dumped ("db: 47/183 tables"); a backup
that has not moved for 15 minutes says it is waiting for background jobs.
- The trash of restore points is on a tab of its own, Bin.
- New screenshots; changelog and plan for 1.0.2.
A file whose data cannot be read or decrypted is left out and listed, and
the command exits with code 2. When the database cannot be restored (its
dump is damaged, or it does not fit this installation), backup:restore:full
leaves the fresh database as it is and puts the data directory and config of
the backup in ngb-salvage-<snapshot> in the data directory. Without --salvage
a restore of damaged data stops as before and points at the option.
- Admin page: an optional label (at most 100 characters) for a backup started with "Back up now";
an (i) by "removed by retention policy" in the trash that explains what it means and until when
the restore point can be brought back. backup:trash says the same below its list.
- backup:status, backup:list, backup:trash and backup:browse show times in the zone of the daily
schedule and name it, instead of the server's UTC.
- backup:restore --to-directory of a file restored nothing: the path got a "/" appended.
- A restore removes "<name>.ngb-restore-<hex>" temporaries that a stopped restore left in the
folders it writes to.
- Roadmap: the cron warning (#13), reading ahead from S3/SMB and --salvage move to 1.0.2.
Backups now record each table's structure in the dump manifest. The full
restore creates tables that do not exist on the new instance (left behind
by apps that were removed or are switched off), lists all problems at
once, and no longer stops on apps that are missing. Foreign keys are not
checked while rows are loaded in name order, and checked again before the
restore commits (PostgreSQL: dropped and added back in the transaction).
--install-apps installs or enables the apps that were enabled at backup
time; --skip-missing-tables leaves out tables of older backups whose
structure is unknown. nextcloud-aio is never required.
backup:target:add without a key points to backup:key:import, and says that
connecting can take a while. Removing a location drops its cached index;
a migration removes the cache of locations that are gone.
The full-restore test now also runs on PostgreSQL with Mail (foreign keys)
and a disabled app.
On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.
backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
Each 30-second step opened the location again and read its list of
index files. Over SMB that cost a request per file, so on a large
repository opening took longer than the step: each step stored one
4 MiB pack and the list grew further. A first backup of nc01 to a
Storage Box over SMB managed 12 GiB in 35 hours.
occ backup:run and system cron now keep the repository open for the
whole run (reopened when another process took a step or the location
was edited), opening no longer counts towards the step's time, and
SMB folders are listed in one request.
occ backup:cancel stops a running backup: no new step starts and the
step in progress finishes first. A run of an earlier version, which
does not hold back its next step, is marked cancelled after two
minutes unless it is recording its snapshot.
A Hetzner Storage Box drops an SFTP connection that carried a lot of data
and then stayed quiet for a few minutes; files_external keeps using the dead
client, on which every file looks missing. StorageBackend now checks the
connection after a minute without requests, and replaces a connection that
is dead after a failed request (once per request; put, move and delete are
not repeated). SftpReadStream closes a connection it left in an unknown state.
The verify cache refresh took more than 6 minutes on PostgreSQL for 27,547
packs, one autocommitted insert per pack: putMany()/replaceAll() write in one
transaction. After a verify the cache holds the index files as they are, so a
blob in two packs stays cached for both.
A run only showed "running (files)", which says nothing during a backup
of a terabyte. The files phase now records an expected size on its first
step: the previous snapshot's size, or for a first backup the users'
files, versions and trash from Nextcloud's file cache (approximate, shown
with a ~). The web interface and occ backup:status show the share of it
done so far, including a large file that is still being read; it stays
below 100% until the run finishes.
A changed password or host name could only be handled by adding a new
location, which starts an empty repository and, with a retention lock at
the provider, keeps the old one billed. Edit now changes a location in
place (web interface and occ backup:target:edit). The settings are only
saved when they lead to the same repository; a failed connection and a
missing repository give separate errors, and the folder is not created
while checking. Stored passwords are never sent to the browser: an empty
password field keeps the stored value.
verifyBlob() caught every exception from loading a blob and counted it as
a failed (corrupted or tampered) blob, so a transient network error during
a deep verify over WebDAV looked like data corruption. Read the range
separately, retry it, and report a read that keeps failing as unreadable
with its error; only a blob that was read but does not decrypt or match
its id counts as failed.
- backup:restore:full refuses on an installation with more than one user
and asks before it starts (--force skips both, also to rerun after a
failure halfway), and warns when the snapshot's database dump was not
one consistent transaction.
- backup:user:restore --mode=replace asks before deleting an existing
account; non-interactive use needs --force.
- The recovery kit file is created owner-only and never over an existing
file or symlink; backup:key:import shows the deletion delay it read from
the kit.
- Adding a location through the web API needs password confirmation, like
removing one.
- Key derivation parameters from a kit or repository config are bounded,
and a stalled network stream no longer spins at full CPU.
- Floating-point columns are dumped exactly instead of with 14 digits.
Storage and repository:
- Listing skipped every name containing ".part-", directories too, so the
exports of a user like "john.part-time" were invisible to prune on S3,
SFTP, FTP and SMB and their data was deleted. Only real temporary uploads
("<name>.part-<8 hex>") are skipped now.
- prune refuses to run while a recorded per-user export's manifest is
missing, instead of freeing that export's data.
- forget() and untrash() copy first, switch the catalog, and delete the
originals last, so a crash can no longer leave the catalog pointing at a
missing snapshot or database dump.
- Writes are checked against the expected size, so a short upload is an
error instead of a truncated pack; raw restores check rename() and touch().
- A catalog write refuses to overwrite a generation another process just
wrote, and the catalog lock lasts 300 s, so opening a large repository
inside it no longer lets the lease run out.
Backup runs and scheduling:
- A file that keeps changing is read again at most three times; then the
last read is kept and reported, instead of restarting forever and
blocking cleanup.
- Unreadable files and folders are recorded, logged, shown by occ
backup:run, stored in the snapshot and notified to admins, instead of
being skipped silently.
- After a failed scheduled backup, at most three attempts a day, an hour
apart, instead of a new run and notification every five minutes.
- The job no longer spins on a run whose lease another process holds, and
the weekly cleanup catches up after a Sunday spent backing up.
- Notifications for the per-user export retention events.
Full restore, raw directory restore, per-user export and the safety export
held a fixed-TTL lease on the location that nothing refreshed. A restore or
export running longer than that TTL let a cleanup take over and delete data
still in use. They now refresh the lease while they work (per blob, per row
batch, per written chunk), throttled to once a minute.
Also:
- backup:prune still applies snapshot retention when user-export retention
fails for a location.
- The recovery kit carries the deletion delay, so a disaster recovery keeps
the original protection instead of falling back to 7 days. Older kits
still import with the default.
- CI checkouts no longer persist the token in the git config.
- recoveryKit() has its real return type; the stale baseline entry is gone.
A full backup opened its repository at the start of a step and wrote the
catalog at the end of it, without the catalog lock. A per-user export or
restore that wrote the catalog in between could lose its entry, or the
snapshot could lose its own.
The final step now only stores the snapshot object and moves the run to a
new 'commit' phase. BackupService then records the snapshot under the
catalog lock, from a repository opened inside that lock; if the lock is
busy, the next step retries the commit. Repository::writeSnapshot() keeps
its old behaviour as putSnapshot() plus recordSnapshot().
Catalog: forgetting a user export or purging trash now writes the catalog
before deleting the objects, so a crash leaves at most an unreferenced file
instead of a catalog entry pointing at nothing. Per-user backup and the
safety export reopen the repository inside the catalog lock, so they build
on the latest catalog generation.
Commands: backup:user:backup and backup:user:restore resolve the location
before asking to install user_migration, and a successful install always
stops with a rerun instruction. backup:retention rejects a malformed
--user-last before changing any policy. backup:key:import fails clearly
when the passphrase file cannot be read.
backup:user:backup and backup:user:restore now ask the administrator before
installing or enabling user_migration (or take --install-user-migration), so
nothing changes on the server without consent. The restore warns that
user_migration imports only its allowlisted app settings.
Also: a clear error for unknown locations instead of a raw database message;
prune reports the number of affected users, not the number of kept ones; and
--user-last rejects values that are not whole numbers. Adds docs/HOWTO.md and
updates the README and PLAN.md for the beta.
backup:verify accepts a user export path (users/<uid>/<export>): every blob its manifest references
is checked the same way as for a snapshot, shallow or --deep.
The SMB backend is refused at backup:target:add when the native smbclient PECL extension is not
loaded, instead of registering and then failing the first backup with "Seek failed": with only the
smbclient binary files_external falls back to a shell stream that cannot seek.
The per-user integration job now runs on Nextcloud 32-35, not just 35: it calls user_migration's
internal UserMigrationService, whose API may change between versions.
PruneService::userExportRetention()/setUserExportRetention(): a single, app-wide "keep the
newest N" setting (default 3, same default as the snapshot policy's "last"), deliberately simpler
than the full daily/weekly/monthly RetentionPolicy -- per-user exports are on-demand only (occ
backup:backup:user, see the fold-out task about whether they ever join the automatic schedule),
so there is no regular cadence yet to bucket by day/week/month.
PruneService::applyUserExports(): groups a location's user exports by uid (read from each
manifest's own 'time' field, the same field snapshot retention already reads), keeps the newest N
per user, forgets the rest. No trash, unlike a snapshot: these are not yet the primary safety net
the deletion delay protects. Runs under the same exclusive lock as snapshot retention.
occ backup:retention gained --user-last=N; occ backup:prune now also applies it for every target
it processes, reporting counts the same way it already does for snapshots and blob packs.
Verified live against the dev rig (four real exports across four users, confirmed exactly the
expected ones were kept/forgotten at two different --user-last values) and added coverage to
tests/integration/user-roundtrip.sh (an account's manual export plus the automatic safety-export
a replace-mode restore makes of it, pruned down to --user-last=2).
Repository::verify(): every blob a snapshot's files and database dump reference must be in a
freshly re-downloaded, re-authenticated index (never trusted from the local cache, same reasoning
as prune()) and have its pack still present on the location -- shallow, cheap, no blob content
read, catches a deleted pack, a tampered/rolled-back index, or a database manifest that went
missing (reported as a failure, not silently skipped). With $deep, each blob is also downloaded
and decrypted, same as loadBlob() already does on every real read (AEAD authentication tag,
content hash against its blob id), done here deliberately for everything in the snapshot rather
than waiting for it to come up during an actual restore. A blob referenced by more than one file
is only checked once; pack presence is cached per pack, not re-checked per blob.
occ backup:verify <target> <snapshot> [--deep], meant to run periodically (cron, a sample of
recent snapshots), per PLAN.md's "Verificatie: periodiek checksums controleren, steekproef-restore".
The lease covering the audit is refreshed roughly every half of its TTL while blobs are checked,
rather than relying on a single long-lived lease like the other new phase-2 commands do.
Verified against a real S3-backed 1.4 GiB / 603-blob snapshot (shallow: instant; --deep: ~28s,
fully decrypted) in addition to the new unit tests and the roundtrip.sh addition (shallow and deep
on a snapshot with real, varied file sizes, now covered on every CI matrix combination).
Local review (Codex) on the first pass found three real issues, all fixed: a missing database
manifest was silently skipped instead of reported; shallow verification trusted the (possibly
stale or unauthenticated) local index cache instead of re-checking the authenticated index on the
location; and the lease had no refresh, so a --deep audit of a very large snapshot could outrun
its TTL and run unprotected against a concurrent prune.
New commands, per PLAN.md 3.7:
- occ backup:key:import: adopt the master key from a recovery kit on a fresh installation
(instead of occ backup:key:init generating a new one), so occ backup:target:add opens the
existing repository on the location rather than creating an empty one.
- occ backup:restore:full: replace the database (consistent, one transaction), restore the data
directory in place, and merge config.php -- refuses up front and lists what's missing when the
Nextcloud major version or any app enabled at backup time doesn't match here (their tables must
exist before the database restore can use them), rather than guessing or partially applying.
Repository::restore() gained an optional $stripPrefix (write "data/<uid>/..." straight into the
real data directory, not a "data/" subdirectory of it) -- restoring "data/" and "config/" this way
needs no temporary copy of what can be the largest part of an instance.
occ backup:target:add now backfills the local snapshot cache (ngb_snapshots, what occ backup:list
actually reads) when it adopts an existing repository: without this, a freshly connected location
showed no restore points at all, even though the repository itself had them. One unreadable
snapshot is skipped rather than aborting the backfill or the target add itself.
Three self-referential problems found and fixed by testing this against two real, separate
Nextcloud instances sharing only the backup location (tests/integration/full-restore-roundtrip.sh
covers this; the per-process nature of these bugs meant they were invisible from inside a single
occ invocation, including the one used to write the fixes, which is exactly why a second, fresh
process re-running occ backup:status etc. was necessary to catch them):
- The full database dump includes ng_backup's own tables and appconfig rows; restoring them
verbatim would overwrite the key/target that is actively performing the restore, on THIS
installation, with the backed-up instance's. Both are now preserved: ng_backup's own tables
(and, on PostgreSQL, their sequences -- excluding a table does not exclude its sequence) are
excluded from what gets restored, and its appconfig rows are captured before the database
restore and written back after, in their own transaction.
- config.php's 'secret' is merged in from the backup (needed so already-encrypted content in the
restored database, e.g. share tokens, stays decryptable), but Target::options, the master key,
the wrapped key and the recovery kit code are all encrypted with that same instance secret
(OCP's ICrypto, or Nextcloud AppConfig's own "sensitive" flag underneath), not with ng_backup's
own key. Changing it without re-encrypting them first left all four permanently undecryptable
the moment any later occ command ran. Fixed with two helpers
(TargetService::reencryptOptionsAround(), KeyService::reencryptForSecretRotation()) that decrypt
before and re-encrypt after whatever callback actually changes the secret; config.php is applied
dead last in restoreFull(), after every other repository access. Not crash-safe (a process kill
mid-rotation would leave a key missing, not just stale) -- disaster recovery is run once by
hand, right after occ maintenance:install, a window too narrow yet to justify a two-phase write.
- A subtler variant of the same bug: Nextcloud's own AppConfig additionally encrypts a "sensitive"
value itself, tied to the same secret, underneath our own encryption. Overwriting an existing
sensitive key makes it read the old value first to compare, which then also fails against the
new secret. Deleting the key before setting the new value avoids that read entirely.
Roadmap (PLAN.md 9): phase 3 (in-place full restore, user self-service) moved out to v1.3.0; phase
2 continues toward v1.2.0 with FTP/SMB, a verification/checksum-audit command, user-export
retention, and deciding whether per-user export joins the automatic schedule.
Wires the existing RepositoryExportDestination/RepositoryImportSource (see
lib/UserMigration/) into user_migration's own UserMigrationService, resolved on demand
through a new UserMigrationServiceLocator since the app is an optional dependency
(PLAN.md 3.8) -- without it, only this feature is unavailable, the full backup still
works.
New: UserBackupService::backupUser() (occ backup:backup:user) streams a user_migration
export straight into the repository, under the same shared location lock as a regular
backup. UserRestoreService::restoreUser() (occ backup:restore:user) never imports over a
live account: --mode=replace safety-exports the current state first, then deletes and
recreates the account from the backup under the same uid; --mode=as-backup restores into
a new "<uid>-bak" account instead, since Nextcloud cannot rename a uid.
The catalog now also tracks user export manifests (Repository::recordUserExport/
listUserExports/forgetUserExport/isRecordedUserExport), rollback-protected like a
snapshot: listUserExports() verifies each one is still present on the location, and
restore only ever reads a manifest the verified catalog actually recorded, never an
arbitrary caller-supplied path. prune()'s existing raw users/ scan for blob retention is
unchanged. A new short exclusive "catalog" lock (PruneService::catalogLockKey())
serializes the catalog write itself, so two concurrent per-user backups (or one
alongside a full backup) can no longer race to write the same next generation and
silently drop one of them -- the broader per-target shared lock only ever protected
against a concurrent cleanup, not against other shared holders.
Two real bugs found by testing this against a live Nextcloud instance, not by unit
tests (RepositoryExportDestination/RepositoryImportSource implement OCP interfaces that
plain PHPUnit can't load, see tests/Unit/Repository/UserExportTest.php's docblock; the
new tests/integration/user-roundtrip.sh covers the real round trip instead):
- RepositoryImportSource::copyToFolder() double-closed the stream it handed to
putContent()/newFile() (both already close it themselves), which throws a TypeError on
PHP 8 and broke every restored file with content.
- A replace-mode restore failed on NotPermittedException: IRootFolder::getUserFolder()
caches a Folder per uid for the process lifetime, so after delete+recreate of the same
uid the import still got the deleted account's (now storage-less) folder. Fixed by
dispatching FilesystemTornDownEvent, which Root already listens for to clear exactly
that cache -- no internal OC\ classes needed.
Local review (CodeRabbit + Codex) against this diff found three more real issues, all
fixed: the catalog write race above, a failed as-backup import leaving a broken stub
account behind (now deleted on failure), and an export stream leak on a read/backend
failure in RepositoryExportDestination::copyFolder() (now closed in a finally).
Deliberately left out: wiring per-user export into the automatic backup schedule
(on-demand via occ only, for now) and retention/pruning of old user exports -- both open
design questions, not yet decided. The lease covering a single export/import still has a
fixed TTL (3600s) with no mid-call refresh, since UserMigrationService exposes no
progress; a resumable, step-based per-user backup (like the main BackupService run)
would remove that cap but is a bigger change than this pass.
dillard