- The admin page and backup:status warn when background jobs do not run on a
system cron, or the cron stopped, with a link to how to set one up.
- Without a working system cron, the open admin page moves a running backup
forward through /api/tick, a short step per poll.
- The database phase shows the tables dumped ("db: 47/183 tables"); a backup
that has not moved for 15 minutes says it is waiting for background jobs.
- The trash of restore points is on a tab of its own, Bin.
- New screenshots; changelog and plan for 1.0.2.
backup:user:restore <location> <uid> only said "is not a recorded user export": the argument
is the manifest path, which only backup:user:backup printed once. backup:user:list shows the
exports again; a bare user id works when that user has exactly one export, otherwise the error
lists them. The export is resolved before the replace confirmation.
Fixes#22
Each 30-second step opened the location again and read its list of
index files. Over SMB that cost a request per file, so on a large
repository opening took longer than the step: each step stored one
4 MiB pack and the list grew further. A first backup of nc01 to a
Storage Box over SMB managed 12 GiB in 35 hours.
occ backup:run and system cron now keep the repository open for the
whole run (reopened when another process took a step or the location
was edited), opening no longer counts towards the step's time, and
SMB folders are listed in one request.
occ backup:cancel stops a running backup: no new step starts and the
step in progress finishes first. A run of an earlier version, which
does not hold back its next step, is marked cancelled after two
minutes unless it is recording its snapshot.
A changed password or host name could only be handled by adding a new
location, which starts an empty repository and, with a retention lock at
the provider, keeps the old one billed. Edit now changes a location in
place (web interface and occ backup:target:edit). The settings are only
saved when they lead to the same repository; a failed connection and a
missing repository give separate errors, and the folder is not created
while checking. Stored passwords are never sent to the browser: an empty
password field keeps the stored value.
Add three screenshots (backups, restore, settings) taken on a demo
instance, plus website, repository and the HOWTO as admin documentation
in info.xml.
Fixes found while taking them:
- the "new folder" restore option lost "<date>" (l10n sanitizes it as
a tag); show the folder name pattern via a placeholder instead
- the restore-point dropdown cut off the time
- passphrase actions stacked vertically in the slot table
- HOWTO: SFTP example used "sftp::password", the auth mechanism is
"password::password"
Repository::verify(): every blob a snapshot's files and database dump reference must be in a
freshly re-downloaded, re-authenticated index (never trusted from the local cache, same reasoning
as prune()) and have its pack still present on the location -- shallow, cheap, no blob content
read, catches a deleted pack, a tampered/rolled-back index, or a database manifest that went
missing (reported as a failure, not silently skipped). With $deep, each blob is also downloaded
and decrypted, same as loadBlob() already does on every real read (AEAD authentication tag,
content hash against its blob id), done here deliberately for everything in the snapshot rather
than waiting for it to come up during an actual restore. A blob referenced by more than one file
is only checked once; pack presence is cached per pack, not re-checked per blob.
occ backup:verify <target> <snapshot> [--deep], meant to run periodically (cron, a sample of
recent snapshots), per PLAN.md's "Verificatie: periodiek checksums controleren, steekproef-restore".
The lease covering the audit is refreshed roughly every half of its TTL while blobs are checked,
rather than relying on a single long-lived lease like the other new phase-2 commands do.
Verified against a real S3-backed 1.4 GiB / 603-blob snapshot (shallow: instant; --deep: ~28s,
fully decrypted) in addition to the new unit tests and the roundtrip.sh addition (shallow and deep
on a snapshot with real, varied file sizes, now covered on every CI matrix combination).
Local review (Codex) on the first pass found three real issues, all fixed: a missing database
manifest was silently skipped instead of reported; shallow verification trusted the (possibly
stale or unauthenticated) local index cache instead of re-checking the authenticated index on the
location; and the lease had no refresh, so a --deep audit of a very large snapshot could outrun
its TTL and run unprotected against a concurrent prune.
New commands, per PLAN.md 3.7:
- occ backup:key:import: adopt the master key from a recovery kit on a fresh installation
(instead of occ backup:key:init generating a new one), so occ backup:target:add opens the
existing repository on the location rather than creating an empty one.
- occ backup:restore:full: replace the database (consistent, one transaction), restore the data
directory in place, and merge config.php -- refuses up front and lists what's missing when the
Nextcloud major version or any app enabled at backup time doesn't match here (their tables must
exist before the database restore can use them), rather than guessing or partially applying.
Repository::restore() gained an optional $stripPrefix (write "data/<uid>/..." straight into the
real data directory, not a "data/" subdirectory of it) -- restoring "data/" and "config/" this way
needs no temporary copy of what can be the largest part of an instance.
occ backup:target:add now backfills the local snapshot cache (ngb_snapshots, what occ backup:list
actually reads) when it adopts an existing repository: without this, a freshly connected location
showed no restore points at all, even though the repository itself had them. One unreadable
snapshot is skipped rather than aborting the backfill or the target add itself.
Three self-referential problems found and fixed by testing this against two real, separate
Nextcloud instances sharing only the backup location (tests/integration/full-restore-roundtrip.sh
covers this; the per-process nature of these bugs meant they were invisible from inside a single
occ invocation, including the one used to write the fixes, which is exactly why a second, fresh
process re-running occ backup:status etc. was necessary to catch them):
- The full database dump includes ng_backup's own tables and appconfig rows; restoring them
verbatim would overwrite the key/target that is actively performing the restore, on THIS
installation, with the backed-up instance's. Both are now preserved: ng_backup's own tables
(and, on PostgreSQL, their sequences -- excluding a table does not exclude its sequence) are
excluded from what gets restored, and its appconfig rows are captured before the database
restore and written back after, in their own transaction.
- config.php's 'secret' is merged in from the backup (needed so already-encrypted content in the
restored database, e.g. share tokens, stays decryptable), but Target::options, the master key,
the wrapped key and the recovery kit code are all encrypted with that same instance secret
(OCP's ICrypto, or Nextcloud AppConfig's own "sensitive" flag underneath), not with ng_backup's
own key. Changing it without re-encrypting them first left all four permanently undecryptable
the moment any later occ command ran. Fixed with two helpers
(TargetService::reencryptOptionsAround(), KeyService::reencryptForSecretRotation()) that decrypt
before and re-encrypt after whatever callback actually changes the secret; config.php is applied
dead last in restoreFull(), after every other repository access. Not crash-safe (a process kill
mid-rotation would leave a key missing, not just stale) -- disaster recovery is run once by
hand, right after occ maintenance:install, a window too narrow yet to justify a two-phase write.
- A subtler variant of the same bug: Nextcloud's own AppConfig additionally encrypts a "sensitive"
value itself, tied to the same secret, underneath our own encryption. Overwriting an existing
sensitive key makes it read the old value first to compare, which then also fails against the
new secret. Deleting the key before setting the new value avoids that read entirely.
Roadmap (PLAN.md 9): phase 3 (in-place full restore, user self-service) moved out to v1.3.0; phase
2 continues toward v1.2.0 with FTP/SMB, a verification/checksum-audit command, user-export
retention, and deciding whether per-user export joins the automatic schedule.
Wires the existing RepositoryExportDestination/RepositoryImportSource (see
lib/UserMigration/) into user_migration's own UserMigrationService, resolved on demand
through a new UserMigrationServiceLocator since the app is an optional dependency
(PLAN.md 3.8) -- without it, only this feature is unavailable, the full backup still
works.
New: UserBackupService::backupUser() (occ backup:backup:user) streams a user_migration
export straight into the repository, under the same shared location lock as a regular
backup. UserRestoreService::restoreUser() (occ backup:restore:user) never imports over a
live account: --mode=replace safety-exports the current state first, then deletes and
recreates the account from the backup under the same uid; --mode=as-backup restores into
a new "<uid>-bak" account instead, since Nextcloud cannot rename a uid.
The catalog now also tracks user export manifests (Repository::recordUserExport/
listUserExports/forgetUserExport/isRecordedUserExport), rollback-protected like a
snapshot: listUserExports() verifies each one is still present on the location, and
restore only ever reads a manifest the verified catalog actually recorded, never an
arbitrary caller-supplied path. prune()'s existing raw users/ scan for blob retention is
unchanged. A new short exclusive "catalog" lock (PruneService::catalogLockKey())
serializes the catalog write itself, so two concurrent per-user backups (or one
alongside a full backup) can no longer race to write the same next generation and
silently drop one of them -- the broader per-target shared lock only ever protected
against a concurrent cleanup, not against other shared holders.
Two real bugs found by testing this against a live Nextcloud instance, not by unit
tests (RepositoryExportDestination/RepositoryImportSource implement OCP interfaces that
plain PHPUnit can't load, see tests/Unit/Repository/UserExportTest.php's docblock; the
new tests/integration/user-roundtrip.sh covers the real round trip instead):
- RepositoryImportSource::copyToFolder() double-closed the stream it handed to
putContent()/newFile() (both already close it themselves), which throws a TypeError on
PHP 8 and broke every restored file with content.
- A replace-mode restore failed on NotPermittedException: IRootFolder::getUserFolder()
caches a Folder per uid for the process lifetime, so after delete+recreate of the same
uid the import still got the deleted account's (now storage-less) folder. Fixed by
dispatching FilesystemTornDownEvent, which Root already listens for to clear exactly
that cache -- no internal OC\ classes needed.
Local review (CodeRabbit + Codex) against this diff found three more real issues, all
fixed: the catalog write race above, a failed as-backup import leaving a broken stub
account behind (now deleted on failure), and an export stream leak on a read/backend
failure in RepositoryExportDestination::copyFolder() (now closed in a finally).
Deliberately left out: wiring per-user export into the automatic backup schedule
(on-demand via occ only, for now) and retention/pruning of old user exports -- both open
design questions, not yet decided. The lease covering a single export/import still has a
fixed TTL (3600s) with no mid-call refresh, since UserMigrationService exposes no
progress; a resumable, step-based per-user backup (like the main BackupService run)
would remove that cap but is a bigger change than this pass.
info.xml's version field is restricted by Nextcloud's own info.xsd to plain
numeric SemVer, no pre-release suffix, so the alpha/beta/rc stage from
PLAN.md's release table lives in the git tag and CHANGELOG instead; noted
that in PLAN.md so it isn't rediscovered later. Bumped the version, wrote
an honest description for an alpha (feature-complete for phase 1, not for
production), and added CHANGELOG.md.
README: new "Storage backend experiences" section, seeded with the Hetzner
Storage Box sub-account root-path quirk found while testing a real SFTP
target (root must be "home", not "/"), open for others to extend via PR.
Show the date of the restore point a restore run used, both in the
backup history table and when confirming a restore. Split the admin
page into a "Backup and restore" tab (locations, backups, restore)
and a "Settings" tab (key, slots, recovery kit), so daily work no
longer sits behind the key setup at the top of the page.
Signed-off-by: Dillard Blom <dillard.blom@ensembia.com>
- after each successful backup the stored size is recorded; at 80% of a
location's limit administrators get a notification (once a day)
- locations list shows stored size and limit; the add form has a
'maximum size' field that explains that empty means unlimited and
billed by the provider; PUT /api/targets/{id}/limit (password
confirmation)
Tested on NC34 with the OVH test bucket: a backup above the limit stops,
a successful one at 81% sends the warning.
A backup step killed by a time limit, OOM or kill -9 kept Nextcloud's
lock on its location for up to an hour, blocking every new start and the
run itself. Locations and runs are now guarded by leases in ngb_leases
with an expiry: a backup step holds them for its duration plus two
minutes, restores refresh theirs every minute, prune holds 30 minutes,
untrash 5. Nextcloud's lock is only used for the milliseconds in which a
lease is checked and written.
Also: starting a backup while one is already running simply continues
it (no exclusive lease needed), and running to completion waits instead
of spinning while a crashed step's lease has not expired.
Tested on NC34: a backup killed four times at random moments resumes on
its own after the lease expires (about two minutes), and restores
bit-identically.
S3 has no bucket quota, so NG Backup enforces one: before a pack is
uploaded, the stored size plus the pack must stay under the location's
limit, otherwise the backup stops with an error (and administrators are
notified through the failed run). Repacking during prune is exempt, as it
is how space is freed. occ backup:target:limit NAME [SIZE|none].
Also ignore credential files in the repository.
Every change to the set of snapshots or trash entries writes an
encrypted catalog generation bound to the repository and its position,
with the hash of the previous generation. The server anchors the newest
generation it has seen in its app config; on open, the location must
show at least that generation and chain back to it, and every snapshot
listed must exist. A rolled-back or tampered location is refused (the
run fails and administrators are notified). Opening with the passphrase
on a new server accepts the newest valid chain.
occ backup:target:trust-current accepts a legitimately older state
(e.g. restored from the provider's snapshot) after typing the location
name; audited and announced to all administrators.
Tests: chain advances, rollback, hidden snapshot, swapped generation,
disaster recovery without anchor. Tested on NC34 by deleting the newest
catalog generation on the SFTP location.
- restore runs from the web are created under the location's exclusive
lock, like backup runs
- raw restore writes to an unpredictable temporary name opened with
exclusive create (never follows a planted symlink) and validates the
path prefix; the restore service rejects non-canonical paths instead
of trimming them (occ normalises user input before calling it)
- restore jobs report progress every minute, so a long restore is not
taken for a dead one
- README: Object Lock turns deletions into delete markers
- one locking protocol per location: backup steps and restores hold a
shared lock, prune and untrash an exclusive one and refuse while any
run (backup, restore, export) exists; starting a backup checks and
creates the run under the exclusive lock, closing the race with prune
- snapshot entries are validated when read (relative path without '.',
'..', NUL, backslash or control characters); raw restore also checks
that the resolved directory stays below the target; the restore API
and service accept canonical paths only
- recovery kit confirmation requires a recent password confirmation and
the confirmation code printed in the current kit file (new code per
kit version), proving the kit was actually downloaded
- prune decides on the authenticated index files on the location and
refreshes the local index cache from them
- restore job marks the run failed on any error; restore runs without
progress for 6 hours are failed so they cannot block cleanup forever
- tests for safe paths and a malicious snapshot path
- removed snapshots go to a trash on the location and their data is only
freed after the deletion delay, which is chosen when the key is created
(default 7 days, 1-365) and cannot be changed afterwards; any admin can
bring them back (occ backup:trash, backup:trash:restore, web UI)
- up to three passphrase slots open the same master key (like LUKS), so
one person leaving or dying does not lock the company out; slot changes
are written to every location, announced to all admins and require a
new recovery kit confirmation, while backups keep running
- occ backup:key:slots, backup:key:slot:add|replace|remove; key init takes
--label and --delete-delay-days
- web UI: deletion delay and first slot at key creation, slot management,
trash per location with restore; all behind password confirmation
- README: key, slots (safe / CTO / head of IT), and protection against
ransomware: object lock in compliance mode (e.g. 90 days), credentials
without delete rights, provider snapshots, off-site copy
- notifications to administrators for failed backup/restore runs and
locations without a successful backup for two days (once a day)
- setup check in the admin overview: error when the key/kit is not set
up, warning for missing, old or failed backups
- security events (recovery kit downloaded, location removed,
append-only turned off, retention changed, snapshots forgotten) go to
the audit log and are announced to all administrators, from the web
and from occ; the scheduled weekly cleanup is logged only
- fix: the recovery-kit download link failed the CSRF check (412); it
now requires a recent password confirmation instead, like creating the
key, removing a location and restoring
- RetentionPolicy (last/daily/weekly/monthly) and PruneService: forget
snapshots the policy drops, delete packs nothing uses and repack mostly
unused ones by copying encrypted blobs as-is; new packs are indexed
before old ones are removed
- a location lock keeps prune and backups apart; append-only locations
are never pruned (cleanup via the location's lifecycle rules)
- weekly prune from the background job; occ backup:prune [--dry-run],
backup:retention, backup:target:append-only
- tests: forget+prune frees space and the remaining snapshot restores
identically, repack of mixed packs, dry-run preview, retention buckets
- AdminApiController (admin only, CSRF-protected): status, create key,
download recovery kit, confirm (statement accepted + exact phrase),
backends with their usable auth mechanisms, add/test/remove locations,
start backups, schedule, snapshots, browse, queue restores
- RestoreJob runs web-initiated restores in the background; backup and
restore runs are kept apart in the run table
- admin section 'NG Backup' with a Vue 3 / @nextcloud/vue 9 page: key and
recovery-kit flow (download, tick statement, 'Yes, I confirm'), location
form built from files_external's parameter definitions, schedule, run
list with polling, snapshot browser and restore with the three modes
- auth mechanisms that depend on a user session or on files_external's
own stored credentials are not offered
API tested with curl on NC34 (status, backends, schedule validation,
confirmation refusal, browse, queued restore and backup completed by
cron); non-admins get 403, anonymous 401.
User files are restored through the Files API so the file cache,
versions and trash stay consistent: modes new-folder (default, into
'Restored <date>/<path>'), merge (overwritten files keep their old
content as a version) and replace (extra files go to the trash). Each
file is staged under a temporary name and moved into place only after
all its blobs were verified. Any snapshot path can also be restored raw
to a local directory (disaster recovery).
Tested on NC34: all three modes restore checksum-identical files, the
overwritten content is kept as a version, the extra file is in the
trash, no temporary files remain.
Opening a repository lists index/ once and downloads only index files
that are not cached; packs written by this server are cached on write,
stale cache rows are dropped. Each resumable step therefore no longer
downloads every index file from the location.
dillard