Switching FOREIGN_KEY_CHECKS back on does not check the rows loaded
meanwhile, and DDL would commit halfway, so on MySQL every foreign key is
checked with a query for rows that point at nothing, before the commit.
The recorded table structure now keeps index options (the prefix lengths
MySQL needs for long text columns) and foreign keys; tables the restore
creates get their foreign keys after the commit.
The full-restore test also runs on MariaDB.
Backups now record each table's structure in the dump manifest. The full
restore creates tables that do not exist on the new instance (left behind
by apps that were removed or are switched off), lists all problems at
once, and no longer stops on apps that are missing. Foreign keys are not
checked while rows are loaded in name order, and checked again before the
restore commits (PostgreSQL: dropped and added back in the transaction).
--install-apps installs or enables the apps that were enabled at backup
time; --skip-missing-tables leaves out tables of older backups whose
structure is unknown. nextcloud-aio is never required.
backup:target:add without a key points to backup:key:import, and says that
connecting can take a while. Removing a location drops its cached index;
a migration removes the cache of locations that are gone.
The full-restore test now also runs on PostgreSQL with Mail (foreign keys)
and a disabled app.
Full restore, raw directory restore, per-user export and the safety export
held a fixed-TTL lease on the location that nothing refreshed. A restore or
export running longer than that TTL let a cleanup take over and delete data
still in use. They now refresh the lease while they work (per blob, per row
batch, per written chunk), throttled to once a minute.
Also:
- backup:prune still applies snapshot retention when user-export retention
fails for a location.
- The recovery kit carries the deletion delay, so a disaster recovery keeps
the original protection instead of falling back to 7 days. Older kits
still import with the default.
- CI checkouts no longer persist the token in the git config.
- recoveryKit() has its real return type; the stale baseline entry is gone.
backup:verify accepts a user export path (users/<uid>/<export>): every blob its manifest references
is checked the same way as for a snapshot, shallow or --deep.
The SMB backend is refused at backup:target:add when the native smbclient PECL extension is not
loaded, instead of registering and then failing the first backup with "Seek failed": with only the
smbclient binary files_external falls back to a shell stream that cannot seek.
The per-user integration job now runs on Nextcloud 32-35, not just 35: it calls user_migration's
internal UserMigrationService, whose API may change between versions.
New commands, per PLAN.md 3.7:
- occ backup:key:import: adopt the master key from a recovery kit on a fresh installation
(instead of occ backup:key:init generating a new one), so occ backup:target:add opens the
existing repository on the location rather than creating an empty one.
- occ backup:restore:full: replace the database (consistent, one transaction), restore the data
directory in place, and merge config.php -- refuses up front and lists what's missing when the
Nextcloud major version or any app enabled at backup time doesn't match here (their tables must
exist before the database restore can use them), rather than guessing or partially applying.
Repository::restore() gained an optional $stripPrefix (write "data/<uid>/..." straight into the
real data directory, not a "data/" subdirectory of it) -- restoring "data/" and "config/" this way
needs no temporary copy of what can be the largest part of an instance.
occ backup:target:add now backfills the local snapshot cache (ngb_snapshots, what occ backup:list
actually reads) when it adopts an existing repository: without this, a freshly connected location
showed no restore points at all, even though the repository itself had them. One unreadable
snapshot is skipped rather than aborting the backfill or the target add itself.
Three self-referential problems found and fixed by testing this against two real, separate
Nextcloud instances sharing only the backup location (tests/integration/full-restore-roundtrip.sh
covers this; the per-process nature of these bugs meant they were invisible from inside a single
occ invocation, including the one used to write the fixes, which is exactly why a second, fresh
process re-running occ backup:status etc. was necessary to catch them):
- The full database dump includes ng_backup's own tables and appconfig rows; restoring them
verbatim would overwrite the key/target that is actively performing the restore, on THIS
installation, with the backed-up instance's. Both are now preserved: ng_backup's own tables
(and, on PostgreSQL, their sequences -- excluding a table does not exclude its sequence) are
excluded from what gets restored, and its appconfig rows are captured before the database
restore and written back after, in their own transaction.
- config.php's 'secret' is merged in from the backup (needed so already-encrypted content in the
restored database, e.g. share tokens, stays decryptable), but Target::options, the master key,
the wrapped key and the recovery kit code are all encrypted with that same instance secret
(OCP's ICrypto, or Nextcloud AppConfig's own "sensitive" flag underneath), not with ng_backup's
own key. Changing it without re-encrypting them first left all four permanently undecryptable
the moment any later occ command ran. Fixed with two helpers
(TargetService::reencryptOptionsAround(), KeyService::reencryptForSecretRotation()) that decrypt
before and re-encrypt after whatever callback actually changes the secret; config.php is applied
dead last in restoreFull(), after every other repository access. Not crash-safe (a process kill
mid-rotation would leave a key missing, not just stale) -- disaster recovery is run once by
hand, right after occ maintenance:install, a window too narrow yet to justify a two-phase write.
- A subtler variant of the same bug: Nextcloud's own AppConfig additionally encrypts a "sensitive"
value itself, tied to the same secret, underneath our own encryption. Overwriting an existing
sensitive key makes it read the old value first to compare, which then also fails against the
new secret. Deleting the key before setting the new value avoids that read entirely.
Roadmap (PLAN.md 9): phase 3 (in-place full restore, user self-service) moved out to v1.3.0; phase
2 continues toward v1.2.0 with FTP/SMB, a verification/checksum-audit command, user-export
retention, and deciding whether per-user export joins the automatic schedule.
Wires the existing RepositoryExportDestination/RepositoryImportSource (see
lib/UserMigration/) into user_migration's own UserMigrationService, resolved on demand
through a new UserMigrationServiceLocator since the app is an optional dependency
(PLAN.md 3.8) -- without it, only this feature is unavailable, the full backup still
works.
New: UserBackupService::backupUser() (occ backup:backup:user) streams a user_migration
export straight into the repository, under the same shared location lock as a regular
backup. UserRestoreService::restoreUser() (occ backup:restore:user) never imports over a
live account: --mode=replace safety-exports the current state first, then deletes and
recreates the account from the backup under the same uid; --mode=as-backup restores into
a new "<uid>-bak" account instead, since Nextcloud cannot rename a uid.
The catalog now also tracks user export manifests (Repository::recordUserExport/
listUserExports/forgetUserExport/isRecordedUserExport), rollback-protected like a
snapshot: listUserExports() verifies each one is still present on the location, and
restore only ever reads a manifest the verified catalog actually recorded, never an
arbitrary caller-supplied path. prune()'s existing raw users/ scan for blob retention is
unchanged. A new short exclusive "catalog" lock (PruneService::catalogLockKey())
serializes the catalog write itself, so two concurrent per-user backups (or one
alongside a full backup) can no longer race to write the same next generation and
silently drop one of them -- the broader per-target shared lock only ever protected
against a concurrent cleanup, not against other shared holders.
Two real bugs found by testing this against a live Nextcloud instance, not by unit
tests (RepositoryExportDestination/RepositoryImportSource implement OCP interfaces that
plain PHPUnit can't load, see tests/Unit/Repository/UserExportTest.php's docblock; the
new tests/integration/user-roundtrip.sh covers the real round trip instead):
- RepositoryImportSource::copyToFolder() double-closed the stream it handed to
putContent()/newFile() (both already close it themselves), which throws a TypeError on
PHP 8 and broke every restored file with content.
- A replace-mode restore failed on NotPermittedException: IRootFolder::getUserFolder()
caches a Folder per uid for the process lifetime, so after delete+recreate of the same
uid the import still got the deleted account's (now storage-less) folder. Fixed by
dispatching FilesystemTornDownEvent, which Root already listens for to clear exactly
that cache -- no internal OC\ classes needed.
Local review (CodeRabbit + Codex) against this diff found three more real issues, all
fixed: the catalog write race above, a failed as-backup import leaving a broken stub
account behind (now deleted on failure), and an export stream leak on a read/backend
failure in RepositoryExportDestination::copyFolder() (now closed in a finally).
Deliberately left out: wiring per-user export into the automatic backup schedule
(on-demand via occ only, for now) and retention/pruning of old user exports -- both open
design questions, not yet decided. The lease covering a single export/import still has a
fixed TTL (3600s) with no mid-call refresh, since UserMigrationService exposes no
progress; a resumable, step-based per-user backup (like the main BackupService run)
would remove that cap but is a bigger change than this pass.
RepositoryImportSource::copyToFolder() called putContent() on the generic
return type of Node::get(), which doesn't declare that method -- only
File does. Existing-file overwrite during a per-user restore would have
fatally errored at runtime; added an instanceof check and a resource leak
fix (finally-close) while there. Also guarded getFileAsStream() against
fopen() returning false.
Everything else Psalm (strict level) currently flags is mechanical
(missing `final` on leaf classes -- applied everywhere safe) or pre-existing
array-shape/type gaps, captured in psalm-baseline.xml so CI fails only on
new issues from here on, not the whole backlog at once.
Added tests/integration/targets-matrix.sh, which runs the full backup/restore
round trip through files_external's AmazonS3 and SFTP backend classes (S3
via a disposable s3mock container, SFTP via atmoz/sftp), since the existing
matrix only ever exercised the local backend.
This caught a real bug: moving a snapshot to the trash read it from the
location and streamed it straight back into a write on the same connection,
which SFTP can't do reliably (phpseclib fails to interleave a read and a
write on one connection). Backends now expose a native move() and the
id-bound snapshot object is moved with that instead of a read+write copy;
the path-bound database dump still goes through decrypt/re-encrypt, since
its AEAD authentication is tied to its path.
- tests/integration/roundtrip.sh: key, kit code and confirmation,
location, full and unchanged incremental backup, restore new-folder and
replace (files identical, extra file in the trash), database dump and
restore, retention, trash and restore from trash, no files_external
errors in the log
- tests/integration/matrix.sh: fresh Nextcloud per version and database
(SQLite, MariaDB, PostgreSQL), removes only its own containers and
their volumes; locally green on NC32 and NC35 with all three databases
- GitHub Actions: unit tests on PHP 8.2-8.4, round trip on NC32-35 x
SQLite/MariaDB/PostgreSQL
Fixes found by the matrix:
- files_external is no longer booted when it is disabled: only its
classes are loaded and a private BackendService gets its backends from
its Application class (booting registered its mount provider, which
failed on installations where its tables were never created)
- SQLite: the database file is excluded from the file backup (it is
covered by the logical dump and copying the live file is inconsistent)
- occ backup:key:confirm accepts --code and --confirm for scripts
dillard