43 Commits

Author	SHA1	Message	Date
dillard	b37a953cfd	Detect a busy run by identity, and clean up a failed kit write ... step() hands back the same Run object when another process holds the lease; comparing update times per second could stop the scheduled job early after a quick step. A failed write of the recovery kit file now closes the handle and removes the partial file.	2026-10-07 00:48:11 +02:00
dillard	16e09f4fde	Fix trash on SFTP and the disaster recovery test after the review fixes ... forget() and untrash() uploaded a remote read stream directly, which the new size check could not judge on SFTP, so cleanup failed there. They now copy through a local buffer, and the size check only trusts local streams (php://temp, php://memory, plain files). The disaster recovery test calls backup:restore:full with --no-interaction: docker exec without a terminal still counts as interactive, so the new confirmation question read an empty answer and cancelled.	2026-10-06 23:59:17 +02:00
dillard	45069994e9	Pre-beta review: safer destructive commands and hardening ... - backup:restore:full refuses on an installation with more than one user and asks before it starts (--force skips both, also to rerun after a failure halfway), and warns when the snapshot's database dump was not one consistent transaction. - backup:user:restore --mode=replace asks before deleting an existing account; non-interactive use needs --force. - The recovery kit file is created owner-only and never over an existing file or symlink; backup:key:import shows the deletion delay it read from the kit. - Adding a location through the web API needs password confirmation, like removing one. - Key derivation parameters from a kit or repository config are bounded, and a stalled network stream no longer spins at full CPU. - Floating-point columns are dumped exactly instead of with 14 digits.	2026-10-06 23:48:41 +02:00
dillard	0f4a238414	Pre-beta review: data safety in storage, catalog and backup runs ... Storage and repository: - Listing skipped every name containing ".part-", directories too, so the exports of a user like "john.part-time" were invisible to prune on S3, SFTP, FTP and SMB and their data was deleted. Only real temporary uploads ("<name>.part-<8 hex>") are skipped now. - prune refuses to run while a recorded per-user export's manifest is missing, instead of freeing that export's data. - forget() and untrash() copy first, switch the catalog, and delete the originals last, so a crash can no longer leave the catalog pointing at a missing snapshot or database dump. - Writes are checked against the expected size, so a short upload is an error instead of a truncated pack; raw restores check rename() and touch(). - A catalog write refuses to overwrite a generation another process just wrote, and the catalog lock lasts 300 s, so opening a large repository inside it no longer lets the lease run out. Backup runs and scheduling: - A file that keeps changing is read again at most three times; then the last read is kept and reported, instead of restarting forever and blocking cleanup. - Unreadable files and folders are recorded, logged, shown by occ backup:run, stored in the snapshot and notified to admins, instead of being skipped silently. - After a failed scheduled backup, at most three attempts a day, an hour apart, instead of a new run and notification every five minutes. - The job no longer spins on a run whose lease another process holds, and the weekly cleanup catches up after a Sunday spent backing up. - Notifications for the per-user export retention events.	2026-10-06 23:48:40 +02:00
dillard	e8ecb77df5	Refresh the location lease while a per-user restore reads its export ... A long import could outlive the lease, letting a cleanup forget the export being restored and free its blobs mid-read.	2026-10-06 22:59:49 +02:00
dillard	70965dd4bd	Keep leases alive during long restores and exports; PR review fixes ... Full restore, raw directory restore, per-user export and the safety export held a fixed-TTL lease on the location that nothing refreshed. A restore or export running longer than that TTL let a cleanup take over and delete data still in use. They now refresh the lease while they work (per blob, per row batch, per written chunk), throttled to once a minute. Also: - backup:prune still applies snapshot retention when user-export retention fails for a location. - The recovery kit carries the deletion delay, so a disaster recovery keeps the original protection instead of falling back to 7 days. Older kits still import with the default. - CI checkouts no longer persist the token in the git config. - recoveryKit() has its real return type; the stale baseline entry is gone.	2026-10-06 22:38:47 +02:00
dillard	cc09d74942	Commit full-backup snapshots to the catalog under the catalog lock ... A full backup opened its repository at the start of a step and wrote the catalog at the end of it, without the catalog lock. A per-user export or restore that wrote the catalog in between could lose its entry, or the snapshot could lose its own. The final step now only stores the snapshot object and moves the run to a new 'commit' phase. BackupService then records the snapshot under the catalog lock, from a repository opened inside that lock; if the lock is busy, the next step retries the commit. Repository::writeSnapshot() keeps its old behaviour as putSnapshot() plus recordSnapshot().	2026-10-06 17:58:21 +02:00
dillard	eb9100a74d	Fix review findings: catalog writes, user restore consent, retention validation ... Catalog: forgetting a user export or purging trash now writes the catalog before deleting the objects, so a crash leaves at most an unreferenced file instead of a catalog entry pointing at nothing. Per-user backup and the safety export reopen the repository inside the catalog lock, so they build on the latest catalog generation. Commands: backup:user:backup and backup:user:restore resolve the location before asking to install user_migration, and a successful install always stops with a rerun instruction. backup:retention rejects a malformed --user-last before changing any policy. backup:key:import fails clearly when the passphrase file cannot be read.	2026-10-06 17:40:51 +02:00
dillard	943c95bf07	Ask before installing user_migration; add the beta how-to ... backup:user:backup and backup:user:restore now ask the administrator before installing or enabling user_migration (or take --install-user-migration), so nothing changes on the server without consent. The restore warns that user_migration imports only its allowlisted app settings. Also: a clear error for unknown locations instead of a raw database message; prune reports the number of affected users, not the number of kept ones; and --user-last rejects values that are not whole numbers. Adds docs/HOWTO.md and updates the README and PLAN.md for the beta.	2026-10-06 16:57:56 +02:00
dillard	ba7f70e0f8	Rename per-user backup/restore commands to backup:user:backup and backup:user:restore	2026-10-06 03:51:35 +02:00
dillard	d70a869070	Phase 2: verify per-user exports, refuse SMB without the native extension, matrix user CI ... backup:verify accepts a user export path (users/<uid>/<export>): every blob its manifest references is checked the same way as for a snapshot, shallow or --deep. The SMB backend is refused at backup:target:add when the native smbclient PECL extension is not loaded, instead of registering and then failing the first backup with "Seek failed": with only the smbclient binary files_external falls back to a shell stream that cannot seek. The per-user integration job now runs on Nextcloud 32-35, not just 35: it calls user_migration's internal UserMigrationService, whose API may change between versions.	2026-10-06 03:47:02 +02:00
dillard	c9f82a79e4	Phase 2: retention for per-user exports (occ backup:retention --user-last) ... PruneService::userExportRetention()/setUserExportRetention(): a single, app-wide "keep the newest N" setting (default 3, same default as the snapshot policy's "last"), deliberately simpler than the full daily/weekly/monthly RetentionPolicy -- per-user exports are on-demand only (occ backup:backup:user, see the fold-out task about whether they ever join the automatic schedule), so there is no regular cadence yet to bucket by day/week/month. PruneService::applyUserExports(): groups a location's user exports by uid (read from each manifest's own 'time' field, the same field snapshot retention already reads), keeps the newest N per user, forgets the rest. No trash, unlike a snapshot: these are not yet the primary safety net the deletion delay protects. Runs under the same exclusive lock as snapshot retention. occ backup:retention gained --user-last=N; occ backup:prune now also applies it for every target it processes, reporting counts the same way it already does for snapshots and blob packs. Verified live against the dev rig (four real exports across four users, confirmed exactly the expected ones were kept/forgotten at two different --user-last values) and added coverage to tests/integration/user-roundtrip.sh (an account's manual export plus the automatic safety-export a replace-mode restore makes of it, pruned down to --user-last=2).	2026-10-06 03:18:14 +02:00
dillard	935c7b6a08	Phase 2: checksum audit (occ backup:verify) ... Repository::verify(): every blob a snapshot's files and database dump reference must be in a freshly re-downloaded, re-authenticated index (never trusted from the local cache, same reasoning as prune()) and have its pack still present on the location -- shallow, cheap, no blob content read, catches a deleted pack, a tampered/rolled-back index, or a database manifest that went missing (reported as a failure, not silently skipped). With $deep, each blob is also downloaded and decrypted, same as loadBlob() already does on every real read (AEAD authentication tag, content hash against its blob id), done here deliberately for everything in the snapshot rather than waiting for it to come up during an actual restore. A blob referenced by more than one file is only checked once; pack presence is cached per pack, not re-checked per blob. occ backup:verify <target> <snapshot> [--deep], meant to run periodically (cron, a sample of recent snapshots), per PLAN.md's "Verificatie: periodiek checksums controleren, steekproef-restore". The lease covering the audit is refreshed roughly every half of its TTL while blobs are checked, rather than relying on a single long-lived lease like the other new phase-2 commands do. Verified against a real S3-backed 1.4 GiB / 603-blob snapshot (shallow: instant; --deep: ~28s, fully decrypted) in addition to the new unit tests and the roundtrip.sh addition (shallow and deep on a snapshot with real, varied file sizes, now covered on every CI matrix combination). Local review (Codex) on the first pass found three real issues, all fixed: a missing database manifest was silently skipped instead of reported; shallow verification trusted the (possibly stale or unauthenticated) local index cache instead of re-checking the authenticated index on the location; and the lease had no refresh, so a --deep audit of a very large snapshot could outrun its TTL and run unprotected against a concurrent prune.	2026-10-05 23:56:16 +02:00
dillard	f18b4f0862	Phase 2: disaster recovery (occ backup:restore:full) ... New commands, per PLAN.md 3.7: - occ backup:key:import: adopt the master key from a recovery kit on a fresh installation (instead of occ backup:key:init generating a new one), so occ backup:target:add opens the existing repository on the location rather than creating an empty one. - occ backup:restore:full: replace the database (consistent, one transaction), restore the data directory in place, and merge config.php -- refuses up front and lists what's missing when the Nextcloud major version or any app enabled at backup time doesn't match here (their tables must exist before the database restore can use them), rather than guessing or partially applying. Repository::restore() gained an optional $stripPrefix (write "data/<uid>/..." straight into the real data directory, not a "data/" subdirectory of it) -- restoring "data/" and "config/" this way needs no temporary copy of what can be the largest part of an instance. occ backup:target:add now backfills the local snapshot cache (ngb_snapshots, what occ backup:list actually reads) when it adopts an existing repository: without this, a freshly connected location showed no restore points at all, even though the repository itself had them. One unreadable snapshot is skipped rather than aborting the backfill or the target add itself. Three self-referential problems found and fixed by testing this against two real, separate Nextcloud instances sharing only the backup location (tests/integration/full-restore-roundtrip.sh covers this; the per-process nature of these bugs meant they were invisible from inside a single occ invocation, including the one used to write the fixes, which is exactly why a second, fresh process re-running occ backup:status etc. was necessary to catch them): - The full database dump includes ng_backup's own tables and appconfig rows; restoring them verbatim would overwrite the key/target that is actively performing the restore, on THIS installation, with the backed-up instance's. Both are now preserved: ng_backup's own tables (and, on PostgreSQL, their sequences -- excluding a table does not exclude its sequence) are excluded from what gets restored, and its appconfig rows are captured before the database restore and written back after, in their own transaction. - config.php's 'secret' is merged in from the backup (needed so already-encrypted content in the restored database, e.g. share tokens, stays decryptable), but Target::options, the master key, the wrapped key and the recovery kit code are all encrypted with that same instance secret (OCP's ICrypto, or Nextcloud AppConfig's own "sensitive" flag underneath), not with ng_backup's own key. Changing it without re-encrypting them first left all four permanently undecryptable the moment any later occ command ran. Fixed with two helpers (TargetService::reencryptOptionsAround(), KeyService::reencryptForSecretRotation()) that decrypt before and re-encrypt after whatever callback actually changes the secret; config.php is applied dead last in restoreFull(), after every other repository access. Not crash-safe (a process kill mid-rotation would leave a key missing, not just stale) -- disaster recovery is run once by hand, right after occ maintenance:install, a window too narrow yet to justify a two-phase write. - A subtler variant of the same bug: Nextcloud's own AppConfig additionally encrypts a "sensitive" value itself, tied to the same secret, underneath our own encryption. Overwriting an existing sensitive key makes it read the old value first to compare, which then also fails against the new secret. Deleting the key before setting the new value avoids that read entirely. Roadmap (PLAN.md 9): phase 3 (in-place full restore, user self-service) moved out to v1.3.0; phase 2 continues toward v1.2.0 with FTP/SMB, a verification/checksum-audit command, user-export retention, and deciding whether per-user export joins the automatic schedule.	2026-10-05 23:31:08 +02:00
dillard	6cbc5d79d4	Phase 2: per-user backup and restore via user_migration ... Wires the existing RepositoryExportDestination/RepositoryImportSource (see lib/UserMigration/) into user_migration's own UserMigrationService, resolved on demand through a new UserMigrationServiceLocator since the app is an optional dependency (PLAN.md 3.8) -- without it, only this feature is unavailable, the full backup still works. New: UserBackupService::backupUser() (occ backup:backup:user) streams a user_migration export straight into the repository, under the same shared location lock as a regular backup. UserRestoreService::restoreUser() (occ backup:restore:user) never imports over a live account: --mode=replace safety-exports the current state first, then deletes and recreates the account from the backup under the same uid; --mode=as-backup restores into a new "<uid>-bak" account instead, since Nextcloud cannot rename a uid. The catalog now also tracks user export manifests (Repository::recordUserExport/ listUserExports/forgetUserExport/isRecordedUserExport), rollback-protected like a snapshot: listUserExports() verifies each one is still present on the location, and restore only ever reads a manifest the verified catalog actually recorded, never an arbitrary caller-supplied path. prune()'s existing raw users/ scan for blob retention is unchanged. A new short exclusive "catalog" lock (PruneService::catalogLockKey()) serializes the catalog write itself, so two concurrent per-user backups (or one alongside a full backup) can no longer race to write the same next generation and silently drop one of them -- the broader per-target shared lock only ever protected against a concurrent cleanup, not against other shared holders. Two real bugs found by testing this against a live Nextcloud instance, not by unit tests (RepositoryExportDestination/RepositoryImportSource implement OCP interfaces that plain PHPUnit can't load, see tests/Unit/Repository/UserExportTest.php's docblock; the new tests/integration/user-roundtrip.sh covers the real round trip instead): - RepositoryImportSource::copyToFolder() double-closed the stream it handed to putContent()/newFile() (both already close it themselves), which throws a TypeError on PHP 8 and broke every restored file with content. - A replace-mode restore failed on NotPermittedException: IRootFolder::getUserFolder() caches a Folder per uid for the process lifetime, so after delete+recreate of the same uid the import still got the deleted account's (now storage-less) folder. Fixed by dispatching FilesystemTornDownEvent, which Root already listens for to clear exactly that cache -- no internal OC\ classes needed. Local review (CodeRabbit + Codex) against this diff found three more real issues, all fixed: the catalog write race above, a failed as-backup import leaving a broken stub account behind (now deleted on failure), and an export stream leak on a read/backend failure in RepositoryExportDestination::copyFolder() (now closed in a finally). Deliberately left out: wiring per-user export into the automatic backup schedule (on-demand via occ only, for now) and retention/pruning of old user exports -- both open design questions, not yet decided. The lease covering a single export/import still has a fixed TTL (3600s) with no mid-call refresh, since UserMigrationService exposes no progress; a resumable, step-based per-user backup (like the main BackupService run) would remove that cap but is a bigger change than this pass.	2026-10-05 17:36:46 +02:00
dillard	807353cbe4	Wire Psalm into CI, fix a real bug it found in user-migration import ... RepositoryImportSource::copyToFolder() called putContent() on the generic return type of Node::get(), which doesn't declare that method -- only File does. Existing-file overwrite during a per-user restore would have fatally errored at runtime; added an instanceof check and a resource leak fix (finally-close) while there. Also guarded getFileAsStream() against fopen() returning false. Everything else Psalm (strict level) currently flags is mechanical (missing `final` on leaf classes -- applied everywhere safe) or pre-existing array-shape/type gaps, captured in psalm-baseline.xml so CI fails only on new issues from here on, not the whole backlog at once.	2026-10-05 11:45:37 +02:00
dillard	48ea6bb33d	CI: round trip against real S3 and SFTP backends, not just a local path ... Added tests/integration/targets-matrix.sh, which runs the full backup/restore round trip through files_external's AmazonS3 and SFTP backend classes (S3 via a disposable s3mock container, SFTP via atmoz/sftp), since the existing matrix only ever exercised the local backend. This caught a real bug: moving a snapshot to the trash read it from the location and streamed it straight back into a write on the same connection, which SFTP can't do reliably (phpseclib fails to interleave a read and a write on one connection). Backends now expose a native move() and the id-bound snapshot object is moved with that instead of a read+write copy; the path-bound database dump still goes through decrypt/re-encrypt, since its AEAD authentication is tied to its path.	2026-10-05 03:15:34 +02:00
dillard	b6ba09b078	Admin UI: restore point dates and a settings/daily-use split ... Show the date of the restore point a restore run used, both in the backup history table and when confirming a restore. Split the admin page into a "Backup and restore" tab (locations, backups, restore) and a "Settings" tab (key, slots, recovery kit), so daily work no longer sits behind the key setup at the top of the page. Signed-off-by: Dillard Blom <dillard.blom@ensembia.com>	2026-10-02 17:11:34 +02:00
dillard	85e835bfc2	Storage limit: 80% warning and limit in the web interface ... - after each successful backup the stored size is recorded; at 80% of a location's limit administrators get a notification (once a day) - locations list shows stored size and limit; the add form has a 'maximum size' field that explains that empty means unlimited and billed by the provider; PUT /api/targets/{id}/limit (password confirmation) Tested on NC34 with the OVH test bucket: a backup above the limit stops, a successful one at 81% sends the warning.	2026-10-02 14:50:16 +02:00
dillard	2610c1ad2f	Expiring leases instead of long-held locks ... A backup step killed by a time limit, OOM or kill -9 kept Nextcloud's lock on its location for up to an hour, blocking every new start and the run itself. Locations and runs are now guarded by leases in ngb_leases with an expiry: a backup step holds them for its duration plus two minutes, restores refresh theirs every minute, prune holds 30 minutes, untrash 5. Nextcloud's lock is only used for the milliseconds in which a lease is checked and written. Also: starting a backup while one is already running simply continues it (no exclusive lease needed), and running to completion waits instead of spinning while a crashed step's lease has not expired. Tested on NC34: a backup killed four times at random moments resumes on its own after the lease expires (about two minutes), and restores bit-identically.	2026-10-02 14:44:52 +02:00
dillard	532b23482e	Storage limit per location ... S3 has no bucket quota, so NG Backup enforces one: before a pack is uploaded, the stored size plus the pack must stay under the location's limit, otherwise the backup stops with an error (and administrators are notified through the failed run). Repacking during prune is exempt, as it is how space is freed. occ backup:target:limit NAME [SIZE|none]. Also ignore credential files in the repository.	2026-10-02 14:23:25 +02:00
dillard	c54b2cebed	M5: end-to-end test matrix and CI ... - tests/integration/roundtrip.sh: key, kit code and confirmation, location, full and unchanged incremental backup, restore new-folder and replace (files identical, extra file in the trash), database dump and restore, retention, trash and restore from trash, no files_external errors in the log - tests/integration/matrix.sh: fresh Nextcloud per version and database (SQLite, MariaDB, PostgreSQL), removes only its own containers and their volumes; locally green on NC32 and NC35 with all three databases - GitHub Actions: unit tests on PHP 8.2-8.4, round trip on NC32-35 x SQLite/MariaDB/PostgreSQL Fixes found by the matrix: - files_external is no longer booted when it is disabled: only its classes are loaded and a private BackendService gets its backends from its Application class (booting registered its mount provider, which failed on installations where its tables were never created) - SQLite: the database file is excluded from the file backup (it is covered by the logical dump and copying the live file is inconsistent) - occ backup:key:confirm accepts --code and --confirm for scripts	2026-10-02 11:56:57 +02:00
dillard	48bb829d45	M5: resumable database dump and faster database restore ... - DbDumper::dumpStep(): tables are dumped until the step's deadline in one REPEATABLE READ transaction and resumed at the last primary key in the next step; the running table checksum is carried over. A dump that fits in one step is fully consistent, otherwise the snapshot records dbConsistent=false. BackupService uses it, so the database phase no longer needs one long request (hosted Nextcloud, web cron) - restore inserts 100 rows per statement and leaves tables whose checksum already matches the dump untouched - spike C: 30-step dump gives the same checksums as the one-shot dump; restore rewrote only the 3 damaged tables of 147; 300k-row table restored in 14.6 s instead of 46 s	2026-10-02 11:34:59 +02:00
dillard	f75ed3aed6	M5: chained catalog against rollback and hidden snapshots ... Every change to the set of snapshots or trash entries writes an encrypted catalog generation bound to the repository and its position, with the hash of the previous generation. The server anchors the newest generation it has seen in its app config; on open, the location must show at least that generation and chain back to it, and every snapshot listed must exist. A rolled-back or tampered location is refused (the run fails and administrators are notified). Opening with the passphrase on a new server accepts the newest valid chain. occ backup:target:trust-current accepts a legitimately older state (e.g. restored from the provider's snapshot) after typing the location name; audited and announced to all administrators. Tests: chain advances, rollback, hidden snapshot, swapped generation, disaster recovery without anchor. Tested on NC34 by deleting the newest catalog generation on the SFTP location.	2026-10-02 11:30:42 +02:00
dillard	131bb6178d	Close remaining review gaps ... - restore runs from the web are created under the location's exclusive lock, like backup runs - raw restore writes to an unpredictable temporary name opened with exclusive create (never follows a planted symlink) and validates the path prefix; the restore service rejects non-canonical paths instead of trimming them (occ normalises user input before calling it) - restore jobs report progress every minute, so a long restore is not taken for a dead one - README: Object Lock turns deletions into delete markers	2026-10-02 11:25:22 +02:00
dillard	ac82b38e37	Fixes from the phase-1 code review ... - one locking protocol per location: backup steps and restores hold a shared lock, prune and untrash an exclusive one and refuse while any run (backup, restore, export) exists; starting a backup checks and creates the run under the exclusive lock, closing the race with prune - snapshot entries are validated when read (relative path without '.', '..', NUL, backslash or control characters); raw restore also checks that the resolved directory stays below the target; the restore API and service accept canonical paths only - recovery kit confirmation requires a recent password confirmation and the confirmation code printed in the current kit file (new code per kit version), proving the kit was actually downloaded - prune decides on the authenticated index files on the location and refreshes the local index cache from them - restore job marks the run failed on any error; restore runs without progress for 6 hours are failed so they cannot block cleanup forever - tests for safe paths and a malicious snapshot path	2026-10-02 11:21:40 +02:00
dillard	d3aa07b106	M5: deletion delay with trash, passphrase slots, ransomware documentation ... - removed snapshots go to a trash on the location and their data is only freed after the deletion delay, which is chosen when the key is created (default 7 days, 1-365) and cannot be changed afterwards; any admin can bring them back (occ backup:trash, backup:trash:restore, web UI) - up to three passphrase slots open the same master key (like LUKS), so one person leaving or dying does not lock the company out; slot changes are written to every location, announced to all admins and require a new recovery kit confirmation, while backups keep running - occ backup:key:slots, backup:key:slot:add|replace|remove; key init takes --label and --delete-delay-days - web UI: deletion delay and first slot at key creation, slot management, trash per location with restore; all behind password confirmation - README: key, slots (safe / CTO / head of IT), and protection against ransomware: object lock in compliance mode (e.g. 90 days), credentials without delete rights, provider snapshots, off-site copy	2026-10-02 03:01:44 +02:00
dillard	b9e8858d76	M5: notifications, setup check, audit log and password confirmation ... - notifications to administrators for failed backup/restore runs and locations without a successful backup for two days (once a day) - setup check in the admin overview: error when the key/kit is not set up, warning for missing, old or failed backups - security events (recovery kit downloaded, location removed, append-only turned off, retention changed, snapshots forgotten) go to the audit log and are announced to all administrators, from the web and from occ; the scheduled weekly cleanup is logged only - fix: the recovery-kit download link failed the CSRF check (412); it now requires a recent password confirmation instead, like creating the key, removing a location and restoring	2026-10-02 02:29:46 +02:00
dillard	38654317a2	M5: retention policy, prune with repacking, append-only locations ... - RetentionPolicy (last/daily/weekly/monthly) and PruneService: forget snapshots the policy drops, delete packs nothing uses and repack mostly unused ones by copying encrypted blobs as-is; new packs are indexed before old ones are removed - a location lock keeps prune and backups apart; append-only locations are never pruned (cleanup via the location's lifecycle rules) - weekly prune from the background job; occ backup:prune [--dry-run], backup:retention, backup:target:append-only - tests: forget+prune frees space and the remaining snapshot restores identically, repack of mixed packs, dry-run preview, retention buckets	2026-10-02 02:21:48 +02:00
dillard	36e6a3fde7	M4: admin settings page and JSON API ... - AdminApiController (admin only, CSRF-protected): status, create key, download recovery kit, confirm (statement accepted + exact phrase), backends with their usable auth mechanisms, add/test/remove locations, start backups, schedule, snapshots, browse, queue restores - RestoreJob runs web-initiated restores in the background; backup and restore runs are kept apart in the run table - admin section 'NG Backup' with a Vue 3 / @nextcloud/vue 9 page: key and recovery-kit flow (download, tick statement, 'Yes, I confirm'), location form built from files_external's parameter definitions, schedule, run list with polling, snapshot browser and restore with the three modes - auth mechanisms that depend on a user session or on files_external's own stored credentials are not offered API tested with curl on NC34 (status, backends, schedule validation, confirmation refusal, browse, queued restore and backup completed by cron); non-admins get 403, anonymous 401.	2026-10-02 01:09:47 +02:00
dillard	c601a02b2b	M3: restore files and folders (occ backup:browse, backup:restore) ... User files are restored through the Files API so the file cache, versions and trash stay consistent: modes new-folder (default, into 'Restored <date>/<path>'), merge (overwritten files keep their old content as a version) and replace (extra files go to the trash). Each file is staged under a temporary name and moved into place only after all its blobs were verified. Any snapshot path can also be restored raw to a local directory (disaster recovery). Tested on NC34: all three modes restore checksum-identical files, the overwritten content is kept as a version, the extra file is in the trash, no temporary files remain.	2026-10-02 00:58:18 +02:00
dillard	370f3c55fa	M2: local cache of pack indexes (ngb_index) ... Opening a repository lists index/ once and downloads only index files that are not cached; packs written by this server are cached on write, stale cache rows are dropped. Each resumable step therefore no longer downloads every index file from the location.	2026-10-02 00:55:46 +02:00
dillard	3f6785bcc0	M2: full backup runs (database, config, data) with occ and cron ... BackupService starts a run per location (database dump in one transaction, then config + data directory as a resumable file run), stores the state in ngb_runs after every step, locks a run while a step executes and records the snapshot. Previews, logs, updater backups and per-user cache/uploads are excluded (TreeWalker now supports patterns). BackupJob (every 5 min) starts the daily schedule and steps running runs with a short budget under web cron and a long one under system cron. occ: backup:run, backup:status, backup:list. Tested on NC34/PostgreSQL: first run 128 files / 1 GiB in 64 s (443 MiB uploaded to S3), unchanged second run 2 s / 0 bytes; a background run to SFTP completed through cron.php.	2026-10-02 00:53:50 +02:00
dillard	4a28bf8790	M2: cursor-based resumable runs with merge-join against the parent ... TreeWalker walks one or more roots in a stable segment-wise order with exclusions and resumes after a cursor; ParentCursor merge-joins the previous snapshot in the same order. BackupRun no longer keeps a file list or the parent's entries in memory or in its state.	2026-10-02 00:46:31 +02:00
dillard	506854d597	M1: key management, backup locations and occ commands ... - migration: ngb_targets, ngb_runs, ngb_snapshots - KeyService: master key stored encrypted with the instance secret (unattended backups), passphrase-wrapped copy in every repository and in the recovery kit, recorded admin confirmation per key fingerprint - TargetService: locations built from files_external backends without a mount; files_external is loaded through IAppManager::loadApp() and its enabled state is left alone; credentials encrypted with ICrypto; existing repositories are adopted only if made with this key, and a repository cannot be registered twice - Repository: initWithKey/openWithKey with an encrypted key check - occ: backup:key:init|kit|confirm, backup:target:backends|add|list|remove	2026-10-02 00:37:18 +02:00
dillard	9bfb6ae334	M1: PHPUnit test suite for crypto, repository and resumable runs ... Dev tooling via composer (phpunit, psalm, nextcloud/ocp); vendor/ is mounted from outside the repository by a helper script. 28 unit tests ported from the phase-0 spikes (round trips, corruption, wrong keys, dedup, incremental runs, partial restore, tampering, object binding, time-boxed steps, file changed between steps).	2026-10-02 00:32:56 +02:00
dillard	7b33347dbc	Phase 0: fixes from review (torn files, pending blobs, exact sequences, UTF-8, storage checks) ... - detect files changing during reading or between resumable steps and re-read them (size/mtime/ctime/inode), spike F overwrites mid-run - drop pending blob ids when a pack or index upload fails - dump and restore PostgreSQL sequence state exactly, bound parameters, validated catalog names; spike C checks all sequences exactly - keep invalid UTF-8 byte-exact (tagged base64) - refuse large tables without a primary key - StorageBackend: check write/flush/close results, validate base path - user_migration import/export: root paths, intermediate folders, partial writes, read errors - spikes: never touch an existing account, own fixtures, 512M note	2026-10-01 18:58:51 +02:00
dillard	93fec0e8f3	Phase 0: resumable time-boxed backup steps (spike F) and results ... BackupRun is a resumable state machine: each step works until a deadline, uploads its pack and tree segment, and returns the state; large files are resumed at the next blob. Snapshot is written last. Spike F: 200 MiB file resumed across steps in separate processes; with 6 random kill -9 the run completes and restores bit-identically. spikes/RESULTS.md summarises phase 0 and the findings for the plan.	2026-10-01 18:39:26 +02:00
dillard	d474050be0	Phase 0: per-user backup/restore through user_migration (spike E) ... RepositoryExportDestination / RepositoryImportSource implement the public OCP\UserMigration interfaces on top of the repository, so user_migration's migrators (calendar, contacts, trashbin, account, files) export straight into the encrypted repository. Spike E (NC34): export 0.8s, unchanged second export stores nothing; after deleting the user, import restores account, files (bit-identical), calendar event and contact. Findings: calendars/address books come back as 'migrated-*' with new object URIs; importing over an existing user duplicates contacts and adds a migrated birthday calendar; creating the user on import needs >128 MB (password_policy word list).	2026-10-01 18:36:57 +02:00
dillard	bc16a186d5	Phase 0: repository on files_external storages without a mount (spike D) ... ExternalStorageFactory builds storages with files_external's own backends and auth mechanisms from an unsaved StorageConfig (as its ConfigAdapter does), and StorageBackend adapts any IStorage to IBackend (writeStream, seekable range reads, temp name + rename except on S3). IBackend::put() now owns and always closes the stream, because files_external's S3 writeStream closes it. Spike D (NC34): S3 (SeaweedFS) and SFTP: 120 MiB backup ~6s, unchanged run uploads nothing, restore bit-identical, no files_external mount created, no temp files (max 12 KiB in /tmp). Works with files_external disabled when its classes are loaded. Backup peak memory 85-89 MiB on these targets (to investigate).	2026-10-01 18:33:15 +02:00
dillard	915e534ab2	Phase 0: logical DB dump/restore without external tools (spike C) ... DbDumper reads every prefixed table through the Nextcloud connection in one REPEATABLE READ (read-only) transaction, with keyset pagination on the primary key, and streams each table as JSON lines into its own blob stream (BlobWriter), so unchanged tables are deduplicated. Restore replaces table contents in one transaction and resets PostgreSQL sequences. Spike C (NC34, PostgreSQL, standard image without pg_dump/pgsql ext): 300k-row table with bytea: dump 6.3s / 50 MiB peak, 94 MiB JSON -> 20 MiB stored; unchanged second dump stores nothing; restore makes every table checksum-identical and all 110 sequences continue above the maximum.	2026-10-01 18:26:27 +02:00
dillard	1839997972	Phase 0: repository format with packs, dedup and incremental runs (spike B) ... Content-addressed repository on an IBackend (local implementation): 4 MiB blobs, deflate when it helps, each blob encrypted with its id as associated data, ~32 MiB packs with an encrypted header, one index file per pack, snapshots as encrypted JSON lines written last. Spike B: 390 MiB tree, peak 54 MiB; unchanged run reads nothing; a 1 MiB change in a 300 MiB file uploads one blob; full and partial restore are bit-identical; tampering is detected.	2026-10-01 18:18:38 +02:00
dillard	07e3d263b4	Phase 0: skeleton, streaming encryption (spike A) ... Repository skeleton (app id ng_backup, AGPL-3.0-or-later), KeyRing with KDF subkeys, keyed blob ids and Argon2id key wrapping, and StreamCipher (secretstream XChaCha20-Poly1305, framed, bounded memory). Spike A: 1 GiB round trip with memory_limit=64M, peak < 2 MiB, ~160/122 MiB/s; tampering, truncation, wrong key/passphrase are rejected.	2026-10-01 18:15:24 +02:00