10 Commits

Author	SHA1	Message	Date
dillard	943c95bf07	Ask before installing user_migration; add the beta how-to ... backup:user:backup and backup:user:restore now ask the administrator before installing or enabling user_migration (or take --install-user-migration), so nothing changes on the server without consent. The restore warns that user_migration imports only its allowlisted app settings. Also: a clear error for unknown locations instead of a raw database message; prune reports the number of affected users, not the number of kept ones; and --user-last rejects values that are not whole numbers. Adds docs/HOWTO.md and updates the README and PLAN.md for the beta.	2026-10-06 16:57:56 +02:00
dillard	d70a869070	Phase 2: verify per-user exports, refuse SMB without the native extension, matrix user CI ... backup:verify accepts a user export path (users/<uid>/<export>): every blob its manifest references is checked the same way as for a snapshot, shallow or --deep. The SMB backend is refused at backup:target:add when the native smbclient PECL extension is not loaded, instead of registering and then failing the first backup with "Seek failed": with only the smbclient binary files_external falls back to a shell stream that cannot seek. The per-user integration job now runs on Nextcloud 32-35, not just 35: it calls user_migration's internal UserMigrationService, whose API may change between versions.	2026-10-06 03:47:02 +02:00
dillard	f18b4f0862	Phase 2: disaster recovery (occ backup:restore:full) ... New commands, per PLAN.md 3.7: - occ backup:key:import: adopt the master key from a recovery kit on a fresh installation (instead of occ backup:key:init generating a new one), so occ backup:target:add opens the existing repository on the location rather than creating an empty one. - occ backup:restore:full: replace the database (consistent, one transaction), restore the data directory in place, and merge config.php -- refuses up front and lists what's missing when the Nextcloud major version or any app enabled at backup time doesn't match here (their tables must exist before the database restore can use them), rather than guessing or partially applying. Repository::restore() gained an optional $stripPrefix (write "data/<uid>/..." straight into the real data directory, not a "data/" subdirectory of it) -- restoring "data/" and "config/" this way needs no temporary copy of what can be the largest part of an instance. occ backup:target:add now backfills the local snapshot cache (ngb_snapshots, what occ backup:list actually reads) when it adopts an existing repository: without this, a freshly connected location showed no restore points at all, even though the repository itself had them. One unreadable snapshot is skipped rather than aborting the backfill or the target add itself. Three self-referential problems found and fixed by testing this against two real, separate Nextcloud instances sharing only the backup location (tests/integration/full-restore-roundtrip.sh covers this; the per-process nature of these bugs meant they were invisible from inside a single occ invocation, including the one used to write the fixes, which is exactly why a second, fresh process re-running occ backup:status etc. was necessary to catch them): - The full database dump includes ng_backup's own tables and appconfig rows; restoring them verbatim would overwrite the key/target that is actively performing the restore, on THIS installation, with the backed-up instance's. Both are now preserved: ng_backup's own tables (and, on PostgreSQL, their sequences -- excluding a table does not exclude its sequence) are excluded from what gets restored, and its appconfig rows are captured before the database restore and written back after, in their own transaction. - config.php's 'secret' is merged in from the backup (needed so already-encrypted content in the restored database, e.g. share tokens, stays decryptable), but Target::options, the master key, the wrapped key and the recovery kit code are all encrypted with that same instance secret (OCP's ICrypto, or Nextcloud AppConfig's own "sensitive" flag underneath), not with ng_backup's own key. Changing it without re-encrypting them first left all four permanently undecryptable the moment any later occ command ran. Fixed with two helpers (TargetService::reencryptOptionsAround(), KeyService::reencryptForSecretRotation()) that decrypt before and re-encrypt after whatever callback actually changes the secret; config.php is applied dead last in restoreFull(), after every other repository access. Not crash-safe (a process kill mid-rotation would leave a key missing, not just stale) -- disaster recovery is run once by hand, right after occ maintenance:install, a window too narrow yet to justify a two-phase write. - A subtler variant of the same bug: Nextcloud's own AppConfig additionally encrypts a "sensitive" value itself, tied to the same secret, underneath our own encryption. Overwriting an existing sensitive key makes it read the old value first to compare, which then also fails against the new secret. Deleting the key before setting the new value avoids that read entirely. Roadmap (PLAN.md 9): phase 3 (in-place full restore, user self-service) moved out to v1.3.0; phase 2 continues toward v1.2.0 with FTP/SMB, a verification/checksum-audit command, user-export retention, and deciding whether per-user export joins the automatic schedule.	2026-10-05 23:31:08 +02:00
dillard	807353cbe4	Wire Psalm into CI, fix a real bug it found in user-migration import ... RepositoryImportSource::copyToFolder() called putContent() on the generic return type of Node::get(), which doesn't declare that method -- only File does. Existing-file overwrite during a per-user restore would have fatally errored at runtime; added an instanceof check and a resource leak fix (finally-close) while there. Also guarded getFileAsStream() against fopen() returning false. Everything else Psalm (strict level) currently flags is mechanical (missing `final` on leaf classes -- applied everywhere safe) or pre-existing array-shape/type gaps, captured in psalm-baseline.xml so CI fails only on new issues from here on, not the whole backlog at once.	2026-10-05 11:45:37 +02:00
dillard	532b23482e	Storage limit per location ... S3 has no bucket quota, so NG Backup enforces one: before a pack is uploaded, the stored size plus the pack must stay under the location's limit, otherwise the backup stops with an error (and administrators are notified through the failed run). Repacking during prune is exempt, as it is how space is freed. occ backup:target:limit NAME [SIZE|none]. Also ignore credential files in the repository.	2026-10-02 14:23:25 +02:00
dillard	c54b2cebed	M5: end-to-end test matrix and CI ... - tests/integration/roundtrip.sh: key, kit code and confirmation, location, full and unchanged incremental backup, restore new-folder and replace (files identical, extra file in the trash), database dump and restore, retention, trash and restore from trash, no files_external errors in the log - tests/integration/matrix.sh: fresh Nextcloud per version and database (SQLite, MariaDB, PostgreSQL), removes only its own containers and their volumes; locally green on NC32 and NC35 with all three databases - GitHub Actions: unit tests on PHP 8.2-8.4, round trip on NC32-35 x SQLite/MariaDB/PostgreSQL Fixes found by the matrix: - files_external is no longer booted when it is disabled: only its classes are loaded and a private BackendService gets its backends from its Application class (booting registered its mount provider, which failed on installations where its tables were never created) - SQLite: the database file is excluded from the file backup (it is covered by the logical dump and copying the live file is inconsistent) - occ backup:key:confirm accepts --code and --confirm for scripts	2026-10-02 11:56:57 +02:00
dillard	f75ed3aed6	M5: chained catalog against rollback and hidden snapshots ... Every change to the set of snapshots or trash entries writes an encrypted catalog generation bound to the repository and its position, with the hash of the previous generation. The server anchors the newest generation it has seen in its app config; on open, the location must show at least that generation and chain back to it, and every snapshot listed must exist. A rolled-back or tampered location is refused (the run fails and administrators are notified). Opening with the passphrase on a new server accepts the newest valid chain. occ backup:target:trust-current accepts a legitimately older state (e.g. restored from the provider's snapshot) after typing the location name; audited and announced to all administrators. Tests: chain advances, rollback, hidden snapshot, swapped generation, disaster recovery without anchor. Tested on NC34 by deleting the newest catalog generation on the SFTP location.	2026-10-02 11:30:42 +02:00
dillard	38654317a2	M5: retention policy, prune with repacking, append-only locations ... - RetentionPolicy (last/daily/weekly/monthly) and PruneService: forget snapshots the policy drops, delete packs nothing uses and repack mostly unused ones by copying encrypted blobs as-is; new packs are indexed before old ones are removed - a location lock keeps prune and backups apart; append-only locations are never pruned (cleanup via the location's lifecycle rules) - weekly prune from the background job; occ backup:prune [--dry-run], backup:retention, backup:target:append-only - tests: forget+prune frees space and the remaining snapshot restores identically, repack of mixed packs, dry-run preview, retention buckets	2026-10-02 02:21:48 +02:00
dillard	370f3c55fa	M2: local cache of pack indexes (ngb_index) ... Opening a repository lists index/ once and downloads only index files that are not cached; packs written by this server are cached on write, stale cache rows are dropped. Each resumable step therefore no longer downloads every index file from the location.	2026-10-02 00:55:46 +02:00
dillard	506854d597	M1: key management, backup locations and occ commands ... - migration: ngb_targets, ngb_runs, ngb_snapshots - KeyService: master key stored encrypted with the instance secret (unattended backups), passphrase-wrapped copy in every repository and in the recovery kit, recorded admin confirmation per key fingerprint - TargetService: locations built from files_external backends without a mount; files_external is loaded through IAppManager::loadApp() and its enabled state is left alone; credentials encrypted with ICrypto; existing repositories are adopted only if made with this key, and a repository cannot be registered twice - Repository: initWithKey/openWithKey with an encrypted key check - occ: backup:key:init|kit|confirm, backup:target:backends|add|list|remove	2026-10-02 00:37:18 +02:00