11 Commits

Author	SHA1	Message	Date
dillard	a11e6c7254	1.0.1: label for "Back up now", retention explained in the trash, occ times in the schedule's zone, single-file restore to a directory, leftovers of a stopped restore removed ... - Admin page: an optional label (at most 100 characters) for a backup started with "Back up now"; an (i) by "removed by retention policy" in the trash that explains what it means and until when the restore point can be brought back. backup:trash says the same below its list. - backup:status, backup:list, backup:trash and backup:browse show times in the zone of the daily schedule and name it, instead of the server's UTC. - backup:restore --to-directory of a file restored nothing: the path got a "/" appended. - A restore removes "<name>.ngb-restore-<hex>" temporaries that a stopped restore left in the folders it writes to. - Roadmap: the cron warning (#13), reading ahead from S3/SMB and --salvage move to 1.0.2.	2026-10-30 19:32:53 +01:00
dillard	4f6a2e5b70	Check MySQL foreign keys explicitly; keep index prefix lengths and foreign keys of created tables ... Switching FOREIGN_KEY_CHECKS back on does not check the rows loaded meanwhile, and DDL would commit halfway, so on MySQL every foreign key is checked with a query for rows that point at nothing, before the commit. The recorded table structure now keeps index options (the prefix lengths MySQL needs for long text columns) and foreign keys; tables the restore creates get their foreign keys after the commit. The full-restore test also runs on MariaDB.	2026-10-18 02:09:00 +02:00
dillard	d8ef50732f	Full restore creates missing tables and suspends foreign keys; --install-apps; clearer messages ... Backups now record each table's structure in the dump manifest. The full restore creates tables that do not exist on the new instance (left behind by apps that were removed or are switched off), lists all problems at once, and no longer stops on apps that are missing. Foreign keys are not checked while rows are loaded in name order, and checked again before the restore commits (PostgreSQL: dropped and added back in the transaction). --install-apps installs or enables the apps that were enabled at backup time; --skip-missing-tables leaves out tables of older backups whose structure is unknown. nextcloud-aio is never required. backup:target:add without a key points to backup:key:import, and says that connecting can take a while. Removing a location drops its cached index; a migration removes the cache of locations that are gone. The full-restore test now also runs on PostgreSQL with Mail (foreign keys) and a disabled app.	2026-10-18 01:45:01 +02:00
dillard	ca003c88ba	Fast S3 verify and snapshot commit; shorter runs list; durations with days ... On S3, index files and the tree segments read back when a snapshot is written are fetched 16 at a time through the storage's S3 client (IBackend::getRanges(), getMany()). Blobs that follow each other in a pack are read as one range on every location. The snapshot is written in a step of its own and the run shows "commit" meanwhile. On a 1 TB production repository the check went from about 4 hours to 3.5 minutes. backup:status and the web interface list the runs of the last 7 days, running ones and at least the newest 5; --all / "Show older runs" list all. Durations of a day or more show the days.	2026-10-14 17:12:41 +02:00
dillard	828f3f142a	Merge branch 'fix/sftp-fresh-connection' into polish	2026-10-11 13:26:47 +02:00
dillard	894258f836	SFTP: open the channel before the first fast read; backup step: survive a takeover by another process	2026-10-11 11:45:27 +02:00
dillard	9775cdc475	Locations: edit connection settings without losing the history ... A changed password or host name could only be handled by adding a new location, which starts an empty repository and, with a retention lock at the provider, keeps the old one billed. Edit now changes a location in place (web interface and occ backup:target:edit). The settings are only saved when they lead to the same repository; a failed connection and a missing repository give separate errors, and the folder is not created while checking. Stored passwords are never sent to the browser: an empty password field keeps the stored value.	2026-10-08 12:56:15 +02:00
dillard	0f4a238414	Pre-beta review: data safety in storage, catalog and backup runs ... Storage and repository: - Listing skipped every name containing ".part-", directories too, so the exports of a user like "john.part-time" were invisible to prune on S3, SFTP, FTP and SMB and their data was deleted. Only real temporary uploads ("<name>.part-<8 hex>") are skipped now. - prune refuses to run while a recorded per-user export's manifest is missing, instead of freeing that export's data. - forget() and untrash() copy first, switch the catalog, and delete the originals last, so a crash can no longer leave the catalog pointing at a missing snapshot or database dump. - Writes are checked against the expected size, so a short upload is an error instead of a truncated pack; raw restores check rename() and touch(). - A catalog write refuses to overwrite a generation another process just wrote, and the catalog lock lasts 300 s, so opening a large repository inside it no longer lets the lease run out. Backup runs and scheduling: - A file that keeps changing is read again at most three times; then the last read is kept and reported, instead of restarting forever and blocking cleanup. - Unreadable files and folders are recorded, logged, shown by occ backup:run, stored in the snapshot and notified to admins, instead of being skipped silently. - After a failed scheduled backup, at most three attempts a day, an hour apart, instead of a new run and notification every five minutes. - The job no longer spins on a run whose lease another process holds, and the weekly cleanup catches up after a Sunday spent backing up. - Notifications for the per-user export retention events.	2026-10-06 23:48:40 +02:00
dillard	70965dd4bd	Keep leases alive during long restores and exports; PR review fixes ... Full restore, raw directory restore, per-user export and the safety export held a fixed-TTL lease on the location that nothing refreshed. A restore or export running longer than that TTL let a cleanup take over and delete data still in use. They now refresh the lease while they work (per blob, per row batch, per written chunk), throttled to once a minute. Also: - backup:prune still applies snapshot retention when user-export retention fails for a location. - The recovery kit carries the deletion delay, so a disaster recovery keeps the original protection instead of falling back to 7 days. Older kits still import with the default. - CI checkouts no longer persist the token in the git config. - recoveryKit() has its real return type; the stale baseline entry is gone.	2026-10-06 22:38:47 +02:00
dillard	f18b4f0862	Phase 2: disaster recovery (occ backup:restore:full) ... New commands, per PLAN.md 3.7: - occ backup:key:import: adopt the master key from a recovery kit on a fresh installation (instead of occ backup:key:init generating a new one), so occ backup:target:add opens the existing repository on the location rather than creating an empty one. - occ backup:restore:full: replace the database (consistent, one transaction), restore the data directory in place, and merge config.php -- refuses up front and lists what's missing when the Nextcloud major version or any app enabled at backup time doesn't match here (their tables must exist before the database restore can use them), rather than guessing or partially applying. Repository::restore() gained an optional $stripPrefix (write "data/<uid>/..." straight into the real data directory, not a "data/" subdirectory of it) -- restoring "data/" and "config/" this way needs no temporary copy of what can be the largest part of an instance. occ backup:target:add now backfills the local snapshot cache (ngb_snapshots, what occ backup:list actually reads) when it adopts an existing repository: without this, a freshly connected location showed no restore points at all, even though the repository itself had them. One unreadable snapshot is skipped rather than aborting the backfill or the target add itself. Three self-referential problems found and fixed by testing this against two real, separate Nextcloud instances sharing only the backup location (tests/integration/full-restore-roundtrip.sh covers this; the per-process nature of these bugs meant they were invisible from inside a single occ invocation, including the one used to write the fixes, which is exactly why a second, fresh process re-running occ backup:status etc. was necessary to catch them): - The full database dump includes ng_backup's own tables and appconfig rows; restoring them verbatim would overwrite the key/target that is actively performing the restore, on THIS installation, with the backed-up instance's. Both are now preserved: ng_backup's own tables (and, on PostgreSQL, their sequences -- excluding a table does not exclude its sequence) are excluded from what gets restored, and its appconfig rows are captured before the database restore and written back after, in their own transaction. - config.php's 'secret' is merged in from the backup (needed so already-encrypted content in the restored database, e.g. share tokens, stays decryptable), but Target::options, the master key, the wrapped key and the recovery kit code are all encrypted with that same instance secret (OCP's ICrypto, or Nextcloud AppConfig's own "sensitive" flag underneath), not with ng_backup's own key. Changing it without re-encrypting them first left all four permanently undecryptable the moment any later occ command ran. Fixed with two helpers (TargetService::reencryptOptionsAround(), KeyService::reencryptForSecretRotation()) that decrypt before and re-encrypt after whatever callback actually changes the secret; config.php is applied dead last in restoreFull(), after every other repository access. Not crash-safe (a process kill mid-rotation would leave a key missing, not just stale) -- disaster recovery is run once by hand, right after occ maintenance:install, a window too narrow yet to justify a two-phase write. - A subtler variant of the same bug: Nextcloud's own AppConfig additionally encrypts a "sensitive" value itself, tied to the same secret, underneath our own encryption. Overwriting an existing sensitive key makes it read the old value first to compare, which then also fails against the new secret. Deleting the key before setting the new value avoids that read entirely. Roadmap (PLAN.md 9): phase 3 (in-place full restore, user self-service) moved out to v1.3.0; phase 2 continues toward v1.2.0 with FTP/SMB, a verification/checksum-audit command, user-export retention, and deciding whether per-user export joins the automatic schedule.	2026-10-05 23:31:08 +02:00
dillard	807353cbe4	Wire Psalm into CI, fix a real bug it found in user-migration import ... RepositoryImportSource::copyToFolder() called putContent() on the generic return type of Node::get(), which doesn't declare that method -- only File does. Existing-file overwrite during a per-user restore would have fatally errored at runtime; added an instanceof check and a resource leak fix (finally-close) while there. Also guarded getFileAsStream() against fopen() returning false. Everything else Psalm (strict level) currently flags is mechanical (missing `final` on leaf classes -- applied everywhere safe) or pre-existing array-shape/type gaps, captured in psalm-baseline.xml so CI fails only on new issues from here on, not the whole backlog at once.	2026-10-05 11:45:37 +02:00