86 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Dillard Blom	54750e9742	Merge pull request #8: Release 0.9.4-beta.5 ... Restore and deep verify: read a pack as one stream; pipelined SFTP reads v0.9.4-beta.5	2026-10-09 12:05:31 +02:00
dillard	fe92ef70bf	Release 0.9.4-beta.5	2026-10-09 11:58:52 +02:00
dillard	b3e3054737	SftpReadStream: fall back to the storage's own stream instead of failing a read	2026-10-09 11:24:37 +02:00
dillard	c775e27dd2	CHANGELOG: SFTP reads with several requests in flight	2026-10-09 09:59:10 +02:00
dillard	d8e772137f	StorageBackend: read SFTP locations with several requests in flight ... files_external's SFTP read stream asks for the next 256 KiB only after the previous answer has arrived, so reading costs one round trip per request whatever the line speed: 4.9 MB/s at 47 ms to a Hetzner Storage Box. SftpReadStream keeps up to eight READ requests in flight over the same phpseclib connection (15 MB/s on that link), starting with one so a small read does not fetch megabytes ahead. A server that answers less than asked is handled by asking again from where the answer ended; when more data follows, that size is used for later reads over the same connection, never below the 32 KiB every server must support. Without the needed connection methods, or when the open fails, the storage's own stream is used as before.	2026-10-09 04:27:21 +02:00
dillard	bcfec17cfe	README: WebDAV request timeout on slow uploads; deep verify no longer fetches per blob	2026-10-08 22:38:32 +02:00
dillard	b309081d1a	Repository: close the pack stream before any other request on the location ... Review notes on the open pack stream: it stayed open while a backup run went on to upload (e.g. the snapshot after reading the file list back), which SFTP cannot do on one connection. Reading moves to PackReader, and Repository reaches the location through PackReaderGuard, which closes the stream before every other request. A pack stream that cannot seek is read up to the offset. Deep verify no longer reports an \Error during decoding as damage.	2026-10-08 22:34:02 +02:00
dillard	5f8d5faee7	Repository: read the blobs of a pack from one open stream ... Restores and deep verifies requested every blob separately: a new file handle and a seek on the location each time. They read the blobs of a snapshot in the order they were written, so the stream of the pack read last now stays open and is read on; a jump backwards, to another pack, or more than 8 MiB ahead opens the pack again at the offset, as before. On a Hetzner Storage Box over SMB, restoring 86 MB in 202 files went from 0.78 MB/s to 4.2 MB/s.	2026-10-08 22:17:53 +02:00
Dillard Blom	4fa7cbf6ef	Merge pull request #7: Release 0.9.3-beta.4 ... Release 0.9.3-beta.4 v0.9.3-beta.4	2026-10-08 21:18:00 +02:00
dillard	f4abfe0a8d	Release 0.9.3-beta.4	2026-10-08 20:46:14 +02:00
dillard	fe1a289c2d	Backend: an existing folder that cannot be listed is an error, not empty ... Read as empty, a catalog/ or index/ folder would make an existing repository look new.	2026-10-08 20:46:14 +02:00
dillard	ab3265e42c	Backup: release the held index cache writes also when the dump transaction fails to start ... holdIndexCache() ran before beginTransaction(), outside the try, and the release came after rollBack(); an exception from either left later cache writes queued for the rest of the run.	2026-10-08 20:38:25 +02:00
dillard	466cff8857	CHANGELOG: fixes for the next hotfix	2026-10-08 18:28:29 +02:00
dillard	6bba0b59fc	Repository: list flat folders without a type check per entry ... Opening a repository lists index/, which holds one file per pack. StorageBackend::list() checked every entry with is_dir(), and on a remote storage each check is a round trip. Over SFTP to a Hetzner Storage Box (33 ms) with a 60 GiB repository, occ backup:run (30 s steps) spent nearly every step on these requests and uploaded 45 KiB/s; with this change it uploads 5.8 MB/s. The cost grows with the number of packs. index/, snapshots/, catalog/ and trash/info/ only ever hold objects: list them with the new IBackend::listFiles(), one listing without per-entry checks.	2026-10-08 18:28:29 +02:00
dillard	62766e1e71	Backup: do not write the index cache inside the read-only dump transaction ... A database dump that fills a whole pack (32 MiB) uploads that pack while the dump transaction is still open, and the pack's index cache row was inserted on the same connection. PostgreSQL refuses that (25006, read-only transaction) and the run failed; on other databases the row was rolled back with the dump transaction. The index now holds those cache writes back until the transaction is over.	2026-10-08 13:03:52 +02:00
Dillard Blom	bfd54447b0	Merge pull request #6: Release 0.9.2-beta.3 v0.9.2-beta.3	2026-10-08 01:28:52 +02:00
dillard	0b50bb6b81	Verify: retry storage exceptions only, let programming errors surface ... Catch \Exception (storage adapters throw their own exception types) instead of \Throwable, so an \Error such as a TypeError is not retried and reported as an unreadable blob.	2026-10-08 01:21:15 +02:00
dillard	f415bab519	Verify, README: address review notes ... An unreadable blob may still be a truncated pack, so don't rule out damage in the verify message; README: Hetzner needs WebDAV/SMB and external reachability enabled for the sub-account.	2026-10-08 01:11:21 +02:00
dillard	27eb0b32fc	Release 0.9.2-beta.3	2026-10-08 01:00:52 +02:00
dillard	b1d4f6a469	README: Hetzner Storage Box over WebDAV and SMB ... Tested both on sub-accounts: works end to end; WebDAV needs an existing root, SMB needs port 445 and the smbclient extension; fixed cost per run and slow deep verify over remote storage.	2026-10-08 01:00:46 +02:00
dillard	fe7ad9b4b5	Verify: report read errors as unreadable, not as corrupted ... verifyBlob() caught every exception from loading a blob and counted it as a failed (corrupted or tampered) blob, so a transient network error during a deep verify over WebDAV looked like data corruption. Read the range separately, retry it, and report a read that keeps failing as unreadable with its error; only a blob that was read but does not decrypt or match its id counts as failed.	2026-10-08 00:53:31 +02:00
dillard	695d972785	Backups: let the runs table scroll on narrow screens ... The table could not shrink below its content width and ran off the page, cutting off the last columns; wrap it in an overflow-x container.	2026-10-08 00:47:47 +02:00
dillard	972a52e665	Settings: align the tabs with the sections ... The tabs had no inline margin, so they sat against the edge while every NcSettingsSection is inset by 7 grid baselines.	2026-10-08 00:47:47 +02:00
dillard	7c08a10621	Locations: ask for password confirmation before adding ... addTarget requires a recent password confirmation, but the form never asked for one. Within 30 minutes of logging in that went unnoticed; after that, adding a location failed with a bare 403.	2026-10-08 00:40:48 +02:00
Dillard Blom	209520165e	Merge pull request #5: Release 0.9.1-beta.2 v0.9.1-beta.2	2026-10-07 18:09:23 +02:00
dillard	9554e774e5	CHANGELOG: correct why info.xml carries no -beta suffix	2026-10-07 18:01:20 +02:00
dillard	8b2dca28e3	Release 0.9.1-beta.2	2026-10-07 17:47:40 +02:00
dillard	5742f77253	App store listing: screenshots, links and UI fixes ... Add three screenshots (backups, restore, settings) taken on a demo instance, plus website, repository and the HOWTO as admin documentation in info.xml. Fixes found while taking them: - the "new folder" restore option lost "<date>" (l10n sanitizes it as a tag); show the folder name pattern via a placeholder instead - the restore-point dropdown cut off the time - passphrase actions stacked vertically in the slot table - HOWTO: SFTP example used "sftp::password", the auth mechanism is "password::password"	2026-10-07 17:43:41 +02:00
Dillard Blom	550249570b	Merge pull request #4: README key rotation note ... README: key rotation for revoking old kits is on the backlog	2026-10-07 02:26:19 +02:00
dillard	0eb23f56e7	README: note that key rotation for revoking old kits is on the backlog	2026-10-07 02:18:45 +02:00
Dillard Blom	a0c1f1cd58	Merge pull request #3: Release 0.9.0-beta.1 ... Release 0.9.0-beta.1 v0.9.0-beta.1	2026-10-07 01:15:19 +02:00
dillard	26cc9a897e	Release 0.9.0-beta.1 ... Version 0.9.0 in info.xml, the app description for the beta, and the CHANGELOG entry for 0.9.0-beta.1.	2026-10-07 01:07:55 +02:00
Dillard Blom	fe0385ad8c	Merge pull request #2: Pre-beta review ... Pre-beta review: data safety, safer commands, known limitations	2026-10-07 01:06:40 +02:00
dillard	b37a953cfd	Detect a busy run by identity, and clean up a failed kit write ... step() hands back the same Run object when another process holds the lease; comparing update times per second could stop the scheduled job early after a quick step. A failed write of the recovery kit file now closes the handle and removes the partial file.	2026-10-07 00:48:11 +02:00
dillard	b89b5b5fa6	Document known limitations for the beta and record the review backlog ... The open points from the pre-beta review are on the backlog in PLAN.md and listed as known limitations in the README; none block 0.9.0-beta or 1.0.0. The how-to warns against pointing a test server at a live location and explains that removing a slot does not revoke access.	2026-10-07 00:19:46 +02:00
dillard	16e09f4fde	Fix trash on SFTP and the disaster recovery test after the review fixes ... forget() and untrash() uploaded a remote read stream directly, which the new size check could not judge on SFTP, so cleanup failed there. They now copy through a local buffer, and the size check only trusts local streams (php://temp, php://memory, plain files). The disaster recovery test calls backup:restore:full with --no-interaction: docker exec without a terminal still counts as interactive, so the new confirmation question read an empty answer and cancelled.	2026-10-06 23:59:17 +02:00
dillard	45069994e9	Pre-beta review: safer destructive commands and hardening ... - backup:restore:full refuses on an installation with more than one user and asks before it starts (--force skips both, also to rerun after a failure halfway), and warns when the snapshot's database dump was not one consistent transaction. - backup:user:restore --mode=replace asks before deleting an existing account; non-interactive use needs --force. - The recovery kit file is created owner-only and never over an existing file or symlink; backup:key:import shows the deletion delay it read from the kit. - Adding a location through the web API needs password confirmation, like removing one. - Key derivation parameters from a kit or repository config are bounded, and a stalled network stream no longer spins at full CPU. - Floating-point columns are dumped exactly instead of with 14 digits.	2026-10-06 23:48:41 +02:00
dillard	0f4a238414	Pre-beta review: data safety in storage, catalog and backup runs ... Storage and repository: - Listing skipped every name containing ".part-", directories too, so the exports of a user like "john.part-time" were invisible to prune on S3, SFTP, FTP and SMB and their data was deleted. Only real temporary uploads ("<name>.part-<8 hex>") are skipped now. - prune refuses to run while a recorded per-user export's manifest is missing, instead of freeing that export's data. - forget() and untrash() copy first, switch the catalog, and delete the originals last, so a crash can no longer leave the catalog pointing at a missing snapshot or database dump. - Writes are checked against the expected size, so a short upload is an error instead of a truncated pack; raw restores check rename() and touch(). - A catalog write refuses to overwrite a generation another process just wrote, and the catalog lock lasts 300 s, so opening a large repository inside it no longer lets the lease run out. Backup runs and scheduling: - A file that keeps changing is read again at most three times; then the last read is kept and reported, instead of restarting forever and blocking cleanup. - Unreadable files and folders are recorded, logged, shown by occ backup:run, stored in the snapshot and notified to admins, instead of being skipped silently. - After a failed scheduled backup, at most three attempts a day, an hour apart, instead of a new run and notification every five minutes. - The job no longer spins on a run whose lease another process holds, and the weekly cleanup catches up after a Sunday spent backing up. - Notifications for the per-user export retention events.	2026-10-06 23:48:40 +02:00
Dillard Blom	f39ca71dab	Merge pull request #1: Phase 2 ... Phase 2: disaster recovery, per-user backup, verify, FTP/SMB/NFS	2026-10-06 23:21:55 +02:00
dillard	e8ecb77df5	Refresh the location lease while a per-user restore reads its export ... A long import could outlive the lease, letting a cleanup forget the export being restored and free its blobs mid-read.	2026-10-06 22:59:49 +02:00
dillard	70965dd4bd	Keep leases alive during long restores and exports; PR review fixes ... Full restore, raw directory restore, per-user export and the safety export held a fixed-TTL lease on the location that nothing refreshed. A restore or export running longer than that TTL let a cleanup take over and delete data still in use. They now refresh the lease while they work (per blob, per row batch, per written chunk), throttled to once a minute. Also: - backup:prune still applies snapshot retention when user-export retention fails for a location. - The recovery kit carries the deletion delay, so a disaster recovery keeps the original protection instead of falling back to 7 days. Older kits still import with the default. - CI checkouts no longer persist the token in the git config. - recoveryKit() has its real return type; the stale baseline entry is gone.	2026-10-06 22:38:47 +02:00
dillard	cc09d74942	Commit full-backup snapshots to the catalog under the catalog lock ... A full backup opened its repository at the start of a step and wrote the catalog at the end of it, without the catalog lock. A per-user export or restore that wrote the catalog in between could lose its entry, or the snapshot could lose its own. The final step now only stores the snapshot object and moves the run to a new 'commit' phase. BackupService then records the snapshot under the catalog lock, from a repository opened inside that lock; if the lock is busy, the next step retries the commit. Repository::writeSnapshot() keeps its old behaviour as putSnapshot() plus recordSnapshot().	2026-10-06 17:58:21 +02:00
dillard	eb9100a74d	Fix review findings: catalog writes, user restore consent, retention validation ... Catalog: forgetting a user export or purging trash now writes the catalog before deleting the objects, so a crash leaves at most an unreferenced file instead of a catalog entry pointing at nothing. Per-user backup and the safety export reopen the repository inside the catalog lock, so they build on the latest catalog generation. Commands: backup:user:backup and backup:user:restore resolve the location before asking to install user_migration, and a successful install always stops with a rerun instruction. backup:retention rejects a malformed --user-last before changing any policy. backup:key:import fails clearly when the passphrase file cannot be read.	2026-10-06 17:40:51 +02:00
dillard	614caf9f63	Document migrated- calendar names in the how-to; add backlog items	2026-10-06 16:59:45 +02:00
dillard	943c95bf07	Ask before installing user_migration; add the beta how-to ... backup:user:backup and backup:user:restore now ask the administrator before installing or enabling user_migration (or take --install-user-migration), so nothing changes on the server without consent. The restore warns that user_migration imports only its allowlisted app settings. Also: a clear error for unknown locations instead of a raw database message; prune reports the number of affected users, not the number of kept ones; and --user-last rejects values that are not whole numbers. Adds docs/HOWTO.md and updates the README and PLAN.md for the beta.	2026-10-06 16:57:56 +02:00
dillard	ba7f70e0f8	Rename per-user backup/restore commands to backup:user:backup and backup:user:restore	2026-10-06 03:51:35 +02:00
dillard	d70a869070	Phase 2: verify per-user exports, refuse SMB without the native extension, matrix user CI ... backup:verify accepts a user export path (users/<uid>/<export>): every blob its manifest references is checked the same way as for a snapshot, shallow or --deep. The SMB backend is refused at backup:target:add when the native smbclient PECL extension is not loaded, instead of registering and then failing the first backup with "Seek failed": with only the smbclient binary files_external falls back to a shell stream that cannot seek. The per-user integration job now runs on Nextcloud 32-35, not just 35: it calls user_migration's internal UserMigrationService, whose API may change between versions.	2026-10-06 03:47:02 +02:00
dillard	c9f82a79e4	Phase 2: retention for per-user exports (occ backup:retention --user-last) ... PruneService::userExportRetention()/setUserExportRetention(): a single, app-wide "keep the newest N" setting (default 3, same default as the snapshot policy's "last"), deliberately simpler than the full daily/weekly/monthly RetentionPolicy -- per-user exports are on-demand only (occ backup:backup:user, see the fold-out task about whether they ever join the automatic schedule), so there is no regular cadence yet to bucket by day/week/month. PruneService::applyUserExports(): groups a location's user exports by uid (read from each manifest's own 'time' field, the same field snapshot retention already reads), keeps the newest N per user, forgets the rest. No trash, unlike a snapshot: these are not yet the primary safety net the deletion delay protects. Runs under the same exclusive lock as snapshot retention. occ backup:retention gained --user-last=N; occ backup:prune now also applies it for every target it processes, reporting counts the same way it already does for snapshots and blob packs. Verified live against the dev rig (four real exports across four users, confirmed exactly the expected ones were kept/forgotten at two different --user-last values) and added coverage to tests/integration/user-roundtrip.sh (an account's manual export plus the automatic safety-export a replace-mode restore makes of it, pruned down to --user-last=2).	2026-10-06 03:18:14 +02:00
dillard	932f60bc5e	Phase 2: FTP, SMB and NFS storage backends ... No ng_backup code changes needed: files_external's generic backend/auth adapter (the same ExternalStorageFactory/StorageBackend already used for S3/SFTP) already covers FTP and SMB, and NFS was never a files_external backend to begin with (PLAN.md 3.5) -- the "local" backend plus an admin-managed OS-level mount already does the job. This commit is test coverage and documentation. tests/integration/targets-matrix.sh now also runs the full round trip against: - FTP (stilliard/pure-ftpd). A first attempt with delfer/alpine-ftp-server (vsftpd) failed with a generic "create directory operation failed": files_external's FTP backend always builds absolute paths, which that server's chroot rejects even via plain php ftp_mkdir() with an absolute path (a relative path on the same server works fine) -- not a files_external or ng_backup bug, just a server-specific quirk, confirmed by switching test servers. - SMB (dperson/samba), via a small Nextcloud image built on the fly with the smbclient PECL extension (libsmbclient-dev + pecl install), in its own temp directory as the build context (never /tmp directly). The backend *lists* as available with just the smbclient CLI binary too (files_external's shell-wrapped fallback), but that fallback's stream cannot seek, and ng_backup reads blobs back by byte range while backing up -- confirmed live, it fails the very first (small) backup with "Seek failed in packs/...". The native extension is genuinely required here, not just preferred. README: new "Storage backend experiences" entries for both findings (and a pointer from the "no extra PHP extensions" intro line, since SMB is the one real exception), plus NFS guidance (mount it yourself, point the local backend at the mount point). PLAN.md: FTP/SMB marked done; SMB/NFS testing on the user's own OMV instance stays open (internal, not something CI can cover). Local review (Codex + CodeRabbit) found three issues, all fixed: the README's first-pass SMB claim ("CLI tool is not enough") was imprecise about *why* -- it lists fine without the extension, it just doesn't work for ng_backup's read pattern; the cleanup trap could abort before removing the network if an earlier docker rm failed, and didn't remove the built image/Dockerfile; and the Docker build used all of /tmp as its context instead of a dedicated directory.	2026-10-06 00:31:29 +02:00
dillard	935c7b6a08	Phase 2: checksum audit (occ backup:verify) ... Repository::verify(): every blob a snapshot's files and database dump reference must be in a freshly re-downloaded, re-authenticated index (never trusted from the local cache, same reasoning as prune()) and have its pack still present on the location -- shallow, cheap, no blob content read, catches a deleted pack, a tampered/rolled-back index, or a database manifest that went missing (reported as a failure, not silently skipped). With $deep, each blob is also downloaded and decrypted, same as loadBlob() already does on every real read (AEAD authentication tag, content hash against its blob id), done here deliberately for everything in the snapshot rather than waiting for it to come up during an actual restore. A blob referenced by more than one file is only checked once; pack presence is cached per pack, not re-checked per blob. occ backup:verify <target> <snapshot> [--deep], meant to run periodically (cron, a sample of recent snapshots), per PLAN.md's "Verificatie: periodiek checksums controleren, steekproef-restore". The lease covering the audit is refreshed roughly every half of its TTL while blobs are checked, rather than relying on a single long-lived lease like the other new phase-2 commands do. Verified against a real S3-backed 1.4 GiB / 603-blob snapshot (shallow: instant; --deep: ~28s, fully decrypted) in addition to the new unit tests and the roundtrip.sh addition (shallow and deep on a snapshot with real, varied file sizes, now covered on every CI matrix combination). Local review (Codex) on the first pass found three real issues, all fixed: a missing database manifest was silently skipped instead of reported; shallow verification trusted the (possibly stale or unauthenticated) local index cache instead of re-checking the authenticated index on the location; and the lease had no refresh, so a --deep audit of a very large snapshot could outrun its TTL and run unprotected against a concurrent prune.	2026-10-05 23:56:16 +02:00