files_external's SFTP read stream asks for the next 256 KiB only after
the previous answer has arrived, so reading costs one round trip per
request whatever the line speed: 4.9 MB/s at 47 ms to a Hetzner Storage
Box. SftpReadStream keeps up to eight READ requests in flight over the
same phpseclib connection (15 MB/s on that link), starting with one so
a small read does not fetch megabytes ahead.
A server that answers less than asked is handled by asking again from
where the answer ended; when more data follows, that size is used for
later reads over the same connection, never below the 32 KiB every
server must support. Without the needed connection methods, or when the
open fails, the storage's own stream is used as before.
Review notes on the open pack stream: it stayed open while a backup run went on to upload
(e.g. the snapshot after reading the file list back), which SFTP cannot do on one connection.
Reading moves to PackReader, and Repository reaches the location through PackReaderGuard,
which closes the stream before every other request. A pack stream that cannot seek is read up
to the offset. Deep verify no longer reports an \Error during decoding as damage.
Restores and deep verifies requested every blob separately: a new file handle and a seek on
the location each time. They read the blobs of a snapshot in the order they were written, so
the stream of the pack read last now stays open and is read on; a jump backwards, to another
pack, or more than 8 MiB ahead opens the pack again at the offset, as before. On a Hetzner
Storage Box over SMB, restoring 86 MB in 202 files went from 0.78 MB/s to 4.2 MB/s.
holdIndexCache() ran before beginTransaction(), outside the try, and the release came after
rollBack(); an exception from either left later cache writes queued for the rest of the run.
Opening a repository lists index/, which holds one file per pack.
StorageBackend::list() checked every entry with is_dir(), and on a
remote storage each check is a round trip. Over SFTP to a Hetzner Storage
Box (33 ms) with a 60 GiB repository, occ backup:run (30 s steps) spent
nearly every step on these requests and uploaded 45 KiB/s; with this
change it uploads 5.8 MB/s. The cost grows with the number of packs.
index/, snapshots/, catalog/ and trash/info/ only ever hold objects:
list them with the new IBackend::listFiles(), one listing without
per-entry checks.
A run only showed "running (files)", which says nothing during a backup
of a terabyte. The files phase now records an expected size on its first
step: the previous snapshot's size, or for a first backup the users'
files, versions and trash from Nextcloud's file cache (approximate, shown
with a ~). The web interface and occ backup:status show the share of it
done so far, including a large file that is still being read; it stays
below 100% until the run finishes.
A database dump that fills a whole pack (32 MiB) uploads that pack while
the dump transaction is still open, and the pack's index cache row was
inserted on the same connection. PostgreSQL refuses that (25006, read-only
transaction) and the run failed; on other databases the row was rolled
back with the dump transaction. The index now holds those cache writes
back until the transaction is over.
A changed password or host name could only be handled by adding a new
location, which starts an empty repository and, with a retention lock at
the provider, keeps the old one billed. Edit now changes a location in
place (web interface and occ backup:target:edit). The settings are only
saved when they lead to the same repository; a failed connection and a
missing repository give separate errors, and the folder is not created
while checking. Stored passwords are never sent to the browser: an empty
password field keeps the stored value.
Catch \Exception (storage adapters throw their own exception types) instead
of \Throwable, so an \Error such as a TypeError is not retried and reported
as an unreadable blob.
An unreadable blob may still be a truncated pack, so don't rule out damage
in the verify message; README: Hetzner needs WebDAV/SMB and external
reachability enabled for the sub-account.
Tested both on sub-accounts: works end to end; WebDAV needs an existing
root, SMB needs port 445 and the smbclient extension; fixed cost per run
and slow deep verify over remote storage.
verifyBlob() caught every exception from loading a blob and counted it as
a failed (corrupted or tampered) blob, so a transient network error during
a deep verify over WebDAV looked like data corruption. Read the range
separately, retry it, and report a read that keeps failing as unreadable
with its error; only a blob that was read but does not decrypt or match
its id counts as failed.
addTarget requires a recent password confirmation, but the form never
asked for one. Within 30 minutes of logging in that went unnoticed; after
that, adding a location failed with a bare 403.
Add three screenshots (backups, restore, settings) taken on a demo
instance, plus website, repository and the HOWTO as admin documentation
in info.xml.
Fixes found while taking them:
- the "new folder" restore option lost "<date>" (l10n sanitizes it as
a tag); show the folder name pattern via a placeholder instead
- the restore-point dropdown cut off the time
- passphrase actions stacked vertically in the slot table
- HOWTO: SFTP example used "sftp::password", the auth mechanism is
"password::password"
step() hands back the same Run object when another process holds the lease;
comparing update times per second could stop the scheduled job early after a
quick step. A failed write of the recovery kit file now closes the handle and
removes the partial file.
The open points from the pre-beta review are on the backlog in PLAN.md and
listed as known limitations in the README; none block 0.9.0-beta or 1.0.0.
The how-to warns against pointing a test server at a live location and
explains that removing a slot does not revoke access.
forget() and untrash() uploaded a remote read stream directly, which the
new size check could not judge on SFTP, so cleanup failed there. They now
copy through a local buffer, and the size check only trusts local streams
(php://temp, php://memory, plain files).
The disaster recovery test calls backup:restore:full with --no-interaction:
docker exec without a terminal still counts as interactive, so the new
confirmation question read an empty answer and cancelled.
- backup:restore:full refuses on an installation with more than one user
and asks before it starts (--force skips both, also to rerun after a
failure halfway), and warns when the snapshot's database dump was not
one consistent transaction.
- backup:user:restore --mode=replace asks before deleting an existing
account; non-interactive use needs --force.
- The recovery kit file is created owner-only and never over an existing
file or symlink; backup:key:import shows the deletion delay it read from
the kit.
- Adding a location through the web API needs password confirmation, like
removing one.
- Key derivation parameters from a kit or repository config are bounded,
and a stalled network stream no longer spins at full CPU.
- Floating-point columns are dumped exactly instead of with 14 digits.
Storage and repository:
- Listing skipped every name containing ".part-", directories too, so the
exports of a user like "john.part-time" were invisible to prune on S3,
SFTP, FTP and SMB and their data was deleted. Only real temporary uploads
("<name>.part-<8 hex>") are skipped now.
- prune refuses to run while a recorded per-user export's manifest is
missing, instead of freeing that export's data.
- forget() and untrash() copy first, switch the catalog, and delete the
originals last, so a crash can no longer leave the catalog pointing at a
missing snapshot or database dump.
- Writes are checked against the expected size, so a short upload is an
error instead of a truncated pack; raw restores check rename() and touch().
- A catalog write refuses to overwrite a generation another process just
wrote, and the catalog lock lasts 300 s, so opening a large repository
inside it no longer lets the lease run out.
Backup runs and scheduling:
- A file that keeps changing is read again at most three times; then the
last read is kept and reported, instead of restarting forever and
blocking cleanup.
- Unreadable files and folders are recorded, logged, shown by occ
backup:run, stored in the snapshot and notified to admins, instead of
being skipped silently.
- After a failed scheduled backup, at most three attempts a day, an hour
apart, instead of a new run and notification every five minutes.
- The job no longer spins on a run whose lease another process holds, and
the weekly cleanup catches up after a Sunday spent backing up.
- Notifications for the per-user export retention events.
Full restore, raw directory restore, per-user export and the safety export
held a fixed-TTL lease on the location that nothing refreshed. A restore or
export running longer than that TTL let a cleanup take over and delete data
still in use. They now refresh the lease while they work (per blob, per row
batch, per written chunk), throttled to once a minute.
Also:
- backup:prune still applies snapshot retention when user-export retention
fails for a location.
- The recovery kit carries the deletion delay, so a disaster recovery keeps
the original protection instead of falling back to 7 days. Older kits
still import with the default.
- CI checkouts no longer persist the token in the git config.
- recoveryKit() has its real return type; the stale baseline entry is gone.
A full backup opened its repository at the start of a step and wrote the
catalog at the end of it, without the catalog lock. A per-user export or
restore that wrote the catalog in between could lose its entry, or the
snapshot could lose its own.
The final step now only stores the snapshot object and moves the run to a
new 'commit' phase. BackupService then records the snapshot under the
catalog lock, from a repository opened inside that lock; if the lock is
busy, the next step retries the commit. Repository::writeSnapshot() keeps
its old behaviour as putSnapshot() plus recordSnapshot().
Catalog: forgetting a user export or purging trash now writes the catalog
before deleting the objects, so a crash leaves at most an unreferenced file
instead of a catalog entry pointing at nothing. Per-user backup and the
safety export reopen the repository inside the catalog lock, so they build
on the latest catalog generation.
Commands: backup:user:backup and backup:user:restore resolve the location
before asking to install user_migration, and a successful install always
stops with a rerun instruction. backup:retention rejects a malformed
--user-last before changing any policy. backup:key:import fails clearly
when the passphrase file cannot be read.
dillard
Dillard Blom