files_external's SFTP read stream asks for the next 256 KiB only after
the previous answer has arrived, so reading costs one round trip per
request whatever the line speed: 4.9 MB/s at 47 ms to a Hetzner Storage
Box. SftpReadStream keeps up to eight READ requests in flight over the
same phpseclib connection (15 MB/s on that link), starting with one so
a small read does not fetch megabytes ahead.
A server that answers less than asked is handled by asking again from
where the answer ended; when more data follows, that size is used for
later reads over the same connection, never below the 32 KiB every
server must support. Without the needed connection methods, or when the
open fails, the storage's own stream is used as before.
Opening a repository lists index/, which holds one file per pack.
StorageBackend::list() checked every entry with is_dir(), and on a
remote storage each check is a round trip. Over SFTP to a Hetzner Storage
Box (33 ms) with a 60 GiB repository, occ backup:run (30 s steps) spent
nearly every step on these requests and uploaded 45 KiB/s; with this
change it uploads 5.8 MB/s. The cost grows with the number of packs.
index/, snapshots/, catalog/ and trash/info/ only ever hold objects:
list them with the new IBackend::listFiles(), one listing without
per-entry checks.
A changed password or host name could only be handled by adding a new
location, which starts an empty repository and, with a retention lock at
the provider, keeps the old one billed. Edit now changes a location in
place (web interface and occ backup:target:edit). The settings are only
saved when they lead to the same repository; a failed connection and a
missing repository give separate errors, and the folder is not created
while checking. Stored passwords are never sent to the browser: an empty
password field keeps the stored value.
forget() and untrash() uploaded a remote read stream directly, which the
new size check could not judge on SFTP, so cleanup failed there. They now
copy through a local buffer, and the size check only trusts local streams
(php://temp, php://memory, plain files).
The disaster recovery test calls backup:restore:full with --no-interaction:
docker exec without a terminal still counts as interactive, so the new
confirmation question read an empty answer and cancelled.
Storage and repository:
- Listing skipped every name containing ".part-", directories too, so the
exports of a user like "john.part-time" were invisible to prune on S3,
SFTP, FTP and SMB and their data was deleted. Only real temporary uploads
("<name>.part-<8 hex>") are skipped now.
- prune refuses to run while a recorded per-user export's manifest is
missing, instead of freeing that export's data.
- forget() and untrash() copy first, switch the catalog, and delete the
originals last, so a crash can no longer leave the catalog pointing at a
missing snapshot or database dump.
- Writes are checked against the expected size, so a short upload is an
error instead of a truncated pack; raw restores check rename() and touch().
- A catalog write refuses to overwrite a generation another process just
wrote, and the catalog lock lasts 300 s, so opening a large repository
inside it no longer lets the lease run out.
Backup runs and scheduling:
- A file that keeps changing is read again at most three times; then the
last read is kept and reported, instead of restarting forever and
blocking cleanup.
- Unreadable files and folders are recorded, logged, shown by occ
backup:run, stored in the snapshot and notified to admins, instead of
being skipped silently.
- After a failed scheduled backup, at most three attempts a day, an hour
apart, instead of a new run and notification every five minutes.
- The job no longer spins on a run whose lease another process holds, and
the weekly cleanup catches up after a Sunday spent backing up.
- Notifications for the per-user export retention events.
RepositoryImportSource::copyToFolder() called putContent() on the generic
return type of Node::get(), which doesn't declare that method -- only
File does. Existing-file overwrite during a per-user restore would have
fatally errored at runtime; added an instanceof check and a resource leak
fix (finally-close) while there. Also guarded getFileAsStream() against
fopen() returning false.
Everything else Psalm (strict level) currently flags is mechanical
(missing `final` on leaf classes -- applied everywhere safe) or pre-existing
array-shape/type gaps, captured in psalm-baseline.xml so CI fails only on
new issues from here on, not the whole backlog at once.
Added tests/integration/targets-matrix.sh, which runs the full backup/restore
round trip through files_external's AmazonS3 and SFTP backend classes (S3
via a disposable s3mock container, SFTP via atmoz/sftp), since the existing
matrix only ever exercised the local backend.
This caught a real bug: moving a snapshot to the trash read it from the
location and streamed it straight back into a write on the same connection,
which SFTP can't do reliably (phpseclib fails to interleave a read and a
write on one connection). Backends now expose a native move() and the
id-bound snapshot object is moved with that instead of a read+write copy;
the path-bound database dump still goes through decrypt/re-encrypt, since
its AEAD authentication is tied to its path.
- AdminApiController (admin only, CSRF-protected): status, create key,
download recovery kit, confirm (statement accepted + exact phrase),
backends with their usable auth mechanisms, add/test/remove locations,
start backups, schedule, snapshots, browse, queue restores
- RestoreJob runs web-initiated restores in the background; backup and
restore runs are kept apart in the run table
- admin section 'NG Backup' with a Vue 3 / @nextcloud/vue 9 page: key and
recovery-kit flow (download, tick statement, 'Yes, I confirm'), location
form built from files_external's parameter definitions, schedule, run
list with polling, snapshot browser and restore with the three modes
- auth mechanisms that depend on a user session or on files_external's
own stored credentials are not offered
API tested with curl on NC34 (status, backends, schedule validation,
confirmation refusal, browse, queued restore and backup completed by
cron); non-admins get 403, anonymous 401.
- detect files changing during reading or between resumable steps and
re-read them (size/mtime/ctime/inode), spike F overwrites mid-run
- drop pending blob ids when a pack or index upload fails
- dump and restore PostgreSQL sequence state exactly, bound parameters,
validated catalog names; spike C checks all sequences exactly
- keep invalid UTF-8 byte-exact (tagged base64)
- refuse large tables without a primary key
- StorageBackend: check write/flush/close results, validate base path
- user_migration import/export: root paths, intermediate folders,
partial writes, read errors
- spikes: never touch an existing account, own fixtures, 512M note
ExternalStorageFactory builds storages with files_external's own
backends and auth mechanisms from an unsaved StorageConfig (as its
ConfigAdapter does), and StorageBackend adapts any IStorage to IBackend
(writeStream, seekable range reads, temp name + rename except on S3).
IBackend::put() now owns and always closes the stream, because
files_external's S3 writeStream closes it.
Spike D (NC34): S3 (SeaweedFS) and SFTP: 120 MiB backup ~6s, unchanged
run uploads nothing, restore bit-identical, no files_external mount
created, no temp files (max 12 KiB in /tmp). Works with files_external
disabled when its classes are loaded. Backup peak memory 85-89 MiB on
these targets (to investigate).
Content-addressed repository on an IBackend (local implementation):
4 MiB blobs, deflate when it helps, each blob encrypted with its id as
associated data, ~32 MiB packs with an encrypted header, one index file
per pack, snapshots as encrypted JSON lines written last.
Spike B: 390 MiB tree, peak 54 MiB; unchanged run reads nothing; a 1 MiB
change in a 300 MiB file uploads one blob; full and partial restore are
bit-identical; tampering is detected.
dillard