SFTP: replace a dropped connection; write the index cache in one transaction

A Hetzner Storage Box drops an SFTP connection that carried a lot of data
and then stayed quiet for a few minutes; files_external keeps using the dead
client, on which every file looks missing. StorageBackend now checks the
connection after a minute without requests, and replaces a connection that
is dead after a failed request (once per request; put, move and delete are
not repeated). SftpReadStream closes a connection it left in an unknown state.

The verify cache refresh took more than 6 minutes on PostgreSQL for 27,547
packs, one autocommitted insert per pack: putMany()/replaceAll() write in one
transaction. After a verify the cache holds the index files as they are, so a
blob in two packs stays cached for both.
This commit is contained in:
2026-10-12 01:58:54 +02:00
parent d659322e60
commit 9c06ab2841
9 changed files with 442 additions and 33 deletions
+12 -3
View File
@@ -49,9 +49,18 @@ instead carried by the app description, the git tag, the release, and this file.
no other process wrote it since the step loaded it (in one database statement), and recording
a snapshot that is already recorded is no longer an error.
- `occ backup:verify` on a large repository failed with "SFTP connection failed while opening
snapshots/…": after downloading the index it refreshed the local index cache with a pass over
all blobs for every pack, which took 29 minutes for 1 TB (27,547 packs, 400,000 blobs). The
location dropped the idle connection in the meantime. It is one pass now (under a second).
snapshots/…" or "Not found: snapshots/…". A Hetzner Storage Box drops an SFTP connection that
carried a lot of data (some 40 MB) and then stayed quiet for a few minutes, and the check went
quiet after downloading the index: it refreshed the local index cache, first in 29 minutes (a
pass over all blobs for every pack: 27,547 packs, 400,000 blobs for 1 TB), then still in more
than 6 minutes (a database transaction per pack on PostgreSQL). Nextcloud's SFTP storage keeps
using a dropped connection, on which every file looks missing. Now:
- an SFTP connection that was not used for a minute is checked first, and one that turns out
to be dead after a failed request is replaced, once per request; uploads, moves and deletes
are only checked first, never repeated;
- the index cache is written in one transaction (27,547 packs in 11 seconds on a test server, against about a minute row by row);
- after a check the cache holds the index files as they are: before, a blob that was in two
packs was only cached for one of them.
## [0.9.4-beta.5] - 2026-10-09