SFTP: replace a dropped connection; write the index cache in one transaction

A Hetzner Storage Box drops an SFTP connection that carried a lot of data
and then stayed quiet for a few minutes; files_external keeps using the dead
client, on which every file looks missing. StorageBackend now checks the
connection after a minute without requests, and replaces a connection that
is dead after a failed request (once per request; put, move and delete are
not repeated). SftpReadStream closes a connection it left in an unknown state.

The verify cache refresh took more than 6 minutes on PostgreSQL for 27,547
packs, one autocommitted insert per pack: putMany()/replaceAll() write in one
transaction. After a verify the cache holds the index files as they are, so a
blob in two packs stays cached for both.
This commit is contained in:
2026-10-12 01:58:54 +02:00
parent d659322e60
commit 9c06ab2841
9 changed files with 442 additions and 33 deletions
+11 -5
View File
@@ -362,6 +362,8 @@ final class SftpReadStream {
self::$failures++;
self::$openBroke = true;
self::$openCause = $cause . self::connectionState($this->sftp);
// unknown state: closed, so that nothing else uses it (and the backend can build a new one)
self::disconnect($this->sftp);
return false;
}
@@ -514,6 +516,9 @@ final class SftpReadStream {
} catch (\Throwable) {
$this->broken = true;
}
if ($this->broken) {
self::disconnect($this->sftp);
}
}
$this->handle = '';
}
@@ -595,19 +600,20 @@ final class SftpReadStream {
private function drain(): void {
$inFlight = $this->inFlight;
$this->inFlight = [];
if ($this->broken) {
return;
}
try {
foreach ($inFlight as [$id]) {
foreach ($this->broken ? [] : $inFlight as [$id]) {
if (!is_string($this->sftp->_get_sftp_packet($id))) {
$this->broken = true;
return;
break;
}
}
} catch (\Throwable) {
$this->broken = true;
}
if ($this->broken) {
// unknown state: closed, so that nothing else uses it (and the backend can build a new one)
self::disconnect($this->sftp);
}
}
private static function chunk(object $sftp): int {