Full restore over SFTP: read packs ahead on four connections at once
A restore read one pack at a time over one SFTP connection, which over a long line is bound by the round trip and by what one TCP connection gets: about 2-3 MiB/s from a Hetzner Storage Box to Canada, with PHP mostly waiting. Repository::restore() now first lists the ranges of packs the restore needs, in order (ReadAheadPlan), and the location reads them ahead into unlinked files in a spool folder in the target (IReadsAhead). Over SFTP, SftpRangeFetcher does that on four connections of its own, 16 requests in flight each, taking the answers of whichever connection has one. A range that fails, a damaged copy, or a blob that moved since the plan is read the usual way, which reports what is wrong. Other locations restore as before. On a test line with 120 ms RTT and 0.2% loss, a 1.6 GiB restore went from 5.1 to 17.8 MiB/s, all files identical.
This commit is contained in:
@@ -466,6 +466,8 @@ final class FakeSftpConnection {
|
||||
public array $reads = [];
|
||||
/** @var array<string, string> */
|
||||
public array $openHandles = [];
|
||||
/** @var list<string> paths of every OPEN request */
|
||||
public array $opened = [];
|
||||
/** Whether the SFTP channel is open: phpseclib 2 opens it on the first SFTP call. */
|
||||
public bool $initialised = false;
|
||||
/** Packets sent */
|
||||
@@ -532,6 +534,7 @@ final class FakeSftpConnection {
|
||||
return false;
|
||||
}
|
||||
$path = substr($data, 4, unpack('N', $data)[1]);
|
||||
$this->opened[] = $path;
|
||||
if (!isset($this->files[$path])) {
|
||||
$this->answers[$id] = [101, pack('N', 2)];
|
||||
break;
|
||||
|
||||
Reference in New Issue
Block a user