A restore read one pack at a time over one SFTP connection, which over a long
line is bound by the round trip and by what one TCP connection gets: about
2-3 MiB/s from a Hetzner Storage Box to Canada, with PHP mostly waiting.
Repository::restore() now first lists the ranges of packs the restore needs,
in order (ReadAheadPlan), and the location reads them ahead into unlinked
files in a spool folder in the target (IReadsAhead). Over SFTP,
SftpRangeFetcher does that on four connections of its own, 16 requests in
flight each, taking the answers of whichever connection has one. A range that
fails, a damaged copy, or a blob that moved since the plan is read the usual
way, which reports what is wrong. Other locations restore as before.
On a test line with 120 ms RTT and 0.2% loss, a 1.6 GiB restore went from
5.1 to 17.8 MiB/s, all files identical.
On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.
backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
Review notes on the open pack stream: it stayed open while a backup run went on to upload
(e.g. the snapshot after reading the file list back), which SFTP cannot do on one connection.
Reading moves to PackReader, and Repository reaches the location through PackReaderGuard,
which closes the stream before every other request. A pack stream that cannot seek is read up
to the offset. Deep verify no longer reports an \Error during decoding as damage.
dillard