Fast S3 verify and snapshot commit; shorter runs list; durations with days
On S3, index files and the tree segments read back when a snapshot is written are fetched 16 at a time through the storage's S3 client (IBackend::getRanges(), getMany()). Blobs that follow each other in a pack are read as one range on every location. The snapshot is written in a step of its own and the run shows "commit" meanwhile. On a 1 TB production repository the check went from about 4 hours to 3.5 minutes. backup:status and the web interface list the runs of the last 7 days, running ones and at least the newest 5; --all / "Show older runs" list all. Durations of a day or more show the days.
This commit is contained in:
@@ -35,6 +35,17 @@ interface IBackend {
|
||||
/** @return string bytes [$offset, $offset + $length) of the object (used to fetch single blobs from a pack) */
|
||||
public function getRange(string $path, int $offset, int $length): string;
|
||||
|
||||
/**
|
||||
* Many byte ranges, in the order given (blobs spread over many packs, such as the tree
|
||||
* segments of a backup). A backend where every request costs a round trip can fetch them
|
||||
* several at a time.
|
||||
*
|
||||
* @param list<array{0:string, 1:int, 2:int}> $ranges [path, offset, length] each
|
||||
* @return \Generator<int, string> contents by position in $ranges
|
||||
* @throws BackendException for a range that cannot be read
|
||||
*/
|
||||
public function getRanges(array $ranges): \Generator;
|
||||
|
||||
public function exists(string $path): bool;
|
||||
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user