Fast S3 verify and snapshot commit; shorter runs list; durations with days
On S3, index files and the tree segments read back when a snapshot is written are fetched 16 at a time through the storage's S3 client (IBackend::getRanges(), getMany()). Blobs that follow each other in a pack are read as one range on every location. The snapshot is written in a step of its own and the run shows "commit" meanwhile. On a 1 TB production repository the check went from about 4 hours to 3.5 minutes. backup:status and the web interface list the runs of the last 7 days, running ones and at least the newest 5; --all / "Show older runs" list all. Durations of a day or more show the days.
This commit is contained in:
@@ -140,21 +140,21 @@ final class BackupRun {
|
||||
$state['parentPos'] = $parent->position();
|
||||
$state['stats'] = $stats;
|
||||
if ($finishedWalk) {
|
||||
// The snapshot is written in a step of its own, with its own time: after a long
|
||||
// run it reads back thousands of segments, and the run reports that it got here.
|
||||
$state['phase'] = 'finish';
|
||||
}
|
||||
return $state;
|
||||
}
|
||||
|
||||
if ($state['phase'] === 'finish') {
|
||||
$segments = $state['segments'];
|
||||
// Every segment ends with a newline, so they read as one stream: fetched together,
|
||||
// several at a time where the location allows it.
|
||||
$repo->putSnapshot($state['snapshot'],
|
||||
['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
|
||||
'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
|
||||
'warnings' => self::warnings($state)] + $state['meta'],
|
||||
(function () use ($repo, $segments) {
|
||||
foreach ($segments as $blobs) {
|
||||
yield from $repo->readLines($blobs);
|
||||
}
|
||||
})());
|
||||
$repo->readLines(array_values(array_merge(...$state['segments']))));
|
||||
$state['phase'] = 'commit';
|
||||
}
|
||||
return $state;
|
||||
|
||||
Reference in New Issue
Block a user