Fast S3 verify and snapshot commit; shorter runs list; durations with days

On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.

backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
This commit is contained in:
2026-10-14 17:12:41 +02:00
parent 9a3bb99d12
commit ca003c88ba
20 changed files with 424 additions and 30 deletions
+11 -1
View File
@@ -7,7 +7,7 @@ declare(strict_types=1);
namespace OCA\NgBackup\Backend;
/** IBackend::getMany() for a backend where reading files one after the other costs nothing extra. */
/** IBackend::getMany() and getRanges() for a backend where reading one after the other costs nothing extra. */
trait GetsOneByOne {
/**
* @param list<string> $paths
@@ -27,4 +27,14 @@ trait GetsOneByOne {
yield $path => $data;
}
}
/**
* @param list<array{0:string, 1:int, 2:int}> $ranges
* @return \Generator<int, string>
*/
public function getRanges(array $ranges): \Generator {
foreach ($ranges as $i => [$path, $offset, $length]) {
yield $i => $this->getRange($path, $offset, $length);
}
}
}