Fast S3 verify and snapshot commit; shorter runs list; durations with days

On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.

backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
This commit is contained in:
2026-10-14 17:12:41 +02:00
parent 9a3bb99d12
commit ca003c88ba
20 changed files with 424 additions and 30 deletions
+3
View File
@@ -225,6 +225,9 @@ final class BackupService {
} else {
$state['files']['estimate'] ??= $this->estimateBytes($run, $state['files']);
$state['files'] = BackupRun::step($repo, $state['files'], $deadline);
if (in_array($state['files']['phase'], ['finish', 'commit'], true)) {
$run->setPhase('commit'); // all files read: writing and recording the snapshot
}
if ($state['files']['phase'] === 'commit') {
// Catalog write under the catalog lock, from a repository opened inside it: a per-user
// export or restore may have written the catalog since this step opened $repo.
+1 -1
View File
@@ -64,7 +64,7 @@ final class VerifyService {
$age < 120 ? $age . ' s' : intdiv($age, 60) . ' min',
$age > 600 ? ' (no progress for a while: the process may have stopped)' : '');
}
$took = gmdate('H:i:s', max(0, (int)($s['finished'] ?? 0) - (int)($s['started'] ?? 0)));
$took = \OCA\NgBackup\Format::duration((int)($s['finished'] ?? 0) - (int)($s['started'] ?? 0));
return match ($s['status'] ?? '') {
'ok' => sprintf('%s OK on %s (took %s): %s files, %s blobs%s', $kind, date('Y-m-d H:i', (int)$s['finished']), $took,
number_format((int)($s['files'] ?? 0)), number_format((int)($s['blobs'] ?? 0)), $deep ? ', ' . $size((int)($s['bytes'] ?? 0)) . ' decrypted' : ''),