Over SMB, read index files in fewer round trips
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled

files_external's fopen() first stats the file and icewind's stream reads 1 MiB at a time. Open,
read and close straight through the native extension's state instead: from Canada to a Hetzner
Storage Box (130 ms) 0.66 instead of 1.05 s per index file, about 5 instead of 8 hours for the
27,000 index files of a 1 TB repository in backup:target:add. Anything unexpected falls back to
the usual way.
This commit is contained in:
2026-11-04 20:18:46 +01:00
parent 3ec23a3072
commit dce93aa75e
2 changed files with 80 additions and 21 deletions
+4
View File
@@ -34,6 +34,10 @@ shown under 1.0.0. The git tag of a release candidate goes after the date.
- Full restores from S3 read the packs ahead, eight ranges at a time, as restores over SFTP do
since 1.0.0-rc.4. Over SMB the packs are read in 8 MiB reads instead of 1 MiB: over a line with
a 130 ms round trip, reading was four times faster in a test (18 instead of 4.5 MiB/s).
- Over SMB the index files, which a new server reads in `backup:target:add`, take fewer round
trips: from Canada to a Hetzner Storage Box (130 ms),
0.66 instead of 1.05 s per file, so about 5 instead of 8 hours for the 27,000 index files of a
1 TB repository.
- `occ backup:target:add` only estimates the time left for reading the index once enough of it
is in. The first file also carries the time to connect, which made it say "about 642 min left"
for something done within a minute.