On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.
backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
Each 30-second step opened the location again and read its list of
index files. Over SMB that cost a request per file, so on a large
repository opening took longer than the step: each step stored one
4 MiB pack and the list grew further. A first backup of nc01 to a
Storage Box over SMB managed 12 GiB in 35 hours.
occ backup:run and system cron now keep the repository open for the
whole run (reopened when another process took a step or the location
was edited), opening no longer counts towards the step's time, and
SMB folders are listed in one request.
occ backup:cancel stops a running backup: no new step starts and the
step in progress finishes first. A run of an earlier version, which
does not hold back its next step, is marked cancelled after two
minutes unless it is recording its snapshot.
With few packs a request per pack is fewer requests than 256 listings. On
FTP every listing takes a data connection; the FTP server of the integration
tests has ten passive ports, and 256 listings in a row slowed it down until a
read failed ("Not found: snapshots/...").
A Hetzner Storage Box drops an SFTP connection that carried a lot of data
and then stayed quiet for a few minutes; files_external keeps using the dead
client, on which every file looks missing. StorageBackend now checks the
connection after a minute without requests, and replaces a connection that
is dead after a failed request (once per request; put, move and delete are
not repeated). SftpReadStream closes a connection it left in an unknown state.
The verify cache refresh took more than 6 minutes on PostgreSQL for 27,547
packs, one autocommitted insert per pack: putMany()/replaceAll() write in one
transaction. After a verify the cache holds the index files as they are, so a
blob in two packs stays cached for both.
files_external's SFTP read stream asks for the next 256 KiB only after
the previous answer has arrived, so reading costs one round trip per
request whatever the line speed: 4.9 MB/s at 47 ms to a Hetzner Storage
Box. SftpReadStream keeps up to eight READ requests in flight over the
same phpseclib connection (15 MB/s on that link), starting with one so
a small read does not fetch megabytes ahead.
A server that answers less than asked is handled by asking again from
where the answer ended; when more data follows, that size is used for
later reads over the same connection, never below the 32 KiB every
server must support. Without the needed connection methods, or when the
open fails, the storage's own stream is used as before.
Review notes on the open pack stream: it stayed open while a backup run went on to upload
(e.g. the snapshot after reading the file list back), which SFTP cannot do on one connection.
Reading moves to PackReader, and Repository reaches the location through PackReaderGuard,
which closes the stream before every other request. A pack stream that cannot seek is read up
to the offset. Deep verify no longer reports an \Error during decoding as damage.
Restores and deep verifies requested every blob separately: a new file handle and a seek on
the location each time. They read the blobs of a snapshot in the order they were written, so
the stream of the pack read last now stays open and is read on; a jump backwards, to another
pack, or more than 8 MiB ahead opens the pack again at the offset, as before. On a Hetzner
Storage Box over SMB, restoring 86 MB in 202 files went from 0.78 MB/s to 4.2 MB/s.
holdIndexCache() ran before beginTransaction(), outside the try, and the release came after
rollBack(); an exception from either left later cache writes queued for the rest of the run.
Opening a repository lists index/, which holds one file per pack.
StorageBackend::list() checked every entry with is_dir(), and on a
remote storage each check is a round trip. Over SFTP to a Hetzner Storage
Box (33 ms) with a 60 GiB repository, occ backup:run (30 s steps) spent
nearly every step on these requests and uploaded 45 KiB/s; with this
change it uploads 5.8 MB/s. The cost grows with the number of packs.
index/, snapshots/, catalog/ and trash/info/ only ever hold objects:
list them with the new IBackend::listFiles(), one listing without
per-entry checks.
A run only showed "running (files)", which says nothing during a backup
of a terabyte. The files phase now records an expected size on its first
step: the previous snapshot's size, or for a first backup the users'
files, versions and trash from Nextcloud's file cache (approximate, shown
with a ~). The web interface and occ backup:status show the share of it
done so far, including a large file that is still being read; it stays
below 100% until the run finishes.
Dillard Blom
dillard