A restore read one pack at a time over one SFTP connection, which over a long
line is bound by the round trip and by what one TCP connection gets: about
2-3 MiB/s from a Hetzner Storage Box to Canada, with PHP mostly waiting.
Repository::restore() now first lists the ranges of packs the restore needs,
in order (ReadAheadPlan), and the location reads them ahead into unlinked
files in a spool folder in the target (IReadsAhead). Over SFTP,
SftpRangeFetcher does that on four connections of its own, 16 requests in
flight each, taking the answers of whichever connection has one. A range that
fails, a damaged copy, or a blob that moved since the plan is read the usual
way, which reports what is wrong. Other locations restore as before.
On a test line with 120 ms RTT and 0.2% loss, a 1.6 GiB restore went from
5.1 to 17.8 MiB/s, all files identical.
Switching FOREIGN_KEY_CHECKS back on does not check the rows loaded
meanwhile, and DDL would commit halfway, so on MySQL every foreign key is
checked with a query for rows that point at nothing, before the commit.
The recorded table structure now keeps index options (the prefix lengths
MySQL needs for long text columns) and foreign keys; tables the restore
creates get their foreign keys after the commit.
The full-restore test also runs on MariaDB.
Backups now record each table's structure in the dump manifest. The full
restore creates tables that do not exist on the new instance (left behind
by apps that were removed or are switched off), lists all problems at
once, and no longer stops on apps that are missing. Foreign keys are not
checked while rows are loaded in name order, and checked again before the
restore commits (PostgreSQL: dropped and added back in the transaction).
--install-apps installs or enables the apps that were enabled at backup
time; --skip-missing-tables leaves out tables of older backups whose
structure is unknown. nextcloud-aio is never required.
backup:target:add without a key points to backup:key:import, and says that
connecting can take a while. Removing a location drops its cached index;
a migration removes the cache of locations that are gone.
The full-restore test now also runs on PostgreSQL with Mail (foreign keys)
and a disabled app.
On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.
backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
Each 30-second step opened the location again and read its list of
index files. Over SMB that cost a request per file, so on a large
repository opening took longer than the step: each step stored one
4 MiB pack and the list grew further. A first backup of nc01 to a
Storage Box over SMB managed 12 GiB in 35 hours.
occ backup:run and system cron now keep the repository open for the
whole run (reopened when another process took a step or the location
was edited), opening no longer counts towards the step's time, and
SMB folders are listed in one request.
occ backup:cancel stops a running backup: no new step starts and the
step in progress finishes first. A run of an earlier version, which
does not hold back its next step, is marked cancelled after two
minutes unless it is recording its snapshot.
With few packs a request per pack is fewer requests than 256 listings. On
FTP every listing takes a data connection; the FTP server of the integration
tests has ten passive ports, and 256 listings in a row slowed it down until a
read failed ("Not found: snapshots/...").
A Hetzner Storage Box drops an SFTP connection that carried a lot of data
and then stayed quiet for a few minutes; files_external keeps using the dead
client, on which every file looks missing. StorageBackend now checks the
connection after a minute without requests, and replaces a connection that
is dead after a failed request (once per request; put, move and delete are
not repeated). SftpReadStream closes a connection it left in an unknown state.
The verify cache refresh took more than 6 minutes on PostgreSQL for 27,547
packs, one autocommitted insert per pack: putMany()/replaceAll() write in one
transaction. After a verify the cache holds the index files as they are, so a
blob in two packs stays cached for both.
files_external's SFTP read stream asks for the next 256 KiB only after
the previous answer has arrived, so reading costs one round trip per
request whatever the line speed: 4.9 MB/s at 47 ms to a Hetzner Storage
Box. SftpReadStream keeps up to eight READ requests in flight over the
same phpseclib connection (15 MB/s on that link), starting with one so
a small read does not fetch megabytes ahead.
A server that answers less than asked is handled by asking again from
where the answer ended; when more data follows, that size is used for
later reads over the same connection, never below the 32 KiB every
server must support. Without the needed connection methods, or when the
open fails, the storage's own stream is used as before.
Review notes on the open pack stream: it stayed open while a backup run went on to upload
(e.g. the snapshot after reading the file list back), which SFTP cannot do on one connection.
Reading moves to PackReader, and Repository reaches the location through PackReaderGuard,
which closes the stream before every other request. A pack stream that cannot seek is read up
to the offset. Deep verify no longer reports an \Error during decoding as damage.
Restores and deep verifies requested every blob separately: a new file handle and a seek on
the location each time. They read the blobs of a snapshot in the order they were written, so
the stream of the pack read last now stays open and is read on; a jump backwards, to another
pack, or more than 8 MiB ahead opens the pack again at the offset, as before. On a Hetzner
Storage Box over SMB, restoring 86 MB in 202 files went from 0.78 MB/s to 4.2 MB/s.
holdIndexCache() ran before beginTransaction(), outside the try, and the release came after
rollBack(); an exception from either left later cache writes queued for the rest of the run.
Opening a repository lists index/, which holds one file per pack.
StorageBackend::list() checked every entry with is_dir(), and on a
remote storage each check is a round trip. Over SFTP to a Hetzner Storage
Box (33 ms) with a 60 GiB repository, occ backup:run (30 s steps) spent
nearly every step on these requests and uploaded 45 KiB/s; with this
change it uploads 5.8 MB/s. The cost grows with the number of packs.
index/, snapshots/, catalog/ and trash/info/ only ever hold objects:
list them with the new IBackend::listFiles(), one listing without
per-entry checks.
A run only showed "running (files)", which says nothing during a backup
of a terabyte. The files phase now records an expected size on its first
step: the previous snapshot's size, or for a first backup the users'
files, versions and trash from Nextcloud's file cache (approximate, shown
with a ~). The web interface and occ backup:status show the share of it
done so far, including a large file that is still being read; it stays
below 100% until the run finishes.
dillard