The DR test of 1.0.2 over SMB stopped after 392 GiB of 983: the connection to the Storage Box
dropped, and the smbclient extension then returns a string of length -1 for a failed read (it
stores smbc_read()'s -1 in an unsigned length). The read loop counted that as data and went
backwards forever at 100% CPU. Reads over SMB are now checked; a failed one is read again the
usual way over a new connection.
backup:restore:full that runs again (--force) leaves the data files already in place with the
snapshot's size and modification time, as a backup takes them as unchanged, and reads ahead only
what is still missing: after 392 GiB it goes on instead of starting over.
files_external's fopen() first stats the file and icewind's stream reads 1 MiB at a time. Open,
read and close straight through the native extension's state instead: from Canada to a Hetzner
Storage Box (130 ms) 0.66 instead of 1.05 s per index file, about 5 instead of 8 hours for the
27,000 index files of a 1 TB repository in backup:target:add. Anything unexpected falls back to
the usual way.
- The admin page and backup:status warn when background jobs do not run on a
system cron, or the cron stopped, with a link to how to set one up.
- Without a working system cron, the open admin page moves a running backup
forward through /api/tick, a short step per poll.
- The database phase shows the tables dumped ("db: 47/183 tables"); a backup
that has not moved for 15 minutes says it is waiting for background jobs.
- The trash of restore points is on a tab of its own, Bin.
- New screenshots; changelog and plan for 1.0.2.
S3: the packs are read ahead, eight ranges at a time, straight into files in
the spool. SMB: ranges are read in 8 MiB reads through libsmbclient instead
of icewind's 1 MiB reads; over a line with a 130 ms round trip, reading was
four times faster in a test (18 instead of 4.5 MiB/s).
A file whose data cannot be read or decrypted is left out and listed, and
the command exits with code 2. When the database cannot be restored (its
dump is damaged, or it does not fit this installation), backup:restore:full
leaves the fresh database as it is and puts the data directory and config of
the backup in ngb-salvage-<snapshot> in the data directory. Without --salvage
a restore of damaged data stops as before and points at the option.
- Admin page: an optional label (at most 100 characters) for a backup started with "Back up now";
an (i) by "removed by retention policy" in the trash that explains what it means and until when
the restore point can be brought back. backup:trash says the same below its list.
- backup:status, backup:list, backup:trash and backup:browse show times in the zone of the daily
schedule and name it, instead of the server's UTC.
- backup:restore --to-directory of a file restored nothing: the path got a "/" appended.
- A restore removes "<name>.ngb-restore-<hex>" temporaries that a stopped restore left in the
folders it writes to.
- Roadmap: the cron warning (#13), reading ahead from S3/SMB and --salvage move to 1.0.2.
SQLite in WAL mode starts a transaction as a reader; a write from another
process between our first read and our first write makes our first write
fail at once, whatever the busy timeout. The SQLite full-restore test now
runs a second writer alongside the restore.
Nextcloud 35 reads keys with phpseclib 3, which refuses an OpenSSH-format
key whose line breaks were lost ("Unable to read key"); phpseclib 2 on
Nextcloud 34 read it. The key is fixed when used, so stored locations work
again without re-entering it.
backup:user:restore <location> <uid> only said "is not a recorded user export": the argument
is the manifest path, which only backup:user:backup printed once. backup:user:list shows the
exports again; a bare user id works when that user has exactly one export, otherwise the error
lists them. The export is resolved before the replace confirmation.
Fixes#22
A restore read one pack at a time over one SFTP connection, which over a long
line is bound by the round trip and by what one TCP connection gets: about
2-3 MiB/s from a Hetzner Storage Box to Canada, with PHP mostly waiting.
Repository::restore() now first lists the ranges of packs the restore needs,
in order (ReadAheadPlan), and the location reads them ahead into unlinked
files in a spool folder in the target (IReadsAhead). Over SFTP,
SftpRangeFetcher does that on four connections of its own, 16 requests in
flight each, taking the answers of whichever connection has one. A range that
fails, a damaged copy, or a blob that moved since the plan is read the usual
way, which reports what is wrong. Other locations restore as before.
On a test line with 120 ms RTT and 0.2% loss, a 1.6 GiB restore went from
5.1 to 17.8 MiB/s, all files identical.
Switching FOREIGN_KEY_CHECKS back on does not check the rows loaded
meanwhile, and DDL would commit halfway, so on MySQL every foreign key is
checked with a query for rows that point at nothing, before the commit.
The recorded table structure now keeps index options (the prefix lengths
MySQL needs for long text columns) and foreign keys; tables the restore
creates get their foreign keys after the commit.
The full-restore test also runs on MariaDB.
Backups now record each table's structure in the dump manifest. The full
restore creates tables that do not exist on the new instance (left behind
by apps that were removed or are switched off), lists all problems at
once, and no longer stops on apps that are missing. Foreign keys are not
checked while rows are loaded in name order, and checked again before the
restore commits (PostgreSQL: dropped and added back in the transaction).
--install-apps installs or enables the apps that were enabled at backup
time; --skip-missing-tables leaves out tables of older backups whose
structure is unknown. nextcloud-aio is never required.
backup:target:add without a key points to backup:key:import, and says that
connecting can take a while. Removing a location drops its cached index;
a migration removes the cache of locations that are gone.
The full-restore test now also runs on PostgreSQL with Mail (foreign keys)
and a disabled app.
On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.
backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
Each 30-second step opened the location again and read its list of
index files. Over SMB that cost a request per file, so on a large
repository opening took longer than the step: each step stored one
4 MiB pack and the list grew further. A first backup of nc01 to a
Storage Box over SMB managed 12 GiB in 35 hours.
occ backup:run and system cron now keep the repository open for the
whole run (reopened when another process took a step or the location
was edited), opening no longer counts towards the step's time, and
SMB folders are listed in one request.
occ backup:cancel stops a running backup: no new step starts and the
step in progress finishes first. A run of an earlier version, which
does not hold back its next step, is marked cancelled after two
minutes unless it is recording its snapshot.
dillard