A notification when a schedule is turned off, then a reminder: once a week while another
location is backed up, every day when none is. The round trip now also checks the text of
these notifications and of the one about names, and that one location never gets two runs.
Off, once a day or every N hours from a time, each in its own time zone. Existing locations
keep the time that held for all of them. A backup that has to wait shows as queued, and its
restore point has the time it really began.
Both live in the catalog on the location, so a new server sees them; the database holds
a local copy for the lists. The retention policy leaves a pinned restore point alone and
does not count it. Available on the restore page and as occ backup:pin and backup:note.
Reported apart from unreadable files, with example paths and the advice to rename
them, instead of advice about permissions. Start of the 1.1.0 plan in PLAN.md.
- Leave out and report a file or folder whose name a snapshot cannot hold
- A staging write failure stops a full restore under --salvage
- Browse reuses its repository; empty folders read along with the entries
- Keep leases alive while empty folders are created; exact-prefix empty folder is the target
- Clean up tables that were partly created or before the transaction started
- Bin: no empty message after a read error; longer pages
A killed restore left its leases for up to two hours, so the first backup of the recovered
server was refused. A backup from before 1.0.3 records no empty folders: the test now expects that.
- Record folders without files in the meta record of a snapshot and restore them (#27)
- Errors writing to the restore directory stop the restore, also with --salvage
- Resume: check the folder before skipping, clean temporaries in folders that are complete
- Clean only read-ahead folders of our own; refresh the lease on skipped files
- Drop tables a failed database restore created; keep --salvage from calling a restored database fresh
- Retry a pack that fails to open on the storage (SMB)
- Without cron, running backups take turns from the open page
- Bin: read in parts with a counter and progress bar, show read errors, refresh when a run ends
The DR test of 1.0.2 over SMB stopped after 392 GiB of 983: the connection to the Storage Box
dropped, and the smbclient extension then returns a string of length -1 for a failed read (it
stores smbc_read()'s -1 in an unsigned length). The read loop counted that as data and went
backwards forever at 100% CPU. Reads over SMB are now checked; a failed one is read again the
usual way over a new connection.
backup:restore:full that runs again (--force) leaves the data files already in place with the
snapshot's size and modification time, as a backup takes them as unchanged, and reads ahead only
what is still missing: after 392 GiB it goes on instead of starting over.
files_external's fopen() first stats the file and icewind's stream reads 1 MiB at a time. Open,
read and close straight through the native extension's state instead: from Canada to a Hetzner
Storage Box (130 ms) 0.66 instead of 1.05 s per index file, about 5 instead of 8 hours for the
27,000 index files of a 1 TB repository in backup:target:add. Anything unexpected falls back to
the usual way.
- The admin page and backup:status warn when background jobs do not run on a
system cron, or the cron stopped, with a link to how to set one up.
- Without a working system cron, the open admin page moves a running backup
forward through /api/tick, a short step per poll.
- The database phase shows the tables dumped ("db: 47/183 tables"); a backup
that has not moved for 15 minutes says it is waiting for background jobs.
- The trash of restore points is on a tab of its own, Bin.
- New screenshots; changelog and plan for 1.0.2.
S3: the packs are read ahead, eight ranges at a time, straight into files in
the spool. SMB: ranges are read in 8 MiB reads through libsmbclient instead
of icewind's 1 MiB reads; over a line with a 130 ms round trip, reading was
four times faster in a test (18 instead of 4.5 MiB/s).
A file whose data cannot be read or decrypted is left out and listed, and
the command exits with code 2. When the database cannot be restored (its
dump is damaged, or it does not fit this installation), backup:restore:full
leaves the fresh database as it is and puts the data directory and config of
the backup in ngb-salvage-<snapshot> in the data directory. Without --salvage
a restore of damaged data stops as before and points at the option.
- Admin page: an optional label (at most 100 characters) for a backup started with "Back up now";
an (i) by "removed by retention policy" in the trash that explains what it means and until when
the restore point can be brought back. backup:trash says the same below its list.
- backup:status, backup:list, backup:trash and backup:browse show times in the zone of the daily
schedule and name it, instead of the server's UTC.
- backup:restore --to-directory of a file restored nothing: the path got a "/" appended.
- A restore removes "<name>.ngb-restore-<hex>" temporaries that a stopped restore left in the
folders it writes to.
- Roadmap: the cron warning (#13), reading ahead from S3/SMB and --salvage move to 1.0.2.
SQLite in WAL mode starts a transaction as a reader; a write from another
process between our first read and our first write makes our first write
fail at once, whatever the busy timeout. The SQLite full-restore test now
runs a second writer alongside the restore.
Nextcloud 35 reads keys with phpseclib 3, which refuses an OpenSSH-format
key whose line breaks were lost ("Unable to read key"); phpseclib 2 on
Nextcloud 34 read it. The key is fixed when used, so stored locations work
again without re-entering it.
backup:user:restore <location> <uid> only said "is not a recorded user export": the argument
is the manifest path, which only backup:user:backup printed once. backup:user:list shows the
exports again; a bare user id works when that user has exactly one export, otherwise the error
lists them. The export is resolved before the replace confirmation.
Fixes#22
A restore read one pack at a time over one SFTP connection, which over a long
line is bound by the round trip and by what one TCP connection gets: about
2-3 MiB/s from a Hetzner Storage Box to Canada, with PHP mostly waiting.
Repository::restore() now first lists the ranges of packs the restore needs,
in order (ReadAheadPlan), and the location reads them ahead into unlinked
files in a spool folder in the target (IReadsAhead). Over SFTP,
SftpRangeFetcher does that on four connections of its own, 16 requests in
flight each, taking the answers of whichever connection has one. A range that
fails, a damaged copy, or a blob that moved since the plan is read the usual
way, which reports what is wrong. Other locations restore as before.
On a test line with 120 ms RTT and 0.2% loss, a 1.6 GiB restore went from
5.1 to 17.8 MiB/s, all files identical.
dillard