Commit Graph

197 Commits

Author SHA1 Message Date
dillard 67cc350b8c Daily backup time in the administrator's time zone (#19); a time saved before is converted once so it shows as it ran 2026-10-22 16:38:54 +02:00
dillard fb5b234e8a Restore: a blob that does not decrypt as first read is read once more on a request of its own before the restore gives up 2026-10-22 14:43:48 +02:00
dillard fd12c586b6 Roadmap: restore mode for hosted Nextcloud, read-ahead for S3 and SMB; changelog for rc6 2026-10-22 13:07:09 +02:00
dillard 6ffe979ea7 Merge branch 'polish' into rc6 2026-10-22 13:06:59 +02:00
dillard a03b7354b3 README: tested disaster recoveries, backend measurements, restores without shell access; guide lists shell access; changelog for rc6 2026-10-22 13:06:49 +02:00
dillard be1c628f07 Full-restore test: install Mail and Deck with up to three tries (the app store is slow at times) 2026-10-22 03:38:30 +02:00
dillard a8355df12e Full restore: tables the restore creates get this installation's charset and collation (utf8mb4 on MySQL/MariaDB); test with an emoji 2026-10-22 03:36:46 +02:00
Dillard Blom f0537f0f2c Merge pull request #18 from dillardblom/rc5
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.5
v1.0.0-rc.5
2026-10-22 02:28:05 +02:00
dillard fd0f786f5f Full restore: a run again after an interrupted restore keeps maintenance mode on when its check refuses; retry text about apps 2026-10-22 01:59:01 +02:00
dillard 2eabdf44d5 README and store description: background jobs must run on system cron, with how to set it up; rc5 date 2026-10-22 01:03:52 +02:00
dillard 61e2a28b67 Full restore: maintenance mode on before the checks (off again when they refuse); the printed retry command is complete 2026-10-22 00:53:08 +02:00
dillard 7ed53ad0eb Full restore: after an error maintenance mode stays on, with the one line to switch it off and run again; maintenance that was on before stays on 2026-10-22 00:36:58 +02:00
dillard bdcb4f0cae Full restore runs in maintenance mode, switched off again at the end (also after an error) 2026-10-22 00:31:34 +02:00
dillard 2616b6e543 Init guard also counts snapshots in the trash and the catalog as backup data 2026-10-21 22:42:43 +02:00
dillard 9bc4e4a4ab Never create a repository where backup data lies but no config could be read 2026-10-21 22:25:48 +02:00
dillard 827e46381e Log a snapshot that cannot be adopted when a location is added; full-restore test prints what target:add and backup:list said when the snapshot is missing 2026-10-21 22:08:14 +02:00
dillard 8eb6c4c033 Index: check known index files spread over the list 2026-10-21 20:56:45 +02:00
dillard d623416306 Index: an empty listing is taken only when known index files are gone too 2026-10-21 20:49:32 +02:00
dillard cd251cffb3 PLAN: SMB index reads one at a time (1.1.0) 2026-10-21 20:02:43 +02:00
dillard 0b4872599d PLAN: the 1.0.1 cron PR closes #13 2026-10-21 19:51:57 +02:00
dillard 95184dadc7 Index: an empty listing is checked (config present, listed once more) before it is taken, so a repository another server emptied is accepted and a hiccup recovers 2026-10-21 19:51:46 +02:00
dillard 9b6bd7d2ab Release 1.0.0-rc.5 (app version 0.9.11) 2026-10-21 13:10:03 +02:00
dillard 2fcd38f478 Full-restore test: Mail and Deck checks only when the origin had them (not on a SQLite origin) 2026-10-21 13:09:54 +02:00
dillard b7d09c9784 PLAN: 1.1.0 faster SMB (local and long-line measurements) 2026-10-21 09:27:15 +02:00
dillard 177361acae Index load keeps the index when it refuses an empty listing; counters of unchanged tables are repaired too, never moved backwards 2026-10-21 09:08:50 +02:00
dillard 24686897c3 PLAN: DR tests across databases (integration for every crossing; S3 sim onto MariaDB, SMB sim onto SQLite before 1.0.0; nc01 onto MariaDB after) 2026-10-20 23:12:19 +02:00
dillard 85ea0acc50 Full-restore test: restore onto another database (to-pgsql, to-mysql, to-sqlite), every crossing in CI 2026-10-20 23:10:25 +02:00
dillard ac57e31c93 Full restore onto PostgreSQL: counters of restored tables go after their data when the backup records none (MySQL, SQLite); test MariaDB onto PostgreSQL 2026-10-20 23:04:53 +02:00
dillard 315e83b847 PLAN: 1.0.1 progress in the database phase and a waiting notice, so a backup does not look stuck 2026-10-20 20:49:44 +02:00
dillard e461c48056 CI: back to Nextcloud 35 for the full-restore test (the failures were a slow app store, not the version) 2026-10-19 22:18:33 +02:00
dillard d504067cd8 Index: an empty listing is an error in checks and cleanups too; refresh() fetches only index files it has not loaded; full-restore CI on Nextcloud 35.0.0 2026-10-19 22:13:26 +02:00
dillard c37b26ea38 Full restore: an empty listing never empties the index; read a failed pack once more; recognise and replace SFTP connections a Storage Box dropped quietly; version 0.9.11 2026-10-19 21:53:05 +02:00
dillard b728db36c7 README: roadmap 2026-10-19 21:35:23 +02:00
dillard 6ed0a555cf PLAN: label field next to Back up now (1.0.1); per-location schedules, every N hours, locations one after the other, editable notes (1.1.0) 2026-10-19 21:26:20 +02:00
dillard 221ed9dc17 PLAN: the cron warning says cron is needed for all backups, links to how to set it up, and that without it a backup started with the button only moves while the page is open 2026-10-19 21:18:23 +02:00
dillard 285153a968 PLAN: 1.0.1 cron warning says to keep the page open under AJAX, with a link to the Nextcloud docs 2026-10-19 21:16:23 +02:00
dillard 6c8a78f6bb PLAN: 1.0.1 background jobs without system cron: warn, and let the open admin page move a backup 2026-10-19 20:41:11 +02:00
Dillard Blom 0accd5a3f2 Merge pull request #16 from dillardblom/rc4
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.4
v1.0.0-rc.4
2026-10-19 11:04:15 +02:00
dillard 583592f69b Release 1.0.0-rc.4 (app version 0.9.10) 2026-10-19 10:34:01 +02:00
dillard a384ab41bb App store description: Ensembia line 2026-10-19 10:31:21 +02:00
dillard 9b85221e50 Read ahead: a plan that cannot be made leaves the restore to the usual reads; only a whole buffered SFTP packet counts as an answer waiting 2026-10-19 08:54:13 +02:00
dillard c95f552cc6 Read ahead over SFTP: one connection with 32 requests in flight by default, more with restore_sftp_connections; a Storage Box gets slower with parallel readers 2026-10-19 07:46:12 +02:00
dillard 6121f684e9 Full restore: say to run occ upgrade before rerunning a restore that stopped after the database; remove empty spool folders of a stopped restore 2026-10-19 05:34:54 +02:00
dillard 43a71684f9 Read ahead over SFTP: take answers phpseclib has read already instead of waiting up to 10 s on the socket 2026-10-19 05:28:26 +02:00
dillard 88b719879d Release candidate 4 (app version 0.9.10): full restore over SFTP reads ahead; SFTP leg in the full-restore test 2026-10-19 03:51:21 +02:00
dillard 1d7f7e6423 Read ahead: resolve paths before opening connections; a failing readAhead() leaves no spool folder 2026-10-19 01:52:58 +02:00
dillard ccbbf01c54 Full restore over SFTP: read packs ahead on four connections at once
A restore read one pack at a time over one SFTP connection, which over a long
line is bound by the round trip and by what one TCP connection gets: about
2-3 MiB/s from a Hetzner Storage Box to Canada, with PHP mostly waiting.

Repository::restore() now first lists the ranges of packs the restore needs,
in order (ReadAheadPlan), and the location reads them ahead into unlinked
files in a spool folder in the target (IReadsAhead). Over SFTP,
SftpRangeFetcher does that on four connections of its own, 16 requests in
flight each, taking the answers of whichever connection has one. A range that
fails, a damaged copy, or a blob that moved since the plan is read the usual
way, which reports what is wrong. Other locations restore as before.

On a test line with 120 ms RTT and 0.2% loss, a 1.6 GiB restore went from
5.1 to 17.8 MiB/s, all files identical.
2026-10-19 01:40:49 +02:00
dillard b789d17636 Screenshot URLs per release tag, so the app store fetches new images 2026-10-18 23:47:32 +02:00
dillard 8e5c8ccda4 Full restore: skip the ID counters of tables left out with --skip-missing-tables; test a backup made by rc.1; version 0.9.9 2026-10-18 23:44:00 +02:00
dillard e0b79939f9 PLAN: 1.0.0 only reliability; 1.0.1 restore speed, cron warning, DR test with table structure 2026-10-18 23:19:18 +02:00