162 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
dillard	d504067cd8	Index: an empty listing is an error in checks and cleanups too; refresh() fetches only index files it has not loaded; full-restore CI on Nextcloud 35.0.0	2026-10-19 22:13:26 +02:00
dillard	c37b26ea38	Full restore: an empty listing never empties the index; read a failed pack once more; recognise and replace SFTP connections a Storage Box dropped quietly; version 0.9.11	2026-10-19 21:53:05 +02:00
Dillard Blom	0accd5a3f2	Merge pull request #16 from dillardblom/rc4 ... Release 1.0.0-rc.4 v1.0.0-rc.4	2026-10-19 11:04:15 +02:00
dillard	583592f69b	Release 1.0.0-rc.4 (app version 0.9.10)	2026-10-19 10:34:01 +02:00
dillard	a384ab41bb	App store description: Ensembia line	2026-10-19 10:31:21 +02:00
dillard	9b85221e50	Read ahead: a plan that cannot be made leaves the restore to the usual reads; only a whole buffered SFTP packet counts as an answer waiting	2026-10-19 08:54:13 +02:00
dillard	c95f552cc6	Read ahead over SFTP: one connection with 32 requests in flight by default, more with restore_sftp_connections; a Storage Box gets slower with parallel readers	2026-10-19 07:46:12 +02:00
dillard	6121f684e9	Full restore: say to run occ upgrade before rerunning a restore that stopped after the database; remove empty spool folders of a stopped restore	2026-10-19 05:34:54 +02:00
dillard	43a71684f9	Read ahead over SFTP: take answers phpseclib has read already instead of waiting up to 10 s on the socket	2026-10-19 05:28:26 +02:00
dillard	88b719879d	Release candidate 4 (app version 0.9.10): full restore over SFTP reads ahead; SFTP leg in the full-restore test	2026-10-19 03:51:21 +02:00
dillard	1d7f7e6423	Read ahead: resolve paths before opening connections; a failing readAhead() leaves no spool folder	2026-10-19 01:52:58 +02:00
dillard	ccbbf01c54	Full restore over SFTP: read packs ahead on four connections at once ... A restore read one pack at a time over one SFTP connection, which over a long line is bound by the round trip and by what one TCP connection gets: about 2-3 MiB/s from a Hetzner Storage Box to Canada, with PHP mostly waiting. Repository::restore() now first lists the ranges of packs the restore needs, in order (ReadAheadPlan), and the location reads them ahead into unlinked files in a spool folder in the target (IReadsAhead). Over SFTP, SftpRangeFetcher does that on four connections of its own, 16 requests in flight each, taking the answers of whichever connection has one. A range that fails, a damaged copy, or a blob that moved since the plan is read the usual way, which reports what is wrong. Other locations restore as before. On a test line with 120 ms RTT and 0.2% loss, a 1.6 GiB restore went from 5.1 to 17.8 MiB/s, all files identical.	2026-10-19 01:40:49 +02:00
dillard	b789d17636	Screenshot URLs per release tag, so the app store fetches new images	2026-10-18 23:47:32 +02:00
dillard	8e5c8ccda4	Full restore: skip the ID counters of tables left out with --skip-missing-tables; test a backup made by rc.1; version 0.9.9	2026-10-18 23:44:00 +02:00
dillard	e0b79939f9	PLAN: 1.0.0 only reliability; 1.0.1 restore speed, cron warning, DR test with table structure	2026-10-18 23:19:18 +02:00
Dillard Blom	39b4fe359b	Merge pull request #15 from dillardblom/release-rc2 ... Release 1.0.0-rc.2 v1.0.0-rc.2	2026-10-18 23:16:32 +02:00
dillard	40e147f3a0	Release 1.0.0-rc.2 (app version 0.9.8)	2026-10-18 23:10:02 +02:00
Dillard Blom	7f6e5c15f8	Merge pull request #14 from dillardblom/rc2 ... 1.0.0-rc.2: disaster recovery fixes	2026-10-18 23:08:55 +02:00
dillard	11ac30635a	target:add: show progress and time left while reading the index of an existing repository	2026-10-18 22:14:45 +02:00
dillard	9e1378c6ab	PLAN: DR finding 16, restore during another server's cleanup	2026-10-18 21:57:17 +02:00
dillard	04e91359a0	Restore: follow blobs that another server's cleanup repacked during the restore	2026-10-18 21:56:58 +02:00
dillard	46eaceb928	DR guide step 9 from the test; PLAN: small DR test with rc2 passed	2026-10-18 08:51:00 +02:00
dillard	9eeaefd938	Refuse a full restore when the table prefix differs	2026-10-18 02:39:14 +02:00
dillard	4f6a2e5b70	Check MySQL foreign keys explicitly; keep index prefix lengths and foreign keys of created tables ... Switching FOREIGN_KEY_CHECKS back on does not check the rows loaded meanwhile, and DDL would commit halfway, so on MySQL every foreign key is checked with a query for rows that point at nothing, before the commit. The recorded table structure now keeps index options (the prefix lengths MySQL needs for long text columns) and foreign keys; tables the restore creates get their foreign keys after the commit. The full-restore test also runs on MariaDB.	2026-10-18 02:09:00 +02:00
dillard	7b735ea941	DR guide step 7 for rc2; changelog; version 0.9.8	2026-10-18 01:45:39 +02:00
dillard	d8ef50732f	Full restore creates missing tables and suspends foreign keys; --install-apps; clearer messages ... Backups now record each table's structure in the dump manifest. The full restore creates tables that do not exist on the new instance (left behind by apps that were removed or are switched off), lists all problems at once, and no longer stops on apps that are missing. Foreign keys are not checked while rows are loaded in name order, and checked again before the restore commits (PostgreSQL: dropped and added back in the transaction). --install-apps installs or enables the apps that were enabled at backup time; --skip-missing-tables leaves out tables of older backups whose structure is unknown. nextcloud-aio is never required. backup:target:add without a key points to backup:key:import, and says that connecting can take a while. Removing a location drops its cached index; a migration removes the cache of locations that are gone. The full-restore test now also runs on PostgreSQL with Mail (foreign keys) and a disabled app.	2026-10-18 01:45:01 +02:00
dillard	5af712acc3	README: BBR for backups over a long distance	2026-10-18 01:26:27 +02:00
dillard	01e16199f7	DR guide: use your own container name	2026-10-18 01:15:43 +02:00
dillard	0924da48ac	PLAN: upload speed over long distances; DR guide container name	2026-10-18 01:15:34 +02:00
dillard	88477a1a9f	PLAN: SFTP restore speed over long distances; foreign keys valid after the restore	2026-10-18 00:59:06 +02:00
dillard	f633de59b4	PLAN: foreign keys block the database restore (Mail)	2026-10-17 23:15:13 +02:00
dillard	523620acde	DR guide: concrete commands for background jobs, copying the kit and the key import	2026-10-17 21:48:31 +02:00
dillard	bbddc83f80	DR guide: background jobs, copying the kit, why not key:init; PLAN: app list and table structure in the backup	2026-10-17 21:38:55 +02:00
dillard	456dffc880	Step-by-step disaster recovery guide	2026-10-17 21:33:15 +02:00
dillard	4d423825a0	PLAN: findings of the DR test, rc2 scope and a second DR test before 1.0.0	2026-10-17 21:29:51 +02:00
dillard	29a56e432b	Mention paid restore help by Ensembia; GitHub sponsor link	2026-10-17 17:48:37 +02:00
dillard	7f29d009f3	App store description for 1.0.0; developed and sponsored by Ensembia	2026-10-17 15:52:33 +02:00
Dillard Blom	511c0fc285	Merge pull request #12: Release 1.0.0-rc.1 (app version 0.9.7) ... Release 1.0.0-rc.1 (app version 0.9.7) v1.0.0-rc.1	2026-10-17 15:04:09 +02:00
dillard	76e8bb0245	Release 1.0.0-rc.1 (app version 0.9.7)	2026-10-17 13:07:41 +02:00
dillard	f32568f83c	Screenshots of the current layout	2026-10-16 00:57:17 +02:00
dillard	ff8619ae6b	PLAN: plan towards 1.0.0 and 1.1.0	2026-10-16 00:28:24 +02:00
dillard	629b8dd315	Passphrases of at least 16 characters; explain that the location holds the key too	2026-10-16 00:07:39 +02:00
dillard	c7cbc0b058	README: primary object storage and External storage mounts are not backed up	2026-10-15 22:47:45 +02:00
dillard	942e8ec003	Web interface: dates and times in the user's Nextcloud locale	2026-10-15 21:43:09 +02:00
dillard	144097d159	Run the backup job at every cron run; skip deflate on compressed data	2026-10-15 20:59:04 +02:00
dillard	5b7dd08e71	Ask for feedback on the admin page, with an experience report form	2026-10-15 10:32:21 +02:00
dillard	30f93fbb8a	README: a backup survives a restart of the containers	2026-10-15 10:20:21 +02:00
dillard	ca003c88ba	Fast S3 verify and snapshot commit; shorter runs list; durations with days ... On S3, index files and the tree segments read back when a snapshot is written are fetched 16 at a time through the storage's S3 client (IBackend::getRanges(), getMany()). Blobs that follow each other in a pack are read as one range on every location. The snapshot is written in a step of its own and the run shows "commit" meanwhile. On a 1 TB production repository the check went from about 4 hours to 3.5 minutes. backup:status and the web interface list the runs of the last 7 days, running ones and at least the newest 5; --all / "Show older runs" list all. Durations of a day or more show the days.	2026-10-14 17:12:41 +02:00
Dillard Blom	9a3bb99d12	Merge pull request #11: Release 0.9.6-beta.7 ... Release 0.9.6-beta.7 v0.9.6-beta.7	2026-10-14 13:20:56 +02:00
dillard	c38cc47534	Release 0.9.6-beta.7	2026-10-14 13:14:46 +02:00