dillard
6c8a78f6bb
PLAN: 1.0.1 background jobs without system cron: warn, and let the open admin page move a backup
2026-10-19 20:41:11 +02:00
0accd5a3f2
Merge pull request #16 from dillardblom/rc4
...
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.4
2026-10-19 11:04:15 +02:00
dillard
583592f69b
Release 1.0.0-rc.4 (app version 0.9.10)
2026-10-19 10:34:01 +02:00
dillard
a384ab41bb
App store description: Ensembia line
2026-10-19 10:31:21 +02:00
dillard
9b85221e50
Read ahead: a plan that cannot be made leaves the restore to the usual reads; only a whole buffered SFTP packet counts as an answer waiting
2026-10-19 08:54:13 +02:00
dillard
c95f552cc6
Read ahead over SFTP: one connection with 32 requests in flight by default, more with restore_sftp_connections; a Storage Box gets slower with parallel readers
2026-10-19 07:46:12 +02:00
dillard
6121f684e9
Full restore: say to run occ upgrade before rerunning a restore that stopped after the database; remove empty spool folders of a stopped restore
2026-10-19 05:34:54 +02:00
dillard
43a71684f9
Read ahead over SFTP: take answers phpseclib has read already instead of waiting up to 10 s on the socket
2026-10-19 05:28:26 +02:00
dillard
88b719879d
Release candidate 4 (app version 0.9.10): full restore over SFTP reads ahead; SFTP leg in the full-restore test
2026-10-19 03:51:21 +02:00
dillard
1d7f7e6423
Read ahead: resolve paths before opening connections; a failing readAhead() leaves no spool folder
2026-10-19 01:52:58 +02:00
dillard
ccbbf01c54
Full restore over SFTP: read packs ahead on four connections at once
...
A restore read one pack at a time over one SFTP connection, which over a long
line is bound by the round trip and by what one TCP connection gets: about
2-3 MiB/s from a Hetzner Storage Box to Canada, with PHP mostly waiting.
Repository::restore() now first lists the ranges of packs the restore needs,
in order (ReadAheadPlan), and the location reads them ahead into unlinked
files in a spool folder in the target (IReadsAhead). Over SFTP,
SftpRangeFetcher does that on four connections of its own, 16 requests in
flight each, taking the answers of whichever connection has one. A range that
fails, a damaged copy, or a blob that moved since the plan is read the usual
way, which reports what is wrong. Other locations restore as before.
On a test line with 120 ms RTT and 0.2% loss, a 1.6 GiB restore went from
5.1 to 17.8 MiB/s, all files identical.
2026-10-19 01:40:49 +02:00
dillard
b789d17636
Screenshot URLs per release tag, so the app store fetches new images
2026-10-18 23:47:32 +02:00
dillard
8e5c8ccda4
Full restore: skip the ID counters of tables left out with --skip-missing-tables; test a backup made by rc.1; version 0.9.9
2026-10-18 23:44:00 +02:00
dillard
e0b79939f9
PLAN: 1.0.0 only reliability; 1.0.1 restore speed, cron warning, DR test with table structure
2026-10-18 23:19:18 +02:00
39b4fe359b
Merge pull request #15 from dillardblom/release-rc2
...
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.2
2026-10-18 23:16:32 +02:00
dillard
40e147f3a0
Release 1.0.0-rc.2 (app version 0.9.8)
2026-10-18 23:10:02 +02:00
7f6e5c15f8
Merge pull request #14 from dillardblom/rc2
...
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
1.0.0-rc.2: disaster recovery fixes
2026-10-18 23:08:55 +02:00
dillard
11ac30635a
target:add: show progress and time left while reading the index of an existing repository
2026-10-18 22:14:45 +02:00
dillard
9e1378c6ab
PLAN: DR finding 16, restore during another server's cleanup
2026-10-18 21:57:17 +02:00
dillard
04e91359a0
Restore: follow blobs that another server's cleanup repacked during the restore
2026-10-18 21:56:58 +02:00
dillard
46eaceb928
DR guide step 9 from the test; PLAN: small DR test with rc2 passed
2026-10-18 08:51:00 +02:00
dillard
9eeaefd938
Refuse a full restore when the table prefix differs
2026-10-18 02:39:14 +02:00
dillard
4f6a2e5b70
Check MySQL foreign keys explicitly; keep index prefix lengths and foreign keys of created tables
...
Switching FOREIGN_KEY_CHECKS back on does not check the rows loaded
meanwhile, and DDL would commit halfway, so on MySQL every foreign key is
checked with a query for rows that point at nothing, before the commit.
The recorded table structure now keeps index options (the prefix lengths
MySQL needs for long text columns) and foreign keys; tables the restore
creates get their foreign keys after the commit.
The full-restore test also runs on MariaDB.
2026-10-18 02:09:00 +02:00
dillard
7b735ea941
DR guide step 7 for rc2; changelog; version 0.9.8
2026-10-18 01:45:39 +02:00
dillard
d8ef50732f
Full restore creates missing tables and suspends foreign keys; --install-apps; clearer messages
...
Backups now record each table's structure in the dump manifest. The full
restore creates tables that do not exist on the new instance (left behind
by apps that were removed or are switched off), lists all problems at
once, and no longer stops on apps that are missing. Foreign keys are not
checked while rows are loaded in name order, and checked again before the
restore commits (PostgreSQL: dropped and added back in the transaction).
--install-apps installs or enables the apps that were enabled at backup
time; --skip-missing-tables leaves out tables of older backups whose
structure is unknown. nextcloud-aio is never required.
backup:target:add without a key points to backup:key:import, and says that
connecting can take a while. Removing a location drops its cached index;
a migration removes the cache of locations that are gone.
The full-restore test now also runs on PostgreSQL with Mail (foreign keys)
and a disabled app.
2026-10-18 01:45:01 +02:00
dillard
5af712acc3
README: BBR for backups over a long distance
2026-10-18 01:26:27 +02:00
dillard
01e16199f7
DR guide: use your own container name
2026-10-18 01:15:43 +02:00
dillard
0924da48ac
PLAN: upload speed over long distances; DR guide container name
2026-10-18 01:15:34 +02:00
dillard
88477a1a9f
PLAN: SFTP restore speed over long distances; foreign keys valid after the restore
2026-10-18 00:59:06 +02:00
dillard
f633de59b4
PLAN: foreign keys block the database restore (Mail)
2026-10-17 23:15:13 +02:00
dillard
523620acde
DR guide: concrete commands for background jobs, copying the kit and the key import
2026-10-17 21:48:31 +02:00
dillard
bbddc83f80
DR guide: background jobs, copying the kit, why not key:init; PLAN: app list and table structure in the backup
2026-10-17 21:38:55 +02:00
dillard
456dffc880
Step-by-step disaster recovery guide
2026-10-17 21:33:15 +02:00
dillard
4d423825a0
PLAN: findings of the DR test, rc2 scope and a second DR test before 1.0.0
2026-10-17 21:29:51 +02:00
dillard
29a56e432b
Mention paid restore help by Ensembia; GitHub sponsor link
2026-10-17 17:48:37 +02:00
dillard
7f29d009f3
App store description for 1.0.0; developed and sponsored by Ensembia
2026-10-17 15:52:33 +02:00
511c0fc285
Merge pull request #12 : Release 1.0.0-rc.1 (app version 0.9.7)
...
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.1 (app version 0.9.7)
2026-10-17 15:04:09 +02:00
dillard
76e8bb0245
Release 1.0.0-rc.1 (app version 0.9.7)
2026-10-17 13:07:41 +02:00
dillard
f32568f83c
Screenshots of the current layout
2026-10-16 00:57:17 +02:00
dillard
ff8619ae6b
PLAN: plan towards 1.0.0 and 1.1.0
2026-10-16 00:28:24 +02:00
dillard
629b8dd315
Passphrases of at least 16 characters; explain that the location holds the key too
2026-10-16 00:07:39 +02:00
dillard
c7cbc0b058
README: primary object storage and External storage mounts are not backed up
2026-10-15 22:47:45 +02:00
dillard
942e8ec003
Web interface: dates and times in the user's Nextcloud locale
2026-10-15 21:43:09 +02:00
dillard
144097d159
Run the backup job at every cron run; skip deflate on compressed data
2026-10-15 20:59:04 +02:00
dillard
5b7dd08e71
Ask for feedback on the admin page, with an experience report form
2026-10-15 10:32:21 +02:00
dillard
30f93fbb8a
README: a backup survives a restart of the containers
2026-10-15 10:20:21 +02:00
dillard
ca003c88ba
Fast S3 verify and snapshot commit; shorter runs list; durations with days
...
On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.
backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
2026-10-14 17:12:41 +02:00
9a3bb99d12
Merge pull request #11 : Release 0.9.6-beta.7
...
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 0.9.6-beta.7
2026-10-14 13:20:56 +02:00
dillard
c38cc47534
Release 0.9.6-beta.7
2026-10-14 13:14:46 +02:00
dillard
dfd3f38d8c
Backups to SMB no longer crawl; add occ backup:cancel
...
Each 30-second step opened the location again and read its list of
index files. Over SMB that cost a request per file, so on a large
repository opening took longer than the step: each step stored one
4 MiB pack and the list grew further. A first backup of nc01 to a
Storage Box over SMB managed 12 GiB in 35 hours.
occ backup:run and system cron now keep the repository open for the
whole run (reopened when another process took a step or the location
was edited), opening no longer counts towards the step's time, and
SMB folders are listed in one request.
occ backup:cancel stops a running backup: no new step starts and the
step in progress finishes first. A run of an earlier version, which
does not hold back its next step, is marked cancelled after two
minutes unless it is recording its snapshot.
2026-10-14 12:15:27 +02:00
dillard
Dillard Blom