dillard
7b735ea941
DR guide step 7 for rc2; changelog; version 0.9.8
2026-10-18 01:45:39 +02:00
dillard
d8ef50732f
Full restore creates missing tables and suspends foreign keys; --install-apps; clearer messages
...
Backups now record each table's structure in the dump manifest. The full
restore creates tables that do not exist on the new instance (left behind
by apps that were removed or are switched off), lists all problems at
once, and no longer stops on apps that are missing. Foreign keys are not
checked while rows are loaded in name order, and checked again before the
restore commits (PostgreSQL: dropped and added back in the transaction).
--install-apps installs or enables the apps that were enabled at backup
time; --skip-missing-tables leaves out tables of older backups whose
structure is unknown. nextcloud-aio is never required.
backup:target:add without a key points to backup:key:import, and says that
connecting can take a while. Removing a location drops its cached index;
a migration removes the cache of locations that are gone.
The full-restore test now also runs on PostgreSQL with Mail (foreign keys)
and a disabled app.
2026-10-18 01:45:01 +02:00
dillard
5af712acc3
README: BBR for backups over a long distance
2026-10-18 01:26:27 +02:00
dillard
01e16199f7
DR guide: use your own container name
2026-10-18 01:15:43 +02:00
dillard
0924da48ac
PLAN: upload speed over long distances; DR guide container name
2026-10-18 01:15:34 +02:00
dillard
88477a1a9f
PLAN: SFTP restore speed over long distances; foreign keys valid after the restore
2026-10-18 00:59:06 +02:00
dillard
f633de59b4
PLAN: foreign keys block the database restore (Mail)
2026-10-17 23:15:13 +02:00
dillard
523620acde
DR guide: concrete commands for background jobs, copying the kit and the key import
2026-10-17 21:48:31 +02:00
dillard
bbddc83f80
DR guide: background jobs, copying the kit, why not key:init; PLAN: app list and table structure in the backup
2026-10-17 21:38:55 +02:00
dillard
456dffc880
Step-by-step disaster recovery guide
2026-10-17 21:33:15 +02:00
dillard
4d423825a0
PLAN: findings of the DR test, rc2 scope and a second DR test before 1.0.0
2026-10-17 21:29:51 +02:00
dillard
29a56e432b
Mention paid restore help by Ensembia; GitHub sponsor link
2026-10-17 17:48:37 +02:00
dillard
7f29d009f3
App store description for 1.0.0; developed and sponsored by Ensembia
2026-10-17 15:52:33 +02:00
511c0fc285
Merge pull request #12 : Release 1.0.0-rc.1 (app version 0.9.7)
...
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.1 (app version 0.9.7)
2026-10-17 15:04:09 +02:00
dillard
76e8bb0245
Release 1.0.0-rc.1 (app version 0.9.7)
2026-10-17 13:07:41 +02:00
dillard
f32568f83c
Screenshots of the current layout
2026-10-16 00:57:17 +02:00
dillard
ff8619ae6b
PLAN: plan towards 1.0.0 and 1.1.0
2026-10-16 00:28:24 +02:00
dillard
629b8dd315
Passphrases of at least 16 characters; explain that the location holds the key too
2026-10-16 00:07:39 +02:00
dillard
c7cbc0b058
README: primary object storage and External storage mounts are not backed up
2026-10-15 22:47:45 +02:00
dillard
942e8ec003
Web interface: dates and times in the user's Nextcloud locale
2026-10-15 21:43:09 +02:00
dillard
144097d159
Run the backup job at every cron run; skip deflate on compressed data
2026-10-15 20:59:04 +02:00
dillard
5b7dd08e71
Ask for feedback on the admin page, with an experience report form
2026-10-15 10:32:21 +02:00
dillard
30f93fbb8a
README: a backup survives a restart of the containers
2026-10-15 10:20:21 +02:00
dillard
ca003c88ba
Fast S3 verify and snapshot commit; shorter runs list; durations with days
...
On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.
backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
2026-10-14 17:12:41 +02:00
9a3bb99d12
Merge pull request #11 : Release 0.9.6-beta.7
...
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 0.9.6-beta.7
2026-10-14 13:20:56 +02:00
dillard
c38cc47534
Release 0.9.6-beta.7
2026-10-14 13:14:46 +02:00
dillard
dfd3f38d8c
Backups to SMB no longer crawl; add occ backup:cancel
...
Each 30-second step opened the location again and read its list of
index files. Over SMB that cost a request per file, so on a large
repository opening took longer than the step: each step stored one
4 MiB pack and the list grew further. A first backup of nc01 to a
Storage Box over SMB managed 12 GiB in 35 hours.
occ backup:run and system cron now keep the repository open for the
whole run (reopened when another process took a step or the location
was edited), opening no longer counts towards the step's time, and
SMB folders are listed in one request.
occ backup:cancel stops a running backup: no new step starts and the
step in progress finishes first. A run of an earlier version, which
does not hold back its next step, is marked cancelled after two
minutes unless it is recording its snapshot.
2026-10-14 12:15:27 +02:00
dillard
92e693cfe6
Runs list: show the total size of each backup next to what it uploaded
2026-10-13 11:02:18 +02:00
dillard
bec070b1b5
Full-restore test: show the output and the log when restore:full fails
2026-10-12 17:15:25 +02:00
89a126999f
Merge pull request #10 : Release 0.9.5-beta.6
...
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
SFTP: open the channel first, replace dropped connections; run takeover; index cache in one transaction; faster checks with progress
2026-10-12 15:05:59 +02:00
dillard
1b986c9d31
Test: no random_bytes(0) in the batched SFTP read test
2026-10-12 14:59:31 +02:00
dillard
025bbfbb62
Verify: list the pack folders only with more packs than folders
...
With few packs a request per pack is fewer requests than 256 listings. On
FTP every listing takes a data connection; the FTP server of the integration
tests has ten passive ports, and 256 listings in a row slowed it down until a
read failed ("Not found: snapshots/...").
2026-10-12 14:50:56 +02:00
dillard
8ed7455619
Release 0.9.5-beta.6
2026-10-12 14:27:39 +02:00
dillard
30a250bc11
Stale-backup notification: "No successful backup to … yet" instead of "since never"
2026-10-12 11:32:42 +02:00
dillard
9c06ab2841
SFTP: replace a dropped connection; write the index cache in one transaction
...
A Hetzner Storage Box drops an SFTP connection that carried a lot of data
and then stayed quiet for a few minutes; files_external keeps using the dead
client, on which every file looks missing. StorageBackend now checks the
connection after a minute without requests, and replaces a connection that
is dead after a failed request (once per request; put, move and delete are
not repeated). SftpReadStream closes a connection it left in an unknown state.
The verify cache refresh took more than 6 minutes on PostgreSQL for 27,547
packs, one autocommitted insert per pack: putMany()/replaceAll() write in one
transaction. After a verify the cache holds the index files as they are, so a
blob in two packs stays cached for both.
2026-10-12 01:58:54 +02:00
dillard
d659322e60
Merge branch 'fix/sftp-fresh-connection' into polish
2026-10-11 17:16:33 +02:00
dillard
7726c179f0
SftpReadStream: say why opening failed on the connection
2026-10-11 15:58:20 +02:00
dillard
df732f185a
Index: refresh the cache after a verify in one pass over the blobs, not one per pack
2026-10-11 14:31:57 +02:00
dillard
b3c8068f8d
Batched SFTP reads: no more batches on a connection whose server did not confirm a close
2026-10-11 14:05:29 +02:00
dillard
0a9bd963fb
Verify and batched SFTP reads: address review notes
2026-10-11 14:00:48 +02:00
dillard
379b81db02
Verify: index files several at a time over SFTP, pack folders listed, progress stored and shown
2026-10-11 13:39:38 +02:00
dillard
828f3f142a
Merge branch 'fix/sftp-fresh-connection' into polish
2026-10-11 13:26:47 +02:00
dillard
c146d700e8
Merge branch 'main' into polish
2026-10-11 13:25:06 +02:00
dillard
8e273ac259
Backup step: reporting on a committed snapshot cannot fail the run; check the snapshot row after a unique violation
2026-10-11 12:30:05 +02:00
dillard
6385925845
Backup step: write the run only if no other process wrote it since it was loaded
2026-10-11 12:24:00 +02:00
dillard
894258f836
SFTP: open the channel before the first fast read; backup step: survive a takeover by another process
2026-10-11 11:45:27 +02:00
54750e9742
Merge pull request #8 : Release 0.9.4-beta.5
...
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Restore and deep verify: read a pack as one stream; pipelined SFTP reads
2026-10-09 12:05:31 +02:00
dillard
fe92ef70bf
Release 0.9.4-beta.5
2026-10-09 11:58:52 +02:00
dillard
b3e3054737
SftpReadStream: fall back to the storage's own stream instead of failing a read
2026-10-09 11:24:37 +02:00
dillard
c775e27dd2
CHANGELOG: SFTP reads with several requests in flight
2026-10-09 09:59:10 +02:00
dillard
Dillard Blom