36 Commits

Author SHA1 Message Date
dillard fb8e8d0694 Review fixes: a malformed key reads as damaged, removing a location always reports it, a key from before 1.1.0 in the recovery guide
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- Invalid base64 or a wrong salt/nonce length in a key gives the damaged-key error, so the recovery options are shown
- Tidying up after a removed location cannot fail the removal anymore, so the removal event is always sent
- Recovery guide: a key made before 1.1.0 counts as used and cannot make way
2026-11-09 00:55:20 +01:00
dillard cf9cf50286 Replace a backup key that was never used by the key of existing backups
A key made by mistake on a server that is being restored left no way back: key:import and the
passphrase route refused because a key existed, and nothing removed it. As long as the key never
protected anything (no location created or adopted with it, no location or run in the database),
occ backup:key:import --replace-unused-key and occ backup:target:add --replace-unused-key replace
it, after the new key was opened. A key that was used is never replaced, also after its location
was removed. The refusal names the option, and all administrators are notified of a replacement.
2026-11-08 11:53:55 +01:00
dillard 569077fa69 Restore without the recovery kit: the key from the location, with a passphrase
On a server without a backup key, backup:target:add finds the backups at
the location and takes their key from its config with one of the old
server's passphrases. Backups then wait until a new recovery kit is
confirmed. The deletion delay and the repository id are now stored on the
location, sealed with the key, so the delay survives a restore without a
kit and --repair-config can write a lost config again from the kit.

The key page warns against creating a new key when restoring a server.
New test: tests/integration/recovery-scenarios.sh, and a no-kit mode in
full-restore-roundtrip.sh.
2026-11-07 12:22:38 +01:00
dillard e58b86fc48 A schedule per location, and locations backed up one after the other
Off, once a day or every N hours from a time, each in its own time zone. Existing locations
keep the time that held for all of them. A backup that has to wait shows as queued, and its
restore point has the time it really began.
2026-11-07 00:04:00 +01:00
dillard 193f65940b Pin a restore point and keep a note on it
Both live in the catalog on the location, so a new server sees them; the database holds
a local copy for the lists. The retention policy leaves a pinned restore point alone and
does not count it. Available on the restore page and as occ backup:pin and backup:note.
2026-11-06 23:49:50 +01:00
dillard dd2e1af997 Roadmap: original modification time for all folders; guide: folders reported as updated 2026-11-06 18:08:34 +01:00
dillard b14b16c66c Empty folders in backup and restore, a full disk stops a restore, progress in the Bin
- Record folders without files in the meta record of a snapshot and restore them (#27)
- Errors writing to the restore directory stop the restore, also with --salvage
- Resume: check the folder before skipping, clean temporaries in folders that are complete
- Clean only read-ahead folders of our own; refresh the lease on skipped files
- Drop tables a failed database restore created; keep --salvage from calling a restored database fresh
- Retry a pack that fails to open on the storage (SMB)
- Without cron, running backups take turns from the open page
- Bin: read in parts with a counter and progress bar, show read errors, refresh when a run ends
2026-11-06 15:50:02 +01:00
dillard d5eb7abbff Guide: after a full disk, run the restore again as in step 8
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-11-06 12:02:38 +01:00
dillard 4c973d5cd8 Docs: a full disk during a restore looks like a damaged backup, to be fixed in 1.0.3 2026-11-06 12:00:52 +01:00
dillard b86b07a49c Disaster recovery guide: --salvage after a restore that stopped halfway 2026-11-06 11:56:10 +01:00
dillard b625fe7666 A restore that stops at a file gone from the location points at --salvage; test a full restore with a damaged and a missing pack
The DR test of 1.0.2 over SMB is done: result in PLAN, empty folders as a known issue (#27).
2026-11-06 10:27:41 +01:00
dillard bf48d339e4 Remove the read-ahead folder of a killed restore with what it kept; changelog and DR guide: a stopped full restore goes on where it was
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-11-05 08:35:28 +01:00
dillard dfe1fe9aad --salvage for backup:restore and backup:restore:full: restore what can be read from a damaged backup
A file whose data cannot be read or decrypted is left out and listed, and
the command exits with code 2. When the database cannot be restored (its
dump is damaged, or it does not fit this installation), backup:restore:full
leaves the fresh database as it is and puts the data directory and config of
the backup in ngb-salvage-<snapshot> in the data directory. Without --salvage
a restore of damaged data stops as before and points at the option.
2026-11-04 04:08:20 +01:00
dillard 95e3dfcf6d User restore by user id, backup:user:list (#22)
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
backup:user:restore <location> <uid> only said "is not a recorded user export": the argument
is the manifest path, which only backup:user:backup printed once. backup:user:list shows the
exports again; a bare user id works when that user has exactly one export, otherwise the error
lists them. The export is resolved before the replace confirmation.

Fixes #22
2026-10-27 00:10:59 +01:00
dillard a03b7354b3 README: tested disaster recoveries, backend measurements, restores without shell access; guide lists shell access; changelog for rc6 2026-10-22 13:06:49 +02:00
dillard fd0f786f5f Full restore: a run again after an interrupted restore keeps maintenance mode on when its check refuses; retry text about apps 2026-10-22 01:59:01 +02:00
dillard 61e2a28b67 Full restore: maintenance mode on before the checks (off again when they refuse); the printed retry command is complete 2026-10-22 00:53:08 +02:00
dillard 7ed53ad0eb Full restore: after an error maintenance mode stays on, with the one line to switch it off and run again; maintenance that was on before stays on 2026-10-22 00:36:58 +02:00
dillard bdcb4f0cae Full restore runs in maintenance mode, switched off again at the end (also after an error) 2026-10-22 00:31:34 +02:00
dillard 6121f684e9 Full restore: say to run occ upgrade before rerunning a restore that stopped after the database; remove empty spool folders of a stopped restore 2026-10-19 05:34:54 +02:00
dillard 11ac30635a target:add: show progress and time left while reading the index of an existing repository 2026-10-18 22:14:45 +02:00
dillard 46eaceb928 DR guide step 9 from the test; PLAN: small DR test with rc2 passed 2026-10-18 08:51:00 +02:00
dillard 7b735ea941 DR guide step 7 for rc2; changelog; version 0.9.8 2026-10-18 01:45:39 +02:00
dillard 01e16199f7 DR guide: use your own container name 2026-10-18 01:15:43 +02:00
dillard 523620acde DR guide: concrete commands for background jobs, copying the kit and the key import 2026-10-17 21:48:31 +02:00
dillard bbddc83f80 DR guide: background jobs, copying the kit, why not key:init; PLAN: app list and table structure in the backup 2026-10-17 21:38:55 +02:00
dillard 456dffc880 Step-by-step disaster recovery guide 2026-10-17 21:33:15 +02:00
dillard 29a56e432b Mention paid restore help by Ensembia; GitHub sponsor link 2026-10-17 17:48:37 +02:00
dillard 629b8dd315 Passphrases of at least 16 characters; explain that the location holds the key too 2026-10-16 00:07:39 +02:00
dillard dfd3f38d8c Backups to SMB no longer crawl; add occ backup:cancel
Each 30-second step opened the location again and read its list of
index files. Over SMB that cost a request per file, so on a large
repository opening took longer than the step: each step stored one
4 MiB pack and the list grew further. A first backup of nc01 to a
Storage Box over SMB managed 12 GiB in 35 hours.

occ backup:run and system cron now keep the repository open for the
whole run (reopened when another process took a step or the location
was edited), opening no longer counts towards the step's time, and
SMB folders are listed in one request.

occ backup:cancel stops a running backup: no new step starts and the
step in progress finishes first. A run of an earlier version, which
does not hold back its next step, is marked cancelled after two
minutes unless it is recording its snapshot.
2026-10-14 12:15:27 +02:00
dillard 9775cdc475 Locations: edit connection settings without losing the history
A changed password or host name could only be handled by adding a new
location, which starts an empty repository and, with a retention lock at
the provider, keeps the old one billed. Edit now changes a location in
place (web interface and occ backup:target:edit). The settings are only
saved when they lead to the same repository; a failed connection and a
missing repository give separate errors, and the folder is not created
while checking. Stored passwords are never sent to the browser: an empty
password field keeps the stored value.
2026-10-08 12:56:15 +02:00
dillard 5742f77253 App store listing: screenshots, links and UI fixes
Add three screenshots (backups, restore, settings) taken on a demo
instance, plus website, repository and the HOWTO as admin documentation
in info.xml.

Fixes found while taking them:
- the "new folder" restore option lost "<date>" (l10n sanitizes it as
  a tag); show the folder name pattern via a placeholder instead
- the restore-point dropdown cut off the time
- passphrase actions stacked vertically in the slot table
- HOWTO: SFTP example used "sftp::password", the auth mechanism is
  "password::password"
2026-10-07 17:43:41 +02:00
dillard b89b5b5fa6 Document known limitations for the beta and record the review backlog
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
The open points from the pre-beta review are on the backlog in PLAN.md and
listed as known limitations in the README; none block 0.9.0-beta or 1.0.0.
The how-to warns against pointing a test server at a live location and
explains that removing a slot does not revoke access.
2026-10-07 00:19:46 +02:00
dillard 45069994e9 Pre-beta review: safer destructive commands and hardening
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- backup:restore:full refuses on an installation with more than one user
  and asks before it starts (--force skips both, also to rerun after a
  failure halfway), and warns when the snapshot's database dump was not
  one consistent transaction.
- backup:user:restore --mode=replace asks before deleting an existing
  account; non-interactive use needs --force.
- The recovery kit file is created owner-only and never over an existing
  file or symlink; backup:key:import shows the deletion delay it read from
  the kit.
- Adding a location through the web API needs password confirmation, like
  removing one.
- Key derivation parameters from a kit or repository config are bounded,
  and a stalled network stream no longer spins at full CPU.
- Floating-point columns are dumped exactly instead of with 14 digits.
2026-10-06 23:48:41 +02:00
dillard 614caf9f63 Document migrated- calendar names in the how-to; add backlog items
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-06 16:59:45 +02:00
dillard 943c95bf07 Ask before installing user_migration; add the beta how-to
backup:user:backup and backup:user:restore now ask the administrator before
installing or enabling user_migration (or take --install-user-migration), so
nothing changes on the server without consent. The restore warns that
user_migration imports only its allowlisted app settings.

Also: a clear error for unknown locations instead of a raw database message;
prune reports the number of affected users, not the number of kept ones; and
--user-last rejects values that are not whole numbers. Adds docs/HOWTO.md and
updates the README and PLAN.md for the beta.
2026-10-06 16:57:56 +02:00