Commit Graph

13 Commits

Author SHA1 Message Date
dillard 51a4a67bc4 Heavy background jobs of other apps pause while a backup runs
Reserved in Nextcloud's job list, so no app is turned off and a job that is running goes on.
At most 6 hours in a row, then they get an hour; Nextcloud ends a reservation by itself after
12 hours, should the end of a pause never come. On by default, with a setting to turn it off.
2026-11-07 01:56:44 +01:00
dillard e58b86fc48 A schedule per location, and locations backed up one after the other
Off, once a day or every N hours from a time, each in its own time zone. Existing locations
keep the time that held for all of them. A backup that has to wait shows as queued, and its
restore point has the time it really began.
2026-11-07 00:04:00 +01:00
dillard 193f65940b Pin a restore point and keep a note on it
Both live in the catalog on the location, so a new server sees them; the database holds
a local copy for the lists. The retention policy leaves a pinned restore point alone and
does not count it. Available on the restore page and as occ backup:pin and backup:note.
2026-11-06 23:49:50 +01:00
dillard 1b621b7149 Backup state stays valid JSON with a name that is not UTF-8; Bin reads on from an id, not a position 2026-11-06 20:02:09 +01:00
dillard b14b16c66c Empty folders in backup and restore, a full disk stops a restore, progress in the Bin
- Record folders without files in the meta record of a snapshot and restore them (#27)
- Errors writing to the restore directory stop the restore, also with --salvage
- Resume: check the folder before skipping, clean temporaries in folders that are complete
- Clean only read-ahead folders of our own; refresh the lease on skipped files
- Drop tables a failed database restore created; keep --salvage from calling a restored database fresh
- Retry a pack that fails to open on the storage (SMB)
- Without cron, running backups take turns from the open page
- Bin: read in parts with a counter and progress bar, show read errors, refresh when a run ends
2026-11-06 15:50:02 +01:00
dillard 36a30732ad Without system cron: warning and a backup that moves on while the page is open (#13); progress of the database phase; Bin tab
- The admin page and backup:status warn when background jobs do not run on a
  system cron, or the cron stopped, with a link to how to set one up.
- Without a working system cron, the open admin page moves a running backup
  forward through /api/tick, a short step per poll.
- The database phase shows the tables dumped ("db: 47/183 tables"); a backup
  that has not moved for 15 minutes says it is waiting for background jobs.
- The trash of restore points is on a tab of its own, Bin.
- New screenshots; changelog and plan for 1.0.2.
2026-11-04 04:08:20 +01:00
dillard 67cc350b8c Daily backup time in the administrator's time zone (#19); a time saved before is converted once so it shows as it ran 2026-10-22 16:38:54 +02:00
dillard ca003c88ba Fast S3 verify and snapshot commit; shorter runs list; durations with days
On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.

backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
2026-10-14 17:12:41 +02:00
dillard 9775cdc475 Locations: edit connection settings without losing the history
A changed password or host name could only be handled by adding a new
location, which starts an empty repository and, with a retention lock at
the provider, keeps the old one billed. Edit now changes a location in
place (web interface and occ backup:target:edit). The settings are only
saved when they lead to the same repository; a failed connection and a
missing repository give separate errors, and the folder is not created
while checking. Stored passwords are never sent to the browser: an empty
password field keeps the stored value.
2026-10-08 12:56:15 +02:00
dillard 85e835bfc2 Storage limit: 80% warning and limit in the web interface
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
- after each successful backup the stored size is recorded; at 80% of a
  location's limit administrators get a notification (once a day)
- locations list shows stored size and limit; the add form has a
  'maximum size' field that explains that empty means unlimited and
  billed by the provider; PUT /api/targets/{id}/limit (password
  confirmation)

Tested on NC34 with the OVH test bucket: a backup above the limit stops,
a successful one at 81% sends the warning.
2026-10-02 14:50:16 +02:00
dillard ac82b38e37 Fixes from the phase-1 code review
- one locking protocol per location: backup steps and restores hold a
  shared lock, prune and untrash an exclusive one and refuse while any
  run (backup, restore, export) exists; starting a backup checks and
  creates the run under the exclusive lock, closing the race with prune
- snapshot entries are validated when read (relative path without '.',
  '..', NUL, backslash or control characters); raw restore also checks
  that the resolved directory stays below the target; the restore API
  and service accept canonical paths only
- recovery kit confirmation requires a recent password confirmation and
  the confirmation code printed in the current kit file (new code per
  kit version), proving the kit was actually downloaded
- prune decides on the authenticated index files on the location and
  refreshes the local index cache from them
- restore job marks the run failed on any error; restore runs without
  progress for 6 hours are failed so they cannot block cleanup forever
- tests for safe paths and a malicious snapshot path
2026-10-02 11:21:40 +02:00
dillard d3aa07b106 M5: deletion delay with trash, passphrase slots, ransomware documentation
- removed snapshots go to a trash on the location and their data is only
  freed after the deletion delay, which is chosen when the key is created
  (default 7 days, 1-365) and cannot be changed afterwards; any admin can
  bring them back (occ backup:trash, backup:trash:restore, web UI)
- up to three passphrase slots open the same master key (like LUKS), so
  one person leaving or dying does not lock the company out; slot changes
  are written to every location, announced to all admins and require a
  new recovery kit confirmation, while backups keep running
- occ backup:key:slots, backup:key:slot:add|replace|remove; key init takes
  --label and --delete-delay-days
- web UI: deletion delay and first slot at key creation, slot management,
  trash per location with restore; all behind password confirmation
- README: key, slots (safe / CTO / head of IT), and protection against
  ransomware: object lock in compliance mode (e.g. 90 days), credentials
  without delete rights, provider snapshots, off-site copy
2026-10-02 03:01:44 +02:00
dillard 36e6a3fde7 M4: admin settings page and JSON API
- AdminApiController (admin only, CSRF-protected): status, create key,
  download recovery kit, confirm (statement accepted + exact phrase),
  backends with their usable auth mechanisms, add/test/remove locations,
  start backups, schedule, snapshots, browse, queue restores
- RestoreJob runs web-initiated restores in the background; backup and
  restore runs are kept apart in the run table
- admin section 'NG Backup' with a Vue 3 / @nextcloud/vue 9 page: key and
  recovery-kit flow (download, tick statement, 'Yes, I confirm'), location
  form built from files_external's parameter definitions, schedule, run
  list with polling, snapshot browser and restore with the three modes
- auth mechanisms that depend on a user session or on files_external's
  own stored credentials are not offered

API tested with curl on NC34 (status, backends, schedule validation,
confirmation refusal, browse, queued restore and backup completed by
cron); non-admins get 403, anonymous 401.
2026-10-02 01:09:47 +02:00