Commit Graph

10 Commits

Author SHA1 Message Date
dillard 99f1655d04 Review fixes: a missed passphrase change follows at the next backup, a key from before 1.1.0 counts as used, schedules edited in their own time zone
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
- A location that a passphrase change could not reach gets it at its next backup
- A key from before 1.1.0 is never replaced as an unused one
- An unreadable settings file on a location is told apart from a missing one
- A schedule is shown and saved in the zone it was saved in
- Saving an unchanged schedule keeps its moment; --timezone needs a schedule
- Schedule coverage uses the time the backup began reading its data
- Tests: catch-up scenario, upgrade test restores the new file, autumn clock change
2026-11-08 17:18:13 +01:00
dillard e58b86fc48 A schedule per location, and locations backed up one after the other
Off, once a day or every N hours from a time, each in its own time zone. Existing locations
keep the time that held for all of them. A backup that has to wait shows as queued, and its
restore point has the time it really began.
2026-11-07 00:04:00 +01:00
dillard 193f65940b Pin a restore point and keep a note on it
Both live in the catalog on the location, so a new server sees them; the database holds
a local copy for the lists. The retention policy leaves a pinned restore point alone and
does not count it. Available on the restore page and as occ backup:pin and backup:note.
2026-11-06 23:49:50 +01:00
dillard d8ef50732f Full restore creates missing tables and suspends foreign keys; --install-apps; clearer messages
Backups now record each table's structure in the dump manifest. The full
restore creates tables that do not exist on the new instance (left behind
by apps that were removed or are switched off), lists all problems at
once, and no longer stops on apps that are missing. Foreign keys are not
checked while rows are loaded in name order, and checked again before the
restore commits (PostgreSQL: dropped and added back in the transaction).

--install-apps installs or enables the apps that were enabled at backup
time; --skip-missing-tables leaves out tables of older backups whose
structure is unknown. nextcloud-aio is never required.

backup:target:add without a key points to backup:key:import, and says that
connecting can take a while. Removing a location drops its cached index;
a migration removes the cache of locations that are gone.

The full-restore test now also runs on PostgreSQL with Mail (foreign keys)
and a disabled app.
2026-10-18 01:45:01 +02:00
dillard 807353cbe4 Wire Psalm into CI, fix a real bug it found in user-migration import
RepositoryImportSource::copyToFolder() called putContent() on the generic
return type of Node::get(), which doesn't declare that method -- only
File does. Existing-file overwrite during a per-user restore would have
fatally errored at runtime; added an instanceof check and a resource leak
fix (finally-close) while there. Also guarded getFileAsStream() against
fopen() returning false.

Everything else Psalm (strict level) currently flags is mechanical
(missing `final` on leaf classes -- applied everywhere safe) or pre-existing
array-shape/type gaps, captured in psalm-baseline.xml so CI fails only on
new issues from here on, not the whole backlog at once.
2026-10-05 11:45:37 +02:00
dillard 2610c1ad2f Expiring leases instead of long-held locks
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
A backup step killed by a time limit, OOM or kill -9 kept Nextcloud's
lock on its location for up to an hour, blocking every new start and the
run itself. Locations and runs are now guarded by leases in ngb_leases
with an expiry: a backup step holds them for its duration plus two
minutes, restores refresh theirs every minute, prune holds 30 minutes,
untrash 5. Nextcloud's lock is only used for the milliseconds in which a
lease is checked and written.

Also: starting a backup while one is already running simply continues
it (no exclusive lease needed), and running to completion waits instead
of spinning while a crashed step's lease has not expired.

Tested on NC34: a backup killed four times at random moments resumes on
its own after the lease expires (about two minutes), and restores
bit-identically.
2026-10-02 14:44:52 +02:00
dillard 532b23482e Storage limit per location
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
S3 has no bucket quota, so NG Backup enforces one: before a pack is
uploaded, the stored size plus the pack must stay under the location's
limit, otherwise the backup stops with an error (and administrators are
notified through the failed run). Repacking during prune is exempt, as it
is how space is freed. occ backup:target:limit NAME [SIZE|none].

Also ignore credential files in the repository.
2026-10-02 14:23:25 +02:00
dillard 38654317a2 M5: retention policy, prune with repacking, append-only locations
- RetentionPolicy (last/daily/weekly/monthly) and PruneService: forget
  snapshots the policy drops, delete packs nothing uses and repack mostly
  unused ones by copying encrypted blobs as-is; new packs are indexed
  before old ones are removed
- a location lock keeps prune and backups apart; append-only locations
  are never pruned (cleanup via the location's lifecycle rules)
- weekly prune from the background job; occ backup:prune [--dry-run],
  backup:retention, backup:target:append-only
- tests: forget+prune frees space and the remaining snapshot restores
  identically, repack of mixed packs, dry-run preview, retention buckets
2026-10-02 02:21:48 +02:00
dillard 370f3c55fa M2: local cache of pack indexes (ngb_index)
Opening a repository lists index/ once and downloads only index files
that are not cached; packs written by this server are cached on write,
stale cache rows are dropped. Each resumable step therefore no longer
downloads every index file from the location.
2026-10-02 00:55:46 +02:00
dillard 506854d597 M1: key management, backup locations and occ commands
- migration: ngb_targets, ngb_runs, ngb_snapshots
- KeyService: master key stored encrypted with the instance secret
  (unattended backups), passphrase-wrapped copy in every repository and
  in the recovery kit, recorded admin confirmation per key fingerprint
- TargetService: locations built from files_external backends without a
  mount; files_external is loaded through IAppManager::loadApp() and its
  enabled state is left alone; credentials encrypted with ICrypto;
  existing repositories are adopted only if made with this key, and a
  repository cannot be registered twice
- Repository: initWithKey/openWithKey with an encrypted key check
- occ: backup:key:init|kit|confirm, backup:target:backends|add|list|remove
2026-10-02 00:37:18 +02:00