Storage and repository:
- Listing skipped every name containing ".part-", directories too, so the
exports of a user like "john.part-time" were invisible to prune on S3,
SFTP, FTP and SMB and their data was deleted. Only real temporary uploads
("<name>.part-<8 hex>") are skipped now.
- prune refuses to run while a recorded per-user export's manifest is
missing, instead of freeing that export's data.
- forget() and untrash() copy first, switch the catalog, and delete the
originals last, so a crash can no longer leave the catalog pointing at a
missing snapshot or database dump.
- Writes are checked against the expected size, so a short upload is an
error instead of a truncated pack; raw restores check rename() and touch().
- A catalog write refuses to overwrite a generation another process just
wrote, and the catalog lock lasts 300 s, so opening a large repository
inside it no longer lets the lease run out.
Backup runs and scheduling:
- A file that keeps changing is read again at most three times; then the
last read is kept and reported, instead of restarting forever and
blocking cleanup.
- Unreadable files and folders are recorded, logged, shown by occ
backup:run, stored in the snapshot and notified to admins, instead of
being skipped silently.
- After a failed scheduled backup, at most three attempts a day, an hour
apart, instead of a new run and notification every five minutes.
- The job no longer spins on a run whose lease another process holds, and
the weekly cleanup catches up after a Sunday spent backing up.
- Notifications for the per-user export retention events.
A full backup opened its repository at the start of a step and wrote the
catalog at the end of it, without the catalog lock. A per-user export or
restore that wrote the catalog in between could lose its entry, or the
snapshot could lose its own.
The final step now only stores the snapshot object and moves the run to a
new 'commit' phase. BackupService then records the snapshot under the
catalog lock, from a repository opened inside that lock; if the lock is
busy, the next step retries the commit. Repository::writeSnapshot() keeps
its old behaviour as putSnapshot() plus recordSnapshot().
TreeWalker walks one or more roots in a stable segment-wise order with
exclusions and resumes after a cursor; ParentCursor merge-joins the
previous snapshot in the same order. BackupRun no longer keeps a file
list or the parent's entries in memory or in its state.
- detect files changing during reading or between resumable steps and
re-read them (size/mtime/ctime/inode), spike F overwrites mid-run
- drop pending blob ids when a pack or index upload fails
- dump and restore PostgreSQL sequence state exactly, bound parameters,
validated catalog names; spike C checks all sequences exactly
- keep invalid UTF-8 byte-exact (tagged base64)
- refuse large tables without a primary key
- StorageBackend: check write/flush/close results, validate base path
- user_migration import/export: root paths, intermediate folders,
partial writes, read errors
- spikes: never touch an existing account, own fixtures, 512M note
BackupRun is a resumable state machine: each step works until a deadline,
uploads its pack and tree segment, and returns the state; large files are
resumed at the next blob. Snapshot is written last.
Spike F: 200 MiB file resumed across steps in separate processes; with 6
random kill -9 the run completes and restores bit-identically.
spikes/RESULTS.md summarises phase 0 and the findings for the plan.
dillard