On S3, index files and the tree segments read back when a snapshot is
written are fetched 16 at a time through the storage's S3 client
(IBackend::getRanges(), getMany()). Blobs that follow each other in a pack
are read as one range on every location. The snapshot is written in a step
of its own and the run shows "commit" meanwhile. On a 1 TB production
repository the check went from about 4 hours to 3.5 minutes.
backup:status and the web interface list the runs of the last 7 days,
running ones and at least the newest 5; --all / "Show older runs" list
all. Durations of a day or more show the days.
Storage and repository:
- Listing skipped every name containing ".part-", directories too, so the
exports of a user like "john.part-time" were invisible to prune on S3,
SFTP, FTP and SMB and their data was deleted. Only real temporary uploads
("<name>.part-<8 hex>") are skipped now.
- prune refuses to run while a recorded per-user export's manifest is
missing, instead of freeing that export's data.
- forget() and untrash() copy first, switch the catalog, and delete the
originals last, so a crash can no longer leave the catalog pointing at a
missing snapshot or database dump.
- Writes are checked against the expected size, so a short upload is an
error instead of a truncated pack; raw restores check rename() and touch().
- A catalog write refuses to overwrite a generation another process just
wrote, and the catalog lock lasts 300 s, so opening a large repository
inside it no longer lets the lease run out.
Backup runs and scheduling:
- A file that keeps changing is read again at most three times; then the
last read is kept and reported, instead of restarting forever and
blocking cleanup.
- Unreadable files and folders are recorded, logged, shown by occ
backup:run, stored in the snapshot and notified to admins, instead of
being skipped silently.
- After a failed scheduled backup, at most three attempts a day, an hour
apart, instead of a new run and notification every five minutes.
- The job no longer spins on a run whose lease another process holds, and
the weekly cleanup catches up after a Sunday spent backing up.
- Notifications for the per-user export retention events.
RepositoryImportSource::copyToFolder() called putContent() on the generic
return type of Node::get(), which doesn't declare that method -- only
File does. Existing-file overwrite during a per-user restore would have
fatally errored at runtime; added an instanceof check and a resource leak
fix (finally-close) while there. Also guarded getFileAsStream() against
fopen() returning false.
Everything else Psalm (strict level) currently flags is mechanical
(missing `final` on leaf classes -- applied everywhere safe) or pre-existing
array-shape/type gaps, captured in psalm-baseline.xml so CI fails only on
new issues from here on, not the whole backlog at once.
- AdminApiController (admin only, CSRF-protected): status, create key,
download recovery kit, confirm (statement accepted + exact phrase),
backends with their usable auth mechanisms, add/test/remove locations,
start backups, schedule, snapshots, browse, queue restores
- RestoreJob runs web-initiated restores in the background; backup and
restore runs are kept apart in the run table
- admin section 'NG Backup' with a Vue 3 / @nextcloud/vue 9 page: key and
recovery-kit flow (download, tick statement, 'Yes, I confirm'), location
form built from files_external's parameter definitions, schedule, run
list with polling, snapshot browser and restore with the three modes
- auth mechanisms that depend on a user session or on files_external's
own stored credentials are not offered
API tested with curl on NC34 (status, backends, schedule validation,
confirmation refusal, browse, queued restore and backup completed by
cron); non-admins get 403, anonymous 401.
BackupService starts a run per location (database dump in one
transaction, then config + data directory as a resumable file run),
stores the state in ngb_runs after every step, locks a run while a step
executes and records the snapshot. Previews, logs, updater backups and
per-user cache/uploads are excluded (TreeWalker now supports patterns).
BackupJob (every 5 min) starts the daily schedule and steps running runs
with a short budget under web cron and a long one under system cron.
occ: backup:run, backup:status, backup:list.
Tested on NC34/PostgreSQL: first run 128 files / 1 GiB in 64 s (443 MiB
uploaded to S3), unchanged second run 2 s / 0 bytes; a background run to
SFTP completed through cron.php.
dillard