- Admin page: an optional label (at most 100 characters) for a backup started with "Back up now";
an (i) by "removed by retention policy" in the trash that explains what it means and until when
the restore point can be brought back. backup:trash says the same below its list.
- backup:status, backup:list, backup:trash and backup:browse show times in the zone of the daily
schedule and name it, instead of the server's UTC.
- backup:restore --to-directory of a file restored nothing: the path got a "/" appended.
- A restore removes "<name>.ngb-restore-<hex>" temporaries that a stopped restore left in the
folders it writes to.
- Roadmap: the cron warning (#13), reading ahead from S3/SMB and --salvage move to 1.0.2.
SQLite in WAL mode starts a transaction as a reader; a write from another
process between our first read and our first write makes our first write
fail at once, whatever the busy timeout. The SQLite full-restore test now
runs a second writer alongside the restore.
backup:user:restore <location> <uid> only said "is not a recorded user export": the argument
is the manifest path, which only backup:user:backup printed once. backup:user:list shows the
exports again; a bare user id works when that user has exactly one export, otherwise the error
lists them. The export is resolved before the replace confirmation.
Fixes#22
Backups now record each table's structure in the dump manifest. The full
restore creates tables that do not exist on the new instance (left behind
by apps that were removed or are switched off), lists all problems at
once, and no longer stops on apps that are missing. Foreign keys are not
checked while rows are loaded in name order, and checked again before the
restore commits (PostgreSQL: dropped and added back in the transaction).
--install-apps installs or enables the apps that were enabled at backup
time; --skip-missing-tables leaves out tables of older backups whose
structure is unknown. nextcloud-aio is never required.
backup:target:add without a key points to backup:key:import, and says that
connecting can take a while. Removing a location drops its cached index;
a migration removes the cache of locations that are gone.
The full-restore test now also runs on PostgreSQL with Mail (foreign keys)
and a disabled app.
The open points from the pre-beta review are on the backlog in PLAN.md and
listed as known limitations in the README; none block 0.9.0-beta or 1.0.0.
The how-to warns against pointing a test server at a live location and
explains that removing a slot does not revoke access.
backup:user:backup and backup:user:restore now ask the administrator before
installing or enabling user_migration (or take --install-user-migration), so
nothing changes on the server without consent. The restore warns that
user_migration imports only its allowlisted app settings.
Also: a clear error for unknown locations instead of a raw database message;
prune reports the number of affected users, not the number of kept ones; and
--user-last rejects values that are not whole numbers. Adds docs/HOWTO.md and
updates the README and PLAN.md for the beta.
PruneService::userExportRetention()/setUserExportRetention(): a single, app-wide "keep the
newest N" setting (default 3, same default as the snapshot policy's "last"), deliberately simpler
than the full daily/weekly/monthly RetentionPolicy -- per-user exports are on-demand only (occ
backup:backup:user, see the fold-out task about whether they ever join the automatic schedule),
so there is no regular cadence yet to bucket by day/week/month.
PruneService::applyUserExports(): groups a location's user exports by uid (read from each
manifest's own 'time' field, the same field snapshot retention already reads), keeps the newest N
per user, forgets the rest. No trash, unlike a snapshot: these are not yet the primary safety net
the deletion delay protects. Runs under the same exclusive lock as snapshot retention.
occ backup:retention gained --user-last=N; occ backup:prune now also applies it for every target
it processes, reporting counts the same way it already does for snapshots and blob packs.
Verified live against the dev rig (four real exports across four users, confirmed exactly the
expected ones were kept/forgotten at two different --user-last values) and added coverage to
tests/integration/user-roundtrip.sh (an account's manual export plus the automatic safety-export
a replace-mode restore makes of it, pruned down to --user-last=2).
No ng_backup code changes needed: files_external's generic backend/auth adapter (the same
ExternalStorageFactory/StorageBackend already used for S3/SFTP) already covers FTP and SMB, and
NFS was never a files_external backend to begin with (PLAN.md 3.5) -- the "local" backend plus an
admin-managed OS-level mount already does the job. This commit is test coverage and documentation.
tests/integration/targets-matrix.sh now also runs the full round trip against:
- FTP (stilliard/pure-ftpd). A first attempt with delfer/alpine-ftp-server (vsftpd) failed with a
generic "create directory operation failed": files_external's FTP backend always builds
absolute paths, which that server's chroot rejects even via plain php ftp_mkdir() with an
absolute path (a relative path on the same server works fine) -- not a files_external or
ng_backup bug, just a server-specific quirk, confirmed by switching test servers.
- SMB (dperson/samba), via a small Nextcloud image built on the fly with the smbclient PECL
extension (libsmbclient-dev + pecl install), in its own temp directory as the build context
(never /tmp directly). The backend *lists* as available with just the smbclient CLI binary too
(files_external's shell-wrapped fallback), but that fallback's stream cannot seek, and ng_backup
reads blobs back by byte range while backing up -- confirmed live, it fails the very first
(small) backup with "Seek failed in packs/...". The native extension is genuinely required here,
not just preferred.
README: new "Storage backend experiences" entries for both findings (and a pointer from the
"no extra PHP extensions" intro line, since SMB is the one real exception), plus NFS guidance
(mount it yourself, point the local backend at the mount point).
PLAN.md: FTP/SMB marked done; SMB/NFS testing on the user's own OMV instance stays open (internal,
not something CI can cover).
Local review (Codex + CodeRabbit) found three issues, all fixed: the README's first-pass SMB claim
("CLI tool is not enough") was imprecise about *why* -- it lists fine without the extension, it
just doesn't work for ng_backup's read pattern; the cleanup trap could abort before removing the
network if an earlier docker rm failed, and didn't remove the built image/Dockerfile; and the
Docker build used all of /tmp as its context instead of a dedicated directory.
New commands, per PLAN.md 3.7:
- occ backup:key:import: adopt the master key from a recovery kit on a fresh installation
(instead of occ backup:key:init generating a new one), so occ backup:target:add opens the
existing repository on the location rather than creating an empty one.
- occ backup:restore:full: replace the database (consistent, one transaction), restore the data
directory in place, and merge config.php -- refuses up front and lists what's missing when the
Nextcloud major version or any app enabled at backup time doesn't match here (their tables must
exist before the database restore can use them), rather than guessing or partially applying.
Repository::restore() gained an optional $stripPrefix (write "data/<uid>/..." straight into the
real data directory, not a "data/" subdirectory of it) -- restoring "data/" and "config/" this way
needs no temporary copy of what can be the largest part of an instance.
occ backup:target:add now backfills the local snapshot cache (ngb_snapshots, what occ backup:list
actually reads) when it adopts an existing repository: without this, a freshly connected location
showed no restore points at all, even though the repository itself had them. One unreadable
snapshot is skipped rather than aborting the backfill or the target add itself.
Three self-referential problems found and fixed by testing this against two real, separate
Nextcloud instances sharing only the backup location (tests/integration/full-restore-roundtrip.sh
covers this; the per-process nature of these bugs meant they were invisible from inside a single
occ invocation, including the one used to write the fixes, which is exactly why a second, fresh
process re-running occ backup:status etc. was necessary to catch them):
- The full database dump includes ng_backup's own tables and appconfig rows; restoring them
verbatim would overwrite the key/target that is actively performing the restore, on THIS
installation, with the backed-up instance's. Both are now preserved: ng_backup's own tables
(and, on PostgreSQL, their sequences -- excluding a table does not exclude its sequence) are
excluded from what gets restored, and its appconfig rows are captured before the database
restore and written back after, in their own transaction.
- config.php's 'secret' is merged in from the backup (needed so already-encrypted content in the
restored database, e.g. share tokens, stays decryptable), but Target::options, the master key,
the wrapped key and the recovery kit code are all encrypted with that same instance secret
(OCP's ICrypto, or Nextcloud AppConfig's own "sensitive" flag underneath), not with ng_backup's
own key. Changing it without re-encrypting them first left all four permanently undecryptable
the moment any later occ command ran. Fixed with two helpers
(TargetService::reencryptOptionsAround(), KeyService::reencryptForSecretRotation()) that decrypt
before and re-encrypt after whatever callback actually changes the secret; config.php is applied
dead last in restoreFull(), after every other repository access. Not crash-safe (a process kill
mid-rotation would leave a key missing, not just stale) -- disaster recovery is run once by
hand, right after occ maintenance:install, a window too narrow yet to justify a two-phase write.
- A subtler variant of the same bug: Nextcloud's own AppConfig additionally encrypts a "sensitive"
value itself, tied to the same secret, underneath our own encryption. Overwriting an existing
sensitive key makes it read the old value first to compare, which then also fails against the
new secret. Deleting the key before setting the new value avoids that read entirely.
Roadmap (PLAN.md 9): phase 3 (in-place full restore, user self-service) moved out to v1.3.0; phase
2 continues toward v1.2.0 with FTP/SMB, a verification/checksum-audit command, user-export
retention, and deciding whether per-user export joins the automatic schedule.
info.xml's version field is restricted by Nextcloud's own info.xsd to plain
numeric SemVer, no pre-release suffix, so the alpha/beta/rc stage from
PLAN.md's release table lives in the git tag and CHANGELOG instead; noted
that in PLAN.md so it isn't rediscovered later. Bumped the version, wrote
an honest description for an alpha (feature-complete for phase 1, not for
production), and added CHANGELOG.md.
README: new "Storage backend experiences" section, seeded with the Hetzner
Storage Box sub-account root-path quirk found while testing a real SFTP
target (root must be "home", not "/"), open for others to extend via PR.
dillard