Phase 2: FTP, SMB and NFS storage backends
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
No ng_backup code changes needed: files_external's generic backend/auth adapter (the same
ExternalStorageFactory/StorageBackend already used for S3/SFTP) already covers FTP and SMB, and
NFS was never a files_external backend to begin with (PLAN.md 3.5) -- the "local" backend plus an
admin-managed OS-level mount already does the job. This commit is test coverage and documentation.
tests/integration/targets-matrix.sh now also runs the full round trip against:
- FTP (stilliard/pure-ftpd). A first attempt with delfer/alpine-ftp-server (vsftpd) failed with a
generic "create directory operation failed": files_external's FTP backend always builds
absolute paths, which that server's chroot rejects even via plain php ftp_mkdir() with an
absolute path (a relative path on the same server works fine) -- not a files_external or
ng_backup bug, just a server-specific quirk, confirmed by switching test servers.
- SMB (dperson/samba), via a small Nextcloud image built on the fly with the smbclient PECL
extension (libsmbclient-dev + pecl install), in its own temp directory as the build context
(never /tmp directly). The backend *lists* as available with just the smbclient CLI binary too
(files_external's shell-wrapped fallback), but that fallback's stream cannot seek, and ng_backup
reads blobs back by byte range while backing up -- confirmed live, it fails the very first
(small) backup with "Seek failed in packs/...". The native extension is genuinely required here,
not just preferred.
README: new "Storage backend experiences" entries for both findings (and a pointer from the
"no extra PHP extensions" intro line, since SMB is the one real exception), plus NFS guidance
(mount it yourself, point the local backend at the mount point).
PLAN.md: FTP/SMB marked done; SMB/NFS testing on the user's own OMV instance stays open (internal,
not something CI can cover).
Local review (Codex + CodeRabbit) found three issues, all fixed: the README's first-pass SMB claim
("CLI tool is not enough") was imprecise about *why* -- it lists fine without the extension, it
just doesn't work for ng_backup's read pattern; the cleanup trap could abort before removing the
network if an earlier docker rm failed, and didn't remove the built image/Dockerfile; and the
Docker build used all of /tmp as its context instead of a dedicated directory.
This commit is contained in:
@@ -339,12 +339,16 @@ Volgorde: fase 1 grondig testen, reviewen en debuggen; dan nadenken over 0.1.0-a
|
||||
minimale volledige restore (`occ backup:restore:full` op een verse installatie) komt vóór de beta.
|
||||
|
||||
**Roadmap-indeling fase 2 (besluit 2026-10-05):** naar v1.2.0 toewerken met in fase 2:
|
||||
1. `backup:restore:full` (disaster recovery, verse installatie) — **in uitvoering**
|
||||
2. FTP/SMB-ondersteuning
|
||||
3. Verificatie/checksum-audit-commando
|
||||
1. `backup:restore:full` (disaster recovery, verse installatie) — **klaar** (2026-10-05)
|
||||
2. FTP/SMB-ondersteuning — **klaar** (2026-10-06): beide liepen al via files_external's generieke
|
||||
adapter zonder enige ng_backup-code; alleen testdekking toegevoegd
|
||||
(`tests/integration/targets-matrix.sh`) + bevindingen in README ("Storage backend experiences").
|
||||
SMB vereist de `smbclient` PECL-extensie (niet standaard in de Nextcloud-image, in CI nu
|
||||
on-the-fly gebouwd). NFS is geen eigen backend: lokaal mounten + `local`-backend erop richten.
|
||||
3. Verificatie/checksum-audit-commando — **klaar** (2026-10-05)
|
||||
4. Retentie/prune voor user-exports
|
||||
5. Automatische user-export inpluggen in het reguliere schema (of expliciet besluit dat het on-demand blijft)
|
||||
9. SMB/NFS intern testen op de eigen OMV-instantie
|
||||
9. SMB/NFS intern testen op de eigen OMV-instantie — nog open (intern, niet in CI)
|
||||
|
||||
Doorgeschoven naar **v1.3.0** (fase 3, niet meer fase 2): in-place volledige restore
|
||||
(eigen restore-modus i.p.v. onderhoudsmodus) en zelfbediening voor gebruikers (eigen bestanden
|
||||
|
||||
@@ -2,8 +2,10 @@
|
||||
|
||||
Encrypted, incremental backups of Nextcloud (database, configuration and data) with restore
|
||||
from the web interface. Works without shell access: no external programs, no extra PHP
|
||||
extensions beyond sodium. Runs on Nextcloud All-in-One, regular installations and hosted
|
||||
Nextcloud.
|
||||
extensions beyond sodium for ng_backup itself. Runs on Nextcloud All-in-One, regular
|
||||
installations and hosted Nextcloud. (Choosing SMB as a storage backend is the one exception:
|
||||
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
||||
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
||||
|
||||
**Status: in development (phase 1). Not ready for production use.** See `PLAN.md` for the design.
|
||||
|
||||
@@ -119,6 +121,29 @@ Contributions welcome via PR — add what you ran into with your own provider.
|
||||
verified whether this also applies to a Hetzner Storage Box **main** account (only tested with a
|
||||
sub-account so far) — if you've tried a main account, please say so in a PR.
|
||||
|
||||
- **FTP, server-dependent absolute-path handling:** files_external's FTP backend always builds
|
||||
absolute paths (e.g. `MKD /repo`) for the configured root, never relative ones. `atmoz/sftp`-style
|
||||
`vsftpd` test servers (chrooted) can reject that with a generic "create directory operation
|
||||
failed", even though a plain `ftp_mkdir()` with a *relative* path on the same server succeeds;
|
||||
`pure-ftpd` accepts the absolute form fine. If a working FTP server rejects ng_backup's writes,
|
||||
try a different FTP server implementation before assuming the backend itself is broken.
|
||||
|
||||
- **SMB/CIFS *lists* as available with just the `smbclient` CLI binary, but does not actually
|
||||
work for ng_backup without the `smbclient` PECL extension too.** files_external falls back to a
|
||||
shell-wrapped `smbclient` process when the native extension is missing, which registers the
|
||||
backend in `occ backup:target:backends` fine, but that fallback's stream cannot seek. ng_backup
|
||||
reads blobs back by byte range while backing up, so even the very first (small) backup fails
|
||||
with `Seek failed in packs/...`. Install the extension (`libsmbclient-dev` + `pecl install
|
||||
smbclient` + `docker-php-ext-enable smbclient` on Debian-based images), not just the CLI tool —
|
||||
the backend being listed is not proof it will work here. Once the extension is present it works
|
||||
the same as any other backend; no ng_backup-specific config beyond host/share/user/password.
|
||||
|
||||
- **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself
|
||||
(e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the
|
||||
mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping
|
||||
mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS
|
||||
can't be driven from PHP directly.
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Purpose |
|
||||
|
||||
@@ -1,17 +1,25 @@
|
||||
#!/bin/bash
|
||||
# Round trip of NG Backup against real S3 (s3mock) and SFTP backends, through files_external's own
|
||||
# backend/auth classes, same as a production location would use. Backend correctness does not
|
||||
# depend on the Nextcloud version or database, so this runs once (not matrixed across NC/DB),
|
||||
# on a recent Nextcloud with sqlite. Each backend gets its own fresh Nextcloud container, since
|
||||
# NG Backup's key can only be initialised once per instance.
|
||||
# Round trip of NG Backup against real S3 (s3mock), SFTP, FTP and SMB backends, through
|
||||
# files_external's own backend/auth classes, same as a production location would use. Backend
|
||||
# correctness does not depend on the Nextcloud version or database, so this runs once (not
|
||||
# matrixed across NC/DB), on a recent Nextcloud with sqlite. Each backend gets its own fresh
|
||||
# Nextcloud container, since NG Backup's key can only be initialised once per instance.
|
||||
# Usage: tests/integration/targets-matrix.sh [nextcloud-version]
|
||||
set -euo pipefail
|
||||
V=${1:-35}
|
||||
APP=$(cd "$(dirname "$0")/../.." && pwd)
|
||||
NET=ngb-targets-net
|
||||
SMB_IMAGE=ngb-targets-smb-nc
|
||||
SMB_BUILD_DIR=$(mktemp -d)
|
||||
RESULTS=()
|
||||
|
||||
cleanup() { docker rm -fv ngb-targets-nc-s3 ngb-targets-nc-sftp ngb-targets-s3 ngb-targets-sftp >/dev/null 2>&1; docker network rm "$NET" >/dev/null 2>&1; }
|
||||
cleanup() {
|
||||
docker rm -fv ngb-targets-nc-s3 ngb-targets-nc-sftp ngb-targets-nc-ftp ngb-targets-nc-smb \
|
||||
ngb-targets-s3 ngb-targets-sftp ngb-targets-ftp ngb-targets-smb >/dev/null 2>&1 || true
|
||||
docker rmi "$SMB_IMAGE" >/dev/null 2>&1 || true
|
||||
docker network rm "$NET" >/dev/null 2>&1 || true
|
||||
rm -rf "$SMB_BUILD_DIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
wait_tcp() { # host port
|
||||
@@ -19,11 +27,11 @@ wait_tcp() { # host port
|
||||
|| { echo "FAIL: $1:$2 never came up"; exit 1; }
|
||||
}
|
||||
|
||||
start_nc() { # container-name
|
||||
start_nc() { # container-name [image]
|
||||
docker run -d --name "$1" --network "$NET" \
|
||||
-e NEXTCLOUD_ADMIN_USER=admin -e NEXTCLOUD_ADMIN_PASSWORD=admin-targets-pw -e NEXTCLOUD_TRUSTED_DOMAINS=localhost \
|
||||
-e SQLITE_DATABASE=nextcloud \
|
||||
-v "$APP:/var/www/html/custom_apps/ng_backup:ro" nextcloud:$V-apache >/dev/null
|
||||
-v "$APP:/var/www/html/custom_apps/ng_backup:ro" "${2:-nextcloud:$V-apache}" >/dev/null
|
||||
for i in $(seq 1 90); do docker exec -u www-data "$1" php occ status 2>/dev/null | grep -q "installed: true" && break; sleep 4; done
|
||||
docker exec "$1" chown www-data:www-data /var/www/html/custom_apps
|
||||
echo "=== NC $V / sqlite ($1): $(docker exec -u www-data "$1" php occ -V 2>/dev/null)"
|
||||
@@ -51,8 +59,37 @@ docker run -d --name ngb-targets-s3 --network "$NET" \
|
||||
docker run -d --name ngb-targets-sftp --network "$NET" \
|
||||
atmoz/sftp ngbackup:ngbackuppw:::upload >/dev/null
|
||||
|
||||
# stilliard/pure-ftpd: delfer/alpine-ftp-server (vsftpd) was tried first and rejects the absolute
|
||||
# paths files_external's FTP backend always builds, inside this server's chroot specifically --
|
||||
# not a files_external bug (confirmed with plain php ftp_mkdir()), but pure-ftpd sidesteps it.
|
||||
docker run -d --name ngb-targets-ftp --network "$NET" \
|
||||
-e "FTP_USER_NAME=ngbackup" -e "FTP_USER_PASS=ngbackuppw" -e "FTP_USER_HOME=/home/ngbackup" \
|
||||
-e "PUBLICHOST=ngb-targets-ftp" \
|
||||
stilliard/pure-ftpd >/dev/null
|
||||
|
||||
# dperson/samba. The *native* smbclient PECL extension is required: files_external's shell-based
|
||||
# fallback (just the smbclient CLI binary) still registers this backend, but its stream cannot
|
||||
# seek, and ng_backup reads blobs back by byte range while backing up -- confirmed live, that
|
||||
# fallback fails the very first (small) backup with "Seek failed in packs/...". The extension is
|
||||
# not in the stock nextcloud image (needs libsmbclient-dev + a compile), so this backend gets its
|
||||
# own small image, built once here, in its own directory (never /tmp directly: that would send
|
||||
# every unrelated file under /tmp to the docker daemon as build context).
|
||||
cat > "$SMB_BUILD_DIR/Dockerfile" <<EOF
|
||||
FROM nextcloud:$V-apache
|
||||
RUN apt-get update -qq && apt-get install -y -qq libsmbclient-dev >/dev/null \\
|
||||
&& pecl install smbclient >/dev/null 2>&1 \\
|
||||
&& docker-php-ext-enable smbclient \\
|
||||
&& apt-get clean
|
||||
EOF
|
||||
docker build -q -t "$SMB_IMAGE" "$SMB_BUILD_DIR" >/dev/null
|
||||
docker run -d --name ngb-targets-smb --network "$NET" \
|
||||
-e USERID=1000 -e GROUPID=1000 \
|
||||
dperson/samba -p -u "ngbackup;ngbackuppw" -s "backup;/share;yes;no;no;ngbackup" >/dev/null
|
||||
|
||||
wait_tcp ngb-targets-s3 9090
|
||||
wait_tcp ngb-targets-sftp 22
|
||||
wait_tcp ngb-targets-ftp 21
|
||||
wait_tcp ngb-targets-smb 445
|
||||
|
||||
start_nc ngb-targets-nc-s3
|
||||
run "S3 (s3mock)" ngb-targets-nc-s3 "amazons3 -a amazons3::accesskey -o bucket=ngbackups -o hostname=ngb-targets-s3 -o port=9090 -o region=us-east-1 -o use_ssl=false -o use_path_style=true -o key=test -o secret=test"
|
||||
@@ -60,6 +97,12 @@ run "S3 (s3mock)" ngb-targets-nc-s3 "amazons3 -a amazons3::accesskey -o bucket=n
|
||||
start_nc ngb-targets-nc-sftp
|
||||
run "SFTP" ngb-targets-nc-sftp "sftp -a password::password -o host=ngb-targets-sftp -o root=/upload -o user=ngbackup -o password=ngbackuppw"
|
||||
|
||||
start_nc ngb-targets-nc-ftp
|
||||
run "FTP" ngb-targets-nc-ftp "ftp -a password::password -o host=ngb-targets-ftp -o user=ngbackup -o password=ngbackuppw"
|
||||
|
||||
start_nc ngb-targets-nc-smb "$SMB_IMAGE"
|
||||
run "SMB" ngb-targets-nc-smb "smb -a password::password -o host=ngb-targets-smb -o share=backup -o user=ngbackup -o password=ngbackuppw"
|
||||
|
||||
printf '%s\n' "${RESULTS[@]}"
|
||||
printf '%s\n' "${RESULTS[@]}" | grep -q FAIL && exit 1
|
||||
exit 0
|
||||
|
||||
Reference in New Issue
Block a user