diff --git a/PLAN.md b/PLAN.md index 34876e3..53ff3a6 100644 --- a/PLAN.md +++ b/PLAN.md @@ -339,12 +339,16 @@ Volgorde: fase 1 grondig testen, reviewen en debuggen; dan nadenken over 0.1.0-a minimale volledige restore (`occ backup:restore:full` op een verse installatie) komt vóór de beta. **Roadmap-indeling fase 2 (besluit 2026-10-05):** naar v1.2.0 toewerken met in fase 2: -1. `backup:restore:full` (disaster recovery, verse installatie) — **in uitvoering** -2. FTP/SMB-ondersteuning -3. Verificatie/checksum-audit-commando +1. `backup:restore:full` (disaster recovery, verse installatie) — **klaar** (2026-10-05) +2. FTP/SMB-ondersteuning — **klaar** (2026-10-06): beide liepen al via files_external's generieke + adapter zonder enige ng_backup-code; alleen testdekking toegevoegd + (`tests/integration/targets-matrix.sh`) + bevindingen in README ("Storage backend experiences"). + SMB vereist de `smbclient` PECL-extensie (niet standaard in de Nextcloud-image, in CI nu + on-the-fly gebouwd). NFS is geen eigen backend: lokaal mounten + `local`-backend erop richten. +3. Verificatie/checksum-audit-commando — **klaar** (2026-10-05) 4. Retentie/prune voor user-exports 5. Automatische user-export inpluggen in het reguliere schema (of expliciet besluit dat het on-demand blijft) -9. SMB/NFS intern testen op de eigen OMV-instantie +9. SMB/NFS intern testen op de eigen OMV-instantie — nog open (intern, niet in CI) Doorgeschoven naar **v1.3.0** (fase 3, niet meer fase 2): in-place volledige restore (eigen restore-modus i.p.v. onderhoudsmodus) en zelfbediening voor gebruikers (eigen bestanden diff --git a/README.md b/README.md index 965d49a..4455b8a 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,10 @@ Encrypted, incremental backups of Nextcloud (database, configuration and data) with restore from the web interface. Works without shell access: no external programs, no extra PHP -extensions beyond sodium. Runs on Nextcloud All-in-One, regular installations and hosted -Nextcloud. +extensions beyond sodium for ng_backup itself. Runs on Nextcloud All-in-One, regular +installations and hosted Nextcloud. (Choosing SMB as a storage backend is the one exception: +see "Storage backend experiences" below — that requirement comes from files_external, not from +ng_backup, and applies to any Nextcloud app that writes to an SMB share.) **Status: in development (phase 1). Not ready for production use.** See `PLAN.md` for the design. @@ -119,6 +121,29 @@ Contributions welcome via PR — add what you ran into with your own provider. verified whether this also applies to a Hetzner Storage Box **main** account (only tested with a sub-account so far) — if you've tried a main account, please say so in a PR. +- **FTP, server-dependent absolute-path handling:** files_external's FTP backend always builds + absolute paths (e.g. `MKD /repo`) for the configured root, never relative ones. `atmoz/sftp`-style + `vsftpd` test servers (chrooted) can reject that with a generic "create directory operation + failed", even though a plain `ftp_mkdir()` with a *relative* path on the same server succeeds; + `pure-ftpd` accepts the absolute form fine. If a working FTP server rejects ng_backup's writes, + try a different FTP server implementation before assuming the backend itself is broken. + +- **SMB/CIFS *lists* as available with just the `smbclient` CLI binary, but does not actually + work for ng_backup without the `smbclient` PECL extension too.** files_external falls back to a + shell-wrapped `smbclient` process when the native extension is missing, which registers the + backend in `occ backup:target:backends` fine, but that fallback's stream cannot seek. ng_backup + reads blobs back by byte range while backing up, so even the very first (small) backup fails + with `Seek failed in packs/...`. Install the extension (`libsmbclient-dev` + `pecl install + smbclient` + `docker-php-ext-enable smbclient` on Debian-based images), not just the CLI tool — + the backend being listed is not proof it will work here. Once the extension is present it works + the same as any other backend; no ng_backup-specific config beyond host/share/user/password. + +- **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself + (e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the + mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping + mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS + can't be driven from PHP directly. + ## Commands | Command | Purpose | diff --git a/tests/integration/targets-matrix.sh b/tests/integration/targets-matrix.sh index e6c6173..602db55 100755 --- a/tests/integration/targets-matrix.sh +++ b/tests/integration/targets-matrix.sh @@ -1,17 +1,25 @@ #!/bin/bash -# Round trip of NG Backup against real S3 (s3mock) and SFTP backends, through files_external's own -# backend/auth classes, same as a production location would use. Backend correctness does not -# depend on the Nextcloud version or database, so this runs once (not matrixed across NC/DB), -# on a recent Nextcloud with sqlite. Each backend gets its own fresh Nextcloud container, since -# NG Backup's key can only be initialised once per instance. +# Round trip of NG Backup against real S3 (s3mock), SFTP, FTP and SMB backends, through +# files_external's own backend/auth classes, same as a production location would use. Backend +# correctness does not depend on the Nextcloud version or database, so this runs once (not +# matrixed across NC/DB), on a recent Nextcloud with sqlite. Each backend gets its own fresh +# Nextcloud container, since NG Backup's key can only be initialised once per instance. # Usage: tests/integration/targets-matrix.sh [nextcloud-version] set -euo pipefail V=${1:-35} APP=$(cd "$(dirname "$0")/../.." && pwd) NET=ngb-targets-net +SMB_IMAGE=ngb-targets-smb-nc +SMB_BUILD_DIR=$(mktemp -d) RESULTS=() -cleanup() { docker rm -fv ngb-targets-nc-s3 ngb-targets-nc-sftp ngb-targets-s3 ngb-targets-sftp >/dev/null 2>&1; docker network rm "$NET" >/dev/null 2>&1; } +cleanup() { + docker rm -fv ngb-targets-nc-s3 ngb-targets-nc-sftp ngb-targets-nc-ftp ngb-targets-nc-smb \ + ngb-targets-s3 ngb-targets-sftp ngb-targets-ftp ngb-targets-smb >/dev/null 2>&1 || true + docker rmi "$SMB_IMAGE" >/dev/null 2>&1 || true + docker network rm "$NET" >/dev/null 2>&1 || true + rm -rf "$SMB_BUILD_DIR" +} trap cleanup EXIT wait_tcp() { # host port @@ -19,11 +27,11 @@ wait_tcp() { # host port || { echo "FAIL: $1:$2 never came up"; exit 1; } } -start_nc() { # container-name +start_nc() { # container-name [image] docker run -d --name "$1" --network "$NET" \ -e NEXTCLOUD_ADMIN_USER=admin -e NEXTCLOUD_ADMIN_PASSWORD=admin-targets-pw -e NEXTCLOUD_TRUSTED_DOMAINS=localhost \ -e SQLITE_DATABASE=nextcloud \ - -v "$APP:/var/www/html/custom_apps/ng_backup:ro" nextcloud:$V-apache >/dev/null + -v "$APP:/var/www/html/custom_apps/ng_backup:ro" "${2:-nextcloud:$V-apache}" >/dev/null for i in $(seq 1 90); do docker exec -u www-data "$1" php occ status 2>/dev/null | grep -q "installed: true" && break; sleep 4; done docker exec "$1" chown www-data:www-data /var/www/html/custom_apps echo "=== NC $V / sqlite ($1): $(docker exec -u www-data "$1" php occ -V 2>/dev/null)" @@ -51,8 +59,37 @@ docker run -d --name ngb-targets-s3 --network "$NET" \ docker run -d --name ngb-targets-sftp --network "$NET" \ atmoz/sftp ngbackup:ngbackuppw:::upload >/dev/null +# stilliard/pure-ftpd: delfer/alpine-ftp-server (vsftpd) was tried first and rejects the absolute +# paths files_external's FTP backend always builds, inside this server's chroot specifically -- +# not a files_external bug (confirmed with plain php ftp_mkdir()), but pure-ftpd sidesteps it. +docker run -d --name ngb-targets-ftp --network "$NET" \ + -e "FTP_USER_NAME=ngbackup" -e "FTP_USER_PASS=ngbackuppw" -e "FTP_USER_HOME=/home/ngbackup" \ + -e "PUBLICHOST=ngb-targets-ftp" \ + stilliard/pure-ftpd >/dev/null + +# dperson/samba. The *native* smbclient PECL extension is required: files_external's shell-based +# fallback (just the smbclient CLI binary) still registers this backend, but its stream cannot +# seek, and ng_backup reads blobs back by byte range while backing up -- confirmed live, that +# fallback fails the very first (small) backup with "Seek failed in packs/...". The extension is +# not in the stock nextcloud image (needs libsmbclient-dev + a compile), so this backend gets its +# own small image, built once here, in its own directory (never /tmp directly: that would send +# every unrelated file under /tmp to the docker daemon as build context). +cat > "$SMB_BUILD_DIR/Dockerfile" </dev/null \\ + && pecl install smbclient >/dev/null 2>&1 \\ + && docker-php-ext-enable smbclient \\ + && apt-get clean +EOF +docker build -q -t "$SMB_IMAGE" "$SMB_BUILD_DIR" >/dev/null +docker run -d --name ngb-targets-smb --network "$NET" \ + -e USERID=1000 -e GROUPID=1000 \ + dperson/samba -p -u "ngbackup;ngbackuppw" -s "backup;/share;yes;no;no;ngbackup" >/dev/null + wait_tcp ngb-targets-s3 9090 wait_tcp ngb-targets-sftp 22 +wait_tcp ngb-targets-ftp 21 +wait_tcp ngb-targets-smb 445 start_nc ngb-targets-nc-s3 run "S3 (s3mock)" ngb-targets-nc-s3 "amazons3 -a amazons3::accesskey -o bucket=ngbackups -o hostname=ngb-targets-s3 -o port=9090 -o region=us-east-1 -o use_ssl=false -o use_path_style=true -o key=test -o secret=test" @@ -60,6 +97,12 @@ run "S3 (s3mock)" ngb-targets-nc-s3 "amazons3 -a amazons3::accesskey -o bucket=n start_nc ngb-targets-nc-sftp run "SFTP" ngb-targets-nc-sftp "sftp -a password::password -o host=ngb-targets-sftp -o root=/upload -o user=ngbackup -o password=ngbackuppw" +start_nc ngb-targets-nc-ftp +run "FTP" ngb-targets-nc-ftp "ftp -a password::password -o host=ngb-targets-ftp -o user=ngbackup -o password=ngbackuppw" + +start_nc ngb-targets-nc-smb "$SMB_IMAGE" +run "SMB" ngb-targets-nc-smb "smb -a password::password -o host=ngb-targets-smb -o share=backup -o user=ngbackup -o password=ngbackuppw" + printf '%s\n' "${RESULTS[@]}" printf '%s\n' "${RESULTS[@]}" | grep -q FAIL && exit 1 exit 0