From 7f29d009f3b507fd0cb3c1afc6607a98be5d93c3 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sat, 17 Oct 2026 15:52:33 +0200 Subject: [PATCH 01/19] App store description for 1.0.0; developed and sponsored by Ensembia --- CHANGELOG.md | 3 +++ README.md | 2 +- appinfo/info.xml | 43 +++++++++++++++++++++++++++++++++++++------ 3 files changed, 41 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 11438ff..ffde11f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,9 @@ instead carried by the app description, the git tag, the release, and this file. ## [Unreleased] +### Changed +- A fuller app store description, ready for 1.0.0. + ## [1.0.0-rc.1] - 2026-10-17 Release candidate for 1.0.0; the app version is 0.9.7. Only bug fixes until 1.0.0. Tested on a diff --git a/README.md b/README.md index 83df76a..62b4dc9 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ ng_backup, and applies to any Nextcloud app that writes to an SMB share.) alongside your current backup, but not yet a drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and `docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. -License: AGPL-3.0-or-later. +License: AGPL-3.0-or-later. Developed and sponsored by [Ensembia](https://www.ensembia.com). ## How it works diff --git a/appinfo/info.xml b/appinfo/info.xml index c317bdb..34359ec 100644 --- a/appinfo/info.xml +++ b/appinfo/info.xml @@ -3,12 +3,43 @@ xsi:noNamespaceSchemaLocation="https://apps.nextcloud.com/schema/apps/info.xsd"> ng_backup NG Backup - Encrypted, incremental backups of your Nextcloud with restore from the web interface - + Encrypted, incremental backups of your Nextcloud — files, database and config — with restore from the web interface + 0.9.7 agpl Dillard Blom From 29a56e432b651e24ff20f53943b683d0f572f613 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sat, 17 Oct 2026 17:48:37 +0200 Subject: [PATCH 02/19] Mention paid restore help by Ensembia; GitHub sponsor link --- .github/FUNDING.yml | 1 + README.md | 2 ++ appinfo/info.xml | 3 ++- docs/HOWTO.md | 4 ++++ 4 files changed, 9 insertions(+), 1 deletion(-) create mode 100644 .github/FUNDING.yml diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 0000000..b2cb40b --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1 @@ +custom: ["https://www.ensembia.com"] diff --git a/README.md b/README.md index 62b4dc9..f02644a 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ alongside your current backup, but not yet a drop-in replacement for it. See "Kn `docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. License: AGPL-3.0-or-later. Developed and sponsored by [Ensembia](https://www.ensembia.com). +Need help restoring a backup after a crash? Ensembia offers paid restore assistance: +[www.ensembia.com](https://www.ensembia.com). ## How it works diff --git a/appinfo/info.xml b/appinfo/info.xml index 34359ec..f0bfb3f 100644 --- a/appinfo/info.xml +++ b/appinfo/info.xml @@ -39,7 +39,8 @@ All-in-One and hosted Nextcloud. Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very welcome: use the "Share your experience" link on the admin page. -Developed and sponsored by [Ensembia](https://www.ensembia.com).]]> +Developed and sponsored by [Ensembia](https://www.ensembia.com), which also offers paid help with +restoring a backup after a crash.]]> 0.9.7 agpl Dillard Blom diff --git a/docs/HOWTO.md b/docs/HOWTO.md index 3f46b45..3c97856 100644 --- a/docs/HOWTO.md +++ b/docs/HOWTO.md @@ -153,6 +153,10 @@ starts; `--force` skips both (also needed to rerun it after it stopped halfway). Afterwards, check that users, files and the database are complete, and run a backup of the restored server. The restored installation keeps its own app configuration and secret. +Stuck? [Ensembia](https://www.ensembia.com), the developer of NG Backup, offers paid help with +restoring a backup after a crash. You keep your passphrase to yourself: it is never needed by +anyone helping you. + ## 8. Retention ``` From 4d423825a020b89261b54e090d492b6ee8dba095 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sat, 17 Oct 2026 21:29:51 +0200 Subject: [PATCH 03/19] PLAN: findings of the DR test, rc2 scope and a second DR test before 1.0.0 --- PLAN.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/PLAN.md b/PLAN.md index bee690d..d42d141 100644 --- a/PLAN.md +++ b/PLAN.md @@ -395,6 +395,28 @@ in README "Known limitations": 3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf, 33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1. +- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met + PostgreSQL 18)** — voor **1.0.0-rc.2**: + 1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE, + AppAPI, een archief-app) blokkeren de restore met één tabelnaam per poging. Standaard + overslaan met één waarschuwing per app; optie om ze toch terug te zetten. + 2. Alle ontbrekende apps en tabellen **vooraf in één lijst**, per app benoemd (nu één per poging, + ook voor uitgeschakelde standaard-apps zoals `admin_audit`). + 3. `--skip-missing-apps`; `nextcloud-aio` standaard overslaan (bestaat buiten AIO niet), zodat + AIO → gewone Nextcloud werkt. + 4. `--install-apps`: ontbrekende apps uit de app store installeren. + 5. Apps buiten de app store (eigen builds, `side_menu`, `nextcloud-aio`): documenteren; later + eventueel `custom_apps` meenemen in de backup. + 6. Tabellen van een app die wél aanstond maar die hun tabel later zelf maakt (Memories: + `memories_planet_geometry`) netjes afhandelen. + 7. `backup:target:add` zonder sleutel verwijst naar `key:init`; als er al een repository staat, + moet het naar `key:import ` wijzen. + 8. `backup:target:add` toont geen voortgang bij het ophalen van de index (27.582 bestanden). + 9. `backup:target:remove` laat de index-cache van die locatie in de database staan; opruimen, + ook bestaande wezen. + **Daarna verplicht:** een tweede volledige restore op een verse VM, alleen met de HOWTO en de + nieuwe opties, zonder handwerk. Pas als die in één keer slaagt: 1.0.0. + - **Later (geen versie):** optie om de sleutel niet op de locatie op te slaan (kit is dan de enige kopie; uitdrukkelijke waarschuwing). Primaire object storage en External-storage-koppelingen meenemen via de opslaglaag (§3.4; nu niet ondersteund, zie README "Known limitations"). From 456dffc880cc2b2054f0989fd977894e2912ce96 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sat, 17 Oct 2026 21:33:15 +0200 Subject: [PATCH 04/19] Step-by-step disaster recovery guide --- README.md | 3 +- docs/DISASTER-RECOVERY.md | 88 +++++++++++++++++++++++++++++++++++++++ docs/HOWTO.md | 2 + 3 files changed, 92 insertions(+), 1 deletion(-) create mode 100644 docs/DISASTER-RECOVERY.md diff --git a/README.md b/README.md index f02644a..119d6fc 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,8 @@ ng_backup, and applies to any Nextcloud app that writes to an SMB share.) **Status: release candidate for 1.0.0 (1.0.0-rc.1, app version 0.9.7).** Tested thoroughly, usable alongside your current backup, but not yet a drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and -`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. +`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See +`docs/DISASTER-RECOVERY.md`. License: AGPL-3.0-or-later. Developed and sponsored by [Ensembia](https://www.ensembia.com). Need help restoring a backup after a crash? Ensembia offers paid restore assistance: diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md new file mode 100644 index 0000000..ba7ddf8 --- /dev/null +++ b/docs/DISASTER-RECOVERY.md @@ -0,0 +1,88 @@ +# Disaster recovery: your server is gone + +Your Nextcloud server is lost. You have a new, empty server. These are the steps to get your +Nextcloud back from an NG Backup location. Do them in this order. + +## What you need + +- [ ] The **recovery kit** file (`ng_backup-kit.json`) and **one passphrase** of it. +- [ ] The **connection details of the backup location**: host, user, password or keys, bucket, folder. +- [ ] A new server with the **same Nextcloud major version** as the old one (for example 34), + a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for + the whole backup plus a margin. +- [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by + hand). Keep their packages somewhere safe beforehand. + +No kit? If the location still holds its `config` file, the key is there too, but this version +can only import it from a kit. Keep the kit off the server, next to the passphrase. + +## Steps + +**1. Install a fresh Nextcloud** of the same major version. Create an admin account. Do not add +users or files. + +**2. Do not start background jobs yet.** Set the mode to cron but do not run cron yet: + + occ background:cron + +This keeps the new server quiet (no mail syncs, no notifications, no backups) until it is ready. + +**3. Install NG Backup** (the same version or newer) and enable it. + +**4. Import the backup key.** You type the passphrase yourself: + + occ backup:key:import /path/to/ng_backup-kit.json + +**5. Add the backup location** with the same details as before. This can take a while on a large +backup (the index is read first) and shows nothing until it is done. + + occ backup:target:add offsite sftp -a password::password -o host=... -o user=... -o password=... -o root=... --path=... + +If it says "No backup key yet", go back to step 4: do **not** run `backup:key:init`, that makes a +new key that cannot read your backups. + +**6. List the restore points** and pick one, normally the newest: + + occ backup:list offsite + +**7. Install the apps the old server had.** Run the restore; before it changes anything it checks +the apps and stops with what is missing: + + occ backup:restore:full offsite + +- "install these apps first: …": install each from the app store (`occ app:install `), or + enable it if it is part of Nextcloud but switched off (`occ app:enable `, for example + `admin_audit`). Apps that are not in the app store: copy them into `custom_apps` and enable them. +- "Table … does not exist here": an app that was removed or switched off long ago left tables + behind. Install that app (for example `deck`, `onlyoffice`), then run the restore again. + +Repeat until the restore starts. + +**8. Let the restore run.** It restores the database, then the data directory, and takes hours for +a large backup. If it stops halfway, fix the cause and run the same command again with `--force`. + +**9. After the restore** + +- Check `config/config.php`: `trusted_domains`, `overwrite.cli.url`, and the cache (Redis) host + must match the new server. The database settings and data directory are kept from the new + installation. +- `occ upgrade` (apps on the new server may be newer than in the backup), then + `occ maintenance:repair`. +- Log in, open some files, a calendar and a contact. +- Start background jobs (cron). + +**10. Point your domain at the new server** and check that clients reconnect. + +**11. Make a backup of the new server** to the same location (`occ backup:run offsite`). The old +server must stay off: two servers must never write to the same location. + +## Never + +- Never let the old server write to the location again if it comes back. Switch its backups off + first. +- Never give your passphrase to anyone. Nobody helping you needs it. + +## Help + +Stuck? [Ensembia](https://www.ensembia.com), the developer of NG Backup, offers paid help with +restoring a backup after a crash. diff --git a/docs/HOWTO.md b/docs/HOWTO.md index 3c97856..afccaec 100644 --- a/docs/HOWTO.md +++ b/docs/HOWTO.md @@ -136,6 +136,8 @@ occ backup:user:restore offsite users// --mode=as-backup ## 7. Disaster recovery: restore the whole server +Server lost? Follow the step-by-step [DISASTER-RECOVERY.md](DISASTER-RECOVERY.md). + Use a fresh Nextcloud installation of the same major version, with NG Backup installed and the same location available. For a test, use a **copy** of the location: two servers writing to one location break each other's catalog. From bbddc83f8016d16deb3f6455e4928f1ce0680b9d Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sat, 17 Oct 2026 21:38:55 +0200 Subject: [PATCH 05/19] DR guide: background jobs, copying the kit, why not key:init; PLAN: app list and table structure in the backup --- PLAN.md | 6 ++++++ docs/DISASTER-RECOVERY.md | 20 ++++++++++++++------ 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/PLAN.md b/PLAN.md index d42d141..385810a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -414,6 +414,12 @@ in README "Known limitations": 8. `backup:target:add` toont geen voortgang bij het ophalen van de index (27.582 bestanden). 9. `backup:target:remove` laat de index-cache van die locatie in de database staan; opruimen, ook bestaande wezen. + 10. Snapshot-meta uitbreiden: ook geïnstalleerde maar uitgeschakelde apps, met per app of hij + uit de app store komt; `occ backup:apps ` toont de lijst leesbaar + (geen platte tekst op de locatie: dat lekt welke apps/versies er draaien). + 11. **Tabelstructuur meedumpen** (kolommen, typen, indexen, primary key), zodat de restore een + ontbrekende tabel zelf aanmaakt met de gegevens erin. Lost restanten van oude apps op zonder + ze over te slaan; een later opnieuw geïnstalleerde app vindt zijn oude gegevens terug. **Daarna verplicht:** een tweede volledige restore op een verse VM, alleen met de HOWTO en de nieuwe opties, zonder handwerk. Pas als die in één keer slaagt: 1.0.0. diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md index ba7ddf8..008a9f3 100644 --- a/docs/DISASTER-RECOVERY.md +++ b/docs/DISASTER-RECOVERY.md @@ -21,25 +21,33 @@ can only import it from a kit. Keep the kit off the server, next to the passphra **1. Install a fresh Nextcloud** of the same major version. Create an admin account. Do not add users or files. -**2. Do not start background jobs yet.** Set the mode to cron but do not run cron yet: +**2. Do not let background jobs run yet.** A new Nextcloud runs them on page visits (AJAX). +Switch that off: occ background:cron -This keeps the new server quiet (no mail syncs, no notifications, no backups) until it is ready. +As long as no cron is set up, nothing runs. If your setup has a cron service or container (Docker, +All-in-One), stop it until step 9. This keeps the new server quiet (no mail syncs, no +notifications, no backups) until it is ready. **3. Install NG Backup** (the same version or newer) and enable it. -**4. Import the backup key.** You type the passphrase yourself: +**4. Import the backup key.** Copy the kit to the new server, readable for the web server user +(for example `scp ng_backup-kit.json newserver:/tmp/` and `chown www-data /tmp/ng_backup-kit.json`). +Then import it; you type the passphrase yourself: - occ backup:key:import /path/to/ng_backup-kit.json + occ backup:key:import /tmp/ng_backup-kit.json + +It shows the key fingerprint: it must match the one of your old server (on the recovery kit). **5. Add the backup location** with the same details as before. This can take a while on a large backup (the index is read first) and shows nothing until it is done. occ backup:target:add offsite sftp -a password::password -o host=... -o user=... -o password=... -o root=... --path=... -If it says "No backup key yet", go back to step 4: do **not** run `backup:key:init`, that makes a -new key that cannot read your backups. +If it says "No backup key yet", go back to step 4. Do **not** run `backup:key:init`: that makes a +brand new key, and your backups were encrypted with the old one. NG Backup then refuses the +location ("created with a different key") and you still need the kit. **6. List the restore points** and pick one, normally the newest: From 523620acde1002f3bc96370d17bfe8d069066fcb Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sat, 17 Oct 2026 21:48:31 +0200 Subject: [PATCH 06/19] DR guide: concrete commands for background jobs, copying the kit and the key import --- docs/DISASTER-RECOVERY.md | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md index 008a9f3..dd3658d 100644 --- a/docs/DISASTER-RECOVERY.md +++ b/docs/DISASTER-RECOVERY.md @@ -26,15 +26,31 @@ Switch that off: occ background:cron -As long as no cron is set up, nothing runs. If your setup has a cron service or container (Docker, -All-in-One), stop it until step 9. This keeps the new server quiet (no mail syncs, no -notifications, no backups) until it is ready. +As long as no cron is set up, nothing runs. Check that none is: + +- Regular server: `sudo crontab -u www-data -l`. If it shows a line with `cron.php`, remove it + for now with `sudo crontab -u www-data -e` (put it back in step 9). +- Docker Compose with a cron service: `docker compose stop cron`. +- Nextcloud All-in-One: cron runs inside the Nextcloud container; leave it. It does no harm here: + the restore keeps NG Backup's own settings of the new server, so it does not start backups. + +This keeps the new server quiet (no mail syncs, no notifications, no backups) until it is ready. **3. Install NG Backup** (the same version or newer) and enable it. -**4. Import the backup key.** Copy the kit to the new server, readable for the web server user -(for example `scp ng_backup-kit.json newserver:/tmp/` and `chown www-data /tmp/ng_backup-kit.json`). -Then import it; you type the passphrase yourself: +**4. Import the backup key.** Copy the kit to the new server so the web server user can read it. + +From your own computer: + + scp ng_backup-kit.json root@newserver:/tmp/ + +Then on the server, one of: + + chown www-data /tmp/ng_backup-kit.json # regular server + docker cp /tmp/ng_backup-kit.json nextcloud:/tmp/ && \ + docker exec nextcloud chown www-data /tmp/ng_backup-kit.json # Docker + +Import it; you type the passphrase yourself: occ backup:key:import /tmp/ng_backup-kit.json @@ -45,9 +61,13 @@ backup (the index is read first) and shows nothing until it is done. occ backup:target:add offsite sftp -a password::password -o host=... -o user=... -o password=... -o root=... --path=... -If it says "No backup key yet", go back to step 4. Do **not** run `backup:key:init`: that makes a -brand new key, and your backups were encrypted with the old one. NG Backup then refuses the -location ("created with a different key") and you still need the kit. +If it says "No backup key yet", the key is not imported. Run step 4: + + occ backup:key:import /tmp/ng_backup-kit.json + +Do **not** run `backup:key:init`, even though the message suggests it: that makes a brand new key, +and your backups were encrypted with the old one. NG Backup then refuses the location ("created +with a different key"). **6. List the restore points** and pick one, normally the newest: From f633de59b4c0b40f37d411de27b39a3b3908a201 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sat, 17 Oct 2026 23:15:13 +0200 Subject: [PATCH 07/19] PLAN: foreign keys block the database restore (Mail) --- PLAN.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/PLAN.md b/PLAN.md index 385810a..e7a2314 100644 --- a/PLAN.md +++ b/PLAN.md @@ -420,6 +420,11 @@ in README "Known limitations": 11. **Tabelstructuur meedumpen** (kolommen, typen, indexen, primary key), zodat de restore een ontbrekende tabel zelf aanmaakt met de gegevens erin. Lost restanten van oude apps op zonder ze over te slaan; een later opnieuw geïnstalleerde app vindt zijn oude gegevens terug. + 12. **Foreign keys**: de restore zet tabellen op alfabet terug; `oc_mail_accounts` verwijst naar + `oc_mail_mailboxes` en PostgreSQL weigert (fk_61a7e87f59a77cdc). Iedereen met Mail loopt hier + vast. Oplossing: in de juiste volgorde terugzetten (afhankelijkheden eerst), of de controle + tijdelijk uitzetten en na afloop terugzetten en controleren. In de test: 15 FK's (alle van + Mail) tijdelijk verwijderd en na de restore teruggezet. **Daarna verplicht:** een tweede volledige restore op een verse VM, alleen met de HOWTO en de nieuwe opties, zonder handwerk. Pas als die in één keer slaagt: 1.0.0. From 88477a1a9fc27dca5df59f3b6e562fd7eef64c2d Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 00:59:06 +0200 Subject: [PATCH 08/19] PLAN: SFTP restore speed over long distances; foreign keys valid after the restore --- PLAN.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/PLAN.md b/PLAN.md index e7a2314..6a92cb7 100644 --- a/PLAN.md +++ b/PLAN.md @@ -424,7 +424,12 @@ in README "Known limitations": `oc_mail_mailboxes` en PostgreSQL weigert (fk_61a7e87f59a77cdc). Iedereen met Mail loopt hier vast. Oplossing: in de juiste volgorde terugzetten (afhankelijkheden eerst), of de controle tijdelijk uitzetten en na afloop terugzetten en controleren. In de test: 15 FK's (alle van - Mail) tijdelijk verwijderd en na de restore teruggezet. + Mail) tijdelijk verwijderd en na de restore teruggezet: alle 15 geldig op de teruggezette + data, dus "uitzetten, terugzetten, achteraf controleren" werkt. + 13. **SFTP-restore over een lange lijn is traag**: 0,75 MiB/s vanaf Hetzner naar Canada (RTT 122 ms), + terwijl OpenSSH `sftp` 16,6 MiB/s haalt. De SFTP-stream van files_external (phpseclib) wacht na + elk klein blok op antwoord. Oplossing: pack-ranges bij SFTP met phpseclib's eigen `get()` + (veel verzoeken tegelijk) lezen in plaats van via de stream, zoals bij S3. 1 TB duurt nu ~2 weken. **Daarna verplicht:** een tweede volledige restore op een verse VM, alleen met de HOWTO en de nieuwe opties, zonder handwerk. Pas als die in één keer slaagt: 1.0.0. From 0924da48ac0a43a8d423ba9a7da8dbdc02ae5a46 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 01:15:34 +0200 Subject: [PATCH 09/19] PLAN: upload speed over long distances; DR guide container name --- PLAN.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/PLAN.md b/PLAN.md index 6a92cb7..bfd91de 100644 --- a/PLAN.md +++ b/PLAN.md @@ -430,6 +430,10 @@ in README "Known limitations": terwijl OpenSSH `sftp` 16,6 MiB/s haalt. De SFTP-stream van files_external (phpseclib) wacht na elk klein blok op antwoord. Oplossing: pack-ranges bij SFTP met phpseclib's eigen `get()` (veel verzoeken tegelijk) lezen in plaats van via de stream, zoals bij S3. 1 TB duurt nu ~2 weken. + 14. **Uploaden over een lange lijn**: backup naar OVH S3 Warschau vanaf Canada (RTT 114 ms) haalt + ~1 MiB/s met één pack tegelijk; `aws s3 cp` (multipart, parallel) 6 MiB/s. Meerdere packs tegelijk + uploaden of S3 multipart. Minder dringend dan 13 (restore). + 15. DR-document: in het Docker-voorbeeld "je containernaam" (`docker ps`) i.p.v. `nextcloud`. **Daarna verplicht:** een tweede volledige restore op een verse VM, alleen met de HOWTO en de nieuwe opties, zonder handwerk. Pas als die in één keer slaagt: 1.0.0. From 01e16199f7c0583be0d7c0e3215e80134fe2b9f8 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 01:15:43 +0200 Subject: [PATCH 10/19] DR guide: use your own container name --- docs/DISASTER-RECOVERY.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md index dd3658d..05f6226 100644 --- a/docs/DISASTER-RECOVERY.md +++ b/docs/DISASTER-RECOVERY.md @@ -47,8 +47,11 @@ From your own computer: Then on the server, one of: chown www-data /tmp/ng_backup-kit.json # regular server - docker cp /tmp/ng_backup-kit.json nextcloud:/tmp/ && \ - docker exec nextcloud chown www-data /tmp/ng_backup-kit.json # Docker + docker cp /tmp/ng_backup-kit.json :/tmp/ && \ + docker exec chown www-data /tmp/ng_backup-kit.json # Docker + +`` is the name of your Nextcloud container: `docker ps` shows it (for example +`nextcloud-aio-nextcloud`, or `myproject-nextcloud-1` with Docker Compose). Import it; you type the passphrase yourself: From 5af712acc362dc652c78e5abb8cba56d5642dafb Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 01:26:27 +0200 Subject: [PATCH 11/19] README: BBR for backups over a long distance --- PLAN.md | 3 +++ README.md | 7 +++++++ 2 files changed, 10 insertions(+) diff --git a/PLAN.md b/PLAN.md index bfd91de..4e768c4 100644 --- a/PLAN.md +++ b/PLAN.md @@ -433,6 +433,9 @@ in README "Known limitations": 14. **Uploaden over een lange lijn**: backup naar OVH S3 Warschau vanaf Canada (RTT 114 ms) haalt ~1 MiB/s met één pack tegelijk; `aws s3 cp` (multipart, parallel) 6 MiB/s. Meerdere packs tegelijk uploaden of S3 multipart. Minder dringend dan 13 (restore). + Oorzaak netwerk (met Sysop): pakketverlies trans-Atlantisch + cubic in de VM; met BBR gaat één + PUT van 2,4 naar 7,5 MiB/s, ng_backup van ~1 naar ~1,6 MiB/s (rest = serieel lezen/uploaden). + Tip over BBR in de README gezet. Let op: PVE02 staat in Canada; een restore in de EU gaat sneller. 15. DR-document: in het Docker-voorbeeld "je containernaam" (`docker ps`) i.p.v. `nextcloud`. **Daarna verplicht:** een tweede volledige restore op een verse VM, alleen met de HOWTO en de nieuwe opties, zonder handwerk. Pas als die in één keer slaagt: 1.0.0. diff --git a/README.md b/README.md index 119d6fc..f2a3a45 100644 --- a/README.md +++ b/README.md @@ -175,6 +175,13 @@ Contributions welcome via PR — add what you ran into with your own provider. the backend being listed is not proof it will work here. Once the extension is present it works the same as any other backend; no ng_backup-specific config beyond host/share/user/password. +- **Backups over a long distance (high latency):** on Linux, the default TCP congestion control + (cubic) slows a single connection down a lot on a long line with some packet loss. From Canada to + an S3 bucket in Warsaw (114 ms), one upload went from 2.4 to 7.5 MiB/s after switching the + server (or the VM/container host where TCP ends) to BBR: + `sysctl -w net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr` + (`modprobe tcp_bbr` first; make it permanent in `/etc/sysctl.d/`). + - **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself (e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping From d8ef50732f0be2a1c193e14daed14a9e890f2ce5 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 01:45:01 +0200 Subject: [PATCH 12/19] Full restore creates missing tables and suspends foreign keys; --install-apps; clearer messages Backups now record each table's structure in the dump manifest. The full restore creates tables that do not exist on the new instance (left behind by apps that were removed or are switched off), lists all problems at once, and no longer stops on apps that are missing. Foreign keys are not checked while rows are loaded in name order, and checked again before the restore commits (PostgreSQL: dropped and added back in the transaction). --install-apps installs or enables the apps that were enabled at backup time; --skip-missing-tables leaves out tables of older backups whose structure is unknown. nextcloud-aio is never required. backup:target:add without a key points to backup:key:import, and says that connecting can take a while. Removing a location drops its cached index; a migration removes the cache of locations that are gone. The full-restore test now also runs on PostgreSQL with Mail (foreign keys) and a disabled app. --- .github/workflows/ci.yml | 2 + PLAN.md | 3 + lib/Command/RestoreFull.php | 54 +++++- lib/Command/TargetAdd.php | 8 +- lib/Db/DbDumper.php | 155 ++++++++++++++++-- lib/Db/DbIndexCache.php | 7 + .../Version000006Date20261018000000.php | 39 +++++ lib/Service/RestoreFullService.php | 53 +++--- lib/Service/TargetService.php | 5 + psalm-baseline.xml | 13 +- tests/integration/full-restore-roundtrip.sh | 66 +++++++- 11 files changed, 359 insertions(+), 46 deletions(-) create mode 100644 lib/Migration/Version000006Date20261018000000.php diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ebfa4a2..7b7fb3e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -104,3 +104,5 @@ jobs: persist-credentials: false - name: Full restore round trip run: tests/integration/full-restore-roundtrip.sh 35-apache + - name: Full restore round trip on PostgreSQL, with Mail (foreign keys) and a disabled app + run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql diff --git a/PLAN.md b/PLAN.md index 4e768c4..f69b0ba 100644 --- a/PLAN.md +++ b/PLAN.md @@ -430,6 +430,9 @@ in README "Known limitations": terwijl OpenSSH `sftp` 16,6 MiB/s haalt. De SFTP-stream van files_external (phpseclib) wacht na elk klein blok op antwoord. Oplossing: pack-ranges bij SFTP met phpseclib's eigen `get()` (veel verzoeken tegelijk) lezen in plaats van via de stream, zoals bij S3. 1 TB duurt nu ~2 weken. + Na meten: de grootste oorzaak was het netwerk (cubic, zie 14): met BBR gaat de restore van 0,75 + naar 2,3 MiB/s, een hele pack lezen 4-5 MiB/s. Meer verzoeken in één stream (DEPTH 16/32) gaf geen + duidelijke winst. Verder komen vraagt meerdere verbindingen tegelijk: naar **1.1.0**. 14. **Uploaden over een lange lijn**: backup naar OVH S3 Warschau vanaf Canada (RTT 114 ms) haalt ~1 MiB/s met één pack tegelijk; `aws s3 cp` (multipart, parallel) 6 MiB/s. Meerdere packs tegelijk uploaden of S3 multipart. Minder dringend dan 13 (restore). diff --git a/lib/Command/RestoreFull.php b/lib/Command/RestoreFull.php index 524fe63..777582d 100644 --- a/lib/Command/RestoreFull.php +++ b/lib/Command/RestoreFull.php @@ -23,6 +23,7 @@ final class RestoreFull extends Command { private TargetService $targets, private RestoreFullService $restoreFull, private IUserManager $userManager, + private \OCP\App\IAppManager $appManager, ) { parent::__construct(); } @@ -33,14 +34,19 @@ final class RestoreFull extends Command { ->addArgument('target', InputArgument::REQUIRED, 'Location name or id') ->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)') ->addOption('force', null, InputOption::VALUE_NONE, 'Also on an installation that already has more than one user, and without asking') + ->addOption('install-apps', null, InputOption::VALUE_NONE, 'First install (from the app store) or enable the apps that were enabled at backup time and are missing here') + ->addOption('skip-missing-tables', null, InputOption::VALUE_NONE, 'Older backups only: leave out tables that do not exist here and whose structure the backup does not record') ->setHelp(<<<'HELP' For a FRESH Nextcloud installation only, same major version as the backup, right after occ maintenance:install, occ backup:key:import and occ backup:target:add -- before anyone else uses this instance. -Every app that was enabled when the snapshot was taken must already be installed here (its -database tables must exist before the restore can write into them); the command lists what is -missing and refuses to continue rather than guess. +Apps: --install-apps first installs (from the app store) or enables every app that was enabled +when the snapshot was taken. Apps that are not in the app store (own builds) must be copied into +custom_apps by hand; the command lists what is still missing. Tables of apps that are missing or +were removed long ago are created from the structure recorded in the backup (backups made with +1.0.0-rc.2 or later), so they do not block the restore. Older backups do not record it: the +command then lists those tables per app, and --skip-missing-tables leaves them out. Steps: the database is replaced (consistent, in one transaction); the data directory is restored directly in place; config.php is merged in -- everything from the backup except the database @@ -75,7 +81,14 @@ HELP); try { $target = $this->targets->get($input->getArgument('target')); $snapshot = (string)$input->getArgument('snapshot'); - $result = $this->restoreFull->restoreFull($target, $snapshot); + $skipMissingTables = (bool)$input->getOption('skip-missing-tables'); + if ($input->getOption('install-apps')) { + $check = $this->restoreFull->checkCompatibility($target, $snapshot, $skipMissingTables); + foreach ($check['missingApps'] as $app) { + $this->installApp($app, $output); + } + } + $result = $this->restoreFull->restoreFull($target, $snapshot, $skipMissingTables); } catch (\Throwable $e) { $output->writeln('' . $e->getMessage() . ''); $output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.'); @@ -88,10 +101,43 @@ HELP); $result['configKeys'], $result['dataFiles'], )); + if ($result['dbReport']['created'] !== []) { + $output->writeln(sprintf('Created %d tables that did not exist here (apps that are missing or were removed): %s', + count($result['dbReport']['created']), implode(', ', $result['dbReport']['created']))); + } + if ($result['dbReport']['missing'] !== []) { + $output->writeln('Left out (structure unknown): ' . implode(', ', $result['dbReport']['missing']) . ''); + } + if ($result['missingApps'] !== []) { + $output->writeln('These apps were enabled on the old server but are not installed here. Their data is restored; install them so they work again:'); + foreach ($result['missingApps'] as $app) { + $output->writeln(" occ app:install $app (not in the app store? copy it into custom_apps and run occ app:enable $app)"); + } + } if (!$result['dbConsistent']) { $output->writeln('Warning: this snapshot\'s database dump was made over several steps (web cron), so it is not one consistent point in time. Check the instance carefully; prefer a snapshot made with system cron.'); } $output->writeln('Next: check trusted_domains and overwritehost/overwriteprotocol match this server, then occ maintenance:repair.'); return 0; } + + /** Enable an app that is here but switched off, or install it from the app store. */ + private function installApp(string $app, OutputInterface $output): void { + try { + $this->appManager->getAppPath($app); + $command = 'app:enable'; // here, but switched off + } catch (\OCP\App\AppPathNotFoundException) { + $command = 'app:install'; + } + $output->writeln("Running occ $command $app ..."); + try { + $code = $this->getApplication()?->find($command)->run(new \Symfony\Component\Console\Input\ArrayInput(['app-id' => $command === 'app:enable' ? [$app] : $app]), $output); + } catch (\Throwable $e) { + $code = 1; + $output->writeln('' . $e->getMessage() . ''); + } + if ($code !== 0) { + $output->writeln("Could not $command $app (not in the app store?). Copy it into custom_apps and enable it, or restore without it."); + } + } } diff --git a/lib/Command/TargetAdd.php b/lib/Command/TargetAdd.php index 2dc9337..d91368c 100644 --- a/lib/Command/TargetAdd.php +++ b/lib/Command/TargetAdd.php @@ -44,7 +44,10 @@ HELP); protected function execute(InputInterface $input, OutputInterface $output): int { if (!$this->keys->isInitialized()) { - $output->writeln('No backup key yet: run occ backup:key:init first.'); + $output->writeln('No backup key yet.'); + $output->writeln('Restoring from a location that already has backups (disaster recovery)? Import the key from the recovery kit first:'); + $output->writeln(' occ backup:key:import /path/to/ng_backup-kit.json'); + $output->writeln('Only for a new location without backups: create a new key with occ backup:key:init. A new key cannot read existing backups.'); return 1; } $options = []; @@ -60,6 +63,9 @@ HELP); $output->writeln('--auth is required (see occ backup:target:backends)'); return 1; } + // Adopting an existing repository reads its whole index first, which takes a while on a + // large backup or a slow line; say so instead of looking stuck. + $output->writeln('Connecting to the location; with existing backups this reads their index first and can take a while ...'); try { $r = $this->targets->add($input->getArgument('name'), $input->getArgument('backend'), $input->getOption('auth'), $options, (string)$input->getOption('path')); } catch (\Throwable $e) { diff --git a/lib/Db/DbDumper.php b/lib/Db/DbDumper.php index 1010cdd..0f27e32 100644 --- a/lib/Db/DbDumper.php +++ b/lib/Db/DbDumper.php @@ -23,8 +23,12 @@ use OCA\NgBackup\Repository\Repository; * sets, so a single big SELECT would not stream). * - Each table becomes its own blob stream of JSON lines: a header line, then one JSON array per * row. A table that did not change produces the same blobs and costs nothing in the next run. - * - The schema itself is not dumped: it comes from Nextcloud's migrations on the instance that is - * restored into (same Nextcloud and app versions). The dump records those versions. + * - The structure of each table (columns, types, primary key, indexes) is recorded in the manifest, + * so a restore can create a table that does not exist on the new instance, for example one left + * behind by an app that was removed long ago. Tables that exist keep the structure the new + * instance's migrations gave them. + * - Foreign keys are not checked while rows are loaded (tables are restored in name order, not in + * dependency order), and are checked again before the restore commits. * * Phase 0 uses the Doctrine connection behind IDBConnection (private API) for schema * introspection; to be replaced by information_schema queries or a public API. @@ -108,7 +112,7 @@ final class DbDumper { break; } $state['result'][$table] = ['blobs' => array_merge(...$state['segments']), 'rows' => $state['rows'], - 'bytes' => $state['bytes'], 'sha256' => hash_final($hash)]; + 'bytes' => $state['bytes'], 'sha256' => hash_final($hash), 'schema' => self::structure($schema)]; $state = array_merge($state, ['i' => $state['i'] + 1, 'after' => null, 'segments' => [], 'rows' => 0, 'bytes' => 0, 'hash' => null]); if (microtime(true) >= $deadline && $state['i'] < count($state['tables'])) { break; @@ -137,18 +141,33 @@ final class DbDumper { * Replace the contents of every table in the dump. Tables that exist but are not in the dump * are left alone (e.g. a newer app); the caller decides whether that is acceptable. * - * @param array{tables: array, rows: int}>} $manifest + * @param array{tables: array, rows: int, sha256: string, schema?: array}>, sequences?: array} $manifest */ - public function restore(Repository $repo, array $manifest, ?array &$report = null, ?callable $heartbeat = null): int { - $sm = $this->db->createSchemaManager(); - $existing = array_flip($sm->listTableNames()); + public function restore(Repository $repo, array $manifest, ?array &$report = null, ?callable $heartbeat = null, bool $skipMissing = false): int { + $report = ['skipped' => [], 'restored' => [], 'created' => [], 'missing' => []]; + $missing = $this->missingTables($manifest); + $unknown = array_values(array_filter($missing, fn (string $t) => !isset($manifest['tables'][$t]['schema']))); + if ($unknown !== [] && !$skipMissing) { + throw new \RuntimeException(self::describeMissing($unknown, $this->prefix)); + } + // Tables the backup knows the structure of are created first (DDL is not transactional + // everywhere); tables it does not know are left out, as asked. + foreach ($missing as $table) { + if (isset($manifest['tables'][$table]['schema'])) { + $this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'])); + $report['created'][] = $table; + } else { + $report['missing'][] = $table; + } + } $total = 0; - $report = ['skipped' => [], 'restored' => []]; + $foreignKeys = []; $this->db->beginTransaction(); try { + $foreignKeys = $this->suspendForeignKeys(); foreach ($manifest['tables'] as $table => $info) { - if (!isset($existing[$table])) { - throw new \RuntimeException("Table $table from the backup does not exist here (different app versions?)"); + if (in_array($table, $report['missing'], true)) { + continue; } // Unchanged since the backup: leave it alone (much faster than delete + insert). $live = $this->checksum($table); @@ -193,14 +212,130 @@ final class DbDumper { $report['restored'][] = $table; } $this->restoreSequences($manifest['sequences'] ?? []); + $this->resumeForeignKeys($foreignKeys); $this->db->commit(); } catch (\Throwable $e) { $this->db->rollBack(); + // PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session + // switch must be turned on again by hand. + if (in_array('SET FOREIGN_KEY_CHECKS = 1', $foreignKeys, true)) { + $this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 1'); + } throw $e; } return $total; } + /** + * Tables in the dump that do not exist on this instance. + * + * @return list + */ + public function missingTables(array $manifest): array { + $existing = array_flip($this->db->createSchemaManager()->listTableNames()); + return array_values(array_filter(array_keys($manifest['tables']), fn (string $t) => !isset($existing[$t]))); + } + + /** One message for all missing tables of an older backup, grouped by the app they probably belong to. */ + public static function describeMissing(array $tables, string $prefix): string { + $groups = []; + foreach ($tables as $t) { + $name = substr($t, strlen($prefix)); + $groups[explode('_', $name, 2)[0]][] = $t; + } + $lines = []; + foreach ($groups as $app => $list) { + $lines[] = '- ' . $app . ': ' . implode(', ', $list); + } + return "These tables from the backup do not exist here, and this backup does not record their structure (made before 1.0.0-rc.2):\n" + . implode("\n", $lines) + . "\nInstall the apps they belong to (often apps that were removed or switched off long ago) and run the restore again, or leave these tables out with --skip-missing-tables."; + } + + /** + * Portable description of a table: enough to create it again, also on another database type. + * + * @return array{columns: list>, pk: list, indexes: list, unique: bool}>} + */ + private static function structure(\Doctrine\DBAL\Schema\Table $table): array { + $columns = []; + foreach ($table->getColumns() as $c) { + $columns[] = ['name' => $c->getName(), 'type' => \Doctrine\DBAL\Types\Type::lookupName($c->getType()), + 'length' => $c->getLength(), 'precision' => $c->getPrecision(), 'scale' => $c->getScale(), + 'unsigned' => $c->getUnsigned(), 'fixed' => $c->getFixed(), 'notnull' => $c->getNotnull(), + 'default' => $c->getDefault(), 'autoincrement' => $c->getAutoincrement()]; + } + $indexes = []; + foreach ($table->getIndexes() as $i) { + if (!$i->isPrimary()) { + $indexes[] = ['name' => $i->getName(), 'columns' => array_values($i->getColumns()), 'unique' => $i->isUnique()]; + } + } + return ['columns' => $columns, 'pk' => array_values($table->getPrimaryKey()?->getColumns() ?? []), 'indexes' => $indexes]; + } + + private static function tableFromStructure(string $name, array $structure): \Doctrine\DBAL\Schema\Table { + $table = new \Doctrine\DBAL\Schema\Table($name); + foreach ($structure['columns'] as $c) { + $table->addColumn($c['name'], $c['type'], array_filter([ + 'length' => $c['length'], 'precision' => $c['precision'], 'scale' => $c['scale'], + 'unsigned' => $c['unsigned'], 'fixed' => $c['fixed'], 'notnull' => $c['notnull'], + 'default' => $c['default'], 'autoincrement' => $c['autoincrement'], + ], fn ($v) => $v !== null)); + } + if ($structure['pk'] !== []) { + $table->setPrimaryKey($structure['pk']); + } + foreach ($structure['indexes'] as $i) { + $i['unique'] ? $table->addUniqueIndex($i['columns'], $i['name']) : $table->addIndex($i['columns'], $i['name']); + } + return $table; + } + + /** + * Stop checking foreign keys while rows are loaded in name order. PostgreSQL has no session + * switch for this without superuser rights, so its foreign keys of Nextcloud tables are dropped + * here and added back (and so checked) by resumeForeignKeys(), inside the same transaction. + * + * @return list statements that restore the foreign keys + */ + private function suspendForeignKeys(): array { + $platform = $this->db->getDatabasePlatform(); + if ($platform instanceof \Doctrine\DBAL\Platforms\PostgreSQLPlatform) { + $rows = $this->db->fetchAllAssociative( + "SELECT conrelid::regclass::text AS tbl, conname, pg_get_constraintdef(oid) AS def + FROM pg_constraint WHERE contype = 'f' AND connamespace = current_schema()::regnamespace"); + $restore = []; + foreach ($rows as $r) { + if (!str_starts_with(trim($r['tbl'], '"'), $this->prefix)) { + continue; + } + $this->db->executeStatement(sprintf('ALTER TABLE %s DROP CONSTRAINT %s', $r['tbl'], $this->db->quoteIdentifier($r['conname']))); + $restore[] = sprintf('ALTER TABLE %s ADD CONSTRAINT %s %s', $r['tbl'], $this->db->quoteIdentifier($r['conname']), $r['def']); + } + return $restore; + } + if ($platform instanceof \Doctrine\DBAL\Platforms\AbstractMySQLPlatform) { + $this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 0'); + return ['SET FOREIGN_KEY_CHECKS = 1']; + } + if ($platform instanceof \Doctrine\DBAL\Platforms\SqlitePlatform) { + $this->db->executeStatement('PRAGMA defer_foreign_keys = ON'); // checked at commit + } + return []; + } + + /** Add the foreign keys back; a row that breaks one fails the restore here, before it commits. */ + private function resumeForeignKeys(array $statements): void { + foreach ($statements as $sql) { + try { + $this->db->executeStatement($sql); + } catch (\Throwable $e) { + throw new \RuntimeException('The restored data breaks a foreign key (' . $sql . '): ' . $e->getMessage(), 0, $e); + } + } + } + /** Same checksum as dump(): sha256 over the row lines in primary-key order. */ public function checksum(string $table): array { $hash = hash_init('sha256'); diff --git a/lib/Db/DbIndexCache.php b/lib/Db/DbIndexCache.php index e70ccbb..d7411e9 100644 --- a/lib/Db/DbIndexCache.php +++ b/lib/Db/DbIndexCache.php @@ -64,6 +64,13 @@ final class DbIndexCache implements IIndexCache { } } + /** Drop the cached index of a repository that no location uses any more. */ + public function forget(string $repositoryId): void { + $qb = $this->db->getQueryBuilder(); + $qb->delete('ngb_index')->where($qb->expr()->eq('repository_id', $qb->createNamedParameter($repositoryId))); + $qb->executeStatement(); + } + /** All or nothing: when it fails, the cache stays as it was. */ public function replaceAll(string $repositoryId, array $packs): void { $this->transaction(function () use ($repositoryId, $packs): void { diff --git a/lib/Migration/Version000006Date20261018000000.php b/lib/Migration/Version000006Date20261018000000.php new file mode 100644 index 0000000..8e90047 --- /dev/null +++ b/lib/Migration/Version000006Date20261018000000.php @@ -0,0 +1,39 @@ +db->getQueryBuilder(); + $used->select('repository_id')->from('ngb_targets')->where($used->expr()->isNotNull('repository_id')); + $keep = array_column($used->executeQuery()->fetchAll(), 'repository_id'); + $delete = $this->db->getQueryBuilder(); + $delete->delete('ngb_index'); + if ($keep !== []) { + $delete->where($delete->expr()->notIn('repository_id', $delete->createNamedParameter($keep, \OCP\DB\QueryBuilder\IQueryBuilder::PARAM_STR_ARRAY))); + } + $removed = $delete->executeStatement(); + if ($removed > 0) { + $output->info("Removed $removed cached index rows of locations that no longer exist"); + } + } +} diff --git a/lib/Service/RestoreFullService.php b/lib/Service/RestoreFullService.php index 1d429ed..467f595 100644 --- a/lib/Service/RestoreFullService.php +++ b/lib/Service/RestoreFullService.php @@ -43,34 +43,49 @@ final class RestoreFullService { ) { } + /** Apps that only exist in a particular setup and are never needed elsewhere. */ + private const ENVIRONMENT_APPS = ['nextcloud-aio']; + /** - * Whether this installation can receive $snapshotId: same Nextcloud major version, and every - * app that was enabled at backup time already installed here (its tables must exist before - * the database restore can write into them). + * Whether this installation can receive $snapshotId, all problems at once: + * - the same Nextcloud major version is required; + * - apps that were enabled at backup time but are not here (they work again once installed); + * - tables in the backup that do not exist here: a backup made from 1.0.0-rc.2 on records + * their structure and the restore creates them, an older one cannot (missingTables). * - * @return array{ok:bool, currentMajor:string, snapshotMajor:string, missingApps:list} + * @return array{ok:bool, currentMajor:string, snapshotMajor:string, missingApps:list, missingTables:list} */ - public function checkCompatibility(Target $target, string $snapshotId): array { - $meta = $this->targets->repository($target)->snapshotMeta($snapshotId); + public function checkCompatibility(Target $target, string $snapshotId, bool $skipMissingTables = false): array { + $repo = $this->targets->repository($target); + $meta = $repo->snapshotMeta($snapshotId); $currentMajor = explode('.', $this->config->getSystemValueString('version'))[0]; $snapshotMajor = explode('.', (string)($meta['nextcloud'] ?? ''))[0]; $enabled = array_flip($this->appManager->getEnabledApps()); - $missingApps = array_values(array_diff(array_keys($meta['apps'] ?? []), array_keys($enabled))); + $missingApps = array_values(array_diff(array_keys($meta['apps'] ?? []), array_keys($enabled), self::ENVIRONMENT_APPS)); + $missingTables = []; + if (isset($meta['db'])) { + $manifest = json_decode($repo->getObject($meta['db']), true, 512, JSON_THROW_ON_ERROR); + $prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_'); + $dumper = new DbDumper($this->db->getInner(), $prefix); + $missingTables = array_values(array_filter($dumper->missingTables($manifest), + fn (string $t) => !isset($manifest['tables'][$t]['schema']) && !str_starts_with($t, $prefix . 'ngb_'))); + } return [ - 'ok' => $currentMajor === $snapshotMajor && $missingApps === [], + 'ok' => $currentMajor === $snapshotMajor && ($missingTables === [] || $skipMissingTables), 'currentMajor' => $currentMajor, 'snapshotMajor' => $snapshotMajor, 'missingApps' => $missingApps, + 'missingTables' => $missingTables, ]; } - /** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool} */ - public function restoreFull(Target $target, string $snapshotId): array { - $check = $this->checkCompatibility($target, $snapshotId); + /** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list} */ + public function restoreFull(Target $target, string $snapshotId, bool $skipMissingTables = false): array { + $check = $this->checkCompatibility($target, $snapshotId, $skipMissingTables); if (!$check['ok']) { throw new \RuntimeException($this->describeBlockers($check)); } - return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId) { + return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $skipMissingTables, $check) { $beat = LeaseService::throttled($refresh); $repo = $this->targets->repository($target); $meta = $repo->snapshotMeta($snapshotId); @@ -102,7 +117,7 @@ final class RestoreFullService { $ownAppConfig = $this->preserveOwnAppConfig(); $dumper = new DbDumper($this->db->getInner(), $prefix); try { - $dumper->restore($repo, $manifest, $dbReport, $beat); + $dumper->restore($repo, $manifest, $dbReport, $beat, $skipMissingTables); } finally { $this->restoreOwnAppConfig($ownAppConfig); } @@ -125,8 +140,8 @@ final class RestoreFullService { ); } - return ['dbReport' => $dbReport ?? ['skipped' => [], 'restored' => []], 'dataFiles' => $dataFiles, 'configKeys' => count($backupConfig), - 'dbConsistent' => (bool)($manifest['consistent'] ?? true)]; + return ['dbReport' => $dbReport ?? ['skipped' => [], 'restored' => [], 'created' => [], 'missing' => []], 'dataFiles' => $dataFiles, 'configKeys' => count($backupConfig), + 'dbConsistent' => (bool)($manifest['consistent'] ?? true), 'missingApps' => $check['missingApps']]; }); } @@ -209,11 +224,11 @@ final class RestoreFullService { if ($check['currentMajor'] !== $check['snapshotMajor']) { $parts[] = "this installation is Nextcloud {$check['currentMajor']}, the backup is from {$check['snapshotMajor']} (same major version required)"; } - if ($check['missingApps'] !== []) { - $parts[] = 'install these apps first (their tables must exist before the database restore can use them): ' - . implode(', ', array_map(fn (string $a) => "occ app:install $a", $check['missingApps'])); + if ($check['missingTables'] !== []) { + $prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_'); + $parts[] = DbDumper::describeMissing($check['missingTables'], $prefix); } - return implode('; ', $parts); + return implode("\n", $parts); } private function withSharedLock(Target $target, callable $fn): mixed { diff --git a/lib/Service/TargetService.php b/lib/Service/TargetService.php index b581301..80d602e 100644 --- a/lib/Service/TargetService.php +++ b/lib/Service/TargetService.php @@ -259,6 +259,11 @@ final class TargetService { /** Remove the target from NG Backup. The backups on the target itself are not touched. */ public function remove(Target $target): void { $this->mapper->delete($target); + // The cached index of its repository is only of use while a location points at it + $repositoryId = $target->getRepositoryId(); + if ($repositoryId !== null && !array_filter($this->list(), fn (Target $t) => $t->getRepositoryId() === $repositoryId)) { + $this->indexCache->forget($repositoryId); + } } public function backend(Target $target): IBackend { diff --git a/psalm-baseline.xml b/psalm-baseline.xml index b243ec9..546574d 100644 --- a/psalm-baseline.xml +++ b/psalm-baseline.xml @@ -153,10 +153,6 @@ - - - - $end[array_search($c, $columns, true)]]]> $q($c)]]> @@ -189,6 +185,10 @@ db]]> db]]> db]]> + db]]> + db]]> + db]]> + db]]> @@ -196,6 +196,10 @@ + + + + @@ -438,6 +442,7 @@ + diff --git a/tests/integration/full-restore-roundtrip.sh b/tests/integration/full-restore-roundtrip.sh index 9e98449..c29f6d0 100755 --- a/tests/integration/full-restore-roundtrip.sh +++ b/tests/integration/full-restore-roundtrip.sh @@ -2,10 +2,14 @@ # Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a # SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup # location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7. -# Usage: full-restore-roundtrip.sh -# e.g. full-restore-roundtrip.sh 35-apache +# Usage: full-restore-roundtrip.sh [sqlite|pgsql] +# e.g. full-restore-roundtrip.sh 35-apache pgsql +# With pgsql the origin also has what broke real disaster recoveries: the Mail app with a synced +# account (foreign keys between its tables) and Deck installed and then switched off (tables of +# an app that is not enabled). The restore instance has neither; --install-apps adds Mail. set -euo pipefail IMAGE="nextcloud:${1:-35-apache}" +DB="${2:-sqlite}" APP=$(cd "$(dirname "$0")/../.." && pwd) NET=ngb-frt-net SHARED=$(mktemp -d) @@ -14,7 +18,7 @@ fail() { echo "FAIL: $*"; exit 1; } ok() { echo "ok $*"; } cleanup() { - docker rm -fv ngb-frt-origin ngb-frt-restore >/dev/null 2>&1 || true + docker rm -fv ngb-frt-origin ngb-frt-restore ngb-frt-db-origin ngb-frt-db-restore ngb-frt-imap >/dev/null 2>&1 || true docker network rm "$NET" >/dev/null 2>&1 || true docker run --rm -v "$SHARED:/target" alpine sh -c 'rm -rf /target/*' >/dev/null 2>&1 || true rmdir "$SHARED" 2>/dev/null || true @@ -24,14 +28,28 @@ trap cleanup EXIT run_on() { docker exec -u www-data "$1" php occ "${@:2}"; } +# Database settings for an instance: SQLite inside the container, or its own PostgreSQL container +db_env() { + if [ "$DB" = pgsql ]; then + docker run -d --name "ngb-frt-db-$1" --network "$NET" -e POSTGRES_DB=nextcloud -e POSTGRES_USER=nextcloud \ + -e POSTGRES_PASSWORD=nextcloud postgres:16-alpine >/dev/null + for i in $(seq 1 30); do docker exec "ngb-frt-db-$1" pg_isready -q 2>/dev/null && break; sleep 2; done + echo "-e POSTGRES_HOST=ngb-frt-db-$1 -e POSTGRES_DB=nextcloud -e POSTGRES_USER=nextcloud -e POSTGRES_PASSWORD=nextcloud" + else + echo "-e SQLITE_DATABASE=nextcloud" + fi +} + docker network create "$NET" >/dev/null -docker run -d --name ngb-frt-origin --network "$NET" -e SQLITE_DATABASE=nextcloud \ +ORIGIN_DB=$(db_env origin) +# shellcheck disable=SC2086 +docker run -d --name ngb-frt-origin --network "$NET" $ORIGIN_DB \ -e NEXTCLOUD_ADMIN_USER=admin -e NEXTCLOUD_ADMIN_PASSWORD=admin \ -v "$APP:/var/www/html/custom_apps/ng_backup:ro" -v "$SHARED:/mnt/shared" "$IMAGE" >/dev/null for i in $(seq 1 90); do run_on ngb-frt-origin status 2>/dev/null | grep -q "installed: true" && break; sleep 3; done docker exec ngb-frt-origin chown www-data:www-data /var/www/html/custom_apps run_on ngb-frt-origin app:enable ng_backup >/dev/null -ok "origin instance up" +ok "origin instance up ($DB)" docker exec ngb-frt-origin sh -c "printf 'full restore roundtrip passphrase 2026\n' > /tmp/pass" run_on ngb-frt-origin backup:key:init --passphrase-file=/tmp/pass --label=CI >/dev/null || fail "key init" @@ -45,6 +63,26 @@ NC_PASS=$(openssl rand -hex 16) docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest" run_on ngb-frt-origin files:scan frtest >/dev/null + +if [ "$DB" = pgsql ]; then + # Mail with a synced account: oc_mail_accounts points at oc_mail_mailboxes (foreign keys), + # while the restore loads tables in name order, accounts before mailboxes. + docker run -d --name ngb-frt-imap --network "$NET" -e MAILNAME=mail.domain.tld -e MAIL_ADDRESS=user@domain.tld \ + -e MAIL_PASS=mypassword ghcr.io/christophwurst/docker-imap-devel:latest >/dev/null + run_on ngb-frt-origin app:install mail >/dev/null || fail "install mail on the origin" + run_on ngb-frt-origin config:system:set app.mail.verify-tls-peer --type=boolean --value=false >/dev/null + sleep 10 + run_on ngb-frt-origin mail:account:create frtest Test user@domain.tld ngb-frt-imap 993 ssl user@domain.tld mypassword \ + ngb-frt-imap 25 tls user@domain.tld mypassword >/dev/null || fail "mail account" + run_on ngb-frt-origin mail:account:sync -f 1 >/dev/null || fail "mail sync" + [ "$(docker exec ngb-frt-db-origin psql -U nextcloud -d nextcloud -Atc 'select count(*) from oc_mail_accounts where drafts_mailbox_id is not null')" = 1 ] \ + || fail "the mail account does not point at its mailboxes" + # Deck installed and then switched off: its tables stay in the database and in the backup + run_on ngb-frt-origin app:install deck >/dev/null || fail "install deck on the origin" + run_on ngb-frt-origin app:disable deck >/dev/null + ok "origin has a synced Mail account and the tables of a disabled app" +fi + RUN_OUT=$(run_on ngb-frt-origin backup:run offsite -l disaster-test) SNAP=$(printf '%s\n' "$RUN_OUT" | head -1 | awk -F'snapshot ' '{print $2}' | tr -d ',') [ -n "$SNAP" ] || fail "no snapshot id from backup:run" @@ -52,7 +90,9 @@ ok "test data backed up, snapshot $SNAP" docker cp ngb-frt-origin:/tmp/kit.json /tmp/ngb-frt-kit.json docker exec ngb-frt-origin sh -c "cat /tmp/pass" > /tmp/ngb-frt-pass -docker run -d --name ngb-frt-restore --network "$NET" -e SQLITE_DATABASE=nextcloud \ +RESTORE_DB=$(db_env restore) +# shellcheck disable=SC2086 +docker run -d --name ngb-frt-restore --network "$NET" $RESTORE_DB \ -e NEXTCLOUD_ADMIN_USER=admin -e NEXTCLOUD_ADMIN_PASSWORD=admin \ -v "$APP:/var/www/html/custom_apps/ng_backup:ro" -v "$SHARED:/mnt/shared" "$IMAGE" >/dev/null for i in $(seq 1 90); do run_on ngb-frt-restore status 2>/dev/null | grep -q "installed: true" && break; sleep 3; done @@ -68,7 +108,7 @@ run_on ngb-frt-restore backup:target:add offsite local -a null::null -o datadir= run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location" ok "recovery kit imported, existing repository adopted, snapshot visible" -OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction 2>&1) || { +OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction --install-apps 2>&1) || { printf '%s\n' "$OUT" | tail -40 docker exec ngb-frt-restore sh -c 'tail -c 6000 /var/www/html/data/nextcloud.log' || true fail "restore:full" @@ -84,7 +124,17 @@ docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.tx run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore" ok "database, files and ng_backup's own state are all intact and usable after the restore" +if [ "$DB" = pgsql ]; then + Q() { docker exec ngb-frt-db-restore psql -U nextcloud -d nextcloud -Atc "$1"; } + [ "$(Q "select count(*) from pg_tables where tablename = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created" + [ "$(Q "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored" + [ "$(Q "select count(*) from pg_constraint where contype = 'f' and conrelid::regclass::text like 'oc_mail_%'")" -gt 0 ] \ + || fail "the foreign keys of Mail are missing after the restore" + printf '%s\n' "$OUT" | grep -q "Created .* tables" || fail "the restore did not report the tables it created" + ok "tables of a disabled app created, Mail restored with its foreign keys" +fi + run_on ngb-frt-restore backup:run offsite -l post-restore-check >/dev/null || fail "a new backup after the restore fails" ok "the restored instance can run a new backup of its own" -echo "FULL RESTORE ROUNDTRIP PASS" +echo "FULL RESTORE ROUNDTRIP PASS ($DB)" From 7b735ea9417c0b95bd49a095e3985035bbbf34b6 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 01:45:39 +0200 Subject: [PATCH 13/19] DR guide step 7 for rc2; changelog; version 0.9.8 --- CHANGELOG.md | 30 +++++++++++++++++++++++++++++- appinfo/info.xml | 2 +- docs/DISASTER-RECOVERY.md | 21 ++++++++++++--------- 3 files changed, 42 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ffde11f..edd6468 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,36 @@ instead carried by the app description, the git tag, the release, and this file. ## [Unreleased] +Fixes from a full disaster recovery test of a production instance (about 1 TB, Nextcloud +All-in-One onto a plain Nextcloud with PostgreSQL). + +### Added +- `docs/DISASTER-RECOVERY.md`: step-by-step guide for when your server is gone. +- `occ backup:restore:full --install-apps` installs (from the app store) or enables the apps the + old server had enabled, before the restore starts. + ### Changed -- A fuller app store description, ready for 1.0.0. +- Backups record the structure of every database table. A full restore creates tables that do not + exist on the new instance, for example those left behind by an app that was removed long ago or + is switched off, instead of stopping at the first one. +- A full restore no longer refuses when apps are missing: it restores their data and lists the + apps to install afterwards. `nextcloud-aio` is never required, so a backup of Nextcloud + All-in-One can be restored onto a plain Nextcloud. +- All problems are listed at once, with the app each missing table probably belongs to. For + backups made before this version, `--skip-missing-tables` leaves out tables whose structure is + unknown. +- A fuller app store description, ready for 1.0.0, and a note on paid restore help by Ensembia. +- README: switching the server to TCP BBR made backups over a long distance about 3 times faster. + +### Fixed +- A full restore failed on foreign keys (for example those of the Mail app), because tables are + restored in name order. Foreign keys are now checked after all rows are loaded, before the + restore commits. +- `occ backup:target:add` without a key pointed to `backup:key:init`, which makes a new key that + cannot read existing backups. It now points to `backup:key:import` for a restore, and says that + connecting to a location with backups can take a while. +- Removing a location left its cached index in the database (tens of thousands of rows for a large + backup). It is removed with the location now, and once for locations that are already gone. ## [1.0.0-rc.1] - 2026-10-17 diff --git a/appinfo/info.xml b/appinfo/info.xml index f0bfb3f..efe6cc8 100644 --- a/appinfo/info.xml +++ b/appinfo/info.xml @@ -41,7 +41,7 @@ welcome: use the "Share your experience" link on the admin page. Developed and sponsored by [Ensembia](https://www.ensembia.com), which also offers paid help with restoring a backup after a crash.]]> - 0.9.7 + 0.9.8 agpl Dillard Blom NgBackup diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md index 05f6226..5861db4 100644 --- a/docs/DISASTER-RECOVERY.md +++ b/docs/DISASTER-RECOVERY.md @@ -76,18 +76,21 @@ with a different key"). occ backup:list offsite -**7. Install the apps the old server had.** Run the restore; before it changes anything it checks -the apps and stops with what is missing: +**7. Start the restore, and let it install the apps.** - occ backup:restore:full offsite + occ backup:restore:full offsite --install-apps -- "install these apps first: …": install each from the app store (`occ app:install `), or - enable it if it is part of Nextcloud but switched off (`occ app:enable `, for example - `admin_audit`). Apps that are not in the app store: copy them into `custom_apps` and enable them. -- "Table … does not exist here": an app that was removed or switched off long ago left tables - behind. Install that app (for example `deck`, `onlyoffice`), then run the restore again. +`--install-apps` first installs every app the old server had enabled from the app store (or +switches it on, if it is part of Nextcloud). Tables of apps that are missing or were removed long +ago are created from the backup, so they do not stop the restore. -Repeat until the restore starts. +At the end it lists the apps it could not install, usually your own builds that are not in the +app store. Copy those into `custom_apps` and enable them (`occ app:enable `); their data is +already restored. + +Backups made before 1.0.0-rc.2 do not record the structure of their tables. If the restore lists +tables that do not exist here, install the apps they belong to and run it again, or leave them +out with `--skip-missing-tables`. **8. Let the restore run.** It restores the database, then the data directory, and takes hours for a large backup. If it stops halfway, fix the cause and run the same command again with `--force`. From 4f6a2e5b70263f7b2f1cd85cc96a53e80addb00c Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 02:09:00 +0200 Subject: [PATCH 14/19] Check MySQL foreign keys explicitly; keep index prefix lengths and foreign keys of created tables Switching FOREIGN_KEY_CHECKS back on does not check the rows loaded meanwhile, and DDL would commit halfway, so on MySQL every foreign key is checked with a query for rows that point at nothing, before the commit. The recorded table structure now keeps index options (the prefix lengths MySQL needs for long text columns) and foreign keys; tables the restore creates get their foreign keys after the commit. The full-restore test also runs on MariaDB. --- .github/workflows/ci.yml | 2 + lib/Command/RestoreFull.php | 3 + lib/Db/DbDumper.php | 70 +++++++++++++++++++-- psalm-baseline.xml | 3 + tests/integration/full-restore-roundtrip.sh | 27 +++++--- 5 files changed, 90 insertions(+), 15 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7b7fb3e..e8df09c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -106,3 +106,5 @@ jobs: run: tests/integration/full-restore-roundtrip.sh 35-apache - name: Full restore round trip on PostgreSQL, with Mail (foreign keys) and a disabled app run: tests/integration/full-restore-roundtrip.sh 35-apache pgsql + - name: Full restore round trip on MariaDB, with Mail (foreign keys) and a disabled app + run: tests/integration/full-restore-roundtrip.sh 35-apache mysql diff --git a/lib/Command/RestoreFull.php b/lib/Command/RestoreFull.php index 777582d..1516391 100644 --- a/lib/Command/RestoreFull.php +++ b/lib/Command/RestoreFull.php @@ -105,6 +105,9 @@ HELP); $output->writeln(sprintf('Created %d tables that did not exist here (apps that are missing or were removed): %s', count($result['dbReport']['created']), implode(', ', $result['dbReport']['created']))); } + foreach ($result['dbReport']['foreignKeyErrors'] ?? [] as $error) { + $output->writeln('Could not add a foreign key to a created table (the data is restored): ' . $error . ''); + } if ($result['dbReport']['missing'] !== []) { $output->writeln('Left out (structure unknown): ' . implode(', ', $result['dbReport']['missing']) . ''); } diff --git a/lib/Db/DbDumper.php b/lib/Db/DbDumper.php index 0f27e32..bc5f981 100644 --- a/lib/Db/DbDumper.php +++ b/lib/Db/DbDumper.php @@ -214,6 +214,21 @@ final class DbDumper { $this->restoreSequences($manifest['sequences'] ?? []); $this->resumeForeignKeys($foreignKeys); $this->db->commit(); + // Foreign keys of the tables created above, now their rows are in. Not inside the + // transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is + // restored either way. + foreach ($report['created'] as $table) { + $structure = $manifest['tables'][$table]['schema'] ?? []; + foreach ($structure['foreignKeys'] ?? [] as $fk) { + try { + $this->db->createSchemaManager()->createForeignKey( + new \Doctrine\DBAL\Schema\ForeignKeyConstraint($fk['columns'], $fk['table'], $fk['foreign'], $fk['name'], $fk['options'] ?? []), + $table); + } catch (\Throwable $e) { + $report['foreignKeyErrors'][] = $table . '.' . $fk['name'] . ': ' . $e->getMessage(); + } + } + } } catch (\Throwable $e) { $this->db->rollBack(); // PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session @@ -268,10 +283,18 @@ final class DbDumper { $indexes = []; foreach ($table->getIndexes() as $i) { if (!$i->isPrimary()) { - $indexes[] = ['name' => $i->getName(), 'columns' => array_values($i->getColumns()), 'unique' => $i->isUnique()]; + // options holds the prefix lengths MySQL needs for indexes on long text columns + $indexes[] = ['name' => $i->getName(), 'columns' => array_values($i->getColumns()), 'unique' => $i->isUnique(), + 'flags' => array_values($i->getFlags()), 'options' => $i->getOptions()]; } } - return ['columns' => $columns, 'pk' => array_values($table->getPrimaryKey()?->getColumns() ?? []), 'indexes' => $indexes]; + $foreignKeys = []; + foreach ($table->getForeignKeys() as $fk) { + $foreignKeys[] = ['name' => $fk->getName(), 'columns' => array_values($fk->getLocalColumns()), + 'table' => $fk->getForeignTableName(), 'foreign' => array_values($fk->getForeignColumns()), 'options' => $fk->getOptions()]; + } + return ['columns' => $columns, 'pk' => array_values($table->getPrimaryKey()?->getColumns() ?? []), 'indexes' => $indexes, + 'foreignKeys' => $foreignKeys]; } private static function tableFromStructure(string $name, array $structure): \Doctrine\DBAL\Schema\Table { @@ -287,7 +310,9 @@ final class DbDumper { $table->setPrimaryKey($structure['pk']); } foreach ($structure['indexes'] as $i) { - $i['unique'] ? $table->addUniqueIndex($i['columns'], $i['name']) : $table->addIndex($i['columns'], $i['name']); + $i['unique'] + ? $table->addUniqueIndex($i['columns'], $i['name'], $i['options'] ?? []) + : $table->addIndex($i['columns'], $i['name'], $i['flags'] ?? [], $i['options'] ?? []); } return $table; } @@ -316,8 +341,30 @@ final class DbDumper { return $restore; } if ($platform instanceof \Doctrine\DBAL\Platforms\AbstractMySQLPlatform) { + // MySQL does not check rows again when FOREIGN_KEY_CHECKS goes back to 1, and its DDL + // would commit the transaction halfway. So every foreign key is checked explicitly, + // with a query that finds rows pointing at nothing (resumeForeignKeys()). + $rows = $this->db->fetchAllAssociative( + 'SELECT TABLE_NAME AS tbl, CONSTRAINT_NAME AS name, COLUMN_NAME AS col, REFERENCED_TABLE_NAME AS rtbl, REFERENCED_COLUMN_NAME AS rcol + FROM information_schema.KEY_COLUMN_USAGE + WHERE TABLE_SCHEMA = DATABASE() AND REFERENCED_TABLE_NAME IS NOT NULL + ORDER BY TABLE_NAME, CONSTRAINT_NAME, ORDINAL_POSITION'); + $keys = []; + foreach ($rows as $r) { + if (str_starts_with((string)$r['tbl'], $this->prefix)) { + $keys[$r['tbl'] . "\0" . $r['name']][] = $r; + } + } + $q = fn (string $i): string => $this->db->quoteIdentifier($i); + $checks = ['SET FOREIGN_KEY_CHECKS = 1']; + foreach ($keys as $parts) { + $join = implode(' AND ', array_map(fn (array $r): string => 'c.' . $q($r['col']) . ' = p.' . $q($r['rcol']), $parts)); + $set = implode(' AND ', array_map(fn (array $r): string => 'c.' . $q($r['col']) . ' IS NOT NULL', $parts)); + $checks[] = sprintf('SELECT COUNT(*) FROM %s c LEFT JOIN %s p ON %s WHERE %s AND p.%s IS NULL /* %s */', + $q($parts[0]['tbl']), $q($parts[0]['rtbl']), $join, $set, $q($parts[0]['rcol']), $parts[0]['name']); + } $this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 0'); - return ['SET FOREIGN_KEY_CHECKS = 1']; + return $checks; } if ($platform instanceof \Doctrine\DBAL\Platforms\SqlitePlatform) { $this->db->executeStatement('PRAGMA defer_foreign_keys = ON'); // checked at commit @@ -325,11 +372,22 @@ final class DbDumper { return []; } - /** Add the foreign keys back; a row that breaks one fails the restore here, before it commits. */ + /** + * Check the foreign keys again; a row that breaks one fails the restore here, before it commits. + * Statements add them back (PostgreSQL) or switch the checks on (MySQL); queries count the rows + * that point at nothing (MySQL) and must find none. + */ private function resumeForeignKeys(array $statements): void { foreach ($statements as $sql) { try { - $this->db->executeStatement($sql); + if (str_starts_with($sql, 'SELECT ')) { + $orphans = (int)$this->db->fetchOne($sql); + if ($orphans > 0) { + throw new \RuntimeException("$orphans rows point at nothing"); + } + } else { + $this->db->executeStatement($sql); + } } catch (\Throwable $e) { throw new \RuntimeException('The restored data breaks a foreign key (' . $sql . '): ' . $e->getMessage(), 0, $e); } diff --git a/psalm-baseline.xml b/psalm-baseline.xml index 546574d..b25f599 100644 --- a/psalm-baseline.xml +++ b/psalm-baseline.xml @@ -189,6 +189,8 @@ db]]> db]]> db]]> + db]]> + db]]> @@ -197,6 +199,7 @@ + diff --git a/tests/integration/full-restore-roundtrip.sh b/tests/integration/full-restore-roundtrip.sh index c29f6d0..e9c3657 100755 --- a/tests/integration/full-restore-roundtrip.sh +++ b/tests/integration/full-restore-roundtrip.sh @@ -2,9 +2,9 @@ # Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a # SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup # location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7. -# Usage: full-restore-roundtrip.sh [sqlite|pgsql] +# Usage: full-restore-roundtrip.sh [sqlite|pgsql|mysql] # e.g. full-restore-roundtrip.sh 35-apache pgsql -# With pgsql the origin also has what broke real disaster recoveries: the Mail app with a synced +# With pgsql or mysql the origin also has what broke real disaster recoveries: the Mail app with a synced # account (foreign keys between its tables) and Deck installed and then switched off (tables of # an app that is not enabled). The restore instance has neither; --install-apps adds Mail. set -euo pipefail @@ -35,6 +35,11 @@ db_env() { -e POSTGRES_PASSWORD=nextcloud postgres:16-alpine >/dev/null for i in $(seq 1 30); do docker exec "ngb-frt-db-$1" pg_isready -q 2>/dev/null && break; sleep 2; done echo "-e POSTGRES_HOST=ngb-frt-db-$1 -e POSTGRES_DB=nextcloud -e POSTGRES_USER=nextcloud -e POSTGRES_PASSWORD=nextcloud" + elif [ "$DB" = mysql ]; then + docker run -d --name "ngb-frt-db-$1" --network "$NET" -e MARIADB_DATABASE=nextcloud -e MARIADB_USER=nextcloud \ + -e MARIADB_PASSWORD=nextcloud -e MARIADB_ROOT_PASSWORD=nextcloud mariadb:11 >/dev/null + for i in $(seq 1 60); do docker exec "ngb-frt-db-$1" mariadb-admin ping -unextcloud -pnextcloud --silent >/dev/null 2>&1 && break; sleep 2; done + echo "-e MYSQL_HOST=ngb-frt-db-$1 -e MYSQL_DATABASE=nextcloud -e MYSQL_USER=nextcloud -e MYSQL_PASSWORD=nextcloud" else echo "-e SQLITE_DATABASE=nextcloud" fi @@ -64,7 +69,12 @@ docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add -- docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest" run_on ngb-frt-origin files:scan frtest >/dev/null -if [ "$DB" = pgsql ]; then +# SQL on an instance's own database (pgsql/mysql only) +sql() { + if [ "$DB" = pgsql ]; then docker exec "ngb-frt-db-$1" psql -U nextcloud -d nextcloud -Atc "$2" + else docker exec "ngb-frt-db-$1" mariadb -unextcloud -pnextcloud -N -B nextcloud -e "$2"; fi +} +if [ "$DB" != sqlite ]; then # Mail with a synced account: oc_mail_accounts points at oc_mail_mailboxes (foreign keys), # while the restore loads tables in name order, accounts before mailboxes. docker run -d --name ngb-frt-imap --network "$NET" -e MAILNAME=mail.domain.tld -e MAIL_ADDRESS=user@domain.tld \ @@ -75,7 +85,7 @@ if [ "$DB" = pgsql ]; then run_on ngb-frt-origin mail:account:create frtest Test user@domain.tld ngb-frt-imap 993 ssl user@domain.tld mypassword \ ngb-frt-imap 25 tls user@domain.tld mypassword >/dev/null || fail "mail account" run_on ngb-frt-origin mail:account:sync -f 1 >/dev/null || fail "mail sync" - [ "$(docker exec ngb-frt-db-origin psql -U nextcloud -d nextcloud -Atc 'select count(*) from oc_mail_accounts where drafts_mailbox_id is not null')" = 1 ] \ + [ "$(sql origin 'select count(*) from oc_mail_accounts where drafts_mailbox_id is not null')" = 1 ] \ || fail "the mail account does not point at its mailboxes" # Deck installed and then switched off: its tables stay in the database and in the backup run_on ngb-frt-origin app:install deck >/dev/null || fail "install deck on the origin" @@ -124,11 +134,10 @@ docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.tx run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore" ok "database, files and ng_backup's own state are all intact and usable after the restore" -if [ "$DB" = pgsql ]; then - Q() { docker exec ngb-frt-db-restore psql -U nextcloud -d nextcloud -Atc "$1"; } - [ "$(Q "select count(*) from pg_tables where tablename = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created" - [ "$(Q "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored" - [ "$(Q "select count(*) from pg_constraint where contype = 'f' and conrelid::regclass::text like 'oc_mail_%'")" -gt 0 ] \ +if [ "$DB" != sqlite ]; then + [ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created" + [ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored" + [ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \ || fail "the foreign keys of Mail are missing after the restore" printf '%s\n' "$OUT" | grep -q "Created .* tables" || fail "the restore did not report the tables it created" ok "tables of a disabled app created, Mail restored with its foreign keys" From 9eeaefd938461ca0ef8977cd4673dc9bde15a1c4 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 02:39:14 +0200 Subject: [PATCH 15/19] Refuse a full restore when the table prefix differs --- CHANGELOG.md | 2 ++ lib/Db/DbDumper.php | 21 +++++++++++++++++++++ lib/Service/RestoreFullService.php | 14 ++++++++++++-- 3 files changed, 35 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index edd6468..054fe91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,8 @@ All-in-One onto a plain Nextcloud with PostgreSQL). - A full restore no longer refuses when apps are missing: it restores their data and lists the apps to install afterwards. `nextcloud-aio` is never required, so a backup of Nextcloud All-in-One can be restored onto a plain Nextcloud. +- A full restore onto an installation with a different table prefix (`dbtableprefix`) is refused + before it starts, with what to change, instead of restoring into tables nobody uses. - All problems are listed at once, with the app each missing table probably belongs to. For backups made before this version, `--skip-missing-tables` leaves out tables whose structure is unknown. diff --git a/lib/Db/DbDumper.php b/lib/Db/DbDumper.php index bc5f981..716b63e 100644 --- a/lib/Db/DbDumper.php +++ b/lib/Db/DbDumper.php @@ -145,6 +145,11 @@ final class DbDumper { */ public function restore(Repository $repo, array $manifest, ?array &$report = null, ?callable $heartbeat = null, bool $skipMissing = false): int { $report = ['skipped' => [], 'restored' => [], 'created' => [], 'missing' => []]; + $source = self::dumpPrefix($manifest); + if ($source !== null && $source !== $this->prefix) { + // Every table would look missing and be created under the old prefix, unused. + throw new \RuntimeException("The backup's tables use the prefix '$source', this installation '{$this->prefix}'. Install Nextcloud again with 'dbtableprefix' => '$source' in config/config.php before occ maintenance:install, and restore again."); + } $missing = $this->missingTables($manifest); $unknown = array_values(array_filter($missing, fn (string $t) => !isset($manifest['tables'][$t]['schema']))); if ($unknown !== [] && !$skipMissing) { @@ -241,6 +246,22 @@ final class DbDumper { return $total; } + /** + * The table prefix of the instance the dump was made on (from its appconfig table), or null + * when the dump has none. + * + */ + public static function dumpPrefix(array $manifest): ?string { + /** @var list $tables table names */ + $tables = array_keys($manifest['tables'] ?? []); + foreach ($tables as $name) { + if (str_ends_with($name, 'appconfig') && !str_ends_with($name, '_ex_appconfig')) { + return substr($name, 0, -strlen('appconfig')); + } + } + return null; + } + /** * Tables in the dump that do not exist on this instance. * diff --git a/lib/Service/RestoreFullService.php b/lib/Service/RestoreFullService.php index 467f595..9a0e279 100644 --- a/lib/Service/RestoreFullService.php +++ b/lib/Service/RestoreFullService.php @@ -63,19 +63,25 @@ final class RestoreFullService { $enabled = array_flip($this->appManager->getEnabledApps()); $missingApps = array_values(array_diff(array_keys($meta['apps'] ?? []), array_keys($enabled), self::ENVIRONMENT_APPS)); $missingTables = []; + $prefixMismatch = null; if (isset($meta['db'])) { $manifest = json_decode($repo->getObject($meta['db']), true, 512, JSON_THROW_ON_ERROR); $prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_'); + $sourcePrefix = DbDumper::dumpPrefix($manifest); + if ($sourcePrefix !== null && $sourcePrefix !== $prefix) { + $prefixMismatch = [$sourcePrefix, $prefix]; + } $dumper = new DbDumper($this->db->getInner(), $prefix); $missingTables = array_values(array_filter($dumper->missingTables($manifest), fn (string $t) => !isset($manifest['tables'][$t]['schema']) && !str_starts_with($t, $prefix . 'ngb_'))); } return [ - 'ok' => $currentMajor === $snapshotMajor && ($missingTables === [] || $skipMissingTables), + 'ok' => $currentMajor === $snapshotMajor && $prefixMismatch === null && ($missingTables === [] || $skipMissingTables), 'currentMajor' => $currentMajor, 'snapshotMajor' => $snapshotMajor, 'missingApps' => $missingApps, - 'missingTables' => $missingTables, + 'missingTables' => $prefixMismatch === null ? $missingTables : [], + 'prefixMismatch' => $prefixMismatch, ]; } @@ -224,6 +230,10 @@ final class RestoreFullService { if ($check['currentMajor'] !== $check['snapshotMajor']) { $parts[] = "this installation is Nextcloud {$check['currentMajor']}, the backup is from {$check['snapshotMajor']} (same major version required)"; } + if (($check['prefixMismatch'] ?? null) !== null) { + [$source, $here] = $check['prefixMismatch']; + $parts[] = "the backup's tables use the prefix '$source', this installation '$here': install Nextcloud again with 'dbtableprefix' => '$source' in config/config.php before occ maintenance:install, and restore again"; + } if ($check['missingTables'] !== []) { $prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_'); $parts[] = DbDumper::describeMissing($check['missingTables'], $prefix); From 46eaceb928a22ebf01b11b7855aa5d1a332c2441 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 08:51:00 +0200 Subject: [PATCH 16/19] DR guide step 9 from the test; PLAN: small DR test with rc2 passed --- PLAN.md | 5 +++++ docs/DISASTER-RECOVERY.md | 20 ++++++++++++-------- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/PLAN.md b/PLAN.md index f69b0ba..b809891 100644 --- a/PLAN.md +++ b/PLAN.md @@ -440,6 +440,11 @@ in README "Known limitations": PUT van 2,4 naar 7,5 MiB/s, ng_backup van ~1 naar ~1,6 MiB/s (rest = serieel lezen/uploaden). Tip over BBR in de README gezet. Let op: PVE02 staat in Canada; een restore in de EU gaat sneller. 15. DR-document: in het Docker-voorbeeld "je containernaam" (`docker ps`) i.p.v. `nextcloud`. + **Kleine DR-test met rc2 (2026-10-18), volautomatisch geslaagd:** bron VM 981 (NC 34, PostgreSQL, + 51 GB, Mail met gesynct account, Deck uitgeschakeld, Calendar/Contacts, admin_audit) → OVH S3 → + verse VM 982 met alleen de DR-gids en `--install-apps`: apps vanzelf geïnstalleerd, 10 Deck-tabellen + aangemaakt, 15 FK's in orde, 23.519/23.519 checksums gelijk, agenda/contact/mail/bord terug. + Restore 1 u 47 vanuit Warschau naar Canada (met BBR). Lokaal slaagt de test ook op SQLite en MariaDB. **Daarna verplicht:** een tweede volledige restore op een verse VM, alleen met de HOWTO en de nieuwe opties, zonder handwerk. Pas als die in één keer slaagt: 1.0.0. diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md index 5861db4..93b1717 100644 --- a/docs/DISASTER-RECOVERY.md +++ b/docs/DISASTER-RECOVERY.md @@ -95,15 +95,19 @@ out with `--skip-missing-tables`. **8. Let the restore run.** It restores the database, then the data directory, and takes hours for a large backup. If it stops halfway, fix the cause and run the same command again with `--force`. -**9. After the restore** +**9. After the restore.** These worked in our tests: -- Check `config/config.php`: `trusted_domains`, `overwrite.cli.url`, and the cache (Redis) host - must match the new server. The database settings and data directory are kept from the new - installation. -- `occ upgrade` (apps on the new server may be newer than in the backup), then - `occ maintenance:repair`. -- Log in, open some files, a calendar and a contact. -- Start background jobs (cron). + occ upgrade + occ maintenance:repair + occ files:scan --all + +- Check `config/config.php`: `trusted_domains`, `overwrite.cli.url` and `overwritehost` come from + the old server; change them if the new server has another name. The database settings and the + data directory are kept from the new installation. +- From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`, + `redis`), which point at AIO's containers. +- Log in, open a few files, a calendar and a contact. +- Start background jobs again (cron), as in step 2 in reverse. **10. Point your domain at the new server** and check that clients reconnect. From 04e91359a0345d67a84881cd70f282da331cb15a Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 21:56:58 +0200 Subject: [PATCH 17/19] Restore: follow blobs that another server's cleanup repacked during the restore --- CHANGELOG.md | 4 +++ lib/Repository/Repository.php | 40 ++++++++++++++++++++++++++--- tests/Unit/Repository/PruneTest.php | 27 +++++++++++++++++++ 3 files changed, 67 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 054fe91..96323af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,10 @@ All-in-One onto a plain Nextcloud with PostgreSQL). - `occ backup:target:add` without a key pointed to `backup:key:init`, which makes a new key that cannot read existing backups. It now points to `backup:key:import` for a restore, and says that connecting to a location with backups can take a while. +- A restore stopped with "Not found: packs/..." when the location was cleaned up while it ran, + for example by the weekly cleanup of the original server during a recovery test on a second + server: blobs that were moved into new packs were still looked for in the old ones. A restore + now reloads the index and reads them from where they are now. - Removing a location left its cached index in the database (tens of thousands of rows for a large backup). It is removed with the location now, and once for locations that are already gone. diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index 9eb0cc7..fdc6be5 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -771,16 +771,48 @@ final class Repository { $ranges[] = [$curPath, $curOffset, $curLength]; $parts[] = $curParts; } - foreach ($this->backend->getRanges($ranges) as $i => $data) { - foreach ($parts[$i] as [$id, $at, $length]) { - yield $this->decodeBlob($id, substr($data, $at, $length)); + $done = 0; + try { + foreach ($this->backend->getRanges($ranges) as $i => $data) { + foreach ($parts[$i] as [$id, $at, $length]) { + yield $this->decodeBlob($id, substr($data, $at, $length)); + $done++; + } + } + } catch (BackendException $e) { + // A pack moved (see loadBlob()): the rest one blob at a time, from the fresh index. + $this->refreshIndex($e); + foreach (array_slice($ids, $done) as $id) { + yield $this->loadBlob($id); } } } private function loadBlob(string $id): string { [$pack, $offset, $length] = $this->index->get($id); - return $this->decodeBlob($id, $this->packReader->read($pack, $offset, $length)); + try { + $encrypted = $this->packReader->read($pack, $offset, $length); + } catch (BackendException $e) { + // Another server's cleanup may have repacked this blob into a new pack since the index + // was loaded (a disaster recovery test while the original server still runs). Reload + // the index from the location; read again only if the blob has moved. + $this->refreshIndex($e); + [$moved, $offset, $length] = $this->index->get($id); + if ($moved === $pack) { + throw $e; + } + $encrypted = $this->packReader->read($moved, $offset, $length); + } + return $this->decodeBlob($id, $encrypted); + } + + /** Reload the index from the location after a failed pack read; rethrows that failure if it cannot. */ + private function refreshIndex(BackendException $cause): void { + try { + $this->index->load(); + } catch (\Throwable) { + throw $cause; + } } /** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */ diff --git a/tests/Unit/Repository/PruneTest.php b/tests/Unit/Repository/PruneTest.php index 6c3a611..2ae37b3 100644 --- a/tests/Unit/Repository/PruneTest.php +++ b/tests/Unit/Repository/PruneTest.php @@ -88,6 +88,33 @@ class PruneTest extends TestCase { $this->assertSame(self::hashTree($src), self::hashTree($target)); } + public function testRestoreFollowsPacksRepackedByAnotherServer(): void { + // A disaster recovery test on a second server reads the location while the original + // server's weekly cleanup repacks it: the restore must not stop at a pack that moved. + $src = $this->tempDir(); + $backend = new LocalBackend($this->tempDir()); + $keys = KeyRing::generate(); + $repo = Repository::initWithKey($backend, $keys, $keys->wrap('pw')); + for ($i = 0; $i < 40; $i++) { + file_put_contents("$src/f$i", random_bytes(500000)); + } + $a = $repo->backupDirectory($src); + for ($i = 0; $i < 40; $i++) { + if ($i % 5 !== 0) { + unlink("$src/f$i"); + } + } + $b = $repo->backupDirectory($src, $a['snapshot']); + $reader = Repository::openWithKey($backend, $keys); // index loaded before the cleanup + $repo->forget($a['snapshot']); + $repo->purgeTrash(0); + $this->assertGreaterThan(0, $repo->prune(false, 0.5)['repacked']); + + $target = $this->tempDir(); + $reader->restore($b['snapshot'], $target); + $this->assertSame(self::hashTree($src), self::hashTree($target)); + } + public function testUntrashRestoresAWorkingSnapshot(): void { $src = $this->tempDir(); $backend = new LocalBackend($this->tempDir()); From 9e1378c6ab00e19caf163714a9e408e87b854e70 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 21:57:17 +0200 Subject: [PATCH 18/19] PLAN: DR finding 16, restore during another server's cleanup --- PLAN.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/PLAN.md b/PLAN.md index b809891..d9bec8e 100644 --- a/PLAN.md +++ b/PLAN.md @@ -440,6 +440,12 @@ in README "Known limitations": PUT van 2,4 naar 7,5 MiB/s, ng_backup van ~1 naar ~1,6 MiB/s (rest = serieel lezen/uploaden). Tip over BBR in de README gezet. Let op: PVE02 staat in Canada; een restore in de EU gaat sneller. 15. DR-document: in het Docker-voorbeeld "je containernaam" (`docker ps`) i.p.v. `nextcloud`. + 16. Grote restore (VM 980, snapshot aa7cce8d van de Storage Box) stopte na 22 uur met + `Not found: packs/26/2642ef39…`. Oorzaak: de wekelijkse opruiming van nc01 (zo 18-10) heeft + dat pakket tijdens de restore omgepakt (blob staat nu in `389567b7…`); de herstelserver had de + index van vóór de opruiming. Productie heel: `backup:verify` op nc01 OK (150.639 bestanden, + 316.830 blobs). Opgelost in rc2: bij een mislukte pakketlezing de index herladen en de blob + lezen waar hij nu staat (test `testRestoreFollowsPacksRepackedByAnotherServer`). **Kleine DR-test met rc2 (2026-10-18), volautomatisch geslaagd:** bron VM 981 (NC 34, PostgreSQL, 51 GB, Mail met gesynct account, Deck uitgeschakeld, Calendar/Contacts, admin_audit) → OVH S3 → verse VM 982 met alleen de DR-gids en `--install-apps`: apps vanzelf geïnstalleerd, 10 Deck-tabellen From 11ac30635a1cd5cd6a5826b68cd178c68041a849 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Sun, 18 Oct 2026 22:14:45 +0200 Subject: [PATCH 19/19] target:add: show progress and time left while reading the index of an existing repository --- CHANGELOG.md | 3 +- docs/DISASTER-RECOVERY.md | 5 +-- lib/Command/TargetAdd.php | 40 ++++++++++++++++++++++-- lib/Repository/Repository.php | 5 +-- lib/Service/TargetService.php | 5 +-- tests/Unit/Repository/IndexCacheTest.php | 18 +++++++++++ 6 files changed, 66 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 96323af..3018ccf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,7 +38,8 @@ All-in-One onto a plain Nextcloud with PostgreSQL). restore commits. - `occ backup:target:add` without a key pointed to `backup:key:init`, which makes a new key that cannot read existing backups. It now points to `backup:key:import` for a restore, and says that - connecting to a location with backups can take a while. + connecting to a location with backups can take a while. While it reads the index of a location + with backups, it shows how far it is and about how long it still takes. - A restore stopped with "Not found: packs/..." when the location was cleaned up while it ran, for example by the weekly cleanup of the original server during a recovery test on a second server: blobs that were moved into new packs were still looked for in the old ones. A restore diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md index 93b1717..fd7bdf1 100644 --- a/docs/DISASTER-RECOVERY.md +++ b/docs/DISASTER-RECOVERY.md @@ -59,8 +59,9 @@ Import it; you type the passphrase yourself: It shows the key fingerprint: it must match the one of your old server (on the recovery kit). -**5. Add the backup location** with the same details as before. This can take a while on a large -backup (the index is read first) and shows nothing until it is done. +**5. Add the backup location** with the same details as before. On a large backup this first reads +the index and shows how far it is, for example +`Reading the index: 6,913/26,791 files (25%), about 12 min left`. occ backup:target:add offsite sftp -a password::password -o host=... -o user=... -o password=... -o root=... --path=... diff --git a/lib/Command/TargetAdd.php b/lib/Command/TargetAdd.php index d91368c..b56d250 100644 --- a/lib/Command/TargetAdd.php +++ b/lib/Command/TargetAdd.php @@ -64,11 +64,15 @@ HELP); return 1; } // Adopting an existing repository reads its whole index first, which takes a while on a - // large backup or a slow line; say so instead of looking stuck. - $output->writeln('Connecting to the location; with existing backups this reads their index first and can take a while ...'); + // large backup or a slow line; show how far it is instead of looking stuck. + $output->writeln('Connecting to the location; with existing backups this reads their index first ...'); try { - $r = $this->targets->add($input->getArgument('name'), $input->getArgument('backend'), $input->getOption('auth'), $options, (string)$input->getOption('path')); + $r = $this->targets->add($input->getArgument('name'), $input->getArgument('backend'), $input->getOption('auth'), $options, + (string)$input->getOption('path'), self::progress($output)); } catch (\Throwable $e) { + if ($output->isDecorated()) { + $output->write("\r\033[K"); // clear an unfinished progress line + } $output->writeln('' . $e->getMessage() . ''); return 1; } @@ -76,4 +80,34 @@ HELP); $r['target']->getRepositoryId(), $r['created'] ? 'created' : 'found and adopted')); return 0; } + + /** + * Progress of reading the index: on a terminal one line that is updated in place, otherwise + * (a log, a pipe) a line every 30 seconds. The time left is estimated from the rate so far. + * + * @return callable(int, int): void + */ + private static function progress(OutputInterface $output): callable { + $start = microtime(true); + $shown = 0.0; + $inPlace = $output->isDecorated(); + return static function (int $done, int $total) use ($output, $start, &$shown, $inPlace): void { + $now = microtime(true); + if ($done < $total && $now - $shown < ($inPlace ? 1 : 30)) { + return; + } + $shown = $now; + $line = sprintf('Reading the index: %s/%s files (%d%%)', number_format($done), number_format($total), intdiv(100 * $done, max(1, $total))); + $elapsed = $now - $start; + if ($done < $total && $done > 0 && $elapsed >= 5) { + $left = (int)ceil($elapsed / (float)$done * (float)($total - $done) / 60.0); + $line .= $left <= 1 ? ', less than a minute left' : sprintf(', about %d min left', $left); + } + if ($inPlace) { + $output->write("\r\033[K" . $line . ($done >= $total ? "\n" : '')); + } else { + $output->writeln($line); + } + }; + } } diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index fdc6be5..83b4abc 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -81,7 +81,8 @@ final class Repository { } /** Open with the master key the server holds (scheduled backups, no passphrase needed). */ - public static function openWithKey(IBackend $backend, KeyRing $keys, ?IIndexCache $cache = null, ?ICatalogAnchor $anchor = null): self { + /** @param ?callable(int, int): void $progress index files downloaded so far, and in all (see BlobIndex::load()) */ + public static function openWithKey(IBackend $backend, KeyRing $keys, ?IIndexCache $cache = null, ?ICatalogAnchor $anchor = null, ?callable $progress = null): self { $config = self::readConfig($backend); try { $ok = isset($config['check']) && (new StreamCipher($keys))->decryptString((string)base64_decode($config['check'], true), 'config-check') === $config['id']; @@ -95,7 +96,7 @@ final class Repository { if ($cache !== null) { $repo->index->setCache($cache, $config['id']); } - $repo->index->load(); + $repo->index->load(false, $progress); $repo->loadCatalog($anchor); return $repo; } diff --git a/lib/Service/TargetService.php b/lib/Service/TargetService.php index 80d602e..b0e914e 100644 --- a/lib/Service/TargetService.php +++ b/lib/Service/TargetService.php @@ -48,15 +48,16 @@ final class TargetService { * installation's key), then save it. * * @param array $options + * @param ?callable(int, int): void $progress reading an existing repository's index: files done, and in all * @return array{target: Target, created: bool} */ - public function add(string $name, string $backend, string $auth, array $options, string $basePath): array { + public function add(string $name, string $backend, string $auth, array $options, string $basePath, ?callable $progress = null): array { $this->validateName($name); $this->requireUsableBackend($backend); $keys = $this->keys->keyRing(); $storageBackend = $this->buildBackend($backend, $auth, $options, $basePath); if ($storageBackend->exists('config')) { - $repo = Repository::openWithKey($storageBackend, $keys, $this->indexCache, $this->anchor); + $repo = Repository::openWithKey($storageBackend, $keys, $this->indexCache, $this->anchor, $progress); $created = false; } else { $repo = Repository::initWithKey($storageBackend, $keys, $this->keys->wrappedKey(), $this->anchor); diff --git a/tests/Unit/Repository/IndexCacheTest.php b/tests/Unit/Repository/IndexCacheTest.php index 3383d63..ff2b112 100644 --- a/tests/Unit/Repository/IndexCacheTest.php +++ b/tests/Unit/Repository/IndexCacheTest.php @@ -54,6 +54,24 @@ class IndexCacheTest extends TestCase { $this->assertCount(12, self::hashTree($target)); } + /** Connecting a new server to a location with backups reports how far reading the index is. */ + public function testOpeningReportsIndexProgress(): void { + $src = $this->tempDir(); + for ($i = 0; $i < 6; $i++) { + file_put_contents("$src/f$i", random_bytes(3 * 1048576)); + } + $backend = new LocalBackend($this->tempDir()); + $keys = KeyRing::generate(); + Repository::initWithKey($backend, $keys, $keys->wrap('pw'))->backupDirectory($src); + $files = count($backend->list('index')); + $seen = []; + Repository::openWithKey($backend, $keys, null, null, function (int $done, int $total) use (&$seen): void { + $seen[] = [$done, $total]; + }); + $this->assertCount($files, $seen); + $this->assertSame([$files, $files], end($seen)); + } + /** The database dump reads in a read-only transaction: a pack that fills up then must not write the cache yet. */ public function testHeldCacheWritesWaitForRelease(): void { $cache = new class implements IIndexCache {