From 6c8a78f6bb6e1b89c0e710d8d1677efd0f2a5805 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Mon, 19 Oct 2026 20:41:11 +0200 Subject: [PATCH 01/26] PLAN: 1.0.1 background jobs without system cron: warn, and let the open admin page move a backup --- PLAN.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/PLAN.md b/PLAN.md index 065a39b..48ba36d 100644 --- a/PLAN.md +++ b/PLAN.md @@ -395,9 +395,19 @@ in README "Known limitations": - **1.0.1:** 1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een restore leest de packs nog één voor één: ook vooruitlezen via de S3-client. - 2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron - staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue - #13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles. + 2. Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19): + a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode` + (`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de + laatste run te lang geleden: melding met de cron-regel erbij ("Backups only move while + someone has Nextcloud open"), en "waiting for background jobs" als een lopende backup lang + geen stap zette. + b. **De pagina laat de backup lopen** zolang ze openstaat: in `ajax`/`webcron` voert de + statusaanroep die de pagina toch al herhaalt ook een korte, begrensde stap uit (zoals de + korte stappen die AJAX-cron nu al krijgt). "Back up now" werkt dan ook zonder cron zolang + het tabblad openstaat; de melding zegt dat geplande backups echte cron nodig hebben. + Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en + `cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop + (vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen). 3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later), zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest. - **1.1.0:** From 285153a96862e1ae2c860d9532da289395461b62 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Mon, 19 Oct 2026 21:16:23 +0200 Subject: [PATCH 02/26] PLAN: 1.0.1 cron warning says to keep the page open under AJAX, with a link to the Nextcloud docs --- PLAN.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/PLAN.md b/PLAN.md index 48ba36d..f9ccdaa 100644 --- a/PLAN.md +++ b/PLAN.md @@ -405,6 +405,11 @@ in README "Known limitations": statusaanroep die de pagina toch al herhaalt ook een korte, begrensde stap uit (zoals de korte stappen die AJAX-cron nu al krijgt). "Back up now" werkt dan ook zonder cron zolang het tabblad openstaat; de melding zegt dat geplande backups echte cron nodig hebben. + Bij `ajax`: in de melding zeggen dat een backup alleen doorloopt zolang de NG Backup-pagina + openstaat, met een link naar de Nextcloud-documentatie over achtergrondjobs voor de + versie van de server (nu + https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html; + het versienummer uit de server halen, niet vast in de tekst). Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en `cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop (vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen). From 221ed9dc1792404556a0ea36f2a6329670dfed44 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Mon, 19 Oct 2026 21:18:23 +0200 Subject: [PATCH 03/26] PLAN: the cron warning says cron is needed for all backups, links to how to set it up, and that without it a backup started with the button only moves while the page is open --- PLAN.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/PLAN.md b/PLAN.md index f9ccdaa..cf0a63f 100644 --- a/PLAN.md +++ b/PLAN.md @@ -405,16 +405,20 @@ in README "Known limitations": statusaanroep die de pagina toch al herhaalt ook een korte, begrensde stap uit (zoals de korte stappen die AJAX-cron nu al krijgt). "Back up now" werkt dan ook zonder cron zolang het tabblad openstaat; de melding zegt dat geplande backups echte cron nodig hebben. - Bij `ajax`: in de melding zeggen dat een backup alleen doorloopt zolang de NG Backup-pagina - openstaat, met een link naar de Nextcloud-documentatie over achtergrondjobs voor de - versie van de server (nu - https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html; - het versienummer uit de server halen, niet vast in de tekst). + De melding zegt (zonder werkende systeem-cron): backups hebben systeem-cron nodig, ook de + gewone nachtelijke; zo stel je cron in (link naar de Nextcloud-documentatie over + achtergrondjobs voor de versie van de server, nu + https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html, + versienummer uit de server, niet vast); zolang er geen cron is, loopt een backup die je + met de knop start alleen door terwijl de NG Backup-pagina openstaat. Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en `cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop (vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen). 3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later), zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest. +- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3, + 4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan + een paar bestanden. - **1.1.0:** 1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van From 6ed0a555cfd41dfd76bfcdb7709909ba1783e398 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Mon, 19 Oct 2026 21:26:20 +0200 Subject: [PATCH 04/26] PLAN: label field next to Back up now (1.0.1); per-location schedules, every N hours, locations one after the other, editable notes (1.1.0) --- PLAN.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/PLAN.md b/PLAN.md index cf0a63f..ea4e7a7 100644 --- a/PLAN.md +++ b/PLAN.md @@ -416,6 +416,10 @@ in README "Known limitations": (vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen). 3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later), zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest. + 4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de + tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop + stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een + bestaande parameter. - **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3, 4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan een paar bestanden. @@ -431,6 +435,15 @@ in README "Known limitations": 3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf, 33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1. + 4. **Schema per locatie** (besluit 2026-10-19): in de instellingen van een locatie "uit", + "dagelijks om HH:MM" of "elke N uur (vanaf HH:MM)". Nu is er één `schedule_time` voor alle + locaties, één keer per dag. Bij de update wordt de huidige tijd het schema van elke + bestaande locatie (voor bestaande gebruikers verandert er niets). + 5. **Locaties na elkaar laten starten** in plaats van tegelijk: twee runs die dezelfde schijven + lezen zijn samen niet sneller (gezien op nc01); wachtende runs zichtbaar als "queued". + 6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot + is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus), + zodat een nieuwe server na een DR hem ook ziet. - **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met PostgreSQL 18)** — voor **1.0.0-rc.2**: 1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE, From b728db36c7d36da233d19e9152a52fc85d193b35 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Mon, 19 Oct 2026 21:35:23 +0200 Subject: [PATCH 05/26] README: roadmap --- README.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/README.md b/README.md index db2595c..71cee7a 100644 --- a/README.md +++ b/README.md @@ -192,6 +192,35 @@ Contributions welcome via PR — add what you ran into with your own provider. mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS can't be driven from PHP directly. +## Roadmap + +What is planned, without dates. Ideas and votes are welcome in the +[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here, +a 👍 on its issue helps decide what comes first. + +**1.0.1** +- A warning on the admin page and in `occ backup:status` when background jobs do not run on a + system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on + while the NG Backup page is open ([#13](https://github.com/dillardblom/ng_backup/issues/13)). +- An optional label for a backup started with "Back up now". +- Faster full restores from S3, reading ahead as restores over SFTP do since 1.0.0-rc.4. + +**1.1.0** +- Restore without the recovery kit: the backup key from the location itself, opened with one of + your passphrases. +- A schedule per location, also "every N hours". +- Locations backed up one after the other instead of at the same time. +- A note on a restore point that you can change afterwards. +- An option to pause heavy background jobs (such as Memories, Recognize or the Preview + Generator) while a backup runs. +- Faster backups: reading and uploading at the same time. + +**Later** +- A full restore onto the existing installation, without first installing a fresh Nextcloud. +- Users restoring their own files from the Files app. +- Primary object storage and External Storage mounts (see Known limitations). +- An option not to keep the backup key on the location (the recovery kit is then the only copy). + ## Known limitations On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it: From 315e83b8474ecd4e25b5ed49a6ff3a04d215d339 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Tue, 20 Oct 2026 20:49:44 +0200 Subject: [PATCH 06/26] PLAN: 1.0.1 progress in the database phase and a waiting notice, so a backup does not look stuck --- PLAN.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/PLAN.md b/PLAN.md index ea4e7a7..f63a753 100644 --- a/PLAN.md +++ b/PLAN.md @@ -411,6 +411,10 @@ in README "Known limitations": https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html, versienummer uit de server, niet vast); zolang er geen cron is, loopt een backup die je met de knop start alleen door terwijl de NG Backup-pagina openstaat. + c. **Voortgang in de databasefase** (issue #13): nu staat er alleen "running (db)", wat er + hetzelfde uitziet als vastzitten. Toon "running (db: 47/183 tables)" zoals de bestandsfase + een percentage toont, en "waiting for background jobs (last ran 12 min ago)" als een run + een tijd geen stap zette, ook bij een cron die is blijven staan. Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en `cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop (vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen). From 24686897c33a2268324a7b226a9b0a4d791d3f22 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Tue, 20 Oct 2026 23:12:19 +0200 Subject: [PATCH 07/26] PLAN: DR tests across databases (integration for every crossing; S3 sim onto MariaDB, SMB sim onto SQLite before 1.0.0; nc01 onto MariaDB after) --- PLAN.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/PLAN.md b/PLAN.md index f63a753..2b19f7a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -424,6 +424,11 @@ in README "Known limitations": tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een bestaande parameter. +- **DR-tests over databasegrenzen** (besluit 2026-10-20; productie nc01 draait PostgreSQL): na + de MariaDB→PostgreSQL-fout (tellers, rc5) elke overstap in de integratietest (`to-pgsql`, + `to-mysql`, `to-sqlite`, ook in CI). Volle DR-tests met echte data: vóór 1.0.0 de S3-simulatie + (MariaDB-bron, 138 GB) nogmaals naar een MariaDB-doel en de SMB-simulatie naar een SQLite-doel; + **na 1.0.0** de 1 TB van nc01 (PostgreSQL) naar een MariaDB-doel. - **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3, 4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan een paar bestanden. From b7d09c97840fd107e15c96f7084f40298f016ca1 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Wed, 21 Oct 2026 09:27:15 +0200 Subject: [PATCH 08/26] PLAN: 1.1.0 faster SMB (local and long-line measurements) --- PLAN.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/PLAN.md b/PLAN.md index 2b19f7a..bd729c2 100644 --- a/PLAN.md +++ b/PLAN.md @@ -450,6 +450,11 @@ in README "Known limitations": bestaande locatie (voor bestaande gebruikers verandert er niets). 5. **Locaties na elkaar laten starten** in plaats van tegelijk: twee runs die dezelfde schijven lezen zijn samen niet sneller (gezien op nc01); wachtende runs zichtbaar als "queued". + 7. **SMB sneller** (gemeten 2026-10-21): lokaal netwerk zonder vertraging (Samba-container op de + laptop, 1,8 GB) back-up 24,9 MB/s, restore 27,6 MiB/s; vanaf Canada naar de Storage Box (130 ms) + 2,6 MiB/s, omdat SMB per blok op het antwoord wacht. Grotere schrijfblokken, en vooruitlezen + bij een restore zoals SFTP (IReadsAhead) of meerdere packs tegelijk. Een NAS met 1 Gbit haalt + ~110 MB/s: daar is winst. 6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus), zodat een nieuwe server na een DR hem ook ziet. From 0b4872599d95cdb8b5e45c4c94e49e8412e830b2 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Wed, 21 Oct 2026 19:51:57 +0200 Subject: [PATCH 09/26] PLAN: the 1.0.1 cron PR closes #13 --- PLAN.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/PLAN.md b/PLAN.md index bd729c2..a240fa2 100644 --- a/PLAN.md +++ b/PLAN.md @@ -395,7 +395,8 @@ in README "Known limitations": - **1.0.1:** 1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een restore leest de packs nog één voor één: ook vooruitlezen via de S3-client. - 2. Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19): + 2. Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit + oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21): a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode` (`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de laatste run te lang geleden: melding met de cron-regel erbij ("Backups only move while From cd251cffb39ce7884f3b39235788ffde66fe2816 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Wed, 21 Oct 2026 20:02:43 +0200 Subject: [PATCH 10/26] PLAN: SMB index reads one at a time (1.1.0) --- PLAN.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/PLAN.md b/PLAN.md index a240fa2..beb046d 100644 --- a/PLAN.md +++ b/PLAN.md @@ -456,6 +456,8 @@ in README "Known limitations": 2,6 MiB/s, omdat SMB per blok op het antwoord wacht. Grotere schrijfblokken, en vooruitlezen bij een restore zoals SFTP (IReadsAhead) of meerdere packs tegelijk. Een NAS met 1 Gbit haalt ~110 MB/s: daar is winst. + Ook de index: `target:add` over SMB vanaf Canada haalt ~1 indexbestand per seconde (4.927 + bestanden ~85 min), want SMB krijgt geen `getMany()` met verzoeken tegelijk zoals S3 en SFTP. 6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus), zodat een nieuwe server na een DR hem ook ziet. From a8355df12ece4fe615e59f82e85439023208420b Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 03:36:46 +0200 Subject: [PATCH 11/26] Full restore: tables the restore creates get this installation's charset and collation (utf8mb4 on MySQL/MariaDB); test with an emoji --- lib/Db/DbDumper.php | 29 ++++++++++++++++++--- tests/integration/full-restore-roundtrip.sh | 3 +++ 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/lib/Db/DbDumper.php b/lib/Db/DbDumper.php index b0918f7..6bd3a40 100644 --- a/lib/Db/DbDumper.php +++ b/lib/Db/DbDumper.php @@ -159,7 +159,7 @@ final class DbDumper { // everywhere); tables it does not know are left out, as asked. foreach ($missing as $table) { if (isset($manifest['tables'][$table]['schema'])) { - $this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'])); + $this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions())); $report['created'][] = $table; } else { $report['missing'][] = $table; @@ -319,8 +319,31 @@ final class DbDumper { 'foreignKeys' => $foreignKeys]; } - private static function tableFromStructure(string $name, array $structure): \Doctrine\DBAL\Schema\Table { - $table = new \Doctrine\DBAL\Schema\Table($name); + /** + * Table options for a table the restore creates: those of this installation's own tables. + * On MySQL/MariaDB that is the character set and collation, as Nextcloud's tables have them + * (utf8mb4, so emoji and other 4-byte characters fit); DBAL alone would use 3-byte utf8. + * + * @return array + * @psalm-suppress UndefinedClass the DBAL classes come with Nextcloud, not with this app + */ + private function tableOptions(): array { + $options = $this->db->createSchemaManager()->createSchemaConfig()->getDefaultTableOptions(); + if ($this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\AbstractMySQLPlatform) { + $collation = $this->db->fetchOne('SELECT TABLE_COLLATION FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ?', + [$this->prefix . 'appconfig']); + if (is_string($collation) && $collation !== '') { + $options['collation'] = $collation; + $options['charset'] = explode('_', $collation, 2)[0]; + unset($options['collate']); + } + } + return $options; + } + + /** @param array $options table options (see tableOptions()) */ + private static function tableFromStructure(string $name, array $structure, array $options = []): \Doctrine\DBAL\Schema\Table { + $table = new \Doctrine\DBAL\Schema\Table($name, [], [], [], [], $options); foreach ($structure['columns'] as $c) { $table->addColumn($c['name'], $c['type'], array_filter([ 'length' => $c['length'], 'precision' => $c['precision'], 'scale' => $c['scale'], diff --git a/tests/integration/full-restore-roundtrip.sh b/tests/integration/full-restore-roundtrip.sh index ab79807..01b46ce 100755 --- a/tests/integration/full-restore-roundtrip.sh +++ b/tests/integration/full-restore-roundtrip.sh @@ -119,6 +119,8 @@ if [ "$DB" != sqlite ]; then || fail "the mail account does not point at its mailboxes" # Deck installed and then switched off: its tables stay in the database and in the backup run_on ngb-frt-origin app:install deck >/dev/null || fail "install deck on the origin" + # a 4-byte character (emoji) in a table the restore has to create: utf8mb4 on MySQL/MariaDB + sql origin "insert into oc_deck_boards (title, owner, color, archived, deleted_at, last_modified) values ('Plan 📌', 'frtest', '0087C5', 0, 0, 0)" >/dev/null run_on ngb-frt-origin app:disable deck >/dev/null ok "origin has a synced Mail account and the tables of a disabled app" fi @@ -202,6 +204,7 @@ if [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ] && [ "$LEGACY" = legacy ok "older backup: tables of the disabled app left out, Mail restored" elif [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ]; then [ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created" + [ "$(sql restore "select title from oc_deck_boards")" = "Plan 📌" ] || fail "a 4-byte character did not survive in a created table" [ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored" [ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \ || fail "the foreign keys of Mail are missing after the restore" From be1c628f07d266137eccf729c85db96f4d5b91ba Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 03:38:30 +0200 Subject: [PATCH 12/26] Full-restore test: install Mail and Deck with up to three tries (the app store is slow at times) --- tests/integration/full-restore-roundtrip.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/integration/full-restore-roundtrip.sh b/tests/integration/full-restore-roundtrip.sh index 01b46ce..923cdc3 100755 --- a/tests/integration/full-restore-roundtrip.sh +++ b/tests/integration/full-restore-roundtrip.sh @@ -47,6 +47,12 @@ if [ "$LEGACY" = legacy ]; then fi run_on() { docker exec -u www-data "$1" php occ "${@:2}"; } +# The app store sends the whole app list (~38 MB) and Nextcloud gives up after 120 s: on a slow day +# an install fails for reasons outside this test. Up to three tries. +install_app() { # container app + for i in 1 2 3; do run_on "$1" app:install "$2" >/dev/null 2>&1 && return 0; sleep 20; done + return 1 +} # Database settings for an instance: SQLite inside the container, or its own PostgreSQL container db_env() { @@ -109,7 +115,7 @@ if [ "$DB" != sqlite ]; then # while the restore loads tables in name order, accounts before mailboxes. docker run -d --name ngb-frt-imap --network "$NET" -e MAILNAME=mail.domain.tld -e MAIL_ADDRESS=user@domain.tld \ -e MAIL_PASS=mypassword ghcr.io/christophwurst/docker-imap-devel:latest >/dev/null - run_on ngb-frt-origin app:install mail >/dev/null || fail "install mail on the origin" + install_app ngb-frt-origin mail || fail "install mail on the origin (app store unreachable?)" run_on ngb-frt-origin config:system:set app.mail.verify-tls-peer --type=boolean --value=false >/dev/null sleep 10 run_on ngb-frt-origin mail:account:create frtest Test user@domain.tld ngb-frt-imap 993 ssl user@domain.tld mypassword \ @@ -118,7 +124,7 @@ if [ "$DB" != sqlite ]; then [ "$(sql origin 'select count(*) from oc_mail_accounts where drafts_mailbox_id is not null')" = 1 ] \ || fail "the mail account does not point at its mailboxes" # Deck installed and then switched off: its tables stay in the database and in the backup - run_on ngb-frt-origin app:install deck >/dev/null || fail "install deck on the origin" + install_app ngb-frt-origin deck || fail "install deck on the origin (app store unreachable?)" # a 4-byte character (emoji) in a table the restore has to create: utf8mb4 on MySQL/MariaDB sql origin "insert into oc_deck_boards (title, owner, color, archived, deleted_at, last_modified) values ('Plan 📌', 'frtest', '0087C5', 0, 0, 0)" >/dev/null run_on ngb-frt-origin app:disable deck >/dev/null From a03b7354b3d21ac6bf8034dd3a59f4dcec9c3ba0 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 13:06:49 +0200 Subject: [PATCH 13/26] README: tested disaster recoveries, backend measurements, restores without shell access; guide lists shell access; changelog for rc6 --- README.md | 51 ++++++++++++++++++++++++++++++++++++--- docs/DISASTER-RECOVERY.md | 2 ++ 2 files changed, 50 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 47ef3c1..aab5a0d 100644 --- a/README.md +++ b/README.md @@ -146,10 +146,31 @@ What you should do on the location (strongly recommended): - **Files and folders** of a user: in the web interface or `occ backup:restore`, into a new folder ("Restored "), merged into the original place, or replacing it. - **Raw restore** of any path (for example `config/`) to a local directory: `occ backup:restore ... --to-directory=DIR`. -- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see the how-to. +- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see + [docs/DISASTER-RECOVERY.md](docs/DISASTER-RECOVERY.md). It runs in maintenance mode, so that + nobody and no background job writes into the instance while it is being replaced. - **One user** (account, settings, files, calendars, contacts via user_migration): `occ backup:user:backup` and `occ backup:user:restore`, replacing the account or restoring it as `-bak`. See the how-to. +**Without shell access** (a hosted Nextcloud where you cannot run `occ`): files and folders can be +restored in the web interface. Restoring a whole user or the whole server needs `occ`, so on such +hosting it is something your provider (or whoever has shell access) runs. A restore mode that +works without maintenance mode, and so from the web interface, is on the roadmap. + +## Tested disaster recoveries + +Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECOVERY.md): + +| Backup | Location | Database | Result | +|---|---|---|---| +| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical | +| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical | +| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical | +| The same | Hetzner Storage Box (SMB) | MariaDB → SQLite | running | +| 51 GB test instance | OVH S3 | PostgreSQL → PostgreSQL | 23,519/23,519 checksums identical | + +The integration tests restore across every pair of PostgreSQL, MariaDB and SQLite. + ## Storage backend experiences Notes from actually running a backend, beyond what's in files_external's own documentation. @@ -206,6 +227,27 @@ Contributions welcome via PR — add what you ran into with your own provider. can be read on more connections: `occ config:app:set ng_backup restore_sftp_connections --value=4` (at most 8; the server must allow that many plus one at once for the user). +- **OVH Object Storage (S3, Warsaw, 2026-10):** works with the `amazons3` backend (path style, + region `waw`). A full backup of 1 TB from Germany ran at about 7 MiB/s, from Canada about + 6 MiB/s with BBR; a check of a 1 TB backup took 3.5 minutes (index files are fetched 16 at a + time); a restore of 138 GB from Warsaw to Canada took about 6 hours. Turn on versioning and + Object Lock on the bucket (see above); OVH charges no egress. + +- **Hetzner Storage Box over SFTP, full restore:** about 13-14 MiB/s from Germany to Canada + (122 ms) with the read-ahead of 1.0.0-rc.4; without it about 2.8 MiB/s. A Storage Box gets + slower with several connections reading at once (4-6 MiB/s for 2-4 connections, it reads from + disks), so one extra connection is the default. A Storage Box drops an SFTP connection that + stayed quiet for a while without telling the client; NG Backup checks and replaces it. + +- **SMB speed:** on a local network (Samba, no delay) a backup ran at about 25 MB/s and a + restore at about 28 MiB/s. Over a long line SMB waits for an answer to each request: from + Canada to a Storage Box in Germany (130 ms) about 2.6 MiB/s for a backup and 1.6 MiB/s for a + restore, and connecting a location with an existing backup reads its index at about one file a + second. SMB is meant for a local network; use SFTP or S3 for a location far away. When + `pecl install smbclient` fails (pecl.php.net was unreachable for hours on 2026-10-21), the + extension builds from its source: `libsmbclient-php` on GitHub, `phpize && ./configure && make + && make install`, then `docker-php-ext-enable smbclient`. + - **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself (e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping @@ -236,8 +278,11 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you - Location passwords passed with `-o password=...` are visible in the process list and shell history while the command runs. - Temporary files of a restore into a user's files end up in that user's trash. -- Restoring the database does not order tables by foreign keys; apps with cascading foreign keys - could lose rows (Nextcloud core does not use them). +- **Restoring a whole user or the whole server needs `occ`** (shell access); see "Without shell + access" above. +- **`--install-apps` depends on the app store.** Nextcloud downloads the whole app list (about + 38 MB) and gives up after 120 seconds; on a slow day apps are not installed. The restore then + lists them: install them afterwards with `occ app:install`, their data is restored already. - A pack whose index upload failed stays on the location until removed by hand. ## Commands diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md index 06bbb47..86e35fc 100644 --- a/docs/DISASTER-RECOVERY.md +++ b/docs/DISASTER-RECOVERY.md @@ -10,6 +10,8 @@ Nextcloud back from an NG Backup location. Do them in this order. - [ ] A new server with the **same Nextcloud major version** as the old one (for example 34), a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for the whole backup plus a margin. +- [ ] **Shell access** to the new server, to run `occ` (on a hosted Nextcloud without it, your + provider runs these steps). - [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by hand). Keep their packages somewhere safe beforehand. From fd12c586b6b1877a3c87a2c451eb6929d6a7b1b8 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 13:07:09 +0200 Subject: [PATCH 14/26] Roadmap: restore mode for hosted Nextcloud, read-ahead for S3 and SMB; changelog for rc6 --- CHANGELOG.md | 15 +++++++++++++++ README.md | 6 ++++-- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e4db1a7..40b80a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,21 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th beta/daily update channel, which would hide the app from most admins. The stability stage is instead carried by the app description, the git tag, the release, and this file. +## [Unreleased] + +### Fixed +- A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table + (of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the + table got 3-byte utf8. Tables the restore creates now get this installation's character set and + collation (utf8mb4). Found in a disaster recovery simulation onto MariaDB. + +### Changed +- README: a roadmap, the tested disaster recoveries, measurements for OVH S3, the Storage Box over + SFTP and SMB, how to build the SMB extension when PECL is down, and what works without shell + access. The guide lists shell access among what you need. +- The full-restore integration test checks a 4-byte character in a table the restore creates, and + retries app installs from the app store. + ## [1.0.0-rc.5] - 2026-10-22 Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP diff --git a/README.md b/README.md index fd80c40..b4fa82d 100644 --- a/README.md +++ b/README.md @@ -265,7 +265,7 @@ a 👍 on its issue helps decide what comes first. system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on while the NG Backup page is open ([#13](https://github.com/dillardblom/ng_backup/issues/13)). - An optional label for a backup started with "Back up now". -- Faster full restores from S3, reading ahead as restores over SFTP do since 1.0.0-rc.4. +- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4. **1.1.0** - Restore without the recovery kit: the backup key from the location itself, opened with one of @@ -278,7 +278,9 @@ a 👍 on its issue helps decide what comes first. - Faster backups: reading and uploading at the same time. **Later** -- A full restore onto the existing installation, without first installing a fresh Nextcloud. +- A full restore onto the existing installation, without first installing a fresh Nextcloud, in a + restore mode of its own instead of maintenance mode, so that it can also run from the web + interface (for hosted Nextcloud without shell access). - Users restoring their own files from the Files app. - Primary object storage and External Storage mounts (see Known limitations). - An option not to keep the backup key on the location (the recovery kit is then the only copy). From fb5b234e8ae274753e353dcd104e510a29bdb5dd Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 14:43:48 +0200 Subject: [PATCH 15/26] Restore: a blob that does not decrypt as first read is read once more on a request of its own before the restore gives up --- lib/Repository/Repository.php | 21 +++- tests/Unit/Repository/DamagedReadTest.php | 116 ++++++++++++++++++++++ 2 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 tests/Unit/Repository/DamagedReadTest.php diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index 2146e20..b23570a 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -895,8 +895,27 @@ final class Repository { } catch (BackendException) { throw $e; } + $pack = $moved; + } + try { + return $this->decodeBlob($id, $encrypted); + } catch (\Exception $damaged) { + // Bytes that do not decrypt: damaged on the location, or a read that went wrong on the + // way (a stream that lost its place: an SMB restore stopped after 17 hours like this). + // Read exactly this blob once more, on a request of its own; only a second failure + // means the data itself is damaged. + $this->packReader->close(); + try { + $again = $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length); + } catch (\Exception) { + throw $damaged; + } + $data = $this->decodeBlob($id, $again); + if (function_exists('OCP\Log\logger')) { + \OCP\Log\logger('ng_backup')->warning('NG Backup: blob ' . $id . ' did not decrypt as first read from pack ' . $pack . '; read again, it is fine'); + } + return $data; } - return $this->decodeBlob($id, $encrypted); } /** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */ diff --git a/tests/Unit/Repository/DamagedReadTest.php b/tests/Unit/Repository/DamagedReadTest.php new file mode 100644 index 0000000..d532496 --- /dev/null +++ b/tests/Unit/Repository/DamagedReadTest.php @@ -0,0 +1,116 @@ +root = $this->tempDir(); + $this->keys = KeyRing::generate(); + $this->src = $this->tempDir(); + for ($i = 0; $i < 6; $i++) { + file_put_contents("$this->src/f$i", random_bytes(3 * 1048576)); + } + $repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw')); + $this->snapshot = $repo->backupDirectory($this->src)['snapshot']; + } + + public function testABlobThatCameInDamagedOnceIsReadAgain(): void { + $backend = new GarblingBackend(new LocalBackend($this->root), streams: 1); + $target = $this->tempDir(); + + $this->assertSame(6, Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $target)); + + $this->assertSame(self::hashTree($this->src), self::hashTree($target)); + $this->assertSame(0, $backend->streams, 'the damaged read happened'); + } + + public function testDataDamagedOnTheLocationStillStopsTheRestore(): void { + $backend = new GarblingBackend(new LocalBackend($this->root), streams: 1000, ranges: 1000); + + $this->expectException(\Exception::class); + Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $this->tempDir()); + } +} + +/** Pack reads that flip a byte: the first $streams pack streams and $ranges range reads. */ +final class GarblingBackend implements IBackend { + use \OCA\NgBackup\Backend\GetsOneByOne; + + public function __construct( + private IBackend $inner, + public int $streams = 0, + public int $ranges = 0, + ) { + } + + public function put(string $path, $stream): void { + $this->inner->put($path, $stream); + } + + public function get(string $path) { + $fh = $this->inner->get($path); + if (!str_starts_with($path, 'packs/') || $this->streams <= 0) { + return $fh; + } + $this->streams--; + $data = stream_get_contents($fh); + fclose($fh); + for ($k = 100; $k < strlen($data); $k += 65536) { + $data[$k] = chr(ord($data[$k]) ^ 0x55); + } + $out = fopen('php://temp', 'w+b'); + fwrite($out, $data); + rewind($out); + return $out; + } + + public function getRange(string $path, int $offset, int $length): string { + $data = $this->inner->getRange($path, $offset, $length); + if (str_starts_with($path, 'packs/') && $this->ranges > 0) { + $this->ranges--; + $data[0] = chr(ord($data[0]) ^ 0x55); + } + return $data; + } + + public function exists(string $path): bool { + return $this->inner->exists($path); + } + + public function move(string $from, string $to): void { + $this->inner->move($from, $to); + } + + public function list(string $prefix): array { + return $this->inner->list($prefix); + } + + public function listFiles(string $dir): array { + return $this->inner->listFiles($dir); + } + + public function delete(string $path): void { + $this->inner->delete($path); + } +} From 67cc350b8c79ad1c14d3fd72c33d42ebff395f14 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 16:38:54 +0200 Subject: [PATCH 16/26] Daily backup time in the administrator's time zone (#19); a time saved before is converted once so it shows as it ran --- CHANGELOG.md | 8 ++ lib/BackgroundJob/BackupJob.php | 11 +-- lib/Controller/AdminApiController.php | 16 ++-- lib/Service/Schedule.php | 108 ++++++++++++++++++++++++++ src/api.js | 2 +- src/components/Backups.vue | 16 ++-- src/format.js | 21 +++++ tests/Unit/Service/ScheduleTest.php | 71 +++++++++++++++++ 8 files changed, 233 insertions(+), 20 deletions(-) create mode 100644 lib/Service/Schedule.php create mode 100644 tests/Unit/Service/ScheduleTest.php diff --git a/CHANGELOG.md b/CHANGELOG.md index 40b80a9..1d37547 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,14 @@ instead carried by the app description, the git tag, the release, and this file. ## [Unreleased] ### Fixed +- The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in + Berlin (#19). The time is now saved with the administrator's time zone and shown with it. A time + saved before is converted once to the zone of the server or an administrator, so it shows as it + effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer) and nothing moves. +- A full restore over SMB stopped after 17 hours with "Decryption failed": a read over a long + line handed out wrong bytes (the backup itself was fine, the same files restored identically + afterwards). A blob that does not decrypt as first read is now read once more on a request of + its own, and only a second failure stops the restore. - A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table (of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the table got 3-byte utf8. Tables the restore creates now get this installation's character set and diff --git a/lib/BackgroundJob/BackupJob.php b/lib/BackgroundJob/BackupJob.php index 4171fe4..76dd5fc 100644 --- a/lib/BackgroundJob/BackupJob.php +++ b/lib/BackgroundJob/BackupJob.php @@ -32,6 +32,7 @@ final class BackupJob extends TimedJob { private RunMapper $runs, private IAppConfig $appConfig, private LoggerInterface $logger, + private \OCA\NgBackup\Service\Schedule $schedule, ) { parent::__construct($time); // Well below the usual 5-minute cron interval: the last run is stamped a moment after cron @@ -101,14 +102,10 @@ final class BackupJob extends TimedJob { $this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due); } - /** Daily schedule: appconfig ng_backup/schedule_time "HH:MM" (server time); empty = no schedule. */ + /** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */ private function startScheduled(): void { - $at = $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''); - if (!preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $at)) { - return; - } - $todayAt = (int)strtotime(date('Y-m-d') . ' ' . $at); - if (time() < $todayAt) { + $todayAt = $this->schedule->todayAt(); + if ($todayAt === null || time() < $todayAt) { return; } foreach ($this->targets->list() as $t) { diff --git a/lib/Controller/AdminApiController.php b/lib/Controller/AdminApiController.php index 1fec3bb..886e415 100644 --- a/lib/Controller/AdminApiController.php +++ b/lib/Controller/AdminApiController.php @@ -47,6 +47,7 @@ final class AdminApiController extends Controller { private \OCA\NgBackup\Service\AlertService $alerts, private \OCA\NgBackup\Service\KeySlotService $slots, private \OCA\NgBackup\Service\PruneService $prune, + private \OCA\NgBackup\Service\Schedule $schedule, ) { parent::__construct(Application::APP_ID, $request); } @@ -72,7 +73,8 @@ final class AdminApiController extends Controller { 'runs' => $runs, 'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown), 'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS, - 'schedule' => $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''), + 'schedule' => $this->schedule->get()['time'], + 'scheduleTimezone' => $this->schedule->get()['timezone'], ]); } @@ -240,12 +242,14 @@ final class AdminApiController extends Controller { return new JSONResponse($this->runJson($run)); } - public function setSchedule(string $time): JSONResponse { - if ($time !== '' && !preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time)) { - return new JSONResponse(['error' => 'Use HH:MM or empty'], Http::STATUS_BAD_REQUEST); + /** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */ + public function setSchedule(string $time, string $timezone = ''): JSONResponse { + try { + $this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']); + } catch (\InvalidArgumentException $e) { + return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST); } - $this->appConfig->setValueString(Application::APP_ID, 'schedule_time', $time); - return new JSONResponse(['schedule' => $time]); + return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]); } public function snapshots(int $targetId): JSONResponse { diff --git a/lib/Service/Schedule.php b/lib/Service/Schedule.php new file mode 100644 index 0000000..4547061 --- /dev/null +++ b/lib/Service/Schedule.php @@ -0,0 +1,108 @@ +appConfig->getValueString(Application::APP_ID, self::TIME_KEY, ''); + $zone = $this->appConfig->getValueString(Application::APP_ID, self::ZONE_KEY, ''); + if ($zone === '' || !self::validZone($zone)) { + $zone = $this->guessZone(); + if ($time !== '' && self::validTime($time)) { + $time = self::convert($time, 'UTC', $zone); + $this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time); + } + $this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone); + } + return ['time' => $time, 'timezone' => $zone]; + } + + /** @throws \InvalidArgumentException for a malformed time or an unknown zone */ + public function set(string $time, string $zone): void { + if ($time !== '' && !self::validTime($time)) { + throw new \InvalidArgumentException('Use HH:MM or empty'); + } + if (!self::validZone($zone)) { + throw new \InvalidArgumentException('Unknown time zone: ' . $zone); + } + $this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone); + $this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time); + } + + /** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */ + public function todayAt(?int $now = null): ?int { + ['time' => $time, 'timezone' => $zone] = $this->get(); + if ($time === '' || !self::validTime($time)) { + return null; + } + $tz = new \DateTimeZone($zone); + $day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d'); + return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp(); + } + + /** + * "HH:MM" in zone $from as "HH:MM" in zone $to, at today's offsets. + * + * @param non-empty-string $from + * @param non-empty-string $to + */ + public static function convert(string $time, string $from, string $to, ?int $now = null): string { + $day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone(new \DateTimeZone($from))->format('Y-m-d'); + return (new \DateTimeImmutable($day . ' ' . $time, new \DateTimeZone($from)))->setTimezone(new \DateTimeZone($to))->format('H:i'); + } + + public static function validTime(string $time): bool { + return preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time) === 1; + } + + /** @psalm-assert-if-true non-empty-string $zone */ + public static function validZone(string $zone): bool { + return in_array($zone, \DateTimeZone::listIdentifiers(), true); + } + + /** + * The server's default_timezone when set, else the first administrator's, else UTC. + * + * @return non-empty-string + */ + private function guessZone(): string { + $zone = $this->config->getSystemValueString('default_timezone', ''); + if ($zone !== '' && $zone !== 'UTC' && self::validZone($zone)) { + return $zone; + } + foreach ($this->groups->get('admin')?->getUsers() ?? [] as $user) { + $zone = $this->config->getUserValue($user->getUID(), 'core', 'timezone', ''); + if ($zone !== '' && self::validZone($zone)) { + return $zone; + } + } + return 'UTC'; + } +} diff --git a/src/api.js b/src/api.js index fad46dd..aa29792 100644 --- a/src/api.js +++ b/src/api.js @@ -33,7 +33,7 @@ export default { testTarget: (id) => call('post', `/targets/${id}/test`), setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }), startBackup: (targetId, label) => call('post', '/backups', { targetId, label }), - setSchedule: (time) => call('put', '/schedule', { time }), + setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }), snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`), browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }), startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }), diff --git a/src/components/Backups.vue b/src/components/Backups.vue index bb500de..82ac031 100644 --- a/src/components/Backups.vue +++ b/src/components/Backups.vue @@ -6,8 +6,8 @@
- - {{ t('ng_backup', 'Save') }} + + {{ t('ng_backup', 'Save') }}
@@ -79,15 +79,19 @@ import NcNoteCard from '@nextcloud/vue/components/NcNoteCard' import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection' import NcTextField from '@nextcloud/vue/components/NcTextField' import api from '../api.js' -import { formatDateTime, formatDate, formatSize } from '../format.js' +import { formatDateTime, formatDate, formatSize, toZone } from '../format.js' import { confirmPassword } from '@nextcloud/password-confirmation' import { reactive } from 'vue' const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } }) const emit = defineEmits(['changed', 'show-all-runs']) -const schedule = ref(props.status.schedule) +// The time is saved in the administrator's own zone (the browser's); a schedule saved in another +// zone is shown in this one, so it reads as the moment it runs here. +const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC' +const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s +const schedule = ref(shown(props.status.schedule)) const error = ref('') -watch(() => props.status.schedule, (s) => { schedule.value = s }) +watch(() => props.status.schedule, (s) => { schedule.value = shown(s) }) const trash = reactive({}) async function loadTrash() { @@ -131,7 +135,7 @@ async function start(target) { async function saveSchedule() { error.value = '' try { - await api.setSchedule(schedule.value.trim()) + await api.setSchedule(schedule.value.trim(), browserZone) emit('changed') } catch (e) { error.value = e.message diff --git a/src/format.js b/src/format.js index add6404..c7496c8 100644 --- a/src/format.js +++ b/src/format.js @@ -26,3 +26,24 @@ export function formatSize(bytes) { } return i === 0 ? `${b} B` : `${b.toFixed(1)} ${units[i]}` } + +/** + * A time of day "HH:MM" in zone `from`, as "HH:MM" in zone `to`, at today's offsets. + * + * @param {string} hhmm the time + * @param {string} from IANA zone it is in + * @param {string} to IANA zone to show it in + * @return {string} + */ +export function toZone(hhmm, from, to) { + const [h, m] = hhmm.split(':').map(Number) + const parts = (zone, date) => Object.fromEntries(new Intl.DateTimeFormat('en-GB', { + timeZone: zone, hourCycle: 'h23', year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit', + }).formatToParts(date).map((p) => [p.type, p.value])) + const today = parts(from, new Date()) + const guess = Date.UTC(+today.year, +today.month - 1, +today.day, h, m) + const seen = parts(from, new Date(guess)) + const offset = Date.UTC(+seen.year, +seen.month - 1, +seen.day, +seen.hour, +seen.minute) - guess + const there = parts(to, new Date(guess - offset)) + return `${there.hour}:${there.minute}` +} diff --git a/tests/Unit/Service/ScheduleTest.php b/tests/Unit/Service/ScheduleTest.php new file mode 100644 index 0000000..db8458d --- /dev/null +++ b/tests/Unit/Service/ScheduleTest.php @@ -0,0 +1,71 @@ +createMock(IAppConfig::class); + $app->method('getValueString')->willReturnCallback(fn (string $a, string $k, string $d = '') => $this->values[$k] ?? $d); + $app->method('setValueString')->willReturnCallback(function (string $a, string $k, string $v): bool { + $this->values[$k] = $v; + return true; + }); + $config = $this->createMock(IConfig::class); + $config->method('getSystemValueString')->willReturn($serverZone); + $config->method('getUserValue')->willReturn($adminZone); + $user = $this->createMock(IUser::class); + $user->method('getUID')->willReturn('admin'); + $group = $this->createMock(IGroup::class); + $group->method('getUsers')->willReturn([$user]); + $groups = $this->createMock(IGroupManager::class); + $groups->method('get')->willReturn($group); + return new Schedule($app, $config, $groups); + } + + public function testATimeSavedWithoutAZoneShowsAsItRanInTheAdministratorsZone(): void { + $this->values = [Schedule::TIME_KEY => '02:30']; + + $s = $this->schedule(adminZone: 'Europe/Berlin')->get(); + + // it ran at 02:30 UTC: in summer that is 04:30 in Berlin, in winter 03:30 + $summer = Schedule::convert('02:30', 'UTC', 'Europe/Berlin', strtotime('2026-07-01 12:00 UTC')); + $this->assertSame('04:30', $summer); + $this->assertSame('Europe/Berlin', $s['timezone']); + $this->assertSame(Schedule::convert('02:30', 'UTC', 'Europe/Berlin'), $s['time']); + } + + public function testTheServersDefaultZoneComesFirst(): void { + $this->values = [Schedule::TIME_KEY => '01:00']; + $this->assertSame('America/Toronto', $this->schedule('America/Toronto', 'Europe/Berlin')->get()['timezone']); + } + + public function testWithoutAnyZoneItStaysUtc(): void { + $this->values = [Schedule::TIME_KEY => '02:30']; + $this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get()); + } + + public function testTodayAtIsTheMomentInTheSchedulesZone(): void { + $this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin']; + $at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC')); + $this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC'); + } + + public function testAnUnknownZoneOrABadTimeIsRefused(): void { + $s = $this->schedule(); + $this->expectException(\InvalidArgumentException::class); + $s->set('03:00', 'Mars/Olympus'); + } +} From cf4bd9db7fd63d06310716c428d78b95391cf42e Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 19:45:49 +0200 Subject: [PATCH 17/26] Review fixes: the database restore also reads a damaged blob once more; schedule conversion right on days the clocks change; the field follows a changed schedule time zone --- lib/Repository/Repository.php | 8 +++++++- src/components/Backups.vue | 2 +- src/format.js | 18 ++++++++++++------ tests/Unit/Repository/DamagedReadTest.php | 17 +++++++++++++++++ 4 files changed, 37 insertions(+), 8 deletions(-) diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index b23570a..1c34ea0 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -845,7 +845,13 @@ final class Repository { try { foreach ($this->backend->getRanges($ranges) as $i => $data) { foreach ($parts[$i] as [$id, $at, $length]) { - yield $this->decodeBlob($id, substr($data, $at, $length)); + try { + $blob = $this->decodeBlob($id, substr($data, $at, $length)); + } catch (\Exception) { + // a read that went wrong on the way (see loadBlob()): this blob on its own + $blob = $this->loadBlob($id); + } + yield $blob; $done++; } } diff --git a/src/components/Backups.vue b/src/components/Backups.vue index 82ac031..e0d915d 100644 --- a/src/components/Backups.vue +++ b/src/components/Backups.vue @@ -91,7 +91,7 @@ const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC' const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s const schedule = ref(shown(props.status.schedule)) const error = ref('') -watch(() => props.status.schedule, (s) => { schedule.value = shown(s) }) +watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) }) const trash = reactive({}) async function loadTrash() { diff --git a/src/format.js b/src/format.js index c7496c8..5774270 100644 --- a/src/format.js +++ b/src/format.js @@ -33,17 +33,23 @@ export function formatSize(bytes) { * @param {string} hhmm the time * @param {string} from IANA zone it is in * @param {string} to IANA zone to show it in + * @param {Date} now the day to convert on (today) * @return {string} */ -export function toZone(hhmm, from, to) { +export function toZone(hhmm, from, to, now = new Date()) { const [h, m] = hhmm.split(':').map(Number) const parts = (zone, date) => Object.fromEntries(new Intl.DateTimeFormat('en-GB', { timeZone: zone, hourCycle: 'h23', year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit', }).formatToParts(date).map((p) => [p.type, p.value])) - const today = parts(from, new Date()) - const guess = Date.UTC(+today.year, +today.month - 1, +today.day, h, m) - const seen = parts(from, new Date(guess)) - const offset = Date.UTC(+seen.year, +seen.month - 1, +seen.day, +seen.hour, +seen.minute) - guess - const there = parts(to, new Date(guess - offset)) + const offsetAt = (at) => { + const seen = parts(from, new Date(at)) + return Date.UTC(+seen.year, +seen.month - 1, +seen.day, +seen.hour, +seen.minute) - at + } + const today = parts(from, now) + const wall = Date.UTC(+today.year, +today.month - 1, +today.day, h, m) + // the offset at the wall time itself: on a day the clocks change it differs from the one at + // midnight, so it is looked up a second time at the moment found first + const at = wall - offsetAt(wall - offsetAt(wall)) + const there = parts(to, new Date(at)) return `${there.hour}:${there.minute}` } diff --git a/tests/Unit/Repository/DamagedReadTest.php b/tests/Unit/Repository/DamagedReadTest.php index d532496..a281fd4 100644 --- a/tests/Unit/Repository/DamagedReadTest.php +++ b/tests/Unit/Repository/DamagedReadTest.php @@ -45,6 +45,23 @@ class DamagedReadTest extends TestCase { $this->assertSame(0, $backend->streams, 'the damaged read happened'); } + public function testATableDumpThatCameInDamagedOnceIsReadAgain(): void { + $repo = Repository::openWithKey(new LocalBackend($this->root), $this->keys); + $stats = []; + $writer = $repo->blobWriter($stats); + $lines = []; + for ($i = 0; $i < 20000; $i++) { + $lines[] = json_encode(['i' => $i, 'pad' => str_repeat('x', $i % 400)]); + } + $writer->write(implode("\n", $lines) . "\n"); + $blobs = $writer->finish(); + $repo->flushPacks(); + $backend = new GarblingBackend(new LocalBackend($this->root), ranges: 1); + + $this->assertSame($lines, iterator_to_array(Repository::openWithKey($backend, $this->keys)->readLines($blobs), false)); + $this->assertSame(0, $backend->ranges, 'the damaged read happened'); + } + public function testDataDamagedOnTheLocationStillStopsTheRestore(): void { $backend = new GarblingBackend(new LocalBackend($this->root), streams: 1000, ranges: 1000); From 133f5f9d34e37ba4bd169c8c86edd992e3e3100a Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 19:47:30 +0200 Subject: [PATCH 18/26] Schedule: say why a server zone of UTC is passed over --- lib/Service/Schedule.php | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/Service/Schedule.php b/lib/Service/Schedule.php index 4547061..326c537 100644 --- a/lib/Service/Schedule.php +++ b/lib/Service/Schedule.php @@ -88,7 +88,9 @@ final class Schedule { } /** - * The server's default_timezone when set, else the first administrator's, else UTC. + * The server's default_timezone when set, else the first administrator's, else UTC. A server + * zone of UTC is passed over on purpose: it is mostly what an image or installer left there, + * while the administrator who set the time read it as their own (#19). * * @return non-empty-string */ From 343389b98bef4791ed66090d99519842ff581502 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 19:48:22 +0200 Subject: [PATCH 19/26] Changelog: the second read covers the database phase too --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d37547..cb96595 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,8 +16,8 @@ instead carried by the app description, the git tag, the release, and this file. effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer) and nothing moves. - A full restore over SMB stopped after 17 hours with "Decryption failed": a read over a long line handed out wrong bytes (the backup itself was fine, the same files restored identically - afterwards). A blob that does not decrypt as first read is now read once more on a request of - its own, and only a second failure stops the restore. + afterwards). A blob that does not decrypt as first read, of a file or of the database, is now + read once more on a request of its own, and only a second failure stops the restore. - A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table (of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the table got 3-byte utf8. Tables the restore creates now get this installation's character set and From b4f20cb0707491b91ca9dea4724adc8bcb46999c Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 19:48:57 +0200 Subject: [PATCH 20/26] Version 0.9.12 (1.0.0-rc.6) --- appinfo/info.xml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/appinfo/info.xml b/appinfo/info.xml index ba68b03..79b83ce 100644 --- a/appinfo/info.xml +++ b/appinfo/info.xml @@ -42,7 +42,7 @@ welcome: use the "Share your experience" link on the admin page. Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if you need help with restoring a backup after a crash.]]> - 0.9.11 + 0.9.12 agpl Dillard Blom NgBackup @@ -53,9 +53,9 @@ you need help with restoring a backup after a crash.]]> https://github.com/dillardblom/ng_backup https://github.com/dillardblom/ng_backup/issues https://github.com/dillardblom/ng_backup - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/1-backups.png - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/2-restore.png - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/3-settings.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/1-backups.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/2-restore.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/3-settings.png sodium From 80324117891b75135129eb9c372c6b4a5be9f92a Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 19:50:27 +0200 Subject: [PATCH 21/26] Restore: the warning for a blob read again says how the first read differed (zeros, shifted, other bytes) --- lib/Repository/Repository.php | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index 1c34ea0..bfd01e9 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -918,12 +918,31 @@ final class Repository { } $data = $this->decodeBlob($id, $again); if (function_exists('OCP\Log\logger')) { - \OCP\Log\logger('ng_backup')->warning('NG Backup: blob ' . $id . ' did not decrypt as first read from pack ' . $pack . '; read again, it is fine'); + \OCP\Log\logger('ng_backup')->warning('NG Backup: blob ' . $id . ' did not decrypt as first read from pack ' . $pack . ' at ' . $offset . '; read again, it is fine. First read: ' . self::howItDiffered($encrypted, $again)); } return $data; } } + /** + * What a read that went wrong looked like next to the right bytes, for the log: where it starts + * to differ, and whether the rest is zeros or the right bytes from another place (a stream that + * lost its place). + */ + private static function howItDiffered(string $wrong, string $right): string { + $at = strspn($wrong ^ $right, "\0"); + $text = strlen($wrong) . ' bytes, differ from byte ' . $at . ' of ' . strlen($right); + $tail = substr($wrong, $at, 64); + if ($tail === '') { + return $text . ' (too short)'; + } + if (trim($tail, "\0") === '') { + return $text . ' (zeros)'; + } + $found = strpos($right, $tail); + return $text . ($found !== false ? ' (the right bytes from ' . ($found - $at) . ' bytes further on)' : ' (other bytes)'); + } + /** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */ private function decodeBlob(string $id, string $encrypted): string { [, , , $raw, $flags] = $this->index->get($id); From 384b2d27927d596aefca4f9aff92520d75106d70 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 20:09:58 +0200 Subject: [PATCH 22/26] Full-restore test: the emoji deck board sets archived as a boolean (PostgreSQL refuses 0) --- tests/integration/full-restore-roundtrip.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/integration/full-restore-roundtrip.sh b/tests/integration/full-restore-roundtrip.sh index 923cdc3..d621713 100755 --- a/tests/integration/full-restore-roundtrip.sh +++ b/tests/integration/full-restore-roundtrip.sh @@ -126,7 +126,7 @@ if [ "$DB" != sqlite ]; then # Deck installed and then switched off: its tables stay in the database and in the backup install_app ngb-frt-origin deck || fail "install deck on the origin (app store unreachable?)" # a 4-byte character (emoji) in a table the restore has to create: utf8mb4 on MySQL/MariaDB - sql origin "insert into oc_deck_boards (title, owner, color, archived, deleted_at, last_modified) values ('Plan 📌', 'frtest', '0087C5', 0, 0, 0)" >/dev/null + sql origin "insert into oc_deck_boards (title, owner, color, archived, deleted_at, last_modified) values ('Plan 📌', 'frtest', '0087C5', false, 0, 0)" >/dev/null run_on ngb-frt-origin app:disable deck >/dev/null ok "origin has a synced Mail account and the tables of a disabled app" fi From 3ac329831358fc1958f46008ef2c0d2a74af8d1e Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Thu, 22 Oct 2026 20:30:18 +0200 Subject: [PATCH 23/26] Database restore: close the pack stream of a blob read again before the next range --- lib/Repository/Repository.php | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index bfd01e9..56a837f 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -850,6 +850,8 @@ final class Repository { } catch (\Exception) { // a read that went wrong on the way (see loadBlob()): this blob on its own $blob = $this->loadBlob($id); + // its pack stream is closed before the next range is read on the same connection + $this->packReader->close(); } yield $blob; $done++; From 9e9234e2a8e9b576fe8bf05f9067e23839cf5936 Mon Sep 17 00:00:00 2001 From: Dillard Blom Date: Fri, 23 Oct 2026 13:29:47 +0200 Subject: [PATCH 24/26] Admin page: sections up to 1400px wide, so the runs table fits without scrolling --- src/AdminSettings.vue | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/AdminSettings.vue b/src/AdminSettings.vue index eae0342..b0ae60d 100644 --- a/src/AdminSettings.vue +++ b/src/AdminSettings.vue @@ -90,7 +90,7 @@ onBeforeUnmount(() => clearTimeout(timer))