Compare commits

..

29 Commits

Author SHA1 Message Date
dillard a0c4abb60a Version 1.0.0: same code as 1.0.0-rc.6; changelog, README status
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-29 01:32:12 +01:00
Dillard Blom 965440d1f7 Merge pull request #20 from dillardblom/rc6
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.6
2026-10-24 01:04:23 +02:00
dillard 3a0f882a38 1.0.0-rc.6: changelog, SMB disaster recovery result in the README, 1.0.1 items (salvage mode, leftover restore files, single-file restore)
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-24 00:40:51 +02:00
dillard 632d2a2838 Admin page: widen the sections through their width; NcSettingsSection sets width, not max-width
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-23 18:10:48 +02:00
dillard 9e9234e2a8 Admin page: sections up to 1400px wide, so the runs table fits without scrolling
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-23 13:29:47 +02:00
dillard 3ac3298313 Database restore: close the pack stream of a blob read again before the next range 2026-10-22 20:30:18 +02:00
dillard 384b2d2792 Full-restore test: the emoji deck board sets archived as a boolean (PostgreSQL refuses 0) 2026-10-22 20:09:58 +02:00
dillard 8032411789 Restore: the warning for a blob read again says how the first read differed (zeros, shifted, other bytes) 2026-10-22 19:50:27 +02:00
dillard b4f20cb070 Version 0.9.12 (1.0.0-rc.6) 2026-10-22 19:48:57 +02:00
dillard 343389b98b Changelog: the second read covers the database phase too 2026-10-22 19:48:22 +02:00
dillard 133f5f9d34 Schedule: say why a server zone of UTC is passed over 2026-10-22 19:47:30 +02:00
dillard cf4bd9db7f Review fixes: the database restore also reads a damaged blob once more; schedule conversion right on days the clocks change; the field follows a changed schedule time zone 2026-10-22 19:45:49 +02:00
dillard 67cc350b8c Daily backup time in the administrator's time zone (#19); a time saved before is converted once so it shows as it ran 2026-10-22 16:38:54 +02:00
dillard fb5b234e8a Restore: a blob that does not decrypt as first read is read once more on a request of its own before the restore gives up 2026-10-22 14:43:48 +02:00
dillard fd12c586b6 Roadmap: restore mode for hosted Nextcloud, read-ahead for S3 and SMB; changelog for rc6 2026-10-22 13:07:09 +02:00
dillard 6ffe979ea7 Merge branch 'polish' into rc6 2026-10-22 13:06:59 +02:00
dillard a03b7354b3 README: tested disaster recoveries, backend measurements, restores without shell access; guide lists shell access; changelog for rc6 2026-10-22 13:06:49 +02:00
dillard be1c628f07 Full-restore test: install Mail and Deck with up to three tries (the app store is slow at times) 2026-10-22 03:38:30 +02:00
dillard a8355df12e Full restore: tables the restore creates get this installation's charset and collation (utf8mb4 on MySQL/MariaDB); test with an emoji 2026-10-22 03:36:46 +02:00
dillard cd251cffb3 PLAN: SMB index reads one at a time (1.1.0) 2026-10-21 20:02:43 +02:00
dillard 0b4872599d PLAN: the 1.0.1 cron PR closes #13 2026-10-21 19:51:57 +02:00
dillard b7d09c9784 PLAN: 1.1.0 faster SMB (local and long-line measurements) 2026-10-21 09:27:15 +02:00
dillard 24686897c3 PLAN: DR tests across databases (integration for every crossing; S3 sim onto MariaDB, SMB sim onto SQLite before 1.0.0; nc01 onto MariaDB after) 2026-10-20 23:12:19 +02:00
dillard 315e83b847 PLAN: 1.0.1 progress in the database phase and a waiting notice, so a backup does not look stuck 2026-10-20 20:49:44 +02:00
dillard b728db36c7 README: roadmap 2026-10-19 21:35:23 +02:00
dillard 6ed0a555cf PLAN: label field next to Back up now (1.0.1); per-location schedules, every N hours, locations one after the other, editable notes (1.1.0) 2026-10-19 21:26:20 +02:00
dillard 221ed9dc17 PLAN: the cron warning says cron is needed for all backups, links to how to set it up, and that without it a backup started with the button only moves while the page is open 2026-10-19 21:18:23 +02:00
dillard 285153a968 PLAN: 1.0.1 cron warning says to keep the page open under AJAX, with a link to the Nextcloud docs 2026-10-19 21:16:23 +02:00
dillard 6c8a78f6bb PLAN: 1.0.1 background jobs without system cron: warn, and let the open admin page move a backup 2026-10-19 20:41:11 +02:00
17 changed files with 638 additions and 41 deletions
+35
View File
@@ -7,6 +7,41 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
beta/daily update channel, which would hide the app from most admins. The stability stage is
instead carried by the app description, the git tag, the release, and this file.
## [1.0.0] - 2026-10-29
First stable release: the same code as 1.0.0-rc.6, which had no blocking problems reported in its
week out (about 220 downloads). Only the version number and the README changed. The fixes for
what was reported (#22, #24) follow in 1.0.1.
## [1.0.0-rc.6] - 2026-10-24
Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery
simulations and a user report. If no problems are reported for about a week, 1.0.0 will be this
same code.
### Fixed
- The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in
Berlin (#19). The time is now saved with the administrator's time zone and shown with it. A time
saved before is converted once to the zone of the server or an administrator, so it shows as it
effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer) and nothing moves.
- A full restore over SMB stopped after 17 hours with "Decryption failed": a read over a long
line handed out wrong bytes (the backup itself was fine, the same files restored identically
afterwards). A blob that does not decrypt as first read, of a file or of the database, is now
read once more on a request of its own, and only a second failure stops the restore.
- A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table
(of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the
table got 3-byte utf8. Tables the restore creates now get this installation's character set and
collation (utf8mb4). Found in a disaster recovery simulation onto MariaDB.
### Changed
- The admin page uses more of the screen (sections up to 1400 pixels instead of 900), so the
list of backup runs fits without scrolling sideways.
- README: a roadmap, the tested disaster recoveries, measurements for OVH S3, the Storage Box over
SFTP and SMB, how to build the SMB extension when PECL is down, and what works without shell
access. The guide lists shell access among what you need.
- The full-restore integration test checks a 4-byte character in a table the restore creates, and
retries app installs from the app store.
## [1.0.0-rc.5] - 2026-10-22
Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
+61 -3
View File
@@ -395,11 +395,53 @@ in README "Known limitations":
- **1.0.1:**
1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron
staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue
#13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles.
2. Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit
oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21):
a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode`
(`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de
laatste run te lang geleden: melding met de cron-regel erbij ("Backups only move while
someone has Nextcloud open"), en "waiting for background jobs" als een lopende backup lang
geen stap zette.
b. **De pagina laat de backup lopen** zolang ze openstaat: in `ajax`/`webcron` voert de
statusaanroep die de pagina toch al herhaalt ook een korte, begrensde stap uit (zoals de
korte stappen die AJAX-cron nu al krijgt). "Back up now" werkt dan ook zonder cron zolang
het tabblad openstaat; de melding zegt dat geplande backups echte cron nodig hebben.
De melding zegt (zonder werkende systeem-cron): backups hebben systeem-cron nodig, ook de
gewone nachtelijke; zo stel je cron in (link naar de Nextcloud-documentatie over
achtergrondjobs voor de versie van de server, nu
https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html,
versienummer uit de server, niet vast); zolang er geen cron is, loopt een backup die je
met de knop start alleen door terwijl de NG Backup-pagina openstaat.
c. **Voortgang in de databasefase** (issue #13): nu staat er alleen "running (db)", wat er
hetzelfde uitziet als vastzitten. Toon "running (db: 47/183 tables)" zoals de bestandsfase
een percentage toont, en "waiting for background jobs (last ran 12 min ago)" als een run
een tijd geen stap zette, ook bij een cron die is blijven staan.
Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en
`cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop
(vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen).
3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later),
zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest.
4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de
tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop
stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een
bestaande parameter.
5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen
wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind
opsommen, in plaats van stoppen.
6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van
rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-<id>` staan; de
geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files.
Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op.
7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden
terug (er komt een `/` achter); nu werkt alleen een map.
- **DR-tests over databasegrenzen** (besluit 2026-10-20; productie nc01 draait PostgreSQL): na
de MariaDB→PostgreSQL-fout (tellers, rc5) elke overstap in de integratietest (`to-pgsql`,
`to-mysql`, `to-sqlite`, ook in CI). Volle DR-tests met echte data: vóór 1.0.0 de S3-simulatie
(MariaDB-bron, 138 GB) nogmaals naar een MariaDB-doel en de SMB-simulatie naar een SQLite-doel;
**na 1.0.0** de 1 TB van nc01 (PostgreSQL) naar een MariaDB-doel.
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
een paar bestanden.
- **1.1.0:**
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
@@ -412,6 +454,22 @@ in README "Known limitations":
3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel
een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf,
33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1.
4. **Schema per locatie** (besluit 2026-10-19): in de instellingen van een locatie "uit",
"dagelijks om HH:MM" of "elke N uur (vanaf HH:MM)". Nu is er één `schedule_time` voor alle
locaties, één keer per dag. Bij de update wordt de huidige tijd het schema van elke
bestaande locatie (voor bestaande gebruikers verandert er niets).
5. **Locaties na elkaar laten starten** in plaats van tegelijk: twee runs die dezelfde schijven
lezen zijn samen niet sneller (gezien op nc01); wachtende runs zichtbaar als "queued".
7. **SMB sneller** (gemeten 2026-10-21): lokaal netwerk zonder vertraging (Samba-container op de
laptop, 1,8 GB) back-up 24,9 MB/s, restore 27,6 MiB/s; vanaf Canada naar de Storage Box (130 ms)
2,6 MiB/s, omdat SMB per blok op het antwoord wacht. Grotere schrijfblokken, en vooruitlezen
bij een restore zoals SFTP (IReadsAhead) of meerdere packs tegelijk. Een NAS met 1 Gbit haalt
~110 MB/s: daar is winst.
Ook de index: `target:add` over SMB vanaf Canada haalt ~1 indexbestand per seconde (4.927
bestanden ~85 min), want SMB krijgt geen `getMany()` met verzoeken tegelijk zoals S3 en SFTP.
6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot
is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus),
zodat een nieuwe server na een DR hem ook ziet.
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
PostgreSQL 18)** — voor **1.0.0-rc.2**:
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
+82 -6
View File
@@ -7,8 +7,8 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
see "Storage backend experiences" below — that requirement comes from files_external, not from
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
**Status: release candidate for 1.0.0 (1.0.0-rc.1, app version 0.9.7).** Tested thoroughly, usable
alongside your current backup, but not yet a drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and
**Status: 1.0.0, stable.** Disaster recovery has been tested on real installations (see "Tested
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
`docs/DISASTER-RECOVERY.md`.
@@ -146,10 +146,31 @@ What you should do on the location (strongly recommended):
- **Files and folders** of a user: in the web interface or `occ backup:restore`, into a new folder
("Restored <date>"), merged into the original place, or replacing it.
- **Raw restore** of any path (for example `config/`) to a local directory: `occ backup:restore ... --to-directory=DIR`.
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see the how-to.
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see
[docs/DISASTER-RECOVERY.md](docs/DISASTER-RECOVERY.md). It runs in maintenance mode, so that
nobody and no background job writes into the instance while it is being replaced.
- **One user** (account, settings, files, calendars, contacts via user_migration): `occ backup:user:backup` and
`occ backup:user:restore`, replacing the account or restoring it as `<uid>-bak`. See the how-to.
**Without shell access** (a hosted Nextcloud where you cannot run `occ`): files and folders can be
restored in the web interface. Restoring a whole user or the whole server needs `occ`, so on such
hosting it is something your provider (or whoever has shell access) runs. A restore mode that
works without maintenance mode, and so from the web interface, is on the roadmap.
## Tested disaster recoveries
Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECOVERY.md):
| Backup | Location | Database | Result |
|---|---|---|---|
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
| 51 GB test instance | OVH S3 | PostgreSQL → PostgreSQL | 23,519/23,519 checksums identical |
The integration tests restore across every pair of PostgreSQL, MariaDB and SQLite.
## Storage backend experiences
Notes from actually running a backend, beyond what's in files_external's own documentation.
@@ -206,15 +227,67 @@ Contributions welcome via PR — add what you ran into with your own provider.
can be read on more connections: `occ config:app:set ng_backup restore_sftp_connections
--value=4` (at most 8; the server must allow that many plus one at once for the user).
- **OVH Object Storage (S3, Warsaw, 2026-10):** works with the `amazons3` backend (path style,
region `waw`). A full backup of 1 TB from Germany ran at about 7 MiB/s, from Canada about
6 MiB/s with BBR; a check of a 1 TB backup took 3.5 minutes (index files are fetched 16 at a
time); a restore of 138 GB from Warsaw to Canada took about 6 hours. Turn on versioning and
Object Lock on the bucket (see above); OVH charges no egress.
- **Hetzner Storage Box over SFTP, full restore:** about 13-14 MiB/s from Germany to Canada
(122 ms) with the read-ahead of 1.0.0-rc.4; without it about 2.8 MiB/s. A Storage Box gets
slower with several connections reading at once (4-6 MiB/s for 2-4 connections, it reads from
disks), so one extra connection is the default. A Storage Box drops an SFTP connection that
stayed quiet for a while without telling the client; NG Backup checks and replaces it.
- **SMB speed:** on a local network (Samba, no delay) a backup ran at about 25 MB/s and a
restore at about 28 MiB/s. Over a long line SMB waits for an answer to each request: from
Canada to a Storage Box in Germany (130 ms) about 2.6 MiB/s for a backup and 1.6 MiB/s for a
restore, and connecting a location with an existing backup reads its index at about one file a
second. SMB is meant for a local network; use SFTP or S3 for a location far away. When
`pecl install smbclient` fails (pecl.php.net was unreachable for hours on 2026-10-21), the
extension builds from its source: `libsmbclient-php` on GitHub, `phpize && ./configure && make
&& make install`, then `docker-php-ext-enable smbclient`.
- **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself
(e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the
mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping
mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS
can't be driven from PHP directly.
## Roadmap
What is planned, without dates. Ideas and votes are welcome in the
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
a 👍 on its issue helps decide what comes first.
**1.0.1**
- A warning on the admin page and in `occ backup:status` when background jobs do not run on a
system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on
while the NG Backup page is open ([#13](https://github.com/dillardblom/ng_backup/issues/13)).
- An optional label for a backup started with "Back up now".
- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4.
**1.1.0**
- Restore without the recovery kit: the backup key from the location itself, opened with one of
your passphrases.
- A schedule per location, also "every N hours".
- Locations backed up one after the other instead of at the same time.
- A note on a restore point that you can change afterwards.
- An option to pause heavy background jobs (such as Memories, Recognize or the Preview
Generator) while a backup runs.
- Faster backups: reading and uploading at the same time.
**Later**
- A full restore onto the existing installation, without first installing a fresh Nextcloud, in a
restore mode of its own instead of maintenance mode, so that it can also run from the web
interface (for hosted Nextcloud without shell access).
- Users restoring their own files from the Files app.
- Primary object storage and External Storage mounts (see Known limitations).
- An option not to keep the backup key on the location (the recovery kit is then the only copy).
## Known limitations
On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it:
On the backlog; know them before you rely on NG Backup:
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
@@ -236,8 +309,11 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
- Location passwords passed with `-o password=...` are visible in the process list and shell
history while the command runs.
- Temporary files of a restore into a user's files end up in that user's trash.
- Restoring the database does not order tables by foreign keys; apps with cascading foreign keys
could lose rows (Nextcloud core does not use them).
- **Restoring a whole user or the whole server needs `occ`** (shell access); see "Without shell
access" above.
- **`--install-apps` depends on the app store.** Nextcloud downloads the whole app list (about
38 MB) and gives up after 120 seconds; on a slow day apps are not installed. The restore then
lists them: install them afterwards with `occ app:install`, their data is restored already.
- A pack whose index upload failed stays on the location until removed by hand.
## Commands
+4 -4
View File
@@ -42,7 +42,7 @@ welcome: use the "Share your experience" link on the admin page.
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
you need help with restoring a backup after a crash.]]></description>
<version>0.9.11</version>
<version>1.0.0</version>
<licence>agpl</licence>
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
<namespace>NgBackup</namespace>
@@ -53,9 +53,9 @@ you need help with restoring a backup after a crash.]]></description>
<website>https://github.com/dillardblom/ng_backup</website>
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/3-settings.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0/screenshots/3-settings.png</screenshot>
<dependencies>
<php min-version="8.1"/>
<lib>sodium</lib>
+2
View File
@@ -10,6 +10,8 @@ Nextcloud back from an NG Backup location. Do them in this order.
- [ ] A new server with the **same Nextcloud major version** as the old one (for example 34),
a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for
the whole backup plus a margin.
- [ ] **Shell access** to the new server, to run `occ` (on a hosted Nextcloud without it, your
provider runs these steps).
- [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by
hand). Keep their packages somewhere safe beforehand.
+4 -7
View File
@@ -32,6 +32,7 @@ final class BackupJob extends TimedJob {
private RunMapper $runs,
private IAppConfig $appConfig,
private LoggerInterface $logger,
private \OCA\NgBackup\Service\Schedule $schedule,
) {
parent::__construct($time);
// Well below the usual 5-minute cron interval: the last run is stamped a moment after cron
@@ -101,14 +102,10 @@ final class BackupJob extends TimedJob {
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
}
/** Daily schedule: appconfig ng_backup/schedule_time "HH:MM" (server time); empty = no schedule. */
/** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */
private function startScheduled(): void {
$at = $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', '');
if (!preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $at)) {
return;
}
$todayAt = (int)strtotime(date('Y-m-d') . ' ' . $at);
if (time() < $todayAt) {
$todayAt = $this->schedule->todayAt();
if ($todayAt === null || time() < $todayAt) {
return;
}
foreach ($this->targets->list() as $t) {
+10 -6
View File
@@ -47,6 +47,7 @@ final class AdminApiController extends Controller {
private \OCA\NgBackup\Service\AlertService $alerts,
private \OCA\NgBackup\Service\KeySlotService $slots,
private \OCA\NgBackup\Service\PruneService $prune,
private \OCA\NgBackup\Service\Schedule $schedule,
) {
parent::__construct(Application::APP_ID, $request);
}
@@ -72,7 +73,8 @@ final class AdminApiController extends Controller {
'runs' => $runs,
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
'schedule' => $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''),
'schedule' => $this->schedule->get()['time'],
'scheduleTimezone' => $this->schedule->get()['timezone'],
]);
}
@@ -240,12 +242,14 @@ final class AdminApiController extends Controller {
return new JSONResponse($this->runJson($run));
}
public function setSchedule(string $time): JSONResponse {
if ($time !== '' && !preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time)) {
return new JSONResponse(['error' => 'Use HH:MM or empty'], Http::STATUS_BAD_REQUEST);
/** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */
public function setSchedule(string $time, string $timezone = ''): JSONResponse {
try {
$this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
} catch (\InvalidArgumentException $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
}
$this->appConfig->setValueString(Application::APP_ID, 'schedule_time', $time);
return new JSONResponse(['schedule' => $time]);
return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]);
}
public function snapshots(int $targetId): JSONResponse {
+26 -3
View File
@@ -159,7 +159,7 @@ final class DbDumper {
// everywhere); tables it does not know are left out, as asked.
foreach ($missing as $table) {
if (isset($manifest['tables'][$table]['schema'])) {
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema']));
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
$report['created'][] = $table;
} else {
$report['missing'][] = $table;
@@ -319,8 +319,31 @@ final class DbDumper {
'foreignKeys' => $foreignKeys];
}
private static function tableFromStructure(string $name, array $structure): \Doctrine\DBAL\Schema\Table {
$table = new \Doctrine\DBAL\Schema\Table($name);
/**
* Table options for a table the restore creates: those of this installation's own tables.
* On MySQL/MariaDB that is the character set and collation, as Nextcloud's tables have them
* (utf8mb4, so emoji and other 4-byte characters fit); DBAL alone would use 3-byte utf8.
*
* @return array<string, mixed>
* @psalm-suppress UndefinedClass the DBAL classes come with Nextcloud, not with this app
*/
private function tableOptions(): array {
$options = $this->db->createSchemaManager()->createSchemaConfig()->getDefaultTableOptions();
if ($this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\AbstractMySQLPlatform) {
$collation = $this->db->fetchOne('SELECT TABLE_COLLATION FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ?',
[$this->prefix . 'appconfig']);
if (is_string($collation) && $collation !== '') {
$options['collation'] = $collation;
$options['charset'] = explode('_', $collation, 2)[0];
unset($options['collate']);
}
}
return $options;
}
/** @param array<string, mixed> $options table options (see tableOptions()) */
private static function tableFromStructure(string $name, array $structure, array $options = []): \Doctrine\DBAL\Schema\Table {
$table = new \Doctrine\DBAL\Schema\Table($name, [], [], [], [], $options);
foreach ($structure['columns'] as $c) {
$table->addColumn($c['name'], $c['type'], array_filter([
'length' => $c['length'], 'precision' => $c['precision'], 'scale' => $c['scale'],
+48 -2
View File
@@ -845,7 +845,15 @@ final class Repository {
try {
foreach ($this->backend->getRanges($ranges) as $i => $data) {
foreach ($parts[$i] as [$id, $at, $length]) {
yield $this->decodeBlob($id, substr($data, $at, $length));
try {
$blob = $this->decodeBlob($id, substr($data, $at, $length));
} catch (\Exception) {
// a read that went wrong on the way (see loadBlob()): this blob on its own
$blob = $this->loadBlob($id);
// its pack stream is closed before the next range is read on the same connection
$this->packReader->close();
}
yield $blob;
$done++;
}
}
@@ -895,8 +903,46 @@ final class Repository {
} catch (BackendException) {
throw $e;
}
$pack = $moved;
}
return $this->decodeBlob($id, $encrypted);
try {
return $this->decodeBlob($id, $encrypted);
} catch (\Exception $damaged) {
// Bytes that do not decrypt: damaged on the location, or a read that went wrong on the
// way (a stream that lost its place: an SMB restore stopped after 17 hours like this).
// Read exactly this blob once more, on a request of its own; only a second failure
// means the data itself is damaged.
$this->packReader->close();
try {
$again = $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
} catch (\Exception) {
throw $damaged;
}
$data = $this->decodeBlob($id, $again);
if (function_exists('OCP\Log\logger')) {
\OCP\Log\logger('ng_backup')->warning('NG Backup: blob ' . $id . ' did not decrypt as first read from pack ' . $pack . ' at ' . $offset . '; read again, it is fine. First read: ' . self::howItDiffered($encrypted, $again));
}
return $data;
}
}
/**
* What a read that went wrong looked like next to the right bytes, for the log: where it starts
* to differ, and whether the rest is zeros or the right bytes from another place (a stream that
* lost its place).
*/
private static function howItDiffered(string $wrong, string $right): string {
$at = strspn($wrong ^ $right, "\0");
$text = strlen($wrong) . ' bytes, differ from byte ' . $at . ' of ' . strlen($right);
$tail = substr($wrong, $at, 64);
if ($tail === '') {
return $text . ' (too short)';
}
if (trim($tail, "\0") === '') {
return $text . ' (zeros)';
}
$found = strpos($right, $tail);
return $text . ($found !== false ? ' (the right bytes from ' . ($found - $at) . ' bytes further on)' : ' (other bytes)');
}
/** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */
+110
View File
@@ -0,0 +1,110 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Service;
use OCA\NgBackup\AppInfo\Application;
use OCP\IAppConfig;
use OCP\IConfig;
use OCP\IGroupManager;
/**
* The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a
* zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone
* is converted once to the zone of the server or of an administrator, at today's offset: it then
* shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
*/
final class Schedule {
public const TIME_KEY = 'schedule_time';
public const ZONE_KEY = 'schedule_timezone';
public function __construct(
private IAppConfig $appConfig,
private IConfig $config,
private IGroupManager $groups,
) {
}
/** @return array{time: string, timezone: non-empty-string} time "" = no schedule */
public function get(): array {
$time = $this->appConfig->getValueString(Application::APP_ID, self::TIME_KEY, '');
$zone = $this->appConfig->getValueString(Application::APP_ID, self::ZONE_KEY, '');
if ($zone === '' || !self::validZone($zone)) {
$zone = $this->guessZone();
if ($time !== '' && self::validTime($time)) {
$time = self::convert($time, 'UTC', $zone);
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
}
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
}
return ['time' => $time, 'timezone' => $zone];
}
/** @throws \InvalidArgumentException for a malformed time or an unknown zone */
public function set(string $time, string $zone): void {
if ($time !== '' && !self::validTime($time)) {
throw new \InvalidArgumentException('Use HH:MM or empty');
}
if (!self::validZone($zone)) {
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
}
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
}
/** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */
public function todayAt(?int $now = null): ?int {
['time' => $time, 'timezone' => $zone] = $this->get();
if ($time === '' || !self::validTime($time)) {
return null;
}
$tz = new \DateTimeZone($zone);
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d');
return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp();
}
/**
* "HH:MM" in zone $from as "HH:MM" in zone $to, at today's offsets.
*
* @param non-empty-string $from
* @param non-empty-string $to
*/
public static function convert(string $time, string $from, string $to, ?int $now = null): string {
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone(new \DateTimeZone($from))->format('Y-m-d');
return (new \DateTimeImmutable($day . ' ' . $time, new \DateTimeZone($from)))->setTimezone(new \DateTimeZone($to))->format('H:i');
}
public static function validTime(string $time): bool {
return preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time) === 1;
}
/** @psalm-assert-if-true non-empty-string $zone */
public static function validZone(string $zone): bool {
return in_array($zone, \DateTimeZone::listIdentifiers(), true);
}
/**
* The server's default_timezone when set, else the first administrator's, else UTC. A server
* zone of UTC is passed over on purpose: it is mostly what an image or installer left there,
* while the administrator who set the time read it as their own (#19).
*
* @return non-empty-string
*/
private function guessZone(): string {
$zone = $this->config->getSystemValueString('default_timezone', '');
if ($zone !== '' && $zone !== 'UTC' && self::validZone($zone)) {
return $zone;
}
foreach ($this->groups->get('admin')?->getUsers() ?? [] as $user) {
$zone = $this->config->getUserValue($user->getUID(), 'core', 'timezone', '');
if ($zone !== '' && self::validZone($zone)) {
return $zone;
}
}
return 'UTC';
}
}
+3 -1
View File
@@ -89,8 +89,10 @@ onBeforeUnmount(() => clearTimeout(timer))
</script>
<style scoped>
/* NcSettingsSection sets its own width (at most 900px): the runs table needs more */
.ng-backup :deep(.settings-section) {
max-width: 900px;
width: min(1400px, 100% - var(--default-grid-baseline) * 7 * 2);
max-width: none;
}
.ngb-feedback {
color: var(--color-text-maxcontrast);
+1 -1
View File
@@ -33,7 +33,7 @@ export default {
testTarget: (id) => call('post', `/targets/${id}/test`),
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
setSchedule: (time) => call('put', '/schedule', { time }),
setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }),
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
+10 -6
View File
@@ -6,8 +6,8 @@
<NcSettingsSection :name="t('ng_backup', 'Backups')"
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
<div class="ngb-row">
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, empty = off)')" />
<NcButton :disabled="schedule === status.schedule" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, {zone}, empty = off)', { zone: browserZone })" />
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
</div>
<div class="ngb-row">
<NcButton v-for="target in status.targets" :key="target.id" :disabled="isRunning(target.id)" @click="start(target)">
@@ -79,15 +79,19 @@ import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
import NcTextField from '@nextcloud/vue/components/NcTextField'
import api from '../api.js'
import { formatDateTime, formatDate, formatSize } from '../format.js'
import { formatDateTime, formatDate, formatSize, toZone } from '../format.js'
import { confirmPassword } from '@nextcloud/password-confirmation'
import { reactive } from 'vue'
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
const emit = defineEmits(['changed', 'show-all-runs'])
const schedule = ref(props.status.schedule)
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
// zone is shown in this one, so it reads as the moment it runs here.
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s
const schedule = ref(shown(props.status.schedule))
const error = ref('')
watch(() => props.status.schedule, (s) => { schedule.value = s })
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
const trash = reactive({})
async function loadTrash() {
@@ -131,7 +135,7 @@ async function start(target) {
async function saveSchedule() {
error.value = ''
try {
await api.setSchedule(schedule.value.trim())
await api.setSchedule(schedule.value.trim(), browserZone)
emit('changed')
} catch (e) {
error.value = e.message
+27
View File
@@ -26,3 +26,30 @@ export function formatSize(bytes) {
}
return i === 0 ? `${b} B` : `${b.toFixed(1)} ${units[i]}`
}
/**
* A time of day "HH:MM" in zone `from`, as "HH:MM" in zone `to`, at today's offsets.
*
* @param {string} hhmm the time
* @param {string} from IANA zone it is in
* @param {string} to IANA zone to show it in
* @param {Date} now the day to convert on (today)
* @return {string}
*/
export function toZone(hhmm, from, to, now = new Date()) {
const [h, m] = hhmm.split(':').map(Number)
const parts = (zone, date) => Object.fromEntries(new Intl.DateTimeFormat('en-GB', {
timeZone: zone, hourCycle: 'h23', year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit',
}).formatToParts(date).map((p) => [p.type, p.value]))
const offsetAt = (at) => {
const seen = parts(from, new Date(at))
return Date.UTC(+seen.year, +seen.month - 1, +seen.day, +seen.hour, +seen.minute) - at
}
const today = parts(from, now)
const wall = Date.UTC(+today.year, +today.month - 1, +today.day, h, m)
// the offset at the wall time itself: on a day the clocks change it differs from the one at
// midnight, so it is looked up a second time at the moment found first
const at = wall - offsetAt(wall - offsetAt(wall))
const there = parts(to, new Date(at))
return `${there.hour}:${there.minute}`
}
+133
View File
@@ -0,0 +1,133 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\IBackend;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/**
* A pack stream that hands out wrong bytes once (an SMB restore stopped after 17 hours with
* "Decryption failed"): the blob is read once more on a request of its own; only data that is
* damaged on the location stops the restore.
*/
class DamagedReadTest extends TestCase {
use TempDirTrait;
private string $root;
private KeyRing $keys;
private string $src;
private string $snapshot;
protected function setUp(): void {
$this->root = $this->tempDir();
$this->keys = KeyRing::generate();
$this->src = $this->tempDir();
for ($i = 0; $i < 6; $i++) {
file_put_contents("$this->src/f$i", random_bytes(3 * 1048576));
}
$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
}
public function testABlobThatCameInDamagedOnceIsReadAgain(): void {
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1);
$target = $this->tempDir();
$this->assertSame(6, Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $target));
$this->assertSame(self::hashTree($this->src), self::hashTree($target));
$this->assertSame(0, $backend->streams, 'the damaged read happened');
}
public function testATableDumpThatCameInDamagedOnceIsReadAgain(): void {
$repo = Repository::openWithKey(new LocalBackend($this->root), $this->keys);
$stats = [];
$writer = $repo->blobWriter($stats);
$lines = [];
for ($i = 0; $i < 20000; $i++) {
$lines[] = json_encode(['i' => $i, 'pad' => str_repeat('x', $i % 400)]);
}
$writer->write(implode("\n", $lines) . "\n");
$blobs = $writer->finish();
$repo->flushPacks();
$backend = new GarblingBackend(new LocalBackend($this->root), ranges: 1);
$this->assertSame($lines, iterator_to_array(Repository::openWithKey($backend, $this->keys)->readLines($blobs), false));
$this->assertSame(0, $backend->ranges, 'the damaged read happened');
}
public function testDataDamagedOnTheLocationStillStopsTheRestore(): void {
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1000, ranges: 1000);
$this->expectException(\Exception::class);
Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $this->tempDir());
}
}
/** Pack reads that flip a byte: the first $streams pack streams and $ranges range reads. */
final class GarblingBackend implements IBackend {
use \OCA\NgBackup\Backend\GetsOneByOne;
public function __construct(
private IBackend $inner,
public int $streams = 0,
public int $ranges = 0,
) {
}
public function put(string $path, $stream): void {
$this->inner->put($path, $stream);
}
public function get(string $path) {
$fh = $this->inner->get($path);
if (!str_starts_with($path, 'packs/') || $this->streams <= 0) {
return $fh;
}
$this->streams--;
$data = stream_get_contents($fh);
fclose($fh);
for ($k = 100; $k < strlen($data); $k += 65536) {
$data[$k] = chr(ord($data[$k]) ^ 0x55);
}
$out = fopen('php://temp', 'w+b');
fwrite($out, $data);
rewind($out);
return $out;
}
public function getRange(string $path, int $offset, int $length): string {
$data = $this->inner->getRange($path, $offset, $length);
if (str_starts_with($path, 'packs/') && $this->ranges > 0) {
$this->ranges--;
$data[0] = chr(ord($data[0]) ^ 0x55);
}
return $data;
}
public function exists(string $path): bool {
return $this->inner->exists($path);
}
public function move(string $from, string $to): void {
$this->inner->move($from, $to);
}
public function list(string $prefix): array {
return $this->inner->list($prefix);
}
public function listFiles(string $dir): array {
return $this->inner->listFiles($dir);
}
public function delete(string $path): void {
$this->inner->delete($path);
}
}
+71
View File
@@ -0,0 +1,71 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Service;
use OCA\NgBackup\Service\Schedule;
use OCP\IAppConfig;
use OCP\IConfig;
use OCP\IGroup;
use OCP\IGroupManager;
use OCP\IUser;
use PHPUnit\Framework\TestCase;
/** The daily time in a time zone (issue #19), and the conversion of a time saved without one. */
class ScheduleTest extends TestCase {
private array $values = [];
private function schedule(string $serverZone = '', string $adminZone = ''): Schedule {
$app = $this->createMock(IAppConfig::class);
$app->method('getValueString')->willReturnCallback(fn (string $a, string $k, string $d = '') => $this->values[$k] ?? $d);
$app->method('setValueString')->willReturnCallback(function (string $a, string $k, string $v): bool {
$this->values[$k] = $v;
return true;
});
$config = $this->createMock(IConfig::class);
$config->method('getSystemValueString')->willReturn($serverZone);
$config->method('getUserValue')->willReturn($adminZone);
$user = $this->createMock(IUser::class);
$user->method('getUID')->willReturn('admin');
$group = $this->createMock(IGroup::class);
$group->method('getUsers')->willReturn([$user]);
$groups = $this->createMock(IGroupManager::class);
$groups->method('get')->willReturn($group);
return new Schedule($app, $config, $groups);
}
public function testATimeSavedWithoutAZoneShowsAsItRanInTheAdministratorsZone(): void {
$this->values = [Schedule::TIME_KEY => '02:30'];
$s = $this->schedule(adminZone: 'Europe/Berlin')->get();
// it ran at 02:30 UTC: in summer that is 04:30 in Berlin, in winter 03:30
$summer = Schedule::convert('02:30', 'UTC', 'Europe/Berlin', strtotime('2026-07-01 12:00 UTC'));
$this->assertSame('04:30', $summer);
$this->assertSame('Europe/Berlin', $s['timezone']);
$this->assertSame(Schedule::convert('02:30', 'UTC', 'Europe/Berlin'), $s['time']);
}
public function testTheServersDefaultZoneComesFirst(): void {
$this->values = [Schedule::TIME_KEY => '01:00'];
$this->assertSame('America/Toronto', $this->schedule('America/Toronto', 'Europe/Berlin')->get()['timezone']);
}
public function testWithoutAnyZoneItStaysUtc(): void {
$this->values = [Schedule::TIME_KEY => '02:30'];
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
}
public function testTodayAtIsTheMomentInTheSchedulesZone(): void {
$this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin'];
$at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC'));
$this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC');
}
public function testAnUnknownZoneOrABadTimeIsRefused(): void {
$s = $this->schedule();
$this->expectException(\InvalidArgumentException::class);
$s->set('03:00', 'Mars/Olympus');
}
}
+11 -2
View File
@@ -47,6 +47,12 @@ if [ "$LEGACY" = legacy ]; then
fi
run_on() { docker exec -u www-data "$1" php occ "${@:2}"; }
# The app store sends the whole app list (~38 MB) and Nextcloud gives up after 120 s: on a slow day
# an install fails for reasons outside this test. Up to three tries.
install_app() { # container app
for i in 1 2 3; do run_on "$1" app:install "$2" >/dev/null 2>&1 && return 0; sleep 20; done
return 1
}
# Database settings for an instance: SQLite inside the container, or its own PostgreSQL container
db_env() {
@@ -109,7 +115,7 @@ if [ "$DB" != sqlite ]; then
# while the restore loads tables in name order, accounts before mailboxes.
docker run -d --name ngb-frt-imap --network "$NET" -e MAILNAME=mail.domain.tld -e MAIL_ADDRESS=user@domain.tld \
-e MAIL_PASS=mypassword ghcr.io/christophwurst/docker-imap-devel:latest >/dev/null
run_on ngb-frt-origin app:install mail >/dev/null || fail "install mail on the origin"
install_app ngb-frt-origin mail || fail "install mail on the origin (app store unreachable?)"
run_on ngb-frt-origin config:system:set app.mail.verify-tls-peer --type=boolean --value=false >/dev/null
sleep 10
run_on ngb-frt-origin mail:account:create frtest Test user@domain.tld ngb-frt-imap 993 ssl user@domain.tld mypassword \
@@ -118,7 +124,9 @@ if [ "$DB" != sqlite ]; then
[ "$(sql origin 'select count(*) from oc_mail_accounts where drafts_mailbox_id is not null')" = 1 ] \
|| fail "the mail account does not point at its mailboxes"
# Deck installed and then switched off: its tables stay in the database and in the backup
run_on ngb-frt-origin app:install deck >/dev/null || fail "install deck on the origin"
install_app ngb-frt-origin deck || fail "install deck on the origin (app store unreachable?)"
# a 4-byte character (emoji) in a table the restore has to create: utf8mb4 on MySQL/MariaDB
sql origin "insert into oc_deck_boards (title, owner, color, archived, deleted_at, last_modified) values ('Plan 📌', 'frtest', '0087C5', false, 0, 0)" >/dev/null
run_on ngb-frt-origin app:disable deck >/dev/null
ok "origin has a synced Mail account and the tables of a disabled app"
fi
@@ -202,6 +210,7 @@ if [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ] && [ "$LEGACY" = legacy
ok "older backup: tables of the disabled app left out, Mail restored"
elif [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ]; then
[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created"
[ "$(sql restore "select title from oc_deck_boards")" = "Plan 📌" ] || fail "a 4-byte character did not survive in a created table"
[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
[ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \
|| fail "the foreign keys of Mail are missing after the restore"