Compare commits

..

33 Commits

Author SHA1 Message Date
dillard 4847ce293d SFTP: restore the line breaks of an OpenSSH private key before use (#24)
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Nextcloud 35 reads keys with phpseclib 3, which refuses an OpenSSH-format
key whose line breaks were lost ("Unable to read key"); phpseclib 2 on
Nextcloud 34 read it. The key is fixed when used, so stored locations work
again without re-entering it.
2026-10-28 01:41:58 +01:00
dillard 90dd97977a CI: build the admin page on every push 2026-10-27 11:34:53 +01:00
dillard 08534975d7 Admin page: Docs tab with links to the how-to, disaster recovery guide and roadmap; drop "in beta" from the feedback line
PLAN: 1.0.1 item 11 (Docs tab, done); 1.1.0 item 9: remove the feedback request in 1.1.0 or 1.2.0.
2026-10-27 11:23:34 +01:00
dillard 95e3dfcf6d User restore by user id, backup:user:list (#22)
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
backup:user:restore <location> <uid> only said "is not a recorded user export": the argument
is the manifest path, which only backup:user:backup printed once. backup:user:list shows the
exports again; a bare user id works when that user has exactly one export, otherwise the error
lists them. The export is resolved before the replace confirmation.

Fixes #22
2026-10-27 00:10:59 +01:00
dillard 7f4cae0e42 Roadmap: pin a restore point (1.1.0); PLAN 1.0.1: explain retention in the trash, time zone in backup:status
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-25 02:40:13 +02:00
Dillard Blom 965440d1f7 Merge pull request #20 from dillardblom/rc6
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
Release 1.0.0-rc.6
2026-10-24 01:04:23 +02:00
dillard 3a0f882a38 1.0.0-rc.6: changelog, SMB disaster recovery result in the README, 1.0.1 items (salvage mode, leftover restore files, single-file restore)
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-24 00:40:51 +02:00
dillard 632d2a2838 Admin page: widen the sections through their width; NcSettingsSection sets width, not max-width
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-23 18:10:48 +02:00
dillard 9e9234e2a8 Admin page: sections up to 1400px wide, so the runs table fits without scrolling
CI / unit (8.2) (push) Has been cancelled
CI / unit (8.3) (push) Has been cancelled
CI / unit (8.4) (push) Has been cancelled
CI / psalm (push) Has been cancelled
CI / integration (mariadb, 32) (push) Has been cancelled
CI / integration (mariadb, 33) (push) Has been cancelled
CI / integration (mariadb, 34) (push) Has been cancelled
CI / integration (mariadb, 35) (push) Has been cancelled
CI / integration (pgsql, 32) (push) Has been cancelled
CI / integration (pgsql, 33) (push) Has been cancelled
CI / integration (pgsql, 34) (push) Has been cancelled
CI / integration (pgsql, 35) (push) Has been cancelled
CI / integration (sqlite, 32) (push) Has been cancelled
CI / integration (sqlite, 33) (push) Has been cancelled
CI / integration (sqlite, 34) (push) Has been cancelled
CI / integration (sqlite, 35) (push) Has been cancelled
CI / integration-targets (push) Has been cancelled
CI / integration-user (32) (push) Has been cancelled
CI / integration-user (33) (push) Has been cancelled
CI / integration-user (34) (push) Has been cancelled
CI / integration-user (35) (push) Has been cancelled
CI / integration-full-restore (push) Has been cancelled
2026-10-23 13:29:47 +02:00
dillard 3ac3298313 Database restore: close the pack stream of a blob read again before the next range 2026-10-22 20:30:18 +02:00
dillard 384b2d2792 Full-restore test: the emoji deck board sets archived as a boolean (PostgreSQL refuses 0) 2026-10-22 20:09:58 +02:00
dillard 8032411789 Restore: the warning for a blob read again says how the first read differed (zeros, shifted, other bytes) 2026-10-22 19:50:27 +02:00
dillard b4f20cb070 Version 0.9.12 (1.0.0-rc.6) 2026-10-22 19:48:57 +02:00
dillard 343389b98b Changelog: the second read covers the database phase too 2026-10-22 19:48:22 +02:00
dillard 133f5f9d34 Schedule: say why a server zone of UTC is passed over 2026-10-22 19:47:30 +02:00
dillard cf4bd9db7f Review fixes: the database restore also reads a damaged blob once more; schedule conversion right on days the clocks change; the field follows a changed schedule time zone 2026-10-22 19:45:49 +02:00
dillard 67cc350b8c Daily backup time in the administrator's time zone (#19); a time saved before is converted once so it shows as it ran 2026-10-22 16:38:54 +02:00
dillard fb5b234e8a Restore: a blob that does not decrypt as first read is read once more on a request of its own before the restore gives up 2026-10-22 14:43:48 +02:00
dillard fd12c586b6 Roadmap: restore mode for hosted Nextcloud, read-ahead for S3 and SMB; changelog for rc6 2026-10-22 13:07:09 +02:00
dillard 6ffe979ea7 Merge branch 'polish' into rc6 2026-10-22 13:06:59 +02:00
dillard a03b7354b3 README: tested disaster recoveries, backend measurements, restores without shell access; guide lists shell access; changelog for rc6 2026-10-22 13:06:49 +02:00
dillard be1c628f07 Full-restore test: install Mail and Deck with up to three tries (the app store is slow at times) 2026-10-22 03:38:30 +02:00
dillard a8355df12e Full restore: tables the restore creates get this installation's charset and collation (utf8mb4 on MySQL/MariaDB); test with an emoji 2026-10-22 03:36:46 +02:00
dillard cd251cffb3 PLAN: SMB index reads one at a time (1.1.0) 2026-10-21 20:02:43 +02:00
dillard 0b4872599d PLAN: the 1.0.1 cron PR closes #13 2026-10-21 19:51:57 +02:00
dillard b7d09c9784 PLAN: 1.1.0 faster SMB (local and long-line measurements) 2026-10-21 09:27:15 +02:00
dillard 24686897c3 PLAN: DR tests across databases (integration for every crossing; S3 sim onto MariaDB, SMB sim onto SQLite before 1.0.0; nc01 onto MariaDB after) 2026-10-20 23:12:19 +02:00
dillard 315e83b847 PLAN: 1.0.1 progress in the database phase and a waiting notice, so a backup does not look stuck 2026-10-20 20:49:44 +02:00
dillard b728db36c7 README: roadmap 2026-10-19 21:35:23 +02:00
dillard 6ed0a555cf PLAN: label field next to Back up now (1.0.1); per-location schedules, every N hours, locations one after the other, editable notes (1.1.0) 2026-10-19 21:26:20 +02:00
dillard 221ed9dc17 PLAN: the cron warning says cron is needed for all backups, links to how to set it up, and that without it a backup started with the button only moves while the page is open 2026-10-19 21:18:23 +02:00
dillard 285153a968 PLAN: 1.0.1 cron warning says to keep the page open under AJAX, with a link to the Nextcloud docs 2026-10-19 21:16:23 +02:00
dillard 6c8a78f6bb PLAN: 1.0.1 background jobs without system cron: warn, and let the open admin page move a backup 2026-10-19 20:41:11 +02:00
25 changed files with 972 additions and 48 deletions
+14
View File
@@ -41,6 +41,20 @@ jobs:
- run: composer install --no-interaction --no-progress
- run: composer psalm
frontend:
# Builds the admin page, so a broken Vue component fails CI instead of showing up after a release.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: npm
- run: npm ci --no-audit --no-fund
- run: npm run build
integration:
# Full round trip (key, kit, location, backup, incremental, restore modes, database dump and
# restore, retention, trash) on a fresh Nextcloud per version and database.
+50
View File
@@ -7,6 +7,56 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
beta/daily update channel, which would hide the app from most admins. The stability stage is
instead carried by the app description, the git tag, the release, and this file.
## [Unreleased]
### Added
- `occ backup:user:list [location] [user]` shows the user exports on a location, with the manifest
path `backup:user:restore` expects. Before, that path was only printed once by
`backup:user:backup` (#22).
### Changed
- `occ backup:user:restore` also takes a user id instead of the manifest path, when that user has
exactly one export on the location. With more, and for a manifest path that is not on the
location, the error lists that user's exports; for a user without exports it names the
expected form (#22). Before, `backup:user:restore <location> <uid>` only said "is not a
recorded user export for this location".
### Fixed
- An SFTP location with an RSA key in OpenSSH format ("BEGIN OPENSSH PRIVATE KEY") whose line
breaks were lost, e.g. turned into spaces on the way in, stopped working after an upgrade to
Nextcloud 35 with "Unable to read key": Nextcloud 35 reads keys with a stricter library. The
line breaks are now restored before the key is used, so the stored location works again without
re-entering it (#24).
## [1.0.0-rc.6] - 2026-10-24
Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery
simulations and a user report. If no problems are reported for about a week, 1.0.0 will be this
same code.
### Fixed
- The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in
Berlin (#19). The time is now saved with the administrator's time zone and shown with it. A time
saved before is converted once to the zone of the server or an administrator, so it shows as it
effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer) and nothing moves.
- A full restore over SMB stopped after 17 hours with "Decryption failed": a read over a long
line handed out wrong bytes (the backup itself was fine, the same files restored identically
afterwards). A blob that does not decrypt as first read, of a file or of the database, is now
read once more on a request of its own, and only a second failure stops the restore.
- A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table
(of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the
table got 3-byte utf8. Tables the restore creates now get this installation's character set and
collation (utf8mb4). Found in a disaster recovery simulation onto MariaDB.
### Changed
- The admin page uses more of the screen (sections up to 1400 pixels instead of 900), so the
list of backup runs fits without scrolling sideways.
- README: a roadmap, the tested disaster recoveries, measurements for OVH S3, the Storage Box over
SFTP and SMB, how to build the SMB extension when PECL is down, and what works without shell
access. The guide lists shell access among what you need.
- The full-restore integration test checks a 4-byte character in a table the restore creates, and
retries app installs from the app store.
## [1.0.0-rc.5] - 2026-10-22
Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
+90 -3
View File
@@ -395,11 +395,73 @@ in README "Known limitations":
- **1.0.1:**
1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron
staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue
#13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles.
2. Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit
oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21):
a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode`
(`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de
laatste run te lang geleden: melding met de cron-regel erbij ("Backups only move while
someone has Nextcloud open"), en "waiting for background jobs" als een lopende backup lang
geen stap zette.
b. **De pagina laat de backup lopen** zolang ze openstaat: in `ajax`/`webcron` voert de
statusaanroep die de pagina toch al herhaalt ook een korte, begrensde stap uit (zoals de
korte stappen die AJAX-cron nu al krijgt). "Back up now" werkt dan ook zonder cron zolang
het tabblad openstaat; de melding zegt dat geplande backups echte cron nodig hebben.
De melding zegt (zonder werkende systeem-cron): backups hebben systeem-cron nodig, ook de
gewone nachtelijke; zo stel je cron in (link naar de Nextcloud-documentatie over
achtergrondjobs voor de versie van de server, nu
https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html,
versienummer uit de server, niet vast); zolang er geen cron is, loopt een backup die je
met de knop start alleen door terwijl de NG Backup-pagina openstaat.
c. **Voortgang in de databasefase** (issue #13): nu staat er alleen "running (db)", wat er
hetzelfde uitziet als vastzitten. Toon "running (db: 47/183 tables)" zoals de bestandsfase
een percentage toont, en "waiting for background jobs (last ran 12 min ago)" als een run
een tijd geen stap zette, ook bij een cron die is blijven staan.
Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en
`cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop
(vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen).
3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later),
zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest.
4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de
tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop
stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een
bestaande parameter.
5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen
wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind
opsommen, in plaats van stoppen.
6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van
rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-<id>` staan; de
geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files.
Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op.
7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden
terug (er komt een `/` achter); nu werkt alleen een map.
8. **Uitleg (i) bij de prullenbak van herstelpunten** (besluit 2026-10-25): bij "removed by
retention policy" een infoknop met wat het betekent: het bewaarbeleid houdt voor oudere dagen
één herstelpunt per dag (het laatste), weken en maanden; de rest gaat naar de prullenbak en
wordt pas na de ingestelde wachttijd (`delete_delay_days`, standaard 30) echt verwijderd, en is
tot die datum terug te zetten. Aanleiding: de beheerder zag "19-10 13:12 – after rc4 upgrade
removed by retention policy" en dacht dat er iets was weggegooid.
9. **Tijdzone in `occ backup:status`**: de tabel toont de servertijd (UTC), de webinterface de
tijd van de beheerder; dat leest als een verkeerde starttijd. Zet de zone in de kolomkop
("started (UTC)") of toon de tijden in de zone van het schema.
10. **User-restore op gebruikersnaam** (issue #22, 2026-10-27): `backup:user:restore <locatie>
New_User` gaf alleen "is not a recorded user export"; het argument moet het manifestpad
zijn, dat alleen `backup:user:backup` eenmalig toonde. Nu: `backup:user:list` toont de
exports, een kale uid werkt bij precies één export, anders een lijst om uit te kiezen —
**klaar** (2026-10-27).
11. **Tabblad "Docs" in de beheerpagina** (besluit 2026-10-27): een derde knop naast "Backup and
restore" en "Settings", met links naar de HOWTO (`docs/HOWTO.md`), de DR-gids
(`docs/DISASTER-RECOVERY.md`) en de roadmap (README, `#roadmap`) op GitHub, elk met een
regel wat erin staat. Nu zijn ze alleen via de README te vinden. Zichtbaar ook zonder
sleutel, zodat de gids vindbaar is vóór de eerste installatie-stap. "NG Backup is in beta"
gaat uit de feedbackregel — **klaar** (2026-10-27).
- **DR-tests over databasegrenzen** (besluit 2026-10-20; productie nc01 draait PostgreSQL): na
de MariaDB→PostgreSQL-fout (tellers, rc5) elke overstap in de integratietest (`to-pgsql`,
`to-mysql`, `to-sqlite`, ook in CI). Volle DR-tests met echte data: vóór 1.0.0 de S3-simulatie
(MariaDB-bron, 138 GB) nogmaals naar een MariaDB-doel en de SMB-simulatie naar een SQLite-doel;
**na 1.0.0** de 1 TB van nc01 (PostgreSQL) naar een MariaDB-doel.
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
een paar bestanden.
- **1.1.0:**
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
@@ -412,6 +474,31 @@ in README "Known limitations":
3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel
een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf,
33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1.
4. **Schema per locatie** (besluit 2026-10-19): in de instellingen van een locatie "uit",
"dagelijks om HH:MM" of "elke N uur (vanaf HH:MM)". Nu is er één `schedule_time` voor alle
locaties, één keer per dag. Bij de update wordt de huidige tijd het schema van elke
bestaande locatie (voor bestaande gebruikers verandert er niets).
5. **Locaties na elkaar laten starten** in plaats van tegelijk: twee runs die dezelfde schijven
lezen zijn samen niet sneller (gezien op nc01); wachtende runs zichtbaar als "queued".
7. **SMB sneller** (gemeten 2026-10-21): lokaal netwerk zonder vertraging (Samba-container op de
laptop, 1,8 GB) back-up 24,9 MB/s, restore 27,6 MiB/s; vanaf Canada naar de Storage Box (130 ms)
2,6 MiB/s, omdat SMB per blok op het antwoord wacht. Grotere schrijfblokken, en vooruitlezen
bij een restore zoals SFTP (IReadsAhead) of meerdere packs tegelijk. Een NAS met 1 Gbit haalt
~110 MB/s: daar is winst.
Ook de index: `target:add` over SMB vanaf Canada haalt ~1 indexbestand per seconde (4.927
bestanden ~85 min), want SMB krijgt geen `getMany()` met verzoeken tegelijk zoals S3 en SFTP.
6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot
is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus),
zodat een nieuwe server na een DR hem ook ziet.
8. **Herstelpunt vastpinnen** (besluit 2026-10-25): een herstelpunt dat het bewaarbeleid
nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met
`occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de
opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly.
9. **Feedbackvraag weghalen** in 1.1.0 of 1.2.0 (besluit 2026-10-27): de regel "We would like
to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot
dan staan (de vermelding "in beta" is er in 1.0.1 al af). Zijn er tegen die
tijd geen noemenswaardige meldingen, dan werkt het: mensen melden eerder wat niet werkt dan
wat gewoon werkt. Het formulier (`experience.yml`) mag blijven voor wie het zelf opzoekt.
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
PostgreSQL 18)** — voor **1.0.0-rc.2**:
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
+86 -4
View File
@@ -146,10 +146,31 @@ What you should do on the location (strongly recommended):
- **Files and folders** of a user: in the web interface or `occ backup:restore`, into a new folder
("Restored <date>"), merged into the original place, or replacing it.
- **Raw restore** of any path (for example `config/`) to a local directory: `occ backup:restore ... --to-directory=DIR`.
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see the how-to.
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see
[docs/DISASTER-RECOVERY.md](docs/DISASTER-RECOVERY.md). It runs in maintenance mode, so that
nobody and no background job writes into the instance while it is being replaced.
- **One user** (account, settings, files, calendars, contacts via user_migration): `occ backup:user:backup` and
`occ backup:user:restore`, replacing the account or restoring it as `<uid>-bak`. See the how-to.
**Without shell access** (a hosted Nextcloud where you cannot run `occ`): files and folders can be
restored in the web interface. Restoring a whole user or the whole server needs `occ`, so on such
hosting it is something your provider (or whoever has shell access) runs. A restore mode that
works without maintenance mode, and so from the web interface, is on the roadmap.
## Tested disaster recoveries
Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECOVERY.md):
| Backup | Location | Database | Result |
|---|---|---|---|
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
| 51 GB test instance | OVH S3 | PostgreSQL → PostgreSQL | 23,519/23,519 checksums identical |
The integration tests restore across every pair of PostgreSQL, MariaDB and SQLite.
## Storage backend experiences
Notes from actually running a backend, beyond what's in files_external's own documentation.
@@ -206,12 +227,70 @@ Contributions welcome via PR — add what you ran into with your own provider.
can be read on more connections: `occ config:app:set ng_backup restore_sftp_connections
--value=4` (at most 8; the server must allow that many plus one at once for the user).
- **OVH Object Storage (S3, Warsaw, 2026-10):** works with the `amazons3` backend (path style,
region `waw`). A full backup of 1 TB from Germany ran at about 7 MiB/s, from Canada about
6 MiB/s with BBR; a check of a 1 TB backup took 3.5 minutes (index files are fetched 16 at a
time); a restore of 138 GB from Warsaw to Canada took about 6 hours. Turn on versioning and
Object Lock on the bucket (see above); OVH charges no egress.
- **Hetzner Storage Box over SFTP, full restore:** about 13-14 MiB/s from Germany to Canada
(122 ms) with the read-ahead of 1.0.0-rc.4; without it about 2.8 MiB/s. A Storage Box gets
slower with several connections reading at once (4-6 MiB/s for 2-4 connections, it reads from
disks), so one extra connection is the default. A Storage Box drops an SFTP connection that
stayed quiet for a while without telling the client; NG Backup checks and replaces it.
- **SMB speed:** on a local network (Samba, no delay) a backup ran at about 25 MB/s and a
restore at about 28 MiB/s. Over a long line SMB waits for an answer to each request: from
Canada to a Storage Box in Germany (130 ms) about 2.6 MiB/s for a backup and 1.6 MiB/s for a
restore, and connecting a location with an existing backup reads its index at about one file a
second. SMB is meant for a local network; use SFTP or S3 for a location far away. When
`pecl install smbclient` fails (pecl.php.net was unreachable for hours on 2026-10-21), the
extension builds from its source: `libsmbclient-php` on GitHub, `phpize && ./configure && make
&& make install`, then `docker-php-ext-enable smbclient`.
- **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself
(e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the
mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping
mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS
can't be driven from PHP directly.
## Roadmap
What is planned, without dates. Ideas and votes are welcome in the
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
a 👍 on its issue helps decide what comes first.
**1.0.1**
- A warning on the admin page and in `occ backup:status` when background jobs do not run on a
system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on
while the NG Backup page is open ([#13](https://github.com/dillardblom/ng_backup/issues/13)).
- An optional label for a backup started with "Back up now".
- An explanation in the trash of restore points of what "removed by retention policy" means and
until when a restore point can be brought back.
- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4.
- A "Docs" tab on the admin page with links to the how-to, the disaster recovery guide and this
roadmap.
**1.1.0**
- Restore without the recovery kit: the backup key from the location itself, opened with one of
your passphrases.
- A schedule per location, also "every N hours".
- Locations backed up one after the other instead of at the same time.
- A note on a restore point that you can change afterwards.
- Pinning a restore point, so the retention policy never removes it (for example the one from
before an upgrade).
- An option to pause heavy background jobs (such as Memories, Recognize or the Preview
Generator) while a backup runs.
- Faster backups: reading and uploading at the same time.
**Later**
- A full restore onto the existing installation, without first installing a fresh Nextcloud, in a
restore mode of its own instead of maintenance mode, so that it can also run from the web
interface (for hosted Nextcloud without shell access).
- Users restoring their own files from the Files app.
- Primary object storage and External Storage mounts (see Known limitations).
- An option not to keep the backup key on the location (the recovery kit is then the only copy).
## Known limitations
On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it:
@@ -236,8 +315,11 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
- Location passwords passed with `-o password=...` are visible in the process list and shell
history while the command runs.
- Temporary files of a restore into a user's files end up in that user's trash.
- Restoring the database does not order tables by foreign keys; apps with cascading foreign keys
could lose rows (Nextcloud core does not use them).
- **Restoring a whole user or the whole server needs `occ`** (shell access); see "Without shell
access" above.
- **`--install-apps` depends on the app store.** Nextcloud downloads the whole app list (about
38 MB) and gives up after 120 seconds; on a slow day apps are not installed. The restore then
lists them: install them afterwards with `occ app:install`, their data is restored already.
- A pack whose index upload failed stays on the location until removed by hand.
## Commands
@@ -254,4 +336,4 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
| `backup:key:import` | import a recovery kit on a fresh installation |
| `backup:user:backup`, `backup:user:restore` | one user via user_migration (asks before installing it) |
| `backup:user:backup`, `backup:user:list`, `backup:user:restore` | one user via user_migration (asks before installing it); list the exports |
+5 -4
View File
@@ -42,7 +42,7 @@ welcome: use the "Share your experience" link on the admin page.
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
you need help with restoring a backup after a crash.]]></description>
<version>0.9.11</version>
<version>0.9.12</version>
<licence>agpl</licence>
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
<namespace>NgBackup</namespace>
@@ -53,9 +53,9 @@ you need help with restoring a backup after a crash.]]></description>
<website>https://github.com/dillardblom/ng_backup</website>
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/3-settings.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/1-backups.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/2-restore.png</screenshot>
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/3-settings.png</screenshot>
<dependencies>
<php min-version="8.1"/>
<lib>sodium</lib>
@@ -92,6 +92,7 @@ you need help with restoring a backup after a crash.]]></description>
<command>OCA\NgBackup\Command\KeySlotRemove</command>
<command>OCA\NgBackup\Command\BackupUser</command>
<command>OCA\NgBackup\Command\RestoreUser</command>
<command>OCA\NgBackup\Command\ListUserExports</command>
<command>OCA\NgBackup\Command\KeyImport</command>
<command>OCA\NgBackup\Command\RestoreFull</command>
<command>OCA\NgBackup\Command\Verify</command>
+2
View File
@@ -10,6 +10,8 @@ Nextcloud back from an NG Backup location. Do them in this order.
- [ ] A new server with the **same Nextcloud major version** as the old one (for example 34),
a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for
the whole backup plus a margin.
- [ ] **Shell access** to the new server, to run `occ` (on a hosted Nextcloud without it, your
provider runs these steps).
- [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by
hand). Keep their packages somewhere safe beforehand.
+6
View File
@@ -114,10 +114,16 @@ Needs the `user_migration` app. NG Backup asks before installing it:
```
occ backup:user:backup offsite <uid>
occ backup:user:list offsite [<uid>] # the exports and their manifest
occ backup:user:restore offsite users/<uid>/<exportId> # replace (default)
occ backup:user:restore offsite users/<uid>/<exportId> --mode=as-backup
```
- `backup:user:restore` needs the manifest path that `backup:user:backup` prints at the end
(`users/<uid>/<exportId>`); `backup:user:list` shows it again. A user id alone also works when
that user has exactly one export on the location. With more (a `replace` restore adds a safety
export), the command lists them and you pass one.
- If `user_migration` is missing, the command asks to install and enable it. For non-interactive
use, add `--install-user-migration` to approve that explicitly. The command stops after the
installation; run it once more.
+22
View File
@@ -72,6 +72,10 @@ final class ExternalStorageFactory {
throw new BackendException("Auth mechanism $authId does not fit backend $backendId");
}
if (is_string($options['private_key'] ?? null)) {
$options['private_key'] = self::normalizePrivateKey($options['private_key']);
}
$config = new StorageConfig();
$config->setBackend($backend);
$config->setAuthMechanism($auth);
@@ -86,4 +90,22 @@ final class ExternalStorageFactory {
$storage = $backend->wrapStorage($storage);
return $auth->wrapStorage($storage);
}
/**
* An OpenSSH-format private key with its line breaks restored. Up to Nextcloud 34
* (phpseclib 2) a key whose newlines were lost or turned into spaces, e.g. on its way through
* a shell or a form, still loaded; Nextcloud 35 (phpseclib 3) refuses it with "Unable to read
* key" (#24). The key itself is not changed, so it is fixed when it is used, not when stored.
* Other formats (PEM, PKCS#8) still load as they are and are returned unchanged.
*/
public static function normalizePrivateKey(string $key): string {
if (!preg_match('/-----BEGIN OPENSSH PRIVATE KEY-----(.*?)-----END OPENSSH PRIVATE KEY-----/s', $key, $m)) {
return $key;
}
$body = (string)preg_replace('/\s+/', '', $m[1]);
if (!preg_match('#^[A-Za-z0-9+/]+=*$#', $body)) {
return $key;
}
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
}
}
+4 -7
View File
@@ -32,6 +32,7 @@ final class BackupJob extends TimedJob {
private RunMapper $runs,
private IAppConfig $appConfig,
private LoggerInterface $logger,
private \OCA\NgBackup\Service\Schedule $schedule,
) {
parent::__construct($time);
// Well below the usual 5-minute cron interval: the last run is stamped a moment after cron
@@ -101,14 +102,10 @@ final class BackupJob extends TimedJob {
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
}
/** Daily schedule: appconfig ng_backup/schedule_time "HH:MM" (server time); empty = no schedule. */
/** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */
private function startScheduled(): void {
$at = $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', '');
if (!preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $at)) {
return;
}
$todayAt = (int)strtotime(date('Y-m-d') . ' ' . $at);
if (time() < $todayAt) {
$todayAt = $this->schedule->todayAt();
if ($todayAt === null || time() < $todayAt) {
return;
}
foreach ($this->targets->list() as $t) {
+64
View File
@@ -0,0 +1,64 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Command;
use OCA\NgBackup\Service\TargetService;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Helper\Table;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
final class ListUserExports extends Command {
public function __construct(
private TargetService $targets,
) {
parent::__construct();
}
protected function configure(): void {
$this->setName('backup:user:list')
->setDescription('List the user exports (backup:user:backup) on a location')
->addArgument('target', InputArgument::OPTIONAL, 'Location name or id (default: all)')
->addArgument('user', InputArgument::OPTIONAL, 'Only the exports of this user id')
->setHelp(<<<'HELP'
The manifest column is what occ backup:user:restore expects. A safety export made by a restore
with --mode=replace is listed here as well.
Example:
occ backup:user:list offsite alice
HELP);
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$name = $input->getArgument('target');
$uid = $input->getArgument('user');
$rows = [];
try {
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
foreach ($targets as $t) {
foreach ($this->targets->repository($t)->listUserExports($uid) as $path) {
$parts = explode('/', $path, 3);
$user = $parts[1] ?? '';
$exportId = $parts[2] ?? '';
$time = \DateTimeImmutable::createFromFormat('Ymd\THis\Z', substr($exportId, 0, 16), new \DateTimeZone('UTC'));
$rows[] = [$t->getName(), $user, $time !== false ? $time->format('Y-m-d H:i') : '', $path];
}
}
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
if ($rows === []) {
$output->writeln('No user exports yet (occ backup:user:backup).');
return 0;
}
(new Table($output))->setHeaders(['location', 'user', 'date (UTC)', 'manifest'])->setRows($rows)->render();
return 0;
}
}
+19 -5
View File
@@ -33,7 +33,7 @@ final class RestoreUser extends Command {
$this->setName('backup:user:restore')
->setDescription('Restore a user from a user_migration export (see backup:user:backup)')
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>)')
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>), or a user id with exactly one export')
->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'replace or as-backup', UserRestoreService::MODE_REPLACE)
->addOption('force', null, InputOption::VALUE_NONE, 'Replace an existing account without asking (required when not interactive)')
->addOption('install-user-migration', null, InputOption::VALUE_NONE, 'Install and enable user_migration if it is missing (non-interactive consent)')
@@ -45,16 +45,31 @@ Never imports over a live account. Modes:
compare and transfer by hand; Nextcloud cannot rename a user id, so the backup gets
the new name, never the existing account
occ backup:user:list shows the exports on a location. A user id alone works when that user
has exactly one export there; with more, the command lists them and you pass one.
Example:
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34
occ backup:user:restore offsite alice
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34 --mode=as-backup
HELP);
}
protected function execute(InputInterface $input, OutputInterface $output): int {
$mode = (string)$input->getOption('mode');
$uid = explode('/', (string)$input->getArgument('manifest'))[1] ?? '';
if ($mode === UserRestoreService::MODE_REPLACE && $uid !== '' && $this->userManager->userExists($uid) && !$input->getOption('force')) {
try {
$target = $this->targets->get($input->getArgument('target'));
$ref = (string)$input->getArgument('manifest');
$manifest = $this->targets->repository($target)->resolveUserExport($ref);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
}
if ($manifest !== $ref) {
$output->writeln("Using $manifest.");
}
$uid = explode('/', $manifest)[1];
if ($mode === UserRestoreService::MODE_REPLACE && $this->userManager->userExists($uid) && !$input->getOption('force')) {
// Replace deletes the live account (after a safety export). Shares to others and data of
// apps without a user_migration migrator do not come back, so never do it unasked.
if (!$input->isInteractive()) {
@@ -69,9 +84,8 @@ HELP);
}
try {
$output->writeln('<comment>Note: on restore, user_migration only imports the app settings on its own allowlist (e.g. calendar view and reminder settings); other app settings of this user are skipped. Reason: NG Backup uses user_migration\'s importer; widening the list would mean building a settings importer of our own.</comment>');
$target = $this->targets->get($input->getArgument('target'));
$this->installer->ensureEnabled($input, $output);
$user = $this->userRestore->restoreUser($target, (string)$input->getArgument('manifest'), $mode);
$user = $this->userRestore->restoreUser($target, $manifest, $mode);
} catch (\Throwable $e) {
$output->writeln('<error>' . $e->getMessage() . '</error>');
return 1;
+10 -6
View File
@@ -47,6 +47,7 @@ final class AdminApiController extends Controller {
private \OCA\NgBackup\Service\AlertService $alerts,
private \OCA\NgBackup\Service\KeySlotService $slots,
private \OCA\NgBackup\Service\PruneService $prune,
private \OCA\NgBackup\Service\Schedule $schedule,
) {
parent::__construct(Application::APP_ID, $request);
}
@@ -72,7 +73,8 @@ final class AdminApiController extends Controller {
'runs' => $runs,
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
'schedule' => $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''),
'schedule' => $this->schedule->get()['time'],
'scheduleTimezone' => $this->schedule->get()['timezone'],
]);
}
@@ -240,12 +242,14 @@ final class AdminApiController extends Controller {
return new JSONResponse($this->runJson($run));
}
public function setSchedule(string $time): JSONResponse {
if ($time !== '' && !preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time)) {
return new JSONResponse(['error' => 'Use HH:MM or empty'], Http::STATUS_BAD_REQUEST);
/** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */
public function setSchedule(string $time, string $timezone = ''): JSONResponse {
try {
$this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
} catch (\InvalidArgumentException $e) {
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
}
$this->appConfig->setValueString(Application::APP_ID, 'schedule_time', $time);
return new JSONResponse(['schedule' => $time]);
return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]);
}
public function snapshots(int $targetId): JSONResponse {
+26 -3
View File
@@ -159,7 +159,7 @@ final class DbDumper {
// everywhere); tables it does not know are left out, as asked.
foreach ($missing as $table) {
if (isset($manifest['tables'][$table]['schema'])) {
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema']));
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
$report['created'][] = $table;
} else {
$report['missing'][] = $table;
@@ -319,8 +319,31 @@ final class DbDumper {
'foreignKeys' => $foreignKeys];
}
private static function tableFromStructure(string $name, array $structure): \Doctrine\DBAL\Schema\Table {
$table = new \Doctrine\DBAL\Schema\Table($name);
/**
* Table options for a table the restore creates: those of this installation's own tables.
* On MySQL/MariaDB that is the character set and collation, as Nextcloud's tables have them
* (utf8mb4, so emoji and other 4-byte characters fit); DBAL alone would use 3-byte utf8.
*
* @return array<string, mixed>
* @psalm-suppress UndefinedClass the DBAL classes come with Nextcloud, not with this app
*/
private function tableOptions(): array {
$options = $this->db->createSchemaManager()->createSchemaConfig()->getDefaultTableOptions();
if ($this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\AbstractMySQLPlatform) {
$collation = $this->db->fetchOne('SELECT TABLE_COLLATION FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ?',
[$this->prefix . 'appconfig']);
if (is_string($collation) && $collation !== '') {
$options['collation'] = $collation;
$options['charset'] = explode('_', $collation, 2)[0];
unset($options['collate']);
}
}
return $options;
}
/** @param array<string, mixed> $options table options (see tableOptions()) */
private static function tableFromStructure(string $name, array $structure, array $options = []): \Doctrine\DBAL\Schema\Table {
$table = new \Doctrine\DBAL\Schema\Table($name, [], [], [], [], $options);
foreach ($structure['columns'] as $c) {
$table->addColumn($c['name'], $c['type'], array_filter([
'length' => $c['length'], 'precision' => $c['precision'], 'scale' => $c['scale'],
+73 -2
View File
@@ -561,6 +561,31 @@ final class Repository {
return isset($this->catalog->userExports[$manifestPath]);
}
/**
* The recorded export $ref names: a manifest path (users/<uid>/<exportId>) as is, or a bare
* user id (also "users/<uid>") when that user has exactly one export here. Anything else
* throws, with the exports to choose from in the message.
*/
public function resolveUserExport(string $ref): string {
if ($this->isRecordedUserExport($ref)) {
return $ref;
}
$rest = trim($ref, '/');
if (str_starts_with($rest, 'users/')) {
$rest = substr($rest, strlen('users/'));
}
$uid = explode('/', $rest)[0];
$exports = $uid !== '' ? $this->listUserExports($uid) : [];
if (!str_contains($rest, '/') && count($exports) === 1) {
return $exports[0];
}
if ($exports === []) {
throw new RepositoryException("$ref is not a recorded user export for this location, and $uid has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports");
}
$which = str_contains($rest, '/') ? "$ref is not a recorded user export for this location" : "$uid has " . count($exports) . ' exports here';
throw new RepositoryException("$which; pass one of these (newest first):\n " . implode("\n ", $exports));
}
/** Permanently delete a user export. Its blobs are freed by the next prune() like any other. */
public function forgetUserExport(string $manifestPath): void {
if (!isset($this->catalog->userExports[$manifestPath])) {
@@ -845,7 +870,15 @@ final class Repository {
try {
foreach ($this->backend->getRanges($ranges) as $i => $data) {
foreach ($parts[$i] as [$id, $at, $length]) {
yield $this->decodeBlob($id, substr($data, $at, $length));
try {
$blob = $this->decodeBlob($id, substr($data, $at, $length));
} catch (\Exception) {
// a read that went wrong on the way (see loadBlob()): this blob on its own
$blob = $this->loadBlob($id);
// its pack stream is closed before the next range is read on the same connection
$this->packReader->close();
}
yield $blob;
$done++;
}
}
@@ -895,8 +928,46 @@ final class Repository {
} catch (BackendException) {
throw $e;
}
$pack = $moved;
}
return $this->decodeBlob($id, $encrypted);
try {
return $this->decodeBlob($id, $encrypted);
} catch (\Exception $damaged) {
// Bytes that do not decrypt: damaged on the location, or a read that went wrong on the
// way (a stream that lost its place: an SMB restore stopped after 17 hours like this).
// Read exactly this blob once more, on a request of its own; only a second failure
// means the data itself is damaged.
$this->packReader->close();
try {
$again = $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
} catch (\Exception) {
throw $damaged;
}
$data = $this->decodeBlob($id, $again);
if (function_exists('OCP\Log\logger')) {
\OCP\Log\logger('ng_backup')->warning('NG Backup: blob ' . $id . ' did not decrypt as first read from pack ' . $pack . ' at ' . $offset . '; read again, it is fine. First read: ' . self::howItDiffered($encrypted, $again));
}
return $data;
}
}
/**
* What a read that went wrong looked like next to the right bytes, for the log: where it starts
* to differ, and whether the rest is zeros or the right bytes from another place (a stream that
* lost its place).
*/
private static function howItDiffered(string $wrong, string $right): string {
$at = strspn($wrong ^ $right, "\0");
$text = strlen($wrong) . ' bytes, differ from byte ' . $at . ' of ' . strlen($right);
$tail = substr($wrong, $at, 64);
if ($tail === '') {
return $text . ' (too short)';
}
if (trim($tail, "\0") === '') {
return $text . ' (zeros)';
}
$found = strpos($right, $tail);
return $text . ($found !== false ? ' (the right bytes from ' . ($found - $at) . ' bytes further on)' : ' (other bytes)');
}
/** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */
+110
View File
@@ -0,0 +1,110 @@
<?php
declare(strict_types=1);
// SPDX-FileCopyrightText: 2026 Dillard Blom
// SPDX-License-Identifier: AGPL-3.0-or-later
namespace OCA\NgBackup\Service;
use OCA\NgBackup\AppInfo\Application;
use OCP\IAppConfig;
use OCP\IConfig;
use OCP\IGroupManager;
/**
* The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a
* zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone
* is converted once to the zone of the server or of an administrator, at today's offset: it then
* shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
*/
final class Schedule {
public const TIME_KEY = 'schedule_time';
public const ZONE_KEY = 'schedule_timezone';
public function __construct(
private IAppConfig $appConfig,
private IConfig $config,
private IGroupManager $groups,
) {
}
/** @return array{time: string, timezone: non-empty-string} time "" = no schedule */
public function get(): array {
$time = $this->appConfig->getValueString(Application::APP_ID, self::TIME_KEY, '');
$zone = $this->appConfig->getValueString(Application::APP_ID, self::ZONE_KEY, '');
if ($zone === '' || !self::validZone($zone)) {
$zone = $this->guessZone();
if ($time !== '' && self::validTime($time)) {
$time = self::convert($time, 'UTC', $zone);
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
}
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
}
return ['time' => $time, 'timezone' => $zone];
}
/** @throws \InvalidArgumentException for a malformed time or an unknown zone */
public function set(string $time, string $zone): void {
if ($time !== '' && !self::validTime($time)) {
throw new \InvalidArgumentException('Use HH:MM or empty');
}
if (!self::validZone($zone)) {
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
}
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
}
/** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */
public function todayAt(?int $now = null): ?int {
['time' => $time, 'timezone' => $zone] = $this->get();
if ($time === '' || !self::validTime($time)) {
return null;
}
$tz = new \DateTimeZone($zone);
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d');
return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp();
}
/**
* "HH:MM" in zone $from as "HH:MM" in zone $to, at today's offsets.
*
* @param non-empty-string $from
* @param non-empty-string $to
*/
public static function convert(string $time, string $from, string $to, ?int $now = null): string {
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone(new \DateTimeZone($from))->format('Y-m-d');
return (new \DateTimeImmutable($day . ' ' . $time, new \DateTimeZone($from)))->setTimezone(new \DateTimeZone($to))->format('H:i');
}
public static function validTime(string $time): bool {
return preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time) === 1;
}
/** @psalm-assert-if-true non-empty-string $zone */
public static function validZone(string $zone): bool {
return in_array($zone, \DateTimeZone::listIdentifiers(), true);
}
/**
* The server's default_timezone when set, else the first administrator's, else UTC. A server
* zone of UTC is passed over on purpose: it is mostly what an image or installer left there,
* while the administrator who set the time read it as their own (#19).
*
* @return non-empty-string
*/
private function guessZone(): string {
$zone = $this->config->getSystemValueString('default_timezone', '');
if ($zone !== '' && $zone !== 'UTC' && self::validZone($zone)) {
return $zone;
}
foreach ($this->groups->get('admin')?->getUsers() ?? [] as $user) {
$zone = $this->config->getUserValue($user->getUID(), 'core', 'timezone', '');
if ($zone !== '' && self::validZone($zone)) {
return $zone;
}
}
return 'UTC';
}
}
+44 -3
View File
@@ -7,7 +7,7 @@
<NcSettingsSection :name="t('ng_backup', 'NG Backup')"
:description="t('ng_backup', 'Encrypted, incremental backups of this Nextcloud (database, configuration and data). Backup locations are never mounted and are not visible in Files.')">
<p class="ngb-feedback">
{{ t('ng_backup', 'NG Backup is in beta. We would like to hear how it works for you, also when everything went fine.') }}
{{ t('ng_backup', 'We would like to hear how NG Backup works for you, also when everything went fine.') }}
<a :href="feedbackUrl" target="_blank" rel="noopener noreferrer">{{ t('ng_backup', 'Share your experience ↗') }}</a>
</p>
<NcLoadingIcon v-if="!status" :size="32" />
@@ -18,9 +18,18 @@
<div class="ngb-tabs">
<NcButton :variant="tab === 'backup' ? 'primary' : 'secondary'" @click="tab = 'backup'">{{ t('ng_backup', 'Backup and restore') }}</NcButton>
<NcButton :variant="tab === 'settings' ? 'primary' : 'secondary'" @click="tab = 'settings'">{{ t('ng_backup', 'Settings') }}</NcButton>
<NcButton :variant="tab === 'docs' ? 'primary' : 'secondary'" @click="tab = 'docs'">{{ t('ng_backup', 'Docs') }}</NcButton>
</div>
<template v-if="tab === 'settings'">
<NcSettingsSection v-if="tab === 'docs'" :name="t('ng_backup', 'Documentation')">
<ul class="ngb-docs">
<li v-for="doc in docs" :key="doc.url">
<a :href="doc.url" target="_blank" rel="noopener noreferrer">{{ doc.name }} ↗</a>
<span>{{ doc.description }}</span>
</li>
</ul>
</NcSettingsSection>
<template v-else-if="tab === 'settings'">
<KeySetup :status="status" @changed="refresh" />
<Locations v-if="status.key.everConfirmed" :status="status" @changed="refresh" />
</template>
@@ -52,6 +61,25 @@ import RestoreBrowser from './components/RestoreBrowser.vue'
// Opens the experience report form in .github/ISSUE_TEMPLATE
const feedbackUrl = 'https://github.com/dillardblom/ng_backup/issues/new?template=experience.yml'
const repoUrl = 'https://github.com/dillardblom/ng_backup'
const docs = [
{
name: t('ng_backup', 'How-to'),
description: t('ng_backup', 'Step by step: from first install to a tested restore.'),
url: repoUrl + '/blob/main/docs/HOWTO.md',
},
{
name: t('ng_backup', 'Disaster recovery'),
description: t('ng_backup', 'Your server is gone: get Nextcloud back on a new server from a backup location.'),
url: repoUrl + '/blob/main/docs/DISASTER-RECOVERY.md',
},
{
name: t('ng_backup', 'Roadmap'),
description: t('ng_backup', 'What is planned for the next versions.'),
url: repoUrl + '#roadmap',
},
]
const status = ref(null)
const allRuns = ref(false) // runs older than the last days too
const error = ref('')
@@ -89,8 +117,10 @@ onBeforeUnmount(() => clearTimeout(timer))
</script>
<style scoped>
/* NcSettingsSection sets its own width (at most 900px): the runs table needs more */
.ng-backup :deep(.settings-section) {
max-width: 900px;
width: min(1400px, 100% - var(--default-grid-baseline) * 7 * 2);
max-width: none;
}
.ngb-feedback {
color: var(--color-text-maxcontrast);
@@ -100,6 +130,17 @@ onBeforeUnmount(() => clearTimeout(timer))
color: var(--color-primary-element);
text-decoration: underline;
}
.ngb-docs li {
margin-bottom: 8px;
}
.ngb-docs a {
color: var(--color-primary-element);
text-decoration: underline;
margin-right: 8px;
}
.ngb-docs span {
color: var(--color-text-maxcontrast);
}
.ngb-tabs {
display: flex;
gap: 8px;
+1 -1
View File
@@ -33,7 +33,7 @@ export default {
testTarget: (id) => call('post', `/targets/${id}/test`),
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
setSchedule: (time) => call('put', '/schedule', { time }),
setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }),
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
+10 -6
View File
@@ -6,8 +6,8 @@
<NcSettingsSection :name="t('ng_backup', 'Backups')"
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
<div class="ngb-row">
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, empty = off)')" />
<NcButton :disabled="schedule === status.schedule" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, {zone}, empty = off)', { zone: browserZone })" />
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
</div>
<div class="ngb-row">
<NcButton v-for="target in status.targets" :key="target.id" :disabled="isRunning(target.id)" @click="start(target)">
@@ -79,15 +79,19 @@ import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
import NcTextField from '@nextcloud/vue/components/NcTextField'
import api from '../api.js'
import { formatDateTime, formatDate, formatSize } from '../format.js'
import { formatDateTime, formatDate, formatSize, toZone } from '../format.js'
import { confirmPassword } from '@nextcloud/password-confirmation'
import { reactive } from 'vue'
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
const emit = defineEmits(['changed', 'show-all-runs'])
const schedule = ref(props.status.schedule)
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
// zone is shown in this one, so it reads as the moment it runs here.
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s
const schedule = ref(shown(props.status.schedule))
const error = ref('')
watch(() => props.status.schedule, (s) => { schedule.value = s })
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
const trash = reactive({})
async function loadTrash() {
@@ -131,7 +135,7 @@ async function start(target) {
async function saveSchedule() {
error.value = ''
try {
await api.setSchedule(schedule.value.trim())
await api.setSchedule(schedule.value.trim(), browserZone)
emit('changed')
} catch (e) {
error.value = e.message
+27
View File
@@ -26,3 +26,30 @@ export function formatSize(bytes) {
}
return i === 0 ? `${b} B` : `${b.toFixed(1)} ${units[i]}`
}
/**
* A time of day "HH:MM" in zone `from`, as "HH:MM" in zone `to`, at today's offsets.
*
* @param {string} hhmm the time
* @param {string} from IANA zone it is in
* @param {string} to IANA zone to show it in
* @param {Date} now the day to convert on (today)
* @return {string}
*/
export function toZone(hhmm, from, to, now = new Date()) {
const [h, m] = hhmm.split(':').map(Number)
const parts = (zone, date) => Object.fromEntries(new Intl.DateTimeFormat('en-GB', {
timeZone: zone, hourCycle: 'h23', year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit',
}).formatToParts(date).map((p) => [p.type, p.value]))
const offsetAt = (at) => {
const seen = parts(from, new Date(at))
return Date.UTC(+seen.year, +seen.month - 1, +seen.day, +seen.hour, +seen.minute) - at
}
const today = parts(from, now)
const wall = Date.UTC(+today.year, +today.month - 1, +today.day, h, m)
// the offset at the wall time itself: on a day the clocks change it differs from the one at
// midnight, so it is looked up a second time at the moment found first
const at = wall - offsetAt(wall - offsetAt(wall))
const there = parts(to, new Date(at))
return `${there.hour}:${there.minute}`
}
@@ -0,0 +1,35 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Backend;
use OCA\NgBackup\Backend\ExternalStorageFactory;
use PHPUnit\Framework\TestCase;
/** ExternalStorageFactory::normalizePrivateKey(): keys that phpseclib 2 read, for phpseclib 3 (#24). */
class ExternalStorageFactoryTest extends TestCase {
/** An OpenSSH-format key laid out as ssh-keygen writes it; the body is random, not a real key. */
private static function key(): string {
$body = base64_encode(random_bytes(1000));
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
}
public function testRestoresLostLineBreaks(): void {
$key = self::key();
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey($key));
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", ' ', $key)), 'newlines as spaces');
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", '', $key)), 'no newlines');
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", "\r\n", $key)), 'CRLF');
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(rtrim($key)), 'no trailing newline');
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(preg_replace('/^/m', ' ', $key)), 'indented');
}
public function testLeavesOtherFormatsAlone(): void {
$pem = "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC,FF44E0A94036766C\n\nAAAA\n-----END RSA PRIVATE KEY-----\n";
$this->assertSame($pem, ExternalStorageFactory::normalizePrivateKey($pem));
$this->assertSame('not a key', ExternalStorageFactory::normalizePrivateKey('not a key'));
$odd = "-----BEGIN OPENSSH PRIVATE KEY-----\\nAAAA\\n-----END OPENSSH PRIVATE KEY-----";
$this->assertSame($odd, ExternalStorageFactory::normalizePrivateKey($odd), 'not base64 inside: unchanged');
}
}
+133
View File
@@ -0,0 +1,133 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Repository;
use OCA\NgBackup\Backend\IBackend;
use OCA\NgBackup\Backend\LocalBackend;
use OCA\NgBackup\Crypto\KeyRing;
use OCA\NgBackup\Repository\Repository;
use OCA\NgBackup\Tests\Unit\TempDirTrait;
use PHPUnit\Framework\TestCase;
/**
* A pack stream that hands out wrong bytes once (an SMB restore stopped after 17 hours with
* "Decryption failed"): the blob is read once more on a request of its own; only data that is
* damaged on the location stops the restore.
*/
class DamagedReadTest extends TestCase {
use TempDirTrait;
private string $root;
private KeyRing $keys;
private string $src;
private string $snapshot;
protected function setUp(): void {
$this->root = $this->tempDir();
$this->keys = KeyRing::generate();
$this->src = $this->tempDir();
for ($i = 0; $i < 6; $i++) {
file_put_contents("$this->src/f$i", random_bytes(3 * 1048576));
}
$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
}
public function testABlobThatCameInDamagedOnceIsReadAgain(): void {
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1);
$target = $this->tempDir();
$this->assertSame(6, Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $target));
$this->assertSame(self::hashTree($this->src), self::hashTree($target));
$this->assertSame(0, $backend->streams, 'the damaged read happened');
}
public function testATableDumpThatCameInDamagedOnceIsReadAgain(): void {
$repo = Repository::openWithKey(new LocalBackend($this->root), $this->keys);
$stats = [];
$writer = $repo->blobWriter($stats);
$lines = [];
for ($i = 0; $i < 20000; $i++) {
$lines[] = json_encode(['i' => $i, 'pad' => str_repeat('x', $i % 400)]);
}
$writer->write(implode("\n", $lines) . "\n");
$blobs = $writer->finish();
$repo->flushPacks();
$backend = new GarblingBackend(new LocalBackend($this->root), ranges: 1);
$this->assertSame($lines, iterator_to_array(Repository::openWithKey($backend, $this->keys)->readLines($blobs), false));
$this->assertSame(0, $backend->ranges, 'the damaged read happened');
}
public function testDataDamagedOnTheLocationStillStopsTheRestore(): void {
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1000, ranges: 1000);
$this->expectException(\Exception::class);
Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $this->tempDir());
}
}
/** Pack reads that flip a byte: the first $streams pack streams and $ranges range reads. */
final class GarblingBackend implements IBackend {
use \OCA\NgBackup\Backend\GetsOneByOne;
public function __construct(
private IBackend $inner,
public int $streams = 0,
public int $ranges = 0,
) {
}
public function put(string $path, $stream): void {
$this->inner->put($path, $stream);
}
public function get(string $path) {
$fh = $this->inner->get($path);
if (!str_starts_with($path, 'packs/') || $this->streams <= 0) {
return $fh;
}
$this->streams--;
$data = stream_get_contents($fh);
fclose($fh);
for ($k = 100; $k < strlen($data); $k += 65536) {
$data[$k] = chr(ord($data[$k]) ^ 0x55);
}
$out = fopen('php://temp', 'w+b');
fwrite($out, $data);
rewind($out);
return $out;
}
public function getRange(string $path, int $offset, int $length): string {
$data = $this->inner->getRange($path, $offset, $length);
if (str_starts_with($path, 'packs/') && $this->ranges > 0) {
$this->ranges--;
$data[0] = chr(ord($data[0]) ^ 0x55);
}
return $data;
}
public function exists(string $path): bool {
return $this->inner->exists($path);
}
public function move(string $from, string $to): void {
$this->inner->move($from, $to);
}
public function list(string $prefix): array {
return $this->inner->list($prefix);
}
public function listFiles(string $dir): array {
return $this->inner->listFiles($dir);
}
public function delete(string $path): void {
$this->inner->delete($path);
}
}
+47
View File
@@ -106,4 +106,51 @@ class UserExportTest extends TestCase {
$this->expectExceptionMessage('missing on the location');
$repo->listUserExports();
}
public function testResolveTakesARecordedPathAsIs(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$path = 'users/alice/20261005T120000Z-aaaa0001';
$this->putManifest($repo, $path);
$this->assertSame($path, $repo->resolveUserExport($path));
}
public function testResolveTakesAUserIdWithOneExport(): void {
// Issue #22: "backup:user:restore <location> New_User" was rejected as not a recorded export.
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$path = 'users/alice/20261005T120000Z-aaaa0001';
$this->putManifest($repo, $path);
$this->putManifest($repo, 'users/alice2/20261005T120000Z-aaaa0002');
$this->assertSame($path, $repo->resolveUserExport('alice'));
$this->assertSame($path, $repo->resolveUserExport('users/alice/'));
}
public function testResolveListsTheExportsOfAUserWithMore(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
$this->putManifest($repo, 'users/alice/20261006T120000Z-aaaa0002');
$this->expectException(RepositoryException::class);
$this->expectExceptionMessage("alice has 2 exports here; pass one of these (newest first):\n users/alice/20261006T120000Z-aaaa0002\n users/alice/20261005T120000Z-aaaa0001");
$repo->resolveUserExport('alice');
}
public function testResolveListsTheExportsOfTheUserOfAWrongPath(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
$this->expectException(RepositoryException::class);
$this->expectExceptionMessage("users/alice/typo is not a recorded user export for this location; pass one of these (newest first):\n users/alice/20261005T120000Z-aaaa0001");
$repo->resolveUserExport('users/alice/typo');
}
public function testResolveNamesTheExpectedFormForAUserWithoutExports(): void {
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
$this->expectException(RepositoryException::class);
$this->expectExceptionMessage('bob has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports');
$repo->resolveUserExport('bob');
}
}
+71
View File
@@ -0,0 +1,71 @@
<?php
declare(strict_types=1);
namespace OCA\NgBackup\Tests\Unit\Service;
use OCA\NgBackup\Service\Schedule;
use OCP\IAppConfig;
use OCP\IConfig;
use OCP\IGroup;
use OCP\IGroupManager;
use OCP\IUser;
use PHPUnit\Framework\TestCase;
/** The daily time in a time zone (issue #19), and the conversion of a time saved without one. */
class ScheduleTest extends TestCase {
private array $values = [];
private function schedule(string $serverZone = '', string $adminZone = ''): Schedule {
$app = $this->createMock(IAppConfig::class);
$app->method('getValueString')->willReturnCallback(fn (string $a, string $k, string $d = '') => $this->values[$k] ?? $d);
$app->method('setValueString')->willReturnCallback(function (string $a, string $k, string $v): bool {
$this->values[$k] = $v;
return true;
});
$config = $this->createMock(IConfig::class);
$config->method('getSystemValueString')->willReturn($serverZone);
$config->method('getUserValue')->willReturn($adminZone);
$user = $this->createMock(IUser::class);
$user->method('getUID')->willReturn('admin');
$group = $this->createMock(IGroup::class);
$group->method('getUsers')->willReturn([$user]);
$groups = $this->createMock(IGroupManager::class);
$groups->method('get')->willReturn($group);
return new Schedule($app, $config, $groups);
}
public function testATimeSavedWithoutAZoneShowsAsItRanInTheAdministratorsZone(): void {
$this->values = [Schedule::TIME_KEY => '02:30'];
$s = $this->schedule(adminZone: 'Europe/Berlin')->get();
// it ran at 02:30 UTC: in summer that is 04:30 in Berlin, in winter 03:30
$summer = Schedule::convert('02:30', 'UTC', 'Europe/Berlin', strtotime('2026-07-01 12:00 UTC'));
$this->assertSame('04:30', $summer);
$this->assertSame('Europe/Berlin', $s['timezone']);
$this->assertSame(Schedule::convert('02:30', 'UTC', 'Europe/Berlin'), $s['time']);
}
public function testTheServersDefaultZoneComesFirst(): void {
$this->values = [Schedule::TIME_KEY => '01:00'];
$this->assertSame('America/Toronto', $this->schedule('America/Toronto', 'Europe/Berlin')->get()['timezone']);
}
public function testWithoutAnyZoneItStaysUtc(): void {
$this->values = [Schedule::TIME_KEY => '02:30'];
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
}
public function testTodayAtIsTheMomentInTheSchedulesZone(): void {
$this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin'];
$at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC'));
$this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC');
}
public function testAnUnknownZoneOrABadTimeIsRefused(): void {
$s = $this->schedule();
$this->expectException(\InvalidArgumentException::class);
$s->set('03:00', 'Mars/Olympus');
}
}
+11 -2
View File
@@ -47,6 +47,12 @@ if [ "$LEGACY" = legacy ]; then
fi
run_on() { docker exec -u www-data "$1" php occ "${@:2}"; }
# The app store sends the whole app list (~38 MB) and Nextcloud gives up after 120 s: on a slow day
# an install fails for reasons outside this test. Up to three tries.
install_app() { # container app
for i in 1 2 3; do run_on "$1" app:install "$2" >/dev/null 2>&1 && return 0; sleep 20; done
return 1
}
# Database settings for an instance: SQLite inside the container, or its own PostgreSQL container
db_env() {
@@ -109,7 +115,7 @@ if [ "$DB" != sqlite ]; then
# while the restore loads tables in name order, accounts before mailboxes.
docker run -d --name ngb-frt-imap --network "$NET" -e MAILNAME=mail.domain.tld -e MAIL_ADDRESS=user@domain.tld \
-e MAIL_PASS=mypassword ghcr.io/christophwurst/docker-imap-devel:latest >/dev/null
run_on ngb-frt-origin app:install mail >/dev/null || fail "install mail on the origin"
install_app ngb-frt-origin mail || fail "install mail on the origin (app store unreachable?)"
run_on ngb-frt-origin config:system:set app.mail.verify-tls-peer --type=boolean --value=false >/dev/null
sleep 10
run_on ngb-frt-origin mail:account:create frtest Test user@domain.tld ngb-frt-imap 993 ssl user@domain.tld mypassword \
@@ -118,7 +124,9 @@ if [ "$DB" != sqlite ]; then
[ "$(sql origin 'select count(*) from oc_mail_accounts where drafts_mailbox_id is not null')" = 1 ] \
|| fail "the mail account does not point at its mailboxes"
# Deck installed and then switched off: its tables stay in the database and in the backup
run_on ngb-frt-origin app:install deck >/dev/null || fail "install deck on the origin"
install_app ngb-frt-origin deck || fail "install deck on the origin (app store unreachable?)"
# a 4-byte character (emoji) in a table the restore has to create: utf8mb4 on MySQL/MariaDB
sql origin "insert into oc_deck_boards (title, owner, color, archived, deleted_at, last_modified) values ('Plan 📌', 'frtest', '0087C5', false, 0, 0)" >/dev/null
run_on ngb-frt-origin app:disable deck >/dev/null
ok "origin has a synced Mail account and the tables of a disabled app"
fi
@@ -202,6 +210,7 @@ if [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ] && [ "$LEGACY" = legacy
ok "older backup: tables of the disabled app left out, Mail restored"
elif [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ]; then
[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created"
[ "$(sql restore "select title from oc_deck_boards")" = "Plan 📌" ] || fail "a 4-byte character did not survive in a created table"
[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
[ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \
|| fail "the foreign keys of Mail are missing after the restore"
+12 -2
View File
@@ -36,8 +36,13 @@ MANIFEST=$(occ backup:user:backup ci "$U" | sed -n 's/.* to \(.*\)\.$/\1/p')
[ -n "$MANIFEST" ] || fail "backup:user:backup printed no manifest path"
ok "backed up $U to $MANIFEST"
# as-backup: restore into <uid>-bak; the original account must stay untouched.
occ backup:user:restore ci "$MANIFEST" --mode=as-backup >/dev/null || fail "restore as-backup"
occ backup:user:list ci "$U" | grep -qF "$MANIFEST" || fail "backup:user:list does not show $MANIFEST"
ok "backup:user:list shows the export"
# as-backup: restore into <uid>-bak; the original account must stay untouched. $U has one export,
# so the bare user id names it (#22).
OUT=$(occ backup:user:restore ci "$U" --mode=as-backup) || fail "restore as-backup by user id"
echo "$OUT" | grep -qF "Using $MANIFEST." || fail "restore by user id did not say which export it used"
occ files:scan "$U-bak" >/dev/null
SUM_BAK=$($X sh -c "cd $DATA/$U-bak/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
[ "$SUM_BEFORE" = "$SUM_BAK" ] || fail "the <uid>-bak account's files differ from the original"
@@ -55,6 +60,11 @@ SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xa
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after a replace restore"
ok "replace restores the original content under the same uid, byte-identical"
# The replace above added a safety export: a bare user id is now ambiguous and must list both.
if OUT=$(occ backup:user:restore ci "$U" --mode=as-backup 2>&1); then fail "restore by an ambiguous user id ran"; fi
echo "$OUT" | grep -qF "$U has 2 exports here" && echo "$OUT" | grep -qF "$MANIFEST" || fail "an ambiguous user id did not list the exports"
ok "an ambiguous user id lists the exports instead of picking one"
# Retention: $U now has the manual export above plus the automatic safety-export the replace
# restore made of it just before deleting it (4 total after these two more), then --user-last=2
# must forget exactly the 2 oldest.