Compare commits
36 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 66615c2a88 | |||
| cc75c7128c | |||
| a0c4abb60a | |||
| 4847ce293d | |||
| 90dd97977a | |||
| 08534975d7 | |||
| 95e3dfcf6d | |||
| 7f4cae0e42 | |||
| 965440d1f7 | |||
| 3a0f882a38 | |||
| 632d2a2838 | |||
| 9e9234e2a8 | |||
| 3ac3298313 | |||
| 384b2d2792 | |||
| 8032411789 | |||
| b4f20cb070 | |||
| 343389b98b | |||
| 133f5f9d34 | |||
| cf4bd9db7f | |||
| 67cc350b8c | |||
| fb5b234e8a | |||
| fd12c586b6 | |||
| 6ffe979ea7 | |||
| a03b7354b3 | |||
| be1c628f07 | |||
| a8355df12e | |||
| cd251cffb3 | |||
| 0b4872599d | |||
| b7d09c9784 | |||
| 24686897c3 | |||
| 315e83b847 | |||
| b728db36c7 | |||
| 6ed0a555cf | |||
| 221ed9dc17 | |||
| 285153a968 | |||
| 6c8a78f6bb |
@@ -41,6 +41,20 @@ jobs:
|
||||
- run: composer install --no-interaction --no-progress
|
||||
- run: composer psalm
|
||||
|
||||
frontend:
|
||||
# Builds the admin page, so a broken Vue component fails CI instead of showing up after a release.
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
persist-credentials: false
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
cache: npm
|
||||
- run: npm ci --no-audit --no-fund
|
||||
- run: npm run build
|
||||
|
||||
integration:
|
||||
# Full round trip (key, kit, location, backup, incremental, restore modes, database dump and
|
||||
# restore, retention, trash) on a fresh Nextcloud per version and database.
|
||||
|
||||
@@ -7,6 +7,62 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
|
||||
beta/daily update channel, which would hide the app from most admins. The stability stage is
|
||||
instead carried by the app description, the git tag, the release, and this file.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- `occ backup:user:list [location] [user]` shows the user exports on a location, with the manifest
|
||||
path `backup:user:restore` expects. Before, that path was only printed once by
|
||||
`backup:user:backup` (#22).
|
||||
|
||||
### Changed
|
||||
- `occ backup:user:restore` also takes a user id instead of the manifest path, when that user has
|
||||
exactly one export on the location. With more, and for a manifest path that is not on the
|
||||
location, the error lists that user's exports; for a user without exports it names the
|
||||
expected form (#22). Before, `backup:user:restore <location> <uid>` only said "is not a
|
||||
recorded user export for this location".
|
||||
|
||||
### Fixed
|
||||
- An SFTP location with an RSA key in OpenSSH format ("BEGIN OPENSSH PRIVATE KEY") whose line
|
||||
breaks were lost, e.g. turned into spaces on the way in, stopped working after an upgrade to
|
||||
Nextcloud 35 with "Unable to read key": Nextcloud 35 reads keys with a stricter library. The
|
||||
line breaks are now restored before the key is used, so the stored location works again without
|
||||
re-entering it (#24).
|
||||
|
||||
## [1.0.0] - 2026-10-29
|
||||
|
||||
First stable release: the same code as 1.0.0-rc.6, which had no blocking problems reported in its
|
||||
week out (about 220 downloads). Only the version number and the README changed. The fixes for
|
||||
what was reported (#22, #24) follow in 1.0.1.
|
||||
|
||||
## [1.0.0-rc.6] - 2026-10-24
|
||||
|
||||
Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery
|
||||
simulations and a user report. If no problems are reported for about a week, 1.0.0 will be this
|
||||
same code.
|
||||
|
||||
### Fixed
|
||||
- The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in
|
||||
Berlin (#19). The time is now saved with the administrator's time zone and shown with it. A time
|
||||
saved before is converted once to the zone of the server or an administrator, so it shows as it
|
||||
effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer) and nothing moves.
|
||||
- A full restore over SMB stopped after 17 hours with "Decryption failed": a read over a long
|
||||
line handed out wrong bytes (the backup itself was fine, the same files restored identically
|
||||
afterwards). A blob that does not decrypt as first read, of a file or of the database, is now
|
||||
read once more on a request of its own, and only a second failure stops the restore.
|
||||
- A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table
|
||||
(of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the
|
||||
table got 3-byte utf8. Tables the restore creates now get this installation's character set and
|
||||
collation (utf8mb4). Found in a disaster recovery simulation onto MariaDB.
|
||||
|
||||
### Changed
|
||||
- The admin page uses more of the screen (sections up to 1400 pixels instead of 900), so the
|
||||
list of backup runs fits without scrolling sideways.
|
||||
- README: a roadmap, the tested disaster recoveries, measurements for OVH S3, the Storage Box over
|
||||
SFTP and SMB, how to build the SMB extension when PECL is down, and what works without shell
|
||||
access. The guide lists shell access among what you need.
|
||||
- The full-restore integration test checks a 4-byte character in a table the restore creates, and
|
||||
retries app installs from the app store.
|
||||
|
||||
## [1.0.0-rc.5] - 2026-10-22
|
||||
|
||||
Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
|
||||
|
||||
@@ -395,11 +395,73 @@ in README "Known limitations":
|
||||
- **1.0.1:**
|
||||
1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
|
||||
restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
|
||||
2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron
|
||||
staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue
|
||||
#13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles.
|
||||
2. Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit
|
||||
oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21):
|
||||
a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode`
|
||||
(`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de
|
||||
laatste run te lang geleden: melding met de cron-regel erbij ("Backups only move while
|
||||
someone has Nextcloud open"), en "waiting for background jobs" als een lopende backup lang
|
||||
geen stap zette.
|
||||
b. **De pagina laat de backup lopen** zolang ze openstaat: in `ajax`/`webcron` voert de
|
||||
statusaanroep die de pagina toch al herhaalt ook een korte, begrensde stap uit (zoals de
|
||||
korte stappen die AJAX-cron nu al krijgt). "Back up now" werkt dan ook zonder cron zolang
|
||||
het tabblad openstaat; de melding zegt dat geplande backups echte cron nodig hebben.
|
||||
De melding zegt (zonder werkende systeem-cron): backups hebben systeem-cron nodig, ook de
|
||||
gewone nachtelijke; zo stel je cron in (link naar de Nextcloud-documentatie over
|
||||
achtergrondjobs voor de versie van de server, nu
|
||||
https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html,
|
||||
versienummer uit de server, niet vast); zolang er geen cron is, loopt een backup die je
|
||||
met de knop start alleen door terwijl de NG Backup-pagina openstaat.
|
||||
c. **Voortgang in de databasefase** (issue #13): nu staat er alleen "running (db)", wat er
|
||||
hetzelfde uitziet als vastzitten. Toon "running (db: 47/183 tables)" zoals de bestandsfase
|
||||
een percentage toont, en "waiting for background jobs (last ran 12 min ago)" als een run
|
||||
een tijd geen stap zette, ook bij een cron die is blijven staan.
|
||||
Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en
|
||||
`cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop
|
||||
(vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen).
|
||||
3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later),
|
||||
zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest.
|
||||
4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de
|
||||
tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop
|
||||
stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een
|
||||
bestaande parameter.
|
||||
5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen
|
||||
wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind
|
||||
opsommen, in plaats van stoppen.
|
||||
6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van
|
||||
rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-<id>` staan; de
|
||||
geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files.
|
||||
Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op.
|
||||
7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden
|
||||
terug (er komt een `/` achter); nu werkt alleen een map.
|
||||
8. **Uitleg (i) bij de prullenbak van herstelpunten** (besluit 2026-10-25): bij "removed by
|
||||
retention policy" een infoknop met wat het betekent: het bewaarbeleid houdt voor oudere dagen
|
||||
één herstelpunt per dag (het laatste), weken en maanden; de rest gaat naar de prullenbak en
|
||||
wordt pas na de ingestelde wachttijd (`delete_delay_days`, standaard 30) echt verwijderd, en is
|
||||
tot die datum terug te zetten. Aanleiding: de beheerder zag "19-10 13:12 – after rc4 upgrade
|
||||
removed by retention policy" en dacht dat er iets was weggegooid.
|
||||
9. **Tijdzone in `occ backup:status`**: de tabel toont de servertijd (UTC), de webinterface de
|
||||
tijd van de beheerder; dat leest als een verkeerde starttijd. Zet de zone in de kolomkop
|
||||
("started (UTC)") of toon de tijden in de zone van het schema.
|
||||
10. **User-restore op gebruikersnaam** (issue #22, 2026-10-27): `backup:user:restore <locatie>
|
||||
New_User` gaf alleen "is not a recorded user export"; het argument moet het manifestpad
|
||||
zijn, dat alleen `backup:user:backup` eenmalig toonde. Nu: `backup:user:list` toont de
|
||||
exports, een kale uid werkt bij precies één export, anders een lijst om uit te kiezen —
|
||||
**klaar** (2026-10-27).
|
||||
11. **Tabblad "Docs" in de beheerpagina** (besluit 2026-10-27): een derde knop naast "Backup and
|
||||
restore" en "Settings", met links naar de HOWTO (`docs/HOWTO.md`), de DR-gids
|
||||
(`docs/DISASTER-RECOVERY.md`) en de roadmap (README, `#roadmap`) op GitHub, elk met een
|
||||
regel wat erin staat. Nu zijn ze alleen via de README te vinden. Zichtbaar ook zonder
|
||||
sleutel, zodat de gids vindbaar is vóór de eerste installatie-stap. "NG Backup is in beta"
|
||||
gaat uit de feedbackregel — **klaar** (2026-10-27).
|
||||
- **DR-tests over databasegrenzen** (besluit 2026-10-20; productie nc01 draait PostgreSQL): na
|
||||
de MariaDB→PostgreSQL-fout (tellers, rc5) elke overstap in de integratietest (`to-pgsql`,
|
||||
`to-mysql`, `to-sqlite`, ook in CI). Volle DR-tests met echte data: vóór 1.0.0 de S3-simulatie
|
||||
(MariaDB-bron, 138 GB) nogmaals naar een MariaDB-doel en de SMB-simulatie naar een SQLite-doel;
|
||||
**na 1.0.0** de 1 TB van nc01 (PostgreSQL) naar een MariaDB-doel.
|
||||
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
|
||||
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
|
||||
een paar bestanden.
|
||||
- **1.1.0:**
|
||||
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
|
||||
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
|
||||
@@ -412,6 +474,31 @@ in README "Known limitations":
|
||||
3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel
|
||||
een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf,
|
||||
33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1.
|
||||
4. **Schema per locatie** (besluit 2026-10-19): in de instellingen van een locatie "uit",
|
||||
"dagelijks om HH:MM" of "elke N uur (vanaf HH:MM)". Nu is er één `schedule_time` voor alle
|
||||
locaties, één keer per dag. Bij de update wordt de huidige tijd het schema van elke
|
||||
bestaande locatie (voor bestaande gebruikers verandert er niets).
|
||||
5. **Locaties na elkaar laten starten** in plaats van tegelijk: twee runs die dezelfde schijven
|
||||
lezen zijn samen niet sneller (gezien op nc01); wachtende runs zichtbaar als "queued".
|
||||
7. **SMB sneller** (gemeten 2026-10-21): lokaal netwerk zonder vertraging (Samba-container op de
|
||||
laptop, 1,8 GB) back-up 24,9 MB/s, restore 27,6 MiB/s; vanaf Canada naar de Storage Box (130 ms)
|
||||
2,6 MiB/s, omdat SMB per blok op het antwoord wacht. Grotere schrijfblokken, en vooruitlezen
|
||||
bij een restore zoals SFTP (IReadsAhead) of meerdere packs tegelijk. Een NAS met 1 Gbit haalt
|
||||
~110 MB/s: daar is winst.
|
||||
Ook de index: `target:add` over SMB vanaf Canada haalt ~1 indexbestand per seconde (4.927
|
||||
bestanden ~85 min), want SMB krijgt geen `getMany()` met verzoeken tegelijk zoals S3 en SFTP.
|
||||
6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot
|
||||
is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus),
|
||||
zodat een nieuwe server na een DR hem ook ziet.
|
||||
8. **Herstelpunt vastpinnen** (besluit 2026-10-25): een herstelpunt dat het bewaarbeleid
|
||||
nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met
|
||||
`occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de
|
||||
opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly.
|
||||
9. **Feedbackvraag weghalen** in 1.1.0 of 1.2.0 (besluit 2026-10-27): de regel "We would like
|
||||
to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot
|
||||
dan staan (de vermelding "in beta" is er in 1.0.1 al af). Zijn er tegen die
|
||||
tijd geen noemenswaardige meldingen, dan werkt het: mensen melden eerder wat niet werkt dan
|
||||
wat gewoon werkt. Het formulier (`experience.yml`) mag blijven voor wie het zelf opzoekt.
|
||||
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
|
||||
PostgreSQL 18)** — voor **1.0.0-rc.2**:
|
||||
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
|
||||
|
||||
@@ -7,8 +7,8 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
|
||||
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
||||
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
||||
|
||||
**Status: release candidate for 1.0.0 (1.0.0-rc.1, app version 0.9.7).** Tested thoroughly, usable
|
||||
alongside your current backup, but not yet a drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and
|
||||
**Status: 1.0.0, stable.** Disaster recovery has been tested on real installations (see "Tested
|
||||
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
|
||||
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
|
||||
`docs/DISASTER-RECOVERY.md`.
|
||||
|
||||
@@ -146,10 +146,31 @@ What you should do on the location (strongly recommended):
|
||||
- **Files and folders** of a user: in the web interface or `occ backup:restore`, into a new folder
|
||||
("Restored <date>"), merged into the original place, or replacing it.
|
||||
- **Raw restore** of any path (for example `config/`) to a local directory: `occ backup:restore ... --to-directory=DIR`.
|
||||
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see the how-to.
|
||||
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see
|
||||
[docs/DISASTER-RECOVERY.md](docs/DISASTER-RECOVERY.md). It runs in maintenance mode, so that
|
||||
nobody and no background job writes into the instance while it is being replaced.
|
||||
- **One user** (account, settings, files, calendars, contacts via user_migration): `occ backup:user:backup` and
|
||||
`occ backup:user:restore`, replacing the account or restoring it as `<uid>-bak`. See the how-to.
|
||||
|
||||
**Without shell access** (a hosted Nextcloud where you cannot run `occ`): files and folders can be
|
||||
restored in the web interface. Restoring a whole user or the whole server needs `occ`, so on such
|
||||
hosting it is something your provider (or whoever has shell access) runs. A restore mode that
|
||||
works without maintenance mode, and so from the web interface, is on the roadmap.
|
||||
|
||||
## Tested disaster recoveries
|
||||
|
||||
Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECOVERY.md):
|
||||
|
||||
| Backup | Location | Database | Result |
|
||||
|---|---|---|---|
|
||||
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
|
||||
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
|
||||
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
|
||||
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
|
||||
| 51 GB test instance | OVH S3 | PostgreSQL → PostgreSQL | 23,519/23,519 checksums identical |
|
||||
|
||||
The integration tests restore across every pair of PostgreSQL, MariaDB and SQLite.
|
||||
|
||||
## Storage backend experiences
|
||||
|
||||
Notes from actually running a backend, beyond what's in files_external's own documentation.
|
||||
@@ -206,15 +227,73 @@ Contributions welcome via PR — add what you ran into with your own provider.
|
||||
can be read on more connections: `occ config:app:set ng_backup restore_sftp_connections
|
||||
--value=4` (at most 8; the server must allow that many plus one at once for the user).
|
||||
|
||||
- **OVH Object Storage (S3, Warsaw, 2026-10):** works with the `amazons3` backend (path style,
|
||||
region `waw`). A full backup of 1 TB from Germany ran at about 7 MiB/s, from Canada about
|
||||
6 MiB/s with BBR; a check of a 1 TB backup took 3.5 minutes (index files are fetched 16 at a
|
||||
time); a restore of 138 GB from Warsaw to Canada took about 6 hours. Turn on versioning and
|
||||
Object Lock on the bucket (see above); OVH charges no egress.
|
||||
|
||||
- **Hetzner Storage Box over SFTP, full restore:** about 13-14 MiB/s from Germany to Canada
|
||||
(122 ms) with the read-ahead of 1.0.0-rc.4; without it about 2.8 MiB/s. A Storage Box gets
|
||||
slower with several connections reading at once (4-6 MiB/s for 2-4 connections, it reads from
|
||||
disks), so one extra connection is the default. A Storage Box drops an SFTP connection that
|
||||
stayed quiet for a while without telling the client; NG Backup checks and replaces it.
|
||||
|
||||
- **SMB speed:** on a local network (Samba, no delay) a backup ran at about 25 MB/s and a
|
||||
restore at about 28 MiB/s. Over a long line SMB waits for an answer to each request: from
|
||||
Canada to a Storage Box in Germany (130 ms) about 2.6 MiB/s for a backup and 1.6 MiB/s for a
|
||||
restore, and connecting a location with an existing backup reads its index at about one file a
|
||||
second. SMB is meant for a local network; use SFTP or S3 for a location far away. When
|
||||
`pecl install smbclient` fails (pecl.php.net was unreachable for hours on 2026-10-21), the
|
||||
extension builds from its source: `libsmbclient-php` on GitHub, `phpize && ./configure && make
|
||||
&& make install`, then `docker-php-ext-enable smbclient`.
|
||||
|
||||
- **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself
|
||||
(e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the
|
||||
mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping
|
||||
mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS
|
||||
can't be driven from PHP directly.
|
||||
|
||||
## Roadmap
|
||||
|
||||
What is planned, without dates. Ideas and votes are welcome in the
|
||||
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
|
||||
a 👍 on its issue helps decide what comes first.
|
||||
|
||||
**1.0.1**
|
||||
- A warning on the admin page and in `occ backup:status` when background jobs do not run on a
|
||||
system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on
|
||||
while the NG Backup page is open ([#13](https://github.com/dillardblom/ng_backup/issues/13)).
|
||||
- An optional label for a backup started with "Back up now".
|
||||
- An explanation in the trash of restore points of what "removed by retention policy" means and
|
||||
until when a restore point can be brought back.
|
||||
- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4.
|
||||
- A "Docs" tab on the admin page with links to the how-to, the disaster recovery guide and this
|
||||
roadmap.
|
||||
|
||||
**1.1.0**
|
||||
- Restore without the recovery kit: the backup key from the location itself, opened with one of
|
||||
your passphrases.
|
||||
- A schedule per location, also "every N hours".
|
||||
- Locations backed up one after the other instead of at the same time.
|
||||
- A note on a restore point that you can change afterwards.
|
||||
- Pinning a restore point, so the retention policy never removes it (for example the one from
|
||||
before an upgrade).
|
||||
- An option to pause heavy background jobs (such as Memories, Recognize or the Preview
|
||||
Generator) while a backup runs.
|
||||
- Faster backups: reading and uploading at the same time.
|
||||
|
||||
**Later**
|
||||
- A full restore onto the existing installation, without first installing a fresh Nextcloud, in a
|
||||
restore mode of its own instead of maintenance mode, so that it can also run from the web
|
||||
interface (for hosted Nextcloud without shell access).
|
||||
- Users restoring their own files from the Files app.
|
||||
- Primary object storage and External Storage mounts (see Known limitations).
|
||||
- An option not to keep the backup key on the location (the recovery kit is then the only copy).
|
||||
|
||||
## Known limitations
|
||||
|
||||
On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it:
|
||||
On the backlog; know them before you rely on NG Backup:
|
||||
|
||||
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
|
||||
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
|
||||
@@ -236,8 +315,11 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
|
||||
- Location passwords passed with `-o password=...` are visible in the process list and shell
|
||||
history while the command runs.
|
||||
- Temporary files of a restore into a user's files end up in that user's trash.
|
||||
- Restoring the database does not order tables by foreign keys; apps with cascading foreign keys
|
||||
could lose rows (Nextcloud core does not use them).
|
||||
- **Restoring a whole user or the whole server needs `occ`** (shell access); see "Without shell
|
||||
access" above.
|
||||
- **`--install-apps` depends on the app store.** Nextcloud downloads the whole app list (about
|
||||
38 MB) and gives up after 120 seconds; on a slow day apps are not installed. The restore then
|
||||
lists them: install them afterwards with `occ app:install`, their data is restored already.
|
||||
- A pack whose index upload failed stays on the location until removed by hand.
|
||||
|
||||
## Commands
|
||||
@@ -254,4 +336,4 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
|
||||
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
|
||||
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
|
||||
| `backup:key:import` | import a recovery kit on a fresh installation |
|
||||
| `backup:user:backup`, `backup:user:restore` | one user via user_migration (asks before installing it) |
|
||||
| `backup:user:backup`, `backup:user:list`, `backup:user:restore` | one user via user_migration (asks before installing it); list the exports |
|
||||
|
||||
+5
-4
@@ -42,7 +42,7 @@ welcome: use the "Share your experience" link on the admin page.
|
||||
|
||||
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
|
||||
you need help with restoring a backup after a crash.]]></description>
|
||||
<version>0.9.11</version>
|
||||
<version>1.0.0</version>
|
||||
<licence>agpl</licence>
|
||||
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
|
||||
<namespace>NgBackup</namespace>
|
||||
@@ -53,9 +53,9 @@ you need help with restoring a backup after a crash.]]></description>
|
||||
<website>https://github.com/dillardblom/ng_backup</website>
|
||||
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
|
||||
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/1-backups.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/2-restore.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/3-settings.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0/screenshots/1-backups.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0/screenshots/2-restore.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0/screenshots/3-settings.png</screenshot>
|
||||
<dependencies>
|
||||
<php min-version="8.1"/>
|
||||
<lib>sodium</lib>
|
||||
@@ -92,6 +92,7 @@ you need help with restoring a backup after a crash.]]></description>
|
||||
<command>OCA\NgBackup\Command\KeySlotRemove</command>
|
||||
<command>OCA\NgBackup\Command\BackupUser</command>
|
||||
<command>OCA\NgBackup\Command\RestoreUser</command>
|
||||
<command>OCA\NgBackup\Command\ListUserExports</command>
|
||||
<command>OCA\NgBackup\Command\KeyImport</command>
|
||||
<command>OCA\NgBackup\Command\RestoreFull</command>
|
||||
<command>OCA\NgBackup\Command\Verify</command>
|
||||
|
||||
@@ -10,6 +10,8 @@ Nextcloud back from an NG Backup location. Do them in this order.
|
||||
- [ ] A new server with the **same Nextcloud major version** as the old one (for example 34),
|
||||
a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for
|
||||
the whole backup plus a margin.
|
||||
- [ ] **Shell access** to the new server, to run `occ` (on a hosted Nextcloud without it, your
|
||||
provider runs these steps).
|
||||
- [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by
|
||||
hand). Keep their packages somewhere safe beforehand.
|
||||
|
||||
|
||||
@@ -114,10 +114,16 @@ Needs the `user_migration` app. NG Backup asks before installing it:
|
||||
|
||||
```
|
||||
occ backup:user:backup offsite <uid>
|
||||
occ backup:user:list offsite [<uid>] # the exports and their manifest
|
||||
occ backup:user:restore offsite users/<uid>/<exportId> # replace (default)
|
||||
occ backup:user:restore offsite users/<uid>/<exportId> --mode=as-backup
|
||||
```
|
||||
|
||||
- `backup:user:restore` needs the manifest path that `backup:user:backup` prints at the end
|
||||
(`users/<uid>/<exportId>`); `backup:user:list` shows it again. A user id alone also works when
|
||||
that user has exactly one export on the location. With more (a `replace` restore adds a safety
|
||||
export), the command lists them and you pass one.
|
||||
|
||||
- If `user_migration` is missing, the command asks to install and enable it. For non-interactive
|
||||
use, add `--install-user-migration` to approve that explicitly. The command stops after the
|
||||
installation; run it once more.
|
||||
|
||||
@@ -72,6 +72,10 @@ final class ExternalStorageFactory {
|
||||
throw new BackendException("Auth mechanism $authId does not fit backend $backendId");
|
||||
}
|
||||
|
||||
if (is_string($options['private_key'] ?? null)) {
|
||||
$options['private_key'] = self::normalizePrivateKey($options['private_key']);
|
||||
}
|
||||
|
||||
$config = new StorageConfig();
|
||||
$config->setBackend($backend);
|
||||
$config->setAuthMechanism($auth);
|
||||
@@ -86,4 +90,22 @@ final class ExternalStorageFactory {
|
||||
$storage = $backend->wrapStorage($storage);
|
||||
return $auth->wrapStorage($storage);
|
||||
}
|
||||
|
||||
/**
|
||||
* An OpenSSH-format private key with its line breaks restored. Up to Nextcloud 34
|
||||
* (phpseclib 2) a key whose newlines were lost or turned into spaces, e.g. on its way through
|
||||
* a shell or a form, still loaded; Nextcloud 35 (phpseclib 3) refuses it with "Unable to read
|
||||
* key" (#24). The key itself is not changed, so it is fixed when it is used, not when stored.
|
||||
* Other formats (PEM, PKCS#8) still load as they are and are returned unchanged.
|
||||
*/
|
||||
public static function normalizePrivateKey(string $key): string {
|
||||
if (!preg_match('/-----BEGIN OPENSSH PRIVATE KEY-----(.*?)-----END OPENSSH PRIVATE KEY-----/s', $key, $m)) {
|
||||
return $key;
|
||||
}
|
||||
$body = (string)preg_replace('/\s+/', '', $m[1]);
|
||||
if (!preg_match('#^[A-Za-z0-9+/]+=*$#', $body)) {
|
||||
return $key;
|
||||
}
|
||||
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,6 +32,7 @@ final class BackupJob extends TimedJob {
|
||||
private RunMapper $runs,
|
||||
private IAppConfig $appConfig,
|
||||
private LoggerInterface $logger,
|
||||
private \OCA\NgBackup\Service\Schedule $schedule,
|
||||
) {
|
||||
parent::__construct($time);
|
||||
// Well below the usual 5-minute cron interval: the last run is stamped a moment after cron
|
||||
@@ -101,14 +102,10 @@ final class BackupJob extends TimedJob {
|
||||
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
|
||||
}
|
||||
|
||||
/** Daily schedule: appconfig ng_backup/schedule_time "HH:MM" (server time); empty = no schedule. */
|
||||
/** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */
|
||||
private function startScheduled(): void {
|
||||
$at = $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', '');
|
||||
if (!preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $at)) {
|
||||
return;
|
||||
}
|
||||
$todayAt = (int)strtotime(date('Y-m-d') . ' ' . $at);
|
||||
if (time() < $todayAt) {
|
||||
$todayAt = $this->schedule->todayAt();
|
||||
if ($todayAt === null || time() < $todayAt) {
|
||||
return;
|
||||
}
|
||||
foreach ($this->targets->list() as $t) {
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
namespace OCA\NgBackup\Command;
|
||||
|
||||
use OCA\NgBackup\Service\TargetService;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
use Symfony\Component\Console\Helper\Table;
|
||||
use Symfony\Component\Console\Input\InputArgument;
|
||||
use Symfony\Component\Console\Input\InputInterface;
|
||||
use Symfony\Component\Console\Output\OutputInterface;
|
||||
|
||||
final class ListUserExports extends Command {
|
||||
public function __construct(
|
||||
private TargetService $targets,
|
||||
) {
|
||||
parent::__construct();
|
||||
}
|
||||
|
||||
protected function configure(): void {
|
||||
$this->setName('backup:user:list')
|
||||
->setDescription('List the user exports (backup:user:backup) on a location')
|
||||
->addArgument('target', InputArgument::OPTIONAL, 'Location name or id (default: all)')
|
||||
->addArgument('user', InputArgument::OPTIONAL, 'Only the exports of this user id')
|
||||
->setHelp(<<<'HELP'
|
||||
The manifest column is what occ backup:user:restore expects. A safety export made by a restore
|
||||
with --mode=replace is listed here as well.
|
||||
|
||||
Example:
|
||||
occ backup:user:list offsite alice
|
||||
HELP);
|
||||
}
|
||||
|
||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||
$name = $input->getArgument('target');
|
||||
$uid = $input->getArgument('user');
|
||||
$rows = [];
|
||||
try {
|
||||
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
|
||||
foreach ($targets as $t) {
|
||||
foreach ($this->targets->repository($t)->listUserExports($uid) as $path) {
|
||||
$parts = explode('/', $path, 3);
|
||||
$user = $parts[1] ?? '';
|
||||
$exportId = $parts[2] ?? '';
|
||||
$time = \DateTimeImmutable::createFromFormat('Ymd\THis\Z', substr($exportId, 0, 16), new \DateTimeZone('UTC'));
|
||||
$rows[] = [$t->getName(), $user, $time !== false ? $time->format('Y-m-d H:i') : '', $path];
|
||||
}
|
||||
}
|
||||
} catch (\Throwable $e) {
|
||||
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||
return 1;
|
||||
}
|
||||
if ($rows === []) {
|
||||
$output->writeln('No user exports yet (occ backup:user:backup).');
|
||||
return 0;
|
||||
}
|
||||
(new Table($output))->setHeaders(['location', 'user', 'date (UTC)', 'manifest'])->setRows($rows)->render();
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
@@ -33,7 +33,7 @@ final class RestoreUser extends Command {
|
||||
$this->setName('backup:user:restore')
|
||||
->setDescription('Restore a user from a user_migration export (see backup:user:backup)')
|
||||
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
||||
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>)')
|
||||
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>), or a user id with exactly one export')
|
||||
->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'replace or as-backup', UserRestoreService::MODE_REPLACE)
|
||||
->addOption('force', null, InputOption::VALUE_NONE, 'Replace an existing account without asking (required when not interactive)')
|
||||
->addOption('install-user-migration', null, InputOption::VALUE_NONE, 'Install and enable user_migration if it is missing (non-interactive consent)')
|
||||
@@ -45,16 +45,31 @@ Never imports over a live account. Modes:
|
||||
compare and transfer by hand; Nextcloud cannot rename a user id, so the backup gets
|
||||
the new name, never the existing account
|
||||
|
||||
occ backup:user:list shows the exports on a location. A user id alone works when that user
|
||||
has exactly one export there; with more, the command lists them and you pass one.
|
||||
|
||||
Example:
|
||||
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34
|
||||
occ backup:user:restore offsite alice
|
||||
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34 --mode=as-backup
|
||||
HELP);
|
||||
}
|
||||
|
||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||
$mode = (string)$input->getOption('mode');
|
||||
$uid = explode('/', (string)$input->getArgument('manifest'))[1] ?? '';
|
||||
if ($mode === UserRestoreService::MODE_REPLACE && $uid !== '' && $this->userManager->userExists($uid) && !$input->getOption('force')) {
|
||||
try {
|
||||
$target = $this->targets->get($input->getArgument('target'));
|
||||
$ref = (string)$input->getArgument('manifest');
|
||||
$manifest = $this->targets->repository($target)->resolveUserExport($ref);
|
||||
} catch (\Throwable $e) {
|
||||
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||
return 1;
|
||||
}
|
||||
if ($manifest !== $ref) {
|
||||
$output->writeln("Using $manifest.");
|
||||
}
|
||||
$uid = explode('/', $manifest)[1];
|
||||
if ($mode === UserRestoreService::MODE_REPLACE && $this->userManager->userExists($uid) && !$input->getOption('force')) {
|
||||
// Replace deletes the live account (after a safety export). Shares to others and data of
|
||||
// apps without a user_migration migrator do not come back, so never do it unasked.
|
||||
if (!$input->isInteractive()) {
|
||||
@@ -69,9 +84,8 @@ HELP);
|
||||
}
|
||||
try {
|
||||
$output->writeln('<comment>Note: on restore, user_migration only imports the app settings on its own allowlist (e.g. calendar view and reminder settings); other app settings of this user are skipped. Reason: NG Backup uses user_migration\'s importer; widening the list would mean building a settings importer of our own.</comment>');
|
||||
$target = $this->targets->get($input->getArgument('target'));
|
||||
$this->installer->ensureEnabled($input, $output);
|
||||
$user = $this->userRestore->restoreUser($target, (string)$input->getArgument('manifest'), $mode);
|
||||
$user = $this->userRestore->restoreUser($target, $manifest, $mode);
|
||||
} catch (\Throwable $e) {
|
||||
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||
return 1;
|
||||
|
||||
@@ -47,6 +47,7 @@ final class AdminApiController extends Controller {
|
||||
private \OCA\NgBackup\Service\AlertService $alerts,
|
||||
private \OCA\NgBackup\Service\KeySlotService $slots,
|
||||
private \OCA\NgBackup\Service\PruneService $prune,
|
||||
private \OCA\NgBackup\Service\Schedule $schedule,
|
||||
) {
|
||||
parent::__construct(Application::APP_ID, $request);
|
||||
}
|
||||
@@ -72,7 +73,8 @@ final class AdminApiController extends Controller {
|
||||
'runs' => $runs,
|
||||
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
|
||||
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
|
||||
'schedule' => $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''),
|
||||
'schedule' => $this->schedule->get()['time'],
|
||||
'scheduleTimezone' => $this->schedule->get()['timezone'],
|
||||
]);
|
||||
}
|
||||
|
||||
@@ -240,12 +242,14 @@ final class AdminApiController extends Controller {
|
||||
return new JSONResponse($this->runJson($run));
|
||||
}
|
||||
|
||||
public function setSchedule(string $time): JSONResponse {
|
||||
if ($time !== '' && !preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time)) {
|
||||
return new JSONResponse(['error' => 'Use HH:MM or empty'], Http::STATUS_BAD_REQUEST);
|
||||
/** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */
|
||||
public function setSchedule(string $time, string $timezone = ''): JSONResponse {
|
||||
try {
|
||||
$this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
|
||||
} catch (\InvalidArgumentException $e) {
|
||||
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||
}
|
||||
$this->appConfig->setValueString(Application::APP_ID, 'schedule_time', $time);
|
||||
return new JSONResponse(['schedule' => $time]);
|
||||
return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]);
|
||||
}
|
||||
|
||||
public function snapshots(int $targetId): JSONResponse {
|
||||
|
||||
+26
-3
@@ -159,7 +159,7 @@ final class DbDumper {
|
||||
// everywhere); tables it does not know are left out, as asked.
|
||||
foreach ($missing as $table) {
|
||||
if (isset($manifest['tables'][$table]['schema'])) {
|
||||
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema']));
|
||||
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
|
||||
$report['created'][] = $table;
|
||||
} else {
|
||||
$report['missing'][] = $table;
|
||||
@@ -319,8 +319,31 @@ final class DbDumper {
|
||||
'foreignKeys' => $foreignKeys];
|
||||
}
|
||||
|
||||
private static function tableFromStructure(string $name, array $structure): \Doctrine\DBAL\Schema\Table {
|
||||
$table = new \Doctrine\DBAL\Schema\Table($name);
|
||||
/**
|
||||
* Table options for a table the restore creates: those of this installation's own tables.
|
||||
* On MySQL/MariaDB that is the character set and collation, as Nextcloud's tables have them
|
||||
* (utf8mb4, so emoji and other 4-byte characters fit); DBAL alone would use 3-byte utf8.
|
||||
*
|
||||
* @return array<string, mixed>
|
||||
* @psalm-suppress UndefinedClass the DBAL classes come with Nextcloud, not with this app
|
||||
*/
|
||||
private function tableOptions(): array {
|
||||
$options = $this->db->createSchemaManager()->createSchemaConfig()->getDefaultTableOptions();
|
||||
if ($this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\AbstractMySQLPlatform) {
|
||||
$collation = $this->db->fetchOne('SELECT TABLE_COLLATION FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ?',
|
||||
[$this->prefix . 'appconfig']);
|
||||
if (is_string($collation) && $collation !== '') {
|
||||
$options['collation'] = $collation;
|
||||
$options['charset'] = explode('_', $collation, 2)[0];
|
||||
unset($options['collate']);
|
||||
}
|
||||
}
|
||||
return $options;
|
||||
}
|
||||
|
||||
/** @param array<string, mixed> $options table options (see tableOptions()) */
|
||||
private static function tableFromStructure(string $name, array $structure, array $options = []): \Doctrine\DBAL\Schema\Table {
|
||||
$table = new \Doctrine\DBAL\Schema\Table($name, [], [], [], [], $options);
|
||||
foreach ($structure['columns'] as $c) {
|
||||
$table->addColumn($c['name'], $c['type'], array_filter([
|
||||
'length' => $c['length'], 'precision' => $c['precision'], 'scale' => $c['scale'],
|
||||
|
||||
@@ -561,6 +561,31 @@ final class Repository {
|
||||
return isset($this->catalog->userExports[$manifestPath]);
|
||||
}
|
||||
|
||||
/**
|
||||
* The recorded export $ref names: a manifest path (users/<uid>/<exportId>) as is, or a bare
|
||||
* user id (also "users/<uid>") when that user has exactly one export here. Anything else
|
||||
* throws, with the exports to choose from in the message.
|
||||
*/
|
||||
public function resolveUserExport(string $ref): string {
|
||||
if ($this->isRecordedUserExport($ref)) {
|
||||
return $ref;
|
||||
}
|
||||
$rest = trim($ref, '/');
|
||||
if (str_starts_with($rest, 'users/')) {
|
||||
$rest = substr($rest, strlen('users/'));
|
||||
}
|
||||
$uid = explode('/', $rest)[0];
|
||||
$exports = $uid !== '' ? $this->listUserExports($uid) : [];
|
||||
if (!str_contains($rest, '/') && count($exports) === 1) {
|
||||
return $exports[0];
|
||||
}
|
||||
if ($exports === []) {
|
||||
throw new RepositoryException("$ref is not a recorded user export for this location, and $uid has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports");
|
||||
}
|
||||
$which = str_contains($rest, '/') ? "$ref is not a recorded user export for this location" : "$uid has " . count($exports) . ' exports here';
|
||||
throw new RepositoryException("$which; pass one of these (newest first):\n " . implode("\n ", $exports));
|
||||
}
|
||||
|
||||
/** Permanently delete a user export. Its blobs are freed by the next prune() like any other. */
|
||||
public function forgetUserExport(string $manifestPath): void {
|
||||
if (!isset($this->catalog->userExports[$manifestPath])) {
|
||||
@@ -845,7 +870,15 @@ final class Repository {
|
||||
try {
|
||||
foreach ($this->backend->getRanges($ranges) as $i => $data) {
|
||||
foreach ($parts[$i] as [$id, $at, $length]) {
|
||||
yield $this->decodeBlob($id, substr($data, $at, $length));
|
||||
try {
|
||||
$blob = $this->decodeBlob($id, substr($data, $at, $length));
|
||||
} catch (\Exception) {
|
||||
// a read that went wrong on the way (see loadBlob()): this blob on its own
|
||||
$blob = $this->loadBlob($id);
|
||||
// its pack stream is closed before the next range is read on the same connection
|
||||
$this->packReader->close();
|
||||
}
|
||||
yield $blob;
|
||||
$done++;
|
||||
}
|
||||
}
|
||||
@@ -895,8 +928,46 @@ final class Repository {
|
||||
} catch (BackendException) {
|
||||
throw $e;
|
||||
}
|
||||
$pack = $moved;
|
||||
}
|
||||
return $this->decodeBlob($id, $encrypted);
|
||||
try {
|
||||
return $this->decodeBlob($id, $encrypted);
|
||||
} catch (\Exception $damaged) {
|
||||
// Bytes that do not decrypt: damaged on the location, or a read that went wrong on the
|
||||
// way (a stream that lost its place: an SMB restore stopped after 17 hours like this).
|
||||
// Read exactly this blob once more, on a request of its own; only a second failure
|
||||
// means the data itself is damaged.
|
||||
$this->packReader->close();
|
||||
try {
|
||||
$again = $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
|
||||
} catch (\Exception) {
|
||||
throw $damaged;
|
||||
}
|
||||
$data = $this->decodeBlob($id, $again);
|
||||
if (function_exists('OCP\Log\logger')) {
|
||||
\OCP\Log\logger('ng_backup')->warning('NG Backup: blob ' . $id . ' did not decrypt as first read from pack ' . $pack . ' at ' . $offset . '; read again, it is fine. First read: ' . self::howItDiffered($encrypted, $again));
|
||||
}
|
||||
return $data;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* What a read that went wrong looked like next to the right bytes, for the log: where it starts
|
||||
* to differ, and whether the rest is zeros or the right bytes from another place (a stream that
|
||||
* lost its place).
|
||||
*/
|
||||
private static function howItDiffered(string $wrong, string $right): string {
|
||||
$at = strspn($wrong ^ $right, "\0");
|
||||
$text = strlen($wrong) . ' bytes, differ from byte ' . $at . ' of ' . strlen($right);
|
||||
$tail = substr($wrong, $at, 64);
|
||||
if ($tail === '') {
|
||||
return $text . ' (too short)';
|
||||
}
|
||||
if (trim($tail, "\0") === '') {
|
||||
return $text . ' (zeros)';
|
||||
}
|
||||
$found = strpos($right, $tail);
|
||||
return $text . ($found !== false ? ' (the right bytes from ' . ($found - $at) . ' bytes further on)' : ' (other bytes)');
|
||||
}
|
||||
|
||||
/** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
namespace OCA\NgBackup\Service;
|
||||
|
||||
use OCA\NgBackup\AppInfo\Application;
|
||||
use OCP\IAppConfig;
|
||||
use OCP\IConfig;
|
||||
use OCP\IGroupManager;
|
||||
|
||||
/**
|
||||
* The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a
|
||||
* zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone
|
||||
* is converted once to the zone of the server or of an administrator, at today's offset: it then
|
||||
* shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
|
||||
*/
|
||||
final class Schedule {
|
||||
public const TIME_KEY = 'schedule_time';
|
||||
public const ZONE_KEY = 'schedule_timezone';
|
||||
|
||||
public function __construct(
|
||||
private IAppConfig $appConfig,
|
||||
private IConfig $config,
|
||||
private IGroupManager $groups,
|
||||
) {
|
||||
}
|
||||
|
||||
/** @return array{time: string, timezone: non-empty-string} time "" = no schedule */
|
||||
public function get(): array {
|
||||
$time = $this->appConfig->getValueString(Application::APP_ID, self::TIME_KEY, '');
|
||||
$zone = $this->appConfig->getValueString(Application::APP_ID, self::ZONE_KEY, '');
|
||||
if ($zone === '' || !self::validZone($zone)) {
|
||||
$zone = $this->guessZone();
|
||||
if ($time !== '' && self::validTime($time)) {
|
||||
$time = self::convert($time, 'UTC', $zone);
|
||||
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
|
||||
}
|
||||
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
|
||||
}
|
||||
return ['time' => $time, 'timezone' => $zone];
|
||||
}
|
||||
|
||||
/** @throws \InvalidArgumentException for a malformed time or an unknown zone */
|
||||
public function set(string $time, string $zone): void {
|
||||
if ($time !== '' && !self::validTime($time)) {
|
||||
throw new \InvalidArgumentException('Use HH:MM or empty');
|
||||
}
|
||||
if (!self::validZone($zone)) {
|
||||
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
|
||||
}
|
||||
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
|
||||
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
|
||||
}
|
||||
|
||||
/** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */
|
||||
public function todayAt(?int $now = null): ?int {
|
||||
['time' => $time, 'timezone' => $zone] = $this->get();
|
||||
if ($time === '' || !self::validTime($time)) {
|
||||
return null;
|
||||
}
|
||||
$tz = new \DateTimeZone($zone);
|
||||
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d');
|
||||
return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp();
|
||||
}
|
||||
|
||||
/**
|
||||
* "HH:MM" in zone $from as "HH:MM" in zone $to, at today's offsets.
|
||||
*
|
||||
* @param non-empty-string $from
|
||||
* @param non-empty-string $to
|
||||
*/
|
||||
public static function convert(string $time, string $from, string $to, ?int $now = null): string {
|
||||
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone(new \DateTimeZone($from))->format('Y-m-d');
|
||||
return (new \DateTimeImmutable($day . ' ' . $time, new \DateTimeZone($from)))->setTimezone(new \DateTimeZone($to))->format('H:i');
|
||||
}
|
||||
|
||||
public static function validTime(string $time): bool {
|
||||
return preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time) === 1;
|
||||
}
|
||||
|
||||
/** @psalm-assert-if-true non-empty-string $zone */
|
||||
public static function validZone(string $zone): bool {
|
||||
return in_array($zone, \DateTimeZone::listIdentifiers(), true);
|
||||
}
|
||||
|
||||
/**
|
||||
* The server's default_timezone when set, else the first administrator's, else UTC. A server
|
||||
* zone of UTC is passed over on purpose: it is mostly what an image or installer left there,
|
||||
* while the administrator who set the time read it as their own (#19).
|
||||
*
|
||||
* @return non-empty-string
|
||||
*/
|
||||
private function guessZone(): string {
|
||||
$zone = $this->config->getSystemValueString('default_timezone', '');
|
||||
if ($zone !== '' && $zone !== 'UTC' && self::validZone($zone)) {
|
||||
return $zone;
|
||||
}
|
||||
foreach ($this->groups->get('admin')?->getUsers() ?? [] as $user) {
|
||||
$zone = $this->config->getUserValue($user->getUID(), 'core', 'timezone', '');
|
||||
if ($zone !== '' && self::validZone($zone)) {
|
||||
return $zone;
|
||||
}
|
||||
}
|
||||
return 'UTC';
|
||||
}
|
||||
}
|
||||
+44
-3
@@ -7,7 +7,7 @@
|
||||
<NcSettingsSection :name="t('ng_backup', 'NG Backup')"
|
||||
:description="t('ng_backup', 'Encrypted, incremental backups of this Nextcloud (database, configuration and data). Backup locations are never mounted and are not visible in Files.')">
|
||||
<p class="ngb-feedback">
|
||||
{{ t('ng_backup', 'NG Backup is in beta. We would like to hear how it works for you, also when everything went fine.') }}
|
||||
{{ t('ng_backup', 'We would like to hear how NG Backup works for you, also when everything went fine.') }}
|
||||
<a :href="feedbackUrl" target="_blank" rel="noopener noreferrer">{{ t('ng_backup', 'Share your experience ↗') }}</a>
|
||||
</p>
|
||||
<NcLoadingIcon v-if="!status" :size="32" />
|
||||
@@ -18,9 +18,18 @@
|
||||
<div class="ngb-tabs">
|
||||
<NcButton :variant="tab === 'backup' ? 'primary' : 'secondary'" @click="tab = 'backup'">{{ t('ng_backup', 'Backup and restore') }}</NcButton>
|
||||
<NcButton :variant="tab === 'settings' ? 'primary' : 'secondary'" @click="tab = 'settings'">{{ t('ng_backup', 'Settings') }}</NcButton>
|
||||
<NcButton :variant="tab === 'docs' ? 'primary' : 'secondary'" @click="tab = 'docs'">{{ t('ng_backup', 'Docs') }}</NcButton>
|
||||
</div>
|
||||
|
||||
<template v-if="tab === 'settings'">
|
||||
<NcSettingsSection v-if="tab === 'docs'" :name="t('ng_backup', 'Documentation')">
|
||||
<ul class="ngb-docs">
|
||||
<li v-for="doc in docs" :key="doc.url">
|
||||
<a :href="doc.url" target="_blank" rel="noopener noreferrer">{{ doc.name }} ↗</a>
|
||||
<span>{{ doc.description }}</span>
|
||||
</li>
|
||||
</ul>
|
||||
</NcSettingsSection>
|
||||
<template v-else-if="tab === 'settings'">
|
||||
<KeySetup :status="status" @changed="refresh" />
|
||||
<Locations v-if="status.key.everConfirmed" :status="status" @changed="refresh" />
|
||||
</template>
|
||||
@@ -52,6 +61,25 @@ import RestoreBrowser from './components/RestoreBrowser.vue'
|
||||
// Opens the experience report form in .github/ISSUE_TEMPLATE
|
||||
const feedbackUrl = 'https://github.com/dillardblom/ng_backup/issues/new?template=experience.yml'
|
||||
|
||||
const repoUrl = 'https://github.com/dillardblom/ng_backup'
|
||||
const docs = [
|
||||
{
|
||||
name: t('ng_backup', 'How-to'),
|
||||
description: t('ng_backup', 'Step by step: from first install to a tested restore.'),
|
||||
url: repoUrl + '/blob/main/docs/HOWTO.md',
|
||||
},
|
||||
{
|
||||
name: t('ng_backup', 'Disaster recovery'),
|
||||
description: t('ng_backup', 'Your server is gone: get Nextcloud back on a new server from a backup location.'),
|
||||
url: repoUrl + '/blob/main/docs/DISASTER-RECOVERY.md',
|
||||
},
|
||||
{
|
||||
name: t('ng_backup', 'Roadmap'),
|
||||
description: t('ng_backup', 'What is planned for the next versions.'),
|
||||
url: repoUrl + '#roadmap',
|
||||
},
|
||||
]
|
||||
|
||||
const status = ref(null)
|
||||
const allRuns = ref(false) // runs older than the last days too
|
||||
const error = ref('')
|
||||
@@ -89,8 +117,10 @@ onBeforeUnmount(() => clearTimeout(timer))
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
/* NcSettingsSection sets its own width (at most 900px): the runs table needs more */
|
||||
.ng-backup :deep(.settings-section) {
|
||||
max-width: 900px;
|
||||
width: min(1400px, 100% - var(--default-grid-baseline) * 7 * 2);
|
||||
max-width: none;
|
||||
}
|
||||
.ngb-feedback {
|
||||
color: var(--color-text-maxcontrast);
|
||||
@@ -100,6 +130,17 @@ onBeforeUnmount(() => clearTimeout(timer))
|
||||
color: var(--color-primary-element);
|
||||
text-decoration: underline;
|
||||
}
|
||||
.ngb-docs li {
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
.ngb-docs a {
|
||||
color: var(--color-primary-element);
|
||||
text-decoration: underline;
|
||||
margin-right: 8px;
|
||||
}
|
||||
.ngb-docs span {
|
||||
color: var(--color-text-maxcontrast);
|
||||
}
|
||||
.ngb-tabs {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
|
||||
+1
-1
@@ -33,7 +33,7 @@ export default {
|
||||
testTarget: (id) => call('post', `/targets/${id}/test`),
|
||||
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
|
||||
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
|
||||
setSchedule: (time) => call('put', '/schedule', { time }),
|
||||
setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }),
|
||||
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
|
||||
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
|
||||
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
|
||||
|
||||
@@ -6,8 +6,8 @@
|
||||
<NcSettingsSection :name="t('ng_backup', 'Backups')"
|
||||
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
|
||||
<div class="ngb-row">
|
||||
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, empty = off)')" />
|
||||
<NcButton :disabled="schedule === status.schedule" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
|
||||
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, {zone}, empty = off)', { zone: browserZone })" />
|
||||
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
|
||||
</div>
|
||||
<div class="ngb-row">
|
||||
<NcButton v-for="target in status.targets" :key="target.id" :disabled="isRunning(target.id)" @click="start(target)">
|
||||
@@ -79,15 +79,19 @@ import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
||||
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
||||
import api from '../api.js'
|
||||
import { formatDateTime, formatDate, formatSize } from '../format.js'
|
||||
import { formatDateTime, formatDate, formatSize, toZone } from '../format.js'
|
||||
import { confirmPassword } from '@nextcloud/password-confirmation'
|
||||
import { reactive } from 'vue'
|
||||
|
||||
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
|
||||
const emit = defineEmits(['changed', 'show-all-runs'])
|
||||
const schedule = ref(props.status.schedule)
|
||||
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
|
||||
// zone is shown in this one, so it reads as the moment it runs here.
|
||||
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
|
||||
const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s
|
||||
const schedule = ref(shown(props.status.schedule))
|
||||
const error = ref('')
|
||||
watch(() => props.status.schedule, (s) => { schedule.value = s })
|
||||
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
|
||||
|
||||
const trash = reactive({})
|
||||
async function loadTrash() {
|
||||
@@ -131,7 +135,7 @@ async function start(target) {
|
||||
async function saveSchedule() {
|
||||
error.value = ''
|
||||
try {
|
||||
await api.setSchedule(schedule.value.trim())
|
||||
await api.setSchedule(schedule.value.trim(), browserZone)
|
||||
emit('changed')
|
||||
} catch (e) {
|
||||
error.value = e.message
|
||||
|
||||
@@ -26,3 +26,30 @@ export function formatSize(bytes) {
|
||||
}
|
||||
return i === 0 ? `${b} B` : `${b.toFixed(1)} ${units[i]}`
|
||||
}
|
||||
|
||||
/**
|
||||
* A time of day "HH:MM" in zone `from`, as "HH:MM" in zone `to`, at today's offsets.
|
||||
*
|
||||
* @param {string} hhmm the time
|
||||
* @param {string} from IANA zone it is in
|
||||
* @param {string} to IANA zone to show it in
|
||||
* @param {Date} now the day to convert on (today)
|
||||
* @return {string}
|
||||
*/
|
||||
export function toZone(hhmm, from, to, now = new Date()) {
|
||||
const [h, m] = hhmm.split(':').map(Number)
|
||||
const parts = (zone, date) => Object.fromEntries(new Intl.DateTimeFormat('en-GB', {
|
||||
timeZone: zone, hourCycle: 'h23', year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit',
|
||||
}).formatToParts(date).map((p) => [p.type, p.value]))
|
||||
const offsetAt = (at) => {
|
||||
const seen = parts(from, new Date(at))
|
||||
return Date.UTC(+seen.year, +seen.month - 1, +seen.day, +seen.hour, +seen.minute) - at
|
||||
}
|
||||
const today = parts(from, now)
|
||||
const wall = Date.UTC(+today.year, +today.month - 1, +today.day, h, m)
|
||||
// the offset at the wall time itself: on a day the clocks change it differs from the one at
|
||||
// midnight, so it is looked up a second time at the moment found first
|
||||
const at = wall - offsetAt(wall - offsetAt(wall))
|
||||
const there = parts(to, new Date(at))
|
||||
return `${there.hour}:${there.minute}`
|
||||
}
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Tests\Unit\Backend;
|
||||
|
||||
use OCA\NgBackup\Backend\ExternalStorageFactory;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
|
||||
/** ExternalStorageFactory::normalizePrivateKey(): keys that phpseclib 2 read, for phpseclib 3 (#24). */
|
||||
class ExternalStorageFactoryTest extends TestCase {
|
||||
/** An OpenSSH-format key laid out as ssh-keygen writes it; the body is random, not a real key. */
|
||||
private static function key(): string {
|
||||
$body = base64_encode(random_bytes(1000));
|
||||
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
|
||||
}
|
||||
|
||||
public function testRestoresLostLineBreaks(): void {
|
||||
$key = self::key();
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey($key));
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", ' ', $key)), 'newlines as spaces');
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", '', $key)), 'no newlines');
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", "\r\n", $key)), 'CRLF');
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(rtrim($key)), 'no trailing newline');
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(preg_replace('/^/m', ' ', $key)), 'indented');
|
||||
}
|
||||
|
||||
public function testLeavesOtherFormatsAlone(): void {
|
||||
$pem = "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC,FF44E0A94036766C\n\nAAAA\n-----END RSA PRIVATE KEY-----\n";
|
||||
$this->assertSame($pem, ExternalStorageFactory::normalizePrivateKey($pem));
|
||||
$this->assertSame('not a key', ExternalStorageFactory::normalizePrivateKey('not a key'));
|
||||
$odd = "-----BEGIN OPENSSH PRIVATE KEY-----\\nAAAA\\n-----END OPENSSH PRIVATE KEY-----";
|
||||
$this->assertSame($odd, ExternalStorageFactory::normalizePrivateKey($odd), 'not base64 inside: unchanged');
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||
|
||||
use OCA\NgBackup\Backend\IBackend;
|
||||
use OCA\NgBackup\Backend\LocalBackend;
|
||||
use OCA\NgBackup\Crypto\KeyRing;
|
||||
use OCA\NgBackup\Repository\Repository;
|
||||
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
|
||||
/**
|
||||
* A pack stream that hands out wrong bytes once (an SMB restore stopped after 17 hours with
|
||||
* "Decryption failed"): the blob is read once more on a request of its own; only data that is
|
||||
* damaged on the location stops the restore.
|
||||
*/
|
||||
class DamagedReadTest extends TestCase {
|
||||
use TempDirTrait;
|
||||
|
||||
private string $root;
|
||||
private KeyRing $keys;
|
||||
private string $src;
|
||||
private string $snapshot;
|
||||
|
||||
protected function setUp(): void {
|
||||
$this->root = $this->tempDir();
|
||||
$this->keys = KeyRing::generate();
|
||||
$this->src = $this->tempDir();
|
||||
for ($i = 0; $i < 6; $i++) {
|
||||
file_put_contents("$this->src/f$i", random_bytes(3 * 1048576));
|
||||
}
|
||||
$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
|
||||
$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
|
||||
}
|
||||
|
||||
public function testABlobThatCameInDamagedOnceIsReadAgain(): void {
|
||||
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1);
|
||||
$target = $this->tempDir();
|
||||
|
||||
$this->assertSame(6, Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $target));
|
||||
|
||||
$this->assertSame(self::hashTree($this->src), self::hashTree($target));
|
||||
$this->assertSame(0, $backend->streams, 'the damaged read happened');
|
||||
}
|
||||
|
||||
public function testATableDumpThatCameInDamagedOnceIsReadAgain(): void {
|
||||
$repo = Repository::openWithKey(new LocalBackend($this->root), $this->keys);
|
||||
$stats = [];
|
||||
$writer = $repo->blobWriter($stats);
|
||||
$lines = [];
|
||||
for ($i = 0; $i < 20000; $i++) {
|
||||
$lines[] = json_encode(['i' => $i, 'pad' => str_repeat('x', $i % 400)]);
|
||||
}
|
||||
$writer->write(implode("\n", $lines) . "\n");
|
||||
$blobs = $writer->finish();
|
||||
$repo->flushPacks();
|
||||
$backend = new GarblingBackend(new LocalBackend($this->root), ranges: 1);
|
||||
|
||||
$this->assertSame($lines, iterator_to_array(Repository::openWithKey($backend, $this->keys)->readLines($blobs), false));
|
||||
$this->assertSame(0, $backend->ranges, 'the damaged read happened');
|
||||
}
|
||||
|
||||
public function testDataDamagedOnTheLocationStillStopsTheRestore(): void {
|
||||
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1000, ranges: 1000);
|
||||
|
||||
$this->expectException(\Exception::class);
|
||||
Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $this->tempDir());
|
||||
}
|
||||
}
|
||||
|
||||
/** Pack reads that flip a byte: the first $streams pack streams and $ranges range reads. */
|
||||
final class GarblingBackend implements IBackend {
|
||||
use \OCA\NgBackup\Backend\GetsOneByOne;
|
||||
|
||||
public function __construct(
|
||||
private IBackend $inner,
|
||||
public int $streams = 0,
|
||||
public int $ranges = 0,
|
||||
) {
|
||||
}
|
||||
|
||||
public function put(string $path, $stream): void {
|
||||
$this->inner->put($path, $stream);
|
||||
}
|
||||
|
||||
public function get(string $path) {
|
||||
$fh = $this->inner->get($path);
|
||||
if (!str_starts_with($path, 'packs/') || $this->streams <= 0) {
|
||||
return $fh;
|
||||
}
|
||||
$this->streams--;
|
||||
$data = stream_get_contents($fh);
|
||||
fclose($fh);
|
||||
for ($k = 100; $k < strlen($data); $k += 65536) {
|
||||
$data[$k] = chr(ord($data[$k]) ^ 0x55);
|
||||
}
|
||||
$out = fopen('php://temp', 'w+b');
|
||||
fwrite($out, $data);
|
||||
rewind($out);
|
||||
return $out;
|
||||
}
|
||||
|
||||
public function getRange(string $path, int $offset, int $length): string {
|
||||
$data = $this->inner->getRange($path, $offset, $length);
|
||||
if (str_starts_with($path, 'packs/') && $this->ranges > 0) {
|
||||
$this->ranges--;
|
||||
$data[0] = chr(ord($data[0]) ^ 0x55);
|
||||
}
|
||||
return $data;
|
||||
}
|
||||
|
||||
public function exists(string $path): bool {
|
||||
return $this->inner->exists($path);
|
||||
}
|
||||
|
||||
public function move(string $from, string $to): void {
|
||||
$this->inner->move($from, $to);
|
||||
}
|
||||
|
||||
public function list(string $prefix): array {
|
||||
return $this->inner->list($prefix);
|
||||
}
|
||||
|
||||
public function listFiles(string $dir): array {
|
||||
return $this->inner->listFiles($dir);
|
||||
}
|
||||
|
||||
public function delete(string $path): void {
|
||||
$this->inner->delete($path);
|
||||
}
|
||||
}
|
||||
@@ -106,4 +106,51 @@ class UserExportTest extends TestCase {
|
||||
$this->expectExceptionMessage('missing on the location');
|
||||
$repo->listUserExports();
|
||||
}
|
||||
|
||||
public function testResolveTakesARecordedPathAsIs(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$path = 'users/alice/20261005T120000Z-aaaa0001';
|
||||
$this->putManifest($repo, $path);
|
||||
|
||||
$this->assertSame($path, $repo->resolveUserExport($path));
|
||||
}
|
||||
|
||||
public function testResolveTakesAUserIdWithOneExport(): void {
|
||||
// Issue #22: "backup:user:restore <location> New_User" was rejected as not a recorded export.
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$path = 'users/alice/20261005T120000Z-aaaa0001';
|
||||
$this->putManifest($repo, $path);
|
||||
$this->putManifest($repo, 'users/alice2/20261005T120000Z-aaaa0002');
|
||||
|
||||
$this->assertSame($path, $repo->resolveUserExport('alice'));
|
||||
$this->assertSame($path, $repo->resolveUserExport('users/alice/'));
|
||||
}
|
||||
|
||||
public function testResolveListsTheExportsOfAUserWithMore(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
|
||||
$this->putManifest($repo, 'users/alice/20261006T120000Z-aaaa0002');
|
||||
|
||||
$this->expectException(RepositoryException::class);
|
||||
$this->expectExceptionMessage("alice has 2 exports here; pass one of these (newest first):\n users/alice/20261006T120000Z-aaaa0002\n users/alice/20261005T120000Z-aaaa0001");
|
||||
$repo->resolveUserExport('alice');
|
||||
}
|
||||
|
||||
public function testResolveListsTheExportsOfTheUserOfAWrongPath(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
|
||||
|
||||
$this->expectException(RepositoryException::class);
|
||||
$this->expectExceptionMessage("users/alice/typo is not a recorded user export for this location; pass one of these (newest first):\n users/alice/20261005T120000Z-aaaa0001");
|
||||
$repo->resolveUserExport('users/alice/typo');
|
||||
}
|
||||
|
||||
public function testResolveNamesTheExpectedFormForAUserWithoutExports(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
|
||||
|
||||
$this->expectException(RepositoryException::class);
|
||||
$this->expectExceptionMessage('bob has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports');
|
||||
$repo->resolveUserExport('bob');
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Tests\Unit\Service;
|
||||
|
||||
use OCA\NgBackup\Service\Schedule;
|
||||
use OCP\IAppConfig;
|
||||
use OCP\IConfig;
|
||||
use OCP\IGroup;
|
||||
use OCP\IGroupManager;
|
||||
use OCP\IUser;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
|
||||
/** The daily time in a time zone (issue #19), and the conversion of a time saved without one. */
|
||||
class ScheduleTest extends TestCase {
|
||||
private array $values = [];
|
||||
|
||||
private function schedule(string $serverZone = '', string $adminZone = ''): Schedule {
|
||||
$app = $this->createMock(IAppConfig::class);
|
||||
$app->method('getValueString')->willReturnCallback(fn (string $a, string $k, string $d = '') => $this->values[$k] ?? $d);
|
||||
$app->method('setValueString')->willReturnCallback(function (string $a, string $k, string $v): bool {
|
||||
$this->values[$k] = $v;
|
||||
return true;
|
||||
});
|
||||
$config = $this->createMock(IConfig::class);
|
||||
$config->method('getSystemValueString')->willReturn($serverZone);
|
||||
$config->method('getUserValue')->willReturn($adminZone);
|
||||
$user = $this->createMock(IUser::class);
|
||||
$user->method('getUID')->willReturn('admin');
|
||||
$group = $this->createMock(IGroup::class);
|
||||
$group->method('getUsers')->willReturn([$user]);
|
||||
$groups = $this->createMock(IGroupManager::class);
|
||||
$groups->method('get')->willReturn($group);
|
||||
return new Schedule($app, $config, $groups);
|
||||
}
|
||||
|
||||
public function testATimeSavedWithoutAZoneShowsAsItRanInTheAdministratorsZone(): void {
|
||||
$this->values = [Schedule::TIME_KEY => '02:30'];
|
||||
|
||||
$s = $this->schedule(adminZone: 'Europe/Berlin')->get();
|
||||
|
||||
// it ran at 02:30 UTC: in summer that is 04:30 in Berlin, in winter 03:30
|
||||
$summer = Schedule::convert('02:30', 'UTC', 'Europe/Berlin', strtotime('2026-07-01 12:00 UTC'));
|
||||
$this->assertSame('04:30', $summer);
|
||||
$this->assertSame('Europe/Berlin', $s['timezone']);
|
||||
$this->assertSame(Schedule::convert('02:30', 'UTC', 'Europe/Berlin'), $s['time']);
|
||||
}
|
||||
|
||||
public function testTheServersDefaultZoneComesFirst(): void {
|
||||
$this->values = [Schedule::TIME_KEY => '01:00'];
|
||||
$this->assertSame('America/Toronto', $this->schedule('America/Toronto', 'Europe/Berlin')->get()['timezone']);
|
||||
}
|
||||
|
||||
public function testWithoutAnyZoneItStaysUtc(): void {
|
||||
$this->values = [Schedule::TIME_KEY => '02:30'];
|
||||
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
|
||||
}
|
||||
|
||||
public function testTodayAtIsTheMomentInTheSchedulesZone(): void {
|
||||
$this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin'];
|
||||
$at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC'));
|
||||
$this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC');
|
||||
}
|
||||
|
||||
public function testAnUnknownZoneOrABadTimeIsRefused(): void {
|
||||
$s = $this->schedule();
|
||||
$this->expectException(\InvalidArgumentException::class);
|
||||
$s->set('03:00', 'Mars/Olympus');
|
||||
}
|
||||
}
|
||||
@@ -47,6 +47,12 @@ if [ "$LEGACY" = legacy ]; then
|
||||
fi
|
||||
|
||||
run_on() { docker exec -u www-data "$1" php occ "${@:2}"; }
|
||||
# The app store sends the whole app list (~38 MB) and Nextcloud gives up after 120 s: on a slow day
|
||||
# an install fails for reasons outside this test. Up to three tries.
|
||||
install_app() { # container app
|
||||
for i in 1 2 3; do run_on "$1" app:install "$2" >/dev/null 2>&1 && return 0; sleep 20; done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Database settings for an instance: SQLite inside the container, or its own PostgreSQL container
|
||||
db_env() {
|
||||
@@ -109,7 +115,7 @@ if [ "$DB" != sqlite ]; then
|
||||
# while the restore loads tables in name order, accounts before mailboxes.
|
||||
docker run -d --name ngb-frt-imap --network "$NET" -e MAILNAME=mail.domain.tld -e MAIL_ADDRESS=user@domain.tld \
|
||||
-e MAIL_PASS=mypassword ghcr.io/christophwurst/docker-imap-devel:latest >/dev/null
|
||||
run_on ngb-frt-origin app:install mail >/dev/null || fail "install mail on the origin"
|
||||
install_app ngb-frt-origin mail || fail "install mail on the origin (app store unreachable?)"
|
||||
run_on ngb-frt-origin config:system:set app.mail.verify-tls-peer --type=boolean --value=false >/dev/null
|
||||
sleep 10
|
||||
run_on ngb-frt-origin mail:account:create frtest Test user@domain.tld ngb-frt-imap 993 ssl user@domain.tld mypassword \
|
||||
@@ -118,7 +124,9 @@ if [ "$DB" != sqlite ]; then
|
||||
[ "$(sql origin 'select count(*) from oc_mail_accounts where drafts_mailbox_id is not null')" = 1 ] \
|
||||
|| fail "the mail account does not point at its mailboxes"
|
||||
# Deck installed and then switched off: its tables stay in the database and in the backup
|
||||
run_on ngb-frt-origin app:install deck >/dev/null || fail "install deck on the origin"
|
||||
install_app ngb-frt-origin deck || fail "install deck on the origin (app store unreachable?)"
|
||||
# a 4-byte character (emoji) in a table the restore has to create: utf8mb4 on MySQL/MariaDB
|
||||
sql origin "insert into oc_deck_boards (title, owner, color, archived, deleted_at, last_modified) values ('Plan 📌', 'frtest', '0087C5', false, 0, 0)" >/dev/null
|
||||
run_on ngb-frt-origin app:disable deck >/dev/null
|
||||
ok "origin has a synced Mail account and the tables of a disabled app"
|
||||
fi
|
||||
@@ -202,6 +210,7 @@ if [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ] && [ "$LEGACY" = legacy
|
||||
ok "older backup: tables of the disabled app left out, Mail restored"
|
||||
elif [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ]; then
|
||||
[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created"
|
||||
[ "$(sql restore "select title from oc_deck_boards")" = "Plan 📌" ] || fail "a 4-byte character did not survive in a created table"
|
||||
[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
|
||||
[ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \
|
||||
|| fail "the foreign keys of Mail are missing after the restore"
|
||||
|
||||
@@ -36,8 +36,13 @@ MANIFEST=$(occ backup:user:backup ci "$U" | sed -n 's/.* to \(.*\)\.$/\1/p')
|
||||
[ -n "$MANIFEST" ] || fail "backup:user:backup printed no manifest path"
|
||||
ok "backed up $U to $MANIFEST"
|
||||
|
||||
# as-backup: restore into <uid>-bak; the original account must stay untouched.
|
||||
occ backup:user:restore ci "$MANIFEST" --mode=as-backup >/dev/null || fail "restore as-backup"
|
||||
occ backup:user:list ci "$U" | grep -qF "$MANIFEST" || fail "backup:user:list does not show $MANIFEST"
|
||||
ok "backup:user:list shows the export"
|
||||
|
||||
# as-backup: restore into <uid>-bak; the original account must stay untouched. $U has one export,
|
||||
# so the bare user id names it (#22).
|
||||
OUT=$(occ backup:user:restore ci "$U" --mode=as-backup) || fail "restore as-backup by user id"
|
||||
echo "$OUT" | grep -qF "Using $MANIFEST." || fail "restore by user id did not say which export it used"
|
||||
occ files:scan "$U-bak" >/dev/null
|
||||
SUM_BAK=$($X sh -c "cd $DATA/$U-bak/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
||||
[ "$SUM_BEFORE" = "$SUM_BAK" ] || fail "the <uid>-bak account's files differ from the original"
|
||||
@@ -55,6 +60,11 @@ SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xa
|
||||
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after a replace restore"
|
||||
ok "replace restores the original content under the same uid, byte-identical"
|
||||
|
||||
# The replace above added a safety export: a bare user id is now ambiguous and must list both.
|
||||
if OUT=$(occ backup:user:restore ci "$U" --mode=as-backup 2>&1); then fail "restore by an ambiguous user id ran"; fi
|
||||
echo "$OUT" | grep -qF "$U has 2 exports here" && echo "$OUT" | grep -qF "$MANIFEST" || fail "an ambiguous user id did not list the exports"
|
||||
ok "an ambiguous user id lists the exports instead of picking one"
|
||||
|
||||
# Retention: $U now has the manual export above plus the automatic safety-export the replace
|
||||
# restore made of it just before deleting it (4 total after these two more), then --user-last=2
|
||||
# must forget exactly the 2 oldest.
|
||||
|
||||
Reference in New Issue
Block a user