diff --git a/CHANGELOG.md b/CHANGELOG.md index bf1ce07..ff2b2cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,60 @@ Each heading starts with that ``, once: the app store shows a release t whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be shown under 1.0.0. The git tag of a release candidate goes after the date. +## [1.1.0] - 2026-11-09 + +### Added +- Pin a restore point, on the restore page or with `occ backup:pin`: the retention policy leaves it + alone, for example the one from before an upgrade. Unpinning notifies all administrators. +- A note on a restore point that you can change afterwards, on the restore page or with + `occ backup:note`. Pins and notes are stored encrypted on the location, so a new server sees them. + An older version of NG Backup can still read the location, but drops them when it writes to it. +- A schedule per location: off, once a day, or every N hours from a given time, on the page under + Backups or with `occ backup:target:schedule`. After the update every location keeps the time that + held for all of them; a new location starts with the schedule of the first one. A schedule is + shown and changed in the time zone it was saved in, which the page names next to the time. +- All folders get their original modification time back on a restore, not only the empty ones, from + the first backup made with this version. An older version of NG Backup can still read new restore + points. A restore through Nextcloud into a folder that is already there leaves its time alone. +- Heavy background jobs (Memories, Recognize, Preview Generator, full text search, the antivirus + background scan) pause while a backup runs, at most 6 hours in a row, then they get an hour. No app + is turned off, and a job that is running goes on. On by default; turn it off under Backups or with + `occ backup:pause-jobs off`. +- Disaster recovery without the recovery kit: on a new server without a backup key, + `occ backup:target:add` finds the backups on the location and takes their key from there with one + of the old server's passphrases. Backups then wait until a new recovery kit is confirmed. The + deletion delay is now also stored on the location, sealed with the key, from its first backup + with this version; a location backed up only by an older version gives the default of 7 days. + When the settings on the location cannot be read, the longest delay (365 days) is taken, so + nothing leaves the Bin sooner than on the old server; the command says so. +- A passphrase change that cannot reach a location (offline at the time) now reaches it at the + location's next backup, so a new server can open it with the new passphrase. +- `occ backup:target:add --repair-config`: when the config file of a location is gone, the recovery + kit writes it again (for locations backed up by this version or newer). Before, the message said + this was possible, but it was not. +- The key page warns against creating a new key when you are restoring a server. A key made there + by mistake can still make way for the key of the backups, as long as it was never used (no + location, no backup): `occ backup:key:import --replace-unused-key` with the kit, or + `occ backup:target:add --replace-unused-key` with a passphrase of the old server. A key that + protected a location is never replaced, and neither is a key from before this version. All + administrators are notified of a replacement. + +### Changed +- The list of backups stays within the page: a long label, note or error wraps onto more lines + instead of making the table scroll sideways. +- Reading files for a backup is much faster on disks where every request waits, such as network and + cloud volumes: files are read in pieces of 4 MiB instead of 8 KiB. On a test server the disk + delivered 27 MiB/s instead of 5 MiB/s. +- Locations are backed up one after the other: a backup that has to wait shows as queued. A restore + point has the time its backup really began, not the time it was queued. +- Choosing a time no longer starts a backup right away when that time has already passed today. +- A location without a schedule no longer gets the notification that it has had no recent backup. + Instead, all administrators are told when a schedule is turned off, and reminded once a week while + another location is still backed up, or every day when none is. +- A file or folder that is left out because of its name now gets a notification of its own, in the + log and in `occ backup:run`. It says the item is missing from the backup, gives example paths and + asks to rename it; before, it was reported as unreadable, with advice about permissions. + ## [1.0.3] - 2026-11-06 ### Fixed diff --git a/PLAN.md b/PLAN.md index a6c9bc9..cf31fa6 100644 --- a/PLAN.md +++ b/PLAN.md @@ -472,9 +472,12 @@ in README "Known limitations": tijd terug. Hoort zo, vooral bij oudere verzamelingen: de maptijd wordt gebruikt om in te schatten wanneer een map is aangemaakt. De backup moet dan de tijd van alle mappen vastleggen en de restore die aan het eind terugzetten; werkt pas vanaf de eerste backup met die versie. Op de roadmap onder 1.1.0. + **1.0.3 uitgebracht (2026-11-06)**, ook op nc01 geïnstalleerd. - **Branches (besluit 2026-11-06):** werk voor een versie op een branch met die naam (`1.0.3`); `polish` alleen voor kleine cosmetische zaken, per geval bekijken of en wanneer die gemerged worden. SMB staat uit op sub5 en sub6; sub6 is het testaccount, eerst aanzetten voor een SMB-test. + Na 1.0.3 gaat het werk naar `1.1.0`. Een spoedfix die niet kan wachten gaat op `1.0.4` vanaf `main`, + krijgt een eigen merge en release; daarna `main` in `1.1.0` mergen, zodat daar met de fix getest wordt. - **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar 1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november, niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij @@ -569,11 +572,33 @@ in README "Known limitations": - **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3, 4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan een paar bestanden. -- **1.1.0:** +- **1.1.0** (geen harde datum: een week was een voorzet, langer mag; de releasefrequentie mag omlaag, + besluit 2026-11-06): + **Betere melding voor namen die een snapshot niet kan bevatten** (backslash, stuurteken, geen UTF-8): + een eigen melding in plaats van "unreadable" met het advies over rechten. Zegt dat deze bestanden of + mappen in de backup ontbreken, geeft voorbeelden van de paden en vraagt ze te hernoemen; de volgende + backup neemt ze mee. Alleen als Nextcloud-melding: de app mailt niet zelf, of meldingen gemaild worden + is de keuze van de beheerder (besluit 2026-11-06). + **Stand op branch `1.1.0` (2026-11-07):** klaar: betere melding, punten 2, 4, 5, 6 en 8, en de maptijden + (`files:scan` na een volledige restore: 0 "Updated" in de simulatie; opslag gepakt in het metarecord). + Punt 2 reserveert de jobs in `oc_jobs` met één tijdstempel, alleen wat Nextcloud op dat moment zou kunnen + starten; 6 uur aan, dan 1 uur vrij; bij een crash geeft Nextcloud de reservering na 12 uur zelf vrij; + de logregels staan op niveau info, dus standaard niet zichtbaar. Pin en notitie staan + in de catalogus, met een kopie in de database voor de lijsten. Een moment van vóór het instellen van + een schema start niets. Geen schema: geen melding "geen recente backup", wel een melding aan alle + beheerders bij het uitzetten en een herinnering, wekelijks zolang een andere locatie een schema heeft, + anders dagelijks (besluit 2026-11-07). Nog niet getest: de update vanaf 1.0.3, de pagina in een browser. 1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van `target:add` en sleutelimport (het gevoeligste pad), daarom niet in rc1. Daarna de DR-test op PVE02 herhalen zonder kit. + **Gebouwd (2026-11-07):** `backup:target:add` op een server zonder sleutel kijkt eerst of de + locatie een repository heeft, vraagt dan een wachtwoordzin (of `--passphrase-file`), opent de + slots uit `config`, controleert de sleutel tegen de `check` en slaat hem pas dan op; geen + kitbevestiging, dus backups wachten op een nieuwe kit. Nieuw bestand `settings` op de locatie, + versleuteld met de hoofdsleutel en eenmalig geschreven (bij `target:add` en bij het begin van + een backup): de verwijdertermijn, anders de standaard 7 dagen. Getest: unit-tests en + `full-restore-roundtrip.sh 35-apache pgsql no-kit`. Nog open: de DR-test op PVE02 zonder kit. 2. Instelling "zware achtergrondjobs pauzeren tijdens een backup" (standaard aan, uit te zetten): Memories, files_antivirus BackgroundScanner, Recognize, Preview Generator, fulltextsearch. Via de reservering/last_run van die jobs (geen apps uitzetten), crashbestendig, maximaal @@ -581,6 +606,14 @@ in README "Known limitations": 3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf, 33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1. + **Gemeten 2026-11-07 op nc01 (alleen lezen):** de schijf (sdb, 2 TB, ROTA=1) haalt met `dd` + 21 MiB/s met 1 stroom, 37 met 4, 55 met 8. PHP las 8 KiB per verzoek: 5,2 MiB/s. Met + `stream_set_read_buffer(0)` en blokken van 4 MiB: 26,8 MiB/s. Hulpprocessen die vooruitlezen + gaven daarbovenop niets (20 MiB/s, ze concurreren). Gebouwd: grote leesverzoeken (commit + "Read files ... 4 MiB"). Niet gebouwd: meerdere uploads tegelijk; SMB bundelt al per 1 MB + (icewind/smb), dus winst over een lange lijn vraagt meerdere packs tegelijk. + **Besluit 2026-11-08: meerdere uploads tegelijk in 1.2.0**, met een instelling (standaard 2) + zodat de beheerder het afstemt op zijn server en de omvang van zijn backup. 4. **Schema per locatie** (besluit 2026-10-19): in de instellingen van een locatie "uit", "dagelijks om HH:MM" of "elke N uur (vanaf HH:MM)". Nu is er één `schedule_time` voor alle locaties, één keer per dag. Bij de update wordt de huidige tijd het schema van elke @@ -601,11 +634,10 @@ in README "Known limitations": nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met `occ backup:pin ` / `--unpin`. Op de locatie bewaren (catalogus), zodat de opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly. - 9. **Feedbackvraag weghalen** in 1.1.0 of 1.2.0 (besluit 2026-10-27): de regel "We would like + 9. **Feedbackvraag weghalen in 1.2.0** (besluit 2026-11-07, niet in 1.1.0): de regel "We would like to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot - dan staan (de vermelding "in beta" is er in 1.0.1 al af). Zijn er tegen die - tijd geen noemenswaardige meldingen, dan werkt het: mensen melden eerder wat niet werkt dan - wat gewoon werkt. Het formulier (`experience.yml`) mag blijven voor wie het zelf opzoekt. + dan staan. Er komt weinig feedback bij zo'n 150 downloads per versie; geen bericht is goed + bericht. Het formulier (`experience.yml`) in de issues blijft. - **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met PostgreSQL 18)** — voor **1.0.0-rc.2**: 1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE, diff --git a/README.md b/README.md index 3211c1d..b2a1699 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on see "Storage backend experiences" below — that requirement comes from files_external, not from ng_backup, and applies to any Nextcloud app that writes to an SMB share.) -**Status: 1.0.3, stable.** Disaster recovery has been tested on real installations (see "Tested +**Status: 1.1.0, stable.** Disaster recovery has been tested on real installations (see "Tested disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and `docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See `docs/DISASTER-RECOVERY.md`. @@ -87,8 +87,9 @@ All backups of an installation are encrypted with one master key. - The **recovery kit** is your own copy of that encrypted key. Still keep it off the server and off the location: an attacker who gets write access to the location can damage or delete its `config` file on purpose, and without the kit that makes every backup there unreadable. -- For a **restore on a new server** (disaster recovery) you currently need the **recovery kit** - and **one passphrase**. +- For a **restore on a new server** (disaster recovery) **one passphrase** is enough: the new + server takes the key from the location. With the **recovery kit** you can restore even when the + location's `config` file is damaged or gone. - Planned: an option to not store the key on the location at all. Then the kit is the only copy, and losing it means losing every backup; that option will come with an explicit warning. - Up to **three passphrases** ("slots") can open the key, so one person leaving or dying does not @@ -170,6 +171,7 @@ Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECO | Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical | | The same, 1.0.2 | Hetzner Storage Box (SMB), Germany → Canada | PostgreSQL → PostgreSQL | 150,755 files restored in about 29 h (7-13 MiB/s), going on after a lost connection; sample identical, empty folders not restored ([#27](https://github.com/dillardblom/ng_backup/issues/27)) | | 1.0.3, small simulations of 2 GB: as planned, with the restore killed halfway, and with a disk that runs full | Local SFTP server | PostgreSQL → PostgreSQL and MariaDB | all 3,101 user files and all empty folders identical; users, a share and a calendar back | +| 1.1.0, the same three simulations | Local SFTP server | PostgreSQL → PostgreSQL and MariaDB | all 3,101 user files identical in content, size and time, all empty folders back; users, a share and a calendar back | | 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical | | The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical | | The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) | @@ -266,18 +268,9 @@ What is planned, without dates. Ideas and votes are welcome in the [issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here, a 👍 on its issue helps decide what comes first. -**1.1.0** -- Restore without the recovery kit: the backup key from the location itself, opened with one of - your passphrases. -- A schedule per location, also "every N hours". -- Locations backed up one after the other instead of at the same time. -- All folders get their original modification time back on a restore, not only the empty ones. -- A note on a restore point that you can change afterwards. -- Pinning a restore point, so the retention policy never removes it (for example the one from - before an upgrade). -- An option to pause heavy background jobs (such as Memories, Recognize or the Preview - Generator) while a backup runs. -- Faster backups: reading and uploading at the same time. +**1.2.0** +- Faster backups over a long line: several uploads at the same time, with a setting + (2 by default) to suit the server and the size of the backup. **Later** - A full restore onto the existing installation, without first installing a fresh Nextcloud, in a @@ -293,8 +286,9 @@ On the backlog; know them before you rely on NG Backup: - **Empty folders only come back from backups made with 1.0.3 or newer.** Older restore points do not contain them. -- **Folders with files in them show the time of the restore**, not their original modification time. - The files keep theirs. On the roadmap. +- **Folders with files in them only get their original modification time back from backups made with + 1.1.0 or newer.** From older restore points they show the time of the restore; the files keep theirs. + A restore through Nextcloud into a folder that is already there leaves that folder's time alone. - **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in `config.php`) is not supported: a backup refuses to start there. Folders mounted through the External storage app are not in the data directory and are **not included**, without a warning. @@ -329,10 +323,13 @@ On the backlog; know them before you rely on NG Backup: | `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation | | `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots | | `backup:target:backends/add/edit/list/remove/append-only` | backup locations | +| `backup:target:schedule` | when a location is backed up: off, daily, or every N hours | | `backup:run`, `backup:status`, `backup:list` | back up, status (runs of the last 7 days; `--all` for all), restore points | | `backup:cancel ` | stop a running backup | | `backup:browse`, `backup:restore` | browse and restore | | `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash | +| `backup:pin`, `backup:note` | keep a restore point, a note you can change later | +| `backup:pause-jobs` | whether heavy background jobs of other apps pause during a backup | | `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) | | `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) | | `backup:key:import` | import a recovery kit on a fresh installation | diff --git a/appinfo/info.xml b/appinfo/info.xml index 6ee5d53..af2804a 100644 --- a/appinfo/info.xml +++ b/appinfo/info.xml @@ -45,7 +45,7 @@ welcome: use the "Share your experience" link on the admin page. Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if you need help with restoring a backup after a crash.]]> - 1.0.3 + 1.1.0 agpl Dillard Blom NgBackup @@ -56,10 +56,10 @@ you need help with restoring a backup after a crash.]]> https://github.com/dillardblom/ng_backup https://github.com/dillardblom/ng_backup/issues https://github.com/dillardblom/ng_backup - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/1-backups.png - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/2-restore.png - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/3-settings.png - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/4-bin.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.1.0/screenshots/1-backups.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.1.0/screenshots/2-restore.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.1.0/screenshots/3-settings.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.1.0/screenshots/4-bin.png sodium @@ -83,11 +83,15 @@ you need help with restoring a backup after a crash.]]> OCA\NgBackup\Command\ListSnapshots OCA\NgBackup\Command\Browse OCA\NgBackup\Command\Restore + OCA\NgBackup\Command\PauseJobs + OCA\NgBackup\Command\Pin + OCA\NgBackup\Command\Note OCA\NgBackup\Command\Prune OCA\NgBackup\Command\Retention OCA\NgBackup\Command\TargetAppendOnly OCA\NgBackup\Command\TargetTrustCurrent OCA\NgBackup\Command\TargetLimit + OCA\NgBackup\Command\TargetSchedule OCA\NgBackup\Command\Trash OCA\NgBackup\Command\TrashRestore OCA\NgBackup\Command\KeySlots diff --git a/appinfo/routes.php b/appinfo/routes.php index b058d68..5714bda 100644 --- a/appinfo/routes.php +++ b/appinfo/routes.php @@ -25,8 +25,11 @@ return [ ['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'], ['name' => 'AdminApi#startBackup', 'url' => '/api/backups', 'verb' => 'POST'], ['name' => 'AdminApi#tick', 'url' => '/api/tick', 'verb' => 'POST'], - ['name' => 'AdminApi#setSchedule', 'url' => '/api/schedule', 'verb' => 'PUT'], + ['name' => 'AdminApi#setPauseHeavyJobs', 'url' => '/api/settings/pause-heavy-jobs', 'verb' => 'PUT'], + ['name' => 'AdminApi#setTargetSchedule', 'url' => '/api/targets/{id}/schedule', 'verb' => 'PUT'], ['name' => 'AdminApi#snapshots', 'url' => '/api/targets/{targetId}/snapshots', 'verb' => 'GET'], + ['name' => 'AdminApi#setSnapshotNote', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/note', 'verb' => 'PUT'], + ['name' => 'AdminApi#setSnapshotPinned', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/pin', 'verb' => 'PUT'], ['name' => 'AdminApi#browse', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/browse', 'verb' => 'GET'], ['name' => 'AdminApi#startRestore', 'url' => '/api/restores', 'verb' => 'POST'], ], diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md index 745f7be..c7ffc48 100644 --- a/docs/DISASTER-RECOVERY.md +++ b/docs/DISASTER-RECOVERY.md @@ -5,7 +5,8 @@ Nextcloud back from an NG Backup location. Do them in this order. ## What you need -- [ ] The **recovery kit** file (`ng_backup-kit.json`) and **one passphrase** of it. +- [ ] **One passphrase** of the old server, and if you have it, the **recovery kit** file + (`ng_backup-kit.json`). - [ ] The **connection details of the backup location**: host, user, password or keys, bucket, folder. - [ ] A new server with the **same Nextcloud major version** as the old one (for example 34), a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for @@ -15,8 +16,10 @@ Nextcloud back from an NG Backup location. Do them in this order. - [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by hand). Keep their packages somewhere safe beforehand. -No kit? If the location still holds its `config` file, the key is there too, but this version -can only import it from a kit. Keep the kit off the server, next to the passphrase. +No kit? Since version 1.1.0 the passphrase is enough: the key is also stored on the location (in +its `config` file), locked with the same passphrases. Skip step 4. The kit is still the safer +copy: it helps when that file on the location is damaged. Keep it off the server, next to the +passphrase. ## Steps @@ -40,7 +43,8 @@ This keeps the new server quiet (no mail syncs, no notifications, no backups) un **3. Install NG Backup** (the same version or newer) and enable it. -**4. Import the backup key.** Copy the kit to the new server so the web server user can read it. +**4. Import the backup key** (with a kit; without one, go to step 5). Copy the kit to the new +server so the web server user can read it. From your own computer: @@ -67,13 +71,29 @@ the index and shows how far it is, for example occ backup:target:add offsite sftp -a password::password -o host=... -o user=... -o password=... -o root=... --path=... -If it says "No backup key yet", the key is not imported. Run step 4: +Without a kit (step 4 skipped), it finds the backups and asks for a passphrase of the old server; +you type it yourself. It then shows the key fingerprint and the deletion delay of the old server. +A wrong passphrase changes nothing: run the command again. Backups made before version 1.1.0 did +not store the deletion delay on the location; it is then set to 7 days. - occ backup:key:import /tmp/ng_backup-kit.json +If it says that the location's **config file is gone**, the key is only in the recovery kit: +import it (step 4), then add the location with `--repair-config` at the end of the same command. +That writes the config again from the kit; it works for locations backed up by version 1.1.0 or +newer. -Do **not** run `backup:key:init`, even though the message suggests it: that makes a brand new key, -and your backups were encrypted with the old one. NG Backup then refuses the location ("created -with a different key"). +If it says "No backup key yet, and no backups at this location", the connection details or +`--path` differ from the old server's. Check them; do **not** run `backup:key:init`: that makes a +brand new key, and your backups were encrypted with the old one. NG Backup then refuses the +location ("created with a different key"). + +**Made a new key by mistake** (with `backup:key:init` or on the page) before adding the location? +As long as that key was never used (no location added with it, no backup), it can make way for the +key of your backups. NG Backup says so when it refuses the location. Add `--replace-unused-key` to +`occ backup:key:import` (with the kit) or to `occ backup:target:add` (without the kit; it asks for +a passphrase of the old server). The passphrases and kit of the unused key then stop working. A key +that was used for a location is never replaced: its own backups depend on it. A key made with a +version before 1.1.0 always counts as used (those versions did not keep track), also when it never +protected a location; recover onto a fresh Nextcloud installation instead. **6. List the restore points** and pick one, normally the newest: @@ -164,6 +184,10 @@ The command then ends with exit code 2 instead of 0. The same option exists for **11. Make a backup of the new server** to the same location (`occ backup:run offsite`). The old server must stay off: two servers must never write to the same location. +Restored without a kit? Then backups wait until you have a new one: download it on the backup +page (or `occ backup:key:kit --output=...`), store it off the server and confirm it +(`occ backup:key:confirm`). + ## Never - Never let the old server write to the location again if it comes back. Switch its backups off diff --git a/docs/HOWTO.md b/docs/HOWTO.md index 3ff6136..ee9b1b1 100644 --- a/docs/HOWTO.md +++ b/docs/HOWTO.md @@ -79,7 +79,9 @@ occ backup:list Later runs upload only changed data. An unchanged run uploads (almost) nothing; the output shows the number of uploaded bytes, so check that value when you test incremental behaviour. -Schedule `occ backup:run` from system cron (or use the Administration settings, if you prefer). +Set when each location is backed up in the Administration settings, or with +`occ backup:target:schedule offsite 02:00` (once a day), `02:00/6` (every 6 hours from 02:00) or `off`. +Locations are backed up one after the other. ## 4. Check the backup regularly @@ -155,6 +157,10 @@ occ backup:list offsite # the snapshots must be lis occ backup:restore:full offsite ``` +Without the kit, leave out `backup:key:import`: `backup:target:add` finds the backups and takes +the key from the location with a passphrase (`--passphrase-file=/path/passphrase`, or it asks). +Backups then wait until a new kit is downloaded and confirmed. + The command refuses on an installation that already has more than one user, and asks before it starts; `--force` skips both (also needed to rerun it after it stopped halfway). @@ -173,8 +179,13 @@ occ backup:prune offsite --dry-run # show first occ backup:prune offsite # apply the policy; forgotten snapshots go to the trash first occ backup:trash # list the trash occ backup:trash:restore offsite +occ backup:pin offsite # keep this restore point; --unpin to undo +occ backup:note offsite "text" # a note you can change later; without text it is removed ``` +The retention policy leaves a pinned restore point alone, and a pinned one does not take the place of +a daily, weekly or monthly one. Pin and note are stored on the location, so a new server sees them too. + Snapshots are kept according to the policy, and their data is deleted only after the deletion delay (set with `backup:key:init --delete-delay-days`). `occ backup:prune offsite --dry-run` shows what would be removed without changing anything. diff --git a/lib/Backend/LocalBackend.php b/lib/Backend/LocalBackend.php index 266b517..f7cc96a 100644 --- a/lib/Backend/LocalBackend.php +++ b/lib/Backend/LocalBackend.php @@ -67,6 +67,9 @@ final class LocalBackend implements IBackend { if ($fh === false) { throw new BackendException('Not found: ' . $path); } + // Packs are read in large pieces: one request each, not 8 KiB at a time (see Repository::openForReading()). + stream_set_read_buffer($fh, 0); + stream_set_chunk_size($fh, 4 * 1048576); return $fh; } diff --git a/lib/BackgroundJob/BackupJob.php b/lib/BackgroundJob/BackupJob.php index 76dd5fc..7a3823a 100644 --- a/lib/BackgroundJob/BackupJob.php +++ b/lib/BackgroundJob/BackupJob.php @@ -33,6 +33,7 @@ final class BackupJob extends TimedJob { private IAppConfig $appConfig, private LoggerInterface $logger, private \OCA\NgBackup\Service\Schedule $schedule, + private \OCA\NgBackup\Service\HeavyJobPause $heavyJobs, ) { parent::__construct($time); // Well below the usual 5-minute cron interval: the last run is stamped a moment after cron @@ -43,25 +44,28 @@ final class BackupJob extends TimedJob { $this->setTimeSensitivity(IJob::TIME_SENSITIVE); } - private const MAX_DAILY_ATTEMPTS = 3; - protected function run($argument): void { $budget = PHP_SAPI === 'cli' ? 240.0 : 20.0; $deadline = microtime(true) + $budget; $this->startScheduled(); - foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) { + $this->heavyJobs->sync($this->backups->runsInTurn() !== []); + // One after the other: runs that read the same disks at the same time are not faster together. + foreach ($this->backups->runsInTurn() as $run) { while (microtime(true) < $deadline && $run->getStatus() === 'running') { $next = $this->backups->step($run, min($deadline, microtime(true) + $budget)); if ($next === $run) { - break; // step() hands back the same object when another process holds the run's lease + break 2; // step() hands back the same object when another process holds the run's lease } $run = $next; } } $this->failStaleRestores(); $this->weeklyPrune(); - foreach ($this->targets->list() as $t) { + $targets = $this->targets->list(); + foreach ($targets as $t) { $this->alerts->checkStale($t); + $othersActive = array_filter($targets, static fn ($o) => $o->getId() !== $t->getId() && (string)$o->getSchedule() !== '') !== []; + $this->alerts->checkOff($t, $othersActive); } } @@ -102,21 +106,18 @@ final class BackupJob extends TimedJob { $this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due); } - /** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */ + /** Start the backups the schedules of the locations ask for; they run in turn (see LocationSchedule). */ private function startScheduled(): void { - $todayAt = $this->schedule->todayAt(); - if ($todayAt === null || time() < $todayAt) { - return; - } + $now = time(); foreach ($this->targets->list() as $t) { - $last = $this->runs->lastFinished($t->getId(), BackupService::KIND_FULL); - if ($this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== [] || ($last !== null && $last->getStartedAt() >= $todayAt)) { + $moment = \OCA\NgBackup\Service\LocationSchedule::lastMoment((string)$t->getSchedule(), $this->schedule->zoneOr($t->getScheduleZone()), $now); + if ($moment === null || $this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== []) { continue; } - // After a failure: retry at most MAX_DAILY_ATTEMPTS times a day, an hour apart, instead - // of starting (and notifying about) a new full run every five minutes all day. - $today = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $todayAt); - if (count($today) >= self::MAX_DAILY_ATTEMPTS || ($today !== [] && time() - (int)$today[0]->getFinishedAt() < 3600)) { + $attempts = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $moment); + if (!\OCA\NgBackup\Service\LocationSchedule::isDue($moment, (int)$t->getScheduleSince(), + $this->dataStarted($this->runs->lastFinished($t->getId(), BackupService::KIND_FULL)), + count($attempts), ($attempts[0] ?? null)?->getFinishedAt(), $now)) { continue; } try { @@ -126,4 +127,15 @@ final class BackupJob extends TimedJob { } } } + + /** When a backup began reading its data: a queued one can start well after it was asked for. */ + private function dataStarted(?\OCA\NgBackup\Db\Run $run): ?int { + if ($run === null) { + return null; + } + $state = json_decode($run->getState(), true); + $started = is_array($state) ? ($state['files']['started'] ?? null) : null; + $at = is_string($started) ? strtotime($started) : false; + return $at !== false ? $at : $run->getStartedAt(); + } } diff --git a/lib/Command/KeyImport.php b/lib/Command/KeyImport.php index f982965..d626ebd 100644 --- a/lib/Command/KeyImport.php +++ b/lib/Command/KeyImport.php @@ -8,6 +8,7 @@ declare(strict_types=1); namespace OCA\NgBackup\Command; use OCA\NgBackup\Service\KeyService; +use OCA\NgBackup\Service\TargetService; use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Input\InputArgument; use Symfony\Component\Console\Input\InputInterface; @@ -18,6 +19,7 @@ use Symfony\Component\Console\Question\Question; final class KeyImport extends Command { public function __construct( private KeyService $keys, + private TargetService $targets, ) { parent::__construct(); } @@ -27,11 +29,19 @@ final class KeyImport extends Command { ->setDescription('Disaster recovery: adopt the backup key from a recovery kit on a fresh installation') ->addArgument('kit-file', InputArgument::REQUIRED, 'The recovery kit file (occ backup:key:kit on the original installation)') ->addOption('passphrase-file', null, InputOption::VALUE_REQUIRED, 'Read the passphrase from this file instead of asking') + ->addOption('replace-unused-key', null, InputOption::VALUE_NONE, 'A new key was made here by mistake: replace it by the kit\'s key, if it was never used (no location, no backup)') ->setHelp(<<<'HELP' For a fresh installation only (occ backup:key:init must not have been run here yet). Reads one of the kit's passphrase slots to recover the original master key, so occ backup:target:add opens the existing repository on the location instead of creating a new, empty one. +A new key was made here by mistake before the location was added? As long as it was never used +(no location, no backup), --replace-unused-key replaces it by the kit's key. A key that was used +is never replaced. + +No kit? Skip this command: occ backup:target:add then takes the key from the location with one +of the old server's passphrases. The kit still helps when the location's config file is damaged. + Example: occ backup:key:import /root/ng_backup-recovery-kit.json occ backup:target:add offsite sftp -a password::password -o host=... -o user=... -o password=... @@ -40,10 +50,24 @@ HELP); } protected function execute(InputInterface $input, OutputInterface $output): int { - if ($this->keys->isInitialized()) { - $output->writeln('A backup key already exists here (fingerprint ' . $this->keys->fingerprint() . '). occ backup:key:import is only for a fresh installation.'); + $replace = (bool)$input->getOption('replace-unused-key'); + if ($replace && !$this->keys->isInitialized()) { + $output->writeln('There is no backup key here to replace: leave out --replace-unused-key.'); return 1; } + if ($this->keys->isInitialized()) { + if (!$this->targets->keyIsUnused()) { + $output->writeln('A backup key already exists here (fingerprint ' . $this->keys->fingerprint() . ') and it has been used for a backup location: it is never replaced.'); + $output->writeln('occ backup:key:import is for a server that is being restored.'); + return 1; + } + if (!$replace) { + $output->writeln('A backup key already exists here (fingerprint ' . $this->keys->fingerprint() . '), but it was never used: no location, no backup.'); + $output->writeln('Made by mistake on a server that is being restored? Run this command again with --replace-unused-key:'); + $output->writeln('the kit\'s key then takes its place, and the passphrases and recovery kit of the unused key stop working.'); + return 1; + } + } $kitFile = (string)$input->getArgument('kit-file'); $raw = @file_get_contents($kitFile); if ($raw === false) { @@ -70,12 +94,16 @@ HELP); $pass = (string)$helper->ask($input, $output, $q); } try { - $this->keys->importRecoveryKit($kit, $pass); + if ($replace) { + $this->targets->importKitReplacingUnusedKey($kit, $pass); + } else { + $this->keys->importRecoveryKit($kit, $pass); + } } catch (\Throwable $e) { $output->writeln('' . $e->getMessage() . ''); return 1; } - $output->writeln('Backup key imported, fingerprint ' . $this->keys->fingerprint() . '.'); + $output->writeln(($replace ? 'The unused key was replaced. ' : '') . 'Backup key imported, fingerprint ' . $this->keys->fingerprint() . '.'); // Read from the kit file, which is not authenticated beyond the wrapped key: show it so a // wrong or altered value stands out. $output->writeln('Deletion delay for forgotten snapshots: ' . $this->keys->deleteDelayDays() . ' days (from the kit).'); diff --git a/lib/Command/ListSnapshots.php b/lib/Command/ListSnapshots.php index 8ad4b96..5cc2892 100644 --- a/lib/Command/ListSnapshots.php +++ b/lib/Command/ListSnapshots.php @@ -40,14 +40,14 @@ final class ListSnapshots extends Command { foreach ($targets as $t) { foreach ($this->snapshots->findForTarget($t->getId()) as $s) { $rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(), - $s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getLabel() ?? '']; + $s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getPinned() ? 'yes' : '', $s->getLabel() ?? '', $s->getNote() ?? '']; } } if ($rows === []) { $output->writeln('No snapshots yet (occ backup:run).'); return 0; } - (new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'label'])->setRows($rows)->render(); + (new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'pinned', 'label', 'note'])->setRows($rows)->render(); return 0; } } diff --git a/lib/Command/Note.php b/lib/Command/Note.php new file mode 100644 index 0000000..bdfcf4c --- /dev/null +++ b/lib/Command/Note.php @@ -0,0 +1,44 @@ +setName('backup:note') + ->setDescription('Set or remove the note on a snapshot') + ->addArgument('target', InputArgument::REQUIRED, 'Location name or id') + ->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)') + ->addArgument('note', InputArgument::OPTIONAL, 'The note, at most ' . PruneService::NOTE_MAX . ' characters; without it the note is removed', ''); + } + + protected function execute(InputInterface $input, OutputInterface $output): int { + $note = (string)$input->getArgument('note'); + try { + $this->prune->setNote($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), $note); + } catch (\Throwable $e) { + $output->writeln('' . $e->getMessage() . ''); + return 1; + } + $output->writeln(trim($note) === '' ? 'Note removed.' : 'Note saved.'); + return 0; + } +} diff --git a/lib/Command/PauseJobs.php b/lib/Command/PauseJobs.php new file mode 100644 index 0000000..232a1ea --- /dev/null +++ b/lib/Command/PauseJobs.php @@ -0,0 +1,45 @@ +setName('backup:pause-jobs') + ->setDescription('Show or set whether heavy background jobs (Memories, Recognize, Preview Generator, ...) pause while a backup runs') + ->addArgument('setting', InputArgument::OPTIONAL, '"on" or "off"; omit to show the current one'); + } + + protected function execute(InputInterface $input, OutputInterface $output): int { + $setting = $input->getArgument('setting'); + if ($setting !== null) { + if ($setting !== 'on' && $setting !== 'off') { + $output->writeln('Use "on" or "off".'); + return 1; + } + $this->pause->setEnabled($setting === 'on'); + } + $status = $this->pause->status(); + $output->writeln('Pausing heavy background jobs during a backup: ' . ($status['enabled'] ? 'on' : 'off')); + if ($status['jobs'] !== []) { + $output->writeln('Paused now: ' . implode(', ', $status['jobs'])); + } + return 0; + } +} diff --git a/lib/Command/Pin.php b/lib/Command/Pin.php new file mode 100644 index 0000000..eedd132 --- /dev/null +++ b/lib/Command/Pin.php @@ -0,0 +1,45 @@ +setName('backup:pin') + ->setDescription('Keep a snapshot: the retention policy does not remove a pinned one') + ->addArgument('target', InputArgument::REQUIRED, 'Location name or id') + ->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)') + ->addOption('unpin', null, InputOption::VALUE_NONE, 'Leave the snapshot to the retention policy again'); + } + + protected function execute(InputInterface $input, OutputInterface $output): int { + $unpin = (bool)$input->getOption('unpin'); + try { + $this->prune->setPinned($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), !$unpin); + } catch (\Throwable $e) { + $output->writeln('' . $e->getMessage() . ''); + return 1; + } + $output->writeln($unpin ? 'Snapshot unpinned; the retention policy applies to it again.' : 'Snapshot pinned; the retention policy leaves it alone.'); + return 0; + } +} diff --git a/lib/Command/RunBackup.php b/lib/Command/RunBackup.php index b5aa06e..f107ea7 100644 --- a/lib/Command/RunBackup.php +++ b/lib/Command/RunBackup.php @@ -79,6 +79,9 @@ HELP); if (($f['unreadable'] ?? 0) > 0) { $output->writeln(sprintf(' warning: %d unreadable files or folders are missing from this backup (see the server log)', $f['unreadable'])); } + if (($f['badNames'] ?? 0) > 0) { + $output->writeln(sprintf(' warning: %d files or folders are missing from this backup because their name is not supported; rename them and the next backup includes them (see the server log)', $f['badNames'])); + } if (($f['changing'] ?? 0) > 0) { $output->writeln(sprintf(' warning: %d files kept changing and were saved as read (see the server log)', $f['changing'])); } diff --git a/lib/Command/Status.php b/lib/Command/Status.php index 65a1260..9f36c5b 100644 --- a/lib/Command/Status.php +++ b/lib/Command/Status.php @@ -31,6 +31,7 @@ final class Status extends Command { private VerifyService $verify, private Schedule $schedule, private CronHealth $cron, + private \OCA\NgBackup\Service\BackupService $backups, ) { parent::__construct(); } @@ -61,9 +62,11 @@ final class Status extends Command { $rows = []; $all = (bool)$input->getOption('all'); $shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400); + $queued = array_map(static fn (\OCA\NgBackup\Db\Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1)); foreach ($shown as $r) { $s = json_decode($r->getStats() ?? '{}', true)['files'] ?? []; - $rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus(), + $rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), + in_array($r->getId(), $queued, true) ? 'queued' : ($r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus()), Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '', $s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '', isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')]; diff --git a/lib/Command/TargetAdd.php b/lib/Command/TargetAdd.php index 0d24c1b..4c6d50a 100644 --- a/lib/Command/TargetAdd.php +++ b/lib/Command/TargetAdd.php @@ -13,7 +13,9 @@ use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Input\InputArgument; use Symfony\Component\Console\Input\InputInterface; use Symfony\Component\Console\Input\InputOption; +use Symfony\Component\Console\Helper\QuestionHelper; use Symfony\Component\Console\Output\OutputInterface; +use Symfony\Component\Console\Question\Question; final class TargetAdd extends Command { public function __construct( @@ -31,10 +33,25 @@ final class TargetAdd extends Command { ->addOption('auth', 'a', InputOption::VALUE_REQUIRED, 'Auth mechanism id (see backup:target:backends)') ->addOption('option', 'o', InputOption::VALUE_REQUIRED | InputOption::VALUE_IS_ARRAY, 'Backend/auth option key=value (repeatable)') ->addOption('path', 'p', InputOption::VALUE_REQUIRED, 'Folder on the location for this repository', 'ng_backup') + ->addOption('passphrase-file', null, InputOption::VALUE_REQUIRED, 'Disaster recovery without the recovery kit: read the passphrase from this file instead of asking') + ->addOption('repair-config', null, InputOption::VALUE_NONE, 'The location\'s config file is gone: write it again with the key of this server or of the imported recovery kit (locations backed up by 1.1.0 or newer)') + ->addOption('replace-unused-key', null, InputOption::VALUE_NONE, 'Disaster recovery after a new key was made here by mistake: replace that key, if it was never used, by the key of the backups at this location (asks for a passphrase of the old server)') ->setHelp(<<<'HELP' The location is reached with files_external's own connection code, but it is never added as an external storage: it does not appear in Files for anyone. Credentials are stored encrypted. +Disaster recovery without the recovery kit: on a server without a backup key, a location that +holds backups gives its key with one of the old server's passphrases. Run this command first +(no occ backup:key:import), then occ backup:list and occ backup:restore:full. + +A new key was made here by mistake (occ backup:key:init or the web page) before the location +was added? As long as that key was never used (no location, no backup), --replace-unused-key +replaces it by the key of the backups. A key that was used is never replaced. + +With the recovery kit: occ backup:key:import first; this command then asks for nothing more. +If the location's config file is gone, only the kit has the key: import it, then add the +location with --repair-config. + Examples: occ backup:target:add offsite amazons3 -a amazons3::accesskey -o bucket=backups -o hostname=s3.example.com -o region=eu-central-1 -o use_ssl=true -o use_path_style=true -o key=AKIA... -o secret=... --path=nextcloud @@ -43,13 +60,6 @@ HELP); } protected function execute(InputInterface $input, OutputInterface $output): int { - if (!$this->keys->isInitialized()) { - $output->writeln('No backup key yet.'); - $output->writeln('Restoring from a location that already has backups (disaster recovery)? Import the key from the recovery kit first:'); - $output->writeln(' occ backup:key:import /path/to/ng_backup-kit.json'); - $output->writeln('Only for a new location without backups: create a new key with occ backup:key:init. A new key cannot read existing backups.'); - return 1; - } $options = []; foreach ($input->getOption('option') as $pair) { if (!str_contains($pair, '=')) { @@ -63,24 +73,125 @@ HELP); $output->writeln('--auth is required (see occ backup:target:backends)'); return 1; } + $name = (string)$input->getArgument('name'); + $backend = (string)$input->getArgument('backend'); + $auth = (string)$input->getOption('auth'); + $path = (string)$input->getOption('path'); + $passphrase = null; + $repair = (bool)$input->getOption('repair-config'); + $replace = (bool)$input->getOption('replace-unused-key'); + if ($replace) { + if ($repair) { + $output->writeln('--replace-unused-key needs the config at the location; with the config gone only the kit helps: occ backup:key:import --replace-unused-key, then --repair-config.'); + return 1; + } + if (!$this->keys->isInitialized()) { + $output->writeln('There is no backup key here to replace: leave out --replace-unused-key.'); + return 1; + } + if (!$this->targets->keyIsUnused()) { + $output->writeln('The backup key of this server (fingerprint ' . $this->keys->fingerprint() . ') has been used for a backup location: it is never replaced.'); + return 1; + } + $output->writeln('The backup key of this server (fingerprint ' . $this->keys->fingerprint() . ') was never used. It is replaced by the key of the backups at'); + $output->writeln('this location, once a passphrase of the old server opens it. Its own passphrases and recovery kit then stop working.'); + $passphrase = $this->passphrase($input, $output); + if ($passphrase === null) { + return 1; + } + } elseif (!$this->keys->isInitialized()) { + try { + $state = $this->targets->locationState($backend, $auth, $options, $path); + } catch (\Throwable $e) { + $output->writeln('' . $e->getMessage() . ''); + return 1; + } + if ($state === TargetService::LOCATION_EMPTY) { + $output->writeln('No backup key yet, and no backups at this location.'); + $output->writeln('Restoring a server (disaster recovery)? Check the connection details and --path: they must be the same as on the old server.'); + $output->writeln('Only for a new location: create a new key first with occ backup:key:init. A new key cannot read existing backups.'); + return 1; + } + if ($state === TargetService::LOCATION_DATA_WITHOUT_CONFIG) { + $output->writeln('This location holds backups, but its config file is gone, and the key was stored in it.'); + $output->writeln('Now only the recovery kit has the key. Import it, then run this command again with --repair-config:'); + $output->writeln(' occ backup:key:import /path/to/ng_backup-kit.json'); + $output->writeln('Do not run occ backup:key:init: a new key cannot read these backups.'); + return 1; + } + if ($repair) { + $output->writeln('The config at this location is there: leave out --repair-config.'); + return 1; + } + $output->writeln('No backup key on this server yet, and this location holds backups. Their key is stored there too,'); + $output->writeln('locked with the passphrases of the old server: one of them unlocks it (the recovery kit is not needed).'); + $passphrase = $this->passphrase($input, $output); + if ($passphrase === null) { + return 1; + } + } // Adopting an existing repository reads its whole index first, which takes a while on a // large backup or a slow line; show how far it is instead of looking stuck. $output->writeln('Connecting to the location; with existing backups this reads their index first ...'); try { - $r = $this->targets->add($input->getArgument('name'), $input->getArgument('backend'), $input->getOption('auth'), $options, - (string)$input->getOption('path'), self::progress($output)); + $r = $passphrase === null + ? $this->targets->add($name, $backend, $auth, $options, $path, self::progress($output), $repair) + : $this->targets->addWithPassphrase($name, $backend, $auth, $options, $path, $passphrase, self::progress($output), $replace); + } catch (\OCA\NgBackup\Crypto\CryptoException $e) { + $output->writeln('' . $e->getMessage() . ''); + if (!$this->keys->isInitialized() || $replace) { + $output->writeln('Nothing was changed. Run the command again to try another passphrase, or use the recovery kit (occ backup:key:import).'); + } + return 1; } catch (\Throwable $e) { if ($output->isDecorated()) { $output->write("\r\033[K"); // clear an unfinished progress line } $output->writeln('' . $e->getMessage() . ''); + if (str_contains($e->getMessage(), 'different key') && !$replace) { + $output->writeln('The backup key here (fingerprint ' . $this->keys->fingerprint() . ') is not the one these backups were made with.'); + $output->writeln('Restoring a server? The recovery kit must be the old server\'s: its fingerprint is in the kit file. Nothing was changed.'); + } return 1; } + if ($repair) { + $output->writeln('The config of the location was written again from the recovery kit.'); + } $output->writeln(sprintf('Location %s added (id %d), repository %s %s.', $r['target']->getName(), $r['target']->getId(), $r['target']->getRepositoryId(), $r['created'] ? 'created' : 'found and adopted')); + if ($passphrase !== null) { + $output->writeln(($replace ? 'The unused key was replaced. ' : '') . 'Backup key taken from the location, fingerprint ' . $this->keys->fingerprint() . '.'); + $days = 'Deletion delay for forgotten snapshots: ' . $this->keys->deleteDelayDays() . ' days'; + $output->writeln(match (true) { + ($r['deleteDelayDays'] ?? null) !== null => $days . ' (from the location).', + $r['settingsUnreadable'] ?? false => '' . $days . ' (the longest: the settings on the location could not be read or hold no valid delay, so nothing leaves the Bin sooner than on the old server; removed restore points take space for that long).', + default => $days . ' (the default: the location has none stored, its backups were made before NG Backup 1.1.0).', + }); + $output->writeln('Next: occ backup:list and occ backup:restore:full. Before backups run again, download a new recovery kit and confirm it (occ backup:key:kit, occ backup:key:confirm).'); + } return 0; } + private function passphrase(InputInterface $input, OutputInterface $output): ?string { + $file = $input->getOption('passphrase-file'); + if ($file !== null) { + $raw = @file_get_contents((string)$file); + if ($raw === false) { + $output->writeln('Cannot read the passphrase file'); + return null; + } + return rtrim($raw, "\r\n"); + } + if (!$input->isInteractive()) { + $output->writeln('Give the passphrase with --passphrase-file, or run the command without --no-interaction to type it.'); + return null; + } + $q = (new Question('Passphrase of the old server (one of its slots): '))->setHidden(true)->setHiddenFallback(false); + /** @var QuestionHelper $helper */ + $helper = $this->getHelper('question'); + return (string)$helper->ask($input, $output, $q); + } + /** * Progress of reading the index: on a terminal one line that is updated in place, otherwise * (a log, a pipe) a line every 30 seconds. The time left is estimated from the rate so far. diff --git a/lib/Command/TargetList.php b/lib/Command/TargetList.php index 058b10c..0e5e068 100644 --- a/lib/Command/TargetList.php +++ b/lib/Command/TargetList.php @@ -31,7 +31,8 @@ final class TargetList extends Base { $rows = []; foreach ($this->targets->list() as $t) { $row = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(), - 'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt())]; + 'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt()), + 'schedule' => (string)$t->getSchedule() !== '' ? $t->getSchedule() : 'off']; if ($input->getOption('test')) { try { $repo = $this->targets->repository($t); diff --git a/lib/Command/TargetSchedule.php b/lib/Command/TargetSchedule.php new file mode 100644 index 0000000..f5bebfb --- /dev/null +++ b/lib/Command/TargetSchedule.php @@ -0,0 +1,54 @@ +setName('backup:target:schedule') + ->setDescription('Show or set when a location is backed up') + ->addArgument('target', InputArgument::REQUIRED, 'Location name or id') + ->addArgument('schedule', InputArgument::OPTIONAL, '"off", HH:MM for once a day, or HH:MM/N for every N hours from HH:MM; omit to show the current one') + ->addOption('timezone', null, InputOption::VALUE_REQUIRED, 'Time zone of the time, e.g. Europe/Amsterdam'); + } + + protected function execute(InputInterface $input, OutputInterface $output): int { + $t = $this->targets->get($input->getArgument('target')); + $schedule = $input->getArgument('schedule'); + if ($schedule === null && $input->getOption('timezone') !== null) { + $output->writeln('--timezone goes with a schedule, e.g. backup:target:schedule ' . $t->getName() . ' 02:30 --timezone=Europe/Amsterdam'); + return 1; + } + $zone = $input->getOption('timezone') ?? $this->schedule->zoneOr($t->getScheduleZone()); + if ($schedule !== null) { + try { + $t = $this->targets->setSchedule($t, $schedule, (string)$zone); + } catch (\InvalidArgumentException $e) { + $output->writeln('' . $e->getMessage() . ''); + return 1; + } + } + $current = (string)$t->getSchedule(); + $output->writeln($t->getName() . ': ' . ($current === '' ? 'off' : $current . ' (' . $zone . ')')); + return 0; + } +} diff --git a/lib/Controller/AdminApiController.php b/lib/Controller/AdminApiController.php index 37995ec..0466191 100644 --- a/lib/Controller/AdminApiController.php +++ b/lib/Controller/AdminApiController.php @@ -48,6 +48,7 @@ final class AdminApiController extends Controller { private \OCA\NgBackup\Service\KeySlotService $slots, private \OCA\NgBackup\Service\PruneService $prune, private \OCA\NgBackup\Service\Schedule $schedule, + private \OCA\NgBackup\Service\HeavyJobPause $heavyJobs, private \OCA\NgBackup\Service\CronHealth $cron, ) { parent::__construct(Application::APP_ID, $request); @@ -59,10 +60,12 @@ final class AdminApiController extends Controller { foreach ($this->targets->list() as $t) { $targets[] = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(), 'repositoryId' => $t->getRepositoryId(), 'createdAt' => $t->getCreatedAt(), 'appendOnly' => (bool)$t->getAppendOnly(), - 'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t)]; + 'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t), + 'schedule' => (string)$t->getSchedule(), 'scheduleZone' => $this->schedule->zoneOr($t->getScheduleZone())]; } $shown = $this->runs->findShown($allRuns ? null : time() - \OCA\NgBackup\Command\Status::RECENT_DAYS * 86400); - $runs = array_map(fn (Run $r) => $this->runJson($r), $shown); + $queued = array_flip(array_map(static fn (Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1))); + $runs = array_map(fn (Run $r) => $this->runJson($r) + ['queued' => isset($queued[$r->getId()])], $shown); return new JSONResponse([ 'key' => ['initialized' => $this->keys->isInitialized(), 'fingerprint' => $this->keys->fingerprint(), 'confirmation' => $this->keys->recoveryKitConfirmation(), 'everConfirmed' => $this->keys->isInitialized() && $this->keys->everConfirmed(), @@ -74,12 +77,16 @@ final class AdminApiController extends Controller { 'runs' => $runs, 'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown), 'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS, - 'schedule' => $this->schedule->get()['time'], - 'scheduleTimezone' => $this->schedule->get()['timezone'], 'cron' => $this->cron->check(), + 'heavyJobs' => $this->heavyJobs->status(), ]); } + public function setPauseHeavyJobs(bool $enabled): JSONResponse { + $this->heavyJobs->setEnabled($enabled); + return new JSONResponse(['enabled' => $enabled]); + } + /** Seconds of backup work one tick() does: well within a normal web request. */ public const TICK_SECONDS = 10.0; /** Seconds one request for the Bin reads, before the page shows how far it is and asks for more. */ @@ -97,16 +104,17 @@ final class AdminApiController extends Controller { } $deadline = microtime(true) + self::TICK_SECONDS; $stepped = 0; - $running = $this->runs->findRunning(null, BackupService::KIND_FULL); - // Longest wait first: a run that takes all of a tick does not keep another from its turn. - usort($running, static fn ($a, $b) => $a->getUpdatedAt() <=> $b->getUpdatedAt()); - foreach ($running as $run) { + $this->heavyJobs->sync($this->backups->runsInTurn() !== []); + foreach ($this->backups->runsInTurn() as $run) { if (microtime(true) >= $deadline) { break; } // step() returns at once when the background job (or another tab) holds the run. - $this->backups->step($run, $deadline); + $next = $this->backups->step($run, $deadline); $stepped++; + if ($next->getStatus() === Run::RUNNING) { + break; // not finished: the next one waits for its turn + } } return new JSONResponse(['stepped' => $stepped]); } @@ -278,25 +286,43 @@ final class AdminApiController extends Controller { return new JSONResponse($this->runJson($run)); } - /** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */ - public function setSchedule(string $time, string $timezone = ''): JSONResponse { + /** @param string $timezone the administrator's zone (the browser's); empty: the default one */ + public function setTargetSchedule(int $id, string $schedule, string $timezone = ''): JSONResponse { try { - $this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']); - } catch (\InvalidArgumentException $e) { + $t = $this->targets->setSchedule($this->targets->get((string)$id), $schedule, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']); + } catch (\Throwable $e) { return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST); } - return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]); + return new JSONResponse(['schedule' => $t->getSchedule(), 'scheduleZone' => $t->getScheduleZone()]); } public function snapshots(int $targetId): JSONResponse { $out = []; foreach ($this->snapshots->findForTarget($targetId) as $s) { $out[] = ['id' => $s->getSnapshotId(), 'kind' => $s->getKind(), 'label' => $s->getLabel(), 'createdAt' => $s->getCreatedAt(), - 'files' => $s->getFiles(), 'bytes' => $s->getBytes()]; + 'files' => $s->getFiles(), 'bytes' => $s->getBytes(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()]; } return new JSONResponse($out); } + public function setSnapshotNote(int $targetId, string $snapshotId, string $note = ''): JSONResponse { + try { + $this->prune->setNote($this->targets->get((string)$targetId), $snapshotId, $note); + } catch (\Throwable $e) { + return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST); + } + return new JSONResponse(['ok' => true]); + } + + public function setSnapshotPinned(int $targetId, string $snapshotId, bool $pinned = true): JSONResponse { + try { + $this->prune->setPinned($this->targets->get((string)$targetId), $snapshotId, $pinned); + } catch (\Throwable $e) { + return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST); + } + return new JSONResponse(['ok' => true]); + } + public function browse(int $targetId, string $snapshotId, string $path = ''): JSONResponse { try { return new JSONResponse($this->restore->browse($this->targets->get((string)$targetId), $snapshotId, $path)); @@ -341,7 +367,7 @@ final class AdminApiController extends Controller { if ($r->getSnapshotId() !== null) { try { $s = $this->snapshots->findOne($r->getTargetId(), $r->getSnapshotId()); - $snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel()]; + $snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()]; } catch (\Throwable) { // Snapshot already pruned or trashed: the run keeps its id, just not the date. } diff --git a/lib/Crypto/KeyRing.php b/lib/Crypto/KeyRing.php index 92de548..9b41d70 100644 --- a/lib/Crypto/KeyRing.php +++ b/lib/Crypto/KeyRing.php @@ -87,14 +87,20 @@ final class KeyRing { foreach ($field['slots'] as $slot) { try { return self::unwrap($slot['key'], $passphrase); - } catch (CryptoException) { + } catch (CryptoException|\SodiumException|\TypeError|\ValueError) { + // a wrong passphrase for this slot, or a damaged slot: the next one may still open } } throw new CryptoException('Wrong passphrase or damaged key'); } - /** @param array{v:int, salt:string, ops:int, mem:int, nonce:string, key:string} $wrapped */ + /** @param array $wrapped from a kit file or a location's config: checked here, not trusted */ public static function unwrap(array $wrapped, #[\SensitiveParameter] string $passphrase): self { + if (!isset($wrapped['salt'], $wrapped['nonce'], $wrapped['key'], $wrapped['ops'], $wrapped['mem']) + || !is_string($wrapped['salt']) || !is_string($wrapped['nonce']) || !is_string($wrapped['key']) + || !is_int($wrapped['ops']) || !is_int($wrapped['mem'])) { + throw new CryptoException('Damaged key file: a part of the key is missing'); + } $b64 = static fn (string $s): string => sodium_base642bin($s, SODIUM_BASE64_VARIANT_ORIGINAL); // ops/mem come from unauthenticated JSON (kit file, repository config): bound them, so a // damaged or crafted file cannot make the KDF use unbounded memory or time. @@ -102,10 +108,22 @@ final class KeyRing { || $wrapped['mem'] < SODIUM_CRYPTO_PWHASH_MEMLIMIT_INTERACTIVE || $wrapped['mem'] > SODIUM_CRYPTO_PWHASH_MEMLIMIT_SENSITIVE) { throw new CryptoException('Unsupported key derivation parameters (damaged or foreign key file)'); } - $kek = sodium_crypto_pwhash(SODIUM_CRYPTO_SECRETBOX_KEYBYTES, $passphrase, $b64($wrapped['salt']), - $wrapped['ops'], $wrapped['mem'], SODIUM_CRYPTO_PWHASH_ALG_ARGON2ID13); - $master = sodium_crypto_secretbox_open($b64($wrapped['key']), $b64($wrapped['nonce']), $kek); - sodium_memzero($kek); + $kek = null; + try { + $salt = $b64($wrapped['salt']); + $sealed = $b64($wrapped['key']); + $nonce = $b64($wrapped['nonce']); + $kek = sodium_crypto_pwhash(SODIUM_CRYPTO_SECRETBOX_KEYBYTES, $passphrase, $salt, + $wrapped['ops'], $wrapped['mem'], SODIUM_CRYPTO_PWHASH_ALG_ARGON2ID13); + $master = sodium_crypto_secretbox_open($sealed, $nonce, $kek); + } catch (\SodiumException|\ValueError $e) { + // invalid base64 or a salt/nonce of the wrong length: damaged, so the caller shows its recovery options + throw new CryptoException('Damaged key file: ' . $e->getMessage(), 0, $e); + } finally { + if ($kek !== null) { + sodium_memzero($kek); + } + } if ($master === false) { throw new CryptoException('Wrong passphrase or damaged key'); } diff --git a/lib/Db/Snapshot.php b/lib/Db/Snapshot.php index 55eb769..47d8014 100644 --- a/lib/Db/Snapshot.php +++ b/lib/Db/Snapshot.php @@ -24,6 +24,10 @@ use OCP\AppFramework\Db\Entity; * @method void setFiles(?int $v) * @method int|null getBytes() * @method void setBytes(?int $v) + * @method string|null getNote() + * @method void setNote(?string $v) + * @method bool|null getPinned() + * @method void setPinned(?bool $v) */ final class Snapshot extends Entity { protected int $targetId = 0; @@ -33,8 +37,11 @@ final class Snapshot extends Entity { protected int $createdAt = 0; protected ?int $files = null; protected ?int $bytes = null; + protected ?string $note = null; + protected ?bool $pinned = null; public function __construct() { + $this->addType('pinned', 'boolean'); $this->addType('targetId', 'integer'); $this->addType('createdAt', 'integer'); $this->addType('files', 'integer'); diff --git a/lib/Db/Target.php b/lib/Db/Target.php index 0930a62..a4d924c 100644 --- a/lib/Db/Target.php +++ b/lib/Db/Target.php @@ -28,6 +28,12 @@ use OCP\AppFramework\Db\Entity; * @method void setMaxBytes(?int $v) * @method int getCreatedAt() * @method void setCreatedAt(int $v) + * @method string|null getSchedule() + * @method void setSchedule(?string $v) + * @method string|null getScheduleZone() + * @method void setScheduleZone(?string $v) + * @method int|null getScheduleSince() + * @method void setScheduleSince(?int $v) */ final class Target extends Entity { protected string $name = ''; @@ -39,8 +45,12 @@ final class Target extends Entity { protected int $createdAt = 0; protected ?bool $appendOnly = null; protected ?int $maxBytes = null; + protected ?string $schedule = null; + protected ?string $scheduleZone = null; + protected ?int $scheduleSince = null; public function __construct() { + $this->addType('scheduleSince', 'integer'); $this->addType('createdAt', 'integer'); $this->addType('appendOnly', 'boolean'); $this->addType('maxBytes', 'integer'); diff --git a/lib/Job/BackupRun.php b/lib/Job/BackupRun.php index 4a0c379..19dbc51 100644 --- a/lib/Job/BackupRun.php +++ b/lib/Job/BackupRun.php @@ -52,19 +52,35 @@ final class BackupRun { $stats = $state['stats']; $walker = new TreeWalker($state['roots'], $state['exclude'], function (string $logical) use (&$state, &$stats): void { self::recordUnreadable($state, $stats, $logical); - }); + }, true); $parent = new ParentCursor($repo, $state['parent'], $state['parentPos']); $lines = []; $emptyDirs = []; + $folders = []; $finishedWalk = true; foreach ($walker->walk($state['after']) as $logical => $path) { - if (!mb_check_encoding($logical, 'UTF-8') || !Repository::isSafePath($logical)) { + $safe = mb_check_encoding($logical, 'UTF-8') && Repository::isSafePath($logical); + if ($walker->atFolderWithContent()) { + // Only its time: what is in it is reported by itself, and a bad name in the path of + // each of those is what gets reported. The cursor stays for such a name, see below. + $mtime = $safe ? @filemtime($path) : false; + if ($safe && $mtime !== false) { + $folders[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR); + $state['after'] = $logical; + if (microtime(true) >= $deadline) { + $finishedWalk = false; + break; + } + } + continue; + } + if (!$safe) { // A name a snapshot cannot hold (a backslash, a control character, not UTF-8). One // such path must not make a backup fail, or the whole snapshot unreadable later. // The cursor stays where it is: the state is JSON, which cannot hold such a name. A // next step meets it again, and it is reported once. - self::recordUnreadable($state, $stats, mb_scrub($logical, 'UTF-8')); + self::recordBadName($state, $stats, $logical); continue; } if (is_dir($path)) { @@ -104,7 +120,7 @@ final class BackupRun { if ($cur === null || $cur['path'] !== $logical) { $cur = ['path' => $logical, 'offset' => 0, 'blobs' => [], 'version' => $version]; } - $fh = @fopen($path, 'rb'); + $fh = Repository::openForReading($path); if ($fh === false) { self::recordUnreadable($state, $stats, $logical); $state['cur'] = null; @@ -164,6 +180,11 @@ final class BackupRun { $writer->write(implode("\n", $emptyDirs) . "\n"); $state['dirSegments'][] = $writer->finish(); } + if ($folders !== []) { + $writer = $repo->blobWriter($stats); + $writer->write(implode("\n", $folders) . "\n"); + $state['folderSegments'][] = $writer->finish(); + } $repo->flushPacks(); $state['parentPos'] = $parent->position(); $state['stats'] = $stats; @@ -186,10 +207,14 @@ final class BackupRun { $emptyDirs[] = json_decode($line, true, 8, JSON_THROW_ON_ERROR); } } + // The times of the other folders too, packed: every look at a snapshot's meta record reads + // this, and a string of a few hundred KiB costs that far less than tens of thousands of arrays. + $folders = isset($state['folderSegments']) ? implode("\n", iterator_to_array($repo->readLines(array_values(array_merge(...$state['folderSegments']))), false)) : ''; $repo->putSnapshot($state['snapshot'], ['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'], 'roots' => array_keys($state['roots']), 'stats' => $state['stats'], - 'warnings' => self::warnings($state)] + ($emptyDirs === [] ? [] : ['emptyDirs' => $emptyDirs]) + $state['meta'], + 'warnings' => self::warnings($state)] + ($emptyDirs === [] ? [] : ['emptyDirs' => $emptyDirs]) + + ($folders === '' ? [] : ['folderTimes' => base64_encode((string)gzdeflate($folders))]) + $state['meta'], $repo->readLines(array_values(array_merge(...$state['segments'])))); $state['phase'] = 'commit'; } @@ -250,10 +275,26 @@ final class BackupRun { $stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1; } - /** @return array{unreadable: list, changing: list} examples for the report (first 50 each) */ + /** + * A file or directory left out because of its name. Remembered like an unreadable one, in a form + * that fits the JSON state and is safe to show: control characters and invalid bytes replaced. + */ + private static function recordBadName(array &$state, array &$stats, string $logical): void { + $shown = (string)preg_replace('/[\x00-\x1f\x7f]/', '?', mb_scrub($logical, 'UTF-8')); + if (isset($state['badNames'][$shown])) { + return; + } + if (count($state['badNames'] ?? []) < 1000) { + $state['badNames'][$shown] = true; + } + $stats['badNames'] = ($stats['badNames'] ?? 0) + 1; + } + + /** @return array{unreadable: list, badNames: list, changing: list} examples for the report (first 50 each) */ public static function warnings(array $state): array { return [ 'unreadable' => array_slice(array_map('strval', array_keys($state['unreadable'] ?? [])), 0, 50), + 'badNames' => array_slice(array_map('strval', array_keys($state['badNames'] ?? [])), 0, 50), 'changing' => array_slice(array_map('strval', array_values($state['changing'] ?? [])), 0, 50), ]; } diff --git a/lib/Job/TreeWalker.php b/lib/Job/TreeWalker.php index 25366dc..bf35c88 100644 --- a/lib/Job/TreeWalker.php +++ b/lib/Job/TreeWalker.php @@ -17,6 +17,8 @@ namespace OCA\NgBackup\Job; * and be merge-joined with the previous snapshot without holding any list in memory. */ final class TreeWalker { + private bool $atFolder = false; + /** * @param array $roots name => absolute directory, walked in this order * @param list $exclude path prefixes ("/") that are skipped entirely; @@ -27,9 +29,16 @@ final class TreeWalker { private array $exclude = [], /** Called with the logical path of a directory that cannot be listed. */ private ?\Closure $onUnreadable = null, + /** Also yield directories with something in them, right before the first thing in them. */ + private bool $withFolders = false, ) { } + /** Whether what was just yielded is a directory with something in it (see $withFolders). */ + public function atFolderWithContent(): bool { + return $this->atFolder; + } + /** @return \Generator logical path => absolute path, strictly after $after */ public function walk(?string $after = null): \Generator { foreach ($this->roots as $name => $dir) { @@ -75,7 +84,14 @@ final class TreeWalker { if (is_link($childAbs)) { continue; // never follow symlinks out of the data directory } - if (is_dir($childAbs)) { + $childIsDir = is_dir($childAbs); + if (($childIsDir || is_file($childAbs)) && $empty && !$this->isExcluded($childLogical, $childIsDir) + && $this->withFolders && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) { + $this->atFolder = true; + yield $logical => $abs; + $this->atFolder = false; + } + if ($childIsDir) { $empty = $empty && $this->isExcluded($childLogical, true); // Skip a directory entirely if everything in it sorts before the cursor. if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) { diff --git a/lib/Migration/Version000007Date20261107000000.php b/lib/Migration/Version000007Date20261107000000.php new file mode 100644 index 0000000..5c8dca0 --- /dev/null +++ b/lib/Migration/Version000007Date20261107000000.php @@ -0,0 +1,33 @@ +getTable('ngb_snapshots'); + $changed = false; + if (!$t->hasColumn('note')) { + $t->addColumn('note', Types::STRING, ['notnull' => false, 'length' => 1000]); + $changed = true; + } + if (!$t->hasColumn('pinned')) { + $t->addColumn('pinned', Types::BOOLEAN, ['notnull' => false, 'default' => false]); + $changed = true; + } + return $changed ? $schema : null; + } +} diff --git a/lib/Migration/Version000008Date20261107010000.php b/lib/Migration/Version000008Date20261107010000.php new file mode 100644 index 0000000..ea4ed14 --- /dev/null +++ b/lib/Migration/Version000008Date20261107010000.php @@ -0,0 +1,49 @@ +getTable('ngb_targets'); + if ($t->hasColumn('schedule')) { + return null; + } + $t->addColumn('schedule', Types::STRING, ['notnull' => false, 'length' => 16]); + $t->addColumn('schedule_zone', Types::STRING, ['notnull' => false, 'length' => 64]); + $t->addColumn('schedule_since', Types::BIGINT, ['notnull' => false, 'unsigned' => true]); + return $schema; + } + + public function postSchemaChange(IOutput $output, Closure $schemaClosure, array $options): void { + ['time' => $time, 'timezone' => $zone] = $this->schedule->get(); + $qb = $this->db->getQueryBuilder(); + $qb->update('ngb_targets') + ->set('schedule', $qb->createNamedParameter(Schedule::validTime($time) ? LocationSchedule::normalize($time) : '')) + ->set('schedule_zone', $qb->createNamedParameter($zone)) + ->where($qb->expr()->isNull('schedule')); + $qb->executeStatement(); + } +} diff --git a/lib/Migration/Version000009Date20261108000000.php b/lib/Migration/Version000009Date20261108000000.php new file mode 100644 index 0000000..bcc6ac4 --- /dev/null +++ b/lib/Migration/Version000009Date20261108000000.php @@ -0,0 +1,31 @@ +keys->isInitialized()) { + $this->keys->markUsed(); + } + } +} diff --git a/lib/Notification/Notifier.php b/lib/Notification/Notifier.php index c1d05ae..69b9581 100644 --- a/lib/Notification/Notifier.php +++ b/lib/Notification/Notifier.php @@ -46,6 +46,10 @@ final class Notifier implements INotifier { $notification->setParsedSubject($l->t('Backup to %1$s finished with warnings: %2$s unreadable, %3$s changed while being read', [$p['target'], $p['unreadable'], $p['changing']])); $notification->setParsedMessage($l->t('Unreadable files and folders are missing from this backup (check their permissions). Files that kept changing were saved as they were while being read. Examples: %s', [$p['examples'] ?? ''])); break; + case 'run_bad_names': + $notification->setParsedSubject($l->n('Backup to %s finished, but %n item is missing because of its name', 'Backup to %s finished, but %n items are missing because of their names', (int)$p['count'], [$p['target']])); + $notification->setParsedMessage($l->t('Names with a backslash or a control character, or that are not valid UTF-8, are not supported, so these files and folders are not in this backup. Rename them; the next backup includes them. Examples: %s', [$p['examples'] ?? ''])); + break; case 'quota_warning': $notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']])); $notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.')); @@ -57,13 +61,21 @@ final class Notifier implements INotifier { : $l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']])); $notification->setParsedMessage($l->t('Check the backup settings and the server log.')); break; + case 'schedule_off': + $notification->setParsedSubject($l->t('Backup location %s has no schedule', [$p['target']])); + $notification->setParsedMessage(($p['others'] ?? '') === '1' + ? $l->t('No backups run to this location until it gets a schedule or someone starts one. If that is what you want, you can ignore this reminder.') + : $l->t('No location has a schedule, so no backups run at all unless someone starts one. Set a schedule under Backups.')); + break; case 'security_event': $who = (string)($p['actor'] ?? ''); $notification->setParsedSubject(match ($p['event'] ?? '') { 'kit_downloaded' => $l->t('%s downloaded the backup recovery kit', [$who]), + 'key_replaced' => $l->t('%1$s replaced the unused backup key %2$s by the key of existing backups, %3$s; download and confirm the new recovery kit', [$who, $p['old'] ?? '', $p['new'] ?? '']), 'target_removed' => $l->t('%1$s removed backup location %2$s', [$who, $p['target'] ?? '']), 'target_changed' => $l->t('%1$s changed the connection settings of backup location %2$s', [$who, $p['target'] ?? '']), - 'append_only_off' => $l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']), + 'schedule_off' => $l->t('%1$s turned off the schedule of backup location %2$s', [$who, $p['target'] ?? '']), + 'append_only_off' =>$l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']), 'retention_changed' => $l->t('%1$s changed the backup retention to %2$s', [$who, $p['policy'] ?? '']), 'snapshots_forgotten' => $l->t('%1$s moved %2$s snapshots of %3$s to the trash; they are deleted permanently after %4$s unless restored', [$who, $p['count'] ?? '', $p['target'] ?? '', $p['until'] ?? '']), 'snapshot_untrashed' => $l->t('%1$s restored snapshot %2$s of %3$s from the trash', [$who, $p['snapshot'] ?? '', $p['target'] ?? '']), diff --git a/lib/Repository/Catalog.php b/lib/Repository/Catalog.php index 7dae152..328cb68 100644 --- a/lib/Repository/Catalog.php +++ b/lib/Repository/Catalog.php @@ -30,6 +30,10 @@ final class Catalog { public array $trash = []; /** @var array manifest path => true, for a per-user export (users//) */ public array $userExports = []; + /** @var array snapshots the retention policy leaves alone */ + public array $pinned = []; + /** @var array snapshot id => note, also while the snapshot is in the trash */ + public array $notes = []; public string $hash = ''; public function __construct( @@ -82,6 +86,8 @@ final class Catalog { // Catalogs written before user exports existed have no such key: default to empty rather // than adopting the location's current users/ listing (that would skip verification). $this->userExports = array_fill_keys($data['userExports'] ?? [], true); + $this->pinned = array_fill_keys($data['pinned'] ?? [], true); + $this->notes = $data['notes'] ?? []; $this->anchor?->set($this->repositoryId, $this->gen, $this->hash); } @@ -93,9 +99,12 @@ final class Catalog { if ($this->backend->exists($this->path($next))) { throw new RepositoryException("Catalog generation $next was written by another process at the same time; try again"); } + // A pin ends when its snapshot leaves the list, a note when the snapshot is gone for good. + $this->pinned = array_intersect_key($this->pinned, $this->snapshots); + $this->notes = array_intersect_key($this->notes, $this->snapshots + $this->trash); $plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash, 'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash), - 'userExports' => array_keys($this->userExports)], JSON_THROW_ON_ERROR); + 'userExports' => array_keys($this->userExports), 'pinned' => array_keys($this->pinned), 'notes' => $this->notes], JSON_THROW_ON_ERROR); $encrypted = $this->cipher->encryptString($plain, $this->ad($next)); $stream = fopen('php://memory', 'w+b'); fwrite($stream, $encrypted); diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index a631000..262a825 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -18,6 +18,7 @@ use OCA\NgBackup\Crypto\StreamCipher; * * Layout on the backend: * config plaintext JSON: format version, repository id, wrapped master key + * settings encrypted JSON: settings fixed at installation (deletion delay) * packs/xx/ encrypted blobs + encrypted pack header * index/ encrypted list of the blobs in that pack * snapshots/ encrypted JSON lines: one meta line, then one line per file @@ -28,6 +29,7 @@ use OCA\NgBackup\Crypto\StreamCipher; final class Repository { public const FORMAT = 1; public const BLOB_SIZE = 4 * 1048576; + private const SETTINGS = 'settings'; private const FLAG_DEFLATE = 1; /** Blobs from this size on get a quick compression test before the full compression. */ private const DEFLATE_PROBE_FROM = 256 * 1024; @@ -79,7 +81,7 @@ final class Repository { // opens this data; refuse instead. foreach (['index', 'snapshots', 'trash/snapshots', 'catalog'] as $dir) { if ($backend->listFiles($dir) !== []) { - throw new RepositoryException("This location holds backup data ($dir/) but no repository config could be read: not creating a new repository over it. Check the connection and the folder; a missing config can be restored from the recovery kit."); + throw new RepositoryException("This location holds backup data ($dir/) but no repository config could be read: not creating a new repository over it. Check the connection and the folder. If the config is really gone, occ backup:target:add with --repair-config writes it again with this server's key (on a new server: import the recovery kit first, occ backup:key:import)."); } } $id = bin2hex(random_bytes(16)); @@ -95,12 +97,7 @@ final class Repository { /** @param ?callable(int, int): void $progress index files downloaded so far, and in all (see BlobIndex::load()) */ public static function openWithKey(IBackend $backend, KeyRing $keys, ?IIndexCache $cache = null, ?ICatalogAnchor $anchor = null, ?callable $progress = null): self { $config = self::readConfig($backend); - try { - $ok = isset($config['check']) && (new StreamCipher($keys))->decryptString((string)base64_decode($config['check'], true), 'config-check') === $config['id']; - } catch (\OCA\NgBackup\Crypto\CryptoException) { - $ok = false; - } - if (!$ok) { + if (!self::configMatches($config, $keys)) { throw new RepositoryException('This repository was created with a different key'); } $repo = new self($backend, $keys, $config['id']); @@ -134,6 +131,49 @@ final class Repository { return $this->catalog->gen; } + /** @return list snapshots the retention policy leaves alone */ + public function pinnedSnapshots(): array { + return array_map('strval', array_keys($this->catalog->pinned)); + } + + /** @return array snapshot id => note */ + public function snapshotNotes(): array { + return $this->catalog->notes; + } + + public function setPinned(string $snapshotId, bool $pinned): void { + $this->requireSnapshot($snapshotId); + if (isset($this->catalog->pinned[$snapshotId]) === $pinned) { + return; + } + if ($pinned) { + $this->catalog->pinned[$snapshotId] = true; + } else { + unset($this->catalog->pinned[$snapshotId]); + } + $this->catalog->write(); + } + + /** A note kept in the catalog: unlike the label inside the snapshot, it can change afterwards. */ + public function setSnapshotNote(string $snapshotId, string $note): void { + $this->requireSnapshot($snapshotId); + if (($this->catalog->notes[$snapshotId] ?? '') === $note) { + return; + } + if ($note === '') { + unset($this->catalog->notes[$snapshotId]); + } else { + $this->catalog->notes[$snapshotId] = $note; + } + $this->catalog->write(); + } + + private function requireSnapshot(string $snapshotId): void { + if (!isset($this->catalog->snapshots[$snapshotId])) { + throw new \InvalidArgumentException('There is no snapshot ' . $snapshotId . ' on this location'); + } + } + /** Index files downloaded when this repository was opened (0 when the local cache was complete). */ public function indexDownloads(): int { return $this->index->downloaded; @@ -153,6 +193,114 @@ final class Repository { return $config; } + /** + * Open a local file to read it in whole blobs. PHP reads a file in pieces of 8 KiB by default; + * on a disk where each request waits (a network or cloud volume) that made reading five times + * slower: 5 MiB/s instead of 27 MiB/s on nc01. Without PHP's read buffer every fread() of a + * blob is one request to the disk. + * + * @return resource|false + */ + public static function openForReading(string $path) { + $fh = @fopen($path, 'rb'); + if ($fh !== false) { + stream_set_read_buffer($fh, 0); + stream_set_chunk_size($fh, self::BLOB_SIZE); + } + return $fh; + } + + private static function configMatches(array $config, KeyRing $keys): bool { + try { + return isset($config['check'], $config['id']) + && (new StreamCipher($keys))->decryptString((string)base64_decode((string)$config['check'], true), 'config-check') === $config['id']; + } catch (\OCA\NgBackup\Crypto\CryptoException) { + return false; + } + } + + /** + * The passphrase-wrapped key slots in the config at this location: disaster recovery without + * the recovery kit, where any one of their passphrases gives back the master key. + */ + public static function wrappedKeyAt(IBackend $backend): array { + $config = self::readConfig($backend); + if (!isset($config['key']) || !is_array($config['key'])) { + throw new RepositoryException('The repository config at this location holds no key'); + } + return $config['key']; + } + + /** Whether the repository at this location was created with this key. Does not read the index. */ + public static function madeWithKey(IBackend $backend, KeyRing $keys): bool { + return self::configMatches(self::readConfig($backend), $keys); + } + + /** + * Settings of the installation that never change (the deletion delay), sealed with the + * master key so that, unlike the config and the recovery kit, nobody can alter them unseen. + * Empty when the location has none yet (made before 1.1.0, until its next backup); only + * ['unreadable' => true] when they are there but cannot be read. + * + * @return array + */ + public static function settingsAt(IBackend $backend, KeyRing $keys): array { + try { + if (!$backend->exists(self::SETTINGS)) { + return []; + } + $fh = $backend->get(self::SETTINGS); + $raw = (string)stream_get_contents($fh); + fclose($fh); + $settings = json_decode((new StreamCipher($keys))->decryptString($raw, 'settings'), true, 16, JSON_THROW_ON_ERROR); + return is_array($settings) ? $settings : ['unreadable' => true]; + } catch (BackendException|\OCA\NgBackup\Crypto\CryptoException|\JsonException) { + return ['unreadable' => true]; + } + } + + /** + * Write the settings once; a location that has them keeps them. The repository id goes with + * them: with it and the key from the recovery kit, a lost config can be written again. + */ + public function recordSettings(array $settings): void { + if ($this->backend->exists(self::SETTINGS)) { + return; + } + $settings['repository'] = $this->repositoryId; + self::putString($this->backend, self::SETTINGS, $this->cipher->encryptString(json_encode($settings, JSON_THROW_ON_ERROR), 'settings')); + } + + /** Whether the location holds backup data, with or without a config. */ + public static function holdsData(IBackend $backend): bool { + foreach (['index', 'snapshots', 'catalog'] as $dir) { + if ($backend->listFiles($dir) !== []) { + return true; + } + } + return false; + } + + /** + * Write a lost config again, from the key (recovery kit) and the repository id in the + * sealed settings. Only where there is no config and the settings open with this key, so + * nothing that works is ever overwritten and a wrong key writes nothing. + */ + public static function repairConfig(IBackend $backend, KeyRing $keys, array $wrappedKey): void { + if ($backend->exists('config')) { + throw new RepositoryException('This location has a config: nothing to repair, leave out --repair-config'); + } + $id = self::settingsAt($backend, $keys)['repository'] ?? null; + if (!is_string($id) || !preg_match('/^[0-9a-f]{32}$/', $id)) { + throw new RepositoryException($backend->exists(self::SETTINGS) + ? 'The settings at this location do not open with this key: the recovery kit is of another server, or the settings are damaged. The config cannot be written again.' + : 'This location has no settings file (it was backed up only by NG Backup before 1.1.0), so its config cannot be written again.'); + } + $config = ['format' => self::FORMAT, 'id' => $id, 'created' => gmdate('c'), 'key' => $wrappedKey, + 'check' => base64_encode((new StreamCipher($keys))->encryptString($id, 'config-check')), 'repaired' => true]; + self::putString($backend, 'config', json_encode($config, JSON_PRETTY_PRINT | JSON_THROW_ON_ERROR)); + } + /** Replace the passphrase-wrapped key slots in this repository's config (after a slot change). */ public function updateWrappedKey(array $wrappedKey): void { $config = self::readConfig($this->backend); @@ -392,9 +540,41 @@ final class Repository { self::restoreEmptyDir($target, $dest, $path, $mtime); } } + // Very last, nothing is written in them any more. Also on a run that goes on after a stop, and + // the deepest first. The time is not worth stopping for: all the files are there by now. + foreach (array_reverse($this->folderTimes($snapshotId)) as [$path, $mtime]) { + if (!self::underPrefix($path, $prefix) || ($stripPrefix && $path === rtrim($prefix, '/'))) { + continue; // the target itself is not ours to change + } + if ($heartbeat !== null) { + $heartbeat(); + } + $dest = self::destination($target, $path, $prefix, $stripPrefix); + @touch(self::directoryInTarget($target, $dest, $path), $mtime); + } return $count; } + /** + * The folders with something in them that a snapshot recorded, parents before children (none + * before 1.1.0). Damage here is no reason to give up on the files of a restore. + * + * @return list path, modification time + */ + public function folderTimes(string $snapshotId): array { + $meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId); + $packed = is_string($meta['folderTimes'] ?? null) ? base64_decode($meta['folderTimes'], true) : false; + $plain = $packed === false ? false : @gzinflate($packed); + $folders = []; + foreach ($plain === false ? [] : explode("\n", $plain) as $line) { + $folder = json_decode($line, true); + if (is_array($folder) && is_string($folder[0] ?? null) && self::isSafePath($folder[0]) && is_int($folder[1] ?? null)) { + $folders[] = [$folder[0], $folder[1]]; + } + } + return $folders; + } + /** * The folders without anything in them that a snapshot recorded (none before 1.0.3). * @@ -985,7 +1165,10 @@ final class Repository { for ($attempt = 1; $attempt <= 3; $attempt++) { $before = self::fileVersion($path); $blobs = []; - $fh = fopen($path, 'rb'); + $fh = self::openForReading($path); + if ($fh === false) { + throw new RepositoryException('Cannot read ' . $path); + } while (($data = self::readFull($fh, self::BLOB_SIZE)) !== '') { $blobs[] = $this->storeBlob($data, $packs, $stats); } diff --git a/lib/Service/AlertService.php b/lib/Service/AlertService.php index ace1dc8..8bc249e 100644 --- a/lib/Service/AlertService.php +++ b/lib/Service/AlertService.php @@ -61,6 +61,16 @@ final class AlertService { 'examples' => mb_strimwidth(implode(', ', array_merge($examples['unreadable'], $examples['changing'])), 0, 300, '…')]); } + /** + * A backup that finished but left out files or folders because of their name. + * + * @param list $examples + */ + public function runBadNames(Run $run, string $targetName, int $count, array $examples): void { + $this->notifyAdmins('run_bad_names', 'run-' . $run->getId(), ['target' => $targetName, 'count' => (string)$count, + 'examples' => mb_strimwidth(implode(', ', $examples), 0, 300, '…')]); + } + /** * Remember how much is stored on a location (for the settings page) and warn administrators * once a day when it is at 80% of its limit or more. @@ -91,9 +101,36 @@ final class AlertService { return $v === '' ? null : (int)$v; } + /** + * Whether it is time to remind administrators of a location without a schedule: once a week + * while another location is still backed up, every day when none is. An hour of margin, so + * that the moment of the cron run does not push a reminder to the next day. + */ + public static function offReminderDue(int $last, int $now, bool $othersActive): bool { + return $now - $last >= ($othersActive ? 7 * 86400 : 86400) - 3600; + } + + public function checkOff(Target $target, bool $othersActive): void { + if ((string)$target->getSchedule() !== '') { + return; + } + $key = 'off_notified_' . $target->getId(); + $now = time(); + $last = max((int)$target->getScheduleSince(), $this->appConfig->getValueInt(Application::APP_ID, $key, 0)); + if (!self::offReminderDue($last, $now, $othersActive)) { + return; + } + $this->appConfig->setValueInt(Application::APP_ID, $key, $now); + $this->notifyAdmins('schedule_off', 'off-' . $target->getId() . '-' . date('Y-m-d', $now), + ['target' => $target->getName(), 'others' => $othersActive ? '1' : '0']); + } + /** At most one notification per location per day. */ public function checkStale(Target $target): void { - $last = $this->runs->lastFinished($target->getId(), BackupService::KIND_FULL); + if ((string)$target->getSchedule() === '') { + return; // no schedule is a choice, as for a location that is only connected now and then + } + $last =$this->runs->lastFinished($target->getId(), BackupService::KIND_FULL); $lastTime = $last?->getFinishedAt() ?? $target->getCreatedAt(); if (time() - $lastTime < self::STALE_AFTER || $this->runs->findRunning($target->getId(), BackupService::KIND_FULL) !== []) { return; diff --git a/lib/Service/BackupService.php b/lib/Service/BackupService.php index 0d45f55..817ff05 100644 --- a/lib/Service/BackupService.php +++ b/lib/Service/BackupService.php @@ -58,6 +58,18 @@ final class BackupService { $this->openedWith = $opened; } + /** + * The running backups in the order they get their turn: first come, first served. The first one + * runs, the others are queued; only one at a time reads the disks. + * + * @return list + */ + public function runsInTurn(): array { + $running = $this->runs->findRunning(null, self::KIND_FULL); + usort($running, static fn (Run $a, Run $b): int => [$a->getStartedAt(), $a->getId()] <=> [$b->getStartedAt(), $b->getId()]); + return $running; + } + public function start(Target $target, string $label = ''): Run { $label = trim($label); if (mb_strlen($label) > self::LABEL_MAX) { @@ -210,6 +222,13 @@ final class BackupService { $state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR); if ($run->getPhase() === 'db') { + if (!isset($state['dump'])) { + // It may have been queued for hours: the restore point has the time of its data. + $state['files']['started'] = gmdate('c'); + // Locations from before 1.1.0 get their sealed settings at their next backup. + $this->targets->recordSettings($repo); + $this->targets->catchUpWrappedKey($target, $repo); + } $dumper = new DbDumper($this->db->getInner(), $this->config->getSystemValueString('dbtableprefix', 'oc_')); $dbStats = $state['dbStats'] ?? []; // With system cron the whole dump usually fits in one step (one transaction, fully @@ -253,8 +272,13 @@ final class BackupService { try { $this->alerts->storedBytes($target, $repo->storedBytes()); $stats = $state['files']['stats']; + $warnings = BackupRun::warnings($state['files']); + if (($stats['badNames'] ?? 0) > 0) { + $this->logger->warning(sprintf('NG Backup run %d to %s: %d left out because their name is not supported (rename them): %s', + $run->getId(), $target->getName(), $stats['badNames'], implode(', ', $warnings['badNames']))); + $this->alerts->runBadNames($run, $target->getName(), (int)$stats['badNames'], $warnings['badNames']); + } if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) { - $warnings = BackupRun::warnings($state['files']); $this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s', $run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']), $stats['changing'] ?? 0, implode(', ', $warnings['changing']))); diff --git a/lib/Service/HeavyJobPause.php b/lib/Service/HeavyJobPause.php new file mode 100644 index 0000000..6a9ad3d --- /dev/null +++ b/lib/Service/HeavyJobPause.php @@ -0,0 +1,156 @@ + what to call it */ + private const JOBS = [ + 'OCA\\Memories\\' => 'Memories', + 'OCA\\Files_Antivirus\\BackgroundJob\\BackgroundScanner' => 'Antivirus background scan', + 'OCA\\Recognize\\' => 'Recognize', + 'OCA\\PreviewGenerator\\' => 'Preview Generator', + 'OCA\\FullTextSearch\\' => 'Full text search', + ]; + + public const NOTHING = 'nothing'; + public const PAUSE = 'pause'; + public const RELEASE = 'release'; + public const REST = 'rest'; + + public function __construct( + private IDBConnection $db, + private IAppConfig $appConfig, + private LoggerInterface $logger, + ) { + } + + /** @return self::NOTHING|self::PAUSE|self::RELEASE|self::REST */ + public static function decide(bool $enabled, bool $backupRunning, int $since, int $restUntil, int $now): string { + if ($since > 0 && $now - $since >= self::MAX_SECONDS) { + return self::REST; + } + if ($enabled && $backupRunning && $now >= $restUntil) { + return self::PAUSE; + } + return $since > 0 ? self::RELEASE : self::NOTHING; + } + + /** Call regularly: pauses or ends the pause as needed. A problem here never stops a backup. */ + public function sync(bool $backupRunning): void { + try { + $this->apply($backupRunning, time()); + } catch (\Throwable $e) { + $this->logger->warning('NG Backup: pausing heavy background jobs failed: ' . $e->getMessage(), ['exception' => $e]); + } + } + + private function apply(bool $backupRunning, int $now): void { + $since = $this->appConfig->getValueInt(Application::APP_ID, self::SINCE_KEY, 0); + $restUntil = $this->appConfig->getValueInt(Application::APP_ID, self::REST_KEY, 0); + switch (self::decide($this->enabled(), $backupRunning, $since, $restUntil, $now)) { + case self::PAUSE: + if ($since === 0) { + $since = $now; + $this->appConfig->setValueInt(Application::APP_ID, self::SINCE_KEY, $since); + $this->logger->info('NG Backup: heavy background jobs paused while a backup runs'); + } + $this->reserve($since, $now); // also the jobs that were added or running until now + break; + case self::REST: + $this->appConfig->setValueInt(Application::APP_ID, self::REST_KEY, $now + self::REST_SECONDS); + $this->release($since); + $this->logger->info('NG Backup: heavy background jobs get their turn for an hour during a long backup'); + break; + case self::RELEASE: + $this->release($since); + $this->logger->info('NG Backup: heavy background jobs run again'); + break; + } + } + + private function reserve(int $stamp, int $now): void { + $qb = $this->db->getQueryBuilder(); + $qb->update('jobs') + ->set('reserved_at', $qb->createNamedParameter($stamp, IQueryBuilder::PARAM_INT)) + ->where($qb->expr()->lte('reserved_at', $qb->createNamedParameter($now - self::RESERVATION_SECONDS, IQueryBuilder::PARAM_INT))) + ->andWhere($this->isHeavy($qb)); + $qb->executeStatement(); + } + + /** Only our own reservations: a job that Nextcloud reserved at another moment is running. */ + private function release(int $stamp): void { + $qb = $this->db->getQueryBuilder(); + $qb->update('jobs') + ->set('reserved_at', $qb->createNamedParameter(0, IQueryBuilder::PARAM_INT)) + ->where($qb->expr()->eq('reserved_at', $qb->createNamedParameter($stamp, IQueryBuilder::PARAM_INT))) + ->andWhere($this->isHeavy($qb)); + $qb->executeStatement(); + $this->appConfig->deleteKey(Application::APP_ID, self::SINCE_KEY); + } + + /** @return \OCP\DB\QueryBuilder\ICompositeExpression */ + private function isHeavy(IQueryBuilder $qb) { + $conditions = []; + foreach (array_keys(self::JOBS) as $class) { + $conditions[] = str_ends_with($class, '\\') + ? $qb->expr()->like('class', $qb->createNamedParameter($this->db->escapeLikeParameter($class) . '%')) + : $qb->expr()->eq('class', $qb->createNamedParameter($class)); + } + return $qb->expr()->orX(...$conditions); + } + + public function enabled(): bool { + return $this->appConfig->getValueString(Application::APP_ID, self::ENABLED_KEY, '1') !== '0'; + } + + public function setEnabled(bool $enabled): void { + $this->appConfig->setValueString(Application::APP_ID, self::ENABLED_KEY, $enabled ? '1' : '0'); + } + + /** @return array{enabled: bool, since: int, jobs: list} what is paused now */ + public function status(): array { + $since = $this->appConfig->getValueInt(Application::APP_ID, self::SINCE_KEY, 0); + $jobs = []; + if ($since > 0) { + $qb = $this->db->getQueryBuilder(); + $qb->selectDistinct('class')->from('jobs') + ->where($qb->expr()->eq('reserved_at', $qb->createNamedParameter($since, IQueryBuilder::PARAM_INT))) + ->andWhere($this->isHeavy($qb)); + $result = $qb->executeQuery(); + while (($class = $result->fetchOne()) !== false) { + foreach (self::JOBS as $start => $name) { + if (str_starts_with((string)$class, $start)) { + $jobs[$name] = true; + } + } + } + $result->closeCursor(); + } + return ['enabled' => $this->enabled(), 'since' => $since, 'jobs' => array_map('strval', array_keys($jobs))]; + } +} diff --git a/lib/Service/KeyService.php b/lib/Service/KeyService.php index cf5c832..4145978 100644 --- a/lib/Service/KeyService.php +++ b/lib/Service/KeyService.php @@ -30,6 +30,9 @@ final class KeyService { private const KEY_DELETE_DELAY = 'delete_delay_days'; private const KEY_KIT_VERSION = 'recovery_kit_version'; private const KEY_KIT_CODE = 'recovery_kit_code'; + /** Set once a location was created or adopted with this key: from then on it is never replaced. */ + private const KEY_USED = 'master_key_used'; + private const KEY_WRAPPED_PENDING = 'master_key_wrapped_pending'; public const DEFAULT_DELETE_DELAY = 7; /** * The location holds the passphrase-wrapped key too, so whoever can read it can try @@ -83,28 +86,100 @@ final class KeyService { * @param array $kit as produced by recoveryKit() * @throws \OCA\NgBackup\Crypto\CryptoException wrong passphrase, or a damaged/foreign kit */ - public function importRecoveryKit(array $kit, #[\SensitiveParameter] string $passphrase): void { - if ($this->isInitialized()) { + public function importRecoveryKit(array $kit, #[\SensitiveParameter] string $passphrase, bool $replaceUnused = false): void { + if ($this->isInitialized() && !$replaceUnused) { throw new \RuntimeException('A backup key already exists'); } if (!isset($kit['wrapped_key'])) { throw new \InvalidArgumentException('Not a NG Backup recovery kit file'); } $keys = KeyRing::unwrapAny($kit['wrapped_key'], $passphrase); - $this->appConfig->setValueString(Application::APP_ID, self::KEY_MASTER, - $this->crypto->encrypt(base64_encode($keys->exportMasterKey())), true, true); - $this->cached = $keys; - $this->appConfig->setValueString(Application::APP_ID, self::KEY_WRAPPED, json_encode($kit['wrapped_key'], JSON_THROW_ON_ERROR), true, true); - $this->appConfig->setValueString(Application::APP_ID, self::KEY_FINGERPRINT, $this->fingerprintOf($keys), true); + $this->discardUnused($keys, $replaceUnused); // Kits from before this field existed fall back to the default. - $delay = (int)($kit['delete_delay_days'] ?? self::DEFAULT_DELETE_DELAY); - $this->appConfig->setValueInt(Application::APP_ID, self::KEY_DELETE_DELAY, ($delay >= 1 && $delay <= 365) ? $delay : self::DEFAULT_DELETE_DELAY, true); - $this->appConfig->setValueInt(Application::APP_ID, self::KEY_KIT_VERSION, (int)($kit['kit_version'] ?? 1), true); + $this->store($keys, $kit['wrapped_key'], (int)($kit['delete_delay_days'] ?? self::DEFAULT_DELETE_DELAY), (int)($kit['kit_version'] ?? 1)); // Having and successfully using a real kit file is stronger proof than the normal // download-and-confirm flow it replaces. $this->confirmRecoveryKit('cli-import'); } + /** + * Disaster recovery without the recovery kit: adopt the master key that a passphrase opened + * from the key slots in a location's config (TargetService::addWithPassphrase). There is no + * kit, so none is confirmed: backups wait until a new kit is downloaded and confirmed. + * + * @param ?int $deleteDelayDays from the location's sealed settings, null when it has none + */ + public function adoptKey(KeyRing $keys, array $wrappedKey, ?int $deleteDelayDays, bool $replaceUnused = false): void { + if ($this->isInitialized() && !$replaceUnused) { + throw new \RuntimeException('A backup key already exists'); + } + $this->discardUnused($keys, $replaceUnused); + $this->store($keys, $wrappedKey, $deleteDelayDays ?? self::DEFAULT_DELETE_DELAY, 1); + } + + /** Whether a location was ever created or adopted with this key (TargetService::keyIsUnused() also looks at the database). */ + public function wasUsed(): bool { + return $this->appConfig->getValueBool(Application::APP_ID, self::KEY_USED, false, true); + } + + public function markUsed(): void { + if (!$this->wasUsed()) { + $this->appConfig->setValueBool(Application::APP_ID, self::KEY_USED, true, true); + } + } + + /** + * Locations that missed the last passphrase change (offline at the time): they still hold + * the previous slots until the change reaches them, which is tried again at each backup. + */ + public function wrappedKeyPending(int $targetId): bool { + return $this->appConfig->getValueBool(Application::APP_ID, self::KEY_WRAPPED_PENDING . '_' . $targetId); + } + + /** One flag per location, so that two processes never overwrite each other's. */ + public function setWrappedKeyPending(int $targetId, bool $pending): void { + $key = self::KEY_WRAPPED_PENDING . '_' . $targetId; + if ($pending) { + $this->appConfig->setValueBool(Application::APP_ID, $key, true); + } elseif ($this->wrappedKeyPending($targetId)) { + $this->appConfig->deleteKey(Application::APP_ID, $key); + } + } + + /** + * A key made by mistake on a server that is being restored: remove it, so the key of the + * backups can take its place. Only after the new key was opened with its passphrase, and + * only when the caller checked that the old key never protected anything + * (TargetService::keyIsUnused()). Its passphrases and recovery kit become worthless. + */ + private function discardUnused(KeyRing $replacement, bool $replaceUnused): void { + if (!$replaceUnused || !$this->isInitialized()) { + return; + } + if ($this->wasUsed()) { + throw new \RuntimeException('The backup key of this server has been used for a backup location: it is never replaced'); + } + if ($this->fingerprintOf($replacement) === $this->fingerprint()) { + throw new \RuntimeException('This server already has this backup key: nothing to replace'); + } + foreach ([self::KEY_MASTER, self::KEY_WRAPPED, self::KEY_CONFIRMED, self::KEY_FINGERPRINT, self::KEY_DELETE_DELAY, + self::KEY_KIT_VERSION, self::KEY_KIT_CODE, self::KEY_USED] as $key) { + $this->appConfig->deleteKey(Application::APP_ID, $key); + } + $this->cached = null; + } + + private function store(KeyRing $keys, array $wrappedKey, int $deleteDelayDays, int $kitVersion): void { + $this->appConfig->setValueString(Application::APP_ID, self::KEY_MASTER, + $this->crypto->encrypt(base64_encode($keys->exportMasterKey())), true, true); + $this->cached = $keys; + $this->appConfig->setValueString(Application::APP_ID, self::KEY_WRAPPED, json_encode($wrappedKey, JSON_THROW_ON_ERROR), true, true); + $this->appConfig->setValueString(Application::APP_ID, self::KEY_FINGERPRINT, $this->fingerprintOf($keys), true); + $this->appConfig->setValueInt(Application::APP_ID, self::KEY_DELETE_DELAY, + ($deleteDelayDays >= 1 && $deleteDelayDays <= 365) ? $deleteDelayDays : self::DEFAULT_DELETE_DELAY, true); + $this->appConfig->setValueInt(Application::APP_ID, self::KEY_KIT_VERSION, max(1, $kitVersion), true); + } + /** * The master key and the recovery kit code are stored encrypted with Nextcloud's own * instance secret (ICrypto), not with themselves; the wrapped key is not (it is sealed with diff --git a/lib/Service/KeySlotService.php b/lib/Service/KeySlotService.php index cf6ed86..2eb2894 100644 --- a/lib/Service/KeySlotService.php +++ b/lib/Service/KeySlotService.php @@ -43,8 +43,10 @@ final class KeySlotService { foreach ($this->targets->list() as $t) { try { $this->targets->repository($t)->updateWrappedKey($wrapped); + $this->keys->setWrappedKeyPending($t->getId(), false); } catch (\Throwable $e) { $failed[] = $t->getName(); + $this->keys->setWrappedKeyPending($t->getId(), true); } } $label = ''; @@ -55,7 +57,7 @@ final class KeySlotService { } $this->alerts->securityEvent($event, ['slot' => (string)$slot, 'label' => $label]); if ($failed !== []) { - throw new \RuntimeException('Saved, but these locations could not be updated (they still accept the previous passphrases): ' . implode(', ', $failed)); + throw new \RuntimeException('Saved, but these locations could not be updated (they still accept the previous passphrases; the change is tried again at their next backup): ' . implode(', ', $failed)); } } } diff --git a/lib/Service/LocationSchedule.php b/lib/Service/LocationSchedule.php new file mode 100644 index 0000000..5726f75 --- /dev/null +++ b/lib/Service/LocationSchedule.php @@ -0,0 +1,78 @@ + 24) { + throw new \InvalidArgumentException('Every 1 to 24 hours'); + } + return $hours === 24 ? $m[1] : $m[1] . '/' . $hours; + } + + /** + * The most recent scheduled moment at or before $now; null without a schedule. + * + * @param non-empty-string $zone + */ + public static function lastMoment(string $schedule, string $zone, int $now): ?int { + if ($schedule === '') { + return null; + } + $tz = new \DateTimeZone($zone); + $today = (new \DateTimeImmutable('@' . $now))->setTimezone($tz); + $moments = array_merge(self::moments($schedule, $today->modify('-1 day'), $tz), self::moments($schedule, $today, $tz)); + $last = null; + foreach ($moments as $moment) { + if ($moment <= $now && ($last === null || $moment > $last)) { + $last = $moment; + } + } + return $last; + } + + /** @return list the moments of one day */ + private static function moments(string $schedule, \DateTimeImmutable $day, \DateTimeZone $tz): array { + [$time, $every] = array_pad(explode('/', $schedule), 2, '24'); + [$hour, $minute] = array_map('intval', explode(':', $time)); + $moments = []; + for ($h = $hour; $h < 24; $h += (int)$every) { + $moments[] = (new \DateTimeImmutable($day->format('Y-m-d') . sprintf(' %02d:%02d', $h, $minute), $tz))->getTimestamp(); + } + return $moments; + } + + /** + * Whether a backup has to start for $moment. One that was made since then is enough, also when + * several moments went by during a long backup. A moment from before the schedule was set does + * not count: choosing a time must not start a backup right away. + */ + public static function isDue(int $moment, int $since, ?int $lastDoneStarted, int $attempts, ?int $lastAttemptFinished, int $now): bool { + if ($moment < $since || ($lastDoneStarted !== null && $lastDoneStarted >= $moment) || $attempts >= self::MAX_ATTEMPTS) { + return false; + } + return $attempts === 0 || $now - (int)$lastAttemptFinished >= self::RETRY_AFTER; + } +} diff --git a/lib/Service/PruneService.php b/lib/Service/PruneService.php index 68852b9..b0f5277 100644 --- a/lib/Service/PruneService.php +++ b/lib/Service/PruneService.php @@ -11,6 +11,7 @@ use OCA\NgBackup\AppInfo\Application; use OCA\NgBackup\Db\RunMapper; use OCA\NgBackup\Db\SnapshotMapper; use OCA\NgBackup\Db\Target; +use OCA\NgBackup\Repository\Repository; use OCP\IAppConfig; use OCP\Lock\ILockingProvider; use OCP\Lock\LockedException; @@ -33,6 +34,54 @@ final class PruneService { ) { } + /** Characters of a note on a snapshot (the column holds 1000) */ + public const NOTE_MAX = 500; + + /** Pin or unpin a snapshot: the retention policy leaves a pinned one alone. */ + public function setPinned(Target $target, string $snapshotId, bool $pinned): void { + $changed = $this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $pinned): bool { + $was = in_array($snapshotId, $repo->pinnedSnapshots(), true); + $repo->setPinned($snapshotId, $pinned); + return $was !== $pinned; + }); + $this->targets->syncSnapshot($target, $snapshotId, pinned: $pinned); + if ($changed) { + // Unpinning opens the way to removing it, so every administrator hears about that. + $this->alerts->securityEvent($pinned ? 'snapshot_pinned' : 'snapshot_unpinned', ['target' => $target->getName(), 'snapshot' => $snapshotId], !$pinned); + } + } + + public function setNote(Target $target, string $snapshotId, string $note): void { + $note = trim((string)preg_replace('/[\x00-\x1f\x7f]+/', ' ', $note)); + if (!mb_check_encoding($note, 'UTF-8')) { + throw new \InvalidArgumentException('A note must be valid UTF-8'); + } + if (mb_strlen($note) > self::NOTE_MAX) { + throw new \InvalidArgumentException('A note has at most ' . self::NOTE_MAX . ' characters'); + } + $this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $note): bool { + $repo->setSnapshotNote($snapshotId, $note); + return true; + }); + $this->targets->syncSnapshot($target, $snapshotId, note: $note); + } + + /** + * Write to the catalog next to a running backup or restore, which only takes it for a moment to + * commit. Not during a cleanup, which holds a repository opened earlier and writes from that. + * + * @param callable(Repository): bool $change + */ + private function changeCatalog(Target $target, callable $change): bool { + try { + return $this->leases->with(self::lockKey($target), LeaseService::SHARED, 300, + fn (): bool => $this->leases->with(self::catalogLockKey($target), LeaseService::EXCLUSIVE, 300, + fn (): bool => $change($this->targets->repository($target)))); + } catch (LockedException) { + throw new \RuntimeException('A cleanup of ' . $target->getName() . ' or another change to its restore points is in progress; try again later'); + } + } + public function untrash(Target $target, string $snapshotId): void { $lease = $this->lockExclusive($target, 300); try { @@ -87,6 +136,7 @@ final class PruneService { $s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now'))); $s->setFiles((int)($meta['stats']['files'] ?? 0)); $s->setBytes((int)($meta['stats']['bytes'] ?? 0)); + $s->setNote($repo->snapshotNotes()[$snapshotId] ?? null); try { $this->snapshots->insert($s); } catch (\Throwable) { @@ -208,7 +258,7 @@ final class PruneService { $meta = $repo->snapshotMeta($id); $times[$id] = (int)strtotime((string)($meta['time'] ?? '@0')); } - $keep = $this->policy()->keep($times); + $keep = $this->policy()->keepWithPinned($times, $repo->pinnedSnapshots()); $forget = array_values(array_diff(array_keys($times), array_keys($keep))); $actor = $scheduled ? 'retention policy' : $this->alerts->actor(); $purged = []; diff --git a/lib/Service/RestoreService.php b/lib/Service/RestoreService.php index 83d3153..e330887 100644 --- a/lib/Service/RestoreService.php +++ b/lib/Service/RestoreService.php @@ -155,6 +155,17 @@ final class RestoreService { $dest = $this->ensureFolder($userFolder, $parentRel); $names = $single ? [basename($rel) => $entries['']] : $entries; + // The folders with something in them that this restore creates get their time back; one that + // was already there keeps its own, as its content changes now. Noted before anything is written. + /** @var array $newFolders a folder called "2026" is an integer key */ + $newFolders = []; + foreach ($single ? [] : $repo->folderTimes($snapshotId) as [$dir, $mtime]) { + $relative = $dir === $source ? '' : (str_starts_with($dir, $source . '/') ? substr($dir, strlen($source) + 1) : null); + if ($relative !== null && ($relative === '' ? !$destExisted : !$dest->nodeExists($relative))) { + $newFolders[$relative] = $mtime; + } + } + $this->cleaned = []; $restored = 0; foreach ($names as $relPath => $entry) { @@ -190,6 +201,13 @@ final class RestoreService { if ($mode === self::MODE_REPLACE && !$single) { $trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)) + $emptyDirs); } + // Last and deepest first: a change in a folder also touches the folders above it. + foreach (array_reverse($newFolders, true) as $dir => $mtime) { + $this->ensureFolder($dest, (string)$dir)->touch($mtime); + if ($progress !== null) { + $progress($restored, count($names), (string)$dir); // keeps the lease alive too + } + } return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())]; } diff --git a/lib/Service/RetentionPolicy.php b/lib/Service/RetentionPolicy.php index fc4b1a7..cd8be0b 100644 --- a/lib/Service/RetentionPolicy.php +++ b/lib/Service/RetentionPolicy.php @@ -28,6 +28,19 @@ final class RetentionPolicy { return ['last' => $this->last, 'daily' => $this->daily, 'weekly' => $this->weekly, 'monthly' => $this->monthly]; } + /** + * keep() for the snapshots that are not pinned, plus the pinned ones. A pinned snapshot does not + * take the place of a daily, weekly or monthly one. + * + * @param array $snapshots id => unix time + * @param list $pinned + * @return array> + */ + public function keepWithPinned(array $snapshots, array $pinned): array { + $pins = array_intersect_key($snapshots, array_flip($pinned)); + return $this->keep(array_diff_key($snapshots, $pins)) + array_fill_keys(array_keys($pins), ['pinned']); + } + /** * @param array $snapshots id => unix time * @return array> id => reasons it is kept (ids not in the result are removed) diff --git a/lib/Service/Schedule.php b/lib/Service/Schedule.php index 326c537..58f80e8 100644 --- a/lib/Service/Schedule.php +++ b/lib/Service/Schedule.php @@ -13,10 +13,11 @@ use OCP\IConfig; use OCP\IGroupManager; /** - * The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a - * zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone - * is converted once to the zone of the server or of an administrator, at today's offset: it then - * shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer). + * The time zone times are shown in, and the daily time from before each location had its own + * schedule (see LocationSchedule). Nextcloud runs PHP in UTC, so a time without a zone meant UTC + * (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone is converted once + * to the zone of the server or of an administrator, at today's offset: it then shows as it + * effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer). */ final class Schedule { public const TIME_KEY = 'schedule_time'; @@ -44,27 +45,9 @@ final class Schedule { return ['time' => $time, 'timezone' => $zone]; } - /** @throws \InvalidArgumentException for a malformed time or an unknown zone */ - public function set(string $time, string $zone): void { - if ($time !== '' && !self::validTime($time)) { - throw new \InvalidArgumentException('Use HH:MM or empty'); - } - if (!self::validZone($zone)) { - throw new \InvalidArgumentException('Unknown time zone: ' . $zone); - } - $this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone); - $this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time); - } - - /** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */ - public function todayAt(?int $now = null): ?int { - ['time' => $time, 'timezone' => $zone] = $this->get(); - if ($time === '' || !self::validTime($time)) { - return null; - } - $tz = new \DateTimeZone($zone); - $day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d'); - return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp(); + /** @return non-empty-string the zone of a location's schedule, or else this one */ + public function zoneOr(?string $zone): string { + return $zone !== null && self::validZone($zone) ? $zone : $this->get()['timezone']; } /** diff --git a/lib/Service/TargetService.php b/lib/Service/TargetService.php index 4fd4922..a00a5b0 100644 --- a/lib/Service/TargetService.php +++ b/lib/Service/TargetService.php @@ -27,6 +27,9 @@ use OCP\Server; * classes but its enabled/disabled state is left as the admin set it. */ final class TargetService { + /** Deletion delay taken when a location's own cannot be read: nothing leaves the Bin sooner than on the old server. */ + public const LONGEST_DELETE_DELAY = 365; + public function __construct( private TargetMapper $mapper, private KeyService $keys, @@ -35,6 +38,8 @@ final class TargetService { private \OCA\NgBackup\Db\DbIndexCache $indexCache, private \OCA\NgBackup\Db\AppConfigCatalogAnchor $anchor, private SnapshotMapper $snapshots, + private AlertService $alerts, + private \OCA\NgBackup\Db\RunMapper $runs, ) { } @@ -51,18 +56,31 @@ final class TargetService { * @param ?callable(int, int): void $progress reading an existing repository's index: files done, and in all * @return array{target: Target, created: bool} */ - public function add(string $name, string $backend, string $auth, array $options, string $basePath, ?callable $progress = null): array { + public function add(string $name, string $backend, string $auth, array $options, string $basePath, ?callable $progress = null, bool $repairConfig = false): array { $this->validateName($name); $this->requireUsableBackend($backend); $keys = $this->keys->keyRing(); - $storageBackend = $this->buildBackend($backend, $auth, $options, $basePath); - if ($storageBackend->exists('config')) { - $repo = Repository::openWithKey($storageBackend, $keys, $this->indexCache, $this->anchor, $progress); - $created = false; - } else { - $repo = Repository::initWithKey($storageBackend, $keys, $this->keys->wrappedKey(), $this->anchor); - $created = true; + $storageBackend = $this->buildBackend($backend, $auth, $options, $basePath, !$repairConfig); + try { + if ($repairConfig) { + Repository::repairConfig($storageBackend, $keys, $this->keys->wrappedKey()); + } + if ($storageBackend->exists('config')) { + $repo = Repository::openWithKey($storageBackend, $keys, $this->indexCache, $this->anchor, $progress); + $created = false; + } else { + $repo = Repository::initWithKey($storageBackend, $keys, $this->keys->wrappedKey(), $this->anchor); + $created = true; + } + } catch (\OCA\NgBackup\Repository\RepositoryException $e) { + $otherKey = str_contains($e->getMessage(), 'different key') || str_contains($e->getMessage(), 'do not open with this key'); + if ($otherKey && $this->keyIsUnused()) { + throw new \OCA\NgBackup\Repository\RepositoryException(rtrim($e->getMessage(), '.') . '. ' . self::UNUSED_KEY_HINT, 0, $e); + } + throw $e; } + $this->keys->markUsed(); + $this->recordSettings($repo); foreach ($this->mapper->findAll() as $existing) { if ($existing->getRepositoryId() === $repo->id()) { @@ -78,6 +96,11 @@ final class TargetService { $target->setBasePath(trim($basePath, '/')); $target->setRepositoryId($repo->id()); $target->setCreatedAt(time()); + // Like the location before it: a backup scheme is usually one scheme. + $first = $this->mapper->findAll()[0] ?? null; + $target->setSchedule($first?->getSchedule() ?? ''); + $target->setScheduleZone($first?->getScheduleZone()); + $target->setScheduleSince(time()); $target = $this->mapper->insert($target); if (!$created) { // Disaster recovery: occ backup:list reads this local cache, not the repository @@ -87,6 +110,128 @@ final class TargetService { return ['target' => $target, 'created' => $created]; } + public const UNUSED_KEY_HINT = 'The backup key of this server was never used (no location, no backup), so it can make way for the key of these backups: ' + . 'with the old server\'s recovery kit, occ backup:key:import --replace-unused-key; without the kit, ' + . 'occ backup:target:add with the same details and --replace-unused-key (asks for a passphrase of the old server)'; + + /** + * A key that never protected anything: no location was created or adopted with it, and the + * database has no location and no run (installations from before the flag existed). Only such + * a key may make way for the key of existing backups, e.g. after a new key was made by + * mistake on a server that is being restored. + */ + public function keyIsUnused(): bool { + return $this->keys->isInitialized() && !$this->keys->wasUsed() + && $this->mapper->findAll() === [] && $this->runs->countAll() === 0; + } + + /** + * occ backup:key:import on a server whose own key was never used: the kit's key takes its place. + * + * @param array $kit + * @throws \OCA\NgBackup\Crypto\CryptoException wrong passphrase: nothing changed + */ + public function importKitReplacingUnusedKey(array $kit, #[\SensitiveParameter] string $passphrase): void { + $old = $this->requireUnusedKey(); + $this->keys->importRecoveryKit($kit, $passphrase, true); + $this->alerts->securityEvent('key_replaced', ['old' => $old, 'new' => $this->keys->fingerprint()]); + } + + /** @return string the fingerprint of the unused key */ + private function requireUnusedKey(): string { + if (!$this->keys->isInitialized()) { + throw new \RuntimeException('There is no backup key here to replace'); + } + if (!$this->keyIsUnused()) { + throw new \RuntimeException('The backup key of this server (fingerprint ' . $this->keys->fingerprint() . ') has been used for a backup location: it is never replaced'); + } + return $this->keys->fingerprint(); + } + + public const LOCATION_REPOSITORY = 'repository'; + public const LOCATION_DATA_WITHOUT_CONFIG = 'data-without-config'; + public const LOCATION_EMPTY = 'empty'; + + /** + * What these settings lead to: a repository, backup data whose config is gone, or nothing + * of NG Backup. Creates nothing. + * + * @return self::LOCATION_* + */ + public function locationState(string $backend, string $auth, array $options, string $basePath): string { + $this->requireUsableBackend($backend); + $storage = $this->buildBackend($backend, $auth, $options, $basePath, false); + if ($storage->exists('config')) { + return self::LOCATION_REPOSITORY; + } + return Repository::holdsData($storage) ? self::LOCATION_DATA_WITHOUT_CONFIG : self::LOCATION_EMPTY; + } + + /** + * Disaster recovery without the recovery kit, on an installation without a backup key: take + * the key from the location's config with one of its passphrases, then add the location as + * usual. The key is only kept when it really is the one the repository was made with. + * With $replaceUnusedKey, a key this server has but never used makes way (keyIsUnused()). + * + * @param array $options + * @param ?callable(int, int): void $progress + * @return array{target: Target, created: bool, deleteDelayDays: ?int, settingsUnreadable: bool} the deletion delay the location had stored, if any, and whether its settings could not be read + * @throws \OCA\NgBackup\Crypto\CryptoException wrong passphrase + */ + public function addWithPassphrase(string $name, string $backend, string $auth, array $options, string $basePath, + #[\SensitiveParameter] string $passphrase, ?callable $progress = null, bool $replaceUnusedKey = false): array { + $old = null; + if ($replaceUnusedKey) { + $old = $this->requireUnusedKey(); + } elseif ($this->keys->isInitialized()) { + throw new \RuntimeException('A backup key already exists here'); + } + $this->validateName($name); + $this->requireUsableBackend($backend); + $storageBackend = $this->buildBackend($backend, $auth, $options, $basePath, false); + if (!$storageBackend->exists('config')) { + throw new \InvalidArgumentException("No NG Backup repository found in folder '" . trim($basePath, '/') . "'"); + } + $wrapped = Repository::wrappedKeyAt($storageBackend); + $keys = \OCA\NgBackup\Crypto\KeyRing::unwrapAny($wrapped, $passphrase); + if (!Repository::madeWithKey($storageBackend, $keys)) { + throw new \RuntimeException('The key in the config at this location does not open its backups: the config is damaged or was replaced. Only the recovery kit can help now: occ backup:key:import , then this command again.'); + } + $settings = Repository::settingsAt($storageBackend, $keys); + $stored = $settings['delete_delay_days'] ?? null; + $delay = is_int($stored) && $stored >= 1 && $stored <= 365 ? $stored : null; + // Settings that are there but cannot be read, or hold no valid delay (also none at all): the old server's delay + // is unknown, so take the longest one; a shorter guess could empty the Bin too soon. + $unknown = ($settings['unreadable'] ?? false) === true || ($settings !== [] && $delay === null); + $this->keys->adoptKey($keys, $wrapped, $delay ?? ($unknown ? self::LONGEST_DELETE_DELAY : null), $replaceUnusedKey); + if ($old !== null) { + $this->alerts->securityEvent('key_replaced', ['old' => $old, 'new' => $this->keys->fingerprint()]); + } + return $this->add($name, $backend, $auth, $options, $basePath, $progress) + ['deleteDelayDays' => $delay, 'settingsUnreadable' => $unknown]; + } + + /** A passphrase change that missed this location (offline at the time) reaches it now. A failure is tried again at the next backup. */ + public function catchUpWrappedKey(Target $target, Repository $repo): void { + if (!$this->keys->wrappedKeyPending($target->getId())) { + return; + } + try { + $repo->updateWrappedKey($this->keys->wrappedKey()); + $this->keys->setWrappedKeyPending($target->getId(), false); + } catch (\Throwable $e) { + Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: the last passphrase change still does not reach ' . $target->getName() + . ' (it keeps the previous passphrases); tried again at its next backup: ' . $e->getMessage(), ['exception' => $e]); + } + } + + /** Seal the settings fixed at installation on the location, once. A failure is tried again at the next backup. */ + public function recordSettings(Repository $repo): void { + try { + $repo->recordSettings(['delete_delay_days' => $this->keys->deleteDelayDays()]); + } catch (\Throwable) { + } + } + /** * Change how an existing location is reached (credentials, host, folder, even the connection * type) or its name. The new settings must lead to the same repository: the history stays @@ -208,7 +353,25 @@ final class TargetService { return $result; } + /** Update the local copy of a snapshot's note or pin, if the snapshot is listed here at all. */ + public function syncSnapshot(Target $target, string $snapshotId, ?string $note = null, ?bool $pinned = null): void { + try { + $s = $this->snapshots->findOne($target->getId(), $snapshotId); + } catch (DoesNotExistException) { + return; + } + if ($note !== null) { + $s->setNote($note !== '' ? $note : null); + } + if ($pinned !== null) { + $s->setPinned($pinned); + } + $this->snapshots->update($s); + } + private function adoptSnapshots(Target $target, Repository $repo): void { + $notes = $repo->snapshotNotes(); + $pinned = array_flip($repo->pinnedSnapshots()); foreach ($repo->listSnapshots() as $snapshotId) { try { $this->snapshots->findOne($target->getId(), $snapshotId); @@ -227,6 +390,8 @@ final class TargetService { $s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now'))); $s->setFiles((int)($meta['stats']['files'] ?? 0)); $s->setBytes((int)($meta['stats']['bytes'] ?? 0)); + $s->setNote($notes[$snapshotId] ?? null); + $s->setPinned(isset($pinned[$snapshotId])); $this->snapshots->insert($s); } catch (\Throwable $e) { Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]); @@ -248,6 +413,30 @@ final class TargetService { } /** Size limit for the data stored on this location (null = none). */ + /** + * @param string $schedule see LocationSchedule + * @throws \InvalidArgumentException + */ + public function setSchedule(Target $target, string $schedule, string $zone): Target { + if (!Schedule::validZone($zone)) { + throw new \InvalidArgumentException('Unknown time zone: ' . $zone); + } + $new = LocationSchedule::normalize($schedule); + $turnedOff = $new === '' && (string)$target->getSchedule() !== ''; + if ($new === (string)$target->getSchedule() && $zone === $target->getScheduleZone()) { + // Saved again unchanged: a moment that already passed still gets its backup + return $target; + } + $target->setSchedule($new); + $target->setScheduleZone($zone); + $target->setScheduleSince(time()); + $target = $this->mapper->update($target); + if ($turnedOff) { + $this->alerts->securityEvent('schedule_off', ['target' => $target->getName()]); + } + return $target; + } + public function setMaxBytes(Target $target, ?int $maxBytes): Target { $target->setMaxBytes($maxBytes); return $this->mapper->update($target); @@ -261,10 +450,29 @@ final class TargetService { /** Remove the target from NG Backup. The backups on the target itself are not touched. */ public function remove(Target $target): void { $this->mapper->delete($target); + // The location is gone now: tidying up must not fail the removal, or the caller would skip + // the removal event and a retry would no longer find the location + $this->tidyUp($target, fn () => $this->keys->setWrappedKeyPending($target->getId(), false)); // The cached index of its repository is only of use while a location points at it - $repositoryId = $target->getRepositoryId(); - if ($repositoryId !== null && !array_filter($this->list(), fn (Target $t) => $t->getRepositoryId() === $repositoryId)) { - $this->indexCache->forget($repositoryId); + $this->tidyUp($target, function () use ($target): void { + $repositoryId = $target->getRepositoryId(); + if ($repositoryId !== null && !array_filter($this->list(), fn (Target $t) => $t->getRepositoryId() === $repositoryId)) { + $this->indexCache->forget($repositoryId); + } + }); + } + + /** One tidy-up step after a removal: a failure is logged (if even that works) and never thrown. */ + private function tidyUp(Target $target, callable $step): void { + try { + $step(); + } catch (\Throwable $e) { + try { + Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: location ' . $target->getName() + . ' removed, tidying up after it failed: ' . $e->getMessage(), ['exception' => $e]); + } catch (\Throwable) { + // nothing left to tell it to + } } } diff --git a/psalm-baseline.xml b/psalm-baseline.xml index d6ad913..0bacdb4 100644 --- a/psalm-baseline.xml +++ b/psalm-baseline.xml @@ -300,6 +300,22 @@ + + + + + + + + + + + + + + + + @@ -351,7 +367,6 @@ - diff --git a/screenshots/1-backups.png b/screenshots/1-backups.png index f5ad83a..549dab7 100644 Binary files a/screenshots/1-backups.png and b/screenshots/1-backups.png differ diff --git a/screenshots/2-restore.png b/screenshots/2-restore.png index bee0170..ae10f12 100644 Binary files a/screenshots/2-restore.png and b/screenshots/2-restore.png differ diff --git a/screenshots/3-settings.png b/screenshots/3-settings.png index 61122a8..28d16f1 100644 Binary files a/screenshots/3-settings.png and b/screenshots/3-settings.png differ diff --git a/screenshots/4-bin.png b/screenshots/4-bin.png index 52ae8f4..1a7b994 100644 Binary files a/screenshots/4-bin.png and b/screenshots/4-bin.png differ diff --git a/src/api.js b/src/api.js index 85b391b..f470947 100644 --- a/src/api.js +++ b/src/api.js @@ -34,8 +34,11 @@ export default { setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }), startBackup: (targetId, label) => call('post', '/backups', { targetId, label }), tick: () => call('post', '/tick'), - setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }), + setPauseHeavyJobs: (enabled) => call('put', '/settings/pause-heavy-jobs', { enabled }), + setTargetSchedule: (id, schedule, timezone) => call('put', `/targets/${id}/schedule`, { schedule, timezone }), snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`), + setSnapshotNote: (targetId, snapshotId, note) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/note`, { note }), + setSnapshotPinned: (targetId, snapshotId, pinned) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/pin`, { pinned }), browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }), startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }), } diff --git a/src/components/Backups.vue b/src/components/Backups.vue index 5866402..84b1117 100644 --- a/src/components/Backups.vue +++ b/src/components/Backups.vue @@ -5,10 +5,11 @@ diff --git a/src/components/RestoreBrowser.vue b/src/components/RestoreBrowser.vue index 6a2d612..da8688a 100644 --- a/src/components/RestoreBrowser.vue +++ b/src/components/RestoreBrowser.vue @@ -11,6 +11,13 @@