Compare commits
22 Commits
v1.0.2
..
a8b5e85c9b
| Author | SHA1 | Date | |
|---|---|---|---|
| a8b5e85c9b | |||
| 36f2427e9a | |||
| e58b86fc48 | |||
| 193f65940b | |||
| 6540a9314a | |||
| f81c5e13e9 | |||
| 1b621b7149 | |||
| 1736005648 | |||
| c7a651f269 | |||
| 8897d8074f | |||
| 0b9fa1ae92 | |||
| ee406c481e | |||
| dd2e1af997 | |||
| d96229f370 | |||
| b14b16c66c | |||
| ce40053cec | |||
| 46e0186f35 | |||
| d5eb7abbff | |||
| 4c973d5cd8 | |||
| e9592cb6c6 | |||
| b86b07a49c | |||
| ecae6e9cdd |
@@ -11,6 +11,62 @@ Each heading starts with that `<version>`, once: the app store shows a release t
|
|||||||
whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
|
whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
|
||||||
shown under 1.0.0. The git tag of a release candidate goes after the date.
|
shown under 1.0.0. The git tag of a release candidate goes after the date.
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Pin a restore point, on the restore page or with `occ backup:pin`: the retention policy leaves it
|
||||||
|
alone, for example the one from before an upgrade. Unpinning notifies all administrators.
|
||||||
|
- A note on a restore point that you can change afterwards, on the restore page or with
|
||||||
|
`occ backup:note`. Pins and notes are stored encrypted on the location, so a new server sees them.
|
||||||
|
An older version of NG Backup can still read the location, but drops them when it writes to it.
|
||||||
|
- A schedule per location: off, once a day, or every N hours from a given time, on the page under
|
||||||
|
Backups or with `occ backup:target:schedule`. After the update every location keeps the time that
|
||||||
|
held for all of them; a new location starts with the schedule of the first one.
|
||||||
|
- All folders get their original modification time back on a restore, not only the empty ones, from
|
||||||
|
the first backup made with this version. An older version of NG Backup can still read new restore
|
||||||
|
points. A restore through Nextcloud into a folder that is already there leaves its time alone.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Locations are backed up one after the other: a backup that has to wait shows as queued. A restore
|
||||||
|
point has the time its backup really began, not the time it was queued.
|
||||||
|
- Choosing a time no longer starts a backup right away when that time has already passed today.
|
||||||
|
- A location without a schedule no longer gets the notification that it has had no recent backup.
|
||||||
|
Instead, all administrators are told when a schedule is turned off, and reminded once a week while
|
||||||
|
another location is still backed up, or every day when none is.
|
||||||
|
- A file or folder that is left out because of its name now gets a notification of its own, in the
|
||||||
|
log and in `occ backup:run`. It says the item is missing from the backup, gives example paths and
|
||||||
|
asks to rename it; before, it was reported as unreadable, with advice about permissions.
|
||||||
|
|
||||||
|
## [1.0.3] - 2026-11-06
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Empty folders are backed up and restored (#27), in `occ backup:restore`, `occ backup:restore:full`
|
||||||
|
and the restore page, where they are shown too. They come back from the first backup made with
|
||||||
|
this version; older restore points do not contain them. An older version of NG Backup can still
|
||||||
|
read new restore points, and then only leaves the empty folders out.
|
||||||
|
- A full disk during a restore no longer looks like a damaged backup. The restore now stops with a
|
||||||
|
message about the disk, also with `--salvage`, and does not point at `--salvage` any more. The same
|
||||||
|
goes for other trouble writing to the restore directory, such as permissions.
|
||||||
|
- A file or folder whose name a snapshot cannot hold (a backslash, a control character, not UTF-8) is
|
||||||
|
left out and reported as unreadable. Before, a backslash or control character could make the whole
|
||||||
|
restore point unreadable, and a name that is not UTF-8 made the backup fail.
|
||||||
|
- A restore that goes on after a stop also removes the temporary files in folders where every file
|
||||||
|
is already in place, and checks that those folders are inside the restore directory before it
|
||||||
|
leaves their files alone.
|
||||||
|
- After a restore that was killed and then ran again, a backup no longer has to wait up to two hours
|
||||||
|
for the locks of the killed run.
|
||||||
|
- A restore only removes read-ahead folders of its own (`.ngb-restore-spool-` with 8 hex digits).
|
||||||
|
- A restore keeps its lock on the location during a long row of files that are already in place.
|
||||||
|
- A database restore that fails no longer leaves behind tables it created, which could stop the
|
||||||
|
installation of that app later. When only NG Backup's own settings cannot be put back after the
|
||||||
|
database was restored, `--salvage` no longer says the database is still the fresh one.
|
||||||
|
- A connection that breaks while opening a pack on the location (seen with SMB) is retried once and
|
||||||
|
is handled by `--salvage`, instead of stopping the restore with the error of the storage.
|
||||||
|
- Without a system cron, several running backups take turns while the page is open.
|
||||||
|
- The Bin tab shows how far it is while it reads the removed restore points, which can take a while.
|
||||||
|
It also shows when it cannot read a location, instead of an empty bin, and refreshes when a run
|
||||||
|
ends.
|
||||||
|
|
||||||
## [1.0.2] - 2026-11-06
|
## [1.0.2] - 2026-11-06
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
@@ -55,6 +111,9 @@ shown under 1.0.0. The git tag of a release candidate goes after the date.
|
|||||||
|
|
||||||
### Known issues
|
### Known issues
|
||||||
- Empty folders are not backed up or restored; to be fixed in 1.0.3 (#27).
|
- Empty folders are not backed up or restored; to be fixed in 1.0.3 (#27).
|
||||||
|
- A full disk during a restore looks like a damaged backup ("Size mismatch"), and with `--salvage`
|
||||||
|
the restore goes on and lists the remaining files. The backup itself is fine: make room and run
|
||||||
|
the restore again. To be fixed in 1.0.3.
|
||||||
|
|
||||||
## [1.0.1] - 2026-11-01
|
## [1.0.1] - 2026-11-01
|
||||||
|
|
||||||
|
|||||||
@@ -437,7 +437,47 @@ in README "Known limitations":
|
|||||||
met vertraging is nog niet gemeten); Bin-tabblad; schatting `target:add`; nieuwe screenshots
|
met vertraging is nog niet gemeten); Bin-tabblad; schatting `target:add`; nieuwe screenshots
|
||||||
(1–3 vervangen, 4-bin erbij; in `info.xml` bij de release naar tag v1.0.2 laten wijzen en
|
(1–3 vervangen, 4-bin erbij; in `info.xml` bij de release naar tag v1.0.2 laten wijzen en
|
||||||
4-bin toevoegen). Getest: unit (208), psalm, integratie NC35 sqlite/pgsql, S3/SFTP/FTP/SMB,
|
4-bin toevoegen). Getest: unit (208), psalm, integratie NC35 sqlite/pgsql, S3/SFTP/FTP/SMB,
|
||||||
volledige restore met `--salvage`. Nog te doen: DR-test, release.
|
volledige restore met `--salvage`. **1.0.2 uitgebracht 2026-11-06** (tag `v1.0.2`, PR #28).
|
||||||
|
- **Naar 1.0.3, uit de reviews van 1.0.2 (achteraf gedraaid, 2026-11-06):**
|
||||||
|
1. Lege mappen (#27).
|
||||||
|
2. Een volle schijf onder `--salvage` telt als beschadigde backup en de restore gaat door; zonder
|
||||||
|
`--salvage` wijst de melding er ten onrechte naar. `fwrite` controleren, schrijffouten stoppen de
|
||||||
|
restore (`Repository::restoreEntries`; ook `RestoreService::writeFile` nakijken). Beide reviews.
|
||||||
|
3. Hervatten: controle op de doelmap vóór het overslaan, en tijdelijke bestanden ook opruimen in
|
||||||
|
een map waar alles al klopt.
|
||||||
|
4. Alleen `.ngb-restore-spool-<hex>` opruimen.
|
||||||
|
5. Salvage-test: tijdelijke bestanden recursief zoeken; `admin` en `frtest` apart controleren.
|
||||||
|
6. Nog nakijken: lease verversen bij overgeslagen bestanden; fout ná de database-commit
|
||||||
|
(`restoreOwnAppConfig`) en tabellen die na een rollback blijven staan; backups zonder cron
|
||||||
|
via de pagina (eerlijke beurten, stappen die de webtimeout halen); Bin verversen en fouten
|
||||||
|
tonen; SMB-uitzondering bij het openen van een pack onder `--salvage`.
|
||||||
|
Niet doen: bij hervatten de inhoud vergelijken (grootte en mtime is ook het criterium van de backup).
|
||||||
|
**Stand op branch `1.0.3` (2026-11-06):** punten 1-6 gebouwd. Lege mappen staan in het metarecord
|
||||||
|
van de snapshot, niet tussen de bestanden: 1.0.2 en ouder lezen zo'n snapshot gewoon en slaan alleen
|
||||||
|
de lege mappen over. Geen webtimeout-wijziging zonder cron: alleen in de code nagelopen, niet getest
|
||||||
|
(Nextcloud zet de PHP-tijdslimiet ruim, een gestopte stap wordt overgedaan); wel eerlijke beurten.
|
||||||
|
Bin leest in delen met teller en balk (besluit 2026-11-06, ook met oog op meerdere backups per
|
||||||
|
dag). Getest: unit (volle schijf op echte tmpfs), integratie NC35 sqlite/pgsql/mysql, sftp, salvage,
|
||||||
|
salvage-files; Bin in een echte browser met vertraagde verzoeken (1 van 7 t/m 7). Nog: reviews
|
||||||
|
vóór de PR, README-regels bij de versiecommit.
|
||||||
|
**Simulaties (2026-11-06, NC 34.0.4, 2 GB, lokale SFTP, `~/dev-local/ngb-sim/sim.sh`):** normaal
|
||||||
|
(pgsql), stop en hervat (pgsql → MariaDB, restore met kill -9 gestopt), volle schijf met `--salvage`:
|
||||||
|
alle drie geslaagd, alle 3.101 gebruikersbestanden en 7 lege mappen gelijk, share, agenda en WebDAV
|
||||||
|
werken. Vond één fout, opgelost: na een kill bleven vergrendelingen tot 2 uur staan, waardoor de
|
||||||
|
backup van stap 11 werd geweigerd. `files:scan` "Updated" zijn in de gemeten run alleen mappen
|
||||||
|
(mtime van mappen wordt niet teruggezet, dus nieuwe mtime en etag; bestanden ongemoeid). "Removed":
|
||||||
|
o.a. `cache` van een gebruiker, die de backup uitsluit. Waarschijnlijk verklaart dit ook de 50.080
|
||||||
|
"updated" op nc01, niet op nc01 zelf vastgesteld.
|
||||||
|
**Naar 1.1.0 of 1.2.0 (besluit 2026-11-06):** alle mappen krijgen bij een restore hun oorspronkelijke
|
||||||
|
tijd terug. Hoort zo, vooral bij oudere verzamelingen: de maptijd wordt gebruikt om in te schatten
|
||||||
|
wanneer een map is aangemaakt. De backup moet dan de tijd van alle mappen vastleggen en de restore
|
||||||
|
die aan het eind terugzetten; werkt pas vanaf de eerste backup met die versie. Op de roadmap onder 1.1.0.
|
||||||
|
**1.0.3 uitgebracht (2026-11-06)**, ook op nc01 geïnstalleerd.
|
||||||
|
- **Branches (besluit 2026-11-06):** werk voor een versie op een branch met die naam (`1.0.3`);
|
||||||
|
`polish` alleen voor kleine cosmetische zaken, per geval bekijken of en wanneer die gemerged
|
||||||
|
worden. SMB staat uit op sub5 en sub6; sub6 is het testaccount, eerst aanzetten voor een SMB-test.
|
||||||
|
Na 1.0.3 gaat het werk naar `1.1.0`. Een spoedfix die niet kan wachten gaat op `1.0.4` vanaf `main`,
|
||||||
|
krijgt een eigen merge en release; daarna `main` in `1.1.0` mergen, zodat daar met de fix getest wordt.
|
||||||
- **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
|
- **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
|
||||||
1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
|
1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
|
||||||
niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij
|
niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij
|
||||||
@@ -532,7 +572,18 @@ in README "Known limitations":
|
|||||||
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
|
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
|
||||||
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
|
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
|
||||||
een paar bestanden.
|
een paar bestanden.
|
||||||
- **1.1.0:**
|
- **1.1.0** (geen harde datum: een week was een voorzet, langer mag; de releasefrequentie mag omlaag,
|
||||||
|
besluit 2026-11-06):
|
||||||
|
**Betere melding voor namen die een snapshot niet kan bevatten** (backslash, stuurteken, geen UTF-8):
|
||||||
|
een eigen melding in plaats van "unreadable" met het advies over rechten. Zegt dat deze bestanden of
|
||||||
|
mappen in de backup ontbreken, geeft voorbeelden van de paden en vraagt ze te hernoemen; de volgende
|
||||||
|
backup neemt ze mee. Alleen als Nextcloud-melding: de app mailt niet zelf, of meldingen gemaild worden
|
||||||
|
is de keuze van de beheerder (besluit 2026-11-06).
|
||||||
|
**Stand op branch `1.1.0` (2026-11-07):** klaar: betere melding, punten 4, 5, 6 en 8. Pin en notitie staan
|
||||||
|
in de catalogus, met een kopie in de database voor de lijsten. Een moment van vóór het instellen van
|
||||||
|
een schema start niets. Geen schema: geen melding "geen recente backup", wel een melding aan alle
|
||||||
|
beheerders bij het uitzetten en een herinnering, wekelijks zolang een andere locatie een schema heeft,
|
||||||
|
anders dagelijks (besluit 2026-11-07). Nog niet getest: de update vanaf 1.0.3, de pagina in een browser.
|
||||||
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
|
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
|
||||||
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
|
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
|
||||||
`target:add` en sleutelimport (het gevoeligste pad), daarom niet in rc1. Daarna de
|
`target:add` en sleutelimport (het gevoeligste pad), daarom niet in rc1. Daarna de
|
||||||
@@ -564,11 +615,10 @@ in README "Known limitations":
|
|||||||
nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met
|
nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met
|
||||||
`occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de
|
`occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de
|
||||||
opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly.
|
opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly.
|
||||||
9. **Feedbackvraag weghalen** in 1.1.0 of 1.2.0 (besluit 2026-10-27): de regel "We would like
|
9. **Feedbackvraag weghalen in 1.2.0** (besluit 2026-11-07, niet in 1.1.0): de regel "We would like
|
||||||
to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot
|
to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot
|
||||||
dan staan (de vermelding "in beta" is er in 1.0.1 al af). Zijn er tegen die
|
dan staan. Er komt weinig feedback bij zo'n 150 downloads per versie; geen bericht is goed
|
||||||
tijd geen noemenswaardige meldingen, dan werkt het: mensen melden eerder wat niet werkt dan
|
bericht. Het formulier (`experience.yml`) in de issues blijft.
|
||||||
wat gewoon werkt. Het formulier (`experience.yml`) mag blijven voor wie het zelf opzoekt.
|
|
||||||
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
|
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
|
||||||
PostgreSQL 18)** — voor **1.0.0-rc.2**:
|
PostgreSQL 18)** — voor **1.0.0-rc.2**:
|
||||||
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
|
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
|
|||||||
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
||||||
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
||||||
|
|
||||||
**Status: 1.0.2, stable.** Disaster recovery has been tested on real installations (see "Tested
|
**Status: 1.0.3, stable.** Disaster recovery has been tested on real installations (see "Tested
|
||||||
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
|
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
|
||||||
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
|
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
|
||||||
`docs/DISASTER-RECOVERY.md`.
|
`docs/DISASTER-RECOVERY.md`.
|
||||||
@@ -169,6 +169,7 @@ Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECO
|
|||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
|
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
|
||||||
| The same, 1.0.2 | Hetzner Storage Box (SMB), Germany → Canada | PostgreSQL → PostgreSQL | 150,755 files restored in about 29 h (7-13 MiB/s), going on after a lost connection; sample identical, empty folders not restored ([#27](https://github.com/dillardblom/ng_backup/issues/27)) |
|
| The same, 1.0.2 | Hetzner Storage Box (SMB), Germany → Canada | PostgreSQL → PostgreSQL | 150,755 files restored in about 29 h (7-13 MiB/s), going on after a lost connection; sample identical, empty folders not restored ([#27](https://github.com/dillardblom/ng_backup/issues/27)) |
|
||||||
|
| 1.0.3, small simulations of 2 GB: as planned, with the restore killed halfway, and with a disk that runs full | Local SFTP server | PostgreSQL → PostgreSQL and MariaDB | all 3,101 user files and all empty folders identical; users, a share and a calendar back |
|
||||||
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
|
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
|
||||||
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
|
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
|
||||||
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
|
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
|
||||||
@@ -265,15 +266,12 @@ What is planned, without dates. Ideas and votes are welcome in the
|
|||||||
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
|
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
|
||||||
a 👍 on its issue helps decide what comes first.
|
a 👍 on its issue helps decide what comes first.
|
||||||
|
|
||||||
**1.0.3**
|
|
||||||
- Empty folders in the backup and the restore
|
|
||||||
([#27](https://github.com/dillardblom/ng_backup/issues/27)).
|
|
||||||
|
|
||||||
**1.1.0**
|
**1.1.0**
|
||||||
- Restore without the recovery kit: the backup key from the location itself, opened with one of
|
- Restore without the recovery kit: the backup key from the location itself, opened with one of
|
||||||
your passphrases.
|
your passphrases.
|
||||||
- A schedule per location, also "every N hours".
|
- A schedule per location, also "every N hours".
|
||||||
- Locations backed up one after the other instead of at the same time.
|
- Locations backed up one after the other instead of at the same time.
|
||||||
|
- All folders get their original modification time back on a restore, not only the empty ones.
|
||||||
- A note on a restore point that you can change afterwards.
|
- A note on a restore point that you can change afterwards.
|
||||||
- Pinning a restore point, so the retention policy never removes it (for example the one from
|
- Pinning a restore point, so the retention policy never removes it (for example the one from
|
||||||
before an upgrade).
|
before an upgrade).
|
||||||
@@ -293,9 +291,11 @@ a 👍 on its issue helps decide what comes first.
|
|||||||
|
|
||||||
On the backlog; know them before you rely on NG Backup:
|
On the backlog; know them before you rely on NG Backup:
|
||||||
|
|
||||||
- **Empty folders do not come back.** A backup only records files, so a folder without files is
|
- **Empty folders only come back from backups made with 1.0.3 or newer.** Older restore points do
|
||||||
not restored. To be fixed in 1.0.3
|
not contain them.
|
||||||
([#27](https://github.com/dillardblom/ng_backup/issues/27)).
|
- **Folders with files in them only get their original modification time back from backups made with
|
||||||
|
1.1.0 or newer.** From older restore points they show the time of the restore; the files keep theirs.
|
||||||
|
A restore through Nextcloud into a folder that is already there leaves that folder's time alone.
|
||||||
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
|
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
|
||||||
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
|
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
|
||||||
External storage app are not in the data directory and are **not included**, without a warning.
|
External storage app are not in the data directory and are **not included**, without a warning.
|
||||||
@@ -330,10 +330,12 @@ On the backlog; know them before you rely on NG Backup:
|
|||||||
| `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation |
|
| `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation |
|
||||||
| `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots |
|
| `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots |
|
||||||
| `backup:target:backends/add/edit/list/remove/append-only` | backup locations |
|
| `backup:target:backends/add/edit/list/remove/append-only` | backup locations |
|
||||||
|
| `backup:target:schedule` | when a location is backed up: off, daily, or every N hours |
|
||||||
| `backup:run`, `backup:status`, `backup:list` | back up, status (runs of the last 7 days; `--all` for all), restore points |
|
| `backup:run`, `backup:status`, `backup:list` | back up, status (runs of the last 7 days; `--all` for all), restore points |
|
||||||
| `backup:cancel <run>` | stop a running backup |
|
| `backup:cancel <run>` | stop a running backup |
|
||||||
| `backup:browse`, `backup:restore` | browse and restore |
|
| `backup:browse`, `backup:restore` | browse and restore |
|
||||||
| `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash |
|
| `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash |
|
||||||
|
| `backup:pin`, `backup:note` | keep a restore point, a note you can change later |
|
||||||
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
|
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
|
||||||
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
|
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
|
||||||
| `backup:key:import` | import a recovery kit on a fresh installation |
|
| `backup:key:import` | import a recovery kit on a fresh installation |
|
||||||
|
|||||||
+8
-5
@@ -45,7 +45,7 @@ welcome: use the "Share your experience" link on the admin page.
|
|||||||
|
|
||||||
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
|
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
|
||||||
you need help with restoring a backup after a crash.]]></description>
|
you need help with restoring a backup after a crash.]]></description>
|
||||||
<version>1.0.2</version>
|
<version>1.0.3</version>
|
||||||
<licence>agpl</licence>
|
<licence>agpl</licence>
|
||||||
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
|
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
|
||||||
<namespace>NgBackup</namespace>
|
<namespace>NgBackup</namespace>
|
||||||
@@ -56,10 +56,10 @@ you need help with restoring a backup after a crash.]]></description>
|
|||||||
<website>https://github.com/dillardblom/ng_backup</website>
|
<website>https://github.com/dillardblom/ng_backup</website>
|
||||||
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
|
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
|
||||||
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
|
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
|
||||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/1-backups.png</screenshot>
|
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/1-backups.png</screenshot>
|
||||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/2-restore.png</screenshot>
|
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/2-restore.png</screenshot>
|
||||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/3-settings.png</screenshot>
|
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/3-settings.png</screenshot>
|
||||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.2/screenshots/4-bin.png</screenshot>
|
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/4-bin.png</screenshot>
|
||||||
<dependencies>
|
<dependencies>
|
||||||
<php min-version="8.1"/>
|
<php min-version="8.1"/>
|
||||||
<lib>sodium</lib>
|
<lib>sodium</lib>
|
||||||
@@ -83,11 +83,14 @@ you need help with restoring a backup after a crash.]]></description>
|
|||||||
<command>OCA\NgBackup\Command\ListSnapshots</command>
|
<command>OCA\NgBackup\Command\ListSnapshots</command>
|
||||||
<command>OCA\NgBackup\Command\Browse</command>
|
<command>OCA\NgBackup\Command\Browse</command>
|
||||||
<command>OCA\NgBackup\Command\Restore</command>
|
<command>OCA\NgBackup\Command\Restore</command>
|
||||||
|
<command>OCA\NgBackup\Command\Pin</command>
|
||||||
|
<command>OCA\NgBackup\Command\Note</command>
|
||||||
<command>OCA\NgBackup\Command\Prune</command>
|
<command>OCA\NgBackup\Command\Prune</command>
|
||||||
<command>OCA\NgBackup\Command\Retention</command>
|
<command>OCA\NgBackup\Command\Retention</command>
|
||||||
<command>OCA\NgBackup\Command\TargetAppendOnly</command>
|
<command>OCA\NgBackup\Command\TargetAppendOnly</command>
|
||||||
<command>OCA\NgBackup\Command\TargetTrustCurrent</command>
|
<command>OCA\NgBackup\Command\TargetTrustCurrent</command>
|
||||||
<command>OCA\NgBackup\Command\TargetLimit</command>
|
<command>OCA\NgBackup\Command\TargetLimit</command>
|
||||||
|
<command>OCA\NgBackup\Command\TargetSchedule</command>
|
||||||
<command>OCA\NgBackup\Command\Trash</command>
|
<command>OCA\NgBackup\Command\Trash</command>
|
||||||
<command>OCA\NgBackup\Command\TrashRestore</command>
|
<command>OCA\NgBackup\Command\TrashRestore</command>
|
||||||
<command>OCA\NgBackup\Command\KeySlots</command>
|
<command>OCA\NgBackup\Command\KeySlots</command>
|
||||||
|
|||||||
+3
-1
@@ -25,8 +25,10 @@ return [
|
|||||||
['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'],
|
['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'],
|
||||||
['name' => 'AdminApi#startBackup', 'url' => '/api/backups', 'verb' => 'POST'],
|
['name' => 'AdminApi#startBackup', 'url' => '/api/backups', 'verb' => 'POST'],
|
||||||
['name' => 'AdminApi#tick', 'url' => '/api/tick', 'verb' => 'POST'],
|
['name' => 'AdminApi#tick', 'url' => '/api/tick', 'verb' => 'POST'],
|
||||||
['name' => 'AdminApi#setSchedule', 'url' => '/api/schedule', 'verb' => 'PUT'],
|
['name' => 'AdminApi#setTargetSchedule', 'url' => '/api/targets/{id}/schedule', 'verb' => 'PUT'],
|
||||||
['name' => 'AdminApi#snapshots', 'url' => '/api/targets/{targetId}/snapshots', 'verb' => 'GET'],
|
['name' => 'AdminApi#snapshots', 'url' => '/api/targets/{targetId}/snapshots', 'verb' => 'GET'],
|
||||||
|
['name' => 'AdminApi#setSnapshotNote', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/note', 'verb' => 'PUT'],
|
||||||
|
['name' => 'AdminApi#setSnapshotPinned', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/pin', 'verb' => 'PUT'],
|
||||||
['name' => 'AdminApi#browse', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/browse', 'verb' => 'GET'],
|
['name' => 'AdminApi#browse', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/browse', 'verb' => 'GET'],
|
||||||
['name' => 'AdminApi#startRestore', 'url' => '/api/restores', 'verb' => 'POST'],
|
['name' => 'AdminApi#startRestore', 'url' => '/api/restores', 'verb' => 'POST'],
|
||||||
],
|
],
|
||||||
|
|||||||
@@ -117,7 +117,18 @@ first. If none is better, `--salvage` restores whatever can be read:
|
|||||||
|
|
||||||
occ backup:restore:full offsite <snapshot> --install-apps --salvage
|
occ backup:restore:full offsite <snapshot> --install-apps --salvage
|
||||||
|
|
||||||
|
If a restore already stopped halfway, run it as in step 8: maintenance mode off, with `--force` and
|
||||||
|
without `--install-apps`:
|
||||||
|
|
||||||
|
occ maintenance:mode --off && occ backup:restore:full offsite <snapshot> --salvage --force
|
||||||
|
|
||||||
- Files that cannot be read are left out and listed at the end; everything else is restored.
|
- Files that cannot be read are left out and listed at the end; everything else is restored.
|
||||||
|
- A full disk is not damage: the restore stops, also with `--salvage`, and says so. Make room and
|
||||||
|
run the restore again as in step 8.
|
||||||
|
- **On version 1.0.2, check the disk first if the reason is "Size mismatch".** There a full disk
|
||||||
|
looks like damaged data: the message points at `--salvage`, and with `--salvage` the restore goes
|
||||||
|
on and lists the remaining files. The backup itself is fine. Update NG Backup, make room and run
|
||||||
|
the restore again as in step 8.
|
||||||
- If the database cannot be restored (its dump is damaged, or it does not fit this
|
- If the database cannot be restored (its dump is damaged, or it does not fit this
|
||||||
installation), the database stays as the fresh installation made it, and the data directory
|
installation), the database stays as the fresh installation made it, and the data directory
|
||||||
and config of the backup go to a folder of their own in the data directory,
|
and config of the backup go to a folder of their own in the data directory,
|
||||||
@@ -140,9 +151,12 @@ The command then ends with exit code 2 instead of 0. The same option exists for
|
|||||||
- From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`,
|
- From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`,
|
||||||
`redis`), which point at AIO's containers.
|
`redis`), which point at AIO's containers.
|
||||||
- Log in, open a few files, a calendar and a contact.
|
- Log in, open a few files, a calendar and a contact.
|
||||||
- Empty folders do not come back: a backup only records files (to be fixed in 1.0.3, see
|
- `occ files:scan` can report many folders as "Updated". That is expected: a restore does not yet put
|
||||||
[issue #27](https://github.com/dillardblom/ng_backup/issues/27)). `occ files:scan` then removes
|
back the modification time of folders with files in them, so they show the time of the restore.
|
||||||
them from the file cache, so it reports them as removed.
|
The files themselves keep their time and content.
|
||||||
|
- Empty folders only come back from backups made with version 1.0.3 or newer. With an older
|
||||||
|
restore point they are gone: `occ files:scan` then removes them from the file cache, so it reports
|
||||||
|
them as removed.
|
||||||
- Start background jobs again (cron), as in step 2 in reverse.
|
- Start background jobs again (cron), as in step 2 in reverse.
|
||||||
|
|
||||||
**10. Point your domain at the new server** and check that clients reconnect.
|
**10. Point your domain at the new server** and check that clients reconnect.
|
||||||
|
|||||||
+8
-1
@@ -79,7 +79,9 @@ occ backup:list
|
|||||||
Later runs upload only changed data. An unchanged run uploads (almost) nothing; the output shows
|
Later runs upload only changed data. An unchanged run uploads (almost) nothing; the output shows
|
||||||
the number of uploaded bytes, so check that value when you test incremental behaviour.
|
the number of uploaded bytes, so check that value when you test incremental behaviour.
|
||||||
|
|
||||||
Schedule `occ backup:run` from system cron (or use the Administration settings, if you prefer).
|
Set when each location is backed up in the Administration settings, or with
|
||||||
|
`occ backup:target:schedule offsite 02:00` (once a day), `02:00/6` (every 6 hours from 02:00) or `off`.
|
||||||
|
Locations are backed up one after the other.
|
||||||
|
|
||||||
## 4. Check the backup regularly
|
## 4. Check the backup regularly
|
||||||
|
|
||||||
@@ -173,8 +175,13 @@ occ backup:prune offsite --dry-run # show first
|
|||||||
occ backup:prune offsite # apply the policy; forgotten snapshots go to the trash first
|
occ backup:prune offsite # apply the policy; forgotten snapshots go to the trash first
|
||||||
occ backup:trash # list the trash
|
occ backup:trash # list the trash
|
||||||
occ backup:trash:restore offsite <id>
|
occ backup:trash:restore offsite <id>
|
||||||
|
occ backup:pin offsite <id> # keep this restore point; --unpin to undo
|
||||||
|
occ backup:note offsite <id> "text" # a note you can change later; without text it is removed
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The retention policy leaves a pinned restore point alone, and a pinned one does not take the place of
|
||||||
|
a daily, weekly or monthly one. Pin and note are stored on the location, so a new server sees them too.
|
||||||
|
|
||||||
Snapshots are kept according to the policy, and their data is deleted only after the deletion delay
|
Snapshots are kept according to the policy, and their data is deleted only after the deletion delay
|
||||||
(set with `backup:key:init --delete-delay-days`). `occ backup:prune offsite --dry-run` shows what would be
|
(set with `backup:key:init --delete-delay-days`). `occ backup:prune offsite --dry-run` shows what would be
|
||||||
removed without changing anything.
|
removed without changing anything.
|
||||||
|
|||||||
@@ -124,7 +124,13 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
|||||||
// null: not supported or failing for now; false: not opened, which need not mean
|
// null: not supported or failing for now; false: not opened, which need not mean
|
||||||
// missing. Either way the storage's own stream decides.
|
// missing. Either way the storage's own stream decides.
|
||||||
}
|
}
|
||||||
|
try {
|
||||||
$fh = $this->storage->fopen($this->abs($path), 'r');
|
$fh = $this->storage->fopen($this->abs($path), 'r');
|
||||||
|
} catch (\Exception $e) {
|
||||||
|
// Some storages throw for what is not a missing file (SMB: a connection that broke).
|
||||||
|
// As a BackendException, the caller reads once more and --salvage can go on.
|
||||||
|
throw new BackendException('Cannot read ' . $path . ': ' . $e->getMessage(), 0, $e);
|
||||||
|
}
|
||||||
if ($fh === false) {
|
if ($fh === false) {
|
||||||
throw new BackendException('Not found: ' . $path);
|
throw new BackendException('Not found: ' . $path);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -43,25 +43,27 @@ final class BackupJob extends TimedJob {
|
|||||||
$this->setTimeSensitivity(IJob::TIME_SENSITIVE);
|
$this->setTimeSensitivity(IJob::TIME_SENSITIVE);
|
||||||
}
|
}
|
||||||
|
|
||||||
private const MAX_DAILY_ATTEMPTS = 3;
|
|
||||||
|
|
||||||
protected function run($argument): void {
|
protected function run($argument): void {
|
||||||
$budget = PHP_SAPI === 'cli' ? 240.0 : 20.0;
|
$budget = PHP_SAPI === 'cli' ? 240.0 : 20.0;
|
||||||
$deadline = microtime(true) + $budget;
|
$deadline = microtime(true) + $budget;
|
||||||
$this->startScheduled();
|
$this->startScheduled();
|
||||||
foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) {
|
// One after the other: runs that read the same disks at the same time are not faster together.
|
||||||
|
foreach ($this->backups->runsInTurn() as $run) {
|
||||||
while (microtime(true) < $deadline && $run->getStatus() === 'running') {
|
while (microtime(true) < $deadline && $run->getStatus() === 'running') {
|
||||||
$next = $this->backups->step($run, min($deadline, microtime(true) + $budget));
|
$next = $this->backups->step($run, min($deadline, microtime(true) + $budget));
|
||||||
if ($next === $run) {
|
if ($next === $run) {
|
||||||
break; // step() hands back the same object when another process holds the run's lease
|
break 2; // step() hands back the same object when another process holds the run's lease
|
||||||
}
|
}
|
||||||
$run = $next;
|
$run = $next;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
$this->failStaleRestores();
|
$this->failStaleRestores();
|
||||||
$this->weeklyPrune();
|
$this->weeklyPrune();
|
||||||
foreach ($this->targets->list() as $t) {
|
$targets = $this->targets->list();
|
||||||
|
foreach ($targets as $t) {
|
||||||
$this->alerts->checkStale($t);
|
$this->alerts->checkStale($t);
|
||||||
|
$othersActive = array_filter($targets, static fn ($o) => $o->getId() !== $t->getId() && (string)$o->getSchedule() !== '') !== [];
|
||||||
|
$this->alerts->checkOff($t, $othersActive);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,21 +104,18 @@ final class BackupJob extends TimedJob {
|
|||||||
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
|
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */
|
/** Start the backups the schedules of the locations ask for; they run in turn (see LocationSchedule). */
|
||||||
private function startScheduled(): void {
|
private function startScheduled(): void {
|
||||||
$todayAt = $this->schedule->todayAt();
|
$now = time();
|
||||||
if ($todayAt === null || time() < $todayAt) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
foreach ($this->targets->list() as $t) {
|
foreach ($this->targets->list() as $t) {
|
||||||
$last = $this->runs->lastFinished($t->getId(), BackupService::KIND_FULL);
|
$moment = \OCA\NgBackup\Service\LocationSchedule::lastMoment((string)$t->getSchedule(), $this->schedule->zoneOr($t->getScheduleZone()), $now);
|
||||||
if ($this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== [] || ($last !== null && $last->getStartedAt() >= $todayAt)) {
|
if ($moment === null || $this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== []) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
// After a failure: retry at most MAX_DAILY_ATTEMPTS times a day, an hour apart, instead
|
$attempts = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $moment);
|
||||||
// of starting (and notifying about) a new full run every five minutes all day.
|
if (!\OCA\NgBackup\Service\LocationSchedule::isDue($moment, (int)$t->getScheduleSince(),
|
||||||
$today = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $todayAt);
|
$this->runs->lastFinished($t->getId(), BackupService::KIND_FULL)?->getStartedAt(),
|
||||||
if (count($today) >= self::MAX_DAILY_ATTEMPTS || ($today !== [] && time() - (int)$today[0]->getFinishedAt() < 3600)) {
|
count($attempts), ($attempts[0] ?? null)?->getFinishedAt(), $now)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
|
|||||||
@@ -40,14 +40,14 @@ final class ListSnapshots extends Command {
|
|||||||
foreach ($targets as $t) {
|
foreach ($targets as $t) {
|
||||||
foreach ($this->snapshots->findForTarget($t->getId()) as $s) {
|
foreach ($this->snapshots->findForTarget($t->getId()) as $s) {
|
||||||
$rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(),
|
$rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(),
|
||||||
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getLabel() ?? ''];
|
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getPinned() ? 'yes' : '', $s->getLabel() ?? '', $s->getNote() ?? ''];
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if ($rows === []) {
|
if ($rows === []) {
|
||||||
$output->writeln('No snapshots yet (occ backup:run).');
|
$output->writeln('No snapshots yet (occ backup:run).');
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
(new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'label'])->setRows($rows)->render();
|
(new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'pinned', 'label', 'note'])->setRows($rows)->render();
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,44 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\PruneService;
|
||||||
|
use OCA\NgBackup\Service\TargetService;
|
||||||
|
use Symfony\Component\Console\Command\Command;
|
||||||
|
use Symfony\Component\Console\Input\InputArgument;
|
||||||
|
use Symfony\Component\Console\Input\InputInterface;
|
||||||
|
use Symfony\Component\Console\Output\OutputInterface;
|
||||||
|
|
||||||
|
final class Note extends Command {
|
||||||
|
public function __construct(
|
||||||
|
private TargetService $targets,
|
||||||
|
private PruneService $prune,
|
||||||
|
) {
|
||||||
|
parent::__construct();
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function configure(): void {
|
||||||
|
$this->setName('backup:note')
|
||||||
|
->setDescription('Set or remove the note on a snapshot')
|
||||||
|
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
||||||
|
->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)')
|
||||||
|
->addArgument('note', InputArgument::OPTIONAL, 'The note, at most ' . PruneService::NOTE_MAX . ' characters; without it the note is removed', '');
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$note = (string)$input->getArgument('note');
|
||||||
|
try {
|
||||||
|
$this->prune->setNote($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), $note);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
$output->writeln(trim($note) === '' ? 'Note removed.' : 'Note saved.');
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\PruneService;
|
||||||
|
use OCA\NgBackup\Service\TargetService;
|
||||||
|
use Symfony\Component\Console\Command\Command;
|
||||||
|
use Symfony\Component\Console\Input\InputArgument;
|
||||||
|
use Symfony\Component\Console\Input\InputInterface;
|
||||||
|
use Symfony\Component\Console\Input\InputOption;
|
||||||
|
use Symfony\Component\Console\Output\OutputInterface;
|
||||||
|
|
||||||
|
final class Pin extends Command {
|
||||||
|
public function __construct(
|
||||||
|
private TargetService $targets,
|
||||||
|
private PruneService $prune,
|
||||||
|
) {
|
||||||
|
parent::__construct();
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function configure(): void {
|
||||||
|
$this->setName('backup:pin')
|
||||||
|
->setDescription('Keep a snapshot: the retention policy does not remove a pinned one')
|
||||||
|
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
||||||
|
->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)')
|
||||||
|
->addOption('unpin', null, InputOption::VALUE_NONE, 'Leave the snapshot to the retention policy again');
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$unpin = (bool)$input->getOption('unpin');
|
||||||
|
try {
|
||||||
|
$this->prune->setPinned($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), !$unpin);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
$output->writeln($unpin ? 'Snapshot unpinned; the retention policy applies to it again.' : 'Snapshot pinned; the retention policy leaves it alone.');
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -79,6 +79,9 @@ HELP);
|
|||||||
if (($f['unreadable'] ?? 0) > 0) {
|
if (($f['unreadable'] ?? 0) > 0) {
|
||||||
$output->writeln(sprintf('<comment> warning: %d unreadable files or folders are missing from this backup (see the server log)</comment>', $f['unreadable']));
|
$output->writeln(sprintf('<comment> warning: %d unreadable files or folders are missing from this backup (see the server log)</comment>', $f['unreadable']));
|
||||||
}
|
}
|
||||||
|
if (($f['badNames'] ?? 0) > 0) {
|
||||||
|
$output->writeln(sprintf('<comment> warning: %d files or folders are missing from this backup because their name is not supported; rename them and the next backup includes them (see the server log)</comment>', $f['badNames']));
|
||||||
|
}
|
||||||
if (($f['changing'] ?? 0) > 0) {
|
if (($f['changing'] ?? 0) > 0) {
|
||||||
$output->writeln(sprintf('<comment> warning: %d files kept changing and were saved as read (see the server log)</comment>', $f['changing']));
|
$output->writeln(sprintf('<comment> warning: %d files kept changing and were saved as read (see the server log)</comment>', $f['changing']));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ final class Status extends Command {
|
|||||||
private VerifyService $verify,
|
private VerifyService $verify,
|
||||||
private Schedule $schedule,
|
private Schedule $schedule,
|
||||||
private CronHealth $cron,
|
private CronHealth $cron,
|
||||||
|
private \OCA\NgBackup\Service\BackupService $backups,
|
||||||
) {
|
) {
|
||||||
parent::__construct();
|
parent::__construct();
|
||||||
}
|
}
|
||||||
@@ -61,9 +62,11 @@ final class Status extends Command {
|
|||||||
$rows = [];
|
$rows = [];
|
||||||
$all = (bool)$input->getOption('all');
|
$all = (bool)$input->getOption('all');
|
||||||
$shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400);
|
$shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400);
|
||||||
|
$queued = array_map(static fn (\OCA\NgBackup\Db\Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1));
|
||||||
foreach ($shown as $r) {
|
foreach ($shown as $r) {
|
||||||
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
|
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
|
||||||
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus(),
|
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(),
|
||||||
|
in_array($r->getId(), $queued, true) ? 'queued' : ($r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus()),
|
||||||
Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
|
Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
|
||||||
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
|
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
|
||||||
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
|
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
|
||||||
|
|||||||
@@ -31,7 +31,8 @@ final class TargetList extends Base {
|
|||||||
$rows = [];
|
$rows = [];
|
||||||
foreach ($this->targets->list() as $t) {
|
foreach ($this->targets->list() as $t) {
|
||||||
$row = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
|
$row = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
|
||||||
'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt())];
|
'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt()),
|
||||||
|
'schedule' => (string)$t->getSchedule() !== '' ? $t->getSchedule() : 'off'];
|
||||||
if ($input->getOption('test')) {
|
if ($input->getOption('test')) {
|
||||||
try {
|
try {
|
||||||
$repo = $this->targets->repository($t);
|
$repo = $this->targets->repository($t);
|
||||||
|
|||||||
@@ -0,0 +1,50 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\Schedule;
|
||||||
|
use OCA\NgBackup\Service\TargetService;
|
||||||
|
use Symfony\Component\Console\Command\Command;
|
||||||
|
use Symfony\Component\Console\Input\InputArgument;
|
||||||
|
use Symfony\Component\Console\Input\InputInterface;
|
||||||
|
use Symfony\Component\Console\Input\InputOption;
|
||||||
|
use Symfony\Component\Console\Output\OutputInterface;
|
||||||
|
|
||||||
|
final class TargetSchedule extends Command {
|
||||||
|
public function __construct(
|
||||||
|
private TargetService $targets,
|
||||||
|
private Schedule $schedule,
|
||||||
|
) {
|
||||||
|
parent::__construct();
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function configure(): void {
|
||||||
|
$this->setName('backup:target:schedule')
|
||||||
|
->setDescription('Show or set when a location is backed up')
|
||||||
|
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
||||||
|
->addArgument('schedule', InputArgument::OPTIONAL, '"off", HH:MM for once a day, or HH:MM/N for every N hours from HH:MM; omit to show the current one')
|
||||||
|
->addOption('timezone', null, InputOption::VALUE_REQUIRED, 'Time zone of the time, e.g. Europe/Amsterdam');
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$t = $this->targets->get($input->getArgument('target'));
|
||||||
|
$schedule = $input->getArgument('schedule');
|
||||||
|
$zone = $input->getOption('timezone') ?? $this->schedule->zoneOr($t->getScheduleZone());
|
||||||
|
if ($schedule !== null) {
|
||||||
|
try {
|
||||||
|
$t = $this->targets->setSchedule($t, $schedule, (string)$zone);
|
||||||
|
} catch (\InvalidArgumentException $e) {
|
||||||
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$current = (string)$t->getSchedule();
|
||||||
|
$output->writeln($t->getName() . ': ' . ($current === '' ? 'off' : $current . ' (' . $zone . ')'));
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -59,10 +59,12 @@ final class AdminApiController extends Controller {
|
|||||||
foreach ($this->targets->list() as $t) {
|
foreach ($this->targets->list() as $t) {
|
||||||
$targets[] = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
|
$targets[] = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
|
||||||
'repositoryId' => $t->getRepositoryId(), 'createdAt' => $t->getCreatedAt(), 'appendOnly' => (bool)$t->getAppendOnly(),
|
'repositoryId' => $t->getRepositoryId(), 'createdAt' => $t->getCreatedAt(), 'appendOnly' => (bool)$t->getAppendOnly(),
|
||||||
'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t)];
|
'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t),
|
||||||
|
'schedule' => (string)$t->getSchedule(), 'scheduleZone' => $this->schedule->zoneOr($t->getScheduleZone())];
|
||||||
}
|
}
|
||||||
$shown = $this->runs->findShown($allRuns ? null : time() - \OCA\NgBackup\Command\Status::RECENT_DAYS * 86400);
|
$shown = $this->runs->findShown($allRuns ? null : time() - \OCA\NgBackup\Command\Status::RECENT_DAYS * 86400);
|
||||||
$runs = array_map(fn (Run $r) => $this->runJson($r), $shown);
|
$queued = array_flip(array_map(static fn (Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1)));
|
||||||
|
$runs = array_map(fn (Run $r) => $this->runJson($r) + ['queued' => isset($queued[$r->getId()])], $shown);
|
||||||
return new JSONResponse([
|
return new JSONResponse([
|
||||||
'key' => ['initialized' => $this->keys->isInitialized(), 'fingerprint' => $this->keys->fingerprint(),
|
'key' => ['initialized' => $this->keys->isInitialized(), 'fingerprint' => $this->keys->fingerprint(),
|
||||||
'confirmation' => $this->keys->recoveryKitConfirmation(), 'everConfirmed' => $this->keys->isInitialized() && $this->keys->everConfirmed(),
|
'confirmation' => $this->keys->recoveryKitConfirmation(), 'everConfirmed' => $this->keys->isInitialized() && $this->keys->everConfirmed(),
|
||||||
@@ -74,14 +76,14 @@ final class AdminApiController extends Controller {
|
|||||||
'runs' => $runs,
|
'runs' => $runs,
|
||||||
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
|
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
|
||||||
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
|
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
|
||||||
'schedule' => $this->schedule->get()['time'],
|
|
||||||
'scheduleTimezone' => $this->schedule->get()['timezone'],
|
|
||||||
'cron' => $this->cron->check(),
|
'cron' => $this->cron->check(),
|
||||||
]);
|
]);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Seconds of backup work one tick() does: well within a normal web request. */
|
/** Seconds of backup work one tick() does: well within a normal web request. */
|
||||||
public const TICK_SECONDS = 10.0;
|
public const TICK_SECONDS = 10.0;
|
||||||
|
/** Seconds one request for the Bin reads, before the page shows how far it is and asks for more. */
|
||||||
|
private const TRASH_PAGE_SECONDS = 5.0;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Move running backups forward from the open admin page, when background jobs do not run on a
|
* Move running backups forward from the open admin page, when background jobs do not run on a
|
||||||
@@ -95,13 +97,16 @@ final class AdminApiController extends Controller {
|
|||||||
}
|
}
|
||||||
$deadline = microtime(true) + self::TICK_SECONDS;
|
$deadline = microtime(true) + self::TICK_SECONDS;
|
||||||
$stepped = 0;
|
$stepped = 0;
|
||||||
foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) {
|
foreach ($this->backups->runsInTurn() as $run) {
|
||||||
if (microtime(true) >= $deadline) {
|
if (microtime(true) >= $deadline) {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
// step() returns at once when the background job (or another tab) holds the run.
|
// step() returns at once when the background job (or another tab) holds the run.
|
||||||
$this->backups->step($run, $deadline);
|
$next = $this->backups->step($run, $deadline);
|
||||||
$stepped++;
|
$stepped++;
|
||||||
|
if ($next->getStatus() === Run::RUNNING) {
|
||||||
|
break; // not finished: the next one waits for its turn
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return new JSONResponse(['stepped' => $stepped]);
|
return new JSONResponse(['stepped' => $stepped]);
|
||||||
}
|
}
|
||||||
@@ -174,15 +179,18 @@ final class AdminApiController extends Controller {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public function trash(int $targetId): JSONResponse {
|
public function trash(int $targetId, string $after = ''): JSONResponse {
|
||||||
try {
|
try {
|
||||||
$delay = $this->keys->deleteDelayDays() * 86400;
|
$delay = $this->keys->deleteDelayDays() * 86400;
|
||||||
|
$repo = $this->targets->repository($this->targets->get((string)$targetId));
|
||||||
|
// In parts, so that the page can show how far it is (a read per removed restore point).
|
||||||
|
$page = $repo->trashPage($after, microtime(true) + self::TRASH_PAGE_SECONDS);
|
||||||
$out = [];
|
$out = [];
|
||||||
foreach ($this->targets->repository($this->targets->get((string)$targetId))->trash() as $id => $info) {
|
foreach ($page['items'] as $id => $info) {
|
||||||
$out[] = ['id' => $id, 'label' => $info['label'], 'time' => $info['time'], 'removedAt' => $info['forgottenAt'],
|
$out[] = ['id' => $id, 'label' => $info['label'], 'time' => $info['time'], 'removedAt' => $info['forgottenAt'],
|
||||||
'by' => $info['by'], 'deleteAfter' => $info['forgottenAt'] + $delay];
|
'by' => $info['by'], 'deleteAfter' => $info['forgottenAt'] + $delay];
|
||||||
}
|
}
|
||||||
return new JSONResponse($out);
|
return new JSONResponse(['total' => $page['total'], 'items' => $out]);
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||||
}
|
}
|
||||||
@@ -270,25 +278,43 @@ final class AdminApiController extends Controller {
|
|||||||
return new JSONResponse($this->runJson($run));
|
return new JSONResponse($this->runJson($run));
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */
|
/** @param string $timezone the administrator's zone (the browser's); empty: the default one */
|
||||||
public function setSchedule(string $time, string $timezone = ''): JSONResponse {
|
public function setTargetSchedule(int $id, string $schedule, string $timezone = ''): JSONResponse {
|
||||||
try {
|
try {
|
||||||
$this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
|
$t = $this->targets->setSchedule($this->targets->get((string)$id), $schedule, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
|
||||||
} catch (\InvalidArgumentException $e) {
|
} catch (\Throwable $e) {
|
||||||
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||||
}
|
}
|
||||||
return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]);
|
return new JSONResponse(['schedule' => $t->getSchedule(), 'scheduleZone' => $t->getScheduleZone()]);
|
||||||
}
|
}
|
||||||
|
|
||||||
public function snapshots(int $targetId): JSONResponse {
|
public function snapshots(int $targetId): JSONResponse {
|
||||||
$out = [];
|
$out = [];
|
||||||
foreach ($this->snapshots->findForTarget($targetId) as $s) {
|
foreach ($this->snapshots->findForTarget($targetId) as $s) {
|
||||||
$out[] = ['id' => $s->getSnapshotId(), 'kind' => $s->getKind(), 'label' => $s->getLabel(), 'createdAt' => $s->getCreatedAt(),
|
$out[] = ['id' => $s->getSnapshotId(), 'kind' => $s->getKind(), 'label' => $s->getLabel(), 'createdAt' => $s->getCreatedAt(),
|
||||||
'files' => $s->getFiles(), 'bytes' => $s->getBytes()];
|
'files' => $s->getFiles(), 'bytes' => $s->getBytes(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()];
|
||||||
}
|
}
|
||||||
return new JSONResponse($out);
|
return new JSONResponse($out);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function setSnapshotNote(int $targetId, string $snapshotId, string $note = ''): JSONResponse {
|
||||||
|
try {
|
||||||
|
$this->prune->setNote($this->targets->get((string)$targetId), $snapshotId, $note);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||||
|
}
|
||||||
|
return new JSONResponse(['ok' => true]);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function setSnapshotPinned(int $targetId, string $snapshotId, bool $pinned = true): JSONResponse {
|
||||||
|
try {
|
||||||
|
$this->prune->setPinned($this->targets->get((string)$targetId), $snapshotId, $pinned);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||||
|
}
|
||||||
|
return new JSONResponse(['ok' => true]);
|
||||||
|
}
|
||||||
|
|
||||||
public function browse(int $targetId, string $snapshotId, string $path = ''): JSONResponse {
|
public function browse(int $targetId, string $snapshotId, string $path = ''): JSONResponse {
|
||||||
try {
|
try {
|
||||||
return new JSONResponse($this->restore->browse($this->targets->get((string)$targetId), $snapshotId, $path));
|
return new JSONResponse($this->restore->browse($this->targets->get((string)$targetId), $snapshotId, $path));
|
||||||
@@ -333,7 +359,7 @@ final class AdminApiController extends Controller {
|
|||||||
if ($r->getSnapshotId() !== null) {
|
if ($r->getSnapshotId() !== null) {
|
||||||
try {
|
try {
|
||||||
$s = $this->snapshots->findOne($r->getTargetId(), $r->getSnapshotId());
|
$s = $this->snapshots->findOne($r->getTargetId(), $r->getSnapshotId());
|
||||||
$snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel()];
|
$snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()];
|
||||||
} catch (\Throwable) {
|
} catch (\Throwable) {
|
||||||
// Snapshot already pruned or trashed: the run keeps its id, just not the date.
|
// Snapshot already pruned or trashed: the run keeps its id, just not the date.
|
||||||
}
|
}
|
||||||
|
|||||||
+41
-13
@@ -157,17 +157,27 @@ final class DbDumper {
|
|||||||
}
|
}
|
||||||
// Tables the backup knows the structure of are created first (DDL is not transactional
|
// Tables the backup knows the structure of are created first (DDL is not transactional
|
||||||
// everywhere); tables it does not know are left out, as asked.
|
// everywhere); tables it does not know are left out, as asked.
|
||||||
|
try {
|
||||||
foreach ($missing as $table) {
|
foreach ($missing as $table) {
|
||||||
if (isset($manifest['tables'][$table]['schema'])) {
|
if (isset($manifest['tables'][$table]['schema'])) {
|
||||||
|
$report['created'][] = $table; // first: creating it can stop halfway (its indexes)
|
||||||
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
|
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
|
||||||
$report['created'][] = $table;
|
|
||||||
} else {
|
} else {
|
||||||
$report['missing'][] = $table;
|
$report['missing'][] = $table;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$this->dropCreatedTables($report);
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
$total = 0;
|
$total = 0;
|
||||||
$foreignKeys = [];
|
$foreignKeys = [];
|
||||||
|
try {
|
||||||
$this->db->beginTransaction();
|
$this->db->beginTransaction();
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$this->dropCreatedTables($report);
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
try {
|
try {
|
||||||
$foreignKeys = $this->suspendForeignKeys(); // first: on SQLite it also takes the write lock
|
$foreignKeys = $this->suspendForeignKeys(); // first: on SQLite it also takes the write lock
|
||||||
foreach ($manifest['tables'] as $table => $info) {
|
foreach ($manifest['tables'] as $table => $info) {
|
||||||
@@ -220,9 +230,19 @@ final class DbDumper {
|
|||||||
$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
|
$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
|
||||||
$this->resumeForeignKeys($foreignKeys);
|
$this->resumeForeignKeys($foreignKeys);
|
||||||
$this->db->commit();
|
$this->db->commit();
|
||||||
// Foreign keys of the tables created above, now their rows are in. Not inside the
|
} catch (\Throwable $e) {
|
||||||
// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is
|
$this->db->rollBack();
|
||||||
// restored either way.
|
// PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session
|
||||||
|
// switch must be turned on again by hand.
|
||||||
|
if (in_array('SET FOREIGN_KEY_CHECKS = 1', $foreignKeys, true)) {
|
||||||
|
$this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 1');
|
||||||
|
}
|
||||||
|
$this->dropCreatedTables($report);
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
// Committed. Foreign keys of the tables created above, now their rows are in. Not inside the
|
||||||
|
// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is restored
|
||||||
|
// either way, and the created tables stay.
|
||||||
foreach ($report['created'] as $table) {
|
foreach ($report['created'] as $table) {
|
||||||
$structure = $manifest['tables'][$table]['schema'] ?? [];
|
$structure = $manifest['tables'][$table]['schema'] ?? [];
|
||||||
foreach ($structure['foreignKeys'] ?? [] as $fk) {
|
foreach ($structure['foreignKeys'] ?? [] as $fk) {
|
||||||
@@ -235,18 +255,26 @@ final class DbDumper {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} catch (\Throwable $e) {
|
|
||||||
$this->db->rollBack();
|
|
||||||
// PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session
|
|
||||||
// switch must be turned on again by hand.
|
|
||||||
if (in_array('SET FOREIGN_KEY_CHECKS = 1', $foreignKeys, true)) {
|
|
||||||
$this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 1');
|
|
||||||
}
|
|
||||||
throw $e;
|
|
||||||
}
|
|
||||||
return $total;
|
return $total;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The restore did not go through: a rollback does not undo CREATE TABLE everywhere. Left behind,
|
||||||
|
* the empty tables of an app would stop that app's installation later ("table already exists").
|
||||||
|
*
|
||||||
|
* @param array{created: list<string>, ...} $report
|
||||||
|
*/
|
||||||
|
private function dropCreatedTables(array &$report): void {
|
||||||
|
foreach ($report['created'] as $table) {
|
||||||
|
try {
|
||||||
|
$this->db->createSchemaManager()->dropTable($table);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
// the error that brought us here is the one to report; a run again uses the table
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$report['created'] = [];
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The table prefix of the instance the dump was made on (from its appconfig table), or null
|
* The table prefix of the instance the dump was made on (from its appconfig table), or null
|
||||||
* when the dump has none.
|
* when the dump has none.
|
||||||
|
|||||||
@@ -24,6 +24,10 @@ use OCP\AppFramework\Db\Entity;
|
|||||||
* @method void setFiles(?int $v)
|
* @method void setFiles(?int $v)
|
||||||
* @method int|null getBytes()
|
* @method int|null getBytes()
|
||||||
* @method void setBytes(?int $v)
|
* @method void setBytes(?int $v)
|
||||||
|
* @method string|null getNote()
|
||||||
|
* @method void setNote(?string $v)
|
||||||
|
* @method bool|null getPinned()
|
||||||
|
* @method void setPinned(?bool $v)
|
||||||
*/
|
*/
|
||||||
final class Snapshot extends Entity {
|
final class Snapshot extends Entity {
|
||||||
protected int $targetId = 0;
|
protected int $targetId = 0;
|
||||||
@@ -33,8 +37,11 @@ final class Snapshot extends Entity {
|
|||||||
protected int $createdAt = 0;
|
protected int $createdAt = 0;
|
||||||
protected ?int $files = null;
|
protected ?int $files = null;
|
||||||
protected ?int $bytes = null;
|
protected ?int $bytes = null;
|
||||||
|
protected ?string $note = null;
|
||||||
|
protected ?bool $pinned = null;
|
||||||
|
|
||||||
public function __construct() {
|
public function __construct() {
|
||||||
|
$this->addType('pinned', 'boolean');
|
||||||
$this->addType('targetId', 'integer');
|
$this->addType('targetId', 'integer');
|
||||||
$this->addType('createdAt', 'integer');
|
$this->addType('createdAt', 'integer');
|
||||||
$this->addType('files', 'integer');
|
$this->addType('files', 'integer');
|
||||||
|
|||||||
@@ -28,6 +28,12 @@ use OCP\AppFramework\Db\Entity;
|
|||||||
* @method void setMaxBytes(?int $v)
|
* @method void setMaxBytes(?int $v)
|
||||||
* @method int getCreatedAt()
|
* @method int getCreatedAt()
|
||||||
* @method void setCreatedAt(int $v)
|
* @method void setCreatedAt(int $v)
|
||||||
|
* @method string|null getSchedule()
|
||||||
|
* @method void setSchedule(?string $v)
|
||||||
|
* @method string|null getScheduleZone()
|
||||||
|
* @method void setScheduleZone(?string $v)
|
||||||
|
* @method int|null getScheduleSince()
|
||||||
|
* @method void setScheduleSince(?int $v)
|
||||||
*/
|
*/
|
||||||
final class Target extends Entity {
|
final class Target extends Entity {
|
||||||
protected string $name = '';
|
protected string $name = '';
|
||||||
@@ -39,8 +45,12 @@ final class Target extends Entity {
|
|||||||
protected int $createdAt = 0;
|
protected int $createdAt = 0;
|
||||||
protected ?bool $appendOnly = null;
|
protected ?bool $appendOnly = null;
|
||||||
protected ?int $maxBytes = null;
|
protected ?int $maxBytes = null;
|
||||||
|
protected ?string $schedule = null;
|
||||||
|
protected ?string $scheduleZone = null;
|
||||||
|
protected ?int $scheduleSince = null;
|
||||||
|
|
||||||
public function __construct() {
|
public function __construct() {
|
||||||
|
$this->addType('scheduleSince', 'integer');
|
||||||
$this->addType('createdAt', 'integer');
|
$this->addType('createdAt', 'integer');
|
||||||
$this->addType('appendOnly', 'boolean');
|
$this->addType('appendOnly', 'boolean');
|
||||||
$this->addType('maxBytes', 'integer');
|
$this->addType('maxBytes', 'integer');
|
||||||
|
|||||||
+80
-3
@@ -52,12 +52,51 @@ final class BackupRun {
|
|||||||
$stats = $state['stats'];
|
$stats = $state['stats'];
|
||||||
$walker = new TreeWalker($state['roots'], $state['exclude'], function (string $logical) use (&$state, &$stats): void {
|
$walker = new TreeWalker($state['roots'], $state['exclude'], function (string $logical) use (&$state, &$stats): void {
|
||||||
self::recordUnreadable($state, $stats, $logical);
|
self::recordUnreadable($state, $stats, $logical);
|
||||||
});
|
}, true);
|
||||||
$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
|
$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
|
||||||
$lines = [];
|
$lines = [];
|
||||||
|
$emptyDirs = [];
|
||||||
|
$folders = [];
|
||||||
$finishedWalk = true;
|
$finishedWalk = true;
|
||||||
|
|
||||||
foreach ($walker->walk($state['after']) as $logical => $path) {
|
foreach ($walker->walk($state['after']) as $logical => $path) {
|
||||||
|
$safe = mb_check_encoding($logical, 'UTF-8') && Repository::isSafePath($logical);
|
||||||
|
if ($walker->atFolderWithContent()) {
|
||||||
|
// Only its time: what is in it is reported by itself, and a bad name in the path of
|
||||||
|
// each of those is what gets reported. The cursor stays for such a name, see below.
|
||||||
|
$mtime = $safe ? @filemtime($path) : false;
|
||||||
|
if ($safe && $mtime !== false) {
|
||||||
|
$folders[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR);
|
||||||
|
$state['after'] = $logical;
|
||||||
|
if (microtime(true) >= $deadline) {
|
||||||
|
$finishedWalk = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!$safe) {
|
||||||
|
// A name a snapshot cannot hold (a backslash, a control character, not UTF-8). One
|
||||||
|
// such path must not make a backup fail, or the whole snapshot unreadable later.
|
||||||
|
// The cursor stays where it is: the state is JSON, which cannot hold such a name. A
|
||||||
|
// next step meets it again, and it is reported once.
|
||||||
|
self::recordBadName($state, $stats, $logical);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (is_dir($path)) {
|
||||||
|
$mtime = @filemtime($path);
|
||||||
|
if ($mtime === false) {
|
||||||
|
continue; // removed while walking
|
||||||
|
}
|
||||||
|
$emptyDirs[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR);
|
||||||
|
$stats['emptyDirs'] = ($stats['emptyDirs'] ?? 0) + 1;
|
||||||
|
$state['after'] = $logical;
|
||||||
|
if (microtime(true) >= $deadline) {
|
||||||
|
$finishedWalk = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
$version = Repository::fileVersion($path);
|
$version = Repository::fileVersion($path);
|
||||||
if ($version === 'missing') {
|
if ($version === 'missing') {
|
||||||
continue; // deleted while walking
|
continue; // deleted while walking
|
||||||
@@ -136,6 +175,16 @@ final class BackupRun {
|
|||||||
$writer->write(implode("\n", $lines) . "\n");
|
$writer->write(implode("\n", $lines) . "\n");
|
||||||
$state['segments'][] = $writer->finish();
|
$state['segments'][] = $writer->finish();
|
||||||
}
|
}
|
||||||
|
if ($emptyDirs !== []) {
|
||||||
|
$writer = $repo->blobWriter($stats);
|
||||||
|
$writer->write(implode("\n", $emptyDirs) . "\n");
|
||||||
|
$state['dirSegments'][] = $writer->finish();
|
||||||
|
}
|
||||||
|
if ($folders !== []) {
|
||||||
|
$writer = $repo->blobWriter($stats);
|
||||||
|
$writer->write(implode("\n", $folders) . "\n");
|
||||||
|
$state['folderSegments'][] = $writer->finish();
|
||||||
|
}
|
||||||
$repo->flushPacks();
|
$repo->flushPacks();
|
||||||
$state['parentPos'] = $parent->position();
|
$state['parentPos'] = $parent->position();
|
||||||
$state['stats'] = $stats;
|
$state['stats'] = $stats;
|
||||||
@@ -150,10 +199,22 @@ final class BackupRun {
|
|||||||
if ($state['phase'] === 'finish') {
|
if ($state['phase'] === 'finish') {
|
||||||
// Every segment ends with a newline, so they read as one stream: fetched together,
|
// Every segment ends with a newline, so they read as one stream: fetched together,
|
||||||
// several at a time where the location allows it.
|
// several at a time where the location allows it.
|
||||||
|
// Empty folders go in the meta record, not between the files: a version that does not
|
||||||
|
// know them reads the snapshot as before, and only restores no empty folders.
|
||||||
|
$emptyDirs = [];
|
||||||
|
foreach (isset($state['dirSegments']) ? $repo->readLines(array_values(array_merge(...$state['dirSegments']))) : [] as $line) {
|
||||||
|
if ($line !== '') {
|
||||||
|
$emptyDirs[] = json_decode($line, true, 8, JSON_THROW_ON_ERROR);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// The times of the other folders too, packed: every look at a snapshot's meta record reads
|
||||||
|
// this, and a string of a few hundred KiB costs that far less than tens of thousands of arrays.
|
||||||
|
$folders = isset($state['folderSegments']) ? implode("\n", iterator_to_array($repo->readLines(array_values(array_merge(...$state['folderSegments']))), false)) : '';
|
||||||
$repo->putSnapshot($state['snapshot'],
|
$repo->putSnapshot($state['snapshot'],
|
||||||
['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
|
['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
|
||||||
'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
|
'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
|
||||||
'warnings' => self::warnings($state)] + $state['meta'],
|
'warnings' => self::warnings($state)] + ($emptyDirs === [] ? [] : ['emptyDirs' => $emptyDirs])
|
||||||
|
+ ($folders === '' ? [] : ['folderTimes' => base64_encode((string)gzdeflate($folders))]) + $state['meta'],
|
||||||
$repo->readLines(array_values(array_merge(...$state['segments']))));
|
$repo->readLines(array_values(array_merge(...$state['segments']))));
|
||||||
$state['phase'] = 'commit';
|
$state['phase'] = 'commit';
|
||||||
}
|
}
|
||||||
@@ -214,10 +275,26 @@ final class BackupRun {
|
|||||||
$stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1;
|
$stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @return array{unreadable: list<string>, changing: list<string>} examples for the report (first 50 each) */
|
/**
|
||||||
|
* A file or directory left out because of its name. Remembered like an unreadable one, in a form
|
||||||
|
* that fits the JSON state and is safe to show: control characters and invalid bytes replaced.
|
||||||
|
*/
|
||||||
|
private static function recordBadName(array &$state, array &$stats, string $logical): void {
|
||||||
|
$shown = (string)preg_replace('/[\x00-\x1f\x7f]/', '?', mb_scrub($logical, 'UTF-8'));
|
||||||
|
if (isset($state['badNames'][$shown])) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (count($state['badNames'] ?? []) < 1000) {
|
||||||
|
$state['badNames'][$shown] = true;
|
||||||
|
}
|
||||||
|
$stats['badNames'] = ($stats['badNames'] ?? 0) + 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @return array{unreadable: list<string>, badNames: list<string>, changing: list<string>} examples for the report (first 50 each) */
|
||||||
public static function warnings(array $state): array {
|
public static function warnings(array $state): array {
|
||||||
return [
|
return [
|
||||||
'unreadable' => array_slice(array_map('strval', array_keys($state['unreadable'] ?? [])), 0, 50),
|
'unreadable' => array_slice(array_map('strval', array_keys($state['unreadable'] ?? [])), 0, 50),
|
||||||
|
'badNames' => array_slice(array_map('strval', array_keys($state['badNames'] ?? [])), 0, 50),
|
||||||
'changing' => array_slice(array_map('strval', array_values($state['changing'] ?? [])), 0, 50),
|
'changing' => array_slice(array_map('strval', array_values($state['changing'] ?? [])), 0, 50),
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|||||||
+28
-3
@@ -9,12 +9,16 @@ namespace OCA\NgBackup\Job;
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Walks one or more source roots in a stable depth-first order (entries of a directory sorted
|
* Walks one or more source roots in a stable depth-first order (entries of a directory sorted
|
||||||
* by name, children right after their directory), yielding "<root>/<relative path>" for files.
|
* by name, children right after their directory), yielding "<root>/<relative path>" for files
|
||||||
|
* and for directories with nothing in them to walk (the caller tells them apart with is_dir()).
|
||||||
|
* A root itself without a name is never yielded.
|
||||||
*
|
*
|
||||||
* The same order is used for comparing paths (compare()), so a walk can resume after a cursor
|
* The same order is used for comparing paths (compare()), so a walk can resume after a cursor
|
||||||
* and be merge-joined with the previous snapshot without holding any list in memory.
|
* and be merge-joined with the previous snapshot without holding any list in memory.
|
||||||
*/
|
*/
|
||||||
final class TreeWalker {
|
final class TreeWalker {
|
||||||
|
private bool $atFolder = false;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @param array<string, string> $roots name => absolute directory, walked in this order
|
* @param array<string, string> $roots name => absolute directory, walked in this order
|
||||||
* @param list<string> $exclude path prefixes ("<root>/<relative>") that are skipped entirely;
|
* @param list<string> $exclude path prefixes ("<root>/<relative>") that are skipped entirely;
|
||||||
@@ -25,9 +29,16 @@ final class TreeWalker {
|
|||||||
private array $exclude = [],
|
private array $exclude = [],
|
||||||
/** Called with the logical path of a directory that cannot be listed. */
|
/** Called with the logical path of a directory that cannot be listed. */
|
||||||
private ?\Closure $onUnreadable = null,
|
private ?\Closure $onUnreadable = null,
|
||||||
|
/** Also yield directories with something in them, right before the first thing in them. */
|
||||||
|
private bool $withFolders = false,
|
||||||
) {
|
) {
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Whether what was just yielded is a directory with something in it (see $withFolders). */
|
||||||
|
public function atFolderWithContent(): bool {
|
||||||
|
return $this->atFolder;
|
||||||
|
}
|
||||||
|
|
||||||
/** @return \Generator<string, string> logical path => absolute path, strictly after $after */
|
/** @return \Generator<string, string> logical path => absolute path, strictly after $after */
|
||||||
public function walk(?string $after = null): \Generator {
|
public function walk(?string $after = null): \Generator {
|
||||||
foreach ($this->roots as $name => $dir) {
|
foreach ($this->roots as $name => $dir) {
|
||||||
@@ -66,27 +77,41 @@ final class TreeWalker {
|
|||||||
}
|
}
|
||||||
$names = array_values(array_diff($names, ['.', '..']));
|
$names = array_values(array_diff($names, ['.', '..']));
|
||||||
sort($names, SORT_STRING);
|
sort($names, SORT_STRING);
|
||||||
|
$empty = true; // no file or directory below it that is walked, whatever the cursor skips
|
||||||
foreach ($names as $name) {
|
foreach ($names as $name) {
|
||||||
$childLogical = $logical === '' ? $name : $logical . '/' . $name;
|
$childLogical = $logical === '' ? $name : $logical . '/' . $name;
|
||||||
$childAbs = $abs . '/' . $name;
|
$childAbs = $abs . '/' . $name;
|
||||||
if (is_link($childAbs)) {
|
if (is_link($childAbs)) {
|
||||||
continue; // never follow symlinks out of the data directory
|
continue; // never follow symlinks out of the data directory
|
||||||
}
|
}
|
||||||
if (is_dir($childAbs)) {
|
$childIsDir = is_dir($childAbs);
|
||||||
|
if (($childIsDir || is_file($childAbs)) && $empty && !$this->isExcluded($childLogical, $childIsDir)
|
||||||
|
&& $this->withFolders && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) {
|
||||||
|
$this->atFolder = true;
|
||||||
|
yield $logical => $abs;
|
||||||
|
$this->atFolder = false;
|
||||||
|
}
|
||||||
|
if ($childIsDir) {
|
||||||
|
$empty = $empty && $this->isExcluded($childLogical, true);
|
||||||
// Skip a directory entirely if everything in it sorts before the cursor.
|
// Skip a directory entirely if everything in it sorts before the cursor.
|
||||||
if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) {
|
if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
yield from $this->walkDir($childAbs, $childLogical, $after);
|
yield from $this->walkDir($childAbs, $childLogical, $after);
|
||||||
} elseif (is_file($childAbs)) {
|
} elseif (is_file($childAbs)) {
|
||||||
|
$excluded = $this->isExcluded($childLogical, false);
|
||||||
|
$empty = $empty && $excluded;
|
||||||
if ($after !== null && self::compare($childLogical, $after) <= 0) {
|
if ($after !== null && self::compare($childLogical, $after) <= 0) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (!$this->isExcluded($childLogical, false)) {
|
if (!$excluded) {
|
||||||
yield $childLogical => $childAbs;
|
yield $childLogical => $childAbs;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if ($empty && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) {
|
||||||
|
yield $logical => $abs;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private function isExcluded(string $logical, bool $isDir): bool {
|
private function isExcluded(string $logical, bool $isDir): bool {
|
||||||
|
|||||||
@@ -0,0 +1,33 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Migration;
|
||||||
|
|
||||||
|
use Closure;
|
||||||
|
use OCP\DB\ISchemaWrapper;
|
||||||
|
use OCP\DB\Types;
|
||||||
|
use OCP\Migration\IOutput;
|
||||||
|
use OCP\Migration\SimpleMigrationStep;
|
||||||
|
|
||||||
|
/** Local copy of what the catalog on the location says about a snapshot: its note and pin. */
|
||||||
|
final class Version000007Date20261107000000 extends SimpleMigrationStep {
|
||||||
|
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
|
||||||
|
/** @var ISchemaWrapper $schema */
|
||||||
|
$schema = $schemaClosure();
|
||||||
|
$t = $schema->getTable('ngb_snapshots');
|
||||||
|
$changed = false;
|
||||||
|
if (!$t->hasColumn('note')) {
|
||||||
|
$t->addColumn('note', Types::STRING, ['notnull' => false, 'length' => 1000]);
|
||||||
|
$changed = true;
|
||||||
|
}
|
||||||
|
if (!$t->hasColumn('pinned')) {
|
||||||
|
$t->addColumn('pinned', Types::BOOLEAN, ['notnull' => false, 'default' => false]);
|
||||||
|
$changed = true;
|
||||||
|
}
|
||||||
|
return $changed ? $schema : null;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Migration;
|
||||||
|
|
||||||
|
use Closure;
|
||||||
|
use OCA\NgBackup\Service\LocationSchedule;
|
||||||
|
use OCA\NgBackup\Service\Schedule;
|
||||||
|
use OCP\DB\ISchemaWrapper;
|
||||||
|
use OCP\DB\Types;
|
||||||
|
use OCP\IDBConnection;
|
||||||
|
use OCP\Migration\IOutput;
|
||||||
|
use OCP\Migration\SimpleMigrationStep;
|
||||||
|
|
||||||
|
/** A schedule per location. Each existing location gets the daily time that held for all of them. */
|
||||||
|
final class Version000008Date20261107010000 extends SimpleMigrationStep {
|
||||||
|
public function __construct(
|
||||||
|
private IDBConnection $db,
|
||||||
|
private Schedule $schedule,
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
||||||
|
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
|
||||||
|
/** @var ISchemaWrapper $schema */
|
||||||
|
$schema = $schemaClosure();
|
||||||
|
$t = $schema->getTable('ngb_targets');
|
||||||
|
if ($t->hasColumn('schedule')) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
$t->addColumn('schedule', Types::STRING, ['notnull' => false, 'length' => 16]);
|
||||||
|
$t->addColumn('schedule_zone', Types::STRING, ['notnull' => false, 'length' => 64]);
|
||||||
|
$t->addColumn('schedule_since', Types::BIGINT, ['notnull' => false, 'unsigned' => true]);
|
||||||
|
return $schema;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function postSchemaChange(IOutput $output, Closure $schemaClosure, array $options): void {
|
||||||
|
['time' => $time, 'timezone' => $zone] = $this->schedule->get();
|
||||||
|
$qb = $this->db->getQueryBuilder();
|
||||||
|
$qb->update('ngb_targets')
|
||||||
|
->set('schedule', $qb->createNamedParameter(Schedule::validTime($time) ? LocationSchedule::normalize($time) : ''))
|
||||||
|
->set('schedule_zone', $qb->createNamedParameter($zone))
|
||||||
|
->where($qb->expr()->isNull('schedule'));
|
||||||
|
$qb->executeStatement();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -46,6 +46,10 @@ final class Notifier implements INotifier {
|
|||||||
$notification->setParsedSubject($l->t('Backup to %1$s finished with warnings: %2$s unreadable, %3$s changed while being read', [$p['target'], $p['unreadable'], $p['changing']]));
|
$notification->setParsedSubject($l->t('Backup to %1$s finished with warnings: %2$s unreadable, %3$s changed while being read', [$p['target'], $p['unreadable'], $p['changing']]));
|
||||||
$notification->setParsedMessage($l->t('Unreadable files and folders are missing from this backup (check their permissions). Files that kept changing were saved as they were while being read. Examples: %s', [$p['examples'] ?? '']));
|
$notification->setParsedMessage($l->t('Unreadable files and folders are missing from this backup (check their permissions). Files that kept changing were saved as they were while being read. Examples: %s', [$p['examples'] ?? '']));
|
||||||
break;
|
break;
|
||||||
|
case 'run_bad_names':
|
||||||
|
$notification->setParsedSubject($l->n('Backup to %s finished, but %n item is missing because of its name', 'Backup to %s finished, but %n items are missing because of their names', (int)$p['count'], [$p['target']]));
|
||||||
|
$notification->setParsedMessage($l->t('Names with a backslash or a control character, or that are not valid UTF-8, are not supported, so these files and folders are not in this backup. Rename them; the next backup includes them. Examples: %s', [$p['examples'] ?? '']));
|
||||||
|
break;
|
||||||
case 'quota_warning':
|
case 'quota_warning':
|
||||||
$notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']]));
|
$notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']]));
|
||||||
$notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.'));
|
$notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.'));
|
||||||
@@ -57,12 +61,19 @@ final class Notifier implements INotifier {
|
|||||||
: $l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']]));
|
: $l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']]));
|
||||||
$notification->setParsedMessage($l->t('Check the backup settings and the server log.'));
|
$notification->setParsedMessage($l->t('Check the backup settings and the server log.'));
|
||||||
break;
|
break;
|
||||||
|
case 'schedule_off':
|
||||||
|
$notification->setParsedSubject($l->t('Backup location %s has no schedule', [$p['target']]));
|
||||||
|
$notification->setParsedMessage(($p['others'] ?? '') === '1'
|
||||||
|
? $l->t('No backups run to this location until it gets a schedule or someone starts one. If that is what you want, you can ignore this reminder.')
|
||||||
|
: $l->t('No location has a schedule, so no backups run at all unless someone starts one. Set a schedule under Backups.'));
|
||||||
|
break;
|
||||||
case 'security_event':
|
case 'security_event':
|
||||||
$who = (string)($p['actor'] ?? '');
|
$who = (string)($p['actor'] ?? '');
|
||||||
$notification->setParsedSubject(match ($p['event'] ?? '') {
|
$notification->setParsedSubject(match ($p['event'] ?? '') {
|
||||||
'kit_downloaded' => $l->t('%s downloaded the backup recovery kit', [$who]),
|
'kit_downloaded' => $l->t('%s downloaded the backup recovery kit', [$who]),
|
||||||
'target_removed' => $l->t('%1$s removed backup location %2$s', [$who, $p['target'] ?? '']),
|
'target_removed' => $l->t('%1$s removed backup location %2$s', [$who, $p['target'] ?? '']),
|
||||||
'target_changed' => $l->t('%1$s changed the connection settings of backup location %2$s', [$who, $p['target'] ?? '']),
|
'target_changed' => $l->t('%1$s changed the connection settings of backup location %2$s', [$who, $p['target'] ?? '']),
|
||||||
|
'schedule_off' => $l->t('%1$s turned off the schedule of backup location %2$s', [$who, $p['target'] ?? '']),
|
||||||
'append_only_off' =>$l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']),
|
'append_only_off' =>$l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']),
|
||||||
'retention_changed' => $l->t('%1$s changed the backup retention to %2$s', [$who, $p['policy'] ?? '']),
|
'retention_changed' => $l->t('%1$s changed the backup retention to %2$s', [$who, $p['policy'] ?? '']),
|
||||||
'snapshots_forgotten' => $l->t('%1$s moved %2$s snapshots of %3$s to the trash; they are deleted permanently after %4$s unless restored', [$who, $p['count'] ?? '', $p['target'] ?? '', $p['until'] ?? '']),
|
'snapshots_forgotten' => $l->t('%1$s moved %2$s snapshots of %3$s to the trash; they are deleted permanently after %4$s unless restored', [$who, $p['count'] ?? '', $p['target'] ?? '', $p['until'] ?? '']),
|
||||||
|
|||||||
@@ -30,6 +30,10 @@ final class Catalog {
|
|||||||
public array $trash = [];
|
public array $trash = [];
|
||||||
/** @var array<string, true> manifest path => true, for a per-user export (users/<uid>/<exportId>) */
|
/** @var array<string, true> manifest path => true, for a per-user export (users/<uid>/<exportId>) */
|
||||||
public array $userExports = [];
|
public array $userExports = [];
|
||||||
|
/** @var array<string, true> snapshots the retention policy leaves alone */
|
||||||
|
public array $pinned = [];
|
||||||
|
/** @var array<string, string> snapshot id => note, also while the snapshot is in the trash */
|
||||||
|
public array $notes = [];
|
||||||
public string $hash = '';
|
public string $hash = '';
|
||||||
|
|
||||||
public function __construct(
|
public function __construct(
|
||||||
@@ -82,6 +86,8 @@ final class Catalog {
|
|||||||
// Catalogs written before user exports existed have no such key: default to empty rather
|
// Catalogs written before user exports existed have no such key: default to empty rather
|
||||||
// than adopting the location's current users/ listing (that would skip verification).
|
// than adopting the location's current users/ listing (that would skip verification).
|
||||||
$this->userExports = array_fill_keys($data['userExports'] ?? [], true);
|
$this->userExports = array_fill_keys($data['userExports'] ?? [], true);
|
||||||
|
$this->pinned = array_fill_keys($data['pinned'] ?? [], true);
|
||||||
|
$this->notes = $data['notes'] ?? [];
|
||||||
$this->anchor?->set($this->repositoryId, $this->gen, $this->hash);
|
$this->anchor?->set($this->repositoryId, $this->gen, $this->hash);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -93,9 +99,12 @@ final class Catalog {
|
|||||||
if ($this->backend->exists($this->path($next))) {
|
if ($this->backend->exists($this->path($next))) {
|
||||||
throw new RepositoryException("Catalog generation $next was written by another process at the same time; try again");
|
throw new RepositoryException("Catalog generation $next was written by another process at the same time; try again");
|
||||||
}
|
}
|
||||||
|
// A pin ends when its snapshot leaves the list, a note when the snapshot is gone for good.
|
||||||
|
$this->pinned = array_intersect_key($this->pinned, $this->snapshots);
|
||||||
|
$this->notes = array_intersect_key($this->notes, $this->snapshots + $this->trash);
|
||||||
$plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash,
|
$plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash,
|
||||||
'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash),
|
'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash),
|
||||||
'userExports' => array_keys($this->userExports)], JSON_THROW_ON_ERROR);
|
'userExports' => array_keys($this->userExports), 'pinned' => array_keys($this->pinned), 'notes' => $this->notes], JSON_THROW_ON_ERROR);
|
||||||
$encrypted = $this->cipher->encryptString($plain, $this->ad($next));
|
$encrypted = $this->cipher->encryptString($plain, $this->ad($next));
|
||||||
$stream = fopen('php://memory', 'w+b');
|
$stream = fopen('php://memory', 'w+b');
|
||||||
fwrite($stream, $encrypted);
|
fwrite($stream, $encrypted);
|
||||||
|
|||||||
+221
-21
@@ -44,6 +44,8 @@ final class Repository {
|
|||||||
private ?int $maxBytes = null;
|
private ?int $maxBytes = null;
|
||||||
private Catalog $catalog;
|
private Catalog $catalog;
|
||||||
private PackReader $packReader;
|
private PackReader $packReader;
|
||||||
|
/** @var array<string, array<string, mixed>> meta record of the snapshot read last */
|
||||||
|
private array $lastMeta = [];
|
||||||
/** verify() lists the pack folders from this many packs on (see presentPacks()); tests lower it. */
|
/** verify() lists the pack folders from this many packs on (see presentPacks()); tests lower it. */
|
||||||
public int $listPacksAbove = 256;
|
public int $listPacksAbove = 256;
|
||||||
|
|
||||||
@@ -132,6 +134,49 @@ final class Repository {
|
|||||||
return $this->catalog->gen;
|
return $this->catalog->gen;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @return list<string> snapshots the retention policy leaves alone */
|
||||||
|
public function pinnedSnapshots(): array {
|
||||||
|
return array_map('strval', array_keys($this->catalog->pinned));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @return array<string, string> snapshot id => note */
|
||||||
|
public function snapshotNotes(): array {
|
||||||
|
return $this->catalog->notes;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function setPinned(string $snapshotId, bool $pinned): void {
|
||||||
|
$this->requireSnapshot($snapshotId);
|
||||||
|
if (isset($this->catalog->pinned[$snapshotId]) === $pinned) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if ($pinned) {
|
||||||
|
$this->catalog->pinned[$snapshotId] = true;
|
||||||
|
} else {
|
||||||
|
unset($this->catalog->pinned[$snapshotId]);
|
||||||
|
}
|
||||||
|
$this->catalog->write();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A note kept in the catalog: unlike the label inside the snapshot, it can change afterwards. */
|
||||||
|
public function setSnapshotNote(string $snapshotId, string $note): void {
|
||||||
|
$this->requireSnapshot($snapshotId);
|
||||||
|
if (($this->catalog->notes[$snapshotId] ?? '') === $note) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if ($note === '') {
|
||||||
|
unset($this->catalog->notes[$snapshotId]);
|
||||||
|
} else {
|
||||||
|
$this->catalog->notes[$snapshotId] = $note;
|
||||||
|
}
|
||||||
|
$this->catalog->write();
|
||||||
|
}
|
||||||
|
|
||||||
|
private function requireSnapshot(string $snapshotId): void {
|
||||||
|
if (!isset($this->catalog->snapshots[$snapshotId])) {
|
||||||
|
throw new \InvalidArgumentException('There is no snapshot ' . $snapshotId . ' on this location');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/** Index files downloaded when this repository was opened (0 when the local cache was complete). */
|
/** Index files downloaded when this repository was opened (0 when the local cache was complete). */
|
||||||
public function indexDownloads(): int {
|
public function indexDownloads(): int {
|
||||||
return $this->index->downloaded;
|
return $this->index->downloaded;
|
||||||
@@ -257,8 +302,8 @@ final class Repository {
|
|||||||
// that was being written when the process was killed stays (8 MiB, DR test 2026-11-05)
|
// that was being written when the process was killed stays (8 MiB, DR test 2026-11-05)
|
||||||
$stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR);
|
$stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR);
|
||||||
foreach ($stale === false ? [] : $stale as $dir) {
|
foreach ($stale === false ? [] : $stale as $dir) {
|
||||||
if (is_link($dir)) {
|
if (is_link($dir) || preg_match('/^\.ngb-restore-spool-[0-9a-f]{8}$/', basename($dir)) !== 1) {
|
||||||
continue;
|
continue; // not a spool of ours, whatever it is called
|
||||||
}
|
}
|
||||||
$files = glob($dir . '/*');
|
$files = glob($dir . '/*');
|
||||||
foreach ($files === false ? [] : $files as $file) {
|
foreach ($files === false ? [] : $files as $file) {
|
||||||
@@ -313,40 +358,39 @@ final class Repository {
|
|||||||
if (!self::underPrefix($entry['p'], $prefix)) {
|
if (!self::underPrefix($entry['p'], $prefix)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
if ($heartbeat !== null) {
|
||||||
|
$heartbeat(); // also for a long row of files that are skipped or have no data
|
||||||
|
}
|
||||||
$dest = self::destination($target, $entry['p'], $prefix, $stripPrefix);
|
$dest = self::destination($target, $entry['p'], $prefix, $stripPrefix);
|
||||||
if ($keepUnchanged && self::inPlace($dest, $entry)) {
|
$dir = self::directoryInTarget($target, dirname($dest), $entry['p']);
|
||||||
// below the target as written by an earlier run of this restore (checked then)
|
if (is_link($dest)) {
|
||||||
$count++;
|
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $entry['p']);
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (!is_dir(dirname($dest))) {
|
|
||||||
mkdir(dirname($dest), 0700, true);
|
|
||||||
}
|
|
||||||
// Defence in depth: the resolved directory must stay below the target (no symlink escapes).
|
|
||||||
$base = realpath($target);
|
|
||||||
$dir = realpath(dirname($dest));
|
|
||||||
if ($base === false || $dir === false || ($dir !== $base && !str_starts_with($dir, $base . '/')) || is_link($dest)) {
|
|
||||||
throw new RepositoryException('Refusing to write outside the restore directory: ' . $entry['p']);
|
|
||||||
}
|
}
|
||||||
if (!isset($cleaned[$dir])) {
|
if (!isset($cleaned[$dir])) {
|
||||||
self::removeStaleTemporaries($dir);
|
self::removeStaleTemporaries($dir);
|
||||||
$cleaned[$dir] = true;
|
$cleaned[$dir] = true;
|
||||||
}
|
}
|
||||||
|
if ($keepUnchanged && self::inPlace($dest, $entry)) {
|
||||||
|
$count++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
// Write next to the destination and rename after all blobs verified: at most one file
|
// Write next to the destination and rename after all blobs verified: at most one file
|
||||||
// is ever present twice.
|
// is ever present twice.
|
||||||
$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
|
$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
|
||||||
$out = @fopen($tmp, 'xb'); // fails if anything (including a symlink) already exists there
|
$out = @fopen($tmp, 'xb'); // fails if anything (including a symlink) already exists there
|
||||||
if ($out === false) {
|
if ($out === false) {
|
||||||
throw new RepositoryException('Cannot create a temporary file for ' . $entry['p']);
|
throw new RestoreTargetException('Cannot create a temporary file for ' . $entry['p'] . ' (disk full, or no permission?)');
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
foreach ($entry['b'] as $id) {
|
foreach ($entry['b'] as $id) {
|
||||||
fwrite($out, $this->restoreBlob($id, $plan));
|
self::writeAll($out, $this->restoreBlob($id, $plan), $entry['p']);
|
||||||
if ($heartbeat !== null) {
|
if ($heartbeat !== null) {
|
||||||
$heartbeat();
|
$heartbeat();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
fclose($out);
|
if (!@fclose($out)) {
|
||||||
|
throw new RestoreTargetException('Cannot write ' . $entry['p'] . ' to the restore directory (disk full?)');
|
||||||
|
}
|
||||||
clearstatcache(true, $tmp);
|
clearstatcache(true, $tmp);
|
||||||
if (filesize($tmp) !== $entry['s']) {
|
if (filesize($tmp) !== $entry['s']) {
|
||||||
throw new RepositoryException('Size mismatch restoring ' . $entry['p']);
|
throw new RepositoryException('Size mismatch restoring ' . $entry['p']);
|
||||||
@@ -363,19 +407,144 @@ final class Repository {
|
|||||||
}
|
}
|
||||||
$skipped[$entry['p']] = $e->getMessage();
|
$skipped[$entry['p']] = $e->getMessage();
|
||||||
continue;
|
continue;
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
if (is_resource($out)) {
|
||||||
|
@fclose($out);
|
||||||
|
}
|
||||||
|
@unlink($tmp);
|
||||||
|
throw $e;
|
||||||
}
|
}
|
||||||
if (!rename($tmp, $dest)) {
|
if (!rename($tmp, $dest)) {
|
||||||
@unlink($tmp);
|
@unlink($tmp);
|
||||||
throw new RepositoryException('Cannot put the restored file in place: ' . $entry['p']);
|
throw new RestoreTargetException('Cannot put the restored file in place: ' . $entry['p']);
|
||||||
}
|
}
|
||||||
if (!touch($dest, $entry['m'])) {
|
if (!touch($dest, $entry['m'])) {
|
||||||
throw new RepositoryException('Cannot set the modification time of ' . $entry['p']);
|
throw new RestoreTargetException('Cannot set the modification time of ' . $entry['p']);
|
||||||
}
|
}
|
||||||
$count++;
|
$count++;
|
||||||
}
|
}
|
||||||
|
// Last: a file can only be restored into a folder that exists, and the meta record was
|
||||||
|
// read along with the files, so this asks nothing more of the location.
|
||||||
|
foreach ($this->emptyDirs($snapshotId) as [$path, $mtime]) {
|
||||||
|
if (self::underPrefix($path, $prefix)) {
|
||||||
|
if ($heartbeat !== null) {
|
||||||
|
$heartbeat();
|
||||||
|
}
|
||||||
|
// The prefix itself, stripped, is the target (for a file it is the file's name there).
|
||||||
|
$dest = $stripPrefix && $path === rtrim($prefix, '/') ? rtrim($target, '/') : self::destination($target, $path, $prefix, $stripPrefix);
|
||||||
|
self::restoreEmptyDir($target, $dest, $path, $mtime);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Very last, nothing is written in them any more. Also on a run that goes on after a stop, and
|
||||||
|
// the deepest first. The time is not worth stopping for: all the files are there by now.
|
||||||
|
foreach (array_reverse($this->folderTimes($snapshotId)) as [$path, $mtime]) {
|
||||||
|
if (!self::underPrefix($path, $prefix) || ($stripPrefix && $path === rtrim($prefix, '/'))) {
|
||||||
|
continue; // the target itself is not ours to change
|
||||||
|
}
|
||||||
|
if ($heartbeat !== null) {
|
||||||
|
$heartbeat();
|
||||||
|
}
|
||||||
|
$dest = self::destination($target, $path, $prefix, $stripPrefix);
|
||||||
|
@touch(self::directoryInTarget($target, $dest, $path), $mtime);
|
||||||
|
}
|
||||||
return $count;
|
return $count;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The folders with something in them that a snapshot recorded, parents before children (none
|
||||||
|
* before 1.1.0). Damage here is no reason to give up on the files of a restore.
|
||||||
|
*
|
||||||
|
* @return list<array{0: string, 1: int}> path, modification time
|
||||||
|
*/
|
||||||
|
public function folderTimes(string $snapshotId): array {
|
||||||
|
$meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId);
|
||||||
|
$packed = is_string($meta['folderTimes'] ?? null) ? base64_decode($meta['folderTimes'], true) : false;
|
||||||
|
$plain = $packed === false ? false : @gzinflate($packed);
|
||||||
|
$folders = [];
|
||||||
|
foreach ($plain === false ? [] : explode("\n", $plain) as $line) {
|
||||||
|
$folder = json_decode($line, true);
|
||||||
|
if (is_array($folder) && is_string($folder[0] ?? null) && self::isSafePath($folder[0]) && is_int($folder[1] ?? null)) {
|
||||||
|
$folders[] = [$folder[0], $folder[1]];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $folders;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The folders without anything in them that a snapshot recorded (none before 1.0.3).
|
||||||
|
*
|
||||||
|
* @return list<array{0: string, 1: int}> path, modification time
|
||||||
|
*/
|
||||||
|
public function emptyDirs(string $snapshotId): array {
|
||||||
|
$meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId);
|
||||||
|
$dirs = [];
|
||||||
|
foreach (is_array($meta['emptyDirs'] ?? null) ? $meta['emptyDirs'] : [] as $dir) {
|
||||||
|
// One folder that makes no sense is no reason to give up on the files of a restore.
|
||||||
|
if (is_array($dir) && is_string($dir[0] ?? null) && self::isSafePath($dir[0]) && is_int($dir[1] ?? null)) {
|
||||||
|
$dirs[] = [$dir[0], $dir[1]];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $dirs;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static function restoreEmptyDir(string $target, string $dest, string $path, int $mtime): void {
|
||||||
|
if (is_link($dest)) {
|
||||||
|
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
|
||||||
|
}
|
||||||
|
$existed = is_dir($dest);
|
||||||
|
self::directoryInTarget($target, $dest, $path);
|
||||||
|
if (!$existed && !@touch($dest, $mtime)) {
|
||||||
|
throw new RestoreTargetException('Cannot set the modification time of ' . $path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Make sure a folder exists below the target, without following a link out of it: what exists
|
||||||
|
* of the path is checked before anything is created.
|
||||||
|
*
|
||||||
|
* @return string the folder's real path
|
||||||
|
*/
|
||||||
|
private static function directoryInTarget(string $target, string $dir, string $path): string {
|
||||||
|
if (!is_dir($target) && !@mkdir($target, 0700, true) && !is_dir($target)) {
|
||||||
|
throw new RestoreTargetException('Cannot create the restore directory ' . $target);
|
||||||
|
}
|
||||||
|
$base = realpath($target);
|
||||||
|
$existing = $dir;
|
||||||
|
while (!file_exists($existing) && !is_link($existing) && dirname($existing) !== $existing) {
|
||||||
|
$existing = dirname($existing);
|
||||||
|
}
|
||||||
|
$real = realpath($existing);
|
||||||
|
if ($base === false || $real === false || ($real !== $base && !str_starts_with($real, $base . '/'))) {
|
||||||
|
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
|
||||||
|
}
|
||||||
|
if ($existing === $dir) {
|
||||||
|
if (!is_dir($real)) {
|
||||||
|
throw new RestoreTargetException('Cannot create the folder for ' . $path . ': a file is in the way');
|
||||||
|
}
|
||||||
|
return $real;
|
||||||
|
}
|
||||||
|
if (!@mkdir($dir, 0700, true) && !is_dir($dir)) {
|
||||||
|
throw new RestoreTargetException('Cannot create the folder for ' . $path . ' (disk full, no permission, or a file in the way?)');
|
||||||
|
}
|
||||||
|
return (string)realpath($dir);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Write all of $data, or stop: a short write leaves a file that looks damaged only by its size.
|
||||||
|
* A write may take only part of the data and still be fine; one that takes nothing is not.
|
||||||
|
*
|
||||||
|
* @param resource $out
|
||||||
|
*/
|
||||||
|
public static function writeAll($out, string $data, string $path): void {
|
||||||
|
while ($data !== '') {
|
||||||
|
$written = @fwrite($out, $data);
|
||||||
|
if ($written === false || $written === 0) {
|
||||||
|
throw new RestoreTargetException('Cannot write ' . $path . ' to the restore directory (disk full?)');
|
||||||
|
}
|
||||||
|
$data = substr($data, $written);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/** Where restore() writes the snapshot path $path. */
|
/** Where restore() writes the snapshot path $path. */
|
||||||
private static function destination(string $target, string $path, string $prefix, bool $stripPrefix): string {
|
private static function destination(string $target, string $path, string $prefix, bool $stripPrefix): string {
|
||||||
$relPath = $path;
|
$relPath = $path;
|
||||||
@@ -722,6 +891,34 @@ final class Repository {
|
|||||||
return array_keys($doomed);
|
return array_keys($doomed);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Part of trash(), for a page that shows how far it is: each entry is a read of its own on the
|
||||||
|
* location. Reads the entries with an id after $after, in order, until $deadline
|
||||||
|
* (microtime(true)), at least one. The id, not a position: a cleanup that changes the trash
|
||||||
|
* between two parts then makes no entry come twice or be missed.
|
||||||
|
*
|
||||||
|
* @return array{total: int, items: array<string, array<string, mixed>>}
|
||||||
|
*/
|
||||||
|
public function trashPage(string $after, float $deadline): array {
|
||||||
|
$ids = array_map('strval', array_keys($this->catalog->trash));
|
||||||
|
sort($ids, SORT_STRING);
|
||||||
|
$items = [];
|
||||||
|
foreach ($ids as $id) {
|
||||||
|
if (strcmp($id, $after) <= 0) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$items[$id] = $this->trashInfo($id);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
throw new RepositoryException("Trash entry $id listed in the catalog is missing on the location (possible tampering)");
|
||||||
|
}
|
||||||
|
if (microtime(true) >= $deadline) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ['total' => count($ids), 'items' => $items];
|
||||||
|
}
|
||||||
|
|
||||||
private function trashInfo(string $snapshotId): array {
|
private function trashInfo(string $snapshotId): array {
|
||||||
return json_decode($this->getObject('trash/info/' . $snapshotId), true, 512, JSON_THROW_ON_ERROR);
|
return json_decode($this->getObject('trash/info/' . $snapshotId), true, 512, JSON_THROW_ON_ERROR);
|
||||||
}
|
}
|
||||||
@@ -1281,7 +1478,10 @@ final class Repository {
|
|||||||
$this->cipher->decrypt($fh, $plain, 'snapshot:' . $snapshotId);
|
$this->cipher->decrypt($fh, $plain, 'snapshot:' . $snapshotId);
|
||||||
fclose($fh);
|
fclose($fh);
|
||||||
rewind($plain);
|
rewind($plain);
|
||||||
fgets($plain); // meta line
|
$meta = json_decode((string)fgets($plain), true);
|
||||||
|
if (is_array($meta)) {
|
||||||
|
$this->lastMeta = [$snapshotId => $meta]; // a snapshot never changes
|
||||||
|
}
|
||||||
while (($line = fgets($plain)) !== false) {
|
while (($line = fgets($plain)) !== false) {
|
||||||
$entry = json_decode($line, true, 512, JSON_THROW_ON_ERROR);
|
$entry = json_decode($line, true, 512, JSON_THROW_ON_ERROR);
|
||||||
if (!is_array($entry) || !is_string($entry['p'] ?? null) || !self::isSafePath($entry['p'])
|
if (!is_array($entry) || !is_string($entry['p'] ?? null) || !self::isSafePath($entry['p'])
|
||||||
|
|||||||
@@ -0,0 +1,15 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Repository;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A restore cannot write to where it restores to (a full disk, permissions, something in the way).
|
||||||
|
* The backup is fine, so this never counts as damaged data: the restore stops, also with --salvage.
|
||||||
|
*/
|
||||||
|
final class RestoreTargetException extends \RuntimeException {
|
||||||
|
}
|
||||||
@@ -61,6 +61,16 @@ final class AlertService {
|
|||||||
'examples' => mb_strimwidth(implode(', ', array_merge($examples['unreadable'], $examples['changing'])), 0, 300, '…')]);
|
'examples' => mb_strimwidth(implode(', ', array_merge($examples['unreadable'], $examples['changing'])), 0, 300, '…')]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A backup that finished but left out files or folders because of their name.
|
||||||
|
*
|
||||||
|
* @param list<string> $examples
|
||||||
|
*/
|
||||||
|
public function runBadNames(Run $run, string $targetName, int $count, array $examples): void {
|
||||||
|
$this->notifyAdmins('run_bad_names', 'run-' . $run->getId(), ['target' => $targetName, 'count' => (string)$count,
|
||||||
|
'examples' => mb_strimwidth(implode(', ', $examples), 0, 300, '…')]);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Remember how much is stored on a location (for the settings page) and warn administrators
|
* Remember how much is stored on a location (for the settings page) and warn administrators
|
||||||
* once a day when it is at 80% of its limit or more.
|
* once a day when it is at 80% of its limit or more.
|
||||||
@@ -91,8 +101,35 @@ final class AlertService {
|
|||||||
return $v === '' ? null : (int)$v;
|
return $v === '' ? null : (int)$v;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether it is time to remind administrators of a location without a schedule: once a week
|
||||||
|
* while another location is still backed up, every day when none is. An hour of margin, so
|
||||||
|
* that the moment of the cron run does not push a reminder to the next day.
|
||||||
|
*/
|
||||||
|
public static function offReminderDue(int $last, int $now, bool $othersActive): bool {
|
||||||
|
return $now - $last >= ($othersActive ? 7 * 86400 : 86400) - 3600;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function checkOff(Target $target, bool $othersActive): void {
|
||||||
|
if ((string)$target->getSchedule() !== '') {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
$key = 'off_notified_' . $target->getId();
|
||||||
|
$now = time();
|
||||||
|
$last = max((int)$target->getScheduleSince(), $this->appConfig->getValueInt(Application::APP_ID, $key, 0));
|
||||||
|
if (!self::offReminderDue($last, $now, $othersActive)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
$this->appConfig->setValueInt(Application::APP_ID, $key, $now);
|
||||||
|
$this->notifyAdmins('schedule_off', 'off-' . $target->getId() . '-' . date('Y-m-d', $now),
|
||||||
|
['target' => $target->getName(), 'others' => $othersActive ? '1' : '0']);
|
||||||
|
}
|
||||||
|
|
||||||
/** At most one notification per location per day. */
|
/** At most one notification per location per day. */
|
||||||
public function checkStale(Target $target): void {
|
public function checkStale(Target $target): void {
|
||||||
|
if ((string)$target->getSchedule() === '') {
|
||||||
|
return; // no schedule is a choice, as for a location that is only connected now and then
|
||||||
|
}
|
||||||
$last =$this->runs->lastFinished($target->getId(), BackupService::KIND_FULL);
|
$last =$this->runs->lastFinished($target->getId(), BackupService::KIND_FULL);
|
||||||
$lastTime = $last?->getFinishedAt() ?? $target->getCreatedAt();
|
$lastTime = $last?->getFinishedAt() ?? $target->getCreatedAt();
|
||||||
if (time() - $lastTime < self::STALE_AFTER || $this->runs->findRunning($target->getId(), BackupService::KIND_FULL) !== []) {
|
if (time() - $lastTime < self::STALE_AFTER || $this->runs->findRunning($target->getId(), BackupService::KIND_FULL) !== []) {
|
||||||
|
|||||||
@@ -58,6 +58,18 @@ final class BackupService {
|
|||||||
$this->openedWith = $opened;
|
$this->openedWith = $opened;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The running backups in the order they get their turn: first come, first served. The first one
|
||||||
|
* runs, the others are queued; only one at a time reads the disks.
|
||||||
|
*
|
||||||
|
* @return list<Run>
|
||||||
|
*/
|
||||||
|
public function runsInTurn(): array {
|
||||||
|
$running = $this->runs->findRunning(null, self::KIND_FULL);
|
||||||
|
usort($running, static fn (Run $a, Run $b): int => [$a->getStartedAt(), $a->getId()] <=> [$b->getStartedAt(), $b->getId()]);
|
||||||
|
return $running;
|
||||||
|
}
|
||||||
|
|
||||||
public function start(Target $target, string $label = ''): Run {
|
public function start(Target $target, string $label = ''): Run {
|
||||||
$label = trim($label);
|
$label = trim($label);
|
||||||
if (mb_strlen($label) > self::LABEL_MAX) {
|
if (mb_strlen($label) > self::LABEL_MAX) {
|
||||||
@@ -79,7 +91,7 @@ final class BackupService {
|
|||||||
try {
|
try {
|
||||||
return $this->leases->with(PruneService::lockKey($target), LeaseService::EXCLUSIVE, 60, fn () => $this->createRun($target, $label));
|
return $this->leases->with(PruneService::lockKey($target), LeaseService::EXCLUSIVE, 60, fn () => $this->createRun($target, $label));
|
||||||
} catch (LockedException) {
|
} catch (LockedException) {
|
||||||
throw new \RuntimeException('A cleanup or another start for ' . $target->getName() . ' is running; try again later');
|
throw new \RuntimeException('A cleanup, a restore or another start for ' . $target->getName() . ' is running, or one was stopped by force (that clears by itself within two hours); try again later');
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -210,6 +222,10 @@ final class BackupService {
|
|||||||
$state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR);
|
$state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR);
|
||||||
|
|
||||||
if ($run->getPhase() === 'db') {
|
if ($run->getPhase() === 'db') {
|
||||||
|
if (!isset($state['dump'])) {
|
||||||
|
// It may have been queued for hours: the restore point has the time of its data.
|
||||||
|
$state['files']['started'] = gmdate('c');
|
||||||
|
}
|
||||||
$dumper = new DbDumper($this->db->getInner(), $this->config->getSystemValueString('dbtableprefix', 'oc_'));
|
$dumper = new DbDumper($this->db->getInner(), $this->config->getSystemValueString('dbtableprefix', 'oc_'));
|
||||||
$dbStats = $state['dbStats'] ?? [];
|
$dbStats = $state['dbStats'] ?? [];
|
||||||
// With system cron the whole dump usually fits in one step (one transaction, fully
|
// With system cron the whole dump usually fits in one step (one transaction, fully
|
||||||
@@ -253,8 +269,13 @@ final class BackupService {
|
|||||||
try {
|
try {
|
||||||
$this->alerts->storedBytes($target, $repo->storedBytes());
|
$this->alerts->storedBytes($target, $repo->storedBytes());
|
||||||
$stats = $state['files']['stats'];
|
$stats = $state['files']['stats'];
|
||||||
if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) {
|
|
||||||
$warnings = BackupRun::warnings($state['files']);
|
$warnings = BackupRun::warnings($state['files']);
|
||||||
|
if (($stats['badNames'] ?? 0) > 0) {
|
||||||
|
$this->logger->warning(sprintf('NG Backup run %d to %s: %d left out because their name is not supported (rename them): %s',
|
||||||
|
$run->getId(), $target->getName(), $stats['badNames'], implode(', ', $warnings['badNames'])));
|
||||||
|
$this->alerts->runBadNames($run, $target->getName(), (int)$stats['badNames'], $warnings['badNames']);
|
||||||
|
}
|
||||||
|
if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) {
|
||||||
$this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s',
|
$this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s',
|
||||||
$run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']),
|
$run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']),
|
||||||
$stats['changing'] ?? 0, implode(', ', $warnings['changing'])));
|
$stats['changing'] ?? 0, implode(', ', $warnings['changing'])));
|
||||||
|
|||||||
@@ -0,0 +1,12 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Service;
|
||||||
|
|
||||||
|
/** A step after the database was committed failed: the database is no longer the fresh one. */
|
||||||
|
final class DatabaseAlreadyRestoredException extends \RuntimeException {
|
||||||
|
}
|
||||||
@@ -76,6 +76,15 @@ final class LeaseService {
|
|||||||
$qb->executeStatement();
|
$qb->executeStatement();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Remove the shared leases on $name, for a caller that knows their holders were stopped. */
|
||||||
|
public function dropShared(string $name): void {
|
||||||
|
$qb = $this->db->getQueryBuilder();
|
||||||
|
$qb->delete('ngb_leases')
|
||||||
|
->where($qb->expr()->eq('name', $qb->createNamedParameter($name)))
|
||||||
|
->andWhere($qb->expr()->eq('mode', $qb->createNamedParameter(self::SHARED)));
|
||||||
|
$qb->executeStatement();
|
||||||
|
}
|
||||||
|
|
||||||
/** Wraps $refresh so calling it often (per file, per row batch) only touches the database every $seconds. */
|
/** Wraps $refresh so calling it often (per file, per row batch) only touches the database every $seconds. */
|
||||||
public static function throttled(callable $refresh, int $seconds = 60): \Closure {
|
public static function throttled(callable $refresh, int $seconds = 60): \Closure {
|
||||||
$last = time();
|
$last = time();
|
||||||
|
|||||||
@@ -0,0 +1,78 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Service;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* When a location is backed up: "" (off), "HH:MM" (daily) or "HH:MM/N" (from HH:MM every N hours,
|
||||||
|
* starting over at HH:MM the next day), in the location's time zone.
|
||||||
|
*/
|
||||||
|
final class LocationSchedule {
|
||||||
|
/** Attempts per moment after a failure, an hour apart, instead of one every few minutes. */
|
||||||
|
public const MAX_ATTEMPTS = 3;
|
||||||
|
public const RETRY_AFTER = 3600;
|
||||||
|
|
||||||
|
/** @throws \InvalidArgumentException */
|
||||||
|
public static function normalize(string $schedule): string {
|
||||||
|
$schedule = trim($schedule);
|
||||||
|
if ($schedule === '' || strtolower($schedule) === 'off') {
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
if (preg_match('/^(([01]\d|2[0-3]):[0-5]\d)(?:\/(\d{1,2}))?$/', $schedule, $m) !== 1) {
|
||||||
|
throw new \InvalidArgumentException('Use "off", HH:MM for once a day, or HH:MM/N for every N hours from HH:MM');
|
||||||
|
}
|
||||||
|
$hours = (int)($m[3] ?? 24);
|
||||||
|
if ($hours < 1 || $hours > 24) {
|
||||||
|
throw new \InvalidArgumentException('Every 1 to 24 hours');
|
||||||
|
}
|
||||||
|
return $hours === 24 ? $m[1] : $m[1] . '/' . $hours;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The most recent scheduled moment at or before $now; null without a schedule.
|
||||||
|
*
|
||||||
|
* @param non-empty-string $zone
|
||||||
|
*/
|
||||||
|
public static function lastMoment(string $schedule, string $zone, int $now): ?int {
|
||||||
|
if ($schedule === '') {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
$tz = new \DateTimeZone($zone);
|
||||||
|
$today = (new \DateTimeImmutable('@' . $now))->setTimezone($tz);
|
||||||
|
$moments = array_merge(self::moments($schedule, $today->modify('-1 day'), $tz), self::moments($schedule, $today, $tz));
|
||||||
|
$last = null;
|
||||||
|
foreach ($moments as $moment) {
|
||||||
|
if ($moment <= $now && ($last === null || $moment > $last)) {
|
||||||
|
$last = $moment;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $last;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @return list<int> the moments of one day */
|
||||||
|
private static function moments(string $schedule, \DateTimeImmutable $day, \DateTimeZone $tz): array {
|
||||||
|
[$time, $every] = array_pad(explode('/', $schedule), 2, '24');
|
||||||
|
[$hour, $minute] = array_map('intval', explode(':', $time));
|
||||||
|
$moments = [];
|
||||||
|
for ($h = $hour; $h < 24; $h += (int)$every) {
|
||||||
|
$moments[] = (new \DateTimeImmutable($day->format('Y-m-d') . sprintf(' %02d:%02d', $h, $minute), $tz))->getTimestamp();
|
||||||
|
}
|
||||||
|
return $moments;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether a backup has to start for $moment. One that was made since then is enough, also when
|
||||||
|
* several moments went by during a long backup. A moment from before the schedule was set does
|
||||||
|
* not count: choosing a time must not start a backup right away.
|
||||||
|
*/
|
||||||
|
public static function isDue(int $moment, int $since, ?int $lastDoneStarted, int $attempts, ?int $lastAttemptFinished, int $now): bool {
|
||||||
|
if ($moment < $since || ($lastDoneStarted !== null && $lastDoneStarted >= $moment) || $attempts >= self::MAX_ATTEMPTS) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return $attempts === 0 || $now - (int)$lastAttemptFinished >= self::RETRY_AFTER;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,6 +11,7 @@ use OCA\NgBackup\AppInfo\Application;
|
|||||||
use OCA\NgBackup\Db\RunMapper;
|
use OCA\NgBackup\Db\RunMapper;
|
||||||
use OCA\NgBackup\Db\SnapshotMapper;
|
use OCA\NgBackup\Db\SnapshotMapper;
|
||||||
use OCA\NgBackup\Db\Target;
|
use OCA\NgBackup\Db\Target;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
use OCP\IAppConfig;
|
use OCP\IAppConfig;
|
||||||
use OCP\Lock\ILockingProvider;
|
use OCP\Lock\ILockingProvider;
|
||||||
use OCP\Lock\LockedException;
|
use OCP\Lock\LockedException;
|
||||||
@@ -33,6 +34,54 @@ final class PruneService {
|
|||||||
) {
|
) {
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Characters of a note on a snapshot (the column holds 1000) */
|
||||||
|
public const NOTE_MAX = 500;
|
||||||
|
|
||||||
|
/** Pin or unpin a snapshot: the retention policy leaves a pinned one alone. */
|
||||||
|
public function setPinned(Target $target, string $snapshotId, bool $pinned): void {
|
||||||
|
$changed = $this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $pinned): bool {
|
||||||
|
$was = in_array($snapshotId, $repo->pinnedSnapshots(), true);
|
||||||
|
$repo->setPinned($snapshotId, $pinned);
|
||||||
|
return $was !== $pinned;
|
||||||
|
});
|
||||||
|
$this->targets->syncSnapshot($target, $snapshotId, pinned: $pinned);
|
||||||
|
if ($changed) {
|
||||||
|
// Unpinning opens the way to removing it, so every administrator hears about that.
|
||||||
|
$this->alerts->securityEvent($pinned ? 'snapshot_pinned' : 'snapshot_unpinned', ['target' => $target->getName(), 'snapshot' => $snapshotId], !$pinned);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public function setNote(Target $target, string $snapshotId, string $note): void {
|
||||||
|
$note = trim((string)preg_replace('/[\x00-\x1f\x7f]+/', ' ', $note));
|
||||||
|
if (!mb_check_encoding($note, 'UTF-8')) {
|
||||||
|
throw new \InvalidArgumentException('A note must be valid UTF-8');
|
||||||
|
}
|
||||||
|
if (mb_strlen($note) > self::NOTE_MAX) {
|
||||||
|
throw new \InvalidArgumentException('A note has at most ' . self::NOTE_MAX . ' characters');
|
||||||
|
}
|
||||||
|
$this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $note): bool {
|
||||||
|
$repo->setSnapshotNote($snapshotId, $note);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
$this->targets->syncSnapshot($target, $snapshotId, note: $note);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Write to the catalog next to a running backup or restore, which only takes it for a moment to
|
||||||
|
* commit. Not during a cleanup, which holds a repository opened earlier and writes from that.
|
||||||
|
*
|
||||||
|
* @param callable(Repository): bool $change
|
||||||
|
*/
|
||||||
|
private function changeCatalog(Target $target, callable $change): bool {
|
||||||
|
try {
|
||||||
|
return $this->leases->with(self::lockKey($target), LeaseService::SHARED, 300,
|
||||||
|
fn (): bool => $this->leases->with(self::catalogLockKey($target), LeaseService::EXCLUSIVE, 300,
|
||||||
|
fn (): bool => $change($this->targets->repository($target))));
|
||||||
|
} catch (LockedException) {
|
||||||
|
throw new \RuntimeException('A cleanup of ' . $target->getName() . ' or another change to its restore points is in progress; try again later');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public function untrash(Target $target, string $snapshotId): void {
|
public function untrash(Target $target, string $snapshotId): void {
|
||||||
$lease = $this->lockExclusive($target, 300);
|
$lease = $this->lockExclusive($target, 300);
|
||||||
try {
|
try {
|
||||||
@@ -87,6 +136,7 @@ final class PruneService {
|
|||||||
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
|
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
|
||||||
$s->setFiles((int)($meta['stats']['files'] ?? 0));
|
$s->setFiles((int)($meta['stats']['files'] ?? 0));
|
||||||
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
|
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
|
||||||
|
$s->setNote($repo->snapshotNotes()[$snapshotId] ?? null);
|
||||||
try {
|
try {
|
||||||
$this->snapshots->insert($s);
|
$this->snapshots->insert($s);
|
||||||
} catch (\Throwable) {
|
} catch (\Throwable) {
|
||||||
@@ -208,7 +258,7 @@ final class PruneService {
|
|||||||
$meta = $repo->snapshotMeta($id);
|
$meta = $repo->snapshotMeta($id);
|
||||||
$times[$id] = (int)strtotime((string)($meta['time'] ?? '@0'));
|
$times[$id] = (int)strtotime((string)($meta['time'] ?? '@0'));
|
||||||
}
|
}
|
||||||
$keep = $this->policy()->keep($times);
|
$keep = $this->policy()->keepWithPinned($times, $repo->pinnedSnapshots());
|
||||||
$forget = array_values(array_diff(array_keys($times), array_keys($keep)));
|
$forget = array_values(array_diff(array_keys($times), array_keys($keep)));
|
||||||
$actor = $scheduled ? 'retention policy' : $this->alerts->actor();
|
$actor = $scheduled ? 'retention policy' : $this->alerts->actor();
|
||||||
$purged = [];
|
$purged = [];
|
||||||
|
|||||||
@@ -151,6 +151,12 @@ final class RestoreFullService {
|
|||||||
// The database cannot go into this installation; the files can.
|
// The database cannot go into this installation; the files can.
|
||||||
$check['dbUnreadable'] = 'the database cannot be restored here: ' . $this->describeBlockers($check);
|
$check['dbUnreadable'] = 'the database cannot be restored here: ' . $this->describeBlockers($check);
|
||||||
}
|
}
|
||||||
|
if ($interrupted) {
|
||||||
|
// A restore that was killed leaves its leases, for up to two hours: they would keep
|
||||||
|
// the first backup of the recovered server waiting after this restore completes.
|
||||||
|
// In maintenance mode, on a partly restored instance, nothing else reads the location.
|
||||||
|
$this->leases->dropShared(PruneService::lockKey($target));
|
||||||
|
}
|
||||||
$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
|
$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
|
||||||
$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check, $salvage);
|
$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check, $salvage);
|
||||||
$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
|
$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
|
||||||
@@ -175,6 +181,8 @@ final class RestoreFullService {
|
|||||||
if ($dbError === null) {
|
if ($dbError === null) {
|
||||||
try {
|
try {
|
||||||
$this->restoreDatabase($repo, $meta, $manifest, $dbReport, $beat, $skipMissingTables);
|
$this->restoreDatabase($repo, $meta, $manifest, $dbReport, $beat, $skipMissingTables);
|
||||||
|
} catch (DatabaseAlreadyRestoredException $e) {
|
||||||
|
throw $e;
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
if (!$salvage) {
|
if (!$salvage) {
|
||||||
throw $e;
|
throw $e;
|
||||||
@@ -201,6 +209,8 @@ final class RestoreFullService {
|
|||||||
try {
|
try {
|
||||||
/** @var array<string, mixed> $backupConfig */
|
/** @var array<string, mixed> $backupConfig */
|
||||||
$backupConfig = $this->fetchBackupConfig($target, $snapshotId, $salvage);
|
$backupConfig = $this->fetchBackupConfig($target, $snapshotId, $salvage);
|
||||||
|
} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
|
||||||
|
throw $e; // nowhere to put it: not a config.php that cannot be read
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
if (!$salvage) {
|
if (!$salvage) {
|
||||||
throw $e;
|
throw $e;
|
||||||
@@ -259,8 +269,20 @@ final class RestoreFullService {
|
|||||||
$dumper = new DbDumper($this->db->getInner(), $prefix);
|
$dumper = new DbDumper($this->db->getInner(), $prefix);
|
||||||
try {
|
try {
|
||||||
$dumper->restore($repo, $manifest, $dbReport, $beat, $skipMissingTables);
|
$dumper->restore($repo, $manifest, $dbReport, $beat, $skipMissingTables);
|
||||||
} finally {
|
} catch (\Throwable $e) {
|
||||||
|
try {
|
||||||
$this->restoreOwnAppConfig($ownAppConfig);
|
$this->restoreOwnAppConfig($ownAppConfig);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
// rolled back, so they are still there: the error that brought us here is the one to report
|
||||||
|
}
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$this->restoreOwnAppConfig($ownAppConfig);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
// The old server's database is in place by now: not a case for "the database stays as
|
||||||
|
// the fresh installation made it" under --salvage. A run again restores these rows.
|
||||||
|
throw new DatabaseAlreadyRestoredException('The database was restored, but NG Backup could not put back its own settings: ' . $e->getMessage(), 0, $e);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -61,7 +61,8 @@ final class RestoreService {
|
|||||||
$prefix = trim($path, '/');
|
$prefix = trim($path, '/');
|
||||||
$prefix = $prefix === '' ? '' : $prefix . '/';
|
$prefix = $prefix === '' ? '' : $prefix . '/';
|
||||||
$children = [];
|
$children = [];
|
||||||
foreach ($this->targets->repository($target)->entries($snapshotId) as $e) {
|
$repo = $this->targets->repository($target);
|
||||||
|
foreach ($repo->entries($snapshotId) as $e) {
|
||||||
if ($prefix !== '' && !str_starts_with($e['p'], $prefix)) {
|
if ($prefix !== '' && !str_starts_with($e['p'], $prefix)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -75,6 +76,12 @@ final class RestoreService {
|
|||||||
}
|
}
|
||||||
$children[$name] = $c;
|
$children[$name] = $c;
|
||||||
}
|
}
|
||||||
|
foreach ($repo->emptyDirs($snapshotId) as [$dir]) { // read along with the entries
|
||||||
|
if ($prefix === '' || str_starts_with($dir, $prefix)) {
|
||||||
|
$name = explode('/', substr($dir, strlen($prefix)), 2)[0];
|
||||||
|
$children[$name] ??= ['name' => $name, 'type' => 'dir', 'size' => 0, 'mtime' => null, 'files' => 0];
|
||||||
|
}
|
||||||
|
}
|
||||||
ksort($children, SORT_STRING);
|
ksort($children, SORT_STRING);
|
||||||
return array_values($children);
|
return array_values($children);
|
||||||
}
|
}
|
||||||
@@ -123,7 +130,17 @@ final class RestoreService {
|
|||||||
$entries[substr($e['p'], strlen($source) + 1)] = $e;
|
$entries[substr($e['p'], strlen($source) + 1)] = $e;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if ($entries === []) {
|
// Folders without anything in them, relative to $source ('' when $source itself is one).
|
||||||
|
/** @var array<array-key, int> $emptyDirs modification times; a folder called "2026" is an integer key */
|
||||||
|
$emptyDirs = [];
|
||||||
|
foreach ($single ? [] : $repo->emptyDirs($snapshotId) as [$dir, $mtime]) {
|
||||||
|
if ($dir === $source) {
|
||||||
|
$emptyDirs[''] = $mtime;
|
||||||
|
} elseif (str_starts_with($dir, $source . '/')) {
|
||||||
|
$emptyDirs[substr($dir, strlen($source) + 1)] = $mtime;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ($entries === [] && $emptyDirs === []) {
|
||||||
throw new NotFoundException("Nothing at $source in snapshot $snapshotId");
|
throw new NotFoundException("Nothing at $source in snapshot $snapshotId");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -134,9 +151,21 @@ final class RestoreService {
|
|||||||
} elseif ($mode !== self::MODE_MERGE && $mode !== self::MODE_REPLACE) {
|
} elseif ($mode !== self::MODE_MERGE && $mode !== self::MODE_REPLACE) {
|
||||||
throw new \InvalidArgumentException("Unknown mode $mode");
|
throw new \InvalidArgumentException("Unknown mode $mode");
|
||||||
}
|
}
|
||||||
|
$destExisted = $parentRel === '' || $userFolder->nodeExists($parentRel);
|
||||||
$dest = $this->ensureFolder($userFolder, $parentRel);
|
$dest = $this->ensureFolder($userFolder, $parentRel);
|
||||||
$names = $single ? [basename($rel) => $entries['']] : $entries;
|
$names = $single ? [basename($rel) => $entries['']] : $entries;
|
||||||
|
|
||||||
|
// The folders with something in them that this restore creates get their time back; one that
|
||||||
|
// was already there keeps its own, as its content changes now. Noted before anything is written.
|
||||||
|
/** @var array<array-key, int> $newFolders a folder called "2026" is an integer key */
|
||||||
|
$newFolders = [];
|
||||||
|
foreach ($single ? [] : $repo->folderTimes($snapshotId) as [$dir, $mtime]) {
|
||||||
|
$relative = $dir === $source ? '' : (str_starts_with($dir, $source . '/') ? substr($dir, strlen($source) + 1) : null);
|
||||||
|
if ($relative !== null && ($relative === '' ? !$destExisted : !$dest->nodeExists($relative))) {
|
||||||
|
$newFolders[$relative] = $mtime;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
$this->cleaned = [];
|
$this->cleaned = [];
|
||||||
$restored = 0;
|
$restored = 0;
|
||||||
foreach ($names as $relPath => $entry) {
|
foreach ($names as $relPath => $entry) {
|
||||||
@@ -156,9 +185,28 @@ final class RestoreService {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
foreach ($emptyDirs as $dir => $mtime) {
|
||||||
|
$dir = (string)$dir;
|
||||||
|
$existed = $dir === '' ? $destExisted : $dest->nodeExists($dir); // one that is there keeps its own time
|
||||||
|
$folder = $this->ensureFolder($dest, $dir);
|
||||||
|
if (!$existed) {
|
||||||
|
$folder->touch($mtime);
|
||||||
|
}
|
||||||
|
if ($progress !== null) {
|
||||||
|
$progress($restored, count($names), $dir); // keeps the lease alive too
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
$trashed = 0;
|
$trashed = 0;
|
||||||
if ($mode === self::MODE_REPLACE && !$single) {
|
if ($mode === self::MODE_REPLACE && !$single) {
|
||||||
$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)));
|
$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)) + $emptyDirs);
|
||||||
|
}
|
||||||
|
// Last and deepest first: a change in a folder also touches the folders above it.
|
||||||
|
foreach (array_reverse($newFolders, true) as $dir => $mtime) {
|
||||||
|
$this->ensureFolder($dest, (string)$dir)->touch($mtime);
|
||||||
|
if ($progress !== null) {
|
||||||
|
$progress($restored, count($names), (string)$dir); // keeps the lease alive too
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())];
|
return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())];
|
||||||
}
|
}
|
||||||
@@ -178,7 +226,7 @@ final class RestoreService {
|
|||||||
throw new \InvalidArgumentException('Invalid path: ' . $prefix);
|
throw new \InvalidArgumentException('Invalid path: ' . $prefix);
|
||||||
}
|
}
|
||||||
if (!is_dir($directory) && !mkdir($directory, 0750, true) && !is_dir($directory)) {
|
if (!is_dir($directory) && !mkdir($directory, 0750, true) && !is_dir($directory)) {
|
||||||
throw new \RuntimeException("Cannot create $directory");
|
throw new \OCA\NgBackup\Repository\RestoreTargetException("Cannot create $directory (disk full, or no permission?)");
|
||||||
}
|
}
|
||||||
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $prefix, $directory, $stripPrefix, $heartbeat, &$skipped, $keepUnchanged): int {
|
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $prefix, $directory, $stripPrefix, $heartbeat, &$skipped, $keepUnchanged): int {
|
||||||
$own = LeaseService::throttled($refresh);
|
$own = LeaseService::throttled($refresh);
|
||||||
@@ -262,7 +310,7 @@ final class RestoreService {
|
|||||||
foreach ($folder->getDirectoryListing() as $node) {
|
foreach ($folder->getDirectoryListing() as $node) {
|
||||||
$rel = $prefix === '' ? $node->getName() : $prefix . '/' . $node->getName();
|
$rel = $prefix === '' ? $node->getName() : $prefix . '/' . $node->getName();
|
||||||
if ($node instanceof Folder) {
|
if ($node instanceof Folder) {
|
||||||
$hasKept = false;
|
$hasKept = isset($keep[$rel]); // a folder that was empty in the snapshot
|
||||||
foreach (array_keys($keep) as $k) {
|
foreach (array_keys($keep) as $k) {
|
||||||
if (str_starts_with((string)$k, $rel . '/')) {
|
if (str_starts_with((string)$k, $rel . '/')) {
|
||||||
$hasKept = true;
|
$hasKept = true;
|
||||||
|
|||||||
@@ -28,6 +28,19 @@ final class RetentionPolicy {
|
|||||||
return ['last' => $this->last, 'daily' => $this->daily, 'weekly' => $this->weekly, 'monthly' => $this->monthly];
|
return ['last' => $this->last, 'daily' => $this->daily, 'weekly' => $this->weekly, 'monthly' => $this->monthly];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* keep() for the snapshots that are not pinned, plus the pinned ones. A pinned snapshot does not
|
||||||
|
* take the place of a daily, weekly or monthly one.
|
||||||
|
*
|
||||||
|
* @param array<string, int> $snapshots id => unix time
|
||||||
|
* @param list<string> $pinned
|
||||||
|
* @return array<string, list<string>>
|
||||||
|
*/
|
||||||
|
public function keepWithPinned(array $snapshots, array $pinned): array {
|
||||||
|
$pins = array_intersect_key($snapshots, array_flip($pinned));
|
||||||
|
return $this->keep(array_diff_key($snapshots, $pins)) + array_fill_keys(array_keys($pins), ['pinned']);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @param array<string, int> $snapshots id => unix time
|
* @param array<string, int> $snapshots id => unix time
|
||||||
* @return array<string, list<string>> id => reasons it is kept (ids not in the result are removed)
|
* @return array<string, list<string>> id => reasons it is kept (ids not in the result are removed)
|
||||||
|
|||||||
@@ -13,10 +13,11 @@ use OCP\IConfig;
|
|||||||
use OCP\IGroupManager;
|
use OCP\IGroupManager;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a
|
* The time zone times are shown in, and the daily time from before each location had its own
|
||||||
* zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone
|
* schedule (see LocationSchedule). Nextcloud runs PHP in UTC, so a time without a zone meant UTC
|
||||||
* is converted once to the zone of the server or of an administrator, at today's offset: it then
|
* (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone is converted once
|
||||||
* shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
|
* to the zone of the server or of an administrator, at today's offset: it then shows as it
|
||||||
|
* effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
|
||||||
*/
|
*/
|
||||||
final class Schedule {
|
final class Schedule {
|
||||||
public const TIME_KEY = 'schedule_time';
|
public const TIME_KEY = 'schedule_time';
|
||||||
@@ -44,27 +45,9 @@ final class Schedule {
|
|||||||
return ['time' => $time, 'timezone' => $zone];
|
return ['time' => $time, 'timezone' => $zone];
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @throws \InvalidArgumentException for a malformed time or an unknown zone */
|
/** @return non-empty-string the zone of a location's schedule, or else this one */
|
||||||
public function set(string $time, string $zone): void {
|
public function zoneOr(?string $zone): string {
|
||||||
if ($time !== '' && !self::validTime($time)) {
|
return $zone !== null && self::validZone($zone) ? $zone : $this->get()['timezone'];
|
||||||
throw new \InvalidArgumentException('Use HH:MM or empty');
|
|
||||||
}
|
|
||||||
if (!self::validZone($zone)) {
|
|
||||||
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
|
|
||||||
}
|
|
||||||
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
|
|
||||||
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */
|
|
||||||
public function todayAt(?int $now = null): ?int {
|
|
||||||
['time' => $time, 'timezone' => $zone] = $this->get();
|
|
||||||
if ($time === '' || !self::validTime($time)) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
$tz = new \DateTimeZone($zone);
|
|
||||||
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d');
|
|
||||||
return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -35,6 +35,7 @@ final class TargetService {
|
|||||||
private \OCA\NgBackup\Db\DbIndexCache $indexCache,
|
private \OCA\NgBackup\Db\DbIndexCache $indexCache,
|
||||||
private \OCA\NgBackup\Db\AppConfigCatalogAnchor $anchor,
|
private \OCA\NgBackup\Db\AppConfigCatalogAnchor $anchor,
|
||||||
private SnapshotMapper $snapshots,
|
private SnapshotMapper $snapshots,
|
||||||
|
private AlertService $alerts,
|
||||||
) {
|
) {
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -78,6 +79,11 @@ final class TargetService {
|
|||||||
$target->setBasePath(trim($basePath, '/'));
|
$target->setBasePath(trim($basePath, '/'));
|
||||||
$target->setRepositoryId($repo->id());
|
$target->setRepositoryId($repo->id());
|
||||||
$target->setCreatedAt(time());
|
$target->setCreatedAt(time());
|
||||||
|
// Like the location before it: a backup scheme is usually one scheme.
|
||||||
|
$first = $this->mapper->findAll()[0] ?? null;
|
||||||
|
$target->setSchedule($first?->getSchedule() ?? '');
|
||||||
|
$target->setScheduleZone($first?->getScheduleZone());
|
||||||
|
$target->setScheduleSince(time());
|
||||||
$target = $this->mapper->insert($target);
|
$target = $this->mapper->insert($target);
|
||||||
if (!$created) {
|
if (!$created) {
|
||||||
// Disaster recovery: occ backup:list reads this local cache, not the repository
|
// Disaster recovery: occ backup:list reads this local cache, not the repository
|
||||||
@@ -208,7 +214,25 @@ final class TargetService {
|
|||||||
return $result;
|
return $result;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Update the local copy of a snapshot's note or pin, if the snapshot is listed here at all. */
|
||||||
|
public function syncSnapshot(Target $target, string $snapshotId, ?string $note = null, ?bool $pinned = null): void {
|
||||||
|
try {
|
||||||
|
$s = $this->snapshots->findOne($target->getId(), $snapshotId);
|
||||||
|
} catch (DoesNotExistException) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if ($note !== null) {
|
||||||
|
$s->setNote($note !== '' ? $note : null);
|
||||||
|
}
|
||||||
|
if ($pinned !== null) {
|
||||||
|
$s->setPinned($pinned);
|
||||||
|
}
|
||||||
|
$this->snapshots->update($s);
|
||||||
|
}
|
||||||
|
|
||||||
private function adoptSnapshots(Target $target, Repository $repo): void {
|
private function adoptSnapshots(Target $target, Repository $repo): void {
|
||||||
|
$notes = $repo->snapshotNotes();
|
||||||
|
$pinned = array_flip($repo->pinnedSnapshots());
|
||||||
foreach ($repo->listSnapshots() as $snapshotId) {
|
foreach ($repo->listSnapshots() as $snapshotId) {
|
||||||
try {
|
try {
|
||||||
$this->snapshots->findOne($target->getId(), $snapshotId);
|
$this->snapshots->findOne($target->getId(), $snapshotId);
|
||||||
@@ -227,6 +251,8 @@ final class TargetService {
|
|||||||
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
|
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
|
||||||
$s->setFiles((int)($meta['stats']['files'] ?? 0));
|
$s->setFiles((int)($meta['stats']['files'] ?? 0));
|
||||||
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
|
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
|
||||||
|
$s->setNote($notes[$snapshotId] ?? null);
|
||||||
|
$s->setPinned(isset($pinned[$snapshotId]));
|
||||||
$this->snapshots->insert($s);
|
$this->snapshots->insert($s);
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]);
|
Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]);
|
||||||
@@ -248,6 +274,26 @@ final class TargetService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** Size limit for the data stored on this location (null = none). */
|
/** Size limit for the data stored on this location (null = none). */
|
||||||
|
/**
|
||||||
|
* @param string $schedule see LocationSchedule
|
||||||
|
* @throws \InvalidArgumentException
|
||||||
|
*/
|
||||||
|
public function setSchedule(Target $target, string $schedule, string $zone): Target {
|
||||||
|
if (!Schedule::validZone($zone)) {
|
||||||
|
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
|
||||||
|
}
|
||||||
|
$new = LocationSchedule::normalize($schedule);
|
||||||
|
$turnedOff = $new === '' && (string)$target->getSchedule() !== '';
|
||||||
|
$target->setSchedule($new);
|
||||||
|
$target->setScheduleZone($zone);
|
||||||
|
$target->setScheduleSince(time());
|
||||||
|
$target = $this->mapper->update($target);
|
||||||
|
if ($turnedOff) {
|
||||||
|
$this->alerts->securityEvent('schedule_off', ['target' => $target->getName()]);
|
||||||
|
}
|
||||||
|
return $target;
|
||||||
|
}
|
||||||
|
|
||||||
public function setMaxBytes(Target $target, ?int $maxBytes): Target {
|
public function setMaxBytes(Target $target, ?int $maxBytes): Target {
|
||||||
$target->setMaxBytes($maxBytes);
|
$target->setMaxBytes($maxBytes);
|
||||||
return $this->mapper->update($target);
|
return $this->mapper->update($target);
|
||||||
|
|||||||
@@ -186,6 +186,7 @@
|
|||||||
<code><![CDATA[$this->db]]></code>
|
<code><![CDATA[$this->db]]></code>
|
||||||
<code><![CDATA[$this->db]]></code>
|
<code><![CDATA[$this->db]]></code>
|
||||||
<code><![CDATA[$this->db]]></code>
|
<code><![CDATA[$this->db]]></code>
|
||||||
|
<code><![CDATA[$this->db]]></code>
|
||||||
<code><![CDATA[TransactionIsolationLevel]]></code>
|
<code><![CDATA[TransactionIsolationLevel]]></code>
|
||||||
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
||||||
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
||||||
@@ -299,6 +300,22 @@
|
|||||||
<code><![CDATA[$t]]></code>
|
<code><![CDATA[$t]]></code>
|
||||||
</UndefinedDocblockClass>
|
</UndefinedDocblockClass>
|
||||||
</file>
|
</file>
|
||||||
|
<file src="lib/Migration/Version000008Date20261107010000.php">
|
||||||
|
<UndefinedDocblockClass>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
</UndefinedDocblockClass>
|
||||||
|
</file>
|
||||||
|
<file src="lib/Migration/Version000007Date20261107000000.php">
|
||||||
|
<UndefinedDocblockClass>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
</UndefinedDocblockClass>
|
||||||
|
</file>
|
||||||
<file src="lib/Migration/Version000005Date20261002040000.php">
|
<file src="lib/Migration/Version000005Date20261002040000.php">
|
||||||
<UndefinedDocblockClass>
|
<UndefinedDocblockClass>
|
||||||
<code><![CDATA[$t]]></code>
|
<code><![CDATA[$t]]></code>
|
||||||
|
|||||||
+4
-2
@@ -21,7 +21,7 @@ export default {
|
|||||||
addSlot: (passphrase, label) => call('post', '/key/slots', { passphrase, label }),
|
addSlot: (passphrase, label) => call('post', '/key/slots', { passphrase, label }),
|
||||||
replaceSlot: (slot, passphrase, label) => call('put', `/key/slots/${slot}`, { passphrase, label }),
|
replaceSlot: (slot, passphrase, label) => call('put', `/key/slots/${slot}`, { passphrase, label }),
|
||||||
removeSlot: (slot) => call('delete', `/key/slots/${slot}`),
|
removeSlot: (slot) => call('delete', `/key/slots/${slot}`),
|
||||||
trash: (targetId) => call('get', `/targets/${targetId}/trash`),
|
trash: (targetId, after = '') => call('get', `/targets/${targetId}/trash`, { after }),
|
||||||
untrash: (targetId, snapshotId) => call('post', `/targets/${targetId}/trash/${snapshotId}`),
|
untrash: (targetId, snapshotId) => call('post', `/targets/${targetId}/trash/${snapshotId}`),
|
||||||
kitUrl: () => url('/key/kit'),
|
kitUrl: () => url('/key/kit'),
|
||||||
confirmKit: (accepted, phrase, code) => call('post', '/key/confirm', { accepted, phrase, code }),
|
confirmKit: (accepted, phrase, code) => call('post', '/key/confirm', { accepted, phrase, code }),
|
||||||
@@ -34,8 +34,10 @@ export default {
|
|||||||
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
|
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
|
||||||
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
|
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
|
||||||
tick: () => call('post', '/tick'),
|
tick: () => call('post', '/tick'),
|
||||||
setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }),
|
setTargetSchedule: (id, schedule, timezone) => call('put', `/targets/${id}/schedule`, { schedule, timezone }),
|
||||||
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
|
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
|
||||||
|
setSnapshotNote: (targetId, snapshotId, note) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/note`, { note }),
|
||||||
|
setSnapshotPinned: (targetId, snapshotId, pinned) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/pin`, { pinned }),
|
||||||
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
|
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
|
||||||
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
|
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,10 +5,7 @@
|
|||||||
<template>
|
<template>
|
||||||
<NcSettingsSection :name="t('ng_backup', 'Backups')"
|
<NcSettingsSection :name="t('ng_backup', 'Backups')"
|
||||||
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
|
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
|
||||||
<div class="ngb-row">
|
<LocationScheduleRow v-for="target in status.targets" :key="target.id" :target="target" @changed="emit('changed')" @error="error = $event" />
|
||||||
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, {zone}, empty = off)', { zone: browserZone })" />
|
|
||||||
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
|
|
||||||
</div>
|
|
||||||
<div class="ngb-row">
|
<div class="ngb-row">
|
||||||
<NcTextField v-model="label" class="ngb-label" :maxlength="100" :label="t('ng_backup', 'Label (optional)')"
|
<NcTextField v-model="label" class="ngb-label" :maxlength="100" :label="t('ng_backup', 'Label (optional)')"
|
||||||
:placeholder="t('ng_backup', 'e.g. before the upgrade to Nextcloud 35')" />
|
:placeholder="t('ng_backup', 'e.g. before the upgrade to Nextcloud 35')" />
|
||||||
@@ -37,7 +34,7 @@
|
|||||||
<td>{{ formatDateTime(run.startedAt) }}</td>
|
<td>{{ formatDateTime(run.startedAt) }}</td>
|
||||||
<td>{{ targetName(run.targetId) }}</td>
|
<td>{{ targetName(run.targetId) }}</td>
|
||||||
<td>{{ run.kind }}</td>
|
<td>{{ run.kind }}</td>
|
||||||
<td>{{ run.snapshot ? formatDateTime(run.snapshot.createdAt) + (run.snapshot.label ? ' – ' + run.snapshot.label : '') : '' }}</td>
|
<td>{{ run.snapshot ? (run.snapshot.pinned ? '📌 ' : '') + formatDateTime(run.snapshot.createdAt) + (run.snapshot.label ? ' – ' + run.snapshot.label : '') + (run.snapshot.note ? ' – ' + run.snapshot.note : '') : '' }}</td>
|
||||||
<td :class="'ngb-' + run.status">{{ runStatus(run) }}{{ run.error ? ': ' + run.error : '' }}</td>
|
<td :class="'ngb-' + run.status">{{ runStatus(run) }}{{ run.error ? ': ' + run.error : '' }}</td>
|
||||||
<td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
|
<td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
|
||||||
<td>{{ run.stats?.files ? formatSize(run.stats.files.bytes) : '' }}</td>
|
<td>{{ run.stats?.files ? formatSize(run.stats.files.bytes) : '' }}</td>
|
||||||
@@ -60,24 +57,19 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { ref, watch } from 'vue'
|
import { ref } from 'vue'
|
||||||
import { n, t } from '@nextcloud/l10n'
|
import { n, t } from '@nextcloud/l10n'
|
||||||
import NcButton from '@nextcloud/vue/components/NcButton'
|
import NcButton from '@nextcloud/vue/components/NcButton'
|
||||||
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
||||||
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||||
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
||||||
|
import LocationScheduleRow from './LocationScheduleRow.vue'
|
||||||
import api from '../api.js'
|
import api from '../api.js'
|
||||||
import { formatDateTime, formatSize, toZone } from '../format.js'
|
import { formatDateTime, formatSize } from '../format.js'
|
||||||
|
|
||||||
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
|
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
|
||||||
const emit = defineEmits(['changed', 'show-all-runs'])
|
const emit = defineEmits(['changed', 'show-all-runs'])
|
||||||
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
|
|
||||||
// zone is shown in this one, so it reads as the moment it runs here.
|
|
||||||
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
|
|
||||||
const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s
|
|
||||||
const schedule = ref(shown(props.status.schedule))
|
|
||||||
const error = ref('')
|
const error = ref('')
|
||||||
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
|
|
||||||
|
|
||||||
const label = ref('')
|
const label = ref('')
|
||||||
|
|
||||||
@@ -97,6 +89,9 @@ const runStatus = (run) => {
|
|||||||
if (run.status !== 'running') {
|
if (run.status !== 'running') {
|
||||||
return run.status
|
return run.status
|
||||||
}
|
}
|
||||||
|
if (run.queued) {
|
||||||
|
return t('ng_backup', 'queued, after the backup that is running now')
|
||||||
|
}
|
||||||
const detail = run.phase + percent(run)
|
const detail = run.phase + percent(run)
|
||||||
if (run.waitingFor) {
|
if (run.waitingFor) {
|
||||||
return t('ng_backup', 'waiting for background jobs ({detail}, last step {minutes} min ago)', { detail, minutes: Math.floor(run.waitingFor / 60) })
|
return t('ng_backup', 'waiting for background jobs ({detail}, last step {minutes} min ago)', { detail, minutes: Math.floor(run.waitingFor / 60) })
|
||||||
@@ -116,21 +111,10 @@ async function start(target) {
|
|||||||
error.value = e.message
|
error.value = e.message
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function saveSchedule() {
|
|
||||||
error.value = ''
|
|
||||||
try {
|
|
||||||
await api.setSchedule(schedule.value.trim(), browserZone)
|
|
||||||
emit('changed')
|
|
||||||
} catch (e) {
|
|
||||||
error.value = e.message
|
|
||||||
}
|
|
||||||
}
|
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
|
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
|
||||||
.ngb-time { max-width: 260px; }
|
|
||||||
.ngb-label { max-width: 360px; }
|
.ngb-label { max-width: 360px; }
|
||||||
.ngb-table-wrap { overflow-x: auto; }
|
.ngb-table-wrap { overflow-x: auto; }
|
||||||
.ngb-table { width: 100%; border-collapse: collapse; }
|
.ngb-table { width: 100%; border-collapse: collapse; }
|
||||||
|
|||||||
+49
-8
@@ -6,8 +6,18 @@
|
|||||||
<NcSettingsSection :name="t('ng_backup', 'Bin')"
|
<NcSettingsSection :name="t('ng_backup', 'Bin')"
|
||||||
:description="t('ng_backup', 'Restore points that were removed, most of them by the retention policy. They can be brought back until they are deleted permanently.')">
|
:description="t('ng_backup', 'Restore points that were removed, most of them by the retention policy. They can be brought back until they are deleted permanently.')">
|
||||||
<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
|
<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
|
||||||
<NcLoadingIcon v-if="loading" :size="32" />
|
<div v-if="reading" class="ngb-reading" role="status">
|
||||||
<NcNoteCard v-else-if="empty" type="info">{{ t('ng_backup', 'The bin is empty.') }}</NcNoteCard>
|
<template v-if="reading.total === null">
|
||||||
|
<NcLoadingIcon :size="20" />
|
||||||
|
<span>{{ t('ng_backup', 'Connecting to {location} …', { location: reading.name }) }}</span>
|
||||||
|
</template>
|
||||||
|
<template v-else>
|
||||||
|
<span>{{ t('ng_backup', 'Reading removed restore points from {location}: {done} of {total}', { location: reading.name, done: reading.done, total: reading.total }) }}</span>
|
||||||
|
<NcProgressBar :value="reading.total ? Math.round(100 * reading.done / reading.total) : 0" />
|
||||||
|
</template>
|
||||||
|
<p class="ngb-muted">{{ t('ng_backup', 'This can take a while: every removed restore point is read from the location.') }}</p>
|
||||||
|
</div>
|
||||||
|
<NcNoteCard v-else-if="empty && !error" type="info">{{ t('ng_backup', 'The bin is empty.') }}</NcNoteCard>
|
||||||
|
|
||||||
<template v-for="target in status.targets" :key="'trash' + target.id">
|
<template v-for="target in status.targets" :key="'trash' + target.id">
|
||||||
<div v-if="trash[target.id]?.length" class="ngb-trash">
|
<div v-if="trash[target.id]?.length" class="ngb-trash">
|
||||||
@@ -39,12 +49,13 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { computed, reactive, ref, watch } from 'vue'
|
import { computed, onBeforeUnmount, reactive, ref, watch } from 'vue'
|
||||||
import { n, t } from '@nextcloud/l10n'
|
import { n, t } from '@nextcloud/l10n'
|
||||||
import { confirmPassword } from '@nextcloud/password-confirmation'
|
import { confirmPassword } from '@nextcloud/password-confirmation'
|
||||||
import NcButton from '@nextcloud/vue/components/NcButton'
|
import NcButton from '@nextcloud/vue/components/NcButton'
|
||||||
import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
|
import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
|
||||||
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
||||||
|
import NcProgressBar from '@nextcloud/vue/components/NcProgressBar'
|
||||||
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||||
import api from '../api.js'
|
import api from '../api.js'
|
||||||
import { formatDate, formatDateTime } from '../format.js'
|
import { formatDate, formatDateTime } from '../format.js'
|
||||||
@@ -56,21 +67,49 @@ const emit = defineEmits(['changed'])
|
|||||||
const RETENTION = 'retention policy'
|
const RETENTION = 'retention policy'
|
||||||
const retentionInfo = ref(false)
|
const retentionInfo = ref(false)
|
||||||
const trash = reactive({})
|
const trash = reactive({})
|
||||||
const loading = ref(true)
|
const reading = ref({ name: '', total: null, done: 0 }) // the location being read; null when done
|
||||||
const error = ref('')
|
const error = ref('')
|
||||||
const empty = computed(() => props.status.targets.every(target => !trash[target.id]?.length))
|
const empty = computed(() => props.status.targets.every(target => !trash[target.id]?.length))
|
||||||
|
|
||||||
|
let current = 0 // the newest load; an older one stops (another took over, or the tab was left)
|
||||||
|
onBeforeUnmount(() => { current++ })
|
||||||
|
|
||||||
|
async function loadTarget(target, load) {
|
||||||
|
reading.value = { name: target.name, total: null, done: 0 }
|
||||||
|
const items = []
|
||||||
|
for (;;) {
|
||||||
|
const page = await api.trash(target.id, items.length ? items[items.length - 1].id : '')
|
||||||
|
if (load !== current) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
items.push(...page.items)
|
||||||
|
reading.value = { name: target.name, total: page.total, done: Math.min(items.length, page.total) }
|
||||||
|
if (!page.items.length || items.length >= page.total) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
trash[target.id] = items.sort((a, b) => String(b.time ?? '').localeCompare(String(a.time ?? '')))
|
||||||
|
}
|
||||||
|
|
||||||
async function loadTrash() {
|
async function loadTrash() {
|
||||||
|
const load = ++current
|
||||||
|
const failed = []
|
||||||
for (const target of props.status.targets) {
|
for (const target of props.status.targets) {
|
||||||
try {
|
try {
|
||||||
trash[target.id] = await api.trash(target.id)
|
await loadTarget(target, load)
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
trash[target.id] = []
|
// keep what was shown: a location that cannot be reached does not mean its bin is empty
|
||||||
|
failed.push(target.name + (e?.message ? ': ' + e.message : ''))
|
||||||
|
}
|
||||||
|
if (load !== current) {
|
||||||
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
loading.value = false
|
error.value = failed.length ? t('ng_backup', 'The bin could not be read from {locations}', { locations: failed.join('; ') }) : ''
|
||||||
|
reading.value = null
|
||||||
}
|
}
|
||||||
watch(() => props.status.targets.map(tg => tg.id).join(','), loadTrash, { immediate: true })
|
// Again when a run ends: a cleanup moves restore points here.
|
||||||
|
watch(() => props.status.targets.map(tg => tg.id).join(',') + '|' + (props.status.runs ?? []).filter(run => run.status !== 'running').map(run => run.id).join(','), loadTrash, { immediate: true })
|
||||||
|
|
||||||
async function untrash(target, item) {
|
async function untrash(target, item) {
|
||||||
error.value = ''
|
error.value = ''
|
||||||
@@ -88,6 +127,8 @@ async function untrash(target, item) {
|
|||||||
<style scoped>
|
<style scoped>
|
||||||
.ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); }
|
.ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); }
|
||||||
.ngb-muted { color: var(--color-text-maxcontrast); }
|
.ngb-muted { color: var(--color-text-maxcontrast); }
|
||||||
|
.ngb-reading { margin: 12px 0; display: flex; flex-direction: column; gap: 6px; max-width: 600px; }
|
||||||
|
.ngb-reading p { margin: 0; }
|
||||||
.ngb-trash { margin: 12px 0; }
|
.ngb-trash { margin: 12px 0; }
|
||||||
.ngb-trash h3 { font-weight: bold; }
|
.ngb-trash h3 { font-weight: bold; }
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -0,0 +1,77 @@
|
|||||||
|
<!--
|
||||||
|
- SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
- SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
-->
|
||||||
|
<template>
|
||||||
|
<div class="ngb-row">
|
||||||
|
<strong class="ngb-name">{{ target.name }}</strong>
|
||||||
|
<NcSelect v-model="kind" :options="kinds" label="label" :input-label="t('ng_backup', 'Schedule')" :clearable="false" />
|
||||||
|
<NcTextField v-if="kind.id !== 'off'" v-model="time" class="ngb-time" :label="kind.id === 'daily' ? t('ng_backup', 'At (HH:MM, {zone})', { zone: browserZone }) : t('ng_backup', 'From (HH:MM, {zone})', { zone: browserZone })" />
|
||||||
|
<NcTextField v-if="kind.id === 'hours'" v-model="hours" class="ngb-hours" type="number" min="1" max="23" :label="t('ng_backup', 'Every … hours')" />
|
||||||
|
<NcButton :disabled="busy || wanted === current" @click="save">{{ t('ng_backup', 'Save') }}</NcButton>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup>
|
||||||
|
import { computed, ref, watch } from 'vue'
|
||||||
|
import { t } from '@nextcloud/l10n'
|
||||||
|
import NcButton from '@nextcloud/vue/components/NcButton'
|
||||||
|
import NcSelect from '@nextcloud/vue/components/NcSelect'
|
||||||
|
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
||||||
|
import api from '../api.js'
|
||||||
|
import { toZone } from '../format.js'
|
||||||
|
|
||||||
|
const props = defineProps({ target: { type: Object, required: true } })
|
||||||
|
const emit = defineEmits(['changed', 'error'])
|
||||||
|
|
||||||
|
const kinds = [
|
||||||
|
{ id: 'off', label: t('ng_backup', 'Off') },
|
||||||
|
{ id: 'daily', label: t('ng_backup', 'Once a day') },
|
||||||
|
{ id: 'hours', label: t('ng_backup', 'Every few hours') },
|
||||||
|
]
|
||||||
|
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
|
||||||
|
// zone is shown in this one, so it reads as the moment it runs here.
|
||||||
|
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
|
||||||
|
const kind = ref(kinds[0])
|
||||||
|
const time = ref('02:00')
|
||||||
|
const hours = ref('6')
|
||||||
|
const busy = ref(false)
|
||||||
|
|
||||||
|
const current = computed(() => {
|
||||||
|
const [at, every] = props.target.schedule.split('/')
|
||||||
|
return at === '' ? '' : (props.target.scheduleZone !== browserZone ? toZone(at, props.target.scheduleZone, browserZone) : at) + (every ? '/' + every : '')
|
||||||
|
})
|
||||||
|
const wanted = computed(() => {
|
||||||
|
if (kind.value.id === 'off') {
|
||||||
|
return ''
|
||||||
|
}
|
||||||
|
return time.value.trim() + (kind.value.id === 'hours' ? '/' + String(hours.value).trim() : '')
|
||||||
|
})
|
||||||
|
|
||||||
|
watch(current, (schedule) => {
|
||||||
|
const [at, every] = schedule.split('/')
|
||||||
|
kind.value = kinds[at === '' ? 0 : (every ? 2 : 1)]
|
||||||
|
time.value = at === '' ? time.value : at
|
||||||
|
hours.value = every ?? hours.value
|
||||||
|
}, { immediate: true })
|
||||||
|
|
||||||
|
async function save() {
|
||||||
|
busy.value = true
|
||||||
|
emit('error', '')
|
||||||
|
try {
|
||||||
|
await api.setTargetSchedule(props.target.id, wanted.value, browserZone)
|
||||||
|
emit('changed')
|
||||||
|
} catch (e) {
|
||||||
|
emit('error', e.message)
|
||||||
|
} finally {
|
||||||
|
busy.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
|
||||||
|
.ngb-name { min-width: 120px; align-self: center; }
|
||||||
|
.ngb-time { max-width: 220px; }
|
||||||
|
.ngb-hours { max-width: 120px; }
|
||||||
|
</style>
|
||||||
@@ -11,6 +11,13 @@
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<template v-if="snapshot">
|
<template v-if="snapshot">
|
||||||
|
<div class="ngb-row ngb-marks">
|
||||||
|
<NcTextField v-model="note" class="ngb-note" :maxlength="500" :label="t('ng_backup', 'Note (can be changed later)')" />
|
||||||
|
<NcButton :disabled="marking || note.trim() === (snapshot.note ?? '')" @click="saveNote">{{ t('ng_backup', 'Save note') }}</NcButton>
|
||||||
|
<NcCheckboxRadioSwitch :model-value="snapshot.pinned" :disabled="marking" @update:model-value="savePinned">
|
||||||
|
{{ t('ng_backup', 'Pin: the retention policy does not remove this restore point') }}
|
||||||
|
</NcCheckboxRadioSwitch>
|
||||||
|
</div>
|
||||||
<div class="ngb-crumbs">
|
<div class="ngb-crumbs">
|
||||||
<NcButton variant="tertiary" @click="open('data')">data</NcButton>
|
<NcButton variant="tertiary" @click="open('data')">data</NcButton>
|
||||||
<template v-for="(crumb, i) in crumbs" :key="crumb.path">
|
<template v-for="(crumb, i) in crumbs" :key="crumb.path">
|
||||||
@@ -49,6 +56,7 @@ import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
|
|||||||
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
||||||
import NcSelect from '@nextcloud/vue/components/NcSelect'
|
import NcSelect from '@nextcloud/vue/components/NcSelect'
|
||||||
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||||
|
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
||||||
import { confirmPassword } from '@nextcloud/password-confirmation'
|
import { confirmPassword } from '@nextcloud/password-confirmation'
|
||||||
import '@nextcloud/password-confirmation/style.css'
|
import '@nextcloud/password-confirmation/style.css'
|
||||||
import api from '../api.js'
|
import api from '../api.js'
|
||||||
@@ -74,7 +82,10 @@ const modes = [
|
|||||||
{ id: 'replace', label: t('ng_backup', 'Into the original place, overwrite and move files that are not in the backup to the trash') },
|
{ id: 'replace', label: t('ng_backup', 'Into the original place, overwrite and move files that are not in the backup to the trash') },
|
||||||
]
|
]
|
||||||
|
|
||||||
const snapshotLabel = (s) => formatDateTime(s.createdAt) + (s.label ? ' – ' + s.label : '')
|
const note = ref('')
|
||||||
|
const marking = ref(false)
|
||||||
|
|
||||||
|
const snapshotLabel = (s) => (s.pinned ? '📌 ' : '') + formatDateTime(s.createdAt) + (s.label ? ' – ' + s.label : '') + (s.note ? ' – ' + s.note : '')
|
||||||
const join = (name) => path.value + '/' + name
|
const join = (name) => path.value + '/' + name
|
||||||
const restorable = (p) => /^data\/[^/]+\/files(\/.*)?$/.test(p)
|
const restorable = (p) => /^data\/[^/]+\/files(\/.*)?$/.test(p)
|
||||||
const crumbs = computed(() => {
|
const crumbs = computed(() => {
|
||||||
@@ -88,7 +99,29 @@ watch(target, async (tg) => {
|
|||||||
snapshots.value = list.map(s => ({ ...s, displayLabel: snapshotLabel(s) }))
|
snapshots.value = list.map(s => ({ ...s, displayLabel: snapshotLabel(s) }))
|
||||||
snapshot.value = snapshots.value[0] ?? null
|
snapshot.value = snapshots.value[0] ?? null
|
||||||
}, { immediate: true })
|
}, { immediate: true })
|
||||||
watch(snapshot, () => { selected.value = ''; open('data') })
|
watch(snapshot, (s) => { selected.value = ''; note.value = s?.note ?? ''; open('data') })
|
||||||
|
|
||||||
|
// Changes the snapshot in place: choosing another one, which opens its folders again, is not needed
|
||||||
|
async function mark(call, change) {
|
||||||
|
const s = snapshot.value
|
||||||
|
marking.value = true
|
||||||
|
error.value = message.value = ''
|
||||||
|
try {
|
||||||
|
await call(target.value.id, s.id)
|
||||||
|
change(s)
|
||||||
|
s.displayLabel = snapshotLabel(s)
|
||||||
|
emit('changed')
|
||||||
|
} catch (e) {
|
||||||
|
error.value = e.message
|
||||||
|
} finally {
|
||||||
|
marking.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const saveNote = () => {
|
||||||
|
const text = note.value.trim()
|
||||||
|
return mark((targetId, id) => api.setSnapshotNote(targetId, id, text), (s) => { s.note = text })
|
||||||
|
}
|
||||||
|
const savePinned = (pinned) => mark((targetId, id) => api.setSnapshotPinned(targetId, id, pinned), (s) => { s.pinned = pinned })
|
||||||
|
|
||||||
async function open(p) {
|
async function open(p) {
|
||||||
if (!snapshot.value) {
|
if (!snapshot.value) {
|
||||||
@@ -125,6 +158,8 @@ async function restore() {
|
|||||||
<style scoped>
|
<style scoped>
|
||||||
.ngb-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 12px; }
|
.ngb-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 12px; }
|
||||||
.ngb-row :deep(.ngb-snapshot) { min-width: min(360px, 100%); }
|
.ngb-row :deep(.ngb-snapshot) { min-width: min(360px, 100%); }
|
||||||
|
.ngb-marks { align-items: flex-end; }
|
||||||
|
.ngb-note { max-width: 360px; }
|
||||||
.ngb-crumbs { display: flex; align-items: center; flex-wrap: wrap; }
|
.ngb-crumbs { display: flex; align-items: center; flex-wrap: wrap; }
|
||||||
.ngb-entries li { display: flex; align-items: center; gap: 12px; min-height: 36px; }
|
.ngb-entries li { display: flex; align-items: center; gap: 12px; min-height: 36px; }
|
||||||
.ngb-entries li.selected { background: var(--color-primary-element-light); border-radius: var(--border-radius); }
|
.ngb-entries li.selected { background: var(--color-primary-element-light); border-radius: var(--border-radius); }
|
||||||
|
|||||||
@@ -28,6 +28,28 @@ class TreeWalkerTest extends TestCase {
|
|||||||
$this->assertSame(['Z', 'a', 'a-b', 'a.txt', 'b', 'skip'], array_values(array_unique(array_map(fn ($p) => explode('/', $p)[0], $paths))));
|
$this->assertSame(['Z', 'a', 'a-b', 'a.txt', 'b', 'skip'], array_values(array_unique(array_map(fn ($p) => explode('/', $p)[0], $paths))));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testEmptyDirectoriesAreYieldedInOrderAndOnResume(): void {
|
||||||
|
$d = $this->tree();
|
||||||
|
mkdir("$d/a/empty");
|
||||||
|
mkdir("$d/a-hollow/inner", 0700, true);
|
||||||
|
symlink("$d/a.txt", "$d/skip/link");
|
||||||
|
mkdir("$d/only-link");
|
||||||
|
symlink("$d/a.txt", "$d/only-link/link");
|
||||||
|
$w = new TreeWalker(['' => $d]);
|
||||||
|
$all = array_keys(iterator_to_array($w->walk()));
|
||||||
|
$this->assertContains('a/empty', $all);
|
||||||
|
$this->assertContains('a-hollow/inner', $all);
|
||||||
|
$this->assertNotContains('a-hollow', $all, 'only the deepest');
|
||||||
|
$this->assertContains('only-link', $all, 'a link is not walked');
|
||||||
|
$this->assertNotContains('skip', $all);
|
||||||
|
$sorted = $all;
|
||||||
|
usort($sorted, [TreeWalker::class, 'compare']);
|
||||||
|
$this->assertSame($sorted, $all);
|
||||||
|
foreach ($all as $i => $cursor) {
|
||||||
|
$this->assertSame(array_slice($all, $i + 1), array_keys(iterator_to_array($w->walk($cursor))), "after $cursor");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public function testResumeAfterCursorYieldsTheRest(): void {
|
public function testResumeAfterCursorYieldsTheRest(): void {
|
||||||
$w = new TreeWalker(['' => $this->tree()]);
|
$w = new TreeWalker(['' => $this->tree()]);
|
||||||
$all = array_keys(iterator_to_array($w->walk()));
|
$all = array_keys(iterator_to_array($w->walk()));
|
||||||
|
|||||||
@@ -50,6 +50,40 @@ class CatalogTest extends TestCase {
|
|||||||
$this->assertSame(4, $again->catalogGeneration());
|
$this->assertSame(4, $again->catalogGeneration());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testTheTrashIsReadInPartsThatTogetherAreAll(): void {
|
||||||
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
|
for ($i = 0; $i < 3; $i++) {
|
||||||
|
$repo->forget($repo->backupDirectory($this->src)['snapshot']);
|
||||||
|
}
|
||||||
|
$again = $this->open();
|
||||||
|
$seen = [];
|
||||||
|
// A deadline already past: one entry per request, as on a slow location
|
||||||
|
$after = '';
|
||||||
|
for ($request = 0; count($seen) < 3 && $request < 5; $request++) {
|
||||||
|
$page = $again->trashPage($after, 0.0);
|
||||||
|
$this->assertSame(3, $page['total']);
|
||||||
|
$this->assertCount(1, $page['items']);
|
||||||
|
$seen += $page['items'];
|
||||||
|
$after = (string)array_key_last($page['items']);
|
||||||
|
}
|
||||||
|
$this->assertEqualsCanonicalizing(array_keys($again->trash()), array_keys($seen));
|
||||||
|
$this->assertSame(['total' => 3, 'items' => []], $again->trashPage($after, 0.0));
|
||||||
|
$this->assertCount(3, $again->trashPage('', microtime(true) + 60)['items']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAnEntryThatLeavesTheTrashBetweenPartsMakesNoneComeTwiceOrBeMissed(): void {
|
||||||
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
|
for ($i = 0; $i < 3; $i++) {
|
||||||
|
$repo->forget($repo->backupDirectory($this->src)['snapshot']);
|
||||||
|
}
|
||||||
|
$reader = $this->open();
|
||||||
|
$ids = array_keys($reader->trashPage('', microtime(true) + 60)['items']);
|
||||||
|
$first = $reader->trashPage('', 0.0)['items'];
|
||||||
|
$this->assertSame([$ids[0]], array_keys($first));
|
||||||
|
$this->open()->untrash($ids[0]); // another administrator, between two parts
|
||||||
|
$this->assertSame([$ids[1], $ids[2]], array_keys($this->open()->trashPage($ids[0], microtime(true) + 60)['items']));
|
||||||
|
}
|
||||||
|
|
||||||
public function testRollbackIsDetected(): void {
|
public function testRollbackIsDetected(): void {
|
||||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
$repo->backupDirectory($this->src);
|
$repo->backupDirectory($this->src);
|
||||||
|
|||||||
@@ -0,0 +1,158 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\LocalBackend;
|
||||||
|
use OCA\NgBackup\Crypto\KeyRing;
|
||||||
|
use OCA\NgBackup\Job\BackupRun;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
|
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** Folders without files are recorded in the snapshot's meta record and come back on a restore. */
|
||||||
|
class EmptyDirsTest extends TestCase {
|
||||||
|
use TempDirTrait;
|
||||||
|
|
||||||
|
private LocalBackend $backend;
|
||||||
|
private KeyRing $keys;
|
||||||
|
private string $src;
|
||||||
|
|
||||||
|
protected function setUp(): void {
|
||||||
|
$this->backend = new LocalBackend($this->tempDir());
|
||||||
|
$this->keys = KeyRing::generate();
|
||||||
|
Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
|
||||||
|
$this->src = $this->tempDir();
|
||||||
|
mkdir("$this->src/data/alice/files/Docs", 0700, true);
|
||||||
|
mkdir("$this->src/data/alice/files/Hollow/Nested", 0700, true);
|
||||||
|
mkdir("$this->src/data/skipped/inner", 0700, true);
|
||||||
|
file_put_contents("$this->src/data/alice/files/Docs/note.txt", "hello\n");
|
||||||
|
touch("$this->src/data/alice/files/Hollow/Nested", 1700000000);
|
||||||
|
}
|
||||||
|
|
||||||
|
private function repo(): Repository {
|
||||||
|
return Repository::openWithKey($this->backend, $this->keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @param list<string> $exclude */
|
||||||
|
private function backup(array $exclude = [], float $seconds = 30.0): string {
|
||||||
|
$state = BackupRun::start(['data' => "$this->src/data"], null, '', $exclude);
|
||||||
|
for ($i = 0; $i < 500 && $state['phase'] !== 'commit'; $i++) {
|
||||||
|
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, microtime(true) + $seconds)), true);
|
||||||
|
}
|
||||||
|
$this->assertSame('commit', $state['phase']);
|
||||||
|
$this->repo()->recordSnapshot($state['snapshot']);
|
||||||
|
return $state['snapshot'];
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOnlyTheDeepestEmptyFolderIsRecordedAndFilesStayAsTheyAre(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
|
||||||
|
$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], $paths);
|
||||||
|
$this->assertSame(['data/alice/files/Docs/note.txt'], array_column(iterator_to_array($this->repo()->entries($snapshot), false), 'p'));
|
||||||
|
$this->assertSame(2, $this->repo()->snapshotMeta($snapshot)['stats']['emptyDirs']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAFolderWithOnlyExcludedContentCountsAsEmpty(): void {
|
||||||
|
$snapshot = $this->backup(['data/skipped/inner']);
|
||||||
|
$this->assertContains('data/skipped', array_column($this->repo()->emptyDirs($snapshot), 0));
|
||||||
|
$this->assertNotContains('data/skipped/inner', array_column($this->repo()->emptyDirs($snapshot), 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAResumedWalkRecordsEveryFolderOnce(): void {
|
||||||
|
for ($i = 0; $i < 30; $i++) {
|
||||||
|
mkdir("$this->src/data/many/d$i", 0700, true);
|
||||||
|
file_put_contents("$this->src/data/many/f$i", 'x');
|
||||||
|
}
|
||||||
|
$snapshot = $this->backup([], 0.0); // a step per entry
|
||||||
|
$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
|
||||||
|
$this->assertCount(32, $paths);
|
||||||
|
$this->assertSame(array_unique($paths), $paths);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testANameASnapshotCannotHoldIsReportedAndTheSnapshotStaysReadable(): void {
|
||||||
|
mkdir("$this->src/data/odd\\folder", 0700, true);
|
||||||
|
mkdir("$this->src/data/alice/files/line\nbreak", 0700, true);
|
||||||
|
file_put_contents("$this->src/data/alice/files/back\\slash.txt", 'x');
|
||||||
|
file_put_contents("$this->src/data/alice/files/\xff.txt", 'x');
|
||||||
|
$state = BackupRun::start(['data' => "$this->src/data"]);
|
||||||
|
for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
|
||||||
|
// A step per entry, state through JSON: as separate processes would
|
||||||
|
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
|
||||||
|
}
|
||||||
|
$this->assertSame('commit', $state['phase']);
|
||||||
|
$this->assertSame(4, $state['stats']['badNames'], 'each reported once, over several steps');
|
||||||
|
$this->assertArrayNotHasKey('unreadable', $state['stats'], 'not a problem with permissions');
|
||||||
|
$shown = BackupRun::warnings($state)['badNames'];
|
||||||
|
$this->assertContains('data/alice/files/line?break', $shown, 'no control character in a message');
|
||||||
|
$this->assertSame($shown, array_filter($shown, fn (string $name): bool => mb_check_encoding($name, 'UTF-8')));
|
||||||
|
$this->repo()->recordSnapshot($state['snapshot']);
|
||||||
|
|
||||||
|
$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], array_column($this->repo()->emptyDirs($state['snapshot']), 0));
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->assertSame(1, $this->repo()->restore($state['snapshot'], $target));
|
||||||
|
$this->assertFileExists("$target/data/alice/files/Docs/note.txt");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAnEmptyFolderThatIsThePrefixItselfIsTheTarget(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target, 'data/skipped/inner/', true);
|
||||||
|
$this->assertSame([], glob("$target/*"), 'not a folder "inner" inside the target');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testRestoreBringsThemBackWithTheirTime(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->assertSame(1, $this->repo()->restore($snapshot, $target));
|
||||||
|
$this->assertDirectoryExists("$target/data/alice/files/Hollow/Nested");
|
||||||
|
clearstatcache();
|
||||||
|
$this->assertSame(1700000000, filemtime("$target/data/alice/files/Hollow/Nested"));
|
||||||
|
$this->assertDirectoryExists("$target/data/skipped/inner");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testRestoreOfAPrefixStripsAndLeavesOthersOut(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target, 'data/alice/', true);
|
||||||
|
$this->assertDirectoryExists("$target/files/Hollow/Nested");
|
||||||
|
$this->assertDirectoryDoesNotExist("$target/skipped");
|
||||||
|
$this->assertDirectoryDoesNotExist("$target/inner");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testRunningAgainLeavesAFolderThatIsThereAlone(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
touch("$target/data/alice/files/Hollow/Nested", 1800000000);
|
||||||
|
file_put_contents("$target/data/alice/files/Hollow/Nested/added", 'kept');
|
||||||
|
touch("$target/data/alice/files/Hollow/Nested", 1800000000);
|
||||||
|
$this->repo()->restore($snapshot, $target, keepUnchanged: true);
|
||||||
|
clearstatcache();
|
||||||
|
$this->assertSame(1800000000, filemtime("$target/data/alice/files/Hollow/Nested"));
|
||||||
|
$this->assertFileExists("$target/data/alice/files/Hollow/Nested/added");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testALinkOutOfTheTargetIsRefused(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$outside = $this->tempDir();
|
||||||
|
mkdir("$target/data/alice/files", 0700, true);
|
||||||
|
symlink($outside, "$target/data/alice/files/Hollow");
|
||||||
|
try {
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
$this->fail('restored through a link');
|
||||||
|
} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
|
||||||
|
$this->assertStringContainsString('outside', $e->getMessage());
|
||||||
|
}
|
||||||
|
$this->assertSame([], glob("$outside/*"));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testASnapshotWithoutThemHasNone(): void {
|
||||||
|
$src = $this->tempDir();
|
||||||
|
file_put_contents("$src/a", 'x');
|
||||||
|
$snapshot = $this->repo()->backupDirectory($src)['snapshot'];
|
||||||
|
$this->assertSame([], $this->repo()->emptyDirs($snapshot));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\LocalBackend;
|
||||||
|
use OCA\NgBackup\Crypto\KeyRing;
|
||||||
|
use OCA\NgBackup\Job\BackupRun;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
|
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** Folders with something in them get their original modification time back on a restore. */
|
||||||
|
class FolderTimesTest extends TestCase {
|
||||||
|
use TempDirTrait;
|
||||||
|
|
||||||
|
private const OLD = 1500000000;
|
||||||
|
private const OLDER = 1400000000;
|
||||||
|
|
||||||
|
private LocalBackend $backend;
|
||||||
|
private KeyRing $keys;
|
||||||
|
private string $src;
|
||||||
|
|
||||||
|
protected function setUp(): void {
|
||||||
|
$this->backend = new LocalBackend($this->tempDir());
|
||||||
|
$this->keys = KeyRing::generate();
|
||||||
|
Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
|
||||||
|
$this->src = $this->tempDir();
|
||||||
|
mkdir("$this->src/data/alice/files/Old/Deep", 0700, true);
|
||||||
|
mkdir("$this->src/data/alice/files/2026", 0700, true);
|
||||||
|
file_put_contents("$this->src/data/alice/files/Old/Deep/a.txt", 'a');
|
||||||
|
file_put_contents("$this->src/data/alice/files/Old/b.txt", 'b');
|
||||||
|
file_put_contents("$this->src/data/alice/files/2026/c.txt", 'c');
|
||||||
|
touch("$this->src/data/alice/files/Old/Deep", self::OLDER);
|
||||||
|
touch("$this->src/data/alice/files/Old", self::OLD);
|
||||||
|
touch("$this->src/data/alice/files/2026", self::OLD);
|
||||||
|
}
|
||||||
|
|
||||||
|
private function repo(): Repository {
|
||||||
|
return Repository::openWithKey($this->backend, $this->keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
private function backup(float $seconds = 30.0): string {
|
||||||
|
$state = BackupRun::start(['data' => "$this->src/data"]);
|
||||||
|
for ($i = 0; $i < 500 && $state['phase'] !== 'commit'; $i++) {
|
||||||
|
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, microtime(true) + $seconds)), true);
|
||||||
|
}
|
||||||
|
$this->assertSame('commit', $state['phase']);
|
||||||
|
$this->repo()->recordSnapshot($state['snapshot']);
|
||||||
|
return $state['snapshot'];
|
||||||
|
}
|
||||||
|
|
||||||
|
private function time(string $path): int {
|
||||||
|
clearstatcache();
|
||||||
|
return (int)filemtime($path);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testEachFolderIsRecordedOnceAlsoInSmallSteps(): void {
|
||||||
|
$expected = ['data', 'data/alice', 'data/alice/files', 'data/alice/files/2026', 'data/alice/files/Old', 'data/alice/files/Old/Deep'];
|
||||||
|
$this->assertSame($expected, array_column($this->repo()->folderTimes($this->backup()), 0));
|
||||||
|
$this->assertSame($expected, array_column($this->repo()->folderTimes($this->backup(0.0)), 0), 'a step per entry');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNoEmptyFolderIsAmongThem(): void {
|
||||||
|
mkdir("$this->src/data/alice/files/Hollow");
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$this->assertNotContains('data/alice/files/Hollow', array_column($this->repo()->folderTimes($snapshot), 0));
|
||||||
|
$this->assertContains('data/alice/files/Hollow', array_column($this->repo()->emptyDirs($snapshot), 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testARestoreBringsThemBack(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/Old"));
|
||||||
|
$this->assertSame(self::OLDER, $this->time("$target/data/alice/files/Old/Deep"));
|
||||||
|
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/2026"));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testARestoreThatGoesOnAfterAStopToo(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
touch("$target/data/alice/files/Old", 1800000000);
|
||||||
|
$this->repo()->restore($snapshot, $target, keepUnchanged: true);
|
||||||
|
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/Old"));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAPrefixLeavesTheTargetAndOthersAlone(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target, 'data/alice/files/Old/', true);
|
||||||
|
$this->assertSame(self::OLDER, $this->time("$target/Deep"));
|
||||||
|
$this->assertNotSame(self::OLD, $this->time($target), 'the directory that was chosen is not ours to change');
|
||||||
|
$this->assertDirectoryDoesNotExist("$target/2026");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testALinkOutOfTheTargetIsRefused(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$outside = $this->tempDir();
|
||||||
|
touch($outside, 1800000000);
|
||||||
|
mkdir("$target/data/alice/files", 0700, true);
|
||||||
|
symlink($outside, "$target/data/alice/files/Old");
|
||||||
|
$this->expectException(\OCA\NgBackup\Repository\RestoreTargetException::class);
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAFolderWithABadNameIsLeftOutAndWhatIsInItReported(): void {
|
||||||
|
mkdir("$this->src/data/odd\\folder");
|
||||||
|
file_put_contents("$this->src/data/odd\\folder/x", 'x');
|
||||||
|
$state = BackupRun::start(['data' => "$this->src/data"]);
|
||||||
|
for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
|
||||||
|
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
|
||||||
|
}
|
||||||
|
$this->assertSame('commit', $state['phase']);
|
||||||
|
$this->assertSame(1, $state['stats']['badNames']);
|
||||||
|
$this->repo()->recordSnapshot($state['snapshot']);
|
||||||
|
$this->assertNotContains("data/odd\\folder", array_column($this->repo()->folderTimes($state['snapshot']), 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testASnapshotWithoutThemHasNone(): void {
|
||||||
|
$snapshot = $this->repo()->backupDirectory($this->src)['snapshot'];
|
||||||
|
$this->assertSame([], $this->repo()->folderTimes($snapshot));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\LocalBackend;
|
||||||
|
use OCA\NgBackup\Command\Restore;
|
||||||
|
use OCA\NgBackup\Crypto\KeyRing;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
|
use OCA\NgBackup\Repository\RestoreTargetException;
|
||||||
|
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** A stream that takes at most 3 bytes per write. */
|
||||||
|
class PartialWriteStream {
|
||||||
|
public static string $written = '';
|
||||||
|
/** @var resource|null set by PHP */
|
||||||
|
public $context;
|
||||||
|
|
||||||
|
public function stream_open(string $path, string $mode, int $options, ?string &$opened): bool {
|
||||||
|
self::$written = '';
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function stream_write(string $data): int {
|
||||||
|
$part = substr($data, 0, 3);
|
||||||
|
self::$written .= $part;
|
||||||
|
return strlen($part);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What goes wrong where a restore writes to is not damage in the backup: it stops the restore, also
|
||||||
|
* with --salvage, and the message does not point at --salvage.
|
||||||
|
*/
|
||||||
|
class RestoreTargetTest extends TestCase {
|
||||||
|
use TempDirTrait;
|
||||||
|
|
||||||
|
private LocalBackend $backend;
|
||||||
|
private KeyRing $keys;
|
||||||
|
private string $snapshot;
|
||||||
|
|
||||||
|
protected function setUp(): void {
|
||||||
|
$this->backend = new LocalBackend($this->tempDir());
|
||||||
|
$this->keys = KeyRing::generate();
|
||||||
|
$src = $this->tempDir();
|
||||||
|
mkdir("$src/dir");
|
||||||
|
file_put_contents("$src/dir/a.bin", random_bytes(6 * 1048576));
|
||||||
|
file_put_contents("$src/dir/b.bin", random_bytes(6 * 1048576));
|
||||||
|
file_put_contents("$src/dir/c.txt", 'small');
|
||||||
|
$this->snapshot = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'))->backupDirectory($src)['snapshot'];
|
||||||
|
}
|
||||||
|
|
||||||
|
private function repo(): Repository {
|
||||||
|
return Repository::openWithKey($this->backend, $this->keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAShortWriteIsATargetError(): void {
|
||||||
|
if (!is_writable('/dev/full')) {
|
||||||
|
$this->markTestSkipped('needs /dev/full');
|
||||||
|
}
|
||||||
|
$out = fopen('/dev/full', 'wb');
|
||||||
|
stream_set_write_buffer($out, 0);
|
||||||
|
$this->expectException(RestoreTargetException::class);
|
||||||
|
Repository::writeAll($out, 'data', 'dir/a.bin');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAWriteThatTakesPartOfTheDataGoesOn(): void {
|
||||||
|
stream_wrapper_register('ngbpartial', PartialWriteStream::class);
|
||||||
|
try {
|
||||||
|
$out = fopen('ngbpartial://x', 'wb');
|
||||||
|
Repository::writeAll($out, 'abcdefgh', 'dir/a.bin');
|
||||||
|
fclose($out);
|
||||||
|
$this->assertSame('abcdefgh', PartialWriteStream::$written);
|
||||||
|
} finally {
|
||||||
|
stream_wrapper_unregister('ngbpartial');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testATargetErrorDoesNotLookLikeDamage(): void {
|
||||||
|
$this->assertFalse(Restore::backupLooksDamaged(new RestoreTargetException('Cannot write')));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A real full disk: run with NGB_TEST_SMALL_DISK set to an empty, writable tmpfs of a few MiB
|
||||||
|
* (docker run --tmpfs /small:size=8m,uid=...), with and without --salvage.
|
||||||
|
*
|
||||||
|
* @dataProvider salvageOrNot
|
||||||
|
*/
|
||||||
|
public function testAFullDiskStopsTheRestoreAndLeavesNoTemporary(bool $salvage): void {
|
||||||
|
$disk = getenv('NGB_TEST_SMALL_DISK');
|
||||||
|
if ($disk === false || $disk === '') {
|
||||||
|
$this->markTestSkipped('needs NGB_TEST_SMALL_DISK');
|
||||||
|
}
|
||||||
|
$target = $disk . '/' . bin2hex(random_bytes(4));
|
||||||
|
$skipped = $salvage ? [] : null;
|
||||||
|
try {
|
||||||
|
$this->repo()->restore($this->snapshot, $target, skipped: $skipped);
|
||||||
|
$this->fail('12 MiB fitted on the small disk');
|
||||||
|
} catch (RestoreTargetException $e) {
|
||||||
|
$this->assertStringContainsString('disk full', $e->getMessage());
|
||||||
|
} finally {
|
||||||
|
$left = glob("$target/dir/*.ngb-restore-*");
|
||||||
|
foreach (glob("$target/dir/*") ?: [] as $f) {
|
||||||
|
unlink($f);
|
||||||
|
}
|
||||||
|
@rmdir("$target/dir");
|
||||||
|
@rmdir($target);
|
||||||
|
}
|
||||||
|
$this->assertSame([], $left);
|
||||||
|
if ($salvage) {
|
||||||
|
$this->assertSame([], $skipped, 'nothing is listed as damaged');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @return array<string, array{bool}> */
|
||||||
|
public static function salvageOrNot(): array {
|
||||||
|
return ['without --salvage' => [false], 'with --salvage' => [true]];
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testResumeChecksTheFolderBeforeSkipping(): void {
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($this->snapshot, $target);
|
||||||
|
$outside = $this->tempDir();
|
||||||
|
rename("$target/dir", "$outside/dir");
|
||||||
|
symlink("$outside/dir", "$target/dir");
|
||||||
|
$this->expectException(RestoreTargetException::class);
|
||||||
|
$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testResumeRemovesTemporariesWhereEverythingIsInPlace(): void {
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($this->snapshot, $target);
|
||||||
|
file_put_contents("$target/dir/a.bin.ngb-restore-0123456789ab", 'left by a stopped restore');
|
||||||
|
$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
|
||||||
|
$this->assertFileDoesNotExist("$target/dir/a.bin.ngb-restore-0123456789ab");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testHeartbeatRunsForFilesThatAreSkipped(): void {
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($this->snapshot, $target);
|
||||||
|
$beats = 0;
|
||||||
|
$this->repo()->restore($this->snapshot, $target, '', false, function () use (&$beats): void {
|
||||||
|
$beats++;
|
||||||
|
}, keepUnchanged: true);
|
||||||
|
$this->assertSame(3, $beats);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\LocalBackend;
|
||||||
|
use OCA\NgBackup\Crypto\KeyRing;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
|
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** A pin and a note on a snapshot live in the catalog on the location, so a new server sees them too. */
|
||||||
|
class SnapshotMarksTest extends TestCase {
|
||||||
|
use TempDirTrait;
|
||||||
|
|
||||||
|
private LocalBackend $backend;
|
||||||
|
private KeyRing $keys;
|
||||||
|
private string $snapshot;
|
||||||
|
|
||||||
|
protected function setUp(): void {
|
||||||
|
$this->backend = new LocalBackend($this->tempDir());
|
||||||
|
$this->keys = KeyRing::generate();
|
||||||
|
$src = $this->tempDir();
|
||||||
|
file_put_contents("$src/a", 'x');
|
||||||
|
$this->snapshot = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'))->backupDirectory($src)['snapshot'];
|
||||||
|
}
|
||||||
|
|
||||||
|
private function repo(): Repository {
|
||||||
|
return Repository::openWithKey($this->backend, $this->keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testTheyAreThereAfterOpeningAgain(): void {
|
||||||
|
$this->repo()->setPinned($this->snapshot, true);
|
||||||
|
$this->repo()->setSnapshotNote($this->snapshot, 'before the upgrade');
|
||||||
|
$this->assertSame([$this->snapshot], $this->repo()->pinnedSnapshots());
|
||||||
|
$this->assertSame([$this->snapshot => 'before the upgrade'], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testTheyCanBeTakenOffAgain(): void {
|
||||||
|
$this->repo()->setPinned($this->snapshot, true);
|
||||||
|
$this->repo()->setSnapshotNote($this->snapshot, 'x');
|
||||||
|
$this->repo()->setPinned($this->snapshot, false);
|
||||||
|
$this->repo()->setSnapshotNote($this->snapshot, '');
|
||||||
|
$this->assertSame([], $this->repo()->pinnedSnapshots());
|
||||||
|
$this->assertSame([], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNoChangeWritesNoCatalog(): void {
|
||||||
|
$repo = $this->repo();
|
||||||
|
$repo->setPinned($this->snapshot, true);
|
||||||
|
$generation = $repo->catalogGeneration();
|
||||||
|
$repo->setPinned($this->snapshot, true);
|
||||||
|
$repo->setSnapshotNote($this->snapshot, '');
|
||||||
|
$this->assertSame($generation, $repo->catalogGeneration());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testASnapshotThatIsNotThereIsRefused(): void {
|
||||||
|
$this->expectException(\InvalidArgumentException::class);
|
||||||
|
$this->repo()->setPinned('0123456789abcdef0123456789abcdef', true);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testATrashedSnapshotLosesItsPinAndKeepsItsNote(): void {
|
||||||
|
$repo = $this->repo();
|
||||||
|
$repo->setPinned($this->snapshot, true);
|
||||||
|
$repo->setSnapshotNote($this->snapshot, 'kept');
|
||||||
|
$repo->forget($this->snapshot);
|
||||||
|
$this->assertSame([], $this->repo()->pinnedSnapshots());
|
||||||
|
$this->assertSame([$this->snapshot => 'kept'], $this->repo()->snapshotNotes());
|
||||||
|
|
||||||
|
$repo = $this->repo();
|
||||||
|
$repo->untrash($this->snapshot);
|
||||||
|
$this->assertSame([], $this->repo()->pinnedSnapshots(), 'back in the list, as a normal one');
|
||||||
|
$this->assertSame([$this->snapshot => 'kept'], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testThePurgeTakesTheNoteAlong(): void {
|
||||||
|
$repo = $this->repo();
|
||||||
|
$repo->setSnapshotNote($this->snapshot, 'kept');
|
||||||
|
$repo->forget($this->snapshot);
|
||||||
|
$repo->purgeTrash(0);
|
||||||
|
$this->assertSame([], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testACatalogFromBeforeThisVersionReadsAsNone(): void {
|
||||||
|
$this->assertSame([], $this->repo()->pinnedSnapshots());
|
||||||
|
$this->assertSame([], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Service;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\LocationSchedule as S;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
class LocationScheduleTest extends TestCase {
|
||||||
|
private const BERLIN = 'Europe/Berlin';
|
||||||
|
|
||||||
|
private static function at(string $time): int {
|
||||||
|
return (int)strtotime($time);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testWhatCanBeSaved(): void {
|
||||||
|
$this->assertSame('', S::normalize('off'));
|
||||||
|
$this->assertSame('', S::normalize(' '));
|
||||||
|
$this->assertSame('02:30', S::normalize('02:30'));
|
||||||
|
$this->assertSame('02:30', S::normalize('02:30/24'));
|
||||||
|
$this->assertSame('00:00/6', S::normalize('00:00/6'));
|
||||||
|
foreach (['2:30', '24:00', '02:30/0', '02:30/25', 'daily', '02:30/x'] as $bad) {
|
||||||
|
try {
|
||||||
|
S::normalize($bad);
|
||||||
|
$this->fail("$bad accepted");
|
||||||
|
} catch (\InvalidArgumentException) {
|
||||||
|
$this->addToAssertionCount(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOffHasNoMoment(): void {
|
||||||
|
$this->assertNull(S::lastMoment('', self::BERLIN, self::at('2026-11-10 12:00 UTC')));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testDailyIsInTheZoneOfTheLocation(): void {
|
||||||
|
$this->assertSame(self::at('2026-11-10 02:00 UTC'), S::lastMoment('03:00', self::BERLIN, self::at('2026-11-10 12:00 UTC')));
|
||||||
|
$this->assertSame(self::at('2026-11-09 02:00 UTC'), S::lastMoment('03:00', self::BERLIN, self::at('2026-11-10 01:00 UTC')), 'not yet today');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testEveryNHoursStartsOverTheNextDay(): void {
|
||||||
|
$zone = 'UTC';
|
||||||
|
$this->assertSame(self::at('2026-11-10 12:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 13:00 UTC')));
|
||||||
|
$this->assertSame(self::at('2026-11-09 22:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 01:59 UTC')), '02, 07, 12, 17, 22, then 02 again');
|
||||||
|
$this->assertSame(self::at('2026-11-10 02:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 02:00 UTC')), 'at the moment itself');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAClockChangeInTheNightStillGivesOneMoment(): void {
|
||||||
|
$this->assertSame(self::at('2026-03-29 01:30 UTC'), S::lastMoment('02:30', self::BERLIN, self::at('2026-03-29 12:00 UTC')), '02:30 does not exist that night');
|
||||||
|
$this->assertNotNull(S::lastMoment('02:30', self::BERLIN, self::at('2026-10-25 12:00 UTC')));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testChoosingATimeDoesNotStartABackup(): void {
|
||||||
|
$moment = self::at('2026-11-10 02:00 UTC');
|
||||||
|
$this->assertFalse(S::isDue($moment, $moment + 1, null, 0, null, $moment + 3600));
|
||||||
|
$this->assertTrue(S::isDue($moment, $moment - 1, null, 0, null, $moment + 3600));
|
||||||
|
$this->assertTrue(S::isDue($moment, 0, null, 0, null, $moment), 'a location from before schedules per location');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testABackupSinceTheMomentIsEnough(): void {
|
||||||
|
$moment = 1000000;
|
||||||
|
$this->assertFalse(S::isDue($moment, 0, $moment, 1, $moment + 10, $moment + 20));
|
||||||
|
$this->assertTrue(S::isDue($moment, 0, $moment - 1, 0, null, $moment + 20), 'it started before this moment: one more');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAfterAFailureOnlyAFewAttemptsAnHourApart(): void {
|
||||||
|
$moment = 1000000;
|
||||||
|
$finished = $moment + 60;
|
||||||
|
$this->assertFalse(S::isDue($moment, 0, null, 1, $finished, $finished + S::RETRY_AFTER - 1));
|
||||||
|
$this->assertTrue(S::isDue($moment, 0, null, 1, $finished, $finished + S::RETRY_AFTER));
|
||||||
|
$this->assertFalse(S::isDue($moment, 0, null, S::MAX_ATTEMPTS, $finished, $finished + 86400));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -25,6 +25,17 @@ class RetentionPolicyTest extends TestCase {
|
|||||||
$this->assertCount(3, $months);
|
$this->assertCount(3, $months);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testAPinnedSnapshotStaysAndDoesNotCount(): void {
|
||||||
|
$policy = new RetentionPolicy(1, 0, 0, 0);
|
||||||
|
$snaps = ['old' => 100, 'pinned' => 300, 'new' => 200];
|
||||||
|
$this->assertSame(['pinned'], array_keys($policy->keep($snaps)), 'without the pin it would take the place of "last"');
|
||||||
|
$keep = $policy->keepWithPinned($snaps, ['pinned', 'gone-already']);
|
||||||
|
$this->assertSame(['pinned'], $keep['pinned']);
|
||||||
|
$this->assertSame(['last'], $keep['new'], 'the newest one that is not pinned');
|
||||||
|
$this->assertArrayNotHasKey('old', $keep);
|
||||||
|
$this->assertArrayNotHasKey('gone-already', $keep);
|
||||||
|
}
|
||||||
|
|
||||||
public function testKeepsEverythingWhenFewSnapshots(): void {
|
public function testKeepsEverythingWhenFewSnapshots(): void {
|
||||||
$keep = (new RetentionPolicy())->keep(['a' => 100, 'b' => 200]);
|
$keep = (new RetentionPolicy())->keep(['a' => 100, 'b' => 200]);
|
||||||
$this->assertSame(['a', 'b'], array_values(array_intersect(['a', 'b'], array_keys($keep))));
|
$this->assertSame(['a', 'b'], array_values(array_intersect(['a', 'b'], array_keys($keep))));
|
||||||
|
|||||||
@@ -0,0 +1,22 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Service;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\AlertService;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** A location without a schedule: a weekly reminder while another is backed up, else a daily one. */
|
||||||
|
class ScheduleOffReminderTest extends TestCase {
|
||||||
|
public function testOnceAWeekWhileAnotherLocationIsActive(): void {
|
||||||
|
$this->assertFalse(AlertService::offReminderDue(0, 6 * 86400, true));
|
||||||
|
$this->assertTrue(AlertService::offReminderDue(0, 7 * 86400, true));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testEveryDayWhenNoLocationIsActive(): void {
|
||||||
|
$this->assertFalse(AlertService::offReminderDue(0, 3600, false));
|
||||||
|
$this->assertTrue(AlertService::offReminderDue(0, 86400, false));
|
||||||
|
$this->assertTrue(AlertService::offReminderDue(0, 86400 - 60, false), 'a cron run a minute earlier than yesterday');
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -57,15 +57,4 @@ class ScheduleTest extends TestCase {
|
|||||||
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
|
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
|
||||||
}
|
}
|
||||||
|
|
||||||
public function testTodayAtIsTheMomentInTheSchedulesZone(): void {
|
|
||||||
$this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin'];
|
|
||||||
$at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC'));
|
|
||||||
$this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC');
|
|
||||||
}
|
|
||||||
|
|
||||||
public function testAnUnknownZoneOrABadTimeIsRefused(): void {
|
|
||||||
$s = $this->schedule();
|
|
||||||
$this->expectException(\InvalidArgumentException::class);
|
|
||||||
$s->set('03:00', 'Mars/Olympus');
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -102,7 +102,7 @@ ok "key, kit and location on the origin"
|
|||||||
|
|
||||||
NC_PASS=$(openssl rand -hex 16)
|
NC_PASS=$(openssl rand -hex 16)
|
||||||
docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
|
docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
|
||||||
docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
|
docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs /var/www/html/data/frtest/files/Empty/Nested && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
|
||||||
if [ "$MODE" = sftp ]; then
|
if [ "$MODE" = sftp ]; then
|
||||||
# about three packs of 32 MiB
|
# about three packs of 32 MiB
|
||||||
docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
|
docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
|
||||||
@@ -214,6 +214,7 @@ if [ "$MODE" = salvage ]; then
|
|||||||
printf '%s\n' "$OUT" | grep "$SALVAGE" >/dev/null || { printf '%s\n' "$OUT" | tail -30; fail "the output does not name the salvage folder"; }
|
printf '%s\n' "$OUT" | grep "$SALVAGE" >/dev/null || { printf '%s\n' "$OUT" | tail -30; fail "the output does not name the salvage folder"; }
|
||||||
[ "$(docker exec ngb-frt-restore cat "$SALVAGE/data/frtest/files/Docs/note.txt")" = "hello disaster recovery" ] || fail "the user's file is not in the salvage folder"
|
[ "$(docker exec ngb-frt-restore cat "$SALVAGE/data/frtest/files/Docs/note.txt")" = "hello disaster recovery" ] || fail "the user's file is not in the salvage folder"
|
||||||
docker exec ngb-frt-restore test -f "$SALVAGE/config/config.php" || fail "config.php is not in the salvage folder"
|
docker exec ngb-frt-restore test -f "$SALVAGE/config/config.php" || fail "config.php is not in the salvage folder"
|
||||||
|
docker exec ngb-frt-restore test -d "$SALVAGE/data/frtest/files/Empty/Nested" || fail "the empty folder is not in the salvage folder"
|
||||||
run_on ngb-frt-restore user:list | grep frtest >/dev/null && fail "the fresh database was changed (the backed-up user is there)"
|
run_on ngb-frt-restore user:list | grep frtest >/dev/null && fail "the fresh database was changed (the backed-up user is there)"
|
||||||
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
|
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
|
||||||
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
|
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
|
||||||
@@ -254,8 +255,8 @@ if [ "$MODE" = salvage-files ]; then
|
|||||||
if (!file_exists($root . $p)) { $bad[] = "not listed and not there: $p"; continue; }
|
if (!file_exists($root . $p)) { $bad[] = "not listed and not there: $p"; continue; }
|
||||||
hash_file("sha256", $root . $p) === $sum ? $same++ : $bad[] = "different: $p";
|
hash_file("sha256", $root . $p) === $sum ? $same++ : $bad[] = "different: $p";
|
||||||
}
|
}
|
||||||
$temps = glob($root . "*/*.ngb-restore-*") ?: [];
|
$walk = new RecursiveIteratorIterator(new RecursiveDirectoryIterator("/var/www/html/data", FilesystemIterator::SKIP_DOTS), RecursiveIteratorIterator::SELF_FIRST);
|
||||||
foreach ($temps as $t) { $bad[] = "temporary left: $t"; }
|
foreach ($walk as $f) { if (str_contains($f->getFilename(), ".ngb-restore-")) { $bad[] = "temporary left: " . $f->getPathname(); } }
|
||||||
echo "same=$same gone=$gone bad=", count($bad), "\n", implode("\n", array_slice($bad, 0, 10)), "\n";
|
echo "same=$same gone=$gone bad=", count($bad), "\n", implode("\n", array_slice($bad, 0, 10)), "\n";
|
||||||
echo "reasons: ", implode(" | ", array_unique(array_map(fn ($r) => substr(preg_replace("#[0-9a-f]{16,}#", "<id>", $r), 0, 70), $reasons))), "\n";')
|
echo "reasons: ", implode(" | ", array_unique(array_map(fn ($r) => substr(preg_replace("#[0-9a-f]{16,}#", "<id>", $r), 0, 70), $reasons))), "\n";')
|
||||||
printf '%s\n' "$CHECK"
|
printf '%s\n' "$CHECK"
|
||||||
@@ -265,6 +266,8 @@ if [ "$MODE" = salvage-files ]; then
|
|||||||
ok "with --salvage: every file listed and absent, or there and identical"
|
ok "with --salvage: every file listed and absent, or there and identical"
|
||||||
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
|
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
|
||||||
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing: the database was not restored"
|
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing: the database was not restored"
|
||||||
|
run_on ngb-frt-restore user:list | grep admin >/dev/null || fail "the admin is missing: the database was not restored"
|
||||||
|
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back with --salvage"
|
||||||
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt 2>/dev/null)" = "hello disaster recovery" ] \
|
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt 2>/dev/null)" = "hello disaster recovery" ] \
|
||||||
|| printf '%s\n' "$OUT" | grep "Docs/note.txt" >/dev/null || fail "note.txt is neither restored nor listed"
|
|| printf '%s\n' "$OUT" | grep "Docs/note.txt" >/dev/null || fail "note.txt is neither restored nor listed"
|
||||||
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
|
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
|
||||||
@@ -323,7 +326,16 @@ run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails aft
|
|||||||
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database"
|
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database"
|
||||||
docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
|
docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
|
||||||
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
|
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
|
||||||
|
if [ "$LEGACY" = legacy ]; then
|
||||||
|
# Backups made before 1.0.3 do not record empty folders
|
||||||
|
docker exec ngb-frt-restore test -e /var/www/html/data/frtest/files/Empty/Nested && fail "an empty folder came back from a backup that does not record them"
|
||||||
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
||||||
|
else
|
||||||
|
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back"
|
||||||
|
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
||||||
|
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "files:scan removed the empty folder"
|
||||||
|
ok "the empty folder came back and stays after files:scan"
|
||||||
|
fi
|
||||||
# New rows after the restore: a new file (file cache) and a new user must not collide with
|
# New rows after the restore: a new file (file cache) and a new user must not collide with
|
||||||
# restored rows (ID counters after the data).
|
# restored rows (ID counters after the data).
|
||||||
docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"
|
docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"
|
||||||
|
|||||||
@@ -23,12 +23,15 @@ ok "key, kit code and confirmation"
|
|||||||
occ backup:target:add ci $TARGET_ARGS --path=repo >/dev/null || fail "target add"
|
occ backup:target:add ci $TARGET_ARGS --path=repo >/dev/null || fail "target add"
|
||||||
ok "location added"
|
ok "location added"
|
||||||
|
|
||||||
# Test data: a user with files.
|
# Test data: a user with files. No skeleton files on the first login, below, in the middle of the checks.
|
||||||
|
occ config:system:set skeletondirectory --value='' >/dev/null
|
||||||
|
occ config:system:set templatedirectory --value='' >/dev/null
|
||||||
$X sh -c "OC_PASS='Roundtrip-pw-2026' php occ user:add --password-from-env $U >/dev/null"
|
$X sh -c "OC_PASS='Roundtrip-pw-2026' php occ user:add --password-from-env $U >/dev/null"
|
||||||
# large.bin (40 MB) fills more than one pack: a restore to a directory reads ahead over several ranges
|
# large.bin (40 MB) fills more than one pack: a restore to a directory reads ahead over several ranges
|
||||||
$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin"
|
$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs/Empty/Nested $DATA/$U/files/Hollow && touch -d @1700000000 $DATA/$U/files/Hollow && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin"
|
||||||
|
$X sh -c "touch -d @1600000000 $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs"
|
||||||
occ files:scan "$U" >/dev/null
|
occ files:scan "$U" >/dev/null
|
||||||
SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
|
||||||
|
|
||||||
occ backup:run ci -l first >/dev/null || fail "first backup"
|
occ backup:run ci -l first >/dev/null || fail "first backup"
|
||||||
SECOND=$(occ backup:run ci -l second)
|
SECOND=$(occ backup:run ci -l second)
|
||||||
@@ -39,14 +42,29 @@ SNAP=$(occ backup:list ci | awk -F'|' '!f && / second /{gsub(/ /,"",$3); print $
|
|||||||
[ -n "$SNAP" ] || fail "snapshot not listed"
|
[ -n "$SNAP" ] || fail "snapshot not listed"
|
||||||
|
|
||||||
# Damage, then restore in each mode.
|
# Damage, then restore in each mode.
|
||||||
$X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt"
|
$X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt && rm -r Docs/Empty && rmdir Hollow"
|
||||||
occ files:scan "$U" >/dev/null
|
occ files:scan "$U" >/dev/null
|
||||||
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m new-folder >/dev/null || fail "restore new-folder"
|
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m new-folder >/dev/null || fail "restore new-folder"
|
||||||
$X sh -c "ls $DATA/$U/files | grep -q '^Restored '" || fail "no Restored folder"
|
$X sh -c "ls $DATA/$U/files | grep -q '^Restored '" || fail "no Restored folder"
|
||||||
|
$X sh -c "test -d $DATA/$U/files/Restored\ */Docs/Empty/Nested" || fail "the empty folder is not in the Restored folder"
|
||||||
|
# Folders with files: on disk and in what Nextcloud shows, the parent as well as the child.
|
||||||
|
RDIR=$($X sh -c "cd $DATA/$U/files && ls -d Restored\ *")
|
||||||
|
[ "$($X sh -c "stat -c %Y $DATA/$U/files/Restored\ */Docs/Sub")" = 1600000000 ] || fail "a folder with files did not get its time back"
|
||||||
|
[ "$($X sh -c "stat -c %Y $DATA/$U/files/Restored\ */Docs")" = 1600000000 ] || fail "the parent folder did not get its time back"
|
||||||
|
DAV=$($X curl -sf -u "$U:Roundtrip-pw-2026" -X PROPFIND -H 'Depth: 1' "http://localhost/remote.php/dav/files/$U/$(php -r 'echo rawurlencode($argv[1]);' "$RDIR")/Docs/") || fail "PROPFIND"
|
||||||
|
[ "$(echo "$DAV" | grep -o 'Sun, 13 Sep 2020 12:26:40 GMT' | wc -l)" = 2 ] || fail "Nextcloud does not show the time of the folders: $(echo "$DAV" | grep -o '<d:getlastmodified>[^<]*' | sort | uniq -c)"
|
||||||
|
occ backup:restore ci "$SNAP" "data/$U/files/Docs/Sub" -m merge >/dev/null || fail "restore into a folder that is there"
|
||||||
|
[ "$($X stat -c %Y "$DATA/$U/files/Docs/Sub")" != 1600000000 ] || fail "a folder that was already there got the old time"
|
||||||
|
occ backup:browse ci "$SNAP" "data/$U/files" | grep Hollow >/dev/null || fail "an empty folder is not shown when browsing the snapshot"
|
||||||
|
occ backup:restore ci "$SNAP" "data/$U/files/Hollow" -m merge >/dev/null || fail "restore of a folder that was empty"
|
||||||
|
$X sh -c "test -d $DATA/$U/files/Hollow" || fail "the empty folder did not come back by itself"
|
||||||
|
[ "$($X stat -c %Y "$DATA/$U/files/Hollow")" = 1700000000 ] || fail "the empty folder did not get its time back"
|
||||||
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m replace >/dev/null || fail "restore replace"
|
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m replace >/dev/null || fail "restore replace"
|
||||||
|
$X sh -c "test -d $DATA/$U/files/Docs/Empty/Nested" || fail "replace did not bring back the empty folder, or moved it to the trash"
|
||||||
|
ok "empty folders: restored in each mode, by themselves and shown when browsing"
|
||||||
$X sh -c "rm -rf '$DATA/$U/files/Restored '*" ; occ files:scan "$U" >/dev/null
|
$X sh -c "rm -rf '$DATA/$U/files/Restored '*" ; occ files:scan "$U" >/dev/null
|
||||||
SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
|
||||||
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore"
|
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore: $(diff <(echo "$SUM_BEFORE") <(echo "$SUM_AFTER") | cut -c1-120)"
|
||||||
$X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash"
|
$X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash"
|
||||||
ok "restore new-folder and replace: files identical, extra file in the trash"
|
ok "restore new-folder and replace: files identical, extra file in the trash"
|
||||||
|
|
||||||
@@ -60,9 +78,10 @@ ok "restore of one file to a directory; the temporary of a stopped restore remov
|
|||||||
|
|
||||||
# All of the user's files to a directory: over S3 and SFTP the packs are read ahead
|
# All of the user's files to a directory: over S3 and SFTP the packs are read ahead
|
||||||
occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-dir >/dev/null || fail "restore of a folder to a directory"
|
occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-dir >/dev/null || fail "restore of a folder to a directory"
|
||||||
SUM_DIR=$($X sh -c "cd /tmp/ngb-rt-dir/data/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
SUM_DIR=$($X sh -c "cd /tmp/ngb-rt-dir/data/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
|
||||||
[ "$SUM_BEFORE" = "$SUM_DIR" ] || fail "the files restored to a directory differ"
|
[ "$SUM_BEFORE" = "$SUM_DIR" ] || fail "the files restored to a directory differ"
|
||||||
$X sh -c "ls -A /tmp/ngb-rt-dir | grep -q ngb-restore-spool" && fail "the read-ahead spool was left in the directory"
|
$X sh -c "ls -A /tmp/ngb-rt-dir | grep -q ngb-restore-spool" && fail "the read-ahead spool was left in the directory"
|
||||||
|
$X sh -c "test -d /tmp/ngb-rt-dir/data/$U/files/Docs/Empty/Nested" || fail "the empty folder is not in the directory"
|
||||||
$X sh -c "rm -rf /tmp/ngb-rt-dir"
|
$X sh -c "rm -rf /tmp/ngb-rt-dir"
|
||||||
ok "restore of all the user's files to a directory: identical, no spool left"
|
ok "restore of all the user's files to a directory: identical, no spool left"
|
||||||
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
|
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
|
||||||
@@ -90,13 +109,74 @@ done
|
|||||||
[ "$RS" = done ] || fail "the backup started through the API ended as '$RS' after $i ticks"
|
[ "$RS" = done ] || fail "the backup started through the API ended as '$RS' after $i ticks"
|
||||||
occ backup:list ci | grep " page " >/dev/null || fail "the snapshot of the ticked backup is not listed"
|
occ backup:list ci | grep " page " >/dev/null || fail "the snapshot of the ticked backup is not listed"
|
||||||
ok "without system cron: warning in backup:status and the API, backup finished by $i ticks of the open page"
|
ok "without system cron: warning in backup:status and the API, backup finished by $i ticks of the open page"
|
||||||
|
# The Bin reads in parts, as the page does (what is in the parts: CatalogTest)
|
||||||
|
api "$API/targets/$TID/trash" | php -r '$p = json_decode(stream_get_contents(STDIN), true); exit(is_int($p["total"] ?? null) && is_array($p["items"] ?? null) && count($p["items"]) <= $p["total"] ? 0 : 1);' || fail "the Bin API does not give a total and items"
|
||||||
|
ok "the Bin API reads in parts, with a total"
|
||||||
|
# Locations are backed up one after the other: the second one started is queued until the first is done.
|
||||||
|
for L in 2 3; do
|
||||||
|
$X sh -c "mkdir -p /tmp/ngb-rt-target$L"
|
||||||
|
occ backup:target:add ci$L local -a null::null -o datadir=/tmp/ngb-rt-target$L --path=repo >/dev/null || fail "target add ci$L"
|
||||||
|
done
|
||||||
|
IDS=$(api "$API/status" | php -r 'foreach (json_decode(stream_get_contents(STDIN), true)["targets"] as $t) { if ($t["name"] === "ci2" || $t["name"] === "ci3") { echo $t["id"], " "; } }')
|
||||||
|
for ID in $IDS; do api -X POST -d "{\"targetId\": $ID, \"label\": \"turn\"}" "$API/backups" >/dev/null || fail "start a backup to location $ID"; done
|
||||||
|
# Never two runs to one location: a second start gets the run that is there.
|
||||||
|
ID=${IDS%% *}
|
||||||
|
RUNS=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
|
||||||
|
api -X POST -d "{\"targetId\": $ID, \"label\": \"again\"}" "$API/backups" >/dev/null || fail "second start"
|
||||||
|
AGAIN=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
|
||||||
|
[ "$RUNS" = 2 ] && [ "$AGAIN" = 2 ] || fail "a second start to one location: $RUNS, then $AGAIN running"
|
||||||
|
occ backup:status | grep "queued" >/dev/null || fail "the second backup is not queued"
|
||||||
|
QUEUED=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["queued"]));')
|
||||||
|
[ "$QUEUED" = 1 ] || fail "$QUEUED backups queued instead of 1"
|
||||||
|
for i in $(seq 1 60); do
|
||||||
|
api -X POST "$API/tick" >/dev/null || fail "tick"
|
||||||
|
RUNNING=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
|
||||||
|
[ "$RUNNING" != 0 ] || break
|
||||||
|
done
|
||||||
|
[ "$RUNNING" = 0 ] || fail "the queued backups did not finish"
|
||||||
|
occ backup:list ci3 | grep " turn " >/dev/null || fail "the queued backup made no snapshot"
|
||||||
|
ok "locations in turn: the second backup queued, and done after the first"
|
||||||
occ background:cron >/dev/null
|
occ background:cron >/dev/null
|
||||||
$X php -f /var/www/html/cron.php >/dev/null 2>&1 || true
|
$X php -f /var/www/html/cron.php >/dev/null 2>&1 || true
|
||||||
occ backup:status | grep "system cron" >/dev/null && fail "backup:status still warns after cron ran"
|
occ backup:status | grep "system cron" >/dev/null && fail "backup:status still warns after cron ran"
|
||||||
sleep 8 # APCu app config cache of the web server
|
sleep 8 # APCu app config cache of the web server
|
||||||
api "$API/status" | php -r 'exit(json_decode(stream_get_contents(STDIN), true)["cron"]["ok"] ? 0 : 1);' || fail "the status API still reports a missing cron after cron ran"
|
api "$API/status" | php -r 'exit(json_decode(stream_get_contents(STDIN), true)["cron"]["ok"] ? 0 : 1);' || fail "the status API still reports a missing cron after cron ran"
|
||||||
ok "with a system cron that ran: no warning in backup:status or the API"
|
ok "with a system cron that ran: no warning in backup:status or the API"
|
||||||
|
|
||||||
|
# A schedule per location. Choosing a time starts nothing; the moment itself does, and only there.
|
||||||
|
JOB=$(occ background-job:list --class='OCA\NgBackup\BackgroundJob\BackupJob' --output=json | php -r 'echo json_decode(stream_get_contents(STDIN), true)[0]["id"];')
|
||||||
|
job() { occ background-job:execute "$JOB" --force-execute >/dev/null; }
|
||||||
|
occ backup:target:schedule ci off >/dev/null
|
||||||
|
occ backup:target:schedule ci3 off >/dev/null
|
||||||
|
occ backup:target:schedule ci2 00:00 --timezone=UTC >/dev/null || fail "set a schedule"
|
||||||
|
occ backup:target:schedule ci2 | grep "00:00 (UTC)" >/dev/null || fail "schedule not shown"
|
||||||
|
job
|
||||||
|
[ "$(occ backup:list ci2 | grep -c ' scheduled ')" = 0 ] || fail "choosing a time started a backup"
|
||||||
|
occ backup:target:schedule ci2 "$($X date -u -d '+1 minute' +%H:%M)/6" --timezone=UTC >/dev/null || fail "set every 6 hours"
|
||||||
|
sleep 65
|
||||||
|
job
|
||||||
|
occ backup:list ci2 | grep " scheduled " >/dev/null || fail "the scheduled moment started no backup"
|
||||||
|
[ "$(occ backup:list ci3 | grep -c ' scheduled ')" = 0 ] || fail "a location that is off was backed up"
|
||||||
|
job
|
||||||
|
[ "$(occ backup:list ci2 | grep -c ' scheduled ')" = 1 ] || fail "one moment started more than one backup"
|
||||||
|
occ backup:target:schedule ci2 nonsense >/dev/null 2>&1 && fail "a bad schedule was accepted"
|
||||||
|
ok "schedule per location: nothing on choosing a time, one backup at the moment, none when off"
|
||||||
|
|
||||||
|
# What administrators are told, as Nextcloud shows it.
|
||||||
|
notes() { api "http://localhost/ocs/v2.php/apps/notifications/api/v2/notifications?format=json"; }
|
||||||
|
occ backup:target:schedule ci2 off >/dev/null
|
||||||
|
$X sh -c "touch '$DATA/ngb-rt-bad\\name.txt'"
|
||||||
|
occ backup:run ci3 -l names | grep "their name is not supported" >/dev/null || fail "occ does not report the name"
|
||||||
|
$X sh -c "rm '$DATA/ngb-rt-bad\\name.txt'"
|
||||||
|
NOTES=$(notes) || fail "read the notifications"
|
||||||
|
echo "$NOTES" | grep "turned off the schedule of backup location ci2" >/dev/null || fail "no notification about the schedule that was turned off"
|
||||||
|
echo "$NOTES" | grep "is missing because of its name" >/dev/null || fail "no notification about the name"
|
||||||
|
echo "$NOTES" | grep "Rename them" >/dev/null || fail "the notification about the name does not ask to rename"
|
||||||
|
job
|
||||||
|
notes | grep "has no schedule" >/dev/null && fail "a reminder right after turning it off"
|
||||||
|
for L in 2 3; do occ backup:target:remove ci$L >/dev/null || fail "target remove ci$L"; done
|
||||||
occ user:delete "$ADM" >/dev/null
|
occ user:delete "$ADM" >/dev/null
|
||||||
|
ok "notifications: schedule turned off, name not supported, no early reminder"
|
||||||
|
|
||||||
# Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob).
|
# Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob).
|
||||||
occ backup:verify ci "$SNAP" | grep "^OK$" >/dev/null || fail "shallow verify"
|
occ backup:verify ci "$SNAP" | grep "^OK$" >/dev/null || fail "shallow verify"
|
||||||
@@ -109,6 +189,17 @@ ok "database dump and restore"
|
|||||||
|
|
||||||
# Retention, trash and catalog.
|
# Retention, trash and catalog.
|
||||||
occ backup:retention --last=1 --daily=0 --weekly=0 --monthly=0 >/dev/null
|
occ backup:retention --last=1 --daily=0 --weekly=0 --monthly=0 >/dev/null
|
||||||
|
# A pinned restore point stays, and a note can change afterwards.
|
||||||
|
FIRST=$(occ backup:list ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')
|
||||||
|
occ backup:pin ci "$FIRST" >/dev/null || fail "pin"
|
||||||
|
occ backup:note ci "$FIRST" "a note made later" >/dev/null || fail "note"
|
||||||
|
occ backup:list ci | grep " first " | grep " yes " | grep "a note made later" >/dev/null || fail "pin and note not listed"
|
||||||
|
occ backup:prune ci --dry-run | grep "would forget 1 " >/dev/null || fail "a pinned snapshot would be removed"
|
||||||
|
occ backup:pin ci "$FIRST" --unpin >/dev/null || fail "unpin"
|
||||||
|
occ backup:prune ci --dry-run | grep "would forget 2 " >/dev/null || fail "an unpinned snapshot would stay"
|
||||||
|
occ backup:note ci "$FIRST" >/dev/null || fail "remove note"
|
||||||
|
occ backup:list ci | grep -q "a note made later" && fail "note not removed"
|
||||||
|
ok "pin and note on a restore point"
|
||||||
occ backup:prune ci | grep "forgot 2" >/dev/null || fail "prune" # first and second; "page" (above) is the last
|
occ backup:prune ci | grep "forgot 2" >/dev/null || fail "prune" # first and second; "page" (above) is the last
|
||||||
occ backup:trash ci | grep " first " >/dev/null || fail "trash"
|
occ backup:trash ci | grep " first " >/dev/null || fail "trash"
|
||||||
TID=$(occ backup:trash ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')
|
TID=$(occ backup:trash ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')
|
||||||
|
|||||||
Reference in New Issue
Block a user