Compare commits
45 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| be3746e1c3 | |||
| f5473495ce | |||
| 3694184674 | |||
| a0fc91a700 | |||
| a11e6c7254 | |||
| 6f1a37f4f4 | |||
| 5cde57bc1b | |||
| 3eec2e9554 | |||
| cdf0b70e25 | |||
| 66615c2a88 | |||
| cc75c7128c | |||
| a0c4abb60a | |||
| 4847ce293d | |||
| 90dd97977a | |||
| 08534975d7 | |||
| 95e3dfcf6d | |||
| 7f4cae0e42 | |||
| 965440d1f7 | |||
| 3a0f882a38 | |||
| 632d2a2838 | |||
| 9e9234e2a8 | |||
| 3ac3298313 | |||
| 384b2d2792 | |||
| 8032411789 | |||
| b4f20cb070 | |||
| 343389b98b | |||
| 133f5f9d34 | |||
| cf4bd9db7f | |||
| 67cc350b8c | |||
| fb5b234e8a | |||
| fd12c586b6 | |||
| 6ffe979ea7 | |||
| a03b7354b3 | |||
| be1c628f07 | |||
| a8355df12e | |||
| cd251cffb3 | |||
| 0b4872599d | |||
| b7d09c9784 | |||
| 24686897c3 | |||
| 315e83b847 | |||
| b728db36c7 | |||
| 6ed0a555cf | |||
| 221ed9dc17 | |||
| 285153a968 | |||
| 6c8a78f6bb |
@@ -41,6 +41,20 @@ jobs:
|
||||
- run: composer install --no-interaction --no-progress
|
||||
- run: composer psalm
|
||||
|
||||
frontend:
|
||||
# Builds the admin page, so a broken Vue component fails CI instead of showing up after a release.
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
persist-credentials: false
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
cache: npm
|
||||
- run: npm ci --no-audit --no-fund
|
||||
- run: npm run build
|
||||
|
||||
integration:
|
||||
# Full round trip (key, kit, location, backup, incremental, restore modes, database dump and
|
||||
# restore, retention, trash) on a fresh Nextcloud per version and database.
|
||||
|
||||
+101
-5
@@ -7,7 +7,97 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
|
||||
beta/daily update channel, which would hide the app from most admins. The stability stage is
|
||||
instead carried by the app description, the git tag, the release, and this file.
|
||||
|
||||
## [1.0.0-rc.5] - 2026-10-22
|
||||
Each heading starts with that `<version>`, once: the app store shows a release the section(s)
|
||||
whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
|
||||
shown under 1.0.0. The git tag of a release candidate goes after the date.
|
||||
|
||||
## [1.0.1] - 2026-11-01
|
||||
|
||||
Fixes for what was reported after 1.0.0 (#22, #24) and small improvements. Nothing changes in
|
||||
your backups or settings.
|
||||
|
||||
### Added
|
||||
- `occ backup:user:list [location] [user]` shows the user exports on a location, with the manifest
|
||||
path `backup:user:restore` expects. Before, that path was only printed once by
|
||||
`backup:user:backup` (#22).
|
||||
- A "Docs" tab on the admin page, with links to the how-to, the disaster recovery guide and the
|
||||
roadmap. It also shows before the backup key is set up.
|
||||
- A label for a backup started with "Back up now" (optional; `occ backup:run -l` already had one).
|
||||
It shows with the restore point.
|
||||
- An explanation in the trash of restore points of what "removed by retention policy" means: the
|
||||
policy keeps the newest restore points and the last one of each recent day, week and month, and
|
||||
moves the others to the trash, where they can be brought back until they are deleted
|
||||
permanently. `occ backup:trash` says the same below its list.
|
||||
|
||||
### Changed
|
||||
- `occ backup:status`, `backup:list`, `backup:trash` and `backup:browse` show times in the time
|
||||
zone of the daily schedule and name it, as the admin page shows them in the administrator's.
|
||||
Before, they showed the server's UTC, which read as a wrong start time.
|
||||
- The admin page no longer says NG Backup is in beta.
|
||||
- `occ backup:user:restore` also takes a user id instead of the manifest path, when that user has
|
||||
exactly one export on the location. With more, and for a manifest path that is not on the
|
||||
location, the error lists that user's exports; for a user without exports it names the
|
||||
expected form (#22). Before, `backup:user:restore <location> <uid>` only said "is not a
|
||||
recorded user export for this location".
|
||||
|
||||
### Fixed
|
||||
- An SFTP location with an RSA key in OpenSSH format ("BEGIN OPENSSH PRIVATE KEY") stopped
|
||||
working after an upgrade to Nextcloud 35 with "Unable to read key", and such a key could not be
|
||||
added there at all (#24). The "Private key" field on the admin page is a single line, so a
|
||||
pasted key lost its line breaks; Nextcloud 34 still read it, Nextcloud 35 reads keys with a
|
||||
stricter library. The line breaks are now restored before the key is used, so a stored location
|
||||
works again without re-entering it.
|
||||
- A full restore onto SQLite could stop at once with "database is locked" when another process
|
||||
wrote to the database just as the restore began (a background job that was already running).
|
||||
The restore now takes the write lock first and waits for the other process instead. The
|
||||
full-restore test on SQLite runs with a second process writing alongside.
|
||||
- `occ backup:restore ... --to-directory` of a single file restored nothing ("Restored 0
|
||||
files"); only a folder worked. A path to a file now restores that file.
|
||||
- A restore that was stopped (a crash, a timeout, a restart) left its half-written temporary file
|
||||
(`<name>.ngb-restore-<id>`) next to the file it was restoring, and a `files:scan` then showed it
|
||||
in Files. A restore now removes such leftovers in each folder it writes to.
|
||||
- The app store showed the notes of every release candidate under 1.0.0, because their headings
|
||||
started with "1.0.0-rc". Each heading now starts with the app version of that release.
|
||||
|
||||
## [1.0.0] - 2026-10-29
|
||||
|
||||
First stable release. NG Backup makes encrypted, incremental backups of your whole Nextcloud
|
||||
(files, database and configuration) to S3, SFTP, WebDAV, SMB and other storage, and restores them
|
||||
from the web interface or with occ. Disaster recovery has been tested on real installations; see
|
||||
"Tested disaster recoveries" in the README, and read "Known limitations" before you rely on it.
|
||||
|
||||
The code is the same as 1.0.0-rc.6 (app version 0.9.12): updating from it changes nothing in your
|
||||
backups or settings.
|
||||
|
||||
## [0.9.12] - 2026-10-24 (1.0.0-rc.6)
|
||||
|
||||
Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery
|
||||
simulations and a user report.
|
||||
|
||||
### Fixed
|
||||
- The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in
|
||||
Berlin (#19). The time is now saved with the administrator's time zone and shown with it. A time
|
||||
saved before is converted once to the zone of the server or an administrator, so it shows as it
|
||||
effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer) and nothing moves.
|
||||
- A full restore over SMB stopped after 17 hours with "Decryption failed": a read over a long
|
||||
line handed out wrong bytes (the backup itself was fine, the same files restored identically
|
||||
afterwards). A blob that does not decrypt as first read, of a file or of the database, is now
|
||||
read once more on a request of its own, and only a second failure stops the restore.
|
||||
- A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table
|
||||
(of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the
|
||||
table got 3-byte utf8. Tables the restore creates now get this installation's character set and
|
||||
collation (utf8mb4). Found in a disaster recovery simulation onto MariaDB.
|
||||
|
||||
### Changed
|
||||
- The admin page uses more of the screen (sections up to 1400 pixels instead of 900), so the
|
||||
list of backup runs fits without scrolling sideways.
|
||||
- README: a roadmap, the tested disaster recoveries, measurements for OVH S3, the Storage Box over
|
||||
SFTP and SMB, how to build the SMB extension when PECL is down, and what works without shell
|
||||
access. The guide lists shell access among what you need.
|
||||
- The full-restore integration test checks a 4-byte character in a table the restore creates, and
|
||||
retries app installs from the app store.
|
||||
|
||||
## [0.9.11] - 2026-10-22 (1.0.0-rc.5)
|
||||
|
||||
Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
|
||||
stopped after 13 hours with "Blob not in index"; the backup itself was fine.
|
||||
@@ -40,7 +130,7 @@ stopped after 13 hours with "Blob not in index"; the backup itself was fine.
|
||||
- The extra connection a full restore reads ahead on is replaced when it breaks (up to 20 times),
|
||||
instead of reading the rest of the restore the slow way.
|
||||
|
||||
## [1.0.0-rc.4] - 2026-10-19
|
||||
## [0.9.10] - 2026-10-19 (1.0.0-rc.4)
|
||||
|
||||
Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not
|
||||
released: its fixes are in here.
|
||||
@@ -64,7 +154,7 @@ released: its fixes are in here.
|
||||
- The app store showed old screenshots: it keeps a copy per URL, and the URLs always pointed at
|
||||
`main`. They now point at the release's tag.
|
||||
|
||||
## [1.0.0-rc.2] - 2026-10-18
|
||||
## [0.9.8] - 2026-10-18 (1.0.0-rc.2)
|
||||
|
||||
Second release candidate for 1.0.0; the app version is 0.9.8. Fixes from a full disaster recovery
|
||||
test of a production instance (about 1 TB, Nextcloud All-in-One onto a plain Nextcloud with
|
||||
@@ -105,7 +195,7 @@ PostgreSQL).
|
||||
- Removing a location left its cached index in the database (tens of thousands of rows for a large
|
||||
backup). It is removed with the location now, and once for locations that are already gone.
|
||||
|
||||
## [1.0.0-rc.1] - 2026-10-17
|
||||
## [0.9.7] - 2026-10-17 (1.0.0-rc.1)
|
||||
|
||||
Release candidate for 1.0.0; the app version is 0.9.7. Only bug fixes until 1.0.0. Tested on a
|
||||
production instance (about 1 TB, 150,000 files): full and incremental backups and a verify on
|
||||
@@ -384,7 +474,13 @@ and not implemented yet.
|
||||
- See the "Storage backend experiences" section of the README for provider-specific quirks found
|
||||
so far (e.g. Hetzner Storage Box sub-accounts).
|
||||
|
||||
[1.0.0-rc.1]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.1
|
||||
[1.0.1]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.1
|
||||
[1.0.0]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0
|
||||
[0.9.12]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.6
|
||||
[0.9.11]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.5
|
||||
[0.9.10]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.4
|
||||
[0.9.8]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.2
|
||||
[0.9.7]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.1
|
||||
[0.9.6-beta.7]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.6-beta.7
|
||||
[0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6
|
||||
[0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5
|
||||
|
||||
@@ -392,14 +392,96 @@ in README "Known limitations":
|
||||
gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een
|
||||
productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0.
|
||||
Problemen of meldingen van gebruikers: rc5.
|
||||
- **1.0.1:**
|
||||
1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
|
||||
- **1.0.1** (klaar voor release 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
|
||||
1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
|
||||
niet aangekondigd en los van de release.
|
||||
1. **Naar 1.0.2.** (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
|
||||
restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
|
||||
2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron
|
||||
staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue
|
||||
#13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles.
|
||||
2. **Naar 1.0.2.** Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit
|
||||
oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21):
|
||||
a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode`
|
||||
(`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de
|
||||
laatste run te lang geleden: melding met de cron-regel erbij ("Backups only move while
|
||||
someone has Nextcloud open"), en "waiting for background jobs" als een lopende backup lang
|
||||
geen stap zette.
|
||||
b. **De pagina laat de backup lopen** zolang ze openstaat: in `ajax`/`webcron` voert de
|
||||
statusaanroep die de pagina toch al herhaalt ook een korte, begrensde stap uit (zoals de
|
||||
korte stappen die AJAX-cron nu al krijgt). "Back up now" werkt dan ook zonder cron zolang
|
||||
het tabblad openstaat; de melding zegt dat geplande backups echte cron nodig hebben.
|
||||
De melding zegt (zonder werkende systeem-cron): backups hebben systeem-cron nodig, ook de
|
||||
gewone nachtelijke; zo stel je cron in (link naar de Nextcloud-documentatie over
|
||||
achtergrondjobs voor de versie van de server, nu
|
||||
https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html,
|
||||
versienummer uit de server, niet vast); zolang er geen cron is, loopt een backup die je
|
||||
met de knop start alleen door terwijl de NG Backup-pagina openstaat.
|
||||
c. **Voortgang in de databasefase** (issue #13): nu staat er alleen "running (db)", wat er
|
||||
hetzelfde uitziet als vastzitten. Toon "running (db: 47/183 tables)" zoals de bestandsfase
|
||||
een percentage toont, en "waiting for background jobs (last ran 12 min ago)" als een run
|
||||
een tijd geen stap zette, ook bij een cron die is blijven staan.
|
||||
Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en
|
||||
`cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop
|
||||
(vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen).
|
||||
3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later),
|
||||
zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest.
|
||||
4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de
|
||||
tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop
|
||||
stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een
|
||||
bestaande parameter. Ten hoogste 100 tekens — **klaar** (2026-10-30).
|
||||
5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen
|
||||
wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind
|
||||
opsommen, in plaats van stoppen. **Naar 1.0.2** (besluit 2026-10-30).
|
||||
6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van
|
||||
rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-<id>` staan; de
|
||||
geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files.
|
||||
Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op (alleen bestanden met
|
||||
precies dat achtervoegsel en 6 of 12 hextekens) — **klaar** (2026-10-30).
|
||||
7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden
|
||||
terug (er komt een `/` achter); nu werkt alleen een map. Gold alleen voor `--to-directory`;
|
||||
naar de bestanden van een gebruiker werkte een los bestand al — **klaar** (2026-10-30).
|
||||
8. **Uitleg (i) bij de prullenbak van herstelpunten** (besluit 2026-10-25): bij "removed by
|
||||
retention policy" een infoknop met wat het betekent: het bewaarbeleid houdt voor oudere dagen
|
||||
één herstelpunt per dag (het laatste), weken en maanden; de rest gaat naar de prullenbak en
|
||||
wordt pas na de ingestelde wachttijd (`delete_delay_days`, standaard 30) echt verwijderd, en is
|
||||
tot die datum terug te zetten. Aanleiding: de beheerder zag "19-10 13:12 – after rc4 upgrade
|
||||
removed by retention policy" en dacht dat er iets was weggegooid. `occ backup:trash` zegt
|
||||
hetzelfde onder de lijst — **klaar** (2026-10-30).
|
||||
9. **Tijdzone in `occ backup:status`**: de tabel toont de servertijd (UTC), de webinterface de
|
||||
tijd van de beheerder; dat leest als een verkeerde starttijd. Zet de zone in de kolomkop
|
||||
("started (UTC)") of toon de tijden in de zone van het schema. Gedaan: beide, en ook in
|
||||
`backup:list`, `backup:trash` en `backup:browse` — **klaar** (2026-10-30).
|
||||
10. **User-restore op gebruikersnaam** (issue #22, 2026-10-27): `backup:user:restore <locatie>
|
||||
New_User` gaf alleen "is not a recorded user export"; het argument moet het manifestpad
|
||||
zijn, dat alleen `backup:user:backup` eenmalig toonde. Nu: `backup:user:list` toont de
|
||||
exports, een kale uid werkt bij precies één export, anders een lijst om uit te kiezen —
|
||||
**klaar** (2026-10-27).
|
||||
11. **Tabblad "Docs" in de beheerpagina** (besluit 2026-10-27): een derde knop naast "Backup and
|
||||
restore" en "Settings", met links naar de HOWTO (`docs/HOWTO.md`), de DR-gids
|
||||
(`docs/DISASTER-RECOVERY.md`) en de roadmap (README, `#roadmap`) op GitHub, elk met een
|
||||
regel wat erin staat. Nu zijn ze alleen via de README te vinden. Zichtbaar ook zonder
|
||||
sleutel, zodat de gids vindbaar is vóór de eerste installatie-stap. "NG Backup is in beta"
|
||||
gaat uit de feedbackregel — **klaar** (2026-10-27).
|
||||
12. **SFTP-sleutel zonder regeleinden** (issue #24, 2026-10-28): het veld "Private key" op de
|
||||
beheerpagina is één regel, dus een geplakte sleutel verliest zijn regeleinden. NC34
|
||||
(phpseclib 2) las dat nog, NC35 (phpseclib 3) weigert een OpenSSH-sleutel dan met "Unable
|
||||
to read key". De regeleinden worden nu teruggezet vlak voor gebruik — **klaar** (2026-10-28).
|
||||
13. **"database is locked" bij het begin van `backup:restore:full` op SQLite** (CI, 2026-10-26
|
||||
bij #22 en 2026-10-29 op `main` na de merge van 1.0.0): de restore stopt meteen na het
|
||||
aanzetten van de onderhoudsmodus. Niet altijd, dus een tweede proces dat op hetzelfde moment
|
||||
de database gebruikt (cron, een webverzoek of de opstart van de container). Kan ook bij een
|
||||
echte restore naar SQLite gebeuren. Oorzaak (2026-10-29): SQLite (WAL) begint een transactie
|
||||
als lezer; schrijft een ander proces tussen ons eerste lezen en ons eerste schrijven, dan
|
||||
faalt dat schrijven direct (geen wachttijd helpt). In CI was dat de installer, die
|
||||
"installed" zet en daarna nog seconden doorschrijft. Nu neemt de restore eerst het
|
||||
schrijfslot en wacht hij op de ander; de SQLite-test draait met een tweede schrijver
|
||||
ernaast — **klaar** (2026-10-29).
|
||||
- **DR-tests over databasegrenzen** (besluit 2026-10-20; productie nc01 draait PostgreSQL): na
|
||||
de MariaDB→PostgreSQL-fout (tellers, rc5) elke overstap in de integratietest (`to-pgsql`,
|
||||
`to-mysql`, `to-sqlite`, ook in CI). Volle DR-tests met echte data: vóór 1.0.0 de S3-simulatie
|
||||
(MariaDB-bron, 138 GB) nogmaals naar een MariaDB-doel en de SMB-simulatie naar een SQLite-doel;
|
||||
**na 1.0.0** de 1 TB van nc01 (PostgreSQL) naar een MariaDB-doel.
|
||||
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
|
||||
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
|
||||
een paar bestanden.
|
||||
- **1.1.0:**
|
||||
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
|
||||
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
|
||||
@@ -412,6 +494,31 @@ in README "Known limitations":
|
||||
3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel
|
||||
een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf,
|
||||
33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1.
|
||||
4. **Schema per locatie** (besluit 2026-10-19): in de instellingen van een locatie "uit",
|
||||
"dagelijks om HH:MM" of "elke N uur (vanaf HH:MM)". Nu is er één `schedule_time` voor alle
|
||||
locaties, één keer per dag. Bij de update wordt de huidige tijd het schema van elke
|
||||
bestaande locatie (voor bestaande gebruikers verandert er niets).
|
||||
5. **Locaties na elkaar laten starten** in plaats van tegelijk: twee runs die dezelfde schijven
|
||||
lezen zijn samen niet sneller (gezien op nc01); wachtende runs zichtbaar als "queued".
|
||||
7. **SMB sneller** (gemeten 2026-10-21): lokaal netwerk zonder vertraging (Samba-container op de
|
||||
laptop, 1,8 GB) back-up 24,9 MB/s, restore 27,6 MiB/s; vanaf Canada naar de Storage Box (130 ms)
|
||||
2,6 MiB/s, omdat SMB per blok op het antwoord wacht. Grotere schrijfblokken, en vooruitlezen
|
||||
bij een restore zoals SFTP (IReadsAhead) of meerdere packs tegelijk. Een NAS met 1 Gbit haalt
|
||||
~110 MB/s: daar is winst.
|
||||
Ook de index: `target:add` over SMB vanaf Canada haalt ~1 indexbestand per seconde (4.927
|
||||
bestanden ~85 min), want SMB krijgt geen `getMany()` met verzoeken tegelijk zoals S3 en SFTP.
|
||||
6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot
|
||||
is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus),
|
||||
zodat een nieuwe server na een DR hem ook ziet.
|
||||
8. **Herstelpunt vastpinnen** (besluit 2026-10-25): een herstelpunt dat het bewaarbeleid
|
||||
nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met
|
||||
`occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de
|
||||
opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly.
|
||||
9. **Feedbackvraag weghalen** in 1.1.0 of 1.2.0 (besluit 2026-10-27): de regel "We would like
|
||||
to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot
|
||||
dan staan (de vermelding "in beta" is er in 1.0.1 al af). Zijn er tegen die
|
||||
tijd geen noemenswaardige meldingen, dan werkt het: mensen melden eerder wat niet werkt dan
|
||||
wat gewoon werkt. Het formulier (`experience.yml`) mag blijven voor wie het zelf opzoekt.
|
||||
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
|
||||
PostgreSQL 18)** — voor **1.0.0-rc.2**:
|
||||
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
|
||||
|
||||
@@ -7,11 +7,14 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
|
||||
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
||||
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
||||
|
||||
**Status: release candidate for 1.0.0 (1.0.0-rc.1, app version 0.9.7).** Tested thoroughly, usable
|
||||
alongside your current backup, but not yet a drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and
|
||||
**Status: 1.0.1, stable.** Disaster recovery has been tested on real installations (see "Tested
|
||||
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
|
||||
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
|
||||
`docs/DISASTER-RECOVERY.md`.
|
||||
|
||||
If you like NG Backup, don't forget to give the app a star in the
|
||||
[app store](https://apps.nextcloud.com/apps/ng_backup) and this repository a star on GitHub!
|
||||
|
||||
License: AGPL-3.0-or-later. Developed and sponsored by [Ensembia](https://www.ensembia.com).
|
||||
Need help restoring a backup after a crash? Ensembia offers paid restore assistance:
|
||||
[www.ensembia.com](https://www.ensembia.com).
|
||||
@@ -146,10 +149,31 @@ What you should do on the location (strongly recommended):
|
||||
- **Files and folders** of a user: in the web interface or `occ backup:restore`, into a new folder
|
||||
("Restored <date>"), merged into the original place, or replacing it.
|
||||
- **Raw restore** of any path (for example `config/`) to a local directory: `occ backup:restore ... --to-directory=DIR`.
|
||||
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see the how-to.
|
||||
- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see
|
||||
[docs/DISASTER-RECOVERY.md](docs/DISASTER-RECOVERY.md). It runs in maintenance mode, so that
|
||||
nobody and no background job writes into the instance while it is being replaced.
|
||||
- **One user** (account, settings, files, calendars, contacts via user_migration): `occ backup:user:backup` and
|
||||
`occ backup:user:restore`, replacing the account or restoring it as `<uid>-bak`. See the how-to.
|
||||
|
||||
**Without shell access** (a hosted Nextcloud where you cannot run `occ`): files and folders can be
|
||||
restored in the web interface. Restoring a whole user or the whole server needs `occ`, so on such
|
||||
hosting it is something your provider (or whoever has shell access) runs. A restore mode that
|
||||
works without maintenance mode, and so from the web interface, is on the roadmap.
|
||||
|
||||
## Tested disaster recoveries
|
||||
|
||||
Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECOVERY.md):
|
||||
|
||||
| Backup | Location | Database | Result |
|
||||
|---|---|---|---|
|
||||
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
|
||||
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
|
||||
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
|
||||
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
|
||||
| 51 GB test instance | OVH S3 | PostgreSQL → PostgreSQL | 23,519/23,519 checksums identical |
|
||||
|
||||
The integration tests restore across every pair of PostgreSQL, MariaDB and SQLite.
|
||||
|
||||
## Storage backend experiences
|
||||
|
||||
Notes from actually running a backend, beyond what's in files_external's own documentation.
|
||||
@@ -206,15 +230,71 @@ Contributions welcome via PR — add what you ran into with your own provider.
|
||||
can be read on more connections: `occ config:app:set ng_backup restore_sftp_connections
|
||||
--value=4` (at most 8; the server must allow that many plus one at once for the user).
|
||||
|
||||
- **OVH Object Storage (S3, Warsaw, 2026-10):** works with the `amazons3` backend (path style,
|
||||
region `waw`). A full backup of 1 TB from Germany ran at about 7 MiB/s, from Canada about
|
||||
6 MiB/s with BBR; a check of a 1 TB backup took 3.5 minutes (index files are fetched 16 at a
|
||||
time); a restore of 138 GB from Warsaw to Canada took about 6 hours. Turn on versioning and
|
||||
Object Lock on the bucket (see above); OVH charges no egress.
|
||||
|
||||
- **Hetzner Storage Box over SFTP, full restore:** about 13-14 MiB/s from Germany to Canada
|
||||
(122 ms) with the read-ahead of 1.0.0-rc.4; without it about 2.8 MiB/s. A Storage Box gets
|
||||
slower with several connections reading at once (4-6 MiB/s for 2-4 connections, it reads from
|
||||
disks), so one extra connection is the default. A Storage Box drops an SFTP connection that
|
||||
stayed quiet for a while without telling the client; NG Backup checks and replaces it.
|
||||
|
||||
- **SMB speed:** on a local network (Samba, no delay) a backup ran at about 25 MB/s and a
|
||||
restore at about 28 MiB/s. Over a long line SMB waits for an answer to each request: from
|
||||
Canada to a Storage Box in Germany (130 ms) about 2.6 MiB/s for a backup and 1.6 MiB/s for a
|
||||
restore, and connecting a location with an existing backup reads its index at about one file a
|
||||
second. SMB is meant for a local network; use SFTP or S3 for a location far away. When
|
||||
`pecl install smbclient` fails (pecl.php.net was unreachable for hours on 2026-10-21), the
|
||||
extension builds from its source: `libsmbclient-php` on GitHub, `phpize && ./configure && make
|
||||
&& make install`, then `docker-php-ext-enable smbclient`.
|
||||
|
||||
- **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself
|
||||
(e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the
|
||||
mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping
|
||||
mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS
|
||||
can't be driven from PHP directly.
|
||||
|
||||
## Roadmap
|
||||
|
||||
What is planned, without dates. Ideas and votes are welcome in the
|
||||
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
|
||||
a 👍 on its issue helps decide what comes first.
|
||||
|
||||
**1.0.2**
|
||||
- A warning on the admin page and in `occ backup:status` when background jobs do not run on a
|
||||
system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on
|
||||
while the NG Backup page is open, and the database phase of a backup shows its progress
|
||||
([#13](https://github.com/dillardblom/ng_backup/issues/13)).
|
||||
- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4.
|
||||
- `--salvage` for a restore: restore what can be read, skip a file that cannot, and list those at
|
||||
the end instead of stopping.
|
||||
|
||||
**1.1.0**
|
||||
- Restore without the recovery kit: the backup key from the location itself, opened with one of
|
||||
your passphrases.
|
||||
- A schedule per location, also "every N hours".
|
||||
- Locations backed up one after the other instead of at the same time.
|
||||
- A note on a restore point that you can change afterwards.
|
||||
- Pinning a restore point, so the retention policy never removes it (for example the one from
|
||||
before an upgrade).
|
||||
- An option to pause heavy background jobs (such as Memories, Recognize or the Preview
|
||||
Generator) while a backup runs.
|
||||
- Faster backups: reading and uploading at the same time.
|
||||
|
||||
**Later**
|
||||
- A full restore onto the existing installation, without first installing a fresh Nextcloud, in a
|
||||
restore mode of its own instead of maintenance mode, so that it can also run from the web
|
||||
interface (for hosted Nextcloud without shell access).
|
||||
- Users restoring their own files from the Files app.
|
||||
- Primary object storage and External Storage mounts (see Known limitations).
|
||||
- An option not to keep the backup key on the location (the recovery kit is then the only copy).
|
||||
|
||||
## Known limitations
|
||||
|
||||
On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it:
|
||||
On the backlog; know them before you rely on NG Backup:
|
||||
|
||||
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
|
||||
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
|
||||
@@ -236,8 +316,11 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
|
||||
- Location passwords passed with `-o password=...` are visible in the process list and shell
|
||||
history while the command runs.
|
||||
- Temporary files of a restore into a user's files end up in that user's trash.
|
||||
- Restoring the database does not order tables by foreign keys; apps with cascading foreign keys
|
||||
could lose rows (Nextcloud core does not use them).
|
||||
- **Restoring a whole user or the whole server needs `occ`** (shell access); see "Without shell
|
||||
access" above.
|
||||
- **`--install-apps` depends on the app store.** Nextcloud downloads the whole app list (about
|
||||
38 MB) and gives up after 120 seconds; on a slow day apps are not installed. The restore then
|
||||
lists them: install them afterwards with `occ app:install`, their data is restored already.
|
||||
- A pack whose index upload failed stays on the location until removed by hand.
|
||||
|
||||
## Commands
|
||||
@@ -254,4 +337,4 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
|
||||
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
|
||||
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
|
||||
| `backup:key:import` | import a recovery kit on a fresh installation |
|
||||
| `backup:user:backup`, `backup:user:restore` | one user via user_migration (asks before installing it) |
|
||||
| `backup:user:backup`, `backup:user:list`, `backup:user:restore` | one user via user_migration (asks before installing it); list the exports |
|
||||
|
||||
+8
-4
@@ -40,9 +40,12 @@ NG Backup needs Nextcloud's background jobs on system cron (a real cron line, no
|
||||
Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very
|
||||
welcome: use the "Share your experience" link on the admin page.
|
||||
|
||||
If you like NG Backup, don't forget to give the app a star in the app store and the
|
||||
[repository](https://github.com/dillardblom/ng_backup) a star on GitHub!
|
||||
|
||||
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
|
||||
you need help with restoring a backup after a crash.]]></description>
|
||||
<version>0.9.11</version>
|
||||
<version>1.0.1</version>
|
||||
<licence>agpl</licence>
|
||||
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
|
||||
<namespace>NgBackup</namespace>
|
||||
@@ -53,9 +56,9 @@ you need help with restoring a backup after a crash.]]></description>
|
||||
<website>https://github.com/dillardblom/ng_backup</website>
|
||||
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
|
||||
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/1-backups.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/2-restore.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/3-settings.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/1-backups.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/2-restore.png</screenshot>
|
||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.1/screenshots/3-settings.png</screenshot>
|
||||
<dependencies>
|
||||
<php min-version="8.1"/>
|
||||
<lib>sodium</lib>
|
||||
@@ -92,6 +95,7 @@ you need help with restoring a backup after a crash.]]></description>
|
||||
<command>OCA\NgBackup\Command\KeySlotRemove</command>
|
||||
<command>OCA\NgBackup\Command\BackupUser</command>
|
||||
<command>OCA\NgBackup\Command\RestoreUser</command>
|
||||
<command>OCA\NgBackup\Command\ListUserExports</command>
|
||||
<command>OCA\NgBackup\Command\KeyImport</command>
|
||||
<command>OCA\NgBackup\Command\RestoreFull</command>
|
||||
<command>OCA\NgBackup\Command\Verify</command>
|
||||
|
||||
@@ -10,6 +10,8 @@ Nextcloud back from an NG Backup location. Do them in this order.
|
||||
- [ ] A new server with the **same Nextcloud major version** as the old one (for example 34),
|
||||
a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for
|
||||
the whole backup plus a margin.
|
||||
- [ ] **Shell access** to the new server, to run `occ` (on a hosted Nextcloud without it, your
|
||||
provider runs these steps).
|
||||
- [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by
|
||||
hand). Keep their packages somewhere safe beforehand.
|
||||
|
||||
|
||||
@@ -114,10 +114,16 @@ Needs the `user_migration` app. NG Backup asks before installing it:
|
||||
|
||||
```
|
||||
occ backup:user:backup offsite <uid>
|
||||
occ backup:user:list offsite [<uid>] # the exports and their manifest
|
||||
occ backup:user:restore offsite users/<uid>/<exportId> # replace (default)
|
||||
occ backup:user:restore offsite users/<uid>/<exportId> --mode=as-backup
|
||||
```
|
||||
|
||||
- `backup:user:restore` needs the manifest path that `backup:user:backup` prints at the end
|
||||
(`users/<uid>/<exportId>`); `backup:user:list` shows it again. A user id alone also works when
|
||||
that user has exactly one export on the location. With more (a `replace` restore adds a safety
|
||||
export), the command lists them and you pass one.
|
||||
|
||||
- If `user_migration` is missing, the command asks to install and enable it. For non-interactive
|
||||
use, add `--install-user-migration` to approve that explicitly. The command stops after the
|
||||
installation; run it once more.
|
||||
|
||||
@@ -72,6 +72,10 @@ final class ExternalStorageFactory {
|
||||
throw new BackendException("Auth mechanism $authId does not fit backend $backendId");
|
||||
}
|
||||
|
||||
if (is_string($options['private_key'] ?? null)) {
|
||||
$options['private_key'] = self::normalizePrivateKey($options['private_key']);
|
||||
}
|
||||
|
||||
$config = new StorageConfig();
|
||||
$config->setBackend($backend);
|
||||
$config->setAuthMechanism($auth);
|
||||
@@ -86,4 +90,22 @@ final class ExternalStorageFactory {
|
||||
$storage = $backend->wrapStorage($storage);
|
||||
return $auth->wrapStorage($storage);
|
||||
}
|
||||
|
||||
/**
|
||||
* An OpenSSH-format private key with its line breaks restored. Up to Nextcloud 34
|
||||
* (phpseclib 2) a key whose newlines were lost or turned into spaces, e.g. on its way through
|
||||
* a shell or a form, still loaded; Nextcloud 35 (phpseclib 3) refuses it with "Unable to read
|
||||
* key" (#24). The key itself is not changed, so it is fixed when it is used, not when stored.
|
||||
* Other formats (PEM, PKCS#8) still load as they are and are returned unchanged.
|
||||
*/
|
||||
public static function normalizePrivateKey(string $key): string {
|
||||
if (!preg_match('/-----BEGIN OPENSSH PRIVATE KEY-----(.*?)-----END OPENSSH PRIVATE KEY-----/s', $key, $m)) {
|
||||
return $key;
|
||||
}
|
||||
$body = (string)preg_replace('/\s+/', '', $m[1]);
|
||||
if (!preg_match('#^[A-Za-z0-9+/]+=*$#', $body)) {
|
||||
return $key;
|
||||
}
|
||||
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,6 +32,7 @@ final class BackupJob extends TimedJob {
|
||||
private RunMapper $runs,
|
||||
private IAppConfig $appConfig,
|
||||
private LoggerInterface $logger,
|
||||
private \OCA\NgBackup\Service\Schedule $schedule,
|
||||
) {
|
||||
parent::__construct($time);
|
||||
// Well below the usual 5-minute cron interval: the last run is stamped a moment after cron
|
||||
@@ -101,14 +102,10 @@ final class BackupJob extends TimedJob {
|
||||
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
|
||||
}
|
||||
|
||||
/** Daily schedule: appconfig ng_backup/schedule_time "HH:MM" (server time); empty = no schedule. */
|
||||
/** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */
|
||||
private function startScheduled(): void {
|
||||
$at = $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', '');
|
||||
if (!preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $at)) {
|
||||
return;
|
||||
}
|
||||
$todayAt = (int)strtotime(date('Y-m-d') . ' ' . $at);
|
||||
if (time() < $todayAt) {
|
||||
$todayAt = $this->schedule->todayAt();
|
||||
if ($todayAt === null || time() < $todayAt) {
|
||||
return;
|
||||
}
|
||||
foreach ($this->targets->list() as $t) {
|
||||
|
||||
@@ -7,7 +7,9 @@ declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Command;
|
||||
|
||||
use OCA\NgBackup\Format;
|
||||
use OCA\NgBackup\Service\RestoreService;
|
||||
use OCA\NgBackup\Service\Schedule;
|
||||
use OCA\NgBackup\Service\TargetService;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
use Symfony\Component\Console\Helper\Table;
|
||||
@@ -19,6 +21,7 @@ final class Browse extends Command {
|
||||
public function __construct(
|
||||
private TargetService $targets,
|
||||
private RestoreService $restore,
|
||||
private Schedule $schedule,
|
||||
) {
|
||||
parent::__construct();
|
||||
}
|
||||
@@ -43,9 +46,10 @@ final class Browse extends Command {
|
||||
$output->writeln('Nothing here.');
|
||||
return 0;
|
||||
}
|
||||
$zone = $this->schedule->get()['timezone'];
|
||||
$rows = array_map(fn ($i) => [$i['type'] === 'dir' ? $i['name'] . '/' : $i['name'], $this->size($i['size']),
|
||||
$i['type'] === 'dir' ? $i['files'] . ' files' : date('Y-m-d H:i', (int)$i['mtime'])], $items);
|
||||
(new Table($output))->setHeaders(['name', 'size', 'files / modified'])->setRows($rows)->render();
|
||||
$i['type'] === 'dir' ? $i['files'] . ' files' : Format::time((int)$i['mtime'], $zone)], $items);
|
||||
(new Table($output))->setHeaders(['name', 'size', "files / modified ($zone)"])->setRows($rows)->render();
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
@@ -8,6 +8,8 @@ declare(strict_types=1);
|
||||
namespace OCA\NgBackup\Command;
|
||||
|
||||
use OCA\NgBackup\Db\SnapshotMapper;
|
||||
use OCA\NgBackup\Format;
|
||||
use OCA\NgBackup\Service\Schedule;
|
||||
use OCA\NgBackup\Service\TargetService;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
use Symfony\Component\Console\Helper\Table;
|
||||
@@ -19,6 +21,7 @@ final class ListSnapshots extends Command {
|
||||
public function __construct(
|
||||
private TargetService $targets,
|
||||
private SnapshotMapper $snapshots,
|
||||
private Schedule $schedule,
|
||||
) {
|
||||
parent::__construct();
|
||||
}
|
||||
@@ -32,10 +35,11 @@ final class ListSnapshots extends Command {
|
||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||
$name = $input->getArgument('target');
|
||||
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
|
||||
$zone = $this->schedule->get()['timezone'];
|
||||
$rows = [];
|
||||
foreach ($targets as $t) {
|
||||
foreach ($this->snapshots->findForTarget($t->getId()) as $s) {
|
||||
$rows[] = [$t->getName(), $s->getSnapshotId(), date('Y-m-d H:i', $s->getCreatedAt()), $s->getKind(),
|
||||
$rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(),
|
||||
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getLabel() ?? ''];
|
||||
}
|
||||
}
|
||||
@@ -43,7 +47,7 @@ final class ListSnapshots extends Command {
|
||||
$output->writeln('No snapshots yet (occ backup:run).');
|
||||
return 0;
|
||||
}
|
||||
(new Table($output))->setHeaders(['location', 'snapshot', 'date', 'kind', 'files', 'size', 'label'])->setRows($rows)->render();
|
||||
(new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'label'])->setRows($rows)->render();
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
namespace OCA\NgBackup\Command;
|
||||
|
||||
use OCA\NgBackup\Service\TargetService;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
use Symfony\Component\Console\Helper\Table;
|
||||
use Symfony\Component\Console\Input\InputArgument;
|
||||
use Symfony\Component\Console\Input\InputInterface;
|
||||
use Symfony\Component\Console\Output\OutputInterface;
|
||||
|
||||
final class ListUserExports extends Command {
|
||||
public function __construct(
|
||||
private TargetService $targets,
|
||||
) {
|
||||
parent::__construct();
|
||||
}
|
||||
|
||||
protected function configure(): void {
|
||||
$this->setName('backup:user:list')
|
||||
->setDescription('List the user exports (backup:user:backup) on a location')
|
||||
->addArgument('target', InputArgument::OPTIONAL, 'Location name or id (default: all)')
|
||||
->addArgument('user', InputArgument::OPTIONAL, 'Only the exports of this user id')
|
||||
->setHelp(<<<'HELP'
|
||||
The manifest column is what occ backup:user:restore expects. A safety export made by a restore
|
||||
with --mode=replace is listed here as well.
|
||||
|
||||
Example:
|
||||
occ backup:user:list offsite alice
|
||||
HELP);
|
||||
}
|
||||
|
||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||
$name = $input->getArgument('target');
|
||||
$uid = $input->getArgument('user');
|
||||
$rows = [];
|
||||
try {
|
||||
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
|
||||
foreach ($targets as $t) {
|
||||
foreach ($this->targets->repository($t)->listUserExports($uid) as $path) {
|
||||
$parts = explode('/', $path, 3);
|
||||
$user = $parts[1] ?? '';
|
||||
$exportId = $parts[2] ?? '';
|
||||
$time = \DateTimeImmutable::createFromFormat('Ymd\THis\Z', substr($exportId, 0, 16), new \DateTimeZone('UTC'));
|
||||
$rows[] = [$t->getName(), $user, $time !== false ? $time->format('Y-m-d H:i') : '', $path];
|
||||
}
|
||||
}
|
||||
} catch (\Throwable $e) {
|
||||
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||
return 1;
|
||||
}
|
||||
if ($rows === []) {
|
||||
$output->writeln('No user exports yet (occ backup:user:backup).');
|
||||
return 0;
|
||||
}
|
||||
(new Table($output))->setHeaders(['location', 'user', 'date (UTC)', 'manifest'])->setRows($rows)->render();
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
@@ -33,7 +33,7 @@ final class RestoreUser extends Command {
|
||||
$this->setName('backup:user:restore')
|
||||
->setDescription('Restore a user from a user_migration export (see backup:user:backup)')
|
||||
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
||||
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>)')
|
||||
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>), or a user id with exactly one export')
|
||||
->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'replace or as-backup', UserRestoreService::MODE_REPLACE)
|
||||
->addOption('force', null, InputOption::VALUE_NONE, 'Replace an existing account without asking (required when not interactive)')
|
||||
->addOption('install-user-migration', null, InputOption::VALUE_NONE, 'Install and enable user_migration if it is missing (non-interactive consent)')
|
||||
@@ -45,16 +45,31 @@ Never imports over a live account. Modes:
|
||||
compare and transfer by hand; Nextcloud cannot rename a user id, so the backup gets
|
||||
the new name, never the existing account
|
||||
|
||||
occ backup:user:list shows the exports on a location. A user id alone works when that user
|
||||
has exactly one export there; with more, the command lists them and you pass one.
|
||||
|
||||
Example:
|
||||
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34
|
||||
occ backup:user:restore offsite alice
|
||||
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34 --mode=as-backup
|
||||
HELP);
|
||||
}
|
||||
|
||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||
$mode = (string)$input->getOption('mode');
|
||||
$uid = explode('/', (string)$input->getArgument('manifest'))[1] ?? '';
|
||||
if ($mode === UserRestoreService::MODE_REPLACE && $uid !== '' && $this->userManager->userExists($uid) && !$input->getOption('force')) {
|
||||
try {
|
||||
$target = $this->targets->get($input->getArgument('target'));
|
||||
$ref = (string)$input->getArgument('manifest');
|
||||
$manifest = $this->targets->repository($target)->resolveUserExport($ref);
|
||||
} catch (\Throwable $e) {
|
||||
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||
return 1;
|
||||
}
|
||||
if ($manifest !== $ref) {
|
||||
$output->writeln("Using $manifest.");
|
||||
}
|
||||
$uid = explode('/', $manifest)[1];
|
||||
if ($mode === UserRestoreService::MODE_REPLACE && $this->userManager->userExists($uid) && !$input->getOption('force')) {
|
||||
// Replace deletes the live account (after a safety export). Shares to others and data of
|
||||
// apps without a user_migration migrator do not come back, so never do it unasked.
|
||||
if (!$input->isInteractive()) {
|
||||
@@ -69,9 +84,8 @@ HELP);
|
||||
}
|
||||
try {
|
||||
$output->writeln('<comment>Note: on restore, user_migration only imports the app settings on its own allowlist (e.g. calendar view and reminder settings); other app settings of this user are skipped. Reason: NG Backup uses user_migration\'s importer; widening the list would mean building a settings importer of our own.</comment>');
|
||||
$target = $this->targets->get($input->getArgument('target'));
|
||||
$this->installer->ensureEnabled($input, $output);
|
||||
$user = $this->userRestore->restoreUser($target, (string)$input->getArgument('manifest'), $mode);
|
||||
$user = $this->userRestore->restoreUser($target, $manifest, $mode);
|
||||
} catch (\Throwable $e) {
|
||||
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||
return 1;
|
||||
|
||||
@@ -8,7 +8,9 @@ declare(strict_types=1);
|
||||
namespace OCA\NgBackup\Command;
|
||||
|
||||
use OCA\NgBackup\Db\RunMapper;
|
||||
use OCA\NgBackup\Format;
|
||||
use OCA\NgBackup\Service\KeyService;
|
||||
use OCA\NgBackup\Service\Schedule;
|
||||
use OCA\NgBackup\Service\TargetService;
|
||||
use OCA\NgBackup\Service\VerifyService;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
@@ -26,6 +28,7 @@ final class Status extends Command {
|
||||
private TargetService $targets,
|
||||
private RunMapper $runs,
|
||||
private VerifyService $verify,
|
||||
private Schedule $schedule,
|
||||
) {
|
||||
parent::__construct();
|
||||
}
|
||||
@@ -36,12 +39,13 @@ final class Status extends Command {
|
||||
}
|
||||
|
||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||
$zone = $this->schedule->get()['timezone'];
|
||||
if (!$this->keys->isInitialized()) {
|
||||
$output->writeln('Key: <error>not created</error> (occ backup:key:init)');
|
||||
} else {
|
||||
$c = $this->keys->recoveryKitConfirmation();
|
||||
$output->writeln('Key: ' . $this->keys->fingerprint() . ', recovery kit '
|
||||
. ($c ? 'confirmed by ' . $c['uid'] . ' on ' . date('Y-m-d H:i', $c['time']) : '<error>not confirmed</error> (occ backup:key:confirm)'));
|
||||
. ($c ? 'confirmed by ' . $c['uid'] . ' on ' . Format::time($c['time'], $zone) . " ($zone)" : '<error>not confirmed</error> (occ backup:key:confirm)'));
|
||||
}
|
||||
$names = [];
|
||||
foreach ($this->targets->list() as $t) {
|
||||
@@ -54,12 +58,12 @@ final class Status extends Command {
|
||||
foreach ($shown as $r) {
|
||||
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
|
||||
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''),
|
||||
date('Y-m-d H:i', $r->getStartedAt()), $r->getFinishedAt() ? \OCA\NgBackup\Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
|
||||
Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
|
||||
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
|
||||
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
|
||||
}
|
||||
if ($rows !== []) {
|
||||
(new Table($output))->setHeaders(['run', 'location', 'kind', 'status', 'started', 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render();
|
||||
(new Table($output))->setHeaders(['run', 'location', 'kind', 'status', "started ($zone)", 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render();
|
||||
}
|
||||
$older = $all ? 0 : $this->runs->countAll() - count($shown);
|
||||
if ($older > 0) {
|
||||
|
||||
@@ -7,7 +7,9 @@ declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Command;
|
||||
|
||||
use OCA\NgBackup\Format;
|
||||
use OCA\NgBackup\Service\KeyService;
|
||||
use OCA\NgBackup\Service\Schedule;
|
||||
use OCA\NgBackup\Service\TargetService;
|
||||
use Symfony\Component\Console\Command\Command;
|
||||
use Symfony\Component\Console\Helper\Table;
|
||||
@@ -19,6 +21,7 @@ final class Trash extends Command {
|
||||
public function __construct(
|
||||
private TargetService $targets,
|
||||
private KeyService $keys,
|
||||
private Schedule $schedule,
|
||||
) {
|
||||
parent::__construct();
|
||||
}
|
||||
@@ -32,11 +35,14 @@ final class Trash extends Command {
|
||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||
$name = $input->getArgument('target');
|
||||
$delay = $this->keys->deleteDelayDays() * 86400;
|
||||
$zone = $this->schedule->get()['timezone'];
|
||||
$rows = [];
|
||||
foreach ($name === null ? $this->targets->list() : [$this->targets->get($name)] as $t) {
|
||||
foreach ($this->targets->repository($t)->trash() as $id => $info) {
|
||||
$rows[] = [$t->getName(), $id, (string)($info['time'] ?? ''), $info['label'], date('Y-m-d H:i', $info['forgottenAt']), $info['by'],
|
||||
date('Y-m-d H:i', $info['forgottenAt'] + $delay)];
|
||||
$time = $info['time'] ?? null;
|
||||
$made = is_string($time) ? strtotime($time) : false;
|
||||
$rows[] = [$t->getName(), $id, $made !== false ? Format::time($made, $zone) : '', $info['label'], Format::time($info['forgottenAt'], $zone), $info['by'],
|
||||
Format::time($info['forgottenAt'] + $delay, $zone)];
|
||||
}
|
||||
}
|
||||
if ($rows === []) {
|
||||
@@ -44,6 +50,7 @@ final class Trash extends Command {
|
||||
return 0;
|
||||
}
|
||||
(new Table($output))->setHeaders(['location', 'snapshot', 'made', 'label', 'removed', 'by', 'deleted after'])->setRows($rows)->render();
|
||||
$output->writeln("Times in $zone. The retention policy keeps the newest snapshots and the last of each recent day, week and month; it moves the others here, and they are deleted permanently after " . $this->keys->deleteDelayDays() . ' days.');
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,6 +47,7 @@ final class AdminApiController extends Controller {
|
||||
private \OCA\NgBackup\Service\AlertService $alerts,
|
||||
private \OCA\NgBackup\Service\KeySlotService $slots,
|
||||
private \OCA\NgBackup\Service\PruneService $prune,
|
||||
private \OCA\NgBackup\Service\Schedule $schedule,
|
||||
) {
|
||||
parent::__construct(Application::APP_ID, $request);
|
||||
}
|
||||
@@ -72,7 +73,8 @@ final class AdminApiController extends Controller {
|
||||
'runs' => $runs,
|
||||
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
|
||||
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
|
||||
'schedule' => $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''),
|
||||
'schedule' => $this->schedule->get()['time'],
|
||||
'scheduleTimezone' => $this->schedule->get()['timezone'],
|
||||
]);
|
||||
}
|
||||
|
||||
@@ -240,12 +242,14 @@ final class AdminApiController extends Controller {
|
||||
return new JSONResponse($this->runJson($run));
|
||||
}
|
||||
|
||||
public function setSchedule(string $time): JSONResponse {
|
||||
if ($time !== '' && !preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time)) {
|
||||
return new JSONResponse(['error' => 'Use HH:MM or empty'], Http::STATUS_BAD_REQUEST);
|
||||
/** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */
|
||||
public function setSchedule(string $time, string $timezone = ''): JSONResponse {
|
||||
try {
|
||||
$this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
|
||||
} catch (\InvalidArgumentException $e) {
|
||||
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||
}
|
||||
$this->appConfig->setValueString(Application::APP_ID, 'schedule_time', $time);
|
||||
return new JSONResponse(['schedule' => $time]);
|
||||
return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]);
|
||||
}
|
||||
|
||||
public function snapshots(int $targetId): JSONResponse {
|
||||
|
||||
+33
-4
@@ -159,7 +159,7 @@ final class DbDumper {
|
||||
// everywhere); tables it does not know are left out, as asked.
|
||||
foreach ($missing as $table) {
|
||||
if (isset($manifest['tables'][$table]['schema'])) {
|
||||
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema']));
|
||||
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
|
||||
$report['created'][] = $table;
|
||||
} else {
|
||||
$report['missing'][] = $table;
|
||||
@@ -169,7 +169,7 @@ final class DbDumper {
|
||||
$foreignKeys = [];
|
||||
$this->db->beginTransaction();
|
||||
try {
|
||||
$foreignKeys = $this->suspendForeignKeys();
|
||||
$foreignKeys = $this->suspendForeignKeys(); // first: on SQLite it also takes the write lock
|
||||
foreach ($manifest['tables'] as $table => $info) {
|
||||
if (in_array($table, $report['missing'], true)) {
|
||||
continue;
|
||||
@@ -319,8 +319,31 @@ final class DbDumper {
|
||||
'foreignKeys' => $foreignKeys];
|
||||
}
|
||||
|
||||
private static function tableFromStructure(string $name, array $structure): \Doctrine\DBAL\Schema\Table {
|
||||
$table = new \Doctrine\DBAL\Schema\Table($name);
|
||||
/**
|
||||
* Table options for a table the restore creates: those of this installation's own tables.
|
||||
* On MySQL/MariaDB that is the character set and collation, as Nextcloud's tables have them
|
||||
* (utf8mb4, so emoji and other 4-byte characters fit); DBAL alone would use 3-byte utf8.
|
||||
*
|
||||
* @return array<string, mixed>
|
||||
* @psalm-suppress UndefinedClass the DBAL classes come with Nextcloud, not with this app
|
||||
*/
|
||||
private function tableOptions(): array {
|
||||
$options = $this->db->createSchemaManager()->createSchemaConfig()->getDefaultTableOptions();
|
||||
if ($this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\AbstractMySQLPlatform) {
|
||||
$collation = $this->db->fetchOne('SELECT TABLE_COLLATION FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ?',
|
||||
[$this->prefix . 'appconfig']);
|
||||
if (is_string($collation) && $collation !== '') {
|
||||
$options['collation'] = $collation;
|
||||
$options['charset'] = explode('_', $collation, 2)[0];
|
||||
unset($options['collate']);
|
||||
}
|
||||
}
|
||||
return $options;
|
||||
}
|
||||
|
||||
/** @param array<string, mixed> $options table options (see tableOptions()) */
|
||||
private static function tableFromStructure(string $name, array $structure, array $options = []): \Doctrine\DBAL\Schema\Table {
|
||||
$table = new \Doctrine\DBAL\Schema\Table($name, [], [], [], [], $options);
|
||||
foreach ($structure['columns'] as $c) {
|
||||
$table->addColumn($c['name'], $c['type'], array_filter([
|
||||
'length' => $c['length'], 'precision' => $c['precision'], 'scale' => $c['scale'],
|
||||
@@ -389,6 +412,12 @@ final class DbDumper {
|
||||
return $checks;
|
||||
}
|
||||
if ($platform instanceof \Doctrine\DBAL\Platforms\SqlitePlatform) {
|
||||
// SQLite (WAL) starts a transaction as a reader. When another process (a cron job, the
|
||||
// installer still finishing) writes between our first read and our first write, that
|
||||
// write fails at once with "database is locked": no waiting helps a reader whose view
|
||||
// is out of date. Take the write lock first, so we wait for the other writer instead
|
||||
// and it waits for us after that.
|
||||
$this->db->executeStatement('DELETE FROM ' . $this->db->quoteIdentifier($this->prefix . 'appconfig') . ' WHERE 0 = 1');
|
||||
$this->db->executeStatement('PRAGMA defer_foreign_keys = ON'); // checked at commit
|
||||
}
|
||||
return [];
|
||||
|
||||
@@ -15,4 +15,14 @@ final class Format {
|
||||
$time = gmdate('H:i:s', $seconds % 86400);
|
||||
return $days > 0 ? $days . 'd ' . $time : $time;
|
||||
}
|
||||
|
||||
/**
|
||||
* "2026-10-29 04:30" in $zone. Nextcloud runs PHP in UTC; occ shows times in the zone of the
|
||||
* schedule, the administrator's, as the web interface does.
|
||||
*
|
||||
* @param non-empty-string $zone
|
||||
*/
|
||||
public static function time(int $timestamp, string $zone): string {
|
||||
return (new \DateTimeImmutable('@' . $timestamp))->setTimezone(new \DateTimeZone($zone))->format('Y-m-d H:i');
|
||||
}
|
||||
}
|
||||
|
||||
@@ -214,7 +214,8 @@ final class Repository {
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore a snapshot (or only paths starting with $prefix) into a local directory.
|
||||
* Restore a snapshot (or only paths starting with $prefix) into a local directory. A $prefix
|
||||
* "dir/file/" also restores the file "dir/file" itself.
|
||||
*
|
||||
* @param bool $stripPrefix write "$target/<path without $prefix>" instead of
|
||||
* "$target/<full logical path>" (disaster recovery: restoring e.g. "data/" or
|
||||
@@ -253,7 +254,7 @@ final class Repository {
|
||||
$plan = new ReadAheadPlan(rtrim($target, '/') . '/.ngb-restore-spool-' . bin2hex(random_bytes(4)));
|
||||
try {
|
||||
foreach ($this->streamEntries($snapshotId) as $entry) {
|
||||
if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
|
||||
if (!self::underPrefix($entry['p'], $prefix)) {
|
||||
continue;
|
||||
}
|
||||
foreach ($entry['b'] as $id) {
|
||||
@@ -286,11 +287,15 @@ final class Repository {
|
||||
|
||||
private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan): int {
|
||||
$count = 0;
|
||||
$cleaned = [];
|
||||
foreach ($this->streamEntries($snapshotId) as $entry) {
|
||||
if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
|
||||
if (!self::underPrefix($entry['p'], $prefix)) {
|
||||
continue;
|
||||
}
|
||||
$relPath = ($stripPrefix && $prefix !== '') ? substr($entry['p'], strlen(rtrim($prefix, '/')) + 1) : $entry['p'];
|
||||
$relPath = $entry['p'];
|
||||
if ($stripPrefix && $prefix !== '') {
|
||||
$relPath = $entry['p'] === rtrim($prefix, '/') ? basename($entry['p']) : substr($entry['p'], strlen($prefix));
|
||||
}
|
||||
$dest = rtrim($target, '/') . '/' . $relPath;
|
||||
if (!is_dir(dirname($dest))) {
|
||||
mkdir(dirname($dest), 0700, true);
|
||||
@@ -301,6 +306,10 @@ final class Repository {
|
||||
if ($base === false || $dir === false || ($dir !== $base && !str_starts_with($dir, $base . '/')) || is_link($dest)) {
|
||||
throw new RepositoryException('Refusing to write outside the restore directory: ' . $entry['p']);
|
||||
}
|
||||
if (!isset($cleaned[$dir])) {
|
||||
self::removeStaleTemporaries($dir);
|
||||
$cleaned[$dir] = true;
|
||||
}
|
||||
// Write next to the destination and rename after all blobs verified: at most one file
|
||||
// is ever present twice.
|
||||
$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
|
||||
@@ -331,6 +340,30 @@ final class Repository {
|
||||
return $count;
|
||||
}
|
||||
|
||||
/** $prefix is "" (everything) or ends with "/"; the path without that "/" is a file to restore too. */
|
||||
private static function underPrefix(string $path, string $prefix): bool {
|
||||
return $prefix === '' || str_starts_with($path, $prefix) || $path === rtrim($prefix, '/');
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove what a restore that was stopped left in a directory this restore writes to:
|
||||
* "<name>.ngb-restore-<hex>" from here, ".<name>.ngb-restore-<hex>" from a restore through
|
||||
* Nextcloud. A files:scan would otherwise make them show up in Files.
|
||||
*/
|
||||
public static function removeStaleTemporaries(string $dir): void {
|
||||
$names = scandir($dir);
|
||||
foreach ($names === false ? [] : $names as $name) {
|
||||
$path = $dir . '/' . $name;
|
||||
if (self::isRestoreTemporary($name) && is_file($path) && !is_link($path)) {
|
||||
@unlink($path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public static function isRestoreTemporary(string $name): bool {
|
||||
return preg_match('/.\.ngb-restore-[0-9a-f]{6}(?:[0-9a-f]{6})?$/', $name) === 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Start writing a generated stream (e.g. a table dump) as blobs. Call flushPacks() when the
|
||||
* run is done, so the last pack is uploaded before anything references it.
|
||||
@@ -561,6 +594,31 @@ final class Repository {
|
||||
return isset($this->catalog->userExports[$manifestPath]);
|
||||
}
|
||||
|
||||
/**
|
||||
* The recorded export $ref names: a manifest path (users/<uid>/<exportId>) as is, or a bare
|
||||
* user id (also "users/<uid>") when that user has exactly one export here. Anything else
|
||||
* throws, with the exports to choose from in the message.
|
||||
*/
|
||||
public function resolveUserExport(string $ref): string {
|
||||
if ($this->isRecordedUserExport($ref)) {
|
||||
return $ref;
|
||||
}
|
||||
$rest = trim($ref, '/');
|
||||
if (str_starts_with($rest, 'users/')) {
|
||||
$rest = substr($rest, strlen('users/'));
|
||||
}
|
||||
$uid = explode('/', $rest)[0];
|
||||
$exports = $uid !== '' ? $this->listUserExports($uid) : [];
|
||||
if (!str_contains($rest, '/') && count($exports) === 1) {
|
||||
return $exports[0];
|
||||
}
|
||||
if ($exports === []) {
|
||||
throw new RepositoryException("$ref is not a recorded user export for this location, and $uid has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports");
|
||||
}
|
||||
$which = str_contains($rest, '/') ? "$ref is not a recorded user export for this location" : "$uid has " . count($exports) . ' exports here';
|
||||
throw new RepositoryException("$which; pass one of these (newest first):\n " . implode("\n ", $exports));
|
||||
}
|
||||
|
||||
/** Permanently delete a user export. Its blobs are freed by the next prune() like any other. */
|
||||
public function forgetUserExport(string $manifestPath): void {
|
||||
if (!isset($this->catalog->userExports[$manifestPath])) {
|
||||
@@ -845,7 +903,15 @@ final class Repository {
|
||||
try {
|
||||
foreach ($this->backend->getRanges($ranges) as $i => $data) {
|
||||
foreach ($parts[$i] as [$id, $at, $length]) {
|
||||
yield $this->decodeBlob($id, substr($data, $at, $length));
|
||||
try {
|
||||
$blob = $this->decodeBlob($id, substr($data, $at, $length));
|
||||
} catch (\Exception) {
|
||||
// a read that went wrong on the way (see loadBlob()): this blob on its own
|
||||
$blob = $this->loadBlob($id);
|
||||
// its pack stream is closed before the next range is read on the same connection
|
||||
$this->packReader->close();
|
||||
}
|
||||
yield $blob;
|
||||
$done++;
|
||||
}
|
||||
}
|
||||
@@ -895,8 +961,46 @@ final class Repository {
|
||||
} catch (BackendException) {
|
||||
throw $e;
|
||||
}
|
||||
$pack = $moved;
|
||||
}
|
||||
return $this->decodeBlob($id, $encrypted);
|
||||
try {
|
||||
return $this->decodeBlob($id, $encrypted);
|
||||
} catch (\Exception $damaged) {
|
||||
// Bytes that do not decrypt: damaged on the location, or a read that went wrong on the
|
||||
// way (a stream that lost its place: an SMB restore stopped after 17 hours like this).
|
||||
// Read exactly this blob once more, on a request of its own; only a second failure
|
||||
// means the data itself is damaged.
|
||||
$this->packReader->close();
|
||||
try {
|
||||
$again = $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length);
|
||||
} catch (\Exception) {
|
||||
throw $damaged;
|
||||
}
|
||||
$data = $this->decodeBlob($id, $again);
|
||||
if (function_exists('OCP\Log\logger')) {
|
||||
\OCP\Log\logger('ng_backup')->warning('NG Backup: blob ' . $id . ' did not decrypt as first read from pack ' . $pack . ' at ' . $offset . '; read again, it is fine. First read: ' . self::howItDiffered($encrypted, $again));
|
||||
}
|
||||
return $data;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* What a read that went wrong looked like next to the right bytes, for the log: where it starts
|
||||
* to differ, and whether the rest is zeros or the right bytes from another place (a stream that
|
||||
* lost its place).
|
||||
*/
|
||||
private static function howItDiffered(string $wrong, string $right): string {
|
||||
$at = strspn($wrong ^ $right, "\0");
|
||||
$text = strlen($wrong) . ' bytes, differ from byte ' . $at . ' of ' . strlen($right);
|
||||
$tail = substr($wrong, $at, 64);
|
||||
if ($tail === '') {
|
||||
return $text . ' (too short)';
|
||||
}
|
||||
if (trim($tail, "\0") === '') {
|
||||
return $text . ' (zeros)';
|
||||
}
|
||||
$found = strpos($right, $tail);
|
||||
return $text . ($found !== false ? ' (the right bytes from ' . ($found - $at) . ' bytes further on)' : ' (other bytes)');
|
||||
}
|
||||
|
||||
/** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */
|
||||
|
||||
@@ -34,6 +34,8 @@ use Psr\Log\LoggerInterface;
|
||||
*/
|
||||
final class BackupService {
|
||||
public const KIND_FULL = 'full';
|
||||
/** Characters of a snapshot label (the column holds 255) */
|
||||
public const LABEL_MAX = 100;
|
||||
|
||||
/** @var \WeakMap<Repository, string> the location settings each kept repository was opened with */
|
||||
private \WeakMap $openedWith;
|
||||
@@ -57,6 +59,10 @@ final class BackupService {
|
||||
}
|
||||
|
||||
public function start(Target $target, string $label = ''): Run {
|
||||
$label = trim($label);
|
||||
if (mb_strlen($label) > self::LABEL_MAX) {
|
||||
throw new \InvalidArgumentException('A label has at most ' . self::LABEL_MAX . ' characters');
|
||||
}
|
||||
if (!$this->keys->everConfirmed()) {
|
||||
throw new \RuntimeException('Confirm the recovery kit first (occ backup:key:confirm)');
|
||||
}
|
||||
|
||||
@@ -33,6 +33,9 @@ final class RestoreService {
|
||||
public const MODE_MERGE = 'merge';
|
||||
public const MODE_REPLACE = 'replace';
|
||||
|
||||
/** @var array<string, true> folders of the current restore already cleared of stale temporaries */
|
||||
private array $cleaned = [];
|
||||
|
||||
public function __construct(
|
||||
private TargetService $targets,
|
||||
private IRootFolder $root,
|
||||
@@ -133,6 +136,7 @@ final class RestoreService {
|
||||
$dest = $this->ensureFolder($userFolder, $parentRel);
|
||||
$names = $single ? [basename($rel) => $entries['']] : $entries;
|
||||
|
||||
$this->cleaned = [];
|
||||
$restored = 0;
|
||||
foreach ($names as $relPath => $entry) {
|
||||
$this->writeFile($repo, $dest, (string)$relPath, $entry);
|
||||
@@ -178,6 +182,10 @@ final class RestoreService {
|
||||
private function writeFile(Repository $repo, Folder $dest, string $relPath, array $entry): void {
|
||||
$dir = dirname($relPath);
|
||||
$folder = $dir === '.' ? $dest : $this->ensureFolder($dest, $dir);
|
||||
if (!isset($this->cleaned[$folder->getPath()])) {
|
||||
$this->removeStaleTemporaries($folder);
|
||||
$this->cleaned[$folder->getPath()] = true;
|
||||
}
|
||||
$name = basename($relPath);
|
||||
// Stage under a temporary name; only rename into place after every blob was verified.
|
||||
$tmpName = '.' . $name . '.ngb-restore-' . bin2hex(random_bytes(3));
|
||||
@@ -223,6 +231,19 @@ final class RestoreService {
|
||||
}
|
||||
}
|
||||
|
||||
/** See Repository::removeStaleTemporaries(): what a restore that was stopped left in this folder. */
|
||||
private function removeStaleTemporaries(Folder $folder): void {
|
||||
foreach ($folder->getDirectoryListing() as $node) {
|
||||
if ($node instanceof File && Repository::isRestoreTemporary($node->getName())) {
|
||||
try {
|
||||
$node->delete();
|
||||
} catch (\Throwable) {
|
||||
// in the way of nothing: the new temporary gets a name of its own
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private function trashExtras(Folder $folder, string $prefix, array $keep): int {
|
||||
$count = 0;
|
||||
foreach ($folder->getDirectoryListing() as $node) {
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
|
||||
namespace OCA\NgBackup\Service;
|
||||
|
||||
use OCA\NgBackup\AppInfo\Application;
|
||||
use OCP\IAppConfig;
|
||||
use OCP\IConfig;
|
||||
use OCP\IGroupManager;
|
||||
|
||||
/**
|
||||
* The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a
|
||||
* zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone
|
||||
* is converted once to the zone of the server or of an administrator, at today's offset: it then
|
||||
* shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
|
||||
*/
|
||||
final class Schedule {
|
||||
public const TIME_KEY = 'schedule_time';
|
||||
public const ZONE_KEY = 'schedule_timezone';
|
||||
|
||||
public function __construct(
|
||||
private IAppConfig $appConfig,
|
||||
private IConfig $config,
|
||||
private IGroupManager $groups,
|
||||
) {
|
||||
}
|
||||
|
||||
/** @return array{time: string, timezone: non-empty-string} time "" = no schedule */
|
||||
public function get(): array {
|
||||
$time = $this->appConfig->getValueString(Application::APP_ID, self::TIME_KEY, '');
|
||||
$zone = $this->appConfig->getValueString(Application::APP_ID, self::ZONE_KEY, '');
|
||||
if ($zone === '' || !self::validZone($zone)) {
|
||||
$zone = $this->guessZone();
|
||||
if ($time !== '' && self::validTime($time)) {
|
||||
$time = self::convert($time, 'UTC', $zone);
|
||||
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
|
||||
}
|
||||
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
|
||||
}
|
||||
return ['time' => $time, 'timezone' => $zone];
|
||||
}
|
||||
|
||||
/** @throws \InvalidArgumentException for a malformed time or an unknown zone */
|
||||
public function set(string $time, string $zone): void {
|
||||
if ($time !== '' && !self::validTime($time)) {
|
||||
throw new \InvalidArgumentException('Use HH:MM or empty');
|
||||
}
|
||||
if (!self::validZone($zone)) {
|
||||
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
|
||||
}
|
||||
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
|
||||
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
|
||||
}
|
||||
|
||||
/** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */
|
||||
public function todayAt(?int $now = null): ?int {
|
||||
['time' => $time, 'timezone' => $zone] = $this->get();
|
||||
if ($time === '' || !self::validTime($time)) {
|
||||
return null;
|
||||
}
|
||||
$tz = new \DateTimeZone($zone);
|
||||
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d');
|
||||
return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp();
|
||||
}
|
||||
|
||||
/**
|
||||
* "HH:MM" in zone $from as "HH:MM" in zone $to, at today's offsets.
|
||||
*
|
||||
* @param non-empty-string $from
|
||||
* @param non-empty-string $to
|
||||
*/
|
||||
public static function convert(string $time, string $from, string $to, ?int $now = null): string {
|
||||
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone(new \DateTimeZone($from))->format('Y-m-d');
|
||||
return (new \DateTimeImmutable($day . ' ' . $time, new \DateTimeZone($from)))->setTimezone(new \DateTimeZone($to))->format('H:i');
|
||||
}
|
||||
|
||||
public static function validTime(string $time): bool {
|
||||
return preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time) === 1;
|
||||
}
|
||||
|
||||
/** @psalm-assert-if-true non-empty-string $zone */
|
||||
public static function validZone(string $zone): bool {
|
||||
return in_array($zone, \DateTimeZone::listIdentifiers(), true);
|
||||
}
|
||||
|
||||
/**
|
||||
* The server's default_timezone when set, else the first administrator's, else UTC. A server
|
||||
* zone of UTC is passed over on purpose: it is mostly what an image or installer left there,
|
||||
* while the administrator who set the time read it as their own (#19).
|
||||
*
|
||||
* @return non-empty-string
|
||||
*/
|
||||
private function guessZone(): string {
|
||||
$zone = $this->config->getSystemValueString('default_timezone', '');
|
||||
if ($zone !== '' && $zone !== 'UTC' && self::validZone($zone)) {
|
||||
return $zone;
|
||||
}
|
||||
foreach ($this->groups->get('admin')?->getUsers() ?? [] as $user) {
|
||||
$zone = $this->config->getUserValue($user->getUID(), 'core', 'timezone', '');
|
||||
if ($zone !== '' && self::validZone($zone)) {
|
||||
return $zone;
|
||||
}
|
||||
}
|
||||
return 'UTC';
|
||||
}
|
||||
}
|
||||
@@ -110,11 +110,6 @@
|
||||
<code><![CDATA[ask]]></code>
|
||||
</UndefinedInterfaceMethod>
|
||||
</file>
|
||||
<file src="lib/Command/Trash.php">
|
||||
<RedundantCastGivenDocblockType>
|
||||
<code><![CDATA[(string)($info['time'] ?? '')]]></code>
|
||||
</RedundantCastGivenDocblockType>
|
||||
</file>
|
||||
<file src="lib/Controller/AdminApiController.php">
|
||||
<InvalidOperand>
|
||||
<code><![CDATA[$maxGb * 1073741824]]></code>
|
||||
|
||||
+44
-3
@@ -7,7 +7,7 @@
|
||||
<NcSettingsSection :name="t('ng_backup', 'NG Backup')"
|
||||
:description="t('ng_backup', 'Encrypted, incremental backups of this Nextcloud (database, configuration and data). Backup locations are never mounted and are not visible in Files.')">
|
||||
<p class="ngb-feedback">
|
||||
{{ t('ng_backup', 'NG Backup is in beta. We would like to hear how it works for you, also when everything went fine.') }}
|
||||
{{ t('ng_backup', 'We would like to hear how NG Backup works for you, also when everything went fine.') }}
|
||||
<a :href="feedbackUrl" target="_blank" rel="noopener noreferrer">{{ t('ng_backup', 'Share your experience ↗') }}</a>
|
||||
</p>
|
||||
<NcLoadingIcon v-if="!status" :size="32" />
|
||||
@@ -18,9 +18,18 @@
|
||||
<div class="ngb-tabs">
|
||||
<NcButton :variant="tab === 'backup' ? 'primary' : 'secondary'" @click="tab = 'backup'">{{ t('ng_backup', 'Backup and restore') }}</NcButton>
|
||||
<NcButton :variant="tab === 'settings' ? 'primary' : 'secondary'" @click="tab = 'settings'">{{ t('ng_backup', 'Settings') }}</NcButton>
|
||||
<NcButton :variant="tab === 'docs' ? 'primary' : 'secondary'" @click="tab = 'docs'">{{ t('ng_backup', 'Docs') }}</NcButton>
|
||||
</div>
|
||||
|
||||
<template v-if="tab === 'settings'">
|
||||
<NcSettingsSection v-if="tab === 'docs'" :name="t('ng_backup', 'Documentation')">
|
||||
<ul class="ngb-docs">
|
||||
<li v-for="doc in docs" :key="doc.url">
|
||||
<a :href="doc.url" target="_blank" rel="noopener noreferrer">{{ doc.name }} ↗</a>
|
||||
<span>{{ doc.description }}</span>
|
||||
</li>
|
||||
</ul>
|
||||
</NcSettingsSection>
|
||||
<template v-else-if="tab === 'settings'">
|
||||
<KeySetup :status="status" @changed="refresh" />
|
||||
<Locations v-if="status.key.everConfirmed" :status="status" @changed="refresh" />
|
||||
</template>
|
||||
@@ -52,6 +61,25 @@ import RestoreBrowser from './components/RestoreBrowser.vue'
|
||||
// Opens the experience report form in .github/ISSUE_TEMPLATE
|
||||
const feedbackUrl = 'https://github.com/dillardblom/ng_backup/issues/new?template=experience.yml'
|
||||
|
||||
const repoUrl = 'https://github.com/dillardblom/ng_backup'
|
||||
const docs = [
|
||||
{
|
||||
name: t('ng_backup', 'How-to'),
|
||||
description: t('ng_backup', 'Step by step: from first install to a tested restore.'),
|
||||
url: repoUrl + '/blob/main/docs/HOWTO.md',
|
||||
},
|
||||
{
|
||||
name: t('ng_backup', 'Disaster recovery'),
|
||||
description: t('ng_backup', 'Your server is gone: get Nextcloud back on a new server from a backup location.'),
|
||||
url: repoUrl + '/blob/main/docs/DISASTER-RECOVERY.md',
|
||||
},
|
||||
{
|
||||
name: t('ng_backup', 'Roadmap'),
|
||||
description: t('ng_backup', 'What is planned for the next versions.'),
|
||||
url: repoUrl + '#roadmap',
|
||||
},
|
||||
]
|
||||
|
||||
const status = ref(null)
|
||||
const allRuns = ref(false) // runs older than the last days too
|
||||
const error = ref('')
|
||||
@@ -89,8 +117,10 @@ onBeforeUnmount(() => clearTimeout(timer))
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
/* NcSettingsSection sets its own width (at most 900px): the runs table needs more */
|
||||
.ng-backup :deep(.settings-section) {
|
||||
max-width: 900px;
|
||||
width: min(1400px, 100% - var(--default-grid-baseline) * 7 * 2);
|
||||
max-width: none;
|
||||
}
|
||||
.ngb-feedback {
|
||||
color: var(--color-text-maxcontrast);
|
||||
@@ -100,6 +130,17 @@ onBeforeUnmount(() => clearTimeout(timer))
|
||||
color: var(--color-primary-element);
|
||||
text-decoration: underline;
|
||||
}
|
||||
.ngb-docs li {
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
.ngb-docs a {
|
||||
color: var(--color-primary-element);
|
||||
text-decoration: underline;
|
||||
margin-right: 8px;
|
||||
}
|
||||
.ngb-docs span {
|
||||
color: var(--color-text-maxcontrast);
|
||||
}
|
||||
.ngb-tabs {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
|
||||
+1
-1
@@ -33,7 +33,7 @@ export default {
|
||||
testTarget: (id) => call('post', `/targets/${id}/test`),
|
||||
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
|
||||
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
|
||||
setSchedule: (time) => call('put', '/schedule', { time }),
|
||||
setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }),
|
||||
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
|
||||
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
|
||||
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
|
||||
|
||||
@@ -6,10 +6,12 @@
|
||||
<NcSettingsSection :name="t('ng_backup', 'Backups')"
|
||||
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
|
||||
<div class="ngb-row">
|
||||
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, empty = off)')" />
|
||||
<NcButton :disabled="schedule === status.schedule" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
|
||||
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, {zone}, empty = off)', { zone: browserZone })" />
|
||||
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
|
||||
</div>
|
||||
<div class="ngb-row">
|
||||
<NcTextField v-model="label" class="ngb-label" :maxlength="100" :label="t('ng_backup', 'Label (optional)')"
|
||||
:placeholder="t('ng_backup', 'e.g. before the upgrade to Nextcloud 35')" />
|
||||
<NcButton v-for="target in status.targets" :key="target.id" :disabled="isRunning(target.id)" @click="start(target)">
|
||||
{{ isRunning(target.id) ? t('ng_backup', 'Backup to {name} running …', { name: target.name }) : t('ng_backup', 'Back up now to {name}', { name: target.name }) }}
|
||||
</NcButton>
|
||||
@@ -24,9 +26,23 @@
|
||||
<li v-for="item in trash[target.id]" :key="item.id">
|
||||
{{ item.time ? formatDateTime(item.time) : item.id }}{{ item.label ? ' – ' + item.label : '' }}
|
||||
<span class="ngb-muted">{{ t('ng_backup', 'removed by {by}, deleted after {date}', { by: item.by, date: formatDate(item.deleteAfter) }) }}</span>
|
||||
<button v-if="item.by === RETENTION"
|
||||
class="ngb-info"
|
||||
:aria-expanded="retentionInfo ? 'true' : 'false'"
|
||||
:aria-label="t('ng_backup', 'What does this mean?')"
|
||||
:title="t('ng_backup', 'What does this mean?')"
|
||||
@click="retentionInfo = !retentionInfo">
|
||||
ⓘ
|
||||
</button>
|
||||
<NcButton variant="tertiary" @click="untrash(target, item)">{{ t('ng_backup', 'Restore') }}</NcButton>
|
||||
</li>
|
||||
</ul>
|
||||
<NcNoteCard v-if="retentionInfo && trash[target.id].some(item => item.by === RETENTION)" type="info">
|
||||
{{ n('ng_backup',
|
||||
'The retention policy keeps the newest restore points, and for older days, weeks and months only the last restore point of each. It moves the others to this trash, where they stay for %n day before they are deleted permanently. Until then you can bring one back with "Restore". Nothing was lost by accident.',
|
||||
'The retention policy keeps the newest restore points, and for older days, weeks and months only the last restore point of each. It moves the others to this trash, where they stay for %n days before they are deleted permanently. Until then you can bring one back with "Restore". Nothing was lost by accident.',
|
||||
status.key.deleteDelayDays) }}
|
||||
</NcNoteCard>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
@@ -79,16 +95,24 @@ import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
||||
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
||||
import api from '../api.js'
|
||||
import { formatDateTime, formatDate, formatSize } from '../format.js'
|
||||
import { formatDateTime, formatDate, formatSize, toZone } from '../format.js'
|
||||
import { confirmPassword } from '@nextcloud/password-confirmation'
|
||||
import { reactive } from 'vue'
|
||||
|
||||
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
|
||||
const emit = defineEmits(['changed', 'show-all-runs'])
|
||||
const schedule = ref(props.status.schedule)
|
||||
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
|
||||
// zone is shown in this one, so it reads as the moment it runs here.
|
||||
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
|
||||
const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s
|
||||
const schedule = ref(shown(props.status.schedule))
|
||||
const error = ref('')
|
||||
watch(() => props.status.schedule, (s) => { schedule.value = s })
|
||||
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
|
||||
|
||||
const label = ref('')
|
||||
// "by" of a restore point the weekly cleanup removed (PruneService)
|
||||
const RETENTION = 'retention policy'
|
||||
const retentionInfo = ref(false)
|
||||
const trash = reactive({})
|
||||
async function loadTrash() {
|
||||
for (const target of props.status.targets) {
|
||||
@@ -121,7 +145,8 @@ const targetName = (id) => props.status.targets.find(tg => tg.id === id)?.name ?
|
||||
async function start(target) {
|
||||
error.value = ''
|
||||
try {
|
||||
await api.startBackup(target.id, '')
|
||||
await api.startBackup(target.id, label.value.trim())
|
||||
label.value = ''
|
||||
emit('changed')
|
||||
} catch (e) {
|
||||
error.value = e.message
|
||||
@@ -131,7 +156,7 @@ async function start(target) {
|
||||
async function saveSchedule() {
|
||||
error.value = ''
|
||||
try {
|
||||
await api.setSchedule(schedule.value.trim())
|
||||
await api.setSchedule(schedule.value.trim(), browserZone)
|
||||
emit('changed')
|
||||
} catch (e) {
|
||||
error.value = e.message
|
||||
@@ -142,6 +167,8 @@ async function saveSchedule() {
|
||||
<style scoped>
|
||||
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
|
||||
.ngb-time { max-width: 260px; }
|
||||
.ngb-label { max-width: 360px; }
|
||||
.ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); }
|
||||
.ngb-table-wrap { overflow-x: auto; }
|
||||
.ngb-table { width: 100%; border-collapse: collapse; }
|
||||
.ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); }
|
||||
|
||||
@@ -26,3 +26,30 @@ export function formatSize(bytes) {
|
||||
}
|
||||
return i === 0 ? `${b} B` : `${b.toFixed(1)} ${units[i]}`
|
||||
}
|
||||
|
||||
/**
|
||||
* A time of day "HH:MM" in zone `from`, as "HH:MM" in zone `to`, at today's offsets.
|
||||
*
|
||||
* @param {string} hhmm the time
|
||||
* @param {string} from IANA zone it is in
|
||||
* @param {string} to IANA zone to show it in
|
||||
* @param {Date} now the day to convert on (today)
|
||||
* @return {string}
|
||||
*/
|
||||
export function toZone(hhmm, from, to, now = new Date()) {
|
||||
const [h, m] = hhmm.split(':').map(Number)
|
||||
const parts = (zone, date) => Object.fromEntries(new Intl.DateTimeFormat('en-GB', {
|
||||
timeZone: zone, hourCycle: 'h23', year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute: '2-digit',
|
||||
}).formatToParts(date).map((p) => [p.type, p.value]))
|
||||
const offsetAt = (at) => {
|
||||
const seen = parts(from, new Date(at))
|
||||
return Date.UTC(+seen.year, +seen.month - 1, +seen.day, +seen.hour, +seen.minute) - at
|
||||
}
|
||||
const today = parts(from, now)
|
||||
const wall = Date.UTC(+today.year, +today.month - 1, +today.day, h, m)
|
||||
// the offset at the wall time itself: on a day the clocks change it differs from the one at
|
||||
// midnight, so it is looked up a second time at the moment found first
|
||||
const at = wall - offsetAt(wall - offsetAt(wall))
|
||||
const there = parts(to, new Date(at))
|
||||
return `${there.hour}:${there.minute}`
|
||||
}
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Tests\Unit\Backend;
|
||||
|
||||
use OCA\NgBackup\Backend\ExternalStorageFactory;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
|
||||
/** ExternalStorageFactory::normalizePrivateKey(): keys that phpseclib 2 read, for phpseclib 3 (#24). */
|
||||
class ExternalStorageFactoryTest extends TestCase {
|
||||
/** An OpenSSH-format key laid out as ssh-keygen writes it; the body is random, not a real key. */
|
||||
private static function key(): string {
|
||||
$body = base64_encode(random_bytes(1000));
|
||||
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
|
||||
}
|
||||
|
||||
public function testRestoresLostLineBreaks(): void {
|
||||
$key = self::key();
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey($key));
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", ' ', $key)), 'newlines as spaces');
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", '', $key)), 'no newlines');
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", "\r\n", $key)), 'CRLF');
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(rtrim($key)), 'no trailing newline');
|
||||
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(preg_replace('/^/m', ' ', $key)), 'indented');
|
||||
}
|
||||
|
||||
public function testLeavesOtherFormatsAlone(): void {
|
||||
$pem = "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC,FF44E0A94036766C\n\nAAAA\n-----END RSA PRIVATE KEY-----\n";
|
||||
$this->assertSame($pem, ExternalStorageFactory::normalizePrivateKey($pem));
|
||||
$this->assertSame('not a key', ExternalStorageFactory::normalizePrivateKey('not a key'));
|
||||
$odd = "-----BEGIN OPENSSH PRIVATE KEY-----\\nAAAA\\n-----END OPENSSH PRIVATE KEY-----";
|
||||
$this->assertSame($odd, ExternalStorageFactory::normalizePrivateKey($odd), 'not base64 inside: unchanged');
|
||||
}
|
||||
}
|
||||
@@ -14,4 +14,11 @@ class FormatTest extends TestCase {
|
||||
$this->assertSame('1d 00:00:00', Format::duration(86400));
|
||||
$this->assertSame('1d 12:10:13', Format::duration(86400 + 12 * 3600 + 10 * 60 + 13));
|
||||
}
|
||||
|
||||
public function testTimeInZone(): void {
|
||||
$t = (int)strtotime('2026-10-19 13:12:00 UTC');
|
||||
$this->assertSame('2026-10-19 13:12', Format::time($t, 'UTC'));
|
||||
$this->assertSame('2026-10-19 15:12', Format::time($t, 'Europe/Berlin')); // summer time
|
||||
$this->assertSame('2026-12-01 01:00', Format::time((int)strtotime('2026-12-01 00:00:00 UTC'), 'Europe/Amsterdam'));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||
|
||||
use OCA\NgBackup\Backend\IBackend;
|
||||
use OCA\NgBackup\Backend\LocalBackend;
|
||||
use OCA\NgBackup\Crypto\KeyRing;
|
||||
use OCA\NgBackup\Repository\Repository;
|
||||
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
|
||||
/**
|
||||
* A pack stream that hands out wrong bytes once (an SMB restore stopped after 17 hours with
|
||||
* "Decryption failed"): the blob is read once more on a request of its own; only data that is
|
||||
* damaged on the location stops the restore.
|
||||
*/
|
||||
class DamagedReadTest extends TestCase {
|
||||
use TempDirTrait;
|
||||
|
||||
private string $root;
|
||||
private KeyRing $keys;
|
||||
private string $src;
|
||||
private string $snapshot;
|
||||
|
||||
protected function setUp(): void {
|
||||
$this->root = $this->tempDir();
|
||||
$this->keys = KeyRing::generate();
|
||||
$this->src = $this->tempDir();
|
||||
for ($i = 0; $i < 6; $i++) {
|
||||
file_put_contents("$this->src/f$i", random_bytes(3 * 1048576));
|
||||
}
|
||||
$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
|
||||
$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
|
||||
}
|
||||
|
||||
public function testABlobThatCameInDamagedOnceIsReadAgain(): void {
|
||||
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1);
|
||||
$target = $this->tempDir();
|
||||
|
||||
$this->assertSame(6, Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $target));
|
||||
|
||||
$this->assertSame(self::hashTree($this->src), self::hashTree($target));
|
||||
$this->assertSame(0, $backend->streams, 'the damaged read happened');
|
||||
}
|
||||
|
||||
public function testATableDumpThatCameInDamagedOnceIsReadAgain(): void {
|
||||
$repo = Repository::openWithKey(new LocalBackend($this->root), $this->keys);
|
||||
$stats = [];
|
||||
$writer = $repo->blobWriter($stats);
|
||||
$lines = [];
|
||||
for ($i = 0; $i < 20000; $i++) {
|
||||
$lines[] = json_encode(['i' => $i, 'pad' => str_repeat('x', $i % 400)]);
|
||||
}
|
||||
$writer->write(implode("\n", $lines) . "\n");
|
||||
$blobs = $writer->finish();
|
||||
$repo->flushPacks();
|
||||
$backend = new GarblingBackend(new LocalBackend($this->root), ranges: 1);
|
||||
|
||||
$this->assertSame($lines, iterator_to_array(Repository::openWithKey($backend, $this->keys)->readLines($blobs), false));
|
||||
$this->assertSame(0, $backend->ranges, 'the damaged read happened');
|
||||
}
|
||||
|
||||
public function testDataDamagedOnTheLocationStillStopsTheRestore(): void {
|
||||
$backend = new GarblingBackend(new LocalBackend($this->root), streams: 1000, ranges: 1000);
|
||||
|
||||
$this->expectException(\Exception::class);
|
||||
Repository::openWithKey($backend, $this->keys)->restore($this->snapshot, $this->tempDir());
|
||||
}
|
||||
}
|
||||
|
||||
/** Pack reads that flip a byte: the first $streams pack streams and $ranges range reads. */
|
||||
final class GarblingBackend implements IBackend {
|
||||
use \OCA\NgBackup\Backend\GetsOneByOne;
|
||||
|
||||
public function __construct(
|
||||
private IBackend $inner,
|
||||
public int $streams = 0,
|
||||
public int $ranges = 0,
|
||||
) {
|
||||
}
|
||||
|
||||
public function put(string $path, $stream): void {
|
||||
$this->inner->put($path, $stream);
|
||||
}
|
||||
|
||||
public function get(string $path) {
|
||||
$fh = $this->inner->get($path);
|
||||
if (!str_starts_with($path, 'packs/') || $this->streams <= 0) {
|
||||
return $fh;
|
||||
}
|
||||
$this->streams--;
|
||||
$data = stream_get_contents($fh);
|
||||
fclose($fh);
|
||||
for ($k = 100; $k < strlen($data); $k += 65536) {
|
||||
$data[$k] = chr(ord($data[$k]) ^ 0x55);
|
||||
}
|
||||
$out = fopen('php://temp', 'w+b');
|
||||
fwrite($out, $data);
|
||||
rewind($out);
|
||||
return $out;
|
||||
}
|
||||
|
||||
public function getRange(string $path, int $offset, int $length): string {
|
||||
$data = $this->inner->getRange($path, $offset, $length);
|
||||
if (str_starts_with($path, 'packs/') && $this->ranges > 0) {
|
||||
$this->ranges--;
|
||||
$data[0] = chr(ord($data[0]) ^ 0x55);
|
||||
}
|
||||
return $data;
|
||||
}
|
||||
|
||||
public function exists(string $path): bool {
|
||||
return $this->inner->exists($path);
|
||||
}
|
||||
|
||||
public function move(string $from, string $to): void {
|
||||
$this->inner->move($from, $to);
|
||||
}
|
||||
|
||||
public function list(string $prefix): array {
|
||||
return $this->inner->list($prefix);
|
||||
}
|
||||
|
||||
public function listFiles(string $dir): array {
|
||||
return $this->inner->listFiles($dir);
|
||||
}
|
||||
|
||||
public function delete(string $path): void {
|
||||
$this->inner->delete($path);
|
||||
}
|
||||
}
|
||||
@@ -87,6 +87,46 @@ class RepositoryTest extends TestCase {
|
||||
$this->assertSame(10, $repo->restore($r['snapshot'], $target, 'docs/d1/'));
|
||||
}
|
||||
|
||||
public function testRestoreOfOneFile(): void {
|
||||
$repo = Repository::init($this->backend, 'pw');
|
||||
$r = $repo->backupDirectory($this->src);
|
||||
$target = $this->tempDir();
|
||||
// occ backup:restore --to-directory passes the path with a "/" appended
|
||||
$this->assertSame(1, $repo->restore($r['snapshot'], $target, 'docs/d1/note-1.txt/'));
|
||||
$this->assertSame(file_get_contents($this->src . '/docs/d1/note-1.txt'), file_get_contents($target . '/docs/d1/note-1.txt'));
|
||||
$this->assertSame(0, $repo->restore($r['snapshot'], $this->tempDir(), 'docs/d1/note-1/'), 'a prefix of a name is not that file');
|
||||
|
||||
$stripped = $this->tempDir();
|
||||
$this->assertSame(1, $repo->restore($r['snapshot'], $stripped, 'docs/d1/note-1.txt/', true));
|
||||
$this->assertFileExists($stripped . '/note-1.txt');
|
||||
}
|
||||
|
||||
public function testRestoreRemovesTemporariesOfAStoppedRestore(): void {
|
||||
$repo = Repository::init($this->backend, 'pw');
|
||||
$r = $repo->backupDirectory($this->src);
|
||||
$target = $this->tempDir();
|
||||
mkdir($target . '/docs/d1', 0700, true);
|
||||
mkdir($target . '/elsewhere', 0700, true);
|
||||
$stale = [
|
||||
$target . '/docs/d1/note-1.txt.ngb-restore-0123456789ab', // from a restore to a directory
|
||||
$target . '/docs/d1/.note-4.txt.ngb-restore-a1b2c3', // from a restore through Nextcloud
|
||||
];
|
||||
$kept = [
|
||||
$target . '/docs/d1/notes.ngb-restore-draft.txt', // not a temporary of ours
|
||||
$target . '/elsewhere/x.ngb-restore-0123456789ab', // not a directory this restore writes to
|
||||
];
|
||||
foreach ([...$stale, ...$kept] as $f) {
|
||||
file_put_contents($f, 'partial');
|
||||
}
|
||||
$repo->restore($r['snapshot'], $target, 'docs/');
|
||||
foreach ($stale as $f) {
|
||||
$this->assertFileDoesNotExist($f);
|
||||
}
|
||||
foreach ($kept as $f) {
|
||||
$this->assertFileExists($f);
|
||||
}
|
||||
}
|
||||
|
||||
public function testWrongPassphrase(): void {
|
||||
Repository::init($this->backend, 'pw');
|
||||
$this->expectException(CryptoException::class);
|
||||
|
||||
@@ -106,4 +106,51 @@ class UserExportTest extends TestCase {
|
||||
$this->expectExceptionMessage('missing on the location');
|
||||
$repo->listUserExports();
|
||||
}
|
||||
|
||||
public function testResolveTakesARecordedPathAsIs(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$path = 'users/alice/20261005T120000Z-aaaa0001';
|
||||
$this->putManifest($repo, $path);
|
||||
|
||||
$this->assertSame($path, $repo->resolveUserExport($path));
|
||||
}
|
||||
|
||||
public function testResolveTakesAUserIdWithOneExport(): void {
|
||||
// Issue #22: "backup:user:restore <location> New_User" was rejected as not a recorded export.
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$path = 'users/alice/20261005T120000Z-aaaa0001';
|
||||
$this->putManifest($repo, $path);
|
||||
$this->putManifest($repo, 'users/alice2/20261005T120000Z-aaaa0002');
|
||||
|
||||
$this->assertSame($path, $repo->resolveUserExport('alice'));
|
||||
$this->assertSame($path, $repo->resolveUserExport('users/alice/'));
|
||||
}
|
||||
|
||||
public function testResolveListsTheExportsOfAUserWithMore(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
|
||||
$this->putManifest($repo, 'users/alice/20261006T120000Z-aaaa0002');
|
||||
|
||||
$this->expectException(RepositoryException::class);
|
||||
$this->expectExceptionMessage("alice has 2 exports here; pass one of these (newest first):\n users/alice/20261006T120000Z-aaaa0002\n users/alice/20261005T120000Z-aaaa0001");
|
||||
$repo->resolveUserExport('alice');
|
||||
}
|
||||
|
||||
public function testResolveListsTheExportsOfTheUserOfAWrongPath(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
|
||||
|
||||
$this->expectException(RepositoryException::class);
|
||||
$this->expectExceptionMessage("users/alice/typo is not a recorded user export for this location; pass one of these (newest first):\n users/alice/20261005T120000Z-aaaa0001");
|
||||
$repo->resolveUserExport('users/alice/typo');
|
||||
}
|
||||
|
||||
public function testResolveNamesTheExpectedFormForAUserWithoutExports(): void {
|
||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
|
||||
|
||||
$this->expectException(RepositoryException::class);
|
||||
$this->expectExceptionMessage('bob has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports');
|
||||
$repo->resolveUserExport('bob');
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
<?php
|
||||
|
||||
declare(strict_types=1);
|
||||
|
||||
namespace OCA\NgBackup\Tests\Unit\Service;
|
||||
|
||||
use OCA\NgBackup\Service\Schedule;
|
||||
use OCP\IAppConfig;
|
||||
use OCP\IConfig;
|
||||
use OCP\IGroup;
|
||||
use OCP\IGroupManager;
|
||||
use OCP\IUser;
|
||||
use PHPUnit\Framework\TestCase;
|
||||
|
||||
/** The daily time in a time zone (issue #19), and the conversion of a time saved without one. */
|
||||
class ScheduleTest extends TestCase {
|
||||
private array $values = [];
|
||||
|
||||
private function schedule(string $serverZone = '', string $adminZone = ''): Schedule {
|
||||
$app = $this->createMock(IAppConfig::class);
|
||||
$app->method('getValueString')->willReturnCallback(fn (string $a, string $k, string $d = '') => $this->values[$k] ?? $d);
|
||||
$app->method('setValueString')->willReturnCallback(function (string $a, string $k, string $v): bool {
|
||||
$this->values[$k] = $v;
|
||||
return true;
|
||||
});
|
||||
$config = $this->createMock(IConfig::class);
|
||||
$config->method('getSystemValueString')->willReturn($serverZone);
|
||||
$config->method('getUserValue')->willReturn($adminZone);
|
||||
$user = $this->createMock(IUser::class);
|
||||
$user->method('getUID')->willReturn('admin');
|
||||
$group = $this->createMock(IGroup::class);
|
||||
$group->method('getUsers')->willReturn([$user]);
|
||||
$groups = $this->createMock(IGroupManager::class);
|
||||
$groups->method('get')->willReturn($group);
|
||||
return new Schedule($app, $config, $groups);
|
||||
}
|
||||
|
||||
public function testATimeSavedWithoutAZoneShowsAsItRanInTheAdministratorsZone(): void {
|
||||
$this->values = [Schedule::TIME_KEY => '02:30'];
|
||||
|
||||
$s = $this->schedule(adminZone: 'Europe/Berlin')->get();
|
||||
|
||||
// it ran at 02:30 UTC: in summer that is 04:30 in Berlin, in winter 03:30
|
||||
$summer = Schedule::convert('02:30', 'UTC', 'Europe/Berlin', strtotime('2026-07-01 12:00 UTC'));
|
||||
$this->assertSame('04:30', $summer);
|
||||
$this->assertSame('Europe/Berlin', $s['timezone']);
|
||||
$this->assertSame(Schedule::convert('02:30', 'UTC', 'Europe/Berlin'), $s['time']);
|
||||
}
|
||||
|
||||
public function testTheServersDefaultZoneComesFirst(): void {
|
||||
$this->values = [Schedule::TIME_KEY => '01:00'];
|
||||
$this->assertSame('America/Toronto', $this->schedule('America/Toronto', 'Europe/Berlin')->get()['timezone']);
|
||||
}
|
||||
|
||||
public function testWithoutAnyZoneItStaysUtc(): void {
|
||||
$this->values = [Schedule::TIME_KEY => '02:30'];
|
||||
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
|
||||
}
|
||||
|
||||
public function testTodayAtIsTheMomentInTheSchedulesZone(): void {
|
||||
$this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin'];
|
||||
$at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC'));
|
||||
$this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC');
|
||||
}
|
||||
|
||||
public function testAnUnknownZoneOrABadTimeIsRefused(): void {
|
||||
$s = $this->schedule();
|
||||
$this->expectException(\InvalidArgumentException::class);
|
||||
$s->set('03:00', 'Mars/Olympus');
|
||||
}
|
||||
}
|
||||
@@ -47,6 +47,12 @@ if [ "$LEGACY" = legacy ]; then
|
||||
fi
|
||||
|
||||
run_on() { docker exec -u www-data "$1" php occ "${@:2}"; }
|
||||
# The app store sends the whole app list (~38 MB) and Nextcloud gives up after 120 s: on a slow day
|
||||
# an install fails for reasons outside this test. Up to three tries.
|
||||
install_app() { # container app
|
||||
for i in 1 2 3; do run_on "$1" app:install "$2" >/dev/null 2>&1 && return 0; sleep 20; done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Database settings for an instance: SQLite inside the container, or its own PostgreSQL container
|
||||
db_env() {
|
||||
@@ -109,7 +115,7 @@ if [ "$DB" != sqlite ]; then
|
||||
# while the restore loads tables in name order, accounts before mailboxes.
|
||||
docker run -d --name ngb-frt-imap --network "$NET" -e MAILNAME=mail.domain.tld -e MAIL_ADDRESS=user@domain.tld \
|
||||
-e MAIL_PASS=mypassword ghcr.io/christophwurst/docker-imap-devel:latest >/dev/null
|
||||
run_on ngb-frt-origin app:install mail >/dev/null || fail "install mail on the origin"
|
||||
install_app ngb-frt-origin mail || fail "install mail on the origin (app store unreachable?)"
|
||||
run_on ngb-frt-origin config:system:set app.mail.verify-tls-peer --type=boolean --value=false >/dev/null
|
||||
sleep 10
|
||||
run_on ngb-frt-origin mail:account:create frtest Test user@domain.tld ngb-frt-imap 993 ssl user@domain.tld mypassword \
|
||||
@@ -118,7 +124,9 @@ if [ "$DB" != sqlite ]; then
|
||||
[ "$(sql origin 'select count(*) from oc_mail_accounts where drafts_mailbox_id is not null')" = 1 ] \
|
||||
|| fail "the mail account does not point at its mailboxes"
|
||||
# Deck installed and then switched off: its tables stay in the database and in the backup
|
||||
run_on ngb-frt-origin app:install deck >/dev/null || fail "install deck on the origin"
|
||||
install_app ngb-frt-origin deck || fail "install deck on the origin (app store unreachable?)"
|
||||
# a 4-byte character (emoji) in a table the restore has to create: utf8mb4 on MySQL/MariaDB
|
||||
sql origin "insert into oc_deck_boards (title, owner, color, archived, deleted_at, last_modified) values ('Plan 📌', 'frtest', '0087C5', false, 0, 0)" >/dev/null
|
||||
run_on ngb-frt-origin app:disable deck >/dev/null
|
||||
ok "origin has a synced Mail account and the tables of a disabled app"
|
||||
fi
|
||||
@@ -165,6 +173,20 @@ if [ "$LEGACY" = legacy ] && [ "$DB" != sqlite ]; then
|
||||
ok "older backup refused without --skip-missing-tables, missing tables named"
|
||||
RESTORE_OPTS+=(--skip-missing-tables)
|
||||
fi
|
||||
if [ "$DB" = sqlite ]; then
|
||||
# Another process writing while the restore starts (a cron job that was already running, or
|
||||
# the installer still finishing after it says "installed"): the restore must wait for it, not
|
||||
# stop with "database is locked". Short write transactions on a table of its own, until the
|
||||
# restore is done (/tmp/frt-side-stop; Nextcloud's own commands after it are not under test).
|
||||
docker exec -d -u www-data ngb-frt-restore php -r '
|
||||
$p = new PDO("sqlite:/var/www/html/data/nextcloud.db"); $p->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
|
||||
$p->exec("CREATE TABLE IF NOT EXISTS frt_side_writer (t REAL)");
|
||||
for ($end = microtime(true) + 300; microtime(true) < $end && !file_exists("/tmp/frt-side-stop");) {
|
||||
$p->beginTransaction(); $p->exec("INSERT INTO frt_side_writer VALUES (" . microtime(true) . ")"); usleep(300000); $p->commit(); usleep(20000);
|
||||
}
|
||||
touch("/tmp/frt-side-done");'
|
||||
sleep 1
|
||||
fi
|
||||
sftp_logins() { docker logs ngb-frt-sftp 2>&1 | grep -c "Accepted password" || true; }
|
||||
[ "$MODE" = sftp ] && LOGINS_BEFORE=$(sftp_logins)
|
||||
OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS[@]}" 2>&1) || {
|
||||
@@ -173,14 +195,22 @@ OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS
|
||||
fail "restore:full"
|
||||
}
|
||||
printf '%s\n' "$OUT" | grep -q "maintenance mode while the restore runs" || fail "the restore did not say it runs in maintenance mode"
|
||||
run_on ngb-frt-restore status 2>&1 | grep -q "maintenance: false" || fail "maintenance mode is still on after the restore"
|
||||
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the restore"
|
||||
run_on ngb-frt-restore config:app:get ng_backup restore_full_started >/dev/null 2>&1 && fail "the restore left its 'started' marker behind"
|
||||
ok "backup:restore:full completed, in maintenance mode and out of it again"
|
||||
if [ "$DB" = sqlite ]; then
|
||||
docker exec ngb-frt-restore touch /tmp/frt-side-stop
|
||||
for i in $(seq 1 20); do docker exec ngb-frt-restore test -f /tmp/frt-side-done && break; sleep 0.5; done
|
||||
WRITES=$(docker exec -u www-data ngb-frt-restore php -r 'echo (new PDO("sqlite:/var/www/html/data/nextcloud.db"))->query("SELECT COUNT(*) FROM frt_side_writer")->fetchColumn();')
|
||||
[ "${WRITES:-0}" -gt 3 ] || fail "the other writer did not run alongside the restore ($WRITES writes)"
|
||||
ok "the restore waited for another process writing to SQLite ($WRITES writes alongside)"
|
||||
fi
|
||||
|
||||
# Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
|
||||
# earlier bugs here only showed up after config.php's secret was rewritten mid-restore.
|
||||
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after restore (own key/target undecryptable?)"
|
||||
run_on ngb-frt-restore user:list | grep -q frtest || fail "the backed-up user is missing from the restored database"
|
||||
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
|
||||
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database"
|
||||
docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
|
||||
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
|
||||
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
||||
@@ -202,6 +232,7 @@ if [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ] && [ "$LEGACY" = legacy
|
||||
ok "older backup: tables of the disabled app left out, Mail restored"
|
||||
elif [ "$ORIGIN_DBTYPE" != sqlite ] && [ "$DB" != sqlite ]; then
|
||||
[ "$(sql restore "select count(*) from information_schema.tables where table_name = 'oc_deck_boards'")" = 1 ] || fail "the tables of the disabled app were not created"
|
||||
[ "$(sql restore "select title from oc_deck_boards")" = "Plan 📌" ] || fail "a 4-byte character did not survive in a created table"
|
||||
[ "$(sql restore "select count(*) from oc_mail_accounts where drafts_mailbox_id is not null")" = 1 ] || fail "the mail account was not restored"
|
||||
[ "$(sql restore "select count(*) from information_schema.table_constraints where constraint_type = 'FOREIGN KEY' and table_name like 'oc_mail_%'")" -gt 0 ] \
|
||||
|| fail "the foreign keys of Mail are missing after the restore"
|
||||
|
||||
@@ -49,9 +49,20 @@ SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xa
|
||||
$X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash"
|
||||
ok "restore new-folder and replace: files identical, extra file in the trash"
|
||||
|
||||
# One file to a local directory, where a restore that was stopped left its temporary.
|
||||
ONE=/tmp/ngb-rt-one/data/$U/files/Docs
|
||||
$X sh -c "mkdir -p $ONE && echo partial > $ONE/f2.bin.ngb-restore-0123456789ab"
|
||||
occ backup:restore ci "$SNAP" "data/$U/files/Docs/f2.bin" -d /tmp/ngb-rt-one | grep "Restored 1 files" >/dev/null || fail "restore of one file to a directory"
|
||||
$X sh -c "cmp -s $ONE/f2.bin $DATA/$U/files/Docs/f2.bin" || fail "the file restored to a directory differs"
|
||||
$X sh -c "ls -A $ONE | grep -q ngb-restore" && fail "the temporary of a stopped restore was left"
|
||||
ok "restore of one file to a directory; the temporary of a stopped restore removed"
|
||||
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
|
||||
occ backup:status | grep "started (" >/dev/null || fail "backup:status does not name the time zone"
|
||||
ok "backup:status names the time zone of its times"
|
||||
|
||||
# Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob).
|
||||
occ backup:verify ci "$SNAP" | grep -q "^OK$" || fail "shallow verify"
|
||||
occ backup:verify ci "$SNAP" --deep | grep -q "^OK$" || fail "deep verify"
|
||||
occ backup:verify ci "$SNAP" | grep "^OK$" >/dev/null || fail "shallow verify"
|
||||
occ backup:verify ci "$SNAP" --deep | grep "^OK$" >/dev/null || fail "deep verify"
|
||||
ok "shallow and deep checksum audit"
|
||||
|
||||
# Database dump/restore round trip (all tables, checksums, sequences on PostgreSQL).
|
||||
|
||||
@@ -36,8 +36,14 @@ MANIFEST=$(occ backup:user:backup ci "$U" | sed -n 's/.* to \(.*\)\.$/\1/p')
|
||||
[ -n "$MANIFEST" ] || fail "backup:user:backup printed no manifest path"
|
||||
ok "backed up $U to $MANIFEST"
|
||||
|
||||
# as-backup: restore into <uid>-bak; the original account must stay untouched.
|
||||
occ backup:user:restore ci "$MANIFEST" --mode=as-backup >/dev/null || fail "restore as-backup"
|
||||
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
|
||||
occ backup:user:list ci "$U" | grep -F "$MANIFEST" >/dev/null || fail "backup:user:list does not show $MANIFEST"
|
||||
ok "backup:user:list shows the export"
|
||||
|
||||
# as-backup: restore into <uid>-bak; the original account must stay untouched. $U has one export,
|
||||
# so the bare user id names it (#22).
|
||||
OUT=$(occ backup:user:restore ci "$U" --mode=as-backup) || fail "restore as-backup by user id"
|
||||
echo "$OUT" | grep -qF "Using $MANIFEST." || fail "restore by user id did not say which export it used"
|
||||
occ files:scan "$U-bak" >/dev/null
|
||||
SUM_BAK=$($X sh -c "cd $DATA/$U-bak/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
||||
[ "$SUM_BEFORE" = "$SUM_BAK" ] || fail "the <uid>-bak account's files differ from the original"
|
||||
@@ -55,6 +61,11 @@ SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xa
|
||||
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after a replace restore"
|
||||
ok "replace restores the original content under the same uid, byte-identical"
|
||||
|
||||
# The replace above added a safety export: a bare user id is now ambiguous and must list both.
|
||||
if OUT=$(occ backup:user:restore ci "$U" --mode=as-backup 2>&1); then fail "restore by an ambiguous user id ran"; fi
|
||||
echo "$OUT" | grep -qF "$U has 2 exports here" && echo "$OUT" | grep -qF "$MANIFEST" || fail "an ambiguous user id did not list the exports"
|
||||
ok "an ambiguous user id lists the exports instead of picking one"
|
||||
|
||||
# Retention: $U now has the manual export above plus the automatic safety-export the replace
|
||||
# restore made of it just before deleting it (4 total after these two more), then --user-last=2
|
||||
# must forget exactly the 2 oldest.
|
||||
|
||||
Reference in New Issue
Block a user