Compare commits
55 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a8b5e85c9b | |||
| 36f2427e9a | |||
| e58b86fc48 | |||
| 193f65940b | |||
| 6540a9314a | |||
| f81c5e13e9 | |||
| 1b621b7149 | |||
| 1736005648 | |||
| c7a651f269 | |||
| 8897d8074f | |||
| 0b9fa1ae92 | |||
| ee406c481e | |||
| dd2e1af997 | |||
| d96229f370 | |||
| b14b16c66c | |||
| ce40053cec | |||
| 46e0186f35 | |||
| d5eb7abbff | |||
| 4c973d5cd8 | |||
| e9592cb6c6 | |||
| b86b07a49c | |||
| ecae6e9cdd | |||
| d36c40e77b | |||
| 50c2636926 | |||
| d1e9a53ce5 | |||
| b625fe7666 | |||
| 74bfa7052f | |||
| bf48d339e4 | |||
| 029b3b9cc7 | |||
| dce93aa75e | |||
| 3ec23a3072 | |||
| 4a395c75f2 | |||
| 36a30732ad | |||
| d7c7ca093b | |||
| 05d02f3c70 | |||
| dfe1fe9aad | |||
| f0293e6d01 | |||
| f7c9894b78 | |||
| be3746e1c3 | |||
| f5473495ce | |||
| 3694184674 | |||
| a0fc91a700 | |||
| a11e6c7254 | |||
| 6f1a37f4f4 | |||
| 5cde57bc1b | |||
| 3eec2e9554 | |||
| cdf0b70e25 | |||
| 66615c2a88 | |||
| cc75c7128c | |||
| a0c4abb60a | |||
| 4847ce293d | |||
| 90dd97977a | |||
| 08534975d7 | |||
| 95e3dfcf6d | |||
| 7f4cae0e42 |
@@ -41,6 +41,20 @@ jobs:
|
|||||||
- run: composer install --no-interaction --no-progress
|
- run: composer install --no-interaction --no-progress
|
||||||
- run: composer psalm
|
- run: composer psalm
|
||||||
|
|
||||||
|
frontend:
|
||||||
|
# Builds the admin page, so a broken Vue component fails CI instead of showing up after a release.
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
- uses: actions/setup-node@v4
|
||||||
|
with:
|
||||||
|
node-version: '22'
|
||||||
|
cache: npm
|
||||||
|
- run: npm ci --no-audit --no-fund
|
||||||
|
- run: npm run build
|
||||||
|
|
||||||
integration:
|
integration:
|
||||||
# Full round trip (key, kit, location, backup, incremental, restore modes, database dump and
|
# Full round trip (key, kit, location, backup, incremental, restore modes, database dump and
|
||||||
# restore, retention, trash) on a fresh Nextcloud per version and database.
|
# restore, retention, trash) on a fresh Nextcloud per version and database.
|
||||||
@@ -124,3 +138,7 @@ jobs:
|
|||||||
run: tests/integration/full-restore-roundtrip.sh 35-apache mysql to-sqlite
|
run: tests/integration/full-restore-roundtrip.sh 35-apache mysql to-sqlite
|
||||||
- name: Full restore of a SQLite backup onto PostgreSQL
|
- name: Full restore of a SQLite backup onto PostgreSQL
|
||||||
run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite to-pgsql
|
run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite to-pgsql
|
||||||
|
- name: Full restore with a damaged database dump (--salvage)
|
||||||
|
run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite salvage
|
||||||
|
- name: Full restore with a damaged and a missing pack (--salvage)
|
||||||
|
run: tests/integration/full-restore-roundtrip.sh 35-apache sqlite salvage-files
|
||||||
|
|||||||
@@ -7,11 +7,176 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
|
|||||||
beta/daily update channel, which would hide the app from most admins. The stability stage is
|
beta/daily update channel, which would hide the app from most admins. The stability stage is
|
||||||
instead carried by the app description, the git tag, the release, and this file.
|
instead carried by the app description, the git tag, the release, and this file.
|
||||||
|
|
||||||
## [1.0.0-rc.6] - 2026-10-24
|
Each heading starts with that `<version>`, once: the app store shows a release the section(s)
|
||||||
|
whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be
|
||||||
|
shown under 1.0.0. The git tag of a release candidate goes after the date.
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Pin a restore point, on the restore page or with `occ backup:pin`: the retention policy leaves it
|
||||||
|
alone, for example the one from before an upgrade. Unpinning notifies all administrators.
|
||||||
|
- A note on a restore point that you can change afterwards, on the restore page or with
|
||||||
|
`occ backup:note`. Pins and notes are stored encrypted on the location, so a new server sees them.
|
||||||
|
An older version of NG Backup can still read the location, but drops them when it writes to it.
|
||||||
|
- A schedule per location: off, once a day, or every N hours from a given time, on the page under
|
||||||
|
Backups or with `occ backup:target:schedule`. After the update every location keeps the time that
|
||||||
|
held for all of them; a new location starts with the schedule of the first one.
|
||||||
|
- All folders get their original modification time back on a restore, not only the empty ones, from
|
||||||
|
the first backup made with this version. An older version of NG Backup can still read new restore
|
||||||
|
points. A restore through Nextcloud into a folder that is already there leaves its time alone.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Locations are backed up one after the other: a backup that has to wait shows as queued. A restore
|
||||||
|
point has the time its backup really began, not the time it was queued.
|
||||||
|
- Choosing a time no longer starts a backup right away when that time has already passed today.
|
||||||
|
- A location without a schedule no longer gets the notification that it has had no recent backup.
|
||||||
|
Instead, all administrators are told when a schedule is turned off, and reminded once a week while
|
||||||
|
another location is still backed up, or every day when none is.
|
||||||
|
- A file or folder that is left out because of its name now gets a notification of its own, in the
|
||||||
|
log and in `occ backup:run`. It says the item is missing from the backup, gives example paths and
|
||||||
|
asks to rename it; before, it was reported as unreadable, with advice about permissions.
|
||||||
|
|
||||||
|
## [1.0.3] - 2026-11-06
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Empty folders are backed up and restored (#27), in `occ backup:restore`, `occ backup:restore:full`
|
||||||
|
and the restore page, where they are shown too. They come back from the first backup made with
|
||||||
|
this version; older restore points do not contain them. An older version of NG Backup can still
|
||||||
|
read new restore points, and then only leaves the empty folders out.
|
||||||
|
- A full disk during a restore no longer looks like a damaged backup. The restore now stops with a
|
||||||
|
message about the disk, also with `--salvage`, and does not point at `--salvage` any more. The same
|
||||||
|
goes for other trouble writing to the restore directory, such as permissions.
|
||||||
|
- A file or folder whose name a snapshot cannot hold (a backslash, a control character, not UTF-8) is
|
||||||
|
left out and reported as unreadable. Before, a backslash or control character could make the whole
|
||||||
|
restore point unreadable, and a name that is not UTF-8 made the backup fail.
|
||||||
|
- A restore that goes on after a stop also removes the temporary files in folders where every file
|
||||||
|
is already in place, and checks that those folders are inside the restore directory before it
|
||||||
|
leaves their files alone.
|
||||||
|
- After a restore that was killed and then ran again, a backup no longer has to wait up to two hours
|
||||||
|
for the locks of the killed run.
|
||||||
|
- A restore only removes read-ahead folders of its own (`.ngb-restore-spool-` with 8 hex digits).
|
||||||
|
- A restore keeps its lock on the location during a long row of files that are already in place.
|
||||||
|
- A database restore that fails no longer leaves behind tables it created, which could stop the
|
||||||
|
installation of that app later. When only NG Backup's own settings cannot be put back after the
|
||||||
|
database was restored, `--salvage` no longer says the database is still the fresh one.
|
||||||
|
- A connection that breaks while opening a pack on the location (seen with SMB) is retried once and
|
||||||
|
is handled by `--salvage`, instead of stopping the restore with the error of the storage.
|
||||||
|
- Without a system cron, several running backups take turns while the page is open.
|
||||||
|
- The Bin tab shows how far it is while it reads the removed restore points, which can take a while.
|
||||||
|
It also shows when it cannot read a location, instead of an empty bin, and refreshes when a run
|
||||||
|
ends.
|
||||||
|
|
||||||
|
## [1.0.2] - 2026-11-06
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- A warning on the admin page and in `occ backup:status` when background jobs do not run on a
|
||||||
|
system cron, or the cron stopped, with a link to how to set one up. Backups, also the scheduled
|
||||||
|
ones, need it (#13).
|
||||||
|
- Without a working system cron, a backup started with "Back up now" moves on while the NG Backup
|
||||||
|
page stays open (#13).
|
||||||
|
- `--salvage` for `occ backup:restore` and `occ backup:restore:full`: restore whatever can be
|
||||||
|
read when a backup is damaged. Files that cannot be read or decrypted are left out and listed,
|
||||||
|
and the command exits with code 2. When the database cannot be restored, `backup:restore:full`
|
||||||
|
leaves the fresh database as it is and puts the data directory and config of the backup in
|
||||||
|
`ngb-salvage-<snapshot>` in the data directory, to put back by hand.
|
||||||
|
- The trash of restore points has a tab of its own, "Bin", instead of being shown below the
|
||||||
|
backups.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- A backup in its database phase shows how far it is, for example "running (db: 47/183 tables)",
|
||||||
|
and one that has not moved for 15 minutes says "waiting for background jobs" with when it last
|
||||||
|
took a step. Before, it only said "running (db)", which looked the same as being stuck (#13).
|
||||||
|
- Full restores from S3 read the packs ahead, eight ranges at a time, as restores over SFTP do
|
||||||
|
since 1.0.0-rc.4. Over SMB the packs are read in 8 MiB reads instead of 1 MiB: over a line with
|
||||||
|
a 130 ms round trip, reading was four times faster in a test (18 instead of 4.5 MiB/s).
|
||||||
|
- Over SMB the index files, which a new server reads in `backup:target:add`, take fewer round
|
||||||
|
trips: from Canada to a Hetzner Storage Box (130 ms), 0.66 instead of 1.05 s per file, so about
|
||||||
|
5 instead of 8 hours for the 27,000 index files of a 1 TB repository.
|
||||||
|
- `occ backup:restore:full` that runs again after it stopped (`--force`) goes on where it was: data
|
||||||
|
files already in place with the size and modification time of the backup are left, and only
|
||||||
|
the rest is read from the location. Before, a stopped restore of 1 TB started over.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- A full restore over SMB could hang at 100% CPU when the connection to the server dropped: a
|
||||||
|
failed read was counted as data. It is now read again over a new connection. Found in a
|
||||||
|
disaster recovery test of 1 TB from Canada to a Hetzner Storage Box, after 392 GiB.
|
||||||
|
- The read-ahead folder of a restore that was killed while it wrote is removed by the next run;
|
||||||
|
it kept one file of up to 8 MiB.
|
||||||
|
- `occ backup:target:add` only estimates the time left for reading the index once enough of it
|
||||||
|
is in. The first file also carries the time to connect, which made it say "about 642 min left"
|
||||||
|
for something done within a minute.
|
||||||
|
- A restore that stops because a file is gone from the location now points at `--salvage`, as it
|
||||||
|
already did for damaged data.
|
||||||
|
|
||||||
|
### Known issues
|
||||||
|
- Empty folders are not backed up or restored; to be fixed in 1.0.3 (#27).
|
||||||
|
- A full disk during a restore looks like a damaged backup ("Size mismatch"), and with `--salvage`
|
||||||
|
the restore goes on and lists the remaining files. The backup itself is fine: make room and run
|
||||||
|
the restore again. To be fixed in 1.0.3.
|
||||||
|
|
||||||
|
## [1.0.1] - 2026-11-01
|
||||||
|
|
||||||
|
Fixes for what was reported after 1.0.0 (#22, #24) and small improvements. Nothing changes in
|
||||||
|
your backups or settings.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- `occ backup:user:list [location] [user]` shows the user exports on a location, with the manifest
|
||||||
|
path `backup:user:restore` expects. Before, that path was only printed once by
|
||||||
|
`backup:user:backup` (#22).
|
||||||
|
- A "Docs" tab on the admin page, with links to the how-to, the disaster recovery guide and the
|
||||||
|
roadmap. It also shows before the backup key is set up.
|
||||||
|
- A label for a backup started with "Back up now" (optional; `occ backup:run -l` already had one).
|
||||||
|
It shows with the restore point.
|
||||||
|
- An explanation in the trash of restore points of what "removed by retention policy" means: the
|
||||||
|
policy keeps the newest restore points and the last one of each recent day, week and month, and
|
||||||
|
moves the others to the trash, where they can be brought back until they are deleted
|
||||||
|
permanently. `occ backup:trash` says the same below its list.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- `occ backup:status`, `backup:list`, `backup:trash` and `backup:browse` show times in the time
|
||||||
|
zone of the daily schedule and name it, as the admin page shows them in the administrator's.
|
||||||
|
Before, they showed the server's UTC, which read as a wrong start time.
|
||||||
|
- The admin page no longer says NG Backup is in beta.
|
||||||
|
- `occ backup:user:restore` also takes a user id instead of the manifest path, when that user has
|
||||||
|
exactly one export on the location. With more, and for a manifest path that is not on the
|
||||||
|
location, the error lists that user's exports; for a user without exports it names the
|
||||||
|
expected form (#22). Before, `backup:user:restore <location> <uid>` only said "is not a
|
||||||
|
recorded user export for this location".
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- An SFTP location with an RSA key in OpenSSH format ("BEGIN OPENSSH PRIVATE KEY") stopped
|
||||||
|
working after an upgrade to Nextcloud 35 with "Unable to read key", and such a key could not be
|
||||||
|
added there at all (#24). The "Private key" field on the admin page is a single line, so a
|
||||||
|
pasted key lost its line breaks; Nextcloud 34 still read it, Nextcloud 35 reads keys with a
|
||||||
|
stricter library. The line breaks are now restored before the key is used, so a stored location
|
||||||
|
works again without re-entering it.
|
||||||
|
- A full restore onto SQLite could stop at once with "database is locked" when another process
|
||||||
|
wrote to the database just as the restore began (a background job that was already running).
|
||||||
|
The restore now takes the write lock first and waits for the other process instead. The
|
||||||
|
full-restore test on SQLite runs with a second process writing alongside.
|
||||||
|
- `occ backup:restore ... --to-directory` of a single file restored nothing ("Restored 0
|
||||||
|
files"); only a folder worked. A path to a file now restores that file.
|
||||||
|
- A restore that was stopped (a crash, a timeout, a restart) left its half-written temporary file
|
||||||
|
(`<name>.ngb-restore-<id>`) next to the file it was restoring, and a `files:scan` then showed it
|
||||||
|
in Files. A restore now removes such leftovers in each folder it writes to.
|
||||||
|
- The app store showed the notes of every release candidate under 1.0.0, because their headings
|
||||||
|
started with "1.0.0-rc". Each heading now starts with the app version of that release.
|
||||||
|
|
||||||
|
## [1.0.0] - 2026-10-29
|
||||||
|
|
||||||
|
First stable release. NG Backup makes encrypted, incremental backups of your whole Nextcloud
|
||||||
|
(files, database and configuration) to S3, SFTP, WebDAV, SMB and other storage, and restores them
|
||||||
|
from the web interface or with occ. Disaster recovery has been tested on real installations; see
|
||||||
|
"Tested disaster recoveries" in the README, and read "Known limitations" before you rely on it.
|
||||||
|
|
||||||
|
The code is the same as 1.0.0-rc.6 (app version 0.9.12): updating from it changes nothing in your
|
||||||
|
backups or settings.
|
||||||
|
|
||||||
|
## [0.9.12] - 2026-10-24 (1.0.0-rc.6)
|
||||||
|
|
||||||
Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery
|
Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery
|
||||||
simulations and a user report. If no problems are reported for about a week, 1.0.0 will be this
|
simulations and a user report.
|
||||||
same code.
|
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
- The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in
|
- The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in
|
||||||
@@ -36,7 +201,7 @@ same code.
|
|||||||
- The full-restore integration test checks a 4-byte character in a table the restore creates, and
|
- The full-restore integration test checks a 4-byte character in a table the restore creates, and
|
||||||
retries app installs from the app store.
|
retries app installs from the app store.
|
||||||
|
|
||||||
## [1.0.0-rc.5] - 2026-10-22
|
## [0.9.11] - 2026-10-22 (1.0.0-rc.5)
|
||||||
|
|
||||||
Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
|
Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP
|
||||||
stopped after 13 hours with "Blob not in index"; the backup itself was fine.
|
stopped after 13 hours with "Blob not in index"; the backup itself was fine.
|
||||||
@@ -69,7 +234,7 @@ stopped after 13 hours with "Blob not in index"; the backup itself was fine.
|
|||||||
- The extra connection a full restore reads ahead on is replaced when it breaks (up to 20 times),
|
- The extra connection a full restore reads ahead on is replaced when it breaks (up to 20 times),
|
||||||
instead of reading the rest of the restore the slow way.
|
instead of reading the rest of the restore the slow way.
|
||||||
|
|
||||||
## [1.0.0-rc.4] - 2026-10-19
|
## [0.9.10] - 2026-10-19 (1.0.0-rc.4)
|
||||||
|
|
||||||
Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not
|
Fourth release candidate for 1.0.0; the app version is 0.9.10. 1.0.0-rc.3 (0.9.9) was not
|
||||||
released: its fixes are in here.
|
released: its fixes are in here.
|
||||||
@@ -93,7 +258,7 @@ released: its fixes are in here.
|
|||||||
- The app store showed old screenshots: it keeps a copy per URL, and the URLs always pointed at
|
- The app store showed old screenshots: it keeps a copy per URL, and the URLs always pointed at
|
||||||
`main`. They now point at the release's tag.
|
`main`. They now point at the release's tag.
|
||||||
|
|
||||||
## [1.0.0-rc.2] - 2026-10-18
|
## [0.9.8] - 2026-10-18 (1.0.0-rc.2)
|
||||||
|
|
||||||
Second release candidate for 1.0.0; the app version is 0.9.8. Fixes from a full disaster recovery
|
Second release candidate for 1.0.0; the app version is 0.9.8. Fixes from a full disaster recovery
|
||||||
test of a production instance (about 1 TB, Nextcloud All-in-One onto a plain Nextcloud with
|
test of a production instance (about 1 TB, Nextcloud All-in-One onto a plain Nextcloud with
|
||||||
@@ -134,7 +299,7 @@ PostgreSQL).
|
|||||||
- Removing a location left its cached index in the database (tens of thousands of rows for a large
|
- Removing a location left its cached index in the database (tens of thousands of rows for a large
|
||||||
backup). It is removed with the location now, and once for locations that are already gone.
|
backup). It is removed with the location now, and once for locations that are already gone.
|
||||||
|
|
||||||
## [1.0.0-rc.1] - 2026-10-17
|
## [0.9.7] - 2026-10-17 (1.0.0-rc.1)
|
||||||
|
|
||||||
Release candidate for 1.0.0; the app version is 0.9.7. Only bug fixes until 1.0.0. Tested on a
|
Release candidate for 1.0.0; the app version is 0.9.7. Only bug fixes until 1.0.0. Tested on a
|
||||||
production instance (about 1 TB, 150,000 files): full and incremental backups and a verify on
|
production instance (about 1 TB, 150,000 files): full and incremental backups and a verify on
|
||||||
@@ -413,7 +578,13 @@ and not implemented yet.
|
|||||||
- See the "Storage backend experiences" section of the README for provider-specific quirks found
|
- See the "Storage backend experiences" section of the README for provider-specific quirks found
|
||||||
so far (e.g. Hetzner Storage Box sub-accounts).
|
so far (e.g. Hetzner Storage Box sub-accounts).
|
||||||
|
|
||||||
[1.0.0-rc.1]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.1
|
[1.0.1]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.1
|
||||||
|
[1.0.0]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0
|
||||||
|
[0.9.12]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.6
|
||||||
|
[0.9.11]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.5
|
||||||
|
[0.9.10]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.4
|
||||||
|
[0.9.8]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.2
|
||||||
|
[0.9.7]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.1
|
||||||
[0.9.6-beta.7]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.6-beta.7
|
[0.9.6-beta.7]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.6-beta.7
|
||||||
[0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6
|
[0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6
|
||||||
[0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5
|
[0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5
|
||||||
|
|||||||
@@ -392,10 +392,102 @@ in README "Known limitations":
|
|||||||
gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een
|
gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een
|
||||||
productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0.
|
productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0.
|
||||||
Problemen of meldingen van gebruikers: rc5.
|
Problemen of meldingen van gebruikers: rc5.
|
||||||
- **1.0.1:**
|
- **1.0.2** (plan akkoord 2026-11-04): punten 1, 2 en 5 hieronder, de Bin, nieuwe screenshots en
|
||||||
1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
|
het polish-punt over de schatting van `target:add`. Besluiten 2026-11-04:
|
||||||
|
- `--salvage` haalt ook bij een kapotte database terug wat kan: bestanden zijn per gebruiker
|
||||||
|
gescheiden en het waardevolst. Kan de database niet terug (dump onleesbaar, of past niet op
|
||||||
|
deze installatie), dan blijft de verse database staan en gaan data en config van de backup
|
||||||
|
naar `<datadir>/ngb-salvage-<snapshot>/`; vanaf daar is het handwerk. Exitcode 2.
|
||||||
|
- De DR-test met een backup mét tabelstructuur, zonder `--skip-missing-tables` (punt 3), is
|
||||||
|
nog niet gedaan en wordt ingepland vóór de release van 1.0.2. Ingepland (2026-11-04): een
|
||||||
|
verse snapshot van nc01 (gemaakt met 1.0.1, dus met tabelstructuur) volgens de DR-gids
|
||||||
|
terugzetten op VM 980 met de 1.0.2-build, over **SMB** (Storage Box; besluit 2026-11-04:
|
||||||
|
SMB in plaats van SFTP, zodat de DR-test de nieuwe SMB-leescode op echte schaal en over de
|
||||||
|
echte lijn vanuit Canada test; vorige meting 2,6 MiB/s, verwachting ~1 dag voor 1 TB), zonder
|
||||||
|
`--skip-missing-tables`; daarna `--salvage` op een deel van de bestanden (moet hetzelfde
|
||||||
|
resultaat geven, exitcode 0). Uitvoeren met Dillard, die VM 980 klaarzet; deze week, vóór
|
||||||
|
de release. Screenshots akkoord (2026-11-04).
|
||||||
|
**DR-test 1.0.2 over SMB (VM 980, snapshot a86271ed van nc01, 983 GiB, gestart 2026-11-04):**
|
||||||
|
index lezen 8 uur (~1 bestand/s; daarna versneld naar 0,66 s/bestand, gemeten vanaf VM 980);
|
||||||
|
restore zonder opties (geen `--install-apps`, geen `--skip-missing-tables`) liep ~12 MiB/s
|
||||||
|
(SFTP eerder 2,8-7,6; SMB vóór 1.0.2 2,6). Na 392 GiB viel de SMB-verbinding weg en bleef de
|
||||||
|
restore op 100% CPU hangen: de smbclient-extensie geeft bij een mislukte read een string met
|
||||||
|
lengte -1. Opgelost (controle + nieuwe verbinding), plus hervatten: `--force` laat bestanden met
|
||||||
|
dezelfde grootte en mtime staan. Hervat 2026-11-05 07:21 UTC na `maintenance:mode --off`
|
||||||
|
volgens de gids; binnen 10 min voorbij de 392 GiB. SMB op sub5 na de test weer uitzetten.
|
||||||
|
**Resultaat (klaar 2026-11-06 02:11 UTC, nazorg 02:27):** hervatte run 18 u 50 min, exit 0, 150.755
|
||||||
|
bestanden; 7-13 MiB/s (langzamer vanaf ~18 uur Duitse tijd, oorzaak niet vastgesteld). Vergeleken
|
||||||
|
met nc01: 150.467 van 150.758 bestanden gelijk in grootte en mtime, 397 op hash zonder verschil;
|
||||||
|
gebruikers, agenda's, contacten, shares en Deck gelijk; bestanden, agenda en contacten via WebDAV
|
||||||
|
getest als gebruiker. `backup:restore --salvage` op 16.904 bestanden: exit 0, geen verschil. Niet
|
||||||
|
vastgesteld: de 50.080 "updated" bij `files:scan`. Zonder `--install-apps` 51 van de 81 apps aan.
|
||||||
|
VM's 980-982 en de sim-backups op OVH en sub6 opgeruimd.
|
||||||
|
**Naar 1.0.3 (besluit 2026-11-06):** lege mappen komen niet terug (3.982 op nc01, 2.076 daarvan in
|
||||||
|
`files`, verklaart vrijwel zeker de 2.370 "removed" bij `files:scan`). De backup legt alleen
|
||||||
|
bestanden vast (`walk()`), dus oudere snapshots hebben ze niet. Voor een DR acceptabel, wel
|
||||||
|
fixen; issue #27. Daarna kleine volledige simulaties; een DR van 1 TB is hiervoor niet nodig.
|
||||||
|
**Salvage vóór de release getest (besluit 2026-11-06):** nieuwe modus `salvage-files` in
|
||||||
|
`full-restore-roundtrip.sh`, ook in CI: één pack beschadigd en één weg. Eerst stopt de restore,
|
||||||
|
daarna `--salvage --force` volgens de gids. Vond één fout: bij een ontbrekend bestand wees de
|
||||||
|
melding niet naar `--salvage` (opgelost).
|
||||||
|
Stand 2026-11-04 (`polish`, gepusht): cron-waarschuwing + tick vanaf de pagina +
|
||||||
|
voortgang van de db-fase; `--salvage` voor `backup:restore` en `backup:restore:full`; S3
|
||||||
|
vooruitlezen (8 ranges tegelijk, met `SaveAs` naar de spool); SMB in leesacties van 8 MiB
|
||||||
|
(gemeten met netem, 130 ms RTT: 4,5 → 18 MiB/s per leesactie; een volledige restore over SMB
|
||||||
|
met vertraging is nog niet gemeten); Bin-tabblad; schatting `target:add`; nieuwe screenshots
|
||||||
|
(1–3 vervangen, 4-bin erbij; in `info.xml` bij de release naar tag v1.0.2 laten wijzen en
|
||||||
|
4-bin toevoegen). Getest: unit (208), psalm, integratie NC35 sqlite/pgsql, S3/SFTP/FTP/SMB,
|
||||||
|
volledige restore met `--salvage`. **1.0.2 uitgebracht 2026-11-06** (tag `v1.0.2`, PR #28).
|
||||||
|
- **Naar 1.0.3, uit de reviews van 1.0.2 (achteraf gedraaid, 2026-11-06):**
|
||||||
|
1. Lege mappen (#27).
|
||||||
|
2. Een volle schijf onder `--salvage` telt als beschadigde backup en de restore gaat door; zonder
|
||||||
|
`--salvage` wijst de melding er ten onrechte naar. `fwrite` controleren, schrijffouten stoppen de
|
||||||
|
restore (`Repository::restoreEntries`; ook `RestoreService::writeFile` nakijken). Beide reviews.
|
||||||
|
3. Hervatten: controle op de doelmap vóór het overslaan, en tijdelijke bestanden ook opruimen in
|
||||||
|
een map waar alles al klopt.
|
||||||
|
4. Alleen `.ngb-restore-spool-<hex>` opruimen.
|
||||||
|
5. Salvage-test: tijdelijke bestanden recursief zoeken; `admin` en `frtest` apart controleren.
|
||||||
|
6. Nog nakijken: lease verversen bij overgeslagen bestanden; fout ná de database-commit
|
||||||
|
(`restoreOwnAppConfig`) en tabellen die na een rollback blijven staan; backups zonder cron
|
||||||
|
via de pagina (eerlijke beurten, stappen die de webtimeout halen); Bin verversen en fouten
|
||||||
|
tonen; SMB-uitzondering bij het openen van een pack onder `--salvage`.
|
||||||
|
Niet doen: bij hervatten de inhoud vergelijken (grootte en mtime is ook het criterium van de backup).
|
||||||
|
**Stand op branch `1.0.3` (2026-11-06):** punten 1-6 gebouwd. Lege mappen staan in het metarecord
|
||||||
|
van de snapshot, niet tussen de bestanden: 1.0.2 en ouder lezen zo'n snapshot gewoon en slaan alleen
|
||||||
|
de lege mappen over. Geen webtimeout-wijziging zonder cron: alleen in de code nagelopen, niet getest
|
||||||
|
(Nextcloud zet de PHP-tijdslimiet ruim, een gestopte stap wordt overgedaan); wel eerlijke beurten.
|
||||||
|
Bin leest in delen met teller en balk (besluit 2026-11-06, ook met oog op meerdere backups per
|
||||||
|
dag). Getest: unit (volle schijf op echte tmpfs), integratie NC35 sqlite/pgsql/mysql, sftp, salvage,
|
||||||
|
salvage-files; Bin in een echte browser met vertraagde verzoeken (1 van 7 t/m 7). Nog: reviews
|
||||||
|
vóór de PR, README-regels bij de versiecommit.
|
||||||
|
**Simulaties (2026-11-06, NC 34.0.4, 2 GB, lokale SFTP, `~/dev-local/ngb-sim/sim.sh`):** normaal
|
||||||
|
(pgsql), stop en hervat (pgsql → MariaDB, restore met kill -9 gestopt), volle schijf met `--salvage`:
|
||||||
|
alle drie geslaagd, alle 3.101 gebruikersbestanden en 7 lege mappen gelijk, share, agenda en WebDAV
|
||||||
|
werken. Vond één fout, opgelost: na een kill bleven vergrendelingen tot 2 uur staan, waardoor de
|
||||||
|
backup van stap 11 werd geweigerd. `files:scan` "Updated" zijn in de gemeten run alleen mappen
|
||||||
|
(mtime van mappen wordt niet teruggezet, dus nieuwe mtime en etag; bestanden ongemoeid). "Removed":
|
||||||
|
o.a. `cache` van een gebruiker, die de backup uitsluit. Waarschijnlijk verklaart dit ook de 50.080
|
||||||
|
"updated" op nc01, niet op nc01 zelf vastgesteld.
|
||||||
|
**Naar 1.1.0 of 1.2.0 (besluit 2026-11-06):** alle mappen krijgen bij een restore hun oorspronkelijke
|
||||||
|
tijd terug. Hoort zo, vooral bij oudere verzamelingen: de maptijd wordt gebruikt om in te schatten
|
||||||
|
wanneer een map is aangemaakt. De backup moet dan de tijd van alle mappen vastleggen en de restore
|
||||||
|
die aan het eind terugzetten; werkt pas vanaf de eerste backup met die versie. Op de roadmap onder 1.1.0.
|
||||||
|
**1.0.3 uitgebracht (2026-11-06)**, ook op nc01 geïnstalleerd.
|
||||||
|
- **Branches (besluit 2026-11-06):** werk voor een versie op een branch met die naam (`1.0.3`);
|
||||||
|
`polish` alleen voor kleine cosmetische zaken, per geval bekijken of en wanneer die gemerged
|
||||||
|
worden. SMB staat uit op sub5 en sub6; sub6 is het testaccount, eerst aanzetten voor een SMB-test.
|
||||||
|
Na 1.0.3 gaat het werk naar `1.1.0`. Een spoedfix die niet kan wachten gaat op `1.0.4` vanaf `main`,
|
||||||
|
krijgt een eigen merge en release; daarna `main` in `1.1.0` mergen, zodat daar met de fix getest wordt.
|
||||||
|
- **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar
|
||||||
|
1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november,
|
||||||
|
niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij
|
||||||
|
"Back up now" en het tabblad Docs staan er nog niet op; besluit 2026-11-01), en de prullenbak
|
||||||
|
van herstelpunten naar een eigen tabblad "Bin" naast "Backup and restore", "Settings" en "Docs":
|
||||||
|
onder de lijst met herstelpunten vervuilt hij het scherm (vooral met wat het bewaarbeleid erin
|
||||||
|
zet; besluit 2026-11-01). Nieuwe screenshots daarna, zodat de Bin er ook op staat.
|
||||||
|
1. **Naar 1.0.2.** (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een
|
||||||
restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
|
restore leest de packs nog één voor één: ook vooruitlezen via de S3-client.
|
||||||
2. Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit
|
2. **Naar 1.0.2.** Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit
|
||||||
oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21):
|
oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21):
|
||||||
a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode`
|
a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode`
|
||||||
(`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de
|
(`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de
|
||||||
@@ -424,16 +516,54 @@ in README "Known limitations":
|
|||||||
4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de
|
4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de
|
||||||
tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop
|
tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop
|
||||||
stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een
|
stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een
|
||||||
bestaande parameter.
|
bestaande parameter. Ten hoogste 100 tekens — **klaar** (2026-10-30).
|
||||||
5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen
|
5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen
|
||||||
wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind
|
wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind
|
||||||
opsommen, in plaats van stoppen.
|
opsommen, in plaats van stoppen. **Naar 1.0.2** (besluit 2026-10-30).
|
||||||
6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van
|
6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van
|
||||||
rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-<id>` staan; de
|
rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-<id>` staan; de
|
||||||
geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files.
|
geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files.
|
||||||
Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op.
|
Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op (alleen bestanden met
|
||||||
|
precies dat achtervoegsel en 6 of 12 hextekens) — **klaar** (2026-10-30).
|
||||||
7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden
|
7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden
|
||||||
terug (er komt een `/` achter); nu werkt alleen een map.
|
terug (er komt een `/` achter); nu werkt alleen een map. Gold alleen voor `--to-directory`;
|
||||||
|
naar de bestanden van een gebruiker werkte een los bestand al — **klaar** (2026-10-30).
|
||||||
|
8. **Uitleg (i) bij de prullenbak van herstelpunten** (besluit 2026-10-25): bij "removed by
|
||||||
|
retention policy" een infoknop met wat het betekent: het bewaarbeleid houdt voor oudere dagen
|
||||||
|
één herstelpunt per dag (het laatste), weken en maanden; de rest gaat naar de prullenbak en
|
||||||
|
wordt pas na de ingestelde wachttijd (`delete_delay_days`, standaard 30) echt verwijderd, en is
|
||||||
|
tot die datum terug te zetten. Aanleiding: de beheerder zag "19-10 13:12 – after rc4 upgrade
|
||||||
|
removed by retention policy" en dacht dat er iets was weggegooid. `occ backup:trash` zegt
|
||||||
|
hetzelfde onder de lijst — **klaar** (2026-10-30).
|
||||||
|
9. **Tijdzone in `occ backup:status`**: de tabel toont de servertijd (UTC), de webinterface de
|
||||||
|
tijd van de beheerder; dat leest als een verkeerde starttijd. Zet de zone in de kolomkop
|
||||||
|
("started (UTC)") of toon de tijden in de zone van het schema. Gedaan: beide, en ook in
|
||||||
|
`backup:list`, `backup:trash` en `backup:browse` — **klaar** (2026-10-30).
|
||||||
|
10. **User-restore op gebruikersnaam** (issue #22, 2026-10-27): `backup:user:restore <locatie>
|
||||||
|
New_User` gaf alleen "is not a recorded user export"; het argument moet het manifestpad
|
||||||
|
zijn, dat alleen `backup:user:backup` eenmalig toonde. Nu: `backup:user:list` toont de
|
||||||
|
exports, een kale uid werkt bij precies één export, anders een lijst om uit te kiezen —
|
||||||
|
**klaar** (2026-10-27).
|
||||||
|
11. **Tabblad "Docs" in de beheerpagina** (besluit 2026-10-27): een derde knop naast "Backup and
|
||||||
|
restore" en "Settings", met links naar de HOWTO (`docs/HOWTO.md`), de DR-gids
|
||||||
|
(`docs/DISASTER-RECOVERY.md`) en de roadmap (README, `#roadmap`) op GitHub, elk met een
|
||||||
|
regel wat erin staat. Nu zijn ze alleen via de README te vinden. Zichtbaar ook zonder
|
||||||
|
sleutel, zodat de gids vindbaar is vóór de eerste installatie-stap. "NG Backup is in beta"
|
||||||
|
gaat uit de feedbackregel — **klaar** (2026-10-27).
|
||||||
|
12. **SFTP-sleutel zonder regeleinden** (issue #24, 2026-10-28): het veld "Private key" op de
|
||||||
|
beheerpagina is één regel, dus een geplakte sleutel verliest zijn regeleinden. NC34
|
||||||
|
(phpseclib 2) las dat nog, NC35 (phpseclib 3) weigert een OpenSSH-sleutel dan met "Unable
|
||||||
|
to read key". De regeleinden worden nu teruggezet vlak voor gebruik — **klaar** (2026-10-28).
|
||||||
|
13. **"database is locked" bij het begin van `backup:restore:full` op SQLite** (CI, 2026-10-26
|
||||||
|
bij #22 en 2026-10-29 op `main` na de merge van 1.0.0): de restore stopt meteen na het
|
||||||
|
aanzetten van de onderhoudsmodus. Niet altijd, dus een tweede proces dat op hetzelfde moment
|
||||||
|
de database gebruikt (cron, een webverzoek of de opstart van de container). Kan ook bij een
|
||||||
|
echte restore naar SQLite gebeuren. Oorzaak (2026-10-29): SQLite (WAL) begint een transactie
|
||||||
|
als lezer; schrijft een ander proces tussen ons eerste lezen en ons eerste schrijven, dan
|
||||||
|
faalt dat schrijven direct (geen wachttijd helpt). In CI was dat de installer, die
|
||||||
|
"installed" zet en daarna nog seconden doorschrijft. Nu neemt de restore eerst het
|
||||||
|
schrijfslot en wacht hij op de ander; de SQLite-test draait met een tweede schrijver
|
||||||
|
ernaast — **klaar** (2026-10-29).
|
||||||
- **DR-tests over databasegrenzen** (besluit 2026-10-20; productie nc01 draait PostgreSQL): na
|
- **DR-tests over databasegrenzen** (besluit 2026-10-20; productie nc01 draait PostgreSQL): na
|
||||||
de MariaDB→PostgreSQL-fout (tellers, rc5) elke overstap in de integratietest (`to-pgsql`,
|
de MariaDB→PostgreSQL-fout (tellers, rc5) elke overstap in de integratietest (`to-pgsql`,
|
||||||
`to-mysql`, `to-sqlite`, ook in CI). Volle DR-tests met echte data: vóór 1.0.0 de S3-simulatie
|
`to-mysql`, `to-sqlite`, ook in CI). Volle DR-tests met echte data: vóór 1.0.0 de S3-simulatie
|
||||||
@@ -442,7 +572,18 @@ in README "Known limitations":
|
|||||||
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
|
- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3,
|
||||||
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
|
4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan
|
||||||
een paar bestanden.
|
een paar bestanden.
|
||||||
- **1.1.0:**
|
- **1.1.0** (geen harde datum: een week was een voorzet, langer mag; de releasefrequentie mag omlaag,
|
||||||
|
besluit 2026-11-06):
|
||||||
|
**Betere melding voor namen die een snapshot niet kan bevatten** (backslash, stuurteken, geen UTF-8):
|
||||||
|
een eigen melding in plaats van "unreadable" met het advies over rechten. Zegt dat deze bestanden of
|
||||||
|
mappen in de backup ontbreken, geeft voorbeelden van de paden en vraagt ze te hernoemen; de volgende
|
||||||
|
backup neemt ze mee. Alleen als Nextcloud-melding: de app mailt niet zelf, of meldingen gemaild worden
|
||||||
|
is de keuze van de beheerder (besluit 2026-11-06).
|
||||||
|
**Stand op branch `1.1.0` (2026-11-07):** klaar: betere melding, punten 4, 5, 6 en 8. Pin en notitie staan
|
||||||
|
in de catalogus, met een kopie in de database voor de lijsten. Een moment van vóór het instellen van
|
||||||
|
een schema start niets. Geen schema: geen melding "geen recente backup", wel een melding aan alle
|
||||||
|
beheerders bij het uitzetten en een herinnering, wekelijks zolang een andere locatie een schema heeft,
|
||||||
|
anders dagelijks (besluit 2026-11-07). Nog niet getest: de update vanaf 1.0.3, de pagina in een browser.
|
||||||
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
|
1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit
|
||||||
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
|
als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van
|
||||||
`target:add` en sleutelimport (het gevoeligste pad), daarom niet in rc1. Daarna de
|
`target:add` en sleutelimport (het gevoeligste pad), daarom niet in rc1. Daarna de
|
||||||
@@ -470,6 +611,14 @@ in README "Known limitations":
|
|||||||
6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot
|
6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot
|
||||||
is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus),
|
is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus),
|
||||||
zodat een nieuwe server na een DR hem ook ziet.
|
zodat een nieuwe server na een DR hem ook ziet.
|
||||||
|
8. **Herstelpunt vastpinnen** (besluit 2026-10-25): een herstelpunt dat het bewaarbeleid
|
||||||
|
nooit opruimt (bijv. "vóór de upgrade naar NC 35"), in de lijst en met
|
||||||
|
`occ backup:pin <snapshot>` / `--unpin`. Op de locatie bewaren (catalogus), zodat de
|
||||||
|
opschoning het ziet, ook na een DR; vastgepinde punten tellen niet mee in last/daily/weekly.
|
||||||
|
9. **Feedbackvraag weghalen in 1.2.0** (besluit 2026-11-07, niet in 1.1.0): de regel "We would like
|
||||||
|
to hear how NG Backup works for you ... Share your experience" in de beheerpagina blijft tot
|
||||||
|
dan staan. Er komt weinig feedback bij zo'n 150 downloads per versie; geen bericht is goed
|
||||||
|
bericht. Het formulier (`experience.yml`) in de issues blijft.
|
||||||
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
|
- **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met
|
||||||
PostgreSQL 18)** — voor **1.0.0-rc.2**:
|
PostgreSQL 18)** — voor **1.0.0-rc.2**:
|
||||||
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
|
1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE,
|
||||||
|
|||||||
@@ -7,11 +7,14 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
|
|||||||
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
||||||
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
||||||
|
|
||||||
**Status: release candidate for 1.0.0 (1.0.0-rc.1, app version 0.9.7).** Tested thoroughly, usable
|
**Status: 1.0.3, stable.** Disaster recovery has been tested on real installations (see "Tested
|
||||||
alongside your current backup, but not yet a drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and
|
disaster recoveries" below). See "Known limitations" below, `PLAN.md` for the design and
|
||||||
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
|
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. Server lost? See
|
||||||
`docs/DISASTER-RECOVERY.md`.
|
`docs/DISASTER-RECOVERY.md`.
|
||||||
|
|
||||||
|
If you like NG Backup, don't forget to give the app a star in the
|
||||||
|
[app store](https://apps.nextcloud.com/apps/ng_backup) and this repository a star on GitHub!
|
||||||
|
|
||||||
License: AGPL-3.0-or-later. Developed and sponsored by [Ensembia](https://www.ensembia.com).
|
License: AGPL-3.0-or-later. Developed and sponsored by [Ensembia](https://www.ensembia.com).
|
||||||
Need help restoring a backup after a crash? Ensembia offers paid restore assistance:
|
Need help restoring a backup after a crash? Ensembia offers paid restore assistance:
|
||||||
[www.ensembia.com](https://www.ensembia.com).
|
[www.ensembia.com](https://www.ensembia.com).
|
||||||
@@ -39,7 +42,8 @@ Need help restoring a backup after a crash? Ensembia offers paid restore assista
|
|||||||
background job, so a backup only moves forward when that job runs. Choosing "Cron" under
|
background job, so a backup only moves forward when that job runs. Choosing "Cron" under
|
||||||
**Administration settings → Basic settings → Background jobs** is not enough on its own: it
|
**Administration settings → Basic settings → Background jobs** is not enough on its own: it
|
||||||
tells Nextcloud that the system will call it, so you also need a cron line that does. Without
|
tells Nextcloud that the system will call it, so you also need a cron line that does. Without
|
||||||
it a backup stays at "running (db)", and so does one started with "Back up now".
|
it scheduled backups do not run; the admin page and `occ backup:status` warn about it, and a
|
||||||
|
backup started with "Back up now" only moves on while the NG Backup page stays open.
|
||||||
`occ backup:run` in a terminal works without cron (it runs in the foreground).
|
`occ backup:run` in a terminal works without cron (it runs in the foreground).
|
||||||
- Regular server: `crontab -u www-data -e` and add
|
- Regular server: `crontab -u www-data -e` and add
|
||||||
`*/5 * * * * php -f /var/www/nextcloud/cron.php` (your Nextcloud path).
|
`*/5 * * * * php -f /var/www/nextcloud/cron.php` (your Nextcloud path).
|
||||||
@@ -164,6 +168,8 @@ Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECO
|
|||||||
| Backup | Location | Database | Result |
|
| Backup | Location | Database | Result |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
|
| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical |
|
||||||
|
| The same, 1.0.2 | Hetzner Storage Box (SMB), Germany → Canada | PostgreSQL → PostgreSQL | 150,755 files restored in about 29 h (7-13 MiB/s), going on after a lost connection; sample identical, empty folders not restored ([#27](https://github.com/dillardblom/ng_backup/issues/27)) |
|
||||||
|
| 1.0.3, small simulations of 2 GB: as planned, with the restore killed halfway, and with a disk that runs full | Local SFTP server | PostgreSQL → PostgreSQL and MariaDB | all 3,101 user files and all empty folders identical; users, a share and a calendar back |
|
||||||
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
|
| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical |
|
||||||
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
|
| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical |
|
||||||
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
|
| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) |
|
||||||
@@ -260,19 +266,15 @@ What is planned, without dates. Ideas and votes are welcome in the
|
|||||||
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
|
[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here,
|
||||||
a 👍 on its issue helps decide what comes first.
|
a 👍 on its issue helps decide what comes first.
|
||||||
|
|
||||||
**1.0.1**
|
|
||||||
- A warning on the admin page and in `occ backup:status` when background jobs do not run on a
|
|
||||||
system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on
|
|
||||||
while the NG Backup page is open ([#13](https://github.com/dillardblom/ng_backup/issues/13)).
|
|
||||||
- An optional label for a backup started with "Back up now".
|
|
||||||
- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4.
|
|
||||||
|
|
||||||
**1.1.0**
|
**1.1.0**
|
||||||
- Restore without the recovery kit: the backup key from the location itself, opened with one of
|
- Restore without the recovery kit: the backup key from the location itself, opened with one of
|
||||||
your passphrases.
|
your passphrases.
|
||||||
- A schedule per location, also "every N hours".
|
- A schedule per location, also "every N hours".
|
||||||
- Locations backed up one after the other instead of at the same time.
|
- Locations backed up one after the other instead of at the same time.
|
||||||
|
- All folders get their original modification time back on a restore, not only the empty ones.
|
||||||
- A note on a restore point that you can change afterwards.
|
- A note on a restore point that you can change afterwards.
|
||||||
|
- Pinning a restore point, so the retention policy never removes it (for example the one from
|
||||||
|
before an upgrade).
|
||||||
- An option to pause heavy background jobs (such as Memories, Recognize or the Preview
|
- An option to pause heavy background jobs (such as Memories, Recognize or the Preview
|
||||||
Generator) while a backup runs.
|
Generator) while a backup runs.
|
||||||
- Faster backups: reading and uploading at the same time.
|
- Faster backups: reading and uploading at the same time.
|
||||||
@@ -287,8 +289,13 @@ a 👍 on its issue helps decide what comes first.
|
|||||||
|
|
||||||
## Known limitations
|
## Known limitations
|
||||||
|
|
||||||
On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it:
|
On the backlog; know them before you rely on NG Backup:
|
||||||
|
|
||||||
|
- **Empty folders only come back from backups made with 1.0.3 or newer.** Older restore points do
|
||||||
|
not contain them.
|
||||||
|
- **Folders with files in them only get their original modification time back from backups made with
|
||||||
|
1.1.0 or newer.** From older restore points they show the time of the restore; the files keep theirs.
|
||||||
|
A restore through Nextcloud into a folder that is already there leaves that folder's time alone.
|
||||||
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
|
- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in
|
||||||
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
|
`config.php`) is not supported: a backup refuses to start there. Folders mounted through the
|
||||||
External storage app are not in the data directory and are **not included**, without a warning.
|
External storage app are not in the data directory and are **not included**, without a warning.
|
||||||
@@ -323,11 +330,13 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
|
|||||||
| `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation |
|
| `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation |
|
||||||
| `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots |
|
| `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots |
|
||||||
| `backup:target:backends/add/edit/list/remove/append-only` | backup locations |
|
| `backup:target:backends/add/edit/list/remove/append-only` | backup locations |
|
||||||
|
| `backup:target:schedule` | when a location is backed up: off, daily, or every N hours |
|
||||||
| `backup:run`, `backup:status`, `backup:list` | back up, status (runs of the last 7 days; `--all` for all), restore points |
|
| `backup:run`, `backup:status`, `backup:list` | back up, status (runs of the last 7 days; `--all` for all), restore points |
|
||||||
| `backup:cancel <run>` | stop a running backup |
|
| `backup:cancel <run>` | stop a running backup |
|
||||||
| `backup:browse`, `backup:restore` | browse and restore |
|
| `backup:browse`, `backup:restore` | browse and restore |
|
||||||
| `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash |
|
| `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash |
|
||||||
|
| `backup:pin`, `backup:note` | keep a restore point, a note you can change later |
|
||||||
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
|
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
|
||||||
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
|
| `backup:restore:full` | whole-server restore onto a fresh installation (after `backup:key:import`) |
|
||||||
| `backup:key:import` | import a recovery kit on a fresh installation |
|
| `backup:key:import` | import a recovery kit on a fresh installation |
|
||||||
| `backup:user:backup`, `backup:user:restore` | one user via user_migration (asks before installing it) |
|
| `backup:user:backup`, `backup:user:list`, `backup:user:restore` | one user via user_migration (asks before installing it); list the exports |
|
||||||
|
|||||||
@@ -40,9 +40,12 @@ NG Backup needs Nextcloud's background jobs on system cron (a real cron line, no
|
|||||||
Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very
|
Read "Known limitations" in the README before you rely on it, and test a restore. Feedback is very
|
||||||
welcome: use the "Share your experience" link on the admin page.
|
welcome: use the "Share your experience" link on the admin page.
|
||||||
|
|
||||||
|
**If you like NG Backup, don't forget to give the app a star in the app store and the
|
||||||
|
[repository](https://github.com/dillardblom/ng_backup) a star on GitHub!**
|
||||||
|
|
||||||
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
|
Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if
|
||||||
you need help with restoring a backup after a crash.]]></description>
|
you need help with restoring a backup after a crash.]]></description>
|
||||||
<version>0.9.12</version>
|
<version>1.0.3</version>
|
||||||
<licence>agpl</licence>
|
<licence>agpl</licence>
|
||||||
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
|
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
|
||||||
<namespace>NgBackup</namespace>
|
<namespace>NgBackup</namespace>
|
||||||
@@ -53,9 +56,10 @@ you need help with restoring a backup after a crash.]]></description>
|
|||||||
<website>https://github.com/dillardblom/ng_backup</website>
|
<website>https://github.com/dillardblom/ng_backup</website>
|
||||||
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
|
<bugs>https://github.com/dillardblom/ng_backup/issues</bugs>
|
||||||
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
|
<repository type="git">https://github.com/dillardblom/ng_backup</repository>
|
||||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/1-backups.png</screenshot>
|
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/1-backups.png</screenshot>
|
||||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/2-restore.png</screenshot>
|
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/2-restore.png</screenshot>
|
||||||
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/3-settings.png</screenshot>
|
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/3-settings.png</screenshot>
|
||||||
|
<screenshot>https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.3/screenshots/4-bin.png</screenshot>
|
||||||
<dependencies>
|
<dependencies>
|
||||||
<php min-version="8.1"/>
|
<php min-version="8.1"/>
|
||||||
<lib>sodium</lib>
|
<lib>sodium</lib>
|
||||||
@@ -79,11 +83,14 @@ you need help with restoring a backup after a crash.]]></description>
|
|||||||
<command>OCA\NgBackup\Command\ListSnapshots</command>
|
<command>OCA\NgBackup\Command\ListSnapshots</command>
|
||||||
<command>OCA\NgBackup\Command\Browse</command>
|
<command>OCA\NgBackup\Command\Browse</command>
|
||||||
<command>OCA\NgBackup\Command\Restore</command>
|
<command>OCA\NgBackup\Command\Restore</command>
|
||||||
|
<command>OCA\NgBackup\Command\Pin</command>
|
||||||
|
<command>OCA\NgBackup\Command\Note</command>
|
||||||
<command>OCA\NgBackup\Command\Prune</command>
|
<command>OCA\NgBackup\Command\Prune</command>
|
||||||
<command>OCA\NgBackup\Command\Retention</command>
|
<command>OCA\NgBackup\Command\Retention</command>
|
||||||
<command>OCA\NgBackup\Command\TargetAppendOnly</command>
|
<command>OCA\NgBackup\Command\TargetAppendOnly</command>
|
||||||
<command>OCA\NgBackup\Command\TargetTrustCurrent</command>
|
<command>OCA\NgBackup\Command\TargetTrustCurrent</command>
|
||||||
<command>OCA\NgBackup\Command\TargetLimit</command>
|
<command>OCA\NgBackup\Command\TargetLimit</command>
|
||||||
|
<command>OCA\NgBackup\Command\TargetSchedule</command>
|
||||||
<command>OCA\NgBackup\Command\Trash</command>
|
<command>OCA\NgBackup\Command\Trash</command>
|
||||||
<command>OCA\NgBackup\Command\TrashRestore</command>
|
<command>OCA\NgBackup\Command\TrashRestore</command>
|
||||||
<command>OCA\NgBackup\Command\KeySlots</command>
|
<command>OCA\NgBackup\Command\KeySlots</command>
|
||||||
@@ -92,6 +99,7 @@ you need help with restoring a backup after a crash.]]></description>
|
|||||||
<command>OCA\NgBackup\Command\KeySlotRemove</command>
|
<command>OCA\NgBackup\Command\KeySlotRemove</command>
|
||||||
<command>OCA\NgBackup\Command\BackupUser</command>
|
<command>OCA\NgBackup\Command\BackupUser</command>
|
||||||
<command>OCA\NgBackup\Command\RestoreUser</command>
|
<command>OCA\NgBackup\Command\RestoreUser</command>
|
||||||
|
<command>OCA\NgBackup\Command\ListUserExports</command>
|
||||||
<command>OCA\NgBackup\Command\KeyImport</command>
|
<command>OCA\NgBackup\Command\KeyImport</command>
|
||||||
<command>OCA\NgBackup\Command\RestoreFull</command>
|
<command>OCA\NgBackup\Command\RestoreFull</command>
|
||||||
<command>OCA\NgBackup\Command\Verify</command>
|
<command>OCA\NgBackup\Command\Verify</command>
|
||||||
|
|||||||
@@ -24,8 +24,11 @@ return [
|
|||||||
['name' => 'AdminApi#testTarget', 'url' => '/api/targets/{id}/test', 'verb' => 'POST'],
|
['name' => 'AdminApi#testTarget', 'url' => '/api/targets/{id}/test', 'verb' => 'POST'],
|
||||||
['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'],
|
['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'],
|
||||||
['name' => 'AdminApi#startBackup', 'url' => '/api/backups', 'verb' => 'POST'],
|
['name' => 'AdminApi#startBackup', 'url' => '/api/backups', 'verb' => 'POST'],
|
||||||
['name' => 'AdminApi#setSchedule', 'url' => '/api/schedule', 'verb' => 'PUT'],
|
['name' => 'AdminApi#tick', 'url' => '/api/tick', 'verb' => 'POST'],
|
||||||
|
['name' => 'AdminApi#setTargetSchedule', 'url' => '/api/targets/{id}/schedule', 'verb' => 'PUT'],
|
||||||
['name' => 'AdminApi#snapshots', 'url' => '/api/targets/{targetId}/snapshots', 'verb' => 'GET'],
|
['name' => 'AdminApi#snapshots', 'url' => '/api/targets/{targetId}/snapshots', 'verb' => 'GET'],
|
||||||
|
['name' => 'AdminApi#setSnapshotNote', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/note', 'verb' => 'PUT'],
|
||||||
|
['name' => 'AdminApi#setSnapshotPinned', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/pin', 'verb' => 'PUT'],
|
||||||
['name' => 'AdminApi#browse', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/browse', 'verb' => 'GET'],
|
['name' => 'AdminApi#browse', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/browse', 'verb' => 'GET'],
|
||||||
['name' => 'AdminApi#startRestore', 'url' => '/api/restores', 'verb' => 'POST'],
|
['name' => 'AdminApi#startRestore', 'url' => '/api/restores', 'verb' => 'POST'],
|
||||||
],
|
],
|
||||||
|
|||||||
@@ -103,12 +103,42 @@ mode on yourself before, it stays on: switch it off with `occ maintenance:mode -
|
|||||||
If it stops halfway, maintenance mode stays on (the instance may be partly restored). Fix the cause,
|
If it stops halfway, maintenance mode stays on (the instance may be partly restored). Fix the cause,
|
||||||
then switch maintenance mode off and run the restore again with `--force`, in one line (the message
|
then switch maintenance mode off and run the restore again with `--force`, in one line (the message
|
||||||
prints it): `occ maintenance:mode --off && occ backup:restore:full <location> <snapshot> --force`.
|
prints it): `occ maintenance:mode --off && occ backup:restore:full <location> <snapshot> --force`.
|
||||||
Leave out `--install-apps` this time, so that the instance is open only for the seconds until the
|
It goes on where it stopped: data files that are already in place with the size and modification
|
||||||
|
time of the backup are left, so only the rest comes from the location again (since 1.0.2; before,
|
||||||
|
it started over). The database is restored again in full, which takes minutes. Leave out `--install-apps` this time, so that the instance is open only for the seconds until the
|
||||||
restore has switched maintenance mode on again; apps the first run could not install are listed at
|
restore has switched maintenance mode on again; apps the first run could not install are listed at
|
||||||
the end of the restore, install those afterwards.
|
the end of the restore, install those afterwards.
|
||||||
If occ then says that Nextcloud or one of the apps requires an upgrade, the database is already
|
If occ then says that Nextcloud or one of the apps requires an upgrade, the database is already
|
||||||
the old server's: run `occ upgrade` first, then the same command with `--force`.
|
the old server's: run `occ upgrade` first, then the same command with `--force`.
|
||||||
|
|
||||||
|
**If the backup itself is damaged** (the restore stops with "Decryption failed", "Blob
|
||||||
|
verification failed" or a file that is missing on the location), try an older restore point
|
||||||
|
first. If none is better, `--salvage` restores whatever can be read:
|
||||||
|
|
||||||
|
occ backup:restore:full offsite <snapshot> --install-apps --salvage
|
||||||
|
|
||||||
|
If a restore already stopped halfway, run it as in step 8: maintenance mode off, with `--force` and
|
||||||
|
without `--install-apps`:
|
||||||
|
|
||||||
|
occ maintenance:mode --off && occ backup:restore:full offsite <snapshot> --salvage --force
|
||||||
|
|
||||||
|
- Files that cannot be read are left out and listed at the end; everything else is restored.
|
||||||
|
- A full disk is not damage: the restore stops, also with `--salvage`, and says so. Make room and
|
||||||
|
run the restore again as in step 8.
|
||||||
|
- **On version 1.0.2, check the disk first if the reason is "Size mismatch".** There a full disk
|
||||||
|
looks like damaged data: the message points at `--salvage`, and with `--salvage` the restore goes
|
||||||
|
on and lists the remaining files. The backup itself is fine. Update NG Backup, make room and run
|
||||||
|
the restore again as in step 8.
|
||||||
|
- If the database cannot be restored (its dump is damaged, or it does not fit this
|
||||||
|
installation), the database stays as the fresh installation made it, and the data directory
|
||||||
|
and config of the backup go to a folder of their own in the data directory,
|
||||||
|
`ngb-salvage-<snapshot>`. Each user's files are in `data/<user>/files` there: create the users
|
||||||
|
again and copy their files back by hand (then `occ files:scan --all`). Calendars, contacts and
|
||||||
|
shares are in the database and cannot be recovered this way.
|
||||||
|
|
||||||
|
The command then ends with exit code 2 instead of 0. The same option exists for
|
||||||
|
`occ backup:restore`, to restore single folders.
|
||||||
|
|
||||||
**9. After the restore.** These worked in our tests:
|
**9. After the restore.** These worked in our tests:
|
||||||
|
|
||||||
occ upgrade
|
occ upgrade
|
||||||
@@ -121,6 +151,12 @@ the old server's: run `occ upgrade` first, then the same command with `--force`.
|
|||||||
- From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`,
|
- From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`,
|
||||||
`redis`), which point at AIO's containers.
|
`redis`), which point at AIO's containers.
|
||||||
- Log in, open a few files, a calendar and a contact.
|
- Log in, open a few files, a calendar and a contact.
|
||||||
|
- `occ files:scan` can report many folders as "Updated". That is expected: a restore does not yet put
|
||||||
|
back the modification time of folders with files in them, so they show the time of the restore.
|
||||||
|
The files themselves keep their time and content.
|
||||||
|
- Empty folders only come back from backups made with version 1.0.3 or newer. With an older
|
||||||
|
restore point they are gone: `occ files:scan` then removes them from the file cache, so it reports
|
||||||
|
them as removed.
|
||||||
- Start background jobs again (cron), as in step 2 in reverse.
|
- Start background jobs again (cron), as in step 2 in reverse.
|
||||||
|
|
||||||
**10. Point your domain at the new server** and check that clients reconnect.
|
**10. Point your domain at the new server** and check that clients reconnect.
|
||||||
|
|||||||
@@ -79,7 +79,9 @@ occ backup:list
|
|||||||
Later runs upload only changed data. An unchanged run uploads (almost) nothing; the output shows
|
Later runs upload only changed data. An unchanged run uploads (almost) nothing; the output shows
|
||||||
the number of uploaded bytes, so check that value when you test incremental behaviour.
|
the number of uploaded bytes, so check that value when you test incremental behaviour.
|
||||||
|
|
||||||
Schedule `occ backup:run` from system cron (or use the Administration settings, if you prefer).
|
Set when each location is backed up in the Administration settings, or with
|
||||||
|
`occ backup:target:schedule offsite 02:00` (once a day), `02:00/6` (every 6 hours from 02:00) or `off`.
|
||||||
|
Locations are backed up one after the other.
|
||||||
|
|
||||||
## 4. Check the backup regularly
|
## 4. Check the backup regularly
|
||||||
|
|
||||||
@@ -114,10 +116,16 @@ Needs the `user_migration` app. NG Backup asks before installing it:
|
|||||||
|
|
||||||
```
|
```
|
||||||
occ backup:user:backup offsite <uid>
|
occ backup:user:backup offsite <uid>
|
||||||
|
occ backup:user:list offsite [<uid>] # the exports and their manifest
|
||||||
occ backup:user:restore offsite users/<uid>/<exportId> # replace (default)
|
occ backup:user:restore offsite users/<uid>/<exportId> # replace (default)
|
||||||
occ backup:user:restore offsite users/<uid>/<exportId> --mode=as-backup
|
occ backup:user:restore offsite users/<uid>/<exportId> --mode=as-backup
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- `backup:user:restore` needs the manifest path that `backup:user:backup` prints at the end
|
||||||
|
(`users/<uid>/<exportId>`); `backup:user:list` shows it again. A user id alone also works when
|
||||||
|
that user has exactly one export on the location. With more (a `replace` restore adds a safety
|
||||||
|
export), the command lists them and you pass one.
|
||||||
|
|
||||||
- If `user_migration` is missing, the command asks to install and enable it. For non-interactive
|
- If `user_migration` is missing, the command asks to install and enable it. For non-interactive
|
||||||
use, add `--install-user-migration` to approve that explicitly. The command stops after the
|
use, add `--install-user-migration` to approve that explicitly. The command stops after the
|
||||||
installation; run it once more.
|
installation; run it once more.
|
||||||
@@ -167,8 +175,13 @@ occ backup:prune offsite --dry-run # show first
|
|||||||
occ backup:prune offsite # apply the policy; forgotten snapshots go to the trash first
|
occ backup:prune offsite # apply the policy; forgotten snapshots go to the trash first
|
||||||
occ backup:trash # list the trash
|
occ backup:trash # list the trash
|
||||||
occ backup:trash:restore offsite <id>
|
occ backup:trash:restore offsite <id>
|
||||||
|
occ backup:pin offsite <id> # keep this restore point; --unpin to undo
|
||||||
|
occ backup:note offsite <id> "text" # a note you can change later; without text it is removed
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The retention policy leaves a pinned restore point alone, and a pinned one does not take the place of
|
||||||
|
a daily, weekly or monthly one. Pin and note are stored on the location, so a new server sees them too.
|
||||||
|
|
||||||
Snapshots are kept according to the policy, and their data is deleted only after the deletion delay
|
Snapshots are kept according to the policy, and their data is deleted only after the deletion delay
|
||||||
(set with `backup:key:init --delete-delay-days`). `occ backup:prune offsite --dry-run` shows what would be
|
(set with `backup:key:init --delete-delay-days`). `occ backup:prune offsite --dry-run` shows what would be
|
||||||
removed without changing anything.
|
removed without changing anything.
|
||||||
|
|||||||
@@ -72,6 +72,10 @@ final class ExternalStorageFactory {
|
|||||||
throw new BackendException("Auth mechanism $authId does not fit backend $backendId");
|
throw new BackendException("Auth mechanism $authId does not fit backend $backendId");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (is_string($options['private_key'] ?? null)) {
|
||||||
|
$options['private_key'] = self::normalizePrivateKey($options['private_key']);
|
||||||
|
}
|
||||||
|
|
||||||
$config = new StorageConfig();
|
$config = new StorageConfig();
|
||||||
$config->setBackend($backend);
|
$config->setBackend($backend);
|
||||||
$config->setAuthMechanism($auth);
|
$config->setAuthMechanism($auth);
|
||||||
@@ -86,4 +90,22 @@ final class ExternalStorageFactory {
|
|||||||
$storage = $backend->wrapStorage($storage);
|
$storage = $backend->wrapStorage($storage);
|
||||||
return $auth->wrapStorage($storage);
|
return $auth->wrapStorage($storage);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* An OpenSSH-format private key with its line breaks restored. Up to Nextcloud 34
|
||||||
|
* (phpseclib 2) a key whose newlines were lost or turned into spaces, e.g. on its way through
|
||||||
|
* a shell or a form, still loaded; Nextcloud 35 (phpseclib 3) refuses it with "Unable to read
|
||||||
|
* key" (#24). The key itself is not changed, so it is fixed when it is used, not when stored.
|
||||||
|
* Other formats (PEM, PKCS#8) still load as they are and are returned unchanged.
|
||||||
|
*/
|
||||||
|
public static function normalizePrivateKey(string $key): string {
|
||||||
|
if (!preg_match('/-----BEGIN OPENSSH PRIVATE KEY-----(.*?)-----END OPENSSH PRIVATE KEY-----/s', $key, $m)) {
|
||||||
|
return $key;
|
||||||
|
}
|
||||||
|
$body = (string)preg_replace('/\s+/', '', $m[1]);
|
||||||
|
if (!preg_match('#^[A-Za-z0-9+/]+=*$#', $body)) {
|
||||||
|
return $key;
|
||||||
|
}
|
||||||
|
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -23,6 +23,11 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
|||||||
private bool $s3Reads;
|
private bool $s3Reads;
|
||||||
/** S3 requests in flight at once for getMany() and getRanges(). */
|
/** S3 requests in flight at once for getMany() and getRanges(). */
|
||||||
public const S3_CONCURRENCY = 16;
|
public const S3_CONCURRENCY = 16;
|
||||||
|
/**
|
||||||
|
* Ranges readAhead() fetches at once over S3, each straight into a file in the spool (a range
|
||||||
|
* is at most ReadAheadPlan::MAX_RANGE: the spool holds at most this many times that).
|
||||||
|
*/
|
||||||
|
public const S3_READ_AHEAD = 8;
|
||||||
/** Most ranges, and bytes, getRanges() holds in memory at once. */
|
/** Most ranges, and bytes, getRanges() holds in memory at once. */
|
||||||
private const RANGES_LOT = 64;
|
private const RANGES_LOT = 64;
|
||||||
private const RANGES_LOT_BYTES = 64 * 1048576;
|
private const RANGES_LOT_BYTES = 64 * 1048576;
|
||||||
@@ -119,7 +124,13 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
|||||||
// null: not supported or failing for now; false: not opened, which need not mean
|
// null: not supported or failing for now; false: not opened, which need not mean
|
||||||
// missing. Either way the storage's own stream decides.
|
// missing. Either way the storage's own stream decides.
|
||||||
}
|
}
|
||||||
|
try {
|
||||||
$fh = $this->storage->fopen($this->abs($path), 'r');
|
$fh = $this->storage->fopen($this->abs($path), 'r');
|
||||||
|
} catch (\Exception $e) {
|
||||||
|
// Some storages throw for what is not a missing file (SMB: a connection that broke).
|
||||||
|
// As a BackendException, the caller reads once more and --salvage can go on.
|
||||||
|
throw new BackendException('Cannot read ' . $path . ': ' . $e->getMessage(), 0, $e);
|
||||||
|
}
|
||||||
if ($fh === false) {
|
if ($fh === false) {
|
||||||
throw new BackendException('Not found: ' . $path);
|
throw new BackendException('Not found: ' . $path);
|
||||||
}
|
}
|
||||||
@@ -128,7 +139,7 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Over SFTP and S3 several at a time (SftpReadStream::readMany(), s3Fetch()); elsewhere one
|
* Over SFTP and S3 several at a time (SftpReadStream::readMany(), s3Fetch()); elsewhere one
|
||||||
* after the other.
|
* after the other, over SMB in fewer round trips (smbWhole()).
|
||||||
*/
|
*/
|
||||||
public function getMany(array $paths): \Generator {
|
public function getMany(array $paths): \Generator {
|
||||||
$size = $this->s3Reads ? 16 * self::S3_CONCURRENCY : 8 * SftpReadStream::BATCH;
|
$size = $this->s3Reads ? 16 * self::S3_CONCURRENCY : 8 * SftpReadStream::BATCH;
|
||||||
@@ -144,7 +155,8 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
|||||||
$fast = $fast === null ? null : array_combine($chunk, array_map(fn (string $a): ?string => $fast[$a] ?? null, $abs));
|
$fast = $fast === null ? null : array_combine($chunk, array_map(fn (string $a): ?string => $fast[$a] ?? null, $abs));
|
||||||
}
|
}
|
||||||
foreach ($chunk as $path) {
|
foreach ($chunk as $path) {
|
||||||
$data = $fast[$path] ?? null;
|
// over SMB in fewer round trips, one after the other (smbWhole())
|
||||||
|
$data = $fast[$path] ?? $this->smbWhole($this->abs($path));
|
||||||
if ($data === null) {
|
if ($data === null) {
|
||||||
// not read in the batch (or no batches here): the usual way, which also
|
// not read in the batch (or no batches here): the usual way, which also
|
||||||
// reports a missing file as such
|
// reports a missing file as such
|
||||||
@@ -211,15 +223,11 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
|||||||
*/
|
*/
|
||||||
private function s3Fetch(array $requests): ?array {
|
private function s3Fetch(array $requests): ?array {
|
||||||
try {
|
try {
|
||||||
/** @psalm-suppress UndefinedMethod, UndefinedThisPropertyFetch */
|
$s3 = $this->s3Client();
|
||||||
[$client, $bucket, $sse] = (function (): array {
|
if ($s3 === null) {
|
||||||
$sse = method_exists($this, 'getServerSideEncryptionParameters') ? $this->getServerSideEncryptionParameters()
|
|
||||||
: (method_exists($this, 'getSSECParameters') ? $this->getSSECParameters() : []);
|
|
||||||
return [$this->getConnection(), $this->getBucket(), $sse];
|
|
||||||
})->call($this->storage);
|
|
||||||
if (!$client instanceof \Aws\S3\S3Client || !is_string($bucket)) {
|
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
[$client, $bucket, $sse] = $s3;
|
||||||
$results = [];
|
$results = [];
|
||||||
$keys = array_keys($requests); // the pool numbers results 0, 1, ...: mapped back below
|
$keys = array_keys($requests); // the pool numbers results 0, 1, ...: mapped back below
|
||||||
$commands = (function () use ($requests, $client, $bucket, $sse): \Generator {
|
$commands = (function () use ($requests, $client, $bucket, $sse): \Generator {
|
||||||
@@ -245,16 +253,111 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* files_external's S3 client, bucket and server-side encryption parameters; null when the
|
||||||
|
* storage is not as expected.
|
||||||
|
*
|
||||||
|
* @return array{0: \Aws\S3\S3Client, 1: string, 2: array}|null
|
||||||
|
* @psalm-suppress UndefinedClass the AWS SDK comes with Nextcloud, not with this app
|
||||||
|
*/
|
||||||
|
private function s3Client(): ?array {
|
||||||
|
/** @psalm-suppress UndefinedMethod, UndefinedThisPropertyFetch */
|
||||||
|
[$client, $bucket, $sse] = (function (): array {
|
||||||
|
$sse = method_exists($this, 'getServerSideEncryptionParameters') ? $this->getServerSideEncryptionParameters()
|
||||||
|
: (method_exists($this, 'getSSECParameters') ? $this->getSSECParameters() : []);
|
||||||
|
return [$this->getConnection(), $this->getBucket(), $sse];
|
||||||
|
})->call($this->storage);
|
||||||
|
return $client instanceof \Aws\S3\S3Client && is_string($bucket) ? [$client, $bucket, is_array($sse) ? $sse : []] : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* readAhead() over S3: S3_READ_AHEAD ranges at a time, fetched together straight into files in
|
||||||
|
* the spool, then handed out in order. One request per range instead of one per blob (with a
|
||||||
|
* new HTTPS connection each), and several at once: a restore is no longer bound by the round
|
||||||
|
* trip. A range that could not be fetched is handed out as null (the caller reads it the
|
||||||
|
* usual way, which reports why).
|
||||||
|
*
|
||||||
|
* @param list<array{0:string, 1:int, 2:int}> $ranges
|
||||||
|
* @return \Generator<int, resource|null>|null
|
||||||
|
* @psalm-suppress UndefinedClass the AWS SDK comes with Nextcloud, not with this app
|
||||||
|
*/
|
||||||
|
private function s3ReadAhead(array $ranges, string $spool): ?\Generator {
|
||||||
|
// before anything is fetched: an invalid path throws
|
||||||
|
$keys = array_map(fn (array $r): string => trim($this->abs($r[0]), '/'), $ranges);
|
||||||
|
try {
|
||||||
|
$s3 = $this->s3Client();
|
||||||
|
} catch (\Throwable) {
|
||||||
|
$s3 = null;
|
||||||
|
}
|
||||||
|
if ($s3 === null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
[$client, $bucket, $sse] = $s3;
|
||||||
|
$this->idleCheck();
|
||||||
|
return (static function () use ($client, $bucket, $sse, $ranges, $keys, $spool): \Generator {
|
||||||
|
$prefix = rtrim($spool, '/') . '/s3-' . bin2hex(random_bytes(4)) . '-';
|
||||||
|
foreach (array_chunk(array_keys($ranges), self::S3_READ_AHEAD) as $lot) {
|
||||||
|
$files = [];
|
||||||
|
$commands = [];
|
||||||
|
foreach ($lot as $n => $i) {
|
||||||
|
$files[$i] = $prefix . $i;
|
||||||
|
$commands[$n] = $client->getCommand('GetObject', ['Bucket' => $bucket, 'Key' => $keys[$i],
|
||||||
|
'Range' => 'bytes=' . $ranges[$i][1] . '-' . ($ranges[$i][1] + $ranges[$i][2] - 1), 'SaveAs' => $files[$i]] + $sse);
|
||||||
|
}
|
||||||
|
$fetched = [];
|
||||||
|
try {
|
||||||
|
(new \Aws\CommandPool($client, $commands, [
|
||||||
|
'concurrency' => self::S3_READ_AHEAD,
|
||||||
|
'fulfilled' => function (mixed $result, int $n) use (&$fetched, $lot): void {
|
||||||
|
$fetched[$lot[$n]] = true;
|
||||||
|
},
|
||||||
|
'rejected' => static function (): void {
|
||||||
|
},
|
||||||
|
]))->promise()->wait();
|
||||||
|
} catch (\Throwable) {
|
||||||
|
// what was not fetched is read the usual way
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
foreach ($lot as $i) {
|
||||||
|
clearstatcache(true, $files[$i]);
|
||||||
|
$fh = isset($fetched[$i]) && @filesize($files[$i]) === $ranges[$i][2] ? @fopen($files[$i], 'rb') : false;
|
||||||
|
@unlink($files[$i]); // the open stream keeps its data
|
||||||
|
yield $i => is_resource($fh) ? $fh : null;
|
||||||
|
if (is_resource($fh)) {
|
||||||
|
fclose($fh);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
// stopped halfway: the rest of this lot
|
||||||
|
foreach ($files as $file) {
|
||||||
|
@unlink($file);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Over SFTP: SftpRangeFetcher, on readAheadConnections new connections of its own (built with the
|
* Over SFTP: SftpRangeFetcher, on readAheadConnections new connections of its own (built with the
|
||||||
* reconnect function), so the storage's connection stays free for other requests. Elsewhere
|
* reconnect function), so the storage's connection stays free for other requests. Over S3:
|
||||||
* not (yet): null.
|
* S3_READ_AHEAD ranges at a time through the S3 client (s3ReadAhead()). Elsewhere not (yet):
|
||||||
|
* null.
|
||||||
*/
|
*/
|
||||||
public function canReadAhead(): bool {
|
public function canReadAhead(): bool {
|
||||||
return $this->sftpReads && $this->reconnect !== null;
|
return ($this->sftpReads && $this->reconnect !== null) || $this->s3Reads || $this->smbNative();
|
||||||
|
}
|
||||||
|
|
||||||
|
private function smbNative(): bool {
|
||||||
|
return $this->storage instanceof \OCA\Files_External\Lib\Storage\SMB && extension_loaded('smbclient');
|
||||||
}
|
}
|
||||||
|
|
||||||
public function readAhead(array $ranges, string $spool): ?\Generator {
|
public function readAhead(array $ranges, string $spool): ?\Generator {
|
||||||
|
if ($ranges !== [] && $this->s3Reads) {
|
||||||
|
return $this->s3ReadAhead($ranges, $spool);
|
||||||
|
}
|
||||||
|
if ($ranges !== [] && $this->smbNative()) {
|
||||||
|
return $this->smbReadAhead($ranges, $spool);
|
||||||
|
}
|
||||||
$reconnect = $this->reconnect;
|
$reconnect = $this->reconnect;
|
||||||
if (!$this->sftpReads || $reconnect === null || $ranges === []) {
|
if (!$this->sftpReads || $reconnect === null || $ranges === []) {
|
||||||
return null;
|
return null;
|
||||||
@@ -306,6 +409,18 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private function readRange(string $path, int $offset, int $length): string {
|
private function readRange(string $path, int $offset, int $length): string {
|
||||||
|
try {
|
||||||
|
$chunks = $this->smbChunks($this->abs($path), $offset, $length);
|
||||||
|
if ($chunks !== null) {
|
||||||
|
$data = implode('', iterator_to_array($chunks, false));
|
||||||
|
if (strlen($data) === $length) {
|
||||||
|
return $data;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (\Throwable) {
|
||||||
|
// read the usual way below, which reports what is wrong, over a new connection
|
||||||
|
$this->smbReconnect();
|
||||||
|
}
|
||||||
$fh = $this->get($path);
|
$fh = $this->get($path);
|
||||||
try {
|
try {
|
||||||
if (fseek($fh, $offset) !== 0) {
|
if (fseek($fh, $offset) !== 0) {
|
||||||
@@ -418,6 +533,218 @@ final class StorageBackend implements IBackend, IReadsAhead {
|
|||||||
*
|
*
|
||||||
* @return list<string>|null
|
* @return list<string>|null
|
||||||
*/
|
*/
|
||||||
|
/**
|
||||||
|
* Bytes per read over SMB (smbChunks()). libsmbclient keeps a large read in flight as many
|
||||||
|
* requests at once; icewind's stream reads 1 MiB at a time. Over a line with a 130 ms round
|
||||||
|
* trip: 4.5 MiB/s in 1 MiB reads, 18 MiB/s in 8 MiB reads (measured 2026-11-04, Samba in a
|
||||||
|
* container behind netem). Without latency it makes no difference.
|
||||||
|
*/
|
||||||
|
public const SMB_READ = 8 * 1048576;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* $length bytes of $abs from $offset over SMB, SMB_READ at a time straight through the native
|
||||||
|
* extension's state of files_external's SMB storage. Null when the storage is not SMB through
|
||||||
|
* the extension, or its internals are not as expected (the usual way reads it then). The
|
||||||
|
* generator throws when a read fails.
|
||||||
|
*
|
||||||
|
* @return \Generator<int, string>|null
|
||||||
|
* @psalm-suppress UndefinedClass icewind/smb comes with files_external, not with this app
|
||||||
|
*/
|
||||||
|
private function smbChunks(string $abs, int $offset, int $length): ?\Generator {
|
||||||
|
$handle = $this->smbHandle($abs);
|
||||||
|
if ($handle === null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
[$state, $url] = $handle;
|
||||||
|
/** @psalm-suppress MixedMethodCall the state is icewind's NativeState */
|
||||||
|
return (static function () use ($state, $url, $offset, $length): \Generator {
|
||||||
|
$fh = $state->open($url, 'r');
|
||||||
|
try {
|
||||||
|
$state->lseek($fh, $offset, SEEK_SET, $url);
|
||||||
|
for ($got = 0; $got < $length;) {
|
||||||
|
$want = min(self::SMB_READ, $length - $got);
|
||||||
|
$chunk = self::smbRead($state, $fh, $want, $url);
|
||||||
|
if ($chunk === '') {
|
||||||
|
throw new BackendException('Short read in ' . $url);
|
||||||
|
}
|
||||||
|
$got += strlen($chunk);
|
||||||
|
yield $chunk;
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
$state->close($fh, $url);
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A whole (small) file over SMB in four round trips: open, read, read up to the end, close.
|
||||||
|
* files_external's fopen() first checks that the file exists (a stat with its attributes) and
|
||||||
|
* icewind's stream reads 1 MiB at a time: about 7 round trips, 1 s per index file from Canada
|
||||||
|
* to the Storage Box (130 ms), so ~8 h for the 27,000 index files of a 1 TB repository. Null
|
||||||
|
* when the storage is not SMB through the extension or anything fails: the usual way then
|
||||||
|
* reads the file and reports a missing one as such.
|
||||||
|
*
|
||||||
|
* @psalm-suppress UndefinedClass icewind/smb comes with files_external, not with this app
|
||||||
|
*/
|
||||||
|
private function smbWhole(string $abs): ?string {
|
||||||
|
$handle = $this->smbHandle($abs);
|
||||||
|
if ($handle === null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
[$state, $url] = $handle;
|
||||||
|
try {
|
||||||
|
/** @psalm-suppress MixedMethodCall the state is icewind's NativeState */
|
||||||
|
$fh = $state->open($url, 'r');
|
||||||
|
} catch (\Throwable) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
$data = '';
|
||||||
|
try {
|
||||||
|
do {
|
||||||
|
$chunk = self::smbRead($state, $fh, self::SMB_READ, $url);
|
||||||
|
$data .= $chunk;
|
||||||
|
} while ($chunk !== '');
|
||||||
|
} catch (\Throwable) {
|
||||||
|
$data = null;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
/** @psalm-suppress MixedMethodCall */
|
||||||
|
$state->close($fh, $url);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
}
|
||||||
|
if ($data === null) {
|
||||||
|
$this->smbReconnect(); // a read failed: the usual way reads it over a new connection
|
||||||
|
}
|
||||||
|
return $data;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One read of at most $bytes through the native state, checked. When libsmbclient fails a read
|
||||||
|
* (seen when the connection to the server dropped during a long restore), the smbclient
|
||||||
|
* extension does not return false but a string whose length is -1, because it stores
|
||||||
|
* smbc_read()'s -1 in an unsigned length; icewind passes it on. Counting that as data made
|
||||||
|
* the read loop go backwards forever at 100% CPU (DR test 2026-11-05, after 392 GiB).
|
||||||
|
*
|
||||||
|
* @param resource $fh
|
||||||
|
* @psalm-suppress MixedMethodCall, MixedArgument the state is icewind's NativeState
|
||||||
|
*/
|
||||||
|
private static function smbRead(object $state, $fh, int $bytes, string $url): string {
|
||||||
|
$chunk = $state->read($fh, $bytes, $url);
|
||||||
|
$n = is_string($chunk) ? strlen($chunk) : -1;
|
||||||
|
if ($n < 0 || $n > $bytes) {
|
||||||
|
throw new BackendException('Read failed in ' . $url);
|
||||||
|
}
|
||||||
|
return $chunk;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* After a failed read: drop icewind's connection state, so the next read (the usual way, or
|
||||||
|
* the next range) connects again instead of failing on a connection that is gone.
|
||||||
|
*
|
||||||
|
* @psalm-suppress UndefinedClass icewind/smb comes with files_external, not with this app
|
||||||
|
*/
|
||||||
|
private function smbReconnect(): void {
|
||||||
|
if (!$this->storage instanceof \OCA\Files_External\Lib\Storage\SMB) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
(function (): void {
|
||||||
|
/** @psalm-suppress UndefinedThisPropertyFetch */
|
||||||
|
$share = $this->share;
|
||||||
|
if ($share instanceof \Icewind\SMB\Native\NativeShare) {
|
||||||
|
(function (): void {
|
||||||
|
/** @psalm-suppress UndefinedThisPropertyAssignment */
|
||||||
|
$this->state = null;
|
||||||
|
})->call($share);
|
||||||
|
}
|
||||||
|
})->call($this->storage);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* icewind's NativeState and the smb:// URL of $abs behind files_external's SMB storage, reached
|
||||||
|
* through bound closures; null when the storage is not SMB through the native extension or its
|
||||||
|
* internals are not as expected.
|
||||||
|
*
|
||||||
|
* @return array{0: object, 1: string}|null
|
||||||
|
* @psalm-suppress UndefinedClass icewind/smb comes with files_external, not with this app
|
||||||
|
*/
|
||||||
|
private function smbHandle(string $abs): ?array {
|
||||||
|
if (!$this->storage instanceof \OCA\Files_External\Lib\Storage\SMB || !extension_loaded('smbclient')) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
/** @var array{0: object, 1: string}|null $handle */
|
||||||
|
$handle = (function (string $path): ?array {
|
||||||
|
/** @psalm-suppress UndefinedThisPropertyFetch */
|
||||||
|
$share = $this->share;
|
||||||
|
if (!$share instanceof \Icewind\SMB\Native\NativeShare) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
/** @psalm-suppress UndefinedMethod */
|
||||||
|
$sharePath = ltrim($this->buildPath($path), '/');
|
||||||
|
/** @psalm-suppress UndefinedMethod */
|
||||||
|
return (fn (string $p): array => [$this->getState(), $this->buildUrl($p)])->call($share, $sharePath);
|
||||||
|
})->call($this->storage, $abs);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return $handle;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* readAhead() over SMB: one range after the other, each in SMB_READ reads into a file in the
|
||||||
|
* spool. Not several at once (PHP cannot wait on more than one libsmbclient call), but whole
|
||||||
|
* ranges of up to ReadAheadPlan::MAX_RANGE in large reads instead of blob by blob in 1 MiB
|
||||||
|
* reads. A range that cannot be read is handed out as null.
|
||||||
|
*
|
||||||
|
* @param list<array{0:string, 1:int, 2:int}> $ranges
|
||||||
|
* @return \Generator<int, resource|null>
|
||||||
|
*/
|
||||||
|
private function smbReadAhead(array $ranges, string $spool): \Generator {
|
||||||
|
// before anything is read: an invalid path throws
|
||||||
|
$abs = array_map(fn (array $r): array => [$this->abs($r[0]), $r[1], $r[2]], $ranges);
|
||||||
|
return (function () use ($abs, $spool): \Generator {
|
||||||
|
$file = rtrim($spool, '/') . '/smb-' . bin2hex(random_bytes(4));
|
||||||
|
try {
|
||||||
|
foreach ($abs as $i => [$path, $offset, $length]) {
|
||||||
|
$fh = null;
|
||||||
|
$out = false;
|
||||||
|
try {
|
||||||
|
$chunks = $this->smbChunks($path, $offset, $length);
|
||||||
|
$out = $chunks === null ? false : fopen($file, 'w+b');
|
||||||
|
if ($chunks !== null && $out !== false) {
|
||||||
|
foreach ($chunks as $chunk) {
|
||||||
|
if (fwrite($out, $chunk) !== strlen($chunk)) {
|
||||||
|
throw new BackendException('Cannot write to the spool');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$fh = ftell($out) === $length && rewind($out) ? $out : null;
|
||||||
|
if ($fh === null) {
|
||||||
|
fclose($out);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (\Throwable) {
|
||||||
|
if (is_resource($out)) {
|
||||||
|
fclose($out);
|
||||||
|
}
|
||||||
|
$fh = null; // read the usual way, which reports why, over a new connection
|
||||||
|
$this->smbReconnect();
|
||||||
|
} finally {
|
||||||
|
@unlink($file); // an open stream keeps its data
|
||||||
|
}
|
||||||
|
yield $i => $fh;
|
||||||
|
if (is_resource($fh)) {
|
||||||
|
fclose($fh);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
@unlink($file);
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
}
|
||||||
|
|
||||||
private function smbNames(string $abs): ?array {
|
private function smbNames(string $abs): ?array {
|
||||||
if (!$this->storage instanceof \OCA\Files_External\Lib\Storage\SMB) {
|
if (!$this->storage instanceof \OCA\Files_External\Lib\Storage\SMB) {
|
||||||
return null;
|
return null;
|
||||||
|
|||||||
@@ -43,25 +43,27 @@ final class BackupJob extends TimedJob {
|
|||||||
$this->setTimeSensitivity(IJob::TIME_SENSITIVE);
|
$this->setTimeSensitivity(IJob::TIME_SENSITIVE);
|
||||||
}
|
}
|
||||||
|
|
||||||
private const MAX_DAILY_ATTEMPTS = 3;
|
|
||||||
|
|
||||||
protected function run($argument): void {
|
protected function run($argument): void {
|
||||||
$budget = PHP_SAPI === 'cli' ? 240.0 : 20.0;
|
$budget = PHP_SAPI === 'cli' ? 240.0 : 20.0;
|
||||||
$deadline = microtime(true) + $budget;
|
$deadline = microtime(true) + $budget;
|
||||||
$this->startScheduled();
|
$this->startScheduled();
|
||||||
foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) {
|
// One after the other: runs that read the same disks at the same time are not faster together.
|
||||||
|
foreach ($this->backups->runsInTurn() as $run) {
|
||||||
while (microtime(true) < $deadline && $run->getStatus() === 'running') {
|
while (microtime(true) < $deadline && $run->getStatus() === 'running') {
|
||||||
$next = $this->backups->step($run, min($deadline, microtime(true) + $budget));
|
$next = $this->backups->step($run, min($deadline, microtime(true) + $budget));
|
||||||
if ($next === $run) {
|
if ($next === $run) {
|
||||||
break; // step() hands back the same object when another process holds the run's lease
|
break 2; // step() hands back the same object when another process holds the run's lease
|
||||||
}
|
}
|
||||||
$run = $next;
|
$run = $next;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
$this->failStaleRestores();
|
$this->failStaleRestores();
|
||||||
$this->weeklyPrune();
|
$this->weeklyPrune();
|
||||||
foreach ($this->targets->list() as $t) {
|
$targets = $this->targets->list();
|
||||||
|
foreach ($targets as $t) {
|
||||||
$this->alerts->checkStale($t);
|
$this->alerts->checkStale($t);
|
||||||
|
$othersActive = array_filter($targets, static fn ($o) => $o->getId() !== $t->getId() && (string)$o->getSchedule() !== '') !== [];
|
||||||
|
$this->alerts->checkOff($t, $othersActive);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,21 +104,18 @@ final class BackupJob extends TimedJob {
|
|||||||
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
|
$this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */
|
/** Start the backups the schedules of the locations ask for; they run in turn (see LocationSchedule). */
|
||||||
private function startScheduled(): void {
|
private function startScheduled(): void {
|
||||||
$todayAt = $this->schedule->todayAt();
|
$now = time();
|
||||||
if ($todayAt === null || time() < $todayAt) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
foreach ($this->targets->list() as $t) {
|
foreach ($this->targets->list() as $t) {
|
||||||
$last = $this->runs->lastFinished($t->getId(), BackupService::KIND_FULL);
|
$moment = \OCA\NgBackup\Service\LocationSchedule::lastMoment((string)$t->getSchedule(), $this->schedule->zoneOr($t->getScheduleZone()), $now);
|
||||||
if ($this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== [] || ($last !== null && $last->getStartedAt() >= $todayAt)) {
|
if ($moment === null || $this->runs->findRunning($t->getId(), BackupService::KIND_FULL) !== []) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
// After a failure: retry at most MAX_DAILY_ATTEMPTS times a day, an hour apart, instead
|
$attempts = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $moment);
|
||||||
// of starting (and notifying about) a new full run every five minutes all day.
|
if (!\OCA\NgBackup\Service\LocationSchedule::isDue($moment, (int)$t->getScheduleSince(),
|
||||||
$today = $this->runs->startedSince($t->getId(), BackupService::KIND_FULL, $todayAt);
|
$this->runs->lastFinished($t->getId(), BackupService::KIND_FULL)?->getStartedAt(),
|
||||||
if (count($today) >= self::MAX_DAILY_ATTEMPTS || ($today !== [] && time() - (int)$today[0]->getFinishedAt() < 3600)) {
|
count($attempts), ($attempts[0] ?? null)?->getFinishedAt(), $now)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
|
|||||||
@@ -7,7 +7,9 @@ declare(strict_types=1);
|
|||||||
|
|
||||||
namespace OCA\NgBackup\Command;
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Format;
|
||||||
use OCA\NgBackup\Service\RestoreService;
|
use OCA\NgBackup\Service\RestoreService;
|
||||||
|
use OCA\NgBackup\Service\Schedule;
|
||||||
use OCA\NgBackup\Service\TargetService;
|
use OCA\NgBackup\Service\TargetService;
|
||||||
use Symfony\Component\Console\Command\Command;
|
use Symfony\Component\Console\Command\Command;
|
||||||
use Symfony\Component\Console\Helper\Table;
|
use Symfony\Component\Console\Helper\Table;
|
||||||
@@ -19,6 +21,7 @@ final class Browse extends Command {
|
|||||||
public function __construct(
|
public function __construct(
|
||||||
private TargetService $targets,
|
private TargetService $targets,
|
||||||
private RestoreService $restore,
|
private RestoreService $restore,
|
||||||
|
private Schedule $schedule,
|
||||||
) {
|
) {
|
||||||
parent::__construct();
|
parent::__construct();
|
||||||
}
|
}
|
||||||
@@ -43,9 +46,10 @@ final class Browse extends Command {
|
|||||||
$output->writeln('Nothing here.');
|
$output->writeln('Nothing here.');
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
$zone = $this->schedule->get()['timezone'];
|
||||||
$rows = array_map(fn ($i) => [$i['type'] === 'dir' ? $i['name'] . '/' : $i['name'], $this->size($i['size']),
|
$rows = array_map(fn ($i) => [$i['type'] === 'dir' ? $i['name'] . '/' : $i['name'], $this->size($i['size']),
|
||||||
$i['type'] === 'dir' ? $i['files'] . ' files' : date('Y-m-d H:i', (int)$i['mtime'])], $items);
|
$i['type'] === 'dir' ? $i['files'] . ' files' : Format::time((int)$i['mtime'], $zone)], $items);
|
||||||
(new Table($output))->setHeaders(['name', 'size', 'files / modified'])->setRows($rows)->render();
|
(new Table($output))->setHeaders(['name', 'size', "files / modified ($zone)"])->setRows($rows)->render();
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -8,6 +8,8 @@ declare(strict_types=1);
|
|||||||
namespace OCA\NgBackup\Command;
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
use OCA\NgBackup\Db\SnapshotMapper;
|
use OCA\NgBackup\Db\SnapshotMapper;
|
||||||
|
use OCA\NgBackup\Format;
|
||||||
|
use OCA\NgBackup\Service\Schedule;
|
||||||
use OCA\NgBackup\Service\TargetService;
|
use OCA\NgBackup\Service\TargetService;
|
||||||
use Symfony\Component\Console\Command\Command;
|
use Symfony\Component\Console\Command\Command;
|
||||||
use Symfony\Component\Console\Helper\Table;
|
use Symfony\Component\Console\Helper\Table;
|
||||||
@@ -19,6 +21,7 @@ final class ListSnapshots extends Command {
|
|||||||
public function __construct(
|
public function __construct(
|
||||||
private TargetService $targets,
|
private TargetService $targets,
|
||||||
private SnapshotMapper $snapshots,
|
private SnapshotMapper $snapshots,
|
||||||
|
private Schedule $schedule,
|
||||||
) {
|
) {
|
||||||
parent::__construct();
|
parent::__construct();
|
||||||
}
|
}
|
||||||
@@ -32,18 +35,19 @@ final class ListSnapshots extends Command {
|
|||||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
$name = $input->getArgument('target');
|
$name = $input->getArgument('target');
|
||||||
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
|
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
|
||||||
|
$zone = $this->schedule->get()['timezone'];
|
||||||
$rows = [];
|
$rows = [];
|
||||||
foreach ($targets as $t) {
|
foreach ($targets as $t) {
|
||||||
foreach ($this->snapshots->findForTarget($t->getId()) as $s) {
|
foreach ($this->snapshots->findForTarget($t->getId()) as $s) {
|
||||||
$rows[] = [$t->getName(), $s->getSnapshotId(), date('Y-m-d H:i', $s->getCreatedAt()), $s->getKind(),
|
$rows[] = [$t->getName(), $s->getSnapshotId(), Format::time($s->getCreatedAt(), $zone), $s->getKind(),
|
||||||
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getLabel() ?? ''];
|
$s->getFiles(), sprintf('%.1f GiB', ($s->getBytes() ?? 0) / 1073741824), $s->getPinned() ? 'yes' : '', $s->getLabel() ?? '', $s->getNote() ?? ''];
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if ($rows === []) {
|
if ($rows === []) {
|
||||||
$output->writeln('No snapshots yet (occ backup:run).');
|
$output->writeln('No snapshots yet (occ backup:run).');
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
(new Table($output))->setHeaders(['location', 'snapshot', 'date', 'kind', 'files', 'size', 'label'])->setRows($rows)->render();
|
(new Table($output))->setHeaders(['location', 'snapshot', "date ($zone)", 'kind', 'files', 'size', 'pinned', 'label', 'note'])->setRows($rows)->render();
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,64 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\TargetService;
|
||||||
|
use Symfony\Component\Console\Command\Command;
|
||||||
|
use Symfony\Component\Console\Helper\Table;
|
||||||
|
use Symfony\Component\Console\Input\InputArgument;
|
||||||
|
use Symfony\Component\Console\Input\InputInterface;
|
||||||
|
use Symfony\Component\Console\Output\OutputInterface;
|
||||||
|
|
||||||
|
final class ListUserExports extends Command {
|
||||||
|
public function __construct(
|
||||||
|
private TargetService $targets,
|
||||||
|
) {
|
||||||
|
parent::__construct();
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function configure(): void {
|
||||||
|
$this->setName('backup:user:list')
|
||||||
|
->setDescription('List the user exports (backup:user:backup) on a location')
|
||||||
|
->addArgument('target', InputArgument::OPTIONAL, 'Location name or id (default: all)')
|
||||||
|
->addArgument('user', InputArgument::OPTIONAL, 'Only the exports of this user id')
|
||||||
|
->setHelp(<<<'HELP'
|
||||||
|
The manifest column is what occ backup:user:restore expects. A safety export made by a restore
|
||||||
|
with --mode=replace is listed here as well.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
occ backup:user:list offsite alice
|
||||||
|
HELP);
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$name = $input->getArgument('target');
|
||||||
|
$uid = $input->getArgument('user');
|
||||||
|
$rows = [];
|
||||||
|
try {
|
||||||
|
$targets = $name === null ? $this->targets->list() : [$this->targets->get($name)];
|
||||||
|
foreach ($targets as $t) {
|
||||||
|
foreach ($this->targets->repository($t)->listUserExports($uid) as $path) {
|
||||||
|
$parts = explode('/', $path, 3);
|
||||||
|
$user = $parts[1] ?? '';
|
||||||
|
$exportId = $parts[2] ?? '';
|
||||||
|
$time = \DateTimeImmutable::createFromFormat('Ymd\THis\Z', substr($exportId, 0, 16), new \DateTimeZone('UTC'));
|
||||||
|
$rows[] = [$t->getName(), $user, $time !== false ? $time->format('Y-m-d H:i') : '', $path];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
if ($rows === []) {
|
||||||
|
$output->writeln('No user exports yet (occ backup:user:backup).');
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
(new Table($output))->setHeaders(['location', 'user', 'date (UTC)', 'manifest'])->setRows($rows)->render();
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\PruneService;
|
||||||
|
use OCA\NgBackup\Service\TargetService;
|
||||||
|
use Symfony\Component\Console\Command\Command;
|
||||||
|
use Symfony\Component\Console\Input\InputArgument;
|
||||||
|
use Symfony\Component\Console\Input\InputInterface;
|
||||||
|
use Symfony\Component\Console\Output\OutputInterface;
|
||||||
|
|
||||||
|
final class Note extends Command {
|
||||||
|
public function __construct(
|
||||||
|
private TargetService $targets,
|
||||||
|
private PruneService $prune,
|
||||||
|
) {
|
||||||
|
parent::__construct();
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function configure(): void {
|
||||||
|
$this->setName('backup:note')
|
||||||
|
->setDescription('Set or remove the note on a snapshot')
|
||||||
|
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
||||||
|
->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)')
|
||||||
|
->addArgument('note', InputArgument::OPTIONAL, 'The note, at most ' . PruneService::NOTE_MAX . ' characters; without it the note is removed', '');
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$note = (string)$input->getArgument('note');
|
||||||
|
try {
|
||||||
|
$this->prune->setNote($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), $note);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
$output->writeln(trim($note) === '' ? 'Note removed.' : 'Note saved.');
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\PruneService;
|
||||||
|
use OCA\NgBackup\Service\TargetService;
|
||||||
|
use Symfony\Component\Console\Command\Command;
|
||||||
|
use Symfony\Component\Console\Input\InputArgument;
|
||||||
|
use Symfony\Component\Console\Input\InputInterface;
|
||||||
|
use Symfony\Component\Console\Input\InputOption;
|
||||||
|
use Symfony\Component\Console\Output\OutputInterface;
|
||||||
|
|
||||||
|
final class Pin extends Command {
|
||||||
|
public function __construct(
|
||||||
|
private TargetService $targets,
|
||||||
|
private PruneService $prune,
|
||||||
|
) {
|
||||||
|
parent::__construct();
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function configure(): void {
|
||||||
|
$this->setName('backup:pin')
|
||||||
|
->setDescription('Keep a snapshot: the retention policy does not remove a pinned one')
|
||||||
|
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
||||||
|
->addArgument('snapshot', InputArgument::REQUIRED, 'Snapshot id (see backup:list)')
|
||||||
|
->addOption('unpin', null, InputOption::VALUE_NONE, 'Leave the snapshot to the retention policy again');
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$unpin = (bool)$input->getOption('unpin');
|
||||||
|
try {
|
||||||
|
$this->prune->setPinned($this->targets->get($input->getArgument('target')), $input->getArgument('snapshot'), !$unpin);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
$output->writeln($unpin ? 'Snapshot unpinned; the retention policy applies to it again.' : 'Snapshot pinned; the retention policy leaves it alone.');
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -31,6 +31,7 @@ final class Restore extends Command {
|
|||||||
->addArgument('path', InputArgument::REQUIRED, 'Path in the snapshot (see backup:browse)')
|
->addArgument('path', InputArgument::REQUIRED, 'Path in the snapshot (see backup:browse)')
|
||||||
->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'new-folder, merge or replace (user files)', RestoreService::MODE_NEW_FOLDER)
|
->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'new-folder, merge or replace (user files)', RestoreService::MODE_NEW_FOLDER)
|
||||||
->addOption('to-directory', 'd', InputOption::VALUE_REQUIRED, 'Write to this local directory instead (any path, raw files)')
|
->addOption('to-directory', 'd', InputOption::VALUE_REQUIRED, 'Write to this local directory instead (any path, raw files)')
|
||||||
|
->addOption('salvage', null, InputOption::VALUE_NONE, 'Leave out a file that cannot be read or decrypted, list it at the end and go on, instead of stopping')
|
||||||
->setHelp(<<<'HELP'
|
->setHelp(<<<'HELP'
|
||||||
User files (data/<user>/files/...) are restored through Nextcloud, so versions, trash and the
|
User files (data/<user>/files/...) are restored through Nextcloud, so versions, trash and the
|
||||||
file cache stay correct. Modes:
|
file cache stay correct. Modes:
|
||||||
@@ -42,30 +43,59 @@ Examples:
|
|||||||
occ backup:restore offsite 537bf7e9... data/alice/files/Documents
|
occ backup:restore offsite 537bf7e9... data/alice/files/Documents
|
||||||
occ backup:restore offsite 537bf7e9... data/alice/files/Documents/plan.odt --mode=merge
|
occ backup:restore offsite 537bf7e9... data/alice/files/Documents/plan.odt --mode=merge
|
||||||
occ backup:restore offsite 537bf7e9... config --to-directory=/root/restored-config
|
occ backup:restore offsite 537bf7e9... config --to-directory=/root/restored-config
|
||||||
|
With --salvage, a file whose data on the location is damaged or missing is left out and listed
|
||||||
|
at the end; everything else is restored. The command then exits with code 2.
|
||||||
HELP);
|
HELP);
|
||||||
}
|
}
|
||||||
|
|
||||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$skipped = $input->getOption('salvage') ? [] : null;
|
||||||
try {
|
try {
|
||||||
$target = $this->targets->get($input->getArgument('target'));
|
$target = $this->targets->get($input->getArgument('target'));
|
||||||
$snapshot = $input->getArgument('snapshot');
|
$snapshot = $input->getArgument('snapshot');
|
||||||
$path = trim((string)$input->getArgument('path'), '/'); // CLI convenience; the service itself only accepts canonical paths
|
$path = trim((string)$input->getArgument('path'), '/'); // CLI convenience; the service itself only accepts canonical paths
|
||||||
if ($dir = $input->getOption('to-directory')) {
|
if ($dir = $input->getOption('to-directory')) {
|
||||||
$n = $this->restore->restoreToDirectory($target, $snapshot, $path, $dir);
|
$n = $this->restore->restoreToDirectory($target, $snapshot, $path, $dir, false, null, $skipped);
|
||||||
$output->writeln("Restored <info>$n</info> files to $dir.");
|
$output->writeln("Restored <info>$n</info> files to $dir.");
|
||||||
return 0;
|
return self::reportSkipped($skipped, $output);
|
||||||
}
|
}
|
||||||
$r = $this->restore->restoreUserFiles($target, $snapshot, $path, (string)$input->getOption('mode'),
|
$r = $this->restore->restoreUserFiles($target, $snapshot, $path, (string)$input->getOption('mode'),
|
||||||
function (int $done, int $total, string $name) use ($output) {
|
function (int $done, int $total, string $name) use ($output) {
|
||||||
if ($output->isVerbose() || $done === $total || $done % 100 === 0) {
|
if ($output->isVerbose() || $done === $total || $done % 100 === 0) {
|
||||||
$output->writeln(" $done/$total $name");
|
$output->writeln(" $done/$total $name");
|
||||||
}
|
}
|
||||||
});
|
}, $skipped);
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
if ($skipped === null && self::backupLooksDamaged($e)) {
|
||||||
|
$output->writeln('The data of a file on the location is damaged or missing. Run the same command with --salvage to restore everything else and list what could not be restored.');
|
||||||
|
}
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
$output->writeln(sprintf('Restored <info>%d</info> files to %s%s.', $r['restored'], $r['target'], $r['trashed'] ? ", moved {$r['trashed']} extra files to the trash" : ''));
|
$output->writeln(sprintf('Restored <info>%d</info> files to %s%s.', $r['restored'], $r['target'], $r['trashed'] ? ", moved {$r['trashed']} extra files to the trash" : ''));
|
||||||
|
return self::reportSkipped($skipped, $output);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Damaged data, or a file that is gone from the location: what --salvage is for. */
|
||||||
|
public static function backupLooksDamaged(\Throwable $e): bool {
|
||||||
|
return $e instanceof \OCA\NgBackup\Repository\RepositoryException
|
||||||
|
|| $e instanceof \OCA\NgBackup\Crypto\CryptoException
|
||||||
|
|| ($e instanceof \OCA\NgBackup\Backend\BackendException && str_starts_with($e->getMessage(), 'Not found'));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The files --salvage left out; exit code 2 when there are any.
|
||||||
|
*
|
||||||
|
* @param array<string, string>|null $skipped
|
||||||
|
*/
|
||||||
|
public static function reportSkipped(?array $skipped, OutputInterface $output): int {
|
||||||
|
if ($skipped === null || $skipped === []) {
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
$output->writeln(sprintf('<comment>%d file%s could not be restored (damaged or missing on the location):</comment>', count($skipped), count($skipped) === 1 ? '' : 's'));
|
||||||
|
foreach ($skipped as $path => $reason) {
|
||||||
|
$output->writeln(" $path: $reason");
|
||||||
|
}
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ final class RestoreFull extends Command {
|
|||||||
->addOption('force', null, InputOption::VALUE_NONE, 'Also on an installation that already has more than one user, and without asking')
|
->addOption('force', null, InputOption::VALUE_NONE, 'Also on an installation that already has more than one user, and without asking')
|
||||||
->addOption('install-apps', null, InputOption::VALUE_NONE, 'First install (from the app store) or enable the apps that were enabled at backup time and are missing here')
|
->addOption('install-apps', null, InputOption::VALUE_NONE, 'First install (from the app store) or enable the apps that were enabled at backup time and are missing here')
|
||||||
->addOption('skip-missing-tables', null, InputOption::VALUE_NONE, 'Older backups only: leave out tables that do not exist here and whose structure the backup does not record')
|
->addOption('skip-missing-tables', null, InputOption::VALUE_NONE, 'Older backups only: leave out tables that do not exist here and whose structure the backup does not record')
|
||||||
|
->addOption('salvage', null, InputOption::VALUE_NONE, 'Restore what can be restored: leave out files that cannot be read, and when the database cannot be restored, put the files in a folder of their own')
|
||||||
->setHelp(<<<'HELP'
|
->setHelp(<<<'HELP'
|
||||||
For a FRESH Nextcloud installation only, same major version as the backup, right after
|
For a FRESH Nextcloud installation only, same major version as the backup, right after
|
||||||
occ maintenance:install, occ backup:key:import and occ backup:target:add -- before anyone else
|
occ maintenance:install, occ backup:key:import and occ backup:target:add -- before anyone else
|
||||||
@@ -55,6 +56,12 @@ here. Other files under config/ (if any) are not restored.
|
|||||||
|
|
||||||
Afterwards: check trusted_domains and overwritehost/overwriteprotocol match this server, and
|
Afterwards: check trusted_domains and overwritehost/overwriteprotocol match this server, and
|
||||||
occ maintenance:repair.
|
occ maintenance:repair.
|
||||||
|
When the backup is damaged: --salvage restores what can be restored. Files whose data cannot be
|
||||||
|
read or decrypted are left out and listed. When the database cannot be restored (its dump is
|
||||||
|
damaged, or it does not fit this installation), the database stays as occ maintenance:install
|
||||||
|
made it, and the data directory and config of the backup go to ngb-salvage-<snapshot> in the
|
||||||
|
data directory instead: from there, the files can be put back by hand. The command then exits
|
||||||
|
with code 2.
|
||||||
|
|
||||||
Example:
|
Example:
|
||||||
occ backup:restore:full offsite 537bf7e9c1a4...
|
occ backup:restore:full offsite 537bf7e9c1a4...
|
||||||
@@ -82,29 +89,41 @@ HELP);
|
|||||||
$target = $this->targets->get($input->getArgument('target'));
|
$target = $this->targets->get($input->getArgument('target'));
|
||||||
$snapshot = (string)$input->getArgument('snapshot');
|
$snapshot = (string)$input->getArgument('snapshot');
|
||||||
$skipMissingTables = (bool)$input->getOption('skip-missing-tables');
|
$skipMissingTables = (bool)$input->getOption('skip-missing-tables');
|
||||||
|
$salvage = (bool)$input->getOption('salvage');
|
||||||
if ($input->getOption('install-apps')) {
|
if ($input->getOption('install-apps')) {
|
||||||
$check = $this->restoreFull->checkCompatibility($target, $snapshot, $skipMissingTables);
|
$check = $this->restoreFull->checkCompatibility($target, $snapshot, $skipMissingTables, $salvage);
|
||||||
foreach ($check['missingApps'] as $app) {
|
foreach ($check['missingApps'] as $app) {
|
||||||
$this->installApp($app, $output);
|
$this->installApp($app, $output);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
$output->writeln('Nextcloud is in maintenance mode while the restore runs; it is switched off when the restore completes.');
|
$output->writeln('Nextcloud is in maintenance mode while the restore runs; it is switched off when the restore completes.');
|
||||||
$result = $this->restoreFull->restoreFull($target, $snapshot, $skipMissingTables);
|
$result = $this->restoreFull->restoreFull($target, $snapshot, $skipMissingTables, $salvage);
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
if (\OCP\Server::get(\OCP\IConfig::class)->getSystemValueBool('maintenance', false)) {
|
if (\OCP\Server::get(\OCP\IConfig::class)->getSystemValueBool('maintenance', false)) {
|
||||||
$output->writeln('The restore had started: the instance may be partly restored, and maintenance mode stays on so that nobody uses it like that.');
|
$output->writeln('The restore had started: the instance may be partly restored, and maintenance mode stays on so that nobody uses it like that.');
|
||||||
$output->writeln('Fix the cause, then switch maintenance mode off and run the same command again with --force, in one line:');
|
$output->writeln('Fix the cause, then switch maintenance mode off and run the same command again with --force, in one line:');
|
||||||
$again = 'occ backup:restore:full ' . escapeshellarg((string)$input->getArgument('target')) . ' ' . escapeshellarg((string)$input->getArgument('snapshot'))
|
$again = 'occ backup:restore:full ' . escapeshellarg((string)$input->getArgument('target')) . ' ' . escapeshellarg((string)$input->getArgument('snapshot'))
|
||||||
. ($input->getOption('skip-missing-tables') ? ' --skip-missing-tables' : '') . ' --force --no-interaction';
|
. ($input->getOption('skip-missing-tables') ? ' --skip-missing-tables' : '') . ($input->getOption('salvage') ? ' --salvage' : '') . ' --force --no-interaction';
|
||||||
$output->writeln(' occ maintenance:mode --off && ' . $again);
|
$output->writeln(' occ maintenance:mode --off && ' . $again);
|
||||||
$output->writeln('(Without --install-apps, so the instance is open only for the seconds until the restore starts again. Apps the first run could not install are listed at the end: install those afterwards.)');
|
$output->writeln('(Without --install-apps, so the instance is open only for the seconds until the restore starts again. Apps the first run could not install are listed at the end: install those afterwards.)');
|
||||||
} else {
|
} else {
|
||||||
$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
|
$output->writeln('If this happened after the restore had started, the instance may be partly restored: fix the cause and run the same command again with --force.');
|
||||||
}
|
}
|
||||||
$output->writeln('If occ then says that Nextcloud needs an upgrade (the database is already the old server\'s), run occ upgrade first.');
|
$output->writeln('If occ then says that Nextcloud needs an upgrade (the database is already the old server\'s), run occ upgrade first.');
|
||||||
|
if (!$input->getOption('salvage') && Restore::backupLooksDamaged($e)) {
|
||||||
|
$output->writeln('The backup itself looks damaged here. --salvage restores everything that can be read and lists the rest.');
|
||||||
|
}
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
|
if ($result['dbError'] !== null) {
|
||||||
|
$output->writeln('<error>Salvage: ' . $result['dbError'] . '</error>');
|
||||||
|
$output->writeln(sprintf('The database stays as occ maintenance:install made it. The data directory and config of the backup (<info>%d</info> data files) are in:', $result['dataFiles']));
|
||||||
|
$output->writeln(' ' . $result['salvageDir']);
|
||||||
|
$output->writeln('Each user\'s files are in data/<user>/files there. Put back what is needed by hand, for example into the files of new users (then occ files:scan).');
|
||||||
|
Restore::reportSkipped($result['skipped'], $output);
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
$output->writeln(sprintf(
|
$output->writeln(sprintf(
|
||||||
'Database restored (<info>%d</info> tables, <info>%d</info> unchanged), config merged (<info>%d</info> keys), <info>%d</info> data files restored.',
|
'Database restored (<info>%d</info> tables, <info>%d</info> unchanged), config merged (<info>%d</info> keys), <info>%d</info> data files restored.',
|
||||||
count($result['dbReport']['restored']),
|
count($result['dbReport']['restored']),
|
||||||
@@ -135,7 +154,12 @@ HELP);
|
|||||||
$output->writeln('<comment>Maintenance mode was on before the restore and stays on: switch it off with occ maintenance:mode --off when the instance is ready.</comment>');
|
$output->writeln('<comment>Maintenance mode was on before the restore and stays on: switch it off with occ maintenance:mode --off when the instance is ready.</comment>');
|
||||||
}
|
}
|
||||||
$output->writeln('Next: check trusted_domains and overwritehost/overwriteprotocol match this server, then occ maintenance:repair.');
|
$output->writeln('Next: check trusted_domains and overwritehost/overwriteprotocol match this server, then occ maintenance:repair.');
|
||||||
return 0;
|
$code = 0;
|
||||||
|
if ($result['configError'] !== null) {
|
||||||
|
$output->writeln('<comment>Salvage: config.php of the backup was not merged (' . $result['configError'] . '). Set at least \'instanceid\', \'passwordsalt\' and \'secret\' to the old server\'s values if you have them, or data encrypted with them (server-side encryption, stored passwords) cannot be read.</comment>');
|
||||||
|
$code = 2;
|
||||||
|
}
|
||||||
|
return max($code, Restore::reportSkipped($result['skipped'], $output));
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Enable an app that is here but switched off, or install it from the app store. */
|
/** Enable an app that is here but switched off, or install it from the app store. */
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ final class RestoreUser extends Command {
|
|||||||
$this->setName('backup:user:restore')
|
$this->setName('backup:user:restore')
|
||||||
->setDescription('Restore a user from a user_migration export (see backup:user:backup)')
|
->setDescription('Restore a user from a user_migration export (see backup:user:backup)')
|
||||||
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
||||||
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>)')
|
->addArgument('manifest', InputArgument::REQUIRED, 'Manifest path printed by backup:user:backup (users/<uid>/<exportId>), or a user id with exactly one export')
|
||||||
->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'replace or as-backup', UserRestoreService::MODE_REPLACE)
|
->addOption('mode', 'm', InputOption::VALUE_REQUIRED, 'replace or as-backup', UserRestoreService::MODE_REPLACE)
|
||||||
->addOption('force', null, InputOption::VALUE_NONE, 'Replace an existing account without asking (required when not interactive)')
|
->addOption('force', null, InputOption::VALUE_NONE, 'Replace an existing account without asking (required when not interactive)')
|
||||||
->addOption('install-user-migration', null, InputOption::VALUE_NONE, 'Install and enable user_migration if it is missing (non-interactive consent)')
|
->addOption('install-user-migration', null, InputOption::VALUE_NONE, 'Install and enable user_migration if it is missing (non-interactive consent)')
|
||||||
@@ -45,16 +45,31 @@ Never imports over a live account. Modes:
|
|||||||
compare and transfer by hand; Nextcloud cannot rename a user id, so the backup gets
|
compare and transfer by hand; Nextcloud cannot rename a user id, so the backup gets
|
||||||
the new name, never the existing account
|
the new name, never the existing account
|
||||||
|
|
||||||
|
occ backup:user:list shows the exports on a location. A user id alone works when that user
|
||||||
|
has exactly one export there; with more, the command lists them and you pass one.
|
||||||
|
|
||||||
Example:
|
Example:
|
||||||
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34
|
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34
|
||||||
|
occ backup:user:restore offsite alice
|
||||||
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34 --mode=as-backup
|
occ backup:user:restore offsite users/alice/20261005T120000Z-ab12cd34 --mode=as-backup
|
||||||
HELP);
|
HELP);
|
||||||
}
|
}
|
||||||
|
|
||||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
$mode = (string)$input->getOption('mode');
|
$mode = (string)$input->getOption('mode');
|
||||||
$uid = explode('/', (string)$input->getArgument('manifest'))[1] ?? '';
|
try {
|
||||||
if ($mode === UserRestoreService::MODE_REPLACE && $uid !== '' && $this->userManager->userExists($uid) && !$input->getOption('force')) {
|
$target = $this->targets->get($input->getArgument('target'));
|
||||||
|
$ref = (string)$input->getArgument('manifest');
|
||||||
|
$manifest = $this->targets->repository($target)->resolveUserExport($ref);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
if ($manifest !== $ref) {
|
||||||
|
$output->writeln("Using $manifest.");
|
||||||
|
}
|
||||||
|
$uid = explode('/', $manifest)[1];
|
||||||
|
if ($mode === UserRestoreService::MODE_REPLACE && $this->userManager->userExists($uid) && !$input->getOption('force')) {
|
||||||
// Replace deletes the live account (after a safety export). Shares to others and data of
|
// Replace deletes the live account (after a safety export). Shares to others and data of
|
||||||
// apps without a user_migration migrator do not come back, so never do it unasked.
|
// apps without a user_migration migrator do not come back, so never do it unasked.
|
||||||
if (!$input->isInteractive()) {
|
if (!$input->isInteractive()) {
|
||||||
@@ -69,9 +84,8 @@ HELP);
|
|||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
$output->writeln('<comment>Note: on restore, user_migration only imports the app settings on its own allowlist (e.g. calendar view and reminder settings); other app settings of this user are skipped. Reason: NG Backup uses user_migration\'s importer; widening the list would mean building a settings importer of our own.</comment>');
|
$output->writeln('<comment>Note: on restore, user_migration only imports the app settings on its own allowlist (e.g. calendar view and reminder settings); other app settings of this user are skipped. Reason: NG Backup uses user_migration\'s importer; widening the list would mean building a settings importer of our own.</comment>');
|
||||||
$target = $this->targets->get($input->getArgument('target'));
|
|
||||||
$this->installer->ensureEnabled($input, $output);
|
$this->installer->ensureEnabled($input, $output);
|
||||||
$user = $this->userRestore->restoreUser($target, (string)$input->getArgument('manifest'), $mode);
|
$user = $this->userRestore->restoreUser($target, $manifest, $mode);
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
return 1;
|
return 1;
|
||||||
|
|||||||
@@ -79,6 +79,9 @@ HELP);
|
|||||||
if (($f['unreadable'] ?? 0) > 0) {
|
if (($f['unreadable'] ?? 0) > 0) {
|
||||||
$output->writeln(sprintf('<comment> warning: %d unreadable files or folders are missing from this backup (see the server log)</comment>', $f['unreadable']));
|
$output->writeln(sprintf('<comment> warning: %d unreadable files or folders are missing from this backup (see the server log)</comment>', $f['unreadable']));
|
||||||
}
|
}
|
||||||
|
if (($f['badNames'] ?? 0) > 0) {
|
||||||
|
$output->writeln(sprintf('<comment> warning: %d files or folders are missing from this backup because their name is not supported; rename them and the next backup includes them (see the server log)</comment>', $f['badNames']));
|
||||||
|
}
|
||||||
if (($f['changing'] ?? 0) > 0) {
|
if (($f['changing'] ?? 0) > 0) {
|
||||||
$output->writeln(sprintf('<comment> warning: %d files kept changing and were saved as read (see the server log)</comment>', $f['changing']));
|
$output->writeln(sprintf('<comment> warning: %d files kept changing and were saved as read (see the server log)</comment>', $f['changing']));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,7 +8,10 @@ declare(strict_types=1);
|
|||||||
namespace OCA\NgBackup\Command;
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
use OCA\NgBackup\Db\RunMapper;
|
use OCA\NgBackup\Db\RunMapper;
|
||||||
|
use OCA\NgBackup\Format;
|
||||||
|
use OCA\NgBackup\Service\CronHealth;
|
||||||
use OCA\NgBackup\Service\KeyService;
|
use OCA\NgBackup\Service\KeyService;
|
||||||
|
use OCA\NgBackup\Service\Schedule;
|
||||||
use OCA\NgBackup\Service\TargetService;
|
use OCA\NgBackup\Service\TargetService;
|
||||||
use OCA\NgBackup\Service\VerifyService;
|
use OCA\NgBackup\Service\VerifyService;
|
||||||
use Symfony\Component\Console\Command\Command;
|
use Symfony\Component\Console\Command\Command;
|
||||||
@@ -26,6 +29,9 @@ final class Status extends Command {
|
|||||||
private TargetService $targets,
|
private TargetService $targets,
|
||||||
private RunMapper $runs,
|
private RunMapper $runs,
|
||||||
private VerifyService $verify,
|
private VerifyService $verify,
|
||||||
|
private Schedule $schedule,
|
||||||
|
private CronHealth $cron,
|
||||||
|
private \OCA\NgBackup\Service\BackupService $backups,
|
||||||
) {
|
) {
|
||||||
parent::__construct();
|
parent::__construct();
|
||||||
}
|
}
|
||||||
@@ -36,12 +42,17 @@ final class Status extends Command {
|
|||||||
}
|
}
|
||||||
|
|
||||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$zone = $this->schedule->get()['timezone'];
|
||||||
|
$cronProblem = CronHealth::describe($this->cron->check());
|
||||||
|
if ($cronProblem !== null) {
|
||||||
|
$output->writeln('<error>' . $cronProblem . '</error>');
|
||||||
|
}
|
||||||
if (!$this->keys->isInitialized()) {
|
if (!$this->keys->isInitialized()) {
|
||||||
$output->writeln('Key: <error>not created</error> (occ backup:key:init)');
|
$output->writeln('Key: <error>not created</error> (occ backup:key:init)');
|
||||||
} else {
|
} else {
|
||||||
$c = $this->keys->recoveryKitConfirmation();
|
$c = $this->keys->recoveryKitConfirmation();
|
||||||
$output->writeln('Key: ' . $this->keys->fingerprint() . ', recovery kit '
|
$output->writeln('Key: ' . $this->keys->fingerprint() . ', recovery kit '
|
||||||
. ($c ? 'confirmed by ' . $c['uid'] . ' on ' . date('Y-m-d H:i', $c['time']) : '<error>not confirmed</error> (occ backup:key:confirm)'));
|
. ($c ? 'confirmed by ' . $c['uid'] . ' on ' . Format::time($c['time'], $zone) . " ($zone)" : '<error>not confirmed</error> (occ backup:key:confirm)'));
|
||||||
}
|
}
|
||||||
$names = [];
|
$names = [];
|
||||||
foreach ($this->targets->list() as $t) {
|
foreach ($this->targets->list() as $t) {
|
||||||
@@ -51,15 +62,17 @@ final class Status extends Command {
|
|||||||
$rows = [];
|
$rows = [];
|
||||||
$all = (bool)$input->getOption('all');
|
$all = (bool)$input->getOption('all');
|
||||||
$shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400);
|
$shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400);
|
||||||
|
$queued = array_map(static fn (\OCA\NgBackup\Db\Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1));
|
||||||
foreach ($shown as $r) {
|
foreach ($shown as $r) {
|
||||||
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
|
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
|
||||||
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''),
|
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(),
|
||||||
date('Y-m-d H:i', $r->getStartedAt()), $r->getFinishedAt() ? \OCA\NgBackup\Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
|
in_array($r->getId(), $queued, true) ? 'queued' : ($r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus()),
|
||||||
|
Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '',
|
||||||
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
|
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
|
||||||
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
|
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
|
||||||
}
|
}
|
||||||
if ($rows !== []) {
|
if ($rows !== []) {
|
||||||
(new Table($output))->setHeaders(['run', 'location', 'kind', 'status', 'started', 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render();
|
(new Table($output))->setHeaders(['run', 'location', 'kind', 'status', "started ($zone)", 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render();
|
||||||
}
|
}
|
||||||
$older = $all ? 0 : $this->runs->countAll() - count($shown);
|
$older = $all ? 0 : $this->runs->countAll() - count($shown);
|
||||||
if ($older > 0) {
|
if ($older > 0) {
|
||||||
@@ -84,8 +97,24 @@ final class Status extends Command {
|
|||||||
return sprintf('%.1f TiB', $b);
|
return sprintf('%.1f TiB', $b);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** " 47/183 tables" in the database phase, " 68%" or " ~68%" in the files phase; empty without a figure */
|
||||||
private static function percent(\OCA\NgBackup\Db\Run $r): string {
|
private static function percent(\OCA\NgBackup\Db\Run $r): string {
|
||||||
$p = \OCA\NgBackup\Job\BackupRun::progressOf($r);
|
$p = \OCA\NgBackup\Job\BackupRun::progressOf($r);
|
||||||
return $p === null ? '' : sprintf(' %s%d%%', $p['approximate'] ? '~' : '', (int)floor($p['fraction'] * 100.0));
|
if ($p === null) {
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
if (isset($p['tables'], $p['totalTables'])) {
|
||||||
|
return sprintf(': %d/%d tables', $p['tables'], $p['totalTables']);
|
||||||
|
}
|
||||||
|
return sprintf(' %s%d%%', $p['approximate'] ? '~' : '', (int)floor($p['fraction'] * 100.0));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** "running (db: 47/183 tables)", or "waiting for background jobs (last step 12 min ago)" */
|
||||||
|
private static function runningStatus(\OCA\NgBackup\Db\Run $r): string {
|
||||||
|
$waiting = $r->getKind() === \OCA\NgBackup\Service\BackupService::KIND_FULL ? CronHealth::waitingFor($r) : null;
|
||||||
|
if ($waiting !== null) {
|
||||||
|
return 'waiting for background jobs (' . $r->getPhase() . self::percent($r) . ', last step ' . CronHealth::ago($waiting) . ')';
|
||||||
|
}
|
||||||
|
return 'running (' . $r->getPhase() . self::percent($r) . ')';
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -88,19 +88,21 @@ HELP);
|
|||||||
* @return callable(int, int): void
|
* @return callable(int, int): void
|
||||||
*/
|
*/
|
||||||
private static function progress(OutputInterface $output): callable {
|
private static function progress(OutputInterface $output): callable {
|
||||||
$start = microtime(true);
|
|
||||||
$shown = 0.0;
|
$shown = 0.0;
|
||||||
|
$first = null; // [time, files done] when the first files were in
|
||||||
$inPlace = $output->isDecorated();
|
$inPlace = $output->isDecorated();
|
||||||
return static function (int $done, int $total) use ($output, $start, &$shown, $inPlace): void {
|
return static function (int $done, int $total) use ($output, &$shown, &$first, $inPlace): void {
|
||||||
$now = microtime(true);
|
$now = microtime(true);
|
||||||
|
if ($first === null && $done > 0) {
|
||||||
|
$first = [$now, $done];
|
||||||
|
}
|
||||||
if ($done < $total && $now - $shown < ($inPlace ? 1 : 30)) {
|
if ($done < $total && $now - $shown < ($inPlace ? 1 : 30)) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
$shown = $now;
|
$shown = $now;
|
||||||
$line = sprintf('Reading the index: %s/%s files (%d%%)', number_format($done), number_format($total), intdiv(100 * $done, max(1, $total)));
|
$line = sprintf('Reading the index: %s/%s files (%d%%)', number_format($done), number_format($total), intdiv(100 * $done, max(1, $total)));
|
||||||
$elapsed = $now - $start;
|
$left = $first === null ? null : self::minutesLeft($now - $first[0], $done - $first[1], $total - $done, $total);
|
||||||
if ($done < $total && $done > 0 && $elapsed >= 5) {
|
if ($left !== null) {
|
||||||
$left = (int)ceil($elapsed / (float)$done * (float)($total - $done) / 60.0);
|
|
||||||
$line .= $left <= 1 ? ', less than a minute left' : sprintf(', about %d min left', $left);
|
$line .= $left <= 1 ? ', less than a minute left' : sprintf(', about %d min left', $left);
|
||||||
}
|
}
|
||||||
if ($inPlace) {
|
if ($inPlace) {
|
||||||
@@ -110,4 +112,17 @@ HELP);
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Minutes left for reading the index, from the rate since the first files were in: the first
|
||||||
|
* file also carries the time to connect, which made an S3 location with 4,568 index files
|
||||||
|
* say "about 642 min left" for something done within a minute. No estimate until 5% of the
|
||||||
|
* files, or 50 files, came in after that, and at least 5 seconds passed.
|
||||||
|
*/
|
||||||
|
public static function minutesLeft(float $elapsed, int $done, int $left, int $total): ?int {
|
||||||
|
if ($left <= 0 || $done <= 0 || $elapsed < 5.0 || ($done < 50 && $done * 20 < $total)) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return (int)ceil($elapsed / (float)$done * (float)$left / 60.0);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -31,7 +31,8 @@ final class TargetList extends Base {
|
|||||||
$rows = [];
|
$rows = [];
|
||||||
foreach ($this->targets->list() as $t) {
|
foreach ($this->targets->list() as $t) {
|
||||||
$row = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
|
$row = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
|
||||||
'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt())];
|
'repository' => $t->getRepositoryId(), 'added' => date('Y-m-d H:i', $t->getCreatedAt()),
|
||||||
|
'schedule' => (string)$t->getSchedule() !== '' ? $t->getSchedule() : 'off'];
|
||||||
if ($input->getOption('test')) {
|
if ($input->getOption('test')) {
|
||||||
try {
|
try {
|
||||||
$repo = $this->targets->repository($t);
|
$repo = $this->targets->repository($t);
|
||||||
|
|||||||
@@ -0,0 +1,50 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\Schedule;
|
||||||
|
use OCA\NgBackup\Service\TargetService;
|
||||||
|
use Symfony\Component\Console\Command\Command;
|
||||||
|
use Symfony\Component\Console\Input\InputArgument;
|
||||||
|
use Symfony\Component\Console\Input\InputInterface;
|
||||||
|
use Symfony\Component\Console\Input\InputOption;
|
||||||
|
use Symfony\Component\Console\Output\OutputInterface;
|
||||||
|
|
||||||
|
final class TargetSchedule extends Command {
|
||||||
|
public function __construct(
|
||||||
|
private TargetService $targets,
|
||||||
|
private Schedule $schedule,
|
||||||
|
) {
|
||||||
|
parent::__construct();
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function configure(): void {
|
||||||
|
$this->setName('backup:target:schedule')
|
||||||
|
->setDescription('Show or set when a location is backed up')
|
||||||
|
->addArgument('target', InputArgument::REQUIRED, 'Location name or id')
|
||||||
|
->addArgument('schedule', InputArgument::OPTIONAL, '"off", HH:MM for once a day, or HH:MM/N for every N hours from HH:MM; omit to show the current one')
|
||||||
|
->addOption('timezone', null, InputOption::VALUE_REQUIRED, 'Time zone of the time, e.g. Europe/Amsterdam');
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$t = $this->targets->get($input->getArgument('target'));
|
||||||
|
$schedule = $input->getArgument('schedule');
|
||||||
|
$zone = $input->getOption('timezone') ?? $this->schedule->zoneOr($t->getScheduleZone());
|
||||||
|
if ($schedule !== null) {
|
||||||
|
try {
|
||||||
|
$t = $this->targets->setSchedule($t, $schedule, (string)$zone);
|
||||||
|
} catch (\InvalidArgumentException $e) {
|
||||||
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$current = (string)$t->getSchedule();
|
||||||
|
$output->writeln($t->getName() . ': ' . ($current === '' ? 'off' : $current . ' (' . $zone . ')'));
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,7 +7,9 @@ declare(strict_types=1);
|
|||||||
|
|
||||||
namespace OCA\NgBackup\Command;
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Format;
|
||||||
use OCA\NgBackup\Service\KeyService;
|
use OCA\NgBackup\Service\KeyService;
|
||||||
|
use OCA\NgBackup\Service\Schedule;
|
||||||
use OCA\NgBackup\Service\TargetService;
|
use OCA\NgBackup\Service\TargetService;
|
||||||
use Symfony\Component\Console\Command\Command;
|
use Symfony\Component\Console\Command\Command;
|
||||||
use Symfony\Component\Console\Helper\Table;
|
use Symfony\Component\Console\Helper\Table;
|
||||||
@@ -19,6 +21,7 @@ final class Trash extends Command {
|
|||||||
public function __construct(
|
public function __construct(
|
||||||
private TargetService $targets,
|
private TargetService $targets,
|
||||||
private KeyService $keys,
|
private KeyService $keys,
|
||||||
|
private Schedule $schedule,
|
||||||
) {
|
) {
|
||||||
parent::__construct();
|
parent::__construct();
|
||||||
}
|
}
|
||||||
@@ -32,11 +35,14 @@ final class Trash extends Command {
|
|||||||
protected function execute(InputInterface $input, OutputInterface $output): int {
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
$name = $input->getArgument('target');
|
$name = $input->getArgument('target');
|
||||||
$delay = $this->keys->deleteDelayDays() * 86400;
|
$delay = $this->keys->deleteDelayDays() * 86400;
|
||||||
|
$zone = $this->schedule->get()['timezone'];
|
||||||
$rows = [];
|
$rows = [];
|
||||||
foreach ($name === null ? $this->targets->list() : [$this->targets->get($name)] as $t) {
|
foreach ($name === null ? $this->targets->list() : [$this->targets->get($name)] as $t) {
|
||||||
foreach ($this->targets->repository($t)->trash() as $id => $info) {
|
foreach ($this->targets->repository($t)->trash() as $id => $info) {
|
||||||
$rows[] = [$t->getName(), $id, (string)($info['time'] ?? ''), $info['label'], date('Y-m-d H:i', $info['forgottenAt']), $info['by'],
|
$time = $info['time'] ?? null;
|
||||||
date('Y-m-d H:i', $info['forgottenAt'] + $delay)];
|
$made = is_string($time) ? strtotime($time) : false;
|
||||||
|
$rows[] = [$t->getName(), $id, $made !== false ? Format::time($made, $zone) : '', $info['label'], Format::time($info['forgottenAt'], $zone), $info['by'],
|
||||||
|
Format::time($info['forgottenAt'] + $delay, $zone)];
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if ($rows === []) {
|
if ($rows === []) {
|
||||||
@@ -44,6 +50,7 @@ final class Trash extends Command {
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
(new Table($output))->setHeaders(['location', 'snapshot', 'made', 'label', 'removed', 'by', 'deleted after'])->setRows($rows)->render();
|
(new Table($output))->setHeaders(['location', 'snapshot', 'made', 'label', 'removed', 'by', 'deleted after'])->setRows($rows)->render();
|
||||||
|
$output->writeln("Times in $zone. The retention policy keeps the newest snapshots and the last of each recent day, week and month; it moves the others here, and they are deleted permanently after " . $this->keys->deleteDelayDays() . ' days.');
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -48,6 +48,7 @@ final class AdminApiController extends Controller {
|
|||||||
private \OCA\NgBackup\Service\KeySlotService $slots,
|
private \OCA\NgBackup\Service\KeySlotService $slots,
|
||||||
private \OCA\NgBackup\Service\PruneService $prune,
|
private \OCA\NgBackup\Service\PruneService $prune,
|
||||||
private \OCA\NgBackup\Service\Schedule $schedule,
|
private \OCA\NgBackup\Service\Schedule $schedule,
|
||||||
|
private \OCA\NgBackup\Service\CronHealth $cron,
|
||||||
) {
|
) {
|
||||||
parent::__construct(Application::APP_ID, $request);
|
parent::__construct(Application::APP_ID, $request);
|
||||||
}
|
}
|
||||||
@@ -58,10 +59,12 @@ final class AdminApiController extends Controller {
|
|||||||
foreach ($this->targets->list() as $t) {
|
foreach ($this->targets->list() as $t) {
|
||||||
$targets[] = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
|
$targets[] = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(),
|
||||||
'repositoryId' => $t->getRepositoryId(), 'createdAt' => $t->getCreatedAt(), 'appendOnly' => (bool)$t->getAppendOnly(),
|
'repositoryId' => $t->getRepositoryId(), 'createdAt' => $t->getCreatedAt(), 'appendOnly' => (bool)$t->getAppendOnly(),
|
||||||
'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t)];
|
'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t),
|
||||||
|
'schedule' => (string)$t->getSchedule(), 'scheduleZone' => $this->schedule->zoneOr($t->getScheduleZone())];
|
||||||
}
|
}
|
||||||
$shown = $this->runs->findShown($allRuns ? null : time() - \OCA\NgBackup\Command\Status::RECENT_DAYS * 86400);
|
$shown = $this->runs->findShown($allRuns ? null : time() - \OCA\NgBackup\Command\Status::RECENT_DAYS * 86400);
|
||||||
$runs = array_map(fn (Run $r) => $this->runJson($r), $shown);
|
$queued = array_flip(array_map(static fn (Run $r): int => $r->getId(), array_slice($this->backups->runsInTurn(), 1)));
|
||||||
|
$runs = array_map(fn (Run $r) => $this->runJson($r) + ['queued' => isset($queued[$r->getId()])], $shown);
|
||||||
return new JSONResponse([
|
return new JSONResponse([
|
||||||
'key' => ['initialized' => $this->keys->isInitialized(), 'fingerprint' => $this->keys->fingerprint(),
|
'key' => ['initialized' => $this->keys->isInitialized(), 'fingerprint' => $this->keys->fingerprint(),
|
||||||
'confirmation' => $this->keys->recoveryKitConfirmation(), 'everConfirmed' => $this->keys->isInitialized() && $this->keys->everConfirmed(),
|
'confirmation' => $this->keys->recoveryKitConfirmation(), 'everConfirmed' => $this->keys->isInitialized() && $this->keys->everConfirmed(),
|
||||||
@@ -73,11 +76,41 @@ final class AdminApiController extends Controller {
|
|||||||
'runs' => $runs,
|
'runs' => $runs,
|
||||||
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
|
'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown),
|
||||||
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
|
'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS,
|
||||||
'schedule' => $this->schedule->get()['time'],
|
'cron' => $this->cron->check(),
|
||||||
'scheduleTimezone' => $this->schedule->get()['timezone'],
|
|
||||||
]);
|
]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Seconds of backup work one tick() does: well within a normal web request. */
|
||||||
|
public const TICK_SECONDS = 10.0;
|
||||||
|
/** Seconds one request for the Bin reads, before the page shows how far it is and asks for more. */
|
||||||
|
private const TRASH_PAGE_SECONDS = 5.0;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Move running backups forward from the open admin page, when background jobs do not run on a
|
||||||
|
* working system cron (#13). The page calls this while a backup runs, so "Back up now" works
|
||||||
|
* without cron as long as the page stays open; scheduled backups still need cron. With a
|
||||||
|
* working system cron this does nothing: the background job does the work.
|
||||||
|
*/
|
||||||
|
public function tick(): JSONResponse {
|
||||||
|
if ($this->cron->check()['ok']) {
|
||||||
|
return new JSONResponse(['stepped' => 0]);
|
||||||
|
}
|
||||||
|
$deadline = microtime(true) + self::TICK_SECONDS;
|
||||||
|
$stepped = 0;
|
||||||
|
foreach ($this->backups->runsInTurn() as $run) {
|
||||||
|
if (microtime(true) >= $deadline) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
// step() returns at once when the background job (or another tab) holds the run.
|
||||||
|
$next = $this->backups->step($run, $deadline);
|
||||||
|
$stepped++;
|
||||||
|
if ($next->getStatus() === Run::RUNNING) {
|
||||||
|
break; // not finished: the next one waits for its turn
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return new JSONResponse(['stepped' => $stepped]);
|
||||||
|
}
|
||||||
|
|
||||||
#[PasswordConfirmationRequired]
|
#[PasswordConfirmationRequired]
|
||||||
public function initKey(string $passphrase, int $deleteDelayDays = \OCA\NgBackup\Service\KeyService::DEFAULT_DELETE_DELAY, string $label = 'Slot 1'): JSONResponse {
|
public function initKey(string $passphrase, int $deleteDelayDays = \OCA\NgBackup\Service\KeyService::DEFAULT_DELETE_DELAY, string $label = 'Slot 1'): JSONResponse {
|
||||||
try {
|
try {
|
||||||
@@ -146,15 +179,18 @@ final class AdminApiController extends Controller {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public function trash(int $targetId): JSONResponse {
|
public function trash(int $targetId, string $after = ''): JSONResponse {
|
||||||
try {
|
try {
|
||||||
$delay = $this->keys->deleteDelayDays() * 86400;
|
$delay = $this->keys->deleteDelayDays() * 86400;
|
||||||
|
$repo = $this->targets->repository($this->targets->get((string)$targetId));
|
||||||
|
// In parts, so that the page can show how far it is (a read per removed restore point).
|
||||||
|
$page = $repo->trashPage($after, microtime(true) + self::TRASH_PAGE_SECONDS);
|
||||||
$out = [];
|
$out = [];
|
||||||
foreach ($this->targets->repository($this->targets->get((string)$targetId))->trash() as $id => $info) {
|
foreach ($page['items'] as $id => $info) {
|
||||||
$out[] = ['id' => $id, 'label' => $info['label'], 'time' => $info['time'], 'removedAt' => $info['forgottenAt'],
|
$out[] = ['id' => $id, 'label' => $info['label'], 'time' => $info['time'], 'removedAt' => $info['forgottenAt'],
|
||||||
'by' => $info['by'], 'deleteAfter' => $info['forgottenAt'] + $delay];
|
'by' => $info['by'], 'deleteAfter' => $info['forgottenAt'] + $delay];
|
||||||
}
|
}
|
||||||
return new JSONResponse($out);
|
return new JSONResponse(['total' => $page['total'], 'items' => $out]);
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||||
}
|
}
|
||||||
@@ -242,25 +278,43 @@ final class AdminApiController extends Controller {
|
|||||||
return new JSONResponse($this->runJson($run));
|
return new JSONResponse($this->runJson($run));
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */
|
/** @param string $timezone the administrator's zone (the browser's); empty: the default one */
|
||||||
public function setSchedule(string $time, string $timezone = ''): JSONResponse {
|
public function setTargetSchedule(int $id, string $schedule, string $timezone = ''): JSONResponse {
|
||||||
try {
|
try {
|
||||||
$this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
|
$t = $this->targets->setSchedule($this->targets->get((string)$id), $schedule, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']);
|
||||||
} catch (\InvalidArgumentException $e) {
|
} catch (\Throwable $e) {
|
||||||
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||||
}
|
}
|
||||||
return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]);
|
return new JSONResponse(['schedule' => $t->getSchedule(), 'scheduleZone' => $t->getScheduleZone()]);
|
||||||
}
|
}
|
||||||
|
|
||||||
public function snapshots(int $targetId): JSONResponse {
|
public function snapshots(int $targetId): JSONResponse {
|
||||||
$out = [];
|
$out = [];
|
||||||
foreach ($this->snapshots->findForTarget($targetId) as $s) {
|
foreach ($this->snapshots->findForTarget($targetId) as $s) {
|
||||||
$out[] = ['id' => $s->getSnapshotId(), 'kind' => $s->getKind(), 'label' => $s->getLabel(), 'createdAt' => $s->getCreatedAt(),
|
$out[] = ['id' => $s->getSnapshotId(), 'kind' => $s->getKind(), 'label' => $s->getLabel(), 'createdAt' => $s->getCreatedAt(),
|
||||||
'files' => $s->getFiles(), 'bytes' => $s->getBytes()];
|
'files' => $s->getFiles(), 'bytes' => $s->getBytes(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()];
|
||||||
}
|
}
|
||||||
return new JSONResponse($out);
|
return new JSONResponse($out);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function setSnapshotNote(int $targetId, string $snapshotId, string $note = ''): JSONResponse {
|
||||||
|
try {
|
||||||
|
$this->prune->setNote($this->targets->get((string)$targetId), $snapshotId, $note);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||||
|
}
|
||||||
|
return new JSONResponse(['ok' => true]);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function setSnapshotPinned(int $targetId, string $snapshotId, bool $pinned = true): JSONResponse {
|
||||||
|
try {
|
||||||
|
$this->prune->setPinned($this->targets->get((string)$targetId), $snapshotId, $pinned);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST);
|
||||||
|
}
|
||||||
|
return new JSONResponse(['ok' => true]);
|
||||||
|
}
|
||||||
|
|
||||||
public function browse(int $targetId, string $snapshotId, string $path = ''): JSONResponse {
|
public function browse(int $targetId, string $snapshotId, string $path = ''): JSONResponse {
|
||||||
try {
|
try {
|
||||||
return new JSONResponse($this->restore->browse($this->targets->get((string)$targetId), $snapshotId, $path));
|
return new JSONResponse($this->restore->browse($this->targets->get((string)$targetId), $snapshotId, $path));
|
||||||
@@ -305,7 +359,7 @@ final class AdminApiController extends Controller {
|
|||||||
if ($r->getSnapshotId() !== null) {
|
if ($r->getSnapshotId() !== null) {
|
||||||
try {
|
try {
|
||||||
$s = $this->snapshots->findOne($r->getTargetId(), $r->getSnapshotId());
|
$s = $this->snapshots->findOne($r->getTargetId(), $r->getSnapshotId());
|
||||||
$snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel()];
|
$snapshot = ['createdAt' => $s->getCreatedAt(), 'label' => $s->getLabel(), 'note' => $s->getNote(), 'pinned' => (bool)$s->getPinned()];
|
||||||
} catch (\Throwable) {
|
} catch (\Throwable) {
|
||||||
// Snapshot already pruned or trashed: the run keeps its id, just not the date.
|
// Snapshot already pruned or trashed: the run keeps its id, just not the date.
|
||||||
}
|
}
|
||||||
@@ -313,6 +367,7 @@ final class AdminApiController extends Controller {
|
|||||||
return ['id' => $r->getId(), 'targetId' => $r->getTargetId(), 'kind' => $r->getKind(), 'status' => $r->getStatus(),
|
return ['id' => $r->getId(), 'targetId' => $r->getTargetId(), 'kind' => $r->getKind(), 'status' => $r->getStatus(),
|
||||||
'phase' => $r->getPhase(), 'snapshotId' => $r->getSnapshotId(), 'snapshot' => $snapshot, 'startedAt' => $r->getStartedAt(),
|
'phase' => $r->getPhase(), 'snapshotId' => $r->getSnapshotId(), 'snapshot' => $snapshot, 'startedAt' => $r->getStartedAt(),
|
||||||
'updatedAt' => $r->getUpdatedAt(), 'finishedAt' => $r->getFinishedAt(), 'error' => $r->getError(),
|
'updatedAt' => $r->getUpdatedAt(), 'finishedAt' => $r->getFinishedAt(), 'error' => $r->getError(),
|
||||||
'stats' => json_decode($r->getStats() ?? 'null', true), 'progress' => \OCA\NgBackup\Job\BackupRun::progressOf($r)];
|
'stats' => json_decode($r->getStats() ?? 'null', true), 'progress' => \OCA\NgBackup\Job\BackupRun::progressOf($r),
|
||||||
|
'waitingFor' => $r->getKind() === BackupService::KIND_FULL ? \OCA\NgBackup\Service\CronHealth::waitingFor($r) : null];
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -157,19 +157,29 @@ final class DbDumper {
|
|||||||
}
|
}
|
||||||
// Tables the backup knows the structure of are created first (DDL is not transactional
|
// Tables the backup knows the structure of are created first (DDL is not transactional
|
||||||
// everywhere); tables it does not know are left out, as asked.
|
// everywhere); tables it does not know are left out, as asked.
|
||||||
|
try {
|
||||||
foreach ($missing as $table) {
|
foreach ($missing as $table) {
|
||||||
if (isset($manifest['tables'][$table]['schema'])) {
|
if (isset($manifest['tables'][$table]['schema'])) {
|
||||||
|
$report['created'][] = $table; // first: creating it can stop halfway (its indexes)
|
||||||
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
|
$this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions()));
|
||||||
$report['created'][] = $table;
|
|
||||||
} else {
|
} else {
|
||||||
$report['missing'][] = $table;
|
$report['missing'][] = $table;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$this->dropCreatedTables($report);
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
$total = 0;
|
$total = 0;
|
||||||
$foreignKeys = [];
|
$foreignKeys = [];
|
||||||
$this->db->beginTransaction();
|
|
||||||
try {
|
try {
|
||||||
$foreignKeys = $this->suspendForeignKeys();
|
$this->db->beginTransaction();
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
$this->dropCreatedTables($report);
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$foreignKeys = $this->suspendForeignKeys(); // first: on SQLite it also takes the write lock
|
||||||
foreach ($manifest['tables'] as $table => $info) {
|
foreach ($manifest['tables'] as $table => $info) {
|
||||||
if (in_array($table, $report['missing'], true)) {
|
if (in_array($table, $report['missing'], true)) {
|
||||||
continue;
|
continue;
|
||||||
@@ -220,9 +230,19 @@ final class DbDumper {
|
|||||||
$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
|
$this->restoreSequences($manifest['sequences'] ?? [], $report['missing'], array_merge($report['restored'], $report['skipped']));
|
||||||
$this->resumeForeignKeys($foreignKeys);
|
$this->resumeForeignKeys($foreignKeys);
|
||||||
$this->db->commit();
|
$this->db->commit();
|
||||||
// Foreign keys of the tables created above, now their rows are in. Not inside the
|
} catch (\Throwable $e) {
|
||||||
// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is
|
$this->db->rollBack();
|
||||||
// restored either way.
|
// PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session
|
||||||
|
// switch must be turned on again by hand.
|
||||||
|
if (in_array('SET FOREIGN_KEY_CHECKS = 1', $foreignKeys, true)) {
|
||||||
|
$this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 1');
|
||||||
|
}
|
||||||
|
$this->dropCreatedTables($report);
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
// Committed. Foreign keys of the tables created above, now their rows are in. Not inside the
|
||||||
|
// transaction: MySQL commits on DDL. A failure is reported, not fatal: the data is restored
|
||||||
|
// either way, and the created tables stay.
|
||||||
foreach ($report['created'] as $table) {
|
foreach ($report['created'] as $table) {
|
||||||
$structure = $manifest['tables'][$table]['schema'] ?? [];
|
$structure = $manifest['tables'][$table]['schema'] ?? [];
|
||||||
foreach ($structure['foreignKeys'] ?? [] as $fk) {
|
foreach ($structure['foreignKeys'] ?? [] as $fk) {
|
||||||
@@ -235,18 +255,26 @@ final class DbDumper {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} catch (\Throwable $e) {
|
|
||||||
$this->db->rollBack();
|
|
||||||
// PostgreSQL rolls the dropped foreign keys back with the transaction; MySQL's session
|
|
||||||
// switch must be turned on again by hand.
|
|
||||||
if (in_array('SET FOREIGN_KEY_CHECKS = 1', $foreignKeys, true)) {
|
|
||||||
$this->db->executeStatement('SET FOREIGN_KEY_CHECKS = 1');
|
|
||||||
}
|
|
||||||
throw $e;
|
|
||||||
}
|
|
||||||
return $total;
|
return $total;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The restore did not go through: a rollback does not undo CREATE TABLE everywhere. Left behind,
|
||||||
|
* the empty tables of an app would stop that app's installation later ("table already exists").
|
||||||
|
*
|
||||||
|
* @param array{created: list<string>, ...} $report
|
||||||
|
*/
|
||||||
|
private function dropCreatedTables(array &$report): void {
|
||||||
|
foreach ($report['created'] as $table) {
|
||||||
|
try {
|
||||||
|
$this->db->createSchemaManager()->dropTable($table);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
// the error that brought us here is the one to report; a run again uses the table
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$report['created'] = [];
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The table prefix of the instance the dump was made on (from its appconfig table), or null
|
* The table prefix of the instance the dump was made on (from its appconfig table), or null
|
||||||
* when the dump has none.
|
* when the dump has none.
|
||||||
@@ -412,6 +440,12 @@ final class DbDumper {
|
|||||||
return $checks;
|
return $checks;
|
||||||
}
|
}
|
||||||
if ($platform instanceof \Doctrine\DBAL\Platforms\SqlitePlatform) {
|
if ($platform instanceof \Doctrine\DBAL\Platforms\SqlitePlatform) {
|
||||||
|
// SQLite (WAL) starts a transaction as a reader. When another process (a cron job, the
|
||||||
|
// installer still finishing) writes between our first read and our first write, that
|
||||||
|
// write fails at once with "database is locked": no waiting helps a reader whose view
|
||||||
|
// is out of date. Take the write lock first, so we wait for the other writer instead
|
||||||
|
// and it waits for us after that.
|
||||||
|
$this->db->executeStatement('DELETE FROM ' . $this->db->quoteIdentifier($this->prefix . 'appconfig') . ' WHERE 0 = 1');
|
||||||
$this->db->executeStatement('PRAGMA defer_foreign_keys = ON'); // checked at commit
|
$this->db->executeStatement('PRAGMA defer_foreign_keys = ON'); // checked at commit
|
||||||
}
|
}
|
||||||
return [];
|
return [];
|
||||||
|
|||||||
@@ -24,6 +24,10 @@ use OCP\AppFramework\Db\Entity;
|
|||||||
* @method void setFiles(?int $v)
|
* @method void setFiles(?int $v)
|
||||||
* @method int|null getBytes()
|
* @method int|null getBytes()
|
||||||
* @method void setBytes(?int $v)
|
* @method void setBytes(?int $v)
|
||||||
|
* @method string|null getNote()
|
||||||
|
* @method void setNote(?string $v)
|
||||||
|
* @method bool|null getPinned()
|
||||||
|
* @method void setPinned(?bool $v)
|
||||||
*/
|
*/
|
||||||
final class Snapshot extends Entity {
|
final class Snapshot extends Entity {
|
||||||
protected int $targetId = 0;
|
protected int $targetId = 0;
|
||||||
@@ -33,8 +37,11 @@ final class Snapshot extends Entity {
|
|||||||
protected int $createdAt = 0;
|
protected int $createdAt = 0;
|
||||||
protected ?int $files = null;
|
protected ?int $files = null;
|
||||||
protected ?int $bytes = null;
|
protected ?int $bytes = null;
|
||||||
|
protected ?string $note = null;
|
||||||
|
protected ?bool $pinned = null;
|
||||||
|
|
||||||
public function __construct() {
|
public function __construct() {
|
||||||
|
$this->addType('pinned', 'boolean');
|
||||||
$this->addType('targetId', 'integer');
|
$this->addType('targetId', 'integer');
|
||||||
$this->addType('createdAt', 'integer');
|
$this->addType('createdAt', 'integer');
|
||||||
$this->addType('files', 'integer');
|
$this->addType('files', 'integer');
|
||||||
|
|||||||
@@ -28,6 +28,12 @@ use OCP\AppFramework\Db\Entity;
|
|||||||
* @method void setMaxBytes(?int $v)
|
* @method void setMaxBytes(?int $v)
|
||||||
* @method int getCreatedAt()
|
* @method int getCreatedAt()
|
||||||
* @method void setCreatedAt(int $v)
|
* @method void setCreatedAt(int $v)
|
||||||
|
* @method string|null getSchedule()
|
||||||
|
* @method void setSchedule(?string $v)
|
||||||
|
* @method string|null getScheduleZone()
|
||||||
|
* @method void setScheduleZone(?string $v)
|
||||||
|
* @method int|null getScheduleSince()
|
||||||
|
* @method void setScheduleSince(?int $v)
|
||||||
*/
|
*/
|
||||||
final class Target extends Entity {
|
final class Target extends Entity {
|
||||||
protected string $name = '';
|
protected string $name = '';
|
||||||
@@ -39,8 +45,12 @@ final class Target extends Entity {
|
|||||||
protected int $createdAt = 0;
|
protected int $createdAt = 0;
|
||||||
protected ?bool $appendOnly = null;
|
protected ?bool $appendOnly = null;
|
||||||
protected ?int $maxBytes = null;
|
protected ?int $maxBytes = null;
|
||||||
|
protected ?string $schedule = null;
|
||||||
|
protected ?string $scheduleZone = null;
|
||||||
|
protected ?int $scheduleSince = null;
|
||||||
|
|
||||||
public function __construct() {
|
public function __construct() {
|
||||||
|
$this->addType('scheduleSince', 'integer');
|
||||||
$this->addType('createdAt', 'integer');
|
$this->addType('createdAt', 'integer');
|
||||||
$this->addType('appendOnly', 'boolean');
|
$this->addType('appendOnly', 'boolean');
|
||||||
$this->addType('maxBytes', 'integer');
|
$this->addType('maxBytes', 'integer');
|
||||||
|
|||||||
@@ -15,4 +15,14 @@ final class Format {
|
|||||||
$time = gmdate('H:i:s', $seconds % 86400);
|
$time = gmdate('H:i:s', $seconds % 86400);
|
||||||
return $days > 0 ? $days . 'd ' . $time : $time;
|
return $days > 0 ? $days . 'd ' . $time : $time;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* "2026-10-29 04:30" in $zone. Nextcloud runs PHP in UTC; occ shows times in the zone of the
|
||||||
|
* schedule, the administrator's, as the web interface does.
|
||||||
|
*
|
||||||
|
* @param non-empty-string $zone
|
||||||
|
*/
|
||||||
|
public static function time(int $timestamp, string $zone): string {
|
||||||
|
return (new \DateTimeImmutable('@' . $timestamp))->setTimezone(new \DateTimeZone($zone))->format('Y-m-d H:i');
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -52,12 +52,51 @@ final class BackupRun {
|
|||||||
$stats = $state['stats'];
|
$stats = $state['stats'];
|
||||||
$walker = new TreeWalker($state['roots'], $state['exclude'], function (string $logical) use (&$state, &$stats): void {
|
$walker = new TreeWalker($state['roots'], $state['exclude'], function (string $logical) use (&$state, &$stats): void {
|
||||||
self::recordUnreadable($state, $stats, $logical);
|
self::recordUnreadable($state, $stats, $logical);
|
||||||
});
|
}, true);
|
||||||
$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
|
$parent = new ParentCursor($repo, $state['parent'], $state['parentPos']);
|
||||||
$lines = [];
|
$lines = [];
|
||||||
|
$emptyDirs = [];
|
||||||
|
$folders = [];
|
||||||
$finishedWalk = true;
|
$finishedWalk = true;
|
||||||
|
|
||||||
foreach ($walker->walk($state['after']) as $logical => $path) {
|
foreach ($walker->walk($state['after']) as $logical => $path) {
|
||||||
|
$safe = mb_check_encoding($logical, 'UTF-8') && Repository::isSafePath($logical);
|
||||||
|
if ($walker->atFolderWithContent()) {
|
||||||
|
// Only its time: what is in it is reported by itself, and a bad name in the path of
|
||||||
|
// each of those is what gets reported. The cursor stays for such a name, see below.
|
||||||
|
$mtime = $safe ? @filemtime($path) : false;
|
||||||
|
if ($safe && $mtime !== false) {
|
||||||
|
$folders[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR);
|
||||||
|
$state['after'] = $logical;
|
||||||
|
if (microtime(true) >= $deadline) {
|
||||||
|
$finishedWalk = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!$safe) {
|
||||||
|
// A name a snapshot cannot hold (a backslash, a control character, not UTF-8). One
|
||||||
|
// such path must not make a backup fail, or the whole snapshot unreadable later.
|
||||||
|
// The cursor stays where it is: the state is JSON, which cannot hold such a name. A
|
||||||
|
// next step meets it again, and it is reported once.
|
||||||
|
self::recordBadName($state, $stats, $logical);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (is_dir($path)) {
|
||||||
|
$mtime = @filemtime($path);
|
||||||
|
if ($mtime === false) {
|
||||||
|
continue; // removed while walking
|
||||||
|
}
|
||||||
|
$emptyDirs[] = json_encode([$logical, $mtime], JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR);
|
||||||
|
$stats['emptyDirs'] = ($stats['emptyDirs'] ?? 0) + 1;
|
||||||
|
$state['after'] = $logical;
|
||||||
|
if (microtime(true) >= $deadline) {
|
||||||
|
$finishedWalk = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
$version = Repository::fileVersion($path);
|
$version = Repository::fileVersion($path);
|
||||||
if ($version === 'missing') {
|
if ($version === 'missing') {
|
||||||
continue; // deleted while walking
|
continue; // deleted while walking
|
||||||
@@ -136,6 +175,16 @@ final class BackupRun {
|
|||||||
$writer->write(implode("\n", $lines) . "\n");
|
$writer->write(implode("\n", $lines) . "\n");
|
||||||
$state['segments'][] = $writer->finish();
|
$state['segments'][] = $writer->finish();
|
||||||
}
|
}
|
||||||
|
if ($emptyDirs !== []) {
|
||||||
|
$writer = $repo->blobWriter($stats);
|
||||||
|
$writer->write(implode("\n", $emptyDirs) . "\n");
|
||||||
|
$state['dirSegments'][] = $writer->finish();
|
||||||
|
}
|
||||||
|
if ($folders !== []) {
|
||||||
|
$writer = $repo->blobWriter($stats);
|
||||||
|
$writer->write(implode("\n", $folders) . "\n");
|
||||||
|
$state['folderSegments'][] = $writer->finish();
|
||||||
|
}
|
||||||
$repo->flushPacks();
|
$repo->flushPacks();
|
||||||
$state['parentPos'] = $parent->position();
|
$state['parentPos'] = $parent->position();
|
||||||
$state['stats'] = $stats;
|
$state['stats'] = $stats;
|
||||||
@@ -150,10 +199,22 @@ final class BackupRun {
|
|||||||
if ($state['phase'] === 'finish') {
|
if ($state['phase'] === 'finish') {
|
||||||
// Every segment ends with a newline, so they read as one stream: fetched together,
|
// Every segment ends with a newline, so they read as one stream: fetched together,
|
||||||
// several at a time where the location allows it.
|
// several at a time where the location allows it.
|
||||||
|
// Empty folders go in the meta record, not between the files: a version that does not
|
||||||
|
// know them reads the snapshot as before, and only restores no empty folders.
|
||||||
|
$emptyDirs = [];
|
||||||
|
foreach (isset($state['dirSegments']) ? $repo->readLines(array_values(array_merge(...$state['dirSegments']))) : [] as $line) {
|
||||||
|
if ($line !== '') {
|
||||||
|
$emptyDirs[] = json_decode($line, true, 8, JSON_THROW_ON_ERROR);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// The times of the other folders too, packed: every look at a snapshot's meta record reads
|
||||||
|
// this, and a string of a few hundred KiB costs that far less than tens of thousands of arrays.
|
||||||
|
$folders = isset($state['folderSegments']) ? implode("\n", iterator_to_array($repo->readLines(array_values(array_merge(...$state['folderSegments']))), false)) : '';
|
||||||
$repo->putSnapshot($state['snapshot'],
|
$repo->putSnapshot($state['snapshot'],
|
||||||
['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
|
['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'],
|
||||||
'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
|
'roots' => array_keys($state['roots']), 'stats' => $state['stats'],
|
||||||
'warnings' => self::warnings($state)] + $state['meta'],
|
'warnings' => self::warnings($state)] + ($emptyDirs === [] ? [] : ['emptyDirs' => $emptyDirs])
|
||||||
|
+ ($folders === '' ? [] : ['folderTimes' => base64_encode((string)gzdeflate($folders))]) + $state['meta'],
|
||||||
$repo->readLines(array_values(array_merge(...$state['segments']))));
|
$repo->readLines(array_values(array_merge(...$state['segments']))));
|
||||||
$state['phase'] = 'commit';
|
$state['phase'] = 'commit';
|
||||||
}
|
}
|
||||||
@@ -176,12 +237,12 @@ final class BackupRun {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* progress() of a run, while it is in its files phase.
|
* progress() of a run in its files phase; in its database phase, the tables dumped so far.
|
||||||
*
|
*
|
||||||
* @return array{fraction: float, approximate: bool}|null
|
* @return array{fraction: float, approximate: bool, tables?: int, totalTables?: int}|null
|
||||||
*/
|
*/
|
||||||
public static function progressOf(\OCA\NgBackup\Db\Run $run): ?array {
|
public static function progressOf(\OCA\NgBackup\Db\Run $run): ?array {
|
||||||
if ($run->getStatus() !== \OCA\NgBackup\Db\Run::RUNNING || $run->getPhase() !== 'files') {
|
if ($run->getStatus() !== \OCA\NgBackup\Db\Run::RUNNING || !in_array($run->getPhase(), ['db', 'files'], true)) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
@@ -189,6 +250,14 @@ final class BackupRun {
|
|||||||
} catch (\JsonException) {
|
} catch (\JsonException) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
if ($run->getPhase() === 'db') {
|
||||||
|
// Tables dumped so far, as of the last step (none known before the first step ends)
|
||||||
|
$tables = count($state['dump']['tables'] ?? []);
|
||||||
|
return $tables > 0
|
||||||
|
? ['fraction' => min(1.0, ($state['dump']['i'] ?? 0) / $tables), 'approximate' => false,
|
||||||
|
'tables' => (int)($state['dump']['i'] ?? 0), 'totalTables' => $tables]
|
||||||
|
: null;
|
||||||
|
}
|
||||||
return is_array($state['files'] ?? null) ? self::progress($state['files']) : null;
|
return is_array($state['files'] ?? null) ? self::progress($state['files']) : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -206,10 +275,26 @@ final class BackupRun {
|
|||||||
$stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1;
|
$stats['unreadable'] = ($stats['unreadable'] ?? 0) + 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @return array{unreadable: list<string>, changing: list<string>} examples for the report (first 50 each) */
|
/**
|
||||||
|
* A file or directory left out because of its name. Remembered like an unreadable one, in a form
|
||||||
|
* that fits the JSON state and is safe to show: control characters and invalid bytes replaced.
|
||||||
|
*/
|
||||||
|
private static function recordBadName(array &$state, array &$stats, string $logical): void {
|
||||||
|
$shown = (string)preg_replace('/[\x00-\x1f\x7f]/', '?', mb_scrub($logical, 'UTF-8'));
|
||||||
|
if (isset($state['badNames'][$shown])) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (count($state['badNames'] ?? []) < 1000) {
|
||||||
|
$state['badNames'][$shown] = true;
|
||||||
|
}
|
||||||
|
$stats['badNames'] = ($stats['badNames'] ?? 0) + 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @return array{unreadable: list<string>, badNames: list<string>, changing: list<string>} examples for the report (first 50 each) */
|
||||||
public static function warnings(array $state): array {
|
public static function warnings(array $state): array {
|
||||||
return [
|
return [
|
||||||
'unreadable' => array_slice(array_map('strval', array_keys($state['unreadable'] ?? [])), 0, 50),
|
'unreadable' => array_slice(array_map('strval', array_keys($state['unreadable'] ?? [])), 0, 50),
|
||||||
|
'badNames' => array_slice(array_map('strval', array_keys($state['badNames'] ?? [])), 0, 50),
|
||||||
'changing' => array_slice(array_map('strval', array_values($state['changing'] ?? [])), 0, 50),
|
'changing' => array_slice(array_map('strval', array_values($state['changing'] ?? [])), 0, 50),
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,12 +9,16 @@ namespace OCA\NgBackup\Job;
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Walks one or more source roots in a stable depth-first order (entries of a directory sorted
|
* Walks one or more source roots in a stable depth-first order (entries of a directory sorted
|
||||||
* by name, children right after their directory), yielding "<root>/<relative path>" for files.
|
* by name, children right after their directory), yielding "<root>/<relative path>" for files
|
||||||
|
* and for directories with nothing in them to walk (the caller tells them apart with is_dir()).
|
||||||
|
* A root itself without a name is never yielded.
|
||||||
*
|
*
|
||||||
* The same order is used for comparing paths (compare()), so a walk can resume after a cursor
|
* The same order is used for comparing paths (compare()), so a walk can resume after a cursor
|
||||||
* and be merge-joined with the previous snapshot without holding any list in memory.
|
* and be merge-joined with the previous snapshot without holding any list in memory.
|
||||||
*/
|
*/
|
||||||
final class TreeWalker {
|
final class TreeWalker {
|
||||||
|
private bool $atFolder = false;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @param array<string, string> $roots name => absolute directory, walked in this order
|
* @param array<string, string> $roots name => absolute directory, walked in this order
|
||||||
* @param list<string> $exclude path prefixes ("<root>/<relative>") that are skipped entirely;
|
* @param list<string> $exclude path prefixes ("<root>/<relative>") that are skipped entirely;
|
||||||
@@ -25,9 +29,16 @@ final class TreeWalker {
|
|||||||
private array $exclude = [],
|
private array $exclude = [],
|
||||||
/** Called with the logical path of a directory that cannot be listed. */
|
/** Called with the logical path of a directory that cannot be listed. */
|
||||||
private ?\Closure $onUnreadable = null,
|
private ?\Closure $onUnreadable = null,
|
||||||
|
/** Also yield directories with something in them, right before the first thing in them. */
|
||||||
|
private bool $withFolders = false,
|
||||||
) {
|
) {
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Whether what was just yielded is a directory with something in it (see $withFolders). */
|
||||||
|
public function atFolderWithContent(): bool {
|
||||||
|
return $this->atFolder;
|
||||||
|
}
|
||||||
|
|
||||||
/** @return \Generator<string, string> logical path => absolute path, strictly after $after */
|
/** @return \Generator<string, string> logical path => absolute path, strictly after $after */
|
||||||
public function walk(?string $after = null): \Generator {
|
public function walk(?string $after = null): \Generator {
|
||||||
foreach ($this->roots as $name => $dir) {
|
foreach ($this->roots as $name => $dir) {
|
||||||
@@ -66,27 +77,41 @@ final class TreeWalker {
|
|||||||
}
|
}
|
||||||
$names = array_values(array_diff($names, ['.', '..']));
|
$names = array_values(array_diff($names, ['.', '..']));
|
||||||
sort($names, SORT_STRING);
|
sort($names, SORT_STRING);
|
||||||
|
$empty = true; // no file or directory below it that is walked, whatever the cursor skips
|
||||||
foreach ($names as $name) {
|
foreach ($names as $name) {
|
||||||
$childLogical = $logical === '' ? $name : $logical . '/' . $name;
|
$childLogical = $logical === '' ? $name : $logical . '/' . $name;
|
||||||
$childAbs = $abs . '/' . $name;
|
$childAbs = $abs . '/' . $name;
|
||||||
if (is_link($childAbs)) {
|
if (is_link($childAbs)) {
|
||||||
continue; // never follow symlinks out of the data directory
|
continue; // never follow symlinks out of the data directory
|
||||||
}
|
}
|
||||||
if (is_dir($childAbs)) {
|
$childIsDir = is_dir($childAbs);
|
||||||
|
if (($childIsDir || is_file($childAbs)) && $empty && !$this->isExcluded($childLogical, $childIsDir)
|
||||||
|
&& $this->withFolders && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) {
|
||||||
|
$this->atFolder = true;
|
||||||
|
yield $logical => $abs;
|
||||||
|
$this->atFolder = false;
|
||||||
|
}
|
||||||
|
if ($childIsDir) {
|
||||||
|
$empty = $empty && $this->isExcluded($childLogical, true);
|
||||||
// Skip a directory entirely if everything in it sorts before the cursor.
|
// Skip a directory entirely if everything in it sorts before the cursor.
|
||||||
if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) {
|
if ($after !== null && self::compare($childLogical, $after) < 0 && !self::isPrefix($childLogical, $after)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
yield from $this->walkDir($childAbs, $childLogical, $after);
|
yield from $this->walkDir($childAbs, $childLogical, $after);
|
||||||
} elseif (is_file($childAbs)) {
|
} elseif (is_file($childAbs)) {
|
||||||
|
$excluded = $this->isExcluded($childLogical, false);
|
||||||
|
$empty = $empty && $excluded;
|
||||||
if ($after !== null && self::compare($childLogical, $after) <= 0) {
|
if ($after !== null && self::compare($childLogical, $after) <= 0) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (!$this->isExcluded($childLogical, false)) {
|
if (!$excluded) {
|
||||||
yield $childLogical => $childAbs;
|
yield $childLogical => $childAbs;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if ($empty && $logical !== '' && ($after === null || self::compare($logical, $after) > 0)) {
|
||||||
|
yield $logical => $abs;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private function isExcluded(string $logical, bool $isDir): bool {
|
private function isExcluded(string $logical, bool $isDir): bool {
|
||||||
|
|||||||
@@ -0,0 +1,33 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Migration;
|
||||||
|
|
||||||
|
use Closure;
|
||||||
|
use OCP\DB\ISchemaWrapper;
|
||||||
|
use OCP\DB\Types;
|
||||||
|
use OCP\Migration\IOutput;
|
||||||
|
use OCP\Migration\SimpleMigrationStep;
|
||||||
|
|
||||||
|
/** Local copy of what the catalog on the location says about a snapshot: its note and pin. */
|
||||||
|
final class Version000007Date20261107000000 extends SimpleMigrationStep {
|
||||||
|
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
|
||||||
|
/** @var ISchemaWrapper $schema */
|
||||||
|
$schema = $schemaClosure();
|
||||||
|
$t = $schema->getTable('ngb_snapshots');
|
||||||
|
$changed = false;
|
||||||
|
if (!$t->hasColumn('note')) {
|
||||||
|
$t->addColumn('note', Types::STRING, ['notnull' => false, 'length' => 1000]);
|
||||||
|
$changed = true;
|
||||||
|
}
|
||||||
|
if (!$t->hasColumn('pinned')) {
|
||||||
|
$t->addColumn('pinned', Types::BOOLEAN, ['notnull' => false, 'default' => false]);
|
||||||
|
$changed = true;
|
||||||
|
}
|
||||||
|
return $changed ? $schema : null;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Migration;
|
||||||
|
|
||||||
|
use Closure;
|
||||||
|
use OCA\NgBackup\Service\LocationSchedule;
|
||||||
|
use OCA\NgBackup\Service\Schedule;
|
||||||
|
use OCP\DB\ISchemaWrapper;
|
||||||
|
use OCP\DB\Types;
|
||||||
|
use OCP\IDBConnection;
|
||||||
|
use OCP\Migration\IOutput;
|
||||||
|
use OCP\Migration\SimpleMigrationStep;
|
||||||
|
|
||||||
|
/** A schedule per location. Each existing location gets the daily time that held for all of them. */
|
||||||
|
final class Version000008Date20261107010000 extends SimpleMigrationStep {
|
||||||
|
public function __construct(
|
||||||
|
private IDBConnection $db,
|
||||||
|
private Schedule $schedule,
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
||||||
|
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
|
||||||
|
/** @var ISchemaWrapper $schema */
|
||||||
|
$schema = $schemaClosure();
|
||||||
|
$t = $schema->getTable('ngb_targets');
|
||||||
|
if ($t->hasColumn('schedule')) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
$t->addColumn('schedule', Types::STRING, ['notnull' => false, 'length' => 16]);
|
||||||
|
$t->addColumn('schedule_zone', Types::STRING, ['notnull' => false, 'length' => 64]);
|
||||||
|
$t->addColumn('schedule_since', Types::BIGINT, ['notnull' => false, 'unsigned' => true]);
|
||||||
|
return $schema;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function postSchemaChange(IOutput $output, Closure $schemaClosure, array $options): void {
|
||||||
|
['time' => $time, 'timezone' => $zone] = $this->schedule->get();
|
||||||
|
$qb = $this->db->getQueryBuilder();
|
||||||
|
$qb->update('ngb_targets')
|
||||||
|
->set('schedule', $qb->createNamedParameter(Schedule::validTime($time) ? LocationSchedule::normalize($time) : ''))
|
||||||
|
->set('schedule_zone', $qb->createNamedParameter($zone))
|
||||||
|
->where($qb->expr()->isNull('schedule'));
|
||||||
|
$qb->executeStatement();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -46,6 +46,10 @@ final class Notifier implements INotifier {
|
|||||||
$notification->setParsedSubject($l->t('Backup to %1$s finished with warnings: %2$s unreadable, %3$s changed while being read', [$p['target'], $p['unreadable'], $p['changing']]));
|
$notification->setParsedSubject($l->t('Backup to %1$s finished with warnings: %2$s unreadable, %3$s changed while being read', [$p['target'], $p['unreadable'], $p['changing']]));
|
||||||
$notification->setParsedMessage($l->t('Unreadable files and folders are missing from this backup (check their permissions). Files that kept changing were saved as they were while being read. Examples: %s', [$p['examples'] ?? '']));
|
$notification->setParsedMessage($l->t('Unreadable files and folders are missing from this backup (check their permissions). Files that kept changing were saved as they were while being read. Examples: %s', [$p['examples'] ?? '']));
|
||||||
break;
|
break;
|
||||||
|
case 'run_bad_names':
|
||||||
|
$notification->setParsedSubject($l->n('Backup to %s finished, but %n item is missing because of its name', 'Backup to %s finished, but %n items are missing because of their names', (int)$p['count'], [$p['target']]));
|
||||||
|
$notification->setParsedMessage($l->t('Names with a backslash or a control character, or that are not valid UTF-8, are not supported, so these files and folders are not in this backup. Rename them; the next backup includes them. Examples: %s', [$p['examples'] ?? '']));
|
||||||
|
break;
|
||||||
case 'quota_warning':
|
case 'quota_warning':
|
||||||
$notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']]));
|
$notification->setParsedSubject($l->t('Backup location %1$s is at %2$s%% of its limit (%3$s of %4$s)', [$p['target'], $p['percent'], $p['used'], $p['limit']]));
|
||||||
$notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.'));
|
$notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.'));
|
||||||
@@ -57,13 +61,20 @@ final class Notifier implements INotifier {
|
|||||||
: $l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']]));
|
: $l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']]));
|
||||||
$notification->setParsedMessage($l->t('Check the backup settings and the server log.'));
|
$notification->setParsedMessage($l->t('Check the backup settings and the server log.'));
|
||||||
break;
|
break;
|
||||||
|
case 'schedule_off':
|
||||||
|
$notification->setParsedSubject($l->t('Backup location %s has no schedule', [$p['target']]));
|
||||||
|
$notification->setParsedMessage(($p['others'] ?? '') === '1'
|
||||||
|
? $l->t('No backups run to this location until it gets a schedule or someone starts one. If that is what you want, you can ignore this reminder.')
|
||||||
|
: $l->t('No location has a schedule, so no backups run at all unless someone starts one. Set a schedule under Backups.'));
|
||||||
|
break;
|
||||||
case 'security_event':
|
case 'security_event':
|
||||||
$who = (string)($p['actor'] ?? '');
|
$who = (string)($p['actor'] ?? '');
|
||||||
$notification->setParsedSubject(match ($p['event'] ?? '') {
|
$notification->setParsedSubject(match ($p['event'] ?? '') {
|
||||||
'kit_downloaded' => $l->t('%s downloaded the backup recovery kit', [$who]),
|
'kit_downloaded' => $l->t('%s downloaded the backup recovery kit', [$who]),
|
||||||
'target_removed' => $l->t('%1$s removed backup location %2$s', [$who, $p['target'] ?? '']),
|
'target_removed' => $l->t('%1$s removed backup location %2$s', [$who, $p['target'] ?? '']),
|
||||||
'target_changed' => $l->t('%1$s changed the connection settings of backup location %2$s', [$who, $p['target'] ?? '']),
|
'target_changed' => $l->t('%1$s changed the connection settings of backup location %2$s', [$who, $p['target'] ?? '']),
|
||||||
'append_only_off' => $l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']),
|
'schedule_off' => $l->t('%1$s turned off the schedule of backup location %2$s', [$who, $p['target'] ?? '']),
|
||||||
|
'append_only_off' =>$l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']),
|
||||||
'retention_changed' => $l->t('%1$s changed the backup retention to %2$s', [$who, $p['policy'] ?? '']),
|
'retention_changed' => $l->t('%1$s changed the backup retention to %2$s', [$who, $p['policy'] ?? '']),
|
||||||
'snapshots_forgotten' => $l->t('%1$s moved %2$s snapshots of %3$s to the trash; they are deleted permanently after %4$s unless restored', [$who, $p['count'] ?? '', $p['target'] ?? '', $p['until'] ?? '']),
|
'snapshots_forgotten' => $l->t('%1$s moved %2$s snapshots of %3$s to the trash; they are deleted permanently after %4$s unless restored', [$who, $p['count'] ?? '', $p['target'] ?? '', $p['until'] ?? '']),
|
||||||
'snapshot_untrashed' => $l->t('%1$s restored snapshot %2$s of %3$s from the trash', [$who, $p['snapshot'] ?? '', $p['target'] ?? '']),
|
'snapshot_untrashed' => $l->t('%1$s restored snapshot %2$s of %3$s from the trash', [$who, $p['snapshot'] ?? '', $p['target'] ?? '']),
|
||||||
|
|||||||
@@ -30,6 +30,10 @@ final class Catalog {
|
|||||||
public array $trash = [];
|
public array $trash = [];
|
||||||
/** @var array<string, true> manifest path => true, for a per-user export (users/<uid>/<exportId>) */
|
/** @var array<string, true> manifest path => true, for a per-user export (users/<uid>/<exportId>) */
|
||||||
public array $userExports = [];
|
public array $userExports = [];
|
||||||
|
/** @var array<string, true> snapshots the retention policy leaves alone */
|
||||||
|
public array $pinned = [];
|
||||||
|
/** @var array<string, string> snapshot id => note, also while the snapshot is in the trash */
|
||||||
|
public array $notes = [];
|
||||||
public string $hash = '';
|
public string $hash = '';
|
||||||
|
|
||||||
public function __construct(
|
public function __construct(
|
||||||
@@ -82,6 +86,8 @@ final class Catalog {
|
|||||||
// Catalogs written before user exports existed have no such key: default to empty rather
|
// Catalogs written before user exports existed have no such key: default to empty rather
|
||||||
// than adopting the location's current users/ listing (that would skip verification).
|
// than adopting the location's current users/ listing (that would skip verification).
|
||||||
$this->userExports = array_fill_keys($data['userExports'] ?? [], true);
|
$this->userExports = array_fill_keys($data['userExports'] ?? [], true);
|
||||||
|
$this->pinned = array_fill_keys($data['pinned'] ?? [], true);
|
||||||
|
$this->notes = $data['notes'] ?? [];
|
||||||
$this->anchor?->set($this->repositoryId, $this->gen, $this->hash);
|
$this->anchor?->set($this->repositoryId, $this->gen, $this->hash);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -93,9 +99,12 @@ final class Catalog {
|
|||||||
if ($this->backend->exists($this->path($next))) {
|
if ($this->backend->exists($this->path($next))) {
|
||||||
throw new RepositoryException("Catalog generation $next was written by another process at the same time; try again");
|
throw new RepositoryException("Catalog generation $next was written by another process at the same time; try again");
|
||||||
}
|
}
|
||||||
|
// A pin ends when its snapshot leaves the list, a note when the snapshot is gone for good.
|
||||||
|
$this->pinned = array_intersect_key($this->pinned, $this->snapshots);
|
||||||
|
$this->notes = array_intersect_key($this->notes, $this->snapshots + $this->trash);
|
||||||
$plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash,
|
$plain = json_encode(['repository' => $this->repositoryId, 'gen' => $next, 'time' => gmdate('c'), 'prev' => $this->hash,
|
||||||
'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash),
|
'snapshots' => array_keys($this->snapshots), 'trash' => array_keys($this->trash),
|
||||||
'userExports' => array_keys($this->userExports)], JSON_THROW_ON_ERROR);
|
'userExports' => array_keys($this->userExports), 'pinned' => array_keys($this->pinned), 'notes' => $this->notes], JSON_THROW_ON_ERROR);
|
||||||
$encrypted = $this->cipher->encryptString($plain, $this->ad($next));
|
$encrypted = $this->cipher->encryptString($plain, $this->ad($next));
|
||||||
$stream = fopen('php://memory', 'w+b');
|
$stream = fopen('php://memory', 'w+b');
|
||||||
fwrite($stream, $encrypted);
|
fwrite($stream, $encrypted);
|
||||||
|
|||||||
@@ -44,6 +44,8 @@ final class Repository {
|
|||||||
private ?int $maxBytes = null;
|
private ?int $maxBytes = null;
|
||||||
private Catalog $catalog;
|
private Catalog $catalog;
|
||||||
private PackReader $packReader;
|
private PackReader $packReader;
|
||||||
|
/** @var array<string, array<string, mixed>> meta record of the snapshot read last */
|
||||||
|
private array $lastMeta = [];
|
||||||
/** verify() lists the pack folders from this many packs on (see presentPacks()); tests lower it. */
|
/** verify() lists the pack folders from this many packs on (see presentPacks()); tests lower it. */
|
||||||
public int $listPacksAbove = 256;
|
public int $listPacksAbove = 256;
|
||||||
|
|
||||||
@@ -132,6 +134,49 @@ final class Repository {
|
|||||||
return $this->catalog->gen;
|
return $this->catalog->gen;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** @return list<string> snapshots the retention policy leaves alone */
|
||||||
|
public function pinnedSnapshots(): array {
|
||||||
|
return array_map('strval', array_keys($this->catalog->pinned));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @return array<string, string> snapshot id => note */
|
||||||
|
public function snapshotNotes(): array {
|
||||||
|
return $this->catalog->notes;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function setPinned(string $snapshotId, bool $pinned): void {
|
||||||
|
$this->requireSnapshot($snapshotId);
|
||||||
|
if (isset($this->catalog->pinned[$snapshotId]) === $pinned) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if ($pinned) {
|
||||||
|
$this->catalog->pinned[$snapshotId] = true;
|
||||||
|
} else {
|
||||||
|
unset($this->catalog->pinned[$snapshotId]);
|
||||||
|
}
|
||||||
|
$this->catalog->write();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A note kept in the catalog: unlike the label inside the snapshot, it can change afterwards. */
|
||||||
|
public function setSnapshotNote(string $snapshotId, string $note): void {
|
||||||
|
$this->requireSnapshot($snapshotId);
|
||||||
|
if (($this->catalog->notes[$snapshotId] ?? '') === $note) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if ($note === '') {
|
||||||
|
unset($this->catalog->notes[$snapshotId]);
|
||||||
|
} else {
|
||||||
|
$this->catalog->notes[$snapshotId] = $note;
|
||||||
|
}
|
||||||
|
$this->catalog->write();
|
||||||
|
}
|
||||||
|
|
||||||
|
private function requireSnapshot(string $snapshotId): void {
|
||||||
|
if (!isset($this->catalog->snapshots[$snapshotId])) {
|
||||||
|
throw new \InvalidArgumentException('There is no snapshot ' . $snapshotId . ' on this location');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/** Index files downloaded when this repository was opened (0 when the local cache was complete). */
|
/** Index files downloaded when this repository was opened (0 when the local cache was complete). */
|
||||||
public function indexDownloads(): int {
|
public function indexDownloads(): int {
|
||||||
return $this->index->downloaded;
|
return $this->index->downloaded;
|
||||||
@@ -214,20 +259,28 @@ final class Repository {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Restore a snapshot (or only paths starting with $prefix) into a local directory.
|
* Restore a snapshot (or only paths starting with $prefix) into a local directory. A $prefix
|
||||||
|
* "dir/file/" also restores the file "dir/file" itself.
|
||||||
*
|
*
|
||||||
* @param bool $stripPrefix write "$target/<path without $prefix>" instead of
|
* @param bool $stripPrefix write "$target/<path without $prefix>" instead of
|
||||||
* "$target/<full logical path>" (disaster recovery: restoring e.g. "data/" or
|
* "$target/<full logical path>" (disaster recovery: restoring e.g. "data/" or
|
||||||
* "config/" directly into the real data/config directory, not a subdirectory
|
* "config/" directly into the real data/config directory, not a subdirectory
|
||||||
* named after the prefix). Ignored when $prefix is '' (nothing to strip).
|
* named after the prefix). Ignored when $prefix is '' (nothing to strip).
|
||||||
|
* @param array<string, string>|null $skipped salvage: an array to collect the files that could
|
||||||
|
* not be read or decrypted (path => reason), which are then left out instead of
|
||||||
|
* stopping the restore; null (default) stops at the first such file
|
||||||
|
* @param bool $keepUnchanged leave a file that is already in place with the size and
|
||||||
|
* modification time of the snapshot, as a backup takes such a file as unchanged:
|
||||||
|
* a restore that was stopped and runs again goes on where it was, without reading
|
||||||
|
* what it already restored from the location again
|
||||||
*/
|
*/
|
||||||
public function restore(string $snapshotId, string $target, string $prefix = '', bool $stripPrefix = false, ?callable $heartbeat = null): int {
|
public function restore(string $snapshotId, string $target, string $prefix = '', bool $stripPrefix = false, ?callable $heartbeat = null, ?array &$skipped = null, bool $keepUnchanged = false): int {
|
||||||
if ($prefix !== '' && !self::isSafePath(rtrim($prefix, '/'))) {
|
if ($prefix !== '' && !self::isSafePath(rtrim($prefix, '/'))) {
|
||||||
throw new RepositoryException('Invalid path prefix');
|
throw new RepositoryException('Invalid path prefix');
|
||||||
}
|
}
|
||||||
$plan = $this->readAhead($snapshotId, $prefix, $target);
|
$plan = $this->readAhead($snapshotId, $prefix, $target, $stripPrefix, $keepUnchanged);
|
||||||
try {
|
try {
|
||||||
return $this->restoreEntries($snapshotId, $target, $prefix, $stripPrefix, $heartbeat, $plan);
|
return $this->restoreEntries($snapshotId, $target, $prefix, $stripPrefix, $heartbeat, $plan, $skipped, $keepUnchanged);
|
||||||
} finally {
|
} finally {
|
||||||
$plan?->close();
|
$plan?->close();
|
||||||
if ($plan !== null) {
|
if ($plan !== null) {
|
||||||
@@ -241,19 +294,32 @@ final class Repository {
|
|||||||
* pass over the snapshot lists the ranges of packs it needs, in order. The spool is a
|
* pass over the snapshot lists the ranges of packs it needs, in order. The spool is a
|
||||||
* folder in the target, which has room for the restore anyway.
|
* folder in the target, which has room for the restore anyway.
|
||||||
*/
|
*/
|
||||||
private function readAhead(string $snapshotId, string $prefix, string $target): ?ReadAheadPlan {
|
private function readAhead(string $snapshotId, string $prefix, string $target, bool $stripPrefix, bool $keepUnchanged): ?ReadAheadPlan {
|
||||||
if (!$this->backend instanceof IReadsAhead || !$this->backend->canReadAhead()) {
|
if (!$this->backend instanceof IReadsAhead || !$this->backend->canReadAhead()) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
// what a restore that was stopped left behind: empty, its files were unlinked when opened
|
// what a restore that was stopped left behind: its files are unlinked once opened, but one
|
||||||
|
// that was being written when the process was killed stays (8 MiB, DR test 2026-11-05)
|
||||||
$stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR);
|
$stale = glob(rtrim($target, '/') . '/.ngb-restore-spool-*', GLOB_ONLYDIR);
|
||||||
foreach ($stale === false ? [] : $stale as $dir) {
|
foreach ($stale === false ? [] : $stale as $dir) {
|
||||||
|
if (is_link($dir) || preg_match('/^\.ngb-restore-spool-[0-9a-f]{8}$/', basename($dir)) !== 1) {
|
||||||
|
continue; // not a spool of ours, whatever it is called
|
||||||
|
}
|
||||||
|
$files = glob($dir . '/*');
|
||||||
|
foreach ($files === false ? [] : $files as $file) {
|
||||||
|
if (is_file($file) && !is_link($file)) {
|
||||||
|
@unlink($file);
|
||||||
|
}
|
||||||
|
}
|
||||||
@rmdir($dir);
|
@rmdir($dir);
|
||||||
}
|
}
|
||||||
$plan = new ReadAheadPlan(rtrim($target, '/') . '/.ngb-restore-spool-' . bin2hex(random_bytes(4)));
|
$plan = new ReadAheadPlan(rtrim($target, '/') . '/.ngb-restore-spool-' . bin2hex(random_bytes(4)));
|
||||||
try {
|
try {
|
||||||
foreach ($this->streamEntries($snapshotId) as $entry) {
|
foreach ($this->streamEntries($snapshotId) as $entry) {
|
||||||
if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
|
if (!self::underPrefix($entry['p'], $prefix)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if ($keepUnchanged && self::inPlace(self::destination($target, $entry['p'], $prefix, $stripPrefix), $entry)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
foreach ($entry['b'] as $id) {
|
foreach ($entry['b'] as $id) {
|
||||||
@@ -284,53 +350,245 @@ final class Repository {
|
|||||||
return $plan;
|
return $plan;
|
||||||
}
|
}
|
||||||
|
|
||||||
private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan): int {
|
/** @param array<string, string>|null $skipped see restore() */
|
||||||
|
private function restoreEntries(string $snapshotId, string $target, string $prefix, bool $stripPrefix, ?callable $heartbeat, ?ReadAheadPlan $plan, ?array &$skipped, bool $keepUnchanged): int {
|
||||||
$count = 0;
|
$count = 0;
|
||||||
|
$cleaned = [];
|
||||||
foreach ($this->streamEntries($snapshotId) as $entry) {
|
foreach ($this->streamEntries($snapshotId) as $entry) {
|
||||||
if ($prefix !== '' && !str_starts_with($entry['p'], $prefix)) {
|
if (!self::underPrefix($entry['p'], $prefix)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
$relPath = ($stripPrefix && $prefix !== '') ? substr($entry['p'], strlen(rtrim($prefix, '/')) + 1) : $entry['p'];
|
if ($heartbeat !== null) {
|
||||||
$dest = rtrim($target, '/') . '/' . $relPath;
|
$heartbeat(); // also for a long row of files that are skipped or have no data
|
||||||
if (!is_dir(dirname($dest))) {
|
|
||||||
mkdir(dirname($dest), 0700, true);
|
|
||||||
}
|
}
|
||||||
// Defence in depth: the resolved directory must stay below the target (no symlink escapes).
|
$dest = self::destination($target, $entry['p'], $prefix, $stripPrefix);
|
||||||
$base = realpath($target);
|
$dir = self::directoryInTarget($target, dirname($dest), $entry['p']);
|
||||||
$dir = realpath(dirname($dest));
|
if (is_link($dest)) {
|
||||||
if ($base === false || $dir === false || ($dir !== $base && !str_starts_with($dir, $base . '/')) || is_link($dest)) {
|
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $entry['p']);
|
||||||
throw new RepositoryException('Refusing to write outside the restore directory: ' . $entry['p']);
|
}
|
||||||
|
if (!isset($cleaned[$dir])) {
|
||||||
|
self::removeStaleTemporaries($dir);
|
||||||
|
$cleaned[$dir] = true;
|
||||||
|
}
|
||||||
|
if ($keepUnchanged && self::inPlace($dest, $entry)) {
|
||||||
|
$count++;
|
||||||
|
continue;
|
||||||
}
|
}
|
||||||
// Write next to the destination and rename after all blobs verified: at most one file
|
// Write next to the destination and rename after all blobs verified: at most one file
|
||||||
// is ever present twice.
|
// is ever present twice.
|
||||||
$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
|
$tmp = $dest . '.ngb-restore-' . bin2hex(random_bytes(6));
|
||||||
$out = @fopen($tmp, 'xb'); // fails if anything (including a symlink) already exists there
|
$out = @fopen($tmp, 'xb'); // fails if anything (including a symlink) already exists there
|
||||||
if ($out === false) {
|
if ($out === false) {
|
||||||
throw new RepositoryException('Cannot create a temporary file for ' . $entry['p']);
|
throw new RestoreTargetException('Cannot create a temporary file for ' . $entry['p'] . ' (disk full, or no permission?)');
|
||||||
}
|
}
|
||||||
|
try {
|
||||||
foreach ($entry['b'] as $id) {
|
foreach ($entry['b'] as $id) {
|
||||||
fwrite($out, $this->restoreBlob($id, $plan));
|
self::writeAll($out, $this->restoreBlob($id, $plan), $entry['p']);
|
||||||
if ($heartbeat !== null) {
|
if ($heartbeat !== null) {
|
||||||
$heartbeat();
|
$heartbeat();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
fclose($out);
|
if (!@fclose($out)) {
|
||||||
|
throw new RestoreTargetException('Cannot write ' . $entry['p'] . ' to the restore directory (disk full?)');
|
||||||
|
}
|
||||||
|
clearstatcache(true, $tmp);
|
||||||
if (filesize($tmp) !== $entry['s']) {
|
if (filesize($tmp) !== $entry['s']) {
|
||||||
unlink($tmp);
|
|
||||||
throw new RepositoryException('Size mismatch restoring ' . $entry['p']);
|
throw new RepositoryException('Size mismatch restoring ' . $entry['p']);
|
||||||
}
|
}
|
||||||
|
} catch (RepositoryException|BackendException|\OCA\NgBackup\Crypto\CryptoException $e) {
|
||||||
|
// What the location gave back for this file is missing or damaged. Writing to the
|
||||||
|
// target (below) is not part of this: a full disk must still stop the restore.
|
||||||
|
if (is_resource($out)) {
|
||||||
|
fclose($out);
|
||||||
|
}
|
||||||
|
@unlink($tmp);
|
||||||
|
if ($skipped === null) {
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
$skipped[$entry['p']] = $e->getMessage();
|
||||||
|
continue;
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
if (is_resource($out)) {
|
||||||
|
@fclose($out);
|
||||||
|
}
|
||||||
|
@unlink($tmp);
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
if (!rename($tmp, $dest)) {
|
if (!rename($tmp, $dest)) {
|
||||||
@unlink($tmp);
|
@unlink($tmp);
|
||||||
throw new RepositoryException('Cannot put the restored file in place: ' . $entry['p']);
|
throw new RestoreTargetException('Cannot put the restored file in place: ' . $entry['p']);
|
||||||
}
|
}
|
||||||
if (!touch($dest, $entry['m'])) {
|
if (!touch($dest, $entry['m'])) {
|
||||||
throw new RepositoryException('Cannot set the modification time of ' . $entry['p']);
|
throw new RestoreTargetException('Cannot set the modification time of ' . $entry['p']);
|
||||||
}
|
}
|
||||||
$count++;
|
$count++;
|
||||||
}
|
}
|
||||||
|
// Last: a file can only be restored into a folder that exists, and the meta record was
|
||||||
|
// read along with the files, so this asks nothing more of the location.
|
||||||
|
foreach ($this->emptyDirs($snapshotId) as [$path, $mtime]) {
|
||||||
|
if (self::underPrefix($path, $prefix)) {
|
||||||
|
if ($heartbeat !== null) {
|
||||||
|
$heartbeat();
|
||||||
|
}
|
||||||
|
// The prefix itself, stripped, is the target (for a file it is the file's name there).
|
||||||
|
$dest = $stripPrefix && $path === rtrim($prefix, '/') ? rtrim($target, '/') : self::destination($target, $path, $prefix, $stripPrefix);
|
||||||
|
self::restoreEmptyDir($target, $dest, $path, $mtime);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Very last, nothing is written in them any more. Also on a run that goes on after a stop, and
|
||||||
|
// the deepest first. The time is not worth stopping for: all the files are there by now.
|
||||||
|
foreach (array_reverse($this->folderTimes($snapshotId)) as [$path, $mtime]) {
|
||||||
|
if (!self::underPrefix($path, $prefix) || ($stripPrefix && $path === rtrim($prefix, '/'))) {
|
||||||
|
continue; // the target itself is not ours to change
|
||||||
|
}
|
||||||
|
if ($heartbeat !== null) {
|
||||||
|
$heartbeat();
|
||||||
|
}
|
||||||
|
$dest = self::destination($target, $path, $prefix, $stripPrefix);
|
||||||
|
@touch(self::directoryInTarget($target, $dest, $path), $mtime);
|
||||||
|
}
|
||||||
return $count;
|
return $count;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The folders with something in them that a snapshot recorded, parents before children (none
|
||||||
|
* before 1.1.0). Damage here is no reason to give up on the files of a restore.
|
||||||
|
*
|
||||||
|
* @return list<array{0: string, 1: int}> path, modification time
|
||||||
|
*/
|
||||||
|
public function folderTimes(string $snapshotId): array {
|
||||||
|
$meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId);
|
||||||
|
$packed = is_string($meta['folderTimes'] ?? null) ? base64_decode($meta['folderTimes'], true) : false;
|
||||||
|
$plain = $packed === false ? false : @gzinflate($packed);
|
||||||
|
$folders = [];
|
||||||
|
foreach ($plain === false ? [] : explode("\n", $plain) as $line) {
|
||||||
|
$folder = json_decode($line, true);
|
||||||
|
if (is_array($folder) && is_string($folder[0] ?? null) && self::isSafePath($folder[0]) && is_int($folder[1] ?? null)) {
|
||||||
|
$folders[] = [$folder[0], $folder[1]];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $folders;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The folders without anything in them that a snapshot recorded (none before 1.0.3).
|
||||||
|
*
|
||||||
|
* @return list<array{0: string, 1: int}> path, modification time
|
||||||
|
*/
|
||||||
|
public function emptyDirs(string $snapshotId): array {
|
||||||
|
$meta = $this->lastMeta[$snapshotId] ?? $this->snapshotMeta($snapshotId);
|
||||||
|
$dirs = [];
|
||||||
|
foreach (is_array($meta['emptyDirs'] ?? null) ? $meta['emptyDirs'] : [] as $dir) {
|
||||||
|
// One folder that makes no sense is no reason to give up on the files of a restore.
|
||||||
|
if (is_array($dir) && is_string($dir[0] ?? null) && self::isSafePath($dir[0]) && is_int($dir[1] ?? null)) {
|
||||||
|
$dirs[] = [$dir[0], $dir[1]];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $dirs;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static function restoreEmptyDir(string $target, string $dest, string $path, int $mtime): void {
|
||||||
|
if (is_link($dest)) {
|
||||||
|
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
|
||||||
|
}
|
||||||
|
$existed = is_dir($dest);
|
||||||
|
self::directoryInTarget($target, $dest, $path);
|
||||||
|
if (!$existed && !@touch($dest, $mtime)) {
|
||||||
|
throw new RestoreTargetException('Cannot set the modification time of ' . $path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Make sure a folder exists below the target, without following a link out of it: what exists
|
||||||
|
* of the path is checked before anything is created.
|
||||||
|
*
|
||||||
|
* @return string the folder's real path
|
||||||
|
*/
|
||||||
|
private static function directoryInTarget(string $target, string $dir, string $path): string {
|
||||||
|
if (!is_dir($target) && !@mkdir($target, 0700, true) && !is_dir($target)) {
|
||||||
|
throw new RestoreTargetException('Cannot create the restore directory ' . $target);
|
||||||
|
}
|
||||||
|
$base = realpath($target);
|
||||||
|
$existing = $dir;
|
||||||
|
while (!file_exists($existing) && !is_link($existing) && dirname($existing) !== $existing) {
|
||||||
|
$existing = dirname($existing);
|
||||||
|
}
|
||||||
|
$real = realpath($existing);
|
||||||
|
if ($base === false || $real === false || ($real !== $base && !str_starts_with($real, $base . '/'))) {
|
||||||
|
throw new RestoreTargetException('Refusing to write outside the restore directory: ' . $path);
|
||||||
|
}
|
||||||
|
if ($existing === $dir) {
|
||||||
|
if (!is_dir($real)) {
|
||||||
|
throw new RestoreTargetException('Cannot create the folder for ' . $path . ': a file is in the way');
|
||||||
|
}
|
||||||
|
return $real;
|
||||||
|
}
|
||||||
|
if (!@mkdir($dir, 0700, true) && !is_dir($dir)) {
|
||||||
|
throw new RestoreTargetException('Cannot create the folder for ' . $path . ' (disk full, no permission, or a file in the way?)');
|
||||||
|
}
|
||||||
|
return (string)realpath($dir);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Write all of $data, or stop: a short write leaves a file that looks damaged only by its size.
|
||||||
|
* A write may take only part of the data and still be fine; one that takes nothing is not.
|
||||||
|
*
|
||||||
|
* @param resource $out
|
||||||
|
*/
|
||||||
|
public static function writeAll($out, string $data, string $path): void {
|
||||||
|
while ($data !== '') {
|
||||||
|
$written = @fwrite($out, $data);
|
||||||
|
if ($written === false || $written === 0) {
|
||||||
|
throw new RestoreTargetException('Cannot write ' . $path . ' to the restore directory (disk full?)');
|
||||||
|
}
|
||||||
|
$data = substr($data, $written);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Where restore() writes the snapshot path $path. */
|
||||||
|
private static function destination(string $target, string $path, string $prefix, bool $stripPrefix): string {
|
||||||
|
$relPath = $path;
|
||||||
|
if ($stripPrefix && $prefix !== '') {
|
||||||
|
$relPath = $path === rtrim($prefix, '/') ? basename($path) : substr($path, strlen($prefix));
|
||||||
|
}
|
||||||
|
return rtrim($target, '/') . '/' . $relPath;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A regular file (no link) at $dest with the size and modification time of the snapshot
|
||||||
|
* entry: what a backup takes as unchanged, and what a finished restore of it leaves.
|
||||||
|
*
|
||||||
|
* @param array{s: int, m: int, ...} $entry
|
||||||
|
*/
|
||||||
|
private static function inPlace(string $dest, array $entry): bool {
|
||||||
|
clearstatcache(true, $dest);
|
||||||
|
return !is_link($dest) && is_file($dest) && filesize($dest) === $entry['s'] && filemtime($dest) === $entry['m'];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** $prefix is "" (everything) or ends with "/"; the path without that "/" is a file to restore too. */
|
||||||
|
private static function underPrefix(string $path, string $prefix): bool {
|
||||||
|
return $prefix === '' || str_starts_with($path, $prefix) || $path === rtrim($prefix, '/');
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Remove what a restore that was stopped left in a directory this restore writes to:
|
||||||
|
* "<name>.ngb-restore-<hex>" from here, ".<name>.ngb-restore-<hex>" from a restore through
|
||||||
|
* Nextcloud. A files:scan would otherwise make them show up in Files.
|
||||||
|
*/
|
||||||
|
public static function removeStaleTemporaries(string $dir): void {
|
||||||
|
$names = scandir($dir);
|
||||||
|
foreach ($names === false ? [] : $names as $name) {
|
||||||
|
$path = $dir . '/' . $name;
|
||||||
|
if (self::isRestoreTemporary($name) && is_file($path) && !is_link($path)) {
|
||||||
|
@unlink($path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static function isRestoreTemporary(string $name): bool {
|
||||||
|
return preg_match('/.\.ngb-restore-[0-9a-f]{6}(?:[0-9a-f]{6})?$/', $name) === 1;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Start writing a generated stream (e.g. a table dump) as blobs. Call flushPacks() when the
|
* Start writing a generated stream (e.g. a table dump) as blobs. Call flushPacks() when the
|
||||||
* run is done, so the last pack is uploaded before anything references it.
|
* run is done, so the last pack is uploaded before anything references it.
|
||||||
@@ -561,6 +819,31 @@ final class Repository {
|
|||||||
return isset($this->catalog->userExports[$manifestPath]);
|
return isset($this->catalog->userExports[$manifestPath]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The recorded export $ref names: a manifest path (users/<uid>/<exportId>) as is, or a bare
|
||||||
|
* user id (also "users/<uid>") when that user has exactly one export here. Anything else
|
||||||
|
* throws, with the exports to choose from in the message.
|
||||||
|
*/
|
||||||
|
public function resolveUserExport(string $ref): string {
|
||||||
|
if ($this->isRecordedUserExport($ref)) {
|
||||||
|
return $ref;
|
||||||
|
}
|
||||||
|
$rest = trim($ref, '/');
|
||||||
|
if (str_starts_with($rest, 'users/')) {
|
||||||
|
$rest = substr($rest, strlen('users/'));
|
||||||
|
}
|
||||||
|
$uid = explode('/', $rest)[0];
|
||||||
|
$exports = $uid !== '' ? $this->listUserExports($uid) : [];
|
||||||
|
if (!str_contains($rest, '/') && count($exports) === 1) {
|
||||||
|
return $exports[0];
|
||||||
|
}
|
||||||
|
if ($exports === []) {
|
||||||
|
throw new RepositoryException("$ref is not a recorded user export for this location, and $uid has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports");
|
||||||
|
}
|
||||||
|
$which = str_contains($rest, '/') ? "$ref is not a recorded user export for this location" : "$uid has " . count($exports) . ' exports here';
|
||||||
|
throw new RepositoryException("$which; pass one of these (newest first):\n " . implode("\n ", $exports));
|
||||||
|
}
|
||||||
|
|
||||||
/** Permanently delete a user export. Its blobs are freed by the next prune() like any other. */
|
/** Permanently delete a user export. Its blobs are freed by the next prune() like any other. */
|
||||||
public function forgetUserExport(string $manifestPath): void {
|
public function forgetUserExport(string $manifestPath): void {
|
||||||
if (!isset($this->catalog->userExports[$manifestPath])) {
|
if (!isset($this->catalog->userExports[$manifestPath])) {
|
||||||
@@ -608,6 +891,34 @@ final class Repository {
|
|||||||
return array_keys($doomed);
|
return array_keys($doomed);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Part of trash(), for a page that shows how far it is: each entry is a read of its own on the
|
||||||
|
* location. Reads the entries with an id after $after, in order, until $deadline
|
||||||
|
* (microtime(true)), at least one. The id, not a position: a cleanup that changes the trash
|
||||||
|
* between two parts then makes no entry come twice or be missed.
|
||||||
|
*
|
||||||
|
* @return array{total: int, items: array<string, array<string, mixed>>}
|
||||||
|
*/
|
||||||
|
public function trashPage(string $after, float $deadline): array {
|
||||||
|
$ids = array_map('strval', array_keys($this->catalog->trash));
|
||||||
|
sort($ids, SORT_STRING);
|
||||||
|
$items = [];
|
||||||
|
foreach ($ids as $id) {
|
||||||
|
if (strcmp($id, $after) <= 0) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$items[$id] = $this->trashInfo($id);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
throw new RepositoryException("Trash entry $id listed in the catalog is missing on the location (possible tampering)");
|
||||||
|
}
|
||||||
|
if (microtime(true) >= $deadline) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ['total' => count($ids), 'items' => $items];
|
||||||
|
}
|
||||||
|
|
||||||
private function trashInfo(string $snapshotId): array {
|
private function trashInfo(string $snapshotId): array {
|
||||||
return json_decode($this->getObject('trash/info/' . $snapshotId), true, 512, JSON_THROW_ON_ERROR);
|
return json_decode($this->getObject('trash/info/' . $snapshotId), true, 512, JSON_THROW_ON_ERROR);
|
||||||
}
|
}
|
||||||
@@ -1167,7 +1478,10 @@ final class Repository {
|
|||||||
$this->cipher->decrypt($fh, $plain, 'snapshot:' . $snapshotId);
|
$this->cipher->decrypt($fh, $plain, 'snapshot:' . $snapshotId);
|
||||||
fclose($fh);
|
fclose($fh);
|
||||||
rewind($plain);
|
rewind($plain);
|
||||||
fgets($plain); // meta line
|
$meta = json_decode((string)fgets($plain), true);
|
||||||
|
if (is_array($meta)) {
|
||||||
|
$this->lastMeta = [$snapshotId => $meta]; // a snapshot never changes
|
||||||
|
}
|
||||||
while (($line = fgets($plain)) !== false) {
|
while (($line = fgets($plain)) !== false) {
|
||||||
$entry = json_decode($line, true, 512, JSON_THROW_ON_ERROR);
|
$entry = json_decode($line, true, 512, JSON_THROW_ON_ERROR);
|
||||||
if (!is_array($entry) || !is_string($entry['p'] ?? null) || !self::isSafePath($entry['p'])
|
if (!is_array($entry) || !is_string($entry['p'] ?? null) || !self::isSafePath($entry['p'])
|
||||||
|
|||||||
@@ -0,0 +1,15 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Repository;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A restore cannot write to where it restores to (a full disk, permissions, something in the way).
|
||||||
|
* The backup is fine, so this never counts as damaged data: the restore stops, also with --salvage.
|
||||||
|
*/
|
||||||
|
final class RestoreTargetException extends \RuntimeException {
|
||||||
|
}
|
||||||
@@ -61,6 +61,16 @@ final class AlertService {
|
|||||||
'examples' => mb_strimwidth(implode(', ', array_merge($examples['unreadable'], $examples['changing'])), 0, 300, '…')]);
|
'examples' => mb_strimwidth(implode(', ', array_merge($examples['unreadable'], $examples['changing'])), 0, 300, '…')]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A backup that finished but left out files or folders because of their name.
|
||||||
|
*
|
||||||
|
* @param list<string> $examples
|
||||||
|
*/
|
||||||
|
public function runBadNames(Run $run, string $targetName, int $count, array $examples): void {
|
||||||
|
$this->notifyAdmins('run_bad_names', 'run-' . $run->getId(), ['target' => $targetName, 'count' => (string)$count,
|
||||||
|
'examples' => mb_strimwidth(implode(', ', $examples), 0, 300, '…')]);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Remember how much is stored on a location (for the settings page) and warn administrators
|
* Remember how much is stored on a location (for the settings page) and warn administrators
|
||||||
* once a day when it is at 80% of its limit or more.
|
* once a day when it is at 80% of its limit or more.
|
||||||
@@ -91,9 +101,36 @@ final class AlertService {
|
|||||||
return $v === '' ? null : (int)$v;
|
return $v === '' ? null : (int)$v;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether it is time to remind administrators of a location without a schedule: once a week
|
||||||
|
* while another location is still backed up, every day when none is. An hour of margin, so
|
||||||
|
* that the moment of the cron run does not push a reminder to the next day.
|
||||||
|
*/
|
||||||
|
public static function offReminderDue(int $last, int $now, bool $othersActive): bool {
|
||||||
|
return $now - $last >= ($othersActive ? 7 * 86400 : 86400) - 3600;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function checkOff(Target $target, bool $othersActive): void {
|
||||||
|
if ((string)$target->getSchedule() !== '') {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
$key = 'off_notified_' . $target->getId();
|
||||||
|
$now = time();
|
||||||
|
$last = max((int)$target->getScheduleSince(), $this->appConfig->getValueInt(Application::APP_ID, $key, 0));
|
||||||
|
if (!self::offReminderDue($last, $now, $othersActive)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
$this->appConfig->setValueInt(Application::APP_ID, $key, $now);
|
||||||
|
$this->notifyAdmins('schedule_off', 'off-' . $target->getId() . '-' . date('Y-m-d', $now),
|
||||||
|
['target' => $target->getName(), 'others' => $othersActive ? '1' : '0']);
|
||||||
|
}
|
||||||
|
|
||||||
/** At most one notification per location per day. */
|
/** At most one notification per location per day. */
|
||||||
public function checkStale(Target $target): void {
|
public function checkStale(Target $target): void {
|
||||||
$last = $this->runs->lastFinished($target->getId(), BackupService::KIND_FULL);
|
if ((string)$target->getSchedule() === '') {
|
||||||
|
return; // no schedule is a choice, as for a location that is only connected now and then
|
||||||
|
}
|
||||||
|
$last =$this->runs->lastFinished($target->getId(), BackupService::KIND_FULL);
|
||||||
$lastTime = $last?->getFinishedAt() ?? $target->getCreatedAt();
|
$lastTime = $last?->getFinishedAt() ?? $target->getCreatedAt();
|
||||||
if (time() - $lastTime < self::STALE_AFTER || $this->runs->findRunning($target->getId(), BackupService::KIND_FULL) !== []) {
|
if (time() - $lastTime < self::STALE_AFTER || $this->runs->findRunning($target->getId(), BackupService::KIND_FULL) !== []) {
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -34,6 +34,8 @@ use Psr\Log\LoggerInterface;
|
|||||||
*/
|
*/
|
||||||
final class BackupService {
|
final class BackupService {
|
||||||
public const KIND_FULL = 'full';
|
public const KIND_FULL = 'full';
|
||||||
|
/** Characters of a snapshot label (the column holds 255) */
|
||||||
|
public const LABEL_MAX = 100;
|
||||||
|
|
||||||
/** @var \WeakMap<Repository, string> the location settings each kept repository was opened with */
|
/** @var \WeakMap<Repository, string> the location settings each kept repository was opened with */
|
||||||
private \WeakMap $openedWith;
|
private \WeakMap $openedWith;
|
||||||
@@ -56,7 +58,23 @@ final class BackupService {
|
|||||||
$this->openedWith = $opened;
|
$this->openedWith = $opened;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The running backups in the order they get their turn: first come, first served. The first one
|
||||||
|
* runs, the others are queued; only one at a time reads the disks.
|
||||||
|
*
|
||||||
|
* @return list<Run>
|
||||||
|
*/
|
||||||
|
public function runsInTurn(): array {
|
||||||
|
$running = $this->runs->findRunning(null, self::KIND_FULL);
|
||||||
|
usort($running, static fn (Run $a, Run $b): int => [$a->getStartedAt(), $a->getId()] <=> [$b->getStartedAt(), $b->getId()]);
|
||||||
|
return $running;
|
||||||
|
}
|
||||||
|
|
||||||
public function start(Target $target, string $label = ''): Run {
|
public function start(Target $target, string $label = ''): Run {
|
||||||
|
$label = trim($label);
|
||||||
|
if (mb_strlen($label) > self::LABEL_MAX) {
|
||||||
|
throw new \InvalidArgumentException('A label has at most ' . self::LABEL_MAX . ' characters');
|
||||||
|
}
|
||||||
if (!$this->keys->everConfirmed()) {
|
if (!$this->keys->everConfirmed()) {
|
||||||
throw new \RuntimeException('Confirm the recovery kit first (occ backup:key:confirm)');
|
throw new \RuntimeException('Confirm the recovery kit first (occ backup:key:confirm)');
|
||||||
}
|
}
|
||||||
@@ -73,7 +91,7 @@ final class BackupService {
|
|||||||
try {
|
try {
|
||||||
return $this->leases->with(PruneService::lockKey($target), LeaseService::EXCLUSIVE, 60, fn () => $this->createRun($target, $label));
|
return $this->leases->with(PruneService::lockKey($target), LeaseService::EXCLUSIVE, 60, fn () => $this->createRun($target, $label));
|
||||||
} catch (LockedException) {
|
} catch (LockedException) {
|
||||||
throw new \RuntimeException('A cleanup or another start for ' . $target->getName() . ' is running; try again later');
|
throw new \RuntimeException('A cleanup, a restore or another start for ' . $target->getName() . ' is running, or one was stopped by force (that clears by itself within two hours); try again later');
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -204,6 +222,10 @@ final class BackupService {
|
|||||||
$state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR);
|
$state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR);
|
||||||
|
|
||||||
if ($run->getPhase() === 'db') {
|
if ($run->getPhase() === 'db') {
|
||||||
|
if (!isset($state['dump'])) {
|
||||||
|
// It may have been queued for hours: the restore point has the time of its data.
|
||||||
|
$state['files']['started'] = gmdate('c');
|
||||||
|
}
|
||||||
$dumper = new DbDumper($this->db->getInner(), $this->config->getSystemValueString('dbtableprefix', 'oc_'));
|
$dumper = new DbDumper($this->db->getInner(), $this->config->getSystemValueString('dbtableprefix', 'oc_'));
|
||||||
$dbStats = $state['dbStats'] ?? [];
|
$dbStats = $state['dbStats'] ?? [];
|
||||||
// With system cron the whole dump usually fits in one step (one transaction, fully
|
// With system cron the whole dump usually fits in one step (one transaction, fully
|
||||||
@@ -247,8 +269,13 @@ final class BackupService {
|
|||||||
try {
|
try {
|
||||||
$this->alerts->storedBytes($target, $repo->storedBytes());
|
$this->alerts->storedBytes($target, $repo->storedBytes());
|
||||||
$stats = $state['files']['stats'];
|
$stats = $state['files']['stats'];
|
||||||
if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) {
|
|
||||||
$warnings = BackupRun::warnings($state['files']);
|
$warnings = BackupRun::warnings($state['files']);
|
||||||
|
if (($stats['badNames'] ?? 0) > 0) {
|
||||||
|
$this->logger->warning(sprintf('NG Backup run %d to %s: %d left out because their name is not supported (rename them): %s',
|
||||||
|
$run->getId(), $target->getName(), $stats['badNames'], implode(', ', $warnings['badNames'])));
|
||||||
|
$this->alerts->runBadNames($run, $target->getName(), (int)$stats['badNames'], $warnings['badNames']);
|
||||||
|
}
|
||||||
|
if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) {
|
||||||
$this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s',
|
$this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s',
|
||||||
$run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']),
|
$run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']),
|
||||||
$stats['changing'] ?? 0, implode(', ', $warnings['changing'])));
|
$stats['changing'] ?? 0, implode(', ', $warnings['changing'])));
|
||||||
|
|||||||
@@ -0,0 +1,97 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Service;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Db\Run;
|
||||||
|
use OCP\Exceptions\AppConfigTypeConflictException;
|
||||||
|
use OCP\IAppConfig;
|
||||||
|
use OCP\IURLGenerator;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether Nextcloud's background jobs run on a system cron, which backups need: a backup moves
|
||||||
|
* forward only when a background job run takes its next step (#13). Without system cron, AJAX
|
||||||
|
* cron runs a job only now and then while someone has Nextcloud open, and webcron only as often as
|
||||||
|
* the outside service calls it.
|
||||||
|
*/
|
||||||
|
final class CronHealth {
|
||||||
|
/** A system cron that has not run for this long has stopped (it runs every 5 minutes). */
|
||||||
|
public const STOPPED_AFTER = 3600;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A running backup that took no step for this long is waiting for background jobs. A step with
|
||||||
|
* system cron takes at most a few minutes (BackupJob) and the job runs every minute.
|
||||||
|
*/
|
||||||
|
public const WAITING_AFTER = 900;
|
||||||
|
|
||||||
|
public function __construct(
|
||||||
|
private IAppConfig $appConfig,
|
||||||
|
private IURLGenerator $urls,
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* mode: ajax, webcron or cron; lastRun: the last background job run (0: never); ok: a system
|
||||||
|
* cron that ran recently; stopped: system cron is set but has not run for STOPPED_AFTER.
|
||||||
|
*
|
||||||
|
* @return array{mode: string, lastRun: int, ok: bool, stopped: bool, docs: string}
|
||||||
|
*/
|
||||||
|
public function check(?int $now = null): array {
|
||||||
|
$now ??= time();
|
||||||
|
$mode = $this->coreString('backgroundjobs_mode', 'ajax');
|
||||||
|
$lastRun = (int)$this->coreString('lastcron', '0');
|
||||||
|
$stopped = $mode === 'cron' && $now - $lastRun > self::STOPPED_AFTER;
|
||||||
|
return [
|
||||||
|
'mode' => $mode,
|
||||||
|
'lastRun' => $lastRun,
|
||||||
|
'ok' => $mode === 'cron' && !$stopped,
|
||||||
|
'stopped' => $stopped,
|
||||||
|
'docs' => $this->urls->linkToDocs('admin-background-jobs'),
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Seconds since a running backup took its last step, when that is long enough to say it waits
|
||||||
|
* for background jobs; null while it moves (or when it is not running).
|
||||||
|
*/
|
||||||
|
public static function waitingFor(Run $run, ?int $now = null): ?int {
|
||||||
|
$idle = ($now ?? time()) - $run->getUpdatedAt();
|
||||||
|
return $run->getStatus() === Run::RUNNING && $idle >= self::WAITING_AFTER ? $idle : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One line for occ and the log; null when system cron runs. */
|
||||||
|
public static function describe(array $health, ?int $now = null): ?string {
|
||||||
|
if ($health['ok']) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
$last = $health['lastRun'] > 0 ? 'last ran ' . self::ago(($now ?? time()) - $health['lastRun']) : 'never ran';
|
||||||
|
$what = $health['stopped']
|
||||||
|
? 'Background jobs are set to system cron, but cron has not run for over an hour (' . $last . ').'
|
||||||
|
: 'Background jobs do not run on a system cron: they are set to "' . ($health['mode'] === 'webcron' ? 'Webcron' : 'AJAX') . '" and ' . $last . '.';
|
||||||
|
return $what . ' Backups, also the scheduled ones, need a system cron that runs every 5 minutes: ' . $health['docs'];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** "12 min ago" style, for durations from a minute to days */
|
||||||
|
public static function ago(int $seconds): string {
|
||||||
|
if ($seconds < 3600) {
|
||||||
|
return max(1, intdiv($seconds, 60)) . ' min ago';
|
||||||
|
}
|
||||||
|
if ($seconds < 172800) {
|
||||||
|
return intdiv($seconds, 3600) . ' h ago';
|
||||||
|
}
|
||||||
|
return intdiv($seconds, 86400) . ' days ago';
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Core stores these with different value types across versions: read whatever is there. */
|
||||||
|
private function coreString(string $key, string $default): string {
|
||||||
|
try {
|
||||||
|
return $this->appConfig->getValueString('core', $key, $default);
|
||||||
|
} catch (AppConfigTypeConflictException) {
|
||||||
|
return (string)$this->appConfig->getValueInt('core', $key, (int)$default);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Service;
|
||||||
|
|
||||||
|
/** A step after the database was committed failed: the database is no longer the fresh one. */
|
||||||
|
final class DatabaseAlreadyRestoredException extends \RuntimeException {
|
||||||
|
}
|
||||||
@@ -76,6 +76,15 @@ final class LeaseService {
|
|||||||
$qb->executeStatement();
|
$qb->executeStatement();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Remove the shared leases on $name, for a caller that knows their holders were stopped. */
|
||||||
|
public function dropShared(string $name): void {
|
||||||
|
$qb = $this->db->getQueryBuilder();
|
||||||
|
$qb->delete('ngb_leases')
|
||||||
|
->where($qb->expr()->eq('name', $qb->createNamedParameter($name)))
|
||||||
|
->andWhere($qb->expr()->eq('mode', $qb->createNamedParameter(self::SHARED)));
|
||||||
|
$qb->executeStatement();
|
||||||
|
}
|
||||||
|
|
||||||
/** Wraps $refresh so calling it often (per file, per row batch) only touches the database every $seconds. */
|
/** Wraps $refresh so calling it often (per file, per row batch) only touches the database every $seconds. */
|
||||||
public static function throttled(callable $refresh, int $seconds = 60): \Closure {
|
public static function throttled(callable $refresh, int $seconds = 60): \Closure {
|
||||||
$last = time();
|
$last = time();
|
||||||
|
|||||||
@@ -0,0 +1,78 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Service;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* When a location is backed up: "" (off), "HH:MM" (daily) or "HH:MM/N" (from HH:MM every N hours,
|
||||||
|
* starting over at HH:MM the next day), in the location's time zone.
|
||||||
|
*/
|
||||||
|
final class LocationSchedule {
|
||||||
|
/** Attempts per moment after a failure, an hour apart, instead of one every few minutes. */
|
||||||
|
public const MAX_ATTEMPTS = 3;
|
||||||
|
public const RETRY_AFTER = 3600;
|
||||||
|
|
||||||
|
/** @throws \InvalidArgumentException */
|
||||||
|
public static function normalize(string $schedule): string {
|
||||||
|
$schedule = trim($schedule);
|
||||||
|
if ($schedule === '' || strtolower($schedule) === 'off') {
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
if (preg_match('/^(([01]\d|2[0-3]):[0-5]\d)(?:\/(\d{1,2}))?$/', $schedule, $m) !== 1) {
|
||||||
|
throw new \InvalidArgumentException('Use "off", HH:MM for once a day, or HH:MM/N for every N hours from HH:MM');
|
||||||
|
}
|
||||||
|
$hours = (int)($m[3] ?? 24);
|
||||||
|
if ($hours < 1 || $hours > 24) {
|
||||||
|
throw new \InvalidArgumentException('Every 1 to 24 hours');
|
||||||
|
}
|
||||||
|
return $hours === 24 ? $m[1] : $m[1] . '/' . $hours;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The most recent scheduled moment at or before $now; null without a schedule.
|
||||||
|
*
|
||||||
|
* @param non-empty-string $zone
|
||||||
|
*/
|
||||||
|
public static function lastMoment(string $schedule, string $zone, int $now): ?int {
|
||||||
|
if ($schedule === '') {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
$tz = new \DateTimeZone($zone);
|
||||||
|
$today = (new \DateTimeImmutable('@' . $now))->setTimezone($tz);
|
||||||
|
$moments = array_merge(self::moments($schedule, $today->modify('-1 day'), $tz), self::moments($schedule, $today, $tz));
|
||||||
|
$last = null;
|
||||||
|
foreach ($moments as $moment) {
|
||||||
|
if ($moment <= $now && ($last === null || $moment > $last)) {
|
||||||
|
$last = $moment;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $last;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @return list<int> the moments of one day */
|
||||||
|
private static function moments(string $schedule, \DateTimeImmutable $day, \DateTimeZone $tz): array {
|
||||||
|
[$time, $every] = array_pad(explode('/', $schedule), 2, '24');
|
||||||
|
[$hour, $minute] = array_map('intval', explode(':', $time));
|
||||||
|
$moments = [];
|
||||||
|
for ($h = $hour; $h < 24; $h += (int)$every) {
|
||||||
|
$moments[] = (new \DateTimeImmutable($day->format('Y-m-d') . sprintf(' %02d:%02d', $h, $minute), $tz))->getTimestamp();
|
||||||
|
}
|
||||||
|
return $moments;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether a backup has to start for $moment. One that was made since then is enough, also when
|
||||||
|
* several moments went by during a long backup. A moment from before the schedule was set does
|
||||||
|
* not count: choosing a time must not start a backup right away.
|
||||||
|
*/
|
||||||
|
public static function isDue(int $moment, int $since, ?int $lastDoneStarted, int $attempts, ?int $lastAttemptFinished, int $now): bool {
|
||||||
|
if ($moment < $since || ($lastDoneStarted !== null && $lastDoneStarted >= $moment) || $attempts >= self::MAX_ATTEMPTS) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return $attempts === 0 || $now - (int)$lastAttemptFinished >= self::RETRY_AFTER;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,6 +11,7 @@ use OCA\NgBackup\AppInfo\Application;
|
|||||||
use OCA\NgBackup\Db\RunMapper;
|
use OCA\NgBackup\Db\RunMapper;
|
||||||
use OCA\NgBackup\Db\SnapshotMapper;
|
use OCA\NgBackup\Db\SnapshotMapper;
|
||||||
use OCA\NgBackup\Db\Target;
|
use OCA\NgBackup\Db\Target;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
use OCP\IAppConfig;
|
use OCP\IAppConfig;
|
||||||
use OCP\Lock\ILockingProvider;
|
use OCP\Lock\ILockingProvider;
|
||||||
use OCP\Lock\LockedException;
|
use OCP\Lock\LockedException;
|
||||||
@@ -33,6 +34,54 @@ final class PruneService {
|
|||||||
) {
|
) {
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Characters of a note on a snapshot (the column holds 1000) */
|
||||||
|
public const NOTE_MAX = 500;
|
||||||
|
|
||||||
|
/** Pin or unpin a snapshot: the retention policy leaves a pinned one alone. */
|
||||||
|
public function setPinned(Target $target, string $snapshotId, bool $pinned): void {
|
||||||
|
$changed = $this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $pinned): bool {
|
||||||
|
$was = in_array($snapshotId, $repo->pinnedSnapshots(), true);
|
||||||
|
$repo->setPinned($snapshotId, $pinned);
|
||||||
|
return $was !== $pinned;
|
||||||
|
});
|
||||||
|
$this->targets->syncSnapshot($target, $snapshotId, pinned: $pinned);
|
||||||
|
if ($changed) {
|
||||||
|
// Unpinning opens the way to removing it, so every administrator hears about that.
|
||||||
|
$this->alerts->securityEvent($pinned ? 'snapshot_pinned' : 'snapshot_unpinned', ['target' => $target->getName(), 'snapshot' => $snapshotId], !$pinned);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public function setNote(Target $target, string $snapshotId, string $note): void {
|
||||||
|
$note = trim((string)preg_replace('/[\x00-\x1f\x7f]+/', ' ', $note));
|
||||||
|
if (!mb_check_encoding($note, 'UTF-8')) {
|
||||||
|
throw new \InvalidArgumentException('A note must be valid UTF-8');
|
||||||
|
}
|
||||||
|
if (mb_strlen($note) > self::NOTE_MAX) {
|
||||||
|
throw new \InvalidArgumentException('A note has at most ' . self::NOTE_MAX . ' characters');
|
||||||
|
}
|
||||||
|
$this->changeCatalog($target, function (Repository $repo) use ($snapshotId, $note): bool {
|
||||||
|
$repo->setSnapshotNote($snapshotId, $note);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
$this->targets->syncSnapshot($target, $snapshotId, note: $note);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Write to the catalog next to a running backup or restore, which only takes it for a moment to
|
||||||
|
* commit. Not during a cleanup, which holds a repository opened earlier and writes from that.
|
||||||
|
*
|
||||||
|
* @param callable(Repository): bool $change
|
||||||
|
*/
|
||||||
|
private function changeCatalog(Target $target, callable $change): bool {
|
||||||
|
try {
|
||||||
|
return $this->leases->with(self::lockKey($target), LeaseService::SHARED, 300,
|
||||||
|
fn (): bool => $this->leases->with(self::catalogLockKey($target), LeaseService::EXCLUSIVE, 300,
|
||||||
|
fn (): bool => $change($this->targets->repository($target))));
|
||||||
|
} catch (LockedException) {
|
||||||
|
throw new \RuntimeException('A cleanup of ' . $target->getName() . ' or another change to its restore points is in progress; try again later');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public function untrash(Target $target, string $snapshotId): void {
|
public function untrash(Target $target, string $snapshotId): void {
|
||||||
$lease = $this->lockExclusive($target, 300);
|
$lease = $this->lockExclusive($target, 300);
|
||||||
try {
|
try {
|
||||||
@@ -87,6 +136,7 @@ final class PruneService {
|
|||||||
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
|
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
|
||||||
$s->setFiles((int)($meta['stats']['files'] ?? 0));
|
$s->setFiles((int)($meta['stats']['files'] ?? 0));
|
||||||
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
|
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
|
||||||
|
$s->setNote($repo->snapshotNotes()[$snapshotId] ?? null);
|
||||||
try {
|
try {
|
||||||
$this->snapshots->insert($s);
|
$this->snapshots->insert($s);
|
||||||
} catch (\Throwable) {
|
} catch (\Throwable) {
|
||||||
@@ -208,7 +258,7 @@ final class PruneService {
|
|||||||
$meta = $repo->snapshotMeta($id);
|
$meta = $repo->snapshotMeta($id);
|
||||||
$times[$id] = (int)strtotime((string)($meta['time'] ?? '@0'));
|
$times[$id] = (int)strtotime((string)($meta['time'] ?? '@0'));
|
||||||
}
|
}
|
||||||
$keep = $this->policy()->keep($times);
|
$keep = $this->policy()->keepWithPinned($times, $repo->pinnedSnapshots());
|
||||||
$forget = array_values(array_diff(array_keys($times), array_keys($keep)));
|
$forget = array_values(array_diff(array_keys($times), array_keys($keep)));
|
||||||
$actor = $scheduled ? 'retention policy' : $this->alerts->actor();
|
$actor = $scheduled ? 'retention policy' : $this->alerts->actor();
|
||||||
$purged = [];
|
$purged = [];
|
||||||
|
|||||||
@@ -58,9 +58,12 @@ final class RestoreFullService {
|
|||||||
* - tables in the backup that do not exist here: a backup made from 1.0.0-rc.2 on records
|
* - tables in the backup that do not exist here: a backup made from 1.0.0-rc.2 on records
|
||||||
* their structure and the restore creates them, an older one cannot (missingTables).
|
* their structure and the restore creates them, an older one cannot (missingTables).
|
||||||
*
|
*
|
||||||
* @return array{ok:bool, currentMajor:string, snapshotMajor:string, missingApps:list<string>, missingTables:list<string>}
|
* With $salvage, a database dump that cannot be read does not block: dbUnreadable says why, and
|
||||||
|
* the restore then salvages the files instead (see restoreFull()).
|
||||||
|
*
|
||||||
|
* @return array{ok:bool, currentMajor:string, snapshotMajor:string, missingApps:list<string>, missingTables:list<string>, dbUnreadable?:string}
|
||||||
*/
|
*/
|
||||||
public function checkCompatibility(Target $target, string $snapshotId, bool $skipMissingTables = false): array {
|
public function checkCompatibility(Target $target, string $snapshotId, bool $skipMissingTables = false, bool $salvage = false): array {
|
||||||
$repo = $this->targets->repository($target);
|
$repo = $this->targets->repository($target);
|
||||||
$meta = $repo->snapshotMeta($snapshotId);
|
$meta = $repo->snapshotMeta($snapshotId);
|
||||||
$currentMajor = explode('.', $this->config->getSystemValueString('version'))[0];
|
$currentMajor = explode('.', $this->config->getSystemValueString('version'))[0];
|
||||||
@@ -69,8 +72,19 @@ final class RestoreFullService {
|
|||||||
$missingApps = array_values(array_diff(array_keys($meta['apps'] ?? []), array_keys($enabled), self::ENVIRONMENT_APPS));
|
$missingApps = array_values(array_diff(array_keys($meta['apps'] ?? []), array_keys($enabled), self::ENVIRONMENT_APPS));
|
||||||
$missingTables = [];
|
$missingTables = [];
|
||||||
$prefixMismatch = null;
|
$prefixMismatch = null;
|
||||||
|
$dbUnreadable = null;
|
||||||
|
$manifest = null;
|
||||||
if (isset($meta['db'])) {
|
if (isset($meta['db'])) {
|
||||||
|
try {
|
||||||
$manifest = json_decode($repo->getObject($meta['db']), true, 512, JSON_THROW_ON_ERROR);
|
$manifest = json_decode($repo->getObject($meta['db']), true, 512, JSON_THROW_ON_ERROR);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
if (!$salvage) {
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
$dbUnreadable = 'the database dump cannot be read: ' . $e->getMessage();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (is_array($manifest)) {
|
||||||
$prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_');
|
$prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_');
|
||||||
$sourcePrefix = DbDumper::dumpPrefix($manifest);
|
$sourcePrefix = DbDumper::dumpPrefix($manifest);
|
||||||
if ($sourcePrefix !== null && $sourcePrefix !== $prefix) {
|
if ($sourcePrefix !== null && $sourcePrefix !== $prefix) {
|
||||||
@@ -87,11 +101,21 @@ final class RestoreFullService {
|
|||||||
'missingApps' => $missingApps,
|
'missingApps' => $missingApps,
|
||||||
'missingTables' => $prefixMismatch === null ? $missingTables : [],
|
'missingTables' => $prefixMismatch === null ? $missingTables : [],
|
||||||
'prefixMismatch' => $prefixMismatch,
|
'prefixMismatch' => $prefixMismatch,
|
||||||
];
|
] + ($dbUnreadable !== null ? ['dbUnreadable' => $dbUnreadable] : []);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>, maintenanceLeftOn:bool} */
|
/**
|
||||||
public function restoreFull(Target $target, string $snapshotId, bool $skipMissingTables = false): array {
|
* With $salvage, restore whatever can be restored instead of stopping:
|
||||||
|
* - a file that cannot be read or decrypted is left out and listed (skipped);
|
||||||
|
* - when the database cannot be restored here (its dump is unreadable or damaged, or it does
|
||||||
|
* not fit this installation), the fresh database stays as it is, and the data directory and
|
||||||
|
* config of the backup go to a folder of their own (salvageDir) instead: the files are what
|
||||||
|
* matters most, and putting Nextcloud back together from there is work by hand;
|
||||||
|
* - a config.php that cannot be read is not merged (configError).
|
||||||
|
*
|
||||||
|
* @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>, maintenanceLeftOn:bool, skipped:array<string, string>, dbError:?string, salvageDir:?string, configError:?string}
|
||||||
|
*/
|
||||||
|
public function restoreFull(Target $target, string $snapshotId, bool $skipMissingTables = false, bool $salvage = false): array {
|
||||||
// Maintenance mode for the whole restore: background jobs (cron.php stops in maintenance
|
// Maintenance mode for the whole restore: background jobs (cron.php stops in maintenance
|
||||||
// mode), page visits (AJAX cron) and users must not write into tables or the data
|
// mode), page visits (AJAX cron) and users must not write into tables or the data
|
||||||
// directory while they are being replaced; a row written in between would collide with a
|
// directory while they are being replaced; a row written in between would collide with a
|
||||||
@@ -114,17 +138,27 @@ final class RestoreFullService {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
try {
|
try {
|
||||||
$check = $this->checkCompatibility($target, $snapshotId, $skipMissingTables);
|
$check = $this->checkCompatibility($target, $snapshotId, $skipMissingTables, $salvage);
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
$reopen();
|
$reopen();
|
||||||
throw $e;
|
throw $e;
|
||||||
}
|
}
|
||||||
if (!$check['ok']) {
|
if (!$check['ok']) {
|
||||||
|
if (!$salvage) {
|
||||||
$reopen();
|
$reopen();
|
||||||
throw new \RuntimeException($this->describeBlockers($check));
|
throw new \RuntimeException($this->describeBlockers($check));
|
||||||
}
|
}
|
||||||
|
// The database cannot go into this installation; the files can.
|
||||||
|
$check['dbUnreadable'] = 'the database cannot be restored here: ' . $this->describeBlockers($check);
|
||||||
|
}
|
||||||
|
if ($interrupted) {
|
||||||
|
// A restore that was killed leaves its leases, for up to two hours: they would keep
|
||||||
|
// the first backup of the recovered server waiting after this restore completes.
|
||||||
|
// In maintenance mode, on a partly restored instance, nothing else reads the location.
|
||||||
|
$this->leases->dropShared(PruneService::lockKey($target));
|
||||||
|
}
|
||||||
$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
|
$appConfig->setValueInt(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY, time());
|
||||||
$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check);
|
$result = $this->restoreInMaintenance($target, $snapshotId, $skipMissingTables, $check, $salvage);
|
||||||
$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
|
$appConfig->deleteKey(\OCA\NgBackup\AppInfo\Application::APP_ID, self::INTERRUPTED_KEY);
|
||||||
if (!$wasInMaintenance) {
|
if (!$wasInMaintenance) {
|
||||||
$this->config->setSystemValue('maintenance', false);
|
$this->config->setSystemValue('maintenance', false);
|
||||||
@@ -132,16 +166,84 @@ final class RestoreFullService {
|
|||||||
return $result + ['maintenanceLeftOn' => $wasInMaintenance];
|
return $result + ['maintenanceLeftOn' => $wasInMaintenance];
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>} */
|
/** @return array{dbReport:array, dataFiles:int, configKeys:int, dbConsistent:bool, missingApps:list<string>, skipped:array<string, string>, dbError:?string, salvageDir:?string, configError:?string} */
|
||||||
private function restoreInMaintenance(Target $target, string $snapshotId, bool $skipMissingTables, array $check): array {
|
private function restoreInMaintenance(Target $target, string $snapshotId, bool $skipMissingTables, array $check, bool $salvage): array {
|
||||||
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $skipMissingTables, $check) {
|
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $skipMissingTables, $check, $salvage) {
|
||||||
$beat = LeaseService::throttled($refresh);
|
$beat = LeaseService::throttled($refresh);
|
||||||
$repo = $this->targets->repository($target);
|
$repo = $this->targets->repository($target);
|
||||||
$meta = $repo->snapshotMeta($snapshotId);
|
$meta = $repo->snapshotMeta($snapshotId);
|
||||||
|
|
||||||
$dbReport = null;
|
$dbReport = null;
|
||||||
$manifest = [];
|
$manifest = [];
|
||||||
if (isset($meta['db'])) {
|
$dbError = $check['dbUnreadable'] ?? null;
|
||||||
|
$skipped = $salvage ? [] : null;
|
||||||
|
$dataDir = rtrim($this->config->getSystemValueString('datadirectory'), '/');
|
||||||
|
if ($dbError === null) {
|
||||||
|
try {
|
||||||
|
$this->restoreDatabase($repo, $meta, $manifest, $dbReport, $beat, $skipMissingTables);
|
||||||
|
} catch (DatabaseAlreadyRestoredException $e) {
|
||||||
|
throw $e;
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
if (!$salvage) {
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
// DbDumper::restore() rolled its transaction back: the database is the fresh one.
|
||||||
|
$dbError = 'the database could not be restored: ' . $e->getMessage();
|
||||||
|
$dbReport = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ($dbError !== null) {
|
||||||
|
$salvageDir = $dataDir . '/ngb-salvage-' . substr($snapshotId, 0, 12);
|
||||||
|
$dataFiles = $this->restore->restoreToDirectory($target, $snapshotId, 'data', $salvageDir . '/data', stripPrefix: true, heartbeat: $beat, skipped: $skipped, keepUnchanged: true);
|
||||||
|
$this->restore->restoreToDirectory($target, $snapshotId, 'config', $salvageDir . '/config', stripPrefix: true, heartbeat: $beat, skipped: $skipped);
|
||||||
|
return ['dbReport' => ['skipped' => [], 'restored' => [], 'created' => [], 'missing' => []], 'dataFiles' => $dataFiles, 'configKeys' => 0,
|
||||||
|
'dbConsistent' => true, 'missingApps' => $check['missingApps'], 'skipped' => $skipped ?? [], 'dbError' => $dbError,
|
||||||
|
'salvageDir' => $salvageDir, 'configError' => null];
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read (but do not yet apply) the backed-up config.php before touching the data
|
||||||
|
// directory: once applied, 'secret' may differ from what ng_backup's own Target
|
||||||
|
// options were encrypted with (see restoreFull()'s docblock), and every further
|
||||||
|
// repository access through $this->targets would fail to decrypt them.
|
||||||
|
$configError = null;
|
||||||
|
try {
|
||||||
|
/** @var array<string, mixed> $backupConfig */
|
||||||
|
$backupConfig = $this->fetchBackupConfig($target, $snapshotId, $salvage);
|
||||||
|
} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
|
||||||
|
throw $e; // nowhere to put it: not a config.php that cannot be read
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
if (!$salvage) {
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
$configError = $e->getMessage();
|
||||||
|
$backupConfig = [];
|
||||||
|
}
|
||||||
|
|
||||||
|
$dataFiles = $this->restore->restoreToDirectory($target, $snapshotId, 'data', $dataDir, stripPrefix: true, heartbeat: $beat, skipped: $skipped, keepUnchanged: true);
|
||||||
|
|
||||||
|
// Last step: nothing after this may need $this->targets->repository($target) again.
|
||||||
|
if ($backupConfig !== []) {
|
||||||
|
$this->targets->reencryptOptionsAround(
|
||||||
|
fn (): mixed => $this->keys->reencryptForSecretRotation(fn (): mixed => $this->config->setSystemValues($backupConfig)),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return ['dbReport' => $dbReport ?? ['skipped' => [], 'restored' => [], 'created' => [], 'missing' => []], 'dataFiles' => $dataFiles, 'configKeys' => count($backupConfig),
|
||||||
|
'dbConsistent' => (bool)($manifest['consistent'] ?? true), 'missingApps' => $check['missingApps'],
|
||||||
|
'skipped' => $skipped ?? [], 'dbError' => null, 'salvageDir' => null, 'configError' => $configError];
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Replace the database with the snapshot's dump (one transaction, see DbDumper::restore()).
|
||||||
|
*
|
||||||
|
* @param array<string, mixed> $meta the snapshot's meta record
|
||||||
|
* @param array $manifest set to the dump's manifest
|
||||||
|
*/
|
||||||
|
private function restoreDatabase(\OCA\NgBackup\Repository\Repository $repo, array $meta, array &$manifest, ?array &$dbReport, callable $beat, bool $skipMissingTables): void {
|
||||||
|
if (!isset($meta['db'])) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
$manifest = json_decode($repo->getObject($meta['db']), true, 512, JSON_THROW_ON_ERROR);
|
$manifest = json_decode($repo->getObject($meta['db']), true, 512, JSON_THROW_ON_ERROR);
|
||||||
$prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_');
|
$prefix = $this->config->getSystemValueString('dbtableprefix', 'oc_');
|
||||||
// ng_backup's own tables (and its own rows in the shared appconfig table) must
|
// ng_backup's own tables (and its own rows in the shared appconfig table) must
|
||||||
@@ -167,38 +269,35 @@ final class RestoreFullService {
|
|||||||
$dumper = new DbDumper($this->db->getInner(), $prefix);
|
$dumper = new DbDumper($this->db->getInner(), $prefix);
|
||||||
try {
|
try {
|
||||||
$dumper->restore($repo, $manifest, $dbReport, $beat, $skipMissingTables);
|
$dumper->restore($repo, $manifest, $dbReport, $beat, $skipMissingTables);
|
||||||
} finally {
|
} catch (\Throwable $e) {
|
||||||
|
try {
|
||||||
$this->restoreOwnAppConfig($ownAppConfig);
|
$this->restoreOwnAppConfig($ownAppConfig);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
// rolled back, so they are still there: the error that brought us here is the one to report
|
||||||
|
}
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$this->restoreOwnAppConfig($ownAppConfig);
|
||||||
|
} catch (\Throwable $e) {
|
||||||
|
// The old server's database is in place by now: not a case for "the database stays as
|
||||||
|
// the fresh installation made it" under --salvage. A run again restores these rows.
|
||||||
|
throw new DatabaseAlreadyRestoredException('The database was restored, but NG Backup could not put back its own settings: ' . $e->getMessage(), 0, $e);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Read (but do not yet apply) the backed-up config.php before touching the data
|
/**
|
||||||
// directory: once applied, 'secret' may differ from what ng_backup's own Target
|
* @return array<string, mixed> config.php's $CONFIG from the backup, minus the connection keys
|
||||||
// options were encrypted with (see restoreFull()'s docblock), and every further
|
* @throws \RuntimeException with $salvage, when config.php itself cannot be read
|
||||||
// repository access through $this->targets would fail to decrypt them.
|
*/
|
||||||
/** @var array<string, mixed> $backupConfig */
|
private function fetchBackupConfig(Target $target, string $snapshotId, bool $salvage = false): array {
|
||||||
$backupConfig = $this->fetchBackupConfig($target, $snapshotId);
|
|
||||||
|
|
||||||
$dataDir = rtrim($this->config->getSystemValueString('datadirectory'), '/');
|
|
||||||
$dataFiles = $this->restore->restoreToDirectory($target, $snapshotId, 'data', $dataDir, stripPrefix: true, heartbeat: $beat);
|
|
||||||
|
|
||||||
// Last step: nothing after this may need $this->targets->repository($target) again.
|
|
||||||
if ($backupConfig !== []) {
|
|
||||||
$this->targets->reencryptOptionsAround(
|
|
||||||
fn (): mixed => $this->keys->reencryptForSecretRotation(fn (): mixed => $this->config->setSystemValues($backupConfig)),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
return ['dbReport' => $dbReport ?? ['skipped' => [], 'restored' => [], 'created' => [], 'missing' => []], 'dataFiles' => $dataFiles, 'configKeys' => count($backupConfig),
|
|
||||||
'dbConsistent' => (bool)($manifest['consistent'] ?? true), 'missingApps' => $check['missingApps']];
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
/** @return array<string, mixed> config.php's $CONFIG from the backup, minus the connection keys */
|
|
||||||
private function fetchBackupConfig(Target $target, string $snapshotId): array {
|
|
||||||
$tmp = sys_get_temp_dir() . '/ngb-restore-config-' . bin2hex(random_bytes(6));
|
$tmp = sys_get_temp_dir() . '/ngb-restore-config-' . bin2hex(random_bytes(6));
|
||||||
try {
|
try {
|
||||||
$this->restore->restoreToDirectory($target, $snapshotId, 'config', $tmp, stripPrefix: true);
|
$skipped = $salvage ? [] : null;
|
||||||
|
$this->restore->restoreToDirectory($target, $snapshotId, 'config', $tmp, stripPrefix: true, skipped: $skipped);
|
||||||
|
if (isset($skipped['config/config.php'])) {
|
||||||
|
throw new \RuntimeException('config.php cannot be read: ' . $skipped['config/config.php']);
|
||||||
|
}
|
||||||
$path = $tmp . '/config.php';
|
$path = $tmp . '/config.php';
|
||||||
if (!is_file($path)) {
|
if (!is_file($path)) {
|
||||||
return []; // nothing to merge; the live config.php stays as occ maintenance:install made it
|
return []; // nothing to merge; the live config.php stays as occ maintenance:install made it
|
||||||
|
|||||||
@@ -33,6 +33,9 @@ final class RestoreService {
|
|||||||
public const MODE_MERGE = 'merge';
|
public const MODE_MERGE = 'merge';
|
||||||
public const MODE_REPLACE = 'replace';
|
public const MODE_REPLACE = 'replace';
|
||||||
|
|
||||||
|
/** @var array<string, true> folders of the current restore already cleared of stale temporaries */
|
||||||
|
private array $cleaned = [];
|
||||||
|
|
||||||
public function __construct(
|
public function __construct(
|
||||||
private TargetService $targets,
|
private TargetService $targets,
|
||||||
private IRootFolder $root,
|
private IRootFolder $root,
|
||||||
@@ -58,7 +61,8 @@ final class RestoreService {
|
|||||||
$prefix = trim($path, '/');
|
$prefix = trim($path, '/');
|
||||||
$prefix = $prefix === '' ? '' : $prefix . '/';
|
$prefix = $prefix === '' ? '' : $prefix . '/';
|
||||||
$children = [];
|
$children = [];
|
||||||
foreach ($this->targets->repository($target)->entries($snapshotId) as $e) {
|
$repo = $this->targets->repository($target);
|
||||||
|
foreach ($repo->entries($snapshotId) as $e) {
|
||||||
if ($prefix !== '' && !str_starts_with($e['p'], $prefix)) {
|
if ($prefix !== '' && !str_starts_with($e['p'], $prefix)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -72,6 +76,12 @@ final class RestoreService {
|
|||||||
}
|
}
|
||||||
$children[$name] = $c;
|
$children[$name] = $c;
|
||||||
}
|
}
|
||||||
|
foreach ($repo->emptyDirs($snapshotId) as [$dir]) { // read along with the entries
|
||||||
|
if ($prefix === '' || str_starts_with($dir, $prefix)) {
|
||||||
|
$name = explode('/', substr($dir, strlen($prefix)), 2)[0];
|
||||||
|
$children[$name] ??= ['name' => $name, 'type' => 'dir', 'size' => 0, 'mtime' => null, 'files' => 0];
|
||||||
|
}
|
||||||
|
}
|
||||||
ksort($children, SORT_STRING);
|
ksort($children, SORT_STRING);
|
||||||
return array_values($children);
|
return array_values($children);
|
||||||
}
|
}
|
||||||
@@ -81,8 +91,8 @@ final class RestoreService {
|
|||||||
*
|
*
|
||||||
* @return array{restored:int, trashed:int, target:string}
|
* @return array{restored:int, trashed:int, target:string}
|
||||||
*/
|
*/
|
||||||
public function restoreUserFiles(Target $target, string $snapshotId, string $logicalPath, string $mode = self::MODE_NEW_FOLDER, ?callable $progress = null): array {
|
public function restoreUserFiles(Target $target, string $snapshotId, string $logicalPath, string $mode = self::MODE_NEW_FOLDER, ?callable $progress = null, ?array &$skipped = null): array {
|
||||||
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $logicalPath, $mode, $progress) {
|
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $logicalPath, $mode, $progress, &$skipped) {
|
||||||
$last = time();
|
$last = time();
|
||||||
// Keep the lease alive while files are being written.
|
// Keep the lease alive while files are being written.
|
||||||
$beat = function (int $done, int $total, string $name) use ($refresh, $progress, &$last): void {
|
$beat = function (int $done, int $total, string $name) use ($refresh, $progress, &$last): void {
|
||||||
@@ -94,11 +104,12 @@ final class RestoreService {
|
|||||||
$progress($done, $total, $name);
|
$progress($done, $total, $name);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
return $this->doRestoreUserFiles($target, $snapshotId, $logicalPath, $mode, $beat);
|
return $this->doRestoreUserFiles($target, $snapshotId, $logicalPath, $mode, $beat, $skipped);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
private function doRestoreUserFiles(Target $target, string $snapshotId, string $logicalPath, string $mode, ?callable $progress): array {
|
/** @param array<string, string>|null $skipped see Repository::restore() */
|
||||||
|
private function doRestoreUserFiles(Target $target, string $snapshotId, string $logicalPath, string $mode, ?callable $progress, ?array &$skipped): array {
|
||||||
if (!Repository::isSafePath($logicalPath) || !preg_match('#^data/([^/]+)/files(?:/(.*))?$#', $logicalPath, $m)) {
|
if (!Repository::isSafePath($logicalPath) || !preg_match('#^data/([^/]+)/files(?:/(.*))?$#', $logicalPath, $m)) {
|
||||||
throw new \InvalidArgumentException('Expected a path like data/<user>/files/<folder or file>');
|
throw new \InvalidArgumentException('Expected a path like data/<user>/files/<folder or file>');
|
||||||
}
|
}
|
||||||
@@ -119,7 +130,17 @@ final class RestoreService {
|
|||||||
$entries[substr($e['p'], strlen($source) + 1)] = $e;
|
$entries[substr($e['p'], strlen($source) + 1)] = $e;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if ($entries === []) {
|
// Folders without anything in them, relative to $source ('' when $source itself is one).
|
||||||
|
/** @var array<array-key, int> $emptyDirs modification times; a folder called "2026" is an integer key */
|
||||||
|
$emptyDirs = [];
|
||||||
|
foreach ($single ? [] : $repo->emptyDirs($snapshotId) as [$dir, $mtime]) {
|
||||||
|
if ($dir === $source) {
|
||||||
|
$emptyDirs[''] = $mtime;
|
||||||
|
} elseif (str_starts_with($dir, $source . '/')) {
|
||||||
|
$emptyDirs[substr($dir, strlen($source) + 1)] = $mtime;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ($entries === [] && $emptyDirs === []) {
|
||||||
throw new NotFoundException("Nothing at $source in snapshot $snapshotId");
|
throw new NotFoundException("Nothing at $source in snapshot $snapshotId");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -130,21 +151,62 @@ final class RestoreService {
|
|||||||
} elseif ($mode !== self::MODE_MERGE && $mode !== self::MODE_REPLACE) {
|
} elseif ($mode !== self::MODE_MERGE && $mode !== self::MODE_REPLACE) {
|
||||||
throw new \InvalidArgumentException("Unknown mode $mode");
|
throw new \InvalidArgumentException("Unknown mode $mode");
|
||||||
}
|
}
|
||||||
|
$destExisted = $parentRel === '' || $userFolder->nodeExists($parentRel);
|
||||||
$dest = $this->ensureFolder($userFolder, $parentRel);
|
$dest = $this->ensureFolder($userFolder, $parentRel);
|
||||||
$names = $single ? [basename($rel) => $entries['']] : $entries;
|
$names = $single ? [basename($rel) => $entries['']] : $entries;
|
||||||
|
|
||||||
|
// The folders with something in them that this restore creates get their time back; one that
|
||||||
|
// was already there keeps its own, as its content changes now. Noted before anything is written.
|
||||||
|
/** @var array<array-key, int> $newFolders a folder called "2026" is an integer key */
|
||||||
|
$newFolders = [];
|
||||||
|
foreach ($single ? [] : $repo->folderTimes($snapshotId) as [$dir, $mtime]) {
|
||||||
|
$relative = $dir === $source ? '' : (str_starts_with($dir, $source . '/') ? substr($dir, strlen($source) + 1) : null);
|
||||||
|
if ($relative !== null && ($relative === '' ? !$destExisted : !$dest->nodeExists($relative))) {
|
||||||
|
$newFolders[$relative] = $mtime;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
$this->cleaned = [];
|
||||||
$restored = 0;
|
$restored = 0;
|
||||||
foreach ($names as $relPath => $entry) {
|
foreach ($names as $relPath => $entry) {
|
||||||
|
try {
|
||||||
$this->writeFile($repo, $dest, (string)$relPath, $entry);
|
$this->writeFile($repo, $dest, (string)$relPath, $entry);
|
||||||
|
} catch (\OCA\NgBackup\Repository\RepositoryException|\OCA\NgBackup\Backend\BackendException|\OCA\NgBackup\Crypto\CryptoException $e) {
|
||||||
|
// Damaged or missing on the location (writeFile removed its temporary)
|
||||||
|
if ($skipped === null) {
|
||||||
|
throw $e;
|
||||||
|
}
|
||||||
|
$skipped[$entry['p']] = $e->getMessage();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
$restored++;
|
$restored++;
|
||||||
if ($progress !== null) {
|
if ($progress !== null) {
|
||||||
$progress($restored, count($names), (string)$relPath);
|
$progress($restored, count($names), (string)$relPath);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
foreach ($emptyDirs as $dir => $mtime) {
|
||||||
|
$dir = (string)$dir;
|
||||||
|
$existed = $dir === '' ? $destExisted : $dest->nodeExists($dir); // one that is there keeps its own time
|
||||||
|
$folder = $this->ensureFolder($dest, $dir);
|
||||||
|
if (!$existed) {
|
||||||
|
$folder->touch($mtime);
|
||||||
|
}
|
||||||
|
if ($progress !== null) {
|
||||||
|
$progress($restored, count($names), $dir); // keeps the lease alive too
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
$trashed = 0;
|
$trashed = 0;
|
||||||
if ($mode === self::MODE_REPLACE && !$single) {
|
if ($mode === self::MODE_REPLACE && !$single) {
|
||||||
$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)));
|
$trashed = $this->trashExtras($dest, '', array_flip(array_keys($names)) + $emptyDirs);
|
||||||
|
}
|
||||||
|
// Last and deepest first: a change in a folder also touches the folders above it.
|
||||||
|
foreach (array_reverse($newFolders, true) as $dir => $mtime) {
|
||||||
|
$this->ensureFolder($dest, (string)$dir)->touch($mtime);
|
||||||
|
if ($progress !== null) {
|
||||||
|
$progress($restored, count($names), (string)$dir); // keeps the lease alive too
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())];
|
return ['restored' => $restored, 'trashed' => $trashed, 'target' => $uid . ':' . $userFolder->getRelativePath($dest->getPath())];
|
||||||
}
|
}
|
||||||
@@ -155,15 +217,18 @@ final class RestoreService {
|
|||||||
* @param bool $stripPrefix write into $directory itself instead of $directory/$prefix (see
|
* @param bool $stripPrefix write into $directory itself instead of $directory/$prefix (see
|
||||||
* Repository::restore()); used to restore "data/" or "config/" straight into the
|
* Repository::restore()); used to restore "data/" or "config/" straight into the
|
||||||
* real data/config directory for occ backup:restore:full.
|
* real data/config directory for occ backup:restore:full.
|
||||||
|
* @param bool $keepUnchanged leave files already in place with the snapshot's size and
|
||||||
|
* modification time (see Repository::restore()): a full restore that runs again
|
||||||
|
* goes on where it stopped
|
||||||
*/
|
*/
|
||||||
public function restoreToDirectory(Target $target, string $snapshotId, string $prefix, string $directory, bool $stripPrefix = false, ?callable $heartbeat = null): int {
|
public function restoreToDirectory(Target $target, string $snapshotId, string $prefix, string $directory, bool $stripPrefix = false, ?callable $heartbeat = null, ?array &$skipped = null, bool $keepUnchanged = false): int {
|
||||||
if ($prefix !== '' && !Repository::isSafePath($prefix)) {
|
if ($prefix !== '' && !Repository::isSafePath($prefix)) {
|
||||||
throw new \InvalidArgumentException('Invalid path: ' . $prefix);
|
throw new \InvalidArgumentException('Invalid path: ' . $prefix);
|
||||||
}
|
}
|
||||||
if (!is_dir($directory) && !mkdir($directory, 0750, true) && !is_dir($directory)) {
|
if (!is_dir($directory) && !mkdir($directory, 0750, true) && !is_dir($directory)) {
|
||||||
throw new \RuntimeException("Cannot create $directory");
|
throw new \OCA\NgBackup\Repository\RestoreTargetException("Cannot create $directory (disk full, or no permission?)");
|
||||||
}
|
}
|
||||||
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $prefix, $directory, $stripPrefix, $heartbeat): int {
|
return $this->withSharedLock($target, function (callable $refresh) use ($target, $snapshotId, $prefix, $directory, $stripPrefix, $heartbeat, &$skipped, $keepUnchanged): int {
|
||||||
$own = LeaseService::throttled($refresh);
|
$own = LeaseService::throttled($refresh);
|
||||||
$beat = static function () use ($own, $heartbeat): void {
|
$beat = static function () use ($own, $heartbeat): void {
|
||||||
$own();
|
$own();
|
||||||
@@ -171,13 +236,17 @@ final class RestoreService {
|
|||||||
$heartbeat();
|
$heartbeat();
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
return $this->targets->repository($target)->restore($snapshotId, $directory, trim($prefix, '/') === '' ? '' : trim($prefix, '/') . '/', $stripPrefix, $beat);
|
return $this->targets->repository($target)->restore($snapshotId, $directory, trim($prefix, '/') === '' ? '' : trim($prefix, '/') . '/', $stripPrefix, $beat, $skipped, $keepUnchanged);
|
||||||
}, 3600);
|
}, 3600);
|
||||||
}
|
}
|
||||||
|
|
||||||
private function writeFile(Repository $repo, Folder $dest, string $relPath, array $entry): void {
|
private function writeFile(Repository $repo, Folder $dest, string $relPath, array $entry): void {
|
||||||
$dir = dirname($relPath);
|
$dir = dirname($relPath);
|
||||||
$folder = $dir === '.' ? $dest : $this->ensureFolder($dest, $dir);
|
$folder = $dir === '.' ? $dest : $this->ensureFolder($dest, $dir);
|
||||||
|
if (!isset($this->cleaned[$folder->getPath()])) {
|
||||||
|
$this->removeStaleTemporaries($folder);
|
||||||
|
$this->cleaned[$folder->getPath()] = true;
|
||||||
|
}
|
||||||
$name = basename($relPath);
|
$name = basename($relPath);
|
||||||
// Stage under a temporary name; only rename into place after every blob was verified.
|
// Stage under a temporary name; only rename into place after every blob was verified.
|
||||||
$tmpName = '.' . $name . '.ngb-restore-' . bin2hex(random_bytes(3));
|
$tmpName = '.' . $name . '.ngb-restore-' . bin2hex(random_bytes(3));
|
||||||
@@ -195,7 +264,7 @@ final class RestoreService {
|
|||||||
throw new \RuntimeException('Write failed for ' . $relPath);
|
throw new \RuntimeException('Write failed for ' . $relPath);
|
||||||
}
|
}
|
||||||
if ($written !== $entry['s']) {
|
if ($written !== $entry['s']) {
|
||||||
throw new \RuntimeException("Size mismatch restoring $relPath");
|
throw new \OCA\NgBackup\Repository\RepositoryException("Size mismatch restoring $relPath");
|
||||||
}
|
}
|
||||||
if ($folder->nodeExists($name)) {
|
if ($folder->nodeExists($name)) {
|
||||||
$existing = $folder->get($name);
|
$existing = $folder->get($name);
|
||||||
@@ -223,12 +292,25 @@ final class RestoreService {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** See Repository::removeStaleTemporaries(): what a restore that was stopped left in this folder. */
|
||||||
|
private function removeStaleTemporaries(Folder $folder): void {
|
||||||
|
foreach ($folder->getDirectoryListing() as $node) {
|
||||||
|
if ($node instanceof File && Repository::isRestoreTemporary($node->getName())) {
|
||||||
|
try {
|
||||||
|
$node->delete();
|
||||||
|
} catch (\Throwable) {
|
||||||
|
// in the way of nothing: the new temporary gets a name of its own
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private function trashExtras(Folder $folder, string $prefix, array $keep): int {
|
private function trashExtras(Folder $folder, string $prefix, array $keep): int {
|
||||||
$count = 0;
|
$count = 0;
|
||||||
foreach ($folder->getDirectoryListing() as $node) {
|
foreach ($folder->getDirectoryListing() as $node) {
|
||||||
$rel = $prefix === '' ? $node->getName() : $prefix . '/' . $node->getName();
|
$rel = $prefix === '' ? $node->getName() : $prefix . '/' . $node->getName();
|
||||||
if ($node instanceof Folder) {
|
if ($node instanceof Folder) {
|
||||||
$hasKept = false;
|
$hasKept = isset($keep[$rel]); // a folder that was empty in the snapshot
|
||||||
foreach (array_keys($keep) as $k) {
|
foreach (array_keys($keep) as $k) {
|
||||||
if (str_starts_with((string)$k, $rel . '/')) {
|
if (str_starts_with((string)$k, $rel . '/')) {
|
||||||
$hasKept = true;
|
$hasKept = true;
|
||||||
|
|||||||
@@ -28,6 +28,19 @@ final class RetentionPolicy {
|
|||||||
return ['last' => $this->last, 'daily' => $this->daily, 'weekly' => $this->weekly, 'monthly' => $this->monthly];
|
return ['last' => $this->last, 'daily' => $this->daily, 'weekly' => $this->weekly, 'monthly' => $this->monthly];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* keep() for the snapshots that are not pinned, plus the pinned ones. A pinned snapshot does not
|
||||||
|
* take the place of a daily, weekly or monthly one.
|
||||||
|
*
|
||||||
|
* @param array<string, int> $snapshots id => unix time
|
||||||
|
* @param list<string> $pinned
|
||||||
|
* @return array<string, list<string>>
|
||||||
|
*/
|
||||||
|
public function keepWithPinned(array $snapshots, array $pinned): array {
|
||||||
|
$pins = array_intersect_key($snapshots, array_flip($pinned));
|
||||||
|
return $this->keep(array_diff_key($snapshots, $pins)) + array_fill_keys(array_keys($pins), ['pinned']);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @param array<string, int> $snapshots id => unix time
|
* @param array<string, int> $snapshots id => unix time
|
||||||
* @return array<string, list<string>> id => reasons it is kept (ids not in the result are removed)
|
* @return array<string, list<string>> id => reasons it is kept (ids not in the result are removed)
|
||||||
|
|||||||
@@ -13,10 +13,11 @@ use OCP\IConfig;
|
|||||||
use OCP\IGroupManager;
|
use OCP\IGroupManager;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The daily backup time: "HH:MM" in a time zone. Nextcloud runs PHP in UTC, so a time without a
|
* The time zone times are shown in, and the daily time from before each location had its own
|
||||||
* zone meant UTC (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone
|
* schedule (see LocationSchedule). Nextcloud runs PHP in UTC, so a time without a zone meant UTC
|
||||||
* is converted once to the zone of the server or of an administrator, at today's offset: it then
|
* (issue #19: 03:00 ran at 05:00 in Berlin). A time saved before there was a zone is converted once
|
||||||
* shows as it effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
|
* to the zone of the server or of an administrator, at today's offset: it then shows as it
|
||||||
|
* effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer).
|
||||||
*/
|
*/
|
||||||
final class Schedule {
|
final class Schedule {
|
||||||
public const TIME_KEY = 'schedule_time';
|
public const TIME_KEY = 'schedule_time';
|
||||||
@@ -44,27 +45,9 @@ final class Schedule {
|
|||||||
return ['time' => $time, 'timezone' => $zone];
|
return ['time' => $time, 'timezone' => $zone];
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @throws \InvalidArgumentException for a malformed time or an unknown zone */
|
/** @return non-empty-string the zone of a location's schedule, or else this one */
|
||||||
public function set(string $time, string $zone): void {
|
public function zoneOr(?string $zone): string {
|
||||||
if ($time !== '' && !self::validTime($time)) {
|
return $zone !== null && self::validZone($zone) ? $zone : $this->get()['timezone'];
|
||||||
throw new \InvalidArgumentException('Use HH:MM or empty');
|
|
||||||
}
|
|
||||||
if (!self::validZone($zone)) {
|
|
||||||
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
|
|
||||||
}
|
|
||||||
$this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone);
|
|
||||||
$this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */
|
|
||||||
public function todayAt(?int $now = null): ?int {
|
|
||||||
['time' => $time, 'timezone' => $zone] = $this->get();
|
|
||||||
if ($time === '' || !self::validTime($time)) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
$tz = new \DateTimeZone($zone);
|
|
||||||
$day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d');
|
|
||||||
return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -35,6 +35,7 @@ final class TargetService {
|
|||||||
private \OCA\NgBackup\Db\DbIndexCache $indexCache,
|
private \OCA\NgBackup\Db\DbIndexCache $indexCache,
|
||||||
private \OCA\NgBackup\Db\AppConfigCatalogAnchor $anchor,
|
private \OCA\NgBackup\Db\AppConfigCatalogAnchor $anchor,
|
||||||
private SnapshotMapper $snapshots,
|
private SnapshotMapper $snapshots,
|
||||||
|
private AlertService $alerts,
|
||||||
) {
|
) {
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -78,6 +79,11 @@ final class TargetService {
|
|||||||
$target->setBasePath(trim($basePath, '/'));
|
$target->setBasePath(trim($basePath, '/'));
|
||||||
$target->setRepositoryId($repo->id());
|
$target->setRepositoryId($repo->id());
|
||||||
$target->setCreatedAt(time());
|
$target->setCreatedAt(time());
|
||||||
|
// Like the location before it: a backup scheme is usually one scheme.
|
||||||
|
$first = $this->mapper->findAll()[0] ?? null;
|
||||||
|
$target->setSchedule($first?->getSchedule() ?? '');
|
||||||
|
$target->setScheduleZone($first?->getScheduleZone());
|
||||||
|
$target->setScheduleSince(time());
|
||||||
$target = $this->mapper->insert($target);
|
$target = $this->mapper->insert($target);
|
||||||
if (!$created) {
|
if (!$created) {
|
||||||
// Disaster recovery: occ backup:list reads this local cache, not the repository
|
// Disaster recovery: occ backup:list reads this local cache, not the repository
|
||||||
@@ -208,7 +214,25 @@ final class TargetService {
|
|||||||
return $result;
|
return $result;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Update the local copy of a snapshot's note or pin, if the snapshot is listed here at all. */
|
||||||
|
public function syncSnapshot(Target $target, string $snapshotId, ?string $note = null, ?bool $pinned = null): void {
|
||||||
|
try {
|
||||||
|
$s = $this->snapshots->findOne($target->getId(), $snapshotId);
|
||||||
|
} catch (DoesNotExistException) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if ($note !== null) {
|
||||||
|
$s->setNote($note !== '' ? $note : null);
|
||||||
|
}
|
||||||
|
if ($pinned !== null) {
|
||||||
|
$s->setPinned($pinned);
|
||||||
|
}
|
||||||
|
$this->snapshots->update($s);
|
||||||
|
}
|
||||||
|
|
||||||
private function adoptSnapshots(Target $target, Repository $repo): void {
|
private function adoptSnapshots(Target $target, Repository $repo): void {
|
||||||
|
$notes = $repo->snapshotNotes();
|
||||||
|
$pinned = array_flip($repo->pinnedSnapshots());
|
||||||
foreach ($repo->listSnapshots() as $snapshotId) {
|
foreach ($repo->listSnapshots() as $snapshotId) {
|
||||||
try {
|
try {
|
||||||
$this->snapshots->findOne($target->getId(), $snapshotId);
|
$this->snapshots->findOne($target->getId(), $snapshotId);
|
||||||
@@ -227,6 +251,8 @@ final class TargetService {
|
|||||||
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
|
$s->setCreatedAt((int)strtotime((string)($meta['time'] ?? 'now')));
|
||||||
$s->setFiles((int)($meta['stats']['files'] ?? 0));
|
$s->setFiles((int)($meta['stats']['files'] ?? 0));
|
||||||
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
|
$s->setBytes((int)($meta['stats']['bytes'] ?? 0));
|
||||||
|
$s->setNote($notes[$snapshotId] ?? null);
|
||||||
|
$s->setPinned(isset($pinned[$snapshotId]));
|
||||||
$this->snapshots->insert($s);
|
$this->snapshots->insert($s);
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]);
|
Server::get(\Psr\Log\LoggerInterface::class)->warning('NG Backup: snapshot ' . $snapshotId . ' on ' . $target->getName() . ' not listed: ' . $e->getMessage(), ['exception' => $e]);
|
||||||
@@ -248,6 +274,26 @@ final class TargetService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** Size limit for the data stored on this location (null = none). */
|
/** Size limit for the data stored on this location (null = none). */
|
||||||
|
/**
|
||||||
|
* @param string $schedule see LocationSchedule
|
||||||
|
* @throws \InvalidArgumentException
|
||||||
|
*/
|
||||||
|
public function setSchedule(Target $target, string $schedule, string $zone): Target {
|
||||||
|
if (!Schedule::validZone($zone)) {
|
||||||
|
throw new \InvalidArgumentException('Unknown time zone: ' . $zone);
|
||||||
|
}
|
||||||
|
$new = LocationSchedule::normalize($schedule);
|
||||||
|
$turnedOff = $new === '' && (string)$target->getSchedule() !== '';
|
||||||
|
$target->setSchedule($new);
|
||||||
|
$target->setScheduleZone($zone);
|
||||||
|
$target->setScheduleSince(time());
|
||||||
|
$target = $this->mapper->update($target);
|
||||||
|
if ($turnedOff) {
|
||||||
|
$this->alerts->securityEvent('schedule_off', ['target' => $target->getName()]);
|
||||||
|
}
|
||||||
|
return $target;
|
||||||
|
}
|
||||||
|
|
||||||
public function setMaxBytes(Target $target, ?int $maxBytes): Target {
|
public function setMaxBytes(Target $target, ?int $maxBytes): Target {
|
||||||
$target->setMaxBytes($maxBytes);
|
$target->setMaxBytes($maxBytes);
|
||||||
return $this->mapper->update($target);
|
return $this->mapper->update($target);
|
||||||
|
|||||||
@@ -110,11 +110,6 @@
|
|||||||
<code><![CDATA[ask]]></code>
|
<code><![CDATA[ask]]></code>
|
||||||
</UndefinedInterfaceMethod>
|
</UndefinedInterfaceMethod>
|
||||||
</file>
|
</file>
|
||||||
<file src="lib/Command/Trash.php">
|
|
||||||
<RedundantCastGivenDocblockType>
|
|
||||||
<code><![CDATA[(string)($info['time'] ?? '')]]></code>
|
|
||||||
</RedundantCastGivenDocblockType>
|
|
||||||
</file>
|
|
||||||
<file src="lib/Controller/AdminApiController.php">
|
<file src="lib/Controller/AdminApiController.php">
|
||||||
<InvalidOperand>
|
<InvalidOperand>
|
||||||
<code><![CDATA[$maxGb * 1073741824]]></code>
|
<code><![CDATA[$maxGb * 1073741824]]></code>
|
||||||
@@ -191,6 +186,7 @@
|
|||||||
<code><![CDATA[$this->db]]></code>
|
<code><![CDATA[$this->db]]></code>
|
||||||
<code><![CDATA[$this->db]]></code>
|
<code><![CDATA[$this->db]]></code>
|
||||||
<code><![CDATA[$this->db]]></code>
|
<code><![CDATA[$this->db]]></code>
|
||||||
|
<code><![CDATA[$this->db]]></code>
|
||||||
<code><![CDATA[TransactionIsolationLevel]]></code>
|
<code><![CDATA[TransactionIsolationLevel]]></code>
|
||||||
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
||||||
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
<code><![CDATA[\Doctrine\DBAL\ParameterType]]></code>
|
||||||
@@ -304,6 +300,22 @@
|
|||||||
<code><![CDATA[$t]]></code>
|
<code><![CDATA[$t]]></code>
|
||||||
</UndefinedDocblockClass>
|
</UndefinedDocblockClass>
|
||||||
</file>
|
</file>
|
||||||
|
<file src="lib/Migration/Version000008Date20261107010000.php">
|
||||||
|
<UndefinedDocblockClass>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
</UndefinedDocblockClass>
|
||||||
|
</file>
|
||||||
|
<file src="lib/Migration/Version000007Date20261107000000.php">
|
||||||
|
<UndefinedDocblockClass>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
<code><![CDATA[$t]]></code>
|
||||||
|
</UndefinedDocblockClass>
|
||||||
|
</file>
|
||||||
<file src="lib/Migration/Version000005Date20261002040000.php">
|
<file src="lib/Migration/Version000005Date20261002040000.php">
|
||||||
<UndefinedDocblockClass>
|
<UndefinedDocblockClass>
|
||||||
<code><![CDATA[$t]]></code>
|
<code><![CDATA[$t]]></code>
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 226 KiB After Width: | Height: | Size: 259 KiB |
|
Before Width: | Height: | Size: 254 KiB After Width: | Height: | Size: 268 KiB |
|
Before Width: | Height: | Size: 227 KiB After Width: | Height: | Size: 252 KiB |
|
After Width: | Height: | Size: 189 KiB |
@@ -7,20 +7,41 @@
|
|||||||
<NcSettingsSection :name="t('ng_backup', 'NG Backup')"
|
<NcSettingsSection :name="t('ng_backup', 'NG Backup')"
|
||||||
:description="t('ng_backup', 'Encrypted, incremental backups of this Nextcloud (database, configuration and data). Backup locations are never mounted and are not visible in Files.')">
|
:description="t('ng_backup', 'Encrypted, incremental backups of this Nextcloud (database, configuration and data). Backup locations are never mounted and are not visible in Files.')">
|
||||||
<p class="ngb-feedback">
|
<p class="ngb-feedback">
|
||||||
{{ t('ng_backup', 'NG Backup is in beta. We would like to hear how it works for you, also when everything went fine.') }}
|
{{ t('ng_backup', 'We would like to hear how NG Backup works for you, also when everything went fine.') }}
|
||||||
<a :href="feedbackUrl" target="_blank" rel="noopener noreferrer">{{ t('ng_backup', 'Share your experience ↗') }}</a>
|
<a :href="feedbackUrl" target="_blank" rel="noopener noreferrer">{{ t('ng_backup', 'Share your experience ↗') }}</a>
|
||||||
</p>
|
</p>
|
||||||
<NcLoadingIcon v-if="!status" :size="32" />
|
<NcLoadingIcon v-if="!status" :size="32" />
|
||||||
<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
|
<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
|
||||||
|
<NcNoteCard v-if="status && !status.cron.ok" type="warning">
|
||||||
|
<p>
|
||||||
|
{{ status.cron.stopped
|
||||||
|
? t('ng_backup', 'Background jobs are set to system cron, but cron has not run since {time}.', { time: lastCron })
|
||||||
|
: t('ng_backup', 'Background jobs do not run on a system cron (they are set to "{mode}").', { mode: status.cron.mode === 'webcron' ? 'Webcron' : 'AJAX' }) }}
|
||||||
|
{{ t('ng_backup', 'Backups need a system cron that runs every 5 minutes, also the scheduled ones.') }}
|
||||||
|
<a :href="status.cron.docs" target="_blank" rel="noopener noreferrer">{{ t('ng_backup', 'How to set up cron ↗') }}</a>
|
||||||
|
</p>
|
||||||
|
<p>{{ t('ng_backup', 'Until then, a backup started with "Back up now" only moves on while this page is open.') }}</p>
|
||||||
|
</NcNoteCard>
|
||||||
</NcSettingsSection>
|
</NcSettingsSection>
|
||||||
|
|
||||||
<template v-if="status">
|
<template v-if="status">
|
||||||
<div class="ngb-tabs">
|
<div class="ngb-tabs">
|
||||||
<NcButton :variant="tab === 'backup' ? 'primary' : 'secondary'" @click="tab = 'backup'">{{ t('ng_backup', 'Backup and restore') }}</NcButton>
|
<NcButton :variant="tab === 'backup' ? 'primary' : 'secondary'" @click="tab = 'backup'">{{ t('ng_backup', 'Backup and restore') }}</NcButton>
|
||||||
|
<NcButton v-if="status.key.everConfirmed && status.targets.length" :variant="tab === 'bin' ? 'primary' : 'secondary'" @click="tab = 'bin'">{{ t('ng_backup', 'Bin') }}</NcButton>
|
||||||
<NcButton :variant="tab === 'settings' ? 'primary' : 'secondary'" @click="tab = 'settings'">{{ t('ng_backup', 'Settings') }}</NcButton>
|
<NcButton :variant="tab === 'settings' ? 'primary' : 'secondary'" @click="tab = 'settings'">{{ t('ng_backup', 'Settings') }}</NcButton>
|
||||||
|
<NcButton :variant="tab === 'docs' ? 'primary' : 'secondary'" @click="tab = 'docs'">{{ t('ng_backup', 'Docs') }}</NcButton>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<template v-if="tab === 'settings'">
|
<NcSettingsSection v-if="tab === 'docs'" :name="t('ng_backup', 'Documentation')">
|
||||||
|
<ul class="ngb-docs">
|
||||||
|
<li v-for="doc in docs" :key="doc.url">
|
||||||
|
<a :href="doc.url" target="_blank" rel="noopener noreferrer">{{ doc.name }} ↗</a>
|
||||||
|
<span>{{ doc.description }}</span>
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
</NcSettingsSection>
|
||||||
|
<Bin v-else-if="tab === 'bin' && status.key.everConfirmed && status.targets.length" :status="status" @changed="refresh" />
|
||||||
|
<template v-else-if="tab === 'settings'">
|
||||||
<KeySetup :status="status" @changed="refresh" />
|
<KeySetup :status="status" @changed="refresh" />
|
||||||
<Locations v-if="status.key.everConfirmed" :status="status" @changed="refresh" />
|
<Locations v-if="status.key.everConfirmed" :status="status" @changed="refresh" />
|
||||||
</template>
|
</template>
|
||||||
@@ -37,7 +58,7 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { onBeforeUnmount, onMounted, ref } from 'vue'
|
import { computed, onBeforeUnmount, onMounted, ref } from 'vue'
|
||||||
import { t } from '@nextcloud/l10n'
|
import { t } from '@nextcloud/l10n'
|
||||||
import NcButton from '@nextcloud/vue/components/NcButton'
|
import NcButton from '@nextcloud/vue/components/NcButton'
|
||||||
import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
|
import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
|
||||||
@@ -45,6 +66,7 @@ import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
|||||||
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||||
import api from './api.js'
|
import api from './api.js'
|
||||||
import Backups from './components/Backups.vue'
|
import Backups from './components/Backups.vue'
|
||||||
|
import Bin from './components/Bin.vue'
|
||||||
import KeySetup from './components/KeySetup.vue'
|
import KeySetup from './components/KeySetup.vue'
|
||||||
import Locations from './components/Locations.vue'
|
import Locations from './components/Locations.vue'
|
||||||
import RestoreBrowser from './components/RestoreBrowser.vue'
|
import RestoreBrowser from './components/RestoreBrowser.vue'
|
||||||
@@ -52,6 +74,25 @@ import RestoreBrowser from './components/RestoreBrowser.vue'
|
|||||||
// Opens the experience report form in .github/ISSUE_TEMPLATE
|
// Opens the experience report form in .github/ISSUE_TEMPLATE
|
||||||
const feedbackUrl = 'https://github.com/dillardblom/ng_backup/issues/new?template=experience.yml'
|
const feedbackUrl = 'https://github.com/dillardblom/ng_backup/issues/new?template=experience.yml'
|
||||||
|
|
||||||
|
const repoUrl = 'https://github.com/dillardblom/ng_backup'
|
||||||
|
const docs = [
|
||||||
|
{
|
||||||
|
name: t('ng_backup', 'How-to'),
|
||||||
|
description: t('ng_backup', 'Step by step: from first install to a tested restore.'),
|
||||||
|
url: repoUrl + '/blob/main/docs/HOWTO.md',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: t('ng_backup', 'Disaster recovery'),
|
||||||
|
description: t('ng_backup', 'Your server is gone: get Nextcloud back on a new server from a backup location.'),
|
||||||
|
url: repoUrl + '/blob/main/docs/DISASTER-RECOVERY.md',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: t('ng_backup', 'Roadmap'),
|
||||||
|
description: t('ng_backup', 'What is planned for the next versions.'),
|
||||||
|
url: repoUrl + '#roadmap',
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
const status = ref(null)
|
const status = ref(null)
|
||||||
const allRuns = ref(false) // runs older than the last days too
|
const allRuns = ref(false) // runs older than the last days too
|
||||||
const error = ref('')
|
const error = ref('')
|
||||||
@@ -77,13 +118,27 @@ async function refresh() {
|
|||||||
} catch (e) {
|
} catch (e) {
|
||||||
error.value = e.message
|
error.value = e.message
|
||||||
}
|
}
|
||||||
// Poll while something runs, so progress shows up without reloading.
|
// Poll while something runs, so progress shows up without reloading. Without a working
|
||||||
|
// system cron, each poll also lets a running backup take a short step (see tick()).
|
||||||
clearTimeout(timer)
|
clearTimeout(timer)
|
||||||
if (status.value?.runs.some(r => r.status === 'running')) {
|
if (status.value?.runs.some(r => r.status === 'running')) {
|
||||||
timer = setTimeout(refresh, 4000)
|
timer = setTimeout(tickAndRefresh, 4000)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function tickAndRefresh() {
|
||||||
|
if (status.value && !status.value.cron.ok && status.value.runs.some(r => r.status === 'running' && r.kind === 'full')) {
|
||||||
|
try {
|
||||||
|
await api.tick()
|
||||||
|
} catch (e) {
|
||||||
|
// The next poll tries again; the error of a failed step shows on its run.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await refresh()
|
||||||
|
}
|
||||||
|
|
||||||
|
const lastCron = computed(() => status.value?.cron.lastRun ? new Date(status.value.cron.lastRun * 1000).toLocaleString() : '')
|
||||||
|
|
||||||
onMounted(refresh)
|
onMounted(refresh)
|
||||||
onBeforeUnmount(() => clearTimeout(timer))
|
onBeforeUnmount(() => clearTimeout(timer))
|
||||||
</script>
|
</script>
|
||||||
@@ -102,6 +157,17 @@ onBeforeUnmount(() => clearTimeout(timer))
|
|||||||
color: var(--color-primary-element);
|
color: var(--color-primary-element);
|
||||||
text-decoration: underline;
|
text-decoration: underline;
|
||||||
}
|
}
|
||||||
|
.ngb-docs li {
|
||||||
|
margin-bottom: 8px;
|
||||||
|
}
|
||||||
|
.ngb-docs a {
|
||||||
|
color: var(--color-primary-element);
|
||||||
|
text-decoration: underline;
|
||||||
|
margin-right: 8px;
|
||||||
|
}
|
||||||
|
.ngb-docs span {
|
||||||
|
color: var(--color-text-maxcontrast);
|
||||||
|
}
|
||||||
.ngb-tabs {
|
.ngb-tabs {
|
||||||
display: flex;
|
display: flex;
|
||||||
gap: 8px;
|
gap: 8px;
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ export default {
|
|||||||
addSlot: (passphrase, label) => call('post', '/key/slots', { passphrase, label }),
|
addSlot: (passphrase, label) => call('post', '/key/slots', { passphrase, label }),
|
||||||
replaceSlot: (slot, passphrase, label) => call('put', `/key/slots/${slot}`, { passphrase, label }),
|
replaceSlot: (slot, passphrase, label) => call('put', `/key/slots/${slot}`, { passphrase, label }),
|
||||||
removeSlot: (slot) => call('delete', `/key/slots/${slot}`),
|
removeSlot: (slot) => call('delete', `/key/slots/${slot}`),
|
||||||
trash: (targetId) => call('get', `/targets/${targetId}/trash`),
|
trash: (targetId, after = '') => call('get', `/targets/${targetId}/trash`, { after }),
|
||||||
untrash: (targetId, snapshotId) => call('post', `/targets/${targetId}/trash/${snapshotId}`),
|
untrash: (targetId, snapshotId) => call('post', `/targets/${targetId}/trash/${snapshotId}`),
|
||||||
kitUrl: () => url('/key/kit'),
|
kitUrl: () => url('/key/kit'),
|
||||||
confirmKit: (accepted, phrase, code) => call('post', '/key/confirm', { accepted, phrase, code }),
|
confirmKit: (accepted, phrase, code) => call('post', '/key/confirm', { accepted, phrase, code }),
|
||||||
@@ -33,8 +33,11 @@ export default {
|
|||||||
testTarget: (id) => call('post', `/targets/${id}/test`),
|
testTarget: (id) => call('post', `/targets/${id}/test`),
|
||||||
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
|
setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }),
|
||||||
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
|
startBackup: (targetId, label) => call('post', '/backups', { targetId, label }),
|
||||||
setSchedule: (time, timezone) => call('put', '/schedule', { time, timezone }),
|
tick: () => call('post', '/tick'),
|
||||||
|
setTargetSchedule: (id, schedule, timezone) => call('put', `/targets/${id}/schedule`, { schedule, timezone }),
|
||||||
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
|
snapshots: (targetId) => call('get', `/targets/${targetId}/snapshots`),
|
||||||
|
setSnapshotNote: (targetId, snapshotId, note) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/note`, { note }),
|
||||||
|
setSnapshotPinned: (targetId, snapshotId, pinned) => call('put', `/targets/${targetId}/snapshots/${snapshotId}/pin`, { pinned }),
|
||||||
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
|
browse: (targetId, snapshotId, path) => call('get', `/targets/${targetId}/snapshots/${snapshotId}/browse`, { path }),
|
||||||
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
|
startRestore: (targetId, snapshotId, path, mode) => call('post', '/restores', { targetId, snapshotId, path, mode }),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,31 +5,16 @@
|
|||||||
<template>
|
<template>
|
||||||
<NcSettingsSection :name="t('ng_backup', 'Backups')"
|
<NcSettingsSection :name="t('ng_backup', 'Backups')"
|
||||||
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
|
:description="t('ng_backup', 'Backups run in the background through Nextcloud\'s cron. Unchanged files are not read again; only new data is uploaded.')">
|
||||||
|
<LocationScheduleRow v-for="target in status.targets" :key="target.id" :target="target" @changed="emit('changed')" @error="error = $event" />
|
||||||
<div class="ngb-row">
|
<div class="ngb-row">
|
||||||
<NcTextField v-model="schedule" class="ngb-time" :label="t('ng_backup', 'Daily at (HH:MM, {zone}, empty = off)', { zone: browserZone })" />
|
<NcTextField v-model="label" class="ngb-label" :maxlength="100" :label="t('ng_backup', 'Label (optional)')"
|
||||||
<NcButton :disabled="schedule === status.schedule && browserZone === status.scheduleTimezone" @click="saveSchedule">{{ t('ng_backup', 'Save') }}</NcButton>
|
:placeholder="t('ng_backup', 'e.g. before the upgrade to Nextcloud 35')" />
|
||||||
</div>
|
|
||||||
<div class="ngb-row">
|
|
||||||
<NcButton v-for="target in status.targets" :key="target.id" :disabled="isRunning(target.id)" @click="start(target)">
|
<NcButton v-for="target in status.targets" :key="target.id" :disabled="isRunning(target.id)" @click="start(target)">
|
||||||
{{ isRunning(target.id) ? t('ng_backup', 'Backup to {name} running …', { name: target.name }) : t('ng_backup', 'Back up now to {name}', { name: target.name }) }}
|
{{ isRunning(target.id) ? t('ng_backup', 'Backup to {name} running …', { name: target.name }) : t('ng_backup', 'Back up now to {name}', { name: target.name }) }}
|
||||||
</NcButton>
|
</NcButton>
|
||||||
</div>
|
</div>
|
||||||
<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
|
<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
|
||||||
|
|
||||||
<template v-for="target in status.targets" :key="'trash' + target.id">
|
|
||||||
<div v-if="trash[target.id]?.length" class="ngb-trash">
|
|
||||||
<h3>{{ t('ng_backup', 'Trash of {name}', { name: target.name }) }}</h3>
|
|
||||||
<p class="ngb-muted">{{ t('ng_backup', 'Removed restore points can be brought back until they are deleted permanently.') }}</p>
|
|
||||||
<ul>
|
|
||||||
<li v-for="item in trash[target.id]" :key="item.id">
|
|
||||||
{{ item.time ? formatDateTime(item.time) : item.id }}{{ item.label ? ' – ' + item.label : '' }}
|
|
||||||
<span class="ngb-muted">{{ t('ng_backup', 'removed by {by}, deleted after {date}', { by: item.by, date: formatDate(item.deleteAfter) }) }}</span>
|
|
||||||
<NcButton variant="tertiary" @click="untrash(target, item)">{{ t('ng_backup', 'Restore') }}</NcButton>
|
|
||||||
</li>
|
|
||||||
</ul>
|
|
||||||
</div>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<div v-if="status.runs.length" class="ngb-table-wrap">
|
<div v-if="status.runs.length" class="ngb-table-wrap">
|
||||||
<table class="ngb-table">
|
<table class="ngb-table">
|
||||||
<thead>
|
<thead>
|
||||||
@@ -49,8 +34,8 @@
|
|||||||
<td>{{ formatDateTime(run.startedAt) }}</td>
|
<td>{{ formatDateTime(run.startedAt) }}</td>
|
||||||
<td>{{ targetName(run.targetId) }}</td>
|
<td>{{ targetName(run.targetId) }}</td>
|
||||||
<td>{{ run.kind }}</td>
|
<td>{{ run.kind }}</td>
|
||||||
<td>{{ run.snapshot ? formatDateTime(run.snapshot.createdAt) + (run.snapshot.label ? ' – ' + run.snapshot.label : '') : '' }}</td>
|
<td>{{ run.snapshot ? (run.snapshot.pinned ? '📌 ' : '') + formatDateTime(run.snapshot.createdAt) + (run.snapshot.label ? ' – ' + run.snapshot.label : '') + (run.snapshot.note ? ' – ' + run.snapshot.note : '') : '' }}</td>
|
||||||
<td :class="'ngb-' + run.status">{{ run.status }}{{ run.status === 'running' ? ' (' + run.phase + percent(run) + ')' : '' }}{{ run.error ? ': ' + run.error : '' }}</td>
|
<td :class="'ngb-' + run.status">{{ runStatus(run) }}{{ run.error ? ': ' + run.error : '' }}</td>
|
||||||
<td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
|
<td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
|
||||||
<td>{{ run.stats?.files ? formatSize(run.stats.files.bytes) : '' }}</td>
|
<td>{{ run.stats?.files ? formatSize(run.stats.files.bytes) : '' }}</td>
|
||||||
<td>{{ run.stats?.files ? formatSize(run.stats.files.uploaded) : '' }}</td>
|
<td>{{ run.stats?.files ? formatSize(run.stats.files.uploaded) : '' }}</td>
|
||||||
@@ -72,70 +57,55 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup>
|
<script setup>
|
||||||
import { ref, watch } from 'vue'
|
import { ref } from 'vue'
|
||||||
import { n, t } from '@nextcloud/l10n'
|
import { n, t } from '@nextcloud/l10n'
|
||||||
import NcButton from '@nextcloud/vue/components/NcButton'
|
import NcButton from '@nextcloud/vue/components/NcButton'
|
||||||
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
||||||
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||||
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
||||||
|
import LocationScheduleRow from './LocationScheduleRow.vue'
|
||||||
import api from '../api.js'
|
import api from '../api.js'
|
||||||
import { formatDateTime, formatDate, formatSize, toZone } from '../format.js'
|
import { formatDateTime, formatSize } from '../format.js'
|
||||||
import { confirmPassword } from '@nextcloud/password-confirmation'
|
|
||||||
import { reactive } from 'vue'
|
|
||||||
|
|
||||||
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
|
const props = defineProps({ status: { type: Object, required: true }, allRuns: { type: Boolean, default: false } })
|
||||||
const emit = defineEmits(['changed', 'show-all-runs'])
|
const emit = defineEmits(['changed', 'show-all-runs'])
|
||||||
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
|
|
||||||
// zone is shown in this one, so it reads as the moment it runs here.
|
|
||||||
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
|
|
||||||
const shown = (s) => (s && props.status.scheduleTimezone && props.status.scheduleTimezone !== browserZone) ? toZone(s, props.status.scheduleTimezone, browserZone) : s
|
|
||||||
const schedule = ref(shown(props.status.schedule))
|
|
||||||
const error = ref('')
|
const error = ref('')
|
||||||
watch(() => [props.status.schedule, props.status.scheduleTimezone], ([s]) => { schedule.value = shown(s) })
|
|
||||||
|
|
||||||
const trash = reactive({})
|
const label = ref('')
|
||||||
async function loadTrash() {
|
|
||||||
for (const target of props.status.targets) {
|
// ": 47/183 tables" in the database phase; " 68%", or " ~68%" when the total is estimated from
|
||||||
try {
|
// Nextcloud's file cache (first backup)
|
||||||
trash[target.id] = await api.trash(target.id)
|
const percent = (run) => {
|
||||||
} catch (e) {
|
if (!run.progress) {
|
||||||
trash[target.id] = []
|
return ''
|
||||||
}
|
}
|
||||||
|
if (run.progress.totalTables) {
|
||||||
|
return ': ' + t('ng_backup', '{done}/{total} tables', { done: run.progress.tables, total: run.progress.totalTables })
|
||||||
}
|
}
|
||||||
|
return ' ' + (run.progress.approximate ? '~' : '') + Math.floor(run.progress.fraction * 100) + '%'
|
||||||
}
|
}
|
||||||
watch(() => props.status.targets.map(tg => tg.id).join(','), loadTrash, { immediate: true })
|
// "running (db: 47/183 tables)", or "waiting for background jobs (…, last step 12 min ago)"
|
||||||
|
const runStatus = (run) => {
|
||||||
async function untrash(target, item) {
|
if (run.status !== 'running') {
|
||||||
error.value = ''
|
return run.status
|
||||||
try {
|
|
||||||
await confirmPassword()
|
|
||||||
await api.untrash(target.id, item.id)
|
|
||||||
await loadTrash()
|
|
||||||
emit('changed')
|
|
||||||
} catch (e) {
|
|
||||||
error.value = e?.message || ''
|
|
||||||
}
|
}
|
||||||
|
if (run.queued) {
|
||||||
|
return t('ng_backup', 'queued, after the backup that is running now')
|
||||||
|
}
|
||||||
|
const detail = run.phase + percent(run)
|
||||||
|
if (run.waitingFor) {
|
||||||
|
return t('ng_backup', 'waiting for background jobs ({detail}, last step {minutes} min ago)', { detail, minutes: Math.floor(run.waitingFor / 60) })
|
||||||
|
}
|
||||||
|
return run.status + ' (' + detail + ')'
|
||||||
}
|
}
|
||||||
|
|
||||||
// " 68%", or " ~68%" when the total is estimated from Nextcloud's file cache (first backup)
|
|
||||||
const percent = (run) => run.progress ? ' ' + (run.progress.approximate ? '~' : '') + Math.floor(run.progress.fraction * 100) + '%' : ''
|
|
||||||
const isRunning = (targetId) => props.status.runs.some(r => r.targetId === targetId && r.kind === 'full' && r.status === 'running')
|
const isRunning = (targetId) => props.status.runs.some(r => r.targetId === targetId && r.kind === 'full' && r.status === 'running')
|
||||||
const targetName = (id) => props.status.targets.find(tg => tg.id === id)?.name ?? '#' + id
|
const targetName = (id) => props.status.targets.find(tg => tg.id === id)?.name ?? '#' + id
|
||||||
|
|
||||||
async function start(target) {
|
async function start(target) {
|
||||||
error.value = ''
|
error.value = ''
|
||||||
try {
|
try {
|
||||||
await api.startBackup(target.id, '')
|
await api.startBackup(target.id, label.value.trim())
|
||||||
emit('changed')
|
label.value = ''
|
||||||
} catch (e) {
|
|
||||||
error.value = e.message
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function saveSchedule() {
|
|
||||||
error.value = ''
|
|
||||||
try {
|
|
||||||
await api.setSchedule(schedule.value.trim(), browserZone)
|
|
||||||
emit('changed')
|
emit('changed')
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
error.value = e.message
|
error.value = e.message
|
||||||
@@ -145,13 +115,11 @@ async function saveSchedule() {
|
|||||||
|
|
||||||
<style scoped>
|
<style scoped>
|
||||||
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
|
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
|
||||||
.ngb-time { max-width: 260px; }
|
.ngb-label { max-width: 360px; }
|
||||||
.ngb-table-wrap { overflow-x: auto; }
|
.ngb-table-wrap { overflow-x: auto; }
|
||||||
.ngb-table { width: 100%; border-collapse: collapse; }
|
.ngb-table { width: 100%; border-collapse: collapse; }
|
||||||
.ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); }
|
.ngb-table th, .ngb-table td { text-align: start; padding: 4px 8px; border-bottom: 1px solid var(--color-border); }
|
||||||
.ngb-failed { color: var(--color-error-text); }
|
.ngb-failed { color: var(--color-error-text); }
|
||||||
.ngb-muted { color: var(--color-text-maxcontrast); }
|
.ngb-muted { color: var(--color-text-maxcontrast); }
|
||||||
.ngb-trash { margin: 12px 0; }
|
|
||||||
.ngb-trash h3 { font-weight: bold; }
|
|
||||||
.ngb-done { color: var(--color-success-text); }
|
.ngb-done { color: var(--color-success-text); }
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -0,0 +1,134 @@
|
|||||||
|
<!--
|
||||||
|
- SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
- SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
-->
|
||||||
|
<template>
|
||||||
|
<NcSettingsSection :name="t('ng_backup', 'Bin')"
|
||||||
|
:description="t('ng_backup', 'Restore points that were removed, most of them by the retention policy. They can be brought back until they are deleted permanently.')">
|
||||||
|
<NcNoteCard v-if="error" type="error">{{ error }}</NcNoteCard>
|
||||||
|
<div v-if="reading" class="ngb-reading" role="status">
|
||||||
|
<template v-if="reading.total === null">
|
||||||
|
<NcLoadingIcon :size="20" />
|
||||||
|
<span>{{ t('ng_backup', 'Connecting to {location} …', { location: reading.name }) }}</span>
|
||||||
|
</template>
|
||||||
|
<template v-else>
|
||||||
|
<span>{{ t('ng_backup', 'Reading removed restore points from {location}: {done} of {total}', { location: reading.name, done: reading.done, total: reading.total }) }}</span>
|
||||||
|
<NcProgressBar :value="reading.total ? Math.round(100 * reading.done / reading.total) : 0" />
|
||||||
|
</template>
|
||||||
|
<p class="ngb-muted">{{ t('ng_backup', 'This can take a while: every removed restore point is read from the location.') }}</p>
|
||||||
|
</div>
|
||||||
|
<NcNoteCard v-else-if="empty && !error" type="info">{{ t('ng_backup', 'The bin is empty.') }}</NcNoteCard>
|
||||||
|
|
||||||
|
<template v-for="target in status.targets" :key="'trash' + target.id">
|
||||||
|
<div v-if="trash[target.id]?.length" class="ngb-trash">
|
||||||
|
<h3>{{ status.targets.length > 1 ? target.name : t('ng_backup', 'Removed restore points') }}</h3>
|
||||||
|
<ul>
|
||||||
|
<li v-for="item in trash[target.id]" :key="item.id">
|
||||||
|
{{ item.time ? formatDateTime(item.time) : item.id }}{{ item.label ? ' – ' + item.label : '' }}
|
||||||
|
<span class="ngb-muted">{{ t('ng_backup', 'removed by {by}, deleted after {date}', { by: item.by, date: formatDate(item.deleteAfter) }) }}</span>
|
||||||
|
<button v-if="item.by === RETENTION"
|
||||||
|
class="ngb-info"
|
||||||
|
:aria-expanded="retentionInfo ? 'true' : 'false'"
|
||||||
|
:aria-label="t('ng_backup', 'What does this mean?')"
|
||||||
|
:title="t('ng_backup', 'What does this mean?')"
|
||||||
|
@click="retentionInfo = !retentionInfo">
|
||||||
|
ⓘ
|
||||||
|
</button>
|
||||||
|
<NcButton variant="tertiary" @click="untrash(target, item)">{{ t('ng_backup', 'Restore') }}</NcButton>
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
<NcNoteCard v-if="retentionInfo && trash[target.id].some(item => item.by === RETENTION)" type="info">
|
||||||
|
{{ n('ng_backup',
|
||||||
|
'The retention policy keeps the newest restore points, and for older days, weeks and months only the last restore point of each. It moves the others to this bin, where they stay for %n day before they are deleted permanently. Until then you can bring one back with "Restore". Nothing was lost by accident.',
|
||||||
|
'The retention policy keeps the newest restore points, and for older days, weeks and months only the last restore point of each. It moves the others to this bin, where they stay for %n days before they are deleted permanently. Until then you can bring one back with "Restore". Nothing was lost by accident.',
|
||||||
|
status.key.deleteDelayDays) }}
|
||||||
|
</NcNoteCard>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
</NcSettingsSection>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup>
|
||||||
|
import { computed, onBeforeUnmount, reactive, ref, watch } from 'vue'
|
||||||
|
import { n, t } from '@nextcloud/l10n'
|
||||||
|
import { confirmPassword } from '@nextcloud/password-confirmation'
|
||||||
|
import NcButton from '@nextcloud/vue/components/NcButton'
|
||||||
|
import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
|
||||||
|
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
||||||
|
import NcProgressBar from '@nextcloud/vue/components/NcProgressBar'
|
||||||
|
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||||
|
import api from '../api.js'
|
||||||
|
import { formatDate, formatDateTime } from '../format.js'
|
||||||
|
|
||||||
|
const props = defineProps({ status: { type: Object, required: true } })
|
||||||
|
const emit = defineEmits(['changed'])
|
||||||
|
|
||||||
|
// "by" of a restore point the weekly cleanup removed (PruneService)
|
||||||
|
const RETENTION = 'retention policy'
|
||||||
|
const retentionInfo = ref(false)
|
||||||
|
const trash = reactive({})
|
||||||
|
const reading = ref({ name: '', total: null, done: 0 }) // the location being read; null when done
|
||||||
|
const error = ref('')
|
||||||
|
const empty = computed(() => props.status.targets.every(target => !trash[target.id]?.length))
|
||||||
|
|
||||||
|
let current = 0 // the newest load; an older one stops (another took over, or the tab was left)
|
||||||
|
onBeforeUnmount(() => { current++ })
|
||||||
|
|
||||||
|
async function loadTarget(target, load) {
|
||||||
|
reading.value = { name: target.name, total: null, done: 0 }
|
||||||
|
const items = []
|
||||||
|
for (;;) {
|
||||||
|
const page = await api.trash(target.id, items.length ? items[items.length - 1].id : '')
|
||||||
|
if (load !== current) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
items.push(...page.items)
|
||||||
|
reading.value = { name: target.name, total: page.total, done: Math.min(items.length, page.total) }
|
||||||
|
if (!page.items.length || items.length >= page.total) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
trash[target.id] = items.sort((a, b) => String(b.time ?? '').localeCompare(String(a.time ?? '')))
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadTrash() {
|
||||||
|
const load = ++current
|
||||||
|
const failed = []
|
||||||
|
for (const target of props.status.targets) {
|
||||||
|
try {
|
||||||
|
await loadTarget(target, load)
|
||||||
|
} catch (e) {
|
||||||
|
// keep what was shown: a location that cannot be reached does not mean its bin is empty
|
||||||
|
failed.push(target.name + (e?.message ? ': ' + e.message : ''))
|
||||||
|
}
|
||||||
|
if (load !== current) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
error.value = failed.length ? t('ng_backup', 'The bin could not be read from {locations}', { locations: failed.join('; ') }) : ''
|
||||||
|
reading.value = null
|
||||||
|
}
|
||||||
|
// Again when a run ends: a cleanup moves restore points here.
|
||||||
|
watch(() => props.status.targets.map(tg => tg.id).join(',') + '|' + (props.status.runs ?? []).filter(run => run.status !== 'running').map(run => run.id).join(','), loadTrash, { immediate: true })
|
||||||
|
|
||||||
|
async function untrash(target, item) {
|
||||||
|
error.value = ''
|
||||||
|
try {
|
||||||
|
await confirmPassword()
|
||||||
|
await api.untrash(target.id, item.id)
|
||||||
|
await loadTrash()
|
||||||
|
emit('changed')
|
||||||
|
} catch (e) {
|
||||||
|
error.value = e?.message || ''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.ngb-info { background: none; border: none; padding: 0 4px; min-height: auto; cursor: pointer; color: var(--color-primary-element); }
|
||||||
|
.ngb-muted { color: var(--color-text-maxcontrast); }
|
||||||
|
.ngb-reading { margin: 12px 0; display: flex; flex-direction: column; gap: 6px; max-width: 600px; }
|
||||||
|
.ngb-reading p { margin: 0; }
|
||||||
|
.ngb-trash { margin: 12px 0; }
|
||||||
|
.ngb-trash h3 { font-weight: bold; }
|
||||||
|
</style>
|
||||||
@@ -12,7 +12,7 @@
|
|||||||
<NcPasswordField v-model="pass2" :label="t('ng_backup', 'Repeat passphrase')" autocomplete="new-password"
|
<NcPasswordField v-model="pass2" :label="t('ng_backup', 'Repeat passphrase')" autocomplete="new-password"
|
||||||
:error="pass2 !== '' && pass1 !== pass2" :helper-text="pass2 !== '' && pass1 !== pass2 ? t('ng_backup', 'The passphrases do not match') : ''" />
|
:error="pass2 !== '' && pass1 !== pass2" :helper-text="pass2 !== '' && pass1 !== pass2 ? t('ng_backup', 'The passphrases do not match') : ''" />
|
||||||
<NcTextField v-model="delay" type="number" min="1" max="365"
|
<NcTextField v-model="delay" type="number" min="1" max="365"
|
||||||
:label="t('ng_backup', 'Days a removed backup stays in the trash before it is really deleted')"
|
:label="t('ng_backup', 'Days a removed backup stays in the Bin before it is really deleted')"
|
||||||
:helper-text="t('ng_backup', 'Protects against a malicious or mistaken cleanup. Cannot be changed later.')" />
|
:helper-text="t('ng_backup', 'Protects against a malicious or mistaken cleanup. Cannot be changed later.')" />
|
||||||
<NcButton type="submit" variant="primary" :disabled="busy || pass1.length < 16 || pass1 !== pass2 || !(delay >= 1 && delay <= 365)">
|
<NcButton type="submit" variant="primary" :disabled="busy || pass1.length < 16 || pass1 !== pass2 || !(delay >= 1 && delay <= 365)">
|
||||||
{{ t('ng_backup', 'Create backup key') }}
|
{{ t('ng_backup', 'Create backup key') }}
|
||||||
@@ -22,7 +22,7 @@
|
|||||||
<template v-else>
|
<template v-else>
|
||||||
<p>
|
<p>
|
||||||
{{ t('ng_backup', 'Key fingerprint: {fingerprint}', { fingerprint: status.key.fingerprint }) }} ·
|
{{ t('ng_backup', 'Key fingerprint: {fingerprint}', { fingerprint: status.key.fingerprint }) }} ·
|
||||||
{{ t('ng_backup', 'Removed backups are kept in the trash for {days} days', { days: status.key.deleteDelayDays }) }}
|
{{ t('ng_backup', 'Removed backups are kept in the Bin for {days} days', { days: status.key.deleteDelayDays }) }}
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<!-- Passphrase slots -->
|
<!-- Passphrase slots -->
|
||||||
|
|||||||
@@ -0,0 +1,77 @@
|
|||||||
|
<!--
|
||||||
|
- SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
- SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
-->
|
||||||
|
<template>
|
||||||
|
<div class="ngb-row">
|
||||||
|
<strong class="ngb-name">{{ target.name }}</strong>
|
||||||
|
<NcSelect v-model="kind" :options="kinds" label="label" :input-label="t('ng_backup', 'Schedule')" :clearable="false" />
|
||||||
|
<NcTextField v-if="kind.id !== 'off'" v-model="time" class="ngb-time" :label="kind.id === 'daily' ? t('ng_backup', 'At (HH:MM, {zone})', { zone: browserZone }) : t('ng_backup', 'From (HH:MM, {zone})', { zone: browserZone })" />
|
||||||
|
<NcTextField v-if="kind.id === 'hours'" v-model="hours" class="ngb-hours" type="number" min="1" max="23" :label="t('ng_backup', 'Every … hours')" />
|
||||||
|
<NcButton :disabled="busy || wanted === current" @click="save">{{ t('ng_backup', 'Save') }}</NcButton>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup>
|
||||||
|
import { computed, ref, watch } from 'vue'
|
||||||
|
import { t } from '@nextcloud/l10n'
|
||||||
|
import NcButton from '@nextcloud/vue/components/NcButton'
|
||||||
|
import NcSelect from '@nextcloud/vue/components/NcSelect'
|
||||||
|
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
||||||
|
import api from '../api.js'
|
||||||
|
import { toZone } from '../format.js'
|
||||||
|
|
||||||
|
const props = defineProps({ target: { type: Object, required: true } })
|
||||||
|
const emit = defineEmits(['changed', 'error'])
|
||||||
|
|
||||||
|
const kinds = [
|
||||||
|
{ id: 'off', label: t('ng_backup', 'Off') },
|
||||||
|
{ id: 'daily', label: t('ng_backup', 'Once a day') },
|
||||||
|
{ id: 'hours', label: t('ng_backup', 'Every few hours') },
|
||||||
|
]
|
||||||
|
// The time is saved in the administrator's own zone (the browser's); a schedule saved in another
|
||||||
|
// zone is shown in this one, so it reads as the moment it runs here.
|
||||||
|
const browserZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC'
|
||||||
|
const kind = ref(kinds[0])
|
||||||
|
const time = ref('02:00')
|
||||||
|
const hours = ref('6')
|
||||||
|
const busy = ref(false)
|
||||||
|
|
||||||
|
const current = computed(() => {
|
||||||
|
const [at, every] = props.target.schedule.split('/')
|
||||||
|
return at === '' ? '' : (props.target.scheduleZone !== browserZone ? toZone(at, props.target.scheduleZone, browserZone) : at) + (every ? '/' + every : '')
|
||||||
|
})
|
||||||
|
const wanted = computed(() => {
|
||||||
|
if (kind.value.id === 'off') {
|
||||||
|
return ''
|
||||||
|
}
|
||||||
|
return time.value.trim() + (kind.value.id === 'hours' ? '/' + String(hours.value).trim() : '')
|
||||||
|
})
|
||||||
|
|
||||||
|
watch(current, (schedule) => {
|
||||||
|
const [at, every] = schedule.split('/')
|
||||||
|
kind.value = kinds[at === '' ? 0 : (every ? 2 : 1)]
|
||||||
|
time.value = at === '' ? time.value : at
|
||||||
|
hours.value = every ?? hours.value
|
||||||
|
}, { immediate: true })
|
||||||
|
|
||||||
|
async function save() {
|
||||||
|
busy.value = true
|
||||||
|
emit('error', '')
|
||||||
|
try {
|
||||||
|
await api.setTargetSchedule(props.target.id, wanted.value, browserZone)
|
||||||
|
emit('changed')
|
||||||
|
} catch (e) {
|
||||||
|
emit('error', e.message)
|
||||||
|
} finally {
|
||||||
|
busy.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.ngb-row { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; margin-bottom: 12px; }
|
||||||
|
.ngb-name { min-width: 120px; align-self: center; }
|
||||||
|
.ngb-time { max-width: 220px; }
|
||||||
|
.ngb-hours { max-width: 120px; }
|
||||||
|
</style>
|
||||||
@@ -11,6 +11,13 @@
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<template v-if="snapshot">
|
<template v-if="snapshot">
|
||||||
|
<div class="ngb-row ngb-marks">
|
||||||
|
<NcTextField v-model="note" class="ngb-note" :maxlength="500" :label="t('ng_backup', 'Note (can be changed later)')" />
|
||||||
|
<NcButton :disabled="marking || note.trim() === (snapshot.note ?? '')" @click="saveNote">{{ t('ng_backup', 'Save note') }}</NcButton>
|
||||||
|
<NcCheckboxRadioSwitch :model-value="snapshot.pinned" :disabled="marking" @update:model-value="savePinned">
|
||||||
|
{{ t('ng_backup', 'Pin: the retention policy does not remove this restore point') }}
|
||||||
|
</NcCheckboxRadioSwitch>
|
||||||
|
</div>
|
||||||
<div class="ngb-crumbs">
|
<div class="ngb-crumbs">
|
||||||
<NcButton variant="tertiary" @click="open('data')">data</NcButton>
|
<NcButton variant="tertiary" @click="open('data')">data</NcButton>
|
||||||
<template v-for="(crumb, i) in crumbs" :key="crumb.path">
|
<template v-for="(crumb, i) in crumbs" :key="crumb.path">
|
||||||
@@ -49,6 +56,7 @@ import NcLoadingIcon from '@nextcloud/vue/components/NcLoadingIcon'
|
|||||||
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
import NcNoteCard from '@nextcloud/vue/components/NcNoteCard'
|
||||||
import NcSelect from '@nextcloud/vue/components/NcSelect'
|
import NcSelect from '@nextcloud/vue/components/NcSelect'
|
||||||
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
import NcSettingsSection from '@nextcloud/vue/components/NcSettingsSection'
|
||||||
|
import NcTextField from '@nextcloud/vue/components/NcTextField'
|
||||||
import { confirmPassword } from '@nextcloud/password-confirmation'
|
import { confirmPassword } from '@nextcloud/password-confirmation'
|
||||||
import '@nextcloud/password-confirmation/style.css'
|
import '@nextcloud/password-confirmation/style.css'
|
||||||
import api from '../api.js'
|
import api from '../api.js'
|
||||||
@@ -74,7 +82,10 @@ const modes = [
|
|||||||
{ id: 'replace', label: t('ng_backup', 'Into the original place, overwrite and move files that are not in the backup to the trash') },
|
{ id: 'replace', label: t('ng_backup', 'Into the original place, overwrite and move files that are not in the backup to the trash') },
|
||||||
]
|
]
|
||||||
|
|
||||||
const snapshotLabel = (s) => formatDateTime(s.createdAt) + (s.label ? ' – ' + s.label : '')
|
const note = ref('')
|
||||||
|
const marking = ref(false)
|
||||||
|
|
||||||
|
const snapshotLabel = (s) => (s.pinned ? '📌 ' : '') + formatDateTime(s.createdAt) + (s.label ? ' – ' + s.label : '') + (s.note ? ' – ' + s.note : '')
|
||||||
const join = (name) => path.value + '/' + name
|
const join = (name) => path.value + '/' + name
|
||||||
const restorable = (p) => /^data\/[^/]+\/files(\/.*)?$/.test(p)
|
const restorable = (p) => /^data\/[^/]+\/files(\/.*)?$/.test(p)
|
||||||
const crumbs = computed(() => {
|
const crumbs = computed(() => {
|
||||||
@@ -88,7 +99,29 @@ watch(target, async (tg) => {
|
|||||||
snapshots.value = list.map(s => ({ ...s, displayLabel: snapshotLabel(s) }))
|
snapshots.value = list.map(s => ({ ...s, displayLabel: snapshotLabel(s) }))
|
||||||
snapshot.value = snapshots.value[0] ?? null
|
snapshot.value = snapshots.value[0] ?? null
|
||||||
}, { immediate: true })
|
}, { immediate: true })
|
||||||
watch(snapshot, () => { selected.value = ''; open('data') })
|
watch(snapshot, (s) => { selected.value = ''; note.value = s?.note ?? ''; open('data') })
|
||||||
|
|
||||||
|
// Changes the snapshot in place: choosing another one, which opens its folders again, is not needed
|
||||||
|
async function mark(call, change) {
|
||||||
|
const s = snapshot.value
|
||||||
|
marking.value = true
|
||||||
|
error.value = message.value = ''
|
||||||
|
try {
|
||||||
|
await call(target.value.id, s.id)
|
||||||
|
change(s)
|
||||||
|
s.displayLabel = snapshotLabel(s)
|
||||||
|
emit('changed')
|
||||||
|
} catch (e) {
|
||||||
|
error.value = e.message
|
||||||
|
} finally {
|
||||||
|
marking.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const saveNote = () => {
|
||||||
|
const text = note.value.trim()
|
||||||
|
return mark((targetId, id) => api.setSnapshotNote(targetId, id, text), (s) => { s.note = text })
|
||||||
|
}
|
||||||
|
const savePinned = (pinned) => mark((targetId, id) => api.setSnapshotPinned(targetId, id, pinned), (s) => { s.pinned = pinned })
|
||||||
|
|
||||||
async function open(p) {
|
async function open(p) {
|
||||||
if (!snapshot.value) {
|
if (!snapshot.value) {
|
||||||
@@ -125,6 +158,8 @@ async function restore() {
|
|||||||
<style scoped>
|
<style scoped>
|
||||||
.ngb-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 12px; }
|
.ngb-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 12px; }
|
||||||
.ngb-row :deep(.ngb-snapshot) { min-width: min(360px, 100%); }
|
.ngb-row :deep(.ngb-snapshot) { min-width: min(360px, 100%); }
|
||||||
|
.ngb-marks { align-items: flex-end; }
|
||||||
|
.ngb-note { max-width: 360px; }
|
||||||
.ngb-crumbs { display: flex; align-items: center; flex-wrap: wrap; }
|
.ngb-crumbs { display: flex; align-items: center; flex-wrap: wrap; }
|
||||||
.ngb-entries li { display: flex; align-items: center; gap: 12px; min-height: 36px; }
|
.ngb-entries li { display: flex; align-items: center; gap: 12px; min-height: 36px; }
|
||||||
.ngb-entries li.selected { background: var(--color-primary-element-light); border-radius: var(--border-radius); }
|
.ngb-entries li.selected { background: var(--color-primary-element-light); border-radius: var(--border-radius); }
|
||||||
|
|||||||
@@ -0,0 +1,35 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Backend;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\ExternalStorageFactory;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** ExternalStorageFactory::normalizePrivateKey(): keys that phpseclib 2 read, for phpseclib 3 (#24). */
|
||||||
|
class ExternalStorageFactoryTest extends TestCase {
|
||||||
|
/** An OpenSSH-format key laid out as ssh-keygen writes it; the body is random, not a real key. */
|
||||||
|
private static function key(): string {
|
||||||
|
$body = base64_encode(random_bytes(1000));
|
||||||
|
return "-----BEGIN OPENSSH PRIVATE KEY-----\n" . chunk_split($body, 70, "\n") . "-----END OPENSSH PRIVATE KEY-----\n";
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testRestoresLostLineBreaks(): void {
|
||||||
|
$key = self::key();
|
||||||
|
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey($key));
|
||||||
|
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", ' ', $key)), 'newlines as spaces');
|
||||||
|
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", '', $key)), 'no newlines');
|
||||||
|
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(str_replace("\n", "\r\n", $key)), 'CRLF');
|
||||||
|
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(rtrim($key)), 'no trailing newline');
|
||||||
|
$this->assertSame($key, ExternalStorageFactory::normalizePrivateKey(preg_replace('/^/m', ' ', $key)), 'indented');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testLeavesOtherFormatsAlone(): void {
|
||||||
|
$pem = "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC,FF44E0A94036766C\n\nAAAA\n-----END RSA PRIVATE KEY-----\n";
|
||||||
|
$this->assertSame($pem, ExternalStorageFactory::normalizePrivateKey($pem));
|
||||||
|
$this->assertSame('not a key', ExternalStorageFactory::normalizePrivateKey('not a key'));
|
||||||
|
$odd = "-----BEGIN OPENSSH PRIVATE KEY-----\\nAAAA\\n-----END OPENSSH PRIVATE KEY-----";
|
||||||
|
$this->assertSame($odd, ExternalStorageFactory::normalizePrivateKey($odd), 'not base64 inside: unchanged');
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Command\TargetAdd;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
class TargetAddEstimateTest extends TestCase {
|
||||||
|
public function testNoEstimateFromTheFirstFewFiles(): void {
|
||||||
|
// 4,568 index files, 10 in after 6 s: too few to say anything
|
||||||
|
$this->assertNull(TargetAdd::minutesLeft(6.0, 10, 4558, 4568));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testEstimateOnceEnoughFilesCameIn(): void {
|
||||||
|
// 300 files in 10 s: 4,268 left at 30 a second is about 2.4 minutes
|
||||||
|
$this->assertSame(3, TargetAdd::minutesLeft(10.0, 300, 4268, 4568));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testFivePercentOfASmallIndexIsEnough(): void {
|
||||||
|
// 6 of 100 files in 6 s: 94 left at one a second is about 1.6 minutes
|
||||||
|
$this->assertSame(2, TargetAdd::minutesLeft(6.0, 6, 94, 100));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNoEstimateInTheFirstSecondsOrWhenDone(): void {
|
||||||
|
$this->assertNull(TargetAdd::minutesLeft(2.0, 500, 4000, 4500));
|
||||||
|
$this->assertNull(TargetAdd::minutesLeft(60.0, 4568, 0, 4568));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -14,4 +14,11 @@ class FormatTest extends TestCase {
|
|||||||
$this->assertSame('1d 00:00:00', Format::duration(86400));
|
$this->assertSame('1d 00:00:00', Format::duration(86400));
|
||||||
$this->assertSame('1d 12:10:13', Format::duration(86400 + 12 * 3600 + 10 * 60 + 13));
|
$this->assertSame('1d 12:10:13', Format::duration(86400 + 12 * 3600 + 10 * 60 + 13));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testTimeInZone(): void {
|
||||||
|
$t = (int)strtotime('2026-10-19 13:12:00 UTC');
|
||||||
|
$this->assertSame('2026-10-19 13:12', Format::time($t, 'UTC'));
|
||||||
|
$this->assertSame('2026-10-19 15:12', Format::time($t, 'Europe/Berlin')); // summer time
|
||||||
|
$this->assertSame('2026-12-01 01:00', Format::time((int)strtotime('2026-12-01 00:00:00 UTC'), 'Europe/Amsterdam'));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -24,4 +24,25 @@ class BackupProgressTest extends TestCase {
|
|||||||
$this->assertNull(BackupRun::progress(['phase' => 'files', 'estimate' => ['bytes' => 0, 'approximate' => true], 'stats' => ['bytes' => 1]]));
|
$this->assertNull(BackupRun::progress(['phase' => 'files', 'estimate' => ['bytes' => 0, 'approximate' => true], 'stats' => ['bytes' => 1]]));
|
||||||
$this->assertNull(BackupRun::progress(['phase' => 'commit', 'estimate' => ['bytes' => 10, 'approximate' => false], 'stats' => ['bytes' => 1]]));
|
$this->assertNull(BackupRun::progress(['phase' => 'commit', 'estimate' => ['bytes' => 10, 'approximate' => false], 'stats' => ['bytes' => 1]]));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static function runIn(string $phase, array $state, string $status = \OCA\NgBackup\Db\Run::RUNNING): \OCA\NgBackup\Db\Run {
|
||||||
|
$run = new \OCA\NgBackup\Db\Run();
|
||||||
|
$run->setStatus($status);
|
||||||
|
$run->setPhase($phase);
|
||||||
|
$run->setState(json_encode($state, JSON_THROW_ON_ERROR));
|
||||||
|
return $run;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testDatabasePhaseCountsTheTablesDumped(): void {
|
||||||
|
$run = self::runIn('db', ['dump' => ['tables' => array_fill(0, 183, 'oc_t'), 'i' => 47]]);
|
||||||
|
$this->assertSame(['fraction' => 47 / 183, 'approximate' => false, 'tables' => 47, 'totalTables' => 183], BackupRun::progressOf($run));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testDatabasePhaseHasNoFigureBeforeItsFirstStep(): void {
|
||||||
|
$this->assertNull(BackupRun::progressOf(self::runIn('db', ['files' => []])));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNoProgressForAFinishedRun(): void {
|
||||||
|
$this->assertNull(BackupRun::progressOf(self::runIn('db', ['dump' => ['tables' => ['oc_a'], 'i' => 0]], \OCA\NgBackup\Db\Run::DONE)));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -28,6 +28,28 @@ class TreeWalkerTest extends TestCase {
|
|||||||
$this->assertSame(['Z', 'a', 'a-b', 'a.txt', 'b', 'skip'], array_values(array_unique(array_map(fn ($p) => explode('/', $p)[0], $paths))));
|
$this->assertSame(['Z', 'a', 'a-b', 'a.txt', 'b', 'skip'], array_values(array_unique(array_map(fn ($p) => explode('/', $p)[0], $paths))));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testEmptyDirectoriesAreYieldedInOrderAndOnResume(): void {
|
||||||
|
$d = $this->tree();
|
||||||
|
mkdir("$d/a/empty");
|
||||||
|
mkdir("$d/a-hollow/inner", 0700, true);
|
||||||
|
symlink("$d/a.txt", "$d/skip/link");
|
||||||
|
mkdir("$d/only-link");
|
||||||
|
symlink("$d/a.txt", "$d/only-link/link");
|
||||||
|
$w = new TreeWalker(['' => $d]);
|
||||||
|
$all = array_keys(iterator_to_array($w->walk()));
|
||||||
|
$this->assertContains('a/empty', $all);
|
||||||
|
$this->assertContains('a-hollow/inner', $all);
|
||||||
|
$this->assertNotContains('a-hollow', $all, 'only the deepest');
|
||||||
|
$this->assertContains('only-link', $all, 'a link is not walked');
|
||||||
|
$this->assertNotContains('skip', $all);
|
||||||
|
$sorted = $all;
|
||||||
|
usort($sorted, [TreeWalker::class, 'compare']);
|
||||||
|
$this->assertSame($sorted, $all);
|
||||||
|
foreach ($all as $i => $cursor) {
|
||||||
|
$this->assertSame(array_slice($all, $i + 1), array_keys(iterator_to_array($w->walk($cursor))), "after $cursor");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public function testResumeAfterCursorYieldsTheRest(): void {
|
public function testResumeAfterCursorYieldsTheRest(): void {
|
||||||
$w = new TreeWalker(['' => $this->tree()]);
|
$w = new TreeWalker(['' => $this->tree()]);
|
||||||
$all = array_keys(iterator_to_array($w->walk()));
|
$all = array_keys(iterator_to_array($w->walk()));
|
||||||
|
|||||||
@@ -50,6 +50,40 @@ class CatalogTest extends TestCase {
|
|||||||
$this->assertSame(4, $again->catalogGeneration());
|
$this->assertSame(4, $again->catalogGeneration());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testTheTrashIsReadInPartsThatTogetherAreAll(): void {
|
||||||
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
|
for ($i = 0; $i < 3; $i++) {
|
||||||
|
$repo->forget($repo->backupDirectory($this->src)['snapshot']);
|
||||||
|
}
|
||||||
|
$again = $this->open();
|
||||||
|
$seen = [];
|
||||||
|
// A deadline already past: one entry per request, as on a slow location
|
||||||
|
$after = '';
|
||||||
|
for ($request = 0; count($seen) < 3 && $request < 5; $request++) {
|
||||||
|
$page = $again->trashPage($after, 0.0);
|
||||||
|
$this->assertSame(3, $page['total']);
|
||||||
|
$this->assertCount(1, $page['items']);
|
||||||
|
$seen += $page['items'];
|
||||||
|
$after = (string)array_key_last($page['items']);
|
||||||
|
}
|
||||||
|
$this->assertEqualsCanonicalizing(array_keys($again->trash()), array_keys($seen));
|
||||||
|
$this->assertSame(['total' => 3, 'items' => []], $again->trashPage($after, 0.0));
|
||||||
|
$this->assertCount(3, $again->trashPage('', microtime(true) + 60)['items']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAnEntryThatLeavesTheTrashBetweenPartsMakesNoneComeTwiceOrBeMissed(): void {
|
||||||
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
|
for ($i = 0; $i < 3; $i++) {
|
||||||
|
$repo->forget($repo->backupDirectory($this->src)['snapshot']);
|
||||||
|
}
|
||||||
|
$reader = $this->open();
|
||||||
|
$ids = array_keys($reader->trashPage('', microtime(true) + 60)['items']);
|
||||||
|
$first = $reader->trashPage('', 0.0)['items'];
|
||||||
|
$this->assertSame([$ids[0]], array_keys($first));
|
||||||
|
$this->open()->untrash($ids[0]); // another administrator, between two parts
|
||||||
|
$this->assertSame([$ids[1], $ids[2]], array_keys($this->open()->trashPage($ids[0], microtime(true) + 60)['items']));
|
||||||
|
}
|
||||||
|
|
||||||
public function testRollbackIsDetected(): void {
|
public function testRollbackIsDetected(): void {
|
||||||
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
$repo->backupDirectory($this->src);
|
$repo->backupDirectory($this->src);
|
||||||
|
|||||||
@@ -0,0 +1,158 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\LocalBackend;
|
||||||
|
use OCA\NgBackup\Crypto\KeyRing;
|
||||||
|
use OCA\NgBackup\Job\BackupRun;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
|
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** Folders without files are recorded in the snapshot's meta record and come back on a restore. */
|
||||||
|
class EmptyDirsTest extends TestCase {
|
||||||
|
use TempDirTrait;
|
||||||
|
|
||||||
|
private LocalBackend $backend;
|
||||||
|
private KeyRing $keys;
|
||||||
|
private string $src;
|
||||||
|
|
||||||
|
protected function setUp(): void {
|
||||||
|
$this->backend = new LocalBackend($this->tempDir());
|
||||||
|
$this->keys = KeyRing::generate();
|
||||||
|
Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
|
||||||
|
$this->src = $this->tempDir();
|
||||||
|
mkdir("$this->src/data/alice/files/Docs", 0700, true);
|
||||||
|
mkdir("$this->src/data/alice/files/Hollow/Nested", 0700, true);
|
||||||
|
mkdir("$this->src/data/skipped/inner", 0700, true);
|
||||||
|
file_put_contents("$this->src/data/alice/files/Docs/note.txt", "hello\n");
|
||||||
|
touch("$this->src/data/alice/files/Hollow/Nested", 1700000000);
|
||||||
|
}
|
||||||
|
|
||||||
|
private function repo(): Repository {
|
||||||
|
return Repository::openWithKey($this->backend, $this->keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @param list<string> $exclude */
|
||||||
|
private function backup(array $exclude = [], float $seconds = 30.0): string {
|
||||||
|
$state = BackupRun::start(['data' => "$this->src/data"], null, '', $exclude);
|
||||||
|
for ($i = 0; $i < 500 && $state['phase'] !== 'commit'; $i++) {
|
||||||
|
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, microtime(true) + $seconds)), true);
|
||||||
|
}
|
||||||
|
$this->assertSame('commit', $state['phase']);
|
||||||
|
$this->repo()->recordSnapshot($state['snapshot']);
|
||||||
|
return $state['snapshot'];
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOnlyTheDeepestEmptyFolderIsRecordedAndFilesStayAsTheyAre(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
|
||||||
|
$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], $paths);
|
||||||
|
$this->assertSame(['data/alice/files/Docs/note.txt'], array_column(iterator_to_array($this->repo()->entries($snapshot), false), 'p'));
|
||||||
|
$this->assertSame(2, $this->repo()->snapshotMeta($snapshot)['stats']['emptyDirs']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAFolderWithOnlyExcludedContentCountsAsEmpty(): void {
|
||||||
|
$snapshot = $this->backup(['data/skipped/inner']);
|
||||||
|
$this->assertContains('data/skipped', array_column($this->repo()->emptyDirs($snapshot), 0));
|
||||||
|
$this->assertNotContains('data/skipped/inner', array_column($this->repo()->emptyDirs($snapshot), 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAResumedWalkRecordsEveryFolderOnce(): void {
|
||||||
|
for ($i = 0; $i < 30; $i++) {
|
||||||
|
mkdir("$this->src/data/many/d$i", 0700, true);
|
||||||
|
file_put_contents("$this->src/data/many/f$i", 'x');
|
||||||
|
}
|
||||||
|
$snapshot = $this->backup([], 0.0); // a step per entry
|
||||||
|
$paths = array_column($this->repo()->emptyDirs($snapshot), 0);
|
||||||
|
$this->assertCount(32, $paths);
|
||||||
|
$this->assertSame(array_unique($paths), $paths);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testANameASnapshotCannotHoldIsReportedAndTheSnapshotStaysReadable(): void {
|
||||||
|
mkdir("$this->src/data/odd\\folder", 0700, true);
|
||||||
|
mkdir("$this->src/data/alice/files/line\nbreak", 0700, true);
|
||||||
|
file_put_contents("$this->src/data/alice/files/back\\slash.txt", 'x');
|
||||||
|
file_put_contents("$this->src/data/alice/files/\xff.txt", 'x');
|
||||||
|
$state = BackupRun::start(['data' => "$this->src/data"]);
|
||||||
|
for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
|
||||||
|
// A step per entry, state through JSON: as separate processes would
|
||||||
|
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
|
||||||
|
}
|
||||||
|
$this->assertSame('commit', $state['phase']);
|
||||||
|
$this->assertSame(4, $state['stats']['badNames'], 'each reported once, over several steps');
|
||||||
|
$this->assertArrayNotHasKey('unreadable', $state['stats'], 'not a problem with permissions');
|
||||||
|
$shown = BackupRun::warnings($state)['badNames'];
|
||||||
|
$this->assertContains('data/alice/files/line?break', $shown, 'no control character in a message');
|
||||||
|
$this->assertSame($shown, array_filter($shown, fn (string $name): bool => mb_check_encoding($name, 'UTF-8')));
|
||||||
|
$this->repo()->recordSnapshot($state['snapshot']);
|
||||||
|
|
||||||
|
$this->assertSame(['data/alice/files/Hollow/Nested', 'data/skipped/inner'], array_column($this->repo()->emptyDirs($state['snapshot']), 0));
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->assertSame(1, $this->repo()->restore($state['snapshot'], $target));
|
||||||
|
$this->assertFileExists("$target/data/alice/files/Docs/note.txt");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAnEmptyFolderThatIsThePrefixItselfIsTheTarget(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target, 'data/skipped/inner/', true);
|
||||||
|
$this->assertSame([], glob("$target/*"), 'not a folder "inner" inside the target');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testRestoreBringsThemBackWithTheirTime(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->assertSame(1, $this->repo()->restore($snapshot, $target));
|
||||||
|
$this->assertDirectoryExists("$target/data/alice/files/Hollow/Nested");
|
||||||
|
clearstatcache();
|
||||||
|
$this->assertSame(1700000000, filemtime("$target/data/alice/files/Hollow/Nested"));
|
||||||
|
$this->assertDirectoryExists("$target/data/skipped/inner");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testRestoreOfAPrefixStripsAndLeavesOthersOut(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target, 'data/alice/', true);
|
||||||
|
$this->assertDirectoryExists("$target/files/Hollow/Nested");
|
||||||
|
$this->assertDirectoryDoesNotExist("$target/skipped");
|
||||||
|
$this->assertDirectoryDoesNotExist("$target/inner");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testRunningAgainLeavesAFolderThatIsThereAlone(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
touch("$target/data/alice/files/Hollow/Nested", 1800000000);
|
||||||
|
file_put_contents("$target/data/alice/files/Hollow/Nested/added", 'kept');
|
||||||
|
touch("$target/data/alice/files/Hollow/Nested", 1800000000);
|
||||||
|
$this->repo()->restore($snapshot, $target, keepUnchanged: true);
|
||||||
|
clearstatcache();
|
||||||
|
$this->assertSame(1800000000, filemtime("$target/data/alice/files/Hollow/Nested"));
|
||||||
|
$this->assertFileExists("$target/data/alice/files/Hollow/Nested/added");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testALinkOutOfTheTargetIsRefused(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$outside = $this->tempDir();
|
||||||
|
mkdir("$target/data/alice/files", 0700, true);
|
||||||
|
symlink($outside, "$target/data/alice/files/Hollow");
|
||||||
|
try {
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
$this->fail('restored through a link');
|
||||||
|
} catch (\OCA\NgBackup\Repository\RestoreTargetException $e) {
|
||||||
|
$this->assertStringContainsString('outside', $e->getMessage());
|
||||||
|
}
|
||||||
|
$this->assertSame([], glob("$outside/*"));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testASnapshotWithoutThemHasNone(): void {
|
||||||
|
$src = $this->tempDir();
|
||||||
|
file_put_contents("$src/a", 'x');
|
||||||
|
$snapshot = $this->repo()->backupDirectory($src)['snapshot'];
|
||||||
|
$this->assertSame([], $this->repo()->emptyDirs($snapshot));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\LocalBackend;
|
||||||
|
use OCA\NgBackup\Crypto\KeyRing;
|
||||||
|
use OCA\NgBackup\Job\BackupRun;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
|
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** Folders with something in them get their original modification time back on a restore. */
|
||||||
|
class FolderTimesTest extends TestCase {
|
||||||
|
use TempDirTrait;
|
||||||
|
|
||||||
|
private const OLD = 1500000000;
|
||||||
|
private const OLDER = 1400000000;
|
||||||
|
|
||||||
|
private LocalBackend $backend;
|
||||||
|
private KeyRing $keys;
|
||||||
|
private string $src;
|
||||||
|
|
||||||
|
protected function setUp(): void {
|
||||||
|
$this->backend = new LocalBackend($this->tempDir());
|
||||||
|
$this->keys = KeyRing::generate();
|
||||||
|
Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'));
|
||||||
|
$this->src = $this->tempDir();
|
||||||
|
mkdir("$this->src/data/alice/files/Old/Deep", 0700, true);
|
||||||
|
mkdir("$this->src/data/alice/files/2026", 0700, true);
|
||||||
|
file_put_contents("$this->src/data/alice/files/Old/Deep/a.txt", 'a');
|
||||||
|
file_put_contents("$this->src/data/alice/files/Old/b.txt", 'b');
|
||||||
|
file_put_contents("$this->src/data/alice/files/2026/c.txt", 'c');
|
||||||
|
touch("$this->src/data/alice/files/Old/Deep", self::OLDER);
|
||||||
|
touch("$this->src/data/alice/files/Old", self::OLD);
|
||||||
|
touch("$this->src/data/alice/files/2026", self::OLD);
|
||||||
|
}
|
||||||
|
|
||||||
|
private function repo(): Repository {
|
||||||
|
return Repository::openWithKey($this->backend, $this->keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
private function backup(float $seconds = 30.0): string {
|
||||||
|
$state = BackupRun::start(['data' => "$this->src/data"]);
|
||||||
|
for ($i = 0; $i < 500 && $state['phase'] !== 'commit'; $i++) {
|
||||||
|
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, microtime(true) + $seconds)), true);
|
||||||
|
}
|
||||||
|
$this->assertSame('commit', $state['phase']);
|
||||||
|
$this->repo()->recordSnapshot($state['snapshot']);
|
||||||
|
return $state['snapshot'];
|
||||||
|
}
|
||||||
|
|
||||||
|
private function time(string $path): int {
|
||||||
|
clearstatcache();
|
||||||
|
return (int)filemtime($path);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testEachFolderIsRecordedOnceAlsoInSmallSteps(): void {
|
||||||
|
$expected = ['data', 'data/alice', 'data/alice/files', 'data/alice/files/2026', 'data/alice/files/Old', 'data/alice/files/Old/Deep'];
|
||||||
|
$this->assertSame($expected, array_column($this->repo()->folderTimes($this->backup()), 0));
|
||||||
|
$this->assertSame($expected, array_column($this->repo()->folderTimes($this->backup(0.0)), 0), 'a step per entry');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNoEmptyFolderIsAmongThem(): void {
|
||||||
|
mkdir("$this->src/data/alice/files/Hollow");
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$this->assertNotContains('data/alice/files/Hollow', array_column($this->repo()->folderTimes($snapshot), 0));
|
||||||
|
$this->assertContains('data/alice/files/Hollow', array_column($this->repo()->emptyDirs($snapshot), 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testARestoreBringsThemBack(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/Old"));
|
||||||
|
$this->assertSame(self::OLDER, $this->time("$target/data/alice/files/Old/Deep"));
|
||||||
|
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/2026"));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testARestoreThatGoesOnAfterAStopToo(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
touch("$target/data/alice/files/Old", 1800000000);
|
||||||
|
$this->repo()->restore($snapshot, $target, keepUnchanged: true);
|
||||||
|
$this->assertSame(self::OLD, $this->time("$target/data/alice/files/Old"));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAPrefixLeavesTheTargetAndOthersAlone(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($snapshot, $target, 'data/alice/files/Old/', true);
|
||||||
|
$this->assertSame(self::OLDER, $this->time("$target/Deep"));
|
||||||
|
$this->assertNotSame(self::OLD, $this->time($target), 'the directory that was chosen is not ours to change');
|
||||||
|
$this->assertDirectoryDoesNotExist("$target/2026");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testALinkOutOfTheTargetIsRefused(): void {
|
||||||
|
$snapshot = $this->backup();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$outside = $this->tempDir();
|
||||||
|
touch($outside, 1800000000);
|
||||||
|
mkdir("$target/data/alice/files", 0700, true);
|
||||||
|
symlink($outside, "$target/data/alice/files/Old");
|
||||||
|
$this->expectException(\OCA\NgBackup\Repository\RestoreTargetException::class);
|
||||||
|
$this->repo()->restore($snapshot, $target);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAFolderWithABadNameIsLeftOutAndWhatIsInItReported(): void {
|
||||||
|
mkdir("$this->src/data/odd\\folder");
|
||||||
|
file_put_contents("$this->src/data/odd\\folder/x", 'x');
|
||||||
|
$state = BackupRun::start(['data' => "$this->src/data"]);
|
||||||
|
for ($i = 0; $i < 50 && $state['phase'] !== 'commit'; $i++) {
|
||||||
|
$state = json_decode(json_encode(BackupRun::step($this->repo(), $state, 0.0), JSON_THROW_ON_ERROR), true, 512, JSON_THROW_ON_ERROR);
|
||||||
|
}
|
||||||
|
$this->assertSame('commit', $state['phase']);
|
||||||
|
$this->assertSame(1, $state['stats']['badNames']);
|
||||||
|
$this->repo()->recordSnapshot($state['snapshot']);
|
||||||
|
$this->assertNotContains("data/odd\\folder", array_column($this->repo()->folderTimes($state['snapshot']), 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testASnapshotWithoutThemHasNone(): void {
|
||||||
|
$snapshot = $this->repo()->backupDirectory($this->src)['snapshot'];
|
||||||
|
$this->assertSame([], $this->repo()->folderTimes($snapshot));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -75,11 +75,14 @@ class ReadAheadTest extends TestCase {
|
|||||||
$this->assertSame(2, $backend->snapshotReads, 'the restore itself read the snapshot again');
|
$this->assertSame(2, $backend->snapshotReads, 'the restore itself read the snapshot again');
|
||||||
}
|
}
|
||||||
|
|
||||||
public function testTheEmptySpoolOfAStoppedRestoreIsRemoved(): void {
|
public function testTheSpoolOfAStoppedRestoreIsRemoved(): void {
|
||||||
$target = $this->tempDir();
|
$target = $this->tempDir();
|
||||||
mkdir("$target/.ngb-restore-spool-0badc0de", 0700); // a restore that was killed
|
mkdir("$target/.ngb-restore-spool-0badc0de", 0700); // a restore that was killed
|
||||||
|
mkdir("$target/.ngb-restore-spool-8badf00d", 0700); // ... while it wrote a range
|
||||||
|
file_put_contents("$target/.ngb-restore-spool-8badf00d/smb-01234567", str_repeat('x', 4096));
|
||||||
|
|
||||||
$this->assertRestored($this->restore(new ReadAheadBackend(new LocalBackend($this->root)), '', $target));
|
$this->assertRestored($this->restore(new ReadAheadBackend(new LocalBackend($this->root)), '', $target));
|
||||||
|
$this->assertSame([], glob("$target/.ngb-restore-spool-*"));
|
||||||
}
|
}
|
||||||
|
|
||||||
public function testARangeThatCouldNotBeReadAheadIsReadTheUsualWay(): void {
|
public function testARangeThatCouldNotBeReadAheadIsReadTheUsualWay(): void {
|
||||||
|
|||||||
@@ -87,6 +87,74 @@ class RepositoryTest extends TestCase {
|
|||||||
$this->assertSame(10, $repo->restore($r['snapshot'], $target, 'docs/d1/'));
|
$this->assertSame(10, $repo->restore($r['snapshot'], $target, 'docs/d1/'));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testRestoreOfOneFile(): void {
|
||||||
|
$repo = Repository::init($this->backend, 'pw');
|
||||||
|
$r = $repo->backupDirectory($this->src);
|
||||||
|
$target = $this->tempDir();
|
||||||
|
// occ backup:restore --to-directory passes the path with a "/" appended
|
||||||
|
$this->assertSame(1, $repo->restore($r['snapshot'], $target, 'docs/d1/note-1.txt/'));
|
||||||
|
$this->assertSame(file_get_contents($this->src . '/docs/d1/note-1.txt'), file_get_contents($target . '/docs/d1/note-1.txt'));
|
||||||
|
$this->assertSame(0, $repo->restore($r['snapshot'], $this->tempDir(), 'docs/d1/note-1/'), 'a prefix of a name is not that file');
|
||||||
|
|
||||||
|
$stripped = $this->tempDir();
|
||||||
|
$this->assertSame(1, $repo->restore($r['snapshot'], $stripped, 'docs/d1/note-1.txt/', true));
|
||||||
|
$this->assertFileExists($stripped . '/note-1.txt');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testARestoreThatRunsAgainKeepsWhatIsInPlaceAndRestoresTheRest(): void {
|
||||||
|
$location = $this->tempDir();
|
||||||
|
$backend = new LocalBackend($location);
|
||||||
|
$repo = Repository::init($backend, 'pw');
|
||||||
|
$r = $repo->backupDirectory($this->src);
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$repo->restore($r['snapshot'], $target);
|
||||||
|
|
||||||
|
// as a stopped restore leaves it: one file not there yet, one with other content
|
||||||
|
unlink($target . '/docs/d1/note-1.txt');
|
||||||
|
file_put_contents($target . '/docs/d0/note-0.txt', 'something else');
|
||||||
|
$this->assertSame(34, $repo->restore($r['snapshot'], $target, keepUnchanged: true));
|
||||||
|
$this->assertSame(self::hashTree($this->src), self::hashTree($target));
|
||||||
|
|
||||||
|
// all in place: nothing is read from the location, which could not give it anyway now
|
||||||
|
self::removeTree($location . '/packs');
|
||||||
|
$this->assertSame(34, Repository::open($backend, 'pw')->restore($r['snapshot'], $target, keepUnchanged: true));
|
||||||
|
$this->expectException(\Throwable::class);
|
||||||
|
Repository::open($backend, 'pw')->restore($r['snapshot'], $target);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static function removeTree(string $dir): void {
|
||||||
|
foreach (new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator($dir, \FilesystemIterator::SKIP_DOTS), \RecursiveIteratorIterator::CHILD_FIRST) as $f) {
|
||||||
|
$f->isDir() ? rmdir($f->getPathname()) : unlink($f->getPathname());
|
||||||
|
}
|
||||||
|
rmdir($dir);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testRestoreRemovesTemporariesOfAStoppedRestore(): void {
|
||||||
|
$repo = Repository::init($this->backend, 'pw');
|
||||||
|
$r = $repo->backupDirectory($this->src);
|
||||||
|
$target = $this->tempDir();
|
||||||
|
mkdir($target . '/docs/d1', 0700, true);
|
||||||
|
mkdir($target . '/elsewhere', 0700, true);
|
||||||
|
$stale = [
|
||||||
|
$target . '/docs/d1/note-1.txt.ngb-restore-0123456789ab', // from a restore to a directory
|
||||||
|
$target . '/docs/d1/.note-4.txt.ngb-restore-a1b2c3', // from a restore through Nextcloud
|
||||||
|
];
|
||||||
|
$kept = [
|
||||||
|
$target . '/docs/d1/notes.ngb-restore-draft.txt', // not a temporary of ours
|
||||||
|
$target . '/elsewhere/x.ngb-restore-0123456789ab', // not a directory this restore writes to
|
||||||
|
];
|
||||||
|
foreach ([...$stale, ...$kept] as $f) {
|
||||||
|
file_put_contents($f, 'partial');
|
||||||
|
}
|
||||||
|
$repo->restore($r['snapshot'], $target, 'docs/');
|
||||||
|
foreach ($stale as $f) {
|
||||||
|
$this->assertFileDoesNotExist($f);
|
||||||
|
}
|
||||||
|
foreach ($kept as $f) {
|
||||||
|
$this->assertFileExists($f);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public function testWrongPassphrase(): void {
|
public function testWrongPassphrase(): void {
|
||||||
Repository::init($this->backend, 'pw');
|
Repository::init($this->backend, 'pw');
|
||||||
$this->expectException(CryptoException::class);
|
$this->expectException(CryptoException::class);
|
||||||
|
|||||||
@@ -0,0 +1,149 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\LocalBackend;
|
||||||
|
use OCA\NgBackup\Command\Restore;
|
||||||
|
use OCA\NgBackup\Crypto\KeyRing;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
|
use OCA\NgBackup\Repository\RestoreTargetException;
|
||||||
|
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** A stream that takes at most 3 bytes per write. */
|
||||||
|
class PartialWriteStream {
|
||||||
|
public static string $written = '';
|
||||||
|
/** @var resource|null set by PHP */
|
||||||
|
public $context;
|
||||||
|
|
||||||
|
public function stream_open(string $path, string $mode, int $options, ?string &$opened): bool {
|
||||||
|
self::$written = '';
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function stream_write(string $data): int {
|
||||||
|
$part = substr($data, 0, 3);
|
||||||
|
self::$written .= $part;
|
||||||
|
return strlen($part);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What goes wrong where a restore writes to is not damage in the backup: it stops the restore, also
|
||||||
|
* with --salvage, and the message does not point at --salvage.
|
||||||
|
*/
|
||||||
|
class RestoreTargetTest extends TestCase {
|
||||||
|
use TempDirTrait;
|
||||||
|
|
||||||
|
private LocalBackend $backend;
|
||||||
|
private KeyRing $keys;
|
||||||
|
private string $snapshot;
|
||||||
|
|
||||||
|
protected function setUp(): void {
|
||||||
|
$this->backend = new LocalBackend($this->tempDir());
|
||||||
|
$this->keys = KeyRing::generate();
|
||||||
|
$src = $this->tempDir();
|
||||||
|
mkdir("$src/dir");
|
||||||
|
file_put_contents("$src/dir/a.bin", random_bytes(6 * 1048576));
|
||||||
|
file_put_contents("$src/dir/b.bin", random_bytes(6 * 1048576));
|
||||||
|
file_put_contents("$src/dir/c.txt", 'small');
|
||||||
|
$this->snapshot = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'))->backupDirectory($src)['snapshot'];
|
||||||
|
}
|
||||||
|
|
||||||
|
private function repo(): Repository {
|
||||||
|
return Repository::openWithKey($this->backend, $this->keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAShortWriteIsATargetError(): void {
|
||||||
|
if (!is_writable('/dev/full')) {
|
||||||
|
$this->markTestSkipped('needs /dev/full');
|
||||||
|
}
|
||||||
|
$out = fopen('/dev/full', 'wb');
|
||||||
|
stream_set_write_buffer($out, 0);
|
||||||
|
$this->expectException(RestoreTargetException::class);
|
||||||
|
Repository::writeAll($out, 'data', 'dir/a.bin');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAWriteThatTakesPartOfTheDataGoesOn(): void {
|
||||||
|
stream_wrapper_register('ngbpartial', PartialWriteStream::class);
|
||||||
|
try {
|
||||||
|
$out = fopen('ngbpartial://x', 'wb');
|
||||||
|
Repository::writeAll($out, 'abcdefgh', 'dir/a.bin');
|
||||||
|
fclose($out);
|
||||||
|
$this->assertSame('abcdefgh', PartialWriteStream::$written);
|
||||||
|
} finally {
|
||||||
|
stream_wrapper_unregister('ngbpartial');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testATargetErrorDoesNotLookLikeDamage(): void {
|
||||||
|
$this->assertFalse(Restore::backupLooksDamaged(new RestoreTargetException('Cannot write')));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A real full disk: run with NGB_TEST_SMALL_DISK set to an empty, writable tmpfs of a few MiB
|
||||||
|
* (docker run --tmpfs /small:size=8m,uid=...), with and without --salvage.
|
||||||
|
*
|
||||||
|
* @dataProvider salvageOrNot
|
||||||
|
*/
|
||||||
|
public function testAFullDiskStopsTheRestoreAndLeavesNoTemporary(bool $salvage): void {
|
||||||
|
$disk = getenv('NGB_TEST_SMALL_DISK');
|
||||||
|
if ($disk === false || $disk === '') {
|
||||||
|
$this->markTestSkipped('needs NGB_TEST_SMALL_DISK');
|
||||||
|
}
|
||||||
|
$target = $disk . '/' . bin2hex(random_bytes(4));
|
||||||
|
$skipped = $salvage ? [] : null;
|
||||||
|
try {
|
||||||
|
$this->repo()->restore($this->snapshot, $target, skipped: $skipped);
|
||||||
|
$this->fail('12 MiB fitted on the small disk');
|
||||||
|
} catch (RestoreTargetException $e) {
|
||||||
|
$this->assertStringContainsString('disk full', $e->getMessage());
|
||||||
|
} finally {
|
||||||
|
$left = glob("$target/dir/*.ngb-restore-*");
|
||||||
|
foreach (glob("$target/dir/*") ?: [] as $f) {
|
||||||
|
unlink($f);
|
||||||
|
}
|
||||||
|
@rmdir("$target/dir");
|
||||||
|
@rmdir($target);
|
||||||
|
}
|
||||||
|
$this->assertSame([], $left);
|
||||||
|
if ($salvage) {
|
||||||
|
$this->assertSame([], $skipped, 'nothing is listed as damaged');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @return array<string, array{bool}> */
|
||||||
|
public static function salvageOrNot(): array {
|
||||||
|
return ['without --salvage' => [false], 'with --salvage' => [true]];
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testResumeChecksTheFolderBeforeSkipping(): void {
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($this->snapshot, $target);
|
||||||
|
$outside = $this->tempDir();
|
||||||
|
rename("$target/dir", "$outside/dir");
|
||||||
|
symlink("$outside/dir", "$target/dir");
|
||||||
|
$this->expectException(RestoreTargetException::class);
|
||||||
|
$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testResumeRemovesTemporariesWhereEverythingIsInPlace(): void {
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($this->snapshot, $target);
|
||||||
|
file_put_contents("$target/dir/a.bin.ngb-restore-0123456789ab", 'left by a stopped restore');
|
||||||
|
$this->repo()->restore($this->snapshot, $target, keepUnchanged: true);
|
||||||
|
$this->assertFileDoesNotExist("$target/dir/a.bin.ngb-restore-0123456789ab");
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testHeartbeatRunsForFilesThatAreSkipped(): void {
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$this->repo()->restore($this->snapshot, $target);
|
||||||
|
$beats = 0;
|
||||||
|
$this->repo()->restore($this->snapshot, $target, '', false, function () use (&$beats): void {
|
||||||
|
$beats++;
|
||||||
|
}, keepUnchanged: true);
|
||||||
|
$this->assertSame(3, $beats);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\LocalBackend;
|
||||||
|
use OCA\NgBackup\Crypto\KeyRing;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
|
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* --salvage: a file whose data is damaged on the location is left out and listed, the rest of the
|
||||||
|
* restore goes on. Without it, the restore stops at that file.
|
||||||
|
*/
|
||||||
|
class SalvageTest extends TestCase {
|
||||||
|
use TempDirTrait;
|
||||||
|
|
||||||
|
private string $root;
|
||||||
|
private KeyRing $keys;
|
||||||
|
private string $src;
|
||||||
|
private string $snapshot;
|
||||||
|
|
||||||
|
protected function setUp(): void {
|
||||||
|
$this->root = $this->tempDir();
|
||||||
|
$this->keys = KeyRing::generate();
|
||||||
|
$this->src = $this->tempDir();
|
||||||
|
mkdir("$this->src/docs");
|
||||||
|
// Incompressible, 2 MiB each: every file is a blob of its own in one pack
|
||||||
|
foreach (['a', 'b', 'c'] as $name) {
|
||||||
|
file_put_contents("$this->src/docs/$name.bin", random_bytes(2 * 1048576));
|
||||||
|
}
|
||||||
|
file_put_contents("$this->src/note.txt", "small\n");
|
||||||
|
$repo = Repository::initWithKey(new LocalBackend($this->root), $this->keys, $this->keys->wrap('pw'));
|
||||||
|
$this->snapshot = $repo->backupDirectory($this->src)['snapshot'];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Flip bytes in the middle of the (only) pack: inside one of the 2 MiB files. */
|
||||||
|
private function damagePack(): void {
|
||||||
|
$packs = glob($this->root . '/packs/*/*');
|
||||||
|
$this->assertCount(1, $packs);
|
||||||
|
$fh = fopen($packs[0], 'r+b');
|
||||||
|
$middle = intdiv(filesize($packs[0]), 2);
|
||||||
|
fseek($fh, $middle);
|
||||||
|
$bytes = fread($fh, 16);
|
||||||
|
fseek($fh, $middle);
|
||||||
|
fwrite($fh, ~$bytes);
|
||||||
|
fclose($fh);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testSalvageLeavesOutTheDamagedFileAndRestoresTheRest(): void {
|
||||||
|
$this->damagePack();
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$skipped = [];
|
||||||
|
|
||||||
|
$restored = Repository::openWithKey(new LocalBackend($this->root), $this->keys)
|
||||||
|
->restore($this->snapshot, $target, '', false, null, $skipped);
|
||||||
|
|
||||||
|
$this->assertSame(3, $restored);
|
||||||
|
$this->assertCount(1, $skipped);
|
||||||
|
$damaged = array_key_first($skipped);
|
||||||
|
$this->assertMatchesRegularExpression('#^docs/[abc]\.bin$#', $damaged);
|
||||||
|
$this->assertFileDoesNotExist("$target/$damaged");
|
||||||
|
foreach (['docs/a.bin', 'docs/b.bin', 'docs/c.bin', 'note.txt'] as $file) {
|
||||||
|
if ($file !== $damaged) {
|
||||||
|
$this->assertFileEquals("$this->src/$file", "$target/$file");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$this->assertSame([], glob("$target/docs/*.ngb-restore-*"), 'no temporary left of the damaged file');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testWithoutSalvageTheRestoreStopsAtTheDamagedFile(): void {
|
||||||
|
$this->damagePack();
|
||||||
|
|
||||||
|
$this->expectException(\Exception::class);
|
||||||
|
Repository::openWithKey(new LocalBackend($this->root), $this->keys)->restore($this->snapshot, $this->tempDir());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testSalvageOfAnUndamagedSnapshotSkipsNothing(): void {
|
||||||
|
$target = $this->tempDir();
|
||||||
|
$skipped = [];
|
||||||
|
|
||||||
|
$this->assertSame(4, Repository::openWithKey(new LocalBackend($this->root), $this->keys)
|
||||||
|
->restore($this->snapshot, $target, '', false, null, $skipped));
|
||||||
|
$this->assertSame([], $skipped);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Repository;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Backend\LocalBackend;
|
||||||
|
use OCA\NgBackup\Crypto\KeyRing;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
|
use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** A pin and a note on a snapshot live in the catalog on the location, so a new server sees them too. */
|
||||||
|
class SnapshotMarksTest extends TestCase {
|
||||||
|
use TempDirTrait;
|
||||||
|
|
||||||
|
private LocalBackend $backend;
|
||||||
|
private KeyRing $keys;
|
||||||
|
private string $snapshot;
|
||||||
|
|
||||||
|
protected function setUp(): void {
|
||||||
|
$this->backend = new LocalBackend($this->tempDir());
|
||||||
|
$this->keys = KeyRing::generate();
|
||||||
|
$src = $this->tempDir();
|
||||||
|
file_put_contents("$src/a", 'x');
|
||||||
|
$this->snapshot = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'))->backupDirectory($src)['snapshot'];
|
||||||
|
}
|
||||||
|
|
||||||
|
private function repo(): Repository {
|
||||||
|
return Repository::openWithKey($this->backend, $this->keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testTheyAreThereAfterOpeningAgain(): void {
|
||||||
|
$this->repo()->setPinned($this->snapshot, true);
|
||||||
|
$this->repo()->setSnapshotNote($this->snapshot, 'before the upgrade');
|
||||||
|
$this->assertSame([$this->snapshot], $this->repo()->pinnedSnapshots());
|
||||||
|
$this->assertSame([$this->snapshot => 'before the upgrade'], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testTheyCanBeTakenOffAgain(): void {
|
||||||
|
$this->repo()->setPinned($this->snapshot, true);
|
||||||
|
$this->repo()->setSnapshotNote($this->snapshot, 'x');
|
||||||
|
$this->repo()->setPinned($this->snapshot, false);
|
||||||
|
$this->repo()->setSnapshotNote($this->snapshot, '');
|
||||||
|
$this->assertSame([], $this->repo()->pinnedSnapshots());
|
||||||
|
$this->assertSame([], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNoChangeWritesNoCatalog(): void {
|
||||||
|
$repo = $this->repo();
|
||||||
|
$repo->setPinned($this->snapshot, true);
|
||||||
|
$generation = $repo->catalogGeneration();
|
||||||
|
$repo->setPinned($this->snapshot, true);
|
||||||
|
$repo->setSnapshotNote($this->snapshot, '');
|
||||||
|
$this->assertSame($generation, $repo->catalogGeneration());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testASnapshotThatIsNotThereIsRefused(): void {
|
||||||
|
$this->expectException(\InvalidArgumentException::class);
|
||||||
|
$this->repo()->setPinned('0123456789abcdef0123456789abcdef', true);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testATrashedSnapshotLosesItsPinAndKeepsItsNote(): void {
|
||||||
|
$repo = $this->repo();
|
||||||
|
$repo->setPinned($this->snapshot, true);
|
||||||
|
$repo->setSnapshotNote($this->snapshot, 'kept');
|
||||||
|
$repo->forget($this->snapshot);
|
||||||
|
$this->assertSame([], $this->repo()->pinnedSnapshots());
|
||||||
|
$this->assertSame([$this->snapshot => 'kept'], $this->repo()->snapshotNotes());
|
||||||
|
|
||||||
|
$repo = $this->repo();
|
||||||
|
$repo->untrash($this->snapshot);
|
||||||
|
$this->assertSame([], $this->repo()->pinnedSnapshots(), 'back in the list, as a normal one');
|
||||||
|
$this->assertSame([$this->snapshot => 'kept'], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testThePurgeTakesTheNoteAlong(): void {
|
||||||
|
$repo = $this->repo();
|
||||||
|
$repo->setSnapshotNote($this->snapshot, 'kept');
|
||||||
|
$repo->forget($this->snapshot);
|
||||||
|
$repo->purgeTrash(0);
|
||||||
|
$this->assertSame([], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testACatalogFromBeforeThisVersionReadsAsNone(): void {
|
||||||
|
$this->assertSame([], $this->repo()->pinnedSnapshots());
|
||||||
|
$this->assertSame([], $this->repo()->snapshotNotes());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -106,4 +106,51 @@ class UserExportTest extends TestCase {
|
|||||||
$this->expectExceptionMessage('missing on the location');
|
$this->expectExceptionMessage('missing on the location');
|
||||||
$repo->listUserExports();
|
$repo->listUserExports();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testResolveTakesARecordedPathAsIs(): void {
|
||||||
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
|
$path = 'users/alice/20261005T120000Z-aaaa0001';
|
||||||
|
$this->putManifest($repo, $path);
|
||||||
|
|
||||||
|
$this->assertSame($path, $repo->resolveUserExport($path));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testResolveTakesAUserIdWithOneExport(): void {
|
||||||
|
// Issue #22: "backup:user:restore <location> New_User" was rejected as not a recorded export.
|
||||||
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
|
$path = 'users/alice/20261005T120000Z-aaaa0001';
|
||||||
|
$this->putManifest($repo, $path);
|
||||||
|
$this->putManifest($repo, 'users/alice2/20261005T120000Z-aaaa0002');
|
||||||
|
|
||||||
|
$this->assertSame($path, $repo->resolveUserExport('alice'));
|
||||||
|
$this->assertSame($path, $repo->resolveUserExport('users/alice/'));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testResolveListsTheExportsOfAUserWithMore(): void {
|
||||||
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
|
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
|
||||||
|
$this->putManifest($repo, 'users/alice/20261006T120000Z-aaaa0002');
|
||||||
|
|
||||||
|
$this->expectException(RepositoryException::class);
|
||||||
|
$this->expectExceptionMessage("alice has 2 exports here; pass one of these (newest first):\n users/alice/20261006T120000Z-aaaa0002\n users/alice/20261005T120000Z-aaaa0001");
|
||||||
|
$repo->resolveUserExport('alice');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testResolveListsTheExportsOfTheUserOfAWrongPath(): void {
|
||||||
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
|
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
|
||||||
|
|
||||||
|
$this->expectException(RepositoryException::class);
|
||||||
|
$this->expectExceptionMessage("users/alice/typo is not a recorded user export for this location; pass one of these (newest first):\n users/alice/20261005T120000Z-aaaa0001");
|
||||||
|
$repo->resolveUserExport('users/alice/typo');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testResolveNamesTheExpectedFormForAUserWithoutExports(): void {
|
||||||
|
$repo = Repository::initWithKey($this->backend, $this->keys, $this->keys->wrap('pw'), $this->anchor);
|
||||||
|
$this->putManifest($repo, 'users/alice/20261005T120000Z-aaaa0001');
|
||||||
|
|
||||||
|
$this->expectException(RepositoryException::class);
|
||||||
|
$this->expectExceptionMessage('bob has no exports here. Expected users/<uid>/<exportId>; occ backup:user:list shows the exports');
|
||||||
|
$repo->resolveUserExport('bob');
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Service;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Db\Run;
|
||||||
|
use OCA\NgBackup\Service\CronHealth;
|
||||||
|
use OCP\Exceptions\AppConfigTypeConflictException;
|
||||||
|
use OCP\IAppConfig;
|
||||||
|
use OCP\IURLGenerator;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
class CronHealthTest extends TestCase {
|
||||||
|
private const NOW = 1_800_000_000;
|
||||||
|
private const DOCS = 'https://docs.example.com/background-jobs';
|
||||||
|
|
||||||
|
private function health(string $mode, ?int $lastCron, bool $lastCronIsInt = false): CronHealth {
|
||||||
|
$config = $this->createMock(IAppConfig::class);
|
||||||
|
$config->method('getValueString')->willReturnCallback(function (string $app, string $key, string $default) use ($mode, $lastCron, $lastCronIsInt): string {
|
||||||
|
if ($key === 'backgroundjobs_mode') {
|
||||||
|
return $mode;
|
||||||
|
}
|
||||||
|
if ($lastCronIsInt) {
|
||||||
|
throw new AppConfigTypeConflictException('stored as int');
|
||||||
|
}
|
||||||
|
return $lastCron === null ? $default : (string)$lastCron;
|
||||||
|
});
|
||||||
|
$config->method('getValueInt')->willReturn($lastCron ?? 0);
|
||||||
|
$urls = $this->createMock(IURLGenerator::class);
|
||||||
|
$urls->method('linkToDocs')->with('admin-background-jobs')->willReturn(self::DOCS);
|
||||||
|
return new CronHealth($config, $urls);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testSystemCronThatRanRecentlyIsFine(): void {
|
||||||
|
$check = $this->health('cron', self::NOW - 120)->check(self::NOW);
|
||||||
|
$this->assertTrue($check['ok']);
|
||||||
|
$this->assertFalse($check['stopped']);
|
||||||
|
$this->assertNull(CronHealth::describe($check, self::NOW));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAjaxAndWebcronAreNotEnough(): void {
|
||||||
|
foreach (['ajax' => 'AJAX', 'webcron' => 'Webcron'] as $mode => $name) {
|
||||||
|
$check = $this->health($mode, self::NOW - 60)->check(self::NOW);
|
||||||
|
$this->assertFalse($check['ok'], $mode);
|
||||||
|
$this->assertFalse($check['stopped'], $mode);
|
||||||
|
$this->assertStringContainsString("they are set to \"$name\" and last ran 1 min ago.", (string)CronHealth::describe($check, self::NOW));
|
||||||
|
$this->assertStringContainsString(self::DOCS, (string)CronHealth::describe($check, self::NOW));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testSystemCronThatStoppedRunning(): void {
|
||||||
|
$check = $this->health('cron', self::NOW - 2 * 3600)->check(self::NOW);
|
||||||
|
$this->assertFalse($check['ok']);
|
||||||
|
$this->assertTrue($check['stopped']);
|
||||||
|
$this->assertStringContainsString('has not run for over an hour (last ran 2 h ago)', (string)CronHealth::describe($check, self::NOW));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testDefaultsToAjaxThatNeverRan(): void {
|
||||||
|
$check = $this->health('ajax', null)->check(self::NOW);
|
||||||
|
$this->assertSame(0, $check['lastRun']);
|
||||||
|
$this->assertStringContainsString('and never ran.', (string)CronHealth::describe($check, self::NOW));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testReadsALastRunStoredAsAnInteger(): void {
|
||||||
|
$check = $this->health('cron', self::NOW - 60, true)->check(self::NOW);
|
||||||
|
$this->assertSame(self::NOW - 60, $check['lastRun']);
|
||||||
|
$this->assertTrue($check['ok']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testARunningBackupWaitsForBackgroundJobsWhenItDidNotMoveForAWhile(): void {
|
||||||
|
$run = new Run();
|
||||||
|
$run->setStatus(Run::RUNNING);
|
||||||
|
$run->setUpdatedAt(self::NOW - 12 * 60);
|
||||||
|
$this->assertNull(CronHealth::waitingFor($run, self::NOW));
|
||||||
|
$run->setUpdatedAt(self::NOW - CronHealth::WAITING_AFTER);
|
||||||
|
$this->assertSame(CronHealth::WAITING_AFTER, CronHealth::waitingFor($run, self::NOW));
|
||||||
|
$run->setStatus(Run::DONE);
|
||||||
|
$this->assertNull(CronHealth::waitingFor($run, self::NOW));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAgo(): void {
|
||||||
|
$this->assertSame('1 min ago', CronHealth::ago(10));
|
||||||
|
$this->assertSame('12 min ago', CronHealth::ago(12 * 60));
|
||||||
|
$this->assertSame('3 h ago', CronHealth::ago(3 * 3600 + 5));
|
||||||
|
$this->assertSame('4 days ago', CronHealth::ago(4 * 86400));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Service;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\LocationSchedule as S;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
class LocationScheduleTest extends TestCase {
|
||||||
|
private const BERLIN = 'Europe/Berlin';
|
||||||
|
|
||||||
|
private static function at(string $time): int {
|
||||||
|
return (int)strtotime($time);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testWhatCanBeSaved(): void {
|
||||||
|
$this->assertSame('', S::normalize('off'));
|
||||||
|
$this->assertSame('', S::normalize(' '));
|
||||||
|
$this->assertSame('02:30', S::normalize('02:30'));
|
||||||
|
$this->assertSame('02:30', S::normalize('02:30/24'));
|
||||||
|
$this->assertSame('00:00/6', S::normalize('00:00/6'));
|
||||||
|
foreach (['2:30', '24:00', '02:30/0', '02:30/25', 'daily', '02:30/x'] as $bad) {
|
||||||
|
try {
|
||||||
|
S::normalize($bad);
|
||||||
|
$this->fail("$bad accepted");
|
||||||
|
} catch (\InvalidArgumentException) {
|
||||||
|
$this->addToAssertionCount(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOffHasNoMoment(): void {
|
||||||
|
$this->assertNull(S::lastMoment('', self::BERLIN, self::at('2026-11-10 12:00 UTC')));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testDailyIsInTheZoneOfTheLocation(): void {
|
||||||
|
$this->assertSame(self::at('2026-11-10 02:00 UTC'), S::lastMoment('03:00', self::BERLIN, self::at('2026-11-10 12:00 UTC')));
|
||||||
|
$this->assertSame(self::at('2026-11-09 02:00 UTC'), S::lastMoment('03:00', self::BERLIN, self::at('2026-11-10 01:00 UTC')), 'not yet today');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testEveryNHoursStartsOverTheNextDay(): void {
|
||||||
|
$zone = 'UTC';
|
||||||
|
$this->assertSame(self::at('2026-11-10 12:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 13:00 UTC')));
|
||||||
|
$this->assertSame(self::at('2026-11-09 22:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 01:59 UTC')), '02, 07, 12, 17, 22, then 02 again');
|
||||||
|
$this->assertSame(self::at('2026-11-10 02:00 UTC'), S::lastMoment('02:00/5', $zone, self::at('2026-11-10 02:00 UTC')), 'at the moment itself');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAClockChangeInTheNightStillGivesOneMoment(): void {
|
||||||
|
$this->assertSame(self::at('2026-03-29 01:30 UTC'), S::lastMoment('02:30', self::BERLIN, self::at('2026-03-29 12:00 UTC')), '02:30 does not exist that night');
|
||||||
|
$this->assertNotNull(S::lastMoment('02:30', self::BERLIN, self::at('2026-10-25 12:00 UTC')));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testChoosingATimeDoesNotStartABackup(): void {
|
||||||
|
$moment = self::at('2026-11-10 02:00 UTC');
|
||||||
|
$this->assertFalse(S::isDue($moment, $moment + 1, null, 0, null, $moment + 3600));
|
||||||
|
$this->assertTrue(S::isDue($moment, $moment - 1, null, 0, null, $moment + 3600));
|
||||||
|
$this->assertTrue(S::isDue($moment, 0, null, 0, null, $moment), 'a location from before schedules per location');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testABackupSinceTheMomentIsEnough(): void {
|
||||||
|
$moment = 1000000;
|
||||||
|
$this->assertFalse(S::isDue($moment, 0, $moment, 1, $moment + 10, $moment + 20));
|
||||||
|
$this->assertTrue(S::isDue($moment, 0, $moment - 1, 0, null, $moment + 20), 'it started before this moment: one more');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAfterAFailureOnlyAFewAttemptsAnHourApart(): void {
|
||||||
|
$moment = 1000000;
|
||||||
|
$finished = $moment + 60;
|
||||||
|
$this->assertFalse(S::isDue($moment, 0, null, 1, $finished, $finished + S::RETRY_AFTER - 1));
|
||||||
|
$this->assertTrue(S::isDue($moment, 0, null, 1, $finished, $finished + S::RETRY_AFTER));
|
||||||
|
$this->assertFalse(S::isDue($moment, 0, null, S::MAX_ATTEMPTS, $finished, $finished + 86400));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -25,6 +25,17 @@ class RetentionPolicyTest extends TestCase {
|
|||||||
$this->assertCount(3, $months);
|
$this->assertCount(3, $months);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testAPinnedSnapshotStaysAndDoesNotCount(): void {
|
||||||
|
$policy = new RetentionPolicy(1, 0, 0, 0);
|
||||||
|
$snaps = ['old' => 100, 'pinned' => 300, 'new' => 200];
|
||||||
|
$this->assertSame(['pinned'], array_keys($policy->keep($snaps)), 'without the pin it would take the place of "last"');
|
||||||
|
$keep = $policy->keepWithPinned($snaps, ['pinned', 'gone-already']);
|
||||||
|
$this->assertSame(['pinned'], $keep['pinned']);
|
||||||
|
$this->assertSame(['last'], $keep['new'], 'the newest one that is not pinned');
|
||||||
|
$this->assertArrayNotHasKey('old', $keep);
|
||||||
|
$this->assertArrayNotHasKey('gone-already', $keep);
|
||||||
|
}
|
||||||
|
|
||||||
public function testKeepsEverythingWhenFewSnapshots(): void {
|
public function testKeepsEverythingWhenFewSnapshots(): void {
|
||||||
$keep = (new RetentionPolicy())->keep(['a' => 100, 'b' => 200]);
|
$keep = (new RetentionPolicy())->keep(['a' => 100, 'b' => 200]);
|
||||||
$this->assertSame(['a', 'b'], array_values(array_intersect(['a', 'b'], array_keys($keep))));
|
$this->assertSame(['a', 'b'], array_values(array_intersect(['a', 'b'], array_keys($keep))));
|
||||||
|
|||||||
@@ -0,0 +1,22 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Tests\Unit\Service;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Service\AlertService;
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
/** A location without a schedule: a weekly reminder while another is backed up, else a daily one. */
|
||||||
|
class ScheduleOffReminderTest extends TestCase {
|
||||||
|
public function testOnceAWeekWhileAnotherLocationIsActive(): void {
|
||||||
|
$this->assertFalse(AlertService::offReminderDue(0, 6 * 86400, true));
|
||||||
|
$this->assertTrue(AlertService::offReminderDue(0, 7 * 86400, true));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testEveryDayWhenNoLocationIsActive(): void {
|
||||||
|
$this->assertFalse(AlertService::offReminderDue(0, 3600, false));
|
||||||
|
$this->assertTrue(AlertService::offReminderDue(0, 86400, false));
|
||||||
|
$this->assertTrue(AlertService::offReminderDue(0, 86400 - 60, false), 'a cron run a minute earlier than yesterday');
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -57,15 +57,4 @@ class ScheduleTest extends TestCase {
|
|||||||
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
|
$this->assertSame(['time' => '02:30', 'timezone' => 'UTC'], $this->schedule()->get());
|
||||||
}
|
}
|
||||||
|
|
||||||
public function testTodayAtIsTheMomentInTheSchedulesZone(): void {
|
|
||||||
$this->values = [Schedule::TIME_KEY => '03:00', Schedule::ZONE_KEY => 'Europe/Berlin'];
|
|
||||||
$at = $this->schedule()->todayAt(strtotime('2026-10-22 12:00 UTC'));
|
|
||||||
$this->assertSame(strtotime('2026-10-22 01:00 UTC'), $at, '03:00 in Berlin (summer time) is 01:00 UTC');
|
|
||||||
}
|
|
||||||
|
|
||||||
public function testAnUnknownZoneOrABadTimeIsRefused(): void {
|
|
||||||
$s = $this->schedule();
|
|
||||||
$this->expectException(\InvalidArgumentException::class);
|
|
||||||
$s->set('03:00', 'Mars/Olympus');
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,13 +2,18 @@
|
|||||||
# Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a
|
# Disaster recovery: occ backup:key:import + backup:target:add + backup:restore:full on a
|
||||||
# SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup
|
# SECOND, fresh Nextcloud instance that never saw the first one, sharing only the backup
|
||||||
# location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7.
|
# location (a local directory, bind-mounted into both containers) -- see PLAN.md 3.7.
|
||||||
# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp|to-pgsql|to-mysql|to-sqlite]
|
# Usage: full-restore-roundtrip.sh <nextcloud-image-tag> [sqlite|pgsql|mysql] [legacy|sftp|salvage|salvage-files|to-pgsql|to-mysql|to-sqlite]
|
||||||
# e.g. full-restore-roundtrip.sh 35-apache pgsql
|
# e.g. full-restore-roundtrip.sh 35-apache pgsql
|
||||||
|
# With "salvage-files" one pack is damaged and another is gone from the location. The restore must
|
||||||
|
# stop and point at --salvage; run again as the guide says, it restores the database and every file
|
||||||
|
# that can be read, and lists the rest (exit 2). NGB_FRT_DATA_MB sets the amount of data (default 200).
|
||||||
# With "sftp" the location is an SFTP server instead of a shared directory, and the origin has a
|
# With "sftp" the location is an SFTP server instead of a shared directory, and the origin has a
|
||||||
# file spread over several packs: the restore reads them ahead on connections of its own.
|
# file spread over several packs: the restore reads them ahead on connections of its own.
|
||||||
# With "to-<db>" the restore instance runs that database whatever the origin runs (a MySQL or
|
# With "to-<db>" the restore instance runs that database whatever the origin runs (a MySQL or
|
||||||
# SQLite backup records no ID counters; after the restore new rows must not collide with restored
|
# SQLite backup records no ID counters; after the restore new rows must not collide with restored
|
||||||
# ones, whatever the database on either side).
|
# ones, whatever the database on either side).
|
||||||
|
# With "salvage" the snapshot's database dump is damaged on the location: the restore must stop
|
||||||
|
# without changing anything, and with --salvage put the files in a folder of their own.
|
||||||
# With "legacy" the origin runs NG Backup 1.0.0-rc.1 (0.9.7), whose backups do not record the
|
# With "legacy" the origin runs NG Backup 1.0.0-rc.1 (0.9.7), whose backups do not record the
|
||||||
# structure of their tables: the restore must then leave the tables of the disabled app out
|
# structure of their tables: the restore must then leave the tables of the disabled app out
|
||||||
# with --skip-missing-tables, as for a real backup made before 1.0.0-rc.2.
|
# with --skip-missing-tables, as for a real backup made before 1.0.0-rc.2.
|
||||||
@@ -34,7 +39,7 @@ cleanup() {
|
|||||||
docker network rm "$NET" >/dev/null 2>&1 || true
|
docker network rm "$NET" >/dev/null 2>&1 || true
|
||||||
docker run --rm -v "$SHARED:/target" alpine sh -c 'rm -rf /target/*' >/dev/null 2>&1 || true
|
docker run --rm -v "$SHARED:/target" alpine sh -c 'rm -rf /target/*' >/dev/null 2>&1 || true
|
||||||
rmdir "$SHARED" 2>/dev/null || true
|
rmdir "$SHARED" 2>/dev/null || true
|
||||||
rm -f /tmp/ngb-frt-kit.json /tmp/ngb-frt-pass
|
rm -f /tmp/ngb-frt-kit.json /tmp/ngb-frt-pass /tmp/ngb-frt-sums
|
||||||
[ -n "${LEGACY_DIR:-}" ] && rm -rf "$LEGACY_DIR"
|
[ -n "${LEGACY_DIR:-}" ] && rm -rf "$LEGACY_DIR"
|
||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
@@ -97,12 +102,23 @@ ok "key, kit and location on the origin"
|
|||||||
|
|
||||||
NC_PASS=$(openssl rand -hex 16)
|
NC_PASS=$(openssl rand -hex 16)
|
||||||
docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
|
docker exec -u www-data -e OC_PASS="$NC_PASS" ngb-frt-origin php occ user:add --password-from-env frtest >/dev/null
|
||||||
docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
|
docker exec ngb-frt-origin sh -c "mkdir -p /var/www/html/data/frtest/files/Docs /var/www/html/data/frtest/files/Empty/Nested && head -c 200000 /dev/urandom > /var/www/html/data/frtest/files/Docs/report.bin && echo 'hello disaster recovery' > /var/www/html/data/frtest/files/Docs/note.txt && chown -R www-data:www-data /var/www/html/data/frtest"
|
||||||
if [ "$MODE" = sftp ]; then
|
if [ "$MODE" = sftp ]; then
|
||||||
# about three packs of 32 MiB
|
# about three packs of 32 MiB
|
||||||
docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
|
docker exec ngb-frt-origin sh -c "head -c 90000000 /dev/urandom > /var/www/html/data/frtest/files/Docs/large.bin && chown www-data:www-data /var/www/html/data/frtest/files/Docs/large.bin"
|
||||||
LARGE_SUM=$(docker exec ngb-frt-origin sha256sum /var/www/html/data/frtest/files/Docs/large.bin | cut -d' ' -f1)
|
LARGE_SUM=$(docker exec ngb-frt-origin sha256sum /var/www/html/data/frtest/files/Docs/large.bin | cut -d' ' -f1)
|
||||||
fi
|
fi
|
||||||
|
if [ "$MODE" = salvage-files ]; then
|
||||||
|
# Half in large files that fill packs of their own, half in files of 1 MiB; incompressible
|
||||||
|
DATA_MB=${NGB_FRT_DATA_MB:-200}
|
||||||
|
docker exec -e DATA_MB="$DATA_MB" ngb-frt-origin sh -c '
|
||||||
|
cd /var/www/html/data/frtest/files && mkdir -p Large Small/a Small/b
|
||||||
|
i=0; while [ $((i * 50)) -lt $((DATA_MB / 2)) ]; do head -c 52428800 /dev/urandom > Large/f$i.bin; i=$((i + 1)); done
|
||||||
|
i=0; while [ $i -lt $((DATA_MB / 2)) ]; do head -c 1048576 /dev/urandom > Small/$([ $((i % 2)) = 0 ] && echo a || echo b)/f$i.bin; i=$((i + 1)); done
|
||||||
|
chown -R www-data:www-data . && find . -type f -print0 | xargs -0 sha256sum > /tmp/frt-sums'
|
||||||
|
docker cp ngb-frt-origin:/tmp/frt-sums /tmp/ngb-frt-sums
|
||||||
|
ok "origin has $DATA_MB MiB of test data, $(wc -l < /tmp/ngb-frt-sums) files"
|
||||||
|
fi
|
||||||
run_on ngb-frt-origin files:scan frtest >/dev/null
|
run_on ngb-frt-origin files:scan frtest >/dev/null
|
||||||
|
|
||||||
# SQL on an instance's own database (pgsql/mysql only)
|
# SQL on an instance's own database (pgsql/mysql only)
|
||||||
@@ -135,6 +151,24 @@ RUN_OUT=$(run_on ngb-frt-origin backup:run offsite -l disaster-test)
|
|||||||
SNAP=$(printf '%s\n' "$RUN_OUT" | head -1 | awk -F'snapshot ' '{print $2}' | tr -d ',')
|
SNAP=$(printf '%s\n' "$RUN_OUT" | head -1 | awk -F'snapshot ' '{print $2}' | tr -d ',')
|
||||||
[ -n "$SNAP" ] || fail "no snapshot id from backup:run"
|
[ -n "$SNAP" ] || fail "no snapshot id from backup:run"
|
||||||
ok "test data backed up, snapshot $SNAP"
|
ok "test data backed up, snapshot $SNAP"
|
||||||
|
if [ "$MODE" = salvage ]; then
|
||||||
|
# The database dump of this snapshot damaged on the location: a byte flipped in its manifest
|
||||||
|
docker exec -u www-data ngb-frt-origin php -r '$p = "/mnt/shared/repo/db/" . $argv[1]; $d = file_get_contents($p); $i = intdiv(strlen($d), 2); $d[$i] = ~$d[$i]; file_put_contents($p, $d);' "$SNAP" \
|
||||||
|
|| fail "damage the database dump"
|
||||||
|
ok "database dump of the snapshot damaged on the location"
|
||||||
|
fi
|
||||||
|
if [ "$MODE" = salvage-files ]; then
|
||||||
|
# The two largest packs hold user files for sure: bytes flipped in one, the other gone
|
||||||
|
docker exec -u www-data ngb-frt-origin php -r '
|
||||||
|
$packs = glob("/mnt/shared/repo/packs/*/*");
|
||||||
|
usort($packs, fn ($a, $b) => filesize($b) <=> filesize($a));
|
||||||
|
count($packs) >= 3 or exit(1);
|
||||||
|
$h = fopen($packs[0], "r+b");
|
||||||
|
for ($o = 4096; $o < filesize($packs[0]); $o += 1048576) { fseek($h, $o); $b = fread($h, 16); fseek($h, $o); fwrite($h, ~$b); }
|
||||||
|
fclose($h);
|
||||||
|
unlink($packs[1]);' || fail "damage the packs"
|
||||||
|
ok "one pack damaged and one pack gone on the location"
|
||||||
|
fi
|
||||||
|
|
||||||
docker cp ngb-frt-origin:/tmp/kit.json /tmp/ngb-frt-kit.json
|
docker cp ngb-frt-origin:/tmp/kit.json /tmp/ngb-frt-kit.json
|
||||||
docker exec ngb-frt-origin sh -c "cat /tmp/pass" > /tmp/ngb-frt-pass
|
docker exec ngb-frt-origin sh -c "cat /tmp/pass" > /tmp/ngb-frt-pass
|
||||||
@@ -164,6 +198,85 @@ printf '%s\n' "$LIST_OUT" | grep -q "$SNAP" || {
|
|||||||
}
|
}
|
||||||
ok "recovery kit imported, existing repository adopted, snapshot visible"
|
ok "recovery kit imported, existing repository adopted, snapshot visible"
|
||||||
|
|
||||||
|
if [ "$MODE" = salvage ]; then
|
||||||
|
set +e
|
||||||
|
OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction 2>&1); CODE=$?
|
||||||
|
set -e
|
||||||
|
[ "$CODE" = 1 ] || { printf '%s\n' "$OUT" | tail -20; fail "a restore with a damaged database dump did not stop (exit $CODE)"; }
|
||||||
|
printf '%s\n' "$OUT" | grep -- "--salvage" >/dev/null || { printf '%s\n' "$OUT" | tail -20; fail "the stopped restore does not point at --salvage"; }
|
||||||
|
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode stayed on after a restore that changed nothing"
|
||||||
|
ok "damaged database dump: the restore stops before changing anything and points at --salvage"
|
||||||
|
set +e
|
||||||
|
OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction --salvage 2>&1); CODE=$?
|
||||||
|
set -e
|
||||||
|
[ "$CODE" = 2 ] || { printf '%s\n' "$OUT" | tail -30; fail "restore:full --salvage exited with $CODE, not 2"; }
|
||||||
|
SALVAGE=/var/www/html/data/ngb-salvage-${SNAP:0:12}
|
||||||
|
printf '%s\n' "$OUT" | grep "$SALVAGE" >/dev/null || { printf '%s\n' "$OUT" | tail -30; fail "the output does not name the salvage folder"; }
|
||||||
|
[ "$(docker exec ngb-frt-restore cat "$SALVAGE/data/frtest/files/Docs/note.txt")" = "hello disaster recovery" ] || fail "the user's file is not in the salvage folder"
|
||||||
|
docker exec ngb-frt-restore test -f "$SALVAGE/config/config.php" || fail "config.php is not in the salvage folder"
|
||||||
|
docker exec ngb-frt-restore test -d "$SALVAGE/data/frtest/files/Empty/Nested" || fail "the empty folder is not in the salvage folder"
|
||||||
|
run_on ngb-frt-restore user:list | grep frtest >/dev/null && fail "the fresh database was changed (the backed-up user is there)"
|
||||||
|
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
|
||||||
|
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
|
||||||
|
ok "with --salvage: the fresh database untouched, data and config of the backup in $SALVAGE (exit 2)"
|
||||||
|
echo "FULL RESTORE ROUNDTRIP PASS ($DB, salvage)"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$MODE" = salvage-files ]; then
|
||||||
|
set +e
|
||||||
|
OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction 2>&1); CODE=$?
|
||||||
|
set -e
|
||||||
|
[ "$CODE" = 1 ] || { printf '%s\n' "$OUT" | tail -20; fail "a restore with damaged data did not stop (exit $CODE)"; }
|
||||||
|
printf '%s\n' "$OUT" | grep -- "--salvage" >/dev/null || { printf '%s\n' "$OUT" | tail -20; fail "the stopped restore does not point at --salvage"; }
|
||||||
|
run_on ngb-frt-restore status 2>&1 | grep "maintenance: true" >/dev/null || fail "maintenance mode is off after a restore that stopped halfway"
|
||||||
|
ok "damaged data: the restore stops, keeps maintenance mode on and points at --salvage"
|
||||||
|
|
||||||
|
# As the guide and the message say: maintenance mode off and again, now with --salvage
|
||||||
|
run_on ngb-frt-restore maintenance:mode --off >/dev/null || fail "maintenance mode off"
|
||||||
|
set +e
|
||||||
|
OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction --salvage --force 2>&1); CODE=$?
|
||||||
|
set -e
|
||||||
|
[ "$CODE" = 2 ] || { printf '%s\n' "$OUT" | tail -30; fail "restore:full --salvage exited with $CODE, not 2"; }
|
||||||
|
printf '%s\n' "$OUT" | grep "could not be restored" >/dev/null || { printf '%s\n' "$OUT" | tail -30; fail "the damaged files are not listed"; }
|
||||||
|
printf '%s\n' "$OUT" | grep "ngb-salvage-" >/dev/null && { printf '%s\n' "$OUT" | tail -30; fail "the database was given up, only files are damaged"; }
|
||||||
|
docker cp /tmp/ngb-frt-sums ngb-frt-restore:/tmp/frt-sums
|
||||||
|
printf '%s\n' "$OUT" | docker exec -i ngb-frt-restore sh -c 'cat > /tmp/frt-out'
|
||||||
|
# Every file: listed and not there, or there and bit-identical
|
||||||
|
CHECK=$(docker exec ngb-frt-restore php -r '
|
||||||
|
$root = "/var/www/html/data/frtest/files/"; $listed = []; $reasons = [];
|
||||||
|
foreach (file("/tmp/frt-out", FILE_IGNORE_NEW_LINES) as $l) {
|
||||||
|
if (preg_match("#^ (?:data/)?frtest/files/(.+?): (.*)$#", $l, $m)) { $listed[$m[1]] = true; $reasons[] = $m[2]; }
|
||||||
|
}
|
||||||
|
$same = 0; $gone = 0; $bad = [];
|
||||||
|
foreach (file("/tmp/frt-sums", FILE_IGNORE_NEW_LINES) as $l) {
|
||||||
|
[$sum, $p] = preg_split("/\s+/", $l, 2); $p = substr($p, 2);
|
||||||
|
if (isset($listed[$p])) { file_exists($root . $p) ? $bad[] = "listed but there: $p" : $gone++; continue; }
|
||||||
|
if (!file_exists($root . $p)) { $bad[] = "not listed and not there: $p"; continue; }
|
||||||
|
hash_file("sha256", $root . $p) === $sum ? $same++ : $bad[] = "different: $p";
|
||||||
|
}
|
||||||
|
$walk = new RecursiveIteratorIterator(new RecursiveDirectoryIterator("/var/www/html/data", FilesystemIterator::SKIP_DOTS), RecursiveIteratorIterator::SELF_FIRST);
|
||||||
|
foreach ($walk as $f) { if (str_contains($f->getFilename(), ".ngb-restore-")) { $bad[] = "temporary left: " . $f->getPathname(); } }
|
||||||
|
echo "same=$same gone=$gone bad=", count($bad), "\n", implode("\n", array_slice($bad, 0, 10)), "\n";
|
||||||
|
echo "reasons: ", implode(" | ", array_unique(array_map(fn ($r) => substr(preg_replace("#[0-9a-f]{16,}#", "<id>", $r), 0, 70), $reasons))), "\n";')
|
||||||
|
printf '%s\n' "$CHECK"
|
||||||
|
printf '%s\n' "$CHECK" | grep -q " bad=0" || fail "files after the salvage are not as listed"
|
||||||
|
printf '%s\n' "$CHECK" | grep -qE "same=[1-9]" || fail "nothing was restored"
|
||||||
|
printf '%s\n' "$CHECK" | grep -qE "gone=[1-9]" || fail "no user file was hit by the damage"
|
||||||
|
ok "with --salvage: every file listed and absent, or there and identical"
|
||||||
|
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the salvage"
|
||||||
|
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing: the database was not restored"
|
||||||
|
run_on ngb-frt-restore user:list | grep admin >/dev/null || fail "the admin is missing: the database was not restored"
|
||||||
|
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back with --salvage"
|
||||||
|
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt 2>/dev/null)" = "hello disaster recovery" ] \
|
||||||
|
|| printf '%s\n' "$OUT" | grep "Docs/note.txt" >/dev/null || fail "note.txt is neither restored nor listed"
|
||||||
|
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after the salvage"
|
||||||
|
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after the salvage"
|
||||||
|
ok "the database and config are the old server's, the instance works"
|
||||||
|
echo "FULL RESTORE ROUNDTRIP PASS ($DB, salvage-files)"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
RESTORE_OPTS=(--no-interaction --install-apps)
|
RESTORE_OPTS=(--no-interaction --install-apps)
|
||||||
if [ "$LEGACY" = legacy ] && [ "$DB" != sqlite ]; then
|
if [ "$LEGACY" = legacy ] && [ "$DB" != sqlite ]; then
|
||||||
# Without --skip-missing-tables an older backup must be refused, naming the app's tables
|
# Without --skip-missing-tables an older backup must be refused, naming the app's tables
|
||||||
@@ -173,6 +286,20 @@ if [ "$LEGACY" = legacy ] && [ "$DB" != sqlite ]; then
|
|||||||
ok "older backup refused without --skip-missing-tables, missing tables named"
|
ok "older backup refused without --skip-missing-tables, missing tables named"
|
||||||
RESTORE_OPTS+=(--skip-missing-tables)
|
RESTORE_OPTS+=(--skip-missing-tables)
|
||||||
fi
|
fi
|
||||||
|
if [ "$DB" = sqlite ]; then
|
||||||
|
# Another process writing while the restore starts (a cron job that was already running, or
|
||||||
|
# the installer still finishing after it says "installed"): the restore must wait for it, not
|
||||||
|
# stop with "database is locked". Short write transactions on a table of its own, until the
|
||||||
|
# restore is done (/tmp/frt-side-stop; Nextcloud's own commands after it are not under test).
|
||||||
|
docker exec -d -u www-data ngb-frt-restore php -r '
|
||||||
|
$p = new PDO("sqlite:/var/www/html/data/nextcloud.db"); $p->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
|
||||||
|
$p->exec("CREATE TABLE IF NOT EXISTS frt_side_writer (t REAL)");
|
||||||
|
for ($end = microtime(true) + 300; microtime(true) < $end && !file_exists("/tmp/frt-side-stop");) {
|
||||||
|
$p->beginTransaction(); $p->exec("INSERT INTO frt_side_writer VALUES (" . microtime(true) . ")"); usleep(300000); $p->commit(); usleep(20000);
|
||||||
|
}
|
||||||
|
touch("/tmp/frt-side-done");'
|
||||||
|
sleep 1
|
||||||
|
fi
|
||||||
sftp_logins() { docker logs ngb-frt-sftp 2>&1 | grep -c "Accepted password" || true; }
|
sftp_logins() { docker logs ngb-frt-sftp 2>&1 | grep -c "Accepted password" || true; }
|
||||||
[ "$MODE" = sftp ] && LOGINS_BEFORE=$(sftp_logins)
|
[ "$MODE" = sftp ] && LOGINS_BEFORE=$(sftp_logins)
|
||||||
OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS[@]}" 2>&1) || {
|
OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS[@]}" 2>&1) || {
|
||||||
@@ -181,17 +308,34 @@ OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" "${RESTORE_OPTS
|
|||||||
fail "restore:full"
|
fail "restore:full"
|
||||||
}
|
}
|
||||||
printf '%s\n' "$OUT" | grep -q "maintenance mode while the restore runs" || fail "the restore did not say it runs in maintenance mode"
|
printf '%s\n' "$OUT" | grep -q "maintenance mode while the restore runs" || fail "the restore did not say it runs in maintenance mode"
|
||||||
run_on ngb-frt-restore status 2>&1 | grep -q "maintenance: false" || fail "maintenance mode is still on after the restore"
|
run_on ngb-frt-restore status 2>&1 | grep "maintenance: false" >/dev/null || fail "maintenance mode is still on after the restore"
|
||||||
run_on ngb-frt-restore config:app:get ng_backup restore_full_started >/dev/null 2>&1 && fail "the restore left its 'started' marker behind"
|
run_on ngb-frt-restore config:app:get ng_backup restore_full_started >/dev/null 2>&1 && fail "the restore left its 'started' marker behind"
|
||||||
ok "backup:restore:full completed, in maintenance mode and out of it again"
|
ok "backup:restore:full completed, in maintenance mode and out of it again"
|
||||||
|
if [ "$DB" = sqlite ]; then
|
||||||
|
docker exec ngb-frt-restore touch /tmp/frt-side-stop
|
||||||
|
for i in $(seq 1 20); do docker exec ngb-frt-restore test -f /tmp/frt-side-done && break; sleep 0.5; done
|
||||||
|
WRITES=$(docker exec -u www-data ngb-frt-restore php -r 'echo (new PDO("sqlite:/var/www/html/data/nextcloud.db"))->query("SELECT COUNT(*) FROM frt_side_writer")->fetchColumn();')
|
||||||
|
[ "${WRITES:-0}" -gt 3 ] || fail "the other writer did not run alongside the restore ($WRITES writes)"
|
||||||
|
ok "the restore waited for another process writing to SQLite ($WRITES writes alongside)"
|
||||||
|
fi
|
||||||
|
|
||||||
# Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
|
# Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
|
||||||
# earlier bugs here only showed up after config.php's secret was rewritten mid-restore.
|
# earlier bugs here only showed up after config.php's secret was rewritten mid-restore.
|
||||||
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after restore (own key/target undecryptable?)"
|
run_on ngb-frt-restore backup:status >/dev/null || fail "backup:status fails after restore (own key/target undecryptable?)"
|
||||||
run_on ngb-frt-restore user:list | grep -q frtest || fail "the backed-up user is missing from the restored database"
|
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
|
||||||
|
run_on ngb-frt-restore user:list | grep frtest >/dev/null || fail "the backed-up user is missing from the restored database"
|
||||||
docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
|
docker exec ngb-frt-restore test -f /var/www/html/data/frtest/files/Docs/note.txt || fail "restored file is missing"
|
||||||
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
|
[ "$(docker exec ngb-frt-restore cat /var/www/html/data/frtest/files/Docs/note.txt)" = "hello disaster recovery" ] || fail "restored file content differs"
|
||||||
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
if [ "$LEGACY" = legacy ]; then
|
||||||
|
# Backups made before 1.0.3 do not record empty folders
|
||||||
|
docker exec ngb-frt-restore test -e /var/www/html/data/frtest/files/Empty/Nested && fail "an empty folder came back from a backup that does not record them"
|
||||||
|
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
||||||
|
else
|
||||||
|
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "the empty folder did not come back"
|
||||||
|
run_on ngb-frt-restore files:scan frtest >/dev/null || fail "files:scan fails after restore"
|
||||||
|
docker exec ngb-frt-restore test -d /var/www/html/data/frtest/files/Empty/Nested || fail "files:scan removed the empty folder"
|
||||||
|
ok "the empty folder came back and stays after files:scan"
|
||||||
|
fi
|
||||||
# New rows after the restore: a new file (file cache) and a new user must not collide with
|
# New rows after the restore: a new file (file cache) and a new user must not collide with
|
||||||
# restored rows (ID counters after the data).
|
# restored rows (ID counters after the data).
|
||||||
docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"
|
docker exec ngb-frt-restore sh -c "echo new > /var/www/html/data/frtest/files/Docs/after-restore.txt && chown www-data /var/www/html/data/frtest/files/Docs/after-restore.txt"
|
||||||
|
|||||||
@@ -23,11 +23,15 @@ ok "key, kit code and confirmation"
|
|||||||
occ backup:target:add ci $TARGET_ARGS --path=repo >/dev/null || fail "target add"
|
occ backup:target:add ci $TARGET_ARGS --path=repo >/dev/null || fail "target add"
|
||||||
ok "location added"
|
ok "location added"
|
||||||
|
|
||||||
# Test data: a user with files.
|
# Test data: a user with files. No skeleton files on the first login, below, in the middle of the checks.
|
||||||
|
occ config:system:set skeletondirectory --value='' >/dev/null
|
||||||
|
occ config:system:set templatedirectory --value='' >/dev/null
|
||||||
$X sh -c "OC_PASS='Roundtrip-pw-2026' php occ user:add --password-from-env $U >/dev/null"
|
$X sh -c "OC_PASS='Roundtrip-pw-2026' php occ user:add --password-from-env $U >/dev/null"
|
||||||
$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin"
|
# large.bin (40 MB) fills more than one pack: a restore to a directory reads ahead over several ranges
|
||||||
|
$X sh -c "mkdir -p $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs/Empty/Nested $DATA/$U/files/Hollow && touch -d @1700000000 $DATA/$U/files/Hollow && for i in 1 2 3 4 5; do head -c 300000 /dev/urandom > $DATA/$U/files/Docs/f\$i.bin; done; echo hello > $DATA/$U/files/Docs/Sub/note.txt; head -c 6000000 /dev/urandom > $DATA/$U/files/big.bin; head -c 40000000 /dev/urandom > $DATA/$U/files/large.bin"
|
||||||
|
$X sh -c "touch -d @1600000000 $DATA/$U/files/Docs/Sub $DATA/$U/files/Docs"
|
||||||
occ files:scan "$U" >/dev/null
|
occ files:scan "$U" >/dev/null
|
||||||
SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
SUM_BEFORE=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
|
||||||
|
|
||||||
occ backup:run ci -l first >/dev/null || fail "first backup"
|
occ backup:run ci -l first >/dev/null || fail "first backup"
|
||||||
SECOND=$(occ backup:run ci -l second)
|
SECOND=$(occ backup:run ci -l second)
|
||||||
@@ -38,20 +42,145 @@ SNAP=$(occ backup:list ci | awk -F'|' '!f && / second /{gsub(/ /,"",$3); print $
|
|||||||
[ -n "$SNAP" ] || fail "snapshot not listed"
|
[ -n "$SNAP" ] || fail "snapshot not listed"
|
||||||
|
|
||||||
# Damage, then restore in each mode.
|
# Damage, then restore in each mode.
|
||||||
$X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt"
|
$X sh -c "cd $DATA/$U/files && rm Docs/Sub/note.txt && echo broken > Docs/f1.bin && echo extra > Docs/extra.txt && rm -r Docs/Empty && rmdir Hollow"
|
||||||
occ files:scan "$U" >/dev/null
|
occ files:scan "$U" >/dev/null
|
||||||
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m new-folder >/dev/null || fail "restore new-folder"
|
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m new-folder >/dev/null || fail "restore new-folder"
|
||||||
$X sh -c "ls $DATA/$U/files | grep -q '^Restored '" || fail "no Restored folder"
|
$X sh -c "ls $DATA/$U/files | grep -q '^Restored '" || fail "no Restored folder"
|
||||||
|
$X sh -c "test -d $DATA/$U/files/Restored\ */Docs/Empty/Nested" || fail "the empty folder is not in the Restored folder"
|
||||||
|
# Folders with files: on disk and in what Nextcloud shows, the parent as well as the child.
|
||||||
|
RDIR=$($X sh -c "cd $DATA/$U/files && ls -d Restored\ *")
|
||||||
|
[ "$($X sh -c "stat -c %Y $DATA/$U/files/Restored\ */Docs/Sub")" = 1600000000 ] || fail "a folder with files did not get its time back"
|
||||||
|
[ "$($X sh -c "stat -c %Y $DATA/$U/files/Restored\ */Docs")" = 1600000000 ] || fail "the parent folder did not get its time back"
|
||||||
|
DAV=$($X curl -sf -u "$U:Roundtrip-pw-2026" -X PROPFIND -H 'Depth: 1' "http://localhost/remote.php/dav/files/$U/$(php -r 'echo rawurlencode($argv[1]);' "$RDIR")/Docs/") || fail "PROPFIND"
|
||||||
|
[ "$(echo "$DAV" | grep -o 'Sun, 13 Sep 2020 12:26:40 GMT' | wc -l)" = 2 ] || fail "Nextcloud does not show the time of the folders: $(echo "$DAV" | grep -o '<d:getlastmodified>[^<]*' | sort | uniq -c)"
|
||||||
|
occ backup:restore ci "$SNAP" "data/$U/files/Docs/Sub" -m merge >/dev/null || fail "restore into a folder that is there"
|
||||||
|
[ "$($X stat -c %Y "$DATA/$U/files/Docs/Sub")" != 1600000000 ] || fail "a folder that was already there got the old time"
|
||||||
|
occ backup:browse ci "$SNAP" "data/$U/files" | grep Hollow >/dev/null || fail "an empty folder is not shown when browsing the snapshot"
|
||||||
|
occ backup:restore ci "$SNAP" "data/$U/files/Hollow" -m merge >/dev/null || fail "restore of a folder that was empty"
|
||||||
|
$X sh -c "test -d $DATA/$U/files/Hollow" || fail "the empty folder did not come back by itself"
|
||||||
|
[ "$($X stat -c %Y "$DATA/$U/files/Hollow")" = 1700000000 ] || fail "the empty folder did not get its time back"
|
||||||
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m replace >/dev/null || fail "restore replace"
|
occ backup:restore ci "$SNAP" "data/$U/files/Docs" -m replace >/dev/null || fail "restore replace"
|
||||||
|
$X sh -c "test -d $DATA/$U/files/Docs/Empty/Nested" || fail "replace did not bring back the empty folder, or moved it to the trash"
|
||||||
|
ok "empty folders: restored in each mode, by themselves and shown when browsing"
|
||||||
$X sh -c "rm -rf '$DATA/$U/files/Restored '*" ; occ files:scan "$U" >/dev/null
|
$X sh -c "rm -rf '$DATA/$U/files/Restored '*" ; occ files:scan "$U" >/dev/null
|
||||||
SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
|
||||||
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore"
|
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after restore: $(diff <(echo "$SUM_BEFORE") <(echo "$SUM_AFTER") | cut -c1-120)"
|
||||||
$X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash"
|
$X sh -c "ls $DATA/$U/files_trashbin/files/ | grep -q '^extra.txt'" || fail "extra file not in trash"
|
||||||
ok "restore new-folder and replace: files identical, extra file in the trash"
|
ok "restore new-folder and replace: files identical, extra file in the trash"
|
||||||
|
|
||||||
|
# One file to a local directory, where a restore that was stopped left its temporary.
|
||||||
|
ONE=/tmp/ngb-rt-one/data/$U/files/Docs
|
||||||
|
$X sh -c "mkdir -p $ONE && echo partial > $ONE/f2.bin.ngb-restore-0123456789ab"
|
||||||
|
occ backup:restore ci "$SNAP" "data/$U/files/Docs/f2.bin" -d /tmp/ngb-rt-one | grep "Restored 1 files" >/dev/null || fail "restore of one file to a directory"
|
||||||
|
$X sh -c "cmp -s $ONE/f2.bin $DATA/$U/files/Docs/f2.bin" || fail "the file restored to a directory differs"
|
||||||
|
$X sh -c "ls -A $ONE | grep -q ngb-restore" && fail "the temporary of a stopped restore was left"
|
||||||
|
ok "restore of one file to a directory; the temporary of a stopped restore removed"
|
||||||
|
|
||||||
|
# All of the user's files to a directory: over S3 and SFTP the packs are read ahead
|
||||||
|
occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-dir >/dev/null || fail "restore of a folder to a directory"
|
||||||
|
SUM_DIR=$($X sh -c "cd /tmp/ngb-rt-dir/data/$U/files && find . -type f -print0 | sort -z | xargs -0 md5sum")
|
||||||
|
[ "$SUM_BEFORE" = "$SUM_DIR" ] || fail "the files restored to a directory differ"
|
||||||
|
$X sh -c "ls -A /tmp/ngb-rt-dir | grep -q ngb-restore-spool" && fail "the read-ahead spool was left in the directory"
|
||||||
|
$X sh -c "test -d /tmp/ngb-rt-dir/data/$U/files/Docs/Empty/Nested" || fail "the empty folder is not in the directory"
|
||||||
|
$X sh -c "rm -rf /tmp/ngb-rt-dir"
|
||||||
|
ok "restore of all the user's files to a directory: identical, no spool left"
|
||||||
|
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
|
||||||
|
occ backup:status | grep "started (" >/dev/null || fail "backup:status does not name the time zone"
|
||||||
|
ok "backup:status names the time zone of its times"
|
||||||
|
|
||||||
|
# Without a system cron (#13): backup:status warns, and the open admin page moves a backup started
|
||||||
|
# with "Back up now" forward through /api/tick. Nothing runs cron in this container, so only the
|
||||||
|
# ticks below can finish this backup.
|
||||||
|
occ background:ajax >/dev/null
|
||||||
|
occ backup:status | grep "do not run on a system cron" >/dev/null || fail "backup:status gives no warning without system cron"
|
||||||
|
ADM=ngbadmin-$RANDOM; ADM_PW="Tick-admin-pw-$RANDOM-2026"
|
||||||
|
$X sh -c "OC_PASS='$ADM_PW' php occ user:add --password-from-env --group=admin $ADM >/dev/null"
|
||||||
|
api() { $X curl -sf -u "$ADM:$ADM_PW" -H 'OCS-APIRequest: true' -H 'Content-Type: application/json' "$@"; }
|
||||||
|
API=http://localhost/index.php/apps/ng_backup/api
|
||||||
|
STATUS=$(api "$API/status") || fail "status API"
|
||||||
|
echo "$STATUS" | php -r '$s = json_decode(stream_get_contents(STDIN), true); exit(!$s["cron"]["ok"] && $s["cron"]["mode"] === "ajax" && $s["cron"]["docs"] !== "" ? 0 : 1);' || fail "status API does not report the missing cron"
|
||||||
|
TID=$(echo "$STATUS" | php -r 'echo json_decode(stream_get_contents(STDIN), true)["targets"][0]["id"];')
|
||||||
|
RUN=$(api -X POST -d "{\"targetId\": $TID, \"label\": \"page\"}" "$API/backups" | php -r 'echo json_decode(stream_get_contents(STDIN), true)["id"];') || fail "start a backup through the API"
|
||||||
|
for i in $(seq 1 60); do
|
||||||
|
api -X POST "$API/tick" >/dev/null || fail "tick"
|
||||||
|
RS=$(api "$API/status" | php -r '$s = json_decode(stream_get_contents(STDIN), true); foreach ($s["runs"] as $r) { if ($r["id"] == '"$RUN"') { echo $r["status"]; } }')
|
||||||
|
[ "$RS" = running ] || break
|
||||||
|
done
|
||||||
|
[ "$RS" = done ] || fail "the backup started through the API ended as '$RS' after $i ticks"
|
||||||
|
occ backup:list ci | grep " page " >/dev/null || fail "the snapshot of the ticked backup is not listed"
|
||||||
|
ok "without system cron: warning in backup:status and the API, backup finished by $i ticks of the open page"
|
||||||
|
# The Bin reads in parts, as the page does (what is in the parts: CatalogTest)
|
||||||
|
api "$API/targets/$TID/trash" | php -r '$p = json_decode(stream_get_contents(STDIN), true); exit(is_int($p["total"] ?? null) && is_array($p["items"] ?? null) && count($p["items"]) <= $p["total"] ? 0 : 1);' || fail "the Bin API does not give a total and items"
|
||||||
|
ok "the Bin API reads in parts, with a total"
|
||||||
|
# Locations are backed up one after the other: the second one started is queued until the first is done.
|
||||||
|
for L in 2 3; do
|
||||||
|
$X sh -c "mkdir -p /tmp/ngb-rt-target$L"
|
||||||
|
occ backup:target:add ci$L local -a null::null -o datadir=/tmp/ngb-rt-target$L --path=repo >/dev/null || fail "target add ci$L"
|
||||||
|
done
|
||||||
|
IDS=$(api "$API/status" | php -r 'foreach (json_decode(stream_get_contents(STDIN), true)["targets"] as $t) { if ($t["name"] === "ci2" || $t["name"] === "ci3") { echo $t["id"], " "; } }')
|
||||||
|
for ID in $IDS; do api -X POST -d "{\"targetId\": $ID, \"label\": \"turn\"}" "$API/backups" >/dev/null || fail "start a backup to location $ID"; done
|
||||||
|
# Never two runs to one location: a second start gets the run that is there.
|
||||||
|
ID=${IDS%% *}
|
||||||
|
RUNS=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
|
||||||
|
api -X POST -d "{\"targetId\": $ID, \"label\": \"again\"}" "$API/backups" >/dev/null || fail "second start"
|
||||||
|
AGAIN=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
|
||||||
|
[ "$RUNS" = 2 ] && [ "$AGAIN" = 2 ] || fail "a second start to one location: $RUNS, then $AGAIN running"
|
||||||
|
occ backup:status | grep "queued" >/dev/null || fail "the second backup is not queued"
|
||||||
|
QUEUED=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["queued"]));')
|
||||||
|
[ "$QUEUED" = 1 ] || fail "$QUEUED backups queued instead of 1"
|
||||||
|
for i in $(seq 1 60); do
|
||||||
|
api -X POST "$API/tick" >/dev/null || fail "tick"
|
||||||
|
RUNNING=$(api "$API/status" | php -r 'echo count(array_filter(json_decode(stream_get_contents(STDIN), true)["runs"], fn ($r) => $r["status"] === "running"));')
|
||||||
|
[ "$RUNNING" != 0 ] || break
|
||||||
|
done
|
||||||
|
[ "$RUNNING" = 0 ] || fail "the queued backups did not finish"
|
||||||
|
occ backup:list ci3 | grep " turn " >/dev/null || fail "the queued backup made no snapshot"
|
||||||
|
ok "locations in turn: the second backup queued, and done after the first"
|
||||||
|
occ background:cron >/dev/null
|
||||||
|
$X php -f /var/www/html/cron.php >/dev/null 2>&1 || true
|
||||||
|
occ backup:status | grep "system cron" >/dev/null && fail "backup:status still warns after cron ran"
|
||||||
|
sleep 8 # APCu app config cache of the web server
|
||||||
|
api "$API/status" | php -r 'exit(json_decode(stream_get_contents(STDIN), true)["cron"]["ok"] ? 0 : 1);' || fail "the status API still reports a missing cron after cron ran"
|
||||||
|
ok "with a system cron that ran: no warning in backup:status or the API"
|
||||||
|
|
||||||
|
# A schedule per location. Choosing a time starts nothing; the moment itself does, and only there.
|
||||||
|
JOB=$(occ background-job:list --class='OCA\NgBackup\BackgroundJob\BackupJob' --output=json | php -r 'echo json_decode(stream_get_contents(STDIN), true)[0]["id"];')
|
||||||
|
job() { occ background-job:execute "$JOB" --force-execute >/dev/null; }
|
||||||
|
occ backup:target:schedule ci off >/dev/null
|
||||||
|
occ backup:target:schedule ci3 off >/dev/null
|
||||||
|
occ backup:target:schedule ci2 00:00 --timezone=UTC >/dev/null || fail "set a schedule"
|
||||||
|
occ backup:target:schedule ci2 | grep "00:00 (UTC)" >/dev/null || fail "schedule not shown"
|
||||||
|
job
|
||||||
|
[ "$(occ backup:list ci2 | grep -c ' scheduled ')" = 0 ] || fail "choosing a time started a backup"
|
||||||
|
occ backup:target:schedule ci2 "$($X date -u -d '+1 minute' +%H:%M)/6" --timezone=UTC >/dev/null || fail "set every 6 hours"
|
||||||
|
sleep 65
|
||||||
|
job
|
||||||
|
occ backup:list ci2 | grep " scheduled " >/dev/null || fail "the scheduled moment started no backup"
|
||||||
|
[ "$(occ backup:list ci3 | grep -c ' scheduled ')" = 0 ] || fail "a location that is off was backed up"
|
||||||
|
job
|
||||||
|
[ "$(occ backup:list ci2 | grep -c ' scheduled ')" = 1 ] || fail "one moment started more than one backup"
|
||||||
|
occ backup:target:schedule ci2 nonsense >/dev/null 2>&1 && fail "a bad schedule was accepted"
|
||||||
|
ok "schedule per location: nothing on choosing a time, one backup at the moment, none when off"
|
||||||
|
|
||||||
|
# What administrators are told, as Nextcloud shows it.
|
||||||
|
notes() { api "http://localhost/ocs/v2.php/apps/notifications/api/v2/notifications?format=json"; }
|
||||||
|
occ backup:target:schedule ci2 off >/dev/null
|
||||||
|
$X sh -c "touch '$DATA/ngb-rt-bad\\name.txt'"
|
||||||
|
occ backup:run ci3 -l names | grep "their name is not supported" >/dev/null || fail "occ does not report the name"
|
||||||
|
$X sh -c "rm '$DATA/ngb-rt-bad\\name.txt'"
|
||||||
|
NOTES=$(notes) || fail "read the notifications"
|
||||||
|
echo "$NOTES" | grep "turned off the schedule of backup location ci2" >/dev/null || fail "no notification about the schedule that was turned off"
|
||||||
|
echo "$NOTES" | grep "is missing because of its name" >/dev/null || fail "no notification about the name"
|
||||||
|
echo "$NOTES" | grep "Rename them" >/dev/null || fail "the notification about the name does not ask to rename"
|
||||||
|
job
|
||||||
|
notes | grep "has no schedule" >/dev/null && fail "a reminder right after turning it off"
|
||||||
|
for L in 2 3; do occ backup:target:remove ci$L >/dev/null || fail "target remove ci$L"; done
|
||||||
|
occ user:delete "$ADM" >/dev/null
|
||||||
|
ok "notifications: schedule turned off, name not supported, no early reminder"
|
||||||
|
|
||||||
# Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob).
|
# Checksum audit: shallow (index presence) and deep (download, decrypt, verify every blob).
|
||||||
occ backup:verify ci "$SNAP" | grep -q "^OK$" || fail "shallow verify"
|
occ backup:verify ci "$SNAP" | grep "^OK$" >/dev/null || fail "shallow verify"
|
||||||
occ backup:verify ci "$SNAP" --deep | grep -q "^OK$" || fail "deep verify"
|
occ backup:verify ci "$SNAP" --deep | grep "^OK$" >/dev/null || fail "deep verify"
|
||||||
ok "shallow and deep checksum audit"
|
ok "shallow and deep checksum audit"
|
||||||
|
|
||||||
# Database dump/restore round trip (all tables, checksums, sequences on PostgreSQL).
|
# Database dump/restore round trip (all tables, checksums, sequences on PostgreSQL).
|
||||||
@@ -60,13 +189,50 @@ ok "database dump and restore"
|
|||||||
|
|
||||||
# Retention, trash and catalog.
|
# Retention, trash and catalog.
|
||||||
occ backup:retention --last=1 --daily=0 --weekly=0 --monthly=0 >/dev/null
|
occ backup:retention --last=1 --daily=0 --weekly=0 --monthly=0 >/dev/null
|
||||||
occ backup:prune ci | grep "forgot 1" >/dev/null || fail "prune"
|
# A pinned restore point stays, and a note can change afterwards.
|
||||||
|
FIRST=$(occ backup:list ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')
|
||||||
|
occ backup:pin ci "$FIRST" >/dev/null || fail "pin"
|
||||||
|
occ backup:note ci "$FIRST" "a note made later" >/dev/null || fail "note"
|
||||||
|
occ backup:list ci | grep " first " | grep " yes " | grep "a note made later" >/dev/null || fail "pin and note not listed"
|
||||||
|
occ backup:prune ci --dry-run | grep "would forget 1 " >/dev/null || fail "a pinned snapshot would be removed"
|
||||||
|
occ backup:pin ci "$FIRST" --unpin >/dev/null || fail "unpin"
|
||||||
|
occ backup:prune ci --dry-run | grep "would forget 2 " >/dev/null || fail "an unpinned snapshot would stay"
|
||||||
|
occ backup:note ci "$FIRST" >/dev/null || fail "remove note"
|
||||||
|
occ backup:list ci | grep -q "a note made later" && fail "note not removed"
|
||||||
|
ok "pin and note on a restore point"
|
||||||
|
occ backup:prune ci | grep "forgot 2" >/dev/null || fail "prune" # first and second; "page" (above) is the last
|
||||||
occ backup:trash ci | grep " first " >/dev/null || fail "trash"
|
occ backup:trash ci | grep " first " >/dev/null || fail "trash"
|
||||||
TID=$(occ backup:trash ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')
|
TID=$(occ backup:trash ci | awk -F'|' '!f && / first /{gsub(/ /,"",$3); print $3; f=1}')
|
||||||
occ backup:trash:restore ci "$TID" >/dev/null || fail "trash restore"
|
occ backup:trash:restore ci "$TID" >/dev/null || fail "trash restore"
|
||||||
[ "$(occ backup:list ci | grep -c ' ci ')" = "2" ] || fail "snapshot not back from the trash"
|
[ "$(occ backup:list ci | grep -c ' ci ')" = "2" ] || fail "snapshot not back from the trash"
|
||||||
ok "retention, trash and restore from trash"
|
ok "retention, trash and restore from trash"
|
||||||
|
|
||||||
|
# Damaged data on the location (last: the location is not used after this). A byte flipped every
|
||||||
|
# MiB of every pack hits big.bin (6 MB) for sure; a small file mostly survives. Without --salvage
|
||||||
|
# the restore stops and points at --salvage; with it, the rest is restored and the damage listed.
|
||||||
|
if [[ "$TARGET_ARGS" == local* ]]; then
|
||||||
|
# "second" went to the trash above: the backup of the page is still there
|
||||||
|
SNAP=$(occ backup:list ci | awk -F'|' '!f && / page /{gsub(/ /,"",$3); print $3; f=1}')
|
||||||
|
[ -n "$SNAP" ] || fail "the snapshot of the ticked backup is not listed"
|
||||||
|
$X php -r 'foreach (glob("/tmp/ngb-rt-target/repo/packs/*/*") as $p) { $h = fopen($p, "r+b"); for ($o = 4096; $o < filesize($p); $o += 1048576) { fseek($h, $o); $b = fread($h, 1); fseek($h, $o); fwrite($h, ~$b); } fclose($h); }'
|
||||||
|
set +e
|
||||||
|
OUT=$(occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-salvage 2>&1); CODE=$?
|
||||||
|
set -e
|
||||||
|
[ "$CODE" = 1 ] || { echo "$OUT"; fail "a restore of damaged data did not stop (exit $CODE)"; }
|
||||||
|
echo "$OUT" | grep -- "--salvage" >/dev/null || { echo "$OUT"; fail "the stopped restore does not point at --salvage"; }
|
||||||
|
set +e
|
||||||
|
OUT=$(occ backup:restore ci "$SNAP" "data/$U/files" -d /tmp/ngb-rt-salvage --salvage 2>&1); CODE=$?
|
||||||
|
set -e
|
||||||
|
[ "$CODE" = 2 ] || { echo "$OUT"; fail "restore --salvage of damaged data exited with $CODE, not 2"; }
|
||||||
|
echo "$OUT" | grep "data/$U/files/big.bin: " >/dev/null || { echo "$OUT"; fail "the damaged big.bin is not listed"; }
|
||||||
|
N=$(echo "$OUT" | sed -n 's/^Restored \([0-9]*\) files.*/\1/p')
|
||||||
|
[ "${N:-0}" -ge 1 ] || { echo "$OUT"; fail "restore --salvage restored nothing"; }
|
||||||
|
$X sh -c "test ! -e /tmp/ngb-rt-salvage/data/$U/files/big.bin" || fail "the damaged big.bin was written anyway"
|
||||||
|
$X sh -c "find /tmp/ngb-rt-salvage -name '*.ngb-restore-*' | grep -q ." && fail "a temporary of a damaged file was left"
|
||||||
|
ok "damaged data: the restore stops and points at --salvage; with --salvage $N files restored and the damaged ones listed (exit 2)"
|
||||||
|
$X sh -c "rm -rf /tmp/ngb-rt-salvage"
|
||||||
|
fi
|
||||||
|
|
||||||
# No errors caused by loading files_external while it is disabled (its tables may not exist).
|
# No errors caused by loading files_external while it is disabled (its tables may not exist).
|
||||||
if $X sh -c "grep -q 'external_mounts' $DATA/nextcloud.log 2>/dev/null"; then fail "files_external table errors in the log"; fi
|
if $X sh -c "grep -q 'external_mounts' $DATA/nextcloud.log 2>/dev/null"; then fail "files_external table errors in the log"; fi
|
||||||
ok "no files_external errors in the log"
|
ok "no files_external errors in the log"
|
||||||
|
|||||||
@@ -36,8 +36,14 @@ MANIFEST=$(occ backup:user:backup ci "$U" | sed -n 's/.* to \(.*\)\.$/\1/p')
|
|||||||
[ -n "$MANIFEST" ] || fail "backup:user:backup printed no manifest path"
|
[ -n "$MANIFEST" ] || fail "backup:user:backup printed no manifest path"
|
||||||
ok "backed up $U to $MANIFEST"
|
ok "backed up $U to $MANIFEST"
|
||||||
|
|
||||||
# as-backup: restore into <uid>-bak; the original account must stay untouched.
|
# not grep -q: it stops reading at the first match, and occ, still writing, then fails (pipefail)
|
||||||
occ backup:user:restore ci "$MANIFEST" --mode=as-backup >/dev/null || fail "restore as-backup"
|
occ backup:user:list ci "$U" | grep -F "$MANIFEST" >/dev/null || fail "backup:user:list does not show $MANIFEST"
|
||||||
|
ok "backup:user:list shows the export"
|
||||||
|
|
||||||
|
# as-backup: restore into <uid>-bak; the original account must stay untouched. $U has one export,
|
||||||
|
# so the bare user id names it (#22).
|
||||||
|
OUT=$(occ backup:user:restore ci "$U" --mode=as-backup) || fail "restore as-backup by user id"
|
||||||
|
echo "$OUT" | grep -qF "Using $MANIFEST." || fail "restore by user id did not say which export it used"
|
||||||
occ files:scan "$U-bak" >/dev/null
|
occ files:scan "$U-bak" >/dev/null
|
||||||
SUM_BAK=$($X sh -c "cd $DATA/$U-bak/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
SUM_BAK=$($X sh -c "cd $DATA/$U-bak/files && find . -type f -print0 | sort -z | xargs -0 md5sum | md5sum")
|
||||||
[ "$SUM_BEFORE" = "$SUM_BAK" ] || fail "the <uid>-bak account's files differ from the original"
|
[ "$SUM_BEFORE" = "$SUM_BAK" ] || fail "the <uid>-bak account's files differ from the original"
|
||||||
@@ -55,6 +61,11 @@ SUM_AFTER=$($X sh -c "cd $DATA/$U/files && find . -type f -print0 | sort -z | xa
|
|||||||
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after a replace restore"
|
[ "$SUM_BEFORE" = "$SUM_AFTER" ] || fail "files differ after a replace restore"
|
||||||
ok "replace restores the original content under the same uid, byte-identical"
|
ok "replace restores the original content under the same uid, byte-identical"
|
||||||
|
|
||||||
|
# The replace above added a safety export: a bare user id is now ambiguous and must list both.
|
||||||
|
if OUT=$(occ backup:user:restore ci "$U" --mode=as-backup 2>&1); then fail "restore by an ambiguous user id ran"; fi
|
||||||
|
echo "$OUT" | grep -qF "$U has 2 exports here" && echo "$OUT" | grep -qF "$MANIFEST" || fail "an ambiguous user id did not list the exports"
|
||||||
|
ok "an ambiguous user id lists the exports instead of picking one"
|
||||||
|
|
||||||
# Retention: $U now has the manual export above plus the automatic safety-export the replace
|
# Retention: $U now has the manual export above plus the automatic safety-export the replace
|
||||||
# restore made of it just before deleting it (4 total after these two more), then --user-last=2
|
# restore made of it just before deleting it (4 total after these two more), then --user-last=2
|
||||||
# must forget exactly the 2 oldest.
|
# must forget exactly the 2 oldest.
|
||||||
|
|||||||