diff --git a/CHANGELOG.md b/CHANGELOG.md index e4db1a7..32be50d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,35 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th beta/daily update channel, which would hide the app from most admins. The stability stage is instead carried by the app description, the git tag, the release, and this file. +## [1.0.0-rc.6] - 2026-10-24 + +Sixth release candidate for 1.0.0; the app version is 0.9.12. Fixes from two disaster recovery +simulations and a user report. If no problems are reported for about a week, 1.0.0 will be this +same code. + +### Fixed +- The daily backup time was taken as UTC (Nextcloud runs PHP in UTC): 03:00 ran at 05:00 in + Berlin (#19). The time is now saved with the administrator's time zone and shown with it. A time + saved before is converted once to the zone of the server or an administrator, so it shows as it + effectively ran (02:30 UTC becomes 04:30 Europe/Berlin in summer) and nothing moves. +- A full restore over SMB stopped after 17 hours with "Decryption failed": a read over a long + line handed out wrong bytes (the backup itself was fine, the same files restored identically + afterwards). A blob that does not decrypt as first read, of a file or of the database, is now + read once more on a request of its own, and only a second failure stops the restore. +- A full restore onto MySQL/MariaDB stopped with "Incorrect string value" when it created a table + (of an app that is not installed here) whose rows hold emoji or other 4-byte characters: the + table got 3-byte utf8. Tables the restore creates now get this installation's character set and + collation (utf8mb4). Found in a disaster recovery simulation onto MariaDB. + +### Changed +- The admin page uses more of the screen (sections up to 1400 pixels instead of 900), so the + list of backup runs fits without scrolling sideways. +- README: a roadmap, the tested disaster recoveries, measurements for OVH S3, the Storage Box over + SFTP and SMB, how to build the SMB extension when PECL is down, and what works without shell + access. The guide lists shell access among what you need. +- The full-restore integration test checks a 4-byte character in a table the restore creates, and + retries app installs from the app store. + ## [1.0.0-rc.5] - 2026-10-22 Fifth release candidate for 1.0.0; the app version is 0.9.11. A full restore of 1 TB over SFTP diff --git a/PLAN.md b/PLAN.md index 065a39b..8c89fed 100644 --- a/PLAN.md +++ b/PLAN.md @@ -395,11 +395,53 @@ in README "Known limitations": - **1.0.1:** 1. (Naar rc4: herstelsnelheid over SFTP.) S3 leest al 16 tegelijk bij index en tree, maar een restore leest de packs nog één voor één: ook vooruitlezen via de S3-client. - 2. Waarschuwing in de beheerpagina en `backup:status` als achtergrondjobs niet op systeem-cron - staan, en "waiting for background jobs" als een lopende backup lang geen stap zette (issue - #13). Cron zelf aanzetten kan de app niet: zonder echte cron-regel stopt dan alles. + 2. Achtergrondjobs zonder systeem-cron (issue #13; besluit 2026-10-19). De PR die dit + oplost zegt "Fixes #13" (bevestigd: #13 was de cron; besluit 2026-10-21): + a. **Waarschuwen** in de beheerpagina en `backup:status`: modus uit `core backgroundjobs_mode` + (`ajax`/`webcron`/`cron`) en de laatste run uit `core lastcron`. Niet op `cron`, of de + laatste run te lang geleden: melding met de cron-regel erbij ("Backups only move while + someone has Nextcloud open"), en "waiting for background jobs" als een lopende backup lang + geen stap zette. + b. **De pagina laat de backup lopen** zolang ze openstaat: in `ajax`/`webcron` voert de + statusaanroep die de pagina toch al herhaalt ook een korte, begrensde stap uit (zoals de + korte stappen die AJAX-cron nu al krijgt). "Back up now" werkt dan ook zonder cron zolang + het tabblad openstaat; de melding zegt dat geplande backups echte cron nodig hebben. + De melding zegt (zonder werkende systeem-cron): backups hebben systeem-cron nodig, ook de + gewone nachtelijke; zo stel je cron in (link naar de Nextcloud-documentatie over + achtergrondjobs voor de versie van de server, nu + https://docs.nextcloud.com/server/34/admin_manual/configuration_server/background_jobs_configuration.html, + versienummer uit de server, niet vast); zolang er geen cron is, loopt een backup die je + met de knop start alleen door terwijl de NG Backup-pagina openstaat. + c. **Voortgang in de databasefase** (issue #13): nu staat er alleen "running (db)", wat er + hetzelfde uitziet als vastzitten. Toon "running (db: 47/183 tables)" zoals de bestandsfase + een percentage toont, en "waiting for background jobs (last ran 12 min ago)" als een run + een tijd geen stap zette, ook bij een cron die is blijven staan. + Niet doen: de cron-modus vanuit de app omzetten (zonder echte cron-regel stopt alles, en + `cron.php` zet de modus zelf op `cron`), en geen `exec php occ backup:run &` vanuit de knop + (vaak uitgeschakeld, niet op hosted, buiten Nextcloud om lastig te volgen en te stoppen). 3. Volledige DR-test met een backup die al de tabelstructuur bevat (gemaakt met rc2 of later), zonder `--skip-missing-tables`: dat pad is op 1 TB nog niet getest. + 4. **Label bij "Back up now"** (besluit 2026-10-19): een optioneel tekstveld naast de knop; de + tekst gaat als label van de snapshot mee (nu kan dat alleen met `occ backup:run -l`; de knop + stuurt een leeg label, geplande runs krijgen "scheduled"). Raakt alleen de UI en een + bestaande parameter. + 5. **`--salvage`** voor `backup:restore` en `backup:restore:full` (besluit 2026-10-22): terughalen + wat kan, een bestand dat niet te lezen of te ontsleutelen is overslaan en aan het eind + opsommen, in plaats van stoppen. + 6. **Tijdelijke bestanden van een afgebroken restore opruimen** (gevonden in de SMB-DR van + rc6, 2026-10-24): de rc5-poging die stopte liet `VID_0009.mp4.ngb-restore-` staan; de + geslaagde restore daarna haalde het niet weg, en `files:scan` maakte het zichtbaar in Files. + Een restore ruimt `*.ngb-restore-*` in de mappen die hij terugzet op. + 7. **Eén bestand terugzetten met `backup:restore`**: een pad naar een bestand zet 0 bestanden + terug (er komt een `/` achter); nu werkt alleen een map. +- **DR-tests over databasegrenzen** (besluit 2026-10-20; productie nc01 draait PostgreSQL): na + de MariaDB→PostgreSQL-fout (tellers, rc5) elke overstap in de integratietest (`to-pgsql`, + `to-mysql`, `to-sqlite`, ook in CI). Volle DR-tests met echte data: vóór 1.0.0 de S3-simulatie + (MariaDB-bron, 138 GB) nogmaals naar een MariaDB-doel en de SMB-simulatie naar een SQLite-doel; + **na 1.0.0** de 1 TB van nc01 (PostgreSQL) naar een MariaDB-doel. +- **Polish:** `target:add` toonde bij het eerste indexbestand al "about 642 min left" (S3, + 4.568 bestanden, daarna binnen een minuut klaar): een schatting pas na ~5% of na meer dan + een paar bestanden. - **1.1.0:** 1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van @@ -412,6 +454,22 @@ in README "Known limitations": 3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf, 33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1. + 4. **Schema per locatie** (besluit 2026-10-19): in de instellingen van een locatie "uit", + "dagelijks om HH:MM" of "elke N uur (vanaf HH:MM)". Nu is er één `schedule_time` voor alle + locaties, één keer per dag. Bij de update wordt de huidige tijd het schema van elke + bestaande locatie (voor bestaande gebruikers verandert er niets). + 5. **Locaties na elkaar laten starten** in plaats van tegelijk: twee runs die dezelfde schijven + lezen zijn samen niet sneller (gezien op nc01); wachtende runs zichtbaar als "queued". + 7. **SMB sneller** (gemeten 2026-10-21): lokaal netwerk zonder vertraging (Samba-container op de + laptop, 1,8 GB) back-up 24,9 MB/s, restore 27,6 MiB/s; vanaf Canada naar de Storage Box (130 ms) + 2,6 MiB/s, omdat SMB per blok op het antwoord wacht. Grotere schrijfblokken, en vooruitlezen + bij een restore zoals SFTP (IReadsAhead) of meerdere packs tegelijk. Een NAS met 1 Gbit haalt + ~110 MB/s: daar is winst. + Ook de index: `target:add` over SMB vanaf Canada haalt ~1 indexbestand per seconde (4.927 + bestanden ~85 min), want SMB krijgt geen `getMany()` met verzoeken tegelijk zoals S3 en SFTP. + 6. **Notitie achteraf bij een restorepunt**, te wijzigen in de lijst. Het label in de snapshot + is versleuteld en ligt vast; de notitie apart op de locatie bewaren (bijv. in de catalogus), + zodat een nieuwe server na een DR hem ook ziet. - **Bevindingen DR-test op PVE02 (2026-10-17, snapshot van nc01, 1 TB, AIO → gewone NC 34 met PostgreSQL 18)** — voor **1.0.0-rc.2**: 1. Tabellen van apps die bij de backup **niet aanstonden** (restanten: Deck, ONLYOFFICE, diff --git a/README.md b/README.md index 47ef3c1..a412d4f 100644 --- a/README.md +++ b/README.md @@ -146,10 +146,31 @@ What you should do on the location (strongly recommended): - **Files and folders** of a user: in the web interface or `occ backup:restore`, into a new folder ("Restored "), merged into the original place, or replacing it. - **Raw restore** of any path (for example `config/`) to a local directory: `occ backup:restore ... --to-directory=DIR`. -- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see the how-to. +- **Whole server** (disaster recovery onto a fresh installation): `occ backup:restore:full`, see + [docs/DISASTER-RECOVERY.md](docs/DISASTER-RECOVERY.md). It runs in maintenance mode, so that + nobody and no background job writes into the instance while it is being replaced. - **One user** (account, settings, files, calendars, contacts via user_migration): `occ backup:user:backup` and `occ backup:user:restore`, replacing the account or restoring it as `-bak`. See the how-to. +**Without shell access** (a hosted Nextcloud where you cannot run `occ`): files and folders can be +restored in the web interface. Restoring a whole user or the whole server needs `occ`, so on such +hosting it is something your provider (or whoever has shell access) runs. A restore mode that +works without maintenance mode, and so from the web interface, is on the roadmap. + +## Tested disaster recoveries + +Full restores onto a fresh server, following only [the guide](docs/DISASTER-RECOVERY.md): + +| Backup | Location | Database | Result | +|---|---|---|---| +| Production, about 1 TB, 150,000 files (Nextcloud All-in-One) | Hetzner Storage Box (SFTP), Germany → Canada | PostgreSQL → PostgreSQL | 150,622 files restored in about 25 h (13-14 MiB/s), sample identical | +| 138 GB simulation: 20 users, calendars, contacts, mail, shares, team folders with an ACL, versions, trash | OVH S3, Warsaw → Canada | MariaDB → PostgreSQL | all 15,772 files and every database count identical | +| The same | OVH S3 | MariaDB → MariaDB | all 15,772 files and every database count identical | +| The same | Hetzner Storage Box (SMB), Germany → Canada | MariaDB → SQLite | all 15,772 files and every database count identical (27.6 h, about 1.5 MB/s) | +| 51 GB test instance | OVH S3 | PostgreSQL → PostgreSQL | 23,519/23,519 checksums identical | + +The integration tests restore across every pair of PostgreSQL, MariaDB and SQLite. + ## Storage backend experiences Notes from actually running a backend, beyond what's in files_external's own documentation. @@ -206,12 +227,64 @@ Contributions welcome via PR — add what you ran into with your own provider. can be read on more connections: `occ config:app:set ng_backup restore_sftp_connections --value=4` (at most 8; the server must allow that many plus one at once for the user). +- **OVH Object Storage (S3, Warsaw, 2026-10):** works with the `amazons3` backend (path style, + region `waw`). A full backup of 1 TB from Germany ran at about 7 MiB/s, from Canada about + 6 MiB/s with BBR; a check of a 1 TB backup took 3.5 minutes (index files are fetched 16 at a + time); a restore of 138 GB from Warsaw to Canada took about 6 hours. Turn on versioning and + Object Lock on the bucket (see above); OVH charges no egress. + +- **Hetzner Storage Box over SFTP, full restore:** about 13-14 MiB/s from Germany to Canada + (122 ms) with the read-ahead of 1.0.0-rc.4; without it about 2.8 MiB/s. A Storage Box gets + slower with several connections reading at once (4-6 MiB/s for 2-4 connections, it reads from + disks), so one extra connection is the default. A Storage Box drops an SFTP connection that + stayed quiet for a while without telling the client; NG Backup checks and replaces it. + +- **SMB speed:** on a local network (Samba, no delay) a backup ran at about 25 MB/s and a + restore at about 28 MiB/s. Over a long line SMB waits for an answer to each request: from + Canada to a Storage Box in Germany (130 ms) about 2.6 MiB/s for a backup and 1.6 MiB/s for a + restore, and connecting a location with an existing backup reads its index at about one file a + second. SMB is meant for a local network; use SFTP or S3 for a location far away. When + `pecl install smbclient` fails (pecl.php.net was unreachable for hours on 2026-10-21), the + extension builds from its source: `libsmbclient-php` on GitHub, `phpize && ./configure && make + && make install`, then `docker-php-ext-enable smbclient`. + - **NFS is not a files_external backend at all.** Mount the NFS export on the server yourself (e.g. via `/etc/fstab` or a systemd `.mount` unit) and point ng_backup's `local` backend at the mount point; ng_backup never mounts or unmounts anything on its own, so an NFS mount dropping mid-run behaves like any other local-path failure (the run resumes). See PLAN.md 3.5 for why NFS can't be driven from PHP directly. +## Roadmap + +What is planned, without dates. Ideas and votes are welcome in the +[issues](https://github.com/dillardblom/ng_backup/issues); if something you need is listed here, +a 👍 on its issue helps decide what comes first. + +**1.0.1** +- A warning on the admin page and in `occ backup:status` when background jobs do not run on a + system cron, with how to set one up. Without cron, a backup started with "Back up now" moves on + while the NG Backup page is open ([#13](https://github.com/dillardblom/ng_backup/issues/13)). +- An optional label for a backup started with "Back up now". +- Faster full restores from S3 and SMB, reading ahead as restores over SFTP do since 1.0.0-rc.4. + +**1.1.0** +- Restore without the recovery kit: the backup key from the location itself, opened with one of + your passphrases. +- A schedule per location, also "every N hours". +- Locations backed up one after the other instead of at the same time. +- A note on a restore point that you can change afterwards. +- An option to pause heavy background jobs (such as Memories, Recognize or the Preview + Generator) while a backup runs. +- Faster backups: reading and uploading at the same time. + +**Later** +- A full restore onto the existing installation, without first installing a fresh Nextcloud, in a + restore mode of its own instead of maintenance mode, so that it can also run from the web + interface (for hosted Nextcloud without shell access). +- Users restoring their own files from the Files app. +- Primary object storage and External Storage mounts (see Known limitations). +- An option not to keep the backup key on the location (the recovery kit is then the only copy). + ## Known limitations On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it: @@ -236,8 +309,11 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you - Location passwords passed with `-o password=...` are visible in the process list and shell history while the command runs. - Temporary files of a restore into a user's files end up in that user's trash. -- Restoring the database does not order tables by foreign keys; apps with cascading foreign keys - could lose rows (Nextcloud core does not use them). +- **Restoring a whole user or the whole server needs `occ`** (shell access); see "Without shell + access" above. +- **`--install-apps` depends on the app store.** Nextcloud downloads the whole app list (about + 38 MB) and gives up after 120 seconds; on a slow day apps are not installed. The restore then + lists them: install them afterwards with `occ app:install`, their data is restored already. - A pack whose index upload failed stays on the location until removed by hand. ## Commands diff --git a/appinfo/info.xml b/appinfo/info.xml index ba68b03..79b83ce 100644 --- a/appinfo/info.xml +++ b/appinfo/info.xml @@ -42,7 +42,7 @@ welcome: use the "Share your experience" link on the admin page. Developed and sponsored by [Ensembia](https://www.ensembia.com). We can offer (paid) assistance if you need help with restoring a backup after a crash.]]> - 0.9.11 + 0.9.12 agpl Dillard Blom NgBackup @@ -53,9 +53,9 @@ you need help with restoring a backup after a crash.]]> https://github.com/dillardblom/ng_backup https://github.com/dillardblom/ng_backup/issues https://github.com/dillardblom/ng_backup - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/1-backups.png - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/2-restore.png - https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.5/screenshots/3-settings.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/1-backups.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/2-restore.png + https://raw.githubusercontent.com/dillardblom/ng_backup/v1.0.0-rc.6/screenshots/3-settings.png sodium diff --git a/docs/DISASTER-RECOVERY.md b/docs/DISASTER-RECOVERY.md index 06bbb47..86e35fc 100644 --- a/docs/DISASTER-RECOVERY.md +++ b/docs/DISASTER-RECOVERY.md @@ -10,6 +10,8 @@ Nextcloud back from an NG Backup location. Do them in this order. - [ ] A new server with the **same Nextcloud major version** as the old one (for example 34), a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for the whole backup plus a margin. +- [ ] **Shell access** to the new server, to run `occ` (on a hosted Nextcloud without it, your + provider runs these steps). - [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by hand). Keep their packages somewhere safe beforehand. diff --git a/lib/BackgroundJob/BackupJob.php b/lib/BackgroundJob/BackupJob.php index 4171fe4..76dd5fc 100644 --- a/lib/BackgroundJob/BackupJob.php +++ b/lib/BackgroundJob/BackupJob.php @@ -32,6 +32,7 @@ final class BackupJob extends TimedJob { private RunMapper $runs, private IAppConfig $appConfig, private LoggerInterface $logger, + private \OCA\NgBackup\Service\Schedule $schedule, ) { parent::__construct($time); // Well below the usual 5-minute cron interval: the last run is stamped a moment after cron @@ -101,14 +102,10 @@ final class BackupJob extends TimedJob { $this->appConfig->setValueString(Application::APP_ID, 'last_prune_week', $due); } - /** Daily schedule: appconfig ng_backup/schedule_time "HH:MM" (server time); empty = no schedule. */ + /** Daily schedule: "HH:MM" in the schedule's time zone (see Schedule); empty = no schedule. */ private function startScheduled(): void { - $at = $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''); - if (!preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $at)) { - return; - } - $todayAt = (int)strtotime(date('Y-m-d') . ' ' . $at); - if (time() < $todayAt) { + $todayAt = $this->schedule->todayAt(); + if ($todayAt === null || time() < $todayAt) { return; } foreach ($this->targets->list() as $t) { diff --git a/lib/Controller/AdminApiController.php b/lib/Controller/AdminApiController.php index 1fec3bb..886e415 100644 --- a/lib/Controller/AdminApiController.php +++ b/lib/Controller/AdminApiController.php @@ -47,6 +47,7 @@ final class AdminApiController extends Controller { private \OCA\NgBackup\Service\AlertService $alerts, private \OCA\NgBackup\Service\KeySlotService $slots, private \OCA\NgBackup\Service\PruneService $prune, + private \OCA\NgBackup\Service\Schedule $schedule, ) { parent::__construct(Application::APP_ID, $request); } @@ -72,7 +73,8 @@ final class AdminApiController extends Controller { 'runs' => $runs, 'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown), 'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS, - 'schedule' => $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''), + 'schedule' => $this->schedule->get()['time'], + 'scheduleTimezone' => $this->schedule->get()['timezone'], ]); } @@ -240,12 +242,14 @@ final class AdminApiController extends Controller { return new JSONResponse($this->runJson($run)); } - public function setSchedule(string $time): JSONResponse { - if ($time !== '' && !preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time)) { - return new JSONResponse(['error' => 'Use HH:MM or empty'], Http::STATUS_BAD_REQUEST); + /** @param string $timezone the administrator's zone (the browser's); empty: keep the current one */ + public function setSchedule(string $time, string $timezone = ''): JSONResponse { + try { + $this->schedule->set($time, $timezone !== '' ? $timezone : $this->schedule->get()['timezone']); + } catch (\InvalidArgumentException $e) { + return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST); } - $this->appConfig->setValueString(Application::APP_ID, 'schedule_time', $time); - return new JSONResponse(['schedule' => $time]); + return new JSONResponse(['schedule' => $time, 'scheduleTimezone' => $this->schedule->get()['timezone']]); } public function snapshots(int $targetId): JSONResponse { diff --git a/lib/Db/DbDumper.php b/lib/Db/DbDumper.php index b0918f7..6bd3a40 100644 --- a/lib/Db/DbDumper.php +++ b/lib/Db/DbDumper.php @@ -159,7 +159,7 @@ final class DbDumper { // everywhere); tables it does not know are left out, as asked. foreach ($missing as $table) { if (isset($manifest['tables'][$table]['schema'])) { - $this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'])); + $this->db->createSchemaManager()->createTable(self::tableFromStructure($table, $manifest['tables'][$table]['schema'], $this->tableOptions())); $report['created'][] = $table; } else { $report['missing'][] = $table; @@ -319,8 +319,31 @@ final class DbDumper { 'foreignKeys' => $foreignKeys]; } - private static function tableFromStructure(string $name, array $structure): \Doctrine\DBAL\Schema\Table { - $table = new \Doctrine\DBAL\Schema\Table($name); + /** + * Table options for a table the restore creates: those of this installation's own tables. + * On MySQL/MariaDB that is the character set and collation, as Nextcloud's tables have them + * (utf8mb4, so emoji and other 4-byte characters fit); DBAL alone would use 3-byte utf8. + * + * @return array + * @psalm-suppress UndefinedClass the DBAL classes come with Nextcloud, not with this app + */ + private function tableOptions(): array { + $options = $this->db->createSchemaManager()->createSchemaConfig()->getDefaultTableOptions(); + if ($this->db->getDatabasePlatform() instanceof \Doctrine\DBAL\Platforms\AbstractMySQLPlatform) { + $collation = $this->db->fetchOne('SELECT TABLE_COLLATION FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = ?', + [$this->prefix . 'appconfig']); + if (is_string($collation) && $collation !== '') { + $options['collation'] = $collation; + $options['charset'] = explode('_', $collation, 2)[0]; + unset($options['collate']); + } + } + return $options; + } + + /** @param array $options table options (see tableOptions()) */ + private static function tableFromStructure(string $name, array $structure, array $options = []): \Doctrine\DBAL\Schema\Table { + $table = new \Doctrine\DBAL\Schema\Table($name, [], [], [], [], $options); foreach ($structure['columns'] as $c) { $table->addColumn($c['name'], $c['type'], array_filter([ 'length' => $c['length'], 'precision' => $c['precision'], 'scale' => $c['scale'], diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index 2146e20..56a837f 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -845,7 +845,15 @@ final class Repository { try { foreach ($this->backend->getRanges($ranges) as $i => $data) { foreach ($parts[$i] as [$id, $at, $length]) { - yield $this->decodeBlob($id, substr($data, $at, $length)); + try { + $blob = $this->decodeBlob($id, substr($data, $at, $length)); + } catch (\Exception) { + // a read that went wrong on the way (see loadBlob()): this blob on its own + $blob = $this->loadBlob($id); + // its pack stream is closed before the next range is read on the same connection + $this->packReader->close(); + } + yield $blob; $done++; } } @@ -895,8 +903,46 @@ final class Repository { } catch (BackendException) { throw $e; } + $pack = $moved; } - return $this->decodeBlob($id, $encrypted); + try { + return $this->decodeBlob($id, $encrypted); + } catch (\Exception $damaged) { + // Bytes that do not decrypt: damaged on the location, or a read that went wrong on the + // way (a stream that lost its place: an SMB restore stopped after 17 hours like this). + // Read exactly this blob once more, on a request of its own; only a second failure + // means the data itself is damaged. + $this->packReader->close(); + try { + $again = $this->backend->getRange('packs/' . substr($pack, 0, 2) . '/' . $pack, $offset, $length); + } catch (\Exception) { + throw $damaged; + } + $data = $this->decodeBlob($id, $again); + if (function_exists('OCP\Log\logger')) { + \OCP\Log\logger('ng_backup')->warning('NG Backup: blob ' . $id . ' did not decrypt as first read from pack ' . $pack . ' at ' . $offset . '; read again, it is fine. First read: ' . self::howItDiffered($encrypted, $again)); + } + return $data; + } + } + + /** + * What a read that went wrong looked like next to the right bytes, for the log: where it starts + * to differ, and whether the rest is zeros or the right bytes from another place (a stream that + * lost its place). + */ + private static function howItDiffered(string $wrong, string $right): string { + $at = strspn($wrong ^ $right, "\0"); + $text = strlen($wrong) . ' bytes, differ from byte ' . $at . ' of ' . strlen($right); + $tail = substr($wrong, $at, 64); + if ($tail === '') { + return $text . ' (too short)'; + } + if (trim($tail, "\0") === '') { + return $text . ' (zeros)'; + } + $found = strpos($right, $tail); + return $text . ($found !== false ? ' (the right bytes from ' . ($found - $at) . ' bytes further on)' : ' (other bytes)'); } /** Decrypts and checks a blob read from its pack; throws when it is corrupted or tampered with. */ diff --git a/lib/Service/Schedule.php b/lib/Service/Schedule.php new file mode 100644 index 0000000..326c537 --- /dev/null +++ b/lib/Service/Schedule.php @@ -0,0 +1,110 @@ +appConfig->getValueString(Application::APP_ID, self::TIME_KEY, ''); + $zone = $this->appConfig->getValueString(Application::APP_ID, self::ZONE_KEY, ''); + if ($zone === '' || !self::validZone($zone)) { + $zone = $this->guessZone(); + if ($time !== '' && self::validTime($time)) { + $time = self::convert($time, 'UTC', $zone); + $this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time); + } + $this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone); + } + return ['time' => $time, 'timezone' => $zone]; + } + + /** @throws \InvalidArgumentException for a malformed time or an unknown zone */ + public function set(string $time, string $zone): void { + if ($time !== '' && !self::validTime($time)) { + throw new \InvalidArgumentException('Use HH:MM or empty'); + } + if (!self::validZone($zone)) { + throw new \InvalidArgumentException('Unknown time zone: ' . $zone); + } + $this->appConfig->setValueString(Application::APP_ID, self::ZONE_KEY, $zone); + $this->appConfig->setValueString(Application::APP_ID, self::TIME_KEY, $time); + } + + /** Today's scheduled moment as a Unix time, in the schedule's zone; null without a schedule. */ + public function todayAt(?int $now = null): ?int { + ['time' => $time, 'timezone' => $zone] = $this->get(); + if ($time === '' || !self::validTime($time)) { + return null; + } + $tz = new \DateTimeZone($zone); + $day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone($tz)->format('Y-m-d'); + return (new \DateTimeImmutable($day . ' ' . $time, $tz))->getTimestamp(); + } + + /** + * "HH:MM" in zone $from as "HH:MM" in zone $to, at today's offsets. + * + * @param non-empty-string $from + * @param non-empty-string $to + */ + public static function convert(string $time, string $from, string $to, ?int $now = null): string { + $day = (new \DateTimeImmutable('@' . ($now ?? time())))->setTimezone(new \DateTimeZone($from))->format('Y-m-d'); + return (new \DateTimeImmutable($day . ' ' . $time, new \DateTimeZone($from)))->setTimezone(new \DateTimeZone($to))->format('H:i'); + } + + public static function validTime(string $time): bool { + return preg_match('/^([01]\d|2[0-3]):[0-5]\d$/', $time) === 1; + } + + /** @psalm-assert-if-true non-empty-string $zone */ + public static function validZone(string $zone): bool { + return in_array($zone, \DateTimeZone::listIdentifiers(), true); + } + + /** + * The server's default_timezone when set, else the first administrator's, else UTC. A server + * zone of UTC is passed over on purpose: it is mostly what an image or installer left there, + * while the administrator who set the time read it as their own (#19). + * + * @return non-empty-string + */ + private function guessZone(): string { + $zone = $this->config->getSystemValueString('default_timezone', ''); + if ($zone !== '' && $zone !== 'UTC' && self::validZone($zone)) { + return $zone; + } + foreach ($this->groups->get('admin')?->getUsers() ?? [] as $user) { + $zone = $this->config->getUserValue($user->getUID(), 'core', 'timezone', ''); + if ($zone !== '' && self::validZone($zone)) { + return $zone; + } + } + return 'UTC'; + } +} diff --git a/src/AdminSettings.vue b/src/AdminSettings.vue index eae0342..b0d7ccc 100644 --- a/src/AdminSettings.vue +++ b/src/AdminSettings.vue @@ -89,8 +89,10 @@ onBeforeUnmount(() => clearTimeout(timer))