diff --git a/.github/ISSUE_TEMPLATE/experience.yml b/.github/ISSUE_TEMPLATE/experience.yml new file mode 100644 index 0000000..5fb29c2 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/experience.yml @@ -0,0 +1,56 @@ +name: Experience report +description: Tell us how NG Backup works for you, also when everything went fine. +title: "Experience: " +body: + - type: markdown + attributes: + value: | + Thanks for trying NG Backup! Reports of what works are just as useful as bug reports: + they tell us which setups are tested. Please leave out host names, user names and keys. + - type: dropdown + id: location + attributes: + label: Backup location + multiple: true + options: + - S3 / S3-compatible + - SFTP + - WebDAV + - SMB/CIFS + - FTP + - Swift + - Local path (disk, NFS mount, ...) + validations: + required: true + - type: input + id: provider + attributes: + label: Storage provider (optional) + placeholder: e.g. Hetzner Storage Box, OVH, Backblaze B2, own NAS + - type: input + id: setup + attributes: + label: Nextcloud version and setup + placeholder: e.g. 32.0.1, AIO / Docker / bare metal, PostgreSQL / MariaDB + - type: input + id: size + attributes: + label: Rough size of the backup + placeholder: e.g. 50 GB, 200,000 files + - type: checkboxes + id: tried + attributes: + label: What did you try? + options: + - label: Full backup + - label: Incremental backup + - label: Verify + - label: Restore of files or folders + - label: Full restore on another server + - type: textarea + id: notes + attributes: + label: How did it go? + description: What worked, what was unclear, slow or broken. + validations: + required: true diff --git a/CHANGELOG.md b/CHANGELOG.md index d7b2dbf..11438ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,49 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th beta/daily update channel, which would hide the app from most admins. The stability stage is instead carried by the app description, the git tag, the release, and this file. +## [Unreleased] + +## [1.0.0-rc.1] - 2026-10-17 + +Release candidate for 1.0.0; the app version is 0.9.7. Only bug fixes until 1.0.0. Tested on a +production instance (about 1 TB, 150,000 files): full and incremental backups and a verify on +SFTP and SMB (Hetzner Storage Box) and S3 (OVH), a deep verify on SFTP, and partial restores from +SMB and S3 checked against checksums. + +### Added +- The admin page asks for feedback, with a link to a short "Experience report" form on GitHub: + which location, setup and size you used, and how it went. + +### Changed +- New passphrases must be at least 16 characters (was 12). The location stores the key + encrypted with each passphrase, so the passphrase is what really protects the backups. + Existing passphrases keep working. The README and the how-to now say so, and why the + recovery kit must still be kept off the server. +- Dates and times in the web interface follow the locale in the user's Nextcloud settings instead + of the browser's language, so a 24-hour locale shows 16:00 rather than 4:00 PM. +- The list of backup runs (web interface and `occ backup:status`) shows the runs of the last 7 + days, those still running, and at least the 5 newest; a failed run stays visible for that week. + `occ backup:status --all` and "Show older runs" in the web interface list all of them. +- `occ backup:status` and `occ backup:verify` show durations of a day or more with the days + ("1d 12:10:13" instead of "12:10:13"). + +### Fixed +- With system cron, a running backup moved forward only every other cron run: Nextcloud + skipped the background job when cron came a few seconds before its 5-minute interval was + over, so a backup was idle 6 of every 10 minutes. The job now runs at every cron run. +- Photos, video and other compressed data are no longer deflated only to throw the result away: + a quick test on three small slices of each block skips it. Compressing was the slowest part of + a backup (about 24 MiB/s). +- Checking a backup on an S3 location (`occ backup:verify`) took hours on a large repository: + the index files were downloaded one at a time, at about two a second. They are now downloaded + 16 at a time: on a production instance (27,585 index files, 1 TB) the check went from about + 4 hours to 3.5 minutes. +- Writing the snapshot at the end of a long backup to S3 took a quarter of an hour without any + progress shown: it read back the file list of every step one at a time. These reads are now + done 16 at a time on S3 (96 reads: 82 seconds before, 2 now), blobs that follow each other in + a pack are read at once on every location, the snapshot is written in a step of its own, and + the run shows "commit" while it does so. + ## [0.9.6-beta.7] - 2026-10-14 Bugfix release for SMB locations, with `occ backup:cancel` to stop a backup that does not get @@ -245,6 +288,7 @@ and not implemented yet. - See the "Storage backend experiences" section of the README for provider-specific quirks found so far (e.g. Hetzner Storage Box sub-accounts). +[1.0.0-rc.1]: https://github.com/dillardblom/ng_backup/releases/tag/v1.0.0-rc.1 [0.9.6-beta.7]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.6-beta.7 [0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6 [0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5 diff --git a/PLAN.md b/PLAN.md index e856e7e..bee690d 100644 --- a/PLAN.md +++ b/PLAN.md @@ -377,6 +377,29 @@ in README "Known limitations": 11. `backup:restore:full` in onderhoudsmodus en hervatbaar maken; nu: opnieuw draaien met `--force`. 12. Config/rename op FTP/SMB-servers die geen rename over een bestaand bestand toestaan. +**Plan richting 1.0.0 en 1.1.0 (besluit 2026-10-15/16):** +- **1.0.0-rc.1:** wat nu op `polish` staat (snelle S3-verify en snapshot-commit, kortere runlijst, + duur met dagen, job bij elke cron-run, deflate-test, feedbackformulier, datums in de locale van + de gebruiker, wachtwoordzin min. 16 tekens). Geen nieuwe functies meer. +- **Vóór 1.0.0 (verplicht):** volledige restore met kit op een tweede server (PVE02), alleen lezend + of op een kopie van de locatie; plus een user-restore. Zonder geslaagde test geen 1.0.0. +- **1.1.0:** + 1. `backup:key:import` zonder kit: de sleutel-slots van de locatie (`config`) lezen, met de kit + als reserve als `config` ontbreekt of niet opent. Vergt een andere volgorde van + `target:add` en sleutelimport (het gevoeligste pad), daarom niet in rc1. Daarna de + DR-test op PVE02 herhalen zonder kit. + 2. Instelling "zware achtergrondjobs pauzeren tijdens een backup" (standaard aan, uit te zetten): + Memories, files_antivirus BackgroundScanner, Recognize, Preview Generator, fulltextsearch. + Via de reservering/last_run van die jobs (geen apps uitzetten), crashbestendig, maximaal + ~6 uur achter elkaar, zichtbaar in UI en log. Op nc01 met de hand getest tijdens SMB-run 10. + 3. Snelheid: lezen en uploaden laten overlappen, meerdere bestanden tegelijk lezen, eventueel + een eigen worker (`occ background-job:worker`). Gemeten op nc01: 55% wachten op de schijf, + 33% op de SMB-upload, ~7% CPU. Vergt extra PHP-processen; botst mogelijk met uitgangspunt 1. +- **Later (geen versie):** optie om de sleutel niet op de locatie op te slaan (kit is dan de enige + kopie; uitdrukkelijke waarschuwing). Primaire object storage en External-storage-koppelingen + meenemen via de opslaglaag (§3.4; nu niet ondersteund, zie README "Known limitations"). +- **Open:** of de bevestigingscode van de kit dubbelop is (nog niet over besloten). + Doorgeschoven naar **v1.3.0** (fase 3, niet meer fase 2): in-place volledige restore (eigen restore-modus i.p.v. onderhoudsmodus) en zelfbediening voor gebruikers (eigen bestanden terugzetten vanuit de Files-app). diff --git a/README.md b/README.md index 85454b9..83df76a 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,8 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on see "Storage backend experiences" below — that requirement comes from files_external, not from ng_backup, and applies to any Nextcloud app that writes to an SMB share.) -**Status: beta (0.9.6).** Tested thoroughly, usable alongside your current backup, but not yet a -drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and +**Status: release candidate for 1.0.0 (1.0.0-rc.1, app version 0.9.7).** Tested thoroughly, usable +alongside your current backup, but not yet a drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and `docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. License: AGPL-3.0-or-later. @@ -23,6 +23,10 @@ License: AGPL-3.0-or-later. leaves the server, and streamed straight to the location: no local copy, no double disk space. Unchanged files are not read again; only new data is uploaded. - The database is dumped through Nextcloud's own connection, inside one consistent transaction. +- A backup runs in short steps from Nextcloud's background jobs, and every step saves where it + got to. A restart of the server or the containers does not lose the run: it continues at the + next background job run. Tested on a production Nextcloud AIO instance: a 1 TB full backup over + SMB carried on through the nightly AIO update, which restarts every container. - Restores go through Nextcloud, so overwritten files keep their old content as a version and removed files go to the trash. @@ -47,9 +51,19 @@ All backups of an installation are encrypted with one master key. - The **server** keeps the master key, encrypted with the instance secret, so scheduled backups and restores work without anyone typing a passphrase. Any administrator can restore from the web interface. Consequence: whoever fully controls the server can read the backups; whoever - only gets hold of the backup location cannot. -- For a **restore on a new server** (disaster recovery) you need the **recovery kit** and **one - passphrase**. The kit contains the master key, encrypted separately with each passphrase. + only gets hold of the backup location cannot, as long as the passphrases are strong (see below). +- The **location holds the key too**, by design (like restic and Borg in repokey mode): its + `config` file contains the master key, encrypted separately with each passphrase (Argon2id), + the same as in the recovery kit. Without a passphrase it is of no use, but whoever can read the + location can try passphrases offline. **Your passphrase is therefore the actual key**: it must + be at least 16 characters; five or more random words are a good choice. +- The **recovery kit** is your own copy of that encrypted key. Still keep it off the server and + off the location: an attacker who gets write access to the location can damage or delete its + `config` file on purpose, and without the kit that makes every backup there unreadable. +- For a **restore on a new server** (disaster recovery) you currently need the **recovery kit** + and **one passphrase**. +- Planned: an option to not store the key on the location at all. Then the kit is the only copy, + and losing it means losing every backup; that option will come with an explicit warning. - Up to **three passphrases** ("slots") can open the key, so one person leaving or dying does not lock the company out. A suggested setup: @@ -168,6 +182,11 @@ Contributions welcome via PR — add what you ran into with your own provider. On the backlog; none of these block the beta or 1.0.0, but know them before you rely on it: +- **Only the data directory is backed up.** Object storage as primary storage (`objectstore` in + `config.php`) is not supported: a backup refuses to start there. Folders mounted through the + External storage app are not in the data directory and are **not included**, without a warning. + Back them up separately. On the roadmap: reading files through Nextcloud's storage layer, which + covers both, with external mounts optional per mount (`PLAN.md` section 3.4). - **Very large instances** (well over 1 TB, millions of files): the blob index, cleanup, browsing a snapshot in the web interface and per-user export manifests are held in memory. Expect high memory use there; reports from such instances are very welcome. @@ -194,7 +213,7 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you | `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation | | `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots | | `backup:target:backends/add/edit/list/remove/append-only` | backup locations | -| `backup:run`, `backup:status`, `backup:list` | back up, status, restore points | +| `backup:run`, `backup:status`, `backup:list` | back up, status (runs of the last 7 days; `--all` for all), restore points | | `backup:cancel ` | stop a running backup | | `backup:browse`, `backup:restore` | browse and restore | | `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash | diff --git a/appinfo/info.xml b/appinfo/info.xml index 12118f4..c317bdb 100644 --- a/appinfo/info.xml +++ b/appinfo/info.xml @@ -4,12 +4,12 @@ ng_backup NG Backup Encrypted, incremental backups of your Nextcloud with restore from the web interface - - 0.9.6 + 0.9.7 agpl Dillard Blom NgBackup diff --git a/docs/HOWTO.md b/docs/HOWTO.md index 4ad750f..3f46b45 100644 --- a/docs/HOWTO.md +++ b/docs/HOWTO.md @@ -12,13 +12,16 @@ occ backup:key:kit --output=/var/tmp/ng_backup-kit.json occ backup:key:confirm -u --code= --confirm="Yes, I confirm" ``` -- The passphrase protects the backup. Keep it somewhere other than this server. +- The passphrase protects the backup: the location stores the key encrypted with it, so whoever + can read the location can try to guess it. Use at least 16 characters (five or more random + words work well) and keep it somewhere other than this server. - Optional: extra passphrases for other people with `occ backup:key:slot:add --label="..."`. Any one passphrase is enough to restore. Removing a slot later does not revoke access for someone who kept an old kit and that passphrase (see "Known limitations" in the README). - The recovery kit (`/var/tmp/ng_backup-kit.json` above) is also required for disaster recovery. The web user writes it there; then move it off the server as root, next to the passphrase, and delete the - copy in `/var/tmp`. Without the kit and a passphrase, the backup cannot be read. + copy in `/var/tmp`. The location holds the same encrypted key, but keep the kit anyway: if + someone damages or deletes the key on the location, the kit is the only way back in. ## 2. Add a backup location diff --git a/lib/Backend/GetsOneByOne.php b/lib/Backend/GetsOneByOne.php index 8385260..132bd99 100644 --- a/lib/Backend/GetsOneByOne.php +++ b/lib/Backend/GetsOneByOne.php @@ -7,7 +7,7 @@ declare(strict_types=1); namespace OCA\NgBackup\Backend; -/** IBackend::getMany() for a backend where reading files one after the other costs nothing extra. */ +/** IBackend::getMany() and getRanges() for a backend where reading one after the other costs nothing extra. */ trait GetsOneByOne { /** * @param list $paths @@ -27,4 +27,14 @@ trait GetsOneByOne { yield $path => $data; } } + + /** + * @param list $ranges + * @return \Generator + */ + public function getRanges(array $ranges): \Generator { + foreach ($ranges as $i => [$path, $offset, $length]) { + yield $i => $this->getRange($path, $offset, $length); + } + } } diff --git a/lib/Backend/IBackend.php b/lib/Backend/IBackend.php index 256e0d7..0d93ca7 100644 --- a/lib/Backend/IBackend.php +++ b/lib/Backend/IBackend.php @@ -35,6 +35,17 @@ interface IBackend { /** @return string bytes [$offset, $offset + $length) of the object (used to fetch single blobs from a pack) */ public function getRange(string $path, int $offset, int $length): string; + /** + * Many byte ranges, in the order given (blobs spread over many packs, such as the tree + * segments of a backup). A backend where every request costs a round trip can fetch them + * several at a time. + * + * @param list $ranges [path, offset, length] each + * @return \Generator contents by position in $ranges + * @throws BackendException for a range that cannot be read + */ + public function getRanges(array $ranges): \Generator; + public function exists(string $path): bool; /** diff --git a/lib/Backend/StorageBackend.php b/lib/Backend/StorageBackend.php index 427b624..c9c0ab3 100644 --- a/lib/Backend/StorageBackend.php +++ b/lib/Backend/StorageBackend.php @@ -19,6 +19,13 @@ final class StorageBackend implements IBackend { private bool $atomicPut; /** SFTP: read through SftpReadStream, which keeps several requests in flight. */ private bool $sftpReads; + /** S3: many small reads go straight to the storage's S3 client, several at a time. */ + private bool $s3Reads; + /** S3 requests in flight at once for getMany() and getRanges(). */ + public const S3_CONCURRENCY = 16; + /** Most ranges, and bytes, getRanges() holds in memory at once. */ + private const RANGES_LOT = 64; + private const RANGES_LOT_BYTES = 64 * 1048576; /** Seconds without a request after which an SFTP connection is checked before it is used. */ public const IDLE_CHECK = 60; private float $lastUsed; @@ -40,6 +47,7 @@ final class StorageBackend implements IBackend { self::validate($this->base, true); $this->atomicPut = $storage->instanceOfStorage(\OCA\Files_External\Lib\Storage\AmazonS3::class); $this->sftpReads = $storage->instanceOfStorage(\OCA\Files_External\Lib\Storage\SFTP::class); + $this->s3Reads = $storage instanceof \OCA\Files_External\Lib\Storage\AmazonS3; if ($createBase && $this->base !== '') { $this->mkdirs($this->base); } @@ -116,11 +124,19 @@ final class StorageBackend implements IBackend { return $fh; } - /** Over SFTP several at a time (SftpReadStream::readMany()); elsewhere one after the other. */ + /** + * Over SFTP and S3 several at a time (SftpReadStream::readMany(), s3Fetch()); elsewhere one + * after the other. + */ public function getMany(array $paths): \Generator { - foreach (array_chunk($paths, 8 * SftpReadStream::BATCH) as $chunk) { + $size = $this->s3Reads ? 16 * self::S3_CONCURRENCY : 8 * SftpReadStream::BATCH; + foreach (array_chunk($paths, $size) as $chunk) { $fast = null; - if ($this->sftpReads) { + if ($this->s3Reads) { + $this->idleCheck(); + $fast = $this->s3Fetch(array_map(fn (string $p): array => [$p, null], $chunk)); + $fast = $fast === null ? null : array_combine($chunk, array_map(fn (int $i): ?string => $fast[$i] ?? null, array_keys($chunk))); + } elseif ($this->sftpReads) { $abs = array_map(fn (string $p): string => $this->abs($p), $chunk); $fast = $this->call(fn () => SftpReadStream::readMany($this->storage, $abs)); $fast = $fast === null ? null : array_combine($chunk, array_map(fn (string $a): ?string => $fast[$a] ?? null, $abs)); @@ -145,6 +161,88 @@ final class StorageBackend implements IBackend { } } + /** Over S3 several at a time (s3Fetch()); elsewhere one after the other. */ + public function getRanges(array $ranges): \Generator { + $lot = []; + $bytes = 0; + foreach ($ranges as $i => $range) { + if ($lot !== [] && (count($lot) >= self::RANGES_LOT || $bytes + $range[2] > self::RANGES_LOT_BYTES)) { + yield from $this->rangesLot($lot); + $lot = []; + $bytes = 0; + } + $lot[$i] = $range; + $bytes += $range[2]; + } + if ($lot !== []) { + yield from $this->rangesLot($lot); + } + } + + /** + * @param array $lot + * @return \Generator + */ + private function rangesLot(array $lot): \Generator { + $fast = null; + if ($this->s3Reads) { + $this->idleCheck(); + $fast = $this->s3Fetch(array_map(fn (array $r): array => [$r[0], $r[1] . '-' . ($r[1] + $r[2] - 1)], $lot)); + } + foreach ($lot as $i => [$path, $offset, $length]) { + $data = $fast[$i] ?? null; + // not read in the batch, or not all of it: the usual way, which also reports why + yield $i => ($data !== null && strlen($data) === $length) ? $data : $this->getRange($path, $offset, $length); + } + } + + /** + * Read objects (or byte ranges of them, "first-last") straight through the S3 client of + * files_external's S3 storage, S3_CONCURRENCY at a time. Its own reads open a new HTTPS + * connection per object, after a HEAD request: one after the other that is about two reads a + * second, which made checking the index of a large repository take hours. What fails here + * is left out of the result, for the caller to read the usual way (which reports the error). + * + * @param array $requests [path, range or null] by key + * @return array|null contents by key; null when the storage is not as expected + * @psalm-suppress UndefinedClass the AWS SDK comes with Nextcloud, not with this app + */ + private function s3Fetch(array $requests): ?array { + try { + /** @psalm-suppress UndefinedMethod, UndefinedThisPropertyFetch */ + [$client, $bucket, $sse] = (function (): array { + $sse = method_exists($this, 'getServerSideEncryptionParameters') ? $this->getServerSideEncryptionParameters() + : (method_exists($this, 'getSSECParameters') ? $this->getSSECParameters() : []); + return [$this->getConnection(), $this->getBucket(), $sse]; + })->call($this->storage); + if (!$client instanceof \Aws\S3\S3Client || !is_string($bucket)) { + return null; + } + $results = []; + $keys = array_keys($requests); // the pool numbers results 0, 1, ...: mapped back below + $commands = (function () use ($requests, $client, $bucket, $sse): \Generator { + foreach (array_values($requests) as $n => [$path, $range]) { + $args = ['Bucket' => $bucket, 'Key' => trim($this->abs($path), '/')] + $sse; + if ($range !== null) { + $args['Range'] = 'bytes=' . $range; + } + yield $n => $client->getCommand('GetObject', $args); + } + })(); + (new \Aws\CommandPool($client, $commands, [ + 'concurrency' => self::S3_CONCURRENCY, + 'fulfilled' => function (mixed $result, int $n) use (&$results, $keys): void { + $results[$keys[$n]] = (string)$result['Body']; + }, + 'rejected' => static function (): void { + }, + ]))->promise()->wait(); + return $results; + } catch (\Throwable) { + return null; + } + } + public function getRange(string $path, int $offset, int $length): string { return $this->call(fn () => $this->readRange($path, $offset, $length)); } diff --git a/lib/BackgroundJob/BackupJob.php b/lib/BackgroundJob/BackupJob.php index cce6c89..4171fe4 100644 --- a/lib/BackgroundJob/BackupJob.php +++ b/lib/BackgroundJob/BackupJob.php @@ -34,7 +34,11 @@ final class BackupJob extends TimedJob { private LoggerInterface $logger, ) { parent::__construct($time); - $this->setInterval(300); + // Well below the usual 5-minute cron interval: the last run is stamped a moment after cron + // starts, so with 300 the next cron run came a few seconds too early and the job ran only + // every other time, idle for 6 of every 10 minutes during a backup. When nothing runs, the + // job only does a few cheap database checks. + $this->setInterval(60); $this->setTimeSensitivity(IJob::TIME_SENSITIVE); } diff --git a/lib/Command/KeyInit.php b/lib/Command/KeyInit.php index aad8aac..5051fc3 100644 --- a/lib/Command/KeyInit.php +++ b/lib/Command/KeyInit.php @@ -50,7 +50,7 @@ HELP); $pass = rtrim((string)@file_get_contents($file), "\r\n"); } else { $helper = $this->getHelper('question'); - $q = (new Question('Passphrase (min. 12 characters): '))->setHidden(true)->setHiddenFallback(false); + $q = (new Question('Passphrase (min. ' . \OCA\NgBackup\Service\KeyService::MIN_PASSPHRASE . ' characters): '))->setHidden(true)->setHiddenFallback(false); $pass = (string)$helper->ask($input, $output, $q); $again = (string)$helper->ask($input, $output, (new Question('Repeat passphrase: '))->setHidden(true)->setHiddenFallback(false)); if ($pass !== $again) { diff --git a/lib/Command/KeySlots.php b/lib/Command/KeySlots.php index 56deb92..aa59df7 100644 --- a/lib/Command/KeySlots.php +++ b/lib/Command/KeySlots.php @@ -88,7 +88,7 @@ HELP; return rtrim((string)@file_get_contents($file), "\r\n"); } $h = $this->getHelper('question'); - $p = (string)$h->ask($input, $output, (new Question('Passphrase (min. 12 characters): '))->setHidden(true)->setHiddenFallback(false)); + $p = (string)$h->ask($input, $output, (new Question('Passphrase (min. ' . \OCA\NgBackup\Service\KeyService::MIN_PASSPHRASE . ' characters): '))->setHidden(true)->setHiddenFallback(false)); if ($p !== (string)$h->ask($input, $output, (new Question('Repeat: '))->setHidden(true)->setHiddenFallback(false))) { throw new \InvalidArgumentException('The passphrases do not match'); } diff --git a/lib/Command/Status.php b/lib/Command/Status.php index be7247c..90491a1 100644 --- a/lib/Command/Status.php +++ b/lib/Command/Status.php @@ -14,9 +14,13 @@ use OCA\NgBackup\Service\VerifyService; use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Helper\Table; use Symfony\Component\Console\Input\InputInterface; +use Symfony\Component\Console\Input\InputOption; use Symfony\Component\Console\Output\OutputInterface; final class Status extends Command { + /** Runs of the last days that are listed by default (failed ones stay visible that long). */ + public const RECENT_DAYS = 7; + public function __construct( private KeyService $keys, private TargetService $targets, @@ -27,7 +31,8 @@ final class Status extends Command { } protected function configure(): void { - $this->setName('backup:status')->setDescription('Show the backup key, locations and recent runs'); + $this->setName('backup:status')->setDescription('Show the backup key, locations and recent runs') + ->addOption('all', null, InputOption::VALUE_NONE, 'List all runs, not only those of the last ' . self::RECENT_DAYS . ' days'); } protected function execute(InputInterface $input, OutputInterface $output): int { @@ -44,16 +49,22 @@ final class Status extends Command { } $output->writeln('Locations: ' . ($names === [] ? 'none' : implode(', ', $names))); $rows = []; - foreach ($this->runs->findRecent(10) as $r) { + $all = (bool)$input->getOption('all'); + $shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400); + foreach ($shown as $r) { $s = json_decode($r->getStats() ?? '{}', true)['files'] ?? []; $rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''), - date('Y-m-d H:i', $r->getStartedAt()), $r->getFinishedAt() ? gmdate('H:i:s', $r->getFinishedAt() - $r->getStartedAt()) : '', + date('Y-m-d H:i', $r->getStartedAt()), $r->getFinishedAt() ? \OCA\NgBackup\Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '', $s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '', isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')]; } if ($rows !== []) { (new Table($output))->setHeaders(['run', 'location', 'kind', 'status', 'started', 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render(); } + $older = $all ? 0 : $this->runs->countAll() - count($shown); + if ($older > 0) { + $output->writeln(sprintf('%d older run%s not shown (--all)', $older, $older === 1 ? '' : 's')); + } foreach ($this->verify->progress() as $targetId => $state) { $output->writeln(sprintf('Verify of %s on %s: %s', mb_strimwidth((string)($state['what'] ?? ''), 0, 40, '…'), $names[$targetId] ?? '#' . $targetId, VerifyService::describe($state))); diff --git a/lib/Controller/AdminApiController.php b/lib/Controller/AdminApiController.php index 26cd430..1fec3bb 100644 --- a/lib/Controller/AdminApiController.php +++ b/lib/Controller/AdminApiController.php @@ -51,14 +51,16 @@ final class AdminApiController extends Controller { parent::__construct(Application::APP_ID, $request); } - public function status(): JSONResponse { + /** @param bool $allRuns all runs instead of those of the last days (Status::RECENT_DAYS) */ + public function status(bool $allRuns = false): JSONResponse { $targets = []; foreach ($this->targets->list() as $t) { $targets[] = ['id' => $t->getId(), 'name' => $t->getName(), 'backend' => $t->getBackend(), 'path' => $t->getBasePath(), 'repositoryId' => $t->getRepositoryId(), 'createdAt' => $t->getCreatedAt(), 'appendOnly' => (bool)$t->getAppendOnly(), 'maxBytes' => $t->getMaxBytes(), 'storedBytes' => $this->alerts->lastStoredBytes($t)]; } - $runs = array_map(fn (Run $r) => $this->runJson($r), $this->runs->findRecent(15)); + $shown = $this->runs->findShown($allRuns ? null : time() - \OCA\NgBackup\Command\Status::RECENT_DAYS * 86400); + $runs = array_map(fn (Run $r) => $this->runJson($r), $shown); return new JSONResponse([ 'key' => ['initialized' => $this->keys->isInitialized(), 'fingerprint' => $this->keys->fingerprint(), 'confirmation' => $this->keys->recoveryKitConfirmation(), 'everConfirmed' => $this->keys->isInitialized() && $this->keys->everConfirmed(), @@ -68,6 +70,8 @@ final class AdminApiController extends Controller { 'confirmationPhrase' => KeyConfirm::CONFIRMATION, 'targets' => $targets, 'runs' => $runs, + 'olderRuns' => $allRuns ? 0 : $this->runs->countAll() - count($shown), + 'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS, 'schedule' => $this->appConfig->getValueString(Application::APP_ID, 'schedule_time', ''), ]); } diff --git a/lib/Db/RunMapper.php b/lib/Db/RunMapper.php index 162f4d7..0f5a300 100644 --- a/lib/Db/RunMapper.php +++ b/lib/Db/RunMapper.php @@ -73,6 +73,40 @@ final class RunMapper extends QBMapper { return $this->findEntities($qb); } + /** + * The runs to list, newest first: those started at or after $since, those still running, and + * at least the newest $atLeast, so the list is never empty after a quiet week. All runs when + * $since is null. + * + * @return list + */ + public function findShown(?int $since, int $atLeast = 5): array { + $qb = $this->db->getQueryBuilder(); + $qb->select('*')->from($this->getTableName())->orderBy('id', 'DESC'); + if ($since === null) { + return $this->findEntities($qb); + } + $qb->where($qb->expr()->orX( + $qb->expr()->gte('started_at', $qb->createNamedParameter($since, IQueryBuilder::PARAM_INT)), + $qb->expr()->eq('status', $qb->createNamedParameter(Run::RUNNING)), + )); + $runs = []; + foreach ([...$this->findEntities($qb), ...$this->findRecent($atLeast)] as $r) { + $runs[$r->getId()] = $r; + } + krsort($runs); + return array_values($runs); + } + + public function countAll(): int { + $qb = $this->db->getQueryBuilder(); + $qb->select($qb->func()->count('*', 'n'))->from($this->getTableName()); + $result = $qb->executeQuery(); + $n = (int)$result->fetchOne(); + $result->closeCursor(); + return $n; + } + /** @return list runs of $kind started at or after $since, newest first */ public function startedSince(int $targetId, string $kind, int $since): array { $qb = $this->db->getQueryBuilder(); diff --git a/lib/Format.php b/lib/Format.php new file mode 100644 index 0000000..6662287 --- /dev/null +++ b/lib/Format.php @@ -0,0 +1,18 @@ + 0 ? $days . 'd ' . $time : $time; + } +} diff --git a/lib/Job/BackupRun.php b/lib/Job/BackupRun.php index 5b6a300..2c9e136 100644 --- a/lib/Job/BackupRun.php +++ b/lib/Job/BackupRun.php @@ -140,21 +140,21 @@ final class BackupRun { $state['parentPos'] = $parent->position(); $state['stats'] = $stats; if ($finishedWalk) { + // The snapshot is written in a step of its own, with its own time: after a long + // run it reads back thousands of segments, and the run reports that it got here. $state['phase'] = 'finish'; } + return $state; } if ($state['phase'] === 'finish') { - $segments = $state['segments']; + // Every segment ends with a newline, so they read as one stream: fetched together, + // several at a time where the location allows it. $repo->putSnapshot($state['snapshot'], ['time' => $state['started'], 'parent' => $state['parent'], 'label' => $state['label'], 'roots' => array_keys($state['roots']), 'stats' => $state['stats'], 'warnings' => self::warnings($state)] + $state['meta'], - (function () use ($repo, $segments) { - foreach ($segments as $blobs) { - yield from $repo->readLines($blobs); - } - })()); + $repo->readLines(array_values(array_merge(...$state['segments'])))); $state['phase'] = 'commit'; } return $state; diff --git a/lib/Repository/PackReaderGuard.php b/lib/Repository/PackReaderGuard.php index 0679298..99577e6 100644 --- a/lib/Repository/PackReaderGuard.php +++ b/lib/Repository/PackReaderGuard.php @@ -38,6 +38,11 @@ final class PackReaderGuard implements IBackend { return $this->inner->getRange($path, $offset, $length); } + public function getRanges(array $ranges): \Generator { + $this->reader->close(); + return $this->inner->getRanges($ranges); + } + public function exists(string $path): bool { $this->reader->close(); return $this->inner->exists($path); diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index 188266c..9eb0cc7 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -28,8 +28,14 @@ final class Repository { public const FORMAT = 1; public const BLOB_SIZE = 4 * 1048576; private const FLAG_DEFLATE = 1; + /** Blobs from this size on get a quick compression test before the full compression. */ + private const DEFLATE_PROBE_FROM = 256 * 1024; + /** Bytes of each of the three slices (start, middle, end) in that test. */ + private const DEFLATE_PROBE_SLICE = 32 * 1024; /** Reads per blob in a deep verify before it is reported as unreadable (not corrupted). */ private const VERIFY_READ_ATTEMPTS = 3; + /** Largest read that loadBlobs() makes of blobs that follow each other in a pack. */ + private const MERGED_READ_MAX = 16 * 1048576; private StreamCipher $cipher; private BlobIndex $index; @@ -288,8 +294,8 @@ final class Repository { */ public function readLines(array $blobIds): \Generator { $carry = ''; - foreach ($blobIds as $id) { - $data = $carry . $this->loadBlob($id); + foreach ($this->loadBlobs($blobIds) as $blob) { + $data = $carry . $blob; $lines = explode("\n", $data); $carry = array_pop($lines); yield from $lines; @@ -701,7 +707,7 @@ final class Repository { } $flags = 0; $payload = $data; - $deflated = gzdeflate($data, 6); + $deflated = self::worthDeflating($data) ? gzdeflate($data, 6) : false; if ($deflated !== false && strlen($deflated) < strlen($data) * 0.95) { $payload = $deflated; $flags |= self::FLAG_DEFLATE; @@ -711,6 +717,67 @@ final class Repository { return $id; } + /** + * Photos, video and archives are compressed already: deflating them takes most of a backup's + * CPU time (about 24 MiB/s) only to be thrown away, as it saves less than the 5% that keeps + * it. A fast deflate of three small slices tells first; below 10% savings there, the full + * deflate is skipped. + */ + private static function worthDeflating(string $data): bool { + $n = strlen($data); + if ($n < self::DEFLATE_PROBE_FROM) { + return true; + } + $k = self::DEFLATE_PROBE_SLICE; + $probe = substr($data, 0, $k) . substr($data, intdiv($n - $k, 2), $k) . substr($data, $n - $k); + $deflated = gzdeflate($probe, 1); + return $deflated !== false && strlen($deflated) * 10 < strlen($probe) * 9; + } + + /** + * Blobs spread over many packs (the tree segments of a run, one per step), through + * IBackend::getRanges(): on a remote location one at a time is a round trip each, which for + * a long run made writing its snapshot take a quarter of an hour. Blobs that follow each + * other in a pack are read as one range. + * + * @param list $ids + * @return \Generator verified blob contents, in order + */ + private function loadBlobs(array $ids): \Generator { + /** @var list $ranges [path, offset, length] of each read */ + $ranges = []; + /** @var list> $parts per read: [blob id, offset in the read, length] */ + $parts = []; + // the read being extended + $curPath = null; + $curOffset = 0; + $curLength = 0; + $curParts = []; + foreach ($ids as $id) { + [$pack, $offset, $length] = $this->index->get($id); + $path = 'packs/' . substr($pack, 0, 2) . '/' . $pack; + if ($curPath === $path && $curOffset + $curLength === $offset && $curLength + $length <= self::MERGED_READ_MAX) { + $curParts[] = [$id, $curLength, $length]; + $curLength += $length; + continue; + } + if ($curPath !== null) { + $ranges[] = [$curPath, $curOffset, $curLength]; + $parts[] = $curParts; + } + [$curPath, $curOffset, $curLength, $curParts] = [$path, $offset, $length, [[$id, 0, $length]]]; + } + if ($curPath !== null) { + $ranges[] = [$curPath, $curOffset, $curLength]; + $parts[] = $curParts; + } + foreach ($this->backend->getRanges($ranges) as $i => $data) { + foreach ($parts[$i] as [$id, $at, $length]) { + yield $this->decodeBlob($id, substr($data, $at, $length)); + } + } + } + private function loadBlob(string $id): string { [$pack, $offset, $length] = $this->index->get($id); return $this->decodeBlob($id, $this->packReader->read($pack, $offset, $length)); diff --git a/lib/Service/BackupService.php b/lib/Service/BackupService.php index a5fef3b..e6c39ea 100644 --- a/lib/Service/BackupService.php +++ b/lib/Service/BackupService.php @@ -225,6 +225,9 @@ final class BackupService { } else { $state['files']['estimate'] ??= $this->estimateBytes($run, $state['files']); $state['files'] = BackupRun::step($repo, $state['files'], $deadline); + if (in_array($state['files']['phase'], ['finish', 'commit'], true)) { + $run->setPhase('commit'); // all files read: writing and recording the snapshot + } if ($state['files']['phase'] === 'commit') { // Catalog write under the catalog lock, from a repository opened inside it: a per-user // export or restore may have written the catalog since this step opened $repo. diff --git a/lib/Service/KeyService.php b/lib/Service/KeyService.php index fca1d40..cf5c832 100644 --- a/lib/Service/KeyService.php +++ b/lib/Service/KeyService.php @@ -31,6 +31,11 @@ final class KeyService { private const KEY_KIT_VERSION = 'recovery_kit_version'; private const KEY_KIT_CODE = 'recovery_kit_code'; public const DEFAULT_DELETE_DELAY = 7; + /** + * The location holds the passphrase-wrapped key too, so whoever can read it can try + * passphrases offline: the passphrase is what really protects the backups. + */ + public const MIN_PASSPHRASE = 16; private ?KeyRing $cached = null; @@ -56,9 +61,7 @@ final class KeyService { if ($this->isInitialized()) { throw new \RuntimeException('A backup key already exists'); } - if (strlen($passphrase) < 12) { - throw new \InvalidArgumentException('The passphrase must be at least 12 characters'); - } + self::checkPassphrase($passphrase); $keys = KeyRing::generate(); $this->appConfig->setValueString(Application::APP_ID, self::KEY_MASTER, $this->crypto->encrypt(base64_encode($keys->exportMasterKey())), true, true); @@ -234,8 +237,8 @@ final class KeyService { } private static function checkPassphrase(string $passphrase): void { - if (strlen($passphrase) < 12) { - throw new \InvalidArgumentException('The passphrase must be at least 12 characters'); + if (strlen($passphrase) < self::MIN_PASSPHRASE) { + throw new \InvalidArgumentException('The passphrase must be at least ' . self::MIN_PASSPHRASE . ' characters'); } } diff --git a/lib/Service/VerifyService.php b/lib/Service/VerifyService.php index 8d7486a..ebec854 100644 --- a/lib/Service/VerifyService.php +++ b/lib/Service/VerifyService.php @@ -64,7 +64,7 @@ final class VerifyService { $age < 120 ? $age . ' s' : intdiv($age, 60) . ' min', $age > 600 ? ' (no progress for a while: the process may have stopped)' : ''); } - $took = gmdate('H:i:s', max(0, (int)($s['finished'] ?? 0) - (int)($s['started'] ?? 0))); + $took = \OCA\NgBackup\Format::duration((int)($s['finished'] ?? 0) - (int)($s['started'] ?? 0)); return match ($s['status'] ?? '') { 'ok' => sprintf('%s OK on %s (took %s): %s files, %s blobs%s', $kind, date('Y-m-d H:i', (int)$s['finished']), $took, number_format((int)($s['files'] ?? 0)), number_format((int)($s['blobs'] ?? 0)), $deep ? ', ' . $size((int)($s['bytes'] ?? 0)) . ' decrypted' : ''), diff --git a/psalm-baseline.xml b/psalm-baseline.xml index 984f91e..b243ec9 100644 --- a/psalm-baseline.xml +++ b/psalm-baseline.xml @@ -218,16 +218,11 @@ - - - - - diff --git a/screenshots/1-backups.png b/screenshots/1-backups.png index 9f15258..959607d 100644 Binary files a/screenshots/1-backups.png and b/screenshots/1-backups.png differ diff --git a/screenshots/2-restore.png b/screenshots/2-restore.png index c4ee2b9..d8928b7 100644 Binary files a/screenshots/2-restore.png and b/screenshots/2-restore.png differ diff --git a/screenshots/3-settings.png b/screenshots/3-settings.png index 2fea888..2cf98da 100644 Binary files a/screenshots/3-settings.png and b/screenshots/3-settings.png differ diff --git a/src/AdminSettings.vue b/src/AdminSettings.vue index c66dcf7..eae0342 100644 --- a/src/AdminSettings.vue +++ b/src/AdminSettings.vue @@ -6,6 +6,10 @@
+

+ {{ t('ng_backup', 'NG Backup is in beta. We would like to hear how it works for you, also when everything went fine.') }} + {{ t('ng_backup', 'Share your experience ↗') }} +

{{ error }} @@ -22,7 +26,7 @@