diff --git a/CHANGELOG.md b/CHANGELOG.md index 139f25a..47868a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,68 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th beta/daily update channel, which would hide the app from most admins. The stability stage is instead carried by the app description, the git tag, the release, and this file. +## [0.9.5-beta.6] - 2026-10-12 + +Bugfix release for SFTP locations and large repositories. Tested on a production instance with a +1 TB repository on a Hetzner Storage Box over SFTP: a check in 5.5 minutes, a deep check of all +316,529 blobs (815 GiB) in 11 hours, and an incremental backup in under 8 minutes. + +### Added +- Progress of a running backup in the web interface and `occ backup:status`, e.g. + "running (files) 68%". The total is the size of the previous snapshot; for a first backup it + is estimated from Nextcloud's file cache (users' files, versions and trash) and shown as + "~68%". +- Edit a backup location (web interface and `occ backup:target:edit`): change the password, host, + folder, connection type or name without losing its history. Before this the only option was a + new location, which starts an empty repository and, with a retention lock at the provider, + keeps paying for the old one. The new settings are only saved when they lead to the same + repository; stored passwords are never sent to the browser, and an empty password field keeps + the stored one. Administrators are notified of the change. +- `occ backup:verify` reports how far it is every 30 seconds, and `occ backup:status` shows the + progress of a running check and how the last check of each location ended. On a large + repository a check takes hours, a deep one days. + +### Changed +- Checking and opening a large repository over SFTP is much faster: the index files (one per + pack, 27,547 for 1 TB) are downloaded 32 at a time instead of one after the other, 114 instead + of 6 files per second to a Hetzner Storage Box (about 4 minutes instead of 77 for 1 TB). A check + of a repository with more than 256 packs also lists the pack folders (256 requests) instead of + asking for every pack. +- App store listing: the author's e-mail address is shown. +- App store listing: the homepage link points to the GitHub repository instead of back to the + app store page itself. +- The notification for a location without a successful backup read "No successful backup to … + since never" when there had never been one; it now reads "No successful backup to … yet". + +### Fixed +- 0.9.4 could not open a repository at the root of an SFTP location (empty path): its faster + SFTP reads sent their first request before phpseclib had opened the SFTP channel, which only + happens on the first SFTP call after login. Every backup then failed with "No repository at + this location". The channel is now opened first. +- A check (`occ backup:verify`) did not renew its hold on the location while it downloaded the + index; on a large repository that takes longer than the hold lasts, so a weekly cleanup could + have started during the check. +- A long first backup could end as failed although its snapshot was complete: when its final + step took longer than the step's lease (150 s with `occ backup:run`), the background job took + the run over, finished it a second time and failed on the snapshot it had already recorded, + overwriting the result. The lease now lasts at least 15 minutes, a step writes the run only if + no other process wrote it since the step loaded it (in one database statement), and recording + a snapshot that is already recorded is no longer an error. +- `occ backup:verify` on a large repository failed with "SFTP connection failed while opening + snapshots/…" or "Not found: snapshots/…". A Hetzner Storage Box drops an SFTP connection that + carried a lot of data (some 40 MB) and then stayed quiet for a few minutes, and the check went + quiet after downloading the index: it refreshed the local index cache, first in 29 minutes (a + pass over all blobs for every pack: 27,547 packs, 400,000 blobs for 1 TB), then still in more + than 6 minutes (a database transaction per pack on PostgreSQL). Nextcloud's SFTP storage keeps + using a dropped connection, on which every file looks missing. Now: + - an SFTP connection that was not used for a minute is checked first, and one that turns out + to be dead after a failed request is replaced, once per request; uploads, moves and deletes + are only checked first, never repeated; + - the index cache is written in one transaction (27,547 packs in 11 seconds on a test server, + against about a minute row by row); + - after a check the cache holds the index files as they are: before, a blob that was in two + packs was only cached for one of them. + ## [0.9.4-beta.5] - 2026-10-09 Bugfix release: faster restores. @@ -154,6 +216,7 @@ and not implemented yet. - See the "Storage backend experiences" section of the README for provider-specific quirks found so far (e.g. Hetzner Storage Box sub-accounts). +[0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6 [0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5 [0.9.3-beta.4]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.3-beta.4 [0.9.2-beta.3]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.2-beta.3 diff --git a/README.md b/README.md index c7bc7e7..3450db9 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on see "Storage backend experiences" below — that requirement comes from files_external, not from ng_backup, and applies to any Nextcloud app that writes to an SMB share.) -**Status: beta (0.9.4).** Tested thoroughly, usable alongside your current backup, but not yet a +**Status: beta (0.9.5).** Tested thoroughly, usable alongside your current backup, but not yet a drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and `docs/HOWTO.md` for a step-by-step guide from setup to a tested restore. @@ -193,7 +193,7 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you |---|---| | `backup:key:init`, `backup:key:kit`, `backup:key:confirm` | create the key, recovery kit, confirmation | | `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots | -| `backup:target:backends/add/list/remove/append-only` | backup locations | +| `backup:target:backends/add/edit/list/remove/append-only` | backup locations | | `backup:run`, `backup:status`, `backup:list` | back up, status, restore points | | `backup:browse`, `backup:restore` | browse and restore | | `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash | diff --git a/appinfo/info.xml b/appinfo/info.xml index 26e0f1d..83fc64f 100644 --- a/appinfo/info.xml +++ b/appinfo/info.xml @@ -9,15 +9,15 @@ directory to S3, SFTP, WebDAV, FTP, SMB or a local path, with restore of files a single users (via user_migration) and of the whole server onto a fresh installation. Tested thoroughly and usable alongside your current backup, but not yet a drop-in replacement for it. See the README for known limitations and the recovery kit before relying on it.]]> - 0.9.4 + 0.9.5 agpl - Dillard Blom + Dillard Blom NgBackup https://github.com/dillardblom/ng_backup/blob/main/docs/HOWTO.md tools - https://apps.nextcloud.com/apps/ng_backup + https://github.com/dillardblom/ng_backup https://github.com/dillardblom/ng_backup/issues https://github.com/dillardblom/ng_backup https://raw.githubusercontent.com/dillardblom/ng_backup/main/screenshots/1-backups.png @@ -37,6 +37,7 @@ See the README for known limitations and the recovery kit before relying on it.] OCA\NgBackup\Command\KeyConfirm OCA\NgBackup\Command\TargetBackends OCA\NgBackup\Command\TargetAdd + OCA\NgBackup\Command\TargetEdit OCA\NgBackup\Command\TargetList OCA\NgBackup\Command\TargetRemove OCA\NgBackup\Command\RunBackup diff --git a/appinfo/routes.php b/appinfo/routes.php index 018dc23..10e7a5d 100644 --- a/appinfo/routes.php +++ b/appinfo/routes.php @@ -18,6 +18,8 @@ return [ ['name' => 'AdminApi#untrash', 'url' => '/api/targets/{targetId}/trash/{snapshotId}', 'verb' => 'POST'], ['name' => 'AdminApi#backends', 'url' => '/api/backends', 'verb' => 'GET'], ['name' => 'AdminApi#addTarget', 'url' => '/api/targets', 'verb' => 'POST'], + ['name' => 'AdminApi#targetSettings', 'url' => '/api/targets/{id}/settings', 'verb' => 'GET'], + ['name' => 'AdminApi#updateTarget', 'url' => '/api/targets/{id}', 'verb' => 'PUT'], ['name' => 'AdminApi#removeTarget', 'url' => '/api/targets/{id}', 'verb' => 'DELETE'], ['name' => 'AdminApi#testTarget', 'url' => '/api/targets/{id}/test', 'verb' => 'POST'], ['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'], diff --git a/docs/HOWTO.md b/docs/HOWTO.md index 6256481..197ffdc 100644 --- a/docs/HOWTO.md +++ b/docs/HOWTO.md @@ -50,6 +50,14 @@ occ backup:target:list occ backup:status ``` +Changed password, new host name, or moved the repository? Edit the location instead of adding a +new one, so its history stays (in the web interface: **Edit** next to the location). The new +settings must lead to the same repository; options you do not pass keep their value: + +``` +occ backup:target:edit nas -o password= +``` + ## 3. Run the first backup ``` diff --git a/lib/Backend/GetsOneByOne.php b/lib/Backend/GetsOneByOne.php new file mode 100644 index 0000000..8385260 --- /dev/null +++ b/lib/Backend/GetsOneByOne.php @@ -0,0 +1,30 @@ + $paths + * @return \Generator + */ + public function getMany(array $paths): \Generator { + foreach ($paths as $path) { + $fh = $this->get($path); + try { + $data = stream_get_contents($fh); + } finally { + fclose($fh); + } + if ($data === false) { + throw new BackendException('Read failed for ' . $path); + } + yield $path => $data; + } + } +} diff --git a/lib/Backend/IBackend.php b/lib/Backend/IBackend.php index 2d7eb6d..256e0d7 100644 --- a/lib/Backend/IBackend.php +++ b/lib/Backend/IBackend.php @@ -22,6 +22,16 @@ interface IBackend { /** @return resource readable stream of the whole object */ public function get(string $path); + /** + * Many small objects whole, in the order given (the repository's index files). A backend + * where every request costs a round trip can fetch them several at a time. + * + * @param list $paths + * @return \Generator contents by path + * @throws BackendException for an object that cannot be read + */ + public function getMany(array $paths): \Generator; + /** @return string bytes [$offset, $offset + $length) of the object (used to fetch single blobs from a pack) */ public function getRange(string $path, int $offset, int $length): string; diff --git a/lib/Backend/LocalBackend.php b/lib/Backend/LocalBackend.php index 1a577bb..266b517 100644 --- a/lib/Backend/LocalBackend.php +++ b/lib/Backend/LocalBackend.php @@ -8,6 +8,8 @@ declare(strict_types=1); namespace OCA\NgBackup\Backend; final class LocalBackend implements IBackend { + use GetsOneByOne; + public function __construct( private string $root, ) { diff --git a/lib/Backend/SftpReadStream.php b/lib/Backend/SftpReadStream.php index c982991..cd91855 100644 --- a/lib/Backend/SftpReadStream.php +++ b/lib/Backend/SftpReadStream.php @@ -28,6 +28,10 @@ final class SftpReadStream { private const PROTOCOL = 'ngbsftpread'; /** Requests in flight once a stream reads sequentially; it starts at one and doubles. */ public const DEPTH = 8; + /** Files read at once by readMany(): each holds a handle on the server while its batch runs. */ + public const BATCH = 32; + /** Larger files are left to the usual read by readMany(), which holds every file in memory. */ + private const MANY_MAX_BYTES = 16 * 1048576; /** OpenSSH and the Hetzner Storage Box both answer at most 255 KiB per READ. */ private const CHUNK = 261120; /** Every SFTP server must handle packets of this size (draft-ietf-secsh-filexfer-02, section 3). */ @@ -40,6 +44,7 @@ final class SftpReadStream { private const FXP_HANDLE = 102; private const FXP_DATA = 103; private const FXF_READ = 1; + private const FX_OK = 0; private const FX_EOF = 1; /** @@ -50,11 +55,20 @@ final class SftpReadStream { */ private static ?\WeakMap $chunks = null; private static int $requestId = 0x4e470000; + /** + * Connections whose server did not confirm closing a batch's files: readMany() is not used on + * them again, so handles cannot pile up. + * + * @var \WeakMap|null + */ + private static ?\WeakMap $noBatches = null; /** Streams in a row that failed; at MAX_FAILURES only the storage's own stream is used. */ private static int $failures = 0; public const MAX_FAILURES = 3; /** Set by stream_open() when the connection failed rather than the file. */ private static bool $openBroke = false; + /** Why the last stream_open() failed on the connection. */ + private static string $openCause = ''; /** @var resource|null set by PHP for stream wrappers */ public $context; @@ -101,14 +115,8 @@ final class SftpReadStream { return null; } try { - if (!is_callable([$storage, 'getConnection']) || !is_callable([$storage, 'getRoot'])) { - return null; - } - $sftp = $storage->getConnection(); - // is_callable and get_object_vars from here only see what is public: phpseclib 3 has - // methods of these names neither public nor at all. - if (!is_object($sftp) || !is_callable([$sftp, '_send_sftp_packet']) || !is_callable([$sftp, '_get_sftp_packet']) - || !is_callable([$sftp, '_realpath']) || !array_key_exists('packet_type', get_object_vars($sftp))) { + $sftp = self::connection($storage); + if ($sftp === null) { return null; } if (!in_array(self::PROTOCOL, stream_get_wrappers(), true)) { @@ -129,14 +137,191 @@ final class SftpReadStream { return null; } if ($fh === false && self::openBroke()) { - throw new BackendException('SFTP connection failed while opening ' . $path); + throw new BackendException('SFTP connection failed while opening ' . $path . ': ' . self::$openCause); } return $fh; } + /** + * The storage's phpseclib connection, with its SFTP channel open, when it offers what these + * streams need; null otherwise. + */ + private static function connection(object $storage): ?object { + if (!is_callable([$storage, 'getConnection']) || !is_callable([$storage, 'getRoot'])) { + return null; + } + $sftp = $storage->getConnection(); + // is_callable and get_object_vars from here only see what is public: phpseclib 3 has + // methods of these names neither public nor at all. + if (!is_object($sftp) || !is_callable([$sftp, '_send_sftp_packet']) || !is_callable([$sftp, '_get_sftp_packet']) + || !is_callable([$sftp, '_realpath']) || !is_callable([$sftp, 'pwd']) + || !array_key_exists('packet_type', get_object_vars($sftp))) { + return null; + } + // phpseclib opens the SFTP channel on its first SFTP call, not at login: on a fresh + // connection a packet sent directly would go nowhere. pwd() opens it like any call does. + if ($sftp->pwd() === false) { + return null; + } + return $sftp; + } + + /** + * Read many small files whole, BATCH at a time: the OPEN, READ and CLOSE requests of a batch + * all go out before their answers are read, so a batch costs a few round trips instead of + * four per file. Meant for files of up to a few hundred KiB (the repository's index files): + * every file is held in memory. + * + * @param list $paths below the storage's root + * @return array|null contents by path; null for a file this could not + * read (missing, or a server error), so the caller + * reads it the usual way; null for all when the + * connection does not offer what this needs + * @throws BackendException when the connection failed while requests were out: the storage's + * own stream must not be tried on it + */ + public static function readMany(object $storage, array $paths): ?array { + if (self::$failures >= self::MAX_FAILURES) { + return null; + } + try { + $sftp = self::connection($storage); + $root = $sftp === null ? '' : rtrim((string)$storage->getRoot(), '/'); + } catch (\Throwable) { + self::$failures++; + return null; + } + if ($sftp === null || self::$noBatches?->offsetExists($sftp) === true) { + return null; + } + $result = array_fill_keys($paths, null); + foreach (array_chunk($paths, self::BATCH) as $batch) { + try { + [$read, $closed] = self::readBatch($sftp, $root, $batch); + } catch (\Throwable $e) { + self::$failures++; + // Answers to requests already sent may still arrive, and files_external would take + // them for its own: end the connection rather than leave it out of step. + self::disconnect($sftp); + throw $e instanceof BackendException ? $e : new BackendException('SFTP connection failed while reading ' . count($batch) . ' files: ' . $e->getMessage(), 0, $e); + } + foreach ($read as $path => $contents) { + $result[$path] = $contents; + } + if (!$closed) { + // The server did not confirm closing these files: no more batches on this + // connection, so handles cannot pile up. The rest is read the usual way. + self::noMoreBatches($sftp); + break; + } + } + self::$failures = 0; + return $result; + } + + private static function noMoreBatches(object $sftp): void { + self::$noBatches ??= new \WeakMap(); + self::$noBatches[$sftp] = true; + } + + private static function disconnect(object $sftp): void { + try { + if (is_callable([$sftp, 'disconnect'])) { + $sftp->disconnect(); + } + } catch (\Throwable) { + } + } + + /** Requests of one kind for a whole batch, then their answers in order. @return list */ + private static function exchange(object $sftp, int $type, array $packets): array { + $ids = []; + foreach ($packets as $packet) { + $id = self::nextId(); + if (!$sftp->_send_sftp_packet($type, $packet, $id)) { + throw new BackendException('SFTP connection failed while sending a request'); + } + $ids[] = $id; + } + $answers = []; + foreach ($ids as $id) { + $response = $sftp->_get_sftp_packet($id); + if (!is_string($response)) { + throw new BackendException('SFTP connection failed while waiting for an answer'); + } + $answers[] = [$sftp->packet_type, $response]; + } + return $answers; + } + + /** + * @param list $batch + * @return array{0: array, 1: bool} the files read whole, and whether the + * server confirmed closing every handle + */ + private static function readBatch(object $sftp, string $root, array $batch): array { + $opens = []; + $openPaths = []; + foreach ($batch as $path) { + $remote = $sftp->_realpath($root . '/' . ltrim($path, '/')); + if (is_string($remote)) { + $opens[] = pack('Na*N2', strlen($remote), $remote, self::FXF_READ, 0); + $openPaths[] = $path; + } + // a path that cannot be resolved is left to the caller's usual read + } + $handles = []; // path => handle + foreach (self::exchange($sftp, self::FXP_OPEN, $opens) as $i => [$type, $response]) { + if ($type === self::FXP_HANDLE) { + $handles[$openPaths[$i]] = substr($response, 4); + } + // any other answer (missing, no permission): the caller reads it the usual way + } + $data = array_fill_keys(array_keys($handles), ''); + $reading = $handles; + while ($reading !== []) { + $length = self::chunk($sftp); + $reads = []; + foreach ($reading as $path => $handle) { + $offset = strlen($data[$path]); + $reads[] = pack('Na*N3', strlen($handle), $handle, intdiv($offset, 4294967296), $offset % 4294967296, $length); + } + $paths = array_keys($reading); + foreach (self::exchange($sftp, self::FXP_READ, $reads) as $i => [$type, $response]) { + $path = $paths[$i]; + $declared = strlen($response) >= 4 ? unpack('N', $response) : false; + if ($type === self::FXP_DATA && is_array($declared) && $declared[1] > 0 && $declared[1] === strlen($response) - 4 + && $declared[1] <= $length && strlen($data[$path]) + $declared[1] <= self::MANY_MAX_BYTES) { + $data[$path] .= substr($response, 4); // more may follow: read on from there + continue; + } + unset($reading[$path]); + $status = $type === self::FXP_STATUS && is_array($declared) ? $declared[1] : null; + if ($status !== self::FX_EOF) { + // a server error, a malformed answer or a file too large for this: the caller + // reads it the usual way + unset($data[$path]); + } + } + } + $closes = []; + foreach ($handles as $handle) { + $closes[] = pack('Na*', strlen($handle), $handle); + } + $closed = true; + foreach (self::exchange($sftp, self::FXP_CLOSE, $closes) as [$type, $response]) { + $status = $type === self::FXP_STATUS && strlen($response) >= 4 ? unpack('N', $response) : false; + if (!is_array($status) || $status[1] !== self::FX_OK) { + $closed = false; + } + } + return [$data, $closed]; + } + /** Test hook: forget the learned READ sizes and earlier failures. */ public static function reset(): void { self::$chunks = null; + self::$noBatches = null; self::$failures = 0; } @@ -156,14 +341,14 @@ final class SftpReadStream { } $id = self::nextId(); if (!$this->sftp->_send_sftp_packet(self::FXP_OPEN, pack('Na*N2', strlen($remote), $remote, self::FXF_READ, 0), $id)) { - return $this->connectionFailed(); + return $this->connectionFailed('the request could not be sent'); } $response = $this->sftp->_get_sftp_packet($id); - } catch (\Throwable) { - return $this->connectionFailed(); + } catch (\Throwable $e) { + return $this->connectionFailed(get_class($e) . ': ' . $e->getMessage()); } if (!is_string($response)) { - return $this->connectionFailed(); + return $this->connectionFailed('no answer'); } if ($this->sftp->packet_type !== self::FXP_HANDLE) { return false; // an answer, but no handle: the file cannot be opened @@ -173,12 +358,39 @@ final class SftpReadStream { } /** Not a missing file but a connection that does not behave as expected. */ - private function connectionFailed(): bool { + private function connectionFailed(string $cause): bool { self::$failures++; self::$openBroke = true; + self::$openCause = $cause . self::connectionState($this->sftp); + // unknown state: closed, so that nothing else uses it (and the backend can build a new one) + self::disconnect($this->sftp); return false; } + /** What phpseclib knows about the connection, for an error message. */ + private static function connectionState(object $sftp): string { + $state = []; + try { + if (is_callable([$sftp, 'isConnected'])) { + $state[] = 'connected: ' . ($sftp->isConnected() ? 'yes' : 'no'); + } + if (is_callable([$sftp, 'getLastError'])) { + $last = (string)$sftp->getLastError(); + if ($last !== '') { + $state[] = 'ssh: ' . $last; + } + } + if (is_callable([$sftp, 'getLastSFTPError'])) { + $last = (string)$sftp->getLastSFTPError(); + if ($last !== '') { + $state[] = 'sftp: ' . $last; + } + } + } catch (\Throwable) { + } + return $state === [] ? '' : ' (' . implode('; ', $state) . ')'; + } + /** Whether the last stream_open() failed on the connection; a call, as fopen() sets it. */ private static function openBroke(): bool { return self::$openBroke; @@ -304,6 +516,9 @@ final class SftpReadStream { } catch (\Throwable) { $this->broken = true; } + if ($this->broken) { + self::disconnect($this->sftp); + } } $this->handle = ''; } @@ -385,19 +600,20 @@ final class SftpReadStream { private function drain(): void { $inFlight = $this->inFlight; $this->inFlight = []; - if ($this->broken) { - return; - } try { - foreach ($inFlight as [$id]) { + foreach ($this->broken ? [] : $inFlight as [$id]) { if (!is_string($this->sftp->_get_sftp_packet($id))) { $this->broken = true; - return; + break; } } } catch (\Throwable) { $this->broken = true; } + if ($this->broken) { + // unknown state: closed, so that nothing else uses it (and the backend can build a new one) + self::disconnect($this->sftp); + } } private static function chunk(object $sftp): int { diff --git a/lib/Backend/StorageBackend.php b/lib/Backend/StorageBackend.php index 95672f3..2ad83c4 100644 --- a/lib/Backend/StorageBackend.php +++ b/lib/Backend/StorageBackend.php @@ -19,16 +19,28 @@ final class StorageBackend implements IBackend { private bool $atomicPut; /** SFTP: read through SftpReadStream, which keeps several requests in flight. */ private bool $sftpReads; + /** Seconds without a request after which an SFTP connection is checked before it is used. */ + public const IDLE_CHECK = 60; + private float $lastUsed; + /** Times the storage was built anew after its connection died (for tests and diagnostics). */ + public int $reconnects = 0; + /** + * @param (\Closure():IStorage)|null $reconnect builds the same storage anew, with a new + * connection + */ public function __construct( private IStorage $storage, private string $base = '', + bool $createBase = true, + private ?\Closure $reconnect = null, ) { + $this->lastUsed = microtime(true); $this->base = trim($base, '/'); self::validate($this->base, true); $this->atomicPut = $storage->instanceOfStorage(\OCA\Files_External\Lib\Storage\AmazonS3::class); $this->sftpReads = $storage->instanceOfStorage(\OCA\Files_External\Lib\Storage\SFTP::class); - if ($this->base !== '') { + if ($createBase && $this->base !== '') { $this->mkdirs($this->base); } } @@ -39,6 +51,8 @@ final class StorageBackend implements IBackend { } public function put(string $path, $stream): void { + // not repeated on a new connection: the stream is (partly) read + $this->idleCheck(); $expected = self::remaining($stream); $target = $this->abs($path); $this->mkdirs(dirname($target)); @@ -82,6 +96,11 @@ final class StorageBackend implements IBackend { } public function get(string $path) { + return $this->call(fn () => $this->open($path)); + } + + /** @return resource */ + private function open(string $path) { if ($this->sftpReads) { $fh = SftpReadStream::open($this->storage, $this->abs($path)); if (is_resource($fh)) { @@ -97,7 +116,40 @@ final class StorageBackend implements IBackend { return $fh; } + /** Over SFTP several at a time (SftpReadStream::readMany()); elsewhere one after the other. */ + public function getMany(array $paths): \Generator { + foreach (array_chunk($paths, 8 * SftpReadStream::BATCH) as $chunk) { + $fast = null; + if ($this->sftpReads) { + $abs = array_map(fn (string $p): string => $this->abs($p), $chunk); + $fast = $this->call(fn () => SftpReadStream::readMany($this->storage, $abs)); + $fast = $fast === null ? null : array_combine($chunk, array_map(fn (string $a): ?string => $fast[$a] ?? null, $abs)); + } + foreach ($chunk as $path) { + $data = $fast[$path] ?? null; + if ($data === null) { + // not read in the batch (or no batches here): the usual way, which also + // reports a missing file as such + $fh = $this->get($path); + try { + $data = stream_get_contents($fh); + } finally { + fclose($fh); + } + if ($data === false) { + throw new BackendException('Read failed for ' . $path); + } + } + yield $path => $data; + } + } + } + public function getRange(string $path, int $offset, int $length): string { + return $this->call(fn () => $this->readRange($path, $offset, $length)); + } + + private function readRange(string $path, int $offset, int $length): string { $fh = $this->get($path); try { if (fseek($fh, $offset) !== 0) { @@ -120,23 +172,40 @@ final class StorageBackend implements IBackend { } } + /** Whether the storage accepts the connection settings (files_external's own check). */ + public function canConnect(): bool { + try { + return !method_exists($this->storage, 'test') || $this->storage->test(); + } catch (\Throwable) { + return false; + } + } + public function exists(string $path): bool { - return $this->storage->file_exists($this->abs($path)); + $abs = $this->abs($path); + return $this->call(fn () => $this->storage->file_exists($abs), fn (bool $found) => !$found); } public function list(string $prefix): array { - $result = []; - $this->listInto($this->abs($prefix), $result); + $abs = $this->abs($prefix); + $result = $this->call(function () use ($abs): array { + $result = []; + $this->listInto($abs, $result); + return $result; + }, fn (array $found) => $found === []); sort($result); $cut = $this->base === '' ? 0 : strlen($this->base) + 1; return array_map(fn ($p) => substr($p, $cut), $result); } public function delete(string $path): void { + $this->idleCheck(); $this->storage->unlink($this->abs($path)); } public function move(string $from, string $to): void { + // not repeated on a new connection: the server may have done it before the connection died + $this->idleCheck(); $target = $this->abs($to); $this->mkdirs(dirname($target)); if (!$this->storage->rename($this->abs($from), $target)) { @@ -146,6 +215,11 @@ final class StorageBackend implements IBackend { public function listFiles(string $dir): array { $abs = $this->abs($dir); + return $this->call(fn () => $this->filesIn($dir, $abs), fn (array $found) => $found === []); + } + + /** @return list */ + private function filesIn(string $dir, string $abs): array { if (!$this->storage->is_dir($abs)) { return []; // not created yet } @@ -166,6 +240,94 @@ final class StorageBackend implements IBackend { return $result; } + /** + * Run $op; when it throws, or $failed says its result may come from a dead connection, and + * the SFTP connection turns out to be dead, run it once more on a new connection. + * + * @template T + * @param \Closure():T $op + * @param (\Closure(T):bool)|null $failed + * @return T + */ + private function call(\Closure $op, ?\Closure $failed = null): mixed { + $this->idleCheck(); + try { + $result = $op(); + } catch (\Throwable $e) { + if (!$this->reopenIfDead()) { + throw $e; + } + return $op(); + } + if ($failed !== null && $failed($result) && $this->reopenIfDead()) { + return $op(); + } + return $result; + } + + /** + * A Hetzner Storage Box drops an SFTP connection that carried a lot of data and then stayed + * quiet for a few minutes, while phpseclib still takes it for open; files_external never + * builds a new one, and every request on it fails as if the file were missing. So after + * such a pause the connection is checked (one round trip) before it is used again. + */ + private function idleCheck(): void { + $now = microtime(true); + if ($this->reconnect !== null && $this->sftpReads && $now - $this->lastUsed >= self::IDLE_CHECK + && $this->alive(true) === false) { + $this->reopen(); + } + $this->lastUsed = microtime(true); + } + + private function reopenIfDead(): bool { + if ($this->reconnect === null || !$this->sftpReads || $this->alive(false) !== false) { + return false; + } + $this->reopen(); + return true; + } + + /** + * Whether the storage's SFTP connection is alive; null when it cannot tell. + * $ask: send a request instead of trusting what phpseclib saw last. + */ + private function alive(bool $ask): ?bool { + try { + if (!is_callable([$this->storage, 'getConnection'])) { + return null; + } + $sftp = $this->storage->getConnection(); + if (!is_object($sftp) || !is_callable([$sftp, 'isConnected'])) { + return null; + } + if (!$sftp->isConnected()) { + return false; + } + if (!$ask || !is_callable([$sftp, 'stat'])) { + return true; + } + if (is_callable([$sftp, 'clearStatCache'])) { + $sftp->clearStatCache(); // or stat() answers from memory + } + $sftp->stat('.'); + return (bool)$sftp->isConnected(); + } catch (\Throwable) { + return false; + } + } + + private function reopen(): void { + /** @var \Closure():IStorage $reconnect */ + $reconnect = $this->reconnect; + $this->storage = $reconnect(); + $this->reconnects++; + $this->lastUsed = microtime(true); + if (function_exists('OCP\Log\logger')) { + \OCP\Log\logger('ng_backup')->info('The SFTP connection to the backup location was lost; reconnected'); + } + } + private function listInto(string $dir, array &$result): void { if (!$this->storage->is_dir($dir)) { return; diff --git a/lib/Command/Status.php b/lib/Command/Status.php index 509e03e..d698e6b 100644 --- a/lib/Command/Status.php +++ b/lib/Command/Status.php @@ -10,6 +10,7 @@ namespace OCA\NgBackup\Command; use OCA\NgBackup\Db\RunMapper; use OCA\NgBackup\Service\KeyService; use OCA\NgBackup\Service\TargetService; +use OCA\NgBackup\Service\VerifyService; use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Helper\Table; use Symfony\Component\Console\Input\InputInterface; @@ -20,6 +21,7 @@ final class Status extends Command { private KeyService $keys, private TargetService $targets, private RunMapper $runs, + private VerifyService $verify, ) { parent::__construct(); } @@ -44,13 +46,23 @@ final class Status extends Command { $rows = []; foreach ($this->runs->findRecent(10) as $r) { $s = json_decode($r->getStats() ?? '{}', true)['files'] ?? []; - $rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . ')' : ''), + $rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''), date('Y-m-d H:i', $r->getStartedAt()), $r->getFinishedAt() ? gmdate('H:i:s', $r->getFinishedAt() - $r->getStartedAt()) : '', $s['files'] ?? '', isset($s['uploaded']) ? sprintf('%.1f MiB', $s['uploaded'] / 1048576) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')]; } if ($rows !== []) { (new Table($output))->setHeaders(['run', 'location', 'kind', 'status', 'started', 'took', 'files', 'uploaded', 'error'])->setRows($rows)->render(); } + foreach ($this->verify->progress() as $targetId => $state) { + $output->writeln(sprintf('Verify of %s on %s: %s', mb_strimwidth((string)($state['what'] ?? ''), 0, 40, '…'), + $names[$targetId] ?? '#' . $targetId, VerifyService::describe($state))); + } return 0; } + + /** " 68%", or " ~68%" when estimated from the file cache; empty without an estimate */ + private static function percent(\OCA\NgBackup\Db\Run $r): string { + $p = \OCA\NgBackup\Job\BackupRun::progressOf($r); + return $p === null ? '' : sprintf(' %s%d%%', $p['approximate'] ? '~' : '', (int)floor($p['fraction'] * 100.0)); + } } diff --git a/lib/Command/TargetEdit.php b/lib/Command/TargetEdit.php new file mode 100644 index 0000000..58a0f39 --- /dev/null +++ b/lib/Command/TargetEdit.php @@ -0,0 +1,71 @@ +setName('backup:target:edit') + ->setDescription('Change the connection settings or the name of a backup location, keeping its backups') + ->addArgument('target', InputArgument::REQUIRED, 'Location name or id') + ->addOption('name', null, InputOption::VALUE_REQUIRED, 'New name') + ->addOption('backend', 'b', InputOption::VALUE_REQUIRED, 'New files_external backend id (see backup:target:backends)') + ->addOption('auth', 'a', InputOption::VALUE_REQUIRED, 'New auth mechanism id') + ->addOption('option', 'o', InputOption::VALUE_REQUIRED | InputOption::VALUE_IS_ARRAY, 'Option key=value to change (repeatable); other options keep their value') + ->addOption('path', 'p', InputOption::VALUE_REQUIRED, 'New folder on the location') + ->setHelp(<<<'HELP' +The new settings must lead to the same repository as before, so the history of the location is +kept. Use it after a password change, a new host name, or after moving the repository. + +Example: + occ backup:target:edit nas -o password=new-secret +HELP); + } + + protected function execute(InputInterface $input, OutputInterface $output): int { + try { + $t = $this->targets->get($input->getArgument('target')); + $options = $this->targets->options($t); + if ($input->getOption('backend') !== null || $input->getOption('auth') !== null) { + $options = []; // a different connection type has its own options + } + foreach ($input->getOption('option') as $pair) { + if (!str_contains($pair, '=')) { + $output->writeln("Option '$pair' must be key=value"); + return 1; + } + $k = (string)strstr($pair, '=', true); + $v = substr($pair, strlen($k) + 1); + $options[$k] = match (strtolower($v)) { 'true' => true, 'false' => false, default => $v }; + } + $before = $t->getName(); + $t = $this->targets->update($t, $input->getOption('name') ?? $t->getName(), $input->getOption('backend') ?? $t->getBackend(), + $input->getOption('auth') ?? $t->getAuth(), $options, $input->getOption('path') ?? $t->getBasePath()); + } catch (\Throwable $e) { + $output->writeln('' . $e->getMessage() . ''); + return 1; + } + $this->alerts->securityEvent('target_changed', ['target' => $before]); + $output->writeln(sprintf('Location %s updated; repository %s found with the new settings.', $t->getName(), (string)$t->getRepositoryId())); + return 0; + } +} diff --git a/lib/Command/Verify.php b/lib/Command/Verify.php index 37997ab..9480f83 100644 --- a/lib/Command/Verify.php +++ b/lib/Command/Verify.php @@ -37,6 +37,9 @@ blob is also downloaded and decrypted, which fails loudly on exactly that even i lists it -- the same verification an actual restore would do, run here deliberately instead of waiting to find out during one. +While it runs it prints how far it is every 30 seconds; occ backup:status shows the same, and +how the last check of each location ended. + Run this periodically (cron, picking a sample of recent snapshots) rather than only right before you'd need to rely on a restore. @@ -51,9 +54,12 @@ HELP); $target = $this->targets->get($input->getArgument('target')); $ref = (string)$input->getArgument('snapshot'); $deep = (bool)$input->getOption('deep'); + $onProgress = static function (array $state) use ($output): void { + $output->writeln(date('H:i:s') . ' ' . VerifyService::describe($state)); + }; $result = str_starts_with($ref, 'users/') - ? $this->verify->verifyUserExport($target, $ref, $deep) - : $this->verify->verify($target, $ref, $deep); + ? $this->verify->verifyUserExport($target, $ref, $deep, $onProgress) + : $this->verify->verify($target, $ref, $deep, $onProgress); } catch (\Throwable $e) { $output->writeln('' . $e->getMessage() . ''); return 1; diff --git a/lib/Controller/AdminApiController.php b/lib/Controller/AdminApiController.php index 3d07195..26cd430 100644 --- a/lib/Controller/AdminApiController.php +++ b/lib/Controller/AdminApiController.php @@ -181,6 +181,28 @@ final class AdminApiController extends Controller { return new JSONResponse(['id' => $r['target']->getId(), 'created' => $r['created']]); } + /** The stored settings for the edit form; password values are left out (null). */ + #[PasswordConfirmationRequired] + public function targetSettings(int $id): JSONResponse { + $t = $this->targets->get((string)$id); + return new JSONResponse(['name' => $t->getName(), 'backend' => $t->getBackend(), 'auth' => $t->getAuth(), + 'options' => $this->targets->editableOptions($t), 'path' => $t->getBasePath()]); + } + + /** Options with a null value keep their stored value. */ + #[PasswordConfirmationRequired] + public function updateTarget(int $id, string $name, string $backend, string $auth, array $options = [], string $path = 'ng_backup'): JSONResponse { + try { + $t = $this->targets->get((string)$id); + $before = $t->getName(); + $t = $this->targets->update($t, $name, $backend, $auth, $options, $path); + } catch (\Throwable $e) { + return new JSONResponse(['error' => $e->getMessage()], Http::STATUS_BAD_REQUEST); + } + $this->alerts->securityEvent('target_changed', ['target' => $before]); + return new JSONResponse(['id' => $t->getId()]); + } + /** Size limit in GiB; null or 0 = unlimited. */ #[PasswordConfirmationRequired] public function setTargetLimit(int $id, ?float $maxGb = null): JSONResponse { @@ -283,6 +305,6 @@ final class AdminApiController extends Controller { return ['id' => $r->getId(), 'targetId' => $r->getTargetId(), 'kind' => $r->getKind(), 'status' => $r->getStatus(), 'phase' => $r->getPhase(), 'snapshotId' => $r->getSnapshotId(), 'snapshot' => $snapshot, 'startedAt' => $r->getStartedAt(), 'updatedAt' => $r->getUpdatedAt(), 'finishedAt' => $r->getFinishedAt(), 'error' => $r->getError(), - 'stats' => json_decode($r->getStats() ?? 'null', true)]; + 'stats' => json_decode($r->getStats() ?? 'null', true), 'progress' => \OCA\NgBackup\Job\BackupRun::progressOf($r)]; } } diff --git a/lib/Db/DbIndexCache.php b/lib/Db/DbIndexCache.php index 4aace2f..e70ccbb 100644 --- a/lib/Db/DbIndexCache.php +++ b/lib/Db/DbIndexCache.php @@ -30,19 +30,84 @@ final class DbIndexCache implements IIndexCache { } public function put(string $repositoryId, string $packId, array $entries): void { + try { + $this->insert($repositoryId, $packId, $entries); + } catch (Exception $e) { + if ($e->getReason() !== Exception::REASON_UNIQUE_CONSTRAINT_VIOLATION) { + throw $e; + } + } + } + + /** + * In one transaction: on PostgreSQL with its default settings every statement of its own + * waits for a disk sync, which made 27,547 rows take more than 6 minutes. Rows already there + * are replaced. Should the transaction fail (another process writing the same rows), the + * rows are written one by one, keeping rows that are already there. + */ + public function putMany(string $repositoryId, array $packs): void { + if ($packs === []) { + return; + } + try { + $this->transaction(function () use ($repositoryId, $packs): void { + $this->remove($repositoryId, array_keys($packs)); + $this->insertAll($repositoryId, $packs); + }); + } catch (\Throwable $e) { + if ($this->db->inTransaction()) { + throw $e; // a transaction of the caller's, which may be spoilt now + } + foreach ($packs as $packId => $entries) { + $this->put($repositoryId, $packId, $entries); + } + } + } + + /** All or nothing: when it fails, the cache stays as it was. */ + public function replaceAll(string $repositoryId, array $packs): void { + $this->transaction(function () use ($repositoryId, $packs): void { + $qb = $this->db->getQueryBuilder(); + $qb->delete('ngb_index')->where($qb->expr()->eq('repository_id', $qb->createNamedParameter($repositoryId))); + $qb->executeStatement(); + $this->insertAll($repositoryId, $packs); + }); + } + + /** Run $write in a transaction of its own, or as part of the caller's when there is one. */ + private function transaction(\Closure $write): void { + if ($this->db->inTransaction()) { + $write(); + return; + } + $this->db->beginTransaction(); + try { + $write(); + $this->db->commit(); + } catch (\Throwable $e) { + if ($this->db->inTransaction()) { + $this->db->rollBack(); + } + throw $e; + } + } + + /** @param array> $packs */ + private function insertAll(string $repositoryId, array $packs): void { + foreach ($packs as $packId => $entries) { + $this->insert($repositoryId, $packId, $entries); + } + } + + /** @param list $entries */ + private function insert(string $repositoryId, string $packId, array $entries): void { $qb = $this->db->getQueryBuilder(); $qb->insert('ngb_index')->values([ 'repository_id' => $qb->createNamedParameter($repositoryId), 'pack_id' => $qb->createNamedParameter($packId), 'entries' => $qb->createNamedParameter(json_encode($entries, JSON_THROW_ON_ERROR)), ]); - try { - $qb->executeStatement(); - } catch (Exception $e) { - if ($e->getReason() !== Exception::REASON_UNIQUE_CONSTRAINT_VIOLATION) { - throw $e; - } - } + $qb->executeStatement(); } public function remove(string $repositoryId, array $packIds): void { diff --git a/lib/Db/RunMapper.php b/lib/Db/RunMapper.php index 9799259..162f4d7 100644 --- a/lib/Db/RunMapper.php +++ b/lib/Db/RunMapper.php @@ -17,6 +17,36 @@ final class RunMapper extends QBMapper { parent::__construct($db, 'ngb_runs', Run::class); } + /** + * Write $run only if it is still running and unchanged since it was loaded (updated_at + * $loadedAt), in one statement: two processes that loaded the same run cannot both write it. + * The new updated_at is always later than $loadedAt, so every write is seen as a change. + * + * @return Run|null the written run, or null when another process wrote it first + */ + public function updateIfUnchanged(Run $run, int $loadedAt): ?Run { + $run->setUpdatedAt(max(time(), $loadedAt + 1)); + $types = $run->getFieldTypes(); + $qb = $this->db->getQueryBuilder(); + $qb->update($this->getTableName()); + foreach (array_keys($run->getUpdatedFields()) as $field) { + if ($field === 'id') { + continue; + } + $value = $run->{'get' . ucfirst($field)}(); + $type = ($types[$field] ?? 'string') === 'integer' ? IQueryBuilder::PARAM_INT : IQueryBuilder::PARAM_STR; + $qb->set($run->propertyToColumn($field), $qb->createNamedParameter($value, $value === null ? IQueryBuilder::PARAM_NULL : $type)); + } + $qb->where($qb->expr()->eq('id', $qb->createNamedParameter($run->getId(), IQueryBuilder::PARAM_INT))) + ->andWhere($qb->expr()->eq('status', $qb->createNamedParameter(Run::RUNNING))) + ->andWhere($qb->expr()->eq('updated_at', $qb->createNamedParameter($loadedAt, IQueryBuilder::PARAM_INT))); + if ($qb->executeStatement() !== 1) { + return null; + } + $run->resetUpdatedFields(); + return $run; + } + public function find(int $id): Run { $qb = $this->db->getQueryBuilder(); $qb->select('*')->from($this->getTableName())->where($qb->expr()->eq('id', $qb->createNamedParameter($id, IQueryBuilder::PARAM_INT))); diff --git a/lib/Job/BackupRun.php b/lib/Job/BackupRun.php index 7edb3f4..5b6a300 100644 --- a/lib/Job/BackupRun.php +++ b/lib/Job/BackupRun.php @@ -160,6 +160,38 @@ final class BackupRun { return $state; } + /** + * Progress of the files phase as a fraction (at most 0.99 until it is done), or null without an + * estimate. Bytes of a large file that is still being read count too. + * + * @return array{fraction: float, approximate: bool}|null + */ + public static function progress(array $state): ?array { + $estimate = $state['estimate'] ?? null; + if (($state['phase'] ?? null) !== 'files' || !is_array($estimate) || ($estimate['bytes'] ?? 0) <= 0) { + return null; + } + $done = (int)($state['stats']['bytes'] ?? 0) + (int)($state['cur']['offset'] ?? 0); + return ['fraction' => min(0.99, $done / (int)$estimate['bytes']), 'approximate' => (bool)($estimate['approximate'] ?? true)]; + } + + /** + * progress() of a run, while it is in its files phase. + * + * @return array{fraction: float, approximate: bool}|null + */ + public static function progressOf(\OCA\NgBackup\Db\Run $run): ?array { + if ($run->getStatus() !== \OCA\NgBackup\Db\Run::RUNNING || $run->getPhase() !== 'files') { + return null; + } + try { + $state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR); + } catch (\JsonException) { + return null; + } + return is_array($state['files'] ?? null) ? self::progress($state['files']) : null; + } + /** * A file or directory that could not be read: it is missing from this snapshot. Remembered (first * 1000, deduplicated across steps) so the run can report it instead of silently succeeding. diff --git a/lib/Notification/Notifier.php b/lib/Notification/Notifier.php index df81fa3..c1d05ae 100644 --- a/lib/Notification/Notifier.php +++ b/lib/Notification/Notifier.php @@ -51,7 +51,10 @@ final class Notifier implements INotifier { $notification->setParsedMessage($l->t('Backups stop when the limit is reached. Raise the limit, or let the retention policy remove older backups.')); break; case 'backup_stale': - $notification->setParsedSubject($l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']])); + // 'never' was stored by 0.9.4 and earlier + $notification->setParsedSubject(in_array($p['since'] ?? '', ['', 'never'], true) + ? $l->t('No successful backup to %s yet', [$p['target']]) + : $l->t('No successful backup to %1$s since %2$s', [$p['target'], $p['since']])); $notification->setParsedMessage($l->t('Check the backup settings and the server log.')); break; case 'security_event': @@ -59,6 +62,7 @@ final class Notifier implements INotifier { $notification->setParsedSubject(match ($p['event'] ?? '') { 'kit_downloaded' => $l->t('%s downloaded the backup recovery kit', [$who]), 'target_removed' => $l->t('%1$s removed backup location %2$s', [$who, $p['target'] ?? '']), + 'target_changed' => $l->t('%1$s changed the connection settings of backup location %2$s', [$who, $p['target'] ?? '']), 'append_only_off' => $l->t('%1$s turned off append-only for backup location %2$s', [$who, $p['target'] ?? '']), 'retention_changed' => $l->t('%1$s changed the backup retention to %2$s', [$who, $p['policy'] ?? '']), 'snapshots_forgotten' => $l->t('%1$s moved %2$s snapshots of %3$s to the trash; they are deleted permanently after %4$s unless restored', [$who, $p['count'] ?? '', $p['target'] ?? '', $p['until'] ?? '']), diff --git a/lib/Repository/BlobIndex.php b/lib/Repository/BlobIndex.php index eb7269e..1fcfc3d 100644 --- a/lib/Repository/BlobIndex.php +++ b/lib/Repository/BlobIndex.php @@ -28,6 +28,8 @@ final class BlobIndex { /** @var list<\Closure():void>|null cache writes held back by holdCache(), or null when writing directly */ private ?array $heldCacheWrites = null; private string $repositoryId = ''; + /** Index files written to the cache per database transaction while loading. */ + private const CACHE_LOT = 256; /** @var int index files downloaded by the last load() (for tests and diagnostics) */ public int $downloaded = 0; @@ -87,47 +89,75 @@ final class BlobIndex { * and only the index files that are not cached yet are downloaded. Cached packs that no * longer exist on the backend are dropped from the cache. */ - public function load(bool $verify = false): void { + public function load(bool $verify = false, ?callable $progress = null): void { $this->blobs = []; $cached = $verify ? [] : ($this->cache?->all($this->repositoryId) ?? []); $present = []; $this->downloaded = 0; + $order = []; // pack ids in listing order: where a blob is in two packs, the last listed wins + $fetch = []; foreach ($this->backend->listFiles('index') as $path) { $packId = basename($path); $present[$packId] = true; - if (isset($cached[$packId])) { - $entries = $cached[$packId]; - } else { - $fh = $this->backend->get($path); - $entries = json_decode($this->cipher->decryptString((string)stream_get_contents($fh), 'index:' . $packId), true, 512, JSON_THROW_ON_ERROR); - fclose($fh); - $this->downloaded++; - $this->cache?->put($this->repositoryId, $packId, $entries); - } - foreach ($entries as $e) { - $this->blobs[$e['id']] = [$packId, $e['offset'], $e['length'], $e['raw'], $e['flags']]; + $order[] = $packId; + if (!isset($cached[$packId])) { + $fetch[] = $path; } } - if ($verify && $this->cache !== null) { - // Refresh the cache from what was just verified. - $this->cache->remove($this->repositoryId, array_keys($this->cache->all($this->repositoryId))); - foreach ($this->packs() as $packId => $_) { - $entries = []; - foreach ($this->blobs as $id => [$pack, $offset, $length, $raw, $flags]) { - if ($pack === $packId) { - $entries[] = ['id' => $id, 'offset' => $offset, 'length' => $length, 'raw' => $raw, 'flags' => $flags]; - } + // The rest is downloaded several at a time where the location allows it: on a large + // repository these are tens of thousands of small files. getMany() keeps their order, so + // the cached packs listed before each one are added first. + $next = 0; + $total = count($fetch); + $toCache = []; // written to the cache a few hundred at a time, each lot in one transaction + foreach ($this->backend->getMany($fetch) as $path => $data) { + $packId = basename($path); + $entries = json_decode($this->cipher->decryptString($data, 'index:' . $packId), true, 512, JSON_THROW_ON_ERROR); + $this->downloaded++; + if ($this->cache !== null) { + $toCache[$packId] = $entries; + // a check writes the whole cache afresh at the end + if (!$verify && count($toCache) >= self::CACHE_LOT) { + $this->cache->putMany($this->repositoryId, $toCache); + $toCache = []; } - $this->cache->put($this->repositoryId, $packId, $entries); } + for (; $next < count($order) && $order[$next] !== $packId; $next++) { + $this->add($order[$next], $cached[$order[$next]] ?? []); + } + $this->add($packId, $entries); + $next++; + if ($progress !== null) { + $progress($this->downloaded, $total); + } + } + for (; $next < count($order); $next++) { + $this->add($order[$next], $cached[$order[$next]] ?? []); + } + if ($verify && $this->cache !== null) { + // Refresh the cache from what was just verified, as the index files list it: a blob in + // two packs is in both rows, so that removing one of the packs leaves the other right. + // In one transaction: one row at a time took more than 6 minutes on a 1 TB repository + // (27,547 packs), long enough for the location to drop the connection. + $this->cache->replaceAll($this->repositoryId, $toCache); return; } + if ($toCache !== []) { + $this->cache?->putMany($this->repositoryId, $toCache); + } $stale = array_keys(array_diff_key($cached, $present)); if ($stale !== []) { $this->cache?->remove($this->repositoryId, $stale); } } + /** @param list $entries */ + private function add(string $packId, array $entries): void { + foreach ($entries as $e) { + $this->blobs[$e['id']] = [$packId, $e['offset'], $e['length'], $e['raw'], $e['flags']]; + } + } + public function has(string $id): bool { return isset($this->blobs[$id]) || isset($this->pending[$id]); } @@ -181,6 +211,15 @@ final class BlobIndex { } } + /** Number of packs that hold blobs of the index. */ + public function packCount(): int { + $packs = []; + foreach ($this->blobs as [$pack]) { + $packs[$pack] = true; + } + return count($packs); + } + /** Bytes of all blobs in the index (encrypted size, i.e. what is stored in packs). */ public function storedBytes(): int { $total = 0; diff --git a/lib/Repository/IIndexCache.php b/lib/Repository/IIndexCache.php index 1a7cff0..a665f7e 100644 --- a/lib/Repository/IIndexCache.php +++ b/lib/Repository/IIndexCache.php @@ -15,6 +15,20 @@ interface IIndexCache { /** @param list $entries */ public function put(string $repositoryId, string $packId, array $entries): void; + /** + * put() for many packs at once (one database transaction). + * + * @param array> $packs packId => entries + */ + public function putMany(string $repositoryId, array $packs): void; + + /** + * Replace everything cached for the repository with $packs (one database transaction). + * + * @param array> $packs packId => entries + */ + public function replaceAll(string $repositoryId, array $packs): void; + /** @param list $packIds */ public function remove(string $repositoryId, array $packIds): void; } diff --git a/lib/Repository/PackReaderGuard.php b/lib/Repository/PackReaderGuard.php index 0004f4e..0679298 100644 --- a/lib/Repository/PackReaderGuard.php +++ b/lib/Repository/PackReaderGuard.php @@ -28,6 +28,11 @@ final class PackReaderGuard implements IBackend { return $this->inner->get($path); } + public function getMany(array $paths): \Generator { + $this->reader->close(); + return $this->inner->getMany($paths); + } + public function getRange(string $path, int $offset, int $length): string { $this->reader->close(); return $this->inner->getRange($path, $offset, $length); diff --git a/lib/Repository/Repository.php b/lib/Repository/Repository.php index 3c75cbf..188266c 100644 --- a/lib/Repository/Repository.php +++ b/lib/Repository/Repository.php @@ -37,6 +37,8 @@ final class Repository { private ?int $maxBytes = null; private Catalog $catalog; private PackReader $packReader; + /** verify() lists the pack folders from this many packs on (see presentPacks()); tests lower it. */ + public int $listPacksAbove = 256; private function __construct( private IBackend $backend, @@ -750,17 +752,23 @@ final class Repository { * referenced by more than one file is only checked once; a missing database manifest is a * failure, not something to silently skip over. * - * @param ?callable $heartbeat called after every blob (shallow) or every blob read (deep); - * callers use it to keep a time-boxed lease alive during a large audit + * @param ?callable(array{phase:string, done:int, total:?int, blobs?:int, bytes?:int}):void $heartbeat + * called often during the check, with how far it is: phase "index" (index files + * downloaded), "packs" (pack folders listed), "files" and "database" (files of the + * snapshot checked); callers use it to keep a time-boxed lease alive and to report * @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list, failed:list, unreadable:list, readError:string} */ public function verify(string $snapshotId, bool $deep = false, ?callable $heartbeat = null): array { - $heartbeat ??= static function (): void { + $heartbeat ??= static function (array $progress): void { }; - $this->index->load(true); + $meta = $this->snapshotMeta($snapshotId); + $filesTotal = isset($meta['stats']['files']) ? (int)$meta['stats']['files'] : null; + $this->index->load(true, static function (int $done, int $total) use ($heartbeat): void { + $heartbeat(['phase' => 'index', 'done' => $done, 'total' => $total]); + }); $stats = ['filesChecked' => 0, 'blobsChecked' => 0, 'bytesChecked' => 0, 'missing' => [], 'failed' => [], 'unreadable' => [], 'readError' => '']; $seen = []; - $packSeen = []; // pack id => exists on the backend; avoids one exists() call per blob + [$packSeen, $listedDirs] = $this->presentPacks($heartbeat); foreach ($this->streamEntries($snapshotId) as $entry) { $stats['filesChecked']++; foreach ($entry['b'] as $id) { @@ -768,11 +776,11 @@ final class Repository { continue; } $seen[$id] = true; - $this->verifyBlob($id, $deep, $stats, $packSeen); - $heartbeat(); + $this->verifyBlob($id, $deep, $stats, $packSeen, $listedDirs); + $heartbeat(['phase' => 'files', 'done' => $stats['filesChecked'], 'total' => $filesTotal, + 'blobs' => $stats['blobsChecked'], 'bytes' => $stats['bytesChecked']]); } } - $meta = $this->snapshotMeta($snapshotId); if (isset($meta['db']) && is_string($meta['db'])) { if (!$this->backend->exists($meta['db'])) { $stats['missing'][] = $meta['db']; @@ -784,8 +792,9 @@ final class Repository { continue; } $seen[$id] = true; - $this->verifyBlob($id, $deep, $stats, $packSeen); - $heartbeat(); + $this->verifyBlob($id, $deep, $stats, $packSeen, $listedDirs); + $heartbeat(['phase' => 'database', 'done' => $stats['filesChecked'], 'total' => $filesTotal, + 'blobs' => $stats['blobsChecked'], 'bytes' => $stats['bytesChecked']]); } } } @@ -793,6 +802,40 @@ final class Repository { return $stats; } + /** + * Which packs are on the location, from one listing per pack folder instead of a request per + * pack. A folder that cannot be listed, or comes back empty (which a storage also reports when + * it could not look), is left out: its packs are then checked one by one. With no more packs + * than folders nothing is listed: a request per pack is fewer requests then, and on FTP every + * request takes a data connection, of which a server may have only a few. + * + * @return array{0: array, 1: array} pack id => present, and the + * folders that were listed + */ + private function presentPacks(callable $heartbeat): array { + $present = []; + $listed = []; + if ($this->index->packCount() <= $this->listPacksAbove) { + return [$present, $listed]; + } + for ($i = 0; $i < 256; $i++) { + $dir = sprintf('%02x', $i); + try { + $files = $this->backend->listFiles('packs/' . $dir); + } catch (BackendException) { + $files = []; + } + foreach ($files as $path) { + $present[basename($path)] = true; + } + if ($files !== []) { + $listed[$dir] = true; + } + $heartbeat(['phase' => 'packs', 'done' => $i + 1, 'total' => 256]); + } + return [$present, $listed]; + } + /** * Same checks as verify(), for a user export: every blob its manifest references must be in * the freshly re-authenticated index with its pack present; $deep also decrypts each. @@ -803,9 +846,11 @@ final class Repository { if (!$this->isRecordedUserExport($manifestPath)) { throw new RepositoryException("$manifestPath is not a recorded user export for this location"); } - $heartbeat ??= static function (): void { + $heartbeat ??= static function (array $progress): void { }; - $this->index->load(true); + $this->index->load(true, static function (int $done, int $total) use ($heartbeat): void { + $heartbeat(['phase' => 'index', 'done' => $done, 'total' => $total]); + }); $stats = ['filesChecked' => 0, 'blobsChecked' => 0, 'bytesChecked' => 0, 'missing' => [], 'failed' => [], 'unreadable' => [], 'readError' => '']; $seen = []; $packSeen = []; @@ -821,20 +866,27 @@ final class Repository { } $seen[$id] = true; $this->verifyBlob($id, $deep, $stats, $packSeen); - $heartbeat(); + $heartbeat(['phase' => 'files', 'done' => $stats['filesChecked'], 'total' => null, + 'blobs' => $stats['blobsChecked'], 'bytes' => $stats['bytesChecked']]); } } return $stats; } - /** @param array $packSeen */ - private function verifyBlob(string $id, bool $deep, array &$stats, array &$packSeen): void { + /** + * @param array $packSeen pack id => present; filled in as packs are checked + * @param array $listedDirs pack folders whose listing is in $packSeen: a pack + * not in it is not there + */ + private function verifyBlob(string $id, bool $deep, array &$stats, array &$packSeen, array $listedDirs = []): void { if (!$this->index->has($id)) { $stats['missing'][] = $id; return; } [$pack] = $this->index->get($id); - $packExists = $packSeen[$pack] ??= $this->backend->exists('packs/' . substr($pack, 0, 2) . '/' . $pack); + $packExists = $packSeen[$pack] ??= isset($listedDirs[substr($pack, 0, 2)]) + ? false + : $this->backend->exists('packs/' . substr($pack, 0, 2) . '/' . $pack); if (!$packExists) { $stats['missing'][] = $id; return; diff --git a/lib/Service/AlertService.php b/lib/Service/AlertService.php index 817e78a..ace1dc8 100644 --- a/lib/Service/AlertService.php +++ b/lib/Service/AlertService.php @@ -104,7 +104,7 @@ final class AlertService { } $this->appConfig->setValueString(Application::APP_ID, $key, date('Y-m-d')); $this->notifyAdmins('backup_stale', 'stale-' . $target->getId() . '-' . date('Y-m-d'), - ['target' => $target->getName(), 'since' => $last ? date('Y-m-d H:i', $lastTime) : 'never']); + ['target' => $target->getName(), 'since' => $last ? date('Y-m-d H:i', $lastTime) : '']); } private function notifyAdmins(string $subject, string $objectId, array $params): void { diff --git a/lib/Service/BackupService.php b/lib/Service/BackupService.php index 702cf78..094e764 100644 --- a/lib/Service/BackupService.php +++ b/lib/Service/BackupService.php @@ -15,8 +15,10 @@ use OCA\NgBackup\Db\SnapshotMapper; use OCA\NgBackup\Db\Target; use OCA\NgBackup\Job\BackupRun; use OCA\NgBackup\Repository\RepositoryException; +use OCP\AppFramework\Db\DoesNotExistException; use OCP\App\IAppManager; use OCP\IConfig; +use OCP\DB\QueryBuilder\IQueryBuilder; use OCP\IDBConnection; use OCP\Lock\ILockingProvider; use OCP\Lock\LockedException; @@ -68,6 +70,46 @@ final class BackupService { } } + /** + * The expected size of the files phase, for the progress shown while it runs: the size of the + * previous snapshot, or for a first backup the users' files, versions and trash as Nextcloud's + * file cache knows them (approximate: other data in the data directory is not counted). + * + * @return array{bytes: int, approximate: bool} + */ + private function estimateBytes(Run $run, array $files): array { + if (($files['parent'] ?? null) !== null) { + try { + return ['bytes' => $this->snapshots->findOne($run->getTargetId(), (string)$files['parent'])->getBytes() ?? 0, 'approximate' => false]; + } catch (\Throwable) { + } + } + try { + $qb = $this->db->getQueryBuilder(); + $qb->select('numeric_id')->from('storages') + ->where($qb->expr()->like('id', $qb->createNamedParameter('home::%'))) + ->orWhere($qb->expr()->like('id', $qb->createNamedParameter('object::user:%'))); + $result = $qb->executeQuery(); + $storages = []; + while (($id = $result->fetchOne()) !== false) { + $storages[] = (int)$id; + } + $result->closeCursor(); + $bytes = 0; + foreach (array_chunk($storages, 500) as $chunk) { + $qb = $this->db->getQueryBuilder(); + $qb->select($qb->func()->sum('size'))->from('filecache') + ->where($qb->expr()->in('storage', $qb->createNamedParameter($chunk, IQueryBuilder::PARAM_INT_ARRAY))) + ->andWhere($qb->expr()->in('path_hash', $qb->createNamedParameter(array_map('md5', ['files', 'files_versions', 'files_trashbin']), IQueryBuilder::PARAM_STR_ARRAY))) + ->andWhere($qb->expr()->gt('size', $qb->createNamedParameter(0, IQueryBuilder::PARAM_INT))); + $bytes += (int)$qb->executeQuery()->fetchOne(); + } + return ['bytes' => $bytes, 'approximate' => true]; + } catch (\Throwable) { + return ['bytes' => 0, 'approximate' => true]; + } + } + private function createRun(Target $target, string $label): Run { $running = $this->runs->findRunning($target->getId(), self::KIND_FULL); if ($running !== []) { @@ -105,7 +147,10 @@ final class BackupService { */ public function step(Run $run, float $deadline): Run { // Leases expire on their own: a step killed by a time limit or OOM blocks nothing for long. - $ttl = (int)min(3600, max(120, $deadline - microtime(true) + 120)); + // A step can run well past its deadline (the commit of a large snapshot writes its tree and + // index), so the lease holds at least 15 minutes; should it still expire and another process + // take the run over, the checks below keep the two from overwriting each other. + $ttl = (int)min(3600, max(900, $deadline - microtime(true) + 600)); try { $runLease = $this->leases->acquire('run/' . $run->getId(), LeaseService::EXCLUSIVE, $ttl); } catch (LockedException) { @@ -118,11 +163,13 @@ final class BackupService { $this->leases->release($runLease); return $run; } + $loadedAt = null; try { $run = $this->runs->find($run->getId()); // fresh state after acquiring the lock if ($run->getStatus() !== Run::RUNNING || $run->getKind() !== self::KIND_FULL) { return $run; } + $loadedAt = $run->getUpdatedAt(); $target = $this->targets->get((string)$run->getTargetId()); $repo = $this->targets->repository($target); $state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR); @@ -147,6 +194,7 @@ final class BackupService { $state['dump'] = $dump; } } else { + $state['files']['estimate'] ??= $this->estimateBytes($run, $state['files']); $state['files'] = BackupRun::step($repo, $state['files'], $deadline); if ($state['files']['phase'] === 'commit') { // Catalog write under the catalog lock, from a repository opened inside it: a per-user @@ -162,14 +210,20 @@ final class BackupService { } if ($state['files']['phase'] === 'done') { $this->recordSnapshot($run, $state['files']); - $this->alerts->storedBytes($target, $repo->storedBytes()); - $stats = $state['files']['stats']; - if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) { - $warnings = BackupRun::warnings($state['files']); - $this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s', - $run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']), - $stats['changing'] ?? 0, implode(', ', $warnings['changing']))); - $this->alerts->runWarnings($run, $target->getName(), (int)($stats['unreadable'] ?? 0), (int)($stats['changing'] ?? 0), $warnings); + // The snapshot is committed: what follows only reports on it and must not turn + // the run into a failure. + try { + $this->alerts->storedBytes($target, $repo->storedBytes()); + $stats = $state['files']['stats']; + if (($stats['unreadable'] ?? 0) > 0 || ($stats['changing'] ?? 0) > 0) { + $warnings = BackupRun::warnings($state['files']); + $this->logger->warning(sprintf('NG Backup run %d to %s: %d unreadable (missing from the snapshot): %s; %d changed while being read: %s', + $run->getId(), $target->getName(), $stats['unreadable'] ?? 0, implode(', ', $warnings['unreadable']), + $stats['changing'] ?? 0, implode(', ', $warnings['changing']))); + $this->alerts->runWarnings($run, $target->getName(), (int)($stats['unreadable'] ?? 0), (int)($stats['changing'] ?? 0), $warnings); + } + } catch (\Throwable $e) { + $this->logger->warning('NG Backup run ' . $run->getId() . ': reporting on the finished backup failed: ' . $e->getMessage(), ['exception' => $e]); } $run->setStatus(Run::DONE); $run->setPhase('done'); @@ -178,15 +232,27 @@ final class BackupService { } $run->setState(json_encode($state, JSON_THROW_ON_ERROR)); $run->setStats(json_encode(['db' => $state['dbStats'] ?? null, 'files' => $state['files']['stats']], JSON_THROW_ON_ERROR)); - $run->setUpdatedAt(time()); - return $this->runs->update($run); + // Another process may have taken the run over after this step's lease expired: then its + // writes stand and this step's are dropped. + return $this->runs->updateIfUnchanged($run, $loadedAt) ?? $this->runs->find($run->getId()); } catch (\Throwable $e) { - $this->logger->error('NG Backup run ' . $run->getId() . ' failed: ' . $e->getMessage(), ['exception' => $e]); + if ($loadedAt === null) { + // The run could not even be loaded: there is nothing fresh to mark as failed, and + // the caller's copy may be stale. + $this->logger->error('NG Backup run ' . $run->getId() . ': step could not load the run: ' . $e->getMessage(), ['exception' => $e]); + return $run; + } $run->setStatus(Run::FAILED); $run->setError($e->getMessage()); $run->setFinishedAt(time()); - $run->setUpdatedAt(time()); - $run = $this->runs->update($run); + $failed = $this->runs->updateIfUnchanged($run, $loadedAt); + if ($failed === null) { + // Another process that took the run over may have finished it: its result stands. + $this->logger->warning('NG Backup run ' . $run->getId() . ': a step that another process had taken over ended with: ' . $e->getMessage()); + return $this->runs->find($run->getId()); + } + $run = $failed; + $this->logger->error('NG Backup run ' . $run->getId() . ' failed: ' . $e->getMessage(), ['exception' => $e]); try { $this->alerts->runFailed($run, isset($target) ? $target->getName() : '#' . $run->getTargetId()); } catch (\Throwable) { @@ -224,6 +290,11 @@ final class BackupService { } private function recordSnapshot(Run $run, array $files): void { + try { + $this->snapshots->findOne($run->getTargetId(), $files['snapshot']); + return; // already recorded, by an earlier attempt at this step + } catch (DoesNotExistException) { + } $s = new Snapshot(); $s->setTargetId($run->getTargetId()); $s->setSnapshotId($files['snapshot']); @@ -232,7 +303,19 @@ final class BackupService { $s->setCreatedAt(time()); $s->setFiles($files['stats']['files']); $s->setBytes($files['stats']['bytes']); - $this->snapshots->insert($s); + try { + $this->snapshots->insert($s); + } catch (\OCP\DB\Exception $e) { + // Recorded by another process between the check above and this insert? + if ($e->getReason() !== \OCP\DB\Exception::REASON_UNIQUE_CONSTRAINT_VIOLATION) { + throw $e; + } + try { + $this->snapshots->findOne($run->getTargetId(), $files['snapshot']); + } catch (DoesNotExistException) { + throw $e; + } + } } /** @return list */ diff --git a/lib/Service/TargetService.php b/lib/Service/TargetService.php index f705414..b581301 100644 --- a/lib/Service/TargetService.php +++ b/lib/Service/TargetService.php @@ -17,6 +17,7 @@ use OCA\NgBackup\Db\TargetMapper; use OCA\NgBackup\Repository\Repository; use OCP\App\IAppManager; use OCP\AppFramework\Db\DoesNotExistException; +use OCP\Files\Storage\IStorage; use OCP\Security\ICrypto; use OCP\Server; @@ -37,7 +38,7 @@ final class TargetService { ) { } - /** @return list, parameters:array}> */ + /** @return list, parameters:array}> */ public function availableBackends(): array { return $this->factory()->describeBackends(); } @@ -50,20 +51,8 @@ final class TargetService { * @return array{target: Target, created: bool} */ public function add(string $name, string $backend, string $auth, array $options, string $basePath): array { - if (!preg_match('/^[A-Za-z0-9][A-Za-z0-9 _.-]{0,63}$/', $name)) { - throw new \InvalidArgumentException('Name: letters, digits, space, _ . - (max 64)'); - } - // Without the native extension files_external falls back to the smbclient binary, which - // registers fine but cannot seek: the first backup then fails with "Seek failed". Refuse - // up front instead of letting the admin find out from a failed run. - if ($backend === 'smb' && !extension_loaded('smbclient')) { - throw new \InvalidArgumentException('The SMB backend needs the PHP smbclient extension (pecl install smbclient); the smbclient binary alone cannot seek and every backup would fail'); - } - try { - $this->mapper->findByName($name); - throw new \InvalidArgumentException("A target named '$name' already exists"); - } catch (\OCP\AppFramework\Db\DoesNotExistException) { - } + $this->validateName($name); + $this->requireUsableBackend($backend); $keys = $this->keys->keyRing(); $storageBackend = $this->buildBackend($backend, $auth, $options, $basePath); if ($storageBackend->exists('config')) { @@ -97,6 +86,103 @@ final class TargetService { return ['target' => $target, 'created' => $created]; } + /** + * Change how an existing location is reached (credentials, host, folder, even the connection + * type) or its name. The new settings must lead to the same repository: the history stays + * and a typo cannot silently start a new, empty repository. A null option value keeps the + * stored one, so the web interface never has to send stored passwords back. + * + * @param array $options + */ + public function update(Target $target, string $name, string $backend, string $auth, array $options, string $basePath): Target { + $this->validateName($name, $target->getId()); + $this->requireUsableBackend($backend); + $stored = $this->options($target); + foreach ($options as $key => $value) { + if ($value === null) { + if (array_key_exists($key, $stored)) { + $options[$key] = $stored[$key]; + } else { + unset($options[$key]); + } + } + } + // Do not create the folder: a typo in the path must not leave an empty folder behind. + $storageBackend = $this->buildBackend($backend, $auth, $options, $basePath, false); + if (!$storageBackend->exists('config')) { + if (!$storageBackend->canConnect()) { + throw new \InvalidArgumentException('Cannot connect with these settings. Check the host, user name and password'); + } + throw new \InvalidArgumentException("No NG Backup repository found in folder '" . trim($basePath, '/') . "'. To start an empty repository, add a new location instead"); + } + $repo = Repository::openWithKey($storageBackend, $this->keys->keyRing(), $this->indexCache, $this->anchor); + if ($target->getRepositoryId() !== null && $repo->id() !== $target->getRepositoryId()) { + throw new \InvalidArgumentException('These settings lead to a different repository than the one of this location'); + } + $target->setName($name); + $target->setBackend($backend); + $target->setAuth($auth); + $target->setOptions($this->crypto->encrypt(json_encode($options, JSON_THROW_ON_ERROR))); + $target->setBasePath(trim($basePath, '/')); + return $this->mapper->update($target); + } + + /** + * The stored options for an edit form: password-type values are replaced by null, which + * update() reads as "keep the stored value". + * + * @return array + */ + public function editableOptions(Target $target): array { + $secret = []; + foreach ($this->availableBackends() as $b) { + if ($b['id'] !== $target->getBackend()) { + continue; + } + $params = $b['parameters']; + foreach ($b['auth'] as $a) { + if ($a['id'] === $target->getAuth()) { + $params += $a['parameters']; + } + } + foreach ($params as $key => $p) { + if (($p['type'] ?? 0) === 2) { + $secret[$key] = true; + } + } + } + $options = $this->options($target); + foreach ($options as $key => $value) { + if (isset($secret[$key])) { + $options[$key] = null; + } + } + return $options; + } + + private function validateName(string $name, ?int $ownId = null): void { + if (!preg_match('/^[A-Za-z0-9][A-Za-z0-9 _.-]{0,63}$/', $name)) { + throw new \InvalidArgumentException('Name: letters, digits, space, _ . - (max 64)'); + } + try { + $other = $this->mapper->findByName($name); + } catch (DoesNotExistException) { + return; + } + if ($other->getId() !== $ownId) { + throw new \InvalidArgumentException("A target named '$name' already exists"); + } + } + + private function requireUsableBackend(string $backend): void { + // Without the native extension files_external falls back to the smbclient binary, which + // registers fine but cannot seek: the first backup then fails with "Seek failed". Refuse + // up front instead of letting the admin find out from a failed run. + if ($backend === 'smb' && !extension_loaded('smbclient')) { + throw new \InvalidArgumentException('The SMB backend needs the PHP smbclient extension (pecl install smbclient); the smbclient binary alone cannot seek and every backup would fail'); + } + } + /** * A target's options are encrypted with Nextcloud's own instance secret (ICrypto), not * ng_backup's key, so changing that secret (occ backup:restore:full merging config.php) @@ -193,8 +279,9 @@ final class TargetService { return json_decode($this->crypto->decrypt($target->getOptions()), true, 512, JSON_THROW_ON_ERROR); } - private function buildBackend(string $backend, string $auth, array $options, string $basePath): StorageBackend { - return new StorageBackend($this->factory()->create($backend, $auth, $options), $basePath); + private function buildBackend(string $backend, string $auth, array $options, string $basePath, bool $createBase = true): StorageBackend { + $create = fn (): IStorage => $this->factory()->create($backend, $auth, $options); + return new StorageBackend($create(), $basePath, $createBase, $create); } private ?ExternalStorageFactory $factory = null; diff --git a/lib/Service/VerifyService.php b/lib/Service/VerifyService.php index fcfd68e..8d7486a 100644 --- a/lib/Service/VerifyService.php +++ b/lib/Service/VerifyService.php @@ -7,42 +7,152 @@ declare(strict_types=1); namespace OCA\NgBackup\Service; +use OCA\NgBackup\AppInfo\Application; use OCA\NgBackup\Db\Target; +use OCP\IAppConfig; use OCP\Lock\LockedException; -/** Checksum audit of a snapshot (occ backup:verify), see Repository::verify(). */ +/** + * Checksum audit of a snapshot (occ backup:verify), see Repository::verify(). On a large + * repository a check takes hours (a deep one days), so how far it is gets stored every + * PROGRESS_SECONDS, together with how it ended: `occ backup:status` shows it. + */ final class VerifyService { + public const PROGRESS_SECONDS = 30; + private const KEY_PREFIX = 'verify_progress_'; + public function __construct( private TargetService $targets, private LeaseService $leases, + private IAppConfig $appConfig, ) { } - /** @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list, failed:list, unreadable:list, readError:string} */ - public function verify(Target $target, string $snapshotId, bool $deep = false): array { - return $this->run($target, $deep, fn (\OCA\NgBackup\Repository\Repository $repo, callable $heartbeat): array => $repo->verify($snapshotId, $deep, $heartbeat)); + /** + * @param ?callable(array):void $onProgress called with the stored progress every PROGRESS_SECONDS + * @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list, failed:list, unreadable:list, readError:string} + */ + public function verify(Target $target, string $snapshotId, bool $deep = false, ?callable $onProgress = null): array { + return $this->run($target, $snapshotId, $deep, $onProgress, fn (\OCA\NgBackup\Repository\Repository $repo, callable $heartbeat): array => $repo->verify($snapshotId, $deep, $heartbeat)); } - /** @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list, failed:list, unreadable:list, readError:string} */ - public function verifyUserExport(Target $target, string $manifestPath, bool $deep = false): array { - return $this->run($target, $deep, fn (\OCA\NgBackup\Repository\Repository $repo, callable $heartbeat): array => $repo->verifyUserExport($manifestPath, $deep, $heartbeat)); + /** + * @param ?callable(array):void $onProgress called with the stored progress every PROGRESS_SECONDS + * @return array{filesChecked:int, blobsChecked:int, bytesChecked:int, missing:list, failed:list, unreadable:list, readError:string} + */ + public function verifyUserExport(Target $target, string $manifestPath, bool $deep = false, ?callable $onProgress = null): array { + return $this->run($target, $manifestPath, $deep, $onProgress, fn (\OCA\NgBackup\Repository\Repository $repo, callable $heartbeat): array => $repo->verifyUserExport($manifestPath, $deep, $heartbeat)); } - private function run(Target $target, bool $deep, callable $fn): array { + /** One line on a check's progress or outcome, for occ (see progress()). */ + public static function describe(array $s): string { + $size = static fn (int $b): string => $b >= 1073741824 ? sprintf('%.1f GiB', (float)$b / 1073741824.0) : sprintf('%.1f MiB', (float)$b / 1048576.0); + $count = static fn (int $done, mixed $total): string => number_format($done) + . (is_int($total) && $total > 0 ? '/' . number_format($total) . sprintf(' (%d%%)', intdiv(100 * $done, $total)) : ''); + $deep = ($s['deep'] ?? false) === true; + $kind = $deep ? 'deep check' : 'check'; + if (($s['status'] ?? '') === 'running') { + $phase = match ($s['phase'] ?? '') { + 'index' => 'index files ' . $count((int)$s['done'], $s['total'] ?? null), + 'packs' => 'pack folders ' . $count((int)$s['done'], $s['total'] ?? null), + 'files', 'database' => 'files ' . $count((int)$s['done'], $s['total'] ?? null) . ', ' . number_format((int)($s['blobs'] ?? 0)) . ' blobs' + . ($deep ? ', ' . $size((int)($s['bytes'] ?? 0)) . ' decrypted' : ''), + default => 'starting', + }; + $age = time() - (int)($s['updated'] ?? 0); + return sprintf('%s running since %s: %s; updated %s ago%s', $kind, date('Y-m-d H:i', (int)$s['started']), $phase, + $age < 120 ? $age . ' s' : intdiv($age, 60) . ' min', + $age > 600 ? ' (no progress for a while: the process may have stopped)' : ''); + } + $took = gmdate('H:i:s', max(0, (int)($s['finished'] ?? 0) - (int)($s['started'] ?? 0))); + return match ($s['status'] ?? '') { + 'ok' => sprintf('%s OK on %s (took %s): %s files, %s blobs%s', $kind, date('Y-m-d H:i', (int)$s['finished']), $took, + number_format((int)($s['files'] ?? 0)), number_format((int)($s['blobs'] ?? 0)), $deep ? ', ' . $size((int)($s['bytes'] ?? 0)) . ' decrypted' : ''), + 'problems' => sprintf('%s found PROBLEMS on %s: %d missing, %d failed, %d unreadable blob(s); run occ backup:verify again for the details', + $kind, date('Y-m-d H:i', (int)$s['finished']), (int)($s['missing'] ?? 0), (int)($s['failedBlobs'] ?? 0), (int)($s['unreadable'] ?? 0)), + default => sprintf('%s FAILED on %s: %s', $kind, date('Y-m-d H:i', (int)($s['finished'] ?? 0)), (string)($s['error'] ?? 'unknown error')), + }; + } + + /** + * The stored state of the last check per location: running (with how far it is) or how it + * ended. + * + * @return array> by location id + */ + public function progress(): array { + $result = []; + foreach ($this->appConfig->getKeys(Application::APP_ID) as $key) { + if (!str_starts_with($key, self::KEY_PREFIX)) { + continue; + } + $state = json_decode($this->appConfig->getValueString(Application::APP_ID, $key, '', lazy: true), true); + if (is_array($state)) { + $result[(int)substr($key, strlen(self::KEY_PREFIX))] = $state; + } + } + ksort($result); + return $result; + } + + private function run(Target $target, string $what, bool $deep, ?callable $onProgress, callable $fn): array { $ttl = $deep ? 600 : 300; + $state = ['what' => $what, 'deep' => $deep, 'status' => 'running', 'started' => time(), 'updated' => time(), + 'pid' => getmypid(), 'phase' => 'starting', 'done' => 0, 'total' => null, 'blobs' => 0, 'bytes' => 0]; try { - return $this->leases->with(PruneService::lockKey($target), LeaseService::SHARED, $ttl, function (callable $refresh) use ($target, $deep, $ttl, $fn): array { - $last = time(); - $heartbeat = function () use ($refresh, $ttl, &$last): void { - if (time() - $last >= intdiv($ttl, 2)) { + $this->store($target, $state); + } catch (\Throwable) { + // progress reports are a convenience; the check itself goes ahead + } + try { + $result = $this->leases->with(PruneService::lockKey($target), LeaseService::SHARED, $ttl, function (callable $refresh) use ($target, $ttl, $fn, $onProgress, &$state): array { + $lastRefresh = time(); + $lastStore = time(); + $heartbeat = function (?array $progress = null) use ($refresh, $ttl, $target, $onProgress, &$lastRefresh, &$lastStore, &$state): void { + $now = time(); + if ($now - $lastRefresh >= intdiv($ttl, 2)) { $refresh(); - $last = time(); + $lastRefresh = $now; + } + if ($progress !== null && $now - $lastStore >= self::PROGRESS_SECONDS) { + $state = array_merge($state, $progress, ['updated' => $now]); + try { + $this->store($target, $state); + } catch (\Throwable) { + // a progress report that cannot be stored must not stop the check + } + $lastStore = $now; + if ($onProgress !== null) { + $onProgress($state); + } } }; return $fn($this->targets->repository($target), $heartbeat); }); } catch (LockedException) { + $this->finish($target, $state, 'failed', ['error' => 'A cleanup of the location was in progress']); throw new \RuntimeException('A cleanup of ' . $target->getName() . ' is in progress; try again later'); + } catch (\Throwable $e) { + $this->finish($target, $state, 'failed', ['error' => $e->getMessage()]); + throw $e; + } + $problems = count($result['missing']) + count($result['failed']) + count($result['unreadable']); + $this->finish($target, $state, $problems === 0 ? 'ok' : 'problems', [ + 'phase' => 'done', 'files' => $result['filesChecked'], 'blobs' => $result['blobsChecked'], 'bytes' => $result['bytesChecked'], + 'missing' => count($result['missing']), 'failedBlobs' => count($result['failed']), 'unreadable' => count($result['unreadable']), + ]); + return $result; + } + + private function finish(Target $target, array $state, string $status, array $extra): void { + try { + $this->store($target, array_merge($state, $extra, ['status' => $status, 'updated' => time(), 'finished' => time()])); + } catch (\Throwable) { + // the check's own result counts; its stored summary is a convenience } } + + private function store(Target $target, array $state): void { + $this->appConfig->setValueString(Application::APP_ID, self::KEY_PREFIX . $target->getId(), json_encode($state, JSON_THROW_ON_ERROR), lazy: true); + } } diff --git a/psalm-baseline.xml b/psalm-baseline.xml index 2718b1c..984f91e 100644 --- a/psalm-baseline.xml +++ b/psalm-baseline.xml @@ -395,7 +395,7 @@ - + @@ -474,12 +474,6 @@ - - factory()->describeBackends()]]> - - - , parameters:array}>]]> - diff --git a/src/api.js b/src/api.js index 0c53981..f30c03b 100644 --- a/src/api.js +++ b/src/api.js @@ -27,6 +27,8 @@ export default { confirmKit: (accepted, phrase, code) => call('post', '/key/confirm', { accepted, phrase, code }), backends: () => call('get', '/backends'), addTarget: (target) => call('post', '/targets', target), + targetSettings: (id) => call('get', `/targets/${id}/settings`), + updateTarget: (id, target) => call('put', `/targets/${id}`, target), removeTarget: (id) => call('delete', `/targets/${id}`), testTarget: (id) => call('post', `/targets/${id}/test`), setTargetLimit: (id, maxGb) => call('put', `/targets/${id}/limit`, { maxGb }), diff --git a/src/components/Backups.vue b/src/components/Backups.vue index 3a09a28..547c9dd 100644 --- a/src/components/Backups.vue +++ b/src/components/Backups.vue @@ -49,7 +49,7 @@ {{ targetName(run.targetId) }} {{ run.kind }} {{ run.snapshot ? new Date(run.snapshot.createdAt * 1000).toLocaleString() + (run.snapshot.label ? ' – ' + run.snapshot.label : '') : '' }} - {{ run.status }}{{ run.status === 'running' ? ' (' + run.phase + ')' : '' }}{{ run.error ? ': ' + run.error : '' }} + {{ run.status }}{{ run.status === 'running' ? ' (' + run.phase + percent(run) + ')' : '' }}{{ run.error ? ': ' + run.error : '' }} {{ run.stats?.files?.files ?? run.stats?.restored ?? '' }} {{ run.stats?.files ? formatSize(run.stats.files.uploaded) : '' }} @@ -101,6 +101,8 @@ async function untrash(target, item) { } } +// " 68%", or " ~68%" when the total is estimated from Nextcloud's file cache (first backup) +const percent = (run) => run.progress ? ' ' + (run.progress.approximate ? '~' : '') + Math.floor(run.progress.fraction * 100) + '%' : '' const isRunning = (targetId) => props.status.runs.some(r => r.targetId === targetId && r.kind === 'full' && r.status === 'running') const targetName = (id) => props.status.targets.find(tg => tg.id === id)?.name ?? '#' + id diff --git a/src/components/Locations.vue b/src/components/Locations.vue index 2131c7c..c29dd1c 100644 --- a/src/components/Locations.vue +++ b/src/components/Locations.vue @@ -14,12 +14,16 @@ {{ tests[target.id].ok ? t('ng_backup', 'OK, {n} snapshots', { n: tests[target.id].snapshots }) : tests[target.id].error }} {{ t('ng_backup', 'Test') }} + {{ t('ng_backup', 'Edit') }} {{ t('ng_backup', 'Remove') }} - {{ t('ng_backup', 'Add location') }} -
+ {{ t('ng_backup', 'Add location') }} + + + {{ t('ng_backup', 'Editing "{name}". The new settings must lead to the same repository, so its backups and history are kept. Leave a password field empty to keep the stored password.', { name: editing.name }) }} + @@ -32,8 +36,8 @@
- {{ t('ng_backup', 'Connect and add') }} - {{ t('ng_backup', 'Cancel') }} + {{ editing ? t('ng_backup', 'Connect and save') : t('ng_backup', 'Connect and add') }} + {{ t('ng_backup', 'Cancel') }}
{{ error }} @@ -41,7 +45,7 @@