Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9a3bb99d12 | |||
| c38cc47534 | |||
| dfd3f38d8c | |||
| 92e693cfe6 | |||
| bec070b1b5 |
@@ -7,6 +7,35 @@ the app store accepts a `-beta` suffix, but Nextcloud servers then only offer th
|
|||||||
beta/daily update channel, which would hide the app from most admins. The stability stage is
|
beta/daily update channel, which would hide the app from most admins. The stability stage is
|
||||||
instead carried by the app description, the git tag, the release, and this file.
|
instead carried by the app description, the git tag, the release, and this file.
|
||||||
|
|
||||||
|
## [0.9.6-beta.7] - 2026-10-14
|
||||||
|
|
||||||
|
Bugfix release for SMB locations, with `occ backup:cancel` to stop a backup that does not get
|
||||||
|
anywhere. Tested on a production instance: a first full backup of 151,021 files (807 GiB) to OVH
|
||||||
|
S3 completed, and a backup to a Hetzner Storage Box over SMB that had slowed to 12 GiB in 35
|
||||||
|
hours ran at about 5.6 MiB/s after the update, while the S3 backup was still reading.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- The list of backup runs (web interface and `occ backup:status`) shows the total size of each
|
||||||
|
backup next to what it uploaded: an incremental backup of 1 TB that uploads 800 MB now shows
|
||||||
|
both. `occ backup:status` shows sizes in GiB where that reads better than MiB.
|
||||||
|
- `occ backup:cancel <run>` stops a running backup; `occ backup:status` shows the run numbers.
|
||||||
|
No new step of the run starts, the step in progress finishes (usually within a minute), then
|
||||||
|
the process working on the run (`occ backup:run` or cron) stops. What it already stored is
|
||||||
|
removed by the next cleanup. A backup started by an earlier version is stopped as well: after
|
||||||
|
two minutes it is marked cancelled while it works, and its process stops at the end of the
|
||||||
|
step.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Backups to SMB locations, and to other slow locations with many packs, slowed to a crawl: each
|
||||||
|
30-second step opened the location again, and reading the list of index files took longer
|
||||||
|
than the step itself. Each step then stored a single 4 MiB pack, which made the list longer
|
||||||
|
still. A first full backup of 0.8 TB to a Hetzner Storage Box over SMB managed 12 GiB in 35
|
||||||
|
hours. `occ backup:run` and system cron now open the location once per run, opening no longer
|
||||||
|
counts towards the step's time, and SMB folders are listed in one request instead of one per
|
||||||
|
file. A backup to an SMB location that is not getting anywhere keeps running in its old
|
||||||
|
process after the update: update first, then stop it with `occ backup:cancel <run>` and start
|
||||||
|
it again. The new backup reuses the packs the old one stored.
|
||||||
|
|
||||||
## [0.9.5-beta.6] - 2026-10-12
|
## [0.9.5-beta.6] - 2026-10-12
|
||||||
|
|
||||||
Bugfix release for SFTP locations and large repositories. Tested on a production instance with a
|
Bugfix release for SFTP locations and large repositories. Tested on a production instance with a
|
||||||
@@ -216,6 +245,7 @@ and not implemented yet.
|
|||||||
- See the "Storage backend experiences" section of the README for provider-specific quirks found
|
- See the "Storage backend experiences" section of the README for provider-specific quirks found
|
||||||
so far (e.g. Hetzner Storage Box sub-accounts).
|
so far (e.g. Hetzner Storage Box sub-accounts).
|
||||||
|
|
||||||
|
[0.9.6-beta.7]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.6-beta.7
|
||||||
[0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6
|
[0.9.5-beta.6]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.5-beta.6
|
||||||
[0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5
|
[0.9.4-beta.5]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.4-beta.5
|
||||||
[0.9.3-beta.4]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.3-beta.4
|
[0.9.3-beta.4]: https://github.com/dillardblom/ng_backup/releases/tag/v0.9.3-beta.4
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ installations and hosted Nextcloud. (Choosing SMB as a storage backend is the on
|
|||||||
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
see "Storage backend experiences" below — that requirement comes from files_external, not from
|
||||||
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
ng_backup, and applies to any Nextcloud app that writes to an SMB share.)
|
||||||
|
|
||||||
**Status: beta (0.9.5).** Tested thoroughly, usable alongside your current backup, but not yet a
|
**Status: beta (0.9.6).** Tested thoroughly, usable alongside your current backup, but not yet a
|
||||||
drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and
|
drop-in replacement for it. See "Known limitations" below, `PLAN.md` for the design and
|
||||||
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore.
|
`docs/HOWTO.md` for a step-by-step guide from setup to a tested restore.
|
||||||
|
|
||||||
@@ -195,6 +195,7 @@ On the backlog; none of these block the beta or 1.0.0, but know them before you
|
|||||||
| `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots |
|
| `backup:key:slots`, `backup:key:slot:add/replace/remove` | passphrase slots |
|
||||||
| `backup:target:backends/add/edit/list/remove/append-only` | backup locations |
|
| `backup:target:backends/add/edit/list/remove/append-only` | backup locations |
|
||||||
| `backup:run`, `backup:status`, `backup:list` | back up, status, restore points |
|
| `backup:run`, `backup:status`, `backup:list` | back up, status, restore points |
|
||||||
|
| `backup:cancel <run>` | stop a running backup |
|
||||||
| `backup:browse`, `backup:restore` | browse and restore |
|
| `backup:browse`, `backup:restore` | browse and restore |
|
||||||
| `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash |
|
| `backup:retention`, `backup:prune`, `backup:trash`, `backup:trash:restore` | retention, cleanup, trash |
|
||||||
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
|
| `backup:verify` | checksum audit of a restore point (`--deep` downloads and decrypts everything) |
|
||||||
|
|||||||
+2
-1
@@ -9,7 +9,7 @@ directory to S3, SFTP, WebDAV, FTP, SMB or a local path, with restore of files a
|
|||||||
single users (via user_migration) and of the whole server onto a fresh installation. Tested
|
single users (via user_migration) and of the whole server onto a fresh installation. Tested
|
||||||
thoroughly and usable alongside your current backup, but not yet a drop-in replacement for it.
|
thoroughly and usable alongside your current backup, but not yet a drop-in replacement for it.
|
||||||
See the README for known limitations and the recovery kit before relying on it.]]></description>
|
See the README for known limitations and the recovery kit before relying on it.]]></description>
|
||||||
<version>0.9.5</version>
|
<version>0.9.6</version>
|
||||||
<licence>agpl</licence>
|
<licence>agpl</licence>
|
||||||
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
|
<author mail="dillard.blom@ensembia.com">Dillard Blom</author>
|
||||||
<namespace>NgBackup</namespace>
|
<namespace>NgBackup</namespace>
|
||||||
@@ -41,6 +41,7 @@ See the README for known limitations and the recovery kit before relying on it.]
|
|||||||
<command>OCA\NgBackup\Command\TargetList</command>
|
<command>OCA\NgBackup\Command\TargetList</command>
|
||||||
<command>OCA\NgBackup\Command\TargetRemove</command>
|
<command>OCA\NgBackup\Command\TargetRemove</command>
|
||||||
<command>OCA\NgBackup\Command\RunBackup</command>
|
<command>OCA\NgBackup\Command\RunBackup</command>
|
||||||
|
<command>OCA\NgBackup\Command\CancelRun</command>
|
||||||
<command>OCA\NgBackup\Command\Status</command>
|
<command>OCA\NgBackup\Command\Status</command>
|
||||||
<command>OCA\NgBackup\Command\ListSnapshots</command>
|
<command>OCA\NgBackup\Command\ListSnapshots</command>
|
||||||
<command>OCA\NgBackup\Command\Browse</command>
|
<command>OCA\NgBackup\Command\Browse</command>
|
||||||
|
|||||||
@@ -50,6 +50,14 @@ occ backup:target:list
|
|||||||
occ backup:status
|
occ backup:status
|
||||||
```
|
```
|
||||||
|
|
||||||
|
A backup that hangs or takes far too long can be stopped. `backup:status` shows the run number;
|
||||||
|
the command waits until the step in progress is finished, then the process working on the run
|
||||||
|
stops, and the next cleanup removes what it stored:
|
||||||
|
|
||||||
|
```
|
||||||
|
occ backup:cancel 7
|
||||||
|
```
|
||||||
|
|
||||||
Changed password, new host name, or moved the repository? Edit the location instead of adding a
|
Changed password, new host name, or moved the repository? Edit the location instead of adding a
|
||||||
new one, so its history stays (in the web interface: **Edit** next to the location). The new
|
new one, so its history stays (in the web interface: **Edit** next to the location). The new
|
||||||
settings must lead to the same repository; options you do not pass keep their value:
|
settings must lead to the same repository; options you do not pass keep their value:
|
||||||
|
|||||||
@@ -223,12 +223,23 @@ final class StorageBackend implements IBackend {
|
|||||||
if (!$this->storage->is_dir($abs)) {
|
if (!$this->storage->is_dir($abs)) {
|
||||||
return []; // not created yet
|
return []; // not created yet
|
||||||
}
|
}
|
||||||
|
$cut = $this->base === '' ? 0 : strlen($this->base) + 1;
|
||||||
|
$names = $this->smbNames($abs);
|
||||||
|
if ($names !== null) {
|
||||||
|
$result = [];
|
||||||
|
foreach ($names as $name) {
|
||||||
|
if (!self::isTemporary($name)) {
|
||||||
|
$result[] = substr($abs . '/' . $name, $cut);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort($result);
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
$dh = $this->storage->opendir($abs);
|
$dh = $this->storage->opendir($abs);
|
||||||
if ($dh === false) {
|
if ($dh === false) {
|
||||||
// An existing folder read as empty would look like a new repository.
|
// An existing folder read as empty would look like a new repository.
|
||||||
throw new BackendException('Cannot list ' . $dir);
|
throw new BackendException('Cannot list ' . $dir);
|
||||||
}
|
}
|
||||||
$cut = $this->base === '' ? 0 : strlen($this->base) + 1;
|
|
||||||
$result = [];
|
$result = [];
|
||||||
while (($name = readdir($dh)) !== false) {
|
while (($name = readdir($dh)) !== false) {
|
||||||
if ($name !== '.' && $name !== '..' && !self::isTemporary($name)) {
|
if ($name !== '.' && $name !== '..' && !self::isTemporary($name)) {
|
||||||
@@ -240,6 +251,46 @@ final class StorageBackend implements IBackend {
|
|||||||
return $result;
|
return $result;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The names in an SMB folder, read in one listing; null for other storages, with access
|
||||||
|
* control lists on, or when the storage's internals are not as expected. The SMB storage's
|
||||||
|
* opendir() asks the server for the attributes of each entry to leave out hidden ones: with
|
||||||
|
* thousands of index files that took minutes. Hidden entries are still left out: names
|
||||||
|
* starting with a dot, and, after asking, any name a repository does not use (letters,
|
||||||
|
* digits, - and _), such as desktop.ini or Thumbs.db. The repository's own files have such
|
||||||
|
* names: they are not checked one by one, which was the cost.
|
||||||
|
*
|
||||||
|
* @return list<string>|null
|
||||||
|
*/
|
||||||
|
private function smbNames(string $abs): ?array {
|
||||||
|
if (!$this->storage instanceof \OCA\Files_External\Lib\Storage\SMB) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
/** @var list<string>|null */
|
||||||
|
return (function (string $path): ?array {
|
||||||
|
/** @psalm-suppress UndefinedThisPropertyFetch */
|
||||||
|
if ($this->checkAcl) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
$names = [];
|
||||||
|
/** @psalm-suppress UndefinedThisPropertyFetch, UndefinedMethod */
|
||||||
|
foreach ($this->share->dir(ltrim($this->buildPath($path), '/')) as $info) {
|
||||||
|
$name = (string)$info->getName();
|
||||||
|
/** @psalm-suppress UndefinedThisPropertyFetch */
|
||||||
|
if (!$this->showHidden && (str_starts_with($name, '.')
|
||||||
|
|| (preg_match('/^[A-Za-z0-9_-]+$/', $name) !== 1 && $info->isHidden()))) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
$names[] = $name;
|
||||||
|
}
|
||||||
|
return $names;
|
||||||
|
})->call($this->storage, $abs);
|
||||||
|
} catch (\Throwable) {
|
||||||
|
return null; // another SMB storage version, or an error: the usual way reports it
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Run $op; when it throws, or $failed says its result may come from a dead connection, and
|
* Run $op; when it throws, or $failed says its result may come from a dead connection, and
|
||||||
* the SFTP connection turns out to be dead, run it once more on a new connection.
|
* the SFTP connection turns out to be dead, run it once more on a new connection.
|
||||||
|
|||||||
@@ -0,0 +1,74 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// SPDX-FileCopyrightText: 2026 Dillard Blom
|
||||||
|
// SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
namespace OCA\NgBackup\Command;
|
||||||
|
|
||||||
|
use OCA\NgBackup\Db\Run;
|
||||||
|
use OCA\NgBackup\Db\RunMapper;
|
||||||
|
use OCA\NgBackup\Service\BackupService;
|
||||||
|
use OCP\AppFramework\Db\DoesNotExistException;
|
||||||
|
use Symfony\Component\Console\Command\Command;
|
||||||
|
use Symfony\Component\Console\Input\InputArgument;
|
||||||
|
use Symfony\Component\Console\Input\InputInterface;
|
||||||
|
use Symfony\Component\Console\Output\OutputInterface;
|
||||||
|
|
||||||
|
final class CancelRun extends Command {
|
||||||
|
public function __construct(
|
||||||
|
private RunMapper $runs,
|
||||||
|
private BackupService $backups,
|
||||||
|
) {
|
||||||
|
parent::__construct();
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function configure(): void {
|
||||||
|
$this->setName('backup:cancel')
|
||||||
|
->setDescription('Stop a running backup')
|
||||||
|
->addArgument('run', InputArgument::REQUIRED, 'Run number, as shown by backup:status')
|
||||||
|
->setHelp("No new step of the run starts; the step in progress finishes first (usually within a
|
||||||
|
"
|
||||||
|
. "minute), so a backup that is just being completed is not cut off. The process working on the
|
||||||
|
"
|
||||||
|
. "run (occ backup:run, or cron) then stops. A backup started by an earlier version is marked
|
||||||
|
"
|
||||||
|
. "cancelled after two minutes and stops at the end of its step. What it already stored is
|
||||||
|
"
|
||||||
|
. "removed by the next cleanup.");
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function execute(InputInterface $input, OutputInterface $output): int {
|
||||||
|
$id = (string)$input->getArgument('run');
|
||||||
|
if (!ctype_digit($id)) {
|
||||||
|
$output->writeln('<error>Give the run number, as shown by backup:status.</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$run = $this->runs->find((int)$id);
|
||||||
|
} catch (DoesNotExistException) {
|
||||||
|
$output->writeln("<error>There is no run $id.</error>");
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
if ($run->getKind() !== BackupService::KIND_FULL) {
|
||||||
|
$output->writeln("<error>Run $id is not a backup; only backups can be cancelled.</error>");
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
if ($run->getStatus() === Run::RUNNING) {
|
||||||
|
$output->writeln("Waiting for the current step of run $id to finish...");
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$run = $this->backups->cancel($run);
|
||||||
|
} catch (\RuntimeException $e) {
|
||||||
|
$output->writeln('<error>' . $e->getMessage() . '</error>');
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
if ($run->getStatus() !== Run::CANCELLED) {
|
||||||
|
$output->writeln("Run $id is not running (status: {$run->getStatus()}); nothing to cancel.");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
$output->writeln("Run $id is cancelled.");
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
+14
-2
@@ -48,10 +48,11 @@ final class Status extends Command {
|
|||||||
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
|
$s = json_decode($r->getStats() ?? '{}', true)['files'] ?? [];
|
||||||
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''),
|
$rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''),
|
||||||
date('Y-m-d H:i', $r->getStartedAt()), $r->getFinishedAt() ? gmdate('H:i:s', $r->getFinishedAt() - $r->getStartedAt()) : '',
|
date('Y-m-d H:i', $r->getStartedAt()), $r->getFinishedAt() ? gmdate('H:i:s', $r->getFinishedAt() - $r->getStartedAt()) : '',
|
||||||
$s['files'] ?? '', isset($s['uploaded']) ? sprintf('%.1f MiB', $s['uploaded'] / 1048576) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
|
$s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '',
|
||||||
|
isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')];
|
||||||
}
|
}
|
||||||
if ($rows !== []) {
|
if ($rows !== []) {
|
||||||
(new Table($output))->setHeaders(['run', 'location', 'kind', 'status', 'started', 'took', 'files', 'uploaded', 'error'])->setRows($rows)->render();
|
(new Table($output))->setHeaders(['run', 'location', 'kind', 'status', 'started', 'took', 'files', 'size', 'uploaded', 'error'])->setRows($rows)->render();
|
||||||
}
|
}
|
||||||
foreach ($this->verify->progress() as $targetId => $state) {
|
foreach ($this->verify->progress() as $targetId => $state) {
|
||||||
$output->writeln(sprintf('Verify of %s on %s: %s', mb_strimwidth((string)($state['what'] ?? ''), 0, 40, '…'),
|
$output->writeln(sprintf('Verify of %s on %s: %s', mb_strimwidth((string)($state['what'] ?? ''), 0, 40, '…'),
|
||||||
@@ -61,6 +62,17 @@ final class Status extends Command {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** " 68%", or " ~68%" when estimated from the file cache; empty without an estimate */
|
/** " 68%", or " ~68%" when estimated from the file cache; empty without an estimate */
|
||||||
|
private static function size(int $bytes): string {
|
||||||
|
$b = (float)$bytes;
|
||||||
|
foreach (['B', 'KiB', 'MiB', 'GiB'] as $u) {
|
||||||
|
if ($b < 1024) {
|
||||||
|
return $u === 'B' ? "$bytes B" : sprintf('%.1f %s', $b, $u);
|
||||||
|
}
|
||||||
|
$b /= 1024.0;
|
||||||
|
}
|
||||||
|
return sprintf('%.1f TiB', $b);
|
||||||
|
}
|
||||||
|
|
||||||
private static function percent(\OCA\NgBackup\Db\Run $r): string {
|
private static function percent(\OCA\NgBackup\Db\Run $r): string {
|
||||||
$p = \OCA\NgBackup\Job\BackupRun::progressOf($r);
|
$p = \OCA\NgBackup\Job\BackupRun::progressOf($r);
|
||||||
return $p === null ? '' : sprintf(' %s%d%%', $p['approximate'] ? '~' : '', (int)floor($p['fraction'] * 100.0));
|
return $p === null ? '' : sprintf(' %s%d%%', $p['approximate'] ? '~' : '', (int)floor($p['fraction'] * 100.0));
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ use OCA\NgBackup\Db\Snapshot;
|
|||||||
use OCA\NgBackup\Db\SnapshotMapper;
|
use OCA\NgBackup\Db\SnapshotMapper;
|
||||||
use OCA\NgBackup\Db\Target;
|
use OCA\NgBackup\Db\Target;
|
||||||
use OCA\NgBackup\Job\BackupRun;
|
use OCA\NgBackup\Job\BackupRun;
|
||||||
|
use OCA\NgBackup\Repository\Repository;
|
||||||
use OCA\NgBackup\Repository\RepositoryException;
|
use OCA\NgBackup\Repository\RepositoryException;
|
||||||
use OCP\AppFramework\Db\DoesNotExistException;
|
use OCP\AppFramework\Db\DoesNotExistException;
|
||||||
use OCP\App\IAppManager;
|
use OCP\App\IAppManager;
|
||||||
@@ -34,6 +35,9 @@ use Psr\Log\LoggerInterface;
|
|||||||
final class BackupService {
|
final class BackupService {
|
||||||
public const KIND_FULL = 'full';
|
public const KIND_FULL = 'full';
|
||||||
|
|
||||||
|
/** @var \WeakMap<Repository, string> the location settings each kept repository was opened with */
|
||||||
|
private \WeakMap $openedWith;
|
||||||
|
|
||||||
public function __construct(
|
public function __construct(
|
||||||
private TargetService $targets,
|
private TargetService $targets,
|
||||||
private KeyService $keys,
|
private KeyService $keys,
|
||||||
@@ -47,6 +51,9 @@ final class BackupService {
|
|||||||
private AlertService $alerts,
|
private AlertService $alerts,
|
||||||
private LeaseService $leases,
|
private LeaseService $leases,
|
||||||
) {
|
) {
|
||||||
|
/** @var \WeakMap<Repository, string> $opened */
|
||||||
|
$opened = new \WeakMap();
|
||||||
|
$this->openedWith = $opened;
|
||||||
}
|
}
|
||||||
|
|
||||||
public function start(Target $target, string $label = ''): Run {
|
public function start(Target $target, string $label = ''): Run {
|
||||||
@@ -144,13 +151,22 @@ final class BackupService {
|
|||||||
/**
|
/**
|
||||||
* Do one step of a run until $deadline. Safe to call concurrently: a second caller finds the
|
* Do one step of a run until $deadline. Safe to call concurrently: a second caller finds the
|
||||||
* run locked and returns immediately.
|
* run locked and returns immediately.
|
||||||
|
*
|
||||||
|
* @param Repository|null $repo the location's repository, opened by an earlier step of this run
|
||||||
|
* in this process (see runToCompletion()); set when this step opens it
|
||||||
*/
|
*/
|
||||||
public function step(Run $run, float $deadline): Run {
|
public function step(Run $run, float $deadline, ?Repository &$repo = null): Run {
|
||||||
// Leases expire on their own: a step killed by a time limit or OOM blocks nothing for long.
|
// Leases expire on their own: a step killed by a time limit or OOM blocks nothing for long.
|
||||||
// A step can run well past its deadline (the commit of a large snapshot writes its tree and
|
// A step can run well past its deadline (the commit of a large snapshot writes its tree and
|
||||||
// index), so the lease holds at least 15 minutes; should it still expire and another process
|
// index), so the lease holds at least 15 minutes; should it still expire and another process
|
||||||
// take the run over, the checks below keep the two from overwriting each other.
|
// take the run over, the checks below keep the two from overwriting each other.
|
||||||
$ttl = (int)min(3600, max(900, $deadline - microtime(true) + 600));
|
$ttl = (int)min(3600, max(900, $deadline - microtime(true) + 600));
|
||||||
|
// No new step once a cancel has begun: it waits for the step in progress, if any.
|
||||||
|
try {
|
||||||
|
$this->leases->release($this->leases->acquire(self::cancelKey($run->getId()), LeaseService::SHARED, 60));
|
||||||
|
} catch (LockedException) {
|
||||||
|
return $run;
|
||||||
|
}
|
||||||
try {
|
try {
|
||||||
$runLease = $this->leases->acquire('run/' . $run->getId(), LeaseService::EXCLUSIVE, $ttl);
|
$runLease = $this->leases->acquire('run/' . $run->getId(), LeaseService::EXCLUSIVE, $ttl);
|
||||||
} catch (LockedException) {
|
} catch (LockedException) {
|
||||||
@@ -164,6 +180,7 @@ final class BackupService {
|
|||||||
return $run;
|
return $run;
|
||||||
}
|
}
|
||||||
$loadedAt = null;
|
$loadedAt = null;
|
||||||
|
$savedBefore = $run->getUpdatedAt();
|
||||||
try {
|
try {
|
||||||
$run = $this->runs->find($run->getId()); // fresh state after acquiring the lock
|
$run = $this->runs->find($run->getId()); // fresh state after acquiring the lock
|
||||||
if ($run->getStatus() !== Run::RUNNING || $run->getKind() !== self::KIND_FULL) {
|
if ($run->getStatus() !== Run::RUNNING || $run->getKind() !== self::KIND_FULL) {
|
||||||
@@ -171,7 +188,19 @@ final class BackupService {
|
|||||||
}
|
}
|
||||||
$loadedAt = $run->getUpdatedAt();
|
$loadedAt = $run->getUpdatedAt();
|
||||||
$target = $this->targets->get((string)$run->getTargetId());
|
$target = $this->targets->get((string)$run->getTargetId());
|
||||||
$repo = $this->targets->repository($target);
|
// Reopen when another process took a step since ours (its packs are not in our copy of
|
||||||
|
// the index) or the location was edited (password, folder, size limit).
|
||||||
|
$settings = json_encode([$target->getBackend(), $target->getAuth(), $target->getOptions(),
|
||||||
|
$target->getBasePath(), $target->getRepositoryId(), $target->getMaxBytes()], JSON_THROW_ON_ERROR);
|
||||||
|
if ($repo === null || $loadedAt !== $savedBefore || ($this->openedWith[$repo] ?? null) !== $settings) {
|
||||||
|
// Opening reads the index listing, which takes minutes on a slow location with many
|
||||||
|
// packs: that time does not count, or each step would store one blob and stop.
|
||||||
|
$opening = microtime(true);
|
||||||
|
$repo = null;
|
||||||
|
$repo = $this->targets->repository($target);
|
||||||
|
$this->openedWith[$repo] = $settings;
|
||||||
|
$deadline += microtime(true) - $opening;
|
||||||
|
}
|
||||||
$state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR);
|
$state = json_decode($run->getState(), true, 512, JSON_THROW_ON_ERROR);
|
||||||
|
|
||||||
if ($run->getPhase() === 'db') {
|
if ($run->getPhase() === 'db') {
|
||||||
@@ -234,8 +263,14 @@ final class BackupService {
|
|||||||
$run->setStats(json_encode(['db' => $state['dbStats'] ?? null, 'files' => $state['files']['stats']], JSON_THROW_ON_ERROR));
|
$run->setStats(json_encode(['db' => $state['dbStats'] ?? null, 'files' => $state['files']['stats']], JSON_THROW_ON_ERROR));
|
||||||
// Another process may have taken the run over after this step's lease expired: then its
|
// Another process may have taken the run over after this step's lease expired: then its
|
||||||
// writes stand and this step's are dropped.
|
// writes stand and this step's are dropped.
|
||||||
return $this->runs->updateIfUnchanged($run, $loadedAt) ?? $this->runs->find($run->getId());
|
$saved = $this->runs->updateIfUnchanged($run, $loadedAt);
|
||||||
|
if ($saved === null) {
|
||||||
|
$repo = null; // the run is another process's now: its packs are not in our index
|
||||||
|
return $this->runs->find($run->getId());
|
||||||
|
}
|
||||||
|
return $saved;
|
||||||
} catch (\Throwable $e) {
|
} catch (\Throwable $e) {
|
||||||
|
$repo = null; // may hold blobs of a half-written pack: the next step opens it afresh
|
||||||
if ($loadedAt === null) {
|
if ($loadedAt === null) {
|
||||||
// The run could not even be loaded: there is nothing fresh to mark as failed, and
|
// The run could not even be loaded: there is nothing fresh to mark as failed, and
|
||||||
// the caller's copy may be stale.
|
// the caller's copy may be stale.
|
||||||
@@ -266,14 +301,16 @@ final class BackupService {
|
|||||||
|
|
||||||
/** Run to completion in this process (occ, system cron), reporting progress. */
|
/** Run to completion in this process (occ, system cron), reporting progress. */
|
||||||
public function runToCompletion(Run $run, ?callable $progress = null, float $stepSeconds = 30): Run {
|
public function runToCompletion(Run $run, ?callable $progress = null, float $stepSeconds = 30): Run {
|
||||||
|
$repo = null; // opened once, by the first step, and kept for the next ones
|
||||||
while ($run->getStatus() === Run::RUNNING) {
|
while ($run->getStatus() === Run::RUNNING) {
|
||||||
$before = $run->getUpdatedAt();
|
$before = $run->getUpdatedAt();
|
||||||
$run = $this->step($run, microtime(true) + $stepSeconds);
|
$run = $this->step($run, microtime(true) + $stepSeconds, $repo);
|
||||||
if ($run->getStatus() === Run::RUNNING && $run->getUpdatedAt() === $before) {
|
if ($run->getStatus() === Run::RUNNING && $run->getUpdatedAt() === $before) {
|
||||||
// Step was not possible (another process holds the run, or a crashed step's lease has
|
// Step was not possible (another process holds the run, or a crashed step's lease has
|
||||||
// not expired yet): wait instead of spinning.
|
// not expired yet): wait instead of spinning.
|
||||||
sleep(5);
|
sleep(5);
|
||||||
$run = $this->runs->find($run->getId());
|
$run = $this->runs->find($run->getId());
|
||||||
|
$repo = null; // another process may have stored packs in the meantime
|
||||||
}
|
}
|
||||||
if ($progress !== null) {
|
if ($progress !== null) {
|
||||||
$progress($run);
|
$progress($run);
|
||||||
@@ -282,11 +319,70 @@ final class BackupService {
|
|||||||
return $run;
|
return $run;
|
||||||
}
|
}
|
||||||
|
|
||||||
public function cancel(Run $run): Run {
|
/**
|
||||||
$run->setStatus(Run::CANCELLED);
|
* Stop a running backup. First no new step may start, then this waits (up to $wait seconds)
|
||||||
$run->setFinishedAt(time());
|
* for the step in progress, so a step that is recording the snapshot finishes first. A process
|
||||||
$run->setUpdatedAt(time());
|
* running the backup then finds it cancelled and stops. The packs it stored stay until the
|
||||||
return $this->runs->update($run);
|
* next cleanup.
|
||||||
|
*
|
||||||
|
* A process of an earlier version does not hold back its next step and may never leave a gap:
|
||||||
|
* after $wait the run is marked cancelled while it works, unless it is recording the snapshot.
|
||||||
|
* Its step then fails to save and the process stops.
|
||||||
|
*/
|
||||||
|
public function cancel(Run $run, int $wait = 120): Run {
|
||||||
|
$id = $run->getId();
|
||||||
|
$until = time() + $wait;
|
||||||
|
$hold = null;
|
||||||
|
for ($try = 0; $hold === null; $try++) {
|
||||||
|
try {
|
||||||
|
$hold = $this->leases->acquire(self::cancelKey($id), LeaseService::EXCLUSIVE, $wait + 300);
|
||||||
|
} catch (LockedException) {
|
||||||
|
if ($try >= 20) {
|
||||||
|
throw new \RuntimeException("Run $id is being cancelled already");
|
||||||
|
}
|
||||||
|
usleep(100000); // a step checking whether it may start
|
||||||
|
}
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$lease = null;
|
||||||
|
while ($lease === null && time() < $until) {
|
||||||
|
try {
|
||||||
|
$lease = $this->leases->acquire('run/' . $id, LeaseService::EXCLUSIVE, 60);
|
||||||
|
} catch (LockedException) {
|
||||||
|
sleep(2);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
for ($try = 0; $try < 10; $try++) {
|
||||||
|
$run = $this->runs->find($id);
|
||||||
|
if ($run->getStatus() !== Run::RUNNING) {
|
||||||
|
return $run;
|
||||||
|
}
|
||||||
|
$state = json_decode($run->getState(), true) ?: [];
|
||||||
|
if ($lease === null && ($state['files']['phase'] ?? '') === 'commit') {
|
||||||
|
throw new \RuntimeException("Run $id is recording its snapshot; try again in a minute");
|
||||||
|
}
|
||||||
|
$loadedAt = $run->getUpdatedAt();
|
||||||
|
$run->setStatus(Run::CANCELLED);
|
||||||
|
$run->setFinishedAt(time());
|
||||||
|
if (($cancelled = $this->runs->updateIfUnchanged($run, $loadedAt)) !== null) {
|
||||||
|
return $cancelled;
|
||||||
|
}
|
||||||
|
usleep(200000); // a step of another process saved just now
|
||||||
|
}
|
||||||
|
throw new \RuntimeException("Run $id keeps changing; try again");
|
||||||
|
} finally {
|
||||||
|
if ($lease !== null) {
|
||||||
|
$this->leases->release($lease);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
$this->leases->release($hold);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static function cancelKey(int $runId): string {
|
||||||
|
return 'run/' . $runId . '/cancel';
|
||||||
}
|
}
|
||||||
|
|
||||||
private function recordSnapshot(Run $run, array $files): void {
|
private function recordSnapshot(Run $run, array $files): void {
|
||||||
|
|||||||
@@ -40,6 +40,7 @@
|
|||||||
<th>{{ t('ng_backup', 'Restore point') }}</th>
|
<th>{{ t('ng_backup', 'Restore point') }}</th>
|
||||||
<th>{{ t('ng_backup', 'Status') }}</th>
|
<th>{{ t('ng_backup', 'Status') }}</th>
|
||||||
<th>{{ t('ng_backup', 'Files') }}</th>
|
<th>{{ t('ng_backup', 'Files') }}</th>
|
||||||
|
<th :title="t('ng_backup', 'Total size of the files in this backup, unchanged files included')">{{ t('ng_backup', 'Size') }}</th>
|
||||||
<th>{{ t('ng_backup', 'Uploaded') }}</th>
|
<th>{{ t('ng_backup', 'Uploaded') }}</th>
|
||||||
</tr>
|
</tr>
|
||||||
</thead>
|
</thead>
|
||||||
@@ -51,6 +52,7 @@
|
|||||||
<td>{{ run.snapshot ? new Date(run.snapshot.createdAt * 1000).toLocaleString() + (run.snapshot.label ? ' – ' + run.snapshot.label : '') : '' }}</td>
|
<td>{{ run.snapshot ? new Date(run.snapshot.createdAt * 1000).toLocaleString() + (run.snapshot.label ? ' – ' + run.snapshot.label : '') : '' }}</td>
|
||||||
<td :class="'ngb-' + run.status">{{ run.status }}{{ run.status === 'running' ? ' (' + run.phase + percent(run) + ')' : '' }}{{ run.error ? ': ' + run.error : '' }}</td>
|
<td :class="'ngb-' + run.status">{{ run.status }}{{ run.status === 'running' ? ' (' + run.phase + percent(run) + ')' : '' }}{{ run.error ? ': ' + run.error : '' }}</td>
|
||||||
<td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
|
<td>{{ run.stats?.files?.files ?? run.stats?.restored ?? '' }}</td>
|
||||||
|
<td>{{ run.stats?.files ? formatSize(run.stats.files.bytes) : '' }}</td>
|
||||||
<td>{{ run.stats?.files ? formatSize(run.stats.files.uploaded) : '' }}</td>
|
<td>{{ run.stats?.files ? formatSize(run.stats.files.uploaded) : '' }}</td>
|
||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
|
|||||||
@@ -10,6 +10,8 @@ use OCA\NgBackup\Tests\Unit\TempDirTrait;
|
|||||||
use OCP\Files\Storage\IStorage;
|
use OCP\Files\Storage\IStorage;
|
||||||
use PHPUnit\Framework\TestCase;
|
use PHPUnit\Framework\TestCase;
|
||||||
|
|
||||||
|
require_once __DIR__ . '/../../stubs/SmbStorage.php';
|
||||||
|
|
||||||
/** On a remote storage every is_dir() is a round trip: listing index/ must not cost one per pack. */
|
/** On a remote storage every is_dir() is a round trip: listing index/ must not cost one per pack. */
|
||||||
class StorageBackendListTest extends TestCase {
|
class StorageBackendListTest extends TestCase {
|
||||||
use TempDirTrait;
|
use TempDirTrait;
|
||||||
@@ -54,4 +56,58 @@ class StorageBackendListTest extends TestCase {
|
|||||||
$this->expectException(BackendException::class);
|
$this->expectException(BackendException::class);
|
||||||
(new StorageBackend($storage, 'repo'))->listFiles('catalog');
|
(new StorageBackend($storage, 'repo'))->listFiles('catalog');
|
||||||
}
|
}
|
||||||
|
public function testSmbFoldersAreListedInOneRequest(): void {
|
||||||
|
$calls = new \ArrayObject();
|
||||||
|
$share = new class($calls) {
|
||||||
|
public function __construct(private \ArrayObject $calls) {
|
||||||
|
}
|
||||||
|
|
||||||
|
public function dir(string $path): array {
|
||||||
|
$this->calls[] = "dir $path";
|
||||||
|
return array_map(fn (string $n) => new class($n, $this->calls) {
|
||||||
|
public function __construct(private string $n, private \ArrayObject $calls) {
|
||||||
|
}
|
||||||
|
|
||||||
|
public function getName(): string {
|
||||||
|
return $this->n;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function isHidden(): bool {
|
||||||
|
$this->calls[] = "attributes {$this->n}";
|
||||||
|
return $this->n === 'desktop.ini';
|
||||||
|
}
|
||||||
|
}, ['b1', 'a2', '.hidden', 'desktop.ini', 'c3.part-0123abcd']);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
$storage = $this->getMockBuilder(\OCA\Files_External\Lib\Storage\SMB::class)->disableOriginalConstructor()->getMock();
|
||||||
|
(fn () => $this->share = $share)->call($storage);
|
||||||
|
$storage->method('instanceOfStorage')->willReturn(false);
|
||||||
|
$storage->method('is_dir')->willReturn(true);
|
||||||
|
$storage->expects($this->never())->method('opendir');
|
||||||
|
|
||||||
|
$this->assertSame(['index/a2', 'index/b1'], (new StorageBackend($storage, 'repo'))->listFiles('index'));
|
||||||
|
$this->assertSame(['dir repo/index', 'attributes desktop.ini', 'attributes c3.part-0123abcd'], $calls->getArrayCopy(),
|
||||||
|
'only names a repository does not use cost a request');
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testSmbWithAccessControlListsIsListedTheUsualWay(): void {
|
||||||
|
$dir = $this->tempDir();
|
||||||
|
touch("$dir/a1");
|
||||||
|
$storage = $this->getMockBuilder(\OCA\Files_External\Lib\Storage\SMB::class)->disableOriginalConstructor()->getMock();
|
||||||
|
(fn () => $this->checkAcl = true)->call($storage);
|
||||||
|
$storage->method('instanceOfStorage')->willReturn(false);
|
||||||
|
$storage->method('is_dir')->willReturn(true);
|
||||||
|
$storage->expects($this->once())->method('opendir')->willReturnCallback(fn () => opendir($dir));
|
||||||
|
$this->assertSame(['index/a1'], (new StorageBackend($storage, 'repo'))->listFiles('index'));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testSmbFallsBackToTheUsualListing(): void {
|
||||||
|
$dir = $this->tempDir();
|
||||||
|
touch("$dir/a1");
|
||||||
|
$storage = $this->getMockBuilder(\OCA\Files_External\Lib\Storage\SMB::class)->disableOriginalConstructor()->getMock(); // no share
|
||||||
|
$storage->method('instanceOfStorage')->willReturn(false);
|
||||||
|
$storage->method('is_dir')->willReturn(true);
|
||||||
|
$storage->method('opendir')->willReturnCallback(fn () => opendir($dir));
|
||||||
|
$this->assertSame(['index/a1'], (new StorageBackend($storage, 'repo'))->listFiles('index'));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -68,7 +68,11 @@ run_on ngb-frt-restore backup:target:add offsite local -a null::null -o datadir=
|
|||||||
run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location"
|
run_on ngb-frt-restore backup:list offsite | grep -q "$SNAP" || fail "the origin's snapshot is not listed after connecting the location"
|
||||||
ok "recovery kit imported, existing repository adopted, snapshot visible"
|
ok "recovery kit imported, existing repository adopted, snapshot visible"
|
||||||
|
|
||||||
run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction >/dev/null || fail "restore:full"
|
OUT=$(run_on ngb-frt-restore backup:restore:full offsite "$SNAP" --no-interaction 2>&1) || {
|
||||||
|
printf '%s\n' "$OUT" | tail -40
|
||||||
|
docker exec ngb-frt-restore sh -c 'tail -c 6000 /var/www/html/data/nextcloud.log' || true
|
||||||
|
fail "restore:full"
|
||||||
|
}
|
||||||
ok "backup:restore:full completed"
|
ok "backup:restore:full completed"
|
||||||
|
|
||||||
# Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
|
# Everything below must work in a FRESH occ process (a new PHP process per docker exec), since
|
||||||
|
|||||||
@@ -0,0 +1,22 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
declare(strict_types=1);
|
||||||
|
|
||||||
|
// Stand-in for the files_external SMB storage in unit tests: only what StorageBackend uses.
|
||||||
|
|
||||||
|
namespace OCA\Files_External\Lib\Storage;
|
||||||
|
|
||||||
|
use OCP\Files\Storage\IStorage;
|
||||||
|
|
||||||
|
if (!class_exists(SMB::class)) {
|
||||||
|
abstract class SMB implements IStorage {
|
||||||
|
/** @var object */
|
||||||
|
protected $share;
|
||||||
|
protected bool $showHidden = false;
|
||||||
|
protected bool $checkAcl = false;
|
||||||
|
|
||||||
|
protected function buildPath(string $path): string {
|
||||||
|
return '/' . trim($path, '/');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user