diff --git a/CHANGELOG.md b/CHANGELOG.md index f162337..5fca3ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,33 @@ Each heading starts with that ``, once: the app store shows a release t whose heading starts with its `x.y.z` and ignores anything after it, so `## [1.0.0-rc.6]` would be shown under 1.0.0. The git tag of a release candidate goes after the date. +## [1.0.2] - unreleased + +### Added +- A warning on the admin page and in `occ backup:status` when background jobs do not run on a + system cron, or the cron stopped, with a link to how to set one up. Backups, also the scheduled + ones, need it (#13). +- Without a working system cron, a backup started with "Back up now" moves on while the NG Backup + page stays open (#13). +- `--salvage` for `occ backup:restore` and `occ backup:restore:full`: restore whatever can be + read when a backup is damaged. Files that cannot be read or decrypted are left out and listed, + and the command exits with code 2. When the database cannot be restored, `backup:restore:full` + leaves the fresh database as it is and puts the data directory and config of the backup in + `ngb-salvage-` in the data directory, to put back by hand. +- The trash of restore points has a tab of its own, "Bin", instead of being shown below the + backups. + +### Changed +- A backup in its database phase shows how far it is, for example "running (db: 47/183 tables)", + and one that has not moved for 15 minutes says "waiting for background jobs" with when it last + took a step. Before, it only said "running (db)", which looked the same as being stuck (#13). +- Full restores from S3 read the packs ahead, eight ranges at a time, as restores over SFTP do + since 1.0.0-rc.4. Over SMB the packs are read in 8 MiB reads instead of 1 MiB: over a line with + a 130 ms round trip, reading was four times faster in a test (18 instead of 4.5 MiB/s). +- `occ backup:target:add` only estimates the time left for reading the index once enough of it + is in. The first file also carries the time to connect, which made it say "about 642 min left" + for something done within a minute. + ## [1.0.1] - 2026-11-01 Fixes for what was reported after 1.0.0 (#22, #24) and small improvements. Nothing changes in diff --git a/PLAN.md b/PLAN.md index 7579e7e..a3c0ff3 100644 --- a/PLAN.md +++ b/PLAN.md @@ -392,6 +392,20 @@ in README "Known limitations": gids (SFTP, Storage Box) → rc4 uitbrengen en om tests vragen → zelf een volledige DR van een productie-instantie van ~250 GB via S3 en via SMB. Drie volledige DR-restores vóór 1.0.0. Problemen of meldingen van gebruikers: rc5. +- **1.0.2** (plan akkoord 2026-11-04): punten 1, 2 en 5 hieronder, de Bin, nieuwe screenshots en + het polish-punt over de schatting van `target:add`. Besluiten 2026-11-04: + - `--salvage` haalt ook bij een kapotte database terug wat kan: bestanden zijn per gebruiker + gescheiden en het waardevolst. Kan de database niet terug (dump onleesbaar, of past niet op + deze installatie), dan blijft de verse database staan en gaan data en config van de backup + naar `/ngb-salvage-/`; vanaf daar is het handwerk. Exitcode 2. + - De DR-test met een backup mét tabelstructuur, zonder `--skip-missing-tables` (punt 3), is + nog niet gedaan en wordt ingepland vóór de release van 1.0.2. + Stand 2026-11-04 (lokaal op `polish`, niet gepusht): cron-waarschuwing + tick vanaf de pagina + + voortgang van de db-fase; `--salvage` voor `backup:restore` en `backup:restore:full`; S3 + vooruitlezen (8 ranges tegelijk, met `SaveAs` naar de spool); SMB in leesacties van 8 MiB + (gemeten met netem, 130 ms RTT: 4,5 → 18 MiB/s per leesactie; een volledige restore over SMB + met vertraging is nog niet gemeten); Bin-tabblad; schatting `target:add`. Nog te doen: + screenshots, DR-test, release. - **1.0.1** (uitgebracht 2026-11-01): alleen de kleine punten (besluit 2026-10-30): 4, 6–13. Punten 1, 2 en 5 gaan naar 1.0.2 (README-roadmap aangepast); punt 3 is een test, ingepland voor de week van 2 november, niet aangekondigd en los van de release. Voor 1.0.2 ook: nieuwe screenshots (het labelveld bij diff --git a/README.md b/README.md index 679f4c8..00987cc 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,8 @@ Need help restoring a backup after a crash? Ensembia offers paid restore assista background job, so a backup only moves forward when that job runs. Choosing "Cron" under **Administration settings → Basic settings → Background jobs** is not enough on its own: it tells Nextcloud that the system will call it, so you also need a cron line that does. Without - it a backup stays at "running (db)", and so does one started with "Back up now". + it scheduled backups do not run; the admin page and `occ backup:status` warn about it, and a + backup started with "Back up now" only moves on while the NG Backup page stays open. `occ backup:run` in a terminal works without cron (it runs in the foreground). - Regular server: `crontab -u www-data -e` and add `*/5 * * * * php -f /var/www/nextcloud/cron.php` (your Nextcloud path). diff --git a/appinfo/routes.php b/appinfo/routes.php index 10e7a5d..b058d68 100644 --- a/appinfo/routes.php +++ b/appinfo/routes.php @@ -24,6 +24,7 @@ return [ ['name' => 'AdminApi#testTarget', 'url' => '/api/targets/{id}/test', 'verb' => 'POST'], ['name' => 'AdminApi#setTargetLimit', 'url' => '/api/targets/{id}/limit', 'verb' => 'PUT'], ['name' => 'AdminApi#startBackup', 'url' => '/api/backups', 'verb' => 'POST'], + ['name' => 'AdminApi#tick', 'url' => '/api/tick', 'verb' => 'POST'], ['name' => 'AdminApi#setSchedule', 'url' => '/api/schedule', 'verb' => 'PUT'], ['name' => 'AdminApi#snapshots', 'url' => '/api/targets/{targetId}/snapshots', 'verb' => 'GET'], ['name' => 'AdminApi#browse', 'url' => '/api/targets/{targetId}/snapshots/{snapshotId}/browse', 'verb' => 'GET'], diff --git a/lib/Command/Status.php b/lib/Command/Status.php index 9d89678..65a1260 100644 --- a/lib/Command/Status.php +++ b/lib/Command/Status.php @@ -9,6 +9,7 @@ namespace OCA\NgBackup\Command; use OCA\NgBackup\Db\RunMapper; use OCA\NgBackup\Format; +use OCA\NgBackup\Service\CronHealth; use OCA\NgBackup\Service\KeyService; use OCA\NgBackup\Service\Schedule; use OCA\NgBackup\Service\TargetService; @@ -29,6 +30,7 @@ final class Status extends Command { private RunMapper $runs, private VerifyService $verify, private Schedule $schedule, + private CronHealth $cron, ) { parent::__construct(); } @@ -40,6 +42,10 @@ final class Status extends Command { protected function execute(InputInterface $input, OutputInterface $output): int { $zone = $this->schedule->get()['timezone']; + $cronProblem = CronHealth::describe($this->cron->check()); + if ($cronProblem !== null) { + $output->writeln('' . $cronProblem . ''); + } if (!$this->keys->isInitialized()) { $output->writeln('Key: not created (occ backup:key:init)'); } else { @@ -57,7 +63,7 @@ final class Status extends Command { $shown = $this->runs->findShown($all ? null : time() - self::RECENT_DAYS * 86400); foreach ($shown as $r) { $s = json_decode($r->getStats() ?? '{}', true)['files'] ?? []; - $rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() . ($r->getStatus() === 'running' ? ' (' . $r->getPhase() . self::percent($r) . ')' : ''), + $rows[] = [$r->getId(), $names[$r->getTargetId()] ?? '#' . $r->getTargetId(), $r->getKind(), $r->getStatus() === 'running' ? self::runningStatus($r) : $r->getStatus(), Format::time($r->getStartedAt(), $zone), $r->getFinishedAt() ? Format::duration($r->getFinishedAt() - $r->getStartedAt()) : '', $s['files'] ?? '', isset($s['bytes']) ? self::size((int)$s['bytes']) : '', isset($s['uploaded']) ? self::size((int)$s['uploaded']) : '', mb_strimwidth((string)$r->getError(), 0, 50, '…')]; @@ -88,8 +94,24 @@ final class Status extends Command { return sprintf('%.1f TiB', $b); } + /** " 47/183 tables" in the database phase, " 68%" or " ~68%" in the files phase; empty without a figure */ private static function percent(\OCA\NgBackup\Db\Run $r): string { $p = \OCA\NgBackup\Job\BackupRun::progressOf($r); - return $p === null ? '' : sprintf(' %s%d%%', $p['approximate'] ? '~' : '', (int)floor($p['fraction'] * 100.0)); + if ($p === null) { + return ''; + } + if (isset($p['tables'], $p['totalTables'])) { + return sprintf(': %d/%d tables', $p['tables'], $p['totalTables']); + } + return sprintf(' %s%d%%', $p['approximate'] ? '~' : '', (int)floor($p['fraction'] * 100.0)); + } + + /** "running (db: 47/183 tables)", or "waiting for background jobs (last step 12 min ago)" */ + private static function runningStatus(\OCA\NgBackup\Db\Run $r): string { + $waiting = $r->getKind() === \OCA\NgBackup\Service\BackupService::KIND_FULL ? CronHealth::waitingFor($r) : null; + if ($waiting !== null) { + return 'waiting for background jobs (' . $r->getPhase() . self::percent($r) . ', last step ' . CronHealth::ago($waiting) . ')'; + } + return 'running (' . $r->getPhase() . self::percent($r) . ')'; } } diff --git a/lib/Controller/AdminApiController.php b/lib/Controller/AdminApiController.php index 886e415..85a2683 100644 --- a/lib/Controller/AdminApiController.php +++ b/lib/Controller/AdminApiController.php @@ -48,6 +48,7 @@ final class AdminApiController extends Controller { private \OCA\NgBackup\Service\KeySlotService $slots, private \OCA\NgBackup\Service\PruneService $prune, private \OCA\NgBackup\Service\Schedule $schedule, + private \OCA\NgBackup\Service\CronHealth $cron, ) { parent::__construct(Application::APP_ID, $request); } @@ -75,9 +76,36 @@ final class AdminApiController extends Controller { 'recentDays' => \OCA\NgBackup\Command\Status::RECENT_DAYS, 'schedule' => $this->schedule->get()['time'], 'scheduleTimezone' => $this->schedule->get()['timezone'], + 'cron' => $this->cron->check(), ]); } + /** Seconds of backup work one tick() does: well within a normal web request. */ + public const TICK_SECONDS = 10.0; + + /** + * Move running backups forward from the open admin page, when background jobs do not run on a + * working system cron (#13). The page calls this while a backup runs, so "Back up now" works + * without cron as long as the page stays open; scheduled backups still need cron. With a + * working system cron this does nothing: the background job does the work. + */ + public function tick(): JSONResponse { + if ($this->cron->check()['ok']) { + return new JSONResponse(['stepped' => 0]); + } + $deadline = microtime(true) + self::TICK_SECONDS; + $stepped = 0; + foreach ($this->runs->findRunning(null, BackupService::KIND_FULL) as $run) { + if (microtime(true) >= $deadline) { + break; + } + // step() returns at once when the background job (or another tab) holds the run. + $this->backups->step($run, $deadline); + $stepped++; + } + return new JSONResponse(['stepped' => $stepped]); + } + #[PasswordConfirmationRequired] public function initKey(string $passphrase, int $deleteDelayDays = \OCA\NgBackup\Service\KeyService::DEFAULT_DELETE_DELAY, string $label = 'Slot 1'): JSONResponse { try { @@ -313,6 +341,7 @@ final class AdminApiController extends Controller { return ['id' => $r->getId(), 'targetId' => $r->getTargetId(), 'kind' => $r->getKind(), 'status' => $r->getStatus(), 'phase' => $r->getPhase(), 'snapshotId' => $r->getSnapshotId(), 'snapshot' => $snapshot, 'startedAt' => $r->getStartedAt(), 'updatedAt' => $r->getUpdatedAt(), 'finishedAt' => $r->getFinishedAt(), 'error' => $r->getError(), - 'stats' => json_decode($r->getStats() ?? 'null', true), 'progress' => \OCA\NgBackup\Job\BackupRun::progressOf($r)]; + 'stats' => json_decode($r->getStats() ?? 'null', true), 'progress' => \OCA\NgBackup\Job\BackupRun::progressOf($r), + 'waitingFor' => $r->getKind() === BackupService::KIND_FULL ? \OCA\NgBackup\Service\CronHealth::waitingFor($r) : null]; } } diff --git a/lib/Job/BackupRun.php b/lib/Job/BackupRun.php index 2c9e136..e7cc3f1 100644 --- a/lib/Job/BackupRun.php +++ b/lib/Job/BackupRun.php @@ -176,12 +176,12 @@ final class BackupRun { } /** - * progress() of a run, while it is in its files phase. + * progress() of a run in its files phase; in its database phase, the tables dumped so far. * - * @return array{fraction: float, approximate: bool}|null + * @return array{fraction: float, approximate: bool, tables?: int, totalTables?: int}|null */ public static function progressOf(\OCA\NgBackup\Db\Run $run): ?array { - if ($run->getStatus() !== \OCA\NgBackup\Db\Run::RUNNING || $run->getPhase() !== 'files') { + if ($run->getStatus() !== \OCA\NgBackup\Db\Run::RUNNING || !in_array($run->getPhase(), ['db', 'files'], true)) { return null; } try { @@ -189,6 +189,14 @@ final class BackupRun { } catch (\JsonException) { return null; } + if ($run->getPhase() === 'db') { + // Tables dumped so far, as of the last step (none known before the first step ends) + $tables = count($state['dump']['tables'] ?? []); + return $tables > 0 + ? ['fraction' => min(1.0, ($state['dump']['i'] ?? 0) / $tables), 'approximate' => false, + 'tables' => (int)($state['dump']['i'] ?? 0), 'totalTables' => $tables] + : null; + } return is_array($state['files'] ?? null) ? self::progress($state['files']) : null; } diff --git a/lib/Service/CronHealth.php b/lib/Service/CronHealth.php new file mode 100644 index 0000000..11d560c --- /dev/null +++ b/lib/Service/CronHealth.php @@ -0,0 +1,97 @@ +coreString('backgroundjobs_mode', 'ajax'); + $lastRun = (int)$this->coreString('lastcron', '0'); + $stopped = $mode === 'cron' && $now - $lastRun > self::STOPPED_AFTER; + return [ + 'mode' => $mode, + 'lastRun' => $lastRun, + 'ok' => $mode === 'cron' && !$stopped, + 'stopped' => $stopped, + 'docs' => $this->urls->linkToDocs('admin-background-jobs'), + ]; + } + + /** + * Seconds since a running backup took its last step, when that is long enough to say it waits + * for background jobs; null while it moves (or when it is not running). + */ + public static function waitingFor(Run $run, ?int $now = null): ?int { + $idle = ($now ?? time()) - $run->getUpdatedAt(); + return $run->getStatus() === Run::RUNNING && $idle >= self::WAITING_AFTER ? $idle : null; + } + + /** One line for occ and the log; null when system cron runs. */ + public static function describe(array $health, ?int $now = null): ?string { + if ($health['ok']) { + return null; + } + $last = $health['lastRun'] > 0 ? 'last ran ' . self::ago(($now ?? time()) - $health['lastRun']) : 'never ran'; + $what = $health['stopped'] + ? 'Background jobs are set to system cron, but cron has not run for over an hour (' . $last . ').' + : 'Background jobs do not run on a system cron: they are set to "' . ($health['mode'] === 'webcron' ? 'Webcron' : 'AJAX') . '" and ' . $last . '.'; + return $what . ' Backups, also the scheduled ones, need a system cron that runs every 5 minutes: ' . $health['docs']; + } + + /** "12 min ago" style, for durations from a minute to days */ + public static function ago(int $seconds): string { + if ($seconds < 3600) { + return max(1, intdiv($seconds, 60)) . ' min ago'; + } + if ($seconds < 172800) { + return intdiv($seconds, 3600) . ' h ago'; + } + return intdiv($seconds, 86400) . ' days ago'; + } + + /** Core stores these with different value types across versions: read whatever is there. */ + private function coreString(string $key, string $default): string { + try { + return $this->appConfig->getValueString('core', $key, $default); + } catch (AppConfigTypeConflictException) { + return (string)$this->appConfig->getValueInt('core', $key, (int)$default); + } + } +} diff --git a/screenshots/1-backups.png b/screenshots/1-backups.png index 959607d..f5ad83a 100644 Binary files a/screenshots/1-backups.png and b/screenshots/1-backups.png differ diff --git a/screenshots/2-restore.png b/screenshots/2-restore.png index d8928b7..bee0170 100644 Binary files a/screenshots/2-restore.png and b/screenshots/2-restore.png differ diff --git a/screenshots/3-settings.png b/screenshots/3-settings.png index 2cf98da..61122a8 100644 Binary files a/screenshots/3-settings.png and b/screenshots/3-settings.png differ diff --git a/screenshots/4-bin.png b/screenshots/4-bin.png new file mode 100644 index 0000000..52ae8f4 Binary files /dev/null and b/screenshots/4-bin.png differ diff --git a/src/AdminSettings.vue b/src/AdminSettings.vue index a70cd42..258d301 100644 --- a/src/AdminSettings.vue +++ b/src/AdminSettings.vue @@ -12,11 +12,22 @@

{{ error }} + +

+ {{ status.cron.stopped + ? t('ng_backup', 'Background jobs are set to system cron, but cron has not run since {time}.', { time: lastCron }) + : t('ng_backup', 'Background jobs do not run on a system cron (they are set to "{mode}").', { mode: status.cron.mode === 'webcron' ? 'Webcron' : 'AJAX' }) }} + {{ t('ng_backup', 'Backups need a system cron that runs every 5 minutes, also the scheduled ones.') }} + {{ t('ng_backup', 'How to set up cron ↗') }} +

+

{{ t('ng_backup', 'Until then, a backup started with "Back up now" only moves on while this page is open.') }}

+