nextcloud-ng_backup/docs/DISASTER-RECOVERY.md

# Disaster recovery: your server is gone

Your Nextcloud server is lost. You have a new, empty server. These are the steps to get your
Nextcloud back from an NG Backup location. Do them in this order.

## What you need

- [ ] **One passphrase** of the old server, and if you have it, the **recovery kit** file
      (`ng_backup-kit.json`).
- [ ] The **connection details of the backup location**: host, user, password or keys, bucket, folder.
- [ ] A new server with the **same Nextcloud major version** as the old one (for example 34),
      a database (PostgreSQL, MySQL/MariaDB or SQLite; it may differ from the old one) and room for
      the whole backup plus a margin.
- [ ] **Shell access** to the new server, to run `occ` (on a hosted Nextcloud without it, your
      provider runs these steps).
- [ ] A list of the apps that are **not in the app store** (your own builds, apps copied in by
      hand). Keep their packages somewhere safe beforehand.

No kit? Since version 1.1.0 the passphrase is enough: the key is also stored on the location (in
its `config` file), locked with the same passphrases. Skip step 4. The kit is still the safer
copy: it helps when that file on the location is damaged. Keep it off the server, next to the
passphrase.

## Steps

**1. Install a fresh Nextcloud** of the same major version. Create an admin account. Do not add
users or files.

**2. Do not let background jobs run yet.** A new Nextcloud runs them on page visits (AJAX).
Switch that off:

    occ background:cron

As long as no cron is set up, nothing runs. Check that none is:

- Regular server: `sudo crontab -u www-data -l`. If it shows a line with `cron.php`, remove it
  for now with `sudo crontab -u www-data -e` (put it back in step 9).
- Docker Compose with a cron service: `docker compose stop cron`.
- Nextcloud All-in-One: cron runs inside the Nextcloud container; leave it. It does no harm here:
  the restore keeps NG Backup's own settings of the new server, so it does not start backups.

This keeps the new server quiet (no mail syncs, no notifications, no backups) until it is ready.

**3. Install NG Backup** (the same version or newer) and enable it.

**4. Import the backup key** (with a kit; without one, go to step 5). Copy the kit to the new
server so the web server user can read it.

From your own computer:

    scp ng_backup-kit.json root@newserver:/tmp/

Then on the server, one of:

    chown www-data /tmp/ng_backup-kit.json                                    # regular server
    docker cp /tmp/ng_backup-kit.json <container>:/tmp/ && \
      docker exec <container> chown www-data /tmp/ng_backup-kit.json          # Docker

`<container>` is the name of your Nextcloud container: `docker ps` shows it (for example
`nextcloud-aio-nextcloud`, or `myproject-nextcloud-1` with Docker Compose).

Import it; you type the passphrase yourself:

    occ backup:key:import /tmp/ng_backup-kit.json

It shows the key fingerprint: it must match the one of your old server (on the recovery kit).

**5. Add the backup location** with the same details as before. On a large backup this first reads
the index and shows how far it is, for example
`Reading the index: 6,913/26,791 files (25%), about 12 min left`.

    occ backup:target:add offsite sftp -a password::password -o host=... -o user=... -o password=... -o root=... --path=...

Without a kit (step 4 skipped), it finds the backups and asks for a passphrase of the old server;
you type it yourself. It then shows the key fingerprint and the deletion delay of the old server.
A wrong passphrase changes nothing: run the command again. Backups made before version 1.1.0 did
not store the deletion delay on the location; it is then set to 7 days.

If it says that the location's **config file is gone**, the key is only in the recovery kit:
import it (step 4), then add the location with `--repair-config` at the end of the same command.
That writes the config again from the kit; it works for locations backed up by version 1.1.0 or
newer.

If it says "No backup key yet, and no backups at this location", the connection details or
`--path` differ from the old server's. Check them; do **not** run `backup:key:init`: that makes a
brand new key, and your backups were encrypted with the old one. NG Backup then refuses the
location ("created with a different key").

**Made a new key by mistake** (with `backup:key:init` or on the page) before adding the location?
As long as that key was never used (no location added with it, no backup), it can make way for the
key of your backups. NG Backup says so when it refuses the location. Add `--replace-unused-key` to
`occ backup:key:import` (with the kit) or to `occ backup:target:add` (without the kit; it asks for
a passphrase of the old server). The passphrases and kit of the unused key then stop working. A key
that was used for a location is never replaced: its own backups depend on it.

**6. List the restore points** and pick one, normally the newest:

    occ backup:list offsite

**7. Start the restore, and let it install the apps.**

    occ backup:restore:full offsite <snapshot> --install-apps

`--install-apps` first installs every app the old server had enabled from the app store (or
switches it on, if it is part of Nextcloud). Tables of apps that are missing or were removed long
ago are created from the backup, so they do not stop the restore.

At the end it lists the apps it could not install, usually your own builds that are not in the
app store. Copy those into `custom_apps` and enable them (`occ app:enable <app>`); their data is
already restored.

Backups made before 1.0.0-rc.2 do not record the structure of their tables. If the restore lists
tables that do not exist here, install the apps they belong to and run it again, or leave them
out with `--skip-missing-tables`.

**8. Let the restore run.** It restores the database, then the data directory, and takes hours for
a large backup. Nextcloud is in maintenance mode meanwhile, so nobody and no background job
writes into it; the restore switches it off when it completes. (If you had switched maintenance
mode on yourself before, it stays on: switch it off with `occ maintenance:mode --off` when ready.)

If it stops halfway, maintenance mode stays on (the instance may be partly restored). Fix the cause,
then switch maintenance mode off and run the restore again with `--force`, in one line (the message
prints it): `occ maintenance:mode --off && occ backup:restore:full <location> <snapshot> --force`.
It goes on where it stopped: data files that are already in place with the size and modification
time of the backup are left, so only the rest comes from the location again (since 1.0.2; before,
it started over). The database is restored again in full, which takes minutes. Leave out `--install-apps` this time, so that the instance is open only for the seconds until the
restore has switched maintenance mode on again; apps the first run could not install are listed at
the end of the restore, install those afterwards.
If occ then says that Nextcloud or one of the apps requires an upgrade, the database is already
the old server's: run `occ upgrade` first, then the same command with `--force`.

**If the backup itself is damaged** (the restore stops with "Decryption failed", "Blob
verification failed" or a file that is missing on the location), try an older restore point
first. If none is better, `--salvage` restores whatever can be read:

    occ backup:restore:full offsite <snapshot> --install-apps --salvage

If a restore already stopped halfway, run it as in step 8: maintenance mode off, with `--force` and
without `--install-apps`:

    occ maintenance:mode --off && occ backup:restore:full offsite <snapshot> --salvage --force

- Files that cannot be read are left out and listed at the end; everything else is restored.
- A full disk is not damage: the restore stops, also with `--salvage`, and says so. Make room and
  run the restore again as in step 8.
- **On version 1.0.2, check the disk first if the reason is "Size mismatch".** There a full disk
  looks like damaged data: the message points at `--salvage`, and with `--salvage` the restore goes
  on and lists the remaining files. The backup itself is fine. Update NG Backup, make room and run
  the restore again as in step 8.
- If the database cannot be restored (its dump is damaged, or it does not fit this
  installation), the database stays as the fresh installation made it, and the data directory
  and config of the backup go to a folder of their own in the data directory,
  `ngb-salvage-<snapshot>`. Each user's files are in `data/<user>/files` there: create the users
  again and copy their files back by hand (then `occ files:scan --all`). Calendars, contacts and
  shares are in the database and cannot be recovered this way.

The command then ends with exit code 2 instead of 0. The same option exists for
`occ backup:restore`, to restore single folders.

**9. After the restore.** These worked in our tests:

    occ upgrade
    occ maintenance:repair
    occ files:scan --all

- Check `config/config.php`: `trusted_domains`, `overwrite.cli.url` and `overwritehost` come from
  the old server; change them if the new server has another name. The database settings and the
  data directory are kept from the new installation.
- From Nextcloud All-in-One to a regular server: also check the cache settings (`memcache.*`,
  `redis`), which point at AIO's containers.
- Log in, open a few files, a calendar and a contact.
- `occ files:scan` can report many folders as "Updated". That is expected: a restore does not yet put
  back the modification time of folders with files in them, so they show the time of the restore.
  The files themselves keep their time and content.
- Empty folders only come back from backups made with version 1.0.3 or newer. With an older
  restore point they are gone: `occ files:scan` then removes them from the file cache, so it reports
  them as removed.
- Start background jobs again (cron), as in step 2 in reverse.

**10. Point your domain at the new server** and check that clients reconnect.

**11. Make a backup of the new server** to the same location (`occ backup:run offsite`). The old
server must stay off: two servers must never write to the same location.

Restored without a kit? Then backups wait until you have a new one: download it on the backup
page (or `occ backup:key:kit --output=...`), store it off the server and confirm it
(`occ backup:key:confirm`).

## Never

- Never let the old server write to the location again if it comes back. Switch its backups off
  first.
- Never give your passphrase to anyone. Nobody helping you needs it.

## Help

Stuck? [Ensembia](https://www.ensembia.com), the developer of NG Backup, offers paid help with
restoring a backup after a crash.