dillard
7b33347dbc
- detect files changing during reading or between resumable steps and re-read them (size/mtime/ctime/inode), spike F overwrites mid-run - drop pending blob ids when a pack or index upload fails - dump and restore PostgreSQL sequence state exactly, bound parameters, validated catalog names; spike C checks all sequences exactly - keep invalid UTF-8 byte-exact (tagged base64) - refuse large tables without a primary key - StorageBackend: check write/flush/close results, validate base path - user_migration import/export: root paths, intermediate folders, partial writes, read errors - spikes: never touch an existing account, own fixtures, 512M note
77 lines
6.5 KiB
Markdown
77 lines
6.5 KiB
Markdown
# Phase 0 — results (2026-10-01)
|
||
|
||
All spikes run on the standard `nextcloud:34-apache` image (PHP 8.5, sodium 1.0.18), without
|
||
`pg_dump`, `mysqldump`, the `pgsql`/`mysqli` extensions, rclone or a shell on the target.
|
||
|
||
| Spike | What | Result |
|
||
|---|---|---|
|
||
| A `01-stream-crypto.php` | Streaming encryption (secretstream XChaCha20-Poly1305), key wrapping (Argon2id) | 1 GiB round trip with `memory_limit=64M`: peak < 2 MiB, 160 MiB/s encrypt, 122 MiB/s decrypt, 0.03 % overhead. Tampering, truncation (also at a frame boundary), trailing data, wrong key/AD/passphrase rejected. |
|
||
| B `02-repository.php` | Repository format: 4 MiB blobs, deflate, packs, index, snapshots, dedup, incremental, restore | 390 MiB tree, peak 54 MiB. Unchanged run: 0.06 s, nothing read or uploaded. 1 MiB change in a 300 MiB file: one blob uploaded. Full and partial restore bit-identical, restore peak 17 MiB. Tampered pack detected. |
|
||
| C `03-db-dump.php` | Logical DB dump/restore via the Nextcloud connection (PostgreSQL) | 300k-row table with bytea (110 MB): dump 6.3 s, peak 50 MiB, 94 MiB JSON → 20 MiB stored. Unchanged second dump stores nothing (per-table blob streams). Restore 46 s: every table checksum-identical, all 110 sequences correct. |
|
||
| D `04-external-storage.php` | Repository on files_external storages without a mount | S3 (SeaweedFS) and SFTP: 120 MiB backup ~6 s, unchanged run uploads nothing, restore bit-identical, no row in `external_mounts`, /tmp stayed at 12 KiB (no local copy). Also works with files_external disabled when its classes are loaded. |
|
||
| E `05-user-migration.php` | Per-user backup/restore through user_migration with own `IExportDestination`/`IImportSource` | Migrators: calendar, contacts, trashbin, account, files. Export 0.8 s, unchanged re-export stores nothing. After deleting the user, import restores account, files (bit-identical), event and contact. |
|
||
| F `06-resumable.php` | Time-boxed, resumable steps in separate processes | 200 MiB file resumed mid-file across steps; with 6 random `kill -9` the run still completes and restores bit-identically; peak 29–53 MiB per step. |
|
||
|
||
## Findings that change or refine the plan
|
||
|
||
1. **Restore over an existing user (user_migration)** is not clean: contacts are duplicated and a
|
||
`migrated-contact_birthdays` calendar is added; calendars/address books come back as
|
||
`migrated-*` with new object URIs (clients resync; shares/subscriptions of those calendars
|
||
break). → Per-user restore in phase 2 needs its own strategy: restore into a fresh account
|
||
(after deleting or renaming the existing one, with a safety export first), or restore
|
||
calendars/contacts at DAV level ourselves and use user_migration only for the rest.
|
||
2. **files_external without enabling it:** its classes can be loaded while the app is disabled
|
||
(`OC_App::loadApp`, private API). Choice to make: rely on that (nothing visible to anyone), or
|
||
require files_external enabled with user mounting off (public path, a bit more visible).
|
||
3. **Peak memory on S3/SFTP backups is 85–89 MiB** (local target 54 MiB): the files_external
|
||
upload path buffers about one extra pack. Fine within 512 MB; to look at in phase 1
|
||
(smaller packs on hosted, or our own S3 multipart adapter).
|
||
4. **DB restore speed:** row-by-row INSERT (46 s for 300k rows). Phase 1: multi-row INSERT,
|
||
skip tables whose checksum matches the dump, disable/re-enable nothing (keep it portable).
|
||
5. **Fixed-size chunking:** a row change in a big table (e.g. `filecache`) or an insertion in a big
|
||
file shifts all following blobs. Phase 1: content-defined chunking (FastCDC) for files and dumps.
|
||
6. **Index and snapshot entries are held in memory** in phase 0. Phase 1: index as a cache table in
|
||
the Nextcloud database (rebuildable from `index/` and the pack headers), streaming comparison
|
||
with the parent snapshot.
|
||
7. **user creation on import needs > 128 MB** (password_policy word list) — document 512 MB as minimum.
|
||
8. **Snowflake ids:** since NC 33 some tables (e.g. `oc_jobs`) have no sequence; the dump/restore
|
||
handles that (no special case needed), the test just must not assume sequences everywhere.
|
||
9. **files_external S3 `writeStream()` closes the stream it is given** → `IBackend::put()` owns and
|
||
closes the stream (contract documented).
|
||
|
||
## Not done in phase 0
|
||
|
||
- MySQL/MariaDB and SQLite runs of spike C (PostgreSQL only so far).
|
||
- Real hosted environment (webcron/AJAX cron, `max_execution_time`); spike F simulates it.
|
||
- Restore of the database while Nextcloud is running (spike C restored with Nextcloud idle).
|
||
- Schema introspection still uses the Doctrine connection behind `IDBConnection` (private API).
|
||
|
||
## Code review (2026-10-01)
|
||
|
||
No way was found to reorder or drop secretstream frames, substitute a blob, or decrypt with the
|
||
wrong key/associated data without detection. Fixed after the review (all spikes re-run, all pass):
|
||
|
||
- Files that change while being read, or between two resumable steps, are detected (size, mtime,
|
||
ctime, inode before/after) and read again instead of producing a torn file; spike F now
|
||
overwrites the big file mid-run and the restore matches the new content.
|
||
- A failed pack/index upload no longer leaves its blobs marked as present.
|
||
- PostgreSQL sequences are dumped and restored exactly (not MAX+1); spike C advances one
|
||
sequence 50 steps ahead of the data and checks all 115 sequences.
|
||
- Text that is not valid UTF-8 is kept byte-exact (tagged base64) instead of being substituted.
|
||
- Sequence SQL uses bound parameters; sequence names from the catalog are validated.
|
||
- Tables without a primary key are refused above 50,000 rows (bounded memory).
|
||
- `StorageBackend` checks `writeStream()`/`fflush()`/`fclose()` results and validates its base path.
|
||
- user_migration import: root-level paths, intermediate folders and partial writes handled;
|
||
export fails on a read error instead of storing a truncated file.
|
||
|
||
Design items for phase 1 (from the review):
|
||
|
||
- **Rollback/freshness:** a malicious backend can replay an older valid object at the same path or
|
||
hide snapshots. Add a chained, MACed generation catalog (repository id + object type/path in the
|
||
associated data) and keep the latest generation id outside the backend (Nextcloud DB + recovery kit).
|
||
- **Decrypt output is uncommitted until the FINAL frame:** callers must stage and only publish
|
||
after success (restore already writes `*.ngb-restore` and renames; make this a rule for all sinks).
|
||
- **Restore mode:** "merge" (current: extra files stay) vs "replace" (target equals snapshot).
|
||
- **Size metadata:** object and pack-header sizes reveal approximate sizes/compressibility;
|
||
optional padding if that matters.
|