Backup & Restore

Simfra can export its entire state to a single portable file and restore it into another Simfra instance. The dump covers everything Simfra manages:

  • Every resource, across all accounts and all regions (queues, topics, IAM entities, EC2/VPC objects, buckets, functions, tables, …).
  • File/blob content: S3 object bodies, Lambda code zips, ECR layer blobs, the root CA (certificate + key), and the STS token-encryption key (so temporary credentials issued before the export still decrypt after import).
  • The account registry (account IDs and their root credentials), so SigV4 authentication keeps working after a restore.

:::warning Dumps are secrets A dump contains plaintext account root credentials, IAM secret access keys, the root CA private key, and the STS token key. Field-encrypted values are decrypted on export and re-encrypted with the target's key on import (so a dump is portable across instances with different SIMFRA_PERSISTENCE_KEYs). Treat dump files as you would any credential store — protect them at rest and in transit.

It works identically whether or not persistence is enabled (SIMFRA_DATA_DIR). The export reads from live in-memory state — not the SQLite file — so even a default, fully in-memory Simfra can be snapshotted and moved.

Usage

Both operations target a running Simfra via the Admin API:

# Export a running Simfra to a file
curl -s http://localhost:4599/_simfra/export -o simfra-dump.tar.gz

# Restore it into the same or a different running Simfra
curl -s -X POST --data-binary @simfra-dump.tar.gz http://localhost:4599/_simfra/import

When SIMFRA_ADMIN_TOKEN is set, both endpoints require Authorization: Bearer <token>.

Load a dump on launch

Instead of POSTing to a running server, you can have Simfra import a dump at startup by pointing SIMFRA_IMPORT at a dump file:

SIMFRA_IMPORT=/path/to/simfra-dump.tar.gz simfra

The dump is loaded before the server starts serving, so resources are queryable immediately — no post-startup POST /_simfra/import required. This is convenient for CI fixtures, demos, and sharing a captured environment.

Semantics are replace on every launch: each start wipes current state and restores the dump. With persistence (SIMFRA_DATA_DIR) enabled, the imported state is reconciled to SQLite, so the dump is authoritative on every restart and any runtime changes made since the last start are discarded. Leave SIMFRA_IMPORT unset (or seed once and then unset it) if you instead want persistence to retain runtime changes across restarts. A missing or malformed dump file aborts startup.

The embeddable Go server exposes the same via simfra.NewServer(...).WithImportFile(path).

Dump format

The dump is a gzipped tar (*.tar.gz) with a stable, inspectable layout:

manifest.json           format version, simfra version, created-at, counts, volume index
accounts.json           account registry records (incl. root credentials)
resources.jsonl         one JSON resource row per line (service, account, region, type, id, version, state)
blobs/<service>/<...>   raw blob bytes (e.g. blobs/s3/<account>/<bucket>/<bodyRef>, blobs/lambda/<key>, blobs/ecr/<...>)
volumes/<ns>/<relPath>  Docker engine named-volume tars (format v2, written strictly last)

You can tar tzf simfra-dump.tar.gz to inspect it.

Format versioning. A dump with no volume section is format: 1 (readable by older importers). As soon as any engine volume is included the dump is format: 2 and the manifest carries a volumes index (namespace + relPath per volume) plus volumeCounts/volumesIncluded. The volumes/ section is always written last; the importer streams volume bodies directly off the tar and rejects any dump whose metadata appears after a volume entry.

Engine volumes (Docker mode). When Docker is enabled, engine data that lives in named volumes — RDS/Redshift/DSQL/Redshift-Serverless (Postgres/MySQL), ElastiCache (Redis/Valkey AOF; memcached is exempt), MSK/Kafka (per broker), Amazon MQ, OpenSearch, EFS (NFS), and CodeCommit git repos — is captured in the volumes/ section and restored on import. RDS standby volumes are excluded (they resync from the primary). Toggle with SIMFRA_EXPORT_VOLUMES (default true) or per request with GET /_simfra/export?volumes=false.

Consistency. Volume export is crash-consistent by default (the live data dir is tar'd; Postgres replays WAL, OpenSearch its translog, Kafka recovers segments, Redis truncates a torn AOF tail on next start). For an exact copy, GET /_simfra/export?quiesce=true stops each volume's holding container for the duration of its tar and restarts it afterward (brief per-engine downtime). Each per-volume tar is spilled to a temp file under $SIMFRA_DATA_DIR (peak extra disk ≈ the largest single volume) so the tar header can carry its exact size.

Semantics

  • Replace, not merge. Import wipes the in-memory state of every service it manages and the blob stores, then applies the dump. The result is exactly the dumped state.
  • Accounts are reconciled. Imported accounts are inserted/overwritten in the account registry (preserving their root credentials). The default account is never deleted.
  • Persistence syncs automatically. When persistence is enabled, the imported state is written to SQLite by the reconciler on its next sweep — no manual flush needed. When disabled, the import simply mutates memory.
  • Corrupt rows are skipped, not fatal; the import summary reports the count.
  • Import is serialized. A live POST /_simfra/import takes a single-flight lock (a second concurrent import gets 409 Conflict) and quiesces tenant API traffic (non-admin requests get 503 + Retry-After) while it clears and restores stores, since that window is not atomic. Launch-time import (SIMFRA_IMPORT) needs neither — nothing is served yet.

How it works

The feature reuses the persistence framework. Each service already exposes its in-memory state to the reconciler via a Source; that abstraction was made symmetric (Snapshot for export, Restore/Clear for import), so every persistable service participates in backup with no extra code. Services that hold file content implement a small BlobArchiver interface; a handful that maintain derived indexes (IAM, Organizations, Route53, Cognito, S3) implement a PostRestore hook to rebuild them after an import.

Limitations

  • Engine volume data requires Docker mode. RDS/Redshift/ElastiCache/Kafka/MQ/ OpenSearch/DSQL/Redshift-Serverless engine data, EFS (NFS) file contents, and CodeCommit git repos are captured as named-volume tars (volumes/ section) only when Docker is enabled. A dump taken with Docker off, or imported into a Docker- off instance, carries/applies resource metadata only — volume entries are skipped and reported in the summary (skippedVolumes). The authoritative map of every on-disk location → archiver/exemption is in Persistence › On-Disk Blob Storage Coverage.
  • Crash-consistent by default. Volume export tars the live data dir; engines replay their logs on next start. Use ?quiesce=true for an exact (stop→tar→restart per volume) copy.
  • Import targets a running server and replaces the services present in the dump; it is not an incremental/merge operation.