Persistence

By default, all Simfra state lives in memory and is lost when the process exits. Setting SIMFRA_DATA_DIR enables persistence to SQLite, so resources survive restarts.

Persistence is driven by a background reconciler: it periodically sweeps every service's in-memory store, compares it against what was last written, and persists only the delta (new/changed resources are saved, deleted ones are removed). A final sweep runs on graceful shutdown. Reads always come from memory — SQLite is never on the request hot path. This means a service is durable simply by exposing its resources to the reconciler; there is no per-operation write code to forget, so coverage is uniform across all services.

Enabling Persistence

export SIMFRA_DATA_DIR=/var/lib/simfra
simfra

This creates:

  • $SIMFRA_DATA_DIR/simfra.db - SQLite database for resource metadata
  • $SIMFRA_DATA_DIR/s3/ - Filesystem storage for S3 object data
  • $SIMFRA_DATA_DIR/bootstrap/ - Terraform state when using Terraform bootstrap
  • $SIMFRA_DATA_DIR/.master_key - Auto-generated encryption key (if SIMFRA_PERSISTENCE_KEY is not set)

What Persists

Resource metadata and configuration for every stateful service are reconciled to SQLite:

  • IAM users, groups, roles, policies, access keys, instance profiles
  • EC2 VPCs, subnets, security groups, route tables, instances, volumes, AMIs, ENIs
  • S3 buckets (metadata and policies; object data stored on the filesystem)
  • SQS queues (configuration, attributes, tags - not messages)
  • SNS topics, subscriptions, and platform applications
  • DynamoDB tables, GSIs, and item data
  • Lambda functions, layers, event source mappings
  • KMS keys, aliases, and grants
  • RDS instances, clusters, subnet groups, parameter groups
  • EKS clusters, node groups, Fargate profiles, add-ons
  • ELBv2 load balancers, target groups, listeners, rules
  • CloudWatch alarms (including current alarm state and bounded alarm history), metric streams, log groups
  • EventBridge rules, targets, event buses
  • Secrets Manager secrets and versions
  • All other service resources with durable state

What Does NOT Persist

Transient runtime state is intentionally excluded:

  • SQS messages - Messages in queues, in-flight state, delayed delivery timers
  • FIFO dedup entries - 5-minute deduplication windows
  • Data key caches - KMS envelope encryption data keys (regenerated on first use after restart)
  • CloudWatch metric data points - Raw time-series datapoint buffers are ephemeral, high-churn data-plane state re-populated by the metric worker (alarm state and alarm history are persisted)
  • HTTP retry queues - Pending async deliveries (SNS, EventBridge targets)
  • Docker container state - Containers are recreated from persisted resource metadata on startup
  • STS session tokens - Temporary credentials are not persisted
  • In-flight async executions - Running deployments, pipeline/state-machine executions, ingestion/customization jobs, and live WebSocket connections are transient; terminal/historical records are persisted where AWS retains them

Startup Behavior

On startup with SIMFRA_DATA_DIR set, Simfra:

  1. Opens simfra.db (creates it if missing)
  2. Loads all persisted resources into memory stores in dependency order (IAM first, then KMS, EC2, etc.)
  3. Reconciles Docker infrastructure (if SIMFRA_DOCKER=true): recreates networks, DNS containers, and service containers from persisted metadata
  4. Skips native bootstrap if persisted state was found (Terraform bootstrap always runs since Terraform state handles idempotency)
  5. Marks the health endpoint as ready

The server does not accept requests until loading is complete.

Encryption at Rest

Sensitive fields in the SQLite database (secrets, private keys, passwords) are encrypted with AES-256-GCM.

Auto-Generated Key

When SIMFRA_PERSISTENCE_KEY is not set, Simfra generates a random 32-byte key and stores it at $SIMFRA_DATA_DIR/.master_key. This is convenient for single-machine setups but means the key is stored alongside the data.

Explicit Key

For production or shared-storage deployments, provide your own key:

# Generate a 32-byte key
openssl rand -hex 32

# Set it as an environment variable
export SIMFRA_PERSISTENCE_KEY=a1b2c3d4e5f6...  # 64 hex characters

The key must be exactly 64 hex characters (32 bytes). If the key changes or is lost, encrypted fields in the database become unreadable.

S3 Object Storage

S3 object data is stored on the filesystem under $SIMFRA_DATA_DIR/s3/, organized by bucket and key. Object metadata (ETags, content types, ACLs) is stored in SQLite alongside other resource metadata. This separation keeps the SQLite database small while supporting arbitrarily large objects.

On-Disk Blob Storage Coverage

Some services store file/blob content on the filesystem under $SIMFRA_DATA_DIR/<namespace>/ rather than in SQLite (object bodies, container images, function code, keys). For a restart these files simply stay on disk. For export/import (/_simfra/export → /_simfra/import, including cross-machine) each location must be covered by a persistence.BlobArchiver so its content travels in the dump. The table below is the authoritative map; every on-disk writer is either archived or explicitly exempt.

On-disk location Owner In export dump? Mechanism
$DATA_DIR/s3/ S3 object bodies Yes BodyStore BlobArchiver (s3)
$DATA_DIR/ecr/ ECR image layers Yes BlobStore BlobArchiver (ecr)
$DATA_DIR/lambda/ Lambda function code Yes CodeStore BlobArchiver (lambda)
$DATA_DIR/ca/ Root CA key/cert Yes CA AdminAdapter BlobArchiver (ca); ClearBlobs is a no-op so a dump without a CA blob keeps the existing root
$DATA_DIR/sts/token.key STS token-encryption key Yes tokenKeyArchiver BlobArchiver (sts); ClearBlobs no-op; import installs the key into the live encryptor so pre-import session tokens still decrypt
$DATA_DIR/.master_key Field-encryption master key No — deliberate Security: the exporter decrypts ENC: fields into the dump and the importer re-encrypts with the local key, so the source key must never travel. Dumps are secrets (plaintext credentials, private keys) — protect them accordingly.
EFS / Transfer-Family NFS content internal/nfs, efs Yes (Docker mode) Named volume simfra-nfs-data-<fsID>; VolumeArchiver namespace efs-nfs. In-place restart also survives via the mounted volume.
CodeCommit git repositories Docker volume Yes (Docker mode) Per-account git volume (SingletonVolumeName); VolumeArchiver namespace codecommit-git.
RDS / Redshift / DSQL / Redshift-Serverless / ElastiCache / Kafka / MQ / OpenSearch engine data Docker named volumes Yes (Docker mode) Each engine mounts a labeled named volume at its data dir (V1); the dump's volumes/ section carries them (dump format v2). Exemptions: memcached (no persistence) and RDS standby (resyncs from primary). Non-Docker mode has no engine volumes.

Engine/volume data is carried in the dump's volumes/<namespace>/ section (format v2) via persistence.VolumeArchiver, the volume analogue of BlobArchiver. It is present only in Docker mode; a dump taken (or imported) with Docker off carries (applies) resource metadata only and reports skipped volumes. Enable/disable with SIMFRA_EXPORT_VOLUMES (default true) or ?volumes=false.

Adding a new on-disk writer? Expose a BlobArchiver (implement persistence.BlobSourcer) with a matching BlobNamespace(), or a VolumeArchiver (persistence.VolumeSourcer) for Docker named volumes, or add a row here documenting the exemption. See docs/persistence-fromscratch-recipe.md.

Backup

To back up Simfra state:

  1. Stop Simfra (or accept a point-in-time snapshot)
  2. Copy $SIMFRA_DATA_DIR/ to your backup location
  3. Include simfra.db, the s3/ directory, and .master_key (or record your SIMFRA_PERSISTENCE_KEY)

The SQLite database uses WAL mode, so copying while Simfra is running produces a consistent snapshot as long as you copy both simfra.db and simfra.db-wal.