Common Issues

Port already in use

Symptom: Simfra fails to start with bind: address already in use on port 4599.

Solutions:

  1. Check for an existing Simfra process:

    ps aux | grep simfra

    Stop it gracefully (with kill <PID> or Ctrl-C), then retry.

  2. Use a different port:

    export SIMFRA_PORT=4600

    Update your AWS_ENDPOINT_URL to match.

  3. If another application is using the port, find it with:

    lsof -i :4599

Warning: Do not run lsof against the port if Docker-backed services are running - it may interfere with Docker containers that have port bindings.

Access Denied errors

Symptom: API calls return AccessDenied or AccessDeniedException.

Simfra enforces IAM policies on every request, matching real AWS behavior. "Access Denied" means the request was authenticated but not authorized.

Check these in order:

  1. Identity policies - Does the IAM user or role have a policy that allows the action? Check attached policies with aws iam list-attached-user-policies or aws iam list-attached-role-policies.

  2. Resource policies - Some services (S3, SQS, SNS, Lambda, KMS) have resource-based policies. An explicit deny in a resource policy overrides any allow.

  3. Permission boundaries - If the IAM entity has a permission boundary, the action must be allowed by both the identity policy and the boundary.

  4. SCPs - If the account is part of an Organization, Service Control Policies restrict what actions are allowed. Check with aws organizations list-policies.

  5. Root account - Root credentials bypass IAM policy evaluation. If you need to quickly verify whether the issue is IAM-related, retry with root credentials:

    export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

S3 operations failing

Symptom: S3 API calls fail with hostname resolution errors or 404s.

Simfra does not support virtual-hosted-style S3 requests (e.g. mybucket.s3.localhost). You must use path-style addressing.

Set the environment variable:

export AWS_S3_USE_PATH_STYLE=true

For AWS CLI:

aws s3api list-objects --bucket mybucket --endpoint-url http://localhost:4599

For SDKs: Enable forcePathStyle / use_path_style in the S3 client configuration.

Health check returns 503

Symptom: GET /_simfra/health returns HTTP 503 with {"status": "bootstrapping"}.

Simfra returns 503 during startup while it loads persisted state and runs any configured bootstrap. This typically takes a few seconds.

Solutions:

  1. Wait and retry. Most startups complete in under 5 seconds.

  2. If bootstrapping is configured (SIMFRA_BOOTSTRAP=standard), the Terraform bootstrap may take longer depending on the resources being created.

  3. Check the Simfra logs for errors:

    # Look for ERROR level messages in the output

Credential errors

Symptom: InvalidClientTokenId, SignatureDoesNotMatch, or IncompleteSignature errors.

Check these:

  1. Access key format - Access keys must be valid SigV4 format. The default root key is AKIAIOSFODNN7EXAMPLE. Custom keys created via IAM or the admin API also work.

  2. Secret key match - The secret key must match the access key. Check that AWS_SECRET_ACCESS_KEY corresponds to the AWS_ACCESS_KEY_ID you are using.

  3. Endpoint URL - Ensure AWS_ENDPOINT_URL points to Simfra (e.g. http://localhost:4599). If you are using HTTPS, the signing will fail unless TLS is properly configured.

  4. Region mismatch - SigV4 signatures include the region. If you sign with us-west-2 but the request routes to a different region, the signature may be invalid.

  5. Clock skew - SigV4 requires the client clock to be within 15 minutes of the server. This is rarely an issue on modern systems but can occur in containers with incorrect time.

  6. Expired temporary credentials - STS temporary credentials (from AssumeRole, GetSessionToken, etc.) expire after their configured duration. Refresh them if expired.

Stale state after restart

Symptom: Resources created in a previous session are missing after restarting Simfra.

By default, all state is in-memory and lost on restart. To persist state across restarts, set SIMFRA_DATA_DIR:

export SIMFRA_DATA_DIR=/path/to/data

This enables SQLite write-through persistence. Resource metadata (queues, topics, IAM entities, tags, etc.) survives restarts. Transient data (SQS messages, in-flight state, FIFO dedup entries) does not persist.

If persistence is enabled but state is still missing:

  1. Verify the data directory exists and is writable.
  2. Check that SIMFRA_DATA_DIR points to the same directory used in the previous session.
  3. Look for errors in the startup logs related to persistence loading.
  4. If you changed SIMFRA_PERSISTENCE_KEY between sessions, encrypted fields will not decrypt correctly.