What Happens When You Make an API Call

This page walks through the exact lifecycle of a request to Simfra. We'll use a concrete example - creating an SQS queue - but the same pipeline applies to every operation across all 88 services.

aws sqs create-queue --queue-name my-queue --endpoint-url http://localhost:4599

The Full Pipeline

HTTP POST localhost:4599/
  Content-Type: application/x-www-form-urlencoded
  Authorization: AWS4-HMAC-SHA256 Credential=AKIA.../us-east-1/sqs/aws4_request, ...
  Body: Action=CreateQueue&QueueName=my-queue

  -> Recovery middleware (panic protection)
  -> Logging middleware (request ID, timing)
  -> CORS middleware (permissive headers for local dev)
  -> dispatch():
      1. Route admin/internal/UI paths (not an admin request, skip)
      2. Create RequestContext (generate UUID request ID, set default region)
      3. Parse SigV4 auth header -> extract access key, region, service
      4. Pre-resolve service from Host header -> SQS
      5. Read and validate request body (size limits, gzip decompression)
      6. Verify SigV4 signature (HMAC-SHA256 against stored secret key)
      7. Resolve caller principal (access key -> account ID, principal ARN, type)
      8. Detect protocol -> Query (form-encoded body with Action= parameter)
      9. Resolve operation -> "CreateQueue" (from Action= parameter)
     10. Parse request body into typed CreateQueueInput struct
     11. IAM policy evaluation (explicit deny, SCPs, resource policy,
         permission boundary, identity policies, session policies)
     12. Call operation handler: func(ctx, *CreateQueueInput) -> (*CreateQueueOutput, error)
     13. Handler creates queue in memory store, writes through to SQLite
     14. Serialize response as XML (Query protocol)
     15. Record CloudTrail event (deferred)
  -> HTTP 200, XML response body

Let's look at each step in detail.

1. Middleware Chain

Every request passes through three middleware layers before dispatch:

Recovery catches panics in any handler and returns a 500 Internal Server Error instead of crashing the process. The panic and stack trace are logged. This means a bug in one service handler won't take down Simfra - other services keep working.

Logging wraps the response writer to capture the HTTP status code, then logs the request after it completes. The log line includes the method, path, status, duration, service name, operation name, and request ID. Internal container-to-Simfra requests (/_internal/*) are logged at DEBUG level to reduce noise.

CORS adds permissive cross-origin headers (Access-Control-Allow-Origin: *) to every response. This allows browser-based tools and the Simfra admin UI to make requests directly to the gateway. OPTIONS preflight requests are answered immediately with 200.

2. Path-Based Routing

Before any AWS processing begins, the dispatcher checks the URL path:

  • /_simfra/* - Admin API (health checks, resource browser, state management). No AWS auth required.
  • /_internal/* - Service-to-service API used by Docker containers (Lambda invoke, config pull, status reporting). Requires an internal bearer token, not SigV4.
  • /v2/* - Docker Registry V2 API for ECR image push/pull. Routed to the ECR registry handler.
  • Everything else - Standard AWS API request. Proceeds through the SigV4 pipeline.

If Simfra has an embedded UI and the request has no AWS auth headers (e.g., a browser navigation), it serves the UI instead. This is how you can open http://localhost:4599 in a browser and see the admin dashboard while SDKs use the same port for API calls.

3. Authentication

SigV4 Parsing

The Authorization header is parsed to extract:

  • Access key ID (e.g., AKIAIOSFODNN7EXAMPLE)
  • Region (e.g., us-east-1)
  • Service (e.g., sqs)
  • Signature (the HMAC-SHA256 value)

The region from the SigV4 credential scope overrides the default region for this request.

Signature Verification

Simfra looks up the secret key for the given access key ID:

  1. Root credentials - checked against the account registry (configured via SIMFRA_ROOT_ACCESS_KEY/SIMFRA_ROOT_SECRET_KEY, or additional accounts created via the admin API).
  2. STS temporary credentials (ASIA* prefix) - the session token (X-Amz-Security-Token header) is decrypted to recover the temporary secret key.
  3. IAM access keys (AKIA* prefix) - looked up in the IAM service's stored access keys.

The signature is then recomputed following the SigV4 signing algorithm and compared. If it doesn't match, the request fails with SignatureDoesNotMatch.

Principal Resolution

After signature verification, Simfra determines who the caller is:

  • Root key - Principal is arn:aws:iam::{account}:root, type is root.
  • STS session - Principal is the assumed role session ARN, type is assumed-role. The original role ARN and session name are extracted from the decrypted session token.
  • IAM access key - Principal is the IAM user ARN, type is user.

The principal ARN, account ID, and principal type are stored on the request context and used for IAM evaluation and CloudTrail recording.

Unauthenticated Operations

A small number of AWS operations don't require authentication (the SDK doesn't send SigV4 headers). These include cognito-identity:GetId, sts:AssumeRoleWithWebIdentity, and a few others. Simfra detects these by checking if the service implements an UnauthenticatedOperationProvider interface. For these operations, the default account is used and IAM evaluation is skipped.

4. Service and Protocol Resolution

The target service is resolved from the request (see How Simfra Works for the resolution order). For our SQS example, the Host header sqs.us-east-1.localhost:4599 maps to the SQS service.

The protocol is determined from the service's declared protocol list and the request's Content-Type:

  • application/x-www-form-urlencoded -> Query protocol
  • application/x-amz-json-1.0 or application/x-amz-json-1.1 -> JSON protocol
  • application/cbor with smithy-protocol: rpc-v2-cbor -> CBOR protocol
  • REST services (S3, Lambda) use route matching regardless of content type

For SQS, the form-encoded body triggers the Query protocol path.

5. Operation Resolution and Input Parsing

Query/EC2 protocol: The operation name comes from the Action parameter in the request body or query string (Action=CreateQueue). The form-encoded body is parsed using a reflection-based parser that maps parameters like QueueName=my-queue and Attribute.1.Name=VisibilityTimeout&Attribute.1.Value=30 to fields on a typed Go struct.

JSON protocol: The operation comes from the X-Amz-Target header (AmazonSQS.CreateQueue). The JSON body is unmarshaled into the input struct.

REST protocols: The operation is resolved by matching the HTTP method and URL path against a routing table (e.g., POST /2015-03-31/functions maps to Lambda's CreateFunction). Path parameters, query parameters, and headers are extracted into the input struct alongside the body.

CBOR protocol: The operation is extracted from the URL path (/service/Operation). The binary CBOR body is decoded into the input struct.

In all cases, the result is a typed Go struct (e.g., *sqs.CreateQueueInput) with all request data populated.

6. IAM Policy Evaluation

With the caller's identity resolved and the target action known, Simfra evaluates whether the request is authorized. The action is formed as {service}:{operation} - in this case, sqs:CreateQueue.

The evaluation follows the AWS IAM reference logic:

  1. Bypass check - A few operations are unconditionally allowed for any authenticated principal (e.g., sts:GetCallerIdentity).
  2. Root principal - Root gets implicit full access. If the account is a member of an Organization, SCPs are still checked.
  3. IAM user or assumed role - The full evaluation chain runs:
    • Gather all identity policies attached to the principal (user policies, group policies, role policies, both inline and managed).
    • Gather the permission boundary if one is set.
    • Gather SCPs and RCPs from Organizations if applicable.
    • Look up resource-based policies on the target resource if it has one.
    • Parse any session policies from the AssumeRole call.
    • Run the policy evaluation engine, which checks for explicit denies, then evaluates each policy layer.
  4. Decision - Allow proceeds to the handler. Deny returns an AccessDenied error with a message indicating which principal was denied which action.

The resource ARN for the evaluation is extracted from the input struct (e.g., the queue ARN derived from the queue name and account/region).

7. Handler Execution

The handler is a Go function with this signature:

func(ctx context.Context, input *CreateQueueInput) (*CreateQueueOutput, error)

The context carries the RequestContext with account ID, region, principal ARN, and other metadata. The handler:

  1. Validates inputs - checks queue name format, attribute values, FIFO suffix requirements, and other AWS-specific constraints.
  2. Acquires locks - per-resource mutexes (not global locks) protect concurrent access.
  3. Creates the resource - the queue is added to the account+region store with correct defaults (visibility timeout, retention period, etc.).
  4. Writes through to persistence - if enabled, the queue metadata is saved to SQLite.
  5. Returns the response - a CreateQueueOutput struct with the queue URL.

If the handler returns an error, it's an *AWSError with the correct error code, message, and HTTP status that AWS would return (e.g., QueueAlreadyExists if the queue already exists with different attributes).

The entire handler runs in a single goroutine - the same goroutine that handles the HTTP request. State changes are atomic per operation because they're protected by mutexes. There's no database transaction to worry about since the in-memory state is the source of truth.

8. Response Serialization

The response is serialized according to the protocol that was used for the request:

Query protocol (SQS, SNS, IAM, STS): The output struct is serialized as XML with an {Operation}Response wrapper and a ResponseMetadata element containing the request ID:

<?xml version="1.0" encoding="UTF-8"?>
<CreateQueueResponse>
  <CreateQueueResult>
    <QueueUrl>http://localhost:4599/000000000000/my-queue</QueueUrl>
  </CreateQueueResult>
  <ResponseMetadata>
    <RequestId>a1b2c3d4-e5f6-7890-abcd-ef1234567890</RequestId>
  </ResponseMetadata>
</CreateQueueResponse>

JSON protocol (DynamoDB, KMS, ECS): The output is serialized as JSON with x-amzn-RequestId in the response header and a CRC32 checksum in X-Amz-Crc32.

REST-JSON (Lambda, API Gateway): JSON body with service-specific status codes (e.g., 201 for resource creation) and response headers mapped from output struct fields.

REST-XML (S3, Route53): XML body with service-specific elements. S3 uses <Error> for errors while other REST-XML services use <ErrorResponse>.

CBOR (CloudWatch): Binary CBOR encoding with smithy-protocol: rpc-v2-cbor header.

Error responses are serialized in the same protocol-specific format. An SQS error is XML, a DynamoDB error is JSON, and a CloudWatch error is CBOR - matching what the corresponding AWS SDK expects to parse.

9. CloudTrail Recording

After the response is sent (via defer), a CloudTrail event is recorded with:

  • Event time, account ID, region
  • Service name and operation
  • Principal ARN, principal type, access key ID
  • Source IP and user agent
  • Request ID
  • Error code and message (if the request failed)
  • Whether the operation is read-only
  • Request parameters (for management events)
  • Response elements (for write management events that succeeded)

This means CloudTrail's LookupEvents API returns real data for calls you've made. You can audit what happened, when, and by whom - the same way you would on real AWS.

Data events (S3 object operations, DynamoDB item operations, Lambda invocations, SQS message operations) are classified separately from management events, matching AWS CloudTrail's data event categorization. Request and response parameters are omitted from data events to avoid storing large payloads.

Concurrency

Each HTTP request runs in its own goroutine. State access is protected by mutexes:

  • Store-level: The AccountRegionStore uses a sync.RWMutex for the account+region map. Getting or creating a store for an account+region pair uses read-lock fast path with write-lock fallback (double-checked locking).
  • Resource-level: Individual services use per-resource mutexes where appropriate. For example, SQS uses per-queue mutexes so operations on different queues don't block each other.

Cross-service side effects (SNS delivering to SQS, Lambda ESM polling, EC2 state transitions) happen asynchronously via background worker goroutines, not inline with the originating request. This matches AWS behavior - when you publish to an SNS topic with SQS subscriptions, the Publish call returns immediately and delivery happens asynchronously.

Request Timeouts

Each request has a context deadline based on the target service. Most services get the configured SIMFRA_REQUEST_TIMEOUT (default 120s). Services with streaming operations or potentially long-running handlers may have different limits. If a handler exceeds the deadline, the context is cancelled and the request returns an error.