Airbyte on EKS

The upstream Airbyte data integration platform deployed on Kind-backed EKS with AWS-native infrastructure controllers. Airbyte's multi-component architecture (server, webapp, worker, cron, Temporal) connects to RDS PostgreSQL and S3 for state and logs. A custom config-init image is built by CodeBuild, pushed to ECR, and deployed alongside upstream Airbyte images. Three IRSA-based controllers (ALB, ExternalDNS, External Secrets Operator) manage AWS resources from Kubernetes manifests.

Services

Service	Role
EKS	Kind-backed cluster hosting Airbyte components
EC2	VPC networking and security groups
ELBv2	ALB created by AWS Load Balancer Controller from Ingress
RDS	PostgreSQL for Airbyte and Temporal metadata, KMS encrypted
S3	Two buckets: logs and state storage, both SSE-KMS
SQS	Operational notification queue
Route53	Hosted zone with records managed by ExternalDNS
ACM	TLS certificate for HTTPS
KMS	Encryption keys for RDS, S3, and EKS secrets
IAM/STS	Eight roles: cluster, node, pod, LB controller, ExternalDNS, ESO, CI/CD
Secrets Manager	Database credentials and application secrets
ECR	Repository for custom config-init image
CodeCommit	Source repository with Go config-init code and deployment manifests
CodeBuild	Builds config-init Docker image, pushes to ECR, renders and applies manifests
CodePipeline	Orchestrates build and deployment
CloudWatch Logs	Pod execution logs

Architecture

Client --> Route53 --> ALB (HTTPS, ACM)
                         |
                         v
                    EKS Kind Cluster
                    ┌────────────────────────────────────┐
                    │ Airbyte components:                 │
                    │  temporal (:7233)                   │
                    │  server (:8001)                     │
                    │  webapp (:8080)                     │
                    │  worker, cron, bootloader           │
                    │  config-init (custom Go image)      │
                    └────────────────────────────────────┘
                       |        |        |        |
                       v        v        v        v
                     RDS      S3(2)    SQS    Secrets
                    (PG)    (logs,   (notify)  Manager
                           state)

Controllers (IRSA):
  AWS LB Controller    --> creates ALB from Ingress
  ExternalDNS          --> creates Route53 A records
  External Secrets Op  --> syncs Secrets Manager --> K8s Secrets

The deployment includes a custom config-init container built from Go source by CodeBuild. This image initializes Airbyte configuration and is pushed to ECR alongside the deployment manifests. External Secrets Operator synchronizes Secrets Manager entries into Kubernetes Secrets across the deployment namespace, providing database credentials and application configuration to Airbyte pods.

What This Validates

• Complex multi-component Kubernetes application (Airbyte + Temporal) on EKS
• External Secrets Operator syncing from Secrets Manager ClusterSecretStore to K8s Secrets
• AWS Load Balancer Controller creating ALB from Ingress annotations
• ExternalDNS creating Route53 records from Ingress annotations
• IRSA with OIDC for three controllers and workload pods
• Custom image build (Go config-init) via CodeBuild with ECR push
• Upstream container images pulled directly at deploy time
• RDS PostgreSQL as shared metadata store for Airbyte and Temporal
• S3 for Airbyte logs and connector state
• SQS operational notifications from within EKS pods

Test Coverage

Tests cover CI/CD pipeline execution and ECR image verification, smoke checks for health endpoints, HTTPS connectivity, and pod status. Integration tests validate Airbyte API version, workspace creation, source and destination definitions, config-init execution, connection setup, and sync job lifecycle. Security tests verify KMS encryption on RDS, S3, and EKS secrets, IRSA credential validation, and ExternalSecret synchronization. Performance tests run 50 concurrent health checks, 10 concurrent API operations, and 5 concurrent workspace calls.