System Design: CI/CD Pipeline

Requirements

Functional Requirements:

• Trigger builds on git push, pull request, or schedule
• Execute parallel build and test jobs across multiple agents
• Cache dependencies and build artifacts to speed up repeat builds
• Manage deployment pipelines across dev, staging, and production environments
• Support blue-green and canary deployment strategies
• Notification of build success/failure via Slack, email, and GitHub status checks

Non-Functional Requirements:

• Build start latency: under 30 seconds from trigger to first job running
• Support 10,000 concurrent build jobs
• Artifact storage: 100 TB for build artifacts and logs
• 99.9% pipeline availability
• Build logs streamed in real time to developers

Scale Estimation

With 10,000 concurrent builds and an average build time of 10 minutes, the system completes 60,000 builds/hour. Build agents: assuming 4 vCPUs per agent and 1 job per agent, 10,000 agents are needed. At $0.10/hour per vCPU (cloud spot pricing), concurrent cost is $4,000/hour for full capacity. Build artifact storage: a compiled Java service produces ~100 MB of JARs; test results (JUnit XML) add 1 MB. At 60,000 builds/hour × 101 MB = 5.8 TB/hour. With 30-day retention, total storage is 4.2 PB. Deduplicated (many builds share unchanged dependencies), effective storage is ~500 TB.

High-Level Architecture

The CI/CD system has five major components: event ingestion (webhook receiver for VCS events), pipeline scheduler (parses YAML, creates job dependency graph), job queue (distributes jobs to agents), agent fleet (executes jobs), and artifact store (stores build outputs). An orchestration layer tracks pipeline state and manages the job execution lifecycle. A notification service sends status updates to configured channels.

VCS webhook → Event Ingestion Service validates the webhook signature, parses the event (push, PR, tag), and publishes to a Kafka topic. The Pipeline Scheduler consumes from Kafka, fetches the CI YAML from the repository (via VCS API), parses the DAG of stages and jobs, resolves dependencies, and creates pipeline + job records in PostgreSQL. Immediately-runnable jobs (no dependencies) are enqueued to the job queue (backed by Redis or a purpose-built queue). Agents poll the queue, accept jobs, and execute them. On job completion, the scheduler evaluates which downstream jobs are now unblocked and enqueues them.

Deployment pipelines add an environment approval gate: after the staging deploy job succeeds, the production deploy job is created but held in a "manual" state (blocked until a human approves). The approval action (via UI or API) transitions the job to "pending", where it is picked up by an agent. Blue-green deployments are implemented as two deploy jobs running in parallel: one targets the blue environment, one targets green. A final "switch" job updates the load balancer to point to the newly deployed environment after health checks pass.

Core Components

Build Caching

Build caching is the most impactful performance optimization. The cache key is a hash of the inputs: dependency manifest files (package.json, pom.xml, go.sum), build tool version, OS image. If the cache key matches a stored cache, the cached directory (e.g., node_modules, .m2/repository) is restored from a local or remote cache store before the build step runs. Fetching a 200 MB node_modules cache from a regional S3 bucket takes 5–10 seconds vs. 120 seconds for npm install from scratch: 10–20x speedup. Cache stores are sharded per repository + branch; main branch cache is shared as a fallback for feature branches that miss their branch-specific cache.

Parallel Job Execution

The scheduler analyzes the job DAG and identifies parallelizable jobs (no dependency relationship between them). These jobs are all enqueued simultaneously. A test suite split across 20 parallel jobs reduces wall-clock test time by 20x. Test splitting uses a historical test duration database to assign similar-runtime test groups to each parallel job (dynamic test grouping), achieving balanced parallel execution. For matrix builds (test against Python 3.9, 3.10, 3.11 × Linux, macOS, Windows), the scheduler generates N×M jobs automatically from the matrix specification, all running in parallel.

Agent Isolation & Security

Each agent executes jobs in isolated environments. Docker-in-Docker (DinD) runs each job in a fresh container: docker run --rm job_image sh -c "<script>". The container has no access to the host network (network=bridge isolated), no host filesystem mounts (except explicitly granted artifact volumes), and is terminated after job completion. Secrets (API keys, deployment credentials) are injected as environment variables from a secrets management service (Vault, AWS Secrets Manager) — never stored in the YAML or artifact store. Ephemeral agents (VMs provisioned per-job and terminated after completion) provide the strongest isolation and prevent state leakage between jobs.

Database Design

Pipeline metadata in PostgreSQL: pipelines (id, project_id, commit_sha, ref, status, created_at, triggered_by), jobs (id, pipeline_id, name, stage, status, agent_id, started_at, finished_at, exit_code, log_key), artifacts (id, job_id, name, size, expire_at, storage_path). Build logs are stored as append-only files in object storage (S3), keyed by log_key (UUID). Logs are streamed to S3 in real time during job execution (multipart upload with 1 MB parts). The UI fetches log chunks via a streaming API backed by S3 byte-range reads. Cache entries are stored in S3 with composite keys: {org}/{repo}/{branch}/{cache_key}.tar.gz. TTL-based expiry via S3 lifecycle rules automatically purges stale caches after 7 days.

API Design

Scaling & Bottlenecks

Agent fleet scaling is the primary operational concern. Auto-scaling rules watch queue depth: if jobs have been waiting >30 seconds, spin up new agents. If agent utilization drops below 20% for 10 minutes, terminate idle agents. Cloud spot/preemptible instances reduce cost by 60–80% but introduce interruption risk — interrupted jobs are automatically retried on a new agent (jobs must be idempotent for retry). Agent warm pools (pre-provisioned agents with base Docker image pre-pulled) reduce cold start latency from 3 minutes (VM boot + Docker pull) to 10 seconds.

Artifact storage cost is dominated by test reports and binary build outputs. Deduplication reduces storage: build artifacts that haven't changed between commits (e.g., a library module that wasn't touched) can reuse the artifact from a previous build by content hash. A content-addressed artifact store (similar to block deduplication in Dropbox) eliminates redundant uploads. Retention policies are critical: production release artifacts are kept indefinitely; PR build artifacts are deleted after 7 days; nightly build artifacts are deleted after 3 days.

Key Trade-offs

• Ephemeral vs. persistent agents: Ephemeral agents (per-job VMs) provide perfect isolation and no state pollution but add 1–3 minutes of provisioning overhead; persistent agents start jobs instantly but accumulate state and risk contamination between jobs
• Build cache freshness vs. correctness: Aggressive caching speeds up builds but risks using stale cached outputs for changed inputs; cryptographic cache keys (hash of all inputs) prevent stale cache hits at the cost of cache miss on any input change
• Blue-green vs. canary vs. rolling deployment: Blue-green is instant switchover (zero downtime) but requires 2x infrastructure; canary gradually shifts traffic (lower risk, gradual rollout) but keeps old and new versions running simultaneously for minutes to hours; rolling updates are resource-efficient but briefly run mixed versions
• Centralized vs. distributed pipeline execution: A central scheduler has full visibility into the pipeline DAG but is a single point of failure; distributed execution (each agent self-schedules based on event subscriptions) is more resilient but harder to reason about and debug