The Orchestration Problem

§ 00 — BEFORE KUBERNETESThe container sprawl problem

Containers were a revolution — until you had hundreds of them. Before orchestration existed, teams discovered a new category of operational pain that Docker itself couldn't solve.

§ 01 — THE QUESTIONThe container orchestration problem

Containers solve the "it works on my machine" problem. Kubernetes solves the "who runs the container on which machine, keeps it alive when the machine dies, and routes traffic to it" problem — at the scale of tens of thousands of machines and hundreds of thousands of containers, with zero human hand-holding.

The question catches most candidates because Kubernetes looks like a deployment tool but is actually a distributed database with a reconciliation engine built on top. A weak answer names the components. A strong answer names the four forces that make this hard — and shows how every design decision exists to survive one of them.

There are exactly four forces:

THE DESIRED-STATE GAP

The cluster is always drifting away from what you declared. Nodes crash, containers OOM-kill, images fail to pull, network partitions split the cluster. The entire control plane architecture exists to continuously close the gap between "what the user declared" (etcd) and "what is actually running" (node status). This is reconciliation loops all the way down.

THE SCHEDULING HARDNESS

Optimal bin-packing is NP-hard. At 10,000 nodes with 50 constraints per pod, a brute-force scheduler would take minutes. Kubernetes solves this with a two-phase filter-then-score pipeline, plus sampling — the scheduler never evaluates all nodes. But "good enough in 10ms" requires deliberately accepting suboptimal placement.

THE ETCD BOTTLENECK

One consistent store for the entire cluster creates a write bottleneck. Every API object — pod, service, endpoint, configmap — is stored in etcd. Every watch (there are thousands of them) is registered against etcd's gRPC watch stream. The API server fans out watch events; etcd itself is sized to handle ~1,000 writes/sec. At 10,000 nodes sending heartbeats every 5s, that's 2,000 writes/sec — you're at the etcd ceiling from heartbeats alone.

THE NETWORK FLATNESS ASSUMPTION

Kubernetes assumes every pod can reach every other pod directly. The CNI spec delegates actual implementation to plugins (Calico, Cilium, Flannel) — Kubernetes itself has no routing table. This assumption powers flat pod networking but means network policy, encryption, and multi-cluster routing all have to be bolted on by the CNI.

Envelope math, volunteered:

§ 02 — PLAIN LANGUAGEThe shipping fleet analogy

Before we dive into etcd and gRPC watches, here is Kubernetes in one analogy that a non-engineer will remember.

Imagine a global shipping fleet. You are the fleet manager. You don't care which specific ship carries which cargo — you care that the cargo arrives, is replaced if the ship sinks, and can be found by other ships that need to connect with it.

THE CAPTAIN (Control Plane)

The control plane is the fleet captain — it knows every ship's capacity, what cargo is loaded on each, and what the manifest says should be loaded. It gives orders (schedule this pod here) but never touches the ships directly. The captain doesn't row.

THE SHIPS (Nodes)

Nodes are the ships — physical or virtual machines that actually run your containers. Each ship has a first mate (kubelet) who receives orders from the captain and actually loads/unloads cargo. The ship reports its capacity and health back to the captain every 5 seconds.

THE CARGO (Pods)

Pods are the shipping containers — the smallest unit of deployment. A pod contains one or more containers that share a network namespace and storage volumes. Just as a shipping container might carry multiple items, a pod might carry an app container plus a sidecar logging agent.

THE MANIFEST (Desired State)

When you run kubectl apply, you hand the captain a manifest: "I want 3 copies of this container, each with 1 CPU and 2GB memory, running port 8080." The captain's entire job is to make reality match the manifest — and to keep matching it, forever, even as ships sink and cargo goes overboard.

What Kubernetes solves

§ 03 — CONTROL PLANEThe brain of the cluster

The control plane is a set of processes that together maintain the desired state of the cluster. None of them run user workloads. All of them communicate exclusively through the API server — no component talks to etcd directly except the API server.

The five control plane components

How they communicate

The golden rule: every component communicates exclusively through the API server. The scheduler does not write to etcd. The controller manager does not call the scheduler. Every action is a write to the API server, which persists to etcd, which triggers a watch event, which wakes up the relevant controller or kubelet.

§ 04 — THE NODEWhat happens on a worker node

Every worker node runs three processes that together receive a pod spec and turn it into running containers.

kubelet

The node agent. Watches the API server for pods scheduled to this node. For each pod: calls the container runtime (via CRI) to pull the image and create containers, mounts volumes, configures the network namespace via CNI, runs health probes, and reports status back to the API server every few seconds. The kubelet is the only process that actually starts containers.

kube-proxy

Programs the node's network rules (iptables or IPVS) to implement Service routing. When a Service is created, kube-proxy watches the Endpoints object and writes iptables/IPVS rules so packets destined for the Service's ClusterIP get redirected to one of the healthy backing pods. Modern clusters use eBPF (Cilium) instead, replacing kube-proxy entirely.

container runtime (containerd / CRI-O)

The low-level runtime that speaks the Container Runtime Interface (CRI) gRPC protocol to kubelet. kubelet says "create this container with this image, these env vars, this resource limit." containerd pulls the image from the registry (OCI), creates the namespace isolation (using Linux namespaces + cgroups), and returns the container ID. Docker itself was removed as a supported runtime in Kubernetes 1.24.

The kubectl apply → pod running sequence

kubectl apply -f deployment.yaml
   │
   ▼
1. API SERVER receives HTTPS request
   ├── Authentication: who are you? (x509 cert / bearer token / OIDC)
   ├── Authorization: are you allowed? (RBAC check)
   ├── Admission control: is this valid? (ValidatingWebhookConfiguration + MutatingWebhookConfiguration)
   ├── Schema validation: is this a valid Deployment spec?
   └── Write to etcd: /registry/deployments/default/my-app  (resource_version increments)

2. DEPLOYMENT CONTROLLER (in controller-manager) notices new Deployment via watch
   └── Creates a ReplicaSet object:  /registry/replicasets/default/my-app-6d4f9c7b2

3. REPLICASET CONTROLLER notices new ReplicaSet
   └── Creates 3 Pod objects, each with spec.nodeName = "" (unscheduled)
       /registry/pods/default/my-app-6d4f9c7b2-xk9p2
       /registry/pods/default/my-app-6d4f9c7b2-m7q4r
       /registry/pods/default/my-app-6d4f9c7b2-p3n8s

4. SCHEDULER notices unscheduled pods via watch (spec.nodeName == "")
   ├── FILTER phase: remove nodes that cannot run this pod
   │     - PodFitsResources: does node have enough CPU/memory?
   │     - PodFitsHostPorts: is the host port available?
   │     - MatchNodeSelector: do labels match nodeSelector/nodeAffinity?
   │     - NoTaint / Toleration: are taints tolerated?
   │     - VolumeZoneConformance: is the PV zone compatible?
   ├── SCORE phase: rank remaining feasible nodes (0–100)
   │     - LeastRequestedPriority: prefer nodes with most free resources
   │     - BalancedResourceAllocation: balance CPU and memory usage
   │     - NodeAffinityPriority: prefer nodes matching preferred affinity
   └── BIND: writes Binding object → API server sets spec.nodeName = "node-42"

5. KUBELET on node-42 notices a pod bound to it via watch
   ├── Calls containerd via CRI: "create sandbox" (pause container for network namespace)
   ├── CNI plugin configures pod network (assigns IP from pod CIDR)
   ├── Calls containerd: "pull image" → overlay filesystem layers stacked
   ├── Calls containerd: "create containers" → cgroups + namespaces set up
   ├── Calls containerd: "start containers"
   ├── Runs init containers sequentially (if any), then all regular containers in parallel
   ├── Starts liveness/readiness/startup probes
   └── Updates pod status in API server: phase=Running, podIP=10.244.42.15

6. ENDPOINT SLICE CONTROLLER notices pod is Running + Ready
   └── Adds pod IP to EndpointSlice for all matching Services

7. kube-proxy on every node notices updated EndpointSlice
   └── Reprograms iptables/IPVS rules — traffic can now reach the new pod

Total time from kubectl apply to pod running: ~2–10 seconds (image pre-pulled) or ~30–90s (cold pull)

§ 05 — POD LIFECYCLEStates, probes, and init containers

A pod transitions through a well-defined set of states, and the probes that kubelet runs determine when a pod is considered healthy — and when it is evicted or restarted.

Init containers and sidecars

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      initContainers:
        - name: wait-for-db
          image: busybox:1.36
          command: ['sh', '-c', 'until nc -z postgres:5432; do sleep 2; done']

      containers:
        - name: payments-api
          image: company/payments-api:v2.1.4
          ports:
            - containerPort: 8080
          resources:
            requests:               # used for scheduling bin-packing
              cpu: "500m"
              memory: "512Mi"
            limits:                 # enforced by cgroups; OOM-kill at limit
              cpu: "2"
              memory: "2Gi"
          livenessProbe:            # fail → restart container
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:           # fail → removed from Service endpoints
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          startupProbe:             # one-time: disables liveness until /startup returns 200
            httpGet:
              path: /startup
              port: 8080
            failureThreshold: 30   # 30 × 10s = 5 minute grace window for slow start
            periodSeconds: 10

§ 06 — NETWORKINGFlat pods, Services, Ingress, DNS

Kubernetes networking rests on one core assumption: every pod has a unique IP, and every pod can reach every other pod directly, without NAT. Everything else — Services, Ingress, NetworkPolicy — is built on top of that flat network.

The four networking layers

DNS: CoreDNS

Every pod gets /etc/resolv.conf pointing at CoreDNS (a cluster-internal DNS server). CoreDNS resolves:

• payments-api → ClusterIP of the payments-api Service in the same namespace
• payments-api.default.svc.cluster.local → ClusterIP (fully-qualified)
• Headless Services (clusterIP: None) → A records for each pod IP directly (used for StatefulSets, Kafka brokers, etc.)

§ 07 — SCHEMA DESIGNetcd as a distributed database

Kubernetes stores every object in etcd as a key-value pair. Understanding the key schema and the watch mechanism is understanding why Kubernetes is architecturally elegant — and where its scalability limits lie.

The etcd key schema

# etcd key format: /registry/{resource-type}/{namespace}/{name}

/registry/pods/default/payments-api-6d4f9c7b2-xk9p2
/registry/pods/kube-system/coredns-74ff55c5b-4rz8k
/registry/deployments/default/payments-api
/registry/replicasets/default/payments-api-6d4f9c7b2
/registry/services/default/payments-api
/registry/endpoints/default/payments-api          # being replaced by EndpointSlices
/registry/endpointslices/default/payments-api-xz9k
/registry/configmaps/default/payments-api-config
/registry/secrets/default/payments-api-tls
/registry/nodes/node-42                           # cluster-scoped, no namespace
/registry/namespaces/production                   # cluster-scoped

# Value: serialized protobuf (not JSON, despite the API accepting JSON)
# Each write atomically increments the resource_version (cluster-wide monotonic counter)

The watch mechanism — Kubernetes as a distributed pub/sub

This is the most important architectural insight in Kubernetes. The entire system is an event-driven, watch-driven pub/sub built on top of a key-value store.

# etcd watch API (simplified)
# Every component establishes a long-lived gRPC watch to the API server

# The scheduler watches for unscheduled pods:
GET /api/v1/pods?fieldSelector=spec.nodeName=&watch=true&resourceVersion=12345
# → receives a stream of ADDED/MODIFIED/DELETED events

# The Deployment controller watches Deployments:
GET /apis/apps/v1/deployments?watch=true&resourceVersion=12345

# Each kubelet watches pods assigned to its node:
GET /api/v1/pods?fieldSelector=spec.nodeName=node-42&watch=true

# The critical insight: API server maintains ONE watch connection to etcd
# and fans out watch events to the potentially THOUSANDS of watchers.
# This "reflector + informer + workqueue" pattern appears in every K8s component:

# Informer lifecycle:
# 1. LIST all objects at startup (paginated, 500 at a time)
# 2. Sync the local in-memory cache
# 3. WATCH from the last seen resourceVersion
# 4. On reconnect, resume from last resourceVersion (no full re-list if etcd has the history)
# 5. If resourceVersion is too old (etcd compacted), fall back to full re-LIST

# resourceVersion is the etcd revision number — a cluster-wide monotonic integer.
# It enables optimistic concurrency:
# GET pod → resource_version=5000
# PUT pod (modify) with resourceVersion=5000
# → API server checks: current etcd revision == 5000? If not (someone else wrote), return 409 Conflict
# This is etcd's compare-and-swap used to prevent lost updates.

What if you replaced etcd with PostgreSQL?

This is a real interview question. etcd is Kubernetes' weakest scalability point — it tops out around 8GB of data and ~3,000 writes/sec. The question forces you to articulate what etcd actually provides.

The K8s community built kine — a shim that lets Kubernetes use MySQL, PostgreSQL, or SQLite as the backing store instead of etcd. K3s uses SQLite for single-node installs via kine. The watch API is the hardest part to replicate — kine implements it via polling + NOTIFY.

§ 08 — SCHEDULINGPlacing pods at scale

The scheduler's job is to assign each pending pod to a node that can run it. At 10,000 nodes, evaluating every node for every pod would be too slow. The solution is a two-phase pipeline with sampling.

The filter → score → bind pipeline

For each unscheduled pod:

PHASE 1: FILTER (predicates — binary pass/fail)
  Remove nodes that CANNOT run this pod:
  ├── PodFitsResources        cpu+memory requests fit within allocatable capacity
  ├── PodFitsHostPorts        no port conflict on the node
  ├── MatchNodeSelector       node labels match pod's nodeSelector / nodeAffinity
  ├── NoDiskConflict          required volumes can be attached
  ├── NoVolumeZoneConflict    PV zone matches node zone
  ├── MaxEBSVolumeCount       AWS: max 39 EBS volumes per node
  ├── MatchInterPodAffinity   pod can coexist with existing pods on this node
  ├── PodToleratesNodeTaints  pod tolerates all NoSchedule taints on the node
  └── (... 20+ predicates total)

  Result: feasibleNodes (could be 0 → Pending, or 1–N)

PHASE 2: SCORE (priorities — 0–100 per node)
  Rank feasible nodes to pick the best:
  ├── LeastRequestedPriority      (100 - (cpu% + mem%) / 2) — prefer emptier nodes
  ├── BalancedResourceAllocation  penalize imbalanced cpu:memory ratio
  ├── NodeAffinityPriority        weight preferred node affinity rules
  ├── InterPodAffinityPriority    prefer/avoid nodes with specific pods
  ├── ImageLocalityPriority       bonus for nodes that already have the image cached
  └── (custom plugins via Scheduling Framework)

  Result: scored nodes sorted descending

PHASE 3: BIND
  Selected node → create Binding object via API server
  API server: SET spec.nodeName = "node-42" in etcd
  This is a compare-and-swap — if two schedulers race, one wins, one gets 409 Conflict

──────────────────────────────────────────────────────────────────────
10,000-NODE PERFORMANCE TRICK: SAMPLING

  Instead of scoring ALL 10,000 feasible nodes:
  percentageOfNodesToScore: 50   (default in large clusters)
  → after passing filters, randomly sample min(50%, 100 nodes) for scoring
  → scoring phase runs on ~100 nodes, not 10,000
  → worst case: slightly suboptimal placement
  → benefit: scheduling latency stays <5ms per pod

  The scheduler also runs in parallel goroutines — ~16 pods scheduled concurrently.

Advanced scheduling constraints

# Node affinity — require or prefer nodes by label
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:   # hard requirement
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values: [us-east-1a, us-east-1b]
    preferredDuringSchedulingIgnoredDuringExecution:  # soft preference
      - weight: 80
        preference:
          matchExpressions:
            - key: node-type
              operator: In
              values: [high-memory]

# Pod anti-affinity — spread pods across zones (prevent all replicas on one node)
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: payments-api
        topologyKey: kubernetes.io/hostname   # no two replicas on same node

# Topology spread constraints (K8s 1.19+) — more flexible than anti-affinity
topologySpreadConstraints:
  - maxSkew: 1                               # max 1 replica difference between zones
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: payments-api

# Taints and tolerations — reserve nodes for specific workloads
# Node: kubectl taint nodes gpu-node-01 dedicated=gpu:NoSchedule
# Pod tolerates the taint:
tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule

§ 09 — AUTOSCALINGHPA, VPA, and Cluster Autoscaler

Kubernetes has three layers of autoscaling, each operating at a different timescale and on different resources.

KEDA — extending HPA with external metrics

# KEDA (Kubernetes Event-Driven Autoscaling) scales on external signals:
# queue depth, Kafka lag, HTTP RPS, cron schedule, custom Prometheus metrics

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: payments-worker
spec:
  scaleTargetRef:
    name: payments-worker
  minReplicaCount: 0             # scale to zero when queue is empty (save $$$)
  maxReplicaCount: 50
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123456789/payments-queue
        queueLength: "10"        # target: 10 messages per replica
        awsRegion: us-east-1
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: payments_queue_depth
        threshold: "100"         # scale up if >100 unprocessed payments per replica

Cluster Autoscaler — adding nodes

When pods are stuck in Pending because no node has enough capacity, the Cluster Autoscaler asks the cloud provider to provision a new node. When nodes have been underutilized for 10+ minutes and all pods could be rescheduled to other nodes, it cordons and drains the node then deletes it.

# Key Cluster Autoscaler behaviors:

# Scale-up trigger:
# pod is Pending AND can be scheduled if one more node of type X exists
# → call cloud provider API: "add 1 node to node group X"
# → node joins cluster in ~90 seconds
# → scheduler places the pending pod

# Scale-down trigger:
# node utilization < 50% of requests for 10 minutes
# AND all pods can be rescheduled elsewhere
# → cordon node (no new pods scheduled)
# → gracefully evict pods (respecting PodDisruptionBudgets)
# → cloud provider: "delete this node"

# PodDisruptionBudget — protect critical workloads from eviction:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api-pdb
spec:
  minAvailable: 2              # always keep at least 2 pods running during disruptions
  selector:
    matchLabels:
      app: payments-api

§ 10 — RBAC & SECURITYWho can do what to which resources

Kubernetes RBAC is a four-part system: Subjects (who), Verbs (what action), Resources (on what), and Scope (namespace or cluster). Every API request goes through RBAC before reaching etcd.

# ServiceAccount — the identity a pod runs as
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-api
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/payments-api  # IRSA for AWS

---
# Role — namespaced permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payments-api-role
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials", "stripe-key"]  # specific secrets only
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]

---
# RoleBinding — bind the Role to the ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-api-binding
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: payments-api
    namespace: payments
roleRef:
  kind: Role
  name: payments-api-role
  apiGroup: rbac.authorization.k8s.io

---
# NetworkPolicy — allow only specific pod-to-pod traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-api-netpol
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: ingress-nginx       # allow from ingress only
      ports:
        - port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres            # allow to postgres only
      ports:
        - port: 5432

Security layers in depth

§ 11 — Q&AFifteen tough interview questions

These questions separate the senior answer from the staff answer — etcd internals, split-brain, rolling strategies, StatefulSet vs Deployment, CRDs, and multi-tenancy.

§ 12 — SUMMARYWhat the strong answer looks like

← Back to Design Scenarios

§ 09 — COMMON MISTAKESWhat trips up experienced engineers

Even engineers who understand Kubernetes conceptually make these operational mistakes. Recognise them before your interviewer does.

§ 10 — WHY NOT?Kubernetes isn't always the answer

Every system design interview expects trade-offs. Know when Kubernetes is the wrong choice — and be able to say so clearly.

§ 11 — ONE-MINUTE ANSWERIf you only remember one thing

When an interviewer asks "Why does Kubernetes exist?" you need to be able to answer in 60 seconds — clearly, with the insight, not just the features.

§ 12 — INTERVIEWER'S MINDWhat they're really testing

When an interviewer asks about Kubernetes, they're probing four levels of understanding. Here's what they want to hear at each level.

§ 13 — THE EVOLUTIONHow we got here

Kubernetes didn't appear in a vacuum. It is the product of a decade of industry convergence around the right abstraction for running software at scale.

§ 14 — WHAT'S NEXT?The frontier beyond orchestration

Kubernetes solved the orchestration problem. Each solution created the next problem. Here is the chain of constraints as the industry has moved up the abstraction ladder.