The Container Problem

§ 00 — BEFORE CONTAINERSThe evolution that made containers inevitable

Containers did not appear in a vacuum. Each step in infrastructure history created a new problem that the next step solved — until containers created their own problem: scale management.

PHYSICAL SERVER

Bare metal: one application per server. Utilization typically 5–15%. A new app means provisioning a new server — weeks of lead time. Upgrade a library and you risk breaking the OS or another tenant. Hardware is expensive; idle hardware is waste.

VIRTUAL MACHINES

Better utilization — run multiple VMs on one host. But each VM carries a full OS kernel: 0.5–2 GB of overhead, 30–120 seconds to boot. Spinning up 100 VMs for a traffic spike takes minutes. VM images are gigabytes; CI pipelines become slow.

DEPENDENCY HELL

"Works on my machine" — the definitive failure mode. App A needs Python 3.9 + OpenSSL 1.1. App B needs Python 3.11 + OpenSSL 3.0. On a shared server, one breaks the other. Virtualenv, rbenv, nvm are band-aids. The problem is the environment is not portable.

CONTAINERS

Package the application with its dependencies — not the full OS — into a portable unit. MBs instead of GBs. Seconds to start instead of minutes. Run identically on a developer's laptop, in CI, and in production. Docker's 2013 launch democratized what Google had been doing internally (Borg, lmctfy) for years.

CONTAINER SPRAWL

Containers solved packaging and isolation. But 500 containers across 20 hosts — how do you restart a failed container? How do you roll out a new version without downtime? How do you balance load? Manual management at scale is chaos. This is the problem Kubernetes was built to solve.

§ 01 — THE QUESTIONContainers are not VMs

A container looks like isolation. It is, in fact, a regular Linux process using kernel namespaces, cgroups, and a union filesystem to create the illusion of a private machine — without the hypervisor overhead that makes VMs take 30+ seconds to boot.

The question catches most candidates because containers sit at an uncomfortable intersection: it's operating systems, distributed storage, networking, and a high-scale CDN problem all at once. A weak answer draws a box labeled "Docker" and describes docker run. A strong answer names the four forces that make the design hard — then shows how every later decision exists to survive one of them.

THE LAYER SHARING PROBLEM

Images are expensive; layers are cheap. A base Ubuntu layer (100 MB) used by 10,000 images should be stored and transferred exactly once — not once per image. The whole registry design flows from this: content-addressable storage by SHA256 digest means a layer appears once in the blob store no matter how many images reference it. Pull deduplication is free.

THE ISOLATION ILLUSION

Containers share the host kernel — they don't own it. Network, PID, mount, UTS, and IPC namespaces create the appearance of isolation. cgroups enforce resource limits. OverlayFS creates the private filesystem view. If any of these kernel primitives is misconfigured, containers can escape to the host. This is why container security is fundamentally harder than VM security.

THE 10B PULLS PROBLEM

A registry is the hottest cache in infrastructure. Every CI job, every pod startup, every autoscale event pulls layers. The bottleneck is not compute — it's bandwidth, and specifically the last-mile latency between the registry and the cluster. CDN, P2P, and lazy-loading each exist to solve a different part of this bandwidth cliff.

THE FORMAT FRAGMENTATION PROBLEM

Docker format ≠ OCI format — but compatibility is expected. Docker's v2 manifest schema and the OCI Image Spec are nearly identical but differ in media types. Multi-arch image support (arm64, amd64, arm/v7) requires a manifest list (OCI: image index). BuildKit, Podman, containerd, and the CRI all have their own quirks. A registry must handle all of them seamlessly.

Envelope math, volunteered:

§ 02 — PLAIN LANGUAGEWhat is a container, really?

Before the engineering, the mental model. Three analogies that actually hold up under scrutiny.

The VM analogy (and why it breaks)

A virtual machine is a full computer running inside your computer — it has its own kernel, its own memory allocator, its own device drivers. Starting one takes 30 seconds because the OS is actually booting. A container is not a virtual machine. It is a process that thinks it has its own computer — it runs directly on the host kernel, uses the host's network stack, and shares memory with the host. The kernel creates an illusion of isolation using namespaces. The illusion is convincing, but it is not a hard wall.

What Docker solves — "works on my machine"

Before Docker, deploying software meant: (1) install the right version of Java/Python/Node on the server, (2) hope it matches your laptop, (3) debug the production difference at 2 AM. Docker's insight was simple: ship the filesystem, not just the code. Package the app, its runtime, its libraries, and its configuration into an image — and that image runs identically on your laptop, in CI, and in production. The image is the deployment unit. "Works on my machine" becomes "this is the machine."

The shipping container analogy

Before standardized shipping containers, loading a cargo ship meant custom-stacking thousands of different-shaped parcels — slow, lossy, requiring specialists at every port. After: everything fits into an ISO-standard container. The crane operator doesn't care what's inside. The port doesn't care about the contents. The ship is fully interchangeable.

Docker containers work identically. The runtime (containerd/runc) doesn't care if your container runs Python, Go, or a Postgres database. The orchestrator (Kubernetes) doesn't care about your app. The cloud (ECS, GKE, AKS) doesn't care about your runtime. The standard interface — OCI Image Spec + OCI Runtime Spec — is the interoperability layer, exactly like ISO 668 for shipping.

§ 03 — IMAGE ANATOMYWhat's actually in an OCI image

An image is a stack of immutable, content-addressed filesystem layers plus a JSON config. The OCI Image Spec formalizes what Docker invented. Understanding the anatomy is the prerequisite for understanding the registry, the runtime, and security.

OverlayFS: the union filesystem

When you run a container, Linux mounts a union filesystem using OverlayFS. OverlayFS takes multiple directory trees ("lower layers") and presents them as a single unified view. Each image layer is a tar archive of filesystem changes. Layers are stacked bottom-up. A container gets a writable "upper" layer on top — and copy-on-write means files from lower layers are only copied to the writable layer when modified.

The OCI Image Spec — three artifacts

MANIFEST

A JSON document that lists the image config blob (SHA256) and all layer blobs (SHA256) with their sizes. This is what a registry returns when you request an image by tag. It is the "table of contents" of the image. For multi-arch, an image index (manifest list) contains multiple manifests, one per platform.

CONFIG

A JSON blob containing environment variables, entrypoint, working directory, exposed ports, and the history of commands used to build the image. The config SHA256 is the image ID you see in docker images. It does not describe what the image does — it describes how to run it.

LAYERS

Each layer is a gzip-compressed tar archive of filesystem changes (adds, modifies, deletes as whiteout files). Layers are content-addressed by the SHA256 of the compressed blob. The same Ubuntu base layer — pulled once — is shared by every image that uses it. This is where the storage efficiency comes from.

# OCI Manifest (simplified)
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:abc123...",
    "size": 7023
  },
  "layers": [
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:1a2b3c...", "size": 30428672 },   // ubuntu base
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:8b1f2e...", "size": 45678901 },   // python runtime
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:3c8a9d...", "size": 12345678 },   // app deps
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f9e2a1...", "size": 234567  }     // app code
  ]
}

# The digest is the SHA256 of the compressed layer tar.
# The diffID (in config) is the SHA256 of the uncompressed tar.
# Content-addressable: if sha256 matches, the blob is valid — no trust required.

§ 04 — THE REGISTRYHow Docker Hub and GHCR work

A container registry is a content-addressable blob store plus a metadata index. The OCI Distribution Spec formalizes the API. The interesting engineering is deduplication, CDN caching, multi-arch support, and garbage collection.

The push/pull protocol (OCI Distribution Spec)

Content-addressable storage — why SHA256 is the key

Every blob in a registry is addressed by its SHA256 digest: sha256:1a2b3c.... This has three profound consequences:

DEDUPLICATION

Any two images that share a layer — identical Ubuntu base, identical Python runtime — share exactly one copy of that layer in the blob store. Storage cost is O(unique layers), not O(images × layers).

CONTENT INTEGRITY

Before using a blob, recompute its SHA256. If it doesn't match the manifest's declared digest, the download is corrupt or tampered. No signature required for layer integrity — the hash is the trust anchor.

CACHE-FRIENDLY CDN

Blobs addressed by SHA256 are immutable — the same digest always means the same content. CDN edge nodes can cache blobs indefinitely with no TTL. A cache hit for sha256:ubuntu-base saves 100 MB of origin bandwidth for every pull worldwide.

Multi-arch: manifest lists (OCI image index)

# docker manifest inspect ubuntu:22.04 --verbose
# An OCI Image Index ("fat manifest") points to per-platform manifests

{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    { "digest": "sha256:amd64...", "platform": {"os":"linux","architecture":"amd64"} },
    { "digest": "sha256:arm64...", "platform": {"os":"linux","architecture":"arm64"} },
    { "digest": "sha256:armv7...", "platform": {"os":"linux","architecture":"arm","variant":"v7"} }
  ]
}

# docker pull resolves the correct manifest for the host platform automatically.
# buildx --platform linux/amd64,linux/arm64 builds and pushes all platforms in one push.

§ 05 — CONTAINER NETWORKINGFrom veth pairs to VXLAN overlays

Container networking is Linux networking — namespaces, veth pairs, bridges, iptables NAT. Multi-host adds VXLAN overlay encapsulation. Understanding the primitives is the prerequisite for debugging any networking issue in Kubernetes.

Single-host: bridge networking (docker0)

Network namespaces and veth pairs — the mechanics

NETWORK NAMESPACE

Each container gets its own network stack: interfaces, routing table, iptables rules, sockets — fully isolated. ip netns exec container1 ip addr shows only that container's interfaces. The host namespace and container namespace are completely separate — a port conflict in one is invisible to the other.

VETH PAIR

A virtual Ethernet cable with two ends. One end is placed inside the container namespace (appears as eth0); the other is placed in the host namespace (appears as vethXXXX) and attached to the docker0 bridge. Traffic flows between them — whatever enters one end exits the other. This is the only "wire" between the container and the host network.

DOCKER0 BRIDGE

A virtual switch at 172.17.0.1/16. All container veth pairs attach here. Container-to-container traffic within the same bridge is direct — no NAT, no kernel routing, just a virtual switch forward. The bridge also has an IP on the host, allowing the host to reach containers.

IPTABLES MASQUERADE

For container-to-internet traffic: the kernel NATs the source IP from 172.17.0.x to the host's public IP. For inbound port publishing (-p 8080:80): iptables DNAT rewrites destination from host:8080 to 172.17.0.x:80. This is why docker run -p works — it's just iptables rules.

Multi-host: VXLAN overlay networking

Single-host bridge networking doesn't cross machines. For multi-host clusters (Docker Swarm, Kubernetes), an overlay network encapsulates container frames inside UDP packets using VXLAN (Virtual Extensible LAN). Container 172.17.0.2 on host A communicating with container 172.17.0.3 on host B: the frame is wrapped in a VXLAN UDP packet (destination: host B's physical IP, port 4789) and unwrapped on arrival. Flannel, Calico, Cilium, and Weave all implement variations of this model.

# VXLAN packet structure (simplified)
Outer Ethernet frame:
  src MAC: host-A NIC
  dst MAC: host-B NIC

Outer IP header:
  src: 10.0.0.1  (host A physical IP)
  dst: 10.0.0.2  (host B physical IP)

UDP header:
  dst port: 4789 (VXLAN)

VXLAN header:
  VNI: 100  (virtual network identifier — which overlay network)

Inner Ethernet frame:
  src MAC: veth in container-A namespace
  dst MAC: veth in container-B namespace

Inner IP:
  src: 10.244.1.2  (pod/container IP on host A)
  dst: 10.244.2.3  (pod/container IP on host B)

# The kernel's VXLAN driver on host B strips the outer headers and
# delivers the inner frame to the correct container namespace.
# To the container, it looks like a normal Ethernet packet.

§ 06 — SCHEMA DESIGNThe registry database

The interesting design is layer deduplication (blobs are shared across images), garbage collection (GC blobs only when no manifest references them), quota enforcement per repository, and pull event analytics. Each table serves a distinct operational requirement.

-- Repositories (e.g. library/ubuntu, ghcr.io/org/app)
CREATE TABLE repository (
    repo_id         BIGSERIAL PRIMARY KEY,
    registry        TEXT NOT NULL,              -- 'docker.io', 'ghcr.io', 'gcr.io'
    namespace        TEXT NOT NULL,             -- 'library', 'myorg'
    name             TEXT NOT NULL,             -- 'ubuntu', 'myapp'
    is_public        BOOLEAN NOT NULL DEFAULT TRUE,
    owner_id         BIGINT NOT NULL,
    storage_bytes    BIGINT NOT NULL DEFAULT 0, -- updated by trigger on blob insert
    quota_bytes      BIGINT,                    -- NULL = unlimited
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (registry, namespace, name)
);

-- Content-addressed blobs — shared across ALL images/repositories
CREATE TABLE blob (
    blob_id          BIGSERIAL PRIMARY KEY,
    digest           TEXT NOT NULL UNIQUE,      -- sha256:hex — THE primary key conceptually
    media_type       TEXT NOT NULL,             -- 'application/vnd.oci.image.layer.v1.tar+gzip' etc.
    size_bytes       BIGINT NOT NULL,
    storage_path     TEXT NOT NULL,             -- s3://blobs/{digest[0:2]}/{digest[2:4]}/{digest}
    uploaded_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    last_referenced_at TIMESTAMPTZ,             -- updated on pull; used by GC
    gc_eligible      BOOLEAN NOT NULL DEFAULT FALSE  -- set true when ref_count → 0
);

-- OCI Manifests (one per image+platform)
CREATE TABLE manifest (
    manifest_id      BIGSERIAL PRIMARY KEY,
    repo_id          BIGINT NOT NULL REFERENCES repository(repo_id),
    digest           TEXT NOT NULL,             -- sha256 of the manifest JSON
    media_type       TEXT NOT NULL,             -- OCI manifest or manifest list
    raw_json         JSONB NOT NULL,            -- full manifest content
    config_digest    TEXT REFERENCES blob(digest),  -- null for manifest lists
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (repo_id, digest)
);

-- Manifest ↔ blob association (many-to-many — core deduplication)
CREATE TABLE manifest_blob (
    manifest_id      BIGINT NOT NULL REFERENCES manifest(manifest_id),
    blob_id          BIGINT NOT NULL REFERENCES blob(blob_id),
    layer_order      INT NOT NULL,              -- 0-based order in manifest layers array
    PRIMARY KEY (manifest_id, blob_id)
);

-- Tags (mutable pointers to manifests)
CREATE TABLE tag (
    tag_id           BIGSERIAL PRIMARY KEY,
    repo_id          BIGINT NOT NULL REFERENCES repository(repo_id),
    name             TEXT NOT NULL,             -- 'latest', 'v1.2.3', 'main'
    manifest_id      BIGINT NOT NULL REFERENCES manifest(manifest_id),
    pushed_at        TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    pushed_by        TEXT NOT NULL,
    UNIQUE (repo_id, name)
);

-- Image index (multi-arch / manifest list → per-platform manifests)
CREATE TABLE image_index_entry (
    parent_manifest_id BIGINT NOT NULL REFERENCES manifest(manifest_id),
    child_manifest_id  BIGINT NOT NULL REFERENCES manifest(manifest_id),
    platform_os        TEXT NOT NULL,           -- 'linux'
    platform_arch      TEXT NOT NULL,           -- 'amd64', 'arm64', 'arm'
    platform_variant   TEXT,                    -- 'v7' for arm/v7
    PRIMARY KEY (parent_manifest_id, child_manifest_id)
);

-- Pull events (analytics + rate limiting)
CREATE TABLE pull_event (
    event_id         BIGSERIAL PRIMARY KEY,
    repo_id          BIGINT NOT NULL REFERENCES repository(repo_id),
    manifest_id      BIGINT REFERENCES manifest(manifest_id),
    tag_name         TEXT,
    puller_id        BIGINT,                    -- authenticated user / org, null for anon
    client_ip        INET NOT NULL,
    pulled_at        TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    bytes_transferred BIGINT NOT NULL DEFAULT 0,
    was_cache_hit    BOOLEAN NOT NULL DEFAULT FALSE
) PARTITION BY RANGE (pulled_at);  -- monthly partitions; archive to S3 after 90 days

CREATE TABLE pull_event_2026_01 PARTITION OF pull_event
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

-- Indexes
CREATE INDEX idx_blob_digest         ON blob (digest);
CREATE INDEX idx_manifest_repo       ON manifest (repo_id, created_at DESC);
CREATE INDEX idx_tag_repo_name       ON tag (repo_id, name);
CREATE INDEX idx_manifest_blob_blob  ON manifest_blob (blob_id);  -- for GC ref-count
CREATE INDEX idx_pull_event_repo     ON pull_event (repo_id, pulled_at DESC);
CREATE INDEX idx_blob_gc             ON blob (gc_eligible, last_referenced_at)
    WHERE gc_eligible = TRUE;

-- Garbage collection query: find blobs with no manifest references
-- Run periodically after tag deletes / manifest deletes
UPDATE blob SET gc_eligible = TRUE
WHERE blob_id NOT IN (SELECT DISTINCT blob_id FROM manifest_blob)
  AND uploaded_at < NOW() - INTERVAL '7 days';  -- soft-delete grace period

Quota enforcement: storage_bytes on repository is updated by a trigger on manifest_blob insert. Before accepting a push, check storage_bytes + new_layer_sizes <= quota_bytes. Deduplication means shared layers don't count against individual repo quotas — only layers unique to that repo.

§ 07 — RUNTIME INTERNALSWhat happens when you run docker run

Eight steps from command to running process. Each step is a distinct subsystem: registry pull, layer unpacking, rootfs construction, namespace setup, cgroup limits, and exec. runc is the last mile — it executes exactly one process.

runc, containerd, CRI — the component stack

§ 08 — SECURITYImage scanning, content trust, and container hardening

Container security has two distinct layers: image security (what's in the image before it runs) and runtime security (what a running container can do). Both require explicit design decisions — the defaults are permissive.

Image scanning: Trivy and Grype

Image scanning tools (Trivy, Grype, Snyk Container) inspect the image's layer contents for known CVEs in OS packages (apt/yum), language runtimes (pip, npm, maven), and application dependencies. They do not run the image — they parse the filesystem statically.

Content trust: Notary v2 / cosign (Sigstore)

An image tag (nginx:latest) is mutable — someone can push a new image and overwrite the tag. Content trust creates a cryptographic signature over a manifest digest, stored alongside the image in the registry. The consumer verifies the signature before pulling.

# cosign (Sigstore) — keyless signing via OIDC (no long-lived key)
# Sign in CI with GitHub Actions OIDC token:
cosign sign --yes ghcr.io/myorg/myapp@sha256:abc123...
# cosign uploads a signature as a separate OCI artifact in the registry

# Verify before pull (in production admission controller):
cosign verify --certificate-identity=https://github.com/myorg/myapp/.github/workflows/build.yaml \
              --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
              ghcr.io/myorg/myapp:latest

# Notary v2 (nv2) — policy-based: only trusted images run
# OPA Gatekeeper or Kyverno enforces policy at admission:
# "any Pod spec.containers[].image must have a valid cosign signature"

Runtime hardening — the defense-in-depth checklist

§ 09 — SCALEServing 10 billion pulls per day

Docker Hub serves ~10B pulls/day. GitHub Container Registry, GCR, and ECR handle similar scale. The architecture has three layers: CDN edge caching, P2P distribution for large-scale cluster pulls, and lazy loading for eliminating unnecessary data transfer entirely.

Layer 1: CDN edge caching

Blobs are addressed by SHA256 digest — they are immutable and can be cached indefinitely. The registry redirects blob GETs to a CDN presigned URL (S3/GCS + CloudFront/Fastly). Cache hit rates for popular base images (ubuntu, alpine, python, node) exceed 99%. This means the origin registry handles only cache misses — roughly 1% of 10B = 100M requests/day.

# Registry blob GET → 307 redirect to CDN URL
GET /v2/library/ubuntu/blobs/sha256:1a2b3c...
→ 307 Location: https://cdn.hub.docker.com/v2/blobs/sha256:1a2b3c...?X-Amz-Expires=3600&X-Amz-Signature=...

# CDN cache key = sha256 digest (immutable — no TTL needed)
# Popular blobs (ubuntu, alpine base layers) served from CDN PoP nearest to client
# Cold miss → CDN fetches from S3, caches permanently

Layer 2: P2P distribution (Dragonfly / Kraken)

When a Kubernetes cluster autoscales from 10 pods to 1000 pods simultaneously, each pod tries to pull the same image. Without P2P, 1000 nodes all hit the CDN/registry simultaneously. With P2P distribution (Alibaba's Dragonfly or Uber's Kraken), nodes form a BitTorrent-like swarm: each node that has pulled a chunk seeds it to peers. The registry/CDN serves only the initial seeder, and the swarm handles the fan-out.

Layer 3: Lazy loading — eStargz and streaming pulls

The fundamental insight: most containers start after pulling 30% of their layers, because only the files accessed during startup are needed immediately. eStargz (CNCF Stargz Snapshotter) reformats image layers so individual files are addressable and fetchable on-demand. Container startup begins before the image is fully downloaded.

# eStargz: Seekable GZIP layer format
# Each file within the layer is independently seekable
# Manifest includes a "stargz" footer with a TOC (table of contents)

# Startup latency comparison (Node.js app, 500 MB image):
# Traditional pull:  download 500 MB → unpack → start    ≈ 60 s on cold node
# eStargz lazy:      download TOC (1 MB) → start → fetch on demand  ≈ 3 s

# Enable in containerd config.toml:
[proxy_plugins]
  [proxy_plugins.stargz]
    type = "snapshot"
    address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"

# Build eStargz-compatible image:
ctr-remote image optimize --oci \
  nginx:alpine \
  ghcr.io/myorg/nginx:alpine-sgz

Registry HA architecture

§ 10 — Q&AFifteen questions the loop actually asks

These separate the staff-level answer from the senior one — layer caching, BuildKit, distroless images, and container escapes.

§ 11 — SUMMARYWhat the strong answer looks like

← Back to Design Scenarios

§ 12 — COMMON MISTAKESWhat candidates get wrong

Three misconceptions that show up repeatedly in interviews — and how to correct them on the spot.

§ 13 — WHY NOT?Containers are not always the answer

The senior answer includes knowing when NOT to use containers. Trade-offs matter more than defaults.

§ 14 — ONE-MINUTE ANSWERThe answer that ends the follow-up

Every interviewer will ask a variant of this. Have the answer internalized, not memorized.

FOLLOW-UP READY: If they ask "what problem does Kubernetes solve?" — scheduling, self-healing, rolling deployments, service discovery, autoscaling. See The Orchestration Problem.

§ 15 — INTERVIEWER'S MINDWhat they are actually testing

The question is about containers. The test is about depth of systems thinking. Four axes that separate junior from staff.

§ 16 — THE EVOLUTIONContainer infrastructure in context

Containers are one milestone in a longer arc. Knowing where they sit — and what came before and after — is the context that makes all other answers land.

§ 17 — WHAT'S NEXT?The problems containers created — and what solves them

Each generation of infrastructure solves one problem and creates the next. Understanding this chain is what makes a strong candidate's answer feel historically grounded rather than tool-focused.

The through-line: every generation of infrastructure abstracts away the complexity of the previous one. Physical servers → VMs → cloud → containers → Kubernetes → platforms → agents. Understanding this arc — not just the current tool — is what a staff engineer brings to the interview.