System Design — Data Platforms.

The 5-minute opening — frame the system in one diagram.

Most candidates fail the data-platform whiteboard in the first 90 seconds — they jump to "use Spark" before establishing the shape of the system. The opening isn't tools; it's the data flow with grain hints. Draw this, then layer in technology choices.

The canonical 5-box diagram

Five rectangles. Five steady-state arrows. Three control-plane services underneath (registry, catalog, lineage). That's the entire opening — repeated for any data-platform prompt: ride-share, ads, payments, e-commerce. The contents of each box change; the shape doesn't.

The five questions to ask BEFORE drawing

The grain-first articulation

After the diagram, say one sentence per box that pins the grain — the level of detail at each stage:

• Sources — "Each source emits events at its own grain. Mobile SDK is per-action; OLTP CDC is per-row-change."
• Ingest — "Kafka topics are partitioned by entity_id; one event = one Kafka message."
• Storage — "Raw zone keeps every event forever; curated zone is per-entity-per-hour rollups."
• Transform — "Star schema in the warehouse — dim_user, fact_event at user-event grain."
• Serve — "BI queries hit per-user-per-day aggregates; ML serving hits per-user-realtime feature store."

Junior engineers say "Spark transforms data". Senior engineers say "the transform layer aggregates from per-event to per-user-per-day grain, with both grains co-existing for different consumer needs."

Lakehouse vs warehouse vs lake — pick the right primitive.

The 2024 reality: most companies are mid-migration from warehouse-only or lake-only to lakehouse. Knowing why each exists and where each wins is the difference between a recommendation and a religious war.

The four flavors

Lakehouse — what makes it different

The lakehouse isn't a product — it's a file format with transactional metadata layered on top of cheap object storage. Three open-source contenders: Apache Iceberg, Delta Lake, Apache Hudi. All three solve the same problems:

Iceberg vs Delta vs Hudi — when to pick which

The 2024 momentum: Iceberg is winning the multi-engine race; Snowflake, BigQuery, and Databricks all support it. Delta is sticky if you're on Databricks. Hudi remains the choice for high-volume streaming upserts.

When NOT to use a lakehouse

The medallion architecture (bronze / silver / gold)

Most lakehouse implementations follow a three-tier "medallion" layering:

Drop the term in the interview: "raw events land in bronze, dbt cleans them into silver with conformed dims, gold is the per-team aggregation marts."

Batch vs streaming vs Lambda vs Kappa — the quadrant.

The interview's hardest moment: "how would you architect this — batch or streaming?". The honest answer is both, with a clear seam. Lambda and Kappa are the two ways to draw the seam.

The four cases

Lambda — the dual-path architecture

Kappa — the single-stream architecture

Kappa eliminates the dual codebase. The stream is the source of truth; batch is just the same job replayed against historical Kafka offsets. Three preconditions to make Kappa work:

1. Durable, retentioned event log — Kafka with infinite retention OR tiered storage (Confluent Cloud, Pulsar)
2. Idempotent / deterministic stream jobs — replay must produce identical output
3. Replay-friendly state backend — RocksDB checkpoints in Flink; Kafka Streams state stores

When to pick which

The 2024+ default — Kappa-leaning hybrid

Most modern platforms aren't pure either. The shape that's winning:

• Kafka as the spine (event-source-of-truth)
• Flink jobs writing to Iceberg in near-real-time (sub-minute)
• "Batch" jobs are dbt incremental models that read the same Iceberg tables
• One transform logic; two query paths (Flink for low-latency views, dbt for analyst SQL)

dbt + Airflow patterns — orchestration that survives audit.

Airflow runs the schedule. dbt builds the SQL. Together they're the most common transformation stack in 2024. The interview question is rarely "what tool" — it's "how do they integrate so a failed step doesn't corrupt the warehouse".

The division of labor

The canonical dbt model lifecycle

One dbt model goes through this lifecycle on every run:

1. Source check     ─▶  dbt source freshness   (is upstream data fresh?)
2. Build            ─▶  dbt run --models +my_model  (run model + upstream deps)
3. Test             ─▶  dbt test --models my_model  (assertions: not_null, unique, accepted_values, custom)
4. Docs / lineage   ─▶  dbt docs generate              (refresh DAG + descriptions)
5. Notify           ─▶  Slack / PagerDuty on failure   (Airflow on_failure_callback)

Materializations — pick the right one

The incremental pattern that survives backfill

{{ config(
    materialized='incremental',
    unique_key='event_id',
    on_schema_change='append_new_columns',
    incremental_strategy='merge'
) }}

SELECT event_id, user_id, event_type, event_at
FROM {{ source('raw', 'events') }}

{% if is_incremental() %}
  WHERE event_at > (SELECT MAX(event_at) FROM {{ this }})
{% endif %}

Two failure modes to anticipate:

• Late-arriving rows — events with event_at earlier than current MAX get skipped. Fix: add a lookback (event_at > MAX(event_at) - INTERVAL '24 hours') and let MERGE deduplicate.
• Reprocessing — the --full-refresh flag rebuilds from scratch. Required when business logic changes; protected by branch-CI gating.

Airflow patterns — the four every senior knows

The "audit-safe" pattern

Compliance-grade pipelines need to prove what ran, when, with what version, and what the input was. The pattern:

1. Versioned dbt manifest — every run captures the dbt project version + git SHA
2. Run logs in a database — Airflow's metadata DB plus dbt's run_results.json uploaded to S3 per run
3. Lineage snapshot — OpenLineage events per task to a central catalog
4. Reproducibility test — quarterly job that rebuilds last quarter's gold table from raw + git SHA, asserts byte-equality

Schema registry & contract-first pipelines.

Three things break a data platform: schema changes, schema changes, and schema changes. A field renamed upstream silently breaks 40 downstream dashboards two weeks later. Schema registry is the contract layer that prevents this.

What a schema registry actually is

A schema registry stores the structure (Avro / Protobuf / JSON Schema) of every Kafka topic. Producers register their schema; consumers fetch it; the registry enforces compatibility rules on every change.

The four compatibility modes

Default to backward for fact streams (consumers are usually dashboards / analytics that lag producers). Default to full for multi-team event APIs.

Allowed schema changes (Avro example)

Renames and type changes are never compatible. The senior pattern: "add the new field, dual-write for a deprecation window, switch consumers, drop the old field."

Contract-first development — the workflow

1. Producer team writes Avro schema in a shared repo (typically a "schemas" monorepo)
2. CI validates compatibility against the registered schema — PR fails if break
3. Schema published on merge to schema registry; producer code generated from it
4. Consumer teams discover schemas via registry browser; generate their own client code
5. Runtime enforcement — Kafka brokers reject messages that don't match registered schema

The tombstone / deprecation pattern

// Schema v1
{ "type": "record", "name": "User", "fields": [
  { "name": "user_id", "type": "long" },
  { "name": "name",    "type": "string" }
]}

// Schema v2 — adding email, deprecating name
{ "type": "record", "name": "User", "fields": [
  { "name": "user_id", "type": "long" },
  { "name": "name",    "type": "string", "deprecated": true },
  { "name": "email",   "type": ["null", "string"], "default": null },
  { "name": "first_name", "type": ["null", "string"], "default": null },
  { "name": "last_name",  "type": ["null", "string"], "default": null }
]}

// 90 days later, schema v3 — drop name
{ "type": "record", "name": "User", "fields": [
  { "name": "user_id", "type": "long" },
  { "name": "first_name", "type": ["null", "string"], "default": null },
  { "name": "last_name",  "type": ["null", "string"], "default": null }
]}

Schema in the warehouse — dbt contracts

dbt 1.5+ added contract enforcement at the model level — runtime validation that the output table matches a declared schema:

models:
  - name: dim_user
    config:
      contract:
        enforced: true
    columns:
      - name: user_id
        data_type: bigint
        constraints: [{ type: not_null }, { type: primary_key }]
      - name: email
        data_type: varchar
        constraints: [{ type: not_null }]

The model fails to build if the output schema drifts. Combined with downstream tests, this is the warehouse-side equivalent of schema registry.

The 90-second articulation script — works on any prompt.

You've drawn the diagram, named the technologies, called out the trade-offs. The closing 90 seconds bring it together as one coherent design. Memorize this template; substitute the specific tools and grain.

Three sentences that signal seniority — in any round

1. "I'd start with the grain question — what's the smallest unit of fact, and what's the consumer grain?"
2. "The contract layer (schema registry + dbt contracts) does more for reliability than the orchestration choice."
3. "Lambda is what you choose when the team can't yet replay confidently; Kappa is the default when they can."