PaddySpeaks · Systems at the Whiteboard · Nº 38

The Multi-Tenant IVR Problem

Design an analytics platform for a contact-center serving 40+ enterprise tenants. Streaming IVR event logs arrive as nested, schema-drifting JSON with ten record types. Dashboards span 16 months, some must be five minutes fresh, and under no circumstances may one tenant ever see another's data. A complete worked answer: ingestion, transformation, storage and serving, the speed layer, structural tenant isolation, hot/cold tiering, scale-out, sizing — and a reasoned AWS vs Microsoft Fabric recommendation.

◆ At a glance the 20-second version — full walk-through below

The senior move

Tenant isolation is structural, not policy-only — and the medallion lake absorbs schema-drifting nested JSON rather than fighting it, so onboarding tenant #41 is config, not code.

The layers

  • Bronze/Silver/Gold medallion over raw JSON
  • Scoped speed layer beside batch (5-min fresh)
  • Hot vs cold tiering across 16 months
  • Per-tenant structural isolation

Key mechanics

  • Absorb schema drift (fields appear/disappear)
  • Speed layer → 5-minute freshness
  • Config-driven tenant onboarding
  • Reasoned AWS vs Microsoft Fabric pick

The trap

Policy-only isolation — one query bug and tenant A sees tenant B's calls; isolation must be structural. And hard-coding today's JSON schema, which breaks the moment a record type changes.

Community question. Shared by a reader after a Data Architect interview at a global contact-center solutions provider. Want to contribute one? Share a question →

§ 01 — THE QUESTIONMulti-tenant IVR analytics platform

Interview Prompt

"You are designing an analytics platform for a contact-center serving 40+ enterprise tenants, with more onboarding over time. Source data is streaming IVR event logs delivered as JSON files. A single file contains multiple record types — call, transfer, prompt, intent, checkpoint, system, interaction, and others. The JSON has nested arrays, and the schema is not stable: fields appear, disappear, and change shape over time. A mid-size tenant produces 30,000–45,000 calls per day. Design the end-to-end architecture, and tell me whether you'd build it on AWS or Microsoft Fabric — and why."

LEVEL · DATA ARCHITECTDURATION · 45–60 MINFORMAT · WHITEBOARD

Every requirement in this prompt is a trap that separates a memorized answer from an architect's. The interviewer isn't testing whether you know Spark — they're testing whether you can hold seven competing constraints in your head at once and make defensible trade-offs. Here are the seven that matter.

The requirementWhat it's really testing
Drifting, nested JSON · 10 record typesDo you fight schema drift or absorb it? Schema-on-read vs brittle ETL.
16-month window, rarely-queried tailHot/cold tiering and retention — not "keep everything in the warehouse."
Trends + drill-down to individual callsPre-aggregation AND raw retrieval — two serving shapes, not one.
≤5-minute freshness on a subsetA speed layer beside the batch layer — lambda architecture, scoped.
Strict tenant isolation, no single misconfigStructural (not policy-only) isolation. Defense in depth.
Per-tenant time zonesStore UTC, present local. Never bake local time into storage.
40+ tenants, growingOnboarding must be config-driven, not a code change per tenant.

§ 02 — DATA SHAPEDo the volume math first

Before drawing boxes, size the problem — it decides whether this is a "big data" build or not. A mid-size tenant does ~40K calls/day. Each call fans out into many events (prompts, transfers, intents, checkpoints) — call it 15–30 events per call.

Back-of-envelope: 40K calls × ~20 events ≈ 800K events/tenant/day. Across 40 tenants ≈ 32M events/day ≈ ~370 events/sec average, bursting to a few thousand/sec at peak. At ~1 KB/event that's ~30 GB/day raw, ~10 TB over 16 months before compression. This is comfortably mid-scale — it does not need exotic infrastructure. The hard parts are schema drift and isolation, not volume.

Stating this out loud is the single highest-signal move in the interview. It tells the interviewer you right-size infrastructure instead of reaching for a 200-node cluster because the word "streaming" appeared. The largest tenants may be 5–10× a mid-size one — hold that thought for the storage-skew discussion in §06.


§ 03 — THE ARCHITECTUREMedallion + a scoped speed layer

The backbone is a medallion (bronze → silver → gold) lake with a parallel speed layer for the five-minute dashboards. One diagram carries the whole answer.

IVR events JSON files per tenant Landing + Ingest autoloader / stream BRONZE raw JSON, as-is SILVER typed, per record type GOLD hourly/daily rollups Warehouse / SQL serving Trend + drill dashboards Near-real-time ≤5-min dashboards SPEED LAYER stream aggregate (scoped tenants)
Fig 1 · Batch backbone (bronze→silver→gold→warehouse) with a parallel, scoped speed layer for ≤5-minute dashboards

The gold rollups answer trend questions cheaply; the warehouse (or a lake-query engine over silver) answers drill-down and export; the speed layer answers only the freshness-critical dashboards. Three serving needs, three purpose-built paths — instead of forcing one engine to do all three badly.


§ 04 — INGESTIONAbsorb schema drift, don't fight it

The killer detail is that fields appear, disappear, and change shape, and one file mixes ten record types. If you design a rigid typed schema up front, you will be paged every time a tenant's vendor ships a new IVR build. The answer is schema-on-read at the boundary.

Bronze = land the raw JSON exactly as received, one immutable copy, partitioned by tenant_id / record_type / ingest_date. Never parse-and-reject at the door. If a record can't be interpreted, it still lands — you can reprocess history later because the raw truth is preserved.

Split by record type early. A router (the stream consumer, or the first bronze job) reads the type discriminator on each record and fans the ten types into ten separate bronze tables/paths. Each type then evolves independently — a new field on intent events never destabilizes call events.

Handle drift with additive schema evolution. Use a table format that supports it — Delta Lake or Apache Iceberg — with mergeSchema-style behavior: new fields are added as nullable columns, old rows keep NULL. Nested arrays are either kept as a variant/JSON column and exploded on read, or flattened into child tables keyed by call_id. Keep a raw _payload column on silver so a field that "disappears" from the typed schema is still recoverable.

Contract tests, not contract enforcement. Run lightweight expectations (row counts, null-rate spikes, new-field detection) and route violations to a quarantine path with an alert — the pipeline keeps running. A schema change becomes a notification, not an outage.


§ 05 — TRANSFORMATION ENGINEThe choice, and what it costs you

The interviewer explicitly asks for the engine "and what it costs you." Name a default, justify it, and be honest about the tax.

EngineWhy choose itWhat it costs you
Spark (Databricks / EMR / Fabric)Handles nested JSON, schema evolution, huge back-fills; one engine for batch + structured streamingCluster cost + tuning; overkill for the smallest tenants; cold-start latency
SQL-in-warehouse (dbt on Snowflake/BigQuery)Simple, cheap for modest volume, analysts can own itWeak at deeply nested/variant JSON and true streaming; drift handling is clumsier
FlinkBest-in-class low-latency streaming for the speed layerOperationally heavy; a second paradigm to run alongside batch
Recommendation: Spark Structured Streaming for bronze→silver→gold because it absorbs nested/drifting JSON natively and unifies batch and streaming in one codebase. The cost you name out loud: you pay for compute and for the expertise to tune it, and you resist the temptation to run it for the speed layer's sub-second needs — for that, a lighter stream aggregator (or Flink) is cheaper and faster.

§ 06 — STORAGE & SERVINGTwo shapes, and the largest tenants

Users need summary trends (hourly/daily), drill from day → hour → individual call, and export raw records. That's pre-aggregation for trends and raw retrieval for drill-down — a single table can't serve both well.

Gold, pre-aggregated: hourly and daily rollups per tenant (calls, transfers, intents, handle-time percentiles). Small, fast, cheap — this powers 95% of dashboard loads. Silver, raw-but-typed: partitioned by tenant_id / event_date / hour so a drill-down to a specific hour prunes to a handful of files, and export is a bounded scan.

The largest-tenant trap: if you partition only by date, a 10× tenant produces 10× bigger partitions and its queries drag on shared compute. Fix it structurally: partition (or cluster) by tenant first, then by date/hour. Compact small files (streaming produces many) on a schedule with Iceberg/Delta OPTIMIZE. For the very largest tenants, the tenant-first layout means their heavy scans touch only their own files, and you can give them dedicated compute without moving data.

§ 07 — 5-MINUTE FRESHNESSA speed layer beside batch

Only a small subset of dashboards needs ≤5-minute data. Don't make the whole pipeline real-time to satisfy a corner of it — that's the classic over-engineering tell. Run a scoped speed layer.

The same ingest stream forks: one branch feeds the batch medallion, the other feeds a streaming aggregator (Spark Structured Streaming with a 1–2 minute trigger, or Flink) that maintains rolling per-tenant counters and writes them to a low-latency store — the warehouse's streaming-ingest table, or a serving KV/OLAP store. The near-real-time dashboards read only these live aggregates. Batch remains the system of record; the speed layer is disposable and rebuildable.

Reconciliation: the speed layer is approximate and gets overwritten by the authoritative batch rollup once it lands. State the tolerance explicitly — "the live tile may be a few percent off for a few minutes, then batch corrects it" — so freshness never silently becomes a correctness claim you can't keep.

§ 08 — TENANT ISOLATIONStructural, not policy-only

The hardest requirement: "no single misconfiguration could cause a leak." That word single rules out relying on one row-level-security predicate. You need defense in depth — several independent layers, each of which alone would prevent a leak.

FOUR INDEPENDENT LAYERS — ANY ONE STOPS A LEAK 1 · STORAGE tenant_id is the top partition / separate prefix or bucket per tenant 2 · IDENTITY per-tenant IAM role scoped to only that tenant's path — no shared read-all role 3 · QUERY row-level security + tenant_id from the auth token, never from the request body 4 · TESTS automated cross-tenant access tests in CI — try to read B as A, assert it fails A leak requires ALL FOUR to be misconfigured at once — no single mistake is sufficient.
Fig 2 · Defense in depth — isolation at storage, identity, query, and test layers
The one rule that prevents most leaks: the tenant_id used to filter every query comes from the authenticated session/token — never from a URL parameter or request body a client could tamper with. Combine that with per-tenant storage prefixes and per-tenant credentials, and a leak needs multiple independent failures.

Time zones fit here too: store every timestamp in UTC, tag each tenant with its zone in config, and convert to local time only at query/presentation time. Never write local time into storage — it makes the 16-month history un-reinterpretable and breaks cross-tenant rollups.


§ 09 — HOT vs COLD16 months, mostly cold

Dashboards cover 16 months but the tail is rarely queried. Keep recent data hot and cheap-to-query; age the rest into cold object storage that stays retrievable but costs a fraction.

HOT · 0–3 months warehouse / cached gold fast dashboards + drill WARM · 3–16 months Iceberg/Delta on object store queried on demand, slower COLD · >16 months archive tier (S3 Glacier / cool) retrievable, rarely touched Lifecycle policies move data down the tiers automatically; raw bronze always retained for reprocessing.
Fig 3 · Age data down the tiers by access pattern, not by deleting it

The gold rollups are tiny and can stay hot for the full 16 months — trend queries over the whole window read pre-aggregated numbers, not raw events. Only raw drill-down/export against old months pays the warm/cold latency, which matches "rarely queried."


§ 10 — SCALE-OUTOnboarding must be config, not code

40+ tenants today, more tomorrow. If adding a tenant means editing pipeline code, you've built a liability. Make onboarding a metadata operation.

A tenant registry (a config table) holds each tenant's id, source location, time zone, SLA tier (does it need the speed layer?), and retention. The pipelines are tenant-agnostic and driven by this registry: a new tenant is a new row plus provisioned storage prefix and scoped credentials — no deploy. This is also the natural place to encode the pooled vs siloed decision: most tenants share pooled compute with tenant-first partitioning; the largest or most sensitive get a dedicated (siloed) path from the same codebase.


§ 11 — SIZING & VALIDATIONProve it before you commit

The prompt asks how you'd size infrastructure and validate the design before committing. Anchor to the §02 numbers and de-risk with a proof of concept.


§ 12 — AWS vs MICROSOFT FABRICThe recommendation

Both can build this. The honest answer names the deciding factor — team and ecosystem — rather than declaring one universally "better."

DimensionAWSMicrosoft Fabric
IngestionKinesis / MSK → S3; Glue or EMR/DatabricksEvent Streams → OneLake; Spark notebooks built in
Lake + table formatS3 + Iceberg/Delta, engine-agnosticOneLake + Delta, unified by default
TransformEMR/Glue Spark, or DatabricksFabric Spark (managed, less tuning)
Serving + BIRedshift/Athena + QuickSight or 3rd-partyWarehouse + Power BI, tightly integrated
Tenant isolationMature, granular IAM + S3 prefixes/bucketsWorkspaces + OneLake security, newer
FreshnessKinesis + Flink/Spark, proven at scaleReal-Time Intelligence (KQL), very strong for this

Recommendation

Default to AWS for this system. The two hardest requirements — structural tenant isolation with no single point of failure, and proven multi-tenant scale — are exactly where AWS's mature, granular IAM and battle-tested S3/streaming primitives shine. You get engine-agnostic storage (Iceberg/Delta on S3), independent scaling of ingest, transform and serving, and the finest-grained isolation controls available.

Choose Microsoft Fabric if the organization is already Microsoft/Power BI-centric and the team is small. Fabric collapses ingest → OneLake → transform → Power BI into one governed SaaS surface with far less glue code, and its Real-Time Intelligence tier handles the 5-minute requirement elegantly. The trade-off you name: less granular control and a younger multi-tenant isolation story — acceptable when time-to-value and a unified BI experience outweigh maximum control.

The senior move is to make the recommendation conditional on the team and existing stack, not to memorize "AWS wins." Say which factor would flip your answer — that's what a Data Architect is paid for.


§ 13 — THE ONE-MINUTE ANSWERIf they cut you off

"I'd land the raw JSON immutably in a bronze lake partitioned by tenant, record type and date — schema-on-read so drift never breaks ingestion. Spark structured streaming promotes it to typed silver (one path per record type, additive schema evolution on Iceberg or Delta) and pre-aggregated gold rollups for trends, while raw silver stays partitioned tenant-first for fast drill-down and export. A scoped speed layer feeds only the ≤5-minute dashboards and is reconciled by batch. Isolation is structural and layered: per-tenant storage prefixes, per-tenant scoped credentials, row-level security keyed off the auth token — never the request — and CI tests that try to cross tenants and must fail. Timestamps stored in UTC, presented per-tenant local. Hot data in the warehouse, warm on object storage, cold in an archive tier, all still retrievable. Onboarding is a config row, not a deploy. I'd build it on AWS for the granular IAM and proven multi-tenant scale — unless the shop is Microsoft-and-Power-BI-first with a small team, where Fabric's unified surface and Real-Time Intelligence win."

~60 SECONDSHITS ALL 8 SUB-QUESTIONSENDS ON A DECISION

§ 14 — COMMON MISTAKESWhat tanks this answer

Rigid schema at ingestion. Designing typed tables the JSON must conform to — then getting paged on every vendor build. Land raw first.
Making everything real-time. Turning the whole pipeline streaming to satisfy a small ≤5-min subset. Scope the speed layer.
Policy-only isolation. Relying on one RLS predicate — a single misconfig leaks. The prompt explicitly forbids this; use defense in depth.
tenant_id from the request. Trusting a client-supplied tenant id instead of the auth token — the most common real-world multi-tenant breach.
Date-only partitioning. Ignoring tenant skew so the biggest tenant degrades everyone. Partition tenant-first.
Local time in storage. Baking per-tenant zones into stored timestamps, making 16 months of history un-reinterpretable.
"AWS is better." Declaring a winner without naming the deciding factor. The recommendation must be conditional and justified.

This was a community submission. Just finished an interview? Share the question that stumped you — anonymous or credited. Share a question →  |  ← Design Index
← paddyspeaks.com