Design an analytics platform for a contact-center serving 40+ enterprise tenants. Streaming IVR event logs arrive as nested, schema-drifting JSON with ten record types. Dashboards span 16 months, some must be five minutes fresh, and under no circumstances may one tenant ever see another's data. A complete worked answer: ingestion, transformation, storage and serving, the speed layer, structural tenant isolation, hot/cold tiering, scale-out, sizing — and a reasoned AWS vs Microsoft Fabric recommendation.
Tenant isolation is structural, not policy-only — and the medallion lake absorbs schema-drifting nested JSON rather than fighting it, so onboarding tenant #41 is config, not code.
Policy-only isolation — one query bug and tenant A sees tenant B's calls; isolation must be structural. And hard-coding today's JSON schema, which breaks the moment a record type changes.
"You are designing an analytics platform for a contact-center serving 40+ enterprise tenants, with more onboarding over time. Source data is streaming IVR event logs delivered as JSON files. A single file contains multiple record types — call, transfer, prompt, intent, checkpoint, system, interaction, and others. The JSON has nested arrays, and the schema is not stable: fields appear, disappear, and change shape over time. A mid-size tenant produces 30,000–45,000 calls per day. Design the end-to-end architecture, and tell me whether you'd build it on AWS or Microsoft Fabric — and why."
Every requirement in this prompt is a trap that separates a memorized answer from an architect's. The interviewer isn't testing whether you know Spark — they're testing whether you can hold seven competing constraints in your head at once and make defensible trade-offs. Here are the seven that matter.
| The requirement | What it's really testing |
|---|---|
| Drifting, nested JSON · 10 record types | Do you fight schema drift or absorb it? Schema-on-read vs brittle ETL. |
| 16-month window, rarely-queried tail | Hot/cold tiering and retention — not "keep everything in the warehouse." |
| Trends + drill-down to individual calls | Pre-aggregation AND raw retrieval — two serving shapes, not one. |
| ≤5-minute freshness on a subset | A speed layer beside the batch layer — lambda architecture, scoped. |
| Strict tenant isolation, no single misconfig | Structural (not policy-only) isolation. Defense in depth. |
| Per-tenant time zones | Store UTC, present local. Never bake local time into storage. |
| 40+ tenants, growing | Onboarding must be config-driven, not a code change per tenant. |
Before drawing boxes, size the problem — it decides whether this is a "big data" build or not. A mid-size tenant does ~40K calls/day. Each call fans out into many events (prompts, transfers, intents, checkpoints) — call it 15–30 events per call.
Stating this out loud is the single highest-signal move in the interview. It tells the interviewer you right-size infrastructure instead of reaching for a 200-node cluster because the word "streaming" appeared. The largest tenants may be 5–10× a mid-size one — hold that thought for the storage-skew discussion in §06.
The backbone is a medallion (bronze → silver → gold) lake with a parallel speed layer for the five-minute dashboards. One diagram carries the whole answer.
The gold rollups answer trend questions cheaply; the warehouse (or a lake-query engine over silver) answers drill-down and export; the speed layer answers only the freshness-critical dashboards. Three serving needs, three purpose-built paths — instead of forcing one engine to do all three badly.
The killer detail is that fields appear, disappear, and change shape, and one file mixes ten record types. If you design a rigid typed schema up front, you will be paged every time a tenant's vendor ships a new IVR build. The answer is schema-on-read at the boundary.
tenant_id / record_type / ingest_date. Never parse-and-reject at the door. If a record can't be interpreted, it still lands — you can reprocess history later because the raw truth is preserved.Split by record type early. A router (the stream consumer, or the first bronze job) reads the type discriminator on each record and fans the ten types into ten separate bronze tables/paths. Each type then evolves independently — a new field on intent events never destabilizes call events.
Handle drift with additive schema evolution. Use a table format that supports it — Delta Lake or Apache Iceberg — with mergeSchema-style behavior: new fields are added as nullable columns, old rows keep NULL. Nested arrays are either kept as a variant/JSON column and exploded on read, or flattened into child tables keyed by call_id. Keep a raw _payload column on silver so a field that "disappears" from the typed schema is still recoverable.
Contract tests, not contract enforcement. Run lightweight expectations (row counts, null-rate spikes, new-field detection) and route violations to a quarantine path with an alert — the pipeline keeps running. A schema change becomes a notification, not an outage.
The interviewer explicitly asks for the engine "and what it costs you." Name a default, justify it, and be honest about the tax.
| Engine | Why choose it | What it costs you |
|---|---|---|
| Spark (Databricks / EMR / Fabric) | Handles nested JSON, schema evolution, huge back-fills; one engine for batch + structured streaming | Cluster cost + tuning; overkill for the smallest tenants; cold-start latency |
| SQL-in-warehouse (dbt on Snowflake/BigQuery) | Simple, cheap for modest volume, analysts can own it | Weak at deeply nested/variant JSON and true streaming; drift handling is clumsier |
| Flink | Best-in-class low-latency streaming for the speed layer | Operationally heavy; a second paradigm to run alongside batch |
Users need summary trends (hourly/daily), drill from day → hour → individual call, and export raw records. That's pre-aggregation for trends and raw retrieval for drill-down — a single table can't serve both well.
Gold, pre-aggregated: hourly and daily rollups per tenant (calls, transfers, intents, handle-time percentiles). Small, fast, cheap — this powers 95% of dashboard loads. Silver, raw-but-typed: partitioned by tenant_id / event_date / hour so a drill-down to a specific hour prunes to a handful of files, and export is a bounded scan.
OPTIMIZE. For the very largest tenants, the tenant-first layout means their heavy scans touch only their own files, and you can give them dedicated compute without moving data.Only a small subset of dashboards needs ≤5-minute data. Don't make the whole pipeline real-time to satisfy a corner of it — that's the classic over-engineering tell. Run a scoped speed layer.
The same ingest stream forks: one branch feeds the batch medallion, the other feeds a streaming aggregator (Spark Structured Streaming with a 1–2 minute trigger, or Flink) that maintains rolling per-tenant counters and writes them to a low-latency store — the warehouse's streaming-ingest table, or a serving KV/OLAP store. The near-real-time dashboards read only these live aggregates. Batch remains the system of record; the speed layer is disposable and rebuildable.
The hardest requirement: "no single misconfiguration could cause a leak." That word single rules out relying on one row-level-security predicate. You need defense in depth — several independent layers, each of which alone would prevent a leak.
tenant_id used to filter every query comes from the authenticated session/token — never from a URL parameter or request body a client could tamper with. Combine that with per-tenant storage prefixes and per-tenant credentials, and a leak needs multiple independent failures.Time zones fit here too: store every timestamp in UTC, tag each tenant with its zone in config, and convert to local time only at query/presentation time. Never write local time into storage — it makes the 16-month history un-reinterpretable and breaks cross-tenant rollups.
Dashboards cover 16 months but the tail is rarely queried. Keep recent data hot and cheap-to-query; age the rest into cold object storage that stays retrievable but costs a fraction.
The gold rollups are tiny and can stay hot for the full 16 months — trend queries over the whole window read pre-aggregated numbers, not raw events. Only raw drill-down/export against old months pays the warm/cold latency, which matches "rarely queried."
40+ tenants today, more tomorrow. If adding a tenant means editing pipeline code, you've built a liability. Make onboarding a metadata operation.
A tenant registry (a config table) holds each tenant's id, source location, time zone, SLA tier (does it need the speed layer?), and retention. The pipelines are tenant-agnostic and driven by this registry: a new tenant is a new row plus provisioned storage prefix and scoped credentials — no deploy. This is also the natural place to encode the pooled vs siloed decision: most tenants share pooled compute with tenant-first partitioning; the largest or most sensitive get a dedicated (siloed) path from the same codebase.
The prompt asks how you'd size infrastructure and validate the design before committing. Anchor to the §02 numbers and de-risk with a proof of concept.
Both can build this. The honest answer names the deciding factor — team and ecosystem — rather than declaring one universally "better."
| Dimension | AWS | Microsoft Fabric |
|---|---|---|
| Ingestion | Kinesis / MSK → S3; Glue or EMR/Databricks | Event Streams → OneLake; Spark notebooks built in |
| Lake + table format | S3 + Iceberg/Delta, engine-agnostic | OneLake + Delta, unified by default |
| Transform | EMR/Glue Spark, or Databricks | Fabric Spark (managed, less tuning) |
| Serving + BI | Redshift/Athena + QuickSight or 3rd-party | Warehouse + Power BI, tightly integrated |
| Tenant isolation | Mature, granular IAM + S3 prefixes/buckets | Workspaces + OneLake security, newer |
| Freshness | Kinesis + Flink/Spark, proven at scale | Real-Time Intelligence (KQL), very strong for this |
Default to AWS for this system. The two hardest requirements — structural tenant isolation with no single point of failure, and proven multi-tenant scale — are exactly where AWS's mature, granular IAM and battle-tested S3/streaming primitives shine. You get engine-agnostic storage (Iceberg/Delta on S3), independent scaling of ingest, transform and serving, and the finest-grained isolation controls available.
Choose Microsoft Fabric if the organization is already Microsoft/Power BI-centric and the team is small. Fabric collapses ingest → OneLake → transform → Power BI into one governed SaaS surface with far less glue code, and its Real-Time Intelligence tier handles the 5-minute requirement elegantly. The trade-off you name: less granular control and a younger multi-tenant isolation story — acceptable when time-to-value and a unified BI experience outweigh maximum control.
The senior move is to make the recommendation conditional on the team and existing stack, not to memorize "AWS wins." Say which factor would flip your answer — that's what a Data Architect is paid for.
"I'd land the raw JSON immutably in a bronze lake partitioned by tenant, record type and date — schema-on-read so drift never breaks ingestion. Spark structured streaming promotes it to typed silver (one path per record type, additive schema evolution on Iceberg or Delta) and pre-aggregated gold rollups for trends, while raw silver stays partitioned tenant-first for fast drill-down and export. A scoped speed layer feeds only the ≤5-minute dashboards and is reconciled by batch. Isolation is structural and layered: per-tenant storage prefixes, per-tenant scoped credentials, row-level security keyed off the auth token — never the request — and CI tests that try to cross tenants and must fail. Timestamps stored in UTC, presented per-tenant local. Hot data in the warehouse, warm on object storage, cold in an archive tier, all still retrievable. Onboarding is a config row, not a deploy. I'd build it on AWS for the granular IAM and proven multi-tenant scale — unless the shop is Microsoft-and-Power-BI-first with a small team, where Fabric's unified surface and Real-Time Intelligence win."