PaddySpeaks · Systems at the Whiteboard · Nº 40

The Architecture Problem

You are a data architect making structural choices from day zero — before code is written, before vendors are locked in, and never on a clean slate. There is always data debt, there are always legacy systems, and the source ecosystem is always messier than the org chart admits. A complete worked answer: the sequential design process and the artifact each step produces, the decision register that converts vague goals into a latency budget, source forensics, one-way versus two-way doors, identity and bitemporal schema design, contracts, the coexistence plan — and the production metrics that would actually make you roll back.

At a glance The three-part answer, compressed

The process

Seven steps, each producing a named artifact. It starts with a decision register, not a requirements document — decisions carry latency and accuracy budgets that you can architect against.

The matrix

Sort every open choice by reversibility. Grain, identity, bitemporality and metric semantics are one-way doors. Orchestrators and BI tools are not. Spend the design budget accordingly.

The metrics

One is non-negotiable: reconciliation variance against the system of record must be zero. The rest are error budgets. Time-to-onboard-a-source is the best proxy for resilience to business change.

The trap

Answering with a stack. The question is about sequencing and trade-offs under ambiguity; naming tools is what candidates reach for when they have no process.

Community question. Shared by a reader after a Data Architecture interview at Citi: "I could answer some but not convincingly when they probed." That last clause is the reason this page exists — the probes are in § 17.

§ 01 — THE QUESTIONAsked as three parts, scored as one

Data Architecture · Hard · Citi

As a data architect, you are making structural choices from day zero, often before code is written or vendors are locked in. I want to understand your exact strategic execution process. Even when designing a new architecture from scratch, there is usually existing data debt, legacy systems, or messy source ecosystems to account for.

The Design Process: What are the sequential steps you take to translate high-level business goals into a conceptual blueprint while navigating existing data constraints?

The Decision Matrix: What foundational rules or trade-offs guide your early choices when deciding on data modeling patterns and the technical stack?

The Evaluation: Once the architecture begins handling production workloads, what specific, non-negotiable metrics do you use to evaluate whether the system is genuinely 'good,' scalable, and resilient against future business shifts?

Three questions, one assessment. The interviewer is not collecting three lists — they are checking whether the same spine runs through all three answers. If your process produces artifacts, your matrix decides using those artifacts, and your metrics measure what the matrix bet on, the answer holds together under probing. If the three parts are independent lists of good practice, the first follow-up will show it.


§ 02 — WHAT IT'S REALLY TESTINGReading the question behind the question

The phrasingWhat it's actually probing
"from day zero, before vendors are locked in"Can you reason without a stack? Candidates who lead with tool names have no process underneath.
"exact strategic execution process"Sequence and artifacts. "Exact" is the tell — they want steps with outputs, not principles.
"there is usually existing data debt"Greenfield is a fiction. They want the coexistence and migration plan treated as part of the design, not a phase-two problem.
"foundational rules"Do you have decision rules that produce an answer, or do you say "it depends" and stop?
"non-negotiable metrics"Which numbers would make you roll back? Most candidates list dashboard metrics that nobody would ever act on.
"resilient against future business shifts"Evolvability, not uptime. Can the architecture absorb a new product line, an acquisition, a regulation?
The single highest-signal move is to say early that you do not design against requirements — you design against decisions. Requirements are a wish list with no budget attached. A decision has an owner, a cadence, a tolerable staleness and a cost of being wrong, and those four facts are what actually determine the architecture. Everything below follows from that.

§ 03 — THE DESIGN PROCESSSeven steps, each with an artifact

A process you can name but not evidence is a philosophy. Every step below ends in a document someone can review, argue with and sign. That is what makes the answer concrete under probing — you can be asked "show me the output of step two" and have something to describe.

THE SPINE — EACH STEP PRODUCES A REVIEWABLE ARTIFACT 1 · Decision inventory not requirements 2 · Source forensics profile, don't ask 3 · Conceptual model vendor-neutral 4 · Reversibility triage where budget goes 5 · Contracts at boundaries schema + SLA + owner 6 · Thin vertical slice → prod one real decision ARTIFACT PRODUCED Decision register latency + accuracy budget per decision Source dossier trust grade per system Conceptual ERD + grain statement per fact Decision matrix one-way vs two-way doors Interface contracts versioned, deprecation policy Proven pattern + measured cost model 7 · Coexistence plan — runs in parallel with every step above, never after them Strangler boundaries · dual-run window · reconciliation gates · decommission criteria for each legacy system the slice invalidates assumptions — the register and the model get rewritten, on purpose The order is load-bearing. Every step consumes the artifact before it. Skip step 1 and step 4 has no budget to triage against; skip step 2 and step 3 models a fiction.

The design spine. Steps 1–6 are sequential; step 7 is continuous. The dashed return path is not failure — the vertical slice exists precisely to break assumptions cheaply.

1 · Decision inventory. Not "what data do you need" — that question returns everything anyone can imagine. Instead: what decisions will this change, who makes them, how often, and what does it cost to be wrong or late? Four columns, and they convert an unbounded wish list into a bounded engineering problem. In practice this is where most of the requested scope evaporates: of forty stated "real-time" needs, two or three usually have a decision behind them that actually changes if the number arrives sooner.

2 · Source forensics. Profile the sources; do not interview people about them. Documentation describes intent, and intent is not what the data does. This is the step that most often changes the architecture, and it is covered on its own in § 05.

3 · Conceptual model. Entities, grain, identity, relationships — deliberately vendor-neutral, deliberately before any stack conversation. The output is a conceptual ERD plus one sentence per fact table stating its grain. If you cannot write the grain sentence, you do not understand the process being modelled yet.

4 · Reversibility triage. The step candidates never mention, and the one that most distinguishes an architect from a senior engineer. Sort every open decision by what it costs to undo. Detailed in § 06.

5 · Contracts at the boundaries. Wherever data crosses a team boundary, publish a contract: schema, semantics, freshness SLA, owner, deprecation policy. Everything inside a boundary stays refactorable; everything crossing one becomes expensive to change. Drawing those boundaries deliberately is the architecture.

6 · Thin vertical slice, in production. One decision from the register, end to end, with real data, real authentication, real users and a real bill. Not a sandbox proof-of-concept — those validate the happy path and none of the things that actually sink programmes: identity, entitlements, network egress, the cost model at volume, and whoever owns the on-call.

7 · Coexistence. The legacy estate does not pause while you build. Treated as an afterthought, it becomes the reason the programme stalls at 70% — see § 13.


§ 04 — THE DECISION REGISTERThe artifact the whole design hangs from

This is the deliverable that makes the rest of the answer defensible. Each row is a real decision a real person makes on a real cadence. The strictest row sets the architecture; every other row rides on infrastructure built for that one.

DecisionOwnerCadenceTolerable stalenessTolerable errorCost of being wrong
Intraday liquidity / funding callTreasuryHourly, market hours15 minZero — must tie to GLRegulatory breach, funding cost
Fraud hold on a transactionFin-crime enginePer eventSub-secondTuned FP/FN rateLoss, or a blocked good customer
Credit line increase offerRiskDaily batch24 hours< 0.5%Credit loss, mis-selling exposure
Regulatory position reportFinanceDaily / quarter-endT+1 closeZero, and reproducibleRestatement, fine, attestation risk
Campaign segment refreshMarketingWeekly7 days~5%Wasted spend
Branch staffing modelOpsMonthly30 days~10%Service level, overtime
What this table buys you in the interview. Three things, and they map exactly onto the three questions. It tells you which decisions justify a streaming path (two of six here — and fraud is an operational system, not the warehouse). It tells you where correctness is absolute rather than statistical, which is what forces bitemporality and reconciliation. And it gives you a defensible answer to "why did you spend money on that?" — because a named owner makes a named decision that a named tolerance requires.

The row that quietly dominates a bank's architecture is the regulatory one, and not because of latency — T+1 is generous. It is the words zero and reproducible. Reproducibility means you must be able to regenerate the report exactly as filed, months later, after the underlying data has been corrected. That single requirement rules out mutable dimension tables and rules in transaction-time history, which is a one-way door decided in week one. That is the kind of chain the interviewer is listening for: a business fact, to a modelling constraint, to a decision you cannot walk back.


§ 05 — SOURCE FORENSICSThe data debt is never where the documentation says

"Account for existing data debt" is the part of the question most candidates gesture at and move past. Be specific instead: here are the pathologies that actually change an architecture, how to detect each one in an afternoon, and what each forces you to build.

PathologyHow you detect itWhat it forces in the design
In-place mutation with no updated_atSnapshot twice a day and diff row hashes; count rows that changed without any timestamp movingIncremental extraction is impossible. Log-based CDC, or full-scan compare — a large, early cost that must be budgeted, not discovered
Soft deletes with an inconsistent flagCOUNT(*) WHERE is_deleted IS NULL — nulls mean the column was added later and history was never backfilledEvery downstream filter is wrong for rows older than the column. Needs an explicit "unknown" state, not a coalesce to false
Recycled natural keysGroup by the key, count distinct open/close date pairs — closed accounts whose numbers are reissuedThe single strongest argument for surrogate keys. A natural key join silently merges two customers years apart
Same party, different id per product siloFuzzy-match a sample across cards, deposits, mortgage; measure the overlap you cannot resolve deterministicallyAn identity resolution layer with its own lifecycle — § 08
Local timestamps with no offsetLook for a missing hour in spring and a duplicated hour in autumnStore UTC plus the original offset. Never store local time alone — DST makes it ambiguous, permanently
Amounts without a currency columnDistribution of magnitudes per source system; a "1,200" that is JPY next to one that is GBPCurrency is part of the grain of every monetary fact. Retrofitting it is a full-history rewrite
Late-arriving backdated correctionsCompare a daily snapshot to the same day re-extracted a month laterYour "immutable" daily fact is not immutable. Either bitemporality or a restatement process — there is no third option
Multi-region duplicates, different collationCase- and accent-insensitive match on name/address across regional instancesDeduplication rules become policy, not code, and need an owner who can arbitrate
Grade each source, and publish the grade. A · reconciles to a system of record, has reliable change capture, an owner who answers. B · usable with known caveats. C · quarantine only — it may feed exploratory work but never a conformed layer or a regulatory number. The grade is not an insult to the source team; it is what lets you publish a number and honestly say what it can be used for. In practice this single artifact prevents more downstream incidents than any amount of pipeline testing.

§ 06 — THE DECISION MATRIXSort by reversibility, not by importance

The second question asks for "foundational rules or trade-offs." The honest organising principle is not a list of dichotomies — it is a single test applied to every open choice: what does it cost to undo this after a year of production data? That test sorts decisions into two piles that deserve completely different amounts of deliberation.

THE TEST — DOES UNDOING IT REQUIRE A HISTORY REWRITE OR A CONSUMER RENEGOTIATION? ONE-WAY DOORS — yes Decide slowly. Write down the reasoning. These are the architecture. Grain of the core factsyou cannot recover detail you never stored Identity and key strategyevery downstream join encodes this choice Bitemporality — keeping transaction timeunrecoverable retroactively; the knowledge is simply gone Semantics of core metricschanging "active customer" invalidates every published trend PII tokenization boundaryre-tokenizing history is a regulated migration of its own Timezone and currency conventionsambiguity here is not fixable after the fact Partitioning of the very large tablesrepartitioning petabytes is a project, not a change TWO-WAY DOORS — no Decide fast, in the slice. Deliberating here is procrastination. Orchestration toola rewrite of DAG definitions, not of data Transformation frameworkthe SQL survives the framework BI / semantic toolpainful, bounded, and no data is lost Compute engine — if storage is open formatthis is exactly why open table formats are worth the tax Cluster sizing and cost tuningcontinuously adjustable; never worth an early argument Naming conventions, folder layoutcheap to rename with lineage in place Batch schedule and cadencea config change, provided the grain supports it

The reversibility matrix. The left column is where an architect earns their title; the right column is where programmes lose months to committee.

The move that reframes the whole answer: "Choosing the warehouse vendor is mostly a two-way door if I keep storage in an open table format — so I decide it quickly and spend the deliberation on grain and identity instead, which are not reversible at any price." That sentence demonstrates you know which choices are load-bearing, and it converts the vendor question from a religious argument into an engineering one.

§ 07 — THE CONCEPTUAL BLUEPRINTLayers, and what changes at each boundary

The question asks for a conceptual blueprint, so draw one — but the value is not the boxes. Every architecture diagram has these boxes. The value is naming what changes at each boundary, and which boundaries carry contracts.

SOURCES Core banking Cards Mortgage CRM General ledger Market data graded A / B / C § 05 LANDING Immutable, raw As-received Partitioned by arrival no business logic ever, on purpose replayable INTEGRATION Conformed, 3NF Bitemporal Identity resolved Surrogate keys Integrity enforced the system of record for meaning § 08 · § 09 · § 10 SERVING Dimensional Wide, denormalized Pre-aggregated Query-shaped rebuildable from integration, always disposable by design CONSUMPTION Regulatory report Risk models Dashboards Feature store Data products C C ◆ = contract boundary: schema + semantics + freshness SLA + named owner + deprecation policy CONTROL PLANE — spans every layer, built in step one, never bolted on Column-level lineage · data quality assertions · reconciliation to system of record · access control and PII tokenization · cost attribution Why two modelling styles rather than one: Integration is normalized because that is where you reconcile and enforce integrity. Serving is dimensional because that is where humans query. Arguing "3NF vs star" as a single global choice is the mistake — they answer different questions at different layers.

The blueprint. Note that serving is explicitly disposable: anything in it must be reproducible from integration, which is what lets you change serving technology without a migration.

Two properties of this blueprint matter more than its shape, and both are worth saying out loud. First, landing is immutable and replayable — every transformation downstream can be rebuilt from it, which converts most future mistakes from incidents into reruns. Second, serving is disposable. If the BI tool changes, or a new consumer needs a different shape, you rebuild serving from integration and nothing is lost. Those two properties together are what make the middle layer worth the effort of getting right.


§ 08 — IDENTITYThe one-way door that decides everything downstream

In a bank, the same human being is a different customer in cards, deposits, mortgage and wealth — with different identifiers, different name spellings and different addresses, each authoritative in its own system. Every meaningful question ("what is our exposure to this client?") depends on resolving that, and the resolution changes over time. This is the hardest schema in the design.

Identity resolution — the cross-reference that outlives every source system
-- The surrogate party key is ours. It never appears in a source system, -- it is never derived from a source id, and it never leaves the platform -- in a form a source system could reuse. Everything joins on this. CREATE TABLE dim_party ( party_key BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, party_type TEXT NOT NULL, -- individual | organisation -- Attributes are deliberately thin here. Anything that varies by -- source system belongs in the xref, not in the golden record. created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); -- The xref is effective-dated because identity DECISIONS change. -- Two parties merge when we learn they are one person; a merge is -- reversed when we learn we were wrong. Neither event may rewrite -- history -- it is recorded as a new assertion. CREATE TABLE party_xref ( xref_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, party_key BIGINT NOT NULL REFERENCES dim_party, source_system TEXT NOT NULL, source_party_id TEXT NOT NULL, -- How we came to believe this, and how strongly. Auditors ask. match_method TEXT NOT NULL, -- deterministic | probabilistic | steward match_score NUMERIC(5,4), matched_on TEXT[], -- ['tax_id','dob','postcode'] steward_id TEXT, -- who approved, when manual -- Business time: when this linkage was true. valid_from TIMESTAMPTZ NOT NULL, valid_to TIMESTAMPTZ NOT NULL DEFAULT 'infinity', -- Knowledge time: when WE learned it. See § 09. recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(), superseded_at TIMESTAMPTZ NOT NULL DEFAULT 'infinity', CONSTRAINT uq_live_link UNIQUE (source_system, source_party_id, valid_from, recorded_at) ); -- A source id may only map to one party at one instant of business time, -- among currently-believed rows. Enforced, not assumed. CREATE UNIQUE INDEX uq_xref_current ON party_xref (source_system, source_party_id) WHERE valid_to = 'infinity' AND superseded_at = 'infinity'; -- Merges and splits are events with a reason, not UPDATEs. -- This table is what you show an auditor who asks "why is this one client?" CREATE TABLE party_merge_log ( event_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, event_type TEXT NOT NULL, -- merge | split | unmerge surviving_key BIGINT NOT NULL REFERENCES dim_party, absorbed_key BIGINT NOT NULL REFERENCES dim_party, reason TEXT NOT NULL, evidence JSONB, actor TEXT NOT NULL, occurred_at TIMESTAMPTZ NOT NULL DEFAULT now() );
Three rules that follow, and that interviewers probe for. One — never join on a source system's identifier, because they are reused, they collide across systems, and they change during migrations. Two — the mapping is effective-dated, because identity is a belief that gets revised, not a fact. Three — a merge must be reversible without rewriting history, which is only possible if you kept knowledge time. That third rule is the bridge into the next section, and it is exactly where the "convincing under probing" is won.

§ 09 — BITEMPORALITYTwo clocks, and why regulation forces the second

Most warehouses track one time: when something was true in the business. Regulated reporting needs a second: when the organisation knew it. Without both, "regenerate the report exactly as it was filed on 31 March" is unanswerable — and that question is asked after every restatement.

A BACKDATED CORRECTION, SEEN ON BOTH CLOCKS VALID TIME — when it was true in the business 10 Mar 31 Mar — period close 30 Apr trade booked, value 1,000 …same trade, corrected value 1,150 — still dated 10 Mar TRANSACTION TIME — when we learned it 10 Mar 05 Apr — report filed 22 Apr — correction arrives As filed — reproducible forever WHERE valid_from <= '31 Mar' AND recorded_at <= '05 Apr' → returns 1,000 As corrected — today's truth WHERE valid_from <= '31 Mar' AND superseded_at = 'infinity' → returns 1,150

Both answers are correct; they answer different questions. A single-clock model can produce only one of them, and silently loses the other the moment the correction lands.

The pattern applied to a slowly-changing dimension
-- Business time answers "what was the customer's risk rating in March?" -- Knowledge time answers "what did we THINK it was when we filed?" CREATE TABLE dim_customer_risk ( party_key BIGINT NOT NULL REFERENCES dim_party, risk_rating TEXT NOT NULL, rating_source TEXT NOT NULL, valid_from TIMESTAMPTZ NOT NULL, -- business time valid_to TIMESTAMPTZ NOT NULL DEFAULT 'infinity', recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(), -- knowledge time superseded_at TIMESTAMPTZ NOT NULL DEFAULT 'infinity', -- Corrections INSERT. Nothing in this table is ever UPDATEd -- except superseded_at, and nothing is ever DELETEd. PRIMARY KEY (party_key, valid_from, recorded_at) ); -- "As we understood it on the filing date" -- the reproducibility query -- that a regulator, an auditor or a restatement review will ask for. SELECT party_key, risk_rating FROM dim_customer_risk WHERE valid_from <= '2026-03-31' AND valid_to > '2026-03-31' AND recorded_at <= '2026-04-05' AND superseded_at > '2026-04-05';

The cost, stated honestly. Bitemporality roughly doubles the row count on corrected entities, makes every query carry four predicates instead of two, and is genuinely harder for analysts to use — so you expose a current-state view over it and reserve the full four-predicate form for reporting and audit. Say that trade-off out loud in the interview. A candidate who applies bitemporality to everything is as unconvincing as one who applies it to nothing: it belongs on entities that feed regulated or restated numbers, and nowhere else.


§ 10 — GRAINThe decision you cannot buy your way out of later

Grain is the first line of any physical design and the most expensive thing to get wrong, because detail you did not store is gone. The rule is simple to state and constantly violated: model at the finest grain the business could ever ask about, and aggregate upward. Aggregation is a query; disaggregation is a data recovery project.

The atomic transaction fact — one row per posting, per currency
-- GRAIN: exactly one row per (transaction, leg, currency). Writing this -- sentence before the DDL is the discipline; if you cannot write it in one -- line, the model is not understood yet. CREATE TABLE fct_transaction ( txn_key BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, -- Surrogate FKs only. No source identifier is ever a join key (§ 08). party_key BIGINT NOT NULL REFERENCES dim_party, account_key BIGINT NOT NULL REFERENCES dim_account, product_key BIGINT NOT NULL REFERENCES dim_product, -- Currency is part of the grain, never implied by the source system. -- Retrofitting this column is a full-history rewrite (§ 05). amount NUMERIC(20,4) NOT NULL, currency_code CHAR(3) NOT NULL, -- Reporting-currency amount is stored, not computed on read, so a -- historical report cannot silently move when FX rates are revised. amount_rpt_ccy NUMERIC(20,4) NOT NULL, fx_rate_key BIGINT NOT NULL REFERENCES dim_fx_rate, -- UTC plus the original offset. Local time alone is ambiguous -- across DST, permanently and unfixably (§ 05). occurred_at_utc TIMESTAMPTZ NOT NULL, occurred_offset INTERVAL NOT NULL, booking_date DATE NOT NULL, -- business date, drives partitioning -- Lineage on the row itself: which extract produced it, and from where. -- This is what makes an incident a 20-minute investigation. source_system TEXT NOT NULL, source_txn_id TEXT NOT NULL, batch_id BIGINT NOT NULL, recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(), CONSTRAINT uq_src UNIQUE (source_system, source_txn_id, currency_code) ) PARTITION BY RANGE (booking_date); -- Partition on the column the filters actually use. Every reporting query -- is bounded by business date; arrival date would prune nothing. -- Repartitioning this later is a one-way door (§ 06). -- The invariant that makes the ledger trustworthy: legs of one event net -- to zero within a currency. Tested continuously, not assumed (§ 15).
The one legitimate exception, and how to phrase it. Cost and regulatory retention can force coarser grain — but the decision is then explicit, dated and signed, not accidental. "We hold atomic detail for 25 months and monthly aggregates for seven years, because retention policy X requires the aggregate and the detail costs Y" is an architectural decision. "We only had monthly" is a discovery someone makes two years later in an audit.

§ 11 — CONTRACTSWhere change becomes expensive, on purpose

Inside a boundary, refactor freely. Across a boundary, you have consumers you cannot see and cannot call. A contract makes that asymmetry explicit — and deciding where the boundaries go is most of what "architecture" means in practice.

A published contract — the minimum that makes a dataset a product
# Not a schema file. A schema says what the columns are; a contract says # what someone may rely on, for how long, and who answers when it breaks. dataset: fct_transaction_daily owner: data-platform/retail-banking # a team, never a person on_call: #retail-data-oncall grain: "one row per party, product, booking_date, currency" sla: freshness: "T+1 by 06:00 UTC" availability: "99.5% of business days within SLA" # error budget, not a wish backfill: "90 days reprocessable within 4 hours" semantics: amount_rpt_ccy: "converted at the rate valid on booking_date, stored at write time -- does not move when rates are revised" party_key: "resolved identity; may merge historically (see party_merge_log)" guarantees: - "additive columns only within a major version" - "reconciles to GL daily; variance = 0 or the dataset is held back" - "no row is ever deleted; corrections arrive as new rows" deprecation: notice: "2 quarters" dual_run: "1 quarter with both versions published" # The guarantees section is the interesting one. "Additive only" is what # lets consumers upgrade lazily. "Held back on variance" is a promise that # the pipeline will fail LOUDLY rather than publish a wrong number quietly -- # and that choice is the difference between a platform people trust and one # they shadow-copy into spreadsheets.

The last comment is the point worth making aloud. Every organisation with an untrusted warehouse has the same symptom: analysts maintaining private extracts. That is not a discipline problem, it is a contract problem — nobody promised them anything, so they insured themselves. Contracts are how you end that, and they cost nothing but the willingness to be held to them.


§ 12 — THE TRADE-OFF RULESEach one resolves to an answer

"It depends" is not a trade-off, it is a refusal to have one. A usable rule names what it depends on and what value tips it. These are the ones worth carrying into an interview, phrased so that they decide something.

The choiceThe rule that decides itWhat tips it
Batch vs streaming Stream only where a registered decision has tolerable staleness below the batch cycle and acting sooner changes the outcome No decision under ~1 hour → batch. Streaming roughly triples operational surface: ordering, state, replay, exactly-once, backfill
Normalize vs denormalize Both, at different layers. 3NF where you reconcile and enforce integrity; dimensional where humans query Not a global choice. Treating it as one is the tell of a candidate who has only worked in one layer
Build vs buy Buy unless a regulator or a customer would recognise the thing as yours Ingestion, orchestration, storage → buy. Identity resolution rules, risk logic, metric semantics → build
Central vs federated (mesh) Federate only when domains have dedicated data engineers and a self-serve platform team already exists Below roughly three staffed domains, mesh is silos with better vocabulary. The platform must precede the federation
ELT vs ETL Land raw first, always. Transform after, in the warehouse, where it is versioned and re-runnable The exception is PII that may not land unmasked — tokenize in flight, and accept the operational cost
Warehouse vs lakehouse Choose the storage format first and the engine second; open table format keeps the engine a two-way door If a vendor's proprietary storage is the only option, the vendor decision has silently become one-way (§ 06)
Real-time identity resolution Resolve asynchronously; serve the last resolved state. Synchronous resolution puts a fuzzy-matching service on the critical path Unless a decision in the register needs it inline — fraud usually does, reporting never does
One warehouse vs many marts One conformed integration layer, many serving marts. Conform the dimensions, not the consumers Marts that bypass integration are how two teams end up with two revenue numbers and no way to adjudicate
How to deploy these under probing. Give the rule, then immediately name its failure mode — "stream only where a decision needs it; the failure mode is that someone senior wants a real-time dashboard nobody acts on, and I would push back by asking what they would do differently at 9am versus noon." Naming the counter-pressure shows you have applied the rule in an organisation rather than read it. That is precisely the register the Citi interviewer was probing for.

§ 13 — COEXISTENCEThe legacy estate does not pause while you build

The question explicitly says there is existing data debt and legacy systems. Programmes rarely fail at the new architecture; they fail at the switch-over, and they stall at roughly 70% — new platform live, old platform still running, nobody willing to sign the decommission. The defence is to make decommissioning a gate with numeric criteria rather than a decision someone has to be brave about.

STRANGLER — CONSUMERS MOVE ONCE, TO A FACADE, AND NEVER AGAIN Sources unchanged Legacy pipeline still authoritative New pipeline shadow, then primary Reconciliation gate row + aggregate diff, every run Facade / semantic layer one stable address Consumers move once, at the start The decommission gate — numeric, agreed in advance, so nobody has to be brave 30 consecutive business days at zero unexplained variance · every consumer migrated to the facade · runbook and on-call transferred · rollback rehearsed once Until every criterion is met the legacy system stays. Once they are all met, decommissioning is administrative rather than a judgement call. Why the facade comes first, before any pipeline work: If consumers point at physical tables, every migration is a negotiation with every consumer, and the cost of change scales with adoption — the worst possible incentive. Point them at a stable logical address on day one and you can replace what is behind it as often as you need to. This is the step that most often gets deferred, and deferring it is what produces the 70% stall.

Strangler with a reconciliation gate. The dual-run period is not waste — it is the only evidence that will let anyone sign the decommission.


§ 14 — THE METRICSOnly one is genuinely non-negotiable

The third question asks which metrics prove the system is good. The strongest framing is to answer a sharper question than the one asked: which numbers would make me roll back a release or hold a publication? Metrics that would never trigger an action are dashboard decoration, and listing them is what makes an answer sound rehearsed.

MetricTargetWhat it tells you that nothing else does
Reconciliation variance vs system of record Zero. Daily. Breaks logged and aged The only truly non-negotiable one. If warehouse revenue ≠ GL revenue, every other metric is measuring the performance of a wrong answer
Freshness, as an error budget 99.5% of business days within the contracted SLA A single threshold hides how often you miss. A budget makes reliability negotiable with a number rather than a feeling
Completeness of expected partitions 100% landed, or the dataset is held back Catches the silent failure that quality tests miss entirely — data that is correct but absent
Time to onboard a new source Trending down; measured per source The single best proxy for "resilient against future business shifts." A rigid architecture shows up here first, long before latency degrades
Schema-change lead time & consumer break rate Additive change same-day; zero unplanned breaks If adding a column takes three weeks or breaks four dashboards, the architecture is brittle no matter how fast it queries
Column-level lineage coverage 100% of published regulatory metrics Directly demanded by BCBS 239-style expectations. Also the difference between a 20-minute incident and a two-day one
Backfill throughput 90 days reprocessed within one business day Your real recovery capability. RPO/RTO on a cluster is meaningless if correcting history takes three weeks
Cost per million transactions processed Flat or declining as volume grows Absolute spend rising is fine when volume rises faster. Unit cost is the metric that reveals whether the architecture actually scales
Query latency p95 / p99 by consumer class Per contract Averages hide the tail, and the tail is what users experience and remember
The two that separate a strong answer from a generic one are time to onboard a new source and schema-change lead time. Everyone lists freshness and latency. Almost nobody names a metric for evolvability — and evolvability is precisely what "resilient against future business shifts" is asking about. An architecture that serves today's queries in 200 ms but takes a quarter to absorb an acquired bank's data has failed the actual question.

§ 15 — THE CONTROLWhat "reconciliation variance = zero" actually is

Claiming a non-negotiable metric invites the obvious follow-up: how is it enforced? Have the mechanism ready, because "we monitor it" is not an answer — monitoring tells you after someone has already used the number.

The gate that runs before publication, not after
-- Runs after load, before the serving layer is swapped. If it fails, the -- previous day's data stays live and someone is paged. Publishing a wrong -- number is strictly worse than publishing yesterday's -- one is a stale -- decision, the other is a wrong one nobody knows is wrong. WITH warehouse AS ( SELECT booking_date, currency_code, SUM(amount) AS amt, COUNT(*) AS rows FROM fct_transaction WHERE booking_date = :business_date GROUP BY 1, 2 ), ledger AS ( SELECT posting_date AS booking_date, currency_code, SUM(posted_amount) AS amt, COUNT(*) AS rows FROM gl_postings -- the system of record WHERE posting_date = :business_date GROUP BY 1, 2 ) SELECT COALESCE(w.booking_date, l.booking_date) AS booking_date, COALESCE(w.currency_code, l.currency_code) AS ccy, COALESCE(w.amt, 0) - COALESCE(l.amt, 0) AS amount_variance, COALESCE(w.rows, 0) - COALESCE(l.rows, 0) AS row_variance FROM warehouse w FULL OUTER JOIN ledger l -- FULL OUTER: a missing side is a break too ON w.booking_date = l.booking_date AND w.currency_code = l.currency_code WHERE COALESCE(w.amt, 0) <> COALESCE(l.amt, 0) OR COALESCE(w.rows, 0) <> COALESCE(l.rows, 0); -- Any row returned = publication halted. Note the three deliberate choices: -- 1. FULL OUTER, so a currency present on only one side is caught. -- 2. Both amount AND row count -- two errors can cancel in a sum. -- 3. Grouped by currency, because a cross-currency total can net to zero -- while both sides are wrong. Reconciling on a single grand total is -- the classic control that passes while the data is broken.

Breaks that are understood and accepted — a known timing difference, a documented in-transit account — go into a break register with an owner and an expiry date, and the gate excludes them explicitly by identifier. What it must never do is widen a tolerance to make the check pass. A tolerance that grows is a control that has been quietly switched off, and in a regulated environment that is the finding an auditor is specifically looking for.


§ 16 — THE ONE-MINUTE ANSWERIf you get sixty seconds, say this

Compressed

"I design against decisions, not requirements. Step one is a decision register — who decides what, how often, how stale it can be, what being wrong costs. That converts vague goals into a latency and accuracy budget. Step two is source forensics: I profile the sources rather than interview people about them, because documentation describes intent and I need behaviour. Then the conceptual model — entities, grain, identity — deliberately before any vendor conversation."

"For trade-offs, I sort every open choice by reversibility. Grain, identity, bitemporality and metric semantics are one-way doors — undoing them means rewriting history or renegotiating with every consumer, so they get the deliberation. Orchestrators, BI tools, even the compute engine if storage is an open format, are two-way doors — I decide those quickly and move on. Most programmes invert this and spend six weeks on a tool bake-off while the grain gets decided by whoever writes the first pipeline."

"In production, one metric is non-negotiable: reconciliation variance against the system of record is zero, checked before publication, not after. Everything else is an error budget — freshness at 99.5% of business days, completeness, p95 latency per contract. But the two I watch for resilience are time to onboard a new source and schema-change lead time, because those are what tell me whether the architecture can absorb a business shift. A platform that answers today's questions in 200 milliseconds but needs a quarter to take on an acquisition has failed the actual test."


§ 17 — THE PROBESWhere this interview is actually decided

The reader who shared this question said they could answer, but not convincingly once the interviewer probed. That is the normal failure mode: the prepared answer holds for ninety seconds, and the follow-up finds the edge. These are the probes that follow this question, and what a convincing reply sounds like.

"What if the business cannot articulate its goals?"
Then stop asking. Watch instead: find decisions currently being made late, badly, or in spreadsheets, and reconstruct the register from those. People cannot specify what they want in the abstract but are precise about what frustrates them. "Which decision did you make last month where you wished you'd had better data, and what did it cost?" gets a usable answer where "what are your requirements?" gets a wish list.
"Our CDO has mandated a data mesh. Would you push back?"
Not on the destination — on the sequence. Mesh needs two prerequisites: domain teams with real data engineering capacity, and a self-serve platform so a domain can publish without rebuilding infrastructure. Without those, federation produces silos with better vocabulary and no conformed dimensions. So: build the platform and the contract standard first, federate the two most mature domains as proof, expand on evidence. That is agreeing with the strategy while refusing to skip its foundation — which is a different thing from resisting it, and interviewers can tell the difference.
"A critical source is unreliable and the owning team will not fix it."
Grade it C and quarantine it — it may feed exploratory analysis, never a conformed layer or a regulatory number. Publish anything derived from it with the caveat attached to the data, not buried in a wiki. This usually resolves itself: the moment a consumer is told a number cannot be used for reporting because of source X, the pressure to fix source X arrives from a direction the data team never had access to.
"Why not just buy a platform that does all of this?"
Often you should, and I would start there for ingestion, orchestration and storage. But name what the purchase decides for you: if it stores in a proprietary format, the engine has silently become a one-way door; if it imposes its own identity model, that is the most expensive decision in the design made by procurement. Buy the commodity layers, keep storage open, and keep identity resolution and metric semantics in your own control — those are the two things a regulator will hold you accountable for, and a vendor cannot be accountable on your behalf.
"Tell me about a time you got a foundational decision wrong."
Have a real one, and make it a one-way door — a two-way door story proves nothing. The strong shape: name the decision, name the signal you missed, name the cost, and name the practice you changed. "We set the fact grain at daily because no stated requirement needed intraday; nine months later treasury needed hourly and there was no detail to recover, so we ran fourteen months of backfill from archives. Now I profile what the source actually emits before setting grain, and store the finest grain available even when nothing has asked for it — because grain is the one thing you cannot add later."
"How do you know when the architecture is done?"
It is not, and saying so is correct rather than evasive — but follow it with the criterion. It is working when the cost of the next change is flat: onboarding the tenth source costs about what the fifth did, and adding a column does not require a committee. Rising marginal cost of change is the signal that the architecture has stopped absorbing the business and started resisting it. That is the same measurement as § 14, which is why those two metrics are in the answer at all.

§ 18 — COMMON MISTAKESWhat tanks this answer

Leading with a stack. Naming Snowflake, dbt and Airflow in the first sentence answers a question about tools that nobody asked. The prompt explicitly says "before vendors are locked in."
A process with no artifacts. "Gather requirements, design, build, iterate" describes every project ever attempted. Each step must produce something reviewable.
Treating greenfield as clean. The prompt says data debt exists. An answer that never mentions coexistence, dual-run or decommission has ignored a third of the question.
Trade-offs that never resolve. "It depends on the use case" for every choice. A rule must name what it depends on and what tips it.
Metrics nobody would act on. Listing uptime and row counts. The useful question is which number halts a release — and reconciliation is the answer.
Confusing resilience with uptime. "Resilient against future business shifts" is about evolvability. Failover time does not measure whether you can absorb an acquisition.
Bitemporality everywhere. Applying it to all entities to sound rigorous. It has real analyst-facing cost; it belongs on regulated and restated entities, and naming that boundary is the signal.
Mesh as a default. Proposing federation without naming its prerequisites. Interviewers at large banks have watched this fail and will probe it hard.

This was a community submission. Just finished an interview? Share the question that stumped you — anonymous or credited. Share a question →  |  ← Design Index
← paddyspeaks.com