The Interview RoomHow a senior data engineer is really tested

Happy to. I'll skip the chronology, if that's all right — I'd rather tell you what kinds of problems I've spent my life on.

Three things, mostly. First, large distributed data platforms: designing them, and then being the person who explains why they're behaving badly. Second, production troubleshooting at the layer where Spark meets the JVM meets object storage — the place where the pretty architecture diagram stops helping you. Third, taking what I learn from a single incident and turning it into something that stops thirty incidents.

Scale, since you'll ask: the largest platform I owned end to end ran about eleven petabytes across roughly nine hundred pipelines, four thousand daily Spark applications, peak concurrency of about three hundred jobs. Roughly two hundred internal teams depended on it.

The work I'm proudest of isn't an architecture. It's a diagnostic. We had a recurring class of failure that took a senior engineer two days per incident. I traced it to a shuffle-fetch pattern, wrote the detector, and it turned a two-day investigation into a fifteen-minute one that a first-year engineer could run.

Fair question. The detection logic was mine — the hypothesis, the event-log parsing, the first ugly version. The productionised service was one of my engineers, and honestly it's better than what I wrote. The thing I'd claim credit for is noticing that the same shape of failure was showing up in tickets that had been filed under six different symptoms.

Honestly? Less than I was five years ago, and I've missed it enough to be sitting here.

But let me answer the question underneath the question, because I think you're asking whether I can still descend. Architecture rots into slideware the moment you can't verify it. The thing that kept mine honest was being able to go all the way down and come back up.

The left-hand column is what a good troubleshooter does. The right-hand column is what stops you doing the left-hand column forever.

Plenty of people can do one or the other. The ones who can only descend become heroes who never fix anything permanently. The ones who can only ascend produce recommendations that don't survive contact with a task log. I want a job that requires both, and escalation engineering at a platform company is the only place I've found where both are the same job.

(a beat) Eleven days ago. A cost-attribution job started spilling. I expected a broadcast on the dimension side and got a sort merge instead, because someone had added a filter that the optimiser evaluated after the size estimate. It wasn't my job to fix. I read it because I wanted to know if I still could.

A retail analytics customer — I'll call them Halvorsen Group — had a nightly pipeline that fed store replenishment. It ran at 01:00, finished by 03:30, and buyers acted on it at 06:00. One Tuesday it didn't finish until 11:00, and the stores ordered on stale numbers.

Their VP of Data got on the bridge and opened with: “Your platform broke our production pipeline, and it has cost us about four hundred thousand in misallocated stock.”

I said: “I understand the impact and I'm treating this as a Sev-1. I'm not going to agree with the diagnosis yet, because I don't have evidence for it, and if I agree with the wrong cause we'll fix the wrong thing.”

It is. It's also the only honest one. The trick isn't the words, it's the split: I acknowledge the impact without accepting the diagnosis. Those are two separate transactions and customers usually only want the first one immediately.

So I split the bridge in two. One track was mitigation — get tonight's run to land before 06:00, whatever it takes, including running it on a bigger cluster we'd eat the cost of. The other track was root cause, which I explicitly said would take days and would not be discussed on this call.

Their upstream vendor had started emitting a new store code — a placeholder for “distribution centre, unassigned”. About forty per cent of the rows carried it. It became the hot key on a join we'd never had trouble with.

So: not our platform. But — and this is the part I insisted on — our platform let it happen silently. The job didn't fail. It just got four times slower with no signal until a human noticed. That was ours. We shipped a data-distribution check on that join, and it caught the same class of thing at two other customers within a quarter.

Not on day one. On day one they thought I was deflecting. What changed their mind wasn't my explanation — it was that I sent them the key-frequency distribution from before and after, with the date the new code appeared circled. Evidence travels better than apology.

They did also point out, correctly, that we'd sold them a platform that was supposed to notice things like this. Which is why I didn't argue about the four hundred thousand.

Support believed there was a product defect in a connector. Engineering believed it was customer misuse. Both had been saying so, more loudly each week, for about a month.

Neither, and I said so, which made me unpopular in two directions at once.

My position is that I don't escalate opinions. I escalate evidence. What Support had was six customers who were angry. What Engineering had was a strong prior that the connector was fine because it had been fine for two years. Neither of those is a bug report.

A chain. It's the same chain every time.

I gave Engineering the four-minute reproduction and the line number I suspected. The defect was confirmed in a day. It had been an argument for a month and a bug for an afternoon.

Then Support would have been wrong, and I'd have had to say so — with the same artefact. That's the deal. You don't get to build an evidence chain and then only publish it when it agrees with you.

My first hypothesis is skew, and now I've got support for it rather than a hunch. Three different executors, so it isn't a bad host. Attempt zero, so it isn't retries. And the slow tasks read fifty to seventy times more data than the fast ones — so it's not environmental, it's the partitioning.

But I want to be careful about the word “skew”, because it's a description of the symptom, not the cause. Something put twenty-eight to forty-three gigabytes into three partitions. I want to know what.

I might well end up there. But salting is an answer, not a diagnosis, and I'd be embarrassed to reach for it before I know which key is hot and why.

Here's my worry. Suppose the hot key turns out to be null. Salting spreads nulls across two hundred partitions and the job gets faster — and we have just spent compute distributing rows that were never going to match anything on the other side of the join. The correct fix is a filter, and it makes the job faster than salting ever would.

Or suppose the hot key is a real, legitimate customer who is genuinely forty per cent of the data. Then salting is right, and it's right permanently, and I should build it into the pipeline rather than patch it.

Or suppose the join is exploding — the “skew” is on the output side, not the input side, because a one-to-many relationship became one-to-many-thousands. Salting does nothing for that at all.

Three causes. Three different fixes. Same symptom.

It's one query, and it runs on a sample.

(quietly) Well. That changes my hypothesis.

Forty-one per cent null, eighteen per cent the literal string UNKNOWN, eleven per cent minus-one. That's not skew in the interesting sense. That's three generations of “we don't have this value” conventions living in one column, and a join that is dutifully shuffling all of them to three partitions.

To a data-quality problem wearing a performance costume.

And it also changes the question I want to ask the customer, which is now: what is the join supposed to do with those rows? Because if the dimension table has no row for null, no row for UNKNOWN, and no row for minus-one — and it almost certainly doesn't — then seventy per cent of the shuffled volume is being moved across the network in order to match nothing.

An inner join drops them after the shuffle. A left join keeps them with nulls on the right, which they could have had without going anywhere.

Then they need them — but they still don't need them in the join. Split the frame: the rows with a resolvable key go through the join, the rows without one bypass it and get unioned back on with the null-filled columns they were always going to get. Same output, seventy per cent less shuffle.

That's the fix I'd propose. And I'd want to say out loud that it's a code change on their side, not a setting, which means it doesn't land tonight.

Tonight I'd verify that adaptive execution is actually enabled and that skew join handling is on, because if it's off — and on older configurations it often is — turning it on may split those three partitions automatically and buy the customer their morning back. That's mitigation, it's reversible, it's one setting, and I can explain exactly what it does.

Then tomorrow we have the conversation about the data model.

Then I want to know why, because that's interesting. Adaptive skew handling splits a partition that's large relative to the median — but it can only split it if the work is splittable. If the downstream operation is an aggregation on that key, you can split the input all you like; the results still converge on one reducer. And if the statistics that AQE uses came from a stage whose sizes were themselves computed before a filter, the partition may not look skewed to the optimiser even though it is.

Which is a long way of saying: if AQE didn't fix it, I stop trusting my mental model and go and read the plan.

Broadcast hash join, obviously — the small side gets collected to the driver, shipped once to every executor, built into a hash table in memory, and then each executor probes it with its own local slice of the fact table. The eight terabytes never moves. That's the whole point.

I don't know yet, and I'd want to see the plan before I speculate. But I can tell you the shortlist I'd work through, in order of how often it's the answer.

One: no statistics. If the table has never been analysed and the format doesn't carry reliable size metadata, the optimiser falls back to a default size estimate — and the default is deliberately enormous, so nothing gets broadcast. This is the answer maybe half the time.

Two: the filter is applied after the estimate. This one catches everyone. The dimension is forty megabytes after segment = 'ENTERPRISE'. If the optimiser's size estimate is based on the table and its selectivity guess for that predicate is wrong — or the predicate is on a column with no statistics — it may believe the built side is four gigabytes rather than forty megabytes.

Three: stale statistics. Somebody analysed the table when it had two hundred rows in it, then loaded a year of data, and nobody re-analysed.

Four: serialised size versus stored size. Forty megabytes of compressed Parquet can be several hundred megabytes as deserialised Java objects in a hash table. If it's near the threshold on disk, it isn't near the threshold in memory, and the estimate the optimiser uses may be either.

Five: the threshold was lowered. Platform teams sometimes reduce the broadcast threshold globally because a driver went out of memory once. Then everything sort-merges and nobody remembers why.

Six: a hint, or the join type. A full outer join can't be broadcast. Nor can most joins where the side you'd need to broadcast is the side that must be preserved.

I'd look at the plan and read the sizeInBytes the optimiser assigned to the built side. That single number tells me whether I'm arguing with a statistics problem or a threshold problem, and those are different conversations.

If it says eight exabytes — and it will say something absurd like that if there are no statistics — I'm in case one, and the fix is to collect statistics rather than to fight with hints.

There. Eight exabytes. That's the “I have no idea how big this is” value — the optimiser's way of saying it has no statistics and is refusing to guess low.

So the mechanism is: no statistics, therefore a maximal size estimate, therefore no broadcast, therefore eight terabytes crosses the network for no reason at all.

I'd add a broadcast hint to prove the mechanism, run it once, and watch the runtime collapse. That's the experiment.

But I wouldn't ship the hint as the fix, and I definitely wouldn't close the ticket with it. A hint fixes one query. Missing statistics is a property of the table, and there are forty other queries against that dimension, all of them making the same bad decision quietly.

The fix is statistics collection on that table, and the interesting question — the escalation-engineer question — is why a table in a managed platform ended up with none. If it's because their ingestion path writes in a way that skips statistics, that's a product conversation, not a customer conversation.

Then my forty-megabyte assumption was wrong and I've learned something valuable for free. Broadcasting collects the built side to the driver first. If it OOMs there, the built side isn't forty megabytes — either the filter isn't as selective as we thought, or it's being applied after the collect, or the deserialised form is far larger than the stored form.

That failure is more informative than the success would have been. It also tells me not to be smug about case four on my list.

To find out what the JVM is doing.

Specifically: when a process is alive but not progressing. CPU is low, nothing is failing, tasks are just… not finishing. That's a thread dump question, because the answer is almost always that threads are parked somewhere and I want to see where.

Blocked threads and what they're blocked on. Deadlocks — the JVM detects most of them for you and says so at the bottom, which people miss. Threads in I/O wait, because that tells me I'm waiting on storage or a service rather than computing. Lock contention, where forty threads are queued on one monitor.

And the single most useful trick: take three dumps, ten seconds apart. One dump is a photograph and photographs lie. Three dumps tell you whether a stack is stuck or merely busy. If the same thread is at the same line in all three, that's your answer. If they're all different, the JVM is working and your problem is elsewhere.

To find out what the JVM is holding. Which is a different question, and I'd want to have already established from the GC log that something is being held — pattern B from before, post-collection occupancy that climbs and stays.

Then: object counts, retained heap rather than shallow size, reference chains back to a GC root. The question I'm answering is “what is keeping this alive?”, and the reference chain is the only thing that answers it.

That's the right question and it's the reason I don't ask for one casually.

A full heap dump stops the world for as long as it takes to write the heap to disk. On a sixteen-gigabyte heap that's tens of seconds of complete freeze, and it produces a file that then has to get off the executor and to me — which on a container that may be killed for being unresponsive is not guaranteed. You also just handed yourself a file containing the customer's actual data, which has consequences I need to have thought about before I ask, not after.

So: heap dump is a deliberate, scheduled act with a named hypothesis. Not a reflex.

Because a heap dump and a thread dump are both snapshots, and half the problems I chase are about timing.

JFR gives me a correlated timeline: allocation by call site, lock contention with durations, I/O with the file or socket that caused it, GC events, and thread states — all on the same clock. When the question is “what was this executor doing during the ninety seconds when the stage stalled”, that's a JFR question and nothing else answers it as well.

It's also low-overhead enough to leave running, which matters when the problem happens once a night.

None of them, probably — and that's the point of asking the question.

If the heap was fine and there's no OutOfMemoryError, the JVM didn't fail. Something killed it from outside. That's the container being terminated for exceeding its memory limit, and the memory that exceeded it wasn't heap — it was off-heap: native allocations, direct byte buffers, memory-mapped files, a Python worker, or the JVM's own metaspace and thread stacks.

So I'd go to the container's kill record and the node's memory accounting, not to a heap dump. A heap dump would have shown me a perfectly healthy heap and cost me two days of believing I had no evidence.

Because the function isn't the expensive part. The commute is.

Spark's execution engine runs on the JVM, in a compact internal row format. A Python UDF can't touch that. So for every batch, rows are serialised out of the JVM, sent over a local socket to a separate Python process, deserialised into Python objects, your function runs one row at a time, the results are serialised back, and the JVM deserialises them again.

Four format conversions and two process boundaries, per batch, to run a function that's usually about four lines long.

There's a second cost people forget: it breaks whole-stage code generation. Spark can fuse a chain of built-in expressions into one generated Java method that runs over the rows with almost no overhead. A Python UDF is an opaque box in the middle of that chain, so the fusion stops on both sides of it.

And a third: the Python process has its own memory, outside the JVM heap, charged against the container. That's how you get an executor killed with a perfectly healthy heap.

No. First I'd establish whether the UDF is actually on the dominant path.

I've watched a team spend three weeks rewriting UDFs in Scala for a pipeline where the UDFs accounted for four per cent of the runtime and the other ninety-six was a shuffle they never looked at. They shipped it. The pipeline got about two per cent faster. Everyone was very tired.

So: which stage is slow, is the UDF in it, and what fraction of that stage's time is the Python boundary. If the answer is “most of it”, then yes — and the options in order of how much I like them are: replace it with built-in expressions so the JVM never leaves home; make it a vectorised, Arrow-based function so the crossing happens per batch instead of per row; or, if the logic genuinely can't be expressed either way, keep it and accept the cost knowingly.

Right, and that's exactly why it's the blind spot.

First I'd look at the platform's own accounting — most Spark distributions expose a Python memory limit and will fail the worker with a clear message rather than letting the container die, if it's configured. Getting that limit set is worth more than any profiling, because it converts a mysterious container kill into a legible error.

Then, from outside: resident set size of the python processes on the executor node over time. That's a shell command and a bit of patience, and it tells you whether the growth is per-batch or monotonic.

From inside, if I can reproduce it: a memory profiler in the worker, or — more often, because it's simpler — I look at what the function does per row. The usual culprits are embarrassingly mundane. Loading a model or a lookup table inside the function instead of once per partition. Accumulating results in a list. Pulling a whole partition into pandas because the API made it easy.

And the structural fix, most of the time, is mapPartitions-shaped thinking: do the expensive setup once per partition, not once per row.

Okay. The obvious first draft is a window function, and I'll write that, but let me say what I'm worried about before I write it so you can see me be wrong in advance.

Worry one: ties. If the same event_id arrives twice with the same updated_at, “the latest” isn't defined and I'll get non-deterministic output. That's the kind of bug that shows up as “the report changes when we re-run it” six months later.

Worry two: nulls in updated_at. Ordering with nulls is a decision, not a default I should accept silently.

Worry three: this is a shuffle on event_id, which is high cardinality, so it should distribute well — but if the upstream duplicates cluster, some keys could be hot.

Let me write the straightforward version.

I've added ingest_seq as a tie-break, assuming there's a monotonic arrival ordering — and if there isn't, that's a question for the customer, not something I invent. Without it, this query is a coin flip on duplicate timestamps and it will silently disagree with itself between runs.

At three billion rows, the row_number is doing more work than the problem requires.

A window function has to shuffle by event_id, then sort within each partition, then assign numbers, then discard everything except rank one. I'm paying for a full ordering of every duplicate group in order to answer a question that only needs a maximum.

If ninety-eight per cent of my event_ids have exactly one row — which is typical for a deduplication job — I'm sorting three billion rows to fix sixty million of them.

An aggregation with a struct, which turns the ordering into a max.

Struct comparison is lexicographic by field order, so the first two fields decide the winner and the rest come along for the ride. It's a partial aggregate — each partition can reduce locally before the shuffle, so the volume crossing the network is proportional to the number of distinct event_ids per partition rather than to the number of rows.

Two honest caveats, because I don't want to oversell it. If the payload is large, dragging it through the struct isn't free, and it may be better to aggregate to the key plus timestamp and then join back. And this reads worse than the window version — the next engineer will have to think about why it's written this way, so it needs a comment saying so.

Then I'd stop optimising the query and go back to looking, because I've now made two changes based on reasoning and zero based on measurement.

My guess would be that the bottleneck was never the dedup at all — it's the scan. Three billion rows a day, if that's landing as a large number of small files, the read is the cost and both of my clever queries are rearranging deck furniture. But that's a guess, and the metrics would take four minutes to check.

Maybe. But adding workers only helps if the thing that can't keep up is parallelisable, and I don't know that yet.

First I need to find which part of the pipeline can't sustain the incoming rate. Because “processing rate is sixty-eight thousand” is an output measurement — it's the number that falls out of everything else. It doesn't tell me where the ceiling is.

What I'd want off the streaming metrics, in order:

Batch duration against trigger interval. If batches are taking twelve seconds on a ten-second trigger, the backlog is inevitable and the question becomes “which phase of the batch”.

The breakdown within the batch — how much is planning, how much is the actual query, how much is the commit. That last one catches people out: if the checkpoint commit is taking four seconds of a twelve-second batch, adding executors will do absolutely nothing, because the commit is a serialised operation against object storage.

State store metrics. Number of keys, size on disk, and — the important one — how much time is spent in state operations per batch. Unbounded state growth is the classic streaming failure: it works beautifully for three weeks and then degrades every day.

Watermark and late data. If the watermark isn't advancing, state is never evicted, and you have a job that is slowly turning into a database.

Source parallelism. If the topic has twenty-four partitions, twenty-five executors buys nothing. That's the case where “add workers” is simply not available to you.

And sink latency, which is where I'd expect this one to be, honestly. A hundred thousand records a second into a table with any kind of upsert semantics is a write-amplification problem long before it's a compute problem.

Then adding workers is precisely the wrong move and I'd say so plainly, because it costs money and buys nothing.

Eight and a half seconds of commit means the checkpoint write is the ceiling. That's usually one of three things: the state is large and being written in full rather than incrementally; the checkpoint location is on storage with poor small-write performance or in a different region from the compute; or the number of state store partitions is high enough that the commit is thousands of small writes rather than dozens of large ones.

The lever people forget is the trigger interval. If the batch takes fourteen seconds and eight of it is fixed cost, then running every thirty seconds instead of every ten roughly triples throughput without adding a single machine — because you're paying the fixed cost a third as often. You buy that with latency, and whether that trade is acceptable is a product question I'd take to the customer rather than decide for them.

(no pause) Because it would work, and that's the problem.

It is, and let me be quick about it because you're right that we've been slow.

The checkpoint holds two things: which offsets you've already consumed, and the state your aggregations have accumulated. Deleting it and starting from latest means the nine hours in your backlog are never processed. Not delayed — never. Those records exist in the source, your table will never contain them, and your eight o'clock deck will be built on numbers that are missing nine hours of activity without saying so anywhere.

And it's worse than a gap, because if any of your aggregations are stateful — running totals, sessions, deduplication — the state resets too. Your dedup window is empty, so records you'd already emitted can be emitted again if they reappear. You could get a gap and a double-count in the same table.

The deck will look complete. That's what makes it dangerous.

Here's what I'd do in the next thirty minutes, and you can decide.

One: raise the trigger interval to thirty seconds now. That's one setting, it's reversible, and based on the numbers on my screen it should roughly triple throughput. If it does, you're draining the backlog rather than growing it, and we can compute when you'll be current.

Two: in parallel, I'd start a separate one-off backfill job reading the same source over the backlog window, writing to a staging table. That's a batch job, it's parallel, and it doesn't care about your trigger interval.

Three: if you decide the deck matters more than the completeness — which is a legitimate business call and not mine to make — then we can start from latest and keep the old checkpoint, so the backlog is recoverable afterwards instead of gone. Move the directory. Don't delete it. That single word is the difference between a decision and an accident.

And whatever we choose, I'd want a line in your deck saying what window the numbers cover. Not because of process. Because someone will act on them.

A financial services customer had intermittent job failures — maybe one run in fifteen, always with a shuffle fetch failure. I had a strong prior, because I'd seen the same signature twice before, and both times it had been network: a saturated cross-zone path.

So I went looking for network evidence. And I found some — there were retransmits, there was elevated latency on some paths, and I built a fairly convincing case. We got their infrastructure team involved. They rearchitected placement so that executors were zone-local. It took eleven days of somebody's time.

The failure rate went from one in fifteen to one in seventeen.

Ouch is right. And the reason it took eleven days is entirely mine: I went looking for evidence that supported my hypothesis instead of evidence that would kill it.

What was actually happening was that executors were occasionally being lost during a scale-down — the autoscaler was removing executors that still held shuffle output, and the fetch failure was a downstream consequence of a node that had gone away. The network noise was real and irrelevant. It was there on the successful runs too. I never checked that, which is the whole error in one sentence.

Two things, one small and one that I think is the most useful thing I know.

The small one: I now always look at a healthy run alongside the failing one. Every signal I plan to blame has to be absent, or meaningfully different, in the run that worked. The network retransmits would have failed that test in ninety seconds.

The bigger one is how I hold a hypothesis. A hypothesis isn't something I defend. It's something I try to disprove. And practically, that means when I state one, I also state what would falsify it — out loud, to whoever's on the call — before I go looking. It's harder to quietly move the goalposts when you've written them down in front of six people.

Yes, and it was a genuinely bad twenty minutes. But their infrastructure team had spent eleven days on my theory and they were entitled to know that the theory was mine and it was wrong. The alternative — quietly letting the real fix land and taking credit for the resolution — was available and I've watched people take it. You only get to do that once before nobody believes your next diagnosis either.

Badly, at first, and then I learned to shut up.

The instinct when someone brings you a broken job is to look at it, see the answer, and give them the answer. It feels generous. It's fast, the customer is unblocked, and everybody says thank you. And it teaches nothing, because next month they bring you an almost-identical problem.

So now, when someone comes to me with “I think we need to increase the executor memory”, I don't say yes or no. I ask: what evidence tells us memory is the bottleneck?

Sometimes they have an answer and it's a good one, in which case I've learned something and they've had thirty seconds of confirmation. More often they pause and say “…the GC time is high”, and then we're in a useful conversation, because now I can ask what the GC time was on the run that worked.

Much slower, the first four times. Then it's much faster forever.

And I do have a rule about it: I don't do Socratic method during a Sev-1. If the customer is down, I give the answer, and we do the teaching in the postmortem when nobody is losing money. Using a live incident as a teaching moment is a form of self-indulgence.

I ask for the falsifying test rather than arguing with the conclusion. “What would we see if that were not true?” It's the same question I use on myself, it's not a status move, and half the time they find their own error while answering it — which is a much better outcome than me being right at them.

And when I'm the one who's wrong, which happens, it costs me nothing because I asked the question in a form that made being wrong survivable.

The first thing I'd do is find out whether anyone is running the incident, and if nobody is, say so and take it — or hand it to whoever's better placed. Nineteen people with no owner is not a bridge, it's a conference call.

Then, roughly in this order:

Preserving evidence, without question. At two in the morning, with a customer down, there is enormous pressure to restart right now — and a restart usually works, and then the evidence is gone and you get to have the exact same incident in three weeks with no more information than you had the first time.

It costs ninety seconds to grab the driver log and a thread dump before you bounce it. I've never once regretted spending that ninety seconds. I have repeatedly regretted not spending it.

Four, and one of them is uncomfortable, so I'll save it.

First: when an escalation reaches your team, where is the biggest diagnostic gap today? Not the biggest product gap — the place where your engineers routinely have to guess because the evidence isn't collectable.

That's useful, thank you. Second: how often does an escalation end in a product change, versus an architecture recommendation to the customer?

Third: what differentiates your strongest escalation engineer from an excellent support engineer? I want the actual behavioural difference, not the level definition.

(laughs) Fourth, and this is the uncomfortable one. How do you measure the problems that don't happen?

If I do this job well, my main output is tickets that never get filed. That's very hard to put in a performance review, and I've watched organisations quietly select against it — you end up rewarding the person who closed two hundred cases over the person who eliminated three hundred.

That's a better answer than a metric would have been, actually.

Can I add one more? Where do you see AI-assisted diagnostics helping in this work — and where would you not trust it?