The Spark Pipeline Debugging & Performance Engineering Handbook

"Walk me through what you do when a production Spark job that normally takes 40 minutes has been running for three hours."

Weak

"I'd check the logs and probably bump the cluster size."

Good

"I'd open the Spark UI, find the stage that's taking the time, and look at whether it's skew."

Senior

"First I'd separate pipeline time from Spark time in the scheduler — a three-hour pipeline may be a 40-minute Spark job behind a two-hour upstream wait. If it really is Spark, I go to the Jobs tab for the active job, then the active stage, then the task table sorted by duration, and I compare the max task against the median on duration, shuffle read and records. Those three numbers tell me in about ninety seconds whether I'm looking at skew, uniform under-partitioning, a straggler node, or an executor that keeps dying. Only then do I look at logs, and only then do I consider a config. And before I change anything I write down the current runtime, input bytes, shuffle bytes and executor-hours, because otherwise I can't tell later whether I helped."

"The pipeline SLA is 90 minutes. Yesterday it took 88. Today it took 140 and the on-call paged you. Where do you start?"

Weak

"Check if the cluster was smaller today."

Good

"Compare today's Spark UI to yesterday's — look for a stage that got much slower."

Senior

"I'd first split the 140 minutes into scheduler time and Spark time — a lot of 'Spark got slower' is actually 'we waited 50 minutes for an upstream table.' If the Spark portion really grew, I compare the two runs on a fixed set of counters: input bytes, output rows, shuffle read/write, executor-hours, task count, and task p50/p95/max for the top three stages. I'm looking for the metric that moved out of proportion to input — that ratio is the diagnosis. If shuffle quadrupled on flat input, I diff the physical plans, because something almost certainly stopped broadcasting. If input doubled, it's not a Spark problem at all and I go talk to the upstream owner."

"You run EXPLAIN and see a BroadcastHashJoin. In production the same query does a SortMergeJoin and takes 40 minutes. How is that possible?"

Weak

"The broadcast threshold must be too low in production."

Good

"Probably different data volumes — the table is bigger in prod than in my dev sample, so it exceeds the broadcast threshold."

Senior

"Several mechanisms could produce that, and I'd distinguish them from evidence. The optimizer picks broadcast from an estimate: file sizes plus whatever catalog statistics exist. If prod stats are stale or absent, the estimate can be wildly wrong in either direction. If the estimate exceeds spark.sql.autoBroadcastJoinThreshold, you get a sort-merge join. It also matters whether the build side is behind a filter the optimizer can't estimate, and whether AQE later converted the join — AQE can promote a sort-merge join to broadcast once it measures the real shuffle output, so the final plan in the SQL tab may differ from both. I'd check the SQL tab of the prod run for the actual join node, look at the build-side size metric, run ANALYZE TABLE … COMPUTE STATISTICS if stats are missing, and only then consider a hint. And I'd be careful: forcing a broadcast on a table that quietly grows is how you turn a slow job into a driver OOM."

"Your job fails with FetchFailedException. A teammate proposes raising spark.shuffle.io.maxRetries from 3 to 10. What do you say?"

Weak

"Sounds reasonable, retries usually help with flaky networks."

Good

"I'd want to check whether an executor died first — retries won't help if the data is gone."

Senior

"A fetch failure means a shuffle block was unreadable, and retries only help if the block is still there and the failure was transient. So the first question is whether the executor holding it is alive. I'd open the Executors tab, find deaths around that timestamp, and read the dead executor's log — if it was OOM-killed, then no retry count fixes it, and worse, each fetch failure forces Spark to re-run the map stage, so we'd be paying that re-execution repeatedly. I'd also check the shuffle block sizes: if the max is orders of magnitude above the median, transfers are timing out because of skew, and the fix is partition sizing, not retry policy. Raising retries is a legitimate resilience setting on a genuinely lossy network — but as the first response to a fetch failure it converts a fast failure into an expensive one and destroys the evidence."

"A PySpark job keeps getting its executors killed with exit code 137. The heap dump shows the JVM heap was only 40% used. What is happening?"

Weak

"Memory leak — increase executor memory."

Good

"It's a container kill, not a heap OOM. Probably memory overhead — I'd increase memoryOverhead."

Senior

"Exit 137 with a half-empty heap means the container limit was hit by something outside the JVM heap, and in PySpark the prime suspect is the Python workers: there is one worker process per concurrent task, so an executor with 8 cores can be running 8 Python processes, each holding its own copy of whatever the UDF materialises. I'd check whether we're using a pandas/Arrow UDF with a large batch size, or something that pulls a whole partition into Python. The cheapest fix is usually reducing spark.executor.cores — fewer simultaneous workers per container — which also gives each remaining task more JVM memory. Then spark.executor.pyspark.memory to make the budget explicit, and raising overhead plus the container size together if it's genuinely needed. And I'd ask whether the UDF needs to be Python at all, because moving it to a native expression removes the whole category."

"An executor shows Task Time 4.2 h and GC Time 1.9 h. Walk me through it."

Weak

"GC is high, so I'd switch to G1GC."

Good

"That's about 45% GC, which is far too high. I'd look at whether we're caching too much and reduce the heap pressure."

Senior

"45% means nearly half the executor's paid time produced nothing, and it also puts us one long pause away from a heartbeat timeout and an executor loss — so a GC problem often shows up in the incident log as a fetch failure, which is why people misdiagnose it. I'd check three things in order: is anything cached that shouldn't be, and does the Storage tab show it pinning most of the heap; are the pauses growing over the stage, which points at a live set that accumulates; and how many cores per executor, since N concurrent tasks allocating into one heap multiplies the pressure. The fixes I'd try are unpersisting, projecting fewer columns before the cache, and reducing cores per executor — all of which shrink the live set. I'd only look at collector settings after those, and only with GC logs showing that pause behaviour, not live-set size, is the limit."

"AQE skew join is enabled and you still have a 40-minute straggler task. What's going on?"

Weak

"AQE must be broken — I'd disable it and salt manually."

Good

"AQE only splits partitions that cross its thresholds. Maybe the partition isn't big enough in absolute bytes to trigger it."

Senior

"Several possibilities, and I'd check them in order from the plan. First: is the skew even in a join? AQE's skew handling targets skewed shuffle partitions in joins — it does nothing for a skewed GROUP BY group, and nothing for skew created by an explode downstream. Second: AQE needs both its relative factor and its absolute byte threshold to be exceeded, so a partition 40× the median can still be ignored if it's below the byte threshold. Third: the skew may be on the read side — one input file or storage partition that's enormous — which is before any shuffle exists to split. Fourth: if the join was already converted to a broadcast, there's no shuffle partition to split at all. I'd look at the SQL tab for AQEShuffleRead nodes and their skew metrics to see what AQE actually did, and only then decide between adjusting thresholds, salting, isolating the hot key, or fixing the file layout."

"What should spark.sql.shuffle.partitions be set to?"

Weak

"200 is the default; I usually set it to 2000."

Good

"It depends on data size — roughly enough partitions that each one is a few hundred megabytes, and at least as many as you have cores."

Senior

"It's the wrong question in a modern Spark, and I'd say why. With AQE on — the default since 3.2 — that setting is the initial partition count, and AQE coalesces down to whatever advisoryPartitionSizeInBytes targets, using the real shuffle sizes it measured. So the practical rule is: set it high enough that no single partition is oversized, then tune the advisory size, which is a size target rather than a count. The asymmetry is why — AQE coalescing merges small partitions, and AQE skew handling can split a skewed partition when it clears both the relative-factor and absolute-byte conditions, but neither rescues a uniformly under-partitioned stage where every partition is simply too big. What determines the right size is the evidence, not a number: if there's disk spill, partitions are too big for the memory each task actually has, which is executor memory divided by cores. If median task duration is in the tens of milliseconds, they're too small and scheduling overhead dominates. And I'd check whether the shuffle should exist at all before sizing it — a stale-statistics sort-merge join that should have been a broadcast is a much bigger win than any partition count."

"Your join output grew from 100 million rows to 500 million rows overnight. The code didn't change. What happened and how do you find it?"

Weak

"The source data must have grown 5×."

Good

"Probably duplicates in the dimension table — I'd check for duplicate keys."

Senior

"A clean 5× on unchanged code is almost never organic growth — organic growth isn't a round multiple. The two candidates are duplicate keys on the dimension side, or a filter that stopped applying, and I'd distinguish them in about two minutes. First, count input rows on both sides: if the fact table is flat and the dimension gained rows, it's the dimension. Then GROUP BY join_key HAVING COUNT(*) > 1 on the dimension — if the modal duplicate count is 5, that's the answer, and the likely cause is either an SCD2 table where the current-record filter was dropped, or a load that ran twice. I'd also check the physical plan against yesterday's, in case a predicate stopped being pushed down. The important part is that this is a correctness incident, not a performance incident: every downstream aggregate has been 5× too high since it started, so the response includes finding out what published on top of it. And the prevention is a uniqueness test on the dimension's key plus an alert on output-rows-per-input-row, because that ratio catches the whole class."

"A job with 500 executors is running at 10% cluster CPU and taking hours. Executors are alive but mostly idle. What's your hypothesis?"

Weak

"Add more executors so it finishes faster."

Good

"Something is bottlenecked outside the executors — maybe the driver, maybe I/O."

Senior

"Idle executors mean work isn't reaching them, so I'd stop thinking about executor capacity entirely. Three candidates, and they're distinguishable. One: the driver is the bottleneck — either it's scheduling millions of tiny tasks, or it's listing an enormous number of files before any stage starts, or the code is submitting thousands of small jobs in a loop. I'd check the Jobs tab for job count and the stage task counts, and look for gaps in the timeline where no stage is active. Two: it's I/O-bound — tasks are running but spending their time waiting on object storage, which shows up as low CPU time relative to task duration. Three: it's a dependency stall — one long-running task or a broadcast being built while everything else waits. Adding executors helps none of these and makes the first one measurably worse, because more executors means more tasks for the same driver to track."

"Someone replaced a SQL CASE expression with a Python UDF because it was 'more readable.' Runtime went from 12 minutes to 3 hours. Explain precisely why."

Weak

"Python is slower than Scala."

Good

"Python UDFs serialize every row out to a Python process, which is much slower than a native expression."

Senior

"Two separate costs, and the second one is usually bigger than people expect. The first is mechanical: a row-at-a-time Python UDF moves every row out of the JVM into a Python worker process and back, so you pay serialization per row and you lose whole-stage code generation for that part of the plan — the fused loop is cut into pieces around the UDF node. The second is optimizer-level: the UDF is a black box, so Catalyst can't push a filter through it, can't eliminate it for rows that don't matter, and can't reorder around it. If that CASE was previously letting a predicate get pushed down into the scan, replacing it may also have turned a pruned read into a full scan — which is a much larger regression than the serialization. I'd confirm from the plan whether PushedFilters changed, and I'd fix it by putting the logic back in SQL. If the logic genuinely can't be expressed natively, a pandas UDF fixes the transfer cost but not the optimizer barrier, so I'd also make sure the filtering happens before it."

"A table has 150 GB in 50,000 files. Reading it takes 25 minutes; the same data in 300 files takes 4. Where does the 21 minutes go?"

Weak

"Small files are slow — Spark has to open more of them."

Good

"Mostly file-open overhead and metadata reads — 50,000 opens with object-store latency each, plus a lot of tiny tasks."

Senior

"It's four separate costs and I'd expect them in roughly this order. First, listing: the driver has to enumerate 50,000 objects before any task starts, and that's serial-ish latency you can watch as a gap before the first stage. Second, per-file open and footer read: Parquet metadata cost scales with file count, not data size, so you pay 50,000 footer reads for the same 150 GB. Third, compression and encoding efficiency: 3 MB files have tiny row groups, so dictionary encoding and run-length encoding barely work and you physically transfer more bytes. Fourth, scheduling: with file-count-driven partitioning you get thousands of tasks each doing milliseconds of work, so per-task overhead becomes a real fraction, and the driver is tracking all of it. There's a fifth, subtler one — per-file statistics over 3 MB are too coarse-grained to skip anything, so predicate pushdown stops helping. The fix is compaction plus fixing whatever writer produces that layout, and I'd raise openCostInBytes as an interim measure so Spark packs more small files into each task."

"You add .coalesce(10) before writing to reduce the number of output files. Runtime goes from 8 minutes to 90. Why?"

Weak

"Ten partitions isn't enough parallelism for writing that much data."

Good

"coalesce reduces parallelism for the whole stage, not just the write, so the computation is now running with 10 tasks."

Senior

"coalesce is deliberately shuffle-free, which means it can't introduce a stage boundary — so the reduced partition count propagates backwards through every narrow dependency in that stage. Whatever transformation chain feeds the write is now running 10-way instead of at full parallelism, and if there's any per-partition memory pressure you've also multiplied the data each task holds by two hundred, so you may now be spilling as well. I'd confirm it in the UI: the final stage will show exactly 10 very long tasks. The fix depends on intent — if I want fewer output files and I'm willing to pay a shuffle, repartition(10) creates a boundary so the upstream work stays parallel; if I want even output sizes without guessing a number, a REBALANCE hint lets AQE size them against the advisory target. And I'd question the premise: 10 files for a large write may itself be too few, since output file size is a band, not a minimum."

"An engineer adds .cache() to a DataFrame and the job gets 30% slower. How is that possible?"

Weak

"They must have cached the wrong DataFrame."

Good

"Caching takes memory from execution, and if the dataset doesn't fit, blocks get evicted and recomputed anyway."

Senior

"Three costs, any of which can exceed the benefit. First, storage and execution share one memory pool, so pinning cache blocks reduces what joins and aggregations can use — if the job was near the edge, it now spills, and spill is a write and a read per byte. Second, cached blocks are long-lived by construction, so they sit in the old generation and every major collection has to scan them; a job that was at 5% GC can climb into the double digits. Third, if it doesn't fit, you get the worst of both worlds — evicted blocks are recomputed on access, so you paid the caching cost and still do the work. I'd check the Storage tab for the fraction actually cached and the size in memory, check whether spill appeared where there was none before, and check GC time. And I'd ask the diagnostic question first: is this dataset read more than once at all? A cache on a dataset used once is pure loss, and that's the most common version of this bug."

"You double the executor count and runtime improves by 8%. What does that tell you?"

Weak

"We need to double it again."

Good

"The job isn't parallelism-bound — something else is the bottleneck, maybe skew or I/O."

Senior

"It tells me the job is not limited by available slots, and it tells me I just doubled the cost for 8%. The candidates: a single stage where the wall clock is set by one enormous task, so extra slots have nothing to do; a serial phase — file listing, plan compilation, the write commit — that doesn't parallelise at all; a downstream bottleneck like object-store throughput on one prefix; or the driver being the constraint, in which case more executors actively hurt. I'd confirm by looking at slot utilisation over time in the Executors tab: if a large fraction of slots are idle during the slow stage, adding capacity was never going to help, and the right response is to revert the change and go find what the stage is actually waiting on. I'd also record the executor-hours for both runs, because 'improved 8%' and 'cost 92% more' are the same sentence."

"5,000 tasks. 4,998 finish in 45 seconds. Two run for 40 minutes. What do you investigate?"

Weak

"Increase executors so the slow ones have more resources."

Good

"Check the slow tasks — probably data skew. I'd look at the join key distribution."

Senior

"First I'd decide whether it's skew or a straggler, because the fixes are unrelated and the discriminator is one column. I open the task table, find those two tasks, and compare their input and shuffle-read bytes to the median. If they read a hundred times more data, it's skew: duration scales with work, and I go find the hot key with a GROUP BY … ORDER BY COUNT(*) DESC, check for sentinels like -1 and NULL, see whether AQE skew handling engaged, and then choose between filtering invalid keys, isolating hot keys, or salting. If they read a normal amount and simply took longer, it's environmental: I check whether both tasks ran on the same executor or host, look at that executor's GC time, and check host disk and network metrics — a degraded disk or a throttled instance produces exactly this. Speculative execution is the right hedge for the second case and useless for the first, which is a good way to state the difference. And the reason this matters is economic: 4,998 slots sat idle for 39 minutes waiting for two tasks, so the cluster was doing almost nothing while being fully billed."

"A job fails after four hours with repeated stage retries and fetch failures. It normally takes 50 minutes. Where do you start?"

Weak

"Increase the number of stage attempts so it can recover."

Good

"Repeated stage retries mean map output keeps being lost — I'd look for executors dying."

Senior

"Four hours for a fifty-minute job with repeated stage attempts is a retry storm, and the important insight is that everything after the first failure is a consequence — so I'd go find the first one and ignore the rest. Shuffle output isn't replicated; it lives on the producing executor's local disk. So each time an executor is lost, Spark must re-run the map tasks that produced its blocks, and if the underlying cause is still present, the re-run kills another executor and the cycle compounds. I'd sort the Executors tab by removal time, take the earliest death, and read that executor's log and exit code: 137 means a memory kill, a disk-full message means local scratch, a clean SIGTERM means preemption or decommissioning. Whatever it is, that's the fix. Raising stage attempts would make the storm last longer and cost more before failing. If the root cause is spot reclamation I'd also ask whether an external shuffle service or shuffle tracking is available, because that decouples shuffle data from executor lifetime and removes the whole cascade."

"Give me three SQL changes that would speed up a slow Spark query, and tell me how you'd prove each one worked."

Weak

"Add more filters, avoid SELECT *, and use broadcast joins."

Good

"Project only needed columns to cut scan and shuffle bytes; make sure the partition filter is on a bare column so pruning works; and pre-aggregate before joining so we shuffle less."

Senior

"I'd order them by which of the three bills they cut, and name the metric that proves each. Cut bytes scanned: project explicit columns and keep the partition predicate on a bare, correctly-typed column — proof is ReadSchema, a populated PartitionFilters, and input bytes dropping from table size to partition size. Cut bytes shuffled: pre-aggregate before the join, or restore a broadcast that stale statistics turned into a sort-merge — proof is shuffle write bytes for that stage and the join node changing in the plan. Cut bytes written: make the job incremental so it touches one partition rather than rewriting history — proof is output bytes and file count. In every case I'd capture the before numbers first, change one thing, and compare, because the failure mode in this work is making three changes and being unable to attribute the improvement to any of them."

"A pipeline that ran in 40 minutes for a year suddenly takes 4 hours. No deploy, no config change, no cluster change. Your first three checks?"

Weak

"Check the Spark UI for a slow stage."

Good

"Compare input volume against previous runs first, then look at the slowest stage, then the plan."

Senior

"With nothing changed on our side, the change is in the data or in the environment, so I'd check those before opening the UI at all. One: input bytes and input rows versus the same weekday last week — a source that reprocessed history, or a duplicate load, explains everything instantly and is not a Spark problem. Two: output rows divided by input rows — if that ratio moved, a join fanned out, which means we also have a correctness incident and everything published since is wrong. Three: input file count and bytes-per-file — if someone changed the upstream writer, we may be reading the same data through fifty times more files, which is a pure overhead event. Only if all three are flat do I go to the Spark UI, and then I'd diff the physical plan against a previous run, because with no code change the remaining candidate is a plan change from stale or refreshed statistics. Cluster-side, I'd also confirm we actually got the executors we asked for — spot reclamation or a quota change can halve effective capacity silently."

"You cut a pipeline's runtime by 40%. Your manager asks whether it was worth it. What do you say?"

Weak

"Yes — it's 40% faster."

Good

"Depends on the cost. I'd check whether executor-hours went up, because a 40% speedup from tripling the cluster isn't a win."

Senior

"I'd answer with three numbers rather than one. Executor-hours before and after, because that's the cost; cost per TB processed, because that normalises for data growth and is the number that should trend down over the year; and whether the SLA was ever at risk, because if the job finished at 04:00 against an 06:00 deadline, the 40% bought nothing anyone can use. The version of this I'd actually be proud of is the one where runtime improved and executor-hours fell — that means I removed work rather than bought parallelism, which usually means a plan or layout fix rather than a sizing change. And I'd say what I'd do with the headroom: if we now finish 90 minutes early with no consumer waiting, the follow-up is to shrink the cluster until we're comfortably inside the SLA and bank the difference."

"You changed five Spark configs and the job got 30% faster. What's the problem?"

Weak

"Nothing — it's faster."

Good

"You don't know which change helped, so you can't undo the ones that didn't."

Senior

"Three problems, in increasing order of seriousness. The obvious one is attribution: with five variables you can't say which mattered, so you carry four unexplained settings forever and nobody dares remove them. The second is that some of those changes may have hurt, and the one big win is masking them — so the real available improvement might have been 50%, not 30%. The third is the one that bites later: a config chosen without a mechanism doesn't survive a change in the data. If I raised memory and the real problem was skew, the fix works until the hot key grows 20% and then fails at 3 a.m., and the runbook says 'increase memory' because that's what worked last time. The discipline that avoids all three is cheap: baseline, one variable, predict what should move, measure the metric that confirms the mechanism rather than just the runtime, then document why. And on Spark specifically I'd re-run before believing a small delta, because run-to-run variance from cluster state and upstream data is real."

"How do you approach a Spark job you've never seen that is failing?"

Weak

"Read the error and fix it."

Good

"Find the exception, work out whether it's driver or executor, then dig into the relevant tab."

Senior

"Three questions before any code. Did the application start at all — if there's no application ID, it's orchestration, provisioning or packaging and Spark is irrelevant. If it started, is the failure driver-side or executor-side, because those share almost no root causes. Then I read the exception chain from the bottom up and keep descending past Spark internals until something names a resource or a value — heap, container limit, disk, a file, a column. If the chain bottoms out in a fetch failure or an executor loss, the real cause is on another host and I go read that executor's log. I also capture the application ID, the input volume and the historical runtime at the start, because I'll need them to prove whatever I do next actually helped."

"What is the difference between repartition and coalesce, and when has that difference hurt you?"

Weak

"repartition increases partitions, coalesce decreases them."

Good

"repartition does a full shuffle and gives even partitions; coalesce avoids the shuffle by merging, so it's cheaper but uneven."

Senior

"The mechanical difference is the shuffle, but the operational difference is that coalesce can't create a stage boundary — precisely because it doesn't shuffle. So reduced parallelism propagates backwards through every narrow dependency in that stage. The classic injury is .coalesce(10) added before a write to reduce file count: the write does produce 10 files, and the entire upstream computation now runs with 10 tasks. I've seen that turn eight minutes into ninety. You spot it in the UI as a final stage with exactly N very long tasks. If I want fewer output files and I'm willing to pay for the boundary, repartition works; if I want even sizes without picking a number, a REBALANCE hint lets AQE target a size instead."

"AQE is on by default. What does that actually change about how you tune?"

Weak

"It optimises things automatically so you don't have to tune."

Good

"It coalesces shuffle partitions, can convert joins to broadcast at runtime, and can split skewed partitions — so hard-coded partition counts matter less."

Senior

"It moves the unit of tuning from a count to a size, and it moves the plan I should be reading from the static EXPLAIN to the SQL tab. Practically: I set the initial shuffle partition count generously, because AQE can merge partitions but cannot split ones that were never created, and then I tune the advisory partition size, which is the actual dial. I stop trusting EXPLAIN output as a description of what ran, because AQE re-plans stage by stage using measured sizes. And I learn what AQE will not do: its skew handling targets skewed shuffle partitions in joins, so it won't help a skewed aggregation group, won't help skew created by an explode downstream, and won't help when one input file is enormous before any shuffle exists. Those are still mine to fix."

"Your job spills 4 TB to disk. Walk me from that number to a fix."

Weak

"Increase executor memory until it stops spilling."

Good

"Spill means partitions don't fit in execution memory — I'd increase the partition count so each one is smaller."

Senior

"I'd first establish whether it's uniform or concentrated, because that changes everything. If max task input is close to median, every partition is too big and the fix is arithmetic: per-task execution memory is roughly heap × memory.fraction ÷ cores, so I compute that, compare it to median partition size, and raise the partition count until the ratio is comfortable — or halve the cores per executor, which doubles per-task memory for free. If spill is concentrated in a few tasks, it's skew and partition count won't help; I go find the hot key. Either way the confirming metric is spill going to zero, not runtime falling — runtime can improve for unrelated reasons and I'd rather know I fixed the mechanism. I'd also check local disk headroom, because sustained spill is what fills scratch volumes and turns a memory problem into an executor-loss cascade."

"How do you find out whether a join is broadcasting, and what would you do if you wanted it to and it isn't?"

Weak

"Add a broadcast hint."

Good

"Check the plan for BroadcastHashJoin. If it's not there, either the table is over the threshold or statistics are stale — I'd run ANALYZE TABLE."

Senior

"I check the SQL tab of the finished run rather than a static EXPLAIN, because AQE can promote a join to broadcast after measuring the real shuffle output — so the final plan is the only authority. If it's a sort-merge join and I think it shouldn't be, I look at the build side's actual size metric versus the threshold. Two common gaps: statistics are missing so the estimate came from a raw footprint much larger than reality, or the estimate is of the in-memory representation, which for compressed columnar data is substantially bigger than the file size people quote. Fixing statistics is the right first move because it lets the optimizer decide correctly everywhere, not just here. I'd reach for a hint only as a last resort and I'd pair it with a size assertion, because a hint overrides the safety check — that's how tables that quietly grow end up taking down the driver."

"When is spark.sql.autoBroadcastJoinThreshold = -1 the right setting?"

Weak

"Never — broadcasting is always faster."

Good

"When broadcasts are causing driver memory problems, disabling them can stabilise the job."

Senior

"It's a blunt instrument and I'd treat it as a diagnostic before treating it as a fix. Setting it to −1 disables auto-broadcast entirely, which is genuinely useful in two situations: as an experiment, to confirm that broadcasts are the cause of driver instability; and in an environment where estimates are systematically unreliable — no statistics, heavily filtered build sides — so the optimizer keeps choosing broadcast for things that aren't small. The cost is that every join in the application now shuffles, including the ones where broadcast was correct and valuable, so it usually trades one incident for a permanent tax. The targeted alternatives are fixing statistics, lowering the threshold rather than disabling it, or using a MERGE/SHUFFLE_HASH hint on the specific offending join."

"A stage has 200 tasks and the cluster has 800 slots. What is wrong and what isn't?"

Weak

"Nothing — 200 tasks is the default."

Good

"We're using a quarter of the cluster. The shuffle partition count is probably at the default of 200 and should be higher."

Senior

"Two things are true. Structurally, 600 slots are idle for the duration of that stage, so we're paying for four times the capacity we're using — that's the cost finding. But whether it's a performance problem depends on the partition size: if each of those 200 partitions is small and the stage finishes in twenty seconds, raising the count buys nothing and adds scheduling overhead. What tells me it's a real problem is spill, or task durations long enough that the idle capacity represents lost wall clock. I'd also check whether the 200 is the default shuffle partition count or the result of AQE coalescing — those look identical in the task table and mean opposite things. If AQE coalesced down to 200 because that hits the advisory size, the system is working correctly and the right lever is the advisory size, not the count."

"Explain what happens physically when Spark shuffles."

Weak

"Data moves between executors over the network."

Good

"Map tasks partition their output by key and write it to local disk; reduce tasks fetch the blocks for their partition from every map task and combine them."

Senior

"Map side: each task partitions its rows by the target partition id, sorts within partitions, serializes and compresses, and writes a data file plus an index file to local disk. Reduce side: each reduce task fetches its partition's block from every map executor, buffering in execution memory and spilling if it doesn't fit. So one shuffle costs CPU for serialization and sorting, memory for buffers on both sides, local disk for the whole volume written and read, and network for every byte crossing once. The consequence people underrate is durability: shuffle output isn't replicated, it lives on the producing executor's disk, so losing that executor means re-running the map stage that produced it. That's why a single memory-killed executor can cascade into hours of re-execution, and why an external shuffle service changes the economics — it decouples shuffle data from executor lifetime."

"Your job reads a 500 GB table but only needs one day of it. How do you confirm pruning is working?"

Weak

"The WHERE clause handles it."

Good

"Check the input bytes in the Spark UI — they should be far less than 500 GB."

Senior

"Input bytes is the outcome; I want the mechanism too, because there are three different things that could be doing the reduction and they fail independently. In EXPLAIN FORMATTED or the SQL tab I look at the scan node for three fields: PartitionFilters should carry the date predicate — that's directory-level pruning and it's the big one; PushedFilters should carry any other predicates that can be evaluated against row-group statistics; and ReadSchema should list only the columns I actually use. Then I check the metrics: number of partitions read, number of files read, and bytes read. If PartitionFilters is empty, something wrapped the partition column in a function or compared it to the wrong type, and I'd fix that before anything else, because it's usually a one-character bug with a very large price."

"When would you deliberately increase the number of shuffle partitions even though it creates more tasks?"

Weak

"When the job is slow."

Good

"When partitions are too big and spilling — smaller partitions fit in memory."

Senior

"Three situations. One: spill — each partition exceeds per-task execution memory, and more partitions is the direct fix. Two: idle capacity — fewer tasks than slots means some of the cluster does nothing regardless of how fast each task is. Three, and this is the AQE-era reason: I set the initial count high deliberately. AQE coalescing merges small partitions, and its skew handling can split a partition that qualifies as skewed, but neither helps when every partition is uniformly oversized — so a generous starting count is cheap insurance and a stingy one is a ceiling. The cost I'm accepting is scheduling overhead per task, more shuffle blocks — M×R grows with R — and potentially more output files. So I'd pair it with a REBALANCE before the write if output file count matters, and I'd watch median task duration: once it's in the tens of milliseconds, I've gone too far."

"Why might a job fail in production but succeed on a 1% sample in development?"

Weak

"Production has more data, so it needs more memory."

Good

"Scale exposes problems the sample doesn't have — skew, memory limits, broadcast thresholds."

Senior

"The interesting answer is that it's usually a threshold being crossed, not a gradual scaling, so the failure is discontinuous. Candidates I'd check in order: the build side of a join crossed the broadcast threshold, so the plan is now a completely different physical plan — a 1% sample and full data can produce different join strategies, which means dev never tested the code path that fails. Skew is invisible at 1%: a key that's 34% of production may be absent or tiny in a sample. Memory is per-partition, so a sample's partitions fit and production's don't. Cardinality effects like fan-out multiply, so a 40× fan-out on 1% is still small. And in PySpark, worker memory scales with concurrency and batch size, not with total data, so a sample may never hit the container limit. The practical response is to compare the two plans, not just the two volumes — a plan difference explains most of these instantly."

"What's the difference between spark.default.parallelism and spark.sql.shuffle.partitions?"

Weak

"They're basically the same thing."

Good

"default.parallelism applies to RDD operations; sql.shuffle.partitions applies to DataFrame and SQL shuffles."

Senior

"They govern different code paths and that trips people up in a specific way: someone sets spark.default.parallelism, sees no change in a DataFrame job, and concludes the setting 'doesn't work'. In the SQL/DataFrame world the shuffle partition count comes from spark.sql.shuffle.partitions, defaulting to 200 in Apache Spark, and with AQE on that's the initial count before coalescing. spark.default.parallelism is the RDD-level default and in cluster mode defaults to the total number of cores. Almost all modern pipeline code is DataFrame or SQL, so the SQL setting is the one that matters — but if you have any RDD operations mixed in, or a library that drops to RDDs, both are live at once, which is worth knowing before you spend an hour confused."

"How would you reduce a pipeline's cost by 40% without breaking its SLA?"

Weak

"Use smaller instances."

Good

"Find the headroom between finish time and SLA, then shrink the cluster and remove wasted work like re-reads and unnecessary shuffles."

Senior

"I'd separate cost into work I can delete and capacity I can stop buying, and attack the first one first because it improves both axes. Work I can delete: bytes scanned — restore pruning, project columns, stop re-reading the same source in three subqueries; bytes shuffled — broadcast what should broadcast, pre-aggregate before joins, remove defensive repartitions; bytes written — make it incremental instead of a full recompute; and re-executed work — any stage attempt above one is paid for twice. Capacity I can stop buying: look at slot utilisation over time, because most clusters are sized for their widest stage and then idle through a long tail, so fixing the tail's skew often lets me cut the cluster substantially with almost no runtime change. Then I'd check the SLA is real — a job finishing at 03:00 against an 06:00 deadline has three hours of headroom that's currently being spent on nothing. I'd report it as executor-hours and cost per TB, not runtime, because those are the numbers that stay honest."

"An executor keeps dying. Give me your classification procedure."

Weak

"Increase memory — it's probably OOM."

Good

"Check whether there's an OOM in the executor log, and check the exit code to see if the container was killed."

Senior

"I want to end up in exactly one of four buckets and each has a different fix. First: does the executor's own log end with a stack trace or mid-sentence? A stack trace means the JVM threw — if it's a heap OOM I go to the skew-versus-uniform question. Ending mid-sentence means it was killed from outside, and then the exit code discriminates: 137 is a memory kill by the container runtime, 143 is a SIGTERM which usually means decommissioning or preemption. Second: is it always the same host? Then it's hardware or a noisy neighbour, and the fix is draining the node, not tuning Spark. Third: did several executors die simultaneously? That's an infrastructure event — spot reclamation, a node group replacement — and the response is architectural, like shuffle tracking or an external shuffle service so we don't lose map output. Fourth: is it PySpark with high cores per executor? Then Python worker memory is the leading hypothesis regardless of what the heap looks like. I'd get to the right bucket in a couple of minutes and I'd resist doing anything before I have."

"What does spark.memory.fraction do and when have you changed it?"

Weak

"It controls how much memory Spark uses. I'd increase it to give Spark more."

Good

"It's the share of usable heap given to unified execution-plus-storage memory, defaulting to 0.6 in Apache Spark. The rest is user memory for your own objects and Spark internals."

Senior

"It splits the usable heap between the unified execution-and-storage pool and everything else — your objects, closures, Spark's own structures. Default 0.6 in Apache Spark, and I've almost never changed it, which I think is the correct answer. Raising it gives joins and aggregations more room, but it takes that room from user memory, and if you're running code that materialises objects — UDFs, typed lambdas, big lookup structures — you can turn a spill into an OOM, which is a strictly worse failure. The reason it's rarely the right knob is that the things it competes with are almost always fixable more directly: smaller partitions, fewer cores per executor, less caching, fewer materialised objects. If I did change it I'd want a specific hypothesis — 'this stage is execution-memory-bound and user memory is provably underused' — and the confirming metric would be spill falling with no new OOM, measured over several runs."

"How do you detect skew without opening the Spark UI?"

Weak

"You can't — you need the UI."

Good

"Profile the key distribution with a GROUP BY … ORDER BY COUNT(*) DESC and look for dominant values."

Senior

"Two approaches and I'd want both. From the data: profile the join or group key with counts and percentage-of-total, and also with bytes per key rather than just rows, because a key with fewer but much wider records can be the expensive one. I specifically look for sentinels — 0, −1, empty string, 'UNKNOWN' — and for NULL, since NULLs all hash together and in an inner join are shuffled and then discarded, which is pure waste. From the run: pull per-stage task quantiles from the Spark REST API rather than the UI, compare shuffle-read bytes at p50 against the max, and store that ratio with every run. That turns skew detection into a monitored metric instead of a manual investigation, which is the difference between finding it during an incident and finding it in a dashboard the morning it starts."

"What would you put in a Spark pipeline's runbook?"

Weak

"How to restart it."

Good

"Restart procedure, the SLA, who owns the upstream data, and the known failure modes with their fixes."

Senior

"The runbook's job is to make an incident boring for someone who has never seen this pipeline. So: what it does and who consumes it; the SLA and what happens if it's missed; the baseline numbers — median runtime, input bytes, shuffle bytes, executor-hours — because triage is comparison and without a baseline there's nothing to compare to; the collection checklist, meaning the exact list of things to capture before changing anything, with the REST API calls to get them; known failure modes with their evidence signatures and fixes, written as 'if you see X in the stage metrics, it's Y'; every non-default configuration with the reason it exists and an expiry date; the upstream dependencies and their owners; and explicitly, what not to do — the knee-jerk fixes that have hurt this pipeline before. That last section is the most valuable and the one nobody writes."

"What's the most important thing you've learned about tuning Spark?"

Weak

"Always enable AQE and use broadcast joins where possible."

Good

"Find the bottleneck before changing anything, and change one thing at a time."

Senior

"That most Spark problems are not Spark problems. The genuinely expensive incidents I've worked came from data — a sentinel key that grew, a dimension that duplicated, an upstream writer that changed file sizes, a source that reprocessed history — and every one of them presented as 'Spark is slow'. So the habit that pays is capturing volume and distribution metrics per run, because they turn a day of Spark UI archaeology into a one-query answer. Second to that: optimise the query and the data layout before you scale resources, because layout fixes compound and capacity fixes just recur at a higher price. And third, tune by evidence, not folklore — every config I keep has a sentence next to it saying which measurement justified it, and the ones that can't produce that sentence get removed."