What to Ask Software Engineers Instead

Part 1 cleared the table. The short version: algorithm rounds, stack filters and syntax trivia have lost most of their signal, because AI either automated the skill or made the test gameable. The timed bug fix was the one exception — there the exercise survives and only the stopwatch has to go.

That is interview drift: the distance between what a process tests and what the job requires. Clearing it out is the easy half. The question is what goes back on the table.

The replacement exercises are not harder or more exotic. They are more honest about what the job now is: specify what should be built, work with a tool that will build it, evaluate whether what came back is correct, and know when something is wrong even if you cannot immediately fix it.

One thing to settle before any of it. Let candidates use AI, and say so at the start. Part 1 made the point that banning it is unenforceable in a remote interview. The exercises below go further — several of them are better when the candidate uses AI, because what you are assessing is the judgement wrapped around the tool, not whether they can outrun it. An interview a candidate can pass by quietly using AI is a bad interview. An interview that hands them AI and still separates people is the one you want.

1. Before they touch a tool

Give a candidate a vague requirement. "Build a notification system." Then stop talking.

What happens in the next thirty seconds tells you a lot.

The strong candidate starts asking questions. What triggers a notification? Does the user configure frequency? What happens if delivery fails — do we retry, and for how long? Is there a preference for in-app, email or push? They are building a mental model of the problem before they build anything else.

The weak candidate asks which framework to use.

The specification is now the primary engineering artefact. When we generate code with AI, the quality of the output is almost entirely determined by the quality of the input. A good specification produces usable code. A vague prompt produces plausible-looking code that fails in ways the developer did not anticipate, because they did not anticipate the edge cases in the first place.

An engineer who cannot specify before implementing is an engineer who feeds vague prompts into AI tools and ships whatever comes back. We have all seen what that looks like six months later.

The exercise is simple: give them a two-sentence requirement and ask them to write down what done looks like, in plain language, before writing any code. You are not grading their prose. You are watching whether they slow down, ask questions, identify ambiguity and name edge cases — or whether they immediately try to build something.

One variation that works well: ask them to identify what is missing from the requirement. A candidate who can tell you what information they would need before starting has already shown more useful judgement than a candidate who can solve a LeetCode hard.

2. Watch them build with it

This is the exercise most processes skip, and on our own scoring it is the highest-validity thing in the whole interview.

Hand them the specification they just wrote and twenty minutes. "Implement the core logic. Use AI freely." Then watch.

You are not grading the output. The output will mostly be fine — that is the entire point of the era. You are grading the process, which is directly visible and cannot be rehearsed:

What do they put in the prompt? Do they carry their own constraints across, or do they paste the two-sentence requirement and hope?
When the first result comes back, do they read it, or do they run it?
When it is wrong, how do they decide what to change? Do they adjust the specification, or do they re-roll the prompt and hope for a different answer?
What do they add that the tool did not produce? The test that matters is usually the one the AI did not think to write.

This is as close as an interview gets to the actual job, which is why the selection research likes it. Work samples and observed work trials have been the best-evidenced hiring methods for decades — the 2022 re-analysis by Sackett and colleagues revised the older effect sizes downward across the board, but it left the ranking intact: watching someone do a realistic version of the work still beats almost everything else you could do with the same hour.

Two things to watch for. A candidate who never opens the AI tool is telling you something. So is a candidate who never reads what it gave them.

3. Reading what came back

The third exercise uses code you provide: AI-generated, forty to sixty lines, with three real problems planted in it. Use the same code and the same three problems for every candidate.

The instruction: read this. Tell me what you would change before shipping it.

You are not asking them to fix it. You are watching how they read.

Most engineers default to style review. They will spot variable names they would choose differently, formatting inconsistencies, a missing comment. That is fine, but it is not what you are looking for.

What you are looking for is behavioural correctness. Do they find the thing that will fail in production? The off-by-one on the retry counter that means retries never actually happen. The edge case where the input is empty and the function returns silently without an error. The assumption baked into the logic that is true in the test environment and false in production.

AI output looks clean. It is almost always syntactically correct and superficially reasonable. This is the entire problem. An engineer who reviews AI output for style is providing no protection at all. Style is the thing AI gets right. Correctness in context is the thing it frequently gets wrong, and only someone with domain knowledge and genuine attention will catch it.

Because the code and the planted problems are fixed, this exercise gives you the one thing the old process never had: a directly comparable number. Three planted problems, and you know how many each candidate found and how long it took them. The spread across candidates is much wider than most panels expect.

4. How does this fail?

Describe a system — a payment retry queue, an email sending service, a user permissions cache. Then ask: how does this fail?

Not "what bugs could it have." What are the failure modes.

Strong candidates go somewhere interesting quickly. They find the silent failures: the case where the queue processes successfully but the payment never lands, and nothing raises an alert. They find the partial failure — the email that sends to nine of ten recipients and silently drops the tenth. They find the failure that only surfaces at three times normal load, after four weeks in production.

And then, if they are very good, they ask the detection question: how would we know if this was failing?

That question separates engineers who prevent problems from engineers who fix them. The failure mode analysis is valuable. The detection analysis is rarer, and more valuable still. Building a system you can observe is at least as important as building one that is correct, because no system stays correct forever and you need to know when it drifts.

This one runs as a conversation. No code required. Five minutes on a system description tells you whether someone's mental model extends beyond the happy path.

5. How do they debug?

Give them something broken and watch how they start.

Part 1 made the case for keeping this exercise and throwing away the stopwatch. This is what to do with it instead.

On the debugging axis, engineers sort into two behaviours. The first forms a hypothesis: they look at the symptom, reason about what could produce it, identify the part of the system most likely responsible, and test that first. The investigation has a shape.

The second opens an AI tool and describes the symptom. They wait. They try whatever comes back. They describe the next symptom. They wait again.

The second approach is not debugging. It is prompt-throwing. It produces results occasionally, by accident, and nothing useful when the problem is subtle or systemic. More importantly, it reveals the absence of a mental model. The engineer does not have a theory about what is wrong — they are outsourcing the theory formation to a tool that knows nothing about this system, this context or this production environment.

This is the same split we described from the engineer's side as the difference between dumping on AI and steering it. In an interview it surfaces faster than anywhere else.

AI generates more code faster, which means more bugs faster. The bottleneck has moved from fixing bugs to finding them, and finding them requires understanding the system well enough to reason about how it could break. An engineer who cannot form a hypothesis cannot debug effectively no matter how many tools they have.

Watch the first two minutes. A hypothesis in the first two minutes is a strong signal. An immediate reach for the AI tool is a different one.

What these five exercises share

They all test the same underlying thing: does this engineer have a mental model of the problem they are working on?

The specification exercise tests whether they build the model before they build. The observed build tests whether they hold onto it while a tool writes for them. The code review tests whether they can read against a model of correct behaviour. The failure analysis tests whether the model extends to what can go wrong. The debugging exercise tests whether they reason from it or abandon it at the first sign of trouble.

This is the skill the old process did not test. Not because it was not important — because it was assumed. When implementation was the bottleneck, the engineer who could implement fast was the valuable one, and the model was a nice-to-have. Now the model is the job.

What this costs to run

Part 1 argued that the volume filter and the quality signal are different jobs, and that the mistake is treating one as the other. Here is where these five sit.

None of them belong at the top of an eight-hundred-applicant funnel. Two of them can be adapted to it: the specification exercise and the code review both work asynchronously, both grade against a fixed rubric, and neither has a public answer bank to prepare against.

The rest are stage-two, and the honest budget is about two hours per candidate: twenty minutes specifying, twenty building, twenty reviewing, and a half hour of conversation across failure modes and debugging. That is more than a ninety-minute panel and it should be. You are spending it on far fewer people, and you are spending it on the thing that actually predicts the job.

These exercises take longer to evaluate as well as to run. That is not a flaw in them. It is the nature of testing judgement. Judgement does not reveal itself in the time it takes to sort a linked list. It reveals itself when the problem is ambiguous, the input is incomplete, and the system could fail in ways nobody wrote down.

Score it the same way every time

One warning, because it is the way this goes wrong in practice.

Open-ended exercises invite unstructured judgement, and unstructured judgement is where bias and impression management do their work. It is also the version that is hardest to defend if a rejected candidate ever asks why.

The fix is the unglamorous part of the method, and it is not optional. Every candidate gets the same requirement, the same forty to sixty lines with the same three planted problems, the same system to break. Criteria are written down before anyone is interviewed. Each interviewer records a score before the group discusses, so the room does not converge on whoever spoke first.

Do that and these exercises are more defensible than the algorithm round was, not less. Skip it and you have replaced a bad measurement with no measurement.

If you want to see where these five sit relative to the old ones, the skill map plots every common interview method on two axes — relevance today, and how well the method measures it. All five cluster in the top right. The tests from Part 1 do not: most of them have slid into the top left, which is the interesting quadrant — still measured accurately, just no longer measuring anything scarce. The interactive version is at the 2026 Interview Skill Map.

There is one more category of signal the interview almost never reaches. It is the most undervalued thing we have found, and almost nobody asks for it. That is Part 3.

The Product Engineer Question Bank runs this as a four-stage interview — 17 questions, each with a construct definition, scoring rubric and red flags. It maps to the exercises above rather than matching them one for one: its Stage 1 is the specification exercise, Stage 2 the observed build, Stage 3 the output review, and Stage 4 a structured behavioural round that checks the exercises against what the candidate has actually done before. The failure-mode and debugging exercises sit in the System Engineer Question Bank. The accompanying scorecard is the one-page tool for running it and reaching a structured hire/no-hire decision.