# The AI-Era Engineering Playbook — full corpus # Source: https://aieraengineering.com — © Gabor Mayer, CC BY 4.0 ==================== # What to Ask Software Engineers Instead (https://aieraengineering.com/hiring/what-to-ask-software-engineers-instead/ — 2026-04-04) [Part 1](/hiring/what-not-to-ask-software-engineers/) cleared the table. The short version: algorithm rounds, stack filters and syntax trivia have lost most of their signal, because AI either automated the skill or made the test gameable. The timed bug fix was the one exception — there the exercise survives and only the stopwatch has to go. That is interview drift: the distance between what a process tests and what the job requires. Clearing it out is the easy half. The question is what goes back on the table. The replacement exercises are not harder or more exotic. They are more honest about what the job now is: specify what should be built, work with a tool that will build it, evaluate whether what came back is correct, and know when something is wrong even if you cannot immediately fix it. One thing to settle before any of it. **Let candidates use AI, and say so at the start.** Part 1 made the point that banning it is unenforceable in a remote interview. The exercises below go further — several of them are *better* when the candidate uses AI, because what you are assessing is the judgement wrapped around the tool, not whether they can outrun it. An interview a candidate can pass by quietly using AI is a bad interview. An interview that hands them AI and still separates people is the one you want. ## 1. Before they touch a tool Give a candidate a vague requirement. "Build a notification system." Then stop talking. What happens in the next thirty seconds tells you a lot. The strong candidate starts asking questions. What triggers a notification? Does the user configure frequency? What happens if delivery fails — do we retry, and for how long? Is there a preference for in-app, email or push? They are building a mental model of the problem before they build anything else. The weak candidate asks which framework to use. The specification is now the primary engineering artefact. When we generate code with AI, the quality of the output is almost entirely determined by the quality of the input. A good specification produces usable code. A vague prompt produces plausible-looking code that fails in ways the developer did not anticipate, because they did not anticipate the edge cases in the first place. An engineer who cannot specify before implementing is an engineer who feeds vague prompts into AI tools and ships whatever comes back. We have all seen what that looks like six months later. The exercise is simple: give them a two-sentence requirement and ask them to write down what done looks like, in plain language, before writing any code. You are not grading their prose. You are watching whether they slow down, ask questions, identify ambiguity and name edge cases — or whether they immediately try to build something. One variation that works well: ask them to identify what is *missing* from the requirement. A candidate who can tell you what information they would need before starting has already shown more useful judgement than a candidate who can solve a LeetCode hard. ## 2. Watch them build with it This is the exercise most processes skip, and on our own scoring it is the highest-validity thing in the whole interview. Hand them the specification they just wrote and twenty minutes. "Implement the core logic. Use AI freely." Then watch. You are not grading the output. The output will mostly be fine — that is the entire point of the era. You are grading the process, which is directly visible and cannot be rehearsed: - What do they put in the prompt? Do they carry their own constraints across, or do they paste the two-sentence requirement and hope? - When the first result comes back, do they read it, or do they run it? - When it is wrong, how do they decide what to change? Do they adjust the specification, or do they re-roll the prompt and hope for a different answer? - What do they add that the tool did not produce? The test that matters is usually the one the AI did not think to write. This is as close as an interview gets to the actual job, which is why the selection research likes it. Work samples and observed work trials have been the best-evidenced hiring methods for decades — the 2022 re-analysis by Sackett and colleagues revised the older effect sizes downward across the board, but it left the ranking intact: watching someone do a realistic version of the work still beats almost everything else you could do with the same hour. Two things to watch for. A candidate who never opens the AI tool is telling you something. So is a candidate who never reads what it gave them. ## 3. Reading what came back The third exercise uses code you provide: AI-generated, forty to sixty lines, with three real problems planted in it. Use the same code and the same three problems for every candidate. The instruction: read this. Tell me what you would change before shipping it. You are not asking them to fix it. You are watching how they read. Most engineers default to style review. They will spot variable names they would choose differently, formatting inconsistencies, a missing comment. That is fine, but it is not what you are looking for. What you are looking for is behavioural correctness. Do they find the thing that will fail in production? The off-by-one on the retry counter that means retries never actually happen. The edge case where the input is empty and the function returns silently without an error. The assumption baked into the logic that is true in the test environment and false in production. AI output looks clean. It is almost always syntactically correct and superficially reasonable. This is the entire problem. An engineer who reviews AI output for style is providing no protection at all. Style is the thing AI gets right. Correctness in context is the thing it frequently gets wrong, and only someone with domain knowledge and genuine attention will catch it. Because the code and the planted problems are fixed, this exercise gives you the one thing the old process never had: a directly comparable number. Three planted problems, and you know how many each candidate found and how long it took them. The spread across candidates is much wider than most panels expect. ## 4. How does this fail? Describe a system — a payment retry queue, an email sending service, a user permissions cache. Then ask: how does this fail? Not "what bugs could it have." What are the failure modes. Strong candidates go somewhere interesting quickly. They find the silent failures: the case where the queue processes successfully but the payment never lands, and nothing raises an alert. They find the partial failure — the email that sends to nine of ten recipients and silently drops the tenth. They find the failure that only surfaces at three times normal load, after four weeks in production. And then, if they are very good, they ask the detection question: how would we know if this was failing? That question separates engineers who prevent problems from engineers who fix them. The failure mode analysis is valuable. The detection analysis is rarer, and more valuable still. Building a system you can observe is at least as important as building one that is correct, because no system stays correct forever and you need to know when it drifts. This one runs as a conversation. No code required. Five minutes on a system description tells you whether someone's mental model extends beyond the happy path. ## 5. How do they debug? Give them something broken and watch how they start. Part 1 made the case for keeping this exercise and throwing away the stopwatch. This is what to do with it instead. On the debugging axis, engineers sort into two behaviours. The first forms a hypothesis: they look at the symptom, reason about what could produce it, identify the part of the system most likely responsible, and test that first. The investigation has a shape. The second opens an AI tool and describes the symptom. They wait. They try whatever comes back. They describe the next symptom. They wait again. The second approach is not debugging. It is prompt-throwing. It produces results occasionally, by accident, and nothing useful when the problem is subtle or systemic. More importantly, it reveals the absence of a mental model. The engineer does not have a theory about what is wrong — they are outsourcing the theory formation to a tool that knows nothing about this system, this context or this production environment. This is the same split we described from the engineer's side as the difference between [dumping on AI and steering it](/engineers/are-you-an-ai-dumper/). In an interview it surfaces faster than anywhere else. AI generates more code faster, which means more bugs faster. The bottleneck has moved from fixing bugs to finding them, and finding them requires understanding the system well enough to reason about how it could break. An engineer who cannot form a hypothesis cannot debug effectively no matter how many tools they have. Watch the first two minutes. A hypothesis in the first two minutes is a strong signal. An immediate reach for the AI tool is a different one. ## What these five exercises share They all test the same underlying thing: does this engineer have a mental model of the problem they are working on? The specification exercise tests whether they build the model before they build. The observed build tests whether they hold onto it while a tool writes for them. The code review tests whether they can read against a model of correct behaviour. The failure analysis tests whether the model extends to what can go wrong. The debugging exercise tests whether they reason from it or abandon it at the first sign of trouble. This is the skill the old process did not test. Not because it was not important — because it was assumed. When implementation was the bottleneck, the engineer who could implement fast was the valuable one, and the model was a nice-to-have. Now the model is the job. ## What this costs to run Part 1 argued that the volume filter and the quality signal are different jobs, and that the mistake is treating one as the other. Here is where these five sit. None of them belong at the top of an eight-hundred-applicant funnel. Two of them can be adapted to it: the specification exercise and the code review both work asynchronously, both grade against a fixed rubric, and neither has a public answer bank to prepare against. The rest are stage-two, and the honest budget is about two hours per candidate: twenty minutes specifying, twenty building, twenty reviewing, and a half hour of conversation across failure modes and debugging. That is more than a ninety-minute panel and it should be. You are spending it on far fewer people, and you are spending it on the thing that actually predicts the job. These exercises take longer to evaluate as well as to run. That is not a flaw in them. It is the nature of testing judgement. Judgement does not reveal itself in the time it takes to sort a linked list. It reveals itself when the problem is ambiguous, the input is incomplete, and the system could fail in ways nobody wrote down. ## Score it the same way every time One warning, because it is the way this goes wrong in practice. Open-ended exercises invite unstructured judgement, and unstructured judgement is where bias and impression management do their work. It is also the version that is hardest to defend if a rejected candidate ever asks why. The fix is the unglamorous part of the method, and it is not optional. Every candidate gets the same requirement, the same forty to sixty lines with the same three planted problems, the same system to break. Criteria are written down before anyone is interviewed. Each interviewer records a score before the group discusses, so the room does not converge on whoever spoke first. Do that and these exercises are more defensible than the algorithm round was, not less. Skip it and you have replaced a bad measurement with no measurement. If you want to see where these five sit relative to the old ones, the skill map plots every common interview method on two axes — relevance today, and how well the method measures it. All five cluster in the top right. The tests from Part 1 do not: most of them have slid into the top *left*, which is the interesting quadrant — still measured accurately, just no longer measuring anything scarce. The interactive version is at the [2026 Interview Skill Map](/interview-skill-map/). There is one more category of signal the interview almost never reaches. It is the most undervalued thing we have found, and almost nobody asks for it. [That is Part 3.](/hiring/the-question-youve-never-thought-to-ask/) --- *The [Product Engineer Question Bank](/product-engineer-question-bank/) runs this as a four-stage interview — 17 questions, each with a construct definition, scoring rubric and red flags. It maps to the exercises above rather than matching them one for one: its Stage 1 is the specification exercise, Stage 2 the observed build, Stage 3 the output review, and Stage 4 a structured behavioural round that checks the exercises against what the candidate has actually done before. The failure-mode and debugging exercises sit in the [System Engineer Question Bank](/system-engineer-question-bank/). The accompanying [scorecard](/product-engineer-scorecard/) is the one-page tool for running it and reaching a structured hire/no-hire decision.* --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # What You Don’t Need to Ask Software Engineers Anymore (https://aieraengineering.com/hiring/what-not-to-ask-software-engineers/ — 2026-04-14) Eighteen minutes for a LeetCode hard. Clean approach, correct complexity reasoning, narrated every step while he wrote it. The panel was impressed. We made the offer that week. Three months later we were quietly cleaning up behind him. The shape of it was always the same: a ticket came back marked done, the feature worked on the happy path, and something surfaced a week later. A retry loop that never actually retried. A total that was right in staging and wrong in production. An error that was caught and then silently swallowed. He could not explain what the code did, because he had not written it. He had been pasting tickets into an AI tool, accepting the first answer, and calling it shipped. We have seen versions of this at more than one company since. The details change. The interview does not. The interview worked exactly as designed. The design is the problem. There is a name for the gap that produced that hire. **Interview drift** is the distance between what a hiring process tests and what the role actually requires. Drift is not the same thing as a sloppy process — ours was well run, consistently scored, and staffed by good engineers. It measured carefully. It measured the wrong things. Drift builds up slowly, and then something moves the target all at once. We call that a **relevance shift**: the point at which a capability becomes abundant enough that testing for it stops telling you anything. Implementation speed crossed that line somewhere around 2022, and most hiring processes have not been re-pointed since. Four of them are worth going through in detail. ## The algorithm question The technical interview has a founding myth: that asking candidates to solve algorithmic puzzles under pressure reveals something real about how they think. It probably did, once. Translating a problem into an efficient algorithm takes structured reasoning, and that signal mattered when writing the algorithm was the bottleneck. But we should be honest about how well it ever worked. When we scored the standard methods on how accurately they measure the thing they claim to measure, the algorithm round came out at 3 out of 10 — and that was its score *before* AI. It was always a weak instrument. What changed is that it is now a weak instrument pointed at a skill that is no longer scarce. We are not going to quote a benchmark at you, because you can check this yourself in about ten minutes. Take the last three problems your panel used and paste them into whatever model your engineers already have open. When we did this with ours, they came back correct on the first attempt, with tidier complexity analysis than most candidates produce out loud under time pressure. The problems where the model still struggles are the genuinely novel ones, and those are not the problems on your list — yours came from the same few hundred that everyone practises. Which points at the real issue. Six weeks of preparation gets a candidate through this round whether or not they can build software. A candidate who passes an algorithm round today has demonstrated one thing clearly: they prepared for the algorithm round. There is a second problem that most panels talk around. Nearly every company bans AI assistance during a live interview. In a remote interview that ban is unenforceable, and everyone in the call knows it. So the round now sorts candidates on two axes: how much they prepared, and how willing they are to ignore a rule nobody can check. Neither is what you were trying to measure. The cost of keeping it is not zero. Every hour spent on algorithm questions is an hour not spent on something that predicts job performance. If you are keeping this round because it is the only thing that scales to your applicant volume, that is a fair reason, and we deal with it further down. ## Stack knowledge "Minimum 5 years of React experience." This requirement feels safe. It is concrete, it is easy to filter on, and it rules out candidates who obviously cannot do the job. It also rules out the people we increasingly need most: domain experts who move between tools, fast learners who have been through two frameworks in three years, engineers who picked up the right new thing before it was obvious enough to hire for. The useful half-life of specific framework knowledge is now roughly one to three years. Not the framework's lifetime — the lifetime of the answer. The specific APIs, the patterns, the "right way to do it in this version" all rotate underneath you, faster now that AI generates most of the boilerplate. React, which feels permanent, will end up where jQuery is. That is worth being precise about, because jQuery is still running on a large share of the web and people are still paid to maintain it. Frameworks do not disappear. They stop being where the value is. Someone with deep React knowledge remains employable — they are simply no longer scarce, and scarcity is the thing a hiring filter is supposed to find. Here is the practical version. AI generates framework code. Give a capable engineer a new framework and a model and they are productive in days, not months. What you pay a premium for when you filter on stack depth is the memorised answer to a question the job rarely asks anymore. The engineers who are genuinely valuable understand why frameworks work the way they do. They can pick up a new one. They bring judgement, not recall. A stack requirement used as a filter selects against exactly that profile. ## Syntax and trivia "Explain how garbage collection works." "What's the difference between a process and a thread?" "What does this keyword do?" This is the one category where the method was never the problem. A syntax question measures recall accurately — it scored 7 out of 10 for measurement quality in our pre-AI baseline, one of the highest on the board. It hit what it aimed at. The trouble was always what it aimed at. Recall stood in for understanding, on the theory that someone who knew the internals would be better at using the surface. Sometimes that held. It was never a tight relationship, and it is now a broken one: the information is a prompt away, and retention of it tells you nothing about judgement, specification quality, or whether someone can evaluate generated code for correctness. A well-built instrument pointed at something that stopped mattering is the cleanest case of drift there is. This category can go. It frees time for questions that cannot be answered by asking an AI. ## The timed bug fix Give the candidate a broken piece of code. Thirty minutes. Fix it. This is the one to be careful with, because it is the easiest to over-correct, and we have watched teams over-correct it. What the timer measured was speed of mechanical implementation: fast hands, pattern recognition under stress, reading unfamiliar code quickly. That is genuinely less relevant now, when a model reads code faster than any human and produces a plausible fix in seconds. What the exercise missed entirely was the part that now matters most. Does this person understand what correct looks like before they see it? Can they describe what the fix should achieve before writing it? Can they notice that the proposed fix addresses the symptom and not the cause? So the thing to remove is the stopwatch, not the exercise. Debugging is one of the few traditional technical skills whose value has gone *up* — AI writes more code faster, which produces more bugs faster, and finding them now requires a mental model that a fast producer of AI output does not have. In our own scoring, a debugging exercise sits in the strongest quadrant both before and after the shift. It is the framing that has to change: stop timing how quickly they reach a fix, start watching whether they form a hypothesis in the first two minutes or start throwing prompts at the symptom. That distinction is the whole of Part 2, so we will leave it there. ## The map Those four are not outliers. They are a pattern: methods built for a bottleneck that moved, still running on inertia. We scored every common interview method on two axes — how relevant the underlying skill is to the job today, and how well the method actually measures that skill — once for the pre-2022 world and once for now. The distance between the two pictures is interview drift, made visible.
The Interview Skill Map — the same interview methods scored against pre-2022 and 2026 skill relevance ← Pre-AI Baseline Map 2026 Skill Map →
A word on what these charts are and are not, because it matters for how much weight you put on them. The positions are our scores. They are not survey results and we are not going to present them as such. The measurement axis leans on the selection-validity research where it exists — work samples and structured interviews have decades of it behind them, and brain teasers have Google's own finding that they predict nothing at all. The relevance axis is a judgement call about 2026. We have written the reasoning for each point into its hover note, so you can disagree with a specific dot rather than with the chart as a whole. That is the more useful argument to have. The pre-2022 picture genuinely does look different. Take-home assignments, pair programming, work samples and structured behavioural interviews sat high and to the right — the right skills, measured reasonably well. Several of them still do. The four methods above are the ones that moved. If you want to score your own process rather than ours, that is what the [Interview Drift Toolkit](/interview-drift-toolkit/) is for. ## "But we screen a thousand people" This is the first objection we get, and it is the right one. The algorithm round did not survive on merit. It survived because it is cheap, runs in parallel, needs no senior time, and produces a number you can sort on. That is a real operational requirement, and any replacement has to clear the same bar. "Have a thoughtful conversation about failure modes" does not scale to eight hundred applicants, and pretending otherwise is how good advice gets ignored. Two things do clear it. A short asynchronous work sample. On our map the work sample scores highest of any method on both axes, before and after the shift. Not a four-hour take-home — a forty-minute task drawn from something your team actually shipped last quarter. It is gradeable against a fixed rubric, it does not need the candidate and the interviewer in a room at the same time, and it is much harder to prepare for than an algorithm round because there is no public bank of them. A review task with planted defects. Give everyone the same forty to sixty lines of AI-generated code with the same three problems planted in it, and ask what they would change before shipping. The objective half grades itself — did they find the three? — which is what makes it survivable at volume. The interesting half, how they read, comes later with far fewer people in the pool. Then keep the funnel honest. The expensive judgement exercises belong at stage two, against a much smaller group. The mistake is not that the algorithm round is at the top of the funnel. The mistake is treating a volume filter as though it were a quality signal. ## "But we need to defend these decisions" The second objection is quieter and usually arrives from HR rather than engineering. Standardised tests exist partly to make hiring decisions consistent and defensible. Replacing them with open-ended judgement exercises sounds like the opposite of that. It is worth separating two things that get bundled together. The risk is not judgement. The risk is *unstructured* judgement — different tasks for different candidates, no rubric, and a decision assembled afterwards out of impressions. Structured interviewing is one of the best-evidenced methods in the whole selection literature, and it is more defensible than an algorithm round, not less. The requirements are unglamorous and non-negotiable: every candidate gets the same task, scored on the same rubric, against criteria written down before anyone was interviewed, with each interviewer recording their score before the group discusses. That is what the [scorecards](/product-engineer-scorecard/) are for. Structure is what makes a decision defensible. Puzzles were never the thing doing that work. ## What this is not This is not an argument that algorithms, frameworks, syntax and debugging speed are worthless. Engineers who understand these things deeply still carry real advantages, and for some System Engineer roles the algorithm intuition is genuinely load-bearing. The question is whether a ninety-minute interview is the right place to measure them, and whether they are the scarce skills worth the weight they currently carry. The interview is a limited resource. Spending it on questions whose signal has degraded is a choice with a cost, and the cost is not spending that time on something that tells you what you need to know. ## What to change first If you do one thing after reading this, make it the smallest one. Take your current process and write down, for each round, the skill it is supposed to measure. Not the format — the skill. Then ask two questions of each line: is that skill still scarce, and does this round actually measure it? Most teams find one round that fails both tests. Remove that round before adding anything. You will free ninety minutes and lose nothing, and it is a great deal easier to get agreement on than a redesign. Then take the stopwatch off the debugging exercise and leave everything else alone until you have read Part 2. What goes in the space you just cleared? [That is Part 2.](/hiring/what-to-ask-software-engineers-instead/) --- *This article builds on an earlier argument about where the role is heading: [The Future Software Engineer Will Not Be a Programmer](https://www.linkedin.com/pulse/future-software-engineer-programmer-gabor-mayer-x32lf/). That piece was about what the job is becoming. This series is about the practical consequence, starting with which interview questions are still worth asking.* *How many of these engineers you actually need, and how teams get structured around them, is a separate question — we took it up in [Why Adding AI to Your Existing Team Structure Doesn't Work](/leaders/why-adding-ai-to-your-existing-team-structure-doesnt-work/).* **Tools referenced in this article:** the [Interview Drift Toolkit](/interview-drift-toolkit/) for scoring your current process, the [Pre-AI Baseline Map](/interview-skill-map-preai/) and [2026 Skill Map](/interview-skill-map/) for the underlying data, the [Hiring Audit Worksheet](/hiring-audit-worksheet/) for running the round-by-round audit above, and the [Product Engineer](/product-engineer-question-bank/) and [System Engineer](/system-engineer-question-bank/) question banks for what to ask instead. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # The Question You’ve Never Thought to Ask (https://aieraengineering.com/hiring/the-question-youve-never-thought-to-ask/ — 2026-04-28) Two engineers. Same years of experience, same rough technical profile, strong performances in the same interview process. You hire both. Six months in, one of them is operating with tools and practices the other hasn't started learning yet. The gap isn't talent. It's not raw intelligence. It's a habit of early adoption — a specific, observable behaviour that compounded quietly over the past year before they ever walked into your office. Most interview processes never surface it. Most job descriptions don't mention it. ## The tooling landscape now changes faster than hiring cycles Consider what's changed in the last twelve months alone. Tools that are standard practice on engineering teams today were experimental or fringe a year ago. Workflows that didn't exist eighteen months ago are now the default at some of the most effective teams. An engineer who finds useful tools before they're mainstream has a structural advantage. Not because early adoption is inherently virtuous, but because of what it produces: months of practice with something before their peers have formed an opinion on it. That compounds. Six months of real experience with a tool before it becomes mainstream is not a marginal edge. In a field moving at this speed, it's the difference between fluency and familiarity. Fluency changes what you can build. Familiarity changes what you can discuss at standup. ## What early adoption actually looks like It's important to be precise here, because the term attracts a lot of noise. Early adoption is a behaviour, not a personality trait. It doesn't mean chasing every hype cycle. It doesn't mean switching tools every three weeks or having strong opinions based on a blog post. That version is actually a liability. The real version looks like this: active but selective monitoring of what's happening at the frontier. Regular low-cost experimentation — trying things in small ways before committing to them. Forming opinions from use, not from articles. And critically, being willing to decide something isn't worth it. That last part is the distinguishing detail. A genuine early adopter has tried things that didn't pan out. They can tell you specifically why. "I tested it for two weeks on a side project, the latency was fine but the context handling was too inconsistent for our use case" is a real answer. "I heard it wasn't great" is not. ## The questions These are the ones we've found most useful. They sound simple. That's intentional. "What have you started using in the last three months that nobody told you to?" "What are you currently experimenting with?" "Tell me about something you tried and decided wasn't worth it." "Where do you find out about things before they're mainstream?" A strong answer is specific. It names actual tools. It contains opinions formed from use, not reputation. It references specific communities — not "I follow tech news" but the Discord server, the researcher's GitHub, the niche newsletter with four thousand subscribers that happened to be right about something important. The weak answer sounds like keeping up. "I try to stay current." It names tools that have been mainstream for two years. Or it lands on the most honest version: "I haven't really had time." ## Why the weak answer is a real signal "I haven't had time" is the answer worth sitting with. In a field changing at this speed, not making time to understand what's changing is not a capacity problem. It's a priority problem. The hours exist. Other things filled them. That's a choice, even when it doesn't feel like one. For roles that involve building with AI tools, making architectural decisions in this environment, or working with AI-assisted workflows — this is close to a disqualifier. Engineers who wait to be told what to learn will always be learning yesterday's curriculum. Here's the arc we've covered in this series. [Part 1](/hiring/what-not-to-ask-software-engineers/): most of what technical interviews test has lost its predictive value. The job changed and the questions didn't. We cleared the table. [Part 2](/hiring/what-to-ask-software-engineers-instead/): we replaced those questions with something that tests what the job actually requires now — specifying clearly, working with AI under observation, evaluating output critically, reasoning about failure, and debugging from a mental model. Part 3: we added the signal nobody was measuring. Not what someone can do today, but whether they have the habit of staying a step ahead of what's required of them. Together, these three categories form an interview that's actually aligned with what you're hiring for. Not performance in an artificial environment. Not recall under pressure. Judgement, in conditions that resemble the real job. That's a harder interview to run. It's also the one worth running. *The interview is only half the problem. The harder question is what happens after you've hired differently — when those engineers arrive at an organisation that wasn't designed for how they work. The next article is about what this environment demands of engineering leaders: not more AI tools, but a clear-eyed look at what your organisation is actually amplifying before you invest further in AI. That article is coming next.* --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # Leadership in the AI Era: What Amplification Means for Engineering Leaders (https://aieraengineering.com/leaders/leadership-in-the-ai-era/ — 2026-05-11) *Leadership in the AI Era* --- In 2023, Klarna announced that AI was handling the work of hundreds of employees. The throughput metrics looked fine. Customer interactions were being processed. Tickets were moving. From the outside — and from most internal dashboards — things were working. Twelve months later, they were partially rehiring. Quality problems had become visible to customers. The terms for the rehired workers were worse than before. What went wrong was not the AI. What went wrong was the assumption that throughput and quality are the same metric. They are not, and they never were. AI made the difference visible faster than anyone expected. --- ## What DORA actually found DORA — the most widely cited annual study of engineering team performance — added a fifth metric for 2025: Rework Rate. It measures the proportion of shipped work that had to be re-implemented within 30 days. This is the first DORA metric that asks whether the output was *correct*, not just whether it was *delivered*. The distinction matters more than it used to. The headline finding that came with it: AI adoption amplifies existing organisational capability in both directions. Teams with strong specification discipline found that AI accelerated correct output. Teams with weak specification discipline found that AI accelerated incorrect output. The tool is neutral. The organisation's existing practices determine the direction. This is the amplifier insight: before asking what AI will do for your team, ask what AI is amplifying in your team. --- ## The three things AI amplifies ### Specification quality If your team writes clear specifications before building, AI-assisted development produces correct output faster. If your team generates code from Jira ticket descriptions, AI-assisted development produces technically plausible but behaviourally wrong output faster. The bottleneck was never implementation speed. It was always specification clarity. AI removed the implementation buffer that used to make bad specs look tolerable. The problems that were always there are now arriving sooner. ### Code review discipline If your team reviews code for domain correctness — not just whether it compiles and passes tests — AI output is caught before production. If your team rubber-stamps code review, AI-generated errors enter production at the same rate as human-generated errors, but at higher volume. Code review in the AI era is a management discipline, not just an engineering practice. Whether your team actually does it, and what they look for when they do, is visible in your rework rate. Most leaders are not looking there yet. ### Domain knowledge density AI generates technically correct code that is domain-wrong regularly. An engineer who knows what correct looks like for their domain catches this. An engineer without deep domain knowledge often does not — the code looks reasonable, it passes review, and the error surfaces in production. The engineers with domain knowledge are the ones who make AI output safe to ship. They are also the ones most likely to leave in a headcount reduction, because their value does not show up in throughput metrics. It shows up six months later, when nobody catches the errors they used to catch. --- ## What Klarna's leaders missed Klarna measured throughput. Throughput looked stable. What they did not measure: defect capture rate, rework rate, domain coverage — the metrics that would have shown quality degrading while the volume numbers held. The engineers who left disproportionately were mid-career domain specialists. Engineers who knew when AI output was wrong in their specific context. That evaluation capacity was invisible in the metrics. Twelve months later, it was visible in production. 55% of organisations that made AI-driven workforce reductions in 2023–2024 reported regretting them within 12 months (Forrester, 2025). The regret was consistently the same thing: we removed the people who caught the errors. The structure of the error is not unique to Klarna. It is the predictable result of optimising for the metric you have when the metric you need does not yet exist on your dashboard. --- ## Three questions before the next AI investment ### 1. What is your current rework rate? If you do not know: measure it now. It is your baseline. Until you have it, you are flying blind on whether AI is helping or compounding. If it is above 25%: further AI investment will accelerate rework before it reduces it. Specification discipline needs to come first. If it is below 15%: your team has the specification quality for AI to amplify correct output. ### 2. Do your engineers know when AI output is wrong? Not "do they use AI" — can they evaluate what it produces against domain requirements? The practical test: give a senior engineer 30 minutes with AI-generated code in their domain and ask them to find the errors. If they find nothing, either the code is genuinely correct or the evaluation depth is not there. Both outcomes are worth knowing. If your team cannot reliably catch AI errors before production, more AI tooling increases your production error rate. That is not a configuration problem. It is a domain knowledge problem. ### 3. What would you lose in a 20% reduction? List the ten engineers you would be most reluctant to lose. Look at what their primary skill is: implementation speed, or domain knowledge and evaluation? If the list is weighted towards evaluation, your team's value is already in the right place. If it is weighted towards implementation speed, your team's value is concentrated in exactly the skill AI replaces fastest. This exercise is useful even if no reduction is planned. It tells you whether your team is positioned for the environment it is operating in. --- ## What this means for the engineering manager right now The EM's job has shifted from delivery management to quality management. Only 20% of engineering teams measure AI impact in any structured way, despite adoption rates above 90% (Jellyfish, 2025). That gap is where the EM's day-to-day work is — not auditing AI use, but tracking the signals that actually reveal whether quality is holding. One practical addition to regular one-on-ones: ask whether the engineer can explain a production issue in their domain. Not as a test. As a proxy for domain ownership. An engineer who cannot explain why something failed in their area probably does not have the depth to evaluate AI output in that area either. If the answer is consistently vague, the domain assignment may be wrong. --- The organisations that will look back on this period as a competitive advantage are not the ones that adopted AI fastest. They are the ones that understood what they were amplifying before they amplified it. That requires measuring quality, not just throughput. It requires knowing who holds domain knowledge and treating that as the asset it is. And it requires the uncomfortable conversation when AI investment decisions were made on throughput metrics and the quality degradation is now in production. That conversation is not evidence that AI was a mistake. It is the beginning of getting it right. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # Why Adding AI to Your Existing Team Structure Doesn't Work (https://aieraengineering.com/leaders/why-adding-ai-to-your-existing-team-structure-doesnt-work/ — 2026-05-18) *The AI-Era Engineering Playbook — Team Structure* --- A company bought Copilot licences for the whole engineering team. Rolled them out in January. By March, velocity metrics were up. More PRs, more tickets closed, more features in the release notes. By August, they were shipping less than before January. Every feature needed rework. The backlog was larger, not smaller. The senior engineers were exhausted because they were reviewing and fixing things that should not have reached review in the first place. Nobody did anything wrong. The tool did what it said it would do. The problem was that the team structure was designed for a different world, and nobody changed the structure when the world changed. --- ## What the old structure was designed for The traditional engineering ladder — Junior, Mid, Senior, Staff, Principal — was designed around one axis: implementation skill. How fast can this person write correct code? How complex a problem can they solve independently? That was the right axis when writing code was the bottleneck. The hierarchy made sense. You put your best implementers at the top, gave them the hardest problems, and organised everyone else around supporting that output. AI moved the bottleneck. Implementation is no longer where teams get stuck. Teams get stuck at two different points now: specifying what to build precisely enough that the output will be correct, and evaluating whether what was built is correct before it ships. Neither of those maps to the old ladder. A junior engineer with deep domain knowledge and strong specification instincts is more valuable on many tasks than a senior who generates code quickly but cannot evaluate whether it does what the business needs. The ladder was not wrong. It was built for constraints that no longer exist in the same form. --- ## The two roles that replace it The AI-era team runs on two primary roles, not one ladder. **The System Engineer** designs the systems that others build within. Architecture, guardrails, constraints, failure handling — the substrate that makes it safe for others to ship without understanding everything underneath. Their work is largely invisible when it is working. It surfaces when it breaks, or when it was never there. **The Product Engineer** owns a business domain and is accountable for what ships within it. Their value is not implementation speed — it is domain knowledge deep enough to specify correct behaviour, and the judgement to evaluate whether the output is right before it goes to production. They use AI tools to build, but the role is defined by ownership and accountability, not by which tools they use. These are not seniority levels. They are different functions requiring different skills. A strong Product Engineer does not automatically become a System Engineer with more experience, any more than a strong designer automatically becomes a product manager. --- ## The pod model The basic team unit is a pod: one System Engineer, two to four Product Engineers, one QA or Behaviour Engineer per product surface. The pod owns the full lifecycle of that surface — from specification to production to on-call. No handoffs to a separate deployment team. No specification thrown over a wall to implementation. The people who specify are the people who build are the people who monitor. This matters because handoffs are where context disappears. In an AI-era team, context disappearing at a handoff means AI-generated code running without anyone who understands the domain it was supposed to serve. The pod model keeps domain knowledge co-located with implementation and accountability. The System Engineer in the pod is not the senior engineer who reviews every pull request. That is a bottleneck, not a structure. The System Engineer's job is to design guardrails that allow Product Engineers to ship autonomously within safe bounds — and to make themselves less necessary over time, not more. --- ## The three anti-patterns that emerge when you skip this **Adding AI to an unrestructured team.** Copilot licences distributed to an existing team with no role changes. Some engineers adopt effectively. Most generate output faster than they can evaluate it. Velocity metrics rise. Quality quietly degrades. Technical debt compounds faster than before. The team is producing more code, most of which will need to be revisited. **Hiring Product Engineers without a System Engineer in place.** Fast initial output. A clean codebase for three months. Then a wall: the system has no coherent architecture, debugging is impossible, every new feature breaks something old. Refactoring costs more than the original build. The missing System Engineer was not a luxury — it was the foundation. **The specification-free team.** Features ship from Jira ticket descriptions, sometimes directly into AI prompts. The AI generates code that is technically coherent but behaviourally wrong, because no one specified what correct behaviour looked like before generation started. QA catches behavioural failures that should have been caught at specification. The team enters permanent fire-fighting mode. The fix for all three is the same: design the structure before you scale the tooling. The tools amplify what is already there. A team with good specification discipline and clear role boundaries gets dramatically faster. A team without those things gets dramatically more wrong. --- ## What to do if you're starting from an existing team You are not rebuilding from scratch. You are making three decisions. First: identify who is already operating as a System Engineer, whether or not they have the title. They are the engineers that other engineers ask about architecture. Put that function on a clear footing. Second: identify which engineers have deep domain knowledge in specific product areas. Those are your Product Engineers, regardless of their current title. The question to ask is not "can they code" but "do they know when what was built is wrong in ways that matter to the business?" Third: add a specification step to the workflow before it is needed. A fifteen-minute spec review before any AI generation starts catches the majority of problems that will take eight hours to fix after ship. It does not require a template or a tool. It requires the habit. The structure change is not about org charts. It is about where accountability sits, and whether the people who are accountable have what they need to actually do the job. --- Teams that add AI to an unchanged structure get faster at producing the same problems. The constraint is not the tool. It is the organisation the tool is running inside. Getting the structure right first is slower in the short term. It is the only approach that compounds in the right direction. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # The Architecture Problem Nobody Fixes When They Adopt AI (https://aieraengineering.com/architecture/the-architecture-problem-nobody-fixes-when-they-adopt-ai/ — 2026-05-25) *The AI-Era Engineering Playbook — Architecture* --- A team spent six months getting the hiring right. They identified their System Engineers, built out a pod structure, added a specification step before every build. Velocity improved. Quality improved. The rework rate dropped. Then they hit a wall. Product Engineers kept pulling in System Engineers for things they should have been able to handle themselves. A bug in the payments feature required someone who understood the authentication layer to diagnose it. A new integration required reading a database schema that nobody had documented. An error message said `ECONNREFUSED 127.0.0.1:5432` and the Product Engineer on call had no idea what it meant or what to do about it. The team structure was right. The system the team was operating had been built for a different kind of team. --- ## What the old architecture assumed The conventional layered system — presentation, application, domain, infrastructure — was designed around a specific assumption: the people changing the system understand everything it touches. A senior engineer modifying the checkout flow knew, without being told, that deleting a user record cascades to payment history. They knew because they had worked in the codebase for two years and built the original implementation. The architecture did not need to tell them. It relied on them knowing. This worked when the system was built and operated by the same small group of people who carried its full mental model. It works less well when the team grows. It breaks almost completely when Product Engineers are operating a system they did not build, using AI tools that generate code without any institutional memory at all. --- ## The three failure patterns **The invisible invariant.** Every production system has constraints that are real but unwritten. We never hard-delete audit records. We always validate at the service boundary. We never call the payment API without an idempotency key. These constraints exist because a senior engineer put them there, and they stay in place because everyone who touches that area knows about them. AI does not know about them. A Product Engineer who joined six months ago does not know about them. The constraint is violated. Nothing catches it at review because it was never written down. It surfaces in production, weeks later, in a form that is very expensive to diagnose. **The test echo chamber.** AI generates code. AI also generates tests. The tests validate the behaviour of the code. But when the code is wrong — not syntactically wrong, not logically wrong, but *behaviourally* wrong, implementing the right function against an incomplete specification — the tests are wrong in exactly the same way. High coverage. Clean CI. Silent correctness failures that pass every gate. The test suite was generated from the same model with the same assumptions as the implementation. Whatever the specification missed, both missed together. **The readable error, the invisible fix.** `Error: ECONNREFUSED 127.0.0.1:5432`. This means the database is unreachable. A System Engineer reads it and knows what to check. A Product Engineer on call at 2am reads it and escalates — because they need to, not because the problem is hard. The information required to act on it is not in the error. It lives in the head of whoever built the infrastructure. Every time a Product Engineer cannot act on an error without a System Engineer, the pod model breaks. The System Engineer becomes a bottleneck that the structure was supposed to eliminate. --- ## What enabling architecture looks like The fix is not a new framework or a new tool. It is applying the same discipline to architecture that the previous articles applied to hiring and team structure: design for the actual constraints, not the ones that existed before AI. **Machine-enforced contracts, not social conventions.** If a convention matters — validate at the boundary, never call the database from the presentation layer, always include an idempotency key — it needs to be enforced by the type system or the CI pipeline, not by institutional memory. A constraint that can be violated without a compiler error will eventually be violated. A type that only accepts validated inputs cannot be misused. This is not theoretical strictness. It is the difference between an invariant that survives team turnover and one that does not. **Errors in domain language.** `OrderLookupError: Could not retrieve order #12345 for customer #678. Database connection unavailable. Check DB_CONNECTION_STATUS on the dashboard; escalate if red.` This error can be acted on by the Product Engineer who owns that feature area. They do not need to understand the deployment topology. They do not need to call anyone. Every error that requires a System Engineer to diagnose is a tax on the team's architecture. It is worth paying to eliminate them in the feature areas Product Engineers own. **The visible surface, the hidden substrate.** Product Engineers should be able to see everything they need to build confidently within their domain: domain APIs, event types, repository interfaces. They should never need to see JWT validation logic, connection pooling, retry handling, or transaction management. Not because they cannot understand it — because they should not need to. The System Engineer's job is to build the substrate that makes this separation real, and to maintain it as the system grows. The architecture does not need to be simple. It needs to be legible at the layer where Product Engineers work. --- ## Getting from here to there Most teams are not starting from scratch. They have an existing system with unwritten conventions, infrastructure-flavoured error messages, and abstractions that leak. The migration is incremental, but the order matters. First: make the dangerous seams explicit. Authentication, database access, the external integrations that carry the most risk. Abstract them so that Product Engineers work against interfaces, not implementations. This has the highest safety return per unit of effort. Second: fix the errors. Error messages that require a System Engineer to interpret are cheap to improve and have immediate impact on Product Engineer autonomy. It is two hours of work per component and pays back every time something breaks. Third: document the constraints that exist only in people's heads. An Architecture Decision Record does not need to be long. It needs to capture what the constraint is, why it exists, and what breaks if it is violated. A Product Engineer who can find this in a reference can follow it. An AI agent that can be given this context will generate code that respects it. The enablement is not an event. It is a direction. Each step expands the range of what Product Engineers can do without System Engineer involvement — and each step makes the System Engineer more useful for the things that actually require them. --- The same logic runs through every article in this series. AI moved the bottleneck. Interviews designed for the old bottleneck stopped working. Team structures designed for the old bottleneck stopped working. And architecture designed for the old bottleneck — where the hard work was in writing the code — stops working when writing the code is no longer the hard work. The constraint now is specification, evaluation, and the judgement to know when what was built is wrong before it ships. The architecture's job is to make that judgement possible. Not to make it unnecessary — that requires the humans — but to give those humans what they need to do it without the system getting in the way. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # Are You an AI Dumper? (https://aieraengineering.com/engineers/are-you-an-ai-dumper/ — 2026-07-08) There's a pattern we've started noticing in code reviews. A pull request lands. It's clean, well-structured, passes CI. The developer who submitted it can walk you through what it does. But when you ask why it handles the empty-input case this way, or why this particular retry interval, or what happens if the upstream service is down for longer than the timeout — the answers get vague. Not because the developer is weak. Because they didn't write it. They prompted it, reviewed it for thirty seconds, and shipped it. The code is someone else's reasoning, and the developer has adopted it without fully inheriting the understanding. This is what we've started calling the Dumper pattern. And it is, by a significant margin, the most common pattern we see in engineering teams right now. --- ## The three groups When we look at how engineers are actually working with AI tools today, three patterns emerge. The first group — the **Avoiders** — are still writing most of their code manually. They use AI occasionally, for boilerplate or documentation, but fundamentally the development model hasn't changed. This group is shrinking fast. The productivity gap between them and AI-assisted engineers is now too large to ignore. The second group — the **Dumpers** — use AI heavily but treat the output as finished work. They prompt, they get code, they check that it compiles and the tests pass, they ship. The review is surface-level. The understanding is borrowed. This is the majority. The third group — the **Steerers** — use AI as a tool they direct rather than a source they consume. They specify carefully before they prompt. They read the output critically, looking for behavioural correctness, not just syntactic coherence. They can explain the agent's architectural choices because they defined the constraints that shaped those choices. Most engineers who read this will recognise themselves in group two. That's not a judgement — it's where the workflow naturally leads when you optimise for speed without building the habits that keep the quality of judgement high. --- ## What the Dumper pattern looks like in practice It's worth being specific, because the Dumper pattern is invisible from the outside. The commits look fine. The velocity is high. The PRs are green. The signs are behavioural: **You prompt before you specify.** The first thing you do with a new task is open the chat window, not write down what done looks like. The specification emerges from the interaction rather than preceding it. **Your review standard is "does it look right."** You're checking structure, naming, format — the things AI gets right almost every time. You're not systematically checking for the failure mode that only shows up under a specific input, or the race condition that only matters at 10× load, or the silent error that drops data without raising an exception. **You can explain what the code does, but not why it was structured this way.** If someone asks you to defend an architectural decision in the code you submitted, you go back to the code rather than drawing on a mental model you built while writing it. **"The tests pass" is your primary quality signal.** Tests check the cases you thought of. They don't check the cases the agent didn't anticipate and you didn't notice it hadn't anticipated. None of these are obvious failures. The code works. The feature ships. The problem surfaces later — in the production incident, the three-weeks-later bug, the review where someone asks a question you can't answer. --- ## Why this is the wrong pattern to be in right now If AI tools were going to do more over time, and the engineer's role were going to shrink, then optimising for output volume would be the right move. But that's not what's happening. The engineer's role is getting harder in a specific way: the implementation is getting easier and the judgement is getting harder. Anyone can generate code. The skill that separates engineers now is the ability to evaluate what was generated — to find the wrong answer in the plausible-looking output, to catch the assumption that won't hold in production, to know when the thing that came back is subtly not the thing you asked for. That skill is exactly what the Dumper pattern doesn't develop. You don't build judgement by accepting output. You build it by interrogating output, which means having a standard against which to interrogate it, which means specifying what you wanted before you see what you got. We mapped this out against the full skill set for the AI era — you can see where these skills land on [the Interview Skill Map](/interview-skill-map/) and compare it against [where they used to sit](/interview-skill-map-preai/) before AI changed the weighting. The skills that dropped in traditional importance (syntax, algorithms from memory, framework knowledge) are exactly the ones the Dumper pattern still exercises. The skills that rose (specification quality, output evaluation, failure-mode reasoning) are exactly the ones it doesn't. --- ## A five-question self-check These aren't trick questions. They're things you should be able to answer about any code you shipped this week. **1. Before you started, did you write down what "done" looks like?** Not a ticket description someone else wrote. A behavioural specification: what inputs, what outputs, what happens in the error cases, what performance assumption is baked in. If you started by prompting, you skipped this. **2. Can you name the failure mode in the last feature you shipped?** Not "it might have bugs." The specific scenario where it behaves incorrectly: the input that breaks it, the load condition that causes it to degrade, the dependency failure that produces a silent wrong answer instead of an exception. **3. Did you read the last PR you approved for behavioural correctness, or for style?** Style is naming, formatting, structure. Behavioural correctness is: does this produce the right output for all valid inputs, and the right error for all invalid ones? These are different reviews. Most people do the first. **4. If someone asked you to explain the retry logic in the last feature you shipped, would you need to re-read the code?** A developer who wrote that code manually would have the answer immediately. A developer who prompted it and accepted it may not. There's no shame in re-reading — but noticing that you need to is the signal. **5. What tool did you most recently evaluate and decide not to adopt?** This one is about early adoption, which turns out to be a stronger career signal than it looks. Genuine early adopters have a list of things they tried and rejected as well as things they adopted. If you can only name things you use, not things you considered and passed on, the adoption pattern is reactive rather than deliberate. If questions 1, 2, and 3 are uncomfortable, you're in the Dumper pattern. That's the starting point, not a verdict. --- ## What to do with this The hiring series [started from the interviewer's perspective](/hiring/what-not-to-ask-software-engineers/) — what interviewers are now testing for, and why the old interview process stopped working. If you want to understand how you'd be evaluated by a team that's thought carefully about this, that's a useful read from the other side. The next article in this series is about something more specific: [what you're quietly losing](/engineers/the-17-points-youre-quietly-losing/), in measurable terms, the longer you stay in the Dumper pattern. The research is uncomfortable. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # The 17 Points You're Quietly Losing (https://aieraengineering.com/engineers/the-17-points-youre-quietly-losing/ — 2026-07-16) Earlier this year, Anthropic ran a randomized controlled trial. Engineers were split into two groups: one worked with AI assistants, one without. Skills were measured before and after. The finding was specific and uncomfortable: the group using AI assistants scored 17% lower on a comprehension quiz covering the skills most relevant to evaluating AI output. Not a drop in productivity. Not a drop in code volume. A drop in debugging ability, code comprehension, and conceptual understanding of systems. The irony is precise: the skills that atrophied are exactly the ones you need most if you're going to work effectively with AI agents. The more you delegate to the agent, the worse you get at knowing whether the agent did it right. --- ## What actually atrophied The three areas that dropped are worth naming specifically, because they're not abstract. **Debugging.** The ability to trace a failure backward from a symptom to a root cause. To read an error, form a hypothesis about where it came from, test it, and narrow the search. This is a skill built through thousands of hours of manual investigation — finding the off-by-one by stepping through the loop, catching the null reference by following the data through the call stack, discovering the race condition by reasoning about thread interleaving. When the agent writes the code and it fails, the debugging path is different. You can ask the agent to debug it, which means the loop that built the skill doesn't run. You're observing someone else's investigation rather than doing one. **Code reading.** The ability to pick up an unfamiliar codebase and build a mental model of it quickly — what does this system do, where does the data go, what can fail and where. This is also learned through practice: you build the skill by reading code slowly at first, then faster as pattern recognition develops. Agent-generated code tends to be structurally coherent and locally readable but architecturally thin. It fits the prompt more than it fits the system. Reading it doesn't build the mental models that reading human-authored systems does, because it doesn't contain the same kinds of structural decisions. **Conceptual understanding.** The ability to hold a system in your head as an abstraction — to reason about it without looking at the code. To know that "if we change the caching layer, it will affect these three downstream consumers" without having to trace through the implementation to find out. This is the most important of the three, and the most fragile. It forms slowly, through the accumulated experience of building and debugging real systems. It degrades when you stop doing the work that builds it. --- ## Why this is a structural trap The trap is that none of this is visible in the short term. Your output volume stays high. Your PRs keep landing. The product keeps shipping. The skills that are degrading don't show up in the daily metrics — they show up in the incident that takes three times as long to diagnose, the architectural decision that turns out to have been wrong in a way nobody anticipated, the junior engineer's question you can no longer answer without looking it up. The Anthropic study measured this over a period of weeks. We don't have good data yet on what the 18-month curve looks like. But the mechanism is clear: skills that aren't exercised degrade. The understanding-while-building loop — the one where you learn the system by building it — has been broken. The understanding has to come from somewhere else, and for most engineers right now, it isn't coming from anywhere. There's a useful comparison point in [the skills shift we mapped for the AI era](/interview-skill-map/). The skills that rose in importance — output evaluation, failure-mode reasoning, specification quality — are precisely the skills that depend on having a strong underlying mental model of how systems work. You can't evaluate whether the agent's caching strategy is correct if you don't have a solid mental model of caching. You can't identify the failure mode in the retry logic if you can't reason about concurrency. The foundation for the new skills is the old skills. And the old skills are atrophying. --- ## The loop that was broken It's worth understanding why this happened structurally, not just that it happened. In traditional development, understanding and building were the same activity. You wrote the function and discovered the edge case in the process. You traced through the data model while implementing and found the schema didn't support what you needed. You wrote the retry loop and noticed, mid-implementation, that you hadn't defined what "failure" meant. This wasn't inefficiency. It was how understanding formed. The act of building was the act of learning. The two were inseparable. When an agent does the implementation, the loop breaks. The agent doesn't discover edge cases as it builds — it either anticipated them because you specified them, or it missed them because you didn't. There's no mid-implementation moment where you pause and catch something unexpected. You see the finished output, not the construction process. The cognitive work that used to happen during building now has to happen before building. And for most engineers, it isn't happening at all — because they were never taught to do it upfront, because the implementation loop always made it unnecessary. This is the subject of [Part 4 of this series](/engineers/before-the-agent-runs/) — the full mechanics of what changes when agents enter the workflow, and what you have to build to replace what you lose. --- ## Three habits that prevent atrophy These aren't silver bullets. They're maintenance. The same way you have to run to maintain the ability to run. **1. No-AI days, once a week minimum on something non-trivial.** Pick a real task — not documentation, not boilerplate — and work through it without the agent. The goal is not to be slower; it's to keep the loop running. You're not testing your productivity. You're testing whether the mental model is still there. If you find yourself stuck in a way you wouldn't have been six months ago, that's data. **2. Interrogate before accepting.** Before merging any agent-generated code, answer three questions without looking at the code: What failure mode does this have? What happens at the boundary conditions? Why did the agent structure it this way rather than the obvious alternative? If you can't answer any of them, you're in the Dumper pattern (see [the previous article in this series](/engineers/are-you-an-ai-dumper/)). Answering them — even briefly — keeps the evaluation muscle active. **3. Explain it to someone.** This one is simple and underrated. Before you merge, explain the key architectural decision in the PR to a colleague — out loud, without referring to the code. If you can do it, the understanding is there. If you can't, it isn't. The explanation test surfaces gaps that reading doesn't, because reading lets you follow along without fully comprehending. None of these are radical. They're the practices that keep the skills from degrading to the point where they're no longer there when you need them. --- ## What the interviewer is testing for If you're interviewing at a company that has thought carefully about this, they are not testing your framework knowledge or your algorithm recall. They are testing whether the atrophy has happened. Specifically: can you read a piece of AI-generated code and find the behavioural problem that isn't visible on the surface? Can you name the failure mode in a system you've been given a two-sentence description of? Can you specify what done looks like before you start building? These are the exercises in [the hiring side of this framework](/hiring/what-to-ask-software-engineers-instead/). Reading it from the engineer's perspective is useful — it shows you exactly what a careful interviewer is looking for, and the skills they're testing are the same ones the atrophy research says you're losing. The next article in this series is precisely about that: [what an interviewer who has read this framework already knows about you](/engineers/what-your-interviewer-already-knows/), before you've said a word. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # What Your Interviewer Already Knows About You (https://aieraengineering.com/engineers/what-your-interviewer-already-knows/ — 2026-07-20) Here is how a code review exercise goes in a well-designed interview. You're given forty lines of AI-generated code and asked: read this, tell me what you'd change before shipping it. No time pressure. No trick questions. Most candidates start the same way. They find the variable names they'd choose differently. They note a missing comment. They suggest a formatting change. They are, in this moment, doing exactly what they do every day — reviewing code for style. Five minutes in, they've generated a list of improvements. None of them would affect the behaviour in production. The interviewer thanks them and asks: did you find anything that would cause this to fail? The question lands differently for different candidates. Some go back immediately — they missed something, they know it, now they're looking for it. Some look at the interviewer, unsure what they're supposed to have found. Some say confidently: no, the code looks solid to me. That moment — the gap between what each candidate sees and what's actually there — is one of the most predictive signals in the entire process. --- ## What a careful interviewer already knows Before you've answered a question, an interviewer who has thought carefully about the AI era already knows several things from your resume and the first five minutes of conversation. They know whether you're in the Avoider, Dumper, or Steerer pattern (see [Part 1 of this series](/engineers/are-you-an-ai-dumper/)). The signals are in how you describe your workflow, not in which tools you name. They know roughly where your skills sit relative to the [new weighting for the AI era](/interview-skill-map/). The skills that now matter most — specification quality, output evaluation, failure-mode reasoning — are almost never tested in traditional interviews, which means most candidates haven't been developing them deliberately. The atrophy research covered in [Part 2](/engineers/the-17-points-youre-quietly-losing/) gives interviewers a framework for understanding what's likely to have degraded and what probably hasn't. And they know what they're going to find out from the exercises: not whether you're smart, but whether you can evaluate output that looks correct but isn't. --- ## What the new interview actually tests The hiring series [covered this from the interviewer's side](/hiring/what-to-ask-software-engineers-instead/) in detail. From the engineer's side, the picture is simpler: there are three things being tested, and they all connect. **Specification quality.** Given a vague requirement, do you slow down and name what's missing before building anything? Do you ask about edge cases, error states, performance assumptions? Or do you start building immediately and discover constraints as you go? The exercise is usually a two-sentence requirement and a blank page. What happens in the first thirty seconds tells the interviewer most of what they want to know. The strong candidate asks questions or writes down assumptions. The weak candidate asks which framework to use. **Output evaluation.** Given code — AI-generated, but you're not always told — can you find the problem that matters? Not the style issue, not the naming inconsistency. The behavioural failure: the case where the function returns the wrong answer silently, the retry logic that never actually retries, the edge case that was never handled and will surface in production. This is the code review exercise from the opening of this article. The [hiring article on this topic](/hiring/what-to-ask-software-engineers-instead/) goes deep on what interviewers are watching for. The short version: they're watching how far you read before you find the real problem, and whether you can tell the difference between a style issue and a correctness issue. **Failure-mode reasoning.** Given a system description — a payment retry queue, an email sending service, a user permissions cache — can you describe how it fails? Not theoretically. Specifically: what's the scenario where it produces a silent wrong answer? What's the failure that only shows up at 3× normal load? What's the dependency assumption baked into the design that will be false in production? Strong candidates go somewhere interesting quickly. They find the failures that don't raise exceptions. The [third article in the hiring series](/hiring/the-question-youve-never-thought-to-ask/) covers why this signal matters more than most traditional interview components. --- ## What it doesn't test This is as important as what it does test. **Syntax and language trivia.** Nobody is testing whether you can write a binary search from memory or name all the methods on a string object. These skills had limited predictive value before AI; they have almost none now. **Framework and library knowledge.** Knowing the React lifecycle hooks, the Spring Boot annotations, the SQLAlchemy relationship parameters — this is retrievable knowledge. The interview is testing judgement, not retrieval. **Algorithm performance.** Big-O analysis, time/space tradeoffs on abstract data structures — these still matter for System Engineering roles at scale, but they're not the primary signal for most engineering positions. [The skill map](/interview-skill-map/) shows exactly where this sits now relative to the other competencies. This shift has a practical implication for preparation. Time spent grinding LeetCode is time not spent developing specification quality and output evaluation. If you're preparing for a serious interview at a company that has updated their process, the ROI on LeetCode is low. --- ## How to prepare The three skills being tested are trainable. None of them require a special environment — you can practice them on your own work, today. **For specification quality:** Before your next task, write down what done looks like. Not a ticket summary. A behavioural specification: what inputs are valid, what outputs are expected, what failure looks like, what performance assumption is baked in. Then identify five things that are missing from the requirement. If you do this consistently for a month, the first thirty seconds of an interview exercise will feel natural rather than unfamiliar. **For output evaluation:** Take the code review exercise seriously. Find a piece of AI-generated code — your own recent work is fine — and review it with one rule: you're not looking for style issues, you're looking for the thing that will fail in production. Give yourself a specific constraint: find one thing that produces a wrong answer, silently, under a valid input. This is harder than it sounds, and that difficulty is what makes it valuable to practice. You can also practice against the [System Engineer Question Bank](/system-engineer-question-bank/) and [Product Engineer Question Bank](/product-engineer-question-bank/) — these are designed as interviewer tools, but reading them as an engineer tells you exactly what's being evaluated and at what level of specificity. **For failure-mode reasoning:** "How does this fail?" as a daily habit. Pick any system you worked with today — your own feature, a third-party API you called, an internal service you depended on. Name three failure modes. Force yourself to be specific: not "it might have bugs" but "if the upstream service returns a 200 with an empty body, this will process successfully and produce a silent wrong answer." Do this every day for two weeks and you'll find your rate of catching production failures before they happen increases noticeably. --- ## Using the scorecards as a mirror The scorecards were built as interviewer tools — the [System Engineer Scorecard](/system-engineer-scorecard/) and the [Product Engineer Scorecard](/product-engineer-scorecard/) give interviewers a structured way to evaluate candidates across the full skill set. Reading them as a candidate is instructive. Each dimension has a description of what a 5/5 looks like and what a 1/5 looks like. Working through them honestly — scoring yourself across each dimension — gives you a clearer picture of where you actually stand than most self-assessments. The difference between a 3 and a 5 on output evaluation, for example, is not about being smarter. It's about having developed the habit of looking for behavioural correctness rather than style. That habit is buildable. The scorecard tells you what the endpoint looks like. --- ## What this series is about We've been writing about the AI era from the hiring side for most of this year — [what interviewers should stop testing](/hiring/what-not-to-ask-software-engineers/), [what they should test instead](/hiring/what-to-ask-software-engineers-instead/), [how teams should be structured](/leaders/why-adding-ai-to-your-existing-team-structure-doesnt-work/), [what leadership looks like](/leaders/leadership-in-the-ai-era/). This series is the other side of that. Same framework, but the question is not "how do I hire for the AI era" — it's "how do I become the engineer who thrives in it." The next article is about the deepest part of that shift: not the habits you need to build, but [the way you have to change how you think before the agent runs](/engineers/before-the-agent-runs/). --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # Before the Agent Runs (https://aieraengineering.com/engineers/before-the-agent-runs/ — 2026-07-28) There's a moment most teams hit a few weeks into serious agentic development. A pull request lands. It looks correct. The agent generated it, the CI is green, and nobody on the team wrote a line of it. The developer who triggered it approves the PR in four minutes. Three weeks later, something fails in production. It's not a crash; it's a silent wrong answer. The retry logic counted from the wrong baseline. The code was coherent, well-structured, and wrong in a way that only someone who deeply understood the domain would catch. Nobody caught it, because the developer who approved it never really understood what it was doing. They read it the way you read a form: they checked that the fields were filled in. This is the failure mode that doesn't show up in demos. --- ## What building code actually taught you For most of us, understanding and building were the same activity. You'd write the first version of a function and discover the edge cases in the process. You'd trace through the data model while implementing and realise the schema didn't support what you needed. You'd write the retry loop and notice, mid-implementation, that you hadn't defined what "failure" meant. This wasn't inefficiency. It was how understanding formed. The act of building was the act of learning. The two were inseparable. That cognitive model was built through repeated manual construction. It was the byproduct of doing the work. When something failed, you knew where to look. When a PR landed, you had the context to read it critically. --- ## Agents break that loop When an agent does the implementation, the loop breaks. The agent doesn't discover edge cases as it builds; it either anticipated them (if you specified them) or it missed them (if you didn't). There's no mid-implementation moment where you pause and catch something unexpected. You see the finished output, not the construction process. This means the cognitive work has to happen before the agent runs, not during. You need to understand the problem, the constraints, the failure modes, and the edge cases before you write the prompt. If you don't, you get back code that looks like it handles everything and actually handles a subset. The confident-looking output is the risk. The shift sounds simple. It isn't. "Understand before you delegate" is a fundamentally different cognitive habit from "understand while you build." Most developers have spent years training the second one. They are not naturally good at the first. --- ## What "think and explain" actually requires The new skill is specification under uncertainty. Before the agent runs, you need to articulate: what does done look like? What inputs are out of range? What failure is silent vs. loud? What assumption am I making that might not hold in production? This is not writing better prompts. This is doing the mental work that the implementation loop used to force you to do automatically. You have to reconstruct the discovery process deliberately, without the act of building to guide you. The clearest test of this skill is the one we covered in [Part 3](/engineers/what-your-interviewer-already-knows/): a two-sentence requirement, and what you do in the first thirty seconds before writing anything. The developer who names five unstated assumptions first is the developer who can work effectively in an agentic environment. The developer who immediately asks which framework to use is not. Experienced developers often underestimate how much of this they were getting for free from implementation. Junior developers often don't yet have the domain model that makes it possible at all. --- ## You cannot just adopt the tools and keep the old habits The failure pattern we see most often: a team adopts an agentic workflow, productivity looks good for the first two weeks, and then the defect rate climbs in ways that are hard to attribute. The root cause is almost always the same. Developers were given the tools without being given the new cognitive model. They kept using the old one (build, check, ship) except now "build" was replaced by "prompt." The checking step stayed the same length it always was (not very long), but it was now covering far more ground. The checking step is now the entire job. It just doesn't feel that way yet. Adopting agentic tools without changing how you think about evaluation is the equivalent of being asked to review architectural drawings but only checking for straight lines. You'll check what you know to check. --- ## A model for building trust incrementally What actually works is treating agents like a new colleague on a probation period; not a feature to deploy. When a new developer joins, you don't give them production access on day one. You give them bounded tasks. You review their output closely. Over several weeks, you expand the scope as confidence builds. The trust is task-specific, not general. The same logic applies to agents, and to yourself learning to work with them. **Start at shadow mode.** Run the task manually and with the agent, then compare outputs. Not to accept either; to understand where they diverge and why. This builds calibration before it builds dependency. **Move to bounded delegation with full review.** Pick one category of task (test generation, documentation, scaffolding) and review every line of agent output before merge. This creates the habit of critical reading under low-stakes conditions. **Expand to workflow-level trust with approval gates.** The agent handles more, but nothing irreversible happens without a human decision point. Deployment, schema migrations, external API calls; these stay gated until you have data on the agent's reliability in your specific context. **Monitor, don't just audit.** Once autonomy is higher, the oversight mechanism shifts from reviewing individual PRs to watching system KPIs. Is the defect rate stable? Are the silent failures increasing? This requires knowing what to measure, which requires having built the mental model first. The teams that rush this process (that go from zero to high autonomy in weeks) are the ones that produce the three-weeks-later production failures. --- ## What atrophies if you don't watch it [Part 2 of this series](/engineers/the-17-points-youre-quietly-losing/) covered the Anthropic research on this. AI assistants reduced developer competency on the skills that matter most for agentic work: debugging, code reading, conceptual understanding. The painful irony is that those are exactly the skills you need to supervise agent output. The more you delegate, the worse you get at evaluating what was delegated. The mitigation is not to avoid the tools. It's to keep the manual muscle active deliberately. Regular no-AI sessions on non-trivial features. Code review habits that require you to explain why the agent's solution works, not just that it does. Asking yourself, before you merge: can I explain the agent's architectural choices to a colleague without looking at the code again? If you can't, you approved something you didn't understand. That's not a safe state. --- ## Security is not a phase In traditional development, security often lived at the end; a review before deployment, a penetration test once a year. In agentic development, that model is broken before you even try it. Agent-generated code moves fast. If you're reviewing security as a gate at the end, you're reviewing a month of generated code in a window that hasn't grown to match. You will miss things. The practical answer is to embed security scanning before the code ever reaches a PR. Secrets interception on prompts; developers regularly leak credentials in natural-language context without realising it. Supply chain scanning that treats hallucinated dependencies the same as real ones. Pre-commit scanning that applies identical standards to agent-generated and human-written code. The structural answer is harder. Agents crash mid-operation. When they do, they leave intermediate states; a payment processed but not recorded, a record updated but the cache not cleared. You cannot solve this by telling the agent to be careful. You solve it by designing tools that are idempotent, with transaction coordination that handles partial failures deterministically. Reliability is an infrastructure problem. It is not a prompting problem. --- ## What this means for you The [Interview Skill Map](/interview-skill-map/) shows where skills have shifted in the AI era. The ones that rose (specification quality, output evaluation, failure-mode reasoning) are exactly the ones that agentic development demands. That's not a coincidence. The skills that make you effective as an agentic developer are the same ones that separate strong engineers from weak ones right now. You develop them the same way: by doing the deliberate work of specifying before building, evaluating output for behavioural correctness rather than style, and asking "how does this fail?" before it fails in production. The next article goes deeper on one part of this: [why the way writing code taught you to think](/engineers/why-writing-code-taught-you-wrong/) is actively the wrong mental model for agentic work, and what to replace it with. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # Why Writing Code Taught You the Wrong Way to Think (https://aieraengineering.com/engineers/why-writing-code-taught-you-the-wrong-way-to-think/ — 2026-08-04) A senior developer we worked with spent two hours on an agent-generated pull request last month. Two hours is a serious review. He read every line. He checked the tests. He traced the main path twice. He approved it, and it was wrong. What made it interesting is that a junior on the same team had flagged the problem in fifteen minutes. Not because she was better. Because she had no idea how the code was built, so she never tried to follow it. She just asked what happens when the list is empty, and went looking. The senior developer had done what he always does. He read the code the way he writes code: from the top, following the flow, building understanding as he went. That method worked for twenty years. It is now the thing that failed him. --- ## What the old loop actually trained We tend to describe the skill as "writing code," which hides what was really being learned. When you build something manually, you work in small steps and you verify continuously. You write a function, you run it, it fails, you fix it. You add the next piece on top of a foundation you have already checked. By the time the feature is done, you have never once been holding a large piece of unverified work. Correctness compounded. That is a specific cognitive habit, and it is a good one. It taught us to think in sequence. To trust progress. To treat "it runs" as meaningful evidence, because when you built it incrementally, it was. It also meant you never had to read a big artifact cold. You already knew what it did. Reading your own code back is not comprehension; it is recall. Everything about that habit assumed one thing: that the person reasoning about the code and the person producing it were the same person, working at the same time. --- ## The assumption broke An agent hands you the finished thing. There were no small steps you verified. There is no foundation you checked. There is a complete artifact, produced by reasoning you did not witness, and your job is to decide whether it is correct. The old habit does not just fail here. It actively misleads, in four specific ways. **"The tests pass" stops being evidence.** It used to be a real signal, because your tests grew alongside your understanding of the problem. Each one encoded something you had discovered while building. Agent-written tests encode what the agent anticipated. They test the same subset of reality the implementation handles. Green means the code is consistent with itself. **Reading top to bottom stops working.** You read that way because you wrote that way, and the sequence carried meaning. Agent-generated code has sequence but not history. Following it linearly gives you a narrative of what happens, which feels like understanding and is not. You end up able to describe the code without being able to predict it. **Structure stops being evidence of thought.** This one is subtle and it catches experienced people hardest. Clean structure used to cost something. If a function was well decomposed with sensible names, a human had thought about it, and that was information you could legitimately use. That signal is now free. Coherent structure tells you nothing about whether anyone reasoned about the domain. **Debugging by re-reading stops working.** When something failed, you reread the code and the memory of writing it came back with it. That is why it worked. Reread code you never wrote and there is nothing to come back. --- ## What the junior developer did differently She did not read the code. She asked what would break it, and then went looking for that specific thing. That is the whole shift, and it sounds smaller than it is. She started from the outside: what are the inputs, what is claimed about the outputs, where is the boundary. Then she went hunting for one case. She was not building a model of the code. She was trying to break a claim. Call it adversarial reading. You begin from the assumption that the artifact is wrong, and your task is to locate where. That is a different posture from building understanding, and it produces a different search pattern. You go to the boundaries first: empty, zero, one, maximum, negative, null, concurrent, the second call. You go to the error paths, which is where agents are weakest, because error handling is the part of a specification people leave out. Here is the kind of thing it finds. This function was in the PR: ```python def apply_discount(items, code): total = sum(i["price"] * i["qty"] for i in items) pct = DISCOUNTS.get(code) if pct: total = total * (1 - pct) return round(total, 2) ``` Read it linearly and it is fine. The names are good, the arithmetic is right, the rounding is there, and a test with a normal cart and a valid code passes. Now attack it. What is `DISCOUNTS.get(code)` when the code is invalid? `None`, and `if pct` is false, so we skip the discount and charge full price. Silently. No exception, no log, no signal. A customer types a promo code that expired, and we charge them the full amount and tell them nothing. That is not a crash; it is a support ticket three weeks later, and a trust problem you will never fully measure. There is a second one in the same three lines. What if a discount is legitimately zero percent? `if pct` is false again, and the two cases are now indistinguishable. The bug and the valid case share a code path. Nothing about reading that function from the top surfaces either problem. You have to arrive with the question. This is exactly the exercise a well-designed interview now runs, which we covered in [Part 3](/engineers/what-your-interviewer-already-knows/). The interviewer is not checking whether you can read. They are checking whether you arrive with the question. --- ## Why seniority makes this harder The uncomfortable part is that the reflex gets stronger with experience. Twenty years of incremental building produces a very good instinct for following code, and that instinct fires automatically. It does not present itself as a choice. You look at a diff and you are already reading it the old way before you have decided anything. Junior developers do not have the reflex yet. That is why the fifteen-minute catch was not luck. She had nothing pulling her towards linear reading, so the obvious question was available to her. This is not an argument that inexperience is an advantage. The senior developer has the domain model that makes the question "what happens with an invalid code" produce a list of consequences rather than a shrug. The advantage is real; it is just being spent on the wrong activity. He used his two hours to reconstruct the code. He should have used ten minutes of it to interrogate the contract. [Part 2 of this series](/engineers/the-17-points-youre-quietly-losing/) covered the research showing which skills atrophy under AI assistance. This is the mechanism underneath the numbers. It is not that people get lazy. It is that a highly trained, previously correct habit keeps running in a situation where it no longer applies. It also explains why the [Interview Skill Map](/interview-skill-map/) moved the way it did. Output evaluation rose to the top not because reading code became more important, but because a specific kind of reading became necessary and almost nobody had been practising it. --- ## What to replace it with Three changes, in the order they are worth making. **Reconstruct the contract before you read the implementation.** Before opening the diff, write down what this code must be true about: valid inputs, expected outputs, what counts as failure, what must never happen. Two minutes, from the ticket rather than the code. Now you are reviewing against something. Without it you are only checking whether the code agrees with itself, which it always will. **Read for the failure, not for the flow.** Pick the boundaries and go straight there. Empty, zero, one, maximum, malformed, absent, repeated. Do not start at line one. Starting at line one is the old habit wearing a review hat. **Treat "I can describe it" as insufficient.** The test is prediction, not description. Can you say what this returns for an input you have not seen? If you can only narrate what the code does, you have followed it rather than understood it. Those feel identical from the inside, which is exactly why the two-hour review approved a broken function. None of this is slower than what most people do now. The senior developer's two hours were not wasted because he was careless. They were wasted because the method was built for a situation that no longer exists. If you want to drill it rather than just agree with it, the [interview prep coach](/ai/interview-prep/) runs this as a scored exercise in your own stack: it plants one real behavioural bug among genuine style flaws and tells you how far you read before you found it. Most people fail the first attempt, which is the point. --- ## The thing this connects to [Part 4](/engineers/before-the-agent-runs/) argued that the cognitive work has to move to before the agent runs. This is the other half: the work that remains after the agent runs is not the work you are trained for either. Specification moved earlier. Evaluation moved later and got harder. The comfortable middle, where you built the thing and understood it by building it, is the part that disappeared, and it was the part that carried both skills for free. If you are not sure which pattern you are in, [Part 1](/engineers/are-you-an-ai-dumper/) has the five-question self-check. The next article is about the first half of that: what a specification an agent can actually execute looks like, and why most of what we call requirements are not specifications at all. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # The Prompt That Reveals Everything (https://aieraengineering.com/engineers/the-prompt-that-reveals-everything/ — 2026-09-03) Two developers on the same team got the same ticket last month: "Add rate limiting to the public API." The first one opened his agent and typed, near enough, the ticket. "Add rate limiting to our public API endpoints." He got back a clean middleware, a token bucket holding its counters in process memory, tests, and a config block. It shipped that afternoon. The second one did not open her agent for about ten minutes. When she did, the prompt was most of a page. Limits per API key rather than per IP, because the customers behind corporate NAT would otherwise share a bucket. State in Redis rather than in process, because we run four instances and an in-memory limiter means the real limit is four times what we advertise. Return 429 with Retry-After. Fail open if Redis is unreachable, log loudly, because a rate limiter that takes down the API when its own dependency dies is worse than no rate limiter. Different limits for the write endpoints. Failed auth attempts limited separately by IP, so the limiter does not leave a credential stuffing hole wide open. Both prompts produced working code. Only one produced correct code. What is interesting is that nobody caught it. Both PRs looked good, and the in-memory counter was findable by anyone who thought to ask what happens with more than one instance. Nobody asked. The gap was not in the code, and it was not in either developer's prompting technique. It was upstream of both, in a decision the ticket never surfaced. --- ## The prompt is not the skill There is an industry built on the idea that prompting is a craft you can learn. Templates, frameworks, role prefixes, "act as a senior engineer," few-shot examples. Some of that helps at the margins. None of it is the thing that separated those two developers. The second developer did not write a better prompt because she knew a better prompt format. She wrote a better prompt because she had thought about corporate NAT, and horizontal scaling, and what a rate limiter should do when its own datastore is down. The prompt was a transcript of thinking that had already happened. This is why we keep saying the prompt is a diagnostic rather than a skill. It does not create understanding. It exposes it. Whatever you have or have not worked out before you start typing, the prompt shows it, in the same way a written specification has always shown it. Which leads to an uncomfortable use for your own history. --- ## Read your last five prompts Go and look. Not the conversation, just the opening message of your last five real tasks. For each one, ask: does this contain anything that was not already in the ticket? The honest answer, for most of us most of the time, is no. We paste the requirement, add a sentence about the stack, and press enter. That is the [Dumper pattern](/engineers/are-you-an-ai-dumper/) wearing a different face: prompt first, understand later, if at all. The prompt is a restatement. And a restatement cannot contain more understanding than the thing it restates, which means the agent is now making every decision the ticket left open, silently, according to whatever is most common in its training data. That is the actual mechanism behind the failure we described in [Part 4](/engineers/before-the-agent-runs/). The agent did not get it wrong. It got it unspecified, and unspecified resolves to plausible. --- ## Requirements and specifications are different objects We use the words as though they are synonyms. They are not, and the difference matters more now than it ever did. A **requirement** describes what someone wants. "Add rate limiting to the public API." It is a statement of intent, written by someone who is allowed to not know how it works. It is complete as a request. A **specification** describes what the system must do, in enough detail that two competent implementers would build the same behaviour. Not the same code; the same behaviour. It answers the questions the requirement left open, and it names the ones it is deliberately leaving open. Almost everything that arrives in a ticket is a requirement. Almost nothing that arrives in a ticket is a specification. For most of us and most of our careers this did not matter much, because the gap got closed during implementation. You started building, you hit the question, you asked someone or you decided. The specification did get written. It just got written incrementally, in your head, in the order the code demanded it, and it never existed as a document. That is the important part: the thinking happened, but always fused to the act of building, never as a separate step you could perform on its own. Some fields always wrote real specifications as documents, and they are about to look prescient. Take away the implementation loop and the gap does not close. It gets filled by the agent instead, using a default you never saw and never approved. --- ## What a specification has that a requirement does not Five categories. Most underspecified prompts are missing several of them. **Valid inputs.** What is in range and what is not. Not the type; the domain. An integer is a type. "Between 1 and 1000, and 0 means unlimited" is a specification. **Expected outputs.** Including the shape of the thing on the boundary. What comes back for an empty result set is a decision, and "empty list" versus "null" versus "404" are three different systems. **Failure definition.** What counts as failure at all, and what the system does about it. This is the one people skip most, and it is where agents are weakest, because they will invent an error path that looks reasonable and is wrong for your domain. Fail open or fail closed is a business decision, not a technical one. **Performance and scale assumptions.** How many, how fast, how concurrent. The rate limiter above is a pure example: the entire correctness of the naive version depends on an assumption about instance count that nobody wrote down. **Dependency behaviour.** What happens when the thing you depend on is slow, absent, or lying. Every system has this and almost no ticket mentions it. You can test this on any requirement in about two minutes. Take it, go through the five, and write down what it does not say. Some tickets genuinely are well specified. Most are not, and the ones that feel most complete are usually the ones worth checking hardest. --- ## The same ticket, three ways Requirement, as it arrived: > Add rate limiting to the public API. The prompt most people write: > Add rate limiting to our public API endpoints. We use Express. The prompt that produces the right code: > Add rate limiting to the public API. > > Scope: per API key, not per IP. Several customers are behind corporate NAT and would otherwise share a bucket. > > Limits: 1000 requests per hour on read endpoints, 100 per hour on writes. > > Failed authentication: cannot be attributed to a key, so limit those separately by IP. This conflicts with the NAT decision above, and we are choosing the tighter option deliberately: 100 failed attempts per hour per IP, which a shared office will not hit under normal use but a credential stuffing run will. Leaving auth failures unmetered is the worse trade. > > State: Redis, not in-process. We run four instances behind the load balancer, so an in-memory counter would let through four times the advertised limit. > > When exceeded: 429 with a Retry-After header giving seconds until the window resets. > > When Redis is unavailable: fail open and log at error level. A limiter that takes the API down when its own dependency dies is worse than no limiter. > > Out of scope for now: per-endpoint overrides, burst allowances. Most of those blocks map onto the five categories. The 429 with Retry-After is the expected output. The failed auth rule and the Redis fallback are failure definition. Four instances is the scale assumption. Redis is the dependency. Two of them do not map, and that is worth noticing rather than forcing. Scope is a prior question the five categories assume you have already answered: what is the unit being measured. And the final block is not a category at all. Notice what the long version is not. It is not longer because it is more polite, or because it uses a better template, or because it tells the agent to think step by step. It is longer because it contains seven decisions that the one-line prompt delegated by accident. Notice also the last line. Naming what you are deliberately not doing is part of a specification, and it is the part that stops an agent from helpfully building it anyway. And notice what it still does not cover. The auth limiter also lives in Redis, so when Redis fails open, credential stuffing goes unmetered at exactly the moment someone is most likely to be probing. That is a real gap, and it is there on purpose: a specification is never complete, only complete enough for the decisions in front of you. The skill is not producing an exhaustive document. It is knowing which unanswered questions are the expensive ones. --- ## Why this is the hardest of the three skills [Part 3](/engineers/what-your-interviewer-already-knows/) laid out what the new interview tests: specification quality, output evaluation, failure-mode reasoning. Of those three, specification is the one people find hardest to develop, for a reason worth naming. Output evaluation has a target. You are looking at something, and either you find the problem or you do not. Failure-mode reasoning has a prompt: someone asks how it breaks, and you answer. Specification has neither a target nor a prompt. You are trying to notice the absence of something, before anything exists, with no external cue that anything is missing. The requirement looks complete. That is what makes it dangerous. Nobody ever feels the gap, because a gap is precisely the thing that does not announce itself. [Part 2](/engineers/the-17-points-youre-quietly-losing/) measured which skills degrade under delegation. Specification is not on that list, and the reason is worth naming: it cannot atrophy the way debugging did, because it was never a standalone habit to begin with. It only ever ran attached to implementation. There is also an uncomfortable diagnostic buried in this. If you cannot specify it, you do not understand it yet. Not the code; the problem. The developer who cannot say what should happen when Redis is down has not thought about what the rate limiter is for. That was always true, and implementation used to hide it, because you could start typing and work it out on the way. You cannot work it out on the way any more. The way is gone. --- ## How to practise it this week **Write the spec before the prompt, in a separate place.** Not in the agent window. Somewhere you cannot start building. Five headings, the five categories above. It takes a few minutes and it will feel like overhead until the first time it catches something expensive. **Count the decisions your prompt delegated.** After you write a prompt and before you send it, read it back and count how many of the five categories it leaves open. That number is how much of the design you just handed to a statistical model. **Practise on tickets you are not going to build.** Take any ticket from your backlog, run it through the five categories, and move on without building anything. It is cheap and fast, and unlike the other two it does not require having real work in front of you. The [interview prep coach](/ai/interview-prep/) runs this as a scored drill in your own domain if you want the feedback loop. **Read the question banks as specifications.** The [Product Engineer Question Bank](/product-engineer-question-bank/) and [System Engineer Question Bank](/system-engineer-question-bank/) were written as interviewer tools, but every question in them has anchored strong and weak answers, which is what a specification of a good answer looks like. Reading them backwards teaches the shape. --- ## Where this lands [Part 5](/engineers/why-writing-code-taught-you-the-wrong-way-to-think/) argued that the reading habit built by writing code is the wrong one for evaluating agent output. This is the same argument on the other side of the work: the specification habit was never built as something you could do on its own, because implementation always carried it. Both halves come down to the same thing. The middle of the job disappeared, and it was carrying two skills that nobody had to teach because everyone acquired them as a side effect. Now they have to be taught. That is what the rest of this series is for. The next article stops describing the problem and starts on the method: [what a specification an agent can actually execute looks like](/engineers/how-to-write-a-spec-an-agent-can-execute/) in practice, section by section, on a real feature. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # How to Write a Spec an Agent Can Actually Execute (https://aieraengineering.com/engineers/how-to-write-a-spec-an-agent-can-execute/ — 2026-09-19) The ticket said: "Users should be able to export their data." An engineer on a team we worked with pasted that into an agent, more or less verbatim, and got back a CSV export button on the account settings page. It worked. It shipped. Within a month, support started getting tickets from users who wanted their data in a format their new tool could actually import, from users who had two accounts and wanted one export covering both, and from one enterprise customer who needed the export to exclude a specific data category for a contractual reason nobody on the engineering team had ever heard of. None of that was in the ticket. None of it was in the prompt either, because the prompt was the ticket. Those were the problems people could see. The ones that didn't arrive as support tickets were worse. --- ## What the one-line prompt actually produced When we read the code properly, after the support tickets had made everyone curious, it looked fine. It was also doing a long list of things nobody had decided. The export query started from the users table and took every column. That included the password hash, the MFA secret and the password reset token. It joined the sessions table and the API keys table, because "their data" plausibly covered both, so the file also contained live credentials. Anyone holding the export could log in as that user. It joined the messages table on conversation rather than on author, so a user's export contained every message other people had sent them. It included the support notes field, which is where the support team writes things like "difficult customer, do not offer discounts." It included rows the user had deleted, because deletion was soft and the query never filtered on it. The endpoint took the account ID as a URL parameter and checked that the caller was logged in, not that the caller owned that account. Change the number, export someone else. It asked for no re-authentication, so a stolen session was one click away from a complete dossier. It sent no notification. It emailed the link to whatever address was on the account at that moment, so an attacker who changed the email first received the export. The link never expired. Then there was the load, and that part did not wait for anyone to read the code. The export ran inside the web request: load everything, build the file, store it, email the link. For a typical account that took two seconds. For a customer who had used the product daily for nine years, with several gigabytes of attachments, it never finished. The load balancer gave up at sixty seconds, but the server kept working. The user saw an error and clicked again, and now two copies were running, because nothing stopped a second export starting while the first was still going. Each copy fetched the account one record at a time, one query per row, which for nine years of activity meant millions of round trips to the database. Each held a single transaction open on the primary for the entire read. A long-open transaction forces the database to keep old versions of every row it might still need to show, so the routine cleanup of those versions stalls and the whole database gradually slows, for every user, not just the one exporting. And each copy built the complete file in memory until the process ran out and was killed, taking down every other request that process happened to be serving. In the first week, two long-time users trying the same thing on the same evening were enough to page someone. The first fix was the obvious one: move the export into a background job. That made the web requests fast again, and it created a fresh set of problems nobody had specified either. There was no limit on how many exports could run at once, so when a product newsletter mentioned the feature a few weeks later, a few hundred users tried it within the same hour and every background worker on the platform filled up with exports. Password reset emails and billing jobs sat in the same queue behind them for most of an afternoon. Failed jobs retried from the beginning, so the largest accounts failed, restarted and failed again, holding a worker each time. And every attempt wrote its temporary file to the job server's local disk and never deleted it, until the disk filled and the server stopped. The agent did nothing unusual. Every one of these is the statistically common way to write an export, which is exactly the problem. None of it looked wrong in a quick review. It was clean, readable code built on decisions nobody had made, and a small test account exercised none of them. --- ## A requirement is not a specification [Part 6](/engineers/the-prompt-that-reveals-everything/) made the case that a requirement and a specification are different objects. This article is about producing the second one. "Users should be able to export their data" is a requirement. It states a desired outcome. It says nothing about format, scope, edge cases, what "their data" includes, or what the feature must never do to the rest of the system. A specification answers the questions a requirement leaves open. Not all of them, not exhaustively, but the ones that change what gets built. The [Enabling Structures](/architecture/) framework we've written about elsewhere calls this the specification gate: a mandatory step before any AI generation begins, where a behavioral spec and its acceptance criteria get written down before a single line of code exists. Teams that skip this step do not get less work. They get the same work, done twice, with a production incident in between. The gate does not require an exhaustive document. It requires the five things Part 6 said a specification has and a requirement does not: valid inputs, expected outputs, failure definition, performance and scale assumptions, and dependency behavior. It also requires the two things Part 6 noticed sit outside those five: the scope, meaning the unit you are working with, and what you are deliberately not building. And one more, which turns the whole thing into something testable: what done means, in terms someone else could verify. The export ticket gestures at one expected output ("their data") and says nothing about the rest. There is a catch in how those categories get filled in. Most of us fill them in for the user in front of the feature. They also have to be filled in for the data behind it and the system around it. Valid inputs covers a user clicking a button. It also covers a stolen session and someone else's account ID typed into the URL. Expected outputs covers the columns a user wants. It also covers the column somebody adds to the schema next year. Performance and scale covers one user waiting for a file. It also covers a nine-year account, and a few hundred users clicking the same button in the same hour. --- ## Rewriting the ticket Here is what a specification for the same feature looks like once those categories get filled in, including the parts the first version never raised. *Scope:* One account per export, for the account the requester is logged into. *Valid inputs:* The account is taken from the session, never from a request parameter. The user must re-enter their password or pass MFA within the ten minutes before requesting. One export per user at a time; a second request while one is running is rejected with a message, not silently queued or silently ignored. *Expected outputs (format):* A ZIP containing one file per data category, because the tools people move their data into import one kind of record at a time, and because a category can then be left out for a customer whose contract excludes it without touching the rest. CSV for flat categories, UTF-8 with a byte order mark so spreadsheet tools read accented characters correctly; JSON for nested ones. Attachments the user uploaded go in as the original files, in a folder of their own. Any CSV cell beginning with `=`, `+`, `-` or `@` is escaped, so a user's own text cannot run as a formula when the file is opened. *Expected outputs (contents):* The contents are an allowlist: profile fields the user entered, their activity history, content and attachments they authored, and billing invoices with bank account numbers masked to the last four digits. Any field not on that list is excluded, including fields added to the schema after this ships. Explicitly excluded: credentials and secrets of any kind (password hashes, MFA secrets, reset and verification tokens, sessions, API keys), content authored by other users, and any category excluded by the customer's contract. Internal fields such as risk scores and support notes, and records the user has deleted, are not in this export either; a user who asks for them goes through the formal data access process, which legal handles separately. *Expected outputs (delivery):* A download link to the registered address, and a notification to the previous address as well if it changed in the last seven days. The download requires the same logged-in user, the link expires after 24 hours, and the file is deleted after seven days. Every export request goes into the audit log. Neither the file contents nor the link are ever written to application logs. Most accounts complete within five minutes of starting; the largest can take up to an hour. When the export pool is full, requests wait in a queue, and the user sees their place in it and the likely wait before they commit. *Failure definition:* A job that fails partway resumes from the last completed category rather than starting again. After three failed attempts the user is notified and no partial file is delivered. A job that has been running for two hours, not counting time spent waiting in the queue, is treated as failed and its worker released. *Performance and scale assumptions:* The largest real account is nine years of activity and several gigabytes of attachments; everything below is sized for that account, not the average one. Exports run as background jobs on their own worker pool, capped at five concurrent jobs across the platform, so they can never starve password resets or billing. They read in batches, never one record at a time, using keyset pagination rather than offsets. They stream straight to object storage instead of building the file in memory or on local disk. No transaction stays open longer than a single batch. *Dependency behavior:* Exports read from a replica, never the primary. If the replica is lagging, the export reflects data up to a few minutes old, and the confirmation says so. If the email provider is down, the file is still available from the settings page and the email is retried. If object storage is unavailable, the job waits and retries; it never falls back to local disk. *Not building now:* Multi-account merging, organisation-wide exports by admins, and exports for deleted or locked accounts. Noted explicitly, rather than left to be discovered later. *Done means:* A user can trigger an export from settings and receive a file that opens cleanly. Verified against a test account built to the size of the largest real one, running alongside four other exports so the pool is full, with primary database latency and password reset delivery times unchanged throughout. Look at where most of that specification goes. The ticket asked for a feature. Most of the spec is about what the feature must not contain and what it must not do to everything around it. There is also no heading called Security, and that is deliberate. Security is not a category; it runs through every one of them. Who may ask is a valid input. What the file may contain is an expected output. Where the link may go, and what gets logged, belong to delivery. A spec that puts security in its own box at the end tends to treat it as a checklist item. Filling in the five categories honestly forces it into every decision instead. The single most important line is *the contents are an allowlist*. A list of fields to exclude is correct on the day it is written and wrong the day someone adds a new sensitive column. A list of fields to include stays correct by default. The agent will almost always produce neither. It takes every column, and the best a reviewer usually adds afterwards is a list of exclusions, because "export everything except the obvious secrets" is what most export code looks like. Every one of those decisions was available before the first line of code. None of it required knowing anything an engineer with domain context didn't already have, or couldn't get by asking one question. It required asking the questions and writing the answers down, instead of generating from the sentence that skipped past them. --- ## Why this got harder to skip Before agents, writing an incomplete spec was survivable, because the engineer building it filled the gaps as they went. They hit the multi-account question three files into the implementation, thought about it for a minute, and made a call. They wrote the query by hand and noticed the password hash in the column list. They ran it against their own long-lived test account and watched it take four minutes. The mental model was under construction the whole time, so the gaps got caught close to where they mattered. An agent does not fill gaps by thinking about your business. It fills gaps by pattern-matching to whatever is statistically common for "data export" in its training data, and it does this instantly and silently. You do not see the moment the gap got filled, because there was no moment. The combined file, the credentials in the export, the account ID in the URL, the one-query-per-row loop on the primary; all of it arrives already decided, wrapped in code that runs and passes a quick smoke test on a small account. The gaps did not go away when the friction of typing every line went away. The friction was doing more work than it looked like it was doing. --- ## What this costs, and what it saves Writing the second version of that spec took about forty minutes longer than pasting the ticket. It is a real cost, and on a busy day it is tempting to skip it, especially when the agent is fast enough that just seeing what comes back feels cheaper than thinking first. It is not cheaper. The team that shipped the one-line version spent days handling the support tickets and patching category exclusion under contractual pressure. Every account that had ever been exported, by its owner or by anyone else, had its sessions and API keys invalidated, its password reset forced and its MFA re-enrolled, because there was no way to know where those files had gone. And because anyone could have exported anyone else's account by changing one number, legal had to treat the whole thing as a possible data breach, on a regulator's 72-hour notification clock. Meanwhile they moved the export into a background job, which fixed the timeouts and then held up password resets for an afternoon, and rebuilt it properly the week after. The forty minutes is not overhead added to the work. It is the work, moved earlier, where it is forty minutes instead of a month of cleanup. --- ## What this means for you Before you next open an agent against a real ticket, try rewriting it against the five categories above, plus scope, what you are not building, and what done means. Do it on paper, before you type anything into the tool. Fill each category in for the data and the system as well as the user. Most tickets leave several of them empty. That gap is not a flaw in whoever wrote the ticket. It is the normal state of a requirement, and turning it into a specification was always your job; it just used to happen invisibly, while you typed the code yourself. The next article is about a faster version of this same check: a sixty-second exercise for finding what's missing from any requirement before you build from it. --- *© Gabor Mayer. Licensed under [Creative Commons Attribution 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Free to share and adapt with attribution.* ==================== # The Interview Skill Map — 2026 (https://aieraengineering.com/interview-skill-map/) The Interview Skill Map Each point = one (skill, method) pair X = how relevant the skill is today · Y = how well the method measures that skill · hover for detail ↖ Good test, wrong skillThe method measures accurately. The skill no longer matters. Wastes interview time selecting for the wrong profile. ↗ Right skill, well measuredThe ideal. Keep and invest. Where new interview methods should land. ↙ Double failureIrrelevant skill, poorly measured. No argument for keeping these. ↘ Right skill, wrong testCritical skills tested with methods that don't capture them. The redesign opportunity. Article 1 — stop asking Article 2 — redesign the test Article 2 & 3 — new methods A two-dimensional map of common software engineering interview methods, scored against 2026 reality. The X axis measures how relevant the underlying skill is after large language models automated most implementation work. The Y axis measures how accurately the method captures that skill. Each dot is one (skill, method) pair — hover for the research behind the placement. ## How to read this chart Each dot is one (skill, method) pair. Hover over any dot to see the skill, the method, the coordinates, and the research behind the placement. The X axis scores how relevant that skill is for software engineering today — far right means essential, far left means obsolete. The Y axis scores how accurately the method measures what it claims to test — top means reliable signal, bottom means noise. Four quadrants: top-right (right skill, well measured — use these), top-left (good test, irrelevant skill — phase out), bottom-right (important skill, poor measurement — redesign these), bottom-left (irrelevant skill, poor measurement — drop immediately). ## The article series This chart is part of a three-article series on hiring software engineers in the AI era. Article 1 — The interview skills that AI made obsolete Article 2 — The top-right quadrant: what still works Article 3 — How to rebuild your hiring process Pre-AI baseline map → ## All methods — full reference table Every (skill, method) pair plotted on the map. Relevance and measurement quality are scored −10 to +10. Hover the chart for the full note and research citation on each point. Method label | Skill tested | How it is tested | Relevance (X) | Measurement quality (Y) | Quadrant | 'What's your biggest weakness?' | Self-awareness | 'What is your biggest weakness?' | -1 | -7 | Double failure | CV walk-through | Self-presentation | CV walk-through / 'tell me about yourself' | -2 | -5 | Double failure | LeetCode | Algorithm reasoning | LeetCode / timed puzzle | -7 | -5 | Double failure | Online coding test (HackerRank / Codility) | Algorithm implementation | Online coding test (HackerRank, Codility) | -7 | -4 | Double failure | Brain teasers | Puzzle-solving / lateral thinking | Brain teasers ('how many golf balls fit in a plane') | -8 | -4 | Double failure | Fermi estimation questions | Estimation / structured thinking | Fermi estimation ('how many X in Y') | -3 | 1 | Good test, wrong skill | Timed bug fix | Implementation speed | Timed bug fix (30 min) | -6 | 5 | Good test, wrong skill | Framework years requirement | Framework knowledge | Job req: '5 years of React' | -8 | 4 | Good test, wrong skill | Syntax & trivia quiz | Language internals / syntax | Trivia quiz ('explain GC', 'what does X do') | -9 | 7 | Good test, wrong skill | Debug — speed only | Debug reasoning | Timed bug fix (speed metric only) | 8 | -4 | Right skill, wrong test | Unstructured behavioural | Communication & problem process | Unstructured behavioural interview | 4 | -3 | Right skill, wrong test | Culture fit interview | Values / cultural alignment | Culture fit interview (unstructured) | 2 | -5 | Right skill, wrong test | Specification test | Specification quality | Give vague req — watch what happens before any tool is touched | 9 | 8 | Right skill, well measured | AI output review | AI output evaluation | Code review of AI-generated code with planted bugs | 9 | 8 | Right skill, well measured | Early adoption question | Early adoption / adaptability | 'What have you tried in the last 3 months that nobody told you to?' | 9 | 7 | Right skill, well measured | Work sample test | Real job performance | Work sample test (actual task from the job) | 9 | 9 | Right skill, well measured | Paid trial period | Actual job performance | Paid trial period (1–5 days of real work) | 9 | 9 | Right skill, well measured | Failure mode exercise | Failure mode reasoning | Describe a system — ask how it fails | 8 | 7 | Right skill, well measured | Debug approach test | Debug reasoning (hypothesis-driven) | Give broken system — observe: hypothesis or paste? | 8 | 7 | Right skill, well measured | Domain scenario test | Domain-contextual judgement | Domain-specific scenario: what's wrong here that tests won't catch? | 7 | 7 | Right skill, well measured | Pair programming | Collaboration + real-time reasoning | Pair programming with interviewer | 7 | 7 | Right skill, well measured | System design whiteboard | Systems thinking & architecture | Whiteboard system design | 6 | 3 | Right skill, well measured | Portfolio / GitHub review | Real engineering capability (past work) | Portfolio / GitHub review | 6 | 5 | Right skill, well measured | Structured behavioural (STAR) | Communication & problem process | Structured behavioural interview (STAR + rubric) | 5 | 6 | Right skill, well measured | Technical presentation (past project) | Deep knowledge of past work | Technical presentation of a past project | 5 | 6 | Right skill, well measured | Take-home assignment | Real implementation ability | Take-home coding assignment (1–7 days) | 1 | 2 | Right skill, well measured | ## Research backing The Y-axis placement of each method is grounded in I/O psychology research. Key sources: Schmidt & Hunter (1998) Psychological Bulletin (meta-analysis of 85 years of selection research); Sackett, Zhang, Berry & Lievens (2022) Journal of Applied Psychology (corrected validity coefficients); Roth, Bobko & McFarland (2005) Personnel Psychology (work sample tests); Rivera (2012) American Sociological Review (cultural fit bias); Laszlo Bock / Google (2013) on brainteasers. The X-axis reflects the documented shift in software engineering work caused by large language models. Implementation tasks (syntax recall, algorithm puzzles, fast bug-fixing) have been automated since 2022. Specification, architectural reasoning, and AI output evaluation have been elevated. The before/after comparison between this chart and the pre-AI baseline quantifies the shift. © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution. ==================== # The Interview Skill Map — Pre-AI Baseline (https://aieraengineering.com/interview-skill-map-preai/) The Interview Skill Map — Pre-AI Baseline Same methods, same axes — scored against pre-2022 skill relevance X = how relevant the skill was then · Y = how well the method measured that skill · hover for detail ↖ Good test, wrong skillAccurately measures something that didn't matter even then. Rare in the pre-AI world — most bad tests were also inaccurate. ↗ Right skill, well measuredMost of the traditional methods sat here. This quadrant largely emptied after 2022 as skills shifted left. ↙ Double failureBad methods for irrelevant skills. These were already wrong before AI — brain teasers, "biggest weakness", culture fit. ↘ Right skill, wrong testSkills that mattered but were measured poorly. The opportunity that existed even then — and still does. Sound method for the era Widely used, had issues Wrong even then Valid but rarely used / didn't exist yet The same interview methods mapped against pre-2022 skill relevance. Before large language models, syntax knowledge, implementation speed, and algorithm skill sat in the top-right quadrant — relevant and well-tested. This chart shows the baseline most hiring processes were designed for, and shows exactly which methods lost their signal when AI arrived. ## How to read this chart Each dot is one (skill, method) pair. Hover over any dot to see the skill, the method, the coordinates, and the research behind the placement. The X axis scores how relevant that skill is for software engineering today — far right means essential, far left means obsolete. The Y axis scores how accurately the method measures what it claims to test — top means reliable signal, bottom means noise. Four quadrants: top-right (right skill, well measured — use these), top-left (good test, irrelevant skill — phase out), bottom-right (important skill, poor measurement — redesign these), bottom-left (irrelevant skill, poor measurement — drop immediately). ## The article series This chart is part of a three-article series on hiring software engineers in the AI era. Article 1 — The interview skills that AI made obsolete Article 2 — The top-right quadrant: what still works Article 3 — How to rebuild your hiring process AI-era map (2026) → ## All methods — full reference table Every (skill, method) pair plotted on the map. Relevance and measurement quality are scored −10 to +10. Hover the chart for the full note and research citation on each point. Method label | Skill tested | How it is tested | Relevance (X) | Measurement quality (Y) | Quadrant | CV walk-through | Self-presentation | CV walk-through / 'tell me about yourself' | -1 | -5 | Double failure | 'What's your biggest weakness?' | Self-awareness | 'What is your biggest weakness?' | -1 | -7 | Double failure | Brain teasers | Puzzle-solving / lateral thinking | Brain teasers ('how many golf balls fit in a plane') | -7 | -4 | Double failure | AI output review | AI output evaluation | Code review of AI-generated code with planted bugs | -9 | -9 | Double failure | Fermi estimation questions | Estimation / structured thinking | Fermi estimation ('how many X in Y') | -1 | 1 | Good test, wrong skill | Unstructured behavioural | Communication & problem process | Unstructured behavioural interview | 4 | -3 | Right skill, wrong test | Culture fit interview | Values / cultural alignment | Culture fit interview (unstructured) | 2 | -5 | Right skill, wrong test | Work sample test | Real job performance | Work sample test (actual task from the job) | 9 | 9 | Right skill, well measured | Paid trial period | Actual job performance | Paid trial period (1–5 days of real work) | 9 | 9 | Right skill, well measured | Take-home assignment | Real implementation ability | Take-home coding assignment (1–7 days) | 8 | 7 | Right skill, well measured | Debug approach test | Debug reasoning (hypothesis-driven) | Give broken system — observe: hypothesis or paste? | 8 | 7 | Right skill, well measured | Framework years requirement | Framework knowledge | Job req: '5 years of React' | 7 | 4 | Right skill, well measured | Failure mode exercise | Failure mode reasoning | Describe a system — ask how it fails | 7 | 7 | Right skill, well measured | Pair programming | Collaboration + real-time reasoning | Pair programming with interviewer | 7 | 7 | Right skill, well measured | LeetCode | Algorithm reasoning | LeetCode / timed puzzle | 6 | 3 | Right skill, well measured | Timed bug fix | Implementation speed | Timed bug fix (30 min) | 6 | 5 | Right skill, well measured | Online coding test (HackerRank / Codility) | Algorithm implementation | Online coding test (HackerRank, Codility) | 6 | 5 | Right skill, well measured | System design whiteboard | Systems thinking & architecture | Whiteboard system design | 6 | 3 | Right skill, well measured | Portfolio / GitHub review | Real engineering capability (past work) | Portfolio / GitHub review | 6 | 5 | Right skill, well measured | Syntax & trivia quiz | Language internals / syntax | Trivia quiz ('explain GC', 'what does X do') | 5 | 7 | Right skill, well measured | Structured behavioural (STAR) | Communication & problem process | Structured behavioural interview (STAR + rubric) | 5 | 6 | Right skill, well measured | Technical presentation (past project) | Deep knowledge of past work | Technical presentation of a past project | 5 | 6 | Right skill, well measured | Domain scenario test | Domain-contextual judgement | Domain-specific scenario: what's wrong here that tests won't catch? | 5 | 7 | Right skill, well measured | Specification test | Specification quality | Give vague req — watch what happens before any tool is touched | 3 | 8 | Right skill, well measured | Early adoption question | Early adoption / adaptability | 'What have you tried in the last 3 months that nobody told you to?' | 1 | 7 | Right skill, well measured | ## Research backing The Y-axis placement of each method is grounded in I/O psychology research. Key sources: Schmidt & Hunter (1998) Psychological Bulletin (meta-analysis of 85 years of selection research); Sackett, Zhang, Berry & Lievens (2022) Journal of Applied Psychology (corrected validity coefficients); Roth, Bobko & McFarland (2005) Personnel Psychology (work sample tests); Rivera (2012) American Sociological Review (cultural fit bias); Laszlo Bock / Google (2013) on brainteasers. The X-axis reflects the documented shift in software engineering work caused by large language models. Implementation tasks (syntax recall, algorithm puzzles, fast bug-fixing) have been automated since 2022. Specification, architectural reasoning, and AI output evaluation have been elevated. The before/after comparison between this chart and the pre-AI baseline quantifies the shift. © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution. ==================== # System Engineer: Question Bank (https://aieraengineering.com/system-engineer-question-bank/) The AI-Era Engineering Playbook — Practitioner Reference ## Design Principles This bank is organised by interview stage rather than by skill dimension. Each stage maps to a distinct assessment method type with its own validity profile. The ordering is intentional: stages run from least to most authentic, ending in the highest-validity method (structured behavioural with past-behaviour evidence). ### Scientific basis for stage design Stage | Assessment Method | Validity (r) | Source | Bloom Level | 1 — Systems Case Study | Work sample (designed scenario) | .33–.54 | Roth et al. 2005; Schmidt & Hunter 1998 | Analyse / Evaluate | 2 — Failure Mode Review | Work sample (incident scenario) | .33–.54 | Roth et al. 2005 | Analyse / Evaluate | 3 — AI Output Audit | Work sample (live code review) | .33–.54 | Roth et al. 2005 | Evaluate / Create | 4 — Structured Behavioural | Structured behavioural interview | .42 | Sackett et al. 2022 | Apply / Analyse | All four stages apply authentic assessment principles (Wiggins & McTighe 1998): the task mirrors the real job condition, not an abstracted proxy. Each question targets constructs at Bloom's analyse, evaluate, or create level — not recall. Key validity references: - Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262–274. - Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(12), 2040–2068. - Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity. Personnel Psychology, 58, 1009–1037. - Wiggins, G., & McTighe, J. (1998). Understanding by Design. ASCD. - Lave, J., & Wenger, E. (1991). Situated Learning. Cambridge University Press. ### Scoring convention All questions use a 1–5 scale. Score based on observed behaviour, not inferred ability. If not demonstrated, score low. Score | Label | Meaning | 5 | Exceptional | Exceeds expected depth; demonstrates mastery with specific, unprompted insight | 4 | Strong | Covers core fully; minor gaps; demonstrates clear competence | 3 | Adequate | Covers the basics; requires prompting to reach depth; no major gaps | 2 | Weak | Partial coverage; significant gaps; cannot reach depth even when prompted | 1 | Absent | Not demonstrated; cannot engage meaningfully with the question | Auto-reject: Any dimension with two or more Q scores of 1. ## Stage 1: Systems Thinking and Architecture Assessment method: Case study discussion Skill map position: System Design (restructured) — X=+6, Y=+6 Validity basis: B — high ecological validity; mirrors actual architectural decision-making Construct: Ability to design systems with explicit constraints, tradeoffs, and failure awareness; ability to design for others Administration notes: - Provide scenario in writing. Give 10 minutes to read and think before discussion. - Do not prompt architectural solutions — let the candidate drive. - Score on reasoning process, not on whether their architecture matches any "correct" answer. - The question is not solvable in 10 minutes. That is intentional. You are watching how they handle incompleteness. Sample scenario (rotate per candidate): A fintech startup wants to build a payment processing system. It needs to handle 1,000 transactions per second at peak, support multiple currencies, integrate with three external payment providers, and have 99.99% uptime. The team building it will include two engineers who are strong AI users but not deep systems people. You are the only System Engineer. ### S1-Q1. "Before talking about implementation — what constraints would you define to make this system safe for the other engineers to build within?" Construct: Constraint-first thinking; designing for others; safety boundary definition Bloom level: Evaluate / Create Chart method: Specification test (X=+9, Y=+8) Research note: The ability to define constraints before implementation is the primary differentiator between senior and junior system design. The specification test has high face and ecological validity (Wiggins & McTighe 1998). Constraint identification precedes implementation in all high-reliability engineering contexts. Scoring rubric: Score | Behavioural description | 5 | Immediately defines constraints as preconditions — not implementation decisions. Distinguishes between invariants (must not be violated) and goals (targets). Explicitly considers the non-expert builders: names constraints that limit the blast radius of errors they will predictably make. Names at least 3 non-obvious constraints. | 4 | Identifies constraints before implementation. Covers core categories (availability, data integrity, failure isolation). Considers the other engineers. May miss 1–2 non-obvious constraint categories. | 3 | Raises constraints when prompted or moves towards them after framing. Covers basic consistency and availability. Does not initially consider the other engineers as a design input. | 2 | Jumps to architecture (microservices, queues, databases) before defining constraints. Cannot enumerate constraints without direct prompting. | 1 | Treats the question as asking for an architecture proposal. Has no framework for constraint definition. Constraints are presented as "best practices" rather than system properties. | Red flags: - Immediately names a technology ("I'd use Kafka for this") - Defines constraints as implementation details ("we'd use idempotency keys") - Does not mention the non-expert builders at all Probing follow-up: "What happens if one of the other engineers misunderstands a constraint? What does the system do?" ### S1-Q2. "What are the hardest tradeoffs in this design? Where would you sacrifice to get something essential?" Construct: Tradeoff reasoning; intellectual honesty about competing constraints Bloom level: Evaluate Chart method: Failure mode exercise (X=+8, Y=+7) Research note: Tradeoff reasoning is a core higher-order thinking skill (Bloom's evaluate). Unlike recall or implementation, it cannot be replicated by AI. Candidates who cannot name tradeoffs have likely never made real architectural decisions under constraint. Scoring rubric: Score | Behavioural description | 5 | Names 2–3 genuine, non-obvious tradeoffs specific to this scenario. Explains what they would sacrifice and why. Demonstrates awareness that tradeoffs are not just technical but organisational (e.g., "strong consistency slows development for the other engineers"). Shows understanding that every architectural decision forecloses other decisions. | 4 | Identifies 2 real tradeoffs. Reasoning is clear. May focus primarily on technical tradeoffs (latency vs. consistency) without organisational dimension. | 3 | Identifies 1 tradeoff. Reasoning is present but shallow ("you trade off availability for consistency"). Does not engage with the specific context of this system. | 2 | States general principles without applying them. Tradeoffs are presented as solved by known patterns rather than as genuine choices under constraint. | 1 | Cannot identify tradeoffs. Describes an architecture as if it has no costs. | Red flags: - "You'd just use eventual consistency" without explaining what that means for payments - Treats CAP theorem as a complete answer rather than a starting point - No acknowledgment that different tradeoffs suit different constraint profiles Probing follow-up: "If you chose consistency over availability and the payment provider is down — what does the user experience? Is that acceptable?" ### S1-Q3. "What would the junior engineers break first? How would you prevent it architecturally rather than through code review?" Construct: Anticipatory failure reasoning; designing for error surfaces; system trust model Bloom level: Analyse / Create Chart method: Failure mode exercise (X=+8, Y=+7) Research note: This question directly tests the shift from "I build for myself" to "I build for others." A System Engineer who cannot anticipate how less experienced engineers will interact with their system is a single point of failure. The architecture must constrain error, not just detect it. Scoring rubric: Score | Behavioural description | 5 | Names 2–3 specific, predictable failure modes non-expert builders will cause. Proposes architectural solutions (not code review solutions) for each: e.g., interface contracts that make the wrong operation impossible, circuit breakers that are automatic rather than opt-in, event schemas that validate at the boundary. Distinguishes between "hard to do wrong" and "reviewable after the fact." | 4 | Identifies 2 predictable errors. Proposes at least one architectural solution rather than a process one. May default to "we'd have review gates" for the harder cases. | 3 | Identifies general risk areas ("they might not handle failures correctly"). Proposes code review or documentation as the primary mitigation. Architecture as a constraint mechanism is not the default. | 2 | Answers in terms of what they would tell the junior engineers, not what the architecture would enforce. | 1 | Cannot anticipate errors non-expert engineers would make. Assumes others will use the system correctly. | Red flags: - "I'd document everything clearly" as the primary answer - "We'd review all their PRs" — this is a process patch, not an architectural solution - No distinction between detecting errors and preventing them structurally Probing follow-up: "If one of them deletes a retry logic because it looks redundant — what happens? Can the architecture stop that from mattering?" ### S1-Q4. "What's the failure mode you're most worried about that would be invisible until production?" Construct: Silent failure identification; prospective systems reasoning Bloom level: Analyse / Evaluate Chart method: Failure mode exercise (X=+8, Y=+7) Research note: Silent failures — those that pass tests, pass review, and only surface under real load or edge conditions — are the highest-cost failure type in production systems. Candidates who enumerate only obvious failures (service down, exception thrown) have not operated systems at scale. Scoring rubric: Score | Behavioural description | 5 | Names a specific silent failure that is non-obvious and realistic for this scenario (e.g., partial duplicate transactions that don't error but don't reconcile; currency rounding that silently accumulates; idempotency keys that collide under specific load patterns). Explains why it would pass all tests. Proposes a detection strategy, not just a fix. | 4 | Names a realistic silent failure. Can explain why it is hard to detect. May not have a detection strategy beyond "we'd need monitoring." | 3 | Names a failure mode that is somewhat visible (slow queries, partial downtime). May describe an obvious failure dressed as a silent one. | 2 | Conflates "rare" with "invisible." Describes loud failures (exceptions, outages) as the most serious. | 1 | Cannot identify silent failures. All failure modes named are outage-level and observable. | Red flags: - Only names failures that would be caught by a test suite - "We'd add logging" as the primary answer — logging only helps if you know what to log for - Conflates failure probability with failure severity Probing follow-up: "How would you design the observability layer to catch this before a customer reports it?" ### S1-Q5. "How would you document this so the next engineer can understand why it was built this way — not just what it does?" Construct: Knowledge transfer; architectural intent preservation; documentation as system design Bloom level: Create Chart method: Structured behavioural (X=+5, Y=+6) Research note: Architectural decisions decay in undocumented systems. The why behind an architectural decision is the most valuable and most commonly absent information in engineering documentation. This question tests mental model transfer, not writing skill. Scoring rubric: Score | Behavioural description | 5 | Distinguishes between what documentation (code comments, API docs) and why documentation (architecture decision records, constraint logs). Has a concrete format: records the decision, the alternatives considered, the constraints that ruled them out, and the conditions under which the decision should be revisited. Has done this before and can describe the outcome. | 4 | Understands the why/what distinction. Proposes ADRs or similar. Has not always done this consistently but understands why it matters. | 3 | Describes README-style documentation. Understands that why is important but does not have a structured approach for capturing it. | 2 | Documents implementation, not decisions. "I comment the tricky parts." | 1 | Does not distinguish between implementation documentation and decision documentation. | Red flags: - "The code should be self-documenting" — applies to implementation, not architectural decisions - Cannot name an example of documentation they've written that explained a tradeoff - "Future engineers should be smart enough to figure it out" Probing follow-up: "Tell me about a system you've inherited where the documentation was good. What specifically made it useful?" ## Stage 2: Failure Mode Reasoning Assessment method: Incident scenario review Skill map position: Debug approach (restructured) — X=+8, Y=+7 Validity basis: B — directly mirrors incident investigation; high ecological validityConstruct: Hypothesis formation; distributed system failure reasoning; the "fix the class, not the bug" distinction Administration notes: - Provide the system description and incident report in writing. Give 10 minutes to review. - Do not reveal additional system details until asked — how they handle information gaps is part of the assessment. - Score on reasoning process: hypothesis formation, diagnosis structure, and the breadth of failure categories enumerated. Sample material (rotate per candidate): System description: A user authentication service. Users log in and receive a JWT token valid for 24 hours. The token is validated on each request. Tokens can be revoked by admins — stored in a Redis cache with a 24-hour TTL matching the token TTL. Incident report: A user's token was revoked due to suspected fraud. Six hours later, they were still making successful API calls. Redis confirmed the token was marked as revoked. The API logs showed successful authentication for all calls. ### S2-Q1. "What's your hypothesis about what went wrong? Walk me through your top three candidates." Construct: Hypothesis generation; distributed system mental model; diagnostic structure Bloom level: Analyse Chart method: Debug approach — hypothesis vs. paste (X=+8, Y=+7) Research note: Hypothesis-driven debugging requires a mental model of the system. Prompt-driven debugging (asking AI "why is this broken") requires no mental model and produces no durable understanding. This question separates the two directly. Scoring rubric: Score | Behavioural description | 5 | Generates 3 distinct, mechanistically plausible hypotheses without prompting. Immediately identifies token caching at the API layer as the primary candidate (not just the revocation mechanism). Distinguishes between "the revocation didn't propagate" and "the revocation propagated but was ignored." Names the specific system components that could cause each hypothesis. | 4 | Generates 2–3 hypotheses. Identifies the API-layer cache as a likely candidate. Reasoning is structured. May need light prompting to reach the third candidate. | 3 | Identifies 1–2 plausible hypotheses. May focus on the revocation side (Redis write failure) rather than the validation side (cache miss on read). | 2 | Generates generic hypotheses ("there's a bug in the token validation"). Does not use the specific system description to constrain the hypothesis space. | 1 | Cannot generate hypotheses. Reads incident as "the system is broken" without mechanistic reasoning. | Red flags: - Focuses only on Redis (the revocation path) and ignores the API (the validation path) - Does not ask clarifying questions about the system before hypothesizing - "There must be a race condition" without explaining the specific race Probing follow-up: "The incident report says Redis showed the token as revoked and the API still authenticated. What does that tell you about where the failure is?" ### S2-Q2. "How would you confirm or eliminate each hypothesis without access to the running system?" Construct: Experimental reasoning; evidence-based diagnosis; system observability awareness Bloom level: Analyse / Evaluate Chart method: Debug approach (X=+8, Y=+7) Research note: This question assesses whether the candidate can reason about evidence structure — what would confirm vs. disconfirm each hypothesis, and how to obtain that evidence. Engineers without distributed system experience tend to reach for "reproduce it" as the only strategy. Scoring rubric: Score | Behavioural description | 5 | For each hypothesis, names a specific observable that would confirm or eliminate it: API access log timestamps vs. Redis revocation timestamps; presence of a token validation cache vs. direct Redis lookup per request; TTL behaviour of any intermediary cache. Understands that absence of evidence is also evidence in some cases. | 4 | Proposes evidence-based tests for most hypotheses. May rely on log correlation without fully specifying the evidence structure. | 3 | Proposes "check the logs" without specifying what to look for or what pattern would confirm or eliminate the hypothesis. | 2 | Proposes reproduction steps as the primary diagnostic strategy. Does not engage with the evidence structure question. | 1 | Cannot propose a diagnostic strategy without access to the running system. | Red flags: - "I'd add more logging" without specifying what the logs should show - Proposes a fix before completing the diagnosis - Conflates "this is where I'd look" with "this is what I'd conclude" Probing follow-up: "What's the simplest piece of evidence that would let you eliminate two hypotheses at once?" ### S2-Q3. "Why wouldn't this have appeared in your test suite?" Construct: Test suite limitations; distributed failure class awareness; the gap between test coverage and correctness Bloom level: Analyse / Evaluate Research note: This question tests whether the candidate understands the structural gap between unit/integration tests and distributed system behaviour. It is a direct application of situated cognition (Lave & Wenger 1991): behaviour in a test harness does not reliably predict behaviour in production under distributed timing. Scoring rubric: Score | Behavioural description | 5 | Identifies that unit tests mock external calls — the Redis lookup is mocked and always returns "revoked" in the test. Integration tests likely test the happy path and a single-call revocation, not a cached revocation under timing. End-to-end tests run against a clean Redis with no pre-existing state. Names what type of test would catch this (a distributed timing test with real TTL behaviour and simulated propagation delay). | 4 | Identifies that tests don't test timing and propagation. Understands mocking as the mechanism of failure. May not identify the specific test type that would catch it. | 3 | Notes that "this is hard to test." Does not engage with the specific mechanism (mocking, TTL simulation, propagation timing). | 2 | "Tests wouldn't catch everything." Cannot explain why. | 1 | Believes a comprehensive test suite would catch this. | Red flags: - "We should add a test for this" without understanding why it wasn't there - Conflates code coverage with behavioural coverage - "The test suite should have caught this" — implying it's a test quality failure, not a structural limitation Probing follow-up: "What would a test for this look like? What would it have to simulate that most test environments don't?" ### S2-Q4. "What architectural change would prevent this class of problem — not just this specific bug?" Construct: Class-level thinking; architectural remediation vs. patch thinking Bloom level: Create Chart method: Failure mode exercise (X=+8, Y=+7) Research note: The distinction between "fix the bug" and "fix the class of bug" is the primary differentiator between senior and junior engineering thinking. A patch closes one hole; an architectural change closes the category. This is the highest-Bloom-level question in this stage. Scoring rubric: Score | Behavioural description | 5 | Identifies the class of problem: "token validation decisions made from a cache that may not reflect current revocation state." Proposes an architectural solution that addresses the class: event-driven invalidation that pushes to all caches; or a pull-through pattern that makes staleness bounded and explicit; or a token design that encodes revocation state directly (short-lived tokens + refresh, eliminating the need for revocation caches). Explicitly names what problem each approach introduces and why one is preferable given the constraints. | 4 | Identifies the class: caches that can be inconsistent with authoritative state. Proposes a sound architectural fix (shorter TTLs, push invalidation). May not enumerate the tradeoffs introduced by the fix. | 3 | Proposes a targeted fix for this specific case ("check Redis more frequently"). Does not identify the class of problem. | 2 | Proposes a monitoring or alerting solution. The system itself does not change. | 1 | Proposes a process fix ("review all revocation flows"). No architectural content. | Red flags: - Proposes a "belt and suspenders" approach that adds checks without changing the architecture - Does not distinguish between solutions that close one hole vs. solutions that seal the class - Cannot explain what problem their architectural fix introduces Probing follow-up: "If you moved to short-lived tokens with refresh — what new failure mode does that introduce? Is it better or worse than the original problem?" ## Stage 3: AI Output Audit Assessment method: Live code review exercise Skill map position: AI output review (X=+9, Y=+8) Validity basis: B — directly mirrors daily job task; high ecological validity Construct: Ability to evaluate AI-generated code for behavioural correctness, security, and architectural fit — not style Administration notes: - Provide 60–100 lines of AI-generated code with exactly 3 planted issues. Do not tell the candidate how many issues are present. - Give 15 minutes to review independently. Then discuss for 30 minutes. - Score on what they find and how they reason about it, not on whether they fix it. - Rotate issues per candidate to prevent question bank leak. Issue categories to plant (choose 3, vary each time): Code issue | Category | What a strong candidate sees | Token validation that checks signature but not expiry independently | Security | "This trusts the embedded expiry claim without verifying it against the server state — the signature can be valid on an expired token if the library is misconfigured" | Amount calculation that silently truncates decimal in currency conversion | Silent logic error | "This loses sub-cent amounts on every conversion — individually small, but accumulates and creates reconciliation gaps at scale" | Retry implementation that uses a tight loop instead of the established exponential backoff pattern | Architectural mismatch | "This doesn't follow the retry contract the rest of the system expects — under load it floods the downstream service" | Null check absent on user preferences lookup | Edge case | "If the user has no preferences set, this silently proceeds with defaults that may not be safe for all event types" | User-supplied field interpolated into query string | SQL injection | "This is injectable — user_id goes directly into the query string" | Mutex that is locked but never released on error path | Resource leak | "If the external call fails, the lock never gets released — this will deadlock under concurrent load" | ### S3-Q1. "Walk me through what you'd want to change before approving this." Construct: Code review judgement; behavioural correctness evaluation; security awareness Bloom level: Evaluate Research note: This question tests whether the review is driven by a mental model of correct behaviour or by surface-level pattern matching. Strong candidates evaluate against specification and expected behaviour; weak candidates evaluate against style conventions. Scoring rubric: Score | Behavioural description | 5 | Finds all 3 planted issues. Explains each in terms of behaviour under the failure condition, not just "this is wrong." Identifies at least one issue as belonging to a class (e.g., "this is an input validation gap — there will be other places in the codebase where this occurs"). May identify additional issues not planted. Does not spend significant time on style. | 4 | Finds 2–3 issues. Reasoning is behavioural ("what would happen if..."). May not identify the class. Proportionate time on substantive vs. style issues. | 3 | Finds 1–2 issues. May include style issues in the count. Reasoning is present but requires prompting to go beyond "this looks wrong." | 2 | Finds style or convention issues primarily. May find 1 substantive issue. Cannot explain the behaviour impact. | 1 | "This looks fine to me." No substantive issues identified. | Red flags: - Spends the majority of time on naming conventions, formatting, or import organisation - Cannot explain why an identified issue is a problem — only that it "looks wrong" - Approves code with a security issue without identifying it Probing follow-up (if they miss an issue): "What happens when [specific edge condition from the planted issue]? Walk me through the execution path." ### S3-Q2. "Is there anything here that would pass all tests but fail in production?" Construct: Test/production gap awareness; silent failure identification; behavioural correctness reasoning Bloom level: Analyse / Evaluate Research note: This question directly activates situated cognition (Lave & Wenger 1991): the candidate must reason about behaviour in a production context, not a test context. It is the same construct as S2-Q3 applied to code rather than architecture. Scoring rubric: Score | Behavioural description | 5 | Identifies at least one planted issue that would pass tests and names specifically why it passes (mocked dependencies, missing edge case in test fixtures, assertion that doesn't cover the failure path). Demonstrates understanding that test suites certify behaviour under tested conditions, not under all conditions. | 4 | Identifies an issue as likely to pass tests. Can explain why in general terms (testing happy path, mock hides the failure). | 3 | "There might be edge cases the tests don't cover." Cannot be specific about which ones or why. | 2 | "Tests should cover this." Does not engage with the structural gap. | 1 | Conflates test passage with production correctness. | Red flags: - "If the tests pass, it's probably fine" - Cannot give a mechanism for why an issue would pass tests - Identifies only issues that would also fail in tests Probing follow-up: "If I run the full test suite and it passes — which of the issues you found would still be present?" ### S3-Q3. "What questions would you ask the author before approving?" Construct: Review as dialogue; specification recovery; understanding intent vs. implementation Bloom level: Evaluate Research note: This question tests whether the candidate understands code review as a bidirectional process for recovering intent, not a unilateral judgement. Strong reviewers use questions to resolve ambiguity; weak reviewers either approve or reject without engaging with intent. Scoring rubric: Score | Behavioural description | 5 | Asks questions that distinguish between "the author made an error" and "the author made a design decision I don't have context for." Questions probe the specification, not just the code: "What behaviour is intended if the user preferences are null?" not "Why did you write it this way?" At least one question targets a planted issue and would reveal whether it was intentional or accidental. | 4 | Asks 2–3 substantive questions. Most are specification-oriented. Some may be phrased as implementation questions rather than intent questions. | 3 | Asks 1–2 questions. Questions are code-level ("why is this hardcoded?") rather than specification-level. | 2 | Would approve without questions, or would reject without questions. Review is a binary judgement, not a process. | 1 | Cannot articulate what they'd ask. | Red flags: - Questions are all about style or convention - "I'd just fix it myself" — bypasses the intent recovery step - Cannot distinguish between issues that need context and issues that are clearly wrong ### S3-Q4. "If you approved this, what would you add to the test suite that isn't there?" Construct: Test design; failure path enumeration; translating behavioural concerns into test cases Bloom level: Create Research note: This question closes the loop between finding an issue and knowing how to verify the fix. It is the create level of Bloom's taxonomy applied to testing — not recall of test patterns, but construction of test cases from behavioural requirements. Scoring rubric: Score | Behavioural description | 5 | Proposes specific test cases (not categories) for each identified issue. Each test specifies the setup condition, the operation, and the expected behaviour. At least one test targets a failure path. Understands the difference between a unit test that mocks the failure and an integration test that simulates it. | 4 | Proposes test cases for most issues. Tests are specific enough to be implementable. May not address whether the test should be unit or integration. | 3 | Proposes test categories ("we need tests for the null case") without specifying the test. | 2 | "Add more tests" without specifics. | 1 | Cannot propose test cases from a behavioural issue. | Red flags: - "The test coverage is already pretty good" after finding issues - Proposes tests that would mock the exact behaviour that caused the issue - Cannot specify what a failing test would look like before the fix ## Stage 4: Structured Behavioural Interview Assessment method: Structured behavioural interview (STAR format with scoring rubric) Skill map position: Structured behavioural interview (X=+5, Y=+6) Validity basis: A — Sackett et al. (2022): r=.42 for structured format; the structure and consistent rubric are essential to reaching this validity level Construct: Past behaviour as predictor of future behaviour; track record of real-world application of the constructs tested in Stages 1–3 Administration notes: - Ask all questions in the same order, to all candidates, without alteration. - Do not help the candidate structure their answer. If they give an incomplete story, probe with the standard follow-ups. - Score independently from your co-interviewer before comparing notes. - The purpose of this stage is to verify that the observable skills in Stages 1–3 are not one-day performances — they should be a pattern across multiple past situations. Why behavioural evidence matters here: Schmidt & Hunter (1998) note that the combination of cognitive/work-sample assessment with structured behavioural interview outperforms either method alone. Stage 4 provides the past-behaviour evidence that validates Stages 1–3. A candidate who performs well on a case study but cannot recall a single real instance of that reasoning raises a construct validity concern. ### S4-Q1. "Tell me about the most architecturally complex system you've designed. What constraints were non-negotiable and why?" Construct: Real-world systems thinking; constraint reasoning under actual production conditions Bloom level: Analyse (retrospective) Scoring rubric: Score | Behavioural description | 5 | Describes a genuinely complex system with specific, named non-negotiable constraints. The constraints are principled (not "the PM said so") — they reflect understanding of why those properties matter for that system. Can explain what would have broken if a constraint had been relaxed. Shows retrospective judgement: what they'd do differently. | 4 | Describes a complex system. Names constraints. The reasoning for why they're non-negotiable is present but may not distinguish between principled and imposed constraints. | 3 | Describes a moderately complex system. Constraints are general ("it had to be reliable, it had to scale"). Cannot name a specific constraint and defend why it was non-negotiable over another. | 2 | Describes a system they maintained rather than designed. Constraints are implicit or absent. | 1 | Cannot name a system they designed or cannot identify constraints within it. | Standard follow-ups: - "What would have happened if you'd relaxed [specific constraint]?" - "What architectural decision did you make that you wouldn't make again? What did you learn?" ### S4-Q2. "Tell me about an architectural decision you made that turned out to be wrong. What did you miss and why?" Construct: Retrospective judgement; intellectual honesty; learning from failure Bloom level: Evaluate (retrospective) Research note: The willingness and ability to critique one's own past decisions is a strong signal of both intellectual honesty and experience. A candidate who cannot recall an architectural error has either not made meaningful architectural decisions, or does not reflect on their work, or is being dishonest. All three are concerning. Scoring rubric: Score | Behavioural description | 5 | Names a specific decision, the reasoning at the time, the specific assumption that was wrong, and what they'd do differently. Does not externalize blame ("requirements changed") — takes ownership of the failure in the reasoning. Derives a generalizable lesson from the specific case. | 4 | Names a specific decision and what went wrong. Reasoning at the time is partially described. Some externalization is present but not dominant. | 3 | Describes a mistake but frames it as bad luck or changing requirements rather than flawed reasoning. The lesson derived is not generalizable. | 2 | Names a minor technical error (wrong library, wrong approach to a small problem) rather than an architectural decision. | 1 | "I can't think of one" or only describes others' decisions. | Standard follow-ups: - "What was the assumption that turned out to be wrong? When did you know it was wrong?" - "What would you tell yourself at the moment you made that decision?" ### S4-Q3. "Tell me about a time you had to prevent someone from building something the wrong way — without just taking over." Construct: Knowledge transfer; teaching under constraint; influence without authority Bloom level: Apply / Create (retrospective) Scoring rubric: Score | Behavioural description | 5 | Describes a specific situation where they used questions, constraints, or reframing to redirect someone rather than overriding them. The outcome demonstrates that the other person understood the why, not just the what. Reflects on what made this approach work. Shows awareness that taking over is a trap: it resolves the immediate problem but creates a dependency. | 4 | Describes a real situation with a clear outcome. The approach was more collaborative than directive. The other person arrived at the right solution. | 3 | Describes a situation but the outcome relied primarily on the candidate's authority or direct correction. The teaching element is thin. | 2 | Describes reviewing code and leaving comments. No interpersonal dynamic. | 1 | Cannot recall a situation or describes taking over as the solution. | Standard follow-ups: - "What would you have done if they hadn't responded to your approach?" - "How did you know it was working?" ### S4-Q4. "Where do you draw the line in your own work between what you delegate to AI and what you own directly? Why there?" Construct: AI integration judgement; self-awareness about dependency risk; system engineer's relationship to AI tooling Bloom level: Evaluate Research note: This question has no equivalent in pre-2023 interview design. It targets a new construct: the engineer's ability to reason about their own AI dependency profile. There is no peer-reviewed validity data for this specific question. It is placed in Stage 4 (behavioural) rather than Stage 1 (case study) because past behaviour is more reliable than hypothetical framing. Scoring rubric: Score | Behavioural description | 5 | Draws a principled line based on consequences: delegates things where errors are visible and reversible, retains ownership of things where errors are invisible or compounding. Gives specific examples of both. Has revised this line as AI capability has changed. Shows awareness that the line is not static. | 4 | Draws a recognizable line with reasoning. Examples are specific. The line may not be fully articulated as a principle but is consistent in practice. | 3 | Draws a line based on comfort or habit ("I just feel more confident when I write certain things myself") without a principle. | 2 | "I use AI for everything / I use AI for nothing" — either extreme without nuance. | 1 | Has not thought about this. Delegates or retains based on workflow convention, not judgement. | Standard follow-ups: - "Has the line moved in the last year? What moved it?" - "What would have to be true for you to delegate something you currently own directly?" ### S4-Q5. "What tools are you currently experimenting with that you haven't brought to a team yet? What's your current assessment?" Construct: Early adoption behaviour; self-directed learning; frontier awareness Bloom level: Evaluate (ongoing) Research note: Early adoption is a behavioural pattern, not a trait. The question is asked in present tense and asks for an ongoing experiment — not a past adoption. A candidate who cannot answer this question is not currently experimenting. The question is resistant to fabrication: claiming a current experiment requires specificity about what the tool does, where it falls short, and what the candidate has tried. Scoring rubric: Score | Behavioural description | 5 | Names a specific tool or technique, describes what they are using it for, gives a concrete current assessment including where it falls short. The assessment is formed from use, not from reviews or announcements. May be evaluating something that has not yet reached mainstream awareness. Shows a systematic approach to evaluation (tries, forms opinion, decides to adopt or reject with reasoning). | 4 | Names a specific tool with a specific assessment. The assessment is from use. May be a tool that is somewhat mainstream. The evaluation is genuine. | 3 | Names a tool that is well-known and mainstream. Assessment is mostly "it's pretty good." Limited specificity about current use or current edge cases. | 2 | "I've been meaning to try X." Not currently experimenting. | 1 | "I use what my team uses." No self-directed experimentation. Disqualifying signal for System Engineer role. | Standard follow-ups: - "What would it have to do for you to bring it to the team?" - "What have you tried that you actively decided not to adopt? Why?" ## Scoring Summary Sheet Stage | Q# | Question summary | Weight | Score (1–5) | Notes | S1 | Q1 | Constraint definition | 1× | | | S1 | Q2 | Tradeoff reasoning | 1× | | | S1 | Q3 | Junior engineer failure paths | 1.5× | | | S1 | Q4 | Silent failure identification | 1.5× | | | S1 | Q5 | Why documentation | 0.5× | | | S2 | Q1 | Hypothesis generation | 1.5× | | | S2 | Q2 | Evidence-based diagnosis | 1× | | | S2 | Q3 | Test suite gap awareness | 1× | | | S2 | Q4 | Class-level architectural fix | 1.5× | | | S3 | Q1 | Code review — find issues | 2× | | | S3 | Q2 | Test/production gap | 1× | | | S3 | Q3 | Questions before approving | 1× | | | S3 | Q4 | Test case design | 1× | | | S4 | Q1 | Most complex system designed | 1× | | | S4 | Q2 | Wrong architectural decision | 1.5× | | | S4 | Q3 | Prevented without taking over | 1× | | | S4 | Q4 | AI delegation line | 1× | | | S4 | Q5 | Current experiments | 1.5× | | | Weighted total: Use the interactive scorecard to calculate your weighted total in real time. Score range | Recommendation | 85–100 | Strong Hire | 70–84 | Hire | 55–69 | Hire with conditions (name the specific gap) | 40–54 | No Hire — gap too large for role | Below 40 | No Hire | Auto-reject conditions (override score): - S4-Q5 scores 1 (no self-directed experimentation) - S3-Q1 scores 1 (approves code with security issues undetected) - Two or more questions in any single stage score 1 ## Reference List - Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262–274. - Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(12), 2040–2068. - Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity. Personnel Psychology, 58, 1009–1037. - Wiggins, G., & McTighe, J. (1998). Understanding by Design. ASCD. - Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge University Press. - Anderson, L. W., & Krathwohl, D. R. (2001). A Taxonomy for Learning, Teaching, and Assessing. Addison Wesley Longman. - Rivera, L. A. (2012). Hiring as cultural matching. American Sociological Review, 77(6), 999–1022. - Weiss, B., & Feldman, R. S. (2006). Looking good and lying to do it. Journal of Applied Social Psychology, 36(4), 1070–1086. Source guides: System Engineer Interview Guide | Evaluation Rubrics © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution. ==================== # Product Engineer: Question Bank (https://aieraengineering.com/product-engineer-question-bank/) # Question Bank: Product Engineer The AI-Era Engineering Playbook — Practitioner Reference ## Design Principles This bank is organised by interview stage. The Product Engineer interview has a structural difference from the System Engineer interview: Stage 2 is a live observed work session, not a static exercise. This is the highest-validity method available and requires specific observation criteria rather than pre-written questions. The four stages move from specification (what happens before any tool is touched) through live observed work (evaluation and accountability in real time) through output review (can they find what's wrong) to behavioural evidence (what their track record shows about domain ownership and correctness). ### Scientific basis for stage design Stage | Assessment Method | Validity (r) | Source | Bloom Level | 1 — Specification Exercise | Work sample (live spec writing) | .33–.54 | Roth et al. 2005; Schmidt & Hunter 1998 | Analyse / Create | 2 — Live AI-Assisted Build | Work sample (observed real-time) | .33–.54 | Roth et al. 2005 | Apply / Evaluate / Create | 3 — Output Review | Work sample (code review exercise) | .33–.54 | Roth et al. 2005 | Evaluate | 4 — Structured Behavioural | Structured behavioural interview | .42 | Sackett et al. 2022 | Apply / Analyse | Stage 2 note: The live AI-assisted build is the closest available method to a paid work trial (Schmidt & Hunter 1998: r=.44). The candidate uses their actual tools on a realistic task while you observe. You are not grading the output. You are grading the process — which is directly visible and cannot be rehearsed. Why early adoption is weighted at 20% for this role: The Product Engineer role requires active self-directed experimentation as a job function. An engineer who waits to be told which tools to use cannot maintain judgement about what AI gets wrong in their domain. Early adoption is a disqualifier at score 1 — not merely a concern. See Early Adopter Framework. Key validity references: - Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262–274. - Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(12), 2040–2068. - Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity. Personnel Psychology, 58, 1009–1037. - Wiggins, G., & McTighe, J. (1998). Understanding by Design. ASCD. - Lave, J., & Wenger, E. (1991). Situated Learning. Cambridge University Press. ### Scoring convention All questions use a 1–5 scale. Score based on observed behaviour, not inferred ability. Score | Label | Meaning | 5 | Exceptional | Exceeds expected depth; demonstrates mastery with specific, unprompted insight | 4 | Strong | Covers core fully; minor gaps; demonstrates clear competence | 3 | Adequate | Covers the basics; requires prompting to reach depth; no major gaps | 2 | Weak | Partial coverage; significant gaps; cannot reach depth even when prompted | 1 | Absent | Not demonstrated; cannot engage meaningfully with the question | Auto-reject: P4-Q5 (early adoption) scores 1; or Stage 2 overall scores average below 2; or two or more questions in any single stage score 1. ## Stage 1: Specification and Mental Modelling Assessment method: Live specification writing exercise Skill map position: Specification test (X=+9, Y=+8) Validity basis: B — directly mirrors the primary daily task; high ecological validity Construct: Ability to build a complete mental model of what should exist before generating anything; ability to translate ambiguity into testable behavioural requirements Administration notes: - Read the requirement aloud. Give it in writing. Give 5 minutes to think silently. - Do not clarify the requirement — ambiguity is the test. Watch what they do with it. - The exercise continues through Q1–Q4 in sequence. Do not skip ahead. - Score on process and output quality, not on whether their spec matches any expected version. Sample requirement (rotate per candidate): "Add a notification feature. Users should be notified about important events." (Intentionally vague. That is the whole point.) ### P1-Q1. "Before writing anything — what do you need to know first?" Construct: Ambiguity recognition; requirement decomposition; question quality as a proxy for mental model depth Bloom level: Analyse Chart method: Specification test (X=+9, Y=+8) Research note: The quality of a Product Engineer's clarifying questions is the primary early signal. Wiggins & McTighe (1998) establish that genuine understanding shows when a person can identify what is missing from a problem statement before attempting to solve it. The specification test has high face and ecological validity: the behaviour directly mirrors the first step of real AI-assisted development. Scoring rubric: Score | Behavioural description | 5 | Asks 4–6 questions that together would fully constrain the specification. Questions are behavioural, not implementational: "Who are the users?" "What counts as an important event — who decides?" "What channels?" "What happens if delivery fails?" "What are the user's notification preferences?" "What's the expected volume?" Does not ask "should we use SendGrid?" or "what database?" before the behavioural questions are answered. | 4 | Asks 3–4 substantive behavioural questions. Covers the core ambiguities (event definition, channel, failure path). May ask one implementation question prematurely. | 3 | Asks 1–2 questions. Misses significant ambiguities (e.g., accepts "important events" without defining what that means; doesn't ask about failure paths). | 2 | Asks clarifying questions about implementation only ("what language?" "what framework?") or asks one vague question ("can you tell me more?") and moves on. | 1 | Does not ask clarifying questions. Begins specifying or implementing the vague requirement as given. | Red flags: - First question is about technology ("should I use a message queue?") - Accepts "important events" as a defined term without unpacking it - Asks only one question and considers the ambiguity resolved Probing follow-up: "Is there anything you'd want to know that I haven't answered yet before you start writing?" ### P1-Q2. "Write the specification. Walk me through it as you go." Construct: Specification completeness; behavioural framing; assumption explicitness; testability Bloom level: Create Chart method: Specification test (X=+9, Y=+8) Research note: This is the core work sample question. The output — the written specification — is directly evaluable against the criterion: is it complete, behavioural, and testable? The "walk me through it" component adds process visibility (Lave & Wenger 1991: knowledge in context) and reveals whether the written output reflects genuine understanding or surface-level pattern matching. Scoring rubric: Score | Behavioural description | 5 | Specification is written in behavioural terms: "When [event X occurs], the system shall [notify user Y via Z], provided [condition A]. If delivery fails after [N] retries, [fallback behaviour]." Assumptions are explicitly labelled. Failure paths are addressed. Edge cases are enumerated. The specification could be handed to an engineer (or an AI) and produce a predictable outcome. Could write acceptance tests from the specification without further discussion. | 4 | Specification covers the happy path and most edge cases. Behavioural framing is mostly present. Some assumptions are implicit. Failure paths are partially addressed. Testable with minor clarification. | 3 | Specification covers the happy path. Edge cases are absent or noted as "TBD." Failure paths are not addressed. Some implementation language present ("we'll use a queue"). Testable for the happy path only. | 2 | Specification describes a partial solution rather than a complete behaviour. More implementation than behaviour. Cannot be used to write acceptance tests without significant additional discussion. | 1 | Writes a design document or code instead of a specification. Cannot distinguish between specifying behaviour and implementing it. | Red flags: - "When the event happens, call the notification service" — describes implementation, not behaviour - No mention of failure paths ("what if the notification doesn't deliver?") - Assumptions are embedded, not labelled ("I'm assuming users have one email address") - Cannot explain what a test for this specification would check Probing follow-up: "Show me one acceptance test that you could write directly from what you've written." ### P1-Q3. "What did you explicitly choose NOT to cover, and why?" Construct: Scope awareness; deliberate constraint; the distinction between MVP and complete specification Bloom level: Evaluate Research note: A specification that attempts to cover everything is not a specification — it is a wish list. Deliberate scoping requires the engineer to have a mental model of the whole problem, identify what is in scope vs. deferred, and be able to defend the line. This is a directly observable form of engineering judgement. Scoring rubric: Score | Behavioural description | 5 | Names 2–3 specific items they chose not to cover and gives principled reasons for each (e.g., "I didn't spec notification preferences because that requires a separate preferences model that would double the scope; I left it as a labelled assumption"). Demonstrates awareness that the excluded items are real requirements that will need to be addressed, not things they missed. | 4 | Names 1–2 exclusions with reasoning. The reasoning is principled, not evasive. | 3 | Notes that they "kept it simple" without naming specific exclusions or articulating why the line was drawn there. | 2 | Exclusions are things they forgot or didn't notice, not things they deliberately deferred. | 1 | "I tried to cover everything." Cannot name an item they chose not to include. | Red flags: - Exclusions are things they missed, not things they chose to exclude - Cannot explain why one thing was in scope and another wasn't - "I'll add more later" without a clear scope boundary now ### P1-Q4. "If I handed this specification to an AI agent and it built something that passed all your tests but behaved wrong — what did you miss?" Construct: Specification gap awareness; the limits of testability; anticipating AI failure modes from underspecification Bloom level: Evaluate Research note: This question targets the most important failure mode of AI-assisted development: specifications that are complete enough to generate code from, but incomplete enough that the code can be technically correct yet behaviourally wrong. This is the construct that most traditional interview methods cannot reach. There is no published validity data specific to this question type — it is a novel construct that emerged post-2022. Scoring rubric: Score | Behavioural description | 5 | Identifies a specific gap in their own specification that an AI would exploit plausibly — not a generic answer. Examples: "I didn't define what 'delivered' means — the AI might mark a notification as delivered when the API returns 200, even if the email bounced." "I didn't specify idempotency — if the event fires twice, the user gets two notifications." Demonstrates that they understand AI generation produces maximally literal interpretations of specifications. | 4 | Identifies a specific gap. The reasoning connects the gap to a plausible AI failure mode. May need light prompting to be specific. | 3 | "There are probably edge cases I missed." Cannot identify a specific one without prompting. | 2 | "My tests would catch any problem." Does not accept the premise that a test-passing system can still be behaviourally wrong. | 1 | Cannot engage with the question. | Red flags: - "I'd just ask the AI to find the gaps" — avoids the metacognitive step - "If it passes the tests it's correct" — conflates test coverage with behavioural correctness - Cannot name a specific gap in their own specification under any prompting Probing follow-up: "What constraint would you add to the specification right now to close that gap?" ## Stage 2: Live AI-Assisted Build and Evaluation Assessment method: Observed real-time work session Skill map position: Work sample test (X=+9, Y=+9) Validity basis: A — closest available method to a paid work trial; Schmidt & Hunter (1998) job tryout r=.44 Construct: Output judgement in real time; evaluation of output against specification and domain requirements; accountability before shipping; diagnostic iteration when output fails Administration notes: - Give the candidate access to their preferred AI coding tool. Use their own environment if possible — ecological validity is the point. - Task: "Using the notification specification you wrote in Stage 1, implement the core logic: a function that takes an event, looks up notification preferences for the affected user, and dispatches the notification through the appropriate channel. You have 20 minutes. Use AI freely." - Observe silently. Do not help, correct, or react. Take notes. - After 20 minutes, conduct the debrief (P2-Q1 through P2-Q4). - You are not grading the output. You are grading the process. A candidate who produces working code via bad process scores lower than a candidate who produces incomplete code via a rigorous process. Observation checklist (fill during the session — not scored separately, informs Q scores): Observable | Strong signal | Weak signal | Output evaluation | Reads the code before running it; checks it against the specification; asks "is this correct for the domain?" | Runs it; if it runs, it ships | Accountability moment | Pauses before approving; asks "would I own this in production?" | No visible pause; immediate progression | Error handling | Looks specifically for absence of error handling; checks edge paths | Does not check error paths | Domain correctness | Checks whether the output satisfies the domain rules, not just the happy path | Evaluates only technical correctness | Iteration trigger | Identifies what was wrong in the specification that caused the wrong output; revises deliberately | Re-prompts randomly; pastes the error message back | Specification use | References their own specification when generating; includes domain constraints | Ignores the specification they wrote; starts fresh | ### P2-Q1. "Walk me through what this function does. What would happen if the user's notification preferences were not found in the database?" Construct: Output comprehension; mental model formation; edge case awareness during evaluation Bloom level: Analyse / Evaluate Research note: The ability to explain AI-generated code is the minimum competence bar for a Product Engineer. A candidate who cannot explain what they just built with AI cannot evaluate its correctness or catch its failures. This is directly observable: either they can walk through the execution path or they cannot. Scoring rubric: Score | Behavioural description | 5 | Explains the function correctly, including the specific execution path for the edge case. Identifies the gap (missing null check, wrong default behaviour, silent failure) without being told there is one. If there is no gap, correctly identifies that and explains why. Connects the behaviour back to the specification they wrote. | 4 | Explains the function correctly. Can trace the edge case path. May not identify the gap without prompting, but immediately understands it when the edge case is named. | 3 | Explains the happy path correctly. Cannot trace the edge case path without significant help. Does not spontaneously check whether the edge case is handled. | 2 | Explanation is vague or partially wrong. Relies on "the AI handled it" without being able to verify. Cannot trace the execution path under the edge condition. | 1 | Cannot explain the function. "I'd have to run it to find out." | Red flags: - "The AI usually handles null cases" — defers to AI correctness as assumption - Reads the code aloud without interpreting it ("it calls getUserPreferences, then...") - Cannot connect the function's behaviour to the specification they wrote ### P2-Q2. "Does this match your specification? Walk me through the differences." Construct: Specification-to-implementation gap detection; output evaluation against an explicit model Bloom level: Evaluate Research note: This is the most direct test of AI output evaluation quality. A specification exists. An implementation exists. The candidate either can or cannot identify the differences. Candidates who cannot answer this question have no effective quality gate over AI-generated code — they are producing output they cannot evaluate. Scoring rubric: Score | Behavioural description | 5 | Conducts a structured comparison between specification and implementation. Identifies specific gaps (missing failure path, wrong retry behaviour, extra scope that wasn't specified). Distinguishes between gaps that matter (behavioural) and gaps that don't (style, naming). Connects each gap back to a specific specification clause. | 4 | Identifies 2–3 substantive gaps. Comparison is structured. May miss one minor gap. | 3 | Identifies 1 gap or identifies gaps only when prompted ("does this handle the failure case?"). Comparison is not spontaneous. | 2 | "I think it matches." Cannot identify differences without being told where to look. Comparison is against intuition, not against the specification document. | 1 | Has not compared the output to the specification. Treats running output as passing output. | Red flags: - Does not refer back to their written specification during the comparison - "It looks right to me" without specifying what "right" means - Identifies only style differences, not behavioural differences Probing follow-up: "Open your specification. Point to the clause that covers this error path. Is it in the code?" ### P2-Q3. "When the output was wrong, how did you decide what to change?" Construct: Diagnostic iteration vs. random re-prompting; specification failure identification Bloom level: Analyse Research note: Random re-prompting is the most common failure mode in AI-assisted development. It produces output churn without convergence. The diagnostic alternative — identifying which part of the specification or prompt caused the wrong output — produces convergent iteration. This question directly reveals which pattern the candidate uses. The observation checklist from the live session provides corroborating evidence. Scoring rubric: Score | Behavioural description | 5 | Describes a diagnostic process: "The output did [X]. My specification says [Y]. The gap is in how I specified [Z]. I changed the prompt to include [specific constraint]." Has a theory about why the output was wrong before making the next attempt. Can reconstruct this reasoning from the live session just completed. | 4 | Mostly diagnostic. Can identify the cause in most cases. May have one instance of random re-prompting. | 3 | Mix of diagnostic and random. Cannot clearly distinguish between "I changed the specification" and "I added more words to the prompt." | 2 | Primarily random re-prompting ("I tried different wording until it worked"). No consistent theory of why changes produced different outputs. | 1 | "I just kept asking until it looked right." No diagnostic process. Cannot reconstruct why any specific change worked. | Red flags: - "I pasted the error message back in" as the primary iteration strategy - Cannot explain why the final output is better than the earlier ones - Iteration was driven by "it looked wrong" without specifying what "wrong" meant ### P2-Q4. "What would you add to the test suite that the AI didn't generate?" Construct: Test gap identification; failure path enumeration; translating behavioural concerns into test cases Bloom level: Create Research note: AI-generated test suites are systematically biased towards happy-path coverage. The code that is easiest to generate — the implementation — is also the easiest to test. The code that is hardest to get right — failure paths, edge cases, concurrency — is the code that is hardest to test and therefore least likely to appear in an AI-generated test suite. A Product Engineer who cannot identify this gap cannot establish a quality gate over their own output. Scoring rubric: Score | Behavioural description | 5 | Names 3+ specific test cases (not categories) covering failure paths and edge cases that the AI test suite does not include. Each test is specific enough to implement immediately: states the setup, the operation, and the expected outcome. At least one test targets a silent failure (wrong behaviour without an error). | 4 | Names 2–3 specific test cases. Tests are implementable. May miss one important category. | 3 | Names test categories ("we need tests for null preferences") without specifying the test setup and assertion. | 2 | "The test coverage looks pretty good." Cannot identify missing tests without being shown the gap explicitly. | 1 | "The AI usually writes comprehensive tests." Cannot propose a single specific missing test. | Red flags: - "I'd run it and see what breaks" — treats production as the test suite - Proposes tests that mock the exact behaviour they should test - Cannot specify what a failing test would look like before the fix is applied ## Stage 3: Output Review and Edge Case Hunting Assessment method: Code review exercise (AI-generated code with planted issues) Skill map position: AI output review (X=+9, Y=+8) Validity basis: B — directly mirrors daily job task; high ecological validity Construct: Ability to identify behavioural, security, and logic errors in AI-generated code; evaluation against correctness, not style Administration notes: - Provide 50–80 lines of AI-generated code. Give 10 minutes to review independently. Then discuss for 30 minutes. - Do not tell the candidate how many issues are present. - Rotate planted issues per candidate. The list below offers more options than you need. - Score on what they find and how they reason about it, not whether they can fix it. Issue categories to plant (choose 4, vary each time): Code issue | Category | What a strong candidate sees | Free trial check that ignores trials started before the billing date | Business logic | "This comparison uses the billing date as the start of the trial window but doesn't account for trials that pre-date the current billing period — users on legacy trials will be incorrectly charged" | No handling for concurrent requests that could double-charge | Race condition | "Two simultaneous requests can both pass the balance check before either commits — there's no locking or idempotency key" | User-supplied field interpolated into query string without parameterization | SQL injection | "This is injectable — the user_id field goes directly into the query without sanitization" | Function assumes external API always returns 200; no handling for 4xx/5xx | Missing error state | "If the payment provider returns anything other than 200, this silently proceeds as if the transaction succeeded" | Currency amount stored as float | Precision error | "Floating-point currency arithmetic introduces rounding errors — this should use integer cents or a decimal type" | Retry loop with no backoff and no max attempts | Reliability | "Under load this will hammer the downstream service — there's no backoff and no ceiling on retries" | Event handler registered inside a loop | Memory leak | "This registers a new event handler on every loop iteration without cleanup — it leaks handlers and fires callbacks multiple times" | ### P3-Q1. "Walk me through what you'd change before shipping this." Construct: Code review judgement; behavioural correctness evaluation; security awareness Bloom level: Evaluate Research note: This question tests whether the review is driven by a mental model of correct behaviour or by surface pattern matching. Strong candidates evaluate against the expected contract and failure conditions; weak candidates evaluate against style conventions. The distinction is directly observable in what they choose to flag. Scoring rubric: Score | Behavioural description | 5 | Finds all planted issues. Explains each in terms of the failure behaviour, not just "this is wrong." Identifies at least one issue as belonging to a class of problem ("this is a missing error-handling pattern — I'd check the rest of the codebase for the same pattern"). Does not spend significant time on style. May find additional unplanted issues. | 4 | Finds 3–4 planted issues. Behavioural reasoning is present. Proportionate attention to substantive vs. style issues. | 3 | Finds 2–3 issues. May include style issues. Reasoning requires prompting to go from "this looks wrong" to "this fails when...". | 2 | Finds 1–2 issues, mostly style or naming. Cannot explain behavioural impact without help. | 1 | "This looks fine." No substantive issues identified. | Red flags: - Opens with "I'd change the variable names" before looking for logic issues - Cannot explain what breaks if they leave a found issue in place - Misses the security issue entirely (SQL injection or equivalent) Probing follow-up (if they miss an issue): "What happens when [specific edge condition]? Walk me through the execution path." ### P3-Q2. "Are there scenarios where this code does the wrong thing but doesn't throw an error?" Construct: Silent failure identification; behavioural correctness beyond exception handling Bloom level: Analyse Research note: Silent failures — wrong behaviour without an error signal — are the most dangerous output from AI-generated code. They pass tests, pass review, and only surface as business impact (incorrect charges, data corruption, missed notifications). This question specifically targets this failure class, which requires a behavioural mental model to detect. Scoring rubric: Score | Behavioural description | 5 | Identifies at least one specific silent failure from the planted issues. Explains the specific execution path: what happens, why no error is raised, what the observable consequence is. Demonstrates that silent failures are a category they actively look for in review. | 4 | Identifies a silent failure. Can explain why it is silent (no exception, wrong success signal). May need light prompting to reach the specific mechanism. | 3 | "There might be edge cases." Cannot name one specifically without prompting. | 2 | "If it doesn't throw an error, it's correct." Conflates absence of error with correct behaviour. | 1 | Cannot engage with the concept of silent failure. | Red flags: - "The exception handling looks good, so it should be fine" - Identifies only issues that would also produce errors - Cannot construct a scenario where the code returns successfully but behaves incorrectly ### P3-Q3. "What would a malicious user be able to do with this code?" Construct: Security mindset; adversarial reasoning; input validation awareness Bloom level: Evaluate Research note: Security issues are the most consistent gap in AI-generated code. Bilkent University (2023) found 30.5% of AI-generated code contains vulnerabilities. A Product Engineer who cannot identify security issues in their own output is a reliable source of security debt. This question tests whether adversarial reasoning is part of their review process or absent from it. Scoring rubric: Score | Behavioural description | 5 | Identifies the planted security issue. Explains the attack vector specifically: what the malicious input is, what it would cause, what data or system it could affect. Raises at least one additional security concern not planted (e.g., authentication boundary, authorization check, rate limiting absent). Does not need to be prompted to think adversarially. | 4 | Identifies the planted security issue. Explains the attack vector. Does not raise additional concerns spontaneously. | 3 | Identifies a security concern but describes it vaguely ("someone could inject something"). Cannot specify the attack or consequence. | 2 | "It looks secure to me." Does not identify the planted security issue. | 1 | Does not think in adversarial terms. "I wouldn't expect users to do that." | Red flags: - "Users won't send malicious input" — trusts input by default - Identifies the issue as "bad practice" without explaining what an attacker would do - "The framework handles security" — delegates security reasoning to tooling ### P3-Q4. "What's missing from the test suite that should be there?" Construct: Test coverage gap identification; failure path enumeration; quality gate design Bloom level: Create Research note: Same construct as P2-Q4, now applied to externally-written code rather than their own output. The question tests whether the candidate can reason about test gaps in a review context — a core skill for any engineer who reviews AI-generated code. Specific test cases are the output; categories are insufficient. Scoring rubric: Score | Behavioural description | 5 | Names 3+ specific missing tests with setup, operation, and expected outcome specified. At least one test covers a security scenario. At least one test covers a silent failure path. Prioritizes by risk, not by ease of writing. | 4 | Names 2–3 specific tests. Tests are implementable. Covers most high-risk gaps. | 3 | Names test categories without specifying the test. "We need tests for the error cases." | 2 | "The test coverage looks adequate." Cannot identify missing tests. | 1 | Cannot propose tests. Defers to "the existing tests cover it." | ## Stage 4: Structured Behavioural Interview Assessment method: Structured behavioural interview (STAR format with rubric) Skill map position: Structured behavioural (X=+5, Y=+6) + Early adoption question (X=+9, Y=+7) Validity basis: A — Sackett et al. (2022): r=.42; structure and consistent rubric are required Construct: Past behavioural evidence for: specification quality, AI collaboration, output evaluation, early adoption, domain knowledge Administration notes: - Ask questions in the same order with all candidates, without alteration. - Do not help candidates structure their answers. Probe only with the listed follow-ups. - Score independently before comparing with your co-interviewer. - Stage 4 validates the process observed in Stages 1–3. A candidate who performs well in the exercises but cannot recall a single real instance is a construct validity concern. ### P4-Q1. "Tell me about a feature you built that behaved wrong in production even though tests passed. What did your specification miss?" Construct: Real-world specification failure; retrospective specification reasoning Bloom level: Analyse (retrospective) Research note: Past-behaviour questions are harder to answer deceptively than hypothetical questions (Weiss & Feldman 2006). This question specifically asks for a specification failure in a production context — not a bug, not a design flaw, but something that traced back to an incomplete or ambiguous specification. Candidates who have never specified before cannot answer this question concretely. Scoring rubric: Score | Behavioural description | 5 | Names a specific feature and describes the production behaviour that was wrong (without being an outage — ideally a silent or business-logic failure). Traces the failure back to a specific gap in the specification: what was underspecified, what assumption was wrong, what edge case was not covered. Derives a generalizable lesson about specification practice. Does not blame the developer who implemented it. | 4 | Names a specific case. Traces back to a specification gap. Lesson is present but specific to the situation. | 3 | Describes a bug or production incident but attributes it to implementation rather than specification. May not be able to trace it back to specification with prompting. | 2 | Describes a production incident that was a system failure or external dependency, not a specification issue. | 1 | "My features have generally worked fine in production." Cannot recall a specification failure. | Standard follow-ups: - "What clause would you add to the specification now to prevent it?" - "Did the test suite cover the case that failed? Why not?" ### P4-Q2. "Describe a time you pushed back on a requirement because it was underspecified. What did you find and what happened?" Construct: Proactive specification quality; upward communication; the ability to see underspecification before it becomes a bug Bloom level: Apply / Analyse (retrospective) Scoring rubric: Score | Behavioural description | 5 | Describes a specific situation where they identified that a requirement could not be safely implemented without clarification. Names what specifically was ambiguous. Describes how they raised it. The outcome: the requirement was clarified, and what was clarified changed the implementation. Demonstrates that this is a habit, not a one-time event. | 4 | Describes a specific instance. The ambiguity was real. The pushback was constructive. The outcome was improved specification. | 3 | Describes asking for clarification on a vague requirement. The ambiguity was minor. The pushback was informal. | 2 | Describes a situation where they implemented a vague requirement and asked for clarification afterward. The specification was not fixed before work began. | 1 | "I usually just build what's asked and see what feedback I get." Cannot recall pushing back on a specification. | Standard follow-ups: - "What would have happened if you'd built it as originally specified?" - "How do you decide when a requirement is underspecified enough to push back vs. making a reasonable assumption?" ### P4-Q3. "Give me an example of a time AI gave you output that looked right but wasn't. How did you catch it?" Construct: AI output evaluation track record; detection mechanism; gap between appearance and correctness Bloom level: Analyse (retrospective) Research note: This question has no equivalent in pre-2023 interview design. It targets a construct that is now central to the Product Engineer role: the ability to catch AI failures that are not surfaced by tests or errors. A candidate who cannot recall an instance where AI output looked correct but wasn't is either not reviewing AI output carefully, or has not used AI in contexts where correctness matters. Both are concerns for this role. Scoring rubric: Score | Behavioural description | 5 | Names a specific instance where AI output was functionally wrong despite looking correct. Explains specifically what made it look right (tests passed, ran without errors, matched the surface pattern). Explains specifically what caught it (manual trace, edge case review, domain knowledge, specification comparison). Derives a principle for catching this class of failure in the future. | 4 | Names a specific instance. The failure was real, not a typo or syntax error. The detection mechanism is described. | 3 | Names an instance but the failure was obvious (syntax error, immediate runtime exception). Does not describe catching a subtle behavioural failure. | 2 | "AI output has generally been good in my experience." Cannot recall a specific failure. | 1 | "I review everything before shipping so there haven't been issues." Cannot name a specific instance where AI was wrong. | Standard follow-ups: - "Would your test suite have caught it if you hadn't reviewed it manually?" - "What do you check for now that you didn't before that incident?" ### P4-Q4. "What do you use AI for in your daily work? What do you deliberately NOT use it for, and why?" Construct: AI collaboration boundaries; judgement about dependency risk; self-awareness about where AI is and isn't reliable Bloom level: Evaluate Research note: This question establishes whether the candidate has a deliberate, principled boundary or an ad-hoc one. Strong Product Engineers know precisely where AI helps and where it introduces risk — and their boundary is based on the consequences of errors, not on comfort or convention. This is distinct from System Engineer S4-Q4, which focuses on delegation in the context of architectural ownership; here the focus is on daily operational judgement. Scoring rubric: Score | Behavioural description | 5 | Describes AI use with specific examples across multiple task types. The "do NOT use" category is principled: things where AI errors are hard to detect, have compounding consequences, or require domain judgement that AI lacks. The line has been revised as AI capability has changed. Demonstrates active management of the boundary. | 4 | Clear use/no-use distinction with specific examples. Reasoning is mostly principled. The line may not be articulated as a principle but is consistent. | 3 | Describes AI use broadly ("I use it for most things"). The no-use category is vague ("sensitive stuff") without a principle. | 2 | "I use AI for everything" or "I only use AI for simple tasks." Extreme positions without nuance. | 1 | Has not thought about this. Use is determined by habit or team convention, not judgement. | Standard follow-ups: - "Has the line moved in the last six months? What moved it?" - "Give me a specific example where you started to use AI for something, then decided not to. What made you stop?" ### P4-Q5. "What have you started using in the last three months that nobody told you to? What's your current honest assessment of it?" Construct: Early adoption behaviour; self-directed learning; opinion quality from use Bloom level: Evaluate (ongoing) Research note: This is the single highest-weight question in the Product Engineer bank. Early adoption is a behavioural pattern that cannot be faked convincingly because specific, recent, concrete examples of tool use are required. A candidate claiming to have used a tool they haven't used will fail on follow-up questions about specifics. The question is asked in present tense and asks for an honest assessment — not a sale. Positive-only responses suggest newsletter reading rather than actual use. See Early Adopter Framework for full scoring context. Scoring rubric: Score | Behavioural description | 5 | Names a specific tool or technique with a specific use case. The assessment is formed from use, not from reviews: includes what it does well, what it fails at, and what they would need to see before recommending it to their team. The tool may not yet be mainstream. Shows a systematic evaluation process. Has also tried at least one thing in the same period that they rejected — and can say why. | 4 | Names a specific tool with a genuine assessment from use. The assessment includes at least one concrete limitation ("it struggles with X"). May be a mainstream tool but evaluated independently. | 3 | Names a mainstream tool with a mostly positive but vague assessment ("it's been really helpful"). Limited specificity about failure modes or current edge cases. Evidence of use is present but shallow. | 2 | "I've been meaning to try [tool]." Not currently experimenting. Assessment is from reading, not use. | 1 | "I use what my team uses." No self-directed experimentation. Disqualifier for Product Engineer role. | Standard follow-ups: - "What would it have to improve for you to start recommending it to your team?" - "What have you tried in the same period that you decided wasn't worth your time? What was wrong with it?" - "Where did you first hear about it? How do you generally find out about new tools?" ## Scoring Summary Sheet Stage | Q# | Question summary | Weight | Score (1–5) | Notes | S1 | Q1 | Clarifying questions before spec | 1.5× | | | S1 | Q2 | Write the specification | 2× | | | S1 | Q3 | What did you choose not to cover | 1× | | | S1 | Q4 | What gap would AI exploit | 1.5× | | | S2 | Q1 | Walk through the function / edge case | 1.5× | | | S2 | Q2 | Does it match the spec | 2× | | | S2 | Q3 | How did you decide what to change | 1.5× | | | S2 | Q4 | What tests did the AI miss | 1× | | | S3 | Q1 | Code review — find issues | 2× | | | S3 | Q2 | Silent failures | 1.5× | | | S3 | Q3 | Malicious user attack surface | 1× | | | S3 | Q4 | Missing tests | 1× | | | S4 | Q1 | Feature that behaved wrong in prod | 1× | | | S4 | Q2 | Pushed back on underspecified req | 1× | | | S4 | Q3 | AI output that looked right but wasn't | 1.5× | | | S4 | Q4 | AI use / no-use boundary | 1× | | | S4 | Q5 | Current experiments (early adoption) | 2× | | | Weighted total: \_\_\_ / 100 Score range | Recommendation | 85–100 | Strong Hire | 70–84 | Hire | 55–69 | Hire with conditions (name the specific gap) | 40–54 | No Hire — gap too large for role | Below 40 | No Hire | Auto-reject conditions (override score): - P4-Q5 scores 1 (no self-directed experimentation — disqualifier for this role) - P2 stage average below 2.0 (cannot collaborate effectively with AI in real time) - Two or more questions in any single stage score 1 ## Reference List - Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262–274. - Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(12), 2040–2068. - Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity. Personnel Psychology, 58, 1009–1037. - Wiggins, G., & McTighe, J. (1998). Understanding by Design. ASCD. - Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge University Press. - Anderson, L. W., & Krathwohl, D. R. (2001). A Taxonomy for Learning, Teaching, and Assessing. Addison Wesley Longman. - Weiss, B., & Feldman, R. S. (2006). Looking good and lying to do it. Journal of Applied Social Psychology, 36(4), 1070–1086. Source guides: Product Engineer interview Guide | Evaluation Rubrics | Early Adopter Framework © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution. ==================== # System Engineer: Interview Scorecard (https://aieraengineering.com/system-engineer-scorecard/) Candidate name Date Interviewer Score 1–5 per question. Rubric anchors: System Engineer question bank → ### Stage 1 — Systems Thinking and Architecture Case study discussion. Scenario provided in writing. 10 min think time, then 35 min discussion. Question | Weight | Score | Notes | S1-Q1 — Constraint definition: what must be defined before implementation begins? | 1× | 12345 | | S1-Q2 — Tradeoff reasoning: where would you sacrifice and why? | 1× | 12345 | | S1-Q3 — Junior failure paths: what breaks first, architecturally? | 1.5× | 12345 | | S1-Q4 — Silent failure: what's invisible until production? | 1.5× | 12345 | | S1-Q5 — Why documentation: intent, not just implementation? | 0.5× | 12345 | | Stage 1: — / 27.5 ### Stage 2 — Failure Mode Reasoning Incident scenario. System description + incident report provided in writing. 10 min review, then 35 min discussion. Question | Weight | Score | Notes | S2-Q1 — Hypothesis generation: top three failure candidates? | 1.5× | 12345 | | S2-Q2 — Evidence-based diagnosis: confirm or eliminate without system access? | 1× | 12345 | | S2-Q3 — Test suite gap: why this failure wouldn't appear in tests? | 1× | 12345 | | S2-Q4 — Class-level fix: architectural change, not a patch? | 1.5× | 12345 | | Stage 2: — / 25 ### Stage 3 — AI Output Audit 60–100 lines of AI-generated code with 3 planted issues. 15 min review, then 30 min discussion. Question | Weight | Score | Notes | S3-Q1 — Code review: find issues, explain failure behaviour (highest weight — auto-reject if 1) | 2× | 12345 | | S3-Q2 — Test/production gap: what passes tests but fails in prod? | 1× | 12345 | | S3-Q3 — Questions before approving: spec recovery, not judgement? | 1× | 12345 | | S3-Q4 — Test case design: specific cases for each found issue? | 1× | 12345 | | Stage 3: — / 25 ### Stage 4 — Structured Behavioural Consistent questions, same order per candidate. Score on observed evidence, not impression. Question | Weight | Score | Notes | S4-Q1 — Most complex system designed: constraints and tradeoffs? | 1× | 12345 | | S4-Q2 — Wrong architectural decision: what was missed and why? | 1.5× | 12345 | | S4-Q3 — Prevented without taking over: teaching under constraint? | 1× | 12345 | | S4-Q4 — AI delegation line: principled boundary, not habit? | 1× | 12345 | | S4-Q5 — Current experiments: active early adoption with specific examples (auto-reject if 1) | 1.5× | 12345 | | Stage 4: — / 30 — / 100 Score questions above to see recommendation #### ⚠ Auto-reject condition triggered — score is overridden S4-Q5 scored 1 — no self-directed experimentation TRIGGERED S3-Q1 scored 1 — AI code with security issues undetected TRIGGERED Two or more questions in a single stage scored 1 TRIGGERED ### Debrief notes Complete before comparing with co-interviewer. Write your answer first — this reduces anchoring to the first opinion expressed. Most decisive positive signal Most decisive negative signal or absence If Hire with conditions — what specifically needs to develop One thing this candidate does that most candidates don't Fill in the scores and debrief notes above, then generate a structured prompt to paste into Claude, ChatGPT, or any LLM for a calibrated debrief summary. Copy AI debrief prompt ✓ Copied to clipboard © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution. ==================== # Product Engineer: Interview Scorecard (https://aieraengineering.com/product-engineer-scorecard/) Candidate name Date Interviewer Score 1–5 per question. Rubric anchors: Product Engineer question bank → ### Stage 1 — Specification and Mental Modelling Live spec writing exercise. Requirement provided verbally and in writing. 5 min think time, then discussion. Requirement used Question | Weight | Score | Notes | P1-Q1 — Clarifying questions: what must be known before specifying? | 1.5× | 12345 | | P1-Q2 — Write the specification: behavioural, testable, complete (highest weight) | 2× | 12345 | | P1-Q3 — Deliberate exclusions: what was scoped out and why? | 1× | 12345 | | P1-Q4 — AI gap anticipation: what would AI exploit or miss in this spec? | 1.5× | 12345 | | Stage 1: — / 30 ### Stage 2 — Live AI-Assisted Build and Evaluation Observed real-time work session. Candidate uses their own AI tool. 20 min build, then debrief. This is the highest-validity stage — observe carefully. Observation checklist (fill during the live session) Observable behaviour | Observed | Notes | Provides context and specification in the prompt before generating | —YesPartialNo | | Reads output carefully before running it | —YesPartialNo | | Checks output against the specification, not just "looks right" | —YesPartialNo | | Iteration is diagnostic — changes are based on a hypothesis | —YesPartialNo | | Pauses to evaluate before generating again | —YesPartialNo | | Planted issues Issue description | Category | Found? | Reasoning quality | | | —YesNo | | | | —YesNo | | | | —YesNo | | Debrief questions (post-session) Question | Weight | Score | Notes | P2-Q1 — Walk through function: including edge case behaviour? | 1.5× | 12345 | | P2-Q2 — Spec match: what's different between spec and output? (highest weight) | 2× | 12345 | | P2-Q3 — Iteration reasoning: how did you decide what to change? | 1.5× | 12345 | | P2-Q4 — Missing tests: specific cases the AI didn't write? | 1× | 12345 | | Stage 2 average: — (auto-reject if below 2.0) Stage 2: — / 30 ### Stage 3 — Output Review and Edge Case Hunting 50–80 lines of AI-generated code with planted issues. 10 min review, then 30 min discussion. Question | Weight | Score | Notes | P3-Q1 — Code review: find issues, explain failure behaviour (highest weight) | 2× | 12345 | | P3-Q2 — Silent failures: wrong but no error raised? | 1.5× | 12345 | | P3-Q3 — Adversarial: what would a malicious user do with this code? | 1× | 12345 | | P3-Q4 — Missing tests: specific cases by risk priority? | 1× | 12345 | | Stage 3: — / 27.5 ### Stage 4 — Structured Behavioural Consistent questions, same order per candidate. Score on observed evidence only. Question | Weight | Score | Notes | P4-Q1 — Feature wrong in production: traced back to specification? | 1× | 12345 | | P4-Q2 — Pushed back on underspecified requirement: what and outcome? | 1× | 12345 | | P4-Q3 — AI output looked right but wasn't: how did you catch it? | 1.5× | 12345 | | P4-Q4 — AI use / no-use boundary: principled, not habitual? | 1× | 12345 | | P4-Q5 — Current experiments: specific, from real use, formed opinions (2×, disqualifier if 1) | 2× | 12345 | | Stage 4: — / 32.5 — / 100 Score questions above to see recommendation #### ⚠ Auto-reject condition triggered — score is overridden P4-Q5 scored 1 — no self-directed experimentation (disqualifier for this role) TRIGGERED Stage 2 average below 2.0 — cannot evaluate output effectively in real time; ships without checking TRIGGERED Two or more questions in a single stage scored 1 TRIGGERED #### Compensation calibration — check if observed Demonstrated early adoption of tools the team is not yet using: +10–15% Deep domain knowledge in the role's specific industry: +15–25% Evidence of bringing AI tooling improvements to previous teams: +10% Demonstrated ability to specify systems used by others: +10% ### Debrief notes Complete before comparing with co-interviewer. Most decisive positive signal Most decisive negative signal or absence If Hire with conditions — what specifically needs to develop One thing this candidate does that most candidates don't Fill in scores and notes above, then generate a structured prompt to paste into Claude, ChatGPT, or any LLM for a calibrated debrief summary. Copy AI debrief prompt ✓ Copied to clipboard © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution. ==================== # Job Description Templates (https://aieraengineering.com/job-description-templates/) The AI-Era Engineering Playbook — Practitioner Reference These templates are designed to attract the right candidates by signalling clearly what the role requires and how you assess it. The "How we assess" and "How we don't assess" sections do two things: they filter out candidates who are optimised for the old interview (not the role), and they tell strong candidates that this company has thought carefully about what the job actually requires. Placeholders are marked [LIKE THIS]. Replace all before posting. ## Template 1: System Engineer ### System Engineer [COMPANY NAME] · [LOCATION OR REMOTE] · [EMPLOYMENT TYPE] #### About the role We're looking for a System Engineer to define the technical backbone of our engineering team. This is not a senior individual contributor role in the traditional sense. The job is not to write the most code — it is to make it safe for others to write code quickly. You will design the systems, define the constraints, own the failure modes, and transfer enough of your mental model to the team that they can operate without you in the room. One excellent System Engineer enables four to eight engineers to work faster, build more reliably, and ship with confidence. That is the leverage point this role sits at. [ONE OR TWO SENTENCES ABOUT THE PRODUCT / DOMAIN / STAGE OF COMPANY] #### What you'll do - Design the system architecture that other engineers build within — data models, API contracts, service boundaries, consistency guarantees - Enumerate how each system can fail before it does, and build the observability that makes failures diagnosable by the team - Define what Product Engineers can deploy autonomously and what requires your review — and build automated guardrails where human review is too slow - Establish standards for AI-generated code entering the codebase; maintain a running catalogue of the patterns AI generates that are unsafe in our context - Transfer your mental model to the team — document the why behind architectural decisions, not just the what; run design reviews as teaching sessions, not approval gates - Own post-mortems end-to-end: the fix must close the class of problem, not just the instance #### What we're looking for We are not filtering on years of experience, specific frameworks, or degrees. We are looking for evidence of specific behaviours. What strong candidates demonstrate: - They talk about what a system must guarantee before they talk about how to build it - They can enumerate the failure modes of systems they've built, unprompted, with specific examples - They have a principled view on what to delegate to AI and what to own — and they've revised that view as AI capability has changed - They have designed systems that non-experts could operate safely — and can show evidence that this worked - They have transferred knowledge to others in ways that made those people genuinely more independent, not just more compliant - They have experimented with new tools and techniques on their own initiative and formed opinions from actual use Technical areas we care about: - Systems design and architectural tradeoffs - Failure mode reasoning and distributed system behaviour - Security fundamentals (OWASP-level minimum; threat modelling preferred) - AI output evaluation — reading and auditing AI-generated code for correctness and safety - Data modelling and API contract design #### How we assess Our interview process is designed to observe the behaviours above directly — not to test your ability to prepare for an interview. Stage | What you'll do | What we're looking for | Systems case study | We give you a scenario in writing. You think, then we discuss. | How you define constraints before architecture; how you reason about tradeoffs; what failure modes you anticipate | Failure mode review | We give you a system description and an incident report. | Whether you form hypotheses or reach for fixes; whether you identify the class of problem or just the instance | AI output audit | We give you 60–100 lines of AI-generated code with issues planted. | Whether you evaluate against behaviour and correctness, or against style; what questions you'd ask before approving | Structured behavioural | We ask the same questions we ask every candidate, in the same order. | Track record of the above behaviours in real situations | The process takes approximately four hours across two sessions. #### How we don't assess We have deliberately removed the following from our process: - Algorithm puzzles (LeetCode-style) — AI solves these reliably on first attempt. Passing them signals interview preparation, not engineering judgement. - Syntax and language trivia — The information is a prompt away. Retention of it tells us nothing about your ability to design or evaluate systems. - "5 years of [framework]" requirements — AI generates framework code. What we need is judgement that outlasts the current framework generation. - Timed coding under pressure — Implementation speed is not the bottleneck. Specification quality and output evaluation are. #### What we offer [COMPENSATION RANGE] Note on compensation: System Engineers command a premium because their leverage is real and their mistakes are expensive. A System Engineer who prevents one bad architectural decision pays for themselves many times over. We price accordingly. [BENEFITS, EQUITY, REMOTE POLICY, ETC.] #### About us [COMPANY DESCRIPTION — 3–5 SENTENCES] To apply: [APPLICATION LINK OR INSTRUCTIONS] ## Template 2: Product Engineer ### Product Engineer [COMPANY NAME] · [LOCATION OR REMOTE] · [EMPLOYMENT TYPE] Also posted as: Software Engineer / Full-Stack Engineer depending on market convention — same role, same criteria #### About the role We're looking for a Product Engineer to own a business domain and be accountable for what ships within it. This role is not defined by which tools you use. It is defined by what you own. You will take business requirements, translate them into precise specifications, evaluate the output for correctness, and be the person accountable when something behaves wrong in production. That accountability requires domain knowledge deep enough to catch what AI gets wrong about your domain — not just technical correctness, but business correctness. The person who ships a feature that runs correctly but behaves wrongly is a Product Engineer who didn't understand the domain well enough. We hire to prevent that. [ONE OR TWO SENTENCES ABOUT THE PRODUCT / DOMAIN / STAGE OF COMPANY] #### What you'll do - Own a business domain — its rules, its edge cases, its history, its failure modes - Take business requirements — often ambiguous ones — and translate them into precise behavioural specifications before any code is generated - Evaluate generated output for correctness: not just "does it run" but "does it behave correctly for this domain under the conditions that matter" - Be the quality gate on your own work — you are accountable for what goes to production, which means you evaluate it against correctness before it gets there - Track the AI tooling landscape on your own initiative; experiment with new tools before they're mainstream; form specific opinions from actual use - Work with System Engineers when a problem exceeds your scope — escalate before it becomes a crisis, not after #### What we're looking for We are not filtering on specific languages, frameworks, or years of experience. We are looking for evidence of specific behaviours. What strong candidates demonstrate: - Before building anything, they ask what the requirement actually means — and they don't stop asking until it's unambiguous - They evaluate generated code against what they specified, not against whether it looks right or tests pass - When output is wrong, they diagnose what the specification missed — they don't re-prompt randomly - They understand their domain well enough to identify domain-specific errors in technically correct code - They have caught output that looked correct but wasn't — and can explain how they caught it - They have experimented with tools on their own initiative in the last 90 days and have a specific, honest assessment of each Background diversity is expected: Strong Product Engineers come from many paths — traditional software engineering, QA, product management, domain expertise in a specialised field (fintech, healthcare, legal, logistics). The common thread is domain ownership and the discipline of evaluating correctness before shipping — not the background. #### How we assess Our interview process observes the behaviours above directly. Stage | What you'll do | What we're looking for | Specification exercise | We give you a vague requirement. You think, then write a specification. | Whether you resolve ambiguity before specifying; whether the spec is behavioural and testable; what you deliberately left out | Live AI-assisted build | You implement part of your specification using your preferred AI tool, with us observing. | Your process; how you evaluate the output; how you iterate when something is wrong | Output review | We give you AI-generated code with issues planted. | Whether you evaluate against correctness and behaviour, or against style; whether you find domain-specific and security issues | Structured behavioural | We ask the same questions we ask every candidate, in the same order. | Track record: specification failures, output catches, early adoption evidence | The process takes approximately three and a half hours across two sessions. #### How we don't assess We have deliberately removed the following from our process: - Algorithm puzzles (LeetCode-style) — AI solves these reliably. The test selects for interview preparation, not engineering judgement. - Framework depth ("5 years of React") — AI generates framework code. What we need is domain knowledge and output evaluation, which transfer across frameworks. - Syntax recall — The information is a prompt away. It tells us nothing about your domain judgement. - Timed bug fix speed — We are not measuring how fast you type. We are measuring whether you understand what correct looks like before you ship it. #### What we offer [COMPENSATION RANGE] Note on compensation: Product Engineers with deep domain expertise in a specialised field command a premium — the domain knowledge is the scarce asset, not the technical execution. Engineers who bring demonstrated early adoption of tools that give the team a compounding advantage are also compensated above the standard range. [BENEFITS, EQUITY, REMOTE POLICY, ETC.] #### About us [COMPANY DESCRIPTION — 3–5 SENTENCES] To apply: [APPLICATION LINK OR INSTRUCTIONS] ## Usage Notes On the "How we don't assess" section: This section does more than filter candidates. It signals to strong candidates — particularly those who have been penalised by legacy interview processes — that this company has thought carefully about what the job actually requires. Engineers who spent years building real systems in real domains instead of practising LeetCode will recognise this and self-select in. That is the intended effect. On the role title: "Product Engineer" maps to what the role actually is — ownership of a product domain. Externally, title the JD to match the seniority level and market convention for your sector. Common external titles: Software Engineer, Full-Stack Engineer, Product Engineer. Avoid titles that suggest deep algorithm or systems knowledge that the role does not require — this drives the wrong applicants and loses the right ones. On the assessment table: The table is intentionally transparent. Candidates who read it and self-select out because they know they haven't been experimenting with AI tools or cannot write a behavioural specification are self-screening correctly. That is a filtering function, not a deterrent. Role profiles: System Engineer | Product Engineer Interview guides: System Engineer | Product Engineer © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution. ==================== # Hiring Audit Worksheet (https://aieraengineering.com/hiring-audit-worksheet/) How signal-efficient is your current interview process? ## What This Is A self-assessment tool for engineering leaders to evaluate their existing technical interview process against the AI-era framework. The output is a single signal-efficiency score and a prioritised list of changes. Time required: 30–45 minutes to complete honestly. Who should complete it: The person who owns your technical hiring process. If that's a committee, complete it independently then compare — disagreements are useful data. ## Part A: Process Inventory List every step in your current technical interview process. Include phone screens, technical screens, live coding sessions, take-homes, system design rounds, behavioural interviews, reference checks — everything. For each step, complete columns 2–6 using the reference tables in Part B. # | Step name | What skill does it test? | Skill relevance today (1–5) | Method validity (1–5) | Effective signal score (col 4 × col 5 ÷ 5) | 1 | | | | | | 2 | | | | | | 3 | | | | | | 4 | | | | | | 5 | | | | | | 6 | | | | | | 7 | | | | | | 8 | | | | | | Total interview hours (approximate): — Sum of effective signal scores: — / — (max = number of steps × 5) Process signal efficiency: sum ÷ max × 100 = — % ## Part B: Reference Tables Use these to fill Part A columns 4 and 5. If your step doesn't appear here, use the definitions at the bottom to assign scores yourself. ### Skill Relevance Today (Column 4) How relevant is the skill being tested to actual job performance in 2026? Skill | Score | Rationale | Specification quality (writing testable behavioural requirements) | 5 | Primary bottleneck in AI-era development | AI output evaluation (reviewing AI-generated code for correctness) | 5 | Daily job for all roles | Failure mode reasoning (what breaks, when, and why) | 5 | Core System Engineer skill; critical for Product Engineers | Early adoption behaviour (self-directed tooling experimentation) | 5 | Highest-value hiring signal; compounds over time | Hypothesis-driven debugging | 4 | AI generates more bugs; debugging is now a differentiator | Systems design and constraint identification | 4 | Higher leverage as implementation moves to AI | Domain knowledge | 4 | Required for specification correctness; not replaceable by AI | Data modelling | 4 | AI generates bad schemas; humans need to catch them | Communication and specification clarity | 4 | Specification is communication; both are now primary skills | Code quality and structural thinking | 3 | Still matters; style automates, structure doesn't | API design | 3 | Relevant but not primary | General problem-solving process | 3 | Relevant but hard to isolate from other signals | Algorithm implementation (DSA / LeetCode) | 1 | AI automates; passes are indistinguishable from AI-assisted passes | Framework-specific knowledge ("5 years of React") | 1 | Half-life 1–3 years; AI generates idiomatic framework code | Language syntax and API recall | 1 | A prompt away; measures memory, not judgement | CV walk-through / "tell me about yourself" | 1 | Measures rehearsal quality, not engineering capability | Brain teasers / Fermi estimation | 1 | Google's own research: zero relationship to job performance | ### Method Validity (Column 5) How well does this method actually measure what it claims to measure? Method | Score | Validity basis | Live AI-assisted build (observed) | 5 | Closest to actual job task; r=.44 (Schmidt & Hunter 1998) | Work sample — realistic task (code review, spec writing) | 5 | r=.33–.54 (Roth et al. 2005); high ecological validity | Structured behavioural interview (consistent questions, rubric) | 4 | r=.42 (Sackett et al. 2022) | Failure mode / incident review | 4 | High ecological validity; directly mirrors real diagnosis work | Technical presentation of past project (deep follow-up) | 4 | High resistance to fabrication; behavioural evidence from past | Specification exercise (write requirements from vague input) | 4 | Observable output; directly testable against criteria | Portfolio / GitHub review (with structured evaluation) | 4 | r=.33+ for well-structured review; evaluator quality is the variable | System design (restructured: constraints + failure modes, not patterns) | 4 | High validity when format is ecological; lower when whiteboard-only | Unstructured behavioural interview | 2 | r=.38 (Schmidt & Hunter 1998); amplifies impression management and bias | Take-home assignment (code output evaluated) | 2 | AI completes these undetectably; output may not reflect candidate at all | Timed coding exercise (speed metric) | 2 | Measures implementation speed, which is no longer the bottleneck | System design (whiteboard, pattern recognition format) | 2 | Extroversion and drawing ability add noise; ecological validity is low | Online coding test (HackerRank / Codility) | 1 | AI assistance is undetectable; filters for AI use, not engineering reasoning | Algorithm puzzle / LeetCode | 1 | Measures interview prep; no published correlation with job performance | Brain teaser | 1 | "Zero relationship" to job performance (Bock, Google, 2013) | CV walk-through | 1 | r≈−0.5 for deception (Weiss & Feldman 2006: 81% of candidates lie) | "What is your biggest weakness?" | 1 | Near-universally answered with a rehearsed non-answer | ## Part C: Signal Analysis ### C1. Redundancy Check Are you testing the same skill more than once? Skill | Steps that test it (from Part A) | Is the duplication adding signal? | | | Y / N | | | Y / N | | | Y / N | | | Y / N | Total redundant steps: — ### C2. Coverage Gap Check Which high-relevance skills are you NOT currently assessing at all? High-relevance skill (score 4–5) | Currently assessed? | Steps that should cover it | Specification quality | Y / N | | AI output evaluation | Y / N | | Failure mode reasoning | Y / N | | Early adoption behaviour | Y / N | | Hypothesis-driven debugging | Y / N | | Domain knowledge | Y / N | | Total gaps (skills rated 4–5 not currently assessed): — ### C3. False Signal Check Which steps are consuming significant interview time with low effective signal? From Part A, list all steps where effective signal score ≤ 1.5: Step | Time spent (hours) | Effective signal score | Candidate impact if removed | | | | | | | | | | | | | Total hours spent on low-signal steps: — ## Part D: Output Summary ### Your Current Score Metric | Your value | Benchmark | Process signal efficiency | — % | Target: ≥65% | Hours on low-signal steps | — hrs | Target: ≤20% of total | Coverage gaps (high-relevance skills untested) | — | Target: 0 | Redundant steps | — | Target: ≤1 | ### Recommended Changes (prioritised) Based on your answers above, list changes in priority order. Start with the highest-impact, lowest-cost changes. Priority 1 — Remove (low signal, replaceable): Step to remove | Hours recovered | Replace with | | | | | | | Priority 2 — Restructure (right skill, wrong method): Step to restructure | Current method | Better method | | | | | | | Priority 3 — Add (high-relevance skill not currently tested): Skill to add | Recommended method | Source | Specification quality | Specification exercise (Stage 1, Product Engineer guide) | 05-product-engineer-question-bank.md | AI output evaluation | Code review with planted issues (Stage 3, both guides) | 04-system-engineer-question-bank.md | Failure mode reasoning | Incident scenario review (Stage 2, SE guide) | 04-system-engineer-question-bank.md | Early adoption behaviour | Structured behavioural Q5 (both guides) | 03-early-adopter-framework.md | ## Part E: Honest Assessment Questions Answer these before finalising your change list. They surface issues the scoring doesn't catch. 1. What does your current process select for — not what you intend, but what candidates who pass it have in common? Answer: 2. Think of the last engineer you hired who underperformed. What would a better interview have caught? Answer: 3. Think of the best engineer you know who would struggle with your current interview. What does that tell you? Answer: 4. What does your current process not test at all that has turned out to matter? Answer: 5. If a candidate spent six weeks specifically preparing for your interview, what skills would they improve that have nothing to do with the job? Answer: ## Part F: Baseline Score Record your pre-change score here. Revisit after implementing changes. | Before | After (date: —) | Signal efficiency % | | | Hours on low-signal steps | | | Coverage gaps | | | Redundant steps | | | ## Reference Full method validity data: 06-skill-map-research-backing.md Assessment theory: 04-assessment-theory.md Skill relevance data: 01-traditional-skills-weighted.md | 02-new-skills-weighted.md Interview guides: System Engineer | Product Engineer © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution. ==================== # Team Realignment Guide (https://aieraengineering.com/team-realignment-guide/) # Team Realignment Guide Restructuring an Existing Engineering Team for the AI Era ## Who This Is For Engineering leaders managing teams built for a different era. You have experienced engineers. Some are adapting well to AI tooling; others aren't. You have a sense that something structural needs to change but haven't named it yet, or you've named it but don't know where to start. This guide is not about letting people go. It is about deliberately shifting how your existing team is structured, what each person's leverage point is, and what you hire for going forward. ## The Starting Point: What You Probably Have Most engineering teams built before 2023 look like this: Engineering Manager ├── Senior Engineers (2–4) — write complex code, review others' work ├── Mid Engineers (3–6) — implement features, participate in design └── Junior Engineers (2–3) — implement well-defined tasks, learn Everyone is on roughly the same track — juniors grow into mids, mids into seniors. Seniority is measured by how fast and how well you write code. This model made sense when writing code was the primary bottleneck. It doesn't anymore. What AI has done to this model: - The junior role as traditionally defined — implement well-scoped tasks, learn by doing — is being absorbed by AI generation. Engineers who adapt build domain knowledge and specification discipline; they grow into Product Engineers. Those who don't become expensive and redundant. - Mid engineers divide sharply into two groups: those who have built genuine mental models of what they're building (they become effective Product Engineers and System Engineer candidates), and those who are now discovering they were faster than they were deep. - Senior engineers face the same bifurcation at higher stakes. The ones who were genuinely deep — who understood systems, failure modes, and architecture — become more valuable. The ones whose seniority was primarily implementation speed lose their primary differentiator. The uncomfortable implication: AI doesn't flatten the team. It makes the difference between real depth and the appearance of depth visible faster than before. ## The Target Model The destination is not a radical reorganization. It is a deliberate clarification of two distinct roles that already exist in your team but aren't currently named or staffed for. System Engineers (own architecture, failure modes, guardrails, mental model transfer) Product Engineers (own a business domain, translate requirements into precise specifications, evaluate output for correctness, are accountable for what ships) These are not a hierarchy. They are parallel tracks with different leverage points and different skill profiles. See Role Taxonomy for full definitions. The ratio goal for most teams: 1 System Engineer for every 3–5 Product Engineers. See Team Ratios for calibration by company type. ## Phase 1: Map Your Current Team Before changing anything, you need an honest picture of where each person sits today. ### The three-group assessment Assess every engineer on your team against the three groups defined in Current State 2025–2026: Group | Behaviour pattern | What this means for realignment | Group 1 — Avoider | Does not use AI tools; treats them with suspicion or ideological resistance | May have deep skills worth protecting; needs a clear path to engagement or an honest conversation about the role's direction | Group 2 — Dumper | Uses AI heavily; produces output they cannot fully evaluate or explain; velocity up, quality variable | The most common group; not a lost cause — the path forward is specification discipline and output evaluation training | Group 3 — Steerer | Uses AI as a directed tool; specifies before generating; evaluates output against a model; forms opinions from use | Your Product Engineer candidates and your System Engineer candidates are both in this group | Complete this table for your team: Name | Group (1/2/3) | Primary strength | Primary risk | Role direction | | | | | | | | | | | | | | | | Role direction options: System Engineer track / Product Engineer track / Needs support / Critical conversation needed ### The System Engineer identification checklist For each engineer you're considering for the System Engineer track, check: - [ ] Can enumerate the failure modes of systems they have built, without prompting - [ ] Has designed systems that other people operated safely — and can show evidence - [ ] Has strong opinions on where AI should and should not touch the codebase - [ ] Actively transfers knowledge; does not hoard expertise - [ ] Has cleaned up after Group 2 engineers and understands the failure patterns - [ ] Can explain architectural decisions and their tradeoffs to non-engineers If fewer than four boxes are checked: This person may be a strong Product Engineer, not a System Engineer. Both are valuable. The roles are not interchangeable. ### The Product Engineer identification checklist For each engineer you're considering for the Product Engineer track, check: - [ ] Has genuine domain knowledge — can identify what is wrong with a technically correct result for their domain - [ ] Builds a mental model before generating — specifies before touching any tool - [ ] Evaluates output against the spec, not against “does it run” - [ ] Asks “would I be comfortable owning this in production?” as a default, not only when prompted - [ ] Knows when to escalate vs. when to ship If fewer than three boxes are checked: This person is likely a Group 2 engineer. The path forward is training and structure, not role change — see Phase 2. ## Phase 2: Stabilise Before Restructuring Do not announce a restructuring before you have done two things: - Identified and secured your System Engineers - Established the minimum enabling structures that Product Engineers need to operate Restructuring without System Engineers in place produces the Anti-Pattern 1: Engineer Shortage Workaround. Restructuring without enabling structures produces Anti-Pattern 4: The Specification-Free Team. Both are expensive. ### Step 1: Protect your System Engineers System Engineers are your highest-retention risk. They are: - Recognizably more valuable than before AI — the market knows this - Being actively recruited by companies that have figured out the new model - The people who, if they leave, take the team's institutional knowledge with them Actions: - Have a direct conversation with each System Engineer candidate about their future role before announcing any changes to the broader team - Understand what they want: more autonomy? Fewer reviews? More architectural scope? Pay adjustment? - Make a retention decision before they receive an external offer, not after The conversation outline: "I want to talk about how your role is evolving. You have skills that are becoming more valuable as the team changes, not less. I want to make sure we structure your role to use those skills properly, and that we're compensating you in a way that reflects that. Here's what I'm thinking..." Then listen. Their answer will tell you whether the retention risk is manageable. ### Step 2: Establish the specification gate Before restructuring roles, establish one process change: no AI-generated code enters the codebase without a written behavioural specification. This is the minimum viable enabling structure. It does three things: - Catches the Group 2 pattern before it produces damage - Creates an observable artifact (the spec) that you can review and improve - Starts shifting the team's mental model towards "specify first" without a confrontational role conversation How to introduce it: - Frame it as a quality change, not a surveillance measure - Start with new features only — don't retrofit existing work - Keep the spec template lightweight: happy path, failure path, definition of done, explicit exclusions - Put the System Engineer in the spec review role, not the approval role — "improve together," not "gate" See Enabling Structures for implementation detail. ## Phase 3: The Realignment Conversation The hardest part of restructuring is telling people their role is changing. The way you frame it determines whether you lose good people or develop them. ### What not to say - "We're restructuring because AI has changed things." — Too abstract. People hear "AI is replacing you." - "We're splitting into senior and junior tracks." — Wrong framing. Product Engineer is not junior. - "Some people will be System Engineers and some won't." — Sounds like winners and losers. It isn't. ### What to say There are three different conversations depending on who you're talking to. #### Conversation A: The System Engineer For engineers whose skills align with the System Engineer profile. "I want to talk about what the next phase of your role looks like here. The work you do — designing systems, catching failure modes, helping others build safely — is becoming more important, not less. I'd like to formalize that. > What I'm proposing is that you move away from implementation output as your primary measure and towards architectural quality and team capability as your primary measure. Concretely: you'd spend more time on design, guardrails, and transfer, and less time on feature delivery. > This is not a step sideways — it's a recognition of where your leverage actually is. I want to make sure your compensation and title reflect that. > Before I get further, I want to hear from you: does this match how you see your own strengths? And is there anything about this that concerns you?" #### Conversation B: The Product Engineer For engineers whose strengths are in specification, domain knowledge, and output evaluation — or who have strong domain expertise from a non-traditional background. “I want to talk about how I see your role evolving, and specifically why I think you're well-positioned for it. > The thing that's becoming more valuable isn't implementation speed — it's knowing what correct looks like and being willing to be accountable for it. That's domain knowledge, specification precision, and the instinct to ask 'would I be comfortable owning this in production?' before shipping. You demonstrate those things. I want to make sure your role reflects that explicitly. > What that means practically: your primary measure shifts from features delivered to domain correctness and outcome accountability. You own a business domain — its rules, its edge cases, its history, its failure modes. When output is technically correct but wrong for the domain, you're the one who catches it. That's not a smaller responsibility than writing the code yourself. It's a different one — and in an AI-generated codebase, it's the harder and more important one. > I want to be direct about what this isn't: it isn't a downgrade, and it isn't asking you to become a prompt engineer. Implementation is the cheap part now. Domain knowledge and outcome accountability are the scarce parts. I want to build a role structure that compensates accordingly. > What questions do you have?” #### Conversation C: The Engineer Who Needs Support For Group 2 engineers who are producing output they can't fully evaluate. This is the most delicate conversation. The person is not failing — they are using the tools available to them in the most natural way. The problem is structural, not personal. Approach it that way. "I want to have an honest conversation about something I've been observing, and I want to hear your perspective on it. > When I look at [specific feature / recent work], I see output that moves fast but that we've had to revisit more than I'd like. I don't think this is a capability problem — I think it's a process problem. Specifically, I think the way you're using AI tools is optimising for speed over correctness. > What I mean by that: it looks like the generation step is happening before the specification step. The output gets produced quickly, but the edge cases and failure paths are found later — in review, in QA, or in production. That's expensive. > I want to work with you on changing that. Not by slowing you down — by changing what 'fast' means. Specifying well upfront is faster in total than specifying vaguely and reworking three times. > I'd like to invest time with you specifically on specification quality. Here's what I'm proposing..." Then propose a concrete path: pairing on specifications with a System Engineer or senior Product Engineer for the next four to six weeks, with explicit review of spec quality rather than output. If this conversation has to happen more than twice with the same person without improvement: the issue may be a deeper unwillingness to change the workflow. At that point the conversation shifts from development to fit. ## Phase 4: Restructure Hiring Once the internal structure is clarifying, align hiring to reinforce it. ### What to stop hiring for - Framework depth as a requirement — "5 years of React" screens out the people you now need and screens in the people you need less - LeetCode as a screen — Signals preparation, not judgement; wastes interview time on the wrong signal - "Strong coder" as the primary hiring filter — Implementation is cheap; judgement and specification are not ### What to start hiring for System Engineers: - Evidence of designing systems others built within safely - Failure mode reasoning demonstrated in interview, not just claimed on CV - Knowledge transfer track record: taught, not just did - Early adoption behaviour — opinion from use, not from announcements Product Engineers: - Domain expertise: deep enough to catch errors that are technically correct but wrong for the business context - Specification discipline: builds mental model before generating, not after - Output evaluation: reads and checks against expected behaviour, not against whether it runs - Accountability instinct: asks "would I own this in production?" before shipping Use the JD templates and interview guides in this toolkit: Tool | Path | System Engineer JD | 08-jd-templates.md | Product Engineer JD | 08-jd-templates.md | System Engineer interview | 01-deep-engineer-interview.md + 04-system-engineer-question-bank.md | Product Engineer interview | 02-product-engineer-interview.md + 05-product-engineer-question-bank.md | ### The critical hiring sequence Do not hire out of sequence. It creates structural debt. - System Engineer first — Without one, everything Product Engineers build will need to be refactored - Senior Product Engineer second — They establish norms; hiring entry-level Product Engineers without senior guidance produces Group 2 patterns - Entry Product Engineers last — They need mentoring infrastructure or they default to dysfunction See Team Ratios for target headcount by team size and stage. ## Phase 5: Build the Enabling Structures Restructuring roles without building the structures that make those roles effective produces confusion and frustration. The System Engineer's primary job in the first 90 days of the new model is not feature delivery — it is building the enabling structures that Product Engineers operate within. The six structures, in build priority order: Priority | Structure | What it does | Time to implement | 1 | Specification gate | Stops spec-free generation from entering the codebase | 1–2 weeks | 2 | Pattern library | Gives Product Engineers on-pattern examples for common tasks | 2–4 weeks | 3 | Escalation protocol | Defines when Product Engineers call System Engineers | 1 week | 4 | Automated guardrails | CI rules that catch architectural violations automatically | 4–8 weeks | 5 | Mental model library | Documented architectural context Product Engineers can reference | Ongoing | 6 | Learning loops | Regular format for transferring knowledge across the team | Ongoing | Full implementation guide: Enabling Structures. ## Phase 6: Track Progress Restructuring is complete when the behaviour changes, not when the org chart changes. Use these metrics. ### Team health metrics (monthly check) Metric | Before realignment | Target | Current | % features requiring rework post-ship | \_\_\_ % | | Avg time to locate a production bug | \_\_\_ hrs | | System Engineer escalation rate | \_\_\_ % | 10–20% | | Spec rejection rate (behavioural mismatch) | \_\_\_ % | | % PRs with > 3 AI-generated code review flags | \_\_\_ % | | Copy baseline from your current state before starting Phase 1. ### Individual development signals (per person, quarterly) For System Engineers: - [ ] Has documented at least one architectural decision record this quarter - [ ] Has identified and closed one class of failure mode (not just one bug) - [ ] Has run at least one knowledge transfer session - [ ] Escalation rate from their pod is in the 10–20% range For Product Engineers: - [ ] Has written specifications that passed first review without behavioural gaps - [ ] Has caught at least one domain correctness error before it reached review (not just a syntax or style issue) - [ ] Can articulate a specific way in which current AI tooling gets their domain wrong — from direct use - [ ] Has reduced System Engineer escalation rate on routine tasks within their domain ## Common Failure Modes in Realignment ### "We announced the structure but nothing changed" The org chart changed; the process didn't. System Engineers are still reviewing every PR. Product Engineers are still generating without specs. No enabling structures were built. Fix: Go back to Phase 2. Structure before restructuring. ### "The System Engineer became a bottleneck" The System Engineer is trying to review everything instead of building guardrails that make review unnecessary for routine work. Fix: Redirect their work. The question is not "did this PR pass review" but "what guardrail would have caught this automatically?" Every manual review should produce either a guardrail or a pattern library entry. ### "The best Product Engineers are frustrated" Strong Product Engineers in an under-structured team are doing System Engineer escalation work for their peers. They're carrying others. Fix: Accelerate the enabling structures. Separate Product Engineer seniority tracks (senior Product Engineers should not be doing System Engineer work by default — they should be raising specification quality across the team). ### "We lost a good engineer during the transition" They felt their role was being downgraded or that the new model didn't value what they brought. Fix: The Conversation C framing failed, or happened too late. The realignment conversation must happen before the person feels demoted by events — not after. If someone only hears about the new model when their title or scope changes, the conversation is already defensive. ## Quick-Start Checklist For teams that need to move quickly: Week 1: - [ ] Complete Phase 1 team assessment (private, honest, no announcements) - [ ] Identify System Engineer candidates — have retention conversation with each Week 2: - [ ] Introduce specification gate for new features only - [ ] Draft escalation protocol with System Engineer(s) Weeks 3–4: - [ ] Have Conversation A and B with individuals (not a group announcement) - [ ] Update JDs for open roles to the new templates Month 2: - [ ] System Engineer begins pattern library - [ ] First round of team health metrics established as baseline - [ ] Conversation C with Group 2 engineers who need support Month 3: - [ ] First automated guardrail in CI - [ ] First knowledge transfer session run formally - [ ] Review team health metrics against baseline © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution. ==================== # The Interview Drift Toolkit (https://aieraengineering.com/interview-drift-toolkit/) Most engineering teams are measuring the right things in the wrong era. The technical skills that interviews have tested for decades — algorithm fluency, framework depth, implementation speed — were built for a bottleneck that no longer exists. AI moved the bottleneck from writing code to specifying and evaluating it. Interview Drift is the measurable distance between what your hiring process tests and what the role actually requires. High drift means you are accurately measuring skills that have been automated. The Relevance Shift — the threshold event when AI capability made implementation skill less scarce — increased interview drift across the industry. Most teams have not measured it. This toolkit exists to close that gap. Four layers: understand the shift, hire for the right signals, structure the team, build the systems they work in. ## Understand the Shift The skill map plots every standard interview method against two axes: how relevant the skill is today, and how well the method actually measures it. The pre-AI baseline shows the same map before the Relevance Shift. The distance between the two maps is your industry-level interview drift. Interview Skill Map — 2026 Edition — interactive, hover each point for detail. Interview Skill Map — Pre-AI Baseline — where the industry was before the Relevance Shift. ## Hire for the Right Signals The AI-era team runs on two primary roles. The System Engineer designs the systems others build within — architecture, guardrails, failure handling. The Product Engineer owns a business domain and is accountable for whether what ships is correct. These are different functions requiring different interview signals. ### System Engineer System Engineer Question Bank — 18 questions across 4 stages, each with construct, scoring rubric, and red flags. System Engineer Scorecard — one-page scoring sheet, weighted to 100, with hire/no-hire signals and debrief prompts. ### Product Engineer Product Engineer Question Bank — 17 questions across 4 stages, including a live observed work session (the highest-validity method in the toolkit). Product Engineer Scorecard — includes the live session observation checklist and output evaluation criteria. ### For Both Roles Job Description Templates — ready-to-post JDs for both roles, with "how we assess / how we don't" sections that set candidate expectations correctly. Hiring Audit Worksheet — self-assessment tool. Scores your current process for signal efficiency, identifies redundancy and coverage gaps, and produces a prioritised change list. ## Structure the Team Adding AI to an unchanged team structure produces the same problems faster. The old engineering ladder was designed around implementation skill — the right axis when writing code was the bottleneck. The AI-era team runs on two roles, not one ladder. The basic unit is a pod: one System Engineer, two to four Product Engineers, full lifecycle ownership of one product surface. Team Realignment Guide — six-phase guide for restructuring an existing team. Includes a three-group assessment, per-role identification checklists, conversation scripts for existing engineers, hiring sequence, and a failure modes catalogue. ## Build the Architecture The system has to change when the team changes. Architecture designed around the old bottleneck — where the hard work was in writing the code — breaks when Product Engineers operate it without the institutional memory the original team carried. Unwritten constraints get violated. Test suites echo the same incomplete specification as the implementation. Errors require a System Engineer to interpret. The Architecture Problem Nobody Fixes When They Adopt AI — the three failure patterns and the six enabling architecture principles that address them. ## The Article Series Six articles on what changed and what to do about it — from hiring signals to leadership to team structure to the systems they build within. 1. What Not to Ask Software Engineers Anymore — four standard interview methods, and why each has lost its signal. 2. What to Ask Instead — five replacement exercises designed for the new bottleneck. 3. The Question You've Never Thought to Ask — the early adopter signal, and why it is the most undervalued question nobody uses. 4. Leadership in the AI Era — AI amplifies what is already there. What that means for the decisions leaders make now. 5. Why Adding AI to Your Existing Team Structure Doesn't Work — the ladder was built for the wrong bottleneck. What replaces it. 6. The Architecture Problem Nobody Fixes When They Adopt AI — the same bottleneck shift applies to systems, not just teams. © Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution.