Product Engineer: Question Bank
# Question Bank: Product Engineer The AI-Era Engineering Playbook — Practitioner Reference
Design Principles
This bank is organised by interview stage. The Product Engineer interview has a structural difference from the System Engineer interview: Stage 2 is a live observed work session, not a static exercise. This is the highest-validity method available and requires specific observation criteria rather than pre-written questions.
The four stages move from specification (what happens before any tool is touched) through live observed work (evaluation and accountability in real time) through output review (can they find what's wrong) to behavioural evidence (what their track record shows about domain ownership and correctness).
Scientific basis for stage design
| Stage | Assessment Method | Validity (r) | Source | Bloom Level |
|---|---|---|---|---|
| 1 — Specification Exercise | Work sample (live spec writing) | .33–.54 | Roth et al. 2005; Schmidt & Hunter 1998 | Analyse / Create |
| 2 — Live AI-Assisted Build | Work sample (observed real-time) | .33–.54 | Roth et al. 2005 | Apply / Evaluate / Create |
| 3 — Output Review | Work sample (code review exercise) | .33–.54 | Roth et al. 2005 | Evaluate |
| 4 — Structured Behavioural | Structured behavioural interview | .42 | Sackett et al. 2022 | Apply / Analyse |
Stage 2 note: The live AI-assisted build is the closest available method to a paid work trial (Schmidt & Hunter 1998: r=.44). The candidate uses their actual tools on a realistic task while you observe. You are not grading the output. You are grading the process — which is directly visible and cannot be rehearsed.
Why early adoption is weighted at 20% for this role: The Product Engineer role requires active self-directed experimentation as a job function. An engineer who waits to be told which tools to use cannot maintain judgement about what AI gets wrong in their domain. Early adoption is a disqualifier at score 1 — not merely a concern. See Early Adopter Framework.
Key validity references:
- Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262–274.
- Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(12), 2040–2068.
- Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity. Personnel Psychology, 58, 1009–1037.
- Wiggins, G., & McTighe, J. (1998). Understanding by Design. ASCD.
- Lave, J., & Wenger, E. (1991). Situated Learning. Cambridge University Press.
Scoring convention
All questions use a 1–5 scale. Score based on observed behaviour, not inferred ability.
| Score | Label | Meaning |
|---|---|---|
| 5 | Exceptional | Exceeds expected depth; demonstrates mastery with specific, unprompted insight |
| 4 | Strong | Covers core fully; minor gaps; demonstrates clear competence |
| 3 | Adequate | Covers the basics; requires prompting to reach depth; no major gaps |
| 2 | Weak | Partial coverage; significant gaps; cannot reach depth even when prompted |
| 1 | Absent | Not demonstrated; cannot engage meaningfully with the question |
Auto-reject: P4-Q5 (early adoption) scores 1; or Stage 2 overall scores average below 2; or two or more questions in any single stage score 1.
Stage 1: Specification and Mental Modelling
Assessment method: Live specification writing exercise
Skill map position: Specification test (X=+9, Y=+8)
Validity basis: B — directly mirrors the primary daily task; high ecological validity
Construct: Ability to build a complete mental model of what should exist before generating anything; ability to translate ambiguity into testable behavioural requirements
Administration notes:
- Read the requirement aloud. Give it in writing. Give 5 minutes to think silently.
- Do not clarify the requirement — ambiguity is the test. Watch what they do with it.
- The exercise continues through Q1–Q4 in sequence. Do not skip ahead.
- Score on process and output quality, not on whether their spec matches any expected version.
Sample requirement (rotate per candidate):
"Add a notification feature. Users should be notified about important events."
(Intentionally vague. That is the whole point.)
P1-Q1. "Before writing anything — what do you need to know first?"
Construct: Ambiguity recognition; requirement decomposition; question quality as a proxy for mental model depth
Bloom level: Analyse
Chart method: Specification test (X=+9, Y=+8)
Research note: The quality of a Product Engineer's clarifying questions is the primary early signal. Wiggins & McTighe (1998) establish that genuine understanding shows when a person can identify what is missing from a problem statement before attempting to solve it. The specification test has high face and ecological validity: the behaviour directly mirrors the first step of real AI-assisted development.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Asks 4–6 questions that together would fully constrain the specification. Questions are behavioural, not implementational: "Who are the users?" "What counts as an important event — who decides?" "What channels?" "What happens if delivery fails?" "What are the user's notification preferences?" "What's the expected volume?" Does not ask "should we use SendGrid?" or "what database?" before the behavioural questions are answered. |
| 4 | Asks 3–4 substantive behavioural questions. Covers the core ambiguities (event definition, channel, failure path). May ask one implementation question prematurely. |
| 3 | Asks 1–2 questions. Misses significant ambiguities (e.g., accepts "important events" without defining what that means; doesn't ask about failure paths). |
| 2 | Asks clarifying questions about implementation only ("what language?" "what framework?") or asks one vague question ("can you tell me more?") and moves on. |
| 1 | Does not ask clarifying questions. Begins specifying or implementing the vague requirement as given. |
Red flags:
- First question is about technology ("should I use a message queue?")
- Accepts "important events" as a defined term without unpacking it
- Asks only one question and considers the ambiguity resolved
Probing follow-up: "Is there anything you'd want to know that I haven't answered yet before you start writing?"
P1-Q2. "Write the specification. Walk me through it as you go."
Construct: Specification completeness; behavioural framing; assumption explicitness; testability
Bloom level: Create
Chart method: Specification test (X=+9, Y=+8)
Research note: This is the core work sample question. The output — the written specification — is directly evaluable against the criterion: is it complete, behavioural, and testable? The "walk me through it" component adds process visibility (Lave & Wenger 1991: knowledge in context) and reveals whether the written output reflects genuine understanding or surface-level pattern matching.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Specification is written in behavioural terms: "When [event X occurs], the system shall [notify user Y via Z], provided [condition A]. If delivery fails after [N] retries, [fallback behaviour]." Assumptions are explicitly labelled. Failure paths are addressed. Edge cases are enumerated. The specification could be handed to an engineer (or an AI) and produce a predictable outcome. Could write acceptance tests from the specification without further discussion. |
| 4 | Specification covers the happy path and most edge cases. Behavioural framing is mostly present. Some assumptions are implicit. Failure paths are partially addressed. Testable with minor clarification. |
| 3 | Specification covers the happy path. Edge cases are absent or noted as "TBD." Failure paths are not addressed. Some implementation language present ("we'll use a queue"). Testable for the happy path only. |
| 2 | Specification describes a partial solution rather than a complete behaviour. More implementation than behaviour. Cannot be used to write acceptance tests without significant additional discussion. |
| 1 | Writes a design document or code instead of a specification. Cannot distinguish between specifying behaviour and implementing it. |
Red flags:
- "When the event happens, call the notification service" — describes implementation, not behaviour
- No mention of failure paths ("what if the notification doesn't deliver?")
- Assumptions are embedded, not labelled ("I'm assuming users have one email address")
- Cannot explain what a test for this specification would check
Probing follow-up: "Show me one acceptance test that you could write directly from what you've written."
P1-Q3. "What did you explicitly choose NOT to cover, and why?"
Construct: Scope awareness; deliberate constraint; the distinction between MVP and complete specification
Bloom level: Evaluate
Research note: A specification that attempts to cover everything is not a specification — it is a wish list. Deliberate scoping requires the engineer to have a mental model of the whole problem, identify what is in scope vs. deferred, and be able to defend the line. This is a directly observable form of engineering judgement.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names 2–3 specific items they chose not to cover and gives principled reasons for each (e.g., "I didn't spec notification preferences because that requires a separate preferences model that would double the scope; I left it as a labelled assumption"). Demonstrates awareness that the excluded items are real requirements that will need to be addressed, not things they missed. |
| 4 | Names 1–2 exclusions with reasoning. The reasoning is principled, not evasive. |
| 3 | Notes that they "kept it simple" without naming specific exclusions or articulating why the line was drawn there. |
| 2 | Exclusions are things they forgot or didn't notice, not things they deliberately deferred. |
| 1 | "I tried to cover everything." Cannot name an item they chose not to include. |
Red flags:
- Exclusions are things they missed, not things they chose to exclude
- Cannot explain why one thing was in scope and another wasn't
- "I'll add more later" without a clear scope boundary now
P1-Q4. "If I handed this specification to an AI agent and it built something that passed all your tests but behaved wrong — what did you miss?"
Construct: Specification gap awareness; the limits of testability; anticipating AI failure modes from underspecification
Bloom level: Evaluate
Research note: This question targets the most important failure mode of AI-assisted development: specifications that are complete enough to generate code from, but incomplete enough that the code can be technically correct yet behaviourally wrong. This is the construct that most traditional interview methods cannot reach. There is no published validity data specific to this question type — it is a novel construct that emerged post-2022.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Identifies a specific gap in their own specification that an AI would exploit plausibly — not a generic answer. Examples: "I didn't define what 'delivered' means — the AI might mark a notification as delivered when the API returns 200, even if the email bounced." "I didn't specify idempotency — if the event fires twice, the user gets two notifications." Demonstrates that they understand AI generation produces maximally literal interpretations of specifications. |
| 4 | Identifies a specific gap. The reasoning connects the gap to a plausible AI failure mode. May need light prompting to be specific. |
| 3 | "There are probably edge cases I missed." Cannot identify a specific one without prompting. |
| 2 | "My tests would catch any problem." Does not accept the premise that a test-passing system can still be behaviourally wrong. |
| 1 | Cannot engage with the question. |
Red flags:
- "I'd just ask the AI to find the gaps" — avoids the metacognitive step
- "If it passes the tests it's correct" — conflates test coverage with behavioural correctness
- Cannot name a specific gap in their own specification under any prompting
Probing follow-up: "What constraint would you add to the specification right now to close that gap?"
Stage 2: Live AI-Assisted Build and Evaluation
Assessment method: Observed real-time work session
Skill map position: Work sample test (X=+9, Y=+9)
Validity basis: A — closest available method to a paid work trial; Schmidt & Hunter (1998) job tryout r=.44
Construct: Output judgement in real time; evaluation of output against specification and domain requirements; accountability before shipping; diagnostic iteration when output fails
Administration notes:
- Give the candidate access to their preferred AI coding tool. Use their own environment if possible — ecological validity is the point.
- Task: "Using the notification specification you wrote in Stage 1, implement the core logic: a function that takes an event, looks up notification preferences for the affected user, and dispatches the notification through the appropriate channel. You have 20 minutes. Use AI freely."
- Observe silently. Do not help, correct, or react. Take notes.
- After 20 minutes, conduct the debrief (P2-Q1 through P2-Q4).
- You are not grading the output. You are grading the process. A candidate who produces working code via bad process scores lower than a candidate who produces incomplete code via a rigorous process.
Observation checklist (fill during the session — not scored separately, informs Q scores):
| Observable | Strong signal | Weak signal |
|---|---|---|
| Output evaluation | Reads the code before running it; checks it against the specification; asks "is this correct for the domain?" | Runs it; if it runs, it ships |
| Accountability moment | Pauses before approving; asks "would I own this in production?" | No visible pause; immediate progression |
| Error handling | Looks specifically for absence of error handling; checks edge paths | Does not check error paths |
| Domain correctness | Checks whether the output satisfies the domain rules, not just the happy path | Evaluates only technical correctness |
| Iteration trigger | Identifies what was wrong in the specification that caused the wrong output; revises deliberately | Re-prompts randomly; pastes the error message back |
| Specification use | References their own specification when generating; includes domain constraints | Ignores the specification they wrote; starts fresh |
P2-Q1. "Walk me through what this function does. What would happen if the user's notification preferences were not found in the database?"
Construct: Output comprehension; mental model formation; edge case awareness during evaluation
Bloom level: Analyse / Evaluate
Research note: The ability to explain AI-generated code is the minimum competence bar for a Product Engineer. A candidate who cannot explain what they just built with AI cannot evaluate its correctness or catch its failures. This is directly observable: either they can walk through the execution path or they cannot.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Explains the function correctly, including the specific execution path for the edge case. Identifies the gap (missing null check, wrong default behaviour, silent failure) without being told there is one. If there is no gap, correctly identifies that and explains why. Connects the behaviour back to the specification they wrote. |
| 4 | Explains the function correctly. Can trace the edge case path. May not identify the gap without prompting, but immediately understands it when the edge case is named. |
| 3 | Explains the happy path correctly. Cannot trace the edge case path without significant help. Does not spontaneously check whether the edge case is handled. |
| 2 | Explanation is vague or partially wrong. Relies on "the AI handled it" without being able to verify. Cannot trace the execution path under the edge condition. |
| 1 | Cannot explain the function. "I'd have to run it to find out." |
Red flags:
- "The AI usually handles null cases" — defers to AI correctness as assumption
- Reads the code aloud without interpreting it ("it calls getUserPreferences, then...")
- Cannot connect the function's behaviour to the specification they wrote
P2-Q2. "Does this match your specification? Walk me through the differences."
Construct: Specification-to-implementation gap detection; output evaluation against an explicit model
Bloom level: Evaluate
Research note: This is the most direct test of AI output evaluation quality. A specification exists. An implementation exists. The candidate either can or cannot identify the differences. Candidates who cannot answer this question have no effective quality gate over AI-generated code — they are producing output they cannot evaluate.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Conducts a structured comparison between specification and implementation. Identifies specific gaps (missing failure path, wrong retry behaviour, extra scope that wasn't specified). Distinguishes between gaps that matter (behavioural) and gaps that don't (style, naming). Connects each gap back to a specific specification clause. |
| 4 | Identifies 2–3 substantive gaps. Comparison is structured. May miss one minor gap. |
| 3 | Identifies 1 gap or identifies gaps only when prompted ("does this handle the failure case?"). Comparison is not spontaneous. |
| 2 | "I think it matches." Cannot identify differences without being told where to look. Comparison is against intuition, not against the specification document. |
| 1 | Has not compared the output to the specification. Treats running output as passing output. |
Red flags:
- Does not refer back to their written specification during the comparison
- "It looks right to me" without specifying what "right" means
- Identifies only style differences, not behavioural differences
Probing follow-up: "Open your specification. Point to the clause that covers this error path. Is it in the code?"
P2-Q3. "When the output was wrong, how did you decide what to change?"
Construct: Diagnostic iteration vs. random re-prompting; specification failure identification
Bloom level: Analyse
Research note: Random re-prompting is the most common failure mode in AI-assisted development. It produces output churn without convergence. The diagnostic alternative — identifying which part of the specification or prompt caused the wrong output — produces convergent iteration. This question directly reveals which pattern the candidate uses. The observation checklist from the live session provides corroborating evidence.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Describes a diagnostic process: "The output did [X]. My specification says [Y]. The gap is in how I specified [Z]. I changed the prompt to include [specific constraint]." Has a theory about why the output was wrong before making the next attempt. Can reconstruct this reasoning from the live session just completed. |
| 4 | Mostly diagnostic. Can identify the cause in most cases. May have one instance of random re-prompting. |
| 3 | Mix of diagnostic and random. Cannot clearly distinguish between "I changed the specification" and "I added more words to the prompt." |
| 2 | Primarily random re-prompting ("I tried different wording until it worked"). No consistent theory of why changes produced different outputs. |
| 1 | "I just kept asking until it looked right." No diagnostic process. Cannot reconstruct why any specific change worked. |
Red flags:
- "I pasted the error message back in" as the primary iteration strategy
- Cannot explain why the final output is better than the earlier ones
- Iteration was driven by "it looked wrong" without specifying what "wrong" meant
P2-Q4. "What would you add to the test suite that the AI didn't generate?"
Construct: Test gap identification; failure path enumeration; translating behavioural concerns into test cases
Bloom level: Create
Research note: AI-generated test suites are systematically biased towards happy-path coverage. The code that is easiest to generate — the implementation — is also the easiest to test. The code that is hardest to get right — failure paths, edge cases, concurrency — is the code that is hardest to test and therefore least likely to appear in an AI-generated test suite. A Product Engineer who cannot identify this gap cannot establish a quality gate over their own output.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names 3+ specific test cases (not categories) covering failure paths and edge cases that the AI test suite does not include. Each test is specific enough to implement immediately: states the setup, the operation, and the expected outcome. At least one test targets a silent failure (wrong behaviour without an error). |
| 4 | Names 2–3 specific test cases. Tests are implementable. May miss one important category. |
| 3 | Names test categories ("we need tests for null preferences") without specifying the test setup and assertion. |
| 2 | "The test coverage looks pretty good." Cannot identify missing tests without being shown the gap explicitly. |
| 1 | "The AI usually writes comprehensive tests." Cannot propose a single specific missing test. |
Red flags:
- "I'd run it and see what breaks" — treats production as the test suite
- Proposes tests that mock the exact behaviour they should test
- Cannot specify what a failing test would look like before the fix is applied
Stage 3: Output Review and Edge Case Hunting
Assessment method: Code review exercise (AI-generated code with planted issues)
Skill map position: AI output review (X=+9, Y=+8)
Validity basis: B — directly mirrors daily job task; high ecological validity
Construct: Ability to identify behavioural, security, and logic errors in AI-generated code; evaluation against correctness, not style
Administration notes:
- Provide 50–80 lines of AI-generated code. Give 10 minutes to review independently. Then discuss for 30 minutes.
- Do not tell the candidate how many issues are present.
- Rotate planted issues per candidate. The list below offers more options than you need.
- Score on what they find and how they reason about it, not whether they can fix it.
Issue categories to plant (choose 4, vary each time):
| Code issue | Category | What a strong candidate sees |
|---|---|---|
| Free trial check that ignores trials started before the billing date | Business logic | "This comparison uses the billing date as the start of the trial window but doesn't account for trials that pre-date the current billing period — users on legacy trials will be incorrectly charged" |
| No handling for concurrent requests that could double-charge | Race condition | "Two simultaneous requests can both pass the balance check before either commits — there's no locking or idempotency key" |
| User-supplied field interpolated into query string without parameterization | SQL injection | "This is injectable — the user_id field goes directly into the query without sanitization" |
| Function assumes external API always returns 200; no handling for 4xx/5xx | Missing error state | "If the payment provider returns anything other than 200, this silently proceeds as if the transaction succeeded" |
| Currency amount stored as float | Precision error | "Floating-point currency arithmetic introduces rounding errors — this should use integer cents or a decimal type" |
| Retry loop with no backoff and no max attempts | Reliability | "Under load this will hammer the downstream service — there's no backoff and no ceiling on retries" |
| Event handler registered inside a loop | Memory leak | "This registers a new event handler on every loop iteration without cleanup — it leaks handlers and fires callbacks multiple times" |
P3-Q1. "Walk me through what you'd change before shipping this."
Construct: Code review judgement; behavioural correctness evaluation; security awareness
Bloom level: Evaluate
Research note: This question tests whether the review is driven by a mental model of correct behaviour or by surface pattern matching. Strong candidates evaluate against the expected contract and failure conditions; weak candidates evaluate against style conventions. The distinction is directly observable in what they choose to flag.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Finds all planted issues. Explains each in terms of the failure behaviour, not just "this is wrong." Identifies at least one issue as belonging to a class of problem ("this is a missing error-handling pattern — I'd check the rest of the codebase for the same pattern"). Does not spend significant time on style. May find additional unplanted issues. |
| 4 | Finds 3–4 planted issues. Behavioural reasoning is present. Proportionate attention to substantive vs. style issues. |
| 3 | Finds 2–3 issues. May include style issues. Reasoning requires prompting to go from "this looks wrong" to "this fails when...". |
| 2 | Finds 1–2 issues, mostly style or naming. Cannot explain behavioural impact without help. |
| 1 | "This looks fine." No substantive issues identified. |
Red flags:
- Opens with "I'd change the variable names" before looking for logic issues
- Cannot explain what breaks if they leave a found issue in place
- Misses the security issue entirely (SQL injection or equivalent)
Probing follow-up (if they miss an issue): "What happens when [specific edge condition]? Walk me through the execution path."
P3-Q2. "Are there scenarios where this code does the wrong thing but doesn't throw an error?"
Construct: Silent failure identification; behavioural correctness beyond exception handling
Bloom level: Analyse
Research note: Silent failures — wrong behaviour without an error signal — are the most dangerous output from AI-generated code. They pass tests, pass review, and only surface as business impact (incorrect charges, data corruption, missed notifications). This question specifically targets this failure class, which requires a behavioural mental model to detect.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Identifies at least one specific silent failure from the planted issues. Explains the specific execution path: what happens, why no error is raised, what the observable consequence is. Demonstrates that silent failures are a category they actively look for in review. |
| 4 | Identifies a silent failure. Can explain why it is silent (no exception, wrong success signal). May need light prompting to reach the specific mechanism. |
| 3 | "There might be edge cases." Cannot name one specifically without prompting. |
| 2 | "If it doesn't throw an error, it's correct." Conflates absence of error with correct behaviour. |
| 1 | Cannot engage with the concept of silent failure. |
Red flags:
- "The exception handling looks good, so it should be fine"
- Identifies only issues that would also produce errors
- Cannot construct a scenario where the code returns successfully but behaves incorrectly
P3-Q3. "What would a malicious user be able to do with this code?"
Construct: Security mindset; adversarial reasoning; input validation awareness
Bloom level: Evaluate
Research note: Security issues are the most consistent gap in AI-generated code. Bilkent University (2023) found 30.5% of AI-generated code contains vulnerabilities. A Product Engineer who cannot identify security issues in their own output is a reliable source of security debt. This question tests whether adversarial reasoning is part of their review process or absent from it.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Identifies the planted security issue. Explains the attack vector specifically: what the malicious input is, what it would cause, what data or system it could affect. Raises at least one additional security concern not planted (e.g., authentication boundary, authorization check, rate limiting absent). Does not need to be prompted to think adversarially. |
| 4 | Identifies the planted security issue. Explains the attack vector. Does not raise additional concerns spontaneously. |
| 3 | Identifies a security concern but describes it vaguely ("someone could inject something"). Cannot specify the attack or consequence. |
| 2 | "It looks secure to me." Does not identify the planted security issue. |
| 1 | Does not think in adversarial terms. "I wouldn't expect users to do that." |
Red flags:
- "Users won't send malicious input" — trusts input by default
- Identifies the issue as "bad practice" without explaining what an attacker would do
- "The framework handles security" — delegates security reasoning to tooling
P3-Q4. "What's missing from the test suite that should be there?"
Construct: Test coverage gap identification; failure path enumeration; quality gate design
Bloom level: Create
Research note: Same construct as P2-Q4, now applied to externally-written code rather than their own output. The question tests whether the candidate can reason about test gaps in a review context — a core skill for any engineer who reviews AI-generated code. Specific test cases are the output; categories are insufficient.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names 3+ specific missing tests with setup, operation, and expected outcome specified. At least one test covers a security scenario. At least one test covers a silent failure path. Prioritizes by risk, not by ease of writing. |
| 4 | Names 2–3 specific tests. Tests are implementable. Covers most high-risk gaps. |
| 3 | Names test categories without specifying the test. "We need tests for the error cases." |
| 2 | "The test coverage looks adequate." Cannot identify missing tests. |
| 1 | Cannot propose tests. Defers to "the existing tests cover it." |
Stage 4: Structured Behavioural Interview
Assessment method: Structured behavioural interview (STAR format with rubric)
Skill map position: Structured behavioural (X=+5, Y=+6) + Early adoption question (X=+9, Y=+7)
Validity basis: A — Sackett et al. (2022): r=.42; structure and consistent rubric are required
Construct: Past behavioural evidence for: specification quality, AI collaboration, output evaluation, early adoption, domain knowledge
Administration notes:
- Ask questions in the same order with all candidates, without alteration.
- Do not help candidates structure their answers. Probe only with the listed follow-ups.
- Score independently before comparing with your co-interviewer.
- Stage 4 validates the process observed in Stages 1–3. A candidate who performs well in the exercises but cannot recall a single real instance is a construct validity concern.
P4-Q1. "Tell me about a feature you built that behaved wrong in production even though tests passed. What did your specification miss?"
Construct: Real-world specification failure; retrospective specification reasoning
Bloom level: Analyse (retrospective)
Research note: Past-behaviour questions are harder to answer deceptively than hypothetical questions (Weiss & Feldman 2006). This question specifically asks for a specification failure in a production context — not a bug, not a design flaw, but something that traced back to an incomplete or ambiguous specification. Candidates who have never specified before cannot answer this question concretely.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names a specific feature and describes the production behaviour that was wrong (without being an outage — ideally a silent or business-logic failure). Traces the failure back to a specific gap in the specification: what was underspecified, what assumption was wrong, what edge case was not covered. Derives a generalizable lesson about specification practice. Does not blame the developer who implemented it. |
| 4 | Names a specific case. Traces back to a specification gap. Lesson is present but specific to the situation. |
| 3 | Describes a bug or production incident but attributes it to implementation rather than specification. May not be able to trace it back to specification with prompting. |
| 2 | Describes a production incident that was a system failure or external dependency, not a specification issue. |
| 1 | "My features have generally worked fine in production." Cannot recall a specification failure. |
Standard follow-ups:
- "What clause would you add to the specification now to prevent it?"
- "Did the test suite cover the case that failed? Why not?"
P4-Q2. "Describe a time you pushed back on a requirement because it was underspecified. What did you find and what happened?"
Construct: Proactive specification quality; upward communication; the ability to see underspecification before it becomes a bug
Bloom level: Apply / Analyse (retrospective)
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Describes a specific situation where they identified that a requirement could not be safely implemented without clarification. Names what specifically was ambiguous. Describes how they raised it. The outcome: the requirement was clarified, and what was clarified changed the implementation. Demonstrates that this is a habit, not a one-time event. |
| 4 | Describes a specific instance. The ambiguity was real. The pushback was constructive. The outcome was improved specification. |
| 3 | Describes asking for clarification on a vague requirement. The ambiguity was minor. The pushback was informal. |
| 2 | Describes a situation where they implemented a vague requirement and asked for clarification afterward. The specification was not fixed before work began. |
| 1 | "I usually just build what's asked and see what feedback I get." Cannot recall pushing back on a specification. |
Standard follow-ups:
- "What would have happened if you'd built it as originally specified?"
- "How do you decide when a requirement is underspecified enough to push back vs. making a reasonable assumption?"
P4-Q3. "Give me an example of a time AI gave you output that looked right but wasn't. How did you catch it?"
Construct: AI output evaluation track record; detection mechanism; gap between appearance and correctness
Bloom level: Analyse (retrospective)
Research note: This question has no equivalent in pre-2023 interview design. It targets a construct that is now central to the Product Engineer role: the ability to catch AI failures that are not surfaced by tests or errors. A candidate who cannot recall an instance where AI output looked correct but wasn't is either not reviewing AI output carefully, or has not used AI in contexts where correctness matters. Both are concerns for this role.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names a specific instance where AI output was functionally wrong despite looking correct. Explains specifically what made it look right (tests passed, ran without errors, matched the surface pattern). Explains specifically what caught it (manual trace, edge case review, domain knowledge, specification comparison). Derives a principle for catching this class of failure in the future. |
| 4 | Names a specific instance. The failure was real, not a typo or syntax error. The detection mechanism is described. |
| 3 | Names an instance but the failure was obvious (syntax error, immediate runtime exception). Does not describe catching a subtle behavioural failure. |
| 2 | "AI output has generally been good in my experience." Cannot recall a specific failure. |
| 1 | "I review everything before shipping so there haven't been issues." Cannot name a specific instance where AI was wrong. |
Standard follow-ups:
- "Would your test suite have caught it if you hadn't reviewed it manually?"
- "What do you check for now that you didn't before that incident?"
P4-Q4. "What do you use AI for in your daily work? What do you deliberately NOT use it for, and why?"
Construct: AI collaboration boundaries; judgement about dependency risk; self-awareness about where AI is and isn't reliable
Bloom level: Evaluate
Research note: This question establishes whether the candidate has a deliberate, principled boundary or an ad-hoc one. Strong Product Engineers know precisely where AI helps and where it introduces risk — and their boundary is based on the consequences of errors, not on comfort or convention. This is distinct from System Engineer S4-Q4, which focuses on delegation in the context of architectural ownership; here the focus is on daily operational judgement.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Describes AI use with specific examples across multiple task types. The "do NOT use" category is principled: things where AI errors are hard to detect, have compounding consequences, or require domain judgement that AI lacks. The line has been revised as AI capability has changed. Demonstrates active management of the boundary. |
| 4 | Clear use/no-use distinction with specific examples. Reasoning is mostly principled. The line may not be articulated as a principle but is consistent. |
| 3 | Describes AI use broadly ("I use it for most things"). The no-use category is vague ("sensitive stuff") without a principle. |
| 2 | "I use AI for everything" or "I only use AI for simple tasks." Extreme positions without nuance. |
| 1 | Has not thought about this. Use is determined by habit or team convention, not judgement. |
Standard follow-ups:
- "Has the line moved in the last six months? What moved it?"
- "Give me a specific example where you started to use AI for something, then decided not to. What made you stop?"
P4-Q5. "What have you started using in the last three months that nobody told you to? What's your current honest assessment of it?"
Construct: Early adoption behaviour; self-directed learning; opinion quality from use
Bloom level: Evaluate (ongoing)
Research note: This is the single highest-weight question in the Product Engineer bank. Early adoption is a behavioural pattern that cannot be faked convincingly because specific, recent, concrete examples of tool use are required. A candidate claiming to have used a tool they haven't used will fail on follow-up questions about specifics. The question is asked in present tense and asks for an honest assessment — not a sale. Positive-only responses suggest newsletter reading rather than actual use. See Early Adopter Framework for full scoring context.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names a specific tool or technique with a specific use case. The assessment is formed from use, not from reviews: includes what it does well, what it fails at, and what they would need to see before recommending it to their team. The tool may not yet be mainstream. Shows a systematic evaluation process. Has also tried at least one thing in the same period that they rejected — and can say why. |
| 4 | Names a specific tool with a genuine assessment from use. The assessment includes at least one concrete limitation ("it struggles with X"). May be a mainstream tool but evaluated independently. |
| 3 | Names a mainstream tool with a mostly positive but vague assessment ("it's been really helpful"). Limited specificity about failure modes or current edge cases. Evidence of use is present but shallow. |
| 2 | "I've been meaning to try [tool]." Not currently experimenting. Assessment is from reading, not use. |
| 1 | "I use what my team uses." No self-directed experimentation. Disqualifier for Product Engineer role. |
Standard follow-ups:
- "What would it have to improve for you to start recommending it to your team?"
- "What have you tried in the same period that you decided wasn't worth your time? What was wrong with it?"
- "Where did you first hear about it? How do you generally find out about new tools?"
Scoring Summary Sheet
| Stage | Q# | Question summary | Weight | Score (1–5) | Notes |
|---|---|---|---|---|---|
| S1 | Q1 | Clarifying questions before spec | 1.5× | ||
| S1 | Q2 | Write the specification | 2× | ||
| S1 | Q3 | What did you choose not to cover | 1× | ||
| S1 | Q4 | What gap would AI exploit | 1.5× | ||
| S2 | Q1 | Walk through the function / edge case | 1.5× | ||
| S2 | Q2 | Does it match the spec | 2× | ||
| S2 | Q3 | How did you decide what to change | 1.5× | ||
| S2 | Q4 | What tests did the AI miss | 1× | ||
| S3 | Q1 | Code review — find issues | 2× | ||
| S3 | Q2 | Silent failures | 1.5× | ||
| S3 | Q3 | Malicious user attack surface | 1× | ||
| S3 | Q4 | Missing tests | 1× | ||
| S4 | Q1 | Feature that behaved wrong in prod | 1× | ||
| S4 | Q2 | Pushed back on underspecified req | 1× | ||
| S4 | Q3 | AI output that looked right but wasn't | 1.5× | ||
| S4 | Q4 | AI use / no-use boundary | 1× | ||
| S4 | Q5 | Current experiments (early adoption) | 2× |
Weighted total: \_\_\_ / 100
| Score range | Recommendation |
|---|---|
| 85–100 | Strong Hire |
| 70–84 | Hire |
| 55–69 | Hire with conditions (name the specific gap) |
| 40–54 | No Hire — gap too large for role |
| Below 40 | No Hire |
Auto-reject conditions (override score):
- P4-Q5 scores 1 (no self-directed experimentation — disqualifier for this role)
- P2 stage average below 2.0 (cannot collaborate effectively with AI in real time)
- Two or more questions in any single stage score 1
Reference List
- Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262–274.
- Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(12), 2040–2068.
- Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity. Personnel Psychology, 58, 1009–1037.
- Wiggins, G., & McTighe, J. (1998). Understanding by Design. ASCD.
- Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge University Press.
- Anderson, L. W., & Krathwohl, D. R. (2001). A Taxonomy for Learning, Teaching, and Assessing. Addison Wesley Longman.
- Weiss, B., & Feldman, R. S. (2006). Looking good and lying to do it. Journal of Applied Social Psychology, 36(4), 1070–1086.
Source guides: Product Engineer interview Guide | Evaluation Rubrics | Early Adopter Framework
© Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution.