System Engineer: Question Bank
The AI-Era Engineering Playbook — Practitioner Reference
Design Principles
This bank is organised by interview stage rather than by skill dimension. Each stage maps to a distinct assessment method type with its own validity profile. The ordering is intentional: stages run from least to most authentic, ending in the highest-validity method (structured behavioural with past-behaviour evidence).
Scientific basis for stage design
| Stage | Assessment Method | Validity (r) | Source | Bloom Level |
|---|---|---|---|---|
| 1 — Systems Case Study | Work sample (designed scenario) | .33–.54 | Roth et al. 2005; Schmidt & Hunter 1998 | Analyse / Evaluate |
| 2 — Failure Mode Review | Work sample (incident scenario) | .33–.54 | Roth et al. 2005 | Analyse / Evaluate |
| 3 — AI Output Audit | Work sample (live code review) | .33–.54 | Roth et al. 2005 | Evaluate / Create |
| 4 — Structured Behavioural | Structured behavioural interview | .42 | Sackett et al. 2022 | Apply / Analyse |
All four stages apply authentic assessment principles (Wiggins & McTighe 1998): the task mirrors the real job condition, not an abstracted proxy. Each question targets constructs at Bloom's analyse, evaluate, or create level — not recall.
Key validity references:
- Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262–274.
- Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(12), 2040–2068.
- Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity. Personnel Psychology, 58, 1009–1037.
- Wiggins, G., & McTighe, J. (1998). Understanding by Design. ASCD.
- Lave, J., & Wenger, E. (1991). Situated Learning. Cambridge University Press.
Scoring convention
All questions use a 1–5 scale. Score based on observed behaviour, not inferred ability. If not demonstrated, score low.
| Score | Label | Meaning |
|---|---|---|
| 5 | Exceptional | Exceeds expected depth; demonstrates mastery with specific, unprompted insight |
| 4 | Strong | Covers core fully; minor gaps; demonstrates clear competence |
| 3 | Adequate | Covers the basics; requires prompting to reach depth; no major gaps |
| 2 | Weak | Partial coverage; significant gaps; cannot reach depth even when prompted |
| 1 | Absent | Not demonstrated; cannot engage meaningfully with the question |
Auto-reject: Any dimension with two or more Q scores of 1.
Stage 1: Systems Thinking and Architecture
Assessment method: Case study discussion
Skill map position: System Design (restructured) — X=+6, Y=+6
Validity basis: B — high ecological validity; mirrors actual architectural decision-making
Construct: Ability to design systems with explicit constraints, tradeoffs, and failure awareness; ability to design for others
Administration notes:
- Provide scenario in writing. Give 10 minutes to read and think before discussion.
- Do not prompt architectural solutions — let the candidate drive.
- Score on reasoning process, not on whether their architecture matches any "correct" answer.
- The question is not solvable in 10 minutes. That is intentional. You are watching how they handle incompleteness.
Sample scenario (rotate per candidate):
A fintech startup wants to build a payment processing system. It needs to handle 1,000 transactions per second at peak, support multiple currencies, integrate with three external payment providers, and have 99.99% uptime. The team building it will include two engineers who are strong AI users but not deep systems people. You are the only System Engineer.
S1-Q1. "Before talking about implementation — what constraints would you define to make this system safe for the other engineers to build within?"
Construct: Constraint-first thinking; designing for others; safety boundary definition
Bloom level: Evaluate / Create
Chart method: Specification test (X=+9, Y=+8)
Research note: The ability to define constraints before implementation is the primary differentiator between senior and junior system design. The specification test has high face and ecological validity (Wiggins & McTighe 1998). Constraint identification precedes implementation in all high-reliability engineering contexts.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Immediately defines constraints as preconditions — not implementation decisions. Distinguishes between invariants (must not be violated) and goals (targets). Explicitly considers the non-expert builders: names constraints that limit the blast radius of errors they will predictably make. Names at least 3 non-obvious constraints. |
| 4 | Identifies constraints before implementation. Covers core categories (availability, data integrity, failure isolation). Considers the other engineers. May miss 1–2 non-obvious constraint categories. |
| 3 | Raises constraints when prompted or moves towards them after framing. Covers basic consistency and availability. Does not initially consider the other engineers as a design input. |
| 2 | Jumps to architecture (microservices, queues, databases) before defining constraints. Cannot enumerate constraints without direct prompting. |
| 1 | Treats the question as asking for an architecture proposal. Has no framework for constraint definition. Constraints are presented as "best practices" rather than system properties. |
Red flags:
- Immediately names a technology ("I'd use Kafka for this")
- Defines constraints as implementation details ("we'd use idempotency keys")
- Does not mention the non-expert builders at all
Probing follow-up: "What happens if one of the other engineers misunderstands a constraint? What does the system do?"
S1-Q2. "What are the hardest tradeoffs in this design? Where would you sacrifice to get something essential?"
Construct: Tradeoff reasoning; intellectual honesty about competing constraints
Bloom level: Evaluate
Chart method: Failure mode exercise (X=+8, Y=+7)
Research note: Tradeoff reasoning is a core higher-order thinking skill (Bloom's evaluate). Unlike recall or implementation, it cannot be replicated by AI. Candidates who cannot name tradeoffs have likely never made real architectural decisions under constraint.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names 2–3 genuine, non-obvious tradeoffs specific to this scenario. Explains what they would sacrifice and why. Demonstrates awareness that tradeoffs are not just technical but organisational (e.g., "strong consistency slows development for the other engineers"). Shows understanding that every architectural decision forecloses other decisions. |
| 4 | Identifies 2 real tradeoffs. Reasoning is clear. May focus primarily on technical tradeoffs (latency vs. consistency) without organisational dimension. |
| 3 | Identifies 1 tradeoff. Reasoning is present but shallow ("you trade off availability for consistency"). Does not engage with the specific context of this system. |
| 2 | States general principles without applying them. Tradeoffs are presented as solved by known patterns rather than as genuine choices under constraint. |
| 1 | Cannot identify tradeoffs. Describes an architecture as if it has no costs. |
Red flags:
- "You'd just use eventual consistency" without explaining what that means for payments
- Treats CAP theorem as a complete answer rather than a starting point
- No acknowledgment that different tradeoffs suit different constraint profiles
Probing follow-up: "If you chose consistency over availability and the payment provider is down — what does the user experience? Is that acceptable?"
S1-Q3. "What would the junior engineers break first? How would you prevent it architecturally rather than through code review?"
Construct: Anticipatory failure reasoning; designing for error surfaces; system trust model
Bloom level: Analyse / Create
Chart method: Failure mode exercise (X=+8, Y=+7)
Research note: This question directly tests the shift from "I build for myself" to "I build for others." A System Engineer who cannot anticipate how less experienced engineers will interact with their system is a single point of failure. The architecture must constrain error, not just detect it.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names 2–3 specific, predictable failure modes non-expert builders will cause. Proposes architectural solutions (not code review solutions) for each: e.g., interface contracts that make the wrong operation impossible, circuit breakers that are automatic rather than opt-in, event schemas that validate at the boundary. Distinguishes between "hard to do wrong" and "reviewable after the fact." |
| 4 | Identifies 2 predictable errors. Proposes at least one architectural solution rather than a process one. May default to "we'd have review gates" for the harder cases. |
| 3 | Identifies general risk areas ("they might not handle failures correctly"). Proposes code review or documentation as the primary mitigation. Architecture as a constraint mechanism is not the default. |
| 2 | Answers in terms of what they would tell the junior engineers, not what the architecture would enforce. |
| 1 | Cannot anticipate errors non-expert engineers would make. Assumes others will use the system correctly. |
Red flags:
- "I'd document everything clearly" as the primary answer
- "We'd review all their PRs" — this is a process patch, not an architectural solution
- No distinction between detecting errors and preventing them structurally
Probing follow-up: "If one of them deletes a retry logic because it looks redundant — what happens? Can the architecture stop that from mattering?"
S1-Q4. "What's the failure mode you're most worried about that would be invisible until production?"
Construct: Silent failure identification; prospective systems reasoning
Bloom level: Analyse / Evaluate
Chart method: Failure mode exercise (X=+8, Y=+7)
Research note: Silent failures — those that pass tests, pass review, and only surface under real load or edge conditions — are the highest-cost failure type in production systems. Candidates who enumerate only obvious failures (service down, exception thrown) have not operated systems at scale.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names a specific silent failure that is non-obvious and realistic for this scenario (e.g., partial duplicate transactions that don't error but don't reconcile; currency rounding that silently accumulates; idempotency keys that collide under specific load patterns). Explains why it would pass all tests. Proposes a detection strategy, not just a fix. |
| 4 | Names a realistic silent failure. Can explain why it is hard to detect. May not have a detection strategy beyond "we'd need monitoring." |
| 3 | Names a failure mode that is somewhat visible (slow queries, partial downtime). May describe an obvious failure dressed as a silent one. |
| 2 | Conflates "rare" with "invisible." Describes loud failures (exceptions, outages) as the most serious. |
| 1 | Cannot identify silent failures. All failure modes named are outage-level and observable. |
Red flags:
- Only names failures that would be caught by a test suite
- "We'd add logging" as the primary answer — logging only helps if you know what to log for
- Conflates failure probability with failure severity
Probing follow-up: "How would you design the observability layer to catch this before a customer reports it?"
S1-Q5. "How would you document this so the next engineer can understand why it was built this way — not just what it does?"
Construct: Knowledge transfer; architectural intent preservation; documentation as system design
Bloom level: Create
Chart method: Structured behavioural (X=+5, Y=+6)
Research note: Architectural decisions decay in undocumented systems. The why behind an architectural decision is the most valuable and most commonly absent information in engineering documentation. This question tests mental model transfer, not writing skill.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Distinguishes between what documentation (code comments, API docs) and why documentation (architecture decision records, constraint logs). Has a concrete format: records the decision, the alternatives considered, the constraints that ruled them out, and the conditions under which the decision should be revisited. Has done this before and can describe the outcome. |
| 4 | Understands the why/what distinction. Proposes ADRs or similar. Has not always done this consistently but understands why it matters. |
| 3 | Describes README-style documentation. Understands that why is important but does not have a structured approach for capturing it. |
| 2 | Documents implementation, not decisions. "I comment the tricky parts." |
| 1 | Does not distinguish between implementation documentation and decision documentation. |
Red flags:
- "The code should be self-documenting" — applies to implementation, not architectural decisions
- Cannot name an example of documentation they've written that explained a tradeoff
- "Future engineers should be smart enough to figure it out"
Probing follow-up: "Tell me about a system you've inherited where the documentation was good. What specifically made it useful?"
Stage 2: Failure Mode Reasoning
Assessment method: Incident scenario review
Skill map position: Debug approach (restructured) — X=+8, Y=+7
Validity basis: B — directly mirrors incident investigation; high ecological validity
Construct: Hypothesis formation; distributed system failure reasoning; the "fix the class, not the bug" distinction
Administration notes:
- Provide the system description and incident report in writing. Give 10 minutes to review.
- Do not reveal additional system details until asked — how they handle information gaps is part of the assessment.
- Score on reasoning process: hypothesis formation, diagnosis structure, and the breadth of failure categories enumerated.
Sample material (rotate per candidate):
System description:
A user authentication service. Users log in and receive a JWT token valid for 24 hours. The token is validated on each request. Tokens can be revoked by admins — stored in a Redis cache with a 24-hour TTL matching the token TTL.
Incident report:
A user's token was revoked due to suspected fraud. Six hours later, they were still making successful API calls. Redis confirmed the token was marked as revoked. The API logs showed successful authentication for all calls.
S2-Q1. "What's your hypothesis about what went wrong? Walk me through your top three candidates."
Construct: Hypothesis generation; distributed system mental model; diagnostic structure
Bloom level: Analyse
Chart method: Debug approach — hypothesis vs. paste (X=+8, Y=+7)
Research note: Hypothesis-driven debugging requires a mental model of the system. Prompt-driven debugging (asking AI "why is this broken") requires no mental model and produces no durable understanding. This question separates the two directly.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Generates 3 distinct, mechanistically plausible hypotheses without prompting. Immediately identifies token caching at the API layer as the primary candidate (not just the revocation mechanism). Distinguishes between "the revocation didn't propagate" and "the revocation propagated but was ignored." Names the specific system components that could cause each hypothesis. |
| 4 | Generates 2–3 hypotheses. Identifies the API-layer cache as a likely candidate. Reasoning is structured. May need light prompting to reach the third candidate. |
| 3 | Identifies 1–2 plausible hypotheses. May focus on the revocation side (Redis write failure) rather than the validation side (cache miss on read). |
| 2 | Generates generic hypotheses ("there's a bug in the token validation"). Does not use the specific system description to constrain the hypothesis space. |
| 1 | Cannot generate hypotheses. Reads incident as "the system is broken" without mechanistic reasoning. |
Red flags:
- Focuses only on Redis (the revocation path) and ignores the API (the validation path)
- Does not ask clarifying questions about the system before hypothesizing
- "There must be a race condition" without explaining the specific race
Probing follow-up: "The incident report says Redis showed the token as revoked and the API still authenticated. What does that tell you about where the failure is?"
S2-Q2. "How would you confirm or eliminate each hypothesis without access to the running system?"
Construct: Experimental reasoning; evidence-based diagnosis; system observability awareness
Bloom level: Analyse / Evaluate
Chart method: Debug approach (X=+8, Y=+7)
Research note: This question assesses whether the candidate can reason about evidence structure — what would confirm vs. disconfirm each hypothesis, and how to obtain that evidence. Engineers without distributed system experience tend to reach for "reproduce it" as the only strategy.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | For each hypothesis, names a specific observable that would confirm or eliminate it: API access log timestamps vs. Redis revocation timestamps; presence of a token validation cache vs. direct Redis lookup per request; TTL behaviour of any intermediary cache. Understands that absence of evidence is also evidence in some cases. |
| 4 | Proposes evidence-based tests for most hypotheses. May rely on log correlation without fully specifying the evidence structure. |
| 3 | Proposes "check the logs" without specifying what to look for or what pattern would confirm or eliminate the hypothesis. |
| 2 | Proposes reproduction steps as the primary diagnostic strategy. Does not engage with the evidence structure question. |
| 1 | Cannot propose a diagnostic strategy without access to the running system. |
Red flags:
- "I'd add more logging" without specifying what the logs should show
- Proposes a fix before completing the diagnosis
- Conflates "this is where I'd look" with "this is what I'd conclude"
Probing follow-up: "What's the simplest piece of evidence that would let you eliminate two hypotheses at once?"
S2-Q3. "Why wouldn't this have appeared in your test suite?"
Construct: Test suite limitations; distributed failure class awareness; the gap between test coverage and correctness
Bloom level: Analyse / Evaluate
Research note: This question tests whether the candidate understands the structural gap between unit/integration tests and distributed system behaviour. It is a direct application of situated cognition (Lave & Wenger 1991): behaviour in a test harness does not reliably predict behaviour in production under distributed timing.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Identifies that unit tests mock external calls — the Redis lookup is mocked and always returns "revoked" in the test. Integration tests likely test the happy path and a single-call revocation, not a cached revocation under timing. End-to-end tests run against a clean Redis with no pre-existing state. Names what type of test would catch this (a distributed timing test with real TTL behaviour and simulated propagation delay). |
| 4 | Identifies that tests don't test timing and propagation. Understands mocking as the mechanism of failure. May not identify the specific test type that would catch it. |
| 3 | Notes that "this is hard to test." Does not engage with the specific mechanism (mocking, TTL simulation, propagation timing). |
| 2 | "Tests wouldn't catch everything." Cannot explain why. |
| 1 | Believes a comprehensive test suite would catch this. |
Red flags:
- "We should add a test for this" without understanding why it wasn't there
- Conflates code coverage with behavioural coverage
- "The test suite should have caught this" — implying it's a test quality failure, not a structural limitation
Probing follow-up: "What would a test for this look like? What would it have to simulate that most test environments don't?"
S2-Q4. "What architectural change would prevent this class of problem — not just this specific bug?"
Construct: Class-level thinking; architectural remediation vs. patch thinking
Bloom level: Create
Chart method: Failure mode exercise (X=+8, Y=+7)
Research note: The distinction between "fix the bug" and "fix the class of bug" is the primary differentiator between senior and junior engineering thinking. A patch closes one hole; an architectural change closes the category. This is the highest-Bloom-level question in this stage.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Identifies the class of problem: "token validation decisions made from a cache that may not reflect current revocation state." Proposes an architectural solution that addresses the class: event-driven invalidation that pushes to all caches; or a pull-through pattern that makes staleness bounded and explicit; or a token design that encodes revocation state directly (short-lived tokens + refresh, eliminating the need for revocation caches). Explicitly names what problem each approach introduces and why one is preferable given the constraints. |
| 4 | Identifies the class: caches that can be inconsistent with authoritative state. Proposes a sound architectural fix (shorter TTLs, push invalidation). May not enumerate the tradeoffs introduced by the fix. |
| 3 | Proposes a targeted fix for this specific case ("check Redis more frequently"). Does not identify the class of problem. |
| 2 | Proposes a monitoring or alerting solution. The system itself does not change. |
| 1 | Proposes a process fix ("review all revocation flows"). No architectural content. |
Red flags:
- Proposes a "belt and suspenders" approach that adds checks without changing the architecture
- Does not distinguish between solutions that close one hole vs. solutions that seal the class
- Cannot explain what problem their architectural fix introduces
Probing follow-up: "If you moved to short-lived tokens with refresh — what new failure mode does that introduce? Is it better or worse than the original problem?"
Stage 3: AI Output Audit
Assessment method: Live code review exercise
Skill map position: AI output review (X=+9, Y=+8)
Validity basis: B — directly mirrors daily job task; high ecological validity
Construct: Ability to evaluate AI-generated code for behavioural correctness, security, and architectural fit — not style
Administration notes:
- Provide 60–100 lines of AI-generated code with exactly 3 planted issues. Do not tell the candidate how many issues are present.
- Give 15 minutes to review independently. Then discuss for 30 minutes.
- Score on what they find and how they reason about it, not on whether they fix it.
- Rotate issues per candidate to prevent question bank leak.
Issue categories to plant (choose 3, vary each time):
| Code issue | Category | What a strong candidate sees |
|---|---|---|
| Token validation that checks signature but not expiry independently | Security | "This trusts the embedded expiry claim without verifying it against the server state — the signature can be valid on an expired token if the library is misconfigured" |
| Amount calculation that silently truncates decimal in currency conversion | Silent logic error | "This loses sub-cent amounts on every conversion — individually small, but accumulates and creates reconciliation gaps at scale" |
| Retry implementation that uses a tight loop instead of the established exponential backoff pattern | Architectural mismatch | "This doesn't follow the retry contract the rest of the system expects — under load it floods the downstream service" |
| Null check absent on user preferences lookup | Edge case | "If the user has no preferences set, this silently proceeds with defaults that may not be safe for all event types" |
| User-supplied field interpolated into query string | SQL injection | "This is injectable — user_id goes directly into the query string" |
| Mutex that is locked but never released on error path | Resource leak | "If the external call fails, the lock never gets released — this will deadlock under concurrent load" |
S3-Q1. "Walk me through what you'd want to change before approving this."
Construct: Code review judgement; behavioural correctness evaluation; security awareness
Bloom level: Evaluate
Research note: This question tests whether the review is driven by a mental model of correct behaviour or by surface-level pattern matching. Strong candidates evaluate against specification and expected behaviour; weak candidates evaluate against style conventions.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Finds all 3 planted issues. Explains each in terms of behaviour under the failure condition, not just "this is wrong." Identifies at least one issue as belonging to a class (e.g., "this is an input validation gap — there will be other places in the codebase where this occurs"). May identify additional issues not planted. Does not spend significant time on style. |
| 4 | Finds 2–3 issues. Reasoning is behavioural ("what would happen if..."). May not identify the class. Proportionate time on substantive vs. style issues. |
| 3 | Finds 1–2 issues. May include style issues in the count. Reasoning is present but requires prompting to go beyond "this looks wrong." |
| 2 | Finds style or convention issues primarily. May find 1 substantive issue. Cannot explain the behaviour impact. |
| 1 | "This looks fine to me." No substantive issues identified. |
Red flags:
- Spends the majority of time on naming conventions, formatting, or import organisation
- Cannot explain why an identified issue is a problem — only that it "looks wrong"
- Approves code with a security issue without identifying it
Probing follow-up (if they miss an issue): "What happens when [specific edge condition from the planted issue]? Walk me through the execution path."
S3-Q2. "Is there anything here that would pass all tests but fail in production?"
Construct: Test/production gap awareness; silent failure identification; behavioural correctness reasoning
Bloom level: Analyse / Evaluate
Research note: This question directly activates situated cognition (Lave & Wenger 1991): the candidate must reason about behaviour in a production context, not a test context. It is the same construct as S2-Q3 applied to code rather than architecture.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Identifies at least one planted issue that would pass tests and names specifically why it passes (mocked dependencies, missing edge case in test fixtures, assertion that doesn't cover the failure path). Demonstrates understanding that test suites certify behaviour under tested conditions, not under all conditions. |
| 4 | Identifies an issue as likely to pass tests. Can explain why in general terms (testing happy path, mock hides the failure). |
| 3 | "There might be edge cases the tests don't cover." Cannot be specific about which ones or why. |
| 2 | "Tests should cover this." Does not engage with the structural gap. |
| 1 | Conflates test passage with production correctness. |
Red flags:
- "If the tests pass, it's probably fine"
- Cannot give a mechanism for why an issue would pass tests
- Identifies only issues that would also fail in tests
Probing follow-up: "If I run the full test suite and it passes — which of the issues you found would still be present?"
S3-Q3. "What questions would you ask the author before approving?"
Construct: Review as dialogue; specification recovery; understanding intent vs. implementation
Bloom level: Evaluate
Research note: This question tests whether the candidate understands code review as a bidirectional process for recovering intent, not a unilateral judgement. Strong reviewers use questions to resolve ambiguity; weak reviewers either approve or reject without engaging with intent.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Asks questions that distinguish between "the author made an error" and "the author made a design decision I don't have context for." Questions probe the specification, not just the code: "What behaviour is intended if the user preferences are null?" not "Why did you write it this way?" At least one question targets a planted issue and would reveal whether it was intentional or accidental. |
| 4 | Asks 2–3 substantive questions. Most are specification-oriented. Some may be phrased as implementation questions rather than intent questions. |
| 3 | Asks 1–2 questions. Questions are code-level ("why is this hardcoded?") rather than specification-level. |
| 2 | Would approve without questions, or would reject without questions. Review is a binary judgement, not a process. |
| 1 | Cannot articulate what they'd ask. |
Red flags:
- Questions are all about style or convention
- "I'd just fix it myself" — bypasses the intent recovery step
- Cannot distinguish between issues that need context and issues that are clearly wrong
S3-Q4. "If you approved this, what would you add to the test suite that isn't there?"
Construct: Test design; failure path enumeration; translating behavioural concerns into test cases
Bloom level: Create
Research note: This question closes the loop between finding an issue and knowing how to verify the fix. It is the create level of Bloom's taxonomy applied to testing — not recall of test patterns, but construction of test cases from behavioural requirements.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Proposes specific test cases (not categories) for each identified issue. Each test specifies the setup condition, the operation, and the expected behaviour. At least one test targets a failure path. Understands the difference between a unit test that mocks the failure and an integration test that simulates it. |
| 4 | Proposes test cases for most issues. Tests are specific enough to be implementable. May not address whether the test should be unit or integration. |
| 3 | Proposes test categories ("we need tests for the null case") without specifying the test. |
| 2 | "Add more tests" without specifics. |
| 1 | Cannot propose test cases from a behavioural issue. |
Red flags:
- "The test coverage is already pretty good" after finding issues
- Proposes tests that would mock the exact behaviour that caused the issue
- Cannot specify what a failing test would look like before the fix
Stage 4: Structured Behavioural Interview
Assessment method: Structured behavioural interview (STAR format with scoring rubric)
Skill map position: Structured behavioural interview (X=+5, Y=+6)
Validity basis: A — Sackett et al. (2022): r=.42 for structured format; the structure and consistent rubric are essential to reaching this validity level
Construct: Past behaviour as predictor of future behaviour; track record of real-world application of the constructs tested in Stages 1–3
Administration notes:
- Ask all questions in the same order, to all candidates, without alteration.
- Do not help the candidate structure their answer. If they give an incomplete story, probe with the standard follow-ups.
- Score independently from your co-interviewer before comparing notes.
- The purpose of this stage is to verify that the observable skills in Stages 1–3 are not one-day performances — they should be a pattern across multiple past situations.
Why behavioural evidence matters here: Schmidt & Hunter (1998) note that the combination of cognitive/work-sample assessment with structured behavioural interview outperforms either method alone. Stage 4 provides the past-behaviour evidence that validates Stages 1–3. A candidate who performs well on a case study but cannot recall a single real instance of that reasoning raises a construct validity concern.
S4-Q1. "Tell me about the most architecturally complex system you've designed. What constraints were non-negotiable and why?"
Construct: Real-world systems thinking; constraint reasoning under actual production conditions
Bloom level: Analyse (retrospective)
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Describes a genuinely complex system with specific, named non-negotiable constraints. The constraints are principled (not "the PM said so") — they reflect understanding of why those properties matter for that system. Can explain what would have broken if a constraint had been relaxed. Shows retrospective judgement: what they'd do differently. |
| 4 | Describes a complex system. Names constraints. The reasoning for why they're non-negotiable is present but may not distinguish between principled and imposed constraints. |
| 3 | Describes a moderately complex system. Constraints are general ("it had to be reliable, it had to scale"). Cannot name a specific constraint and defend why it was non-negotiable over another. |
| 2 | Describes a system they maintained rather than designed. Constraints are implicit or absent. |
| 1 | Cannot name a system they designed or cannot identify constraints within it. |
Standard follow-ups:
- "What would have happened if you'd relaxed [specific constraint]?"
- "What architectural decision did you make that you wouldn't make again? What did you learn?"
S4-Q2. "Tell me about an architectural decision you made that turned out to be wrong. What did you miss and why?"
Construct: Retrospective judgement; intellectual honesty; learning from failure
Bloom level: Evaluate (retrospective)
Research note: The willingness and ability to critique one's own past decisions is a strong signal of both intellectual honesty and experience. A candidate who cannot recall an architectural error has either not made meaningful architectural decisions, or does not reflect on their work, or is being dishonest. All three are concerning.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names a specific decision, the reasoning at the time, the specific assumption that was wrong, and what they'd do differently. Does not externalize blame ("requirements changed") — takes ownership of the failure in the reasoning. Derives a generalizable lesson from the specific case. |
| 4 | Names a specific decision and what went wrong. Reasoning at the time is partially described. Some externalization is present but not dominant. |
| 3 | Describes a mistake but frames it as bad luck or changing requirements rather than flawed reasoning. The lesson derived is not generalizable. |
| 2 | Names a minor technical error (wrong library, wrong approach to a small problem) rather than an architectural decision. |
| 1 | "I can't think of one" or only describes others' decisions. |
Standard follow-ups:
- "What was the assumption that turned out to be wrong? When did you know it was wrong?"
- "What would you tell yourself at the moment you made that decision?"
S4-Q3. "Tell me about a time you had to prevent someone from building something the wrong way — without just taking over."
Construct: Knowledge transfer; teaching under constraint; influence without authority
Bloom level: Apply / Create (retrospective)
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Describes a specific situation where they used questions, constraints, or reframing to redirect someone rather than overriding them. The outcome demonstrates that the other person understood the why, not just the what. Reflects on what made this approach work. Shows awareness that taking over is a trap: it resolves the immediate problem but creates a dependency. |
| 4 | Describes a real situation with a clear outcome. The approach was more collaborative than directive. The other person arrived at the right solution. |
| 3 | Describes a situation but the outcome relied primarily on the candidate's authority or direct correction. The teaching element is thin. |
| 2 | Describes reviewing code and leaving comments. No interpersonal dynamic. |
| 1 | Cannot recall a situation or describes taking over as the solution. |
Standard follow-ups:
- "What would you have done if they hadn't responded to your approach?"
- "How did you know it was working?"
S4-Q4. "Where do you draw the line in your own work between what you delegate to AI and what you own directly? Why there?"
Construct: AI integration judgement; self-awareness about dependency risk; system engineer's relationship to AI tooling
Bloom level: Evaluate
Research note: This question has no equivalent in pre-2023 interview design. It targets a new construct: the engineer's ability to reason about their own AI dependency profile. There is no peer-reviewed validity data for this specific question. It is placed in Stage 4 (behavioural) rather than Stage 1 (case study) because past behaviour is more reliable than hypothetical framing.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Draws a principled line based on consequences: delegates things where errors are visible and reversible, retains ownership of things where errors are invisible or compounding. Gives specific examples of both. Has revised this line as AI capability has changed. Shows awareness that the line is not static. |
| 4 | Draws a recognizable line with reasoning. Examples are specific. The line may not be fully articulated as a principle but is consistent in practice. |
| 3 | Draws a line based on comfort or habit ("I just feel more confident when I write certain things myself") without a principle. |
| 2 | "I use AI for everything / I use AI for nothing" — either extreme without nuance. |
| 1 | Has not thought about this. Delegates or retains based on workflow convention, not judgement. |
Standard follow-ups:
- "Has the line moved in the last year? What moved it?"
- "What would have to be true for you to delegate something you currently own directly?"
S4-Q5. "What tools are you currently experimenting with that you haven't brought to a team yet? What's your current assessment?"
Construct: Early adoption behaviour; self-directed learning; frontier awareness
Bloom level: Evaluate (ongoing)
Research note: Early adoption is a behavioural pattern, not a trait. The question is asked in present tense and asks for an ongoing experiment — not a past adoption. A candidate who cannot answer this question is not currently experimenting. The question is resistant to fabrication: claiming a current experiment requires specificity about what the tool does, where it falls short, and what the candidate has tried.
Scoring rubric:
| Score | Behavioural description |
|---|---|
| 5 | Names a specific tool or technique, describes what they are using it for, gives a concrete current assessment including where it falls short. The assessment is formed from use, not from reviews or announcements. May be evaluating something that has not yet reached mainstream awareness. Shows a systematic approach to evaluation (tries, forms opinion, decides to adopt or reject with reasoning). |
| 4 | Names a specific tool with a specific assessment. The assessment is from use. May be a tool that is somewhat mainstream. The evaluation is genuine. |
| 3 | Names a tool that is well-known and mainstream. Assessment is mostly "it's pretty good." Limited specificity about current use or current edge cases. |
| 2 | "I've been meaning to try X." Not currently experimenting. |
| 1 | "I use what my team uses." No self-directed experimentation. Disqualifying signal for System Engineer role. |
Standard follow-ups:
- "What would it have to do for you to bring it to the team?"
- "What have you tried that you actively decided not to adopt? Why?"
Scoring Summary Sheet
| Stage | Q# | Question summary | Weight | Score (1–5) | Notes |
|---|---|---|---|---|---|
| S1 | Q1 | Constraint definition | 1× | ||
| S1 | Q2 | Tradeoff reasoning | 1× | ||
| S1 | Q3 | Junior engineer failure paths | 1.5× | ||
| S1 | Q4 | Silent failure identification | 1.5× | ||
| S1 | Q5 | Why documentation | 0.5× | ||
| S2 | Q1 | Hypothesis generation | 1.5× | ||
| S2 | Q2 | Evidence-based diagnosis | 1× | ||
| S2 | Q3 | Test suite gap awareness | 1× | ||
| S2 | Q4 | Class-level architectural fix | 1.5× | ||
| S3 | Q1 | Code review — find issues | 2× | ||
| S3 | Q2 | Test/production gap | 1× | ||
| S3 | Q3 | Questions before approving | 1× | ||
| S3 | Q4 | Test case design | 1× | ||
| S4 | Q1 | Most complex system designed | 1× | ||
| S4 | Q2 | Wrong architectural decision | 1.5× | ||
| S4 | Q3 | Prevented without taking over | 1× | ||
| S4 | Q4 | AI delegation line | 1× | ||
| S4 | Q5 | Current experiments | 1.5× |
Weighted total: Use the interactive scorecard to calculate your weighted total in real time.
| Score range | Recommendation |
|---|---|
| 85–100 | Strong Hire |
| 70–84 | Hire |
| 55–69 | Hire with conditions (name the specific gap) |
| 40–54 | No Hire — gap too large for role |
| Below 40 | No Hire |
Auto-reject conditions (override score):
- S4-Q5 scores 1 (no self-directed experimentation)
- S3-Q1 scores 1 (approves code with security issues undetected)
- Two or more questions in any single stage score 1
Reference List
- Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262–274.
- Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(12), 2040–2068.
- Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity. Personnel Psychology, 58, 1009–1037.
- Wiggins, G., & McTighe, J. (1998). Understanding by Design. ASCD.
- Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge University Press.
- Anderson, L. W., & Krathwohl, D. R. (2001). A Taxonomy for Learning, Teaching, and Assessing. Addison Wesley Longman.
- Rivera, L. A. (2012). Hiring as cultural matching. American Sociological Review, 77(6), 999–1022.
- Weiss, B., & Feldman, R. S. (2006). Looking good and lying to do it. Journal of Applied Social Psychology, 36(4), 1070–1086.
Source guides: System Engineer Interview Guide | Evaluation Rubrics
© Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution.