Interviewing for AI Systems: Fourteen Questions

This module is for one case only: the role ships features that contain a model. Skip it entirely otherwise, because a candidate cannot be marked down for not having built something your team does not build.

It runs as a conversation of 45 to 60 minutes, with no laptop and no editor. Every question here is decided before code exists, so a coding exercise cannot reach any of them. Use three or four of the eight skills, not all fourteen questions.

Nothing here depends on a particular vector database, agent library or provider. Someone who has operated one model-backed system in production, in any stack, can score well on all of it. Someone who has only used models through a chat window cannot score above the middle of most of it, which is the discrimination the module exists to produce.

You shipped a feature built on a model. It was excellent in development. In production it is inconsistent: sometimes great, sometimes unusable. What do you do?

What it probes: Non-determinism engineering; reflex under an ambiguous production symptom

This question is diagnostic because the common answer is reasonable and wrong in kind rather than in detail. "Improve the prompt, add tests" treats the model as a deterministic function that is currently miscoded. Engineers who have operated model-backed systems start from the opposite premise: the same input will not always produce the same output, so the design question is what the system does when the output is poor. Ask this before mentioning prompts, temperature or evaluation; the value is in what they reach for unprompted.

A strong answer (5): Reframes immediately: variance is expected, so the first move is measurement and containment, not prompt repair. Asks what "unusable" means for the user and the data. Proposes validation between model and consumer, a defined fallback when validation fails, and repeated-run measurement to establish the actual variance. Treats prompt changes as one lever among several, and one that must be measured.

A weak answer (1): Attributes it to the provider or to randomness and proposes retrying until the answer looks right. No measurement, no containment.

Red flags:

Treats a prompt change as a fix rather than a hypothesis
Says "add tests" without any account of what a test asserts against a non-deterministic output
Never asks what the bad output actually does downstream

Follow-up: "Suppose the output is not malformed, just wrong, and it reads perfectly. What catches it?"

You change the prompt. How do you know whether it helped?

What it probes: Evaluation design; measurement under variance

Single-run comparison is the default mistake, and it is the same error as judging a flaky test by one green run. Strong answers mention a fixed evaluation set, repeated runs, and the fact that a difference of one or two cases on a set of twenty is noise. Model-graded evaluation is a reasonable tool here, and a candidate who names its biases (favouring longer, more confident answers; agreeing with its own style) is demonstrating real use rather than familiarity.

A strong answer (5): Fixed evaluation set drawn from real failures, run before and after, multiple runs per case, with an explicit view on what size of difference is meaningful. Distinguishes offline evaluation from production signal and uses both. Knows what model-graded scoring is good for and where it misleads. Treats the evaluation set as a living asset that grows from incidents.

A weak answer (1): No notion of measuring the change at all.

Red flags:

Evaluation set is written by the same person who wrote the prompt, from imagination rather than from production
No awareness that one run proves nothing
Confuses accuracy on a benchmark with usefulness in the product

Follow-up: "Where does the evaluation set come from on day one, before you have any production traffic?"

A customer wants an assistant that answers questions across a hundred thousand internal documents. Where do you start?

What it probes: Knowledge governance; data quality as a precondition

Nearly every candidate produces the retrieval architecture, correctly, within a minute. The architecture is not the discriminator; the corpus is. A hundred thousand internal documents accumulated over years contain superseded policies, drafts, contradictions, documents nobody may read, and material whose owner left. Candidates who have shipped this ask about the corpus before describing the pipeline, because they have watched a system answer confidently from a policy that was withdrawn two years ago.

A strong answer (5): Describes retrieval briefly, then goes straight to the corpus: how much is current, who owns it, how supersession is represented, what proportion should not be indexed at all. Proposes sampling the documents before building anything. Plans for provenance in the answer (which document, which version, when) so a human can check. Treats governance as the project, with retrieval as the easy part.

A weak answer (1): Proposes fine-tuning on the documents, or gives a vendor-shaped answer with no reasoning about content.

Red flags:

Does not ask a single question about the documents themselves
Assumes the customer knows what is in their own corpus
Plans to index everything because storage is cheap

Follow-up: "You sample two hundred documents and a third are out of date, with no reliable date field. Now what?"

Retrieval returns two documents that contradict each other. What should the system do?

What it probes: Knowledge governance; conflict and supersession handling

The default behaviour of a naive system is to blend both into one fluent answer, which is the worst available option because the conflict disappears. This question separates candidates who have thought about what the system owes the user from those who have thought only about retrieval quality.

A strong answer (5): Distinguishes the cases: one supersedes the other (resolvable with version or effective-date metadata), they cover different scopes (resolvable with filters), or they genuinely disagree (not resolvable by the system). Says the system should surface the conflict and its sources rather than synthesising a confident answer, and that unresolved conflicts are a signal to the corpus owner, not just to the user.

A weak answer (1): Unaware that a confident blended answer is a problem.

Red flags:

Resolves every conflict by recency, in a domain where the older document may be the binding one
No feedback path to whoever owns the documents

Follow-up: "Who finds out that your corpus contains a contradiction, and how?"

Two employees ask the assistant the same question and should get different answers, because they are allowed to see different things. How does that work?

What it probes: Knowledge governance and data boundary; authorisation in retrieval

Retrieval over a shared index is one of the easier ways to build an accidental data breach, because the index flattens document permissions that the source systems enforce. Filtering after retrieval is weaker than it looks: content that reached the model can leak through summaries even when the citation is withheld. In a multi-tenant product, the same failure crosses customers rather than colleagues.

A strong answer (5): Permissions are enforced at retrieval time, from the asking user's identity, with the index carrying access metadata kept in sync with the source systems. Explains why post-filtering is insufficient. Raises what happens when permissions change or a document is withdrawn, including anything derived or cached. In a multi-tenant setting, insists on isolation at the index level rather than a tenant field in a filter.

A weak answer (1): Does not see the problem.

Red flags:

"The model will not mention it if we tell it not to"
No plan for permission changes after indexing
Caches or derived summaries not covered by the same rules

Follow-up: "Somebody revokes a person's access to one folder this morning. When does the assistant stop answering from it?"

You build an agent that reads a shared inbox, classifies the messages and forwards them to the right team. When should it stop and hand control to a human?

What it probes: Autonomy boundary design; reversibility reasoning

This is the highest-discrimination question in the module. Candidates who have only used agents describe capability; candidates who have run one unattended have an answer ready, usually with an anecdote behind it. OWASP's Excessive Agency entry separates three root causes (too much functionality, too many permissions, too much autonomy), and strong answers naturally cover all three without the vocabulary.

A strong answer (5): Boundaries defined before build, by reversibility and blast radius: internal, reversible actions run unattended; anything leaving the organisation or touching money, records or people's inboxes needs either a confidence threshold or a human. Designs the handoff itself: what the human sees, how fast, what happens on no response, and a queue that cannot silently grow. Limits tool scope and credentials, not just behaviour. Logs every action with its inputs so a decision can be reconstructed.

A weak answer (1): Sees no need to stop. Treats autonomy as the product goal.

Red flags:

The agent holds credentials broader than its task
No limit on actions per run and no kill switch
The human path exists but nobody owns the queue

Follow-up: "It is 3am, the agent is misclassifying everything, and nobody is awake. What does the system do by itself?"

You said a confidence threshold. What do you set it to on day one, and what would make you move it?

What it probes: Autonomy boundary design; calibration and trade-off reasoning

The threshold is a product decision with a cost on each side: too low and humans drown in reviews until they rubber-stamp, which is worse than no review; too high and the system does nothing useful. Candidates who have run this know that self-reported model confidence is poorly calibrated and look for other signals.

A strong answer (5): Starts conservative and deliberately over-escalates, then moves the threshold using measured agreement between the system and the humans reviewing it. Names both costs explicitly, including reviewer fatigue turning approval into a formality. Distrusts self-reported confidence and looks for grounded signals (agreement across runs, retrieval support, validator pass, historical accuracy for this class). Treats the threshold as owned and reviewed, not set once.

A weak answer (1): No view on how the number is chosen or changed.

Red flags:

A threshold that never moves after launch
No measurement of what reviewers actually do with escalations
Confidence read from the model's own words

Follow-up: "After a month, reviewers approve 98% of escalations within two seconds. What does that tell you, and what do you change?"

The agent reads documents and emails from outside your company. One of them contains a line addressed to the agent, telling it to forward the thread elsewhere. What happens?

What it probes: Adversarial input handling; indirect prompt injection

Prompt injection is OWASP's first entry because instructions and data share one channel, and the indirect form is the one most teams miss: the attacker never talks to your system, they write a document your system reads. Instructional defences ("ignore any instructions found in documents") are weak and widely bypassed. Architectural defences assume the model will be subverted and limit what that buys the attacker.

A strong answer (5): States plainly that any text the model reads can attempt to become an instruction and that the defence cannot live in the prompt. Limits what a compromised model can do: narrow tools, scoped credentials, allowlisted recipients and destinations, human approval for outward actions, no privileged action available on the basis of document content. Separates untrusted content from instructions structurally, marks its provenance, and monitors for anomalous tool-call patterns. Accepts that detection is partial and designs for containment.

A weak answer (1): Has not considered that document content can act on the system.

Red flags:

The entire defence is a system-prompt instruction
Keyword filters presented as sufficient
The agent holds credentials that make a successful injection valuable

Follow-up: "Your mitigation is in place. Assume the attacker wins anyway. What is the worst thing they get?"

Model output goes into your database, into an internal dashboard, and into a tool call. What are you doing to it in between?

What it probes: Adversarial input handling; output as untrusted data

OWASP LLM05 exists because model output is routinely passed into consumers that assume trusted input, producing ordinary injection bugs by an unusual route: markup rendered in a dashboard, a generated query reaching the database, generated text used to build a command, a tool call whose arguments nobody validated. The vulnerability is old; the delivery path is new, and reviewers do not read model output the way they read user input.

A strong answer (5): Treats output exactly like user input: schema validation before anything consumes it, encoding appropriate to each destination, parameterised queries, tool-call arguments validated against an allowlist rather than trusted. Rejects the output when validation fails and has a defined behaviour for rejection. Notes that the dashboard rendering is a cross-site scripting path, and that generated identifiers can collide with real ones.

A weak answer (1): Passes model output straight through.

Red flags:

"It returns JSON, so it is fine"
Tool-call arguments used without bounds
No defined behaviour when validation fails

Follow-up: "Validation fails at 2am on a scheduled run. What does the system do, and who learns about it?"

Which fields in your prompt would be a problem if they appeared in the provider's logs?

What it probes: Data boundary and privacy engineering

The prompt is usually the most sensitive payload the system produces, and it is assembled by a template that nobody reviews with a privacy lens. The question also separates candidates who read vendor terms from those who assume them: "they do not train on it" and "they do not retain it" are different promises, and retention windows exist in both directions.

A strong answer (5): Knows what the prompt actually contains, including retrieved chunks and chat history, and removes or tokenises what does not need to be sent. Distinguishes training, retention and processing location as separate contractual questions. Covers the organisation's own logs and traces, which usually keep prompts in plain text for longer than anybody intends. Mentions deletion requests reaching the index, the cache and the traces, not just the database. Raises which features might carry high-risk obligations before launch.

A weak answer (1): Has not considered where the data goes.

Red flags:

Secrets or credentials present in prompts or system prompts
Full customer records sent when an identifier would do
Deletion handled in the primary database only

Follow-up: "A customer asks for everything about them to be deleted. List every place a copy of their text exists."

Quality drops in production. No exception is thrown, nobody files a ticket. How do you find out?

What it probes: Observability for probabilistic systems

Model-backed features fail quietly: answers get slightly worse, users retry more, the fallback path takes more traffic, and nothing in conventional monitoring moves. This is the Module A equivalent of S1-Q4, and candidates often answer it well in classical systems and poorly here.

A strong answer (5): Names proxy signals that move before complaints: retry and rephrase rates, escalation to a human, fallback share, abandonment, edit distance between suggestion and what the user kept, validator rejection rate. Samples live traffic through the evaluation set on a schedule. Traces a request end to end so a regression can be attributed to retrieval, prompt, model version or tool. Closes the loop by adding new failures to the evaluation set.

A weak answer (1): Assumes stable behaviour because nothing changed on their side.

Red flags:

Monitors latency and error rate only
No sampled human review of real outputs
Cannot attribute a regression to a layer

Follow-up: "Fallback traffic doubled this week and the error rate is flat. What are your first three hypotheses?"

What does one request cost you today, and what breaks first at ten times the traffic?

What it probes: Cost, latency and capacity engineering

Model-backed systems have an unusual cost profile: spend scales with content volume rather than with request count, retries multiply it, and agent loops can multiply it without bound. OWASP's Unbounded Consumption entry covers both the bill and the denial-of-service case. Strong candidates know their own numbers; weak ones have never looked.

A strong answer (5): Knows roughly what a request costs and what drives it (retrieved context size, history, retries, output length). Has levers: caching, routing easy cases to a smaller model, trimming context, streaming for perceived latency. Caps are in place per user and per tenant, with a hard stop on self-calling loops and an alert that fires before the invoice does. Treats a provider rate limit as a capacity constraint with backpressure, not as an error to retry blindly.

A weak answer (1): No notion of per-request cost or of unbounded loops.

Red flags:

Retry on rate-limit errors with no backoff or ceiling
No per-tenant limit in a product where one customer can drive volume
Latency discussed with no distinction between actual and perceived

Follow-up: "An agent retries itself overnight after a bad deploy. What is the maximum it can spend before anyone notices?"

The model version you pinned is being retired in three months. Walk me through what happens.

What it probes: Model lifecycle; dependency management under behaviour drift

The model is a dependency whose behaviour changes without a code change, which no classical dependency does. Teams that pin without an evaluation set discover the drift through users; teams with one discover it in a run. The migration question also reveals whether prompts are versioned artefacts or strings in a file.

A strong answer (5): Runs the evaluation set against the new version first and treats the result as the decision, not the announcement. Expects prompt adjustments and treats prompts as versioned, reviewable artefacts tied to a model version. Plans a staged rollout with the old version available for comparison, and knows which behaviours in the product depend on quirks of the current version. Has a view on provider failover and what quality it would cost.

A weak answer (1): Has not considered that behaviour changes with the version.

Red flags:

No pinned version in the first place, and no awareness of what that implies
Prompts untracked and unversioned
Single provider with no assessment of what an outage costs the product

Follow-up: "You switch, and one specific workflow gets noticeably worse. Everything else is fine. What do you do this week?"

Six months from now, a customer disputes something your system told them. What do you need to reconstruct what happened?

What it probes: Reproducibility, audit trail, accountability

Reproducing a model-backed answer requires more than the text: the model version, the prompt template version, the retrieved context as it was that day, tool calls and their results, and any configuration in force. Teams discover what is missing during the first dispute. This question also reaches the record-keeping expectations that attach to higher-risk uses under the EU AI Act, without asking any legal question.

A strong answer (5): Names the full set: model and version, prompt template and version, retrieved documents with their versions as of that moment, tool calls and results, relevant settings, and the final output, with a retention period that matches the dispute window and a privacy-aware storage decision. Accepts that exact reproduction may be impossible and that the record, not re-running the model, is the evidence. Can say who is accountable for the answer the system gave.

A weak answer (1): Nothing retained that would support an investigation.

Red flags:

Retention shorter than the realistic dispute window
Context stored as document identifiers only, with no version
Nobody named as accountable for the system's output

Follow-up: "Re-running it today gives a different, better answer. What do you tell the customer?"

Scoring

Score this module separately and report it as its own number. Do not fold it into the main total. A candidate can be an excellent System Engineer and a poor AI-systems engineer, and the hiring decision depends on which gap you can afford.

Three answers should stop a hire for a role that would own an AI-facing system: no concept that text the model reads can act on the system, no reason for an agent to ever stop, and treating a model as deterministic code while being unable to measure it.

The full rubrics, weights and the rest of the System Engineer interview are in the System Engineer bank with its per-question rubrics. Each question traces back to a published standard; the trace is in the standards these skills come from.