The Eight Skills Nobody Interviews For

A team ships an assistant over its own documents. Another runs an agent that reads a shared inbox and routes the messages. A third adds a summary box to a dashboard. In each case the engineers were interviewed on algorithms, system design and past projects, and not one question touched what actually decides whether the thing holds up in production.

That is not negligence. It is a gap in the model of what engineering is. Interviews test building software with the help of a model. These features are software that contains a model, which is a different job.

Eight competencies carry that job. They are architectural: every one of them is decided before the code exists, which is why no coding exercise reaches them. They are also externally anchored, mapping onto the OWASP Top 10 for LLM Applications and the NIST Generative AI Profile, so a hiring team can defend them to a CTO or an auditor without appealing to anyone's framework.

1. Non-Determinism Engineering

The ability to design systems whose core component does not return the same answer to the same input, and to build for that property rather than against it.

Treating variance as a system property to be measured, not a defect to be prompted away
Placing validation between the model and anything that consumes its output
Designing fallbacks for the case where the model is confidently wrong, not only for the case where it errors
Measuring consistency across repeated runs, not accuracy on a single run
Building an evaluation set that can be re-run on every change, the way a test suite is

What its absence sounds like: A candidate who responds to "it is inconsistent in production" with "improve the prompt and add tests." That answer treats the model as a deterministic function that is currently miscoded. The system reasoning starts from the opposite premise.

How to test it: Give a symptom ("excellent in dev, erratic in production") and ask what they change, before offering any detail about the prompt

Where it comes from: OWASP LLM09 Misinformation. NIST function: Measure, Manage.

2. Knowledge Governance

The ability to reason about the corpus a system retrieves from as a governed asset rather than a folder.

Freshness: how old is this document, and does the system know
Versioning and supersession: what happens when policy v3 and policy v7 both match the query
Provenance: can the answer be traced to a source a human can open and check
Access scope: whether retrieval respects the permissions of the person asking
Removal: what happens when a document must be withdrawn, including from anything derived from it

What its absence sounds like: "Put the documents in a vector database and retrieve the relevant ones." That is a correct description of the mechanism and says nothing about whether the answers can be trusted.

How to test it: Give a corpus with a realistic defect profile (a hundred thousand documents, a decade of edits, no owner) and ask what they do before building anything

Where it comes from: OWASP LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation. NIST function: Map.

3. Autonomy Boundary Design

The ability to decide, in advance and in writing, which actions a system may take on its own and which require a human.

Classifying actions by reversibility and by blast radius, not by technical difficulty
Gating anything with an external effect (a message sent, money moved, a record changed outside the system) behind an explicit condition
Treating a confidence threshold as a product decision with a cost on both sides, not as a tuning constant
Designing the handoff itself: what the human sees, how long they have, and what happens if nobody responds
Keeping an audit trail that can answer, later, why the system did this

What its absence sounds like: A candidate who has used agents but never run one unattended. They describe capability ("it can read the inbox and route the messages") and have no answer for when it should stop.

How to test it: Describe an agent that performs real actions and ask where it must stop and hand back

Where it comes from: OWASP LLM06 Excessive Agency. NIST function: Govern, Manage.

4. Adversarial Input and Output Handling

The ability to treat the model's input channel as an attack surface and its output as untrusted data.

Recognising that instructions and content arrive through the same channel, so any text the model reads can try to become an instruction
Taking indirect injection seriously: the hostile text arrives inside a retrieved document, a web page, an email, a PDF, a code comment, not from the user in front of you
Treating model output as untrusted input to whatever consumes it next, including shells, queries, browsers, templates and tool calls
Keeping authority outside the prompt: what a tool is allowed to do is enforced by the tool, not by an instruction asking the model to behave
Assuming the system prompt is readable and designing so that reading it costs nothing

How to test it: Describe an agent that summarises incoming email or retrieved documents, then ask what an attacker can do

Where it comes from: OWASP LLM01 Prompt Injection, LLM05 Improper Output Handling, LLM07 System Prompt Leakage. NIST function: Map, Manage.

5. Data Boundary and Privacy Engineering

The ability to reason about what data leaves the organisation, what is retained, and who can see what.

Knowing which provider receives which fields, what the contract says about retention and training, and where processing happens
Removing or tokenising the data that does not need to be sent at all
Tenant and permission isolation in shared indexes: an answer assembled from another customer's documents is a breach, not a defect
Logging discipline: prompts and completions often contain the most sensitive text in the system, and they end up in traces
Knowing which features plausibly fall under high-risk obligations, and raising that before launch rather than after

How to test it: Ask what fields are in the prompt, and which of them would be a problem in a vendor's logs

Where it comes from: OWASP LLM02 Sensitive Information Disclosure, LLM08 Vector and Embedding Weaknesses. NIST function: Map, Govern.

6. Evaluation and Observability Design

The ability to answer "did that change make it better" with evidence.

Building a labelled evaluation set from real failures, and growing it every time production surprises you
Knowing what a model-graded evaluation is good for and where it is biased, including its preference for longer and more confident answers
Running evaluations as a gate, the way a test suite gates a merge, with a known pass bar
Tracing a single request end to end: retrieval, prompt, tool calls, output, user reaction
Detecting silent quality regressions, which produce no exception and no alert, only a slow drift in whatever the users do next

How to test it: Ask what they would measure before and after a prompt change, and how many runs

Where it comes from: OWASP LLM09 Misinformation (detection side). NIST function: Measure.

7. Cost, Latency and Capacity Engineering

The ability to design model-backed systems that stay affordable and responsive under real traffic.

Knowing the per-request token budget, and what drives it (context stuffing, chat history, retrieved chunks, retries)
Caching at the right layer, and knowing what cannot be cached
Routing: small model for the easy majority, large model for the hard minority, and a way to tell them apart
Designing for perceived latency (streaming, partial results) as well as actual latency
Backpressure and caps: per-tenant budgets, rate limits, and a hard stop on agent loops that can retry themselves into a four-figure overnight bill

How to test it: Ask what one request costs today, and what the answer would be at ten times the traffic

Where it comes from: OWASP LLM10 Unbounded Consumption. NIST function: Manage.

8. Model Lifecycle and Supply Chain

The ability to treat the model as a dependency with a version, an owner and an end of life.

Pinning versions, and knowing what happens when the pin is retired
Expecting behaviour drift on provider updates, and having an evaluation set that detects it
Migration and failover: a second provider or a smaller local model, with the quality trade-off measured rather than assumed
Reproducibility: being able to say, months later, which model, prompt and retrieved context produced a given output
Provenance of things you did not build: models, fine-tuning data, embeddings, plugins and third-party agents, and what licence each one carries

How to test it: Ask what happens on the day the pinned model version is deprecated

Where it comes from: OWASP LLM03 Supply Chain, LLM04 Data and Model Poisoning. NIST function: Govern, Manage.

How to use these

Not all eight, and not for every role. Pick the three or four that match what your team actually ships. A team adding an assistant over internal documents is assessed mainly on knowledge governance, adversarial input, data boundary and evaluation. A team running unattended agents is assessed mainly on autonomy boundaries, adversarial input, evaluation and cost.

Where the organisation ships no model-backed features, all eight score zero and the eight core skills stand alone.

The questions that test them, with scoring rubrics, are in Module A of the System Engineer interview. The mapping to OWASP, NIST and the EU AI Act is in the standards mapping.