The AI-Era Engineering Playbook

The AI-Era Engineer's Playbook — Part 11

Find the Silent Failure

Find the Silent Failure

A drill this time. Set a timer, read the function, and note the moment you find the bug. The time matters as much as the answer, so be honest about it.

The prompt that produced this was reasonable: "add a timeout and a sensible default so a slow preferences service doesn't block the page load."

def get_notification_preferences(user_id, timeout_ms=800):
    try:
        prefs = preferences_service.fetch(user_id, timeout=timeout_ms)
        return prefs
    except Exception:
        return DEFAULT_PREFERENCES  # all notifications on

Six lines. No style issues worth the name. A linter passes it, a reviewer skims it, and a test that stubs a slow service proves it works.

Stop here if you are doing the drill.


What the function actually promises

It promises two things, and only one of them is true.

The true one: the page will not hang. The timeout is set, the exception is caught, the caller always gets an object back.

The false one, which nobody wrote down and everybody assumes: that when this function returns DEFAULT_PREFERENCES, the service was slow or down.

except Exception does not mean "the service was slow". It means every failure, including ones that have nothing to do with the service being slow:

  • The service answered promptly, and the response did not parse.
  • The service answered, and the record used an older shape than the code expects, so an attribute access raised.
  • The credential used to call the service expired, and the error came back as an exception rather than a status.
  • preferences_service was imported but not configured in this environment.
  • A typo somewhere inside the try block raised a NameError or a TypeError, in the code you wrote, not in the service at all.

Every one of those returns DEFAULT_PREFERENCES. And the comment on that line tells you exactly why this is the dangerous kind of bug: all notifications on.

So the function's real behaviour is this. If anything at all goes wrong, including a mistake in your own code, re-enable every notification for that user, including the ones they explicitly turned off, and tell nobody.

A user who opted out of marketing email receives marketing email again. There is no error, no alert, no log line, and nothing in the metrics moves, because from the system's point of view nothing failed. The fallback worked. That is the whole problem.


The pattern, not the example

The bug is not "they used a broad except". Broad handlers are sometimes right. The bug is a mismatch between two things that are written in different places and never compared:

How wide the handler is, and how neutral the default is.

A neutral default is one that is merely inconvenient when it fires for the wrong reason. Show a spinner, retry, degrade to a simpler view, return an empty list that the caller treats as "unknown". Pair that with a broad handler and the worst case is an annoyed user.

A non-neutral default makes a decision on the user's behalf. All notifications on. Access allowed. Discount valid. Balance sufficient. Feature enabled. Those are correct in exactly one failure mode, the one the author pictured, and actively wrong in most of the others. Pair that with a broad handler and you have built a machine that converts any unexpected error into a silent policy change.

This is old ground, which is part of why it is worth knowing. In 1975, Saltzer and Schroeder listed fail-safe defaults among their design principles for secure systems in The Protection of Information in Computer Systems: base decisions on permission rather than exclusion, so that the default is denial and a mistake fails towards safety. Half a century later both halves of this function have catalogue numbers of their own. The default is CWE-636, Not Failing Securely. The handler is CWE-396, Declaration of Catch for Generic Exception. Neither entry is new, neither is obscure, and the agent produced both in six lines from a prompt nobody would have objected to in review.

Part 6 of this series, the prompt that reveals everything, had a deliberate version of the same decision: a rate limiter that fails open when Redis dies, because a limiter that takes down the API when its own dependency fails is worse than no limiter. That is defensible, and notice what made it defensible. Somebody chose it, wrote down why, and logged it loudly when it happened. The difference between that and this function is not the behaviour. It is whether anybody decided.


Why this one needs a drill of its own

Part 10 was about a race: two requests inside the same guard. You find that by asking what runs twice.

This one is invisible to that question. The function is single-threaded, has no shared state, and would behave identically under any amount of concurrency. You find it by asking a different question: what can this handler actually catch, and is the default right for all of it?

That is two categories. There are more. Nobody holds them all in their head at once, which is the argument for a written list rather than a careful read. The careful read is what already failed; you are reading this because the code looked fine.

So here is where the drill time matters. If you found it in the first thirty seconds, you have met this shape before and it is in your list. If you found it after a minute, you were reading rather than checking. If you did not find it, nothing is wrong with you, because six lines of clean Python with a sensible comment is a genuinely hard place to see a policy change.


What this means for you

Three things, in order of how much they cost.

Narrow the handler or split the default. Catch the timeout specifically and return the permissive default for that. Let everything else raise, or return a value that means "unknown" so the caller can decide. The parse failure and the expired credential are not the same event as a slow service and should not share an outcome.

Make the fallback visible. Log which exception fired and count it. The OWASP Error Handling Cheat Sheet states the split this function gets wrong: a generic response goes to the user, the error details go to the server log for investigation. This function has the generic response and no log. A fallback path that nobody can see firing is indistinguishable from a working system, right up until a support agent spots a pattern in complaints.

Add it to your list. Broad handler plus non-neutral default is now a shape you can look for in ten seconds, in any language, for the rest of your career. The list is the deliverable here, not this one bug.

And write the failure case into the request next time. Writing a spec an agent can execute names failure definition as one of the five things a prompt has to carry. "On a timeout, return the stored preferences or raise; never substitute defaults" is one sentence, and it is the sentence missing from the prompt that produced this.

The next article turns the list into a habit: asking "how does this fail?" of anything before it ships, rather than only of the exception handlers.


Sources

© Gabor Mayer. Licensed under Creative Commons Attribution 4.0 (CC BY 4.0). Free to share and adapt with attribution.