The gap nobody talks about
Working and reliable are not the same thing.
A system that works is one where the main use case produces the expected output. You can demonstrate it. It passes a test. The demo goes well. Working is a threshold that most systems cross early, sometimes in the first week.
Software projects that succeed
The Standish Group CHAOS Report, tracking software project outcomes since 1994, consistently finds that only around a third of projects are delivered on time, on budget, and with the required features. Most post-launch failures are not technical. They are operational: missing exception handling, unclear ownership, and systems that cannot be maintained without the original builder.
Source: Standish Group CHAOS Report (2020)
The first decision
Build or buy: the question that shapes everything that follows.
The decision between building a custom system and configuring an existing tool is one of the most consequential choices in operational systems work, and it is usually made too quickly. Teams default to tools because tools feel lower-risk. The risk with tools is not the tool itself but the assumption that a general-purpose tool will fit the specific operational logic of a particular workflow without major rework.
The decision framework
Build when the logic is specific. Buy when the problem is generic.
A tool solves a well-defined, broadly applicable problem well. If your problem fits that definition exactly, buy the tool. If your problem has specific exceptions, unusual logic, or custom data requirements that the tool does not natively support, the configuration work approaches the complexity of building.
Custom systems are justified when the operational logic is unique, when the data model requires precision that off-the-shelf tools cannot provide, or when long-term maintainability matters more than speed to launch.
The cheapest solution is not always the tool with the lowest subscription cost. A tool that requires four workarounds to fit the actual workflow costs more in ongoing maintenance than a simpler custom solution that matches the workflow exactly.
The right question is not "what tool could we use for this?" but "what are the actual rules, triggers, exceptions and ownership requirements of this workflow, and which approach - tool, custom, or hybrid - handles all of them cleanly?"
Build or buy is really build or script
A script solves the task once. A system survives the second person, the exception, and the process change.
The build-or-buy choice usually hides a third option: a quick script that works today and quietly becomes the thing nobody can safely change later.
Why the thing that works once is not yet the thing a team can run.
Script vs System: What Is the Difference? ->Skip a phase, pay for it later
Building a system that lasts requires four phases done in order.
The most common reason operational systems fail is that phases are skipped or reversed. The build starts before the workflow is understood. Testing covers only the happy path. The handover is rushed. Each shortcut taken early creates compounding problems later.
Phase 1
Understand the real workflow, not the documented one
Every organisation has a formal version of how a workflow runs and an actual version of how people do it. The formal version is what was designed. The actual version is what adapted to the constraints of real tools, real team habits, and real exceptions that were never formally handled.
A system built against the formal version will break against the actual version. The first phase is understanding the actual version: what people actually do, which shortcuts are load-bearing, which steps are skipped when things are busy, which exceptions are handled informally because the formal process does not cover them. This cannot be learned from documentation. It requires watching the workflow run and asking the people who operate it.
Phase 2
Separate what should be automated from what needs a person
Not all work that runs through an operational system should be automated. Some work requires context, judgement, or accountability that cannot be reliably encoded in rules. Automating this work does not remove the judgement - it removes the visibility of where the judgement is happening, which makes failures harder to find and fix.
The design question for every step in a workflow is: does this step require human judgement, or does it require reliable execution of a rule? Steps that require reliable execution of a clear rule are candidates for automation. Steps that require reading context, making a judgement call, or accepting accountability for a decision should remain human - with the system supporting the decision rather than making it.
Phase 3
Build and test the failure paths first
Happy path testing confirms that the system works when everything goes as expected. It does not confirm that the system is reliable. Production runs on the happy path some of the time. It runs on everything else the rest of the time: records with missing fields, inputs that arrive out of order, steps completed by a new team member who does not know the convention, values that are technically valid but operationally wrong.
Testing the failure paths before launch - what happens when a required field is missing, when a record arrives twice, when the upstream system delivers data in an unexpected format - is what separates systems that work in demos from systems that work in production. Most failures that occur in the first month of production were visible during testing if anyone had thought to look for them.
Phase 4
Hand over something operable, not just something built
A successful handover means the people who will operate the system can do so without the person who built it. Not without ever asking a question - but without needing the builder present to handle normal operations, diagnose common failures, or make routine adjustments.
The test for a good handover is simple: could someone who was not involved in building this system diagnose the most likely failure and resolve it using only the documentation and the system itself? If the answer is no, the documentation is incomplete, the error messages are not descriptive enough, or the failure paths are not handled clearly enough to be diagnosed without expert knowledge.
What breaks after launch
The failures that arrive six months after a system goes live are predictable. These are the most common.
Most post-launch system failures are not caused by technical bugs. They are caused by the gap between the conditions the system was designed for and the conditions it actually encounters in sustained production use.
Post-launch failure 1
The exception that was handled informally becomes a recurring problem
When the system was launched, there was one unusual case that was handled manually because it did not fit the standard flow. Everyone knew about it. It was not a big deal at launch. Eighteen months later, there are twelve variants of that exception, none of them formally handled, and the manual workarounds consume more time than the system saves.
Exceptions that are handled informally at launch have a way of multiplying. Each informal exception sets a precedent for the next one. The fix is designing an explicit exception path from the beginning: not a path that handles every possible exception, but a path that makes exceptions visible and routes them to a human decision rather than disappearing into informal practice.
Post-launch failure 2
The system becomes impossible to change without risk
A year after launch, the business process the system supports has changed. The changes are modest. But making them in the system requires understanding the full logic of how it was originally built, because the components are entangled enough that changing one thing risks breaking another. Nobody wants to touch it.
Systems that accumulate this kind of fragility usually do so because the original design did not separate concerns clearly. The configuration was mixed with the logic. The exception handling was woven through the main flow rather than handled at defined boundary points. Untangling it now is expensive. The lesson for the next build is to treat separation of concerns as a maintainability requirement, not an architectural preference.
Post-launch failure 3
The institutional knowledge is lost when the owner changes
The person who understood how the system worked - not just what it was supposed to do but why specific decisions were made, which edge cases existed, which parts were fragile - has moved on. The person who inherited it has the documentation but not the understanding. The documentation covers the happy path but not the operational nuances.
This failure is almost entirely prevented by better handover documentation. Not longer documentation - more targeted documentation. The goal is not to explain everything the system does. It is to explain the things that are non-obvious: why the logic works this way instead of the simpler way, which cases the system handles poorly and why, what the most likely failure is and how to diagnose it.
Post-launch failure 4
The system silently stops working
An upstream data source changes its format. A scheduled job stops running because a credential expired. A dependency is updated in a way that breaks compatibility. The system stops producing correct output. Nobody notices for weeks because the failure is silent.
Silent failures are the most expensive post-launch failure mode because the cost compounds with time. Every day the system runs incorrectly, more data is produced in the wrong state and more decisions are made from incorrect information. The fix for silent failures is monitoring that runs continuously and alerts when the system stops doing what it should - not when it throws an error, but when its output diverges from expectations.
The second standard
The demo was the easy part
Working vs reliable
Working means the demo went well. Reliable means someone other than the builder can diagnose it when something goes wrong.
A system that is reliable is one that produces the expected output consistently, over time, under conditions that were not anticipated when it was built. Reliable means the failure paths are handled. It means someone other than the original builder can diagnose a problem when one occurs. It means new team members can operate it without a personal walkthrough. It means the system's behaviour is documented well enough that a decision about changing it can be made without the person who built it in the room.
The builder's responsibility
The work is finished when failures are visible, ownership is clear, and someone other than the builder can run it.
This is the standard that separates a technical prototype from an operational system. A prototype demonstrates the happy path. A system handles the full range of conditions the workflow will actually encounter, with enough visibility into what is happening that failures can be found and fixed by the people who operate it, not only by the person who built it.
What good handover looks like
Handover is part of the build. These are the four things it must include.
Good handover is not a document. It is a state of the system in which the people responsible for operating it have what they need to do so reliably, without depending on the original builder being available.
01
The logic is documented in plain language
Every rule, trigger, exception path and decision point in the system should be describable in plain language by someone who has read the documentation. Not pseudocode - language. If the logic cannot be described in plain language, it is either too complex for the people who will maintain it or not well-understood enough to have been built reliably.
02
Failures are visible and diagnosable
When something goes wrong, the system should make it visible. Not just that something went wrong - what went wrong, on which record, at which step, and with enough context to understand why. A failure that produces a cryptic error or fails silently is a failure that will recur because it cannot be diagnosed without the original builder.
03
Ownership is explicit
For every part of the system - every data source it reads from, every action it takes, every exception queue it produces - there is a named owner. Not a team. A person. The owner is the one who notices when their part of the system is behaving unexpectedly and takes responsibility for investigating. Without named ownership, problems get noticed by everyone and fixed by no one.
04
Normal changes can be made without the original builder
Business requirements change. The trigger conditions that made sense at launch may need adjustment six months later. The message template may need updating. The exception rules may need expanding. These changes should be makeable by the person responsible for the system without requiring the original builder. If they cannot be, the system has a hidden dependency that will cause problems the first time the builder is unavailable.
Keeping it maintainable
Systems that remain reliable over time are designed for maintainability from the start.
Maintainability is not a property that can be added after the fact. It comes from design decisions made during the build: how the logic is structured, how the configuration is separated from the code, how the error handling is built, how the documentation is written.
The most maintainable systems are not the most technically elaborate ones. They are the ones that a new person can understand, operate, and modify without needing to understand the full history of why every decision was made. Simplicity that serves the operator is a higher goal than cleverness that serves the builder.
This means the best operational systems are often not the ones that required the most intricate engineering to build. They are the ones built with enough restraint that the people who will operate them - and eventually modify or replace them - can understand what they are doing and why. This requires resisting a natural instinct during the build: the instinct to solve the problem in the most technically interesting way available. The technically interesting solution is often harder to maintain, harder to explain, and harder to modify when the requirements change. The solution that is obvious to someone reading the code for the first time - even if it is not the tidiest - is usually the one that will serve the team best over three years of changing requirements and different owners.
Practical maintainability also means thinking carefully about where complexity lives. Complexity that is in the business logic is necessary - it reflects the real complexity of the operational problem. Complexity that is in the implementation is a cost with no corresponding benefit. The goal is a system that handles complex operational logic in the simplest way that correctly expresses that logic, and nothing more. Every additional layer of abstraction, every clever pattern, every structural decision that requires explanation is a small tax paid by everyone who has to understand the system later.
Design for the next person
Five maintainability properties worth designing in from the start
Configuration is separated from logic: changing a threshold, a recipient list, or a template does not require modifying the code.
Error messages describe the problem in terms the operator understands, not in terms of the internal implementation.
The system does one thing per component: components that try to do too many things become impossible to change without risk of breaking something unrelated.
Dependencies are documented: every external service, API, or data source the system reads from or writes to is listed, with a note on what happens if it is unavailable.
There is a test for the most likely failure: not exhaustive test coverage, but at least a way to confirm the system is still working correctly that does not require running the full production workflow.
Common questions
Questions about operational systems
A working system produces correct output when conditions match what the demo covered. A reliable system produces correct output under the range of conditions the workflow actually encounters, including edge cases, missing data, exceptions, and situations the original builder did not anticipate. The gap between them is where most operational systems live, and it is invisible until something goes wrong in production.
Four things: the logic documented in plain language, failures that are visible and diagnosable without the original builder, named ownership for every part of the system, and the ability to make normal operational changes without calling the person who built it. If any of those four are missing, the system has a hidden dependency that will surface at the worst possible time.
Ask three questions: can someone other than the original builder diagnose a failure from what the system logs? Are exceptions routed to a visible queue or do they disappear? Can a normal operational change, updating a threshold, changing a recipient, adjusting a rule, be made without touching the underlying code? If the answer to any of these is no, the system was built to the demo standard, not the production standard.
Fix when the core logic is sound and the problems are in the operational layer, error handling, monitoring, documentation, exception paths. Rebuild when the core logic no longer reflects how the business actually works, when the system has accumulated so many workarounds that understanding it requires more effort than starting clean, or when the data model it was built on is wrong in ways that cannot be corrected from the outside.
If your operational system works in demos but breaks under real conditions, it was not built for production.
Bring the workflow, the current state of the system, and the failure that prompted the conversation. A good second pass starts with understanding what the system was designed for versus what it actually encounters - and identifying which gap between those two things is causing the most cost. That gap is usually findable in the first conversation.
Bring the page, report, or workflow as it is now.
We reply with the clearest next step, or an honest no.
