Why Event-Driven Systems Fail When We Treat Patterns Like Plumbing

The real problem is not moving messages

Most teams think the hard part of distributed systems is getting data from place to place. It is not. The hard part is deciding what kind of relationship you want between the things that exchange that data.

That sounds abstract until a system breaks in a familiar way. A checkout service waits on inventory, inventory waits on a fraud check, the fraud check waits on a third party, and suddenly a supposedly simple request has become a chain of invisible dependencies. Or a queue builds up, retries happen, duplicates appear, and now the question is not whether the message arrived, but whether the system can survive seeing it twice.

This is where architectural patterns are often misunderstood. They are treated like recipes for transport, as if the main question were whether to use a queue, an event bus, or an orchestrator. But the deeper question is more human and more consequential: how much coupling are you willing to accept in exchange for certainty, speed, and control?

That is the tension running through synchronous and asynchronous communication, fire and forget delivery, idempotency, event routing, and orchestration. These are not separate technical tricks. They are answers to one design question: what kind of coordination does your system need, and what kind of failure can it tolerate?

---

Patterns are not plumbing, they are contracts about time

Enterprise integration patterns are valuable precisely because they describe recurring problems. But patterns become truly useful only when we stop thinking about them as implementation shortcuts and start seeing them as contracts about time, responsibility, and uncertainty.

A synchronous call says, in effect, "I will wait for you, and my next move depends on your answer." That is a very strong contract. It is simple, direct, and often the right choice when the user needs an immediate result, such as validating a payment or checking authorization. Yet that simplicity comes at a cost: the caller and callee become temporally coupled. If the callee slows down, the caller slows down. If the callee fails, the caller fails in the same moment.

An asynchronous pattern makes a different promise: "I will hand off responsibility and continue." This is the logic behind fire and forget behavior. The sender does not need to block, which improves scalability and resilience. But this freedom changes the nature of correctness. Once the sender stops waiting, it can no longer rely on the next line of code as proof that the work happened. The system has moved from immediate certainty to eventual coordination.

That shift is not just technical. It is philosophical. Synchronous systems optimize for shared present tense, while asynchronous systems optimize for distributed patience.

The important design choice is not whether a message moves. It is whether the system can afford to stop caring about the exact moment the message is processed.

This is why many teams get stuck. They adopt event-driven architecture for scalability, but retain a synchronous mindset for correctness. They want all the benefits of decoupling without paying the cognitive cost of designing for uncertainty. The result is a brittle architecture that looks modern but behaves like a chain of hidden dependencies.

---

The hidden enemy is not failure, it is ambiguity

Once a system becomes asynchronous, the old assumptions about uniqueness and sequence stop being reliable. A message may be duplicated. It may arrive later than expected. It may be processed twice. A downstream service may be temporarily unavailable while the upstream service has already moved on.

This is why idempotency is not a niche implementation detail. It is the moral center of asynchronous design.

Idempotency means that processing the same event multiple times produces the same final result. In practical terms, it turns chaos into tolerable repetition. If an order is marked shipped twice, the system should not create two shipments. If a payment event is replayed, the customer should not be charged again. If a notification is sent twice, the user should not be punished for the architecture's uncertainty.

The deeper insight here is that idempotency is how systems buy the right to be asynchronous. Without it, every duplicate becomes a potential incident. With it, duplication becomes a manageable fact of life rather than a source of existential fear.

Think of a restaurant kitchen. A reliable ticketing system does not assume the waiter will never repeat an order. It assumes repetition is possible and builds procedures so the same ticket does not produce two meals. The kitchen does not demand perfect behavior from the messenger. It designs for the reality of imperfect delivery.

That is the mindset event-driven systems require. Ambiguity is not an exception to be eliminated at all costs. It is a condition to be designed against.

This also changes how we think about debugging. In synchronous systems, a failure is often visible where it happens. In asynchronous systems, the failure may appear elsewhere, or much later. The system needs traceability, correlation, and durable state transitions, because correctness is no longer a single moment in time. It is a sequence of defensible states.

---

Routing is coordination, not just delivery

Once you accept that messages can be late, duplicated, or rerouted, the next question is not merely where they go. The question is who decides where they go, and when that decision is made.

This is the power of event routing. A routing layer such as EventBridge separates producers from consumers so that one service can publish intent without needing to know every downstream reaction. That may sound like a simple convenience, but it is actually a profound architectural move. Routing converts direct dependency into policy.

A direct API call says, "I know exactly who should handle this." A routed event says, "Here is something that happened, now let the system decide which interested parties should care." The producer becomes less like a manager giving orders and more like a witness making a statement.

This distinction matters because many systems are not really integration problems. They are attention problems. As the number of consumers grows, the challenge is no longer sending the event. It is controlling the blast radius of that event, the selectivity of its audience, and the maintainability of the relationships it creates.

Routing becomes especially powerful when combined with patterns that are otherwise hard to implement cleanly. A single event can fan out to multiple targets. A transformation step can normalize data before it reaches downstream services. A pipe can connect sources and destinations without a bespoke glue layer for every pair.

That is where infrastructure starts to express architecture. A routing service does not merely move data. It encodes the organization of responsibility. Which service owns which reaction? Which events should be observed but not acted upon? Which workflow should branch, and which should remain linear?

Here is a useful mental model: routing is the nervous system of a distributed application, while orchestration is its executive function. Routing senses and distributes signals. Orchestration decides sequences, dependencies, and completion criteria. Confusing the two leads to systems that are either too passive to coordinate work or too centralized to scale gracefully.

---

Orchestration is valuable when uncertainty must be domesticated

There is a temptation in event-driven design to treat orchestration as a failure of purity. If events are supposed to be decoupled, why centralize control in a workflow engine at all? The answer is that some processes are too consequential to leave to emergent behavior.

Step Functions and similar orchestration tools exist for a reason. They give shape to long-running, multi-step processes where order matters, retries matter, compensation matters, and human visibility matters. A complex workflow is not just a chain of messages. It is a story with branching paths, checkpoints, and a clear notion of completion.

Consider an insurance claim. Documents arrive, validation occurs, fraud checks run, an adjuster may be involved, and payment may be issued or denied. This is not a loose swarm of independent reactions. It is a process with explicit states and business rules. Trying to model it purely as a set of loosely coupled events often creates hidden orchestration in application code, which is the worst of both worlds: centralized control without centralized visibility.

Orchestration makes the hidden visible. It tells you where the system is, what is waiting, what has failed, and what happens next. In that sense, orchestration is not the enemy of event-driven architecture. It is the tool that makes event-driven systems usable when outcomes must be coordinated deliberately.

The real decision, then, is not "events or workflows." It is where should the system absorb complexity, and where should it expose complexity?

If every service contains fragments of workflow logic, complexity is hidden and duplicated. If one orchestration layer owns all decisions, the system can become rigid and overcentralized. Mature architecture finds the boundary: local autonomy for simple reactions, explicit orchestration for consequential journeys.

---

A useful framework: three questions for every integration

When teams design distributed systems, they often start with tools. A better starting point is to ask three questions about the business behavior itself.

1. Must the caller know the result now?

If yes, synchronous communication may be the right default. Authentication, validation, and immediate user feedback often belong here. The cost of waiting is acceptable because the user experience depends on immediate certainty.

If no, asynchronous delivery opens the door to resilience and scale. Shipping notifications, audit trails, analytics events, and background processing usually do not need the caller to stall.

2. What happens if the event is repeated?

If repetition is dangerous, the downstream service must be idempotent or guarded by a deduplication strategy. This is not optional in asynchronous systems. It is the condition that makes retry and replay safe.

If repetition is harmless, the system can tolerate more elasticity. But even then, it is wise to design as if duplicates will happen, because eventually they will.

3. Is this a reaction or a journey?

If the task is a reaction, such as notifying a team or updating a read model, routing may be enough. If the task is a journey, such as onboarding a customer or processing a claim, orchestration provides the structure.

This question is often the most revealing. Many teams use heavy orchestration for simple reactions and fragile event chains for complex journeys. The architectural mistake is not a lack of sophistication. It is a mismatch between the shape of the business process and the shape of the software.

Good architecture does not merely move information efficiently. It aligns the level of coordination with the level of business consequence.

---

Key Takeaways

1. Treat patterns as relationship contracts, not transport tricks. Decide how much coupling, delay, and uncertainty your system can tolerate before choosing a pattern.

2. Assume duplication will happen. Build idempotency into every asynchronous workflow that touches money, state, or user-facing side effects.

3. Use routing to separate producers from consumers. Let events describe what happened, not prescribe every downstream reaction.

4. Use orchestration when the process has real business meaning. When order, retries, and compensation matter, make the workflow explicit instead of hiding it in scattered services.

5. Ask whether the problem is a reaction or a journey. This one distinction often reveals whether you need a queue, an event bus, a pipe, or a workflow engine.

---

The deepest shift: from code that does work to systems that negotiate uncertainty

The most important lesson in event-driven architecture is not about AWS services, queues, or workflow engines. It is about how software participates in reality.

Real systems are messy. Messages get duplicated. Dependencies fail. External services slow down. Users click twice. Partners send inconsistent data. In that environment, the goal is not perfect control. The goal is well-designed negotiation with uncertainty.

That is why patterns matter. They are not just reusable shapes. They are ways of deciding where certainty lives, where responsibility moves, and where the system should be resilient instead of reactive. Synchronous communication gives you immediacy, asynchronous communication gives you freedom, idempotency gives you safety, routing gives you decoupling, and orchestration gives you visibility. None of them is sufficient alone. The craft lies in composing them according to the real shape of the problem.

So the next time someone says they are "just wiring up an event," ask a more interesting question: what relationship are they creating between time, failure, and responsibility? The answer will tell you more about the architecture than the choice of service ever will.