What Happens to Messages
This page states what the platform does with a message from the moment it arrives until delivery, and what happens when any step fails. Read it before you decide whether a flow needs a retry strategy, a dead-letter handler, or a different source connector.
The lifecycle
Every message passes through the same phases in the same order. At each step, either the message continues or the platform resolves it (acknowledged or discarded) and stops.
- Route matching: The adapter evaluates each route's conditions in order.
- If a route's conditions match, the message runs through that route's stages. Later routes never see it.
- If a route's conditions do not match, the message moves on to the next route.
- If no route matches, the message is acknowledged and discarded.
- Transforms: The matched route's transforms run in order. Each one reshapes the message.
- Transform fails: the message is acknowledged and discarded silently (the same outcome as no route matching).
- Processors: The matched route's processors run in order. They can enrich the message, record it, or trigger a side effect.
- Processor fails: behaves like a delivery failure (see below).
- Send via a Send Processor: A Send Processor is the pipeline step that actually forwards a message to the route's Target (see Pipeline Designer). It is the component that delivers the message downstream.
- No Send Processor on the route: the message is acknowledged and nothing is sent downstream. This is an intentional shape for audit or enrichment routes.
- Send succeeds: the message is acknowledged back to the source.
- Send fails: the outcome depends on the source type. See Delivery failure.
Unmatched messages are silently discarded
A message that no route claims is acknowledged and dropped. The platform does not log this by default, does not keep a copy, and does not emit a warning.
This is the correct behavior for a condition-based routing system. A route that does not want a message simply does not claim it, and the message falls through to the next route. A logged entry per unclaimed message would defeat the purpose on a high-volume feed. If you need to know when messages go unclaimed, add an explicit catch-all route at the end (a final route with no conditions, so it claims every message that earlier routes did not).
If messages you expected downstream are missing, verify two things in the Pipeline Designer before looking upstream.
- Do any route's conditions match the message? If not, add or relax a condition.
- Does the matched route include a Send Processor? If not, add one. A route without a Send Processor acknowledges the message but sends nothing.
Delivery failure
The guarantee the platform gives when a Send Processor cannot deliver depends on whether the source connector exposes a replayable position. Know which type your source is before you reason about the failure.
How to tell which type your source is
You do not have to guess. Each connector's plugin documentation states its delivery type: offset-tracked (it commits a replayable position for each message) or session-based (it delivers messages inline on a live connection with no replayable position). Look up the connector you configured under Official Plugins and read its delivery guarantee before you design a retry or dead-letter strategy around it.
Sources with offset tracking
Offset-tracked sources (topic consumers that commit a position per message) give the strongest guarantee: a failed message is not acknowledged, its offset is not committed, and the next poll redelivers it.
Guarantees:
- No loss from the source to the pipeline, provided the source connector tracks offsets and the platform has not been asked to commit a failed message.
- At-least-once delivery downstream, provided the destination is idempotent (processing the same message twice has the same effect as processing it once) or the downstream system deduplicates on a reference identifier the pipeline preserves.
- Automatic recovery from transient failures (network blip, target temporarily unavailable). The message is retried on the next poll with no operator action.
Trade-off:
- Head-of-line blocking on persistent failures. Head-of-line blocking means one stuck item holds up everything behind it. If the same message fails every attempt, the source makes no forward progress and new messages queue behind it.
Operator action when a source stalls on one message:
- Open the event log and look for repeated error-level entries on the same message.
- Decide whether the message is recoverable (a transient downstream issue) or poison (a message that will never succeed no matter how many times it is retried).
- For poison messages, either fix the downstream so the message can be accepted, or build a custom error handler via the Plugin SDK that isolates and skips them.
Session-based sources
Session-based sources deliver messages inline on a live session and do not expose a per-message replayable position. If the Send Processor cannot deliver a message from a session-based source, the platform has no way to redeliver it.
Guarantee:
- A message that fails to deliver is dropped. There is no retry and no preservation.
Implication:
- A session-based source cannot offer automatic redelivery on a downstream failure. If that matters for your flow, either route the session's traffic to an offset-tracked target first (and let a second adapter deliver from that offset-tracked feed), or build durability into the target so you can replay from a system you control.
Failover during processing
The platform runs only one copy of an adapter against a session at a time. When that copy has to hand the session over (for example, because the server running it lost connectivity), the in-flight message is never counted as delivered. The standby takes over and continues from a known-safe point.
Guarantees:
- No double delivery across a failover, provided the source is offset-tracked. The standby re-reads the message from the source and delivers it normally.
- No redelivery across a failover for session-based sources. The in-flight message is dropped; the standby resumes with whatever the session presents next.
Operator action: none required. Failover and recovery are automatic. For the mechanics of how the platform picks a single active copy and detects failure, see Deployment Topologies.
There is no built-in dead letter queue
The platform does not ship a DLQ. The two failure shapes above describe every path a failed message can take.
- Offset-tracked source: the message stays at the head of the source and is retried until it succeeds.
- Session-based source: the message is dropped.
If you need poison-message isolation or a dead-letter sink, build it as a custom error handler plugin using the Plugin SDK.
Investigating a missing message
Work through these checks in order. Each one rules out a class of cause.
- Event log on the adapter. Look for error-level entries around the time the message was expected. An entry naming the message is a delivery or pipeline failure; follow Event Log Reference to resolve it.
- Route conditions in the Pipeline Designer. If no route's conditions claim the message, it is acknowledged and discarded. Add or relax a condition, or add a catch-all route at the end.
- Send Processor on the matched route. If a route matches but has no Send Processor, the message is acknowledged and nothing leaves the adapter. Add a Send Processor that targets the destination you expect.
- The source system. If the source never saw the message, the platform was never asked to handle it; the problem is upstream of the adapter.
- Message flow on the source connector. No new receive activity in the event log on an offset-tracked source, while new messages are arriving upstream, is the signature of a stalled source. Apply the offset-tracked troubleshooting above.
See also
- Event Log Reference: exact descriptions of every event type and the mechanical fix for each.
- Monitoring: reading adapter status and the event log.
- Pipeline Designer: how conditions, transforms, processors, and Send Processors combine into routes.