Skip to main content

What Happens to Messages

This page explains what the platform does with messages as they flow through a pipeline, including what happens when something goes wrong.

The normal path​

A message arriving at an adapter goes through the pipeline in order:

  1. Filter: does this message match the route's conditions? If not, it is acknowledged and discarded. No error, no record in the event log; it's working as designed.
  2. Transform: the message is reformatted. If a transformer can't process the message, it is discarded (same as a filter, silent by design).
  3. Process: custom logic runs. The message may be enriched, validated, or routed.
  4. Deliver: the message is sent to the target.

Once a message is successfully delivered, it is acknowledged back to the source.

What "filtered" means​

If a message is filtered out by a condition (for example, "only process certain message types, drop everything else"), it is silently acknowledged. The message is gone. There is no dead letter queue, no copy is kept, and nothing appears in the event log.

This is correct behavior. Filtering is intentional. If messages are being filtered that you expected to pass through, check your pipeline conditions in the Pipeline Designer.

What happens when delivery fails​

If the platform cannot deliver a message to the target, the behavior depends on the source connector:

Sources with offset tracking​

For source connectors that track message position (such as topic consumers), a failed delivery means the message is not acknowledged. The offset is not committed, so the message will be redelivered on the next poll. This means:

  • Transient failures (network blip, target temporarily unavailable) will self-heal, and the message will be retried automatically
  • Persistent failures can cause the source to stall. If a message cannot be processed repeatedly, the source will keep retrying the same message and make no forward progress

If an adapter stops processing new messages, check the event log for repeated errors on the same message.

Session-based sources​

For session-based source connectors, there is no built-in redelivery mechanism. If a message from a session-based source cannot be delivered downstream, the message is dropped.

What happens when the platform loses its lease mid-message​

If an adapter loses its HA lease while processing a message (for example, due to a network partition), it will halt the pipeline immediately and nack the message rather than risk delivering it twice. The adapter then shuts down.

For sources with offset tracking, this means the message will be redelivered to another adapter pod that successfully claims the session.

There is no built-in dead letter queue​

Conncentric does not have a built-in DLQ. Messages that fail to process on an offset-tracked source are retried by the source. Messages that fail on a session-based source are dropped.

If you need poison-message handling (isolate and skip messages that consistently fail), this can be implemented as a custom error handler plugin using the Plugin SDK.

How to investigate a message that went missing​

  1. Check the event log on the adapter. Look for errors around the time the message was expected.
  2. Check your pipeline conditions in the Pipeline Designer. The message may have been intentionally filtered.
  3. Check the source system. Verify the message was actually sent to the platform.
  4. Check your source connector metrics. If the adapter is stuck retrying a poison message, the source will show no forward progress.