Kafka Patterns

This page walks through two patterns that come up routinely in production Kafka deployments: multicasting a single inbound message to several topics simultaneously (commonly used for Drop Copy feeds), and building a non-destructive Dead Letter Queue that preserves the exact original payload for safe replay. It assumes you are already familiar with the bridge topologies described in the global Protocol Bridging and Fan-Out and Drop Copy use cases.

Pattern 1: Multi-topic multicast for Drop Copy

A Drop Copy is an auxiliary feed that receives the same messages as a primary processing path but is consumed by a different downstream system. In a typical trading integration the primary feed lands on an order processing topic while the Drop Copy lands on a compliance archive topic, a real-time risk topic, and a data warehouse ingest topic. All four feeds carry the same bytes, so all four consumers see a consistent view of what actually arrived.

Why a single multi-topic processor instead of several parallel routes

A Drop Copy is conceptually a fan-out, and the platform has two ways to express it. You can define one route that multicasts to N topics, or you can define N separate routes, one per topic, all sharing the same source.

Approach	When to pick it
Single route, multi-topic send	All Drop Copy destinations consume the same transform of the payload. Failures on one destination topic should not prevent the other destinations from receiving the message. The order of publication across topics is deterministic.
Multiple parallel routes	Each destination requires a materially different transform, a different source filter, or a fundamentally different lifecycle (for example, enabling one destination for a limited-duration A/B test).

The single-route approach is the right default. It keeps the message bytes identical across destinations, which is what most downstream systems rely on for reconciliation, and it minimizes the operational surface area.

Configuring the multi-topic send

In the Portal's Pipeline Designer, add a Kafka Send to Topic processor to the route and configure its topics list with every destination.

processors:
  - id: drop-copy-multicast
    type: kafka-send-processor
    config:
      topics:
        - primary.orders.inbound
        - compliance.archive
        - risk.realtime
        - warehouse.ingest

The processor iterates the topics list in order. A failure on any single topic does not abort the remaining topics. After every topic has been attempted, if one or more failed, the processor raises a single aggregated failure that carries the per-topic causes. This preserves two important properties.

1. A downstream topic that is healthy today continues to receive its copy even when a sibling topic is unreachable.
2. The route's own error handler chain, or the circuit breaker, sees a single failure event rather than one failure per destination topic, which simplifies both alerting and retry logic.

Ordering and keying

Each multicast dispatch uses the same partition key derived from the source message. If the source route sets routing_kafkaKey, every destination topic sees the message on the partition derived from that key. Two messages that share the same key are published in order on every destination topic, which is the guarantee most Drop Copy consumers rely on for stream replay.

If the source route does not set a key, messages are distributed across partitions by the broker's default assignment. Set a key explicitly for any Drop Copy that downstream consumers use to replay by entity (account, symbol, session).

The Kafka Send to Topic processor's keyStrategy field defaults to PRESERVE_EXISTING. On every dispatch the processor forwards the message to each destination topic with the routing_kafkaKey that is already on it; it does not modify the key.

Value	Behavior	When to use
PRESERVE_EXISTING (default)	Keeps the routing_kafkaKey already on the message, so a key set upstream carries through to every destination topic and preserves per-key ordering. This is the behavior the connector applies today for either value of the field.	Any Drop Copy whose consumers replay by entity (account, symbol, session).
NONE	Not yet honored by the connector. The value renders in the Portal, but the send processor does not strip the key when it is selected: the existing routing_kafkaKey is still forwarded. To publish without a key, leave routing_kafkaKey unset on the source route rather than selecting this value.	To send without a key today, do not set a key upstream.

Recommended patterns

Pure Drop Copy. One source, N identical copies to separate topics. Use this for compliance archive, risk feeds, and data lake ingest where every consumer wants the same bytes.

processors:
  - id: copy
    type: kafka-send-processor
    config:
      topics: ["compliance.archive", "warehouse.ingest"]

Selective fan-out. One source, different subsets of messages to different topics. Use separate routes with source-level filters when a compliance topic only cares about a subset of messages, and the warehouse wants everything.

Single-topic send. The degenerate case. topics is a single-element list, equivalent to the legacy single-target routing.

processors:
  - id: send
    type: kafka-send-processor
    config:
      topics: ["primary.orders.inbound"]

Pattern 2: A pristine, non-destructive Dead Letter Queue

Every production pipeline needs a place to put messages that could not be processed. A well-built Dead Letter Queue has two properties.

1. The payload that goes to the DLQ is byte-for-byte identical to the payload that originally arrived. No envelope, no wrapping, no added fields.
2. All error context (the exception, the stack trace, the original topic, the original offset, a timestamp) lives in a separate, structured channel.

Together these properties make DLQ replay a one-step operation: read the record, re-publish the bytes to the original topic, and move on. An envelope around the payload would force every downstream consumer to learn an extra serialization format or an extra unwrap step. That is how DLQs become sources of their own bugs.

How the platform achieves the pristine property

The Kafka Dead Letter Queue error handler publishes the failed message's payload as-is to the configured DLQ topic. The error context is attached as native Kafka record headers, prefixed x-dlq-*, rather than being folded into the payload body. Consumers that only care about the bytes can ignore the headers entirely. Consumers that need the failure context can read it without parsing the payload.

The header set includes the error class, the error message, the full stack trace, the original topic, the original partition, the original offset, a dead-letter timestamp, and the route identifier. This is enough to drive most replay workflows without coordinating with any external system. The x-dlq-original-topic, x-dlq-original-partition, and x-dlq-original-offset headers are populated only when the failed message originated from Kafka; a message that entered the pipeline from another source (a FIX session, the Traffic Source connector) carries no Kafka source coordinates, so those headers are absent.

Configuring the DLQ

In the Portal's Pipeline Designer, add a Kafka Dead Letter Queue error handler to the route's error handler chain and configure the DLQ topic.

errorHandlers:
  - id: dlq
    type: kafka-dlq-processor
    config:
      topic: pipeline.deadletter

The DLQ is installed as an error handler, not as a processor. It only runs when a prior pipeline stage fails and the route's error handler chain is consulted.

Interaction with the circuit breaker

The circuit breaker and the DLQ handle orthogonal failure modes, and their interaction is deliberate.

Failure type	Where it lands
A per-message fault (malformed payload, validation failure, business rule rejection)	The route's error handler chain runs, which means the DLQ writes the failed message and the pipeline continues processing healthy messages behind it.
A configured infrastructure fault (target broker unreachable, socket timeout)	The circuit breaker trips after the configured failure threshold. The DLQ is not invoked for these failures, because writing a persistent stream of DLQ entries about a dead downstream is usually less useful than halting the source and waiting for an operator.

For details on configuring which exception types count as infrastructure faults, see Resilience and Backpressure.

DLQ topic sizing and retention

Property	Recommendation
Partitions	At least as many as the source topic. Keyed messages preserve their partition affinity, so the DLQ must be able to fan out to match.
Retention	Long enough that an operator can investigate and replay within the same business day, with a safety margin for weekends. Seven days is a common starting point.
Replication factor	Match the source topic. DLQ data loss defeats the purpose of having a DLQ.
Compaction	Disable. DLQ records are typed by offset and should never be compacted on key.
Access control	Read and write restricted to the adapter service identity and the operator role that performs replays.

Replaying DLQ records

A single-record replay, given a DLQ record whose x-dlq-original-topic header names the source topic, is equivalent to re-publishing the record's payload to that topic with the same key. No transformation is required. The downstream consumer receives an identical copy of what it would have received if the pipeline had succeeded on the first attempt.

Replay workflows worth standardizing.

Scope	Approach
One record	Use a standard Kafka producer tool to read the DLQ record and publish its payload bytes to the topic named in x-dlq-original-topic, preserving the record's key.
One batch defined by time window	Drain DLQ records within a time range into a file, then drive the per-record replay above for every entry. Useful after an incident whose blast radius is time-bounded.
Full DLQ drain	Appropriate only after an incident where every DLQ entry should be replayed. Coordinate with downstream consumers to avoid duplicate processing.

Recommended configurations

Drop Copy with DLQ. The canonical production pattern. Multicast to primary plus compliance, with a DLQ to catch processing failures on either route.

processors:
  - id: copy
    type: kafka-send-processor
    config:
      topics: ["primary.orders", "compliance.archive"]
errorHandlers:
  - id: dlq
    type: kafka-dlq-processor
    config:
      topic: pipeline.deadletter

Single-topic producer, no DLQ. Appropriate for development environments and for pipelines where failed messages should surface in the event log rather than being parked.

processors:
  - id: send
    type: kafka-send-processor
    config:
      topics: ["dev.events"]

DLQ only, no multicast. Useful when the route produces to one topic but the business requires a formal failure path.

processors:
  - id: send
    type: kafka-send-processor
    config:
      topics: ["orders.inbound"]
errorHandlers:
  - id: dlq
    type: kafka-dlq-processor
    config:
      topic: orders.deadletter

Troubleshooting

Symptom	Likely cause	Action
Multicast route consistently fails on one of its topics while others succeed	The failing topic does not exist, has an ACL mismatch, or has partition count mismatches.	Inspect the aggregated failure's per-topic causes. Fix the failing topic independently; the healthy topics are already receiving their copies.
DLQ records are missing the original payload	The payload could not be serialized. The processor will have logged a serialization error and returned the message to the pipeline for nack.	Inspect the error log. Ensure the payload type is compatible with byte serialization.
DLQ consumers see extra fields around the payload	Your consumer is reading a DLQ populated by a different tool that wraps the payload. The platform's DLQ never wraps.	Verify that the DLQ is being written by the platform's dead letter handler, not by an external process.
Records in the DLQ cannot be replayed because the original topic is not recorded	The DLQ record is older than the schema that captures original-topic headers, or the DLQ is being written by a different tool.	Replay older records manually against whichever topic is appropriate based on the record's key or other headers.
Circuit breaker keeps opening while messages still arrive at the DLQ	The breaker considers some exception type infrastructure but the error handler is still handling it.	Verify that the misbehaving exception is not listed both as "included" in the breaker and as recoverable by the DLQ. The two should be disjoint.

Monitoring signals

Signal	What it tells you	Alert when
Write rate to the DLQ topic	How frequently messages are being parked	Any sustained non-zero rate during steady-state operation. A healthy pipeline should DLQ only on edge-case payloads.
Multicast per-topic send failure rate	Whether any destination is flaky	Per-destination failures above a small tolerance
DLQ consumer lag	Whether DLQ records are being reviewed and replayed	Growth without bound implies nobody is draining the DLQ. Record operational owners and an SLO for review.

See also

• Protocol Bridging
• Fan-Out and Drop Copy
• Kafka Configuration and Tuning
• Resilience and Backpressure