Kafka Plugin
The Kafka plugin provides consumer and producer connectivity to Apache Kafka and the compatible brokers listed under Supported brokers. It ships as an integration-adapter plugin only; Kafka configuration has no control-plane state that an orchestrator plugin needs to manage.
Provided components
When building pipelines in the Portal, these components appear in the dropdown menus. Their configuration fields are rendered dynamically based on the plugin version installed on your Orchestrator.
| Component ID | Name | Type | What it does |
|---|---|---|---|
kafka-connector | Kafka Connector | Connector | Reads from or writes to a Kafka topic. Tracks per-partition offsets on the consumer side and participates in a consumer group when multiple adapters share a group identifier; on the producer side, declares a Kafka destination for a route. |
kafka-send-processor | Kafka Send to Topic | Processor | Sends the current message to one or more Kafka topics in a single step. A failure on any single topic does not abort the remaining topics; per-topic failures are aggregated into one error for the route's error handler chain. See Kafka Patterns. |
kafka-dlq-processor | Kafka Dead Letter Queue | Error Handler | Publishes a failed message's payload to a configured DLQ topic with error context attached as record headers, so the original bytes can be replayed without unwrapping. |
kafka-retry-dlq | Kafka Retry & DLQ | Error Processor | Retries transient errors in place by nacking to rewind the partition, with no retry topic. When the retry budget (retryDurationMs, default 60000) is exhausted, or the error is non-transient, it writes the failed message to dlqTopic. |
Supported brokers
| Broker | Supported |
|---|---|
| Apache Kafka 3.0+ | Yes |
| Redpanda 23.2 | Yes (newer versions expected to work) |
| Confluent Platform 7.0+ | Yes |
For encrypted or authenticated broker connections, configure TLS at the network or load balancer layer using your organization's standard approach.
Deployment-mode reference
Kafka is natively concurrent-friendly for consumers, but any adapter that pairs a Kafka connector with a stateful target (a FIX session, an exclusive-consumer queue) inherits the target's single-writer requirement. The table below summarizes the safe mode for each common pairing. For the rule that produces these answers, see the Weakest Link rule.
| Pairing on a single adapter | Safe deployment mode |
|---|---|
| Kafka Consumer (source) with a Kafka Producer (target) | Scale Out. Both sides are concurrent-safe. |
| Kafka Consumer (source) with a FIX Initiator or Acceptor (target) | Single Writer. The FIX target forces exclusive access. |
| FIX Initiator or Acceptor (source) with a Kafka Producer (target) | Single Writer. The FIX source forces exclusive access. |
In this section
- Configuration & Tuning: broker connection, offset management, backpressure model, producer tuning, deployment-mode safety, and troubleshooting.
- Kafka Patterns: multi-topic multicast (Drop Copy) and non-destructive Dead Letter Queue patterns.
End-to-end pipelines
This plugin documents the Kafka connector and the Kafka-specific portal fields. End-to-end integration patterns (protocol bridging through a Kafka topic, fan-out, drop copy, multi-venue aggregation) are protocol-agnostic and live in the global Use Cases section. Start there to see how a Kafka adapter slots into a complete pipeline.