Kafka Adapter
Overview
The Kafka integration-adapter plugin provides consumer and producer connectivity to Apache Kafka and the compatible brokers listed under Supported brokers. Each adapter instance reads from or writes to one or more Kafka topics, hands messages to the pipeline for configured processors and transformers, and routes results to downstream targets.
For the plugin's component catalog and a deployment-mode quick reference, see the Kafka Plugin overview.
Supported brokers
| Broker | Supported |
|---|---|
| Apache Kafka 3.0+ | Yes |
| Redpanda 23.2 | Yes (newer versions expected to work) |
| Confluent Platform 7.0+ | Yes |
Broker connection
bootstrapServers is a required field on each Kafka connector, entered in the Portal when you configure the connection (for example, kafka.example.com:9092) or carried in a bundle definition as a concrete value. There is no adapter.kafka block in the Helm chart.
Broker authentication is not currently configurable on the connector: the fields below cover connection and tuning only. Reaching a broker that requires SASL or client-certificate authentication needs those fields added first, so raise it with Connamara before planning that integration. Where the broker itself does not require authentication, terminate TLS at the load balancer or network layer using your organization's standard approach, and use Kubernetes NetworkPolicies to restrict traffic between adapter pods and the broker.
Topic creation
autoCreateTopics defaults to true. On startup the connector checks the topics it is configured with directly, producerDefaultTopic and the consumer subscribe topics, and creates any that are missing using defaultPartitions and defaultReplicationFactor. This startup check is not exhaustive: topics named on a Kafka Send to Topic processor (for example, the Drop Copy multicast pattern) are resolved per message and are not pre-validated, so a typo in a processor topic surfaces at runtime rather than at startup.
| Field | Default | Description |
|---|---|---|
autoCreateTopics | true | Create missing topics on startup. Set to false when you provision topics yourself. |
defaultPartitions | 1 | Partition count for topics the connector creates. |
defaultReplicationFactor | 1 | Replication factor for topics the connector creates. |
In production, where topic partitioning and replication are usually provisioned and governed outside the adapter, set autoCreateTopics: false. A typo in a startup-validated topic name (producerDefaultTopic or a consumer subscribe topic) then fails the connector at startup, naming the missing topic, instead of silently creating a single-partition, single-replica topic that under-provisions the workload. A typo in a topic named on a send processor is not covered by this check and still surfaces at runtime.
Supported Roles
| Role | Direction | What it does |
|---|---|---|
Kafka Consumer | Source | Reads messages from a Kafka topic and delivers them to the pipeline. Tracks per-partition offsets and joins a consumer group when multiple adapters share a group identifier. |
Kafka Producer | Target | Declares a Kafka destination for a route. Delivery is executed by a Kafka Send to Topic processor on the same route. |
Role is selected per connection in the adapter configuration. A single adapter can pair a consumer source with a producer target to bridge topics, or pair a Kafka role with another protocol's role to bridge across protocols.
Deployment-mode safety
Kafka consumer groups are natively concurrent: the broker assigns partitions to every consumer in the group and rebalances as consumers join or leave. The safe deployment mode for a Kafka adapter depends on every connector on the adapter, not just the Kafka one.
If any connector on the adapter, source or target, requires exclusive single-writer access, the entire adapter must run as Single Writer.
Deploying such an adapter in Scale Out produces duplicate concurrent logons on the single-writer leg and sequence-number corruption on any stateful protocol. The Kafka consumer-group mechanic cannot protect the non-Kafka side of the adapter; the broker does not know that leg exists.
| Pairing on a single adapter | Safe deployment mode | Why |
|---|---|---|
| Kafka Consumer (source) with a Kafka Producer (target) | Scale Out | Both connectors tolerate concurrent replicas. The consumer group assigns partitions per-pod; the producer has no single-writer constraint. |
| Kafka Consumer (source) with a FIX Initiator (target) | Single Writer | The FIX Initiator holds an exclusive session with the venue. Running two replicas in parallel produces duplicate logons and corrupts sequence numbers. |
| Kafka Consumer (source) with a FIX Acceptor (target) | Single Writer | The FIX Acceptor listens for a single inbound session per identity. Two replicas attempting to accept the same identity conflict at the TCP and session layers. |
| FIX Initiator or Acceptor (source) with a Kafka Producer (target) | Single Writer | The FIX source holds an exclusive session. The Kafka producer side is incidentally concurrent-safe, but the weakest link (FIX) forces Single Writer for the adapter as a whole. |
| Kafka Consumer (source) with any custom connector requiring a single owner | Single Writer | Same rule: the single-owner target constrains the whole adapter. |
To scale throughput on a stateful leg, shard the work into multiple single-owner sessions and run one Single Writer adapter per shard. See Session sharding for the sharding pattern. See Protocol Bridging for worked examples of broker-to-broker (Scale Out) and stateful-to-broker (Single Writer) bridges.
Behavior Under Load
When messages arrive faster than the pipeline can drain them, the adapter must choose between accumulating in memory until the pod is killed or applying backpressure that the broker can interpret. The platform applies a graduated response that protects the pod from memory exhaustion and surfaces the slowdown to the broker through standard Kafka mechanics.
| Layer | What happens | What it protects against |
|---|---|---|
| Pipeline throttle | When the completion pipeline falls behind, completion tasks run inline on the Kafka I/O thread. The consumer stops calling poll(), which the broker observes as a stalled consumer and stops sending records. | Runaway in-memory queue growth on the consumer side. |
| Heap-pressure throttle | When JVM heap usage crosses the platform's pressure threshold, the same inline-execution behavior engages regardless of queue depth. | Slow heap consumption hidden by a downstream buffer. |
| Producer buffer pause | When the producer's internal batching buffer fills, producer.send() blocks the caller. The pause propagates upstream into the pipeline and on into the consumer's poll loop. | Silent heap consumption by the producer's own batching buffer. |
| Consumer-group rebalance | A pod that stays paused long enough is removed from the consumer group by the broker. Surviving pods pick up its partitions and continue draining. | Indefinite stall on a wedged pod. The group itself keeps moving. |
The platform sizes its queues and pressure thresholds automatically from the pod's available memory and CPU. Operators tune the response by adjusting pod resource limits, not by editing connector fields. For pod sizing and resource limits, refer to the Scaling and Helm reference guides.
Message loss guarantees
The adapter's durability posture under backpressure depends on the producer's acknowledgement configuration. The table below summarizes the loss characteristics per setting. Verify the behavior against your broker's replication configuration before relying on it in production.
| Configuration | Loss guarantee under backpressure |
|---|---|
producerAcks: all (default) + transactionalIdPrefix unset | The adapter waits for broker confirmation before acknowledging the upstream caller. Under backpressure, producer.send() blocks, which propagates the wait upstream. Broker-level replication guarantees apply beyond that point. |
producerAcks: all + transactionalIdPrefix set (exactly-once semantics) | Atomic consume-process-produce transactions. Either all messages in a batch are committed or none are. Strongest guarantee. |
producerAcks: 1 | The leader acknowledges, but if the leader fails before replicating, the message is lost. Not recommended for production. |
producerAcks: 0 (fire-and-forget) | No acknowledgement at all. Any producer failure loses messages. Do not use in production. |
For the strongest durability, use the defaults (acks=all, transactional mode when the consumer logic warrants it).
Monitoring signals
| Signal | Meaning | Alert when |
|---|---|---|
| "Pipeline backpressure" WARNING event | Emitted (rate-limited) when the completion pipeline is throttling, or when JVM heap is above the pressure threshold. | The event appears and keeps recurring rather than clearing. |
| Producer record-send-total (Prometheus) | Records sent per second. | Drops unexpectedly during a workload that should be steady. |
| Producer buffer-available-bytes (Prometheus) | Bytes remaining in the producer buffer. | Trends toward zero and stays there. |
| Consumer records-lag-max (Prometheus) | Highest lag across consumed partitions. | Grows unbounded; the adapter or downstream cannot keep up. |
| Adapter pod memory usage | RSS of the adapter container. | Above 80 percent of the container limit for more than a few seconds. |
Offset tracking
Each adapter instance tracks its own position in the Kafka topic. When a pod restarts or is replaced, it resumes from its last committed offset. It does not re-process messages that were already handled, and it does not skip messages that were not.
A message is only marked as processed after it has been fully handled and delivered to the downstream target. This prevents loss on a pod crash, but it does mean a message can be delivered twice in edge cases. Design downstream systems to be idempotent where that matters.
Offset commit strategies
The consumerCommitMode value controls when processed offsets are written back to Kafka. Pick the strategy that matches your durability-versus-throughput trade-off.
| Mode | Behavior | When to use |
|---|---|---|
ASYNC_BATCH (default) | Batched, non-blocking commits. Best throughput. A pod crash can lose at most one batch of already-acked offsets, which are redelivered on restart. | Most workloads where downstream consumers are idempotent. |
SYNC_BATCH | Batched, blocking commits. Slightly slower; each commit waits for broker acknowledgement. Redelivery window is smaller than async. | Workloads where you want tighter bounds on redelivery. |
MANUAL_PER_MESSAGE | Commits each message's offset immediately on processing completion. Highest durability; lowest throughput. | Financial postings, billing events, or any per-message operation where double-processing is more expensive than redelivery. |
Offset tracking strategy
The offsetTrackingStrategy value controls how the adapter handles out-of-order acknowledgements from the pipeline.
| Strategy | Behavior | When to use |
|---|---|---|
CONTIGUOUS (default) | Commits only up to the lowest unacknowledged offset per partition. Strict at-least-once semantics; no message is skipped even if its neighbour is acked first. | All production workloads. Safe default. |
LATEST | Commits the highest acked offset regardless of gaps. Higher throughput; messages between a gap can be skipped on a crash. | Only when an upstream partitioner guarantees message order and message loss at crash boundaries is acceptable. |
Producer reference
| Field | Default | Description |
|---|---|---|
producerAcks | all | Broker acknowledgement policy. Do not change unless you have a specific latency-versus-durability trade-off in mind. |
producerLingerMs | 5 | Milliseconds to wait before sending a batch, to allow batching. Higher values improve throughput; 0 gives lowest latency. |
producerCompressionType | lz4 | Compression codec. lz4 is the best speed-to-ratio balance for financial message payloads. |
transactionalIdPrefix | unset | Enables Kafka Exactly-Once Semantics when set. Consumer isolation is automatically set to read_committed. |
producerDefaultTopic | unset | Topic used when a message on a producer route carries no target-topic routing header. Set it for a producer connector that always writes to one topic; leave it unset when a send processor supplies the topic. |
producerBufferMemoryBytes | 8388608 (8 MB) | Upper bound on the producer's internal batching buffer. Kafka's own default is 32 MB, which can consume a large share of a small container's heap before queue-depth backpressure engages. Raise it only when the pod has heap headroom and needs larger batches. |
Consumer reference
| Field | Default | Description |
|---|---|---|
consumerTopics | required | Topics the connector subscribes to (Portal label "Subscribe Topics"). Required to consume: consumer threads start only when both consumerTopics and consumerGroupId are set. |
consumerGroupId | required | Consumer group the connector joins (Portal label "Consumer Group ID"). Required when subscribing. |
consumerConcurrency | 1 | Number of parallel consumer threads. Must not exceed the topic's partition count. |
consumerCommitMode | ASYNC_BATCH | Offset commit strategy. See the table above. |
offsetTrackingStrategy | CONTIGUOUS | Out-of-order acknowledgement handling. See the table above. |
consumerAutoOffsetReset | earliest | Where a new consumer group starts when it has no committed offset. Use earliest to process the full backlog on first connect; use latest to start from live traffic and ignore history; use none to make the consumer fail with an error instead of resetting, for workloads where starting without a committed offset is itself a fault to investigate. |
consumerMaxPollRecords | 500 | Maximum records returned per poll. Lower it when per-message processing is heavy and you want the consumer to yield to backpressure sooner; raise it for lightweight, high-throughput workloads. |
Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| Venue reports duplicate logons or sequence-number violations after scaling up a Kafka-to-FIX adapter | The adapter was set to Scale Out. The Kafka source side allowed concurrent replicas; the FIX target side did not. | Change the adapter's deployment mode to Single Writer. See the Weakest Link rule. Investigate whether messages were double-processed during the window. |
Adapter pod restarts with exit code 137 (OOMKilled) while processing Kafka traffic | Pod memory limit too low for the workload, or too many concurrent connectors sharing a pod. | Raise adapter.resources.limits.memory in Helm values. See the capacity table in Scaling. |
| Producer send latency climbs steadily even at constant input rate | Producer is filling its batching buffer faster than the broker can acknowledge. | Check broker health, network latency to brokers, and whether compression is enabled. If the pod has heap headroom, raise the pod's resource limits and let the platform's auto-sized buffer absorb more. |
| Consumer lag grows unbounded | Consumer cannot keep up with the topic's production rate. | Increase consumerConcurrency up to the partition count of the source topic. Add adapter replicas (only on Kafka-to-Kafka adapters; see the Weakest Link rule before scaling adapters whose targets are stateful). Check downstream target capacity. |
A recurring Pipeline backpressure WARNING event with Kafka as the downstream | Downstream Kafka broker cannot absorb the write rate. | Broker-side issue: check broker CPU, disk IO, partition count. The adapter is doing what it should (holding back); the broker is the bottleneck. |
| Messages appear duplicated downstream after a pod restart | Expected behavior under ASYNC_BATCH commit mode. | Design downstream consumers to be idempotent, or switch to MANUAL_PER_MESSAGE commit mode for tighter redelivery bounds. |
| Transactional producer cannot commit | Broker is not configured for transactions, or the transactional ID collides with another adapter. | Verify the broker supports transactions (Apache Kafka 0.11+). Ensure transactionalIdPrefix is unique per adapter instance across your deployment. |
See also
- FIX Adapter for the FIX plugin's behavior under load.
- Protocol Bridging for multi-adapter bridge patterns, including the deployment-mode rule applied to stateful-to-broker-to-stateful pipelines.
- Scaling for pod resource sizing guidance.
- Helm reference for adapter pod resource limits.
- Traffic Source Connector for benchmarking Kafka adapters at a target rate.
- Troubleshooting for broader diagnostic patterns.