Skip to main content

Advanced Kafka Topology

This guide explains how to design and deploy complex multi-adapter topologies using Kafka as the message backbone between FIX sessions.

The FIX-to-Kafka Bridge Pattern​

The most common enterprise topology bridges inbound FIX connectivity to a Kafka cluster for downstream consumption, and routes outbound messages from Kafka back to a FIX venue. Kafka acts as the durable buffer between the two legs.

Venue ──FIX──▶ Inbound Adapter ──▶ Kafka topic ──▶ Outbound Adapter ──FIX──▶ Venue

Inbound Adapter (FIX → Kafka)​

  • Connector type: FIX Initiator (or Acceptor, depending on the venue)
  • Pipeline: Receive FIX message → transform to your internal format → send to Kafka topic
  • Deployment mode: active_passive, since only one instance should hold the FIX session at a time

Outbound Adapter (Kafka → FIX)​

  • Connector type: Kafka Consumer
  • Pipeline: Read from Kafka topic → transform to FIX → send to venue
  • Deployment mode: active_passive for FIX Initiators (prevents sequence number conflicts); scale_out is appropriate only if the target supports multiple concurrent connections

Deploying the Topology​

You can deploy a two-adapter topology using either the Portal or the Custom Bundle workflow described in Custom Bundles & Extensibility.

Option 1: Configure in the Portal​

Create each adapter directly in the Portal. Use the Pipeline Designer to wire the inbound FIX connector to a Kafka Producer target, and a separate adapter to wire a Kafka Consumer connector to a FIX session target.

Option 2: Deploy via Custom Bundle and Helm​

For repeatable, environment-parameterized deployments, package your adapter manifests, plugins, and artifacts into a Custom Bundle .zip and deploy via Helm. See Custom Bundles & Extensibility for the full step-by-step workflow.

Scaling and High Availability​

Active-Passive for FIX Sessions​

Use deploymentMode: "active_passive" for any adapter that holds a FIX session (Initiator or Acceptor). This ensures only one adapter instance is connected to the venue at a time, preventing sequence number conflicts. If the active instance fails, another claims the session within the lease timeout.

Scale-Out for Kafka Consumers​

Use deploymentMode: "scale_out" for Kafka Consumer adapters where the destination allows concurrent writers. Each replica joins the same Kafka Consumer Group, and Kafka distributes partitions across all active replicas automatically.

Do not use scale_out for FIX Initiators. Multiple simultaneous connections to the same venue session will cause sequence number conflicts.

Monitoring the Topology​

Use the Portal to monitor both legs of the bridge:

  • Check the Event Log on each adapter for session status, logon events, and errors
  • Use the Portal's Adapters list to get an operational summary across all adapters
  • Watch for adapters stuck in CLAIMING, which indicates a lease conflict or Orchestrator connectivity issue

See Troubleshooting for a diagnostic checklist.