Deployment Topologies
This page is for site reliability engineers deciding how to deploy adapter pods against a given integration. It covers the two supported topologies (Single Writer and Scale Out), explains which topology fits which integration pattern, and shows how to configure each. If you picked the wrong topology for your protocol, your pipeline is either fragile or artificially serialized, so picking correctly up front matters more than most individual configuration choices.
How High Availability works on this platform
High availability is not a separate mode. Both topologies achieve HA the same way: provision more than one pod. The platform handles the rest.
Two independent configuration planes are in play, and they live in different places. Get them straight before you read the table below.
| Setting | Where it lives | What it controls |
|---|---|---|
| Deployment mode (Single Writer or Scale Out) | Per adapter, selected in the Portal when you create the adapter, or set in the bundle definition. Not a Helm value. | Whether the orchestrator arbitrates a single-writer lease for the adapter, or lets every pod serve it concurrently. |
adapter.replicaCount, adapter.leasePartitionFenceMs | Helm values.yaml | How many pods the pool runs, and the self-fence window. |
deploymentMode is a property of the adapter, not of the chart. There is no adapter.deploymentMode key in values.yaml. Pasting one there is silently ignored: the pool keeps whatever mode the adapter definition already carries, so you can end up running the wrong topology (for example, concurrent writers against a single-owner protocol) with no error to warn you. Choose the mode where the adapter is defined: the Portal create form, or the bundle definition. In the bundle JSON the field serializes lowercase as single or scale_out.
| Goal | Deployment mode (per adapter, Portal or bundle definition) | Helm values.yaml |
|---|---|---|
| HA for a single-owner protocol (FIX, exclusive consumer, custom TCP) | Single Writer | adapter.replicaCount: 2 or higher. The orchestrator's per-adapter lease keeps a single pod active at a time; the others stand by, polling, and claim on failover. |
| HA for a partitioned or stateless protocol (Kafka consumer group, load-balanced HTTP) | Scale Out | adapter.replicaCount: 2 or higher. Every pod is active; the external system distributes work and absorbs pod failures by rebalancing. |
| No HA, accept restart-scoped downtime | Either mode | adapter.replicaCount: 1. Demo, development, and any production integration where a brief outage during pod restart is acceptable. |
The rest of this page details the mechanisms behind each topology and the failure modes they cover.
The two topologies at a glance
| Topology | Number of simultaneously serving pods | Who arbitrates ownership | Choose when |
|---|---|---|---|
| Single Writer | One pod actively serves the integration at a time. Additional pods stand by, polling for the lease. | The platform's orchestrator, via a per-adapter lease. | The integration protocol requires a single owner of the connection, OR you accept restart-scoped downtime in exchange for the simplest topology. |
| Scale Out | Two or more simultaneously. | The external system itself (broker, load balancer, shared queue). | The integration protocol is either natively partitionable, or stateless enough that concurrent workers do not interfere. |
Choosing comes down to one question about your protocol: can two adapter pods talk to the same external system at the same time without causing damage? If yes, Scale Out is usually cheaper, faster, and simpler. If no, Single Writer is the safer choice.
Single Writer
Single Writer means a single pod actively serves the integration at a time. If you run additional pods against the same adapter, they sit in standby (polling the orchestrator for an unclaimed lease) and take over automatically if the active pod fails. If you run only one pod, the integration is offline during pod restarts.
Why Single Writer exists
Some protocols require a single owner of the connection. The signatures are protocol-specific but the underlying pattern is the same: the protocol maintains state that only one endpoint can meaningfully hold at a time. Examples include:
- Stateful, monotonically sequenced session protocols (this is the pattern behind standard financial message protocols like FIX, where message loss or duplicate processing is detected by the counterparty as a sequence number violation).
- Exclusive-consumer message queues that refuse to dispatch the same message to more than one subscriber.
- Custom TCP integrations that require a single long-lived connection per counterparty.
Running two active pods against these protocols can corrupt the protocol's state. For sequenced sessions, two pods incrementing the same outbound counter produce duplicate numbers, which the counterparty typically rejects or silently drops. For exclusive consumers, two connections contend for dispatch and the outcome is unpredictable.
The platform protects against this with a lease.
How the lease works
The orchestrator issues each adapter a lease that is held by one pod at a time. A pod must hold a valid lease to serve the integration. The pod refreshes its lease by checking in with the orchestrator on a periodic interval. If the orchestrator stops hearing from the pod, it marks the lease stale and frees the adapter, at which point any standby pod can claim it on its next heartbeat cycle.
The platform is designed to preserve single-writer semantics whenever the orchestrator is reachable by at least one adapter pod in the pool. Three failure modes show how that property holds.
| Failure mode | Platform behavior |
|---|---|
| Explicit revoke on check-in | The pod immediately stops serving traffic, closes network listeners, and exits cleanly. |
| Configuration or artifact drift | The pod cooperatively releases the lease so a standby with the updated configuration can take over without a forced outage window. |
| Orchestrator unreachable | By default the pod self-fences after adapter.leasePartitionFenceMs of continuous check-in failure (90s in the shipped chart), closing the split-brain window for asymmetric partitions. Set the value to 0 to disable the fence, in which case the pod keeps serving on its last-valid lease instead (see below). |
How HA is achieved with one adapter row
The Single Writer topology uses one logical adapter (one row in the orchestrator's catalog), and you achieve high availability by running multiple pods against it. Every pod boots, polls for unclaimed adapters, and tries to claim. The orchestrator's lease arbitration grants the lease to one of them. The losers stay in standby and poll on their heartbeat interval. When the active pod releases (cooperatively on shutdown, or via staleness on a crash), the next polling pod wins the lease and takes over.
This is structurally simpler than maintaining a cluster of distinct adapter rows: there is no notion of "primary" vs "secondary" registered with the platform, no per-slot configuration drift, and no policy required to keep cluster members in lock-step. Operators reason about a single adapter and tune redundancy by changing the pod replica count.
The split-brain hazard and the self-fence
In a symmetric outage (orchestrator is down or unreachable from every pod) the system stays safe: while the orchestrator is unreachable no standby can be granted the lease, so the active pod continues serving on its existing lease.
In an asymmetric partition, the active pod loses its path to the orchestrator while a standby in a different network segment still has connectivity. The orchestrator eventually marks the active pod's lease stale and hands ownership to the standby. Until the original pod regains orchestrator connectivity and learns that its lease was revoked, both pods are serving traffic. This is the split-brain window.
The platform closes this window with adapter.leasePartitionFenceMs. When set to a positive value (the shipped chart default is 90000), the pod self-fences after that many milliseconds of continuous check-in failure, even without an explicit revoke signal. Setting it to 0 disables the fence.
| Setting | Behavior |
|---|---|
90000 (default, armed) | The pod self-fences after this many milliseconds of continuous check-in failure. The value must exceed the orchestrator's staleness detection window (orchestrator.staleness.thresholdSeconds, default 75, so 75000 ms) plus the longest planned control-plane rollout duration; otherwise every legitimate orchestrator restart cascades a full-fleet fence. The 90s default clears the 75s staleness default with 15s of headroom. |
0 (opt out) | Disabled. Unreachability alone does not fence the pod; only an explicit revoke does. Appropriate for single-cluster deployments where pods and orchestrator share fate. Not recommended in production. |
When to use Single Writer
Choose Single Writer when any of the following are true for your integration.
- The protocol requires a single owner of the connection.
- The protocol is stateful in a way that two concurrent writers cannot safely share (sequence numbers, exclusive sessions, monotonic counters).
- The external system is a single endpoint that will reject or misbehave under concurrent connections from the same logical identity.
- You explicitly accept restart-scoped downtime as a non-issue (development workloads, demos, or production integrations where a brief outage during pod restarts is acceptable). In this case, run a single pod and skip the standby cost.
For HA, provision at least two pods so there is always a standby. More standby pods reduce the probability that a failover has to wait for a pod startup, at the cost of idle infrastructure.
Configuration
For HA (recommended for production):
adapter:
replicaCount: 2
For demo or development workloads where downtime is acceptable:
adapter:
replicaCount: 1
Select Single Writer as the deployment mode when creating the adapter. No additional lease configuration is required; the platform defaults are appropriate for most deployments, including the self-fence, which is armed at 90s by default. Change adapter.leasePartitionFenceMs only to raise it for a longer control-plane rollout budget, or set it to 0 for single-cluster deployments where pods and orchestrator share fate.
Scale Out
Scale Out runs two or more pods that simultaneously serve the integration. Work is divided between them by the external system, not by the platform.
Why Scale Out exists
Some protocols do not have the single-owner property. Two adapter pods can safely hold independent connections to the same external system at the same time, because the external system distributes work between them. Two common patterns qualify.
Natively partitioned consumers. The external system assigns disjoint partitions to each consumer. Each pod receives its own slice of the stream, there is no overlap, and adding or removing pods triggers a rebalance without any platform-level coordination. This is how consumer groups work in queue systems with broker-managed partition assignment, and how several other data-streaming protocols distribute load.
Stateless request-response endpoints. Each inbound request is self-contained. Any pod can handle any request. An external load balancer routes each request to whichever pod has capacity. Typical examples include inbound HTTP receivers, webhook handlers, and pull-based pollers against queryable APIs.
In both cases, the external system, not the platform, is the authority for work distribution. Forcing these protocols into Single Writer is an anti-pattern: it limits throughput to a single pod's capacity, turns every pod failure into a failover delay, and prevents horizontal scaling.
How work distribution works
Scale Out is modeled as multiple adapter rows that share one configuration. Every adapter row runs the same configuration, every pod connects to the external system, and the platform allows all of them to be claimed simultaneously. The external system, not the platform, decides which pod processes which slice of work.
This is distinct from Single Writer in two important ways.
- The orchestrator does not gate ongoing message flow for these adapters. The only reason the orchestrator matters is for configuration changes and observability.
- There is no meaningful split-brain hazard at the protocol layer. The external system is the source of truth for which pod handles which work, and any pod temporarily "talking too much" (for example, because it thinks it was kicked but the broker still considers it present) is a protocol-layer situation the external system resolves automatically.
When to use Scale Out
Choose Scale Out when your integration protocol is natively partitioned (the broker assigns partitions to consumers), or stateless at the request level (each inbound request is independent and can be handled by any pod).
The trade-off is throughput: Scale Out scales linearly with pod count up to the external system's partition or concurrency limits. Three pods process approximately three times as much as one. Single Writer caps out at a single pod's capacity regardless of how many standbys you run.
Worked example: natively partitioned source
Consider an adapter that consumes from a partitioned queue. The external broker assigns partitions to each pod that joins the same consumer group. Three pods sharing a group identifier receive the stream of messages split three ways. If a pod fails, the broker rebalances its partitions onto the surviving pods without platform-level coordination. If a fourth pod is added, the broker rebalances again to include it.
For a worked example of a partitioned broker source running Scale Out (consumer group identifier, partition rebalancing behavior, throughput planning), see Protocol Bridging.
The same principle applies to any protocol with broker-managed partition assignment.
Worked example: stateless inbound endpoint
Consider a custom plugin that exposes an inbound HTTP endpoint. The adapter pool runs behind a standard cluster Service. The Service's load balancer distributes inbound requests across the pod set. Each pod handles its assigned request independently, with no shared state.
This pattern does not require platform-level arbitration for ownership. Operationally, the pool is managed like any other stateless HTTP workload: readiness probes keep dead pods out of rotation, and the load balancer handles request distribution across the running pods.
Configuration
adapter:
replicaCount: 3
Select Scale Out as the deployment mode when creating the adapter. This tells the platform that every adapter in the cluster runs the configuration concurrently. The plugin must be one that supports concurrent pods against the same external system; check the plugin's own documentation for supported modes.
Mixed deployments
A single adapter pool commonly runs multiple adapters with different topologies. The platform allows this. Each adapter carries its own deployment mode, and the orchestrator applies lease arbitration only to the Single Writer adapters in the pool. Scale Out adapters are claimed by every pod in their cluster; Single Writer adapters are claimed by one pod.
In a mixed pool the replica count must be at least two, so that every Single Writer adapter has a standby. The Scale Out adapters gain no benefit from the standby, but they also pay no cost.
Decision heuristic
Use this tree to pick the topology for a new integration.
Recommended configurations
Three canonical patterns cover most production deployments.
Stateful single-owner integration (for example, standard financial session protocols). Minimum two pods, Single Writer, default lease configuration. The self-fence is armed at 90s by default; keep it armed when the deployment spans network segments with realistic risk of asymmetric partition.
adapter:
replicaCount: 2
leasePartitionFenceMs: 90000
Deployment mode: Single Writer.
Natively partitioned consumer integration (for example, a consumer group against a shared queue broker). Two or more pods, Scale Out, scaled according to partition count.
adapter:
replicaCount: 3
Deployment mode: Scale Out. See the plugin's documentation for broker-side configuration (consumer group identifier, partition strategy).
Stateless inbound endpoint integration. Two or more pods, Scale Out, behind a standard load balancer Service.
adapter:
replicaCount: 3
Deployment mode: Scale Out.
Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| Counterparty reports sequence number violations or duplicate processing after a failover | Two pods served the integration simultaneously for a portion of the failover window. The deployment mode is set to Scale Out on a single-owner protocol. | Change the deployment mode to Single Writer. Investigate whether any messages were double-processed during the window. |
| Scaling up the pool does not increase throughput | The deployment mode is Single Writer. Only one pod is serving traffic regardless of replica count. | If the protocol supports it, change to Scale Out. If it does not, throughput is bound by one pod's capacity. |
| Pods stay in standby indefinitely after a failover | The orchestrator cannot reach the standby pods, or the pods cannot reach the orchestrator. | Check pod reachability from the control plane. Inspect the pod events and lifecycle state. |
| Cross-zone network event triggers double-processing for a brief window | The self-fence was disabled (adapter.leasePartitionFenceMs: 0) during an asymmetric partition. | Re-arm adapter.leasePartitionFenceMs (default 90000) with a value above orchestrator.staleness.thresholdSeconds (default 75, so 75000 ms) plus rollout budget. |
| Scale Out deployment has one pod doing all the work | The external system's partition assignment is degenerate (fewer partitions than pods, or all partitions assigned to one consumer). | Inspect the external system's distribution. Add partitions or rebalance consumers. This is a protocol-layer concern, not a platform concern. |
| Orchestrator restart causes every Single Writer pod to self-fence simultaneously | adapter.leasePartitionFenceMs is set below orchestrator.staleness.thresholdSeconds (default 75, so 75000 ms) plus rollout duration. | Raise it above the staleness threshold plus the observed rollout duration, or set it to 0 if asymmetric partition is not a real concern for this deployment. |
| Attempting to add a second adapter to an existing Single Writer adapter's configuration is rejected with a 409 | Single Writer adapters are exclusive per configuration; HA is achieved by adding pods, not by adding adapter rows. | Increase replicaCount instead, or recreate the cluster as Scale Out if concurrent multi-writer is what you actually need. |
Monitoring signals
| Signal | What it tells you | Alert when |
|---|---|---|
| Count of active pods per adapter | Whether the topology is healthy | For Single Writer: one is expected. Zero for more than one check-in interval indicates a failover failure; more than one points to a protocol-layer violation. For Scale Out: equal to replicaCount; lower values indicate pods are failing to join the pool. |
| Self-fence events | Whether the self-fence has engaged | Any self-fence during a planned orchestrator restart, or more than one per pod per day |
| Counterparty disconnect rate | Whether the upstream is seeing the pool as stable | Sustained disconnects not correlated with planned maintenance |
| Consumer group rebalance rate (for natively partitioned integrations) | Stability of the Scale Out pool | Frequent rebalances indicate pod churn; investigate pod lifecycle events |