Resilience and Backpressure

This page is for site reliability engineers. It explains how the platform absorbs infrastructure faults during a downstream outage, how backpressure flows upstream from a slow target back to the source, and what to do when an alert fires. It is written in infrastructure terms (network timeouts, broker availability, socket backpressure) rather than in programming terms, because the operator-facing contract is infrastructure-level.

The shock absorber

The pipeline is the shock absorber between a source system that produces work at the rate the source wants and a target system that accepts work at whatever rate the target is currently capable of. Most of the time those rates are aligned. When they drift, the pipeline has to choose between two bad options: drop the excess, or fill memory until the pod is killed. Neither is acceptable, so instead the pipeline makes the source slow down to match the target. That is backpressure.

Backpressure is a layered story, not a single mechanism.

1. Per-message flow control at the egress boundary. The pipeline processes one message at a time per worker, and a worker does not pick up the next message until the target has accepted the current one. When the target slows down, workers linger on each message longer, and the pipeline as a whole accepts fewer new messages per second. That reduction flows backward.
2. Bounded internal queue. The pipeline's in-pod queue has a fixed, configurable size. When it fills, the adapter stops pulling new messages off the source and lets the source's own protocol apply its native stall: a queue consumer pauses polling, a TCP-based session stops reading its receive buffer and signals a zero-window condition to the sender.
3. Circuit breaker. If the target remains broken for long enough that the bounded queue fills entirely or fills repeatedly, the breaker trips and the pipeline halts the source outright. This is a deliberate, loud, operator-visible signal that something downstream needs attention.

The first two layers are transparent to most operators most of the time. The third is the one you monitor and respond to.

The circuit breaker

Why the breaker exists

If the only defense against a downstream outage is the bounded queue, then a long outage eventually fills the queue, stalls the source's protocol, and piles up work at the source's own buffer. Depending on the source protocol, that pile grows on a counterparty's disk, on a broker's partitions, or on a network device's queue. None of these are the adapter's problem to fix, but all of them get worse the longer the pipeline pretends the target is healthy. The breaker's job is to stop pretending.

A route is a single configured path through the pipeline: one source connector feeding messages through its processing chain to one target. A pipeline can carry several routes, and the breaker tracks each one independently.

When the target has failed enough times in a row to be declared unhealthy, the breaker halts the source for that route. The source's own backpressure mechanisms kick in immediately: the counterparty sees the stall, the broker's consumer group rebalances without this adapter, the load balancer routes incoming requests elsewhere. The pipeline is now cleanly out of the data path and an operator can investigate without fighting for system resources.

The two operationally visible states

The breaker lives in one of two states that matter to an operator.

Closed is the normal operating state. Every message the pipeline produces for the target is sent. If the target accepts the message, the breaker stays closed and any previous failure count resets. If the target fails, the breaker classifies the failure (see below) and either counts it or ignores it.

Open is tripped. The pipeline has halted the source for that route and stopped sending to the target. The breaker stays in this state until an operator intervenes. It does not self-heal: because the source is halted, no further messages reach the breaker to probe the downstream, so there is no automatic path back to Closed. Recovery requires restarting the adapter, which reinitializes the pipeline and resets every breaker in the pool to Closed. The expected workflow is: verify the downstream is healthy, then restart the adapter.

Infrastructure faults versus per-message faults

The breaker is not a generic error detector. It only counts the kind of failure that indicates the target itself is broken. A bad payload on an otherwise healthy target is not the breaker's problem; that kind of failure is the route's error-handler chain's problem.

Failure category	Example causes	Who handles it
Infrastructure fault	Network timeout reaching the target. Connection refused. Broker unavailable. DNS resolution failure. Target returned a transport-level error.	The circuit breaker counts it. If the count hits the threshold, the breaker trips.
Per-message fault	Payload fails validation. Required field missing. Unsupported message type. Counterparty rejected an otherwise well-formed message on business grounds.	The route's configured error-handler chain (for example, a dead-letter queue) processes it. The breaker ignores it entirely.

The platform categorizes these two kinds of failure automatically. Operators do not maintain a list of exception class names or map error codes to categories. If the failure looks like "I could not reach the target" at the network or transport layer, it counts. If it looks like "the target rejected this specific message," it does not.

This is a deliberate usability choice. The prior generation of this feature required operators to enumerate every exception class that should trip the breaker, which turned the breaker into a programming exercise disguised as an operational setting. The current design keeps the contract at the infrastructure level, where operators actually work.

Configuration

The breaker exposes two operationally meaningful settings.

Setting	Default	What it does
enabled	true	Master switch. When false, the breaker is bypassed entirely: every failure, infrastructure or otherwise, flows to the route's error-handler chain.
failure-threshold	5	Number of consecutive infrastructure faults that trip the breaker.

These settings live in the adapter's application configuration under the pipeline.circuit-breaker prefix and apply to every route on the adapter pod. They are not currently surfaced as dedicated Helm chart values: the defaults shown ship baked into the adapter image, and you override them per deployment through the PIPELINE_CIRCUIT_BREAKER_ENABLED and PIPELINE_CIRCUIT_BREAKER_FAILURE_THRESHOLD environment variables. The in-pod pipeline sizing knobs that the chart does expose are separate and live under adapter.pipeline (see Scaling).

Two canonical patterns.

Production default. Enabled, five faults to trip. Appropriate for most routes against reliable targets.

pipeline:
  circuit-breaker:
    enabled: true
    failure-threshold: 5

Aggressive halt for sensitive integrations. One fault and the breaker trips. Use this when any sustained downstream fault is an operational event, and the cost of halting the source is lower than the cost of sending even one more message into a broken target.

pipeline:
  circuit-breaker:
    enabled: true
    failure-threshold: 1

Circuit breaker versus dead-letter queue

These two mechanisms are often conflated. They are not interchangeable, and understanding the boundary is essential to reasoning about an incident.

Characteristic	Circuit breaker	Dead-letter queue
What it catches	Infrastructure faults: the target system is unreachable, slow, or broken.	Per-message faults: a specific message cannot be processed.
Blast radius	Route-wide. Halts the source so no further traffic enters the pipeline.	Per message. Other messages continue through the pipeline normally.
Intended duration	Until the target recovers, an operator verifies, then restarts the adapter to clear the halt.	Until the dead-letter record is inspected and either replayed or discarded.
Operator action	Diagnose the downstream outage. Restart the adapter once the target is healthy.	Review the dead-letter record. Fix the payload or the producing system. Replay if appropriate.
Configuration location	Adapter-level, under the pipeline.circuit-breaker prefix in the adapter's application configuration; applies to every route on the pod. See Configuration above.	Per-route error handler (see the relevant plugin guide).

An incident almost never requires both at the same time. If infrastructure is on fire, the breaker fires and the route is halted; the dead-letter queue is irrelevant because nothing is getting to the target anyway. If a few messages are malformed, the dead-letter queue absorbs them and the breaker stays closed because the target is healthy. The two mechanisms share no state and have no retry coordination; they are defenses against different kinds of failure.

Backpressure to the source

The moment the breaker trips, the pipeline stops observing the source connector for that route. The source's own protocol-layer backpressure takes over. The exact mechanism depends on the source.

Source type	What backpressure looks like
Persistent-connection session protocols (stateful, sequence-tracked)	The adapter stops reading the socket. The receive buffer fills. The TCP stack signals a zero-window condition to the peer, stalling further sends at the transport layer. No messages are lost: whatever has been acknowledged by the counterparty stays acknowledged, and whatever has not is still queued on the counterparty's side.
Queue-consumer protocols (partitioned, offset-tracked)	The adapter stops polling. The broker notices the missed heartbeats and rebalances partitions to other consumers in the group. No offsets are committed for unprocessed messages, so the replacement consumer reads them from where this consumer stopped.
Pull-based request-response protocols	The adapter stops issuing requests. The remote system sees a gap. Scheduled rechecks pick up the work when the adapter is restarted.
Push-based request-response protocols (HTTP inbound, webhooks)	The adapter's readiness probe flips to not-ready. Upstream load balancers stop routing new requests to this pod. Other pods in the pool continue serving if they exist.

In every case, the source protocol's own backpressure is what provides the stall. The pipeline does not invent a new stall mechanism; it delegates to the protocol that was already designed to handle this.

Drain budget

Separate from the breaker, a second configuration governs graceful shutdown.

Setting	Default	What it does
pipeline.drain.timeout-ms	30000	How long the pipeline waits for inflight messages to complete during shutdown before it forces termination.

Graceful shutdown is triggered whenever a pod receives its termination signal: planned pod eviction, rolling upgrades, or an operator-initiated disable from the Portal. At shutdown, the source stops feeding new messages, inflight messages continue toward their targets, and each message is acknowledged back to the source only after the target confirms acceptance. The drain budget is the upper bound on how long this phase is allowed to take. Raise it in tandem with the container platform's termination grace period whenever the downstream has high tail latency.

Health signal

The platform's health endpoint, /actuator/health on the adapter's management port, reports a per-route view of every breaker under the pipelineCircuitBreakerHealth component.

{
  "status": "UP",
  "components": {
    "pipelineCircuitBreakerHealth": {
      "status": "UP",
      "details": {
        "enabled": true,
        "failureThreshold": 5,
        "resetTimeoutMs": 30000,
        "openRoutes": 0,
        "halfOpenRoutes": 0,
        "totalRoutes": 3,
        "breakers": {
          "route-a": { "state": "CLOSED", "failureCount": 0 },
          "route-b": { "state": "CLOSED", "failureCount": 0 },
          "route-c": { "state": "CLOSED", "failureCount": 0 }
        }
      }
    }
  }
}

Overall status transitions to DOWN while any route is in OPEN. Health probes on the cluster platform can use this directly; paging rules can alert on it. While a route is halted the source is detached, so the reset timer never fires on its own; recovery is operator-driven (restart the adapter).

Field	Meaning	Alert when
openRoutes	Number of routes whose breaker has tripped	>= 1. Every trip requires operator action.
failureCount per route	Failures accumulated in the current closed window	Raise an informational alert at fifty percent of the threshold; raise a paging alert at the threshold itself.

Runbook: the circuit breaker health check is failing

Follow this sequence when the health endpoint reports one or more open routes, or when a paging alert fires on openRoutes >= 1.

Step 1. Identify which routes are affected

Read the health endpoint at /actuator/health on the adapter's management port (consistent with Metrics). Under the pipelineCircuitBreakerHealth component, the breakers detail lists every route and its state. Note every route whose state is OPEN. If more than one route is affected and they share a downstream target, treat the event as a single downstream outage, not as independent route failures.

Step 2. Confirm the downstream is unhealthy

An open breaker is a diagnosis, not a cause. The breaker is signaling that the target refused or timed out on requests. Verify this independently by inspecting the downstream system itself. Check broker health, reachability of the target endpoint, any recent deployments or network changes in the downstream's environment.

Downstream type	First check
Message broker	Broker cluster health dashboard. Broker logs for rejected or timed-out produce calls from this adapter.
Network endpoint	Reachability from the adapter pod. Recent changes to network policy, firewall, or DNS.
Gateway or load balancer	Upstream component health. Backend pool status if the gateway fronts multiple instances.

Step 3. Read the adapter's own event log

The pipeline records a structured event every time a breaker trips. Find the event, confirm the timestamp matches the incident, and read the route identifier and source connector from the event details. This is the definitive record of what the pipeline observed.

Step 4. Wait for the downstream to recover

Leave the route halted until the downstream is verified healthy. The breaker is preventing a pile-up against a broken target. Do not restart the adapter while the downstream is still unhealthy: the breaker will simply trip again, and you will have used your restart budget without making progress.

Step 5. Restart the adapter

Once the downstream is confirmed healthy, restart the adapter from the Portal. Restart discards the current breaker state and reinitializes the pipeline: every route begins from Closed, and traffic resumes through the restored target. Verify by watching the health endpoint transition back to UP with zero open routes.

Step 6. Postmortem

Every breaker trip in production is worth a brief postmortem. Record the downstream cause, the elapsed time from trip to restart, and whether the source applied backpressure cleanly. Adjust failure-threshold if the defaults did not fit the failure profile of this target.

Troubleshooting

Symptom	Likely cause	Action
Breaker is open but downstream systems report healthy	The failure threshold was reached on a transient fault that has since cleared.	Confirm the downstream is healthy, then restart the adapter to clear the halted state.
Breaker never trips despite obvious downstream outage	The downstream is returning a fault that the platform is classifying as per-message rather than infrastructure (for example, an application-level error response that the connector does not raise as an infrastructure-level failure).	Inspect the error stream and, in the relevant plugin, have the connector raise an infrastructure-level failure for this condition.
Per-message faults are tripping the breaker	A plugin is raising infrastructure-typed failures for per-message conditions.	Work with the plugin maintainer to correct the classification. Infrastructure-typed failures must only be raised when the target itself is unreachable.
Adapter restarts but the breaker trips again shortly after	The downstream is still unhealthy. Restart alone does not fix the underlying infrastructure problem.	Leave the route halted and resolve the downstream cause before the next restart.
Drain budget exhausted during graceful shutdown	The drain budget is shorter than the target's tail latency for accepting a message.	Raise pipeline.drain.timeout-ms. Raise the container platform's termination grace period in tandem.
Pod forcibly terminated during shutdown with inflight messages lost	Container platform termination grace period is shorter than the drain budget.	Raise the termination grace period above the drain budget with headroom for the remaining shutdown phases.

Monitoring signals

Signal	What it tells you	Alert when
Per-route breaker state	Whether any route is halted	Any route in OPEN. Every trip requires operator action.
Rate of infrastructure-classified failures	Trajectory toward a trip	Sustained non-zero rate without corresponding successes
Drain budget exhausted events	Shutdowns that could not complete within budget	Any occurrence. Graceful shutdowns should almost never exhaust the budget.
Circuit-breaker-open events	Fleet-wide infrastructure health	More than one per route per day, or correlated opens across multiple routes in the same window

See also

• Deployment Topologies
• Scaling
• Troubleshooting