Disaster Recovery
This page is strictly about disaster recovery: catastrophic data loss, full-zone or full-region outages, and the procedures for rebuilding the platform from scratch onto new infrastructure. It is distinct from routine failover and high availability. Local pod failover, orchestrator leases, and adapter-pool fencing are covered in Deployment Topologies.
Use this page when the thing that has failed is large enough that healthy adapter pods cannot recover on their own: the cluster is gone, the availability zone is offline, the database region has been lost, or backups need to be restored.
Jump straight to The three failure classes. It routes you to the matching procedure based on what actually failed, so you land on your steps before reading the concepts. The sections in between explain the state model and recovery objectives those procedures rely on.
What state the platform keeps
The platform's durable state is consolidated in the relational database that every component shares. Pods are stateless with respect to disaster recovery: rebuilding them from container images restores no useful information on its own. The database is the source of truth.
| Data | Where it lives | How to back it up |
|---|---|---|
| Adapter, route, and connector configurations | Relational database | Provider database backups |
| Artifacts (data dictionaries, schemas, transformer resources) | Relational database | Provider database backups |
| Installed plugin metadata | Relational database | Provider database backups |
| Session state for session-based protocols (sequence numbers, last-flushed positions, message log) | Relational database | Provider database backups |
| Deployment values (Helm values, environment-specific overrides) | Your source control system | Keep in Git with reviewed changes |
| Container images | Your container registry | Registry provider backups, or mirror to a second registry |
The contract is: if you can restore the database and retrieve the container images, you can restore the platform on any new infrastructure that meets the runtime prerequisites.
Recovery objectives
Two industry-standard metrics govern disaster recovery planning.
| Metric | Definition | What it depends on |
|---|---|---|
| Recovery Point Objective (RPO) | Maximum data loss measured in wall-clock time, from the moment of failure back to the last recoverable state. | The frequency and durability of your database backups plus the lag of any cross-region replication. |
| Recovery Time Objective (RTO) | Maximum time from the start of a disaster event to the platform being operational again on recovered infrastructure. | The combined duration of provisioning new infrastructure, restoring the database, and bringing pods online. |
The platform does not prescribe a specific RPO or RTO. Both are a function of your database provider's backup cadence, your cluster provisioning speed, and how much investment your organization has made in pre-provisioned warm standbys. The recommended configurations below are patterns: match them to your own targets.
Recommended patterns
Baseline, single-region. A single database backed by the cloud provider's point-in-time recovery. Recovery uses the backup to restore state and the deployment values from source control to re-stand up the platform on fresh infrastructure.
| Target | Typical baseline |
|---|---|
| RPO | Bounded by provider point-in-time recovery granularity (typically small) |
| RTO | Bounded by new-cluster provisioning time plus database restore duration |
Hot standby, multi-region. A standby database in a second region, continuously replicated from the primary. A standby cluster provisioned but with adapter pods scaled to zero. Failover promotes the standby database and scales adapter pods up.
| Target | Typical baseline |
|---|---|
| RPO | Bounded by cross-region replication lag |
| RTO | Bounded by database promotion plus scale-up time |
Choose between these based on the cost of downtime for your integrations. Many financial integrations have regulated RTO targets; cross-check the pattern against your compliance commitments before committing.
Example starting point
The values below are an illustration to adapt to your own compliance obligations, not a platform commitment. Use them as a first draft and tighten or relax each one against your regulator's requirements.
| Parameter | Example starting value | Tune against |
|---|---|---|
| Backup cadence | Provider point-in-time recovery enabled continuously, plus a daily full snapshot retained for 30 days | Your retention and audit obligations |
| RPO | Bounded by point-in-time-recovery granularity for single-region (commonly on the order of a few minutes); bounded by replication lag for a cross-region hot standby (typically tighter) | The maximum data loss your integrations can absorb |
| RTO | New-cluster rebuild plus database restore for single-region; standby promotion plus adapter scale-up for hot standby (typically much shorter) | The maximum downtime your integrations can absorb |
Treat your first drill against real infrastructure as authoritative: measure the RPO and RTO you actually achieve and replace these example values with the observed timings.
Durability posture
How durably an individual message is retained depends on the integration's protocol. Session-based protocols (stateful, sequence-tracked, such as financial messaging sessions) persist each message so a failover resumes from exactly where it left off. Broker-based protocols rely on the broker as the system of record together with committed offsets, so the broker's own replication governs their durability and the platform keeps no separate per-message copy. What durabilityProfile controls is how far the platform's own durable writes (session state and the message archive) reach before an acknowledgement returns, trading protection against a full availability-zone loss against acknowledgement latency. It takes effect only when the database is configured for synchronous cross-zone commit.
durabilityProfile is a top-level Helm value, set alongside other cluster-wide settings rather than nested under adapter::
# values.yaml (top level, not under adapter:)
durabilityProfile: durability-first # or: latency-first
| Posture | What it guarantees | When to choose it |
|---|---|---|
Maximum durability (durability-first, default) | A write is acknowledged only after it is durably stored in a way that survives the loss of an entire availability zone, so no write the platform has acknowledged is lost even if a whole zone fails. | Compliance and regulatory integrations that must not lose durably recorded state, including during a zone or regional event, where acknowledgement latency is not the primary concern. |
Latency-optimized (latency-first) | A write is acknowledged once it is durably stored within its primary zone. Routine failures (a single pod or node) lose nothing. A full-zone disaster is recovered from a replicated copy in another region, leaving a recovery-point window of the most recent writes to reconcile. | Latency-sensitive integrations where faster acknowledgement matters and a full-zone disaster is an accepted, recovery-handled event rather than a zero-loss requirement. |
The latency-first posture requires the database to be co-located with the adapters in a single availability zone, with the disaster-recovery copy in a second region (see Recommended patterns above). It does not change routine failover: a single pod or node loss is recovered with no loss of committed data under either posture.
At startup the platform inspects the database it is connected to and reports the durability posture (the effective durabilityProfile) it can actually deliver. If the requested posture cannot be honored by that database's configuration, it logs a clear warning rather than silently behaving differently. This matters most when you bring your own database: some managed-database configurations confirm every write across zones regardless of the requested posture, in which case latency-first has no effect and the startup log says so.
The three failure classes
Disaster recovery procedures depend on what exactly failed. The platform's behavior differs across three escalating blast radii.
Cluster or zone loss means the Kubernetes cluster itself is unavailable: the orchestrator and adapter pods cannot be reached, but the database survives. The procedure is to provision a new cluster and redeploy; adapter pods resume with existing lease state from the database.
Database failure means the database is unreachable but the cluster survives. Adapter pods continue operating on their last-valid lease for as long as they can remain healthy. Do not pursue disaster recovery procedures during a transient database outage: the platform is designed to tolerate brief unreachability. If the outage is extended, use the provider's managed failover if available, or restore from backup onto a replacement database and update connection strings.
Full site loss is both of the above simultaneously: cluster and database gone. This is the scenario the multi-region warm-standby pattern is designed for.
Procedure: full cluster loss with intact database
Use this procedure when the cluster is lost but the database in its original region is still reachable.
Step 1. Provision replacement infrastructure
Stand up a new Kubernetes cluster matching the prerequisites of the platform version you intend to run. See Prerequisites for the required runtime surface.
Step 2. Restore deployment values
From source control, retrieve the Helm values file for the environment that was lost. Update connection strings only if the database endpoint has changed (for example, if you have been using an IP address rather than a DNS name).
Step 3. Deploy the platform
Deploy using the saved Helm values. The platform pods start, connect to the existing database, and the adapter pool begins claiming leases. All configurations, artifacts, routes, and session sequence state are already present because they live in the database.
Step 4. Verify
From the Portal, confirm that every adapter returns to its expected lifecycle state. Check the operational events log for any lease conflicts or claim failures. Counterparties reconnect using their normal reconnect logic; the platform resumes from the state that was in the database at the moment of the cluster loss.
Procedure: full site loss (database plus cluster)
Use this procedure when the database is also gone. Recovery requires restoring the database before the platform can be rebuilt.
Step 1. Provision a new database
Follow your database provider's restore procedure using the most recent backup (or the standby database, if a hot standby was pre-provisioned). Confirm the restore is complete, the endpoint is reachable, and any required network policies or firewall rules are in place.
Step 2. Provision replacement cluster infrastructure
As in the single-loss procedure above.
Step 3. Update deployment values
Update the database connection configuration in the Helm values to reference the new endpoint. If this is a cross-region failover, verify other endpoint-dependent values (identity provider, container registry, external secret stores) as well.
Step 4. Deploy the platform
Deploy using the updated values. Pods start, read configuration from the restored database, and the adapter pool begins claiming leases.
Step 5. Reconcile sequence state with counterparties
This step is the one that distinguishes disaster recovery from routine failover. Any counterparty that uses session-based sequence numbers (most financial messaging protocols) now has two views of the world: its own, advanced to wherever it was at the moment of the disaster, and the platform's, restored from whatever was in the last backup. If the backup is not fresh enough to match the counterparty's view, the session will fail to resume.
| Backup freshness relative to disaster | Likely outcome |
|---|---|
| Backup captured the last message before the disaster | Sessions resume cleanly on reconnect. |
| Backup lags the disaster by some amount | The counterparty's expected sequence is ahead of the platform's. The session will refuse logon until the numbers are reconciled. |
| Backup is significantly stale, or the counterparty has reset their side | Full sequence reset, typically coordinated with the counterparty's operations team. |
For the reconciliation procedure per session, see the protocol-specific operations guide (for example, Sequence number recovery in FIX Operations for the standard financial messaging protocol).
Planned maintenance windows
Planned maintenance is not disaster recovery, but it shares the same concern about bringing sessions back up cleanly. For a controlled, operator-supervised shutdown before invasive maintenance:
- Disable every adapter in the Portal. Each adapter stops serving traffic and drains inflight work through its usual graceful-shutdown phases.
- Wait for every adapter to report its operational state as inactive.
- Perform the maintenance.
- Re-enable the adapters. The platform resumes from the durable state that was persisted at shutdown.
This is usually unnecessary. Rolling upgrades documented in Upgrades are designed to be zero-downtime, and planned maintenance that does not require database downtime or cluster eviction can use the rolling upgrade path instead.
Monitoring signals
| Signal | What it tells you | Alert when |
|---|---|---|
| Last-successful database backup timestamp | How stale your recovery point is | Age exceeds your RPO target |
| Cross-region replication lag | How far behind your standby database is | Lag exceeds your RPO target |
| Database endpoint reachability from the cluster | Whether the platform's lifeline is available | Any sustained unreachability. Short blips are expected and tolerated; sustained unreachability is a disaster-recovery-class event. |
| Time from disaster declaration to platform operational | Actual achieved RTO for the event | Compare post-event against your RTO target and adjust procedure |
Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| After full cluster rebuild, adapter pods stay in a pre-active state and never claim leases | Database reachability from the new cluster is blocked; network policy or firewall has not been reapplied. | Verify database connectivity from a pod in the new cluster. Reapply network policies as needed. |
| After database restore, counterparty rejects logon with a sequence-too-low error | Backup lag: the counterparty saw messages past the restored sequence position. | Coordinate sequence reconciliation with the counterparty. See the protocol-specific durability guide. |
| Standby region takes over but sessions rebuild with outdated configuration | Configuration changes made after the last replication cycle did not reach the standby. | Reapply configuration from source control to the promoted region once operational. |
| Planned failover drill causes longer-than-expected downtime | RTO target is looser than the actual procedure achieves. | Treat the drill result as the authoritative RTO. Update recovery-plan documentation to match observed timings. |
| A disaster recovery procedure succeeds but some adapters remain disabled | The adapters were disabled before the disaster and their state was preserved by the backup. | Re-enable from the Portal. |