Upgrades
This runbook takes a running deployment from its current version to a new one with no downtime. Work through the phases in order; each ends with an exit check, and you should not move on until it passes. Read "Before you start" first: the no-data-loss guarantee holds only when the conditions there are met, and if they cannot be met you use the maintenance-window path at the end of this page instead.
Before you start
A rolling upgrade loses no inflight message as pods rotate, provided all of the following hold. Confirm each before you begin.
| Condition | How to confirm |
|---|---|
| The migration note for the target release describes the path from the version you are running. | Compare the running and target versions; read the note for every release in between. |
| The adapter pool has enough pods to hand off leases during rotation: at least two in Single Writer, at least two plus one headroom in sharded pools. | Check adapter.replicaCount against your topology. See Deployment Topologies. |
| The termination grace period exceeds the full shutdown budget: the pipeline drain timeout, plus the preStop sleep, plus headroom for connection close and the final flush. | The chart defaults (60s grace, 30s drain, 10s preStop) already satisfy this. If you raised adapter.pipeline.drainTimeoutMs, raise adapter.terminationGracePeriodSeconds with it. |
adapter.pipeline.drainTimeoutMs is at least the downstream target's worst-case acknowledgement time. | Set it from the slowest target's tail latency (cross-region brokers, slow third-party APIs). |
| No schema change in the release needs manual data migration. | Column and index additions are automatic; any manual step is called out in that release's migration note. |
If any condition cannot be met, the rolling upgrade still completes, but the no-loss guarantee does not apply. Use the maintenance-window path instead.
Phase 1: Plan the upgrade
Read the migration note Connamara ships with each release, covering every release between the one you are running and the target. Look for:
- Changed Helm value names (renames are always documented).
- Manual steps (rare; always explicitly marked).
- Schema-migration notes (usually automatic, occasionally requiring operator involvement).
If the jump spans more than one version increment, upgrade one version at a time unless the combined path is explicitly documented as supported.
Exit check: you have read the notes for every intervening version, every "Before you start" condition holds, and no manual migration step is outstanding. If a condition fails, switch to the maintenance-window path.
Phase 2: Apply the upgrade
Update the image tags in your values file:
orchestrator:
image:
tag: "1.5.0"
adapter:
image:
tag: "1.5.0"
portal:
image:
tag: "1.5.0"
A helm upgrade does more than rotate pods. A pre-upgrade migration Job (app.kubernetes.io/component=migrator) applies any schema changes before the new pods start, and a post-upgrade installer Job (app.kubernetes.io/component=installer) re-uploads plugin JARs and re-applies your bundles once the new pods are healthy.
Because the installer re-runs on every upgrade, the values file you pass with -f must still carry installer.customBundleUrls. If your original install set it but the upgrade's values file omits it, the installer re-applies only the base distribution, and your custom bundles and plugin selections silently drop. For a narrow patch, prefer --reuse-values (below), which carries the existing installer settings forward.
Run the Helm upgrade:
helm upgrade conncentric oci://<your-registry>/conncentric/charts/conncentric --version <release-tag> \
-n conncentric \
-f my-values.yaml \
--atomic \
--timeout 5m
--atomic rolls back automatically if any pod fails to become healthy within the timeout. Size the timeout to at least the termination grace period times the pool size, to leave room for sequential pod rotation.
To patch only one component, upgrade it in isolation and keep every other setting from the current release:
helm upgrade conncentric oci://<your-registry>/conncentric/charts/conncentric --version <release-tag> \
-n conncentric \
--set orchestrator.image.tag=1.5.1 \
--reuse-values
--reuse-values is the safest pattern for a narrow patch, because it eliminates the chance of a mistake in a values file affecting components you are not upgrading.
Exit check: helm reports the release upgraded, and rollout completes for all three deployments:
kubectl rollout status deployment/conncentric-orchestrator -n conncentric
kubectl rollout status deployment/conncentric-adapter -n conncentric
kubectl rollout status deployment/conncentric-portal -n conncentric
Phase 3: Verify
Open the Portal and confirm every adapter that was Running before the upgrade has returned to Running. The Portal's per-adapter event log is the authoritative record of what the platform did to each adapter and when (see Event Log Reference); use it to confirm a clean lease transition for every pod in the pool. Counterparties connected through persistent-session protocols reconnect automatically once the pod owning their session has been replaced.
A clean lease transition reads, per rotated configuration, as an Upgrade shutdown initiated entry followed by Lease released on the outgoing pod, and a matching Lease acquired entry on the pod that took over. A Fencing triggered entry, or a claim that lands only after the staleness window rather than immediately, means the cooperative handoff did not happen and the slower fallback path ran instead. Investigate that configuration before you continue.
An upgrade also runs two short-lived Jobs, identifiable by component label: the pre-upgrade migration Job (app.kubernetes.io/component=migrator) and the post-upgrade installer Job (app.kubernetes.io/component=installer). Both delete themselves on success, so read their logs while the upgrade is in flight, or after a failed run when the Job is retained for inspection:
kubectl logs -l app.kubernetes.io/component=migrator -n conncentric
kubectl logs -l app.kubernetes.io/component=installer -n conncentric
A clean installer run ends with Installer finished successfully.
Exit check: every adapter that was Running before the upgrade is Running again, and for each rotated pod the event log shows the Upgrade shutdown initiated, Lease released, Lease acquired sequence with no Fencing triggered entry. The upgrade is complete.
What happens during a rolling upgrade
This section describes what the platform does to each pod during the rollout, so you can interpret what you see in Phase 3.
The sequence breaks down into five stages. Each has a specific purpose and a specific configuration knob. These stages are the anatomy of a single pod's shutdown; they are numbered separately from the runbook's Phase 1 through Phase 3 above.
Stage 1. Pre-stop network un-publishing
When the cluster platform decides to replace a pod, it runs the pod's pre-stop hook before sending the termination signal. The hook marks the pod as not ready and then sleeps for a configured delay window.
The sleep matters because the readiness change has to propagate through the routing layer. Without the delay, the pod would start tearing itself down while the cluster's Service object was still advertising its endpoint, so inbound traffic would keep arriving at a pod that is already shutting down. The sleep gives the endpoint controller time to remove the pod from every Service and load balancer that references it.
This stage protects inbound traffic, specifically any load-balanced inbound integration (request-response endpoints, webhook receivers) from receiving requests that would be cut off mid-processing.
Stage 2. Pipeline drain
After the pre-stop window elapses, the container platform sends the termination signal, and the pod begins its internal shutdown. The first substantive step is the pipeline drain.
The pipeline stops consuming new messages from its sources. Messages already inside the pipeline continue toward their targets. The pod waits for each inflight message to be accepted by its target before releasing the source-side acknowledgement. The maximum duration the pod waits for this stage is adapter.pipeline.drainTimeoutMs.
| Setting | Default | When to adjust |
|---|---|---|
adapter.pipeline.drainTimeoutMs | Thirty seconds | Raise for deployments with high downstream tail latency (for example cross-region brokers or slow third-party APIs). Lower for latency-sensitive deployments against local targets. |
Raise the container platform's termination grace period in tandem. The grace period must exceed the drain timeout plus the preStop sleep plus headroom for the remaining shutdown stages (connection close and the final flush). The chart defaults already hold this relationship: a thirty-second drain budget and a ten-second preStop sleep paired with a sixty-second termination grace period.
Stage 3. Downstream connection closure
With the pipeline drained, the pod closes its connections to downstream target systems cleanly. The behavior depends on the target's protocol.
For queue-based targets, the producer is closed with a grace period that lets any remaining in-flight sends be acknowledged by the broker. For persistent-connection session protocols, the session is disconnected according to the protocol's close semantics.
Persistent-session close behavior (protocol specific)
For protocols that define a close handshake (for example standard financial messaging protocols with an explicit session-logout message), clean closure ideally involves sending the close message, waiting for the counterparty's acknowledgement, and only then closing the socket.
The platform sends the logout message and closes the socket without waiting for the counterparty's acknowledgement round trip. Sessions are still closed cleanly from the platform's perspective (no pending writes are lost), but the counterparty's log reflects the asymmetric close. For operators running integrations where counterparty logs are reviewed, this is the explanation.
Stage 4. Final persistence flush
The pod performs a final synchronous flush of any session state held in memory on a periodic sync cadence. For stateful session protocols this is the sequence-number checkpoint; for partitioned consumers it is the last offset commit. After this stage, the database reflects exactly what the pod has observed.
This is the stage that makes the subsequent lease handoff safe: whichever pod claims the lease next finds durable state that matches what the counterparty or external system has already seen.
Stage 5. Cooperative lease release
After durable state is flushed (Stage 4), and before the pod exits, the outgoing pod sends an explicit handoff signal to the orchestrator. The orchestrator frees the lease on the spot: the active-instance record is cleared and the configuration becomes claimable. A standby pod picks up the lease on its next heartbeat cycle and proceeds with startup.
This handoff fires for every graceful shutdown path, including a standard Kubernetes rolling upgrade, a Portal-initiated shutdown, and an orchestrator-signaled upgrade. As a result, rolling upgrades do not pay the orchestrator's staleness window as an availability tax: that window is reserved for the case where the pod does not get the chance to signal, typically a hard crash or node loss.
The expected total interruption per counterparty during a standard rolling upgrade, for a persistent session, is the sum of: drain time (bounded by the drain timeout), the time for a standby pod to receive the lease and complete its own startup, and the protocol's own reconnect cycle.
Staleness-window fallback
Cooperative release requires the outgoing pod to be running when shutdown is triggered and the orchestrator to be reachable at the moment the handoff is sent. When either is absent (a hard crash, a kernel OOM kill, a node failure, or an orchestrator outage concurrent with the pod's exit), the lease is not released cooperatively. The orchestrator's staleness detector eventually notices the absence of check-ins from the exited pod and frees the lease itself, at which point a standby claims it.
Every cluster therefore has two failover paths: the fast cooperative path for graceful shutdowns, and the staleness-window fallback for ungraceful ones. The staleness window, tuned through orchestrator.staleness.thresholdSeconds, determines the worst-case failover delay for the fallback path.
Recovering from a bad upgrade
The platform is designed to be rolled forward, not rolled back. The schema convention is expand-and-contract: a version increment adds columns and indexes but never renames or drops existing ones in the same release, and the previous version ignores columns it does not recognize. Because a new release never destroys the old schema, you recover from a misbehaving upgrade by applying a corrected version (forward-fix), not by reverting the running one.
helm rollback is reserved for one narrow case: clearing a release stuck in pending-install or pending-upgrade (for example after an interrupted helm upgrade). It is not a version-reversion strategy. Reverting application code onto an already-migrated schema is neither supported nor necessary under expand-and-contract.
Releases that cannot follow expand-and-contract (for example a forced destructive migration) are called out explicitly in the migration note together with their recovery procedure. Those cases require a data restore, not a Helm rollback.
The maintenance-window path
Live rolling upgrades are the default. Some organizations prefer a controlled window for operational risk or regulatory alignment, and it is the required path when a "Before you start" condition cannot be met. It sacrifices zero-downtime but gives the operator full control.
- Disable every adapter in the Portal. Each adapter stops serving traffic and drains inflight work through its usual graceful-shutdown stages.
- Wait for every adapter to report inactive.
- Run the upgrade.
- Verify the new version is running in each component.
- Re-enable adapters. Each resumes from its durable state.
Use this path when:
- The upgrade spans more than one minor version and has not been validated end-to-end as a single rolling transition.
- Your compliance posture requires a documented, operator-supervised transition.
- A manual migration step is required as part of the upgrade.
- You have not yet configured the termination grace period appropriately for the drain budget.
Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| Pods forcibly terminated during shutdown with inflight messages lost | Termination grace period is shorter than the full shutdown budget. | Raise the termination grace period above the drain budget plus the preStop sleep plus headroom (the chart default is 60s for a 30s drain and a 10s preStop). |
| Counterparty sees sequence-number conflicts after an upgrade | A pod was killed without completing its final persistence flush, and a standby took over from stale database state. | Follow the reconciliation runbook in the protocol's durability guide. Verify the termination grace period is long enough to cover every shutdown stage. |
| Counterparty logs "unexpected disconnect" during a planned rollout | The platform closes the connection without waiting for a logout acknowledgement (see the note above); the counterparty observes that asymmetric close. | No action required on the platform side. The data path is unaffected. |
Rolling upgrade stalls with --atomic timeout triggered | Pod startup is slow (image pull, slow dependency readiness), exceeding the Helm timeout. | Raise the --timeout value, or investigate image pull and startup dependencies. |
| Configuration state looks inconsistent after a version change | A configuration change was made through the Portal during the upgrade window; configuration lives in the database, independent of the deployed code version. | Reconcile configuration state in the Portal, and suspend Portal configuration changes during upgrade windows. |
| New pods come up but never claim leases | Configuration fails to load on the new version, or the new pods cannot reach the orchestrator. | Inspect the new pod's startup events for configuration errors. Verify orchestrator reachability from the new pods. |
| Custom bundles or plugin selections disappear after an upgrade | The values file used for the upgrade omitted installer.customBundleUrls, so the re-run installer applied only the base distribution. | Restore installer.customBundleUrls to the values file (or use --reuse-values for a narrow patch), then re-run helm upgrade. Confirm with kubectl logs -l app.kubernetes.io/component=installer -n conncentric. |
Upgrade aborts before pods roll, or the release sticks in pending-upgrade | The pre-upgrade migration Job failed, so Helm stopped before applying the new pods. | Read the migration Job log: kubectl logs -l app.kubernetes.io/component=migrator -n conncentric. Resolve the schema error, then re-run the upgrade. |
Monitoring signals
Build these alerts from the metrics documented in Metrics & Monitoring. Not every rollout signal has a dedicated platform metric; where none exists, the "Where to observe" column names the kubectl command or Portal view that carries the same information.
| Signal | What it tells you | Alert when | Where to observe |
|---|---|---|---|
| Pod rotation duration during the rollout | Whether each pod is completing its shutdown stages within budget | Any pod rotation exceeding the termination grace period | No platform metric. Watch kubectl rollout status and pod lifecycle events (kubectl describe pod); a pod force-terminated at grace-period expiry overran its budget. |
| Inflight message count during drain | Whether the drain budget is adequate | Non-zero at end of drain window | No dedicated metric. A pod force-killed at grace-period expiry (the row above) is the signal that drain did not finish; the Portal event log for that adapter corroborates it. |
| Time from outgoing-pod exit to standby claim | Whether failover is completing in the expected amount of time | Exceeds the sum of the staleness threshold plus a small margin (the fallback path ran instead of the cooperative one) | adapter_lifecycle_state (see Metrics & Monitoring): the standby moves through state="provisioning" to state="active". The Portal event log records the matching Lease acquired entry. |
| Counterparty reconnect duration after pod rotation | Observed end-user visibility of the rollout | Exceeds the target's documented reconnect tolerance | No platform metric (protocol-specific). Observe via the protocol plugin's own reference metrics and the counterparty's session logs. |