Production Deployment Runbook

This runbook is the end-to-end sequence for taking Conncentric from nothing to a running deployment with your first live session. Work through the phases in order. Each phase ends with an exit check; do not move on until it passes.

The per-topic pages linked below carry the full detail. This page is the checklist you execute on deployment day, with the gates and the common failure points called out.

Before you start

Connamara provides three things when your deployment is provisioned:

You receive	Purpose
Pull access to the platform container images	Mirror them into your own registry and point the chart at it. See Platform Images.
The Conncentric Helm chart	Contains `values.yaml` (production defaults). You layer your own overrides on top of it (see Phase 2).
Your plugin entitlements	Which protocol plugins your license covers (for example Essentials, FIX, Kafka). You enable these in `installer.plugins`.

You provide the infrastructure. Conncentric follows a Bring Your Own Infrastructure model: it runs inside your own Kubernetes cluster, against your own PostgreSQL database and identity provider.

Phase 1: Provision infrastructure

These four steps have no ordering dependency on each other, so run them in parallel.

Step	What to do	Detail
Kubernetes cluster	Confirm the cluster meets the supported version and has capacity for the control-plane and adapter pools.	Prerequisites, Compatibility
PostgreSQL	Provision the database, create the database and an application user, and record the host, name, and credentials.	Database
Identity provider	Register an OIDC application for the Portal. Record the issuer URI and client ID. Set the redirect URI to the Portal's external URL exactly, with no trailing slash.	Authentication
Networking	Open HTTPS ingress for the Portal and API. If any adapter runs in acceptor mode, open the acceptor TCP port range (default 6700 to 6750) and restrict it to known counterparty addresses.	Networking & Ingress

Portal hostname, certificate, and redirect URI (settle these up front). Choose the Portal's external hostname now (for example conncentric.example.com), provision its TLS certificate, and register that exact URL, with no trailing slash, as the OIDC redirect URI with your identity provider before you install. The hostname is baked into portal.auth.redirectUri and must match what the identity provider expects, so it cannot be decided after the fact. The one thing you cannot do yet is create the DNS record: your ingress or load balancer has no address until after the install, so you point DNS at it in Phase 3 once that address exists.

Exit check: the cluster is reachable with your tooling, the database accepts a connection from inside the cluster, the OIDC application is registered with the Portal hostname as its redirect URI, and your networking team has confirmed the ingress and the port range.

Phase 2: Configure Helm values

Keep your own overrides file (for example values.production.yaml) with only your environment's settings. You do not need a local copy of the chart's values.yaml: the chart carries the production defaults and applies them automatically, so you layer only your overrides on top at install time. A chart upgrade never overwrites your settings, because they live in your file, not in the chart.

Start your overrides file from this minimal skeleton and replace every placeholder:

global:
  imageRegistry: "<your-registry>"          # registry that hosts the platform images
  imagePullSecrets:                         # created below; omit only if nodes pull via IAM
    - conncentric-pull                      # a Secret name (string), not a { name: ... } object
# Pin every image to a released version, never "latest".
orchestrator:
  image: { tag: "<version>" }
adapter:
  image: { tag: "<version>" }
portal:
  image: { tag: "<version>" }
  auth:
    provider: "<oidc-provider>"             # must match security.provider
    authority: "<oidc-issuer-uri>"          # must match security.jwt.issuerUri
    clientId: "<portal-oidc-client-id>"
    redirectUri: "https://<your-portal-domain>"   # EXACT, no trailing slash
installer:
  image: { tag: "<version>" }
  plugins:                                  # enable only what your license covers
    essentials: true
    fix: false
    kafka: false
security:
  provider: "<oidc-provider>"               # cognito | okta | auth0 | entra
  jwt:
    issuerUri: "<oidc-issuer-uri>"
  internalApiKeyExistingSecret: "conncentric-internal-api-key"   # pre-created Secret (created below)
database:
  host: "<postgres-host>"
  name: "conncentric"
  auth:
    username: "<db-username>"
    existingSecret: "conncentric-db-credentials"   # pre-created Secret (created below)
durabilityProfile: durability-first         # see "Choosing a durability profile" below

Neither secret lives in your overrides file. The database password lives in the conncentric-db-credentials Secret, and the internal service-to-service key lives in the conncentric-internal-api-key Secret. You create both below, before you install, and the skeleton references them by name. Both are injected into the pods by reference and never appear as a plaintext pod env value. If you prefer, you can pass the internal key inline at install time instead (--set security.internalApiKey="$INTERNAL_API_KEY"), which the chart synthesizes into a Secret for you; pre-creating it keeps the key out of your values file and shell history.

The full values.yaml, with every default and inline note, ships inside the chart; run helm show values oci://<your-registry>/conncentric/charts/conncentric --version <release-tag> to print it. Each field above is described next, and the complete reference is in the Helm Reference.

Setting	Value
`global.imageRegistry`	The registry that hosts the platform images.
Image tags	Pin the `orchestrator`, `adapter`, `portal`, and `installer` image tags to a released version. Do not use `latest` in production.
`security.provider`, `security.jwt.issuerUri`	Your OIDC provider identifier and issuer URI.
`security.internalApiKeyExistingSecret`	The name of a Kubernetes Secret holding the internal service-to-service key under key `internal-api-key`. Generate the key with `openssl rand -hex 32` and create the Secret before installing.
`database.host`, `database.name`, `database.auth.username`	Your database connection details.
`database.auth.existingSecret`	The name of a Kubernetes Secret holding the database password. Create it before installing.
`portal.auth.*`	Provider, authority, client ID, and redirect URI for the Portal. `provider` must match `security.provider`, and `authority` must match `security.jwt.issuerUri`.
`installer.plugins.*`	Enable the plugins your license covers.
`durabilityProfile`	See the "choosing a durability profile" note below.

Create both Secrets before installing. The install references them by name, so they must exist first. Create the namespace, then the two Secrets:

kubectl create namespace conncentric

# Image pull secret. Needed unless the cluster pulls via node IAM (for example an
# IAM role attached to the node group). Referenced by global.imagePullSecrets above.
kubectl create secret docker-registry conncentric-pull \
  --docker-server=<your-registry> \
  --docker-username=<user> --docker-password=<pass> \
  --namespace conncentric

# Database credentials (keys: username, password)
kubectl create secret generic conncentric-db-credentials \
  --from-literal=username=<username> --from-literal=password=<password> \
  --namespace conncentric

# Internal service-to-service key (key: internal-api-key)
kubectl create secret generic conncentric-internal-api-key \
  --from-literal=internal-api-key="$(openssl rand -hex 32)" \
  --namespace conncentric

Choosing a durability profile

durabilityProfile sets how message-archive writes are committed. It changes behavior only when your database is multi-availability-zone synchronous (has a synchronous standby). Without one, both profiles commit to the primary only and behave identically, so the choice does not matter.

On a multi-availability-zone synchronous database:

Profile	Behavior
`durability-first` (default)	Inherits the synchronous cross-AZ commit: no loss of committed data on a full availability-zone outage (in-flight, not-yet-acknowledged messages are not covered), at higher commit latency.
`latency-first`	Commits to the primary only, without waiting for the standby: lower latency, but the most recent archive writes can be lost on a full primary or availability-zone loss. Pair it with your own disaster-recovery replica.

The complete configuration surface is in the Helm Reference.

Exit check: run the preflight script, which ships inside the chart. Pull and extract the chart, then run the script against your namespace and overrides file:

helm pull oci://<your-registry>/conncentric/charts/conncentric --version <release-tag> --untar
cd conncentric && bash preflight.sh -n conncentric -f ../values.production.yaml

Invoke it with bash as shown. Extracting a chart does not preserve the execute bit, so ./preflight.sh fails with a permission error.

The script defaults its chart path to the current directory, so run it from inside the extracted conncentric chart directory as shown (or pass -c <chart-path>). Running it from the parent directory makes its chart-render check false-fail because it cannot find the chart there.

It fails fast on missing tooling, an unreachable cluster, an out-of-support Kubernetes version, a missing database Secret, an unreachable OIDC issuer, or unreplaced placeholders. Proceed when it reports no errors.

Phase 3: Install and verify

Install with the pre-created Secrets and your overrides file. The chart is an OCI artifact in your own registry, so <your-registry> is the same value you set for global.imageRegistry:

helm install conncentric oci://<your-registry>/conncentric/charts/conncentric --version <release-tag> -n conncentric --create-namespace -f values.production.yaml

The chart applies changes in order:

A pre-install schema migration brings the database up to date. If it fails, the install stops before any application pod starts.
The control-plane (orchestrator) and data-plane (adapter) pods start.
A one-shot installer job applies the base distribution with your selected plugins, then any custom bundles you listed.

Once the install returns, read the address of your ingress or load balancer and create the DNS record for the Portal hostname you chose in Phase 1, pointing it at that address:

kubectl get ingress -n conncentric    # if you front the Portal with an Ingress
kubectl get svc -n conncentric         # if you expose it through a LoadBalancer Service

Until that DNS record resolves and the certificate is attached at the ingress, the Portal is not reachable in a browser and the sign-in check below will not pass.

Then verify. Watch the pods reach their expected state:

kubectl get pods -n conncentric

Check	Expected result
Pods	The orchestrator and portal pods reach a running state.
Installer	The installer job removes itself once it succeeds, so it is normal for `kubectl get jobs -n conncentric` to come back empty after a clean install. It is retained only when it fails, which is when its log is worth reading: `kubectl logs -l app.kubernetes.io/component=installer -n conncentric`. Confirm what it applied from the Portal, where the connectors and pipeline components it registered become available when you build your first adapter.
Adapter pods	Adapter pods that hold no lease report not ready. This is expected: an idle adapter is a standby, not a fault. See HA & Leasing.
Portal	You can open the Portal in your browser and sign in through your identity provider.

Exit check: you are signed in to the Portal through your identity provider. Deployment is complete.

Phase 4: Your first session

Adapters do not connect on their own. Every adapter is created disabled, and enabling it is an explicit action, so the platform never opens a session you did not intend.

In the Portal, create your first adapter. The Creating an Adapter guide walks through connector type, connection details, and the message pipeline.
Enable the adapter in the Portal. It claims a lease, and its pod becomes ready.
Confirm traffic on the adapter's monitoring view.

To roll out configurations you have already built in another environment, export them from the Portal, package them as a .zip bundle, host it at a URL your cluster can reach, set installer.customBundleUrls in your values, and run helm upgrade. See Custom Bundles & Extensibility.

Go-live readiness

Item	Confirm
Image tags	Pinned to a released version, not `latest`.
Secrets	The database password and the internal service-to-service key each live in a pre-created Kubernetes Secret, referenced by name, never in a values file.
Durability	`durabilityProfile` matches your PostgreSQL topology.
Backups	Database backups are configured and a restore has been tested. See HA & Disaster Recovery.
Monitoring	Metrics scraping is enabled and alerts are wired. See Metrics & Monitoring.
Networking	The acceptor port range is restricted to known counterparty addresses.
Upgrades	The rolling-upgrade procedure is understood. See Upgrades.

Troubleshooting

Symptom	Likely cause	Action
Pods stay in `ImagePullBackOff`	Registry access or an image tag is wrong.	Confirm `global.imageRegistry`, that the pinned image tags exist in the registry, and that the image pull secret is present in the namespace.
Install stops before pods start	The schema-migration step failed.	Read the migration job logs. Confirm the database is reachable from the cluster and the credentials in the Secret are correct.
Portal loads but sign-in fails or loops	OIDC redirect URI mismatch.	Confirm the redirect URI registered with your identity provider exactly matches the Portal's external URL, with no trailing slash. This is the most common setup failure. See Authentication.
`helm install` fails with "no kind ServiceMonitor"	`prometheus.serviceMonitor.enabled` is true but the cluster has no Prometheus Operator.	Set it to false, or install the Prometheus Operator first. See Metrics & Monitoring.
Adapter pods report not ready	No lease is held yet.	Expected for a standby. If no adapter ever becomes ready, confirm at least one adapter is enabled in the Portal.
The installer job did not apply your custom bundle	The bundle URL is not reachable from the cluster, or its checksum did not match.	Confirm the URL resolves from inside the cluster and the bundle is intact, then re-run `helm upgrade`.

Before you start​

Phase 1: Provision infrastructure​

Phase 2: Configure Helm values​

Choosing a durability profile​

Phase 3: Install and verify​

Phase 4: Your first session​

Go-live readiness​

Troubleshooting​

See also​