Metrics & Monitoring
Conncentric exposes metrics in Prometheus format on a standard endpoint. Any backend that can scrape a Prometheus endpoint consumes them directly, and backends such as Datadog or CloudWatch can ingest them through their own Prometheus-compatible agents.
Prometheus Scrape Endpoints
Each component exposes a Prometheus-compatible endpoint on its management port. These endpoints are always available for scraping. If your cluster runs the Prometheus Operator, the chart can create the ServiceMonitor resources for you; see below.
| Component | Path | Port |
|---|---|---|
| Orchestrator | /actuator/prometheus | 8080 (application port) |
| Adapter | /actuator/prometheus | 8081 (management port) |
The Adapter exposes metrics on a dedicated management port (8081), separate from the dynamic application port used for protocol sessions. The Orchestrator exposes metrics on its application port (8080). Neither metrics endpoint is exposed outside the cluster; Prometheus scrapes them over the cluster-internal pod network.
ServiceMonitor
If your cluster runs the Prometheus Operator, enable the chart's built-in ServiceMonitors rather than hand-authoring them:
prometheus:
serviceMonitor:
enabled: true
interval: 30s # tighten to 15s for latency-critical fleets
scrapeTimeout: 10s # must be < interval
additionalLabels: {} # e.g. { release: prometheus } to match your Operator's selector
This creates two ServiceMonitor resources: one for the Orchestrator (scraped on its application port) and one for the adapter fleet. The adapter ServiceMonitor selects a dedicated headless metrics Service, because the adapter's main Service carries only the protocol acceptor session ports; scraping therefore does not depend on adapter.networking.portRange.
Set additionalLabels to whatever label your Prometheus Operator's serviceMonitorSelector matches (commonly release: <your-prometheus-release>). Without a matching label, the resources are created but never scraped.
If you do not run the Prometheus Operator, point your Prometheus scrape_configs at path /actuator/prometheus on port 8080 for Orchestrator pods and port 8081 for Adapter pods in the conncentric namespace.
Orchestrator Metrics
conncentric_enabled_adapters_count
Type: Gauge
The total number of adapters in the ENABLED administrative state. Adapters are not horizontally autoscaled: the replica count is a topology decision the operator sets explicitly (see adapter.replicaCount in the Helm Reference), not an elasticity lever. Treat this gauge as an operational health signal to alert on, not a scaling input.
Tags: none
Use: Alert if this unexpectedly drops to 0 in production.
adapter_lifecycle_state
Type: Gauge (state-set pattern, one gauge per adapter per state value)
Reports which lifecycle stage each adapter is currently in. The gauge for the active state reports 1; all others report 0.
Tags:
| Tag | Description |
|---|---|
adapter_id | The adapter's logical ID |
display_name | Human-readable name |
state | The state this gauge represents |
State values:
| Value | Meaning |
|---|---|
provisioning | Adapter has a lease and is downloading plugins/config |
active | Adapter is running the message pipeline |
standby | Adapter is healthy but waiting; a sibling node is active |
paused | Adapter was intentionally stopped by a user |
releasing | Adapter is gracefully stopping and releasing its lease |
port_exhaustion | Adapter cannot start due to a port conflict on the host |
inactive | Adapter has no lease and is not participating |
Example query (PromQL):
# Count of active adapters
sum(adapter_lifecycle_state{state="active"})
# Alert: any adapter stuck in port_exhaustion
adapter_lifecycle_state{state="port_exhaustion"} == 1
adapter_operational_status
Type: Gauge (state-set pattern)
Reports the business health of each adapter. Only meaningful when lifecycle state is active.
Tags:
| Tag | Description |
|---|---|
adapter_id | The adapter's logical ID |
display_name | Human-readable name |
status | The status this gauge represents |
Status values:
| Value | Meaning |
|---|---|
healthy | All connectors and pipeline routes are operational |
degraded | Partial failure, e.g. one of two connectors is down |
unhealthy | Total failure or critical internal error |
na | Not applicable (adapter is inactive or on standby) |
Example query (PromQL):
# Any adapter in unhealthy state
adapter_operational_status{status="unhealthy"} == 1
# Any adapter reporting degraded
adapter_operational_status{status="degraded"} == 1
A degraded or unhealthy gauge reads 1 only while that is the adapter's current operational status, and the platform reports a status other than na only for an active adapter, so the bare == 1 needs no join against adapter_lifecycle_state. If you want to make the active gate explicit, intersect the two gauges on their shared identity label rather than ignoring a label: adapter_operational_status{status="degraded"} == 1 and on(adapter_id) adapter_lifecycle_state{state="active"} == 1.
The Portal shows each adapter's status as a single Operational Status label (Running, Provisioning, Standby, Error, Inactive, and so on), whereas the metrics above expose two separate dimensions: adapter_lifecycle_state and adapter_operational_status. The Portal derives its label from both:
| Portal label | Metric state |
|---|---|
Running | lifecycle_state = active with operational_status = healthy |
Degraded | lifecycle_state = active with operational_status = degraded |
Error | lifecycle_state = active with operational_status = unhealthy |
Provisioning / Releasing | lifecycle_state = provisioning / releasing (the platform is placing or removing the adapter) |
Standby | lifecycle_state = standby (awaiting the primary lease) |
Paused | lifecycle_state = paused |
Ports Exhausted | lifecycle_state = port_exhaustion |
Inactive | lifecycle_state = inactive |
The Portal is authoritative for the exact label an adapter shows: it combines both dimensions into the single label on its Adapters list. When correlating a Portal status with an alert, use this table to find the underlying metric state. See Troubleshooting for the diagnostic workflow that references the Portal labels.
adapter_pipeline_open_circuit_breakers
Type: Gauge
Number of pipeline routes whose circuit breaker is currently OPEN. When a route's breaker trips (repeated infrastructure faults sending to the target), the route is halted and messages stop flowing, even though the source connector may still report as connected. This gauge makes that halted state visible to metric-based alerting. Per-route detail is also available on /actuator/health under pipelineCircuitBreakerHealth.
Example query (PromQL):
# Any route halted by an open circuit breaker
adapter_pipeline_open_circuit_breakers > 0
Use: Alert when greater than 0. A halted route does not clear on its own: restart the adapter once the downstream fault is resolved. See the circuit breaker runbook.
Pipeline and Boundary Metrics (per adapter)
Every message crosses a fixed set of platform seams, and each seam is measured for every protocol; a plugin does not need to emit anything for its route to be fully attributable. All latency series are histogram buckets (use histogram_quantile()), with exact boundaries at 1/2/3/5/10/25 ms.
| Series | Measures | Alert when |
|---|---|---|
pipeline_messages_{in,out,filtered,errored}_total | Per-route throughput and outcomes. | errored rate is non-zero in steady state. |
pipeline_processing_latency_seconds | One message through the route, entry to ack. | p99 exceeds your route's latency budget. |
pipeline_stage_latency_seconds{stage} | Each stage's share: conditions, transformers, processors (includes target dispatch), errorHandlers. | One stage dominates a latency regression. |
pipeline_target_dispatch_latency_seconds | Hand-off to the target connector: the transport, broker, or counterparty accept time. | High while stages are quiet: the slowness is the target side, not the platform. |
pipeline_settlement_latency_seconds | The durability transition at ack (for sources that declare one, such as FIX write-ahead logging). | High: the deployment's database or storage, not platform code. |
fix_{inbound,outbound}_backpressure_blocks_total and ..._block_ms | How often and for how long producers were deliberately throttled by a full queue. | Sustained non-zero block time: the system is saturated downstream of that queue, and latency percentiles are inflated by design (the platform trades latency for zero loss). |
For a one-call view of all of the above with each number's scope stated inline, read GET /api/v1/adapters/{adapterId}/perf on the platform API; the same payload ships in every Support Diagnostics bundle as perf.json.
Plugin Metrics
Plugins can emit their own metrics, which appear alongside the platform metrics in the same scrape endpoint. Plugin metrics are tagged automatically with adapter ID and plugin key.
The specific metric names emitted by each plugin are documented in each plugin's own reference material.
Dashboards & Alerting
Conncentric does not ship pre-built Grafana dashboards or alerting rules; build them in your monitoring stack from the metrics on this page. Alerting policy (thresholds, durations, routing, escalation) is yours to own, because the right values depend on your traffic profile and session schedule. The PromQL examples below cover the most critical platform-level alert conditions:
| What to alert on | Query |
|---|---|
| Any adapter unhealthy | adapter_operational_status{status="unhealthy"} == 1 |
| Any adapter degraded | adapter_operational_status{status="degraded"} == 1 |
| Route halted by circuit breaker | adapter_pipeline_open_circuit_breakers > 0 |
| Route erroring continuously | rate(pipeline_messages_errored_total[5m]) > 0 for > 10 minutes |
| Port exhaustion | adapter_lifecycle_state{state="port_exhaustion"} == 1 |
| All enabled adapters lost | conncentric_enabled_adapters_count == 0 |
| Adapter stuck provisioning | adapter_lifecycle_state{state="provisioning"} == 1 for > 2 minutes |
| Heavy GC pause time | increase(jvm_gc_pause_seconds_sum[5m]) > 5 (more than 5 seconds of pauses per 5-minute window inflates every latency percentile; check pod memory sizing) |
Protocol-level alert conditions (session connectivity, disconnects, rejects, backpressure) are documented in each plugin's operations page alongside the metrics it emits.
Each entry in the table is an alert expression. A bare == 1 (or > 0) fires as soon as the condition is true. To require the condition to hold for a duration, as the provisioning row does, wrap the expression in a Prometheus alerting rule and set its for: clause. Provisioning is expected to be brief, so alert only if an adapter stays in it:
groups:
- name: conncentric-adapters
rules:
- alert: AdapterStuckProvisioning
expr: adapter_lifecycle_state{state="provisioning"} == 1
for: 2m
labels:
severity: warning
annotations:
summary: "Adapter {{ $labels.adapter_id }} stuck provisioning for over 2 minutes"
The for: 2m clause holds the alert in a pending state until the expression has been continuously true for two minutes, then fires it. Apply the same pattern to any of the instantaneous expressions above when a brief, self-clearing spike should not page.