Traffic Source Connector
The Traffic Source is not supported as part of a production data flow. Use it to load-test and measure a deployment, not to originate or receive real traffic. Its configuration surface and behavior may change between releases without a migration path, so a pipeline built around it can break on upgrade.
The Traffic Source (traffic-source-connector) ships with the Essentials plugin and is the platform's built-in benchmarking tool. Use it to answer three kinds of questions: capacity (can this deployment sustain X messages per second), tail latency (what are the p99 and p99.9 under load), and regression (did the last code or configuration change degrade performance).
This page covers how to run a test, how to read the results, and how to measure latency across a protocol boundary.
Modes
Each Traffic Source instance runs in exactly one mode, selected at adapter configuration time. There are two modes, plus an echoBack flag that changes what the RECEIVER does with each message.
| Mode | Role | What the connector does |
|---|---|---|
| GENERATOR | Source | Emits messages at the configured rate and hands them to the pipeline. Operator-controlled: Start and Stop from the dashboard. Stamps a send timestamp (perf_sent_ts) on every message that downstream tooling can carry across a boundary (see Cross-protocol testing). |
RECEIVER (echoBack=false, default) | One-way sink | Always listening (no Start/Stop). Receives whatever the pipeline delivers, acknowledges it durably, and drops it. Counts received messages. A run auto-starts on the first received message and auto-completes after an idle window. |
RECEIVER (echoBack=true) | Echo counterparty | Same as above, but re-dispatches every inbound message to the route's observers (typically a return FIX leg). Used as the far end of a round-trip test whose measuring adapter is a separate GENERATOR. Counts both received and re-sent. |
The connector itself no longer computes latency or throughput. It tracks only lightweight run-lifecycle counts (Sent, Received, Errors). Percentiles and throughput are recorded by the adapter's pipeline as Micrometer metrics and read from /actuator/metrics. See Interpreting results.
RECEIVER mode auto-completes a run after receiverIdleAutoCompleteSeconds (default 30) of no new received messages, then auto-starts a fresh run on the next inbound message. This cleanly brackets back-to-back tests. Set it to 0 to keep the receiver run open until adapter shutdown.
Topologies
A single mode is only a role. What a measurement means depends on how the two roles are wired across the pipeline. Pick the topology that matches your question.
| Question | Topology | What it measures |
|---|---|---|
| One-way rate and latency across a protocol hop | GENERATOR on the producer-side adapter, RECEIVER on the consumer-side adapter, with the send timestamp carried across the hop (see Cross-protocol testing). | One-way processing from producer to consumer across whatever hops sit between them. |
| Is downstream processing keeping up with live inbound traffic? | RECEIVER (echoBack=false) reading from the live downstream. | Inbound processing rate only. |
| Round-trip latency on a single clock | GENERATOR on adapter A sending through a RECEIVER (echoBack=true) on adapter B that returns the message to A's inbound leg. | Full round trip. Because A stamps the send time and A measures the return, cross-host clock skew is out of the picture. |
| Pipeline overhead only, no protocol egress | GENERATOR routed to a RECEIVER in the same adapter (same JVM), no protocol connector in the pipeline. | Pipeline overhead (transformers, processors) with no kernel network or protocol encoding. Single clock. |
| Fan-in behavior under contention | Two GENERATORs against one shared RECEIVER. | How the consumer behaves when two producers contend for it. |
| Has a recent change regressed performance? | Any of the above, run against the baseline build and again after the change with identical parameters. | Compare the percentiles and throughput from /actuator/metrics before and after. |
When a one-way result tells you something is slow but not where, point the RECEIVER at successive stages of a multi-adapter chain to measure up to each stage in isolation.
Running a test from the Portal
- Open the adapter's dashboard page. The Run Status card shows the connector's Mode.
- For a GENERATOR, use the Generator Controls panel and set:
- Rate (msg/sec): the target rate. Start at a rate you believe the platform can comfortably hold, then increase.
- Iterations: how many times to replay the dataset. Total messages sent is iterations times dataset size.
- Pacing Preset: see Pacing presets.
- Traffic Shape: see Traffic shapes.
- Click Start Traffic. Watch the Sent counter tick up to confirm the run is progressing.
- For a RECEIVER, there are no controls. The run auto-starts when the first message arrives and the Received counter begins to move.
- Read percentiles and throughput from
/actuator/metrics(see below). The dashboard card is for run liveness and counts, not performance numbers.
Pacing presets
Pacing presets are measurement methodologies, not deployment configurations. Production traffic on this platform is event-driven: orders fire when a strategy decides, market data arrives at whatever rate the upstream sends, fills land when matches happen. Neither preset ships into a production traffic path. They exist to drive synthetic load on the test side so the platform's behavior can be characterized along two axes: per-message timing precision, and aggregate volume ceiling. Pick the preset that matches the question you are trying to answer.
| Preset | Mechanism | When to use |
|---|---|---|
LATENCY (default) | Park most of the wait, spin the last few tens of microseconds for tight per-message precision. One message per dispatch. | "What is the platform's per-message timing at sustained rate X?" Use this for sustained-rate tests where the headline answer is a percentile (p50, p95, p99). The reported timing reflects the platform, not pacer jitter. |
THROUGHPUT | Batched dispatch in 10ms windows. Yields CPU between bursts and amortizes per-message dispatch overhead across the batch size. | "What is the platform's maximum sustained throughput?" Use this to find the volume ceiling. Per-message timing precision is not the goal of this preset: messages within a 10ms window are dispatched together. |
Choosing the wrong preset for the question gives a misleading number. Running a throughput-ceiling test with LATENCY understates the ceiling because the per-message dispatch path does not amortize. Running a percentile-precision test with THROUGHPUT overstates per-message timing because the batched dispatch contaminates each message's individual measurement with batch-window scheduling delay.
The platform's actual production behavior depends on production traffic shape, durability mode, session settings, and resource sizing. The pacer presets characterize the platform's measurable boundaries; they do not configure how production runs.
Traffic shapes
The pacing preset controls per-message timing mechanics. The traffic shape controls the volume profile over the run. A shape is optional: when unset, the connector emits at a steady rate driven by the target rate and preset, which is backward compatible with configurations that set only targetRate. Choose a shape to model a specific arrival pattern.
| Shape | Mechanism | When to use |
|---|---|---|
CONSTANT_RATE | Emits at a sustained messagesPerSecond. | The default. Steady-state capacity and latency measurement, where you hold one rate and read the percentiles. |
BURSTY | Emits burstSize messages back-to-back, idles for delayBetweenBurstsMs, and repeats. | Model low-volume systems that publish batched micro-bursts, and test how the pipeline absorbs a pulse followed by quiet. |
RAMP | Linearly scales from startRate to endRate over rampDurationSeconds, then holds at endRate. | Find the knee of the throughput and latency curve: the rate at which latency starts to climb. |
The dashboard's Traffic Shape selector on the Start action picks the shape with reasonable defaults for its secondary parameters (for example, a bursty shape idles one second between bursts, and a ramp scales from one message per second to the target rate over ten seconds). For fine control over burst spacing or ramp duration, set the shape in the connector's static configuration, which exposes every parameter.
Interpreting results
The run dashboard
The Run Status card is a liveness and progress view, not a performance report. It shows only the fields below.
| Field | Meaning |
|---|---|
| Mode | GENERATOR or RECEIVER. |
| State | RUNNING if a run is active, else STOPPED. |
| Run ID | The current run's identifier, or empty when stopped. |
| Duration | Seconds since the active run started. |
| Sent | Messages emitted in the current run. GENERATOR increments this. |
| Received | Messages received in the current run. RECEIVER increments this. |
| Errors | Errors recorded in the current run. |
For an echoBack=true RECEIVER, Sent and Received move in lockstep: every inbound message is received and then re-dispatched to the return leg. All counts reset to zero when the next run starts.
Use these fields to confirm a test is alive and progressing. For the actual performance numbers, read the metrics endpoint.
Percentiles and throughput
Latency percentiles and throughput are recorded by the adapter's pipeline as Micrometer summaries and are read from the adapter's metrics endpoint (/actuator/metrics), or scraped into Prometheus from /actuator/prometheus. These percentiles describe processing latency measured inside a single adapter; they are not an end-to-end latency across a protocol boundary. The platform does not compute an end-to-end latency from perf_sent_ts. The generator stamps that value and the transports can carry it, but turning it into a latency distribution is not something the platform provides today; it requires your own downstream consumer (see Cross-protocol testing).
Read percentiles rather than averages. Exchange-grade SLAs are written in percentiles, and a mean hides the tail: p50 and p95 can look fine while p99.9 is orders of magnitude larger because a small number of outliers (garbage collection, kernel scheduler blips, a single slow downstream call) dominate the tail without moving the average. A p99.9 close to p99 means the tail is bounded; a p99.9 far above p99 means outliers are present.
Read the shape, not the absolute numbers. The absolute numbers depend on your deployment, your downstream targets, and your target rate. Compare them against your own baseline, not against claimed benchmarks.
What these numbers do and do not measure
Every latency number is trustworthy within its stated scope; the common failure is reading a number outside its scope. The bounds:
| Number | Measures | Does NOT measure |
|---|---|---|
| Pipeline latency percentiles | One message through one adapter, from pipeline entry to ack. | Wire transit, counterparty processing, other pods, or time queued before pipeline entry. |
| Outbound / inbound decomposition | The database and storage share (WAL) vs engine and socket share, per pod. | Anything outside that pod. |
| Pipeline stage decomposition | Each stage's share (conditions, transformers, processors) plus the hand-off to the target connector, for any protocol. Processors include the dispatch time; the dispatch series isolates it. | Work inside the target transport after it accepts the message. |
| Session RTT probe | Request/response round-trip on one pod's clock, including the counterparty's processing time. | Pure wire transit: halving the RTT overstates one-way time. The probe must be enabled (rttProbeIntervalMs above 0); the perf sample definitions ship with it on. |
| Backpressure blocks and block time | How long producers were deliberately throttled because a queue was full. Non-zero means the latency percentiles are inflated by design: the platform trades latency for zero loss. | Which downstream component caused it: read the settlement, dispatch, and WAL numbers to attribute. |
| Percentiles generally | Exact at the 1/2/3/5/10/25ms boundaries. | Values below 0.1ms (the platform's measurement floor), and quantiles are unreliable when a window holds too few samples (short runs at low rates). |
There is no single cross-pod end-to-end number, deliberately: clocks on different pods cannot be compared. Reconstruct end-to-end as outbound latency + RTT/2 + inbound latency, each measured on one clock.
For a one-call view of all of this, read GET /api/v1/adapters/{adapterId}/perf on the platform API (the orchestrator forwards to the running instance, so no pod names or ports are needed; in-cluster, the same payload is on the adapter's management port at /actuator/perf). It returns offered/dropped counts, the latency decomposition with each number's scope stated inline, and the RTT section (or an explicit note that the probe is off). Each run also ends with a completion event carrying a one-word verdict: CLEAN (numbers describe the system under test), GENERATOR_STARVED (the load source was the bottleneck; the platform was not fully exercised), or LOSSY (messages were dropped; discard the run).
Offered load vs delivered load
The generator publishes what it attempted to send as counters, independent of what the system under test received:
| Series | Meaning |
|---|---|
perf_generator_sent_total | Messages the generator dispatched downstream (offered load). |
perf_generator_dropped_total | Messages lost to retry exhaustion or retry-queue overflow during a downstream outage. |
Compare rate(perf_generator_sent_total[1m]) against the receive-side throughput series. When they diverge, the generator is being starved (CPU throttling, pacing limits); when they match but latency climbs, the system under test is saturating. A non-zero dropped count means the run lost messages and its results should be discarded.
See Metrics & Monitoring for the endpoints, scrape configuration, and how to query connector metrics.
Capacity testing workflow
A repeatable capacity test protocol:
- Find the ceiling. Run a short test at an intentionally high target rate (for example twice your expected production rate) with the
THROUGHPUTpreset. Watch the Received throughput on the metrics endpoint level off: that plateau is your effective ceiling on this deployment. - Warm up. Run at roughly 75 percent of the observed ceiling and discard the result. This lets the JIT compiler optimize hot paths and fills the plugin caches.
- Measure. Run a fresh test at or below 75 percent of the observed ceiling. Run long enough that the percentiles stabilize.
- Record the results. Capture p50, p95, p99, p99.9, and sustained throughput from
/actuator/metrics. These are your reference numbers. - Rerun under different conditions. Vary the target rate, the number of concurrent sessions, or the traffic shape to stress different dimensions.
Running a regression test
To catch performance regressions between platform versions or configuration changes:
- Run the capacity test protocol above on the baseline build. Record p99 and p99.9 from
/actuator/metrics. - Apply the change.
- Rerun the same test with the same parameters (same rate, same iterations, same preset, same shape).
- Compare the new p99 and p99.9 against the baseline. A change of more than 10 percent in either direction is worth investigating.
Compare only runs measured under the same conditions. A run that did not reach steady state, or that ran on a contended node, says nothing about the change.
Cross-protocol testing
The GENERATOR stamps a send timestamp itself: on every dispatch it writes perf_sent_ts onto the message headers. No separate enricher component is involved. The platform does not turn this value into a latency measurement. Carrying it across a boundary is only useful if you run your own downstream consumer that reads it; this section describes how to carry the header across each protocol so that such a consumer can see it.
In the simplest single-JVM topology (a GENERATOR routed to a RECEIVER in the same adapter), the header rides on the in-memory message object and reaches the receive side unchanged. No pipeline transformers are needed.
Any test that sends traffic across a protocol boundary (a FIX session, a Kafka topic, a cross-pod RECEIVER) serializes the message to the wire, and the perf_sent_ts value is dropped unless it is carried across that boundary explicitly. Different protocols expose metadata in different places, so the mechanism is protocol-specific:
- FIX has no separate metadata channel: every field lives inside the message body. Carry
perf_sent_tswith the FIX Header Tunnel (fix-metadata-transformer), which injects the header into a user-defined FIX tag on egress and extracts it back to a header on ingress. It defaults to Tag 6400 and header keyperf_sent_ts. Place it inINJECTmode on the outbound route andEXTRACTmode on the inbound route. - Kafka ships a native record-header channel that travels alongside the payload. The
perf_sent_tsheader is carried natively as a record header; no injection transformer is required. - Custom protocols built on the Plugin SDK carry the header through whatever the protocol's wire format exposes for metadata (a record header, a reserved body field, a trailing envelope). The plugin author writes a transformer that moves
perf_sent_tsinto that channel on egress and back out on ingress. The FIX Header Tunnel is a reference implementation to model on.
| Test topology | What carries perf_sent_ts across the boundary |
|---|---|
| GENERATOR to RECEIVER in the same adapter (same JVM) | Nothing extra. The in-memory message carries the header directly. |
| GENERATOR to a Kafka topic, RECEIVER reading that topic | Kafka's native record header. No transformer needed. |
| GENERATOR to a FIX session, RECEIVER reading that session | FIX Header Tunnel: INJECT on the outbound route, EXTRACT on the inbound route. |
| Cross-protocol bridge (FIX to Kafka to FIX) | FIX Header Tunnel on each FIX hop; Kafka preserves the header natively on the Kafka hop. |
| Custom protocol on the Plugin SDK | A custom transformer that injects perf_sent_ts into the protocol's metadata channel on egress and extracts it on ingress. |
If perf_sent_ts is missing at the receive side of a cross-protocol test, the transformer chain is the first thing to inspect. See the troubleshooting matrix below.
The perf_sent_ts header: value contract
Whenever a test crosses a protocol boundary, the perf_sent_ts header is what carries the send timestamp. The contract is one value format, everywhere, for every topology:
perf_sent_ts is a signed decimal integer nanosecond reading from the sending JVM's monotonic clock (System.nanoTime()), captured at the message's intended (scheduled) send time.
The GENERATOR sources this value from the pacer's scheduled send time, not from a wall-clock read at the moment of dispatch, so a catch-up burst after a pause does not skew the reading. Because it is a monotonic-clock reading and not a wall-clock (Unix-epoch) time, it is meaningful only within the JVM that produced it: two System.nanoTime() readings are comparable only when they come from the same process. Readings taken on different hosts share no common origin and cannot be subtracted to yield a latency. That cross-host incomparability is exactly why the platform provides no cross-host end-to-end latency metric. Within a single JVM (a GENERATOR and RECEIVER in the same adapter), the readings are directly comparable.
Because the value is a monotonic-clock reading, carry it verbatim across a boundary. A custom plugin that moves the header through a protocol's metadata channel should transport the exact integer it received, without reinterpreting or rescaling it. Deriving a wall-clock timestamp from a protocol field (for example a FIX SendingTime field) is a different measurement and must not be written into perf_sent_ts.
Clock synchronization and what a one-way measurement means
Any cross-process latency measurement is bounded in accuracy by how well the sender's and receiver's clocks agree. Two separate hosts never have perfectly identical clocks; the practical question is whether the skew is smaller than the latency you are trying to measure. The two time-sync protocols that show up in practice are NTP (Network Time Protocol, the standard service running on most hosts) and PTP (Precision Time Protocol, a higher-accuracy alternative that requires network-interface hardware support and is common in low-latency trading networks).
| Time-sync protocol | Typical achievable skew | What a one-way measurement can resolve |
|---|---|---|
| Unsynchronized (default on an unconfigured host) | Seconds or more, drifting further over time. | Nothing. Do not attempt cross-process latency measurement until time sync is configured and healthy. |
| NTP against a reasonable-quality server | A few milliseconds, sometimes tens on congested networks or virtualized hosts. | Measurements whose true latency is significantly larger than the skew. A sub-millisecond true latency will be buried in millisecond skew. |
| PTP with hardware timestamping | Sub-microsecond on a properly configured LAN. | Microsecond-scale latency, the regime where the percentiles are the point of the test. |
Two practical takeaways.
First, before quoting a one-way (GENERATOR-to-RECEIVER across pods) number, verify the current skew between the sender and receiver clocks through your normal time-sync monitoring. A reading of p99 = 900 microseconds from an NTP-only deployment may be entirely skew and zero real work.
Second, if the latency you want to measure is smaller than your realistic time-sync precision, use a round-trip topology instead of a one-way cross-pod measurement. A GENERATOR sending through a RECEIVER (echoBack=true) that returns the message to the originating adapter captures both the send time and the return time on the same adapter's clock, so skew between hosts is not in the picture. The trade-off is that you measure a full round trip rather than a one-way hop; if the two directions are approximately symmetric, halving the round-trip number is a defensible one-way estimate.
A GENERATOR routed to a RECEIVER within a single JVM (where the message is never serialized) is exempt from this discussion because both ends share the same monotonic clock by definition.
Common failure modes
| Symptom | Likely cause | Action |
|---|---|---|
| Received throughput plateaus well below the target rate | The pipeline is at its ceiling: the target rate exceeds what this deployment can sustain. | Note the plateau as the effective ceiling. Re-run at or below 75 percent of that value to measure latency in a regime the platform can hold. |
| Wide latency tail (p99.9 much larger than p50) on a run that otherwise holds its rate | Pipeline jitter from group-commit batching, network read coalescing, or infrequent GC pauses. | Cross-reference with node-level GC and CPU telemetry. If the tail must be tightened, the trade-off is throughput; re-run with the LATENCY preset and a lower rate. |
| Sent counter well below the expected total (target rate times iterations times dataset size) | The generator could not maintain the target rate: it is the bottleneck. | Check the adapter's CPU limits. Confirm the THROUGHPUT preset for volume tests; LATENCY caps at one message per dispatch. |
| Sent counter correct but Received counter low | The pipeline or downstream is behind the generator. | Check the downstream target system, node contention, and database write latency. |
| p50 and p95 close but p99 very high | Bimodal distribution: a subset of messages takes a slow path. | Profile the slow path. Check for conditional logic in processors that fires only for some messages. |
perf_sent_ts missing at the receive side of a cross-protocol or cross-pod test, with messages received | The send timestamp is not being carried across the boundary. | Confirm the FIX Header Tunnel is on both routes (INJECT outbound, EXTRACT inbound), or that Kafka record headers are preserved. For custom protocols, confirm the transformer moves the header on egress and restores it on ingress. The platform does not compute a latency from this value; see Cross-protocol testing. |
| RECEIVER run never completes between test bursts | receiverIdleAutoCompleteSeconds is set to 0 (disabled), or bursts arrive closer together than the idle window. | Set a non-zero idle window to auto-bracket back-to-back tests, or widen the gap between bursts. |
See also
- Essentials Plugin overview
- Metrics & Monitoring for the Prometheus endpoint and how to query connector metrics.
- Scaling for adapter resource sizing guidance.
- Troubleshooting for broader diagnostic patterns.