Skip to main content

Pipeline Designer

A pipeline is a set of rules that runs on every message passing through an adapter. You use it to control which messages get through, change their format, or add extra information before they reach their destination.

Pipelines are optional. If you just want to forward messages exactly as they arrive, you don't need one.

How a message moves through a pipeline​

Every message goes through the same four stages, in order:

  1. Filter: decide whether this message should be processed at all
  2. Transform: change the message's content or format
  3. Process: take additional actions (logging, lookups, enrichment)
  4. Deliver: send the message to its destination

Each stage is optional. You only add the stages you need.

Filters​

Filters let you ignore messages you don't care about. A message that doesn't pass the filter is silently discarded.

Examples:

  • Only pass through trade confirmations, ignoring everything else
  • Only process messages for a specific account
  • Drop messages that are missing a required field

You can add multiple filters. A message must pass all of them to continue.

Transforms​

Transforms change the message itself. Each transform is applied in the order you set, so the result of one feeds into the next.

Examples:

  • Rename a field (e.g., map the external system's field name to your internal field name)
  • Set a field to a fixed value on all outbound messages
  • Add a reference ID to help track a message as it moves through your systems
  • Remove fields you don't want to forward downstream

Processors​

Processors take actions on a message without necessarily changing it.

Examples:

  • Write an entry to an audit log every time an order is cancelled
  • Look up a client's account name from your internal system and add it to the message
  • Track how many messages are being processed per minute

Routes​

If your adapter handles different types of messages that need to go to different places, you use routes to split them.

Each route has its own filter, transforms, and destination. The adapter checks routes in order and uses the first one that matches. Messages that don't match any route are discarded.

Example: An adapter receiving messages from an external system:

  • Route 1: Trade confirmations → publish to the confirmations topic
  • Route 2: Order cancellations → publish to the cancellations topic
  • Route 3: Everything else → discard

Adding pipeline components​

In the wizard, on the Pipeline step:

  1. Click Add Route if you need more than one destination
  2. Within a route, click Add Filter / Transform / Processor to add components
  3. Select the component from the list (what's available depends on your installed plugins)
  4. Fill in its settings
  5. Drag to reorder within a stage (order matters for transforms)

Checking your pipeline is working​

Once the adapter is running, open it and go to the Event Log tab. Each entry shows what happened to that message, including which stage it was in when it was processed or discarded.

If messages are being dropped unexpectedly, look for entries marked PIPELINE_DROPPED. They'll show which filter caused the message to be discarded.