Pipeline Designer
A pipeline is the set of rules that runs inside an adapter every time a message passes through. Use it to match traffic to routes, reshape messages, run additional processing, and deliver to a target.
A route delivers a message only if it includes a Send Processor, added from the Actions & Sinks group into the route's Outbound Actions stage. A route with no Send Processor still runs its conditions and transforms, then acknowledges the message and sends nothing downstream. The conditions and transforms are optional; the delivery step is not.
How a message moves through the pipeline
Every message entering an adapter with a pipeline goes through two phases in order. The pipeline picks a route first, then runs that route's stages.
| Phase | Answers the question | What happens |
|---|---|---|
| Route matching | Which route claims this message? | Conditions on each route are checked in order. The first route whose conditions match wins. If no route matches, the message is acknowledged and discarded. |
| Transform | What should the message look like downstream? | The matched route's transforms run in order. Each one reshapes the message. |
| Process | What should the adapter actually do with this message? | The matched route's processors run in order. Delivery to the destination happens only if a Send Processor is present. |
Route matching with Conditions
A Condition on a route decides whether that route claims the message. Conditions on a single route combine as one test: the message matches the route only if it satisfies every condition on that route.
Routes are evaluated in the order you place them. For each incoming message, the adapter walks the list from top to bottom:
- If the first route's conditions match, the message runs through that route's stages and stops. Later routes never see it.
- If the first route's conditions do not match, the message is not discarded. It falls through to the next route for evaluation.
- Only when the message reaches the end of the route list without any route claiming it does the platform discard it.
This is the single most common source of confusion in the Pipeline Designer. A non-matching route sends the message along to the next route. It does not destroy it.
Worked example
Three routes in order:
- Route 1 claims a specific message type for account A.
- Route 2 claims any message type for account A.
- Route 3 claims every remaining message.
A message for account A of a different type fails Route 1, matches Route 2, and stops there. A message for a different account fails Route 1 and Route 2, then matches the catch-all in Route 3. A message that matches none of the three (not possible here because Route 3 is a catch-all, but would apply if Route 3 were removed) would be discarded.
Put the most specific conditions at the top and the broadest or catch-all route at the bottom. A permissive condition placed ahead of a specific one will silently claim traffic you expected a later route to handle.
Without a catch-all route at the end, any message that no route claims is acknowledged and dropped. No error, no entry in the event log by default, no copy kept. If messages you expect downstream are missing, verify that some route is actually claiming them.
Authoring a condition
You add a condition from the Component Library on the left of the Pipeline Designer. Conditions live in the Sources & Filters group. Click the one you want and its configuration dialog opens.
Each condition component brings its own configuration, so the dialog shows the inputs that the component you picked defines rather than one fixed set of fields. A component that matches on a message field, for example, asks you to name the field to inspect and the value it must hold; a different component exposes a different set of inputs, and some need no configuration at all. The fields you can match on, and how you express the match, depend on the component and on the message format your connector uses, described in that component's plugin documentation.
Fill in the settings and click Done to add the condition. It appears in the route's Inbound Filters stage on the canvas. Add a second condition to the same route to narrow further; the route matches only when every condition on it passes.
Transform: rewrite the message
Use a transform to change the message itself. Transforms are the mutation step of a route: the shape, field names, or values going out are not required to match what came in.
Typical uses:
- Rename fields so the external system's identifiers map to your internal names.
- Set a field to a fixed value on every outbound message.
- Add a reference identifier that lets you correlate the message across downstream systems.
- Remove fields the destination does not need.
Transforms on a single route run in the order you place them. The output of one transform is the input of the next, so order decides whether a later transform sees the original field or a value an earlier transform already rewrote.
Authoring a transform
You add a transform from the Transforms group in the Component Library. Click the one you want and its configuration dialog opens.
As with conditions, there is no single fixed form: the dialog shows the inputs the transform component defines. A transform that sets a field to a fixed value, for example, asks for the field to write and the value to write onto it, while a transform that renames or removes fields exposes the inputs it needs for that. Fill in the settings and click Done; the transform appears in the route's Transformation stage on the canvas.
Process: act on the message and deliver it
A processor is a step the adapter runs for each message that reaches this phase. A route's processors run in the order you place them, after the transforms.
Processors cover two different jobs.
- Acting on the message without forwarding it. A processor can enrich the message with data from another system, record a counter, or trigger an action elsewhere. Nothing about this sends the message downstream.
- Forwarding the message to the route's destination. A special kind of processor called a Send Processor is the component that actually delivers the message. If you want the message to reach the configured destination, a Send Processor must appear on the route.
Choosing a destination is not the same as forwarding to it
It is tempting to think that choosing the route's destination is enough to forward each matched message. It is not. The destination names where the message can go; a Send Processor is the step that actually sends it there.
You set the destination with the Destination Point selector at the bottom of the Pipeline Designer canvas. A Send Processor is added like any other processor, from the Actions & Sinks group in the Component Library, and lands in the route's Outbound Actions stage.
| In the designer | What it does |
|---|---|
| Destination Point | Names the destination for this route (for example, an outbound queue, a topic, or a session). On its own, choosing a destination does not move any message. |
| Send Processor | The processor that actually forwards the message to the destination. Add one to the route's Outbound Actions stage to make delivery happen. |
Which Send Processor to use depends on the destination. Pick the forwarding processor that corresponds to the destination you selected: the platform's default sender for a standard destination, or the sender contributed by the connectivity plugin that owns that destination. Matching the Send Processor to the destination sends the message out through the right connection.
Why delivery is explicit
Decoupling the destination from a Send Processor gives you explicit control over what leaves the adapter. You can:
- Build a route that enriches or records a message without forwarding it. Leave out the Send Processor entirely.
- Build a route whose only purpose is side-effect work (lookups, metrics, internal signals) and produces no downstream output.
If a route has no Send Processor, the message is acknowledged and nothing is sent from that route. This is a valid configuration; it is how "match but do not forward" is expressed.
Typical uses
- Record an operational log entry when a specific action occurs.
- Look up additional data from an internal system and attach it to the message.
- Count messages for operational metrics.
- Forward the message to the destination with a Send Processor.
Choosing between Conditions, Transforms, and Processors
| If your goal is to | Use |
|---|---|
| Decide which route claims a given message | A Condition on the route |
| Change what the destination receives | A Transform |
| Take an action alongside the message (enrich, log, count) | A Processor |
| Actually send the message to the route's destination | A Send Processor (a specific kind of Processor) |
Routes: send different messages to different destinations
A route is a named pipeline inside an adapter. One adapter can carry multiple routes, each with its own conditions, transforms, processors, and destination. In the wizard, routes are managed on the Data Pipelines step, where each route is listed as a pipeline card and opened for editing in the Pipeline Designer.
Fall-through matching (described above) means the adapter always runs exactly one route per message: the first whose conditions match. A message never runs through two routes.
Typical uses:
- Split an incoming feed so each message type lands on a different downstream topic.
- Route by account or region so different customers or geographies go to separate targets.
- End with a catch-all route that either sends whatever earlier routes did not claim (with a Send Processor) or drops it explicitly (without a Send Processor).
Adding pipeline components
The Data Pipelines step lists every route on the adapter, each shown as a pipeline card. To build one:
- Click Add Pipeline to open the Pipeline Designer. (To change an existing route, use the pencil icon on its card.)
- In the designer, the Component Library on the left lists the components you can add, grouped into Sources & Filters (conditions), Transforms, Actions & Sinks (processors), and Error Handlers. The list is populated from the plugins installed on the platform; a component you cannot see is one your platform does not have.
- Click a component to open its configuration dialog, fill in its settings, and click Done. A component that needs no settings shows nothing to configure; click Done to add it. The component lands in the matching stage on the canvas: Inbound Filters, Transformation, Outbound Actions, or On Error.
- Include a Send Processor in Outbound Actions to forward the message to the route's destination.
- Choose the route's Origin Point (source) and Destination Point (target) with the selectors at the top and bottom of the canvas.
- On each step, use the up and down arrows to reorder it, the pencil icon to reopen its configuration, and the trash icon to remove it. Order matters: transforms run top to bottom (the output of one is the input of the next), and every stage runs its steps in the order shown.
- Name the route in the header, then click Save Route to return to the Data Pipelines step.
Verifying the pipeline
Once the adapter is running, open it and switch to the Event Log tab. The log shows what happened to each message as it moved through the pipeline, including which route claimed the message and whether a Send Processor forwarded it.
When messages are disappearing unexpectedly, remember that filtered or unmatched messages are acknowledged and dropped without a log entry by default. To make drops visible, add a catch-all route that matches anything the earlier routes did not and watch which messages it claims: that tells you whether a condition is wrong, a route ordering issue is at play, or a Send Processor is missing.
See also
- Creating an Adapter: configuring a pipeline during the wizard flow.
- What Happens to Messages: how the platform handles delivery failures and retries after the pipeline stages run.
- Monitoring: watching live traffic through the pipeline.