Avoiding loops and maximizing pipeline efficiency

Learn how controlling event and message jumps across pipelines ensures efficiency and prevents performance-impacting loops.

Understanding the “jump” concept

A “jump” occurs when an event is triggered across different pipelines or even within the same pipeline. The Digibee Integration Platform tracks the number of times an event is triggered, with a maximum of 25 jumps allowed in the event’s lifecycle.

Practical examples

Scenario 1: Excessive jumps between two pipelines

To better understand how jumps work, consider the following scenario:

1. Pipeline A publishes an event with ID “123” to Pipeline B.

2. When event 123 is published to Pipeline B, the first jump occurs, leaving 24 jumps remaining.

3. Pipeline B is configured to trigger 30 times per event.

This operation won’t succeed because event 123 has only 24 jumps remaining, making it impossible to complete all 30 triggers.

Scenario 2: Excessive jumps in a pipeline chain

While infinite loops that exhaust the event lifecycle are rare (as pipelines tend to self-trigger fewer times), they are more likely in scenarios with long pipeline chains, each triggering the next.

Here’s an example to illustrate:

1. Pipeline A publishes event 123 to Pipeline B, consuming the first jump and leaving 24 jumps.

2. Pipeline B is configured to trigger 15 times per event, leaving 9 jumps remaining after processing.

3. Pipeline B then triggers Pipeline C, consuming another jump and leaving 8 jumps.

4. Pipeline C is configured to trigger 15 times per event.

In this case, the event 123 won’t complete all 15 triggers in Pipeline C, as only 8 jumps are available. The event’s lifecycle limit of 25 jumps is insufficient for the 32 jumps required by this chain of pipelines.

Managing jump control

Even in parallel executions, the event lifecycle is tracked. This jump count applies to messages generated by both the Event Publisher and Pipeline Executor connectors.

The examples discussed here don't represent infinite loops. Instead, they assume that pipelines are designed with safeguards to prevent unlimited self-triggering. Without such controls, an infinite loop could occur, where the pipeline continuously triggers itself, negatively impacting system performance. Implementing such controls is crucial for managing event lifecycles, avoiding loops, and ensuring system stability.