How to resolve common pipeline issues

Learn how to identify and resolve common errors in your pipelines.

What qualifies as an error?

In integrations, an error is a failure or unexpected behavior that interrupts the normal flow of data between systems, services, or connectors. These errors can occur at any stage of execution.

Examples of errors in integrations

Timeout

Description: Occurs when the pipeline exceeds the configured execution timeout, forcing the process to stop.

Example: A pipeline that queries an external database takes more than 15 minutes to complete the request due to a large volume of data.

{
  "timestamp": 1730373482528,
  "error": "Pipeline execution timed-out.",
  "code": 500
}

Out of Memory (OOM)

Description: This error occurs when the pipeline consumes more memory or CPU than the configuration allows and stops execution.

Example: A pipeline that processes a large file or queries a high volume API without pagination, resulting in excessive memory usage. Learn how to resolve the "Pipeline execution was aborted" error.

{
  "timestamp": 1730438185028,
  "error": "This pipeline execution was aborted and cannot be automatically retried because it was configured to disallow redeliveries.",
  "code": 500
}

Connection failures

Description: Communication error between the pipeline and an external service or database, such as authentication failures or unavailable servers.

Example: A pipeline tries to access an external API but encounters an authentication error or cannot connect because the server is temporarily unavailable.

{
  "timestamp": 1734254417051,
  "error": "java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 60000ms.",
  "code": 500
}

Configuration errors

Description: Errors caused by incorrect settings, such as invalid endpoints, missing parameters, or incorrect size limits.

Example: A pipeline with an incorrect endpoint for a REST API cannot send data, resulting in an error with an invalid URL.

{
    "timestamp": 1734134205495,
    "error": "An internal error has occurred. Exception: java.lang.NullPointerException: Cannot invoke \"String.length()\" because \"sql\" is null",
    "code": 500
}

Unexpected restarts

Description: When the integration infrastructure, such as pipeline containers, is restarted due to internal problems such as OOM errors.

Example: A pipeline that consumes a lot of memory leads to an automatic restart by Kubernetes and generates a “recycled” warning in the Runtime screen.

Message expiration in event queues

Description: Messages expire if they are not processed in time, which leads to their elimination.

Example: A message expires before it is consumed by the next pipeline, especially if the queue expiration time is short and the data load is high.

Step-by-step guide to identifying, analyzing, and solving issues

Identifying the issue

1

Gather initial information

Make sure you have the execution key from the error logs or find it by filtering the logs by time and relevant fields.

2

Access the pipeline and execution

Open two tabs: one for the examined pipeline and another for the associated execution on the Completed Executions page. This allows you to compare the configured integration (pipeline) with what actually happened (execution).

• Build: Shows how your integration is configured.

• Monitor: Shows execution details and possible error points.

3

Analyze the logs

Check the execution logs to trace the integration path, to find the point at which the integration was aborted, or to identify the error. The logs can be analyzed individually or across multiple executions. You can view the logs in two ways:

• Completed Executions: Displays the execution key, the input/output messages, and the start and end timestamps.
• Pipeline Logs: Provides logs of the steps that were executed from the request to the pipeline response.

Not all pipeline connectors automatically generate logs. Connectors such as Log and external interaction connectors (for example, REST, Google Storage, Stream DB) always log messages. However, the results of data transformation or external connectors are only displayed if they are explicitly logged.

Analyzing the issue

Method 1

Compare a successful integration with a failed integration. Look for data that has been integrated correctly (for example, an order number) and data that has failed to integrate. If similar data follows different flows, compare the logs to find the point of divergence.

Method 2

Identify patterns in execution failures. For example, if concurrent executions fail with a timeout, analyze the logs to determine which step caused the issue and which connector or service is associated with it.

Resolving common issues

Timeout

What it is: A timeout occurs when a pipeline exceeds the execution time, usually 15 minutes (900,000 milliseconds).

How to fix it:

• Increase the timeout in the pipeline’s trigger, but avoid exceeding the maximum limit.

• Identify where the timeout occurs in the flow. Adjust the timeout in the connector for specific requests such as REST or SOAP.
• Restructure pipelines that are approaching the 10-minute limit by processing the data in smaller chunks using pagination.

Out of Memory (OOM)

What it is: Occurs when the pipeline exceeds its configured memory limit.

How to fix it:

1. Identify the cause: Use the Monitor page to analyze the logs and determine where the error is occurring. Adding logs can narrow down the issue.

2. Optimize the flow: Implement pagination to reduce the data flow or split the flow into primary and secondary pipelines to implement enrichment and data submission logic.

3. Adjust the deployment: After you restructure the pipeline, test it and then deploy it to Production. If the error persists, increase the pipeline size and deploy it again.

Trigger configuration and queue expiration

What it is: The pipeline trigger causes execution to start and the queue expiration determines how long a message can wait to be processed.

How you can fix it:

• For low-volume scenarios, set shorter expiration times (2-5 minutes) to avoid unnecessary message accumulation.

• For high-volume scenarios, extend the inspiration time to 10-15 minutes, depending on how much time the messages take to consume and process.

• Adjust the expiration time to avoid the loss of messages that can occur when queues expire before processing.

Suggestions:

• Make sure the trigger is configured according to the expected load before running the pipeline.

• Monitor the queues regularly to adjust the expiration time to the volume and expected behavior of the pipeline.

Delay in Object Store or Session Management

What it is: A delay in Object Store or Session Management occurs when the volume of data stored increases significantly without a cleanup or deletion routine in place. This overloads the database, slowing down queries and pipeline execution and affecting the overall performance of the pipeline.

How to fix it:

• Check the data volume: Access the Object Store and analyze the current size of the database. If the data volume is high, implement a cleanup routine to improve performance.

• Implement a cleanup routine: Configure a cleanup routine to remove old or unnecessary data. This will avoid the accumulation of excessive data and maintain more efficient storage.

• Divide the processing into smaller parts: For pipelines that process large amounts of data, use pagination or split the flow into different pipelines. One pipeline performs the first query and sends the data to other pipelines. This keeps the Object Store clearer and the integration process scalable.

• Test and monitor: After you have implemented a cleanup routine or restructured the pipeline, monitor performance in a Test environment. Observe whether the delay has decreased and, if necessary, adjust the cleanup process or the integration flow before deploying it in the Production environment.

External tools support

Sometimes you can use external tools to help you with certain aspects of the integration:

• SSL certificate validation: If you encounter connection issues, it can be helpful to check that certificates are correctly updated and active.

• XML validation and comparison: For debugging errors in XML workflows and comparing successful executions with failed ones.

• HTML generator: For creating HTML messages for use in connectors such as Email and Template Transformer.
• JSON Schema generator: For generating schemas from a valid JSON file.

• Mock data generator: Useful for testing with large amounts of data.

• AI tools: For simplifying logical solutions such as the creation of RegEx.

Improving diagnostics

Pay attention to logs

• Logs are crucial to quickly identify where errors occur in the pipeline. For effective diagnostics, we recommend placing the Log connector at strategic points in the pipeline. Place it before and after critical connectors, such as data transformations, calls to external services, and flow decisions. This will allow you to track the exact path of execution and pinpoint specific issues.
• Monitor these logs regularly to identify error patterns or signs of poor performance. With the Log connector configured in key areas, you get a more detailed overview of execution, making it easier to track problems.

Check trigger and queue settings

• Make sure that the pipeline trigger is configured correctly and that the queue expiration time is appropriate for the volume of data you are processing. Errors in the trigger or an expiry time that is too short can lead to the loss of important messages.

• Configure the trigger to meet the requirements of the data flow and adjust the time in the queue if necessary to ensure that the messages have enough time to be processed. Pay attention to the expiration time of the event queue and regularly check the lifetime of the messages in the queues.

Review pipeline implementation

• Analyze each pipeline connector to identify possible points of failure. If an error is detected in a specific connector (for example, REST or SOAP), check its configurations, such as timeout settings and the data structure sent or received.
• You can use an Assert to check whether the data returned by external services matches the expected format, which ensures a more robust integration.

Monitor deployment settings

• Deployment settings such as the number of Replicas, Concurrent executions, and memory allocation (Pipeline size) have a significant impact on pipeline performance. Ensure that the memory and replicas are appropriately sized for the expected workload, especially for pipelines with high demand.
• Adjust these settings as needed to ensure stable performance and minimize execution errors.

Use Digibee monitoring tools

The Digibee Integration Platform provides monitoring tools and insights to detect issues such as increased response times or unusual error rates. Enable these tools to monitor pipeline health in real time and receive proactive alerts about performance or configuration issues.

• Alerts Configuration: On this page, you can customize integration performance notifications based on metrics to take quick action to minimize operational impact. Learn more in the Alerts documentation.

• Monitor Insights: This feature provides you with insights into the health of completed integrations. Selecting an insight takes the user to the metrics of the executions that require attention, already filtered by the time period of occurrence. Further information can be found in the Monitor Insights documentation.

Manually reprocess executions

• This feature allows you to retry previously processed calls, which is useful in scenarios where the original call failed due to temporary issues such as network interruptions or errors in the target system.

• Manual reprocessing requires identifying the failed call and checking the previously used parameters. Depending on the configuration, you can keep the same parameters or change them. After reprocessing the call, perform a new search on the Pipeline logs page; it will be displayed with the value "manual" in the Source field.

Although manual reprocessing is an effective tool, it should be used with caution. In some cases, it may be necessary to investigate the causes of errors before simply retrying calls to avoid repeating the same errors.

Still having issues with your integration?

If so, open a ticket with our Support team. Here you can find out how to open a ticket and how long it is likely to take.

When you open the ticket, make sure you provide the relevant information:

• Pipeline name or Execution ID

• Previous analysis performed by the customer with proof of testing

• Other pertinent information to help identify and analyze

• Expected behavior versus observed behavior

• Error message with date and time

• Pipeline key (for errors or incidents)

• Environment (Prod or Test)

• Project name (if applicable)

Conclusion

By following the recommendations in this guide, you can troubleshoot pipelines on the Digibee Integration Platform more efficiently and independently. With an organized approach and the application of best practices — such as detailed log analysis, proper queue monitoring, and adjusting deployment settings — you can quickly identify and resolve common errors that affect integration execution.

The Digibee Integration Platform provides tools that, along with the troubleshooting procedures outlined, allow you to resolve many issues on your own. When necessary, use resources such as manual reprocessing and monitoring tools to validate corrections and ensure pipeline stability.