Digibee Documentation
Request documentationBook a demo
English
English
  • Quick start
  • Highlights
    • Release notes
      • Release notes 2025
        • April
        • March
        • February
        • January
      • Release notes 2024
        • December
        • November
        • October
        • September
        • August
          • Connectors release 08/20/2024
        • July
        • June
        • May
        • April
        • March
        • February
        • January
      • Release notes 2023
        • December
        • November
        • October
        • September
        • August
        • July
        • June
        • May
        • April
        • March
        • February
        • January
      • Release notes 2022
        • December
        • November
        • October
        • September
        • August
        • July
        • June
        • May
        • April
        • March
        • February
        • January
      • Release notes 2021
      • Release notes 2020
    • AI Pair Programmer
    • Digibeectl
      • Getting started
        • How to install Digibeectl on Windows
      • Digibeectl syntax
      • Digibeectl operations
  • Digibee in action
    • Use Cases in Action
      • Improving integration performance with API pagination
      • Automating file storage with Digibee
      • Reprocessing strategy in event-driven integrations
      • Key practices for securing sensitive information in pipelines with Digibee
      • OAuth2 for secure API access
      • Secure your APIs with JWT in Digibee
      • Integration best practices for developers on the Digibee Integration Platform
      • How to use Event-driven architecture on the Digibee Integration Platform
      • Dynamic file download with Digibee
      • Microservices: Circuit Breaker pattern for improving resilience
      • Error handling strategy in event-driven integrations
    • Troubleshooting
      • Integration guidance
        • How to resolve common pipeline issues
        • How to resolve Error 409: “You cannot update a pipeline that is not on draft mode”
        • How to resolve the "Pipeline execution was aborted" error
        • Integrated authentication with Microsoft Entra ID
        • How to resolve the "Failed to initialize pool: ONS configuration failed" error
        • How to perform IP address mapping with Progress database
        • How to build integration flows that send error notifications
        • How to send logs to external services
        • How JSONPath differs in connectors and the Execution panel
        • Using JSONPath to validate numbers with specific initial digits
        • How to analyze the "Network error: Failed to fetch" in the Execution panel
        • How to handle request payloads larger than 5MB
        • How to configure Microsoft Entra ID to display groups on the Digibee Integration Platform
        • How to build an HL7 message
      • Connectors behavior and configuration
        • Timeout in the Pipeline Executor connector
        • How to use DISTINCT and COUNT in the Object Store
        • Understanding @@DGB_TRUNCATED@@ on the Digibee Integration Platform
        • How to resolve names without a DNS - REST, SOAP, SAP (web protocols)
        • How to read and write files inside a folder
        • AuthToken Reuse for Salesforce connector
        • How to resolve the "Invalid payload" error in API Integration
        • Supported databases
          • Functions and uses for databases
      • Connectors implementation and usage examples
        • Google Storage: Usage scenarios
        • DB V2: Usage scenarios
        • For Each: Usage example
        • Template and its uses
        • Digibee JWT implementation
        • Email V1: Usage example (Deprecated)
      • JOLT applications
        • Transformer: Getting to know JOLT
        • Transformer: Transformations with JOLT
        • Transformer: Add values to list elements
        • Transformer: Operations overview
        • Transformer: Date formatting using split and concat
        • Transformer: Simple IF-ELSE logic with JOLT
      • Platform access and performance tips
        • How to solve login problems on the Digibee Integration Platform
        • How to receive updates from Digibee Status Page
        • How to clean the Digibee Integration Platform cache
      • Governance troubleshooting guidance
        • How to consume Internal API pipelines using ZTNA
        • How to use Internal API with and without a VPN
        • How to generate, convert, and register SSH Keys
        • mTLS authentication
          • How to configure mTLS on the Digibee Integration Platform
          • FAQs: Certificates in mTLS
        • How to connect Digibee to Oracle RAC
        • How to connect Digibee to SAP
        • How to connect Digibee to MongoDB Atlas using VPN
        • How to manage IPs on the Digibee Integration Platform
        • Configuring the Dropbox account
        • How to use your Gmail account with the Digibee email component (SMTP)
        • How to use the CORS policy on the Digibee Integration Platform
      • Deployment scenarios
        • Solving the “Out of memory” errors in deployment
        • Warning of route conflicts
    • Best practices
      • Best practices for building a pipeline
      • Best practices on validating messages in a consumer pipeline
      • Avoiding loops and maximizing pipeline efficiency
      • Naming: Global, Accounts, and API Keys
      • Pagination tutorial
        • Pagination tutorial - part 1
        • Pagination tutorial - part 2
        • Pagination tutorial - part 3
        • Pagination tutorial - part 4
      • Pagination example
      • Event-driven architecture
      • Notification model in event-driven integrations
      • OAuth2 integration model with Digibee
      • Best practices for error handling in pipelines
    • Digibee Academy
      • Integration Developer Bootcamp
  • Reference guides
    • Connectors
      • AWS
        • S3 Storage
        • SQS
        • AWS Secrets Manager
        • AWS Athena
        • AWS CloudWatch
        • AWS Elastic Container Service (ECS)
        • AWS Eventbridge
        • AWS Identity and Access Management (IAM)
        • AWS Kinesis
        • AWS Kinesis Firehose
        • AWS Key Management Service (KMS)
        • AWS Lambda
        • AWS MQ
        • AWS Simple Email Service (SES)
        • AWS Simple Notification System (SNS)
        • AWS Security Token Service (STS)
        • AWS Translate
      • Azure
        • Azure CosmosDB
        • Azure Event Hubs
        • Azure Key Vault
        • Azure ServiceBus
        • Azure Storage DataLake Service
        • Azure Storage Queue Service
      • Enterprise applications
        • SAP
        • Salesforce
        • Braintree
        • Facebook
        • GitHub
        • Jira
        • ServiceNow
        • Slack
        • Telegram
        • Twilio
        • WhatsApp
        • Wordpress
        • Workday
        • Zendesk
      • File storage
        • Blob Storage (Azure)
        • Digibee Storage
        • Dropbox
        • FTP
        • Google Drive
        • Google Storage
        • OneDrive
        • SFTP
        • WebDav V2
        • WebDav (Deprecated)
      • Files
        • Append Files
        • Avro File Reader
        • Avro File Writer
        • CSV to Excel
        • Excel
        • File Reader
        • File Writer
        • GZIP V2
        • GZIP V1 (Deprecated)
        • Parquet File Reader
        • Parquet File Writer
        • Stream Avro File Reader
        • Stream Excel
        • Stream File Reader
        • Stream File Reader Pattern
        • Stream JSON File Reader
        • Stream Parquet File Reader
        • Stream XML File Reader
        • XML Schema Validator
        • ZIP File
        • NFS
      • Flow
        • Delayer
      • Google/GCP
        • Google BigQuery
        • Google BigQuery Standard SQL
        • Google Calendar
        • Google Cloud Functions
        • Google Mail
        • Google PubSub
        • Google Secret Manager
        • Google Sheets
      • Industry solutions
        • HL7
        • Orderful
        • FHIR (Beta)
        • Mailgun API
        • Gupy Public API
        • Protheus: Billing and Inventory of Cost
        • HubSpot: Sales and CMS
      • Logic
        • Block Execution
        • Choice
        • Do While
        • For Each
        • Retry
        • Parallel Execution
      • Queues and messaging
        • Event Publisher
        • JMS
        • Kafka
        • RabbitMQ
      • Security
        • AES Cryptography
        • Asymmetric Cryptography
        • CMS
        • Digital Signature
        • JWT (Deprecated)
        • JWT V2
        • Google IAP Token
        • Hash
        • Digibee JWT (Generate and Decode)
        • LDAP
        • PBE Cryptography
        • PGP
        • RSA Cryptography
        • Symmetric Cryptography
      • Structured data
        • CassandraDB
        • DB V2
        • DB V1 (Deprecated)
        • DynamoDB
        • Google Big Table
        • Memcached
        • MongoDB
        • Object Store
        • Relationship
        • Session Management
        • Stored Procedure
        • Stream DB V3
        • Stream DB V1 (Deprecated)
        • ArangoDb
        • Caffeine Cache
        • Caffeine LoadCache
        • Couchbase
        • CouchDB
        • Ehcache
        • InfluxDB
      • Tools
        • Assert V2
        • Assert V1 (Deprecated)
        • Base64
        • CSV to JSON V2
        • CSV to JSON V1 (Deprecated)
        • HL7 Message Transformer (Beta)
        • HTML to PDF
        • Transformer (JOLT) V2
        • JSLT
        • JSON String to JSON Transformer
        • JSON to JSON String Transformer
        • JSON to XML Transformer
        • JSON to CSV V2
        • JSON to CSV Transformer (Deprecated)
        • JSON Path Transformer V2
        • JSON Path Transformer
        • JSON Transformer
        • Log
        • Pipeline Executor
        • SSH Remote Command
        • Script (JavaScript)
        • Secure PDF
        • Store Account
        • Template Transformer
        • Throw Error
        • Transformer (JOLT)
        • Validator V1 (Deprecated)
        • Validator V2
        • XML to JSON Transformer
        • XML Transformer
        • JSON Generator (Mock)
      • Web protocols
        • Email V2
        • Email V1 (Deprecated)
        • REST V2
        • REST V1 (Deprecated)
        • SOAP V1 (Deprecated)
        • SOAP V2
        • SOAP V3
        • WGet (Download HTTP)
        • gRPC
    • Triggers
      • Web Protocols
        • API Trigger
        • Email Trigger
        • Email Trigger V2
        • HTTP Trigger
        • HTTP File Trigger
          • HTTP File Trigger - Downloads
          • HTTP File Trigger - Uploads
        • REST Trigger
      • Scheduling
        • Scheduler Trigger
      • Messaging and Events
        • Event Trigger
        • JMS Trigger
        • Kafka Trigger
        • RabbitMQ Trigger
      • Others
        • DynamoDB Streams Trigger
        • HL7 Trigger
        • Salesforce Trigger - Events
    • Double Braces
      • How to reference data using Double Braces
      • Double Braces functions
        • Math functions
        • Utilities functions
        • Numerical functions
        • String functions
        • JSON functions
        • Date functions
        • Comparison functions
        • File functions
        • Conditional functions
      • Double Braces autocomplete
  • Development cycle
    • Build
      • Canvas
        • AI Assistant
        • Smart Connector User Experience
        • Execution panel
        • Design and Inspect Mode
        • Linter: Canvas building validation
        • Connector Mocking
      • Pipeline
        • How to create a pipeline
        • How to scaffold a pipeline using an OpenAPI specification
        • How to create a project
        • Pipeline version history
        • Pipeline versioning
        • Messages processing
        • Subpipelines
        • Pipeline documentation with AI
      • Capsules
        • How to use Capsules
          • How to create a Capsule collection
            • Capsule header dimensions
          • How to create a Capsule group
          • How to configure a Capsule
          • How to build a Capsule
          • How to test a Capsule
          • How to save a Capsule
          • How to publish a Capsule
          • How to change a Capsule collection or group
          • How to archive and restore a Capsule
        • Capsules versioning
        • Public capsules
          • SAP
          • Digibee Tools
          • Google Sheets
          • Gupy
          • Send notifications via email
          • Totvs Live
          • Canvas LMS
        • AI Assistant for Capsules Docs Generation
    • Run
      • Run concepts
        • Autoscalling
      • Deployment
        • Deploying a pipeline
        • How to redeploy a pipeline
        • How to promote pipelines across environments
        • How to check the pipeline deployment History
        • How to rollback to a previous deployment version
        • Using deployment history advanced functions
        • Pipeline deployment status
      • How warnings work on pipelines in Run
    • Monitor
      • Monitor Insights (Beta)
      • Completed executions
        • Pipeline execution logs download
      • Pipeline logs
      • Pipeline Metrics
        • Pipeline Metrics API
          • How to set up Digibee API metrics with Datadog
          • How to set up Digibee API metrics with Prometheus
        • Connector Latency
      • Alerts
        • How to create an alert
        • How to edit an alert
        • How to activate, deactivate or duplicate an alert
        • How to delete an alert
        • How to configure alerts on Slack
        • How to configure alerts on Telegram
        • How to configure alerts through a webhook
        • Available metrics
        • Best practices about alerts
        • Use cases for alerts
      • VPN connections monitoring
        • Alerts for VPN metrics
  • Connectivity management
    • Connectivity
    • Zero Trust Network Access (ZTNA)
      • Prerequisites for using ZTNA
      • How to view connections (Edge Routers)
      • How to view the Network Mappings associated with an Edge Router
      • How to add new ZTNA connections (Edge Routers)
      • How to delete connections (Edge Routers)
      • How to view routes (Network Mapping)
      • How to add new routes (Network Mapping)
      • How to add routes in batch for ZTNA
      • How to edit routes (Network Mapping)
      • How to delete routes (Network Mapping)
      • How to generate new keys (Edge Router)
      • How to change the environment of Edge routers
      • ZTNA Inverse Flow
      • ZTNA Groups
    • Virtual Private Network (VPN)
  • Platform administration
    • Administration
      • Audit
      • Access control
        • Users
        • Groups
        • Roles
          • List of permissions by service
          • Roles and responsibilities: Governance and key stakeholder identification
      • Identity provider integration
        • How to integrate an identity provider
        • Authentication rules
        • Integration of IdP groups with Digibee groups
          • How to create a group integration
          • How to test a group integration
          • How to enable group integrations
          • How to edit a group integration
          • How to delete a group integration
      • User authentication and authorization
        • How to activate and deactivate two-factor authentication
        • Login flow
      • Organization groups
    • Settings
      • Globals
        • How to create Globals
        • How to edit or delete Globals
        • How to use Globals
      • Accounts
        • Configuring each account type
        • Monitor changes to account settings in deployed pipelines
        • OAuth2 Architecture
          • Registration of new OAuth providers
      • Consumers (API Keys)
      • Relationship model
      • Multi-Instance
        • Deploying a multi-instance pipeline
      • Log Streaming
        • How to use Log Streaming with Datadog
    • Governance
      • Policies
        • Security
          • Internal API access policy
          • External API access policy
          • Sensitive fields policy
        • Transformation
          • Custom HTTP header
          • CORS HTTP header
        • Limit of Replicas policy
    • Licensing
      • Licensing models
        • Consumption Based model
      • Capacity and quotas
      • License consumption
    • Digibee APIs
      • How to create API credentials
  • Digibee concepts
    • Pipeline Engine
      • Digibee Integration Platform Pipeline Engine v2
      • Support Dynamic Accounts (Restricted Beta)
    • Digibee Integration Platform Dedicated SaaS
      • Digibee Integration Platform architecture on Dedicated Saas model
      • Requirements for Digibee Dedicated Saas model
      • Site-to-Site VPN for dedicated SaaS customer support
      • Dedicated Saas customer responsibilities
      • Custom Images of Kubernetes Nodes
      • Digibee Dedicated SaaS installation on AWS
        • How to install requirements before installing Digibee Integration Platform on EKS
        • Permissions to use Digibee Integration Platform on EKS
        • How to create custom nodes for EKS (Golden Images)
    • Introduction to ZTNA
  • Help & FAQ
    • Digibee Customer Support
    • Request documentation, suggest features, or send feedback
    • Beta Program
    • Security and compliance
    • About Digibee
Powered by GitBook
On this page
  • Using Fail On Error and OnException
  • Fail On Error
  • OnException
  • Error summary in loop connectors (For Each, Do While)
  • Reviewing timeouts for external connection connectors in relation to the trigger
  • Importance of aligning timeouts
  • Impact of response times on the pipeline
  • Tips for setting timeout values
  • Error validation in pipelines with heavy Object Store usage
  • Preventing errors in concurrent operations
  • Implementation example
  • Specific behavior of the Retry connector
  • “Connection redundancy” in the DB V2 connector
  • Database connection management
  • Optimization recommendations
  • Conclusion

Was this helpful?

  1. Digibee in action
  2. Best practices

Best practices for error handling in pipelines

Learn how to implement efficient error handling in your pipeline and improve the reliability of your integrations.

PreviousOAuth2 integration model with DigibeeNextDigibee Academy

Last updated 22 days ago

Was this helpful?

Proper error handling is essential to ensure the stability, predictability, and reliability of the integrations built on the Digibee Integration Platform. Without effective error management, pipelines can behave unexpectedly, making maintenance difficult and impacting critical processes. This article presents best practices for efficient error handling.

Using Fail On Error and OnException

Fail On Error

This option is available in the configurations of most Platform connectors. When enabled, the pipeline execution is interrupted. In connectors that use , an exception is sent to the OnException flow.

OnException

is present in some connectors with subpipelines such as , , and , and plays an important role in capturing controlled errors. It allows error handling to be centralized at a single point in the pipeline, making future adjustments, analysis, and additions easier, and avoiding repetition of logic throughout the pipeline.

Error summary in loop connectors (For Each, Do While)

The OnException of loop connectors such as For Each doesn’t interrupt the execution of the loop. This makes it possible to collect or summarize all exceptions that occur during execution and handle them after the loop ends. For more information, see the official .

A recommended practice is to store an error summary in the pipeline memory to display it later using the Add item to session (array) capsule. This capsule inserts records into an array called resultList, which can be retrieved later.

A JSON object must be included in the payload to properly add an item to the array:

Reviewing timeouts for external connection connectors in relation to the trigger

Importance of aligning timeouts

The trigger timeout should be greater than the sum of the timeouts of all external connectors. This ensures that the pipeline has enough time to handle potential errors before it’s terminated.

Impact of response times on the pipeline

If external connectors have long timeouts and the trigger has a lower value, the pipeline may terminate before the requests are completed, leading to inconsistencies and making error identification more difficult.

Tips for setting timeout values

  • Calculate the sum of timeouts for external connectors and add a safety margin to the trigger timeout.

  • Avoid excessively high values, as they may affect the platform's performance.

  • Consider internal pipeline processes, such as transformations and error handling, when defining the trigger timeout.

  • Regularly monitor connector execution time and adjust timeouts as needed.

Error validation in pipelines with heavy Object Store usage

Preventing errors in concurrent operations

Consider the following practices:

  • Enable the Fail On Error option in connectors accessing the Object Store to automatically capture failures.

  • Monitor simultaneous access to the Object Store to avoid overload and concurrency-related failures.

Implementation example

If a pipeline reads from the Object Store and then updates the data, an error may occur if another process modifies or deletes that data before the update. To mitigate this risk, it’s recommended to:

  • Verify the data after reading, ensuring it’s still available.

These practices help minimize failures and improve reliability in Object Store interactions.

Specific behavior of the Retry connector

Recommendation: To ensure Retry continues its attempts until the configured limit is reached, avoid using Throw Error in OnException. This allows the pipeline to continue executing retries before completely failing.

“Connection redundancy” in the DB V2 connector

For example, with two servers and a 30-second timeout:

  • The first attempt will use 30 seconds to connect to the first server.

  • If it fails, the connector will try the second server, applying another 30 seconds.

Database connection management

Connection management should align with integration flow needs. Poor configuration may lead to:

  • Too many open connections, affecting both database and pipeline performance.

  • Unnecessary resource use, compromising scalability and stability.

Optimization recommendations

Reviewing and adjusting connection parameters can improve pipeline performance and reduce database impact. Key parameters include:

  • Pool Size By Actual Consumers – Sets pool size based on the number of actual consumers.

  • Exclusive DB Pool – Indicates whether the pipeline has a dedicated pool.

  • Custom Pool – Allows defining a custom pool.

  • Keep Connection – Keeps the connection active between executions, which may impact resource usage.

For high-volume operation flows, reviewing these settings is essential for performance and avoiding overload.

Conclusion

Proper error management in the Digibee Integration Platform is essential to ensure integration stability and efficiency. As we've seen throughout this article, it's important to:

  • Properly configure the Fail On Error and OnException connectors.

  • Understand specific behaviors of the Retry and DB V2 connectors.

  • Define appropriate timeouts for external connectors and align them with the trigger.

  • Use validations in concurrent flows with Object Store to avoid inconsistencies.

Applying these best practices makes the pipeline more resilient and helps with faster issue detection and resolution. We also recommend periodically reviewing your error-handling strategies and configurations to keep up with evolving integration needs.

With a structured approach, you can ensure that the solutions developed on Digibee are robust, reliable, and scalable.

If the capsule is not available, it’s also possible to perform this directly in the pipeline using other available connectors, like and :

This approach should be used with caution, as memory consumption increases as records are added to the array. For large iteration volumes, it's recommended to use with the Isolated parameter enabled, as it’s a more appropriate alternative.

When configuring a pipeline, it’s crucial to consider the timeout of external connection connectors (, , , ) in relation to the timeout defined in the trigger.

For example, if a pipeline has three connectors, each with a 30-second timeout, the total execution time can reach 90 seconds. If the trigger is configured with a timeout shorter than this value, the pipeline will terminate before the connectors finish their execution.

When a pipeline performs multiple CRUD (Create, Read, Update, Delete) operations on an collection, especially concurrently across different flows, it’s critical to follow best practices to avoid inconsistencies and unexpected failures.

Implement validations after each operation using or connectors to ensure data was correctly manipulated.

Use the connector to validate the presence of expected data.

Apply the connector to define alternative routes if the data has been changed or removed.

The Retry connector has its own rules for error handling, described in . It's important to understand how the OnProcess and OnException blocks behave in the Retry context.

If OnProcess contains a connector or has the Fail On Error parameter enabled, and OnException also includes a Throw Error, Retry will immediately stop its attempts, even if there are retries remaining.

When the connector’s connection string includes two or more servers, the timeout defined in the Custom Connection Properties field is applied individually to each server.

Session Management
JSON Generator
Object Store
REST
DB
SOAP
MongoDB
REST
Object Store
Assert
Choice
Assert
Choice
Throw Error
DB V2
subflows
Do While
For Each
Retry
For Each documentation
OnException
this documentation