Error handling strategy in event-driven integrations
PreviousMicroservices: Circuit Breaker pattern for improving resilienceNextBest practices for building a pipeline
Last updated
Was this helpful?
Last updated
Was this helpful?
Was this helpful?
Event-driven architectures (EDA) enable integrations to be structured as multiple pipelines, each responsible for a specific part of the process. These pipelines work together, forming a cohesive integration that handles data flow and processing. To maintain reliability, error management is essential, especially as integrations grow in complexity.
This use case explores how an error-handling pipeline interacts with other pipelines, presenting its benefits and implementation within the Digibee Integration Platform as a global error management solution.
Instead of each pipeline managing errors individually, they standardize error formatting and send them to a centralized error-handling pipeline. This approach ensures consistent validation, traceability, and reduces duplication across the integration.
During processing, errors are classified to determine the appropriate handling strategy:
Retriable errors (server-side): Caused by temporary issues like network outages or service downtime. These errors can be reprocessed before being sent to the error-handling pipeline if retries fail.
Non-retriable errors (client-side): Malformed payloads or missing required data. These errors are published to the error-handling pipeline, where they are logged and processed accordingly.
By centralizing error handling, pipelines avoid duplicating logic, ensuring a uniform approach to error management while reducing redundancy.
Consider an event-driven integration involving pipelines for querying records, processing events, and reprocessing failed events. Temporary issues, like network outages or service unavailability, can cause events to fail in the processing pipeline, necessitating error handling.
For instance, in a retail integration, purchase orders go through pipelines for inventory checks, payment validation, and shipping. If a network issue prevents payment validation, the processing pipeline logs the failure, and the reprocessing pipeline retries. If retries are exhausted, the error is published to Digibee’s internal event broker, where the error-handling pipeline listens for these events and processes them asynchronously.
Depending on the architecture, multiple pipelines can subscribe to the same error event. This decoupling enhances performance and scalability by removing error-handling logic from the main processing workflows.
A standardized error message ensures consistency across pipelines. Below is an example payload used in the error-handling pipeline. This structure includes attributes such as pipeline metadata, process identifiers, and timestamps to facilitate traceability:
{
"errorType": "server",
"message": "Timeout while calling payment service",
"payload": {"orderId": "12345", "amount": 100.00},
"errorDate": "2025-01-31 12:45:00",
"orderId": "12345",
"processId": "abc-987",
"email": "[email protected]"
}Once an error occurs, the processing pipeline publishes it to the error-handling pipeline with a standardized format. The error-handling pipeline follows these key steps:
Event Trigger: In this microservices and EDA approach, multiple pipelines publish their errors to the centralized error-handling pipeline via the Event Publisher. The Event Trigger listens for these events and initiates the error-handling process, without modifying the incoming data.
Validation: As a best practice in EDA, validation ensures that errors conform to a standardized format, allowing multiple publishers to send structured error messages to a single destination. This error payload is validated with a Validator connector against predefined rules to ensure proper formatting and completeness.
Control update: If used, the status of the failed event is updated in a temporary control database, such as Object Store, to prevent reprocessing, ensuring errors are stored only as long as needed.
Optional routing: Depending on the integration's requirements, errors can trigger additional actions, such as notifying stakeholders, updating error tracking databases, forwarding logs to monitoring systems, or, in this example, directing the error to an external API for error statistics management using the REST connector.
Validation of API response: The API response should be validated using an Assert v2 Connector or a Choice Connector to check for a 200 OK status. If unsuccessful, alternative handling, such as logging or retrying, can be triggered.
Implementing an error-handling pipeline in an event-driven architecture improves reliability by centralizing error management across multiple pipelines. With Digibee’s Integration Platform, this process becomes faster and more intuitive, eliminating the need for complex custom implementations.
By leveraging Digibee's capabilities, teams can seamlessly integrate error processing, ensuring structured validation, logging, treatment, or routing without disrupting business workflows. The Platform’s intuitive configuration options simplify implementation while maintaining control over how errors are handled.
For a better understanding of related strategies, check out our use cases:
Reprocessing strategy in event-driven integrations: Learn how to retry failed events before escalating them to the error-handling pipeline, preventing unnecessary failures.
How to use Event-driven architecture on the Digibee Integration Platform: Understand how to design scalable, decoupled integrations using event-driven principles.
By integrating this pipeline into the Digibee Integration Platform, teams can enhance the reliability and maintainability of their integrations while reducing operational complexity.
To learn more about error-handling pipelines and event-driven architectures, explore our Documentation Portal or visit the Digibee Academy for hands-on challenges and courses.
For feedback or suggestions on this Use Case, feel free to share your thoughts through our Feedback Form.