Was this helpful?
Was this helpful?
When managing integrations that require segregation, splitting the integration into multiple pipelines can improve clarity and efficiency. Below is an example scenario and recommended practices to follow.
Imagine an integration that processes e-commerce orders. To handle the high volume of orders efficiently, you decide to divide the integration into two pipelines:
Pipeline A: Retrieves orders from an API.
Pipeline B (consumer): Receives product data extracted from the orders and sends it to another API.
When integrating processes that require segregation, use Events to delegate specific tasks to another pipeline (the consumer).
In these cases, it's essential to:
Define precisely what data the consumer will receive.
Establish a clear contract between the processes.
This ensures that the message transmitted is accurate, concise, and valid, facilitating communication and reducing potential errors between the two processes.
In Pipeline A, the orders API returns the following JSON:
However, for Pipeline B, only the "products" array is needed:
Sending the entire JSON can clutter Pipeline B with irrelevant data, possibly including hundreds of rows when only a fraction is relevant. Limiting the data transfer to essential information improves efficiency, regardless of data volume.
Additional benefits:
Execution logs: Smaller messages result in clearer, more manageable logs.
Maintainability: Streamlined data makes troubleshooting and updating easier, especially when different people are involved in maintaining the integration.
For robust data validation, Pipeline B should use:
If these conditions are not met, an error flow will be triggered and interrupt the process because it has not fulfilled the specified contract.
Sending the complete JSON to Pipeline B would introduce unnecessary data. Instead, use a , , or connector to process the JSON so only essential “products” information is sent.
: Use a JSON Schema to precisely define the JSON required by the consumer.
: Use JSONPath to validate specific elements in the received message. For example, using expressions like $.produtos
or $.produtos[?(@.codigo && @.nome)]
ensures only valid data passes through.
Learn the best practices for passing messages between processes and validating messages received by a consumer.