# Best practices on validating messages in a consumer pipeline

When managing integrations that require segregation, splitting the integration into multiple pipelines can improve clarity and efficiency. Below is an example scenario and recommended practices to follow.

## **Example scenario: Processing purchase orders in e-commerce**

Imagine an integration that processes e-commerce orders. To handle the high volume of orders efficiently, you decide to divide the integration into two pipelines:

* **Pipeline A:** Retrieves orders from an API.
* **Pipeline B (consumer):** Receives product data extracted from the orders and sends it to another API.

1. ### **Using Events for process segregation**

When integrating processes that require segregation, use **Events** to delegate specific tasks to another pipeline (the consumer).

In these cases, it's essential to:

* Define precisely what data the consumer will receive.
* Establish a clear contract between the processes.

This ensures that the message transmitted is accurate, concise, and valid, facilitating communication and reducing potential errors between the two processes.

2. ### **Optimizing the data sent to the consumer**

In **Pipeline A**, the orders API returns the following JSON:

```json
{
  "status": 200,
  "body": {
    "orderNum": "001",
    "products": [
      {
        "code": "123",
        "name": "Product A"
      },
      {
        "code": "456",
        "name": "Product B"
      }
    ],
    "customer": {
      "name": "Example Customer",
      "cpf": "123.456.789-10",
      "email": "customer@email.com",
      "addresses": {
        "street": "Example Street",
        "number": "10",
        "zipCode": "01010-10"
      }
    }
  },
  "headers": {
    "Cache-Control": "no-cache,must-revalidate,max-age=0,no-store,private",
    "Content-Type": "application/json;charset=UTF-8",
    "Date": "Wed, 01 Jul 2020 19:04:46 GMT",
    "Expires": "Thu, 01 Jan 1970 00:00:00 GMT",
    "Set-Cookie": "BrowserId=up7LXrwv46wesv5NEeg9ps_4AgB_",
    "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
    "Transfer-Encoding": "chunked",
    "Vary": "Accept-Encoding",
    "X-B3-Sampled": "0",
    "X-B3-SpanId": "8c419a93i3bsi0=9d8e54316",
    "X-B3-TraceId": "8c419a9389gba9y54316",
    "X-Content-Type-Options": "nosniff",
    "X-ReadOnlyMode": "false",
    "X-XSS-Protection": "1; mode=block"
  }
}
```

However, for **Pipeline B**, only the "products" array is needed:

```json
{
  "products": [
    {
      "code": "123",
      "name": "Product A"
    },
    {
      "code": "456",
      "name": "Product B"
    }
  ]
}
```

Sending the complete JSON to **Pipeline B** would introduce unnecessary data. Instead, use a [**Transformer (JOLT)**](/documentation/connectors-and-triggers/connectors/tools/jolt-v2.md), [**JSON Generator**](/documentation/connectors-and-triggers/connectors/tools/json-generator.md), or [**Event**](/documentation/connectors-and-triggers/connectors/queues-and-messaging/event-publisher.md) connector to process the JSON so only essential “products” information is sent.

#### **Why avoid sending full JSON payloads?**

Sending the entire JSON can clutter **Pipeline B** with irrelevant data, possibly including hundreds of rows when only a fraction is relevant. Limiting the data transfer to essential information improves efficiency, regardless of data volume.

Additional benefits:

* **Execution logs:** Smaller messages result in clearer, more manageable logs.
* **Maintainability:** Streamlined data makes troubleshooting and updating easier, especially when different people are involved in maintaining the integration.

3. ### **Validating data and contracts**

For robust data validation, **Pipeline B** should use:

* [**Validator connector**](/documentation/connectors-and-triggers/connectors/tools/validator-v2.md)**:** Use a JSON Schema to precisely define the JSON required by the consumer.
* [**Choice connector**](/documentation/connectors-and-triggers/connectors/logic/choice.md)**:** Use JSONPath to validate specific elements in the received message. For example, using expressions like `$.produtos` or `$.produtos[?(@.codigo && @.nome)]` ensures only valid data passes through.

If these conditions are not met, an error flow will be triggered and interrupt the process because it has not fulfilled the specified contract.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.digibee.com/documentation/resources/best-practices/validating-messages-on-a-consumer.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.