# Best practices on validating messages in a consumer pipeline

When managing integrations that require segregation, splitting the integration into multiple pipelines can improve clarity and efficiency. Below is an example scenario and recommended practices to follow.

## **Example scenario: Processing purchase orders in e-commerce**

Imagine an integration that processes e-commerce orders. To handle the high volume of orders efficiently, you decide to divide the integration into two pipelines:

* **Pipeline A:** Retrieves orders from an API.
* **Pipeline B (consumer):** Receives product data extracted from the orders and sends it to another API.

1. ### **Using Events for process segregation**

When integrating processes that require segregation, use **Events** to delegate specific tasks to another pipeline (the consumer).

In these cases, it's essential to:

* Define precisely what data the consumer will receive.
* Establish a clear contract between the processes.

This ensures that the message transmitted is accurate, concise, and valid, facilitating communication and reducing potential errors between the two processes.

2. ### **Optimizing the data sent to the consumer**

In **Pipeline A**, the orders API returns the following JSON:

```json
{
  "status": 200,
  "body": {
    "orderNum": "001",
    "products": [
      {
        "code": "123",
        "name": "Product A"
      },
      {
        "code": "456",
        "name": "Product B"
      }
    ],
    "customer": {
      "name": "Example Customer",
      "cpf": "123.456.789-10",
      "email": "customer@email.com",
      "addresses": {
        "street": "Example Street",
        "number": "10",
        "zipCode": "01010-10"
      }
    }
  },
  "headers": {
    "Cache-Control": "no-cache,must-revalidate,max-age=0,no-store,private",
    "Content-Type": "application/json;charset=UTF-8",
    "Date": "Wed, 01 Jul 2020 19:04:46 GMT",
    "Expires": "Thu, 01 Jan 1970 00:00:00 GMT",
    "Set-Cookie": "BrowserId=up7LXrwv46wesv5NEeg9ps_4AgB_",
    "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
    "Transfer-Encoding": "chunked",
    "Vary": "Accept-Encoding",
    "X-B3-Sampled": "0",
    "X-B3-SpanId": "8c419a93i3bsi0=9d8e54316",
    "X-B3-TraceId": "8c419a9389gba9y54316",
    "X-Content-Type-Options": "nosniff",
    "X-ReadOnlyMode": "false",
    "X-XSS-Protection": "1; mode=block"
  }
}
```

However, for **Pipeline B**, only the "products" array is needed:

```json
{
  "products": [
    {
      "code": "123",
      "name": "Product A"
    },
    {
      "code": "456",
      "name": "Product B"
    }
  ]
}
```

Sending the complete JSON to **Pipeline B** would introduce unnecessary data. Instead, use a [**Transformer (JOLT)**](https://app.gitbook.com/s/EKM2LD3uNAckQgy1OUyZ/connectors/tools/jolt-v2), [**JSON Generator**](https://app.gitbook.com/s/EKM2LD3uNAckQgy1OUyZ/connectors/tools/json-generator), or [**Event**](https://app.gitbook.com/s/EKM2LD3uNAckQgy1OUyZ/connectors/queues-and-messaging/event-publisher) connector to process the JSON so only essential “products” information is sent.

#### **Why avoid sending full JSON payloads?**

Sending the entire JSON can clutter **Pipeline B** with irrelevant data, possibly including hundreds of rows when only a fraction is relevant. Limiting the data transfer to essential information improves efficiency, regardless of data volume.

Additional benefits:

* **Execution logs:** Smaller messages result in clearer, more manageable logs.
* **Maintainability:** Streamlined data makes troubleshooting and updating easier, especially when different people are involved in maintaining the integration.

3. ### **Validating data and contracts**

For robust data validation, **Pipeline B** should use:

* [**Validator connector**](https://app.gitbook.com/s/EKM2LD3uNAckQgy1OUyZ/connectors/tools/validator-v2)**:** Use a JSON Schema to precisely define the JSON required by the consumer.
* [**Choice connector**](https://app.gitbook.com/s/EKM2LD3uNAckQgy1OUyZ/connectors/logic/choice)**:** Use JSONPath to validate specific elements in the received message. For example, using expressions like `$.produtos` or `$.produtos[?(@.codigo && @.nome)]` ensures only valid data passes through.

If these conditions are not met, an error flow will be triggered and interrupt the process because it has not fulfilled the specified contract.