Best practices on validating messages in a consumer pipeline

Learn the best practices for passing messages between processes and validating messages received by a consumer.

When managing integrations that require segregation, splitting the integration into multiple pipelines can improve clarity and efficiency. Below is an example scenario and recommended practices to follow.

Example scenario: Processing purchase orders in e-commerce

Imagine an integration that processes e-commerce orders. To handle the high volume of orders efficiently, you decide to divide the integration into two pipelines:

• Pipeline A: Retrieves orders from an API.

• Pipeline B (consumer): Receives product data extracted from the orders and sends it to another API.

1. Using Events for process segregation

When integrating processes that require segregation, use Events to delegate specific tasks to another pipeline (the consumer).

In these cases, it's essential to:

• Define precisely what data the consumer will receive.

• Establish a clear contract between the processes.

This ensures that the message transmitted is accurate, concise, and valid, facilitating communication and reducing potential errors between the two processes.

1. Optimizing the data sent to the consumer

In Pipeline A, the orders API returns the following JSON:

{
  "status": 200,
  "body": {
    "orderNum": "001",
    "products": [
      {
        "code": "123",
        "name": "Product A"
      },
      {
        "code": "456",
        "name": "Product B"
      }
    ],
    "customer": {
      "name": "Example Customer",
      "cpf": "123.456.789-10",
      "email": "[e-mail]",
      "addresses": {
        "street": "Example Street",
        "number": "10",
        "zipCode": "01010-10"
      }
    }
  },
  "headers": {
    "Cache-Control": "no-cache,must-revalidate,max-age=0,no-store,private",
    "Content-Type": "application/json;charset=UTF-8",
    "Date": "Wed, 01 Jul 2020 19:04:46 GMT",
    "Expires": "Thu, 01 Jan 1970 00:00:00 GMT",
    "Set-Cookie": "BrowserId=up7LXrwv46wesv5NEeg9ps_4AgB_",
    "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
    "Transfer-Encoding": "chunked",
    "Vary": "Accept-Encoding",
    "X-B3-Sampled": "0",
    "X-B3-SpanId": "8c419a93i3bsi0=9d8e54316",
    "X-B3-TraceId": "8c419a9389gba9y54316",
    "X-Content-Type-Options": "nosniff",
    "X-ReadOnlyMode": "false",
    "X-XSS-Protection": "1; mode=block"
  }
}

However, for Pipeline B, only the "products" array is needed:

{
  "products": [
    {
      "code": "123",
      "name": "Product A"
    },
    {
      "code": "456",
      "name": "Product B"
    }
  ]
}

Sending the complete JSON to Pipeline B would introduce unnecessary data. Instead, use a Transformer (JOLT), JSON Generator, or Event connector to process the JSON so only essential “products” information is sent.

Why avoid sending full JSON payloads?

Sending the entire JSON can clutter Pipeline B with irrelevant data, possibly including hundreds of rows when only a fraction is relevant. Limiting the data transfer to essential information improves efficiency, regardless of data volume.

Additional benefits:

• Execution logs: Smaller messages result in clearer, more manageable logs.

• Maintainability: Streamlined data makes troubleshooting and updating easier, especially when different people are involved in maintaining the integration.

1. Validating data and contracts

For robust data validation, Pipeline B should use:

• Validator connector: Use a JSON Schema to precisely define the JSON required by the consumer.

• Choice connector: Use JSONPath to validate specific elements in the received message. For example, using expressions like $.produtos or $.produtos[?(@.codigo && @.nome)] ensures only valid data passes through.

If these conditions are not met, an error flow will be triggered and interrupt the process because it has not fulfilled the specified contract.