Centralized reprocessing engine model on Digibee

Reprocessing request

An event-type pipeline receives the message and stores it in a temporary Object Store named reprocess. This pipeline can be scaled both horizontally (by increasing replicas) and vertically (by increasing size) to support your company’s request volume and message sizes.

It ensures the payload is correctly formatted for reprocessing and determines key information, such as the retry timestamp,based on the intervalRetry field, which represents the number of minutes to wait before the next attempt.

In the diagram below, this step is handled by the add-reprocess-event pipeline:

Example payload sent to the reprocessing engine:

{
  "payload": {
    "customerId": 123456,
    "customerName": "Raimundo Nonato",
    "customerCity": "Maranguape",
    "customerState": "CE",
    "customerJob": "Professor"
  },
  "pipeOrEventName": "update-customer-event",
  "pipeType": "event",
  "maxRetry": 3,
  "intervalRetry": 5
}

The message must include, in addition to the payload itself:

• The name of the original event or pipeline (if not an event-type).

• The trigger type of the source pipeline (pipeType).

• Maximum number of reprocessing attempts.

• The interval (in minutes) between attempts.

Listing and publishing pending reprocesses

Once the messages are properly stored in the reprocess Object Store, the next steps are:

1. List messages ready for retry A scheduler-type pipeline is responsible for listing messages whose retry interval has been reached. The scheduler must run at the smallest interval required for any reprocessing scenario. For example, if the shortest retry interval is 5 minutes, the scheduler should execute every 5 minutes.

2. Publish message ID Each retrieved message ID is then published to the next event. Because every message includes a retry timestamp, the scheduler filters and publishes only those that are due. This process is handled by the reprocess-timer pipeline (as shown in the diagram):

The reprocess-timer pipeline should list pending reprocessing IDs (in a paginated manner) and publish them individually to the reprocessing-event. It can be configured with minimal resources, as its function is limited to listing and publishing.

Reprocessing and resubmitting to the original pipeline

The reprocessing-event is consumed by a pipeline of the same name. For each published ID in the event, the pipeline:

• Retrieves the original payload from the reprocess Object Store.

• Checks whether the retry limit has been reached.

If the retry limit has not been reached, the message is resent to the original flow:

• If the original flow is event-based, it's sent via an event.

• Otherwise, the Pipeline Executor connector is used.

If the retry limit has been reached, the pipeline should publish a message to the company’s standard notification mechanism (if one exists, or one can be built).

The original flow must be prepared to receive the payload from the reprocessing engine. For example, if the original pipeline is a scheduler that assembles the payload throughout the flow from a file server or database, a new route may be needed to receive the input and continue the flow once the payload is ready.

The diagram below represents the reprocess-event pipeline:

Example payload sent back to the original flow:

{
  "customerId": 123456,
  "customerName": "Raimundo Nonato",
  "customerCity": "Maranguape",
  "customerState": "CE",
  "customerJob": "Professor",
  "_reprocess": {
    "forwardTo": {
      "name": "update-invoice-event",
      "type": "event"
    },
    "retry": {
      "intervalRetry": 5,
      "max": 2,
      "current": 1,
      "lastRetry": 1687462056888
    }
  },
  "_oId": "b1ba4fdd-4c0c-4991-a153-b715ebfe4c9c"
}

Note that the payload sent back to the original processing flow includes a retry field with detailed retry information. The reprocess-event pipeline should increment the current retry value each time the message is resent. If the original flow needs to return the message to the reprocessing engine, it should include this value in the next call.

Like the input pipeline, this return pipeline can be scaled horizontally and vertically to meet performance and volume demands.