How to integrate Distribution Centers using Multi-Instance

Learn how Multi-Instance enables a single pipeline to process messages from multiple Distribution Centers with independent queue configurations.

Overview

This article explains how to use the Digibee Integration Platform's Multi-Instance feature to solve a scenario in which a retail company's core system needs to consume stock information from multiple Distribution Centers, each with its own messaging queue.

Scenario

Imagine a retail network where Distribution Centers (DCs) publish stock updates to a RabbitMQ broker, which then sends this information to the ERP (Enterprise Resource Planning).

Normally, a single pipeline with a RabbitMQ Trigger pointing to the broker would be enough. However, if each DC has a different queue or even a different host, the trigger cannot consume messages from multiple sources.

The Multi-Instance feature solves this challenge by allowing you to:

• Maintain a single pipeline with the same processing logic.

• Deploy dedicated instances for each DC, with specific parameters (host, port, queue, among others).

This way, processing remains standardized while configurations vary as needed.

The pipeline logic must be identical across all instances. It’s a best practice to keep the same stages in every instance, avoiding conditional processing based on the instance.

Architecture

Each Distribution Center (DC1, DC2, and DC3) sends stock updates through RabbitMQ (AMQP).

In Digibee, a Multi-Instance pipeline configured with small size, 10 consumers, and 1 replica collects the messages, processes the information, and sends it to the ERP, where it’s stored in a SQL database.

This architecture ensures centralization, automation, and scalability for stock management.

Putting the theory into practice

Follow the steps below to build the solution presented above.

1

Create the Multi-Instance model

On the Platform home page, click the Settings icon in the upper-right corner, access the Multi-Instance menu, and click Create.

Fill in the fields:

• Name: warehouse-stock-queue

• Description: Add a description explaining the purpose of this model, for example: “Multi-Instance model for consuming independent RabbitMQ broker queues, with a dedicated instance for each distribution center.”

• Fields: Add the following fields, pressing Enter after each one:

  • warehouse-cd: Distribution center code.

  • host: RabbitMQ server IP for the DC.

  • port: RabbitMQ server port.

  • virtual-host: RabbitMQ virtual host.

  • queue: RabbitMQ queue with the DC’s messages.

Click Save to create the model.

2

Register instances and fill in the data

1. With the model created, click Configure Multi-Instance (second icon in the Actions column) and then click Create.

2. Register the instances corresponding to each DC’s queue (dc1, dc2, and dc3):

• Fill in the Name field.

• Fill in the other parameters: port, host, virtual-host, warehouse-cd, and queue, according to each environment’s configuration.

1. Click Save.

2. Repeat the process for each DC.

For each DC, there is an instance configured according to its RabbitMQ queue, both for testing and production.

3

Link the Multi-Instance model to the pipeline

With the Multi-Instance model created and configured, the next step is to link it to the pipeline that will use this data.

A single Multi-Instance model can be linked to as many pipelines as needed.

To link it:

1. Open the desired pipeline.

2. Open the Settings menu.

3. Select It’s Multi-Instance.

4. Select the warehouse-stock-queue model.

After defining a pipeline as Multi-Instance, it cannot be reverted. However, any pipeline, deployed or not, can be converted into a Multi-Instance pipeline.

4

Reference Multi-Instance fields

As described in the documentation on creating Multi-Instance pipelines, to reference the value of a field configured in an instance, use the following notation:

{{replica.instance_variable_name}}

In the example from this document, these fields are used in the RabbitMQ trigger as follows:

• Hostname: {{replica.host}}

• Port: {{replica.port}}

• The RabbitMQ Queue To Consume Messages From: {{replica.queue}}

This way, when deploying the pipeline for an instance (dc1, dc2, or dc3), the trigger automatically uses the configured values.

Multi-Instance fields can also be used in other connectors.

Example with JSON Generator:

{
  "warehouse": "{{replica.warehouse-cd}}",
  "host": "{{replica.host}}",
  "port": {{replica.port}},
  "queue": "{{replica.queue}}"
}

5

Test the Multi-Instance pipeline

To test the pipeline, go to the Execution Panel. Before starting the test, select the desired instance in the Multi-Instance field and click Execute to see the result for that instance.

6

Deploy the pipeline instances

The deployment of a Multi-Instance pipeline is done per instance. Although the pipeline is unique, each instance is deployed independently (dc1, dc2, dc3).

During deployment, select the desired instance in the Instance field of the form.

This allows you to scale based on the demand of each DC.

For example, if DC3 has a higher transaction volume, add more pipeline replicas for that instance only (horizontal scaling).

Each deployed instance receives the instance name as a suffix on the pipeline (dc1, dc2, dc3). Although they share the same logic, instances behave as independent execution units.

Final considerations

The Multi-Instance feature centralizes pipeline maintenance and provides flexibility through advanced parameterization.

In the scenario presented, each Distribution Center runs the same pipeline with its own queue and processing configuration, enabling scalability, operational simplicity, and reduced maintenance effort.

With the new Consumption-Based Model, the number of deployed instances doesn’t directly affect costs, keeping the model aligned with real transaction demand.

Key benefits of Multi-Instance:

• Instance-level scalability: Each instance can scale independently, horizontally or vertically.

• Centralized maintenance: A single pipeline serves all instances, simplifying management.

• Flexible parameterization: Multi-Instance fields allow configuring specific variations.

• Easy expansion: To include a new DC, simply create a new instance in the model and deploy it.

• Gradual rollout: Ideal for partial rollouts and progressive adoption.