How to run a load test in Digibee

Overview

This document provides guidance for running load tests on pipelines in the Digibee Integration Platform, assessing performance, response time, and stability to ensure a safe go-live in production.

Here you will find the recommended strategy, the ideal settings for the planned load, and best practices.

Support materials are also available:

• Test Plan spreadsheet template, which helps organize and document the load test.

• Checklist with the main steps of the Test Plan, for quick reference whenever needed.

Context

The guidance in this document is based on a real case: testing a synchronous REST API pipeline that needed to support 4,000 simultaneous users (threads) with a maximum response time of 4,000 ms. This load occurred only on the fifth business day of the month; on other days, the peak was 10 threads.

Important considerations

Disabling autoscaling

During the test, autoscaling must be disabled, since cold start affects metrics but doesn’t represent real use, as it only occurs on the first request. After the test, configure the minimum number of replicas to ensure good performance for initial requests and for below-average loads.

Warm up on new replicas

Each incremental test aims to identify limits and adjust either horizontal scaling (replicas) or vertical scaling (pipeline size). When increasing replicas, disregard the first execution, since it triggers the warm up process on new replicas, which distorts performance analysis.

Load test costs

Costs depend on the licensing model used (Licensing models):

• Pipeline-Based (Licenses): Production and Test consume the same package.

• Subscription-Based (RTUs): Production and Test consume different packages.

• Consumption-Based (Credits): Based on actual usage; load tests can consume credits quickly due to the high volume of requests.

Digibee team involvement

The load test must include Digibee team support starting in Phase 2, ensuring assistance during execution and making necessary adjustments. More information can be found in the section Phase 2: Preparing the test environment.

Phase 1: Test planning

A test plan is essential to assess the environment, execute scenarios, analyze results, and successfully complete the load test.

Load parameters

The first step of the plan is to record the general conditions for the test, covering the following topics:

• What will be tested

• Relevant parameters for test execution (simultaneous requests, message size, example of the test message, expected response time)

• Notes (tool used for running the tests, information about peak transactions and average load outside the peak)

See an example:

Test rehearsal

The Test Plan should include scenarios with gradual load growth, starting with minimal resources and increasing with each run until reaching the final goal. This process is called a “rehearsal” because its results help define the parameters for the final test.

Below is an example of the planning in a Test Plan spreadsheet page:

Phase 2: Preparing the test environment

With the plan defined, it’s time to prepare the environment together with Digibee. At this stage, it’s essential to align responsibilities among the customer team, Digibee internal team, and Product team, ensuring that all necessary adjustments are made before starting the tests.

Customer team

Responsible for enabling test execution and providing the necessary external resources.

• Define the tool to be used for test execution (for example, JMeter).

• Assign people responsible for running and analyzing results together with the Digibee team.

• Provide resources for simulating external access steps, even with simulated responses, such as:

  • Request endpoints

  • Databases

  • File server

  • Messaging brokers

  • Others

• Participate in the pipeline pre-analysis to identify components requiring scale.

Digibee internal team

Acts as an intermediary between the customer and the Product team, coordinating actions and ensuring the environment is ready for the load test.

For this, one of the following teams must be contacted:

1. Professional Services: Specialized technical support, if previously contracted.

2. Customer Success: Oversees the process, ensuring alignment with the customer.

Main activities:

• Verify completion of infrastructure requirements.

• Confirm availability of external access resources or validate alternatives (for example, Mock).

• Conduct pre-analysis of pipelines and functional tests.

• Assess environment capacity.

Digibee Product team

Responsible for evaluating and calibrating the test environment, ensuring resources are adequate before execution.

Main activities:

• Analyze load parameters and the test plan, identifying necessary adjustments.

• Review and calibrate Platform components such as AWS EC2, GCE, Gateway (Kong), Trigger, RabbitMQ, Object Store, and Digibee Store.

• Define test monitoring: team presence, execution window, reports and metrics, access, and other relevant details.

Initial configurations can be recorded in the “general” page of the Test Plan spreadsheet.

Phase 3: Execution

With the previous phases completed, the planned tests should be executed sequentially.

Updating general environment data

During testing, configuration adjustments may be required.

Whenever this happens, it’s a good practice to update the general conditions page so that the final analysis reflects the actual scenario, as illustrated below:

Points to note

During testing, it is common to observe relevant behaviors that should be recorded for evaluation or future consideration.

For this, we recommend creating a dedicated page in the spreadsheet:

Final test

After rehearsals and necessary adjustments, the team and customer select the most relevant tests to repeat, usually those with the best initial results in terms of success rate and response time.

In the example, the five best results from the initial scenario are highlighted, considering the success rate and response time.

Phase 4: Final result analysis and action plan

Final analysis of tests

Based on the final test, analyses are conducted to assess results, considering metrics such as RPS (requests per second), response time, and success rate, correlated with the pipeline deployment size (size × replicas × threads).

Test conclusion

This phase concludes with a report that includes need, context, results, test summary, success rate observations, platform configuration, adjustments made (rehearsal and final tests), points of attention, and general notes.

The report can be added in Markdown to the “conclusion” page of the Test Plan spreadsheet.

Final considerations

The strategy outlined in this document ensures the execution of a structured, controlled, and iterative load test, capable of indicating not only pipeline scalability needs but also possible platform capacity adjustments.

This process is essential to validate scenarios with high transaction volumes, where response time and stability are critical factors for ensuring a safe production go-live without risks.

Support materials

Test Plan spreadsheet template

To accompany the tests, we recommend using a structured spreadsheet, based on the following model.

Checklist for the load test

Use the checklist below as a guide to create a test plan according to the instructions in this document.