Best practices for building a pipeline

Follow these best practices to ensure your pipelines are secure, efficient, and well-structured.

Before deploying a pipeline, review these key points to safeguard your integration flows:

Use API Keys

If your pipeline includes HTTP, REST, or HTTP File triggers, the services created will be exposed to the internet. To improve security, the Digibee Integration Platform requires these triggers to be used with an API key by default. Learn how to create an API key.

Use a separate API key for each system consuming an API and restrict access to only the necessary pipelines.

For additional security, we recommend implementing JWT (JSON Web Token) authentication.

Exception: If you need to publish a pipeline with these triggers without an API key (not recommended), submit a request via chat using the following template:

I request the inclusion of the following pipelines in the whitelist to not require an API key.

• Realm name

• Names of the pipelines to be included in the realm’s whitelist

• Reason for the request

Store usernames and passwords securely

Never expose login credentials directly in connector settings. Instead, store them on the Accounts page and reference them in the configuration form of the connector.

Obfuscate sensitive fields

Configure sensitive fields in the pipeline settings or the Sensitive Fields Policy to automatically hide them in logs and messages. This prevents data exposure during integration, reducing security risks. Obfuscated fields are replaced with "***".

Learn how to obfuscate sensitive fields at the pipeline and realm levels.

Use HTTPS instead of HTTP

Whenever possible, use HTTPS for external service requests to improve security.

Encrypt and clear Object Store data

• Encrypt sensitive data stored in an Object Store using cryptography connectors.
• Clear the Object Store periodically. Object Stores are auxiliary databases that support you in the development of integration flows. They are not intended to store large amounts of data. If you don’t clear your Object Store regularly, this can lead to errors in your integration flows.

Use the Script connector only when necessary

The Script (JavaScript) connector runs on a separate infrastructure outside the pipeline. This configuration can slow down processing and, since the infrastructure is shared, intensive use can cause delays in multiple pipelines.

To avoid this, use dedicated connectors like Transformer (JOLT), JSLT, JSON Transformer, and JSON Generator (Mock) for data transformation and mapping whenever possible. These connectors are built into the pipeline and work more efficiently. Only use Script (JavaScript) when JavaScript is the only option.

Not sure if you need the Script (JavaScript) connector? Contact our team. We may have a better solution using other connectors.

Validate responses from external services

When integrating with external services, such as an API or database, always verify that the response type matches expectations. If it doesn’t, take appropriate action, such as reprocessing or handling the error.

For more details, refer to our documentation on event-driven architecture.