JOLT and Double Braces on Digibee: Choosing the right method for data transformation
PreviousBest practices for error handling in pipelinesNextNotification model in event-driven integrations
Last updated
Was this helpful?
Last updated
Was this helpful?
Was this helpful?
Digibee empowers users to connect diverse systems and orchestrate complex data flows. An important part of this process is data transformation, as it allows you to reshape and adapt data to meet the structure and requirements of target systems.
Within Digibee, two primary methods are available for transforming data:
JOLT: JSON to JSON Transformation
Double Braces: Digibee’s own expression language
Understanding when to use each of these approaches helps you design clear and maintainable transformations. This article explores the strengths and use cases of both JOLT and Double Braces, guiding you to choose the right one for your specific transformation needs.
JOLT is a JSON-based transformation language designed for reshaping entire JSON documents. It works through a series of specifications that define how the input JSON should be transformed into the desired output structure. JOLT can handle complex structural changes by combining multiple specs within a single transformation, allowing you to move, rename, and reorganize data. Digibee provides the Transformer (JOLT) connector for using JOLT transformations.
JOLT provides a range of operations to reshape JSON data, including:
shift: Move and rename fields.
default: Insert default values for missing fields.
remove: Delete fields.
sort: Sort fields alphabetically .
cardinality: Switch between single items and arrays.
modify-default-beta / modify-overwrite-beta: Add or update fields dynamically with functions.
JOLT also supports wildcards like *, @, $, and others. This makes it flexible to evolve JSON structures without knowing all the field names in advance.
To explore each operation in detail, check out the Getting to Know JOLT article.
Standardizing data from multiple sources: Imagine receiving customer data from different systems, each with its own JSON structure. JOLT's shift operation can map fields like customerName and nameClient to customer.fullName in your data model.
// Input from System A
{
"customerName": "Alice Smith",
"email": "[email protected]"
}
// Input from System B
{
"nameClient": "Bob Johnson",
"correo": "[email protected]"
}
// JOLT Transformation
[
{
"operation": "shift",
"spec": {
"customerName|nameClient": "customer.fullName",
"email|correo": "customer.email"
}
}
]
// Output (for System A)
{
"customer": {
"fullName": "Alice Smith",
"email": "[email protected]"
}
}
// Output (for System B)
{
"customer": {
"fullName": "Bob Johnson",
"email": "[email protected]"
}
}Flattening nested JSON structures: When dealing with APIs that return nested data, JOLT's dot notation in the shift operation can flatten these structures for easier processing.
// Input
{
"order": {
"details": {
"orderId": "12345",
"totalAmount": 100.5
},
"customer": {
"name": "Charlie Brown"
}
}
}
// JOLT Transformation
[
{
"operation": "shift",
"spec": {
"order": {
"details": {
"orderId": "orderId",
"totalAmount": "totalAmount"
},
"customer": {
"name": "customerName"
}
}
}
}
]
// Output
{
"orderId" : "12345",
"totalAmount" : 100.5,
"customerName" : "Charlie Brown"
}Processing arrays of objects: JOLT can transform arrays of objects. Instead of explicit iteration, its specifications apply to each object in the array. This is useful for standardizing data within a list, extracting specific information from each item, or restructuring collections of data.
Imagine receiving a list of product objects from different suppliers, each using different field names. JOLT can standardize this list into a consistent format.
// Input from Supplier A
{
"products": [
{"productName": "Laptop", "priceUSD": 1200},
{"productName": "Mouse", "cost": 25}
]
}
// Input from Supplier B
{
"items": [
{"name": "Keyboard", "value": 75},
{"name": "Monitor", "price_eur": 300}
]
}
// JOLT Transformation
[
{
"operation": "shift",
"spec": {
"products": {
"*": {
"productName|name": "productsOut[].name",
"priceUSD|cost|value|price_eur": "productsOut[].price"
}
},
"items": {
"*": {
"productName|name": "productsOut[].name",
"priceUSD|cost|value|price_eur": "productsOut[].price"
}
}
}
}
]
// Output (for Supplier A)
{
"productsOut": [
{"name": "Laptop", "price": 1200},
{"name": "Mouse", "price": 25}
]
}
// Output (for Supplier B)
{
"productsOut": [
{"name": "Keyboard", "price": 75},
{"name": "Monitor", "price": 300}
]
}Extracting data from a list of users: JOLT can extract just the id and email from an array of user objects.
// Input
{
"users": [
{"userId": "user123", "emailAddress": "[email protected]", "details": {"age":40 }},
{"userId": "user456", "emailAddress": "[email protected]", "info": {"country":"BR" }}
]
}
// JOLT Transformation
[
{
"operation": "shift",
"spec": {
"users": {
"*": {
"userId": "usersOut[&1].id",
"emailAddress": "usersOut[&1].email"
}
}
}
}
]
// Output
{
"usersOut": [
{"id": "user123", "email": "[email protected]"},
{"id": "user456", "email": "[email protected]"}
]
}To explore more real-world examples and advanced use cases, check out the Transformations with JOLT article.
Double Braces is Digibee's expression language. It’s designed for dynamic data access and manipulation inside the configuration of Digibee connectors and capsules. Unlike JOLT, which transforms entire JSON structures using separate specifications, Double Braces allows you to reference and manipulate data inline within parameters of almost all Digibee connectors, such as JSON Generator, JSON Transformer, and many others.
A key characteristic of Double Braces is its function-based approach and its ability to access various aspects of the execution context, including the incoming message payload, pipeline metadata, accounts, and global variables.
Dynamically constructing JSON with JSON Generator: The JSON Generator (Mock) connector is a good example of how Double Braces can be used to create JSON objects dynamically, allowing you to:
Insert values from the incoming message by embedding fields directly from the input JSON using {{ message.fieldName }}.
Transform data using many built-in Double Braces functions, like string manipulation, conditional logic, math, date formatting, and more.
Add static values by using functions like CONCAT to combine fixed strings with dynamic values, such as {{ CONCAT("User ", message.userId) }}.
Access global variables and accounts by retrieving predefined values using {{global.global-name}} or {{ account.username }}.
Combining fields and calculating value: Double Braces functions can be used to both concatenate field values and perform dynamic calculations in the same transformation step.
// Input
{
"firstName": "Carlos",
"lastName": "Silva",
"orderTotal": "95.756"
}
// JSON Generator configuration with Double Braces
{
"fullName": {{ CONCAT(message.firstName, " ", message.lastName) }},
"taxAmount": {{ TONUMBER(message.orderTotal, "###.##") }},
"country": "Brazil"
}
// Output
{
"fullName": "Carlos Silva",
"taxAmount": 95.756,
"country": "Brazil"
}Conditional data processing: With Double Braces, you can decide how to proceed with the pipeline execution or how to transform the data.
// Incoming message (example)
{
"countryCode": "US"
}
// JSON Generator configuration with Double Braces
{
"shippingRegion": {{ SWITCHCASE("Other", EQUALTO(message.countryCode, "US"), "North America", EQUALTO(message.countryCode, "CA"), "North America",EQUALTO(message.countryCode, "EU"), "Europe") }}
}
// Output (with added property)
{
"shippingRegion": "North America"
}Pro tip: The JSON Transformer is an alternative for manipulating data without changing the original JSON structure — unless you want to remove something. It lets you centralize transformations in a single connector, keeping flows cleaner and easier to maintain.
Primary goal
Restructure JSON documents
Manipulate individual field values and access context
Scope
Entire JSON structure
Specific field values or pipeline context
Complexity
Handle complex structural changes
Handle granular logic
Context
Focus on the JSON payload
Access broader Digibee pipeline context
Digibee usage
Dedicated transformation step
Embedded within connector and capsules parameters
Your main task is to reshape the JSON structure (flattening, nesting, mapping elements in arrays, and so on).
You need to rename and move a significant number of fields.
You are dealing with different or unknown JSON structures and need dynamic transformations using wildcards.
You need to transform arrays of objects by applying consistent rules to each element.
You need to perform data type conversions (date formatting).
You need to implement conditional logic to assign values based on data.
You need to perform string manipulations (concatenation, substring).
You need to access specific elements within JSON arrays or objects.
You need to integrate dynamic values based on the pipeline metadata.
You need to use values stored in session variables.
While this article focuses on JOLT and Double Braces as the primary transformation methods available within Digibee, feel free to explore the JSLT connector, an alternative transformation language with JavaScript-like syntax. It lets you modify, extract, remove, sort, and access data from JSON content.
Combining both JOLT and Double Braces is a key step toward creating flexible and maintainable data transformations within the Digibee Integration Platform. By understanding the strengths of each approach and knowing when to use them individually or together, you can simplify your transformation logic, reduce complexity, and keep your pipelines adaptable as requirements change.