Object Store

Discover more about the Object Store connector and how to use it on the Digibee Integration Platform.

Object Store performs operations to store any document in Digibee's Object Store. It's a simple, quick way to save useful JSON-type information, offering operations that support multiple use cases during pipeline creation.

Object Store isn't enabled by default in new realms. Contact Digibee support to request it.

Parameters

The table below lists all configuration parameters for the connector. Parameters that support Double Braces expressions are marked with ✅ in the Supports DB column.

Parameter

Description

Type

Supports DB

Default value

Visible when

Account

Account to be used by the connector. This item can't be changed.

String

❌

Digibee Object Store Account

—

Operation

Operation to be executed inside Object Store: Find by Object ID, Find By Query, Insert, Aggregate, Update By Object ID, Update By Query, Delete By Object ID, Delete By Query, Create Index, List Indexes, and Drop Index.

String

❌

Find by Object ID

—

Object Store Name

Name of the collection to be used to record or read information. If it doesn't exist, the connector creates it automatically.

String

❌

sales

—

Expire After Seconds

Defines the number of seconds after which a document from the Object Store expires.

Integer

✅

0

Operation is Create Index

Object ID

Identifier of the object to be stored or searched. It can be a unique number or a UUID.

String

✅

1

Operation is Find by Object ID, Insert, Update by Object ID, or Delete by Object ID

Query

JSON object that defines the query.

String

✅

N/A

Operation is Find by Query, Aggregate, Update by Query, Delete by Query, Create Index, or Drop Index

Document

JSON field with the object's data or update instructions for the operation.

String

✅

N/A

Operation is Insert, Update by Object ID, or Update by Query

Limit

Maximum number of objects to be returned in a search.

Integer

✅

0

Operation is Find by Object ID or Find by Query

Skip

Number of objects to skip before the query returns results. Use this parameter together with Limit to enable pagination.

Integer

✅

0

Operation is Find by Object ID or Find by Query

Sort

Defines the sort order for the query.

String

✅

N/A

Operation is Find by Object ID or Find by Query

queryTimeoutMs

Maximum time, in milliseconds, that a query can run on the server before it's aborted. Applies only to the Find by Object ID, Find By Query, and Aggregate operations; write operations aren't affected.

Integer

✅

295000

Operation is Find by Object ID, Find by Query, or Aggregate

totalTimeoutMs

Total time limit in milliseconds for the whole read (initial query plus every batch fetched from the cursor, including retries). Empty or 0 means no limit.

Integer

✅

N/A

Operation is Find by Object ID, Find by Query, or Aggregate

Unique Index

When enabled, the connector creates an index that accepts unique Object ID values only. When disabled, it creates a non-unique index.

Boolean

❌

true

—

Isolated

When enabled, the connector scopes all queries to the current execution.

Boolean

❌

false

—

Upsert

When enabled, the connector inserts the provided object into the collection if it doesn't already exist.

Boolean

❌

false

Operation is Update By Object ID or Update By Query

Fail On Error

When enabled, the connector stops pipeline execution if an error occurs. When disabled, execution continues, but the result sets the "success" property to false.

Boolean

❌

false

—

Parameter

Description

Type

Supports DB

Default value

Documentation

Optional field to describe the connector configuration and any relevant business rules.

String

❌

N/A

How the timeout parameters work

queryTimeoutMs limits the first command sent to the server, and totalTimeoutMs limits the entire read, including every batch fetched afterward.

Invalid values in either parameter never cause a pipeline error. The connector ignores them and logs a warning, so you can safely resolve both parameters from Double Braces expressions at runtime.

queryTimeoutMs

queryTimeoutMs accepts values greater than 0 and up to 295000 milliseconds (295 seconds). You can use it to reduce this limit, but not to raise it. For values outside this range or non-numeric values, the connector uses the default value.

When a query reaches the timeout, the connector returns an error immediately and does not retry it as a transient failure. As a result, slow queries fail fast and explicitly and don't block the step for several minutes.

If totalTimeoutMs is also set, the time left in the total limit caps queryTimeoutMs.

totalTimeoutMs

queryTimeoutMs covers only the first command. Large results are then fetched in batches, and without totalTimeoutMs, this fetching has no time limit.

If you leave the field empty or set it to 0, reads have no time limit. Negative and non-numeric values are ignored. The limit is counted per message, and any retries of that message draw from the same time budget.

When a read hits the limit, the connector closes the cursor and returns a timeout error with the number of documents read up to that point. This error isn't retried. If Fail On Error is disabled, the step returns "success": false.

If a batch request is already in progress when the limit is reached, only the global socket timeout (300 seconds) can interrupt it.

Because of this, totalTimeoutMs works best as a safeguard, not as the main way to control long reads. For large collections, use paging with Limit and Skip, plus an index on the Sort field.

Best practices

Object Store is an auxiliary NoSQL database for integrations. It makes integration development faster and more practical. The following best practices show how to use the connector effectively:

The Object Store connector acts as an intermediary database, that is, it mediates information between the flows of an integration. Use it only to store information relevant to that integration.
Object Store is a temporary database. After information has served its purpose in the integration flow, remove old and unnecessary data regularly. You can remove it manually or create an index with a TTL mechanism to expire it automatically.
Because it's an auxiliary database, use the Object Store connector only in specific cases to support integration development, not as a permanent database.
The Digibee Integration Platform stores all data securely. However, we recommend encrypting sensitive data stored in Object Store using the encryption connectors available on the canvas.

Messages flow

Input

For this connector, the only mandatory input is a JSON object. The input parameter supports Double Braces syntax, so you can send the received message directly to the connector.

Output

Insert

{
    "data": [],
    "updateCount": 1
}

Find

{
   "data": [
       {
           "name": "Galaxy s20",
           "uuid": "123",
           "_oId": "1"
       }
   ],
   "rowCount": 1
}

Update

{
    "data": [],
    "updateCount": 1
}

Delete

{
    "data": [],
    "updateCount": 1
}

Aggregate

{
    "data": [],
    "rowCount": 0
}

Object Store in action

The examples above showed the output of each operation. The following examples demonstrate how to configure the connector correctly to get a specific result:

Insert multiple items at once in a collection

When you send an array of objects in the Document field, the connector inserts each item separately into the selected collection.

Configure the connector with the following parameters:

Operation: Insert
Unique Index: False
Document:

[
   {
       "id": 1,
       "name": "Galaxy s20",
       "price": 5000
   },
   {
       "id": 2,
       "name": "Samsung 4k 55\"",
       "price": 5000
   },
   {
       "id": 3,
       "name": "Galaxy A10",
       "price": 699
   },
   {
       "id": 4,
       "name": "Galaxy A51",
       "price": 1620
   }
]

Output

{
    "data": [],
    "updateCount": 4
}

You can only insert multiple objects at once in collections created with Unique Index set to False. You set the Unique Index property when you create the collection, and you can't change it afterward.

Find items from a determined query

For example, consider an Object Store that already has product-type items registered, each with a name and a price.

Configure the connector with the following parameters:

Operation: Find By Query
Query:

{
    "product.price": { $gt: 2000 }
}

Output

{
   "data": [
       {
           "product": {
               "id": 1,
               "name": "Galaxy s20",
               "price": 5000
           },
           "_oId": "1"
       },
       {
           "product": {
               "id": 2,
               "name": "Samsung 4k 55\"",
               "price": 5000
           },
           "_oId": "2"
       }
   ],
   "rowCount": 2
}

Find all the items from a query

For example, consider an Object Store that already has product-type items registered, each with a name and a price.

Configure the connector with the following parameters:

Operation: Find By Query
Limit: 10
Query: {}

Output

{
   "data": [
       {
           "product": {
               "id": 1,
               "name": "Galaxy s20",
               "price": 5000
           },
           "_oId": "1"
       },
       {
           "product": {
               "id": 2,
               "name": "Samsung 4k 55\"",
               "price": 5000
           },
           "_oId": "2"
       },
       {
           "product": {
               "id": 3,
               "name": "Galaxy A10",
               "price": 699
           },
           "_oId": "3"
       },
       {
           "product": {
               "id": 4,
               "name": "Galaxy A51",
               "price": 1620
           },
           "_oId": "4"
       }
   ],
   "rowCount": 4
}

In this example, the Limit parameter prevents an unnecessary overload when returning objects from Object Store. Without it, the pipeline can throw an "Out Of Memory" error. Setting Limit lets you control how many objects the response returns.

Update an item from a specific ID

For example, consider an Object Store that already has product-type items registered, each with a name and a price.

Configure the connector with the following parameters:

Operation: Update By Object ID
Object ID: 3
Document:

{
   $set: {
       "product": {
         "id": 3,
         "name": "Galaxy A10",
         "price": 605
       }
   }
}

Output

{
    "data": [],
    "updateCount": 1
}

In this example, the output only identifies that an update happened. To confirm the object updated correctly, repeat the ID search scenario.

When the Object Store connector updates records inside an iteration connector, such as For Each or Stream File Reader, and runs in parallel, identical update instructions can create a race condition. In that case, one instruction returns "updateCount": 1 and the other returns "updateCount": 0.

This happens when two identical records enter the Object Store operation pool at the same time, and their update or insert instructions (with Upsert enabled) run sequentially. The first instruction updates the record. The second instruction finds the record already updated, determines that no change is needed, and returns "updateCount": 0.

Remove an item from a specific ID

For example, consider an Object Store that already has product-type items registered, each with a name and a price.

Configure the connector with the following parameters:

Operation: Delete By Object Id
Object ID: 4

Output

{
    "data": [],
    "updateCount": 1
}

In this example, the output only identifies that a deletion happened. To confirm the object was deleted correctly, repeat the ID search scenario.

Aggregation to copy the collection

For example, consider an Object Store called product that already has product-type items registered, each with a name and a price. From that collection, create a new Object Store called product-backup and copy all the items into it.

Provide an object array with the query aggregation pipeline in the Document parameter.

Configure the connector with the following parameters:

Operation: Aggregate
Query:

[
   {
       $merge: {
           into: "product-backup",
           on: "_id",
           whenMatched: "replace",
           whenNotMatched: "insert"
       }
   }
]

In this example, the query replaces existing items with the new ones in the collection.

Output

{
    "data": [],
    "rowCount": 0
}

To confirm the collection was created correctly with the expected items, repeat the search scenario for all items, using the new collection.

Aggregation to filter collection items

Provide an object array with the query aggregation pipeline in the Document parameter.

Configure the connector with the following parameters:

Operation: Aggregate
Query:

[
   {
       $match: {
           "product.price": {
               $gt: 3000
           }
       }
   },
   {
       $group: {
           _id: null,
           count: {
               $sum: 1
           }
       }
   }
]

In this example, the query searches for products with a price above a specific value and returns their sum.

Output

{
    "data": [{
        "count": 2
    }],
    "rowCount": 0
}

Create an index with expiration time

Object Store lets you create an index with a TTL (Time to live) attribute, so you can define an expiration strategy for your documents. The Expire after seconds parameter controls this behavior. The following examples show how to create indexes with expiration strategies:

Create an index with constant TTL

Configure the connector with the following parameters:

Operation: Create Index
Expire After Seconds: 300
Query: {"expiresAt": 1}

The field name must match the date attribute in your document that controls the object's lifetime. The platform automatically deletes all documents with this field after the number of seconds specified in the Expire after seconds parameter.

Output

{
    "data": "expiresAt_1",
    "rowCount": 0
}

Create an index with custom dates

Configure the connector with the following parameters:

Operation: Create Index
Expire After Seconds: 0
Query: {"expiresAt": 1}

The field name must match the date attribute in your document that controls the object's lifetime. Also set the Expire after seconds parameter to 0.

In this scenario, the expiration date comes from the datetime value of the corresponding field in the Query parameter.

When you insert new documents, make sure the field you configured in the index has a date value.

{
    "id": 1,
    "name": "Galaxy s20",
    "price": 5000,
    "expiresAt": new Date()
}

The TTL index doesn't guarantee that expired data is deleted immediately. There can be a delay between when a document expires and when Object Store removes it from the database, because the background task that removes expired documents runs every 60 seconds.

List all indexes

This operation lists all indexes you created in an Object Store.

Output

{
  "data": [
    {
      "v": 2,
      "key": {
        "expiresAt": 1
      },
      "name": "expiresAt_1",
      "expireAfterSeconds": 0
    }
  ],
  "rowCount": 1
}

Drop an existing index

This operation lets you drop an index you created previously.

Configure the connector with the following parameters:

Operation: Drop Index
Query: {"expiresAt": 1}

Output

{
    "data": "expiresAt_1",
    "rowCount": 0
}

Technology

Object Store uses search operators and object aggregation similar to MongoDB syntax. See the MongoDB documentation to learn more.

Object Store data is isolated between environments. However, the Test environment shares data with the Execution panel.

PreviousMongoDB

NextHow to use DISTINCT and COUNT in the Object Store

Last updated 14 days ago

Was this helpful?