DynamoDB

Discover more about the DynamoDB connector and how to use it on the Digibee Integration Platform.

The DynamoDB connector allows pipelines to execute operations against DynamoDB tables in AWS. The following operations are currently available:

• PutItem: creates or replaces an item in a DynamoDB table.

• GetItem: fetches attributes from an existing item in a DynamoDB table by primary key.

• UpdateItem: edits an existing item's attributes or adds a new item to a DynamoDB table.

• DeleteItem: deletes a single item in a table by primary key

• Query: fetches all items with the same partition key from a table or from an index (Local secondary index, or global secondary index).

• Scan: fetches all items from a table or from an index (local secondary index, or global secondary index). A filter expression may be applied to limit the number of returned items.

Parameters

The available parameters are divided into four tabs and may vary according to the chosen operation. Parameters supported by Double Braces expressions are marked with (DB).

General Tab

Parameter	Description	Default value	Data type
Fail On Error	If the option is activated, the pipeline's execution with an error will be interrupted. Otherwise, the pipeline execution proceeds, but the result will show a false value for the "success" property.	False	Boolean

Authentication tab

Parameter	Description	Default value	Data type
Account Type	Defines the account type of the main account used for the connection. Supported types: AWS V4 and Basic.	Basic	String
Use Dynamic Account	When active, the connector will use the account dynamically. Otherwise, it will use the account statically.	False	Boolean
Scoped	When active, the stored account is isolated from other subprocesses. In this case, the subprocesses see their own version of the stored account data. It is only available if the Use Dynamic Account parameter is active.	False	Boolean
Account	The account used to connect the connector to the target DynamoDB table. It is only available if the Use Dynamic Account parameter is inactive.	Null	BASIC, AWS-V4
Account Name (DB)	The name of the dynamic account used to connect the connector to the DynamoDB target table. This account must have been previously configured in a Store Account connector in the pipeline for this process to take effect. It is only available if the Use Dynamic Account parameter is active.	Empty	String
AWS Region	The AWS region where the target table is available. It is used only when the account type is set to Basic.	us-east-1	String
Use Assume Role	When active, it allows the use of an AWS Role account to be assumed by the Account. It can only be used if the account type is set to AWS V4.	False	Boolean
AWS Role Account	The account used to delegate temporary permissions to the Account through the Assume Role operation.	NULL	AWS-ROLE
AWS Role Account Name (DB)	The name of the dynamic account used to delegate temporary permissions to the account referenced in the Account Name field through the Assume Role operation. It is only available if the Use Dynamic Account parameter is active.	Empty	String

Operation Settings tab

Parameter	Description	Default value	Data type
Operation	The operation to be executed.	PutItem	String
Table Name	Name of the table against which the operation is being executed.	NULL	String

PutItem operation parameters

Parameter	Description	Default value	Data type
Upsert	When active, it completely replaces an existing item with the same primary key. Otherwise, the operation will fail when an item with the specified primary key already exists.	True	Boolean
Body (DB)	JSON object to be used by the operation. JSON arrays, and other valid JSON definitions are not allowed.	{{ message.$ }}	JSON

GetItem operation parameters

Parameter	Description	Default value	Data type
Partition Key Value (DB)	Value of the target item's partition key. This parameter is mandatory.	NULL	String
Sort Key Value (DB)	Value of the target item's sort key. Only needed when the target table uses a composite primary key (partition key + sort key).	NULL	String
Attributes to Return (DB)	Comma-separated list of attributes names to be returned by the operation.	NULL	String
Consistent Read	This parameter overrides DynamoDB's default eventually consistent behavior when active.	False	Boolean

UpdateItem operation parameters

Parameter	Description	Default value	Data type
Partition Key Value (DB)	Value of the target item's partition key. This parameter is mandatory.	NULL	String
Sort Key Value (DB)	Value of the target item's sort key. Only needed when the target table uses a composite primary key (partition key + sort key).	NULL	String
Return Values	List of options on how to get the attributes' values, before or after the update operation is performed. Options are: ALL NEW (Return all values as they are, after the update), ALL OLD (All values as they were before the update), NONE (Nothing is returned), UPDATED NEW (Only updated values are returned as they are after the update), and UPDATED OLD (Only updated values as they were before the update).	NONE	String
Body (DB)	JSON object to be used by the operation. JSON arrays, and other valid JSON definitions are not allowed.	{{ message.$ }}	JSON

DeleteItem operation parameters

Parameter	Description	Default value	Data type
Partition Key Value (DB)	Value of the target item's partition key. This parameter is mandatory.	NULL	String
Sort Key Value (DB)	Value of the target item's sort key. Only needed when the target table uses a composite primary key (partition key + sort key).	NULL	String

Query operation parameters

Parameter	Description	Default value	Data type
Index Name	Name of the index to be used in this operation. The index must belong to the table specified in the Table Name field.	NULL	String
Partition Key Value (DB)	Value of the target item's partition key. This will refer to an Index Partition Key, if the field Index Name is specified. This parameter is mandatory.	NULL	String
Sort Key Expression (DB)	A condition expression to be applied on the Sort Key attribute. This will refer to an Index Sort Key, if the field Index Name is specified.	NULL	String
Attributes to Return (DB)	List of attribute names to be returned by the operation.	NULL	String
Consistent Read	When active, this parameter overrides DynamoDB's default eventually consistent behavior. It cannot be used when the query operation is used against a Global Secondary Index (GSI).	False	Boolean
Filter Expression (DB)	A string that contains conditions that DynamoDB applies after the Query operation, but before the data is returned to you.	NULL	String
Exclusive Start Partition Key Value (DB)	The partition key of the first item that this operation will evaluate. This attribute is mandatory only when fetching paginated results.	NULL	String
Exclusive Start Sort Key Value (DB)	The sort key of the first item that this operation will evaluate. This attribute is mandatory only when fetching paginated results.	NULL	String
Exclusive Start Index Partition Key Value (DB)	The index partition key of the first item that this operation will evaluate. This attribute is mandatory only when fetching paginated results from an Index.	NULL	String
Exclusive Start Index Sort Key Value (DB)	The index sort key of the first item that this operation will evaluate. This attribute is mandatory only when fetching paginated results from an Index.	NULL	String
Limit	Maximum number of items to be returned by the operation. Minimum allowed is 1 and maximum allowed is 1000.	1000	Integer
Automatic Pagination	If active, this parameter will automatically fetch all pages when DynamoDB returns paginated responses.	False	Boolean
Scan Forward	If active, this parameter returns the result in its original ordering (based on the sort key). Otherwise, the order is inverted.	True	Boolean
Expression Attribute Values (DB)	A list of attribute values used in the conditional expression fields (Sort Key Expression and Filter Expression). Each attribute value must specify the Placeholder used, the Type of the data, and the Value to replace the placeholder.	Empty	List of Attribute Values

Scan operation parameters

Parameter	Description	Default value	Data type
Index Name	Name of the index to be used in this operation. The index must belong to the table specified in the Table Name field.	NULL	String
Attributes to Return (DB)	List of attribute names to be returned by the operation.	NULL	String
Consistent Read	When active, this parameter overrides DynamoDB's default eventually consistent behavior. It cannot be used when the scan operation is used against a Global Secondary Index (GSI).	False	Boolean
Filter Expression (DB)	A string that contains conditions that DynamoDB applies after the Scan operation, but before the data is returned to you.	NULL	String
Exclusive Start Partition Key Value (DB)	The partition key of the first item that this operation will evaluate. This attribute is mandatory only when fetching paginated results.	NULL	String
Exclusive Start Sort Key Value (DB)	The sort key of the first item that this operation will evaluate. This attribute is mandatory only when fetching paginated results.	NULL	String
Exclusive Start Index Partition Key Value (DB)	The index partition key of the first item that this operation will evaluate. This attribute is mandatory only when fetching paginated results from an Index.	NULL	String
Exclusive Start Index Sort Key Value (DB)	The index sort key of the first item that this operation will evaluate. This attribute is mandatory only when fetching paginated results from an Index.	NULL	String
Limit	Maximum number of items to be returned by the operation. Minimum allowed is 1 and maximum allowed is 1000.	1000	Integer
Automatic Pagination	If active, this parameter will automatically fetch all pages when DynamoDB returns paginated responses.	False	Boolean
Scan Forward	If active, this parameter returns the result in its original ordering (based on the sort key). Otherwise, the order is inverted.	True	Boolean
Expression Attribute Values (DB)	A list of attribute values used in the conditional expression fields (Filter Expression). Each attribute value must specify the Placeholder used, the Type of the data, and the Value to replace the placeholder.	Empty	List of Attribute Values

Output

Every operation returns:

• A "success" boolean attribute to indicate if the operation was performed successfully (true), or if it failed (false).

• A counter attribute, indicating how many items were affected by the operation. This parameter is named after the operation in the following format: <operation's name>+"Count".

• A "data" attribute containing an array of returned item records. This is restricted to operations that return something.

GetItem example

{
	"success": true,
	"getItemCount": 1,
	"data": [
		{
	"Age": 8,
	"Colors": [
		"White", 
		"Brown",
		"Black"
],
"Name": "Fido",
"Vaccinations": {
	"Rabies": [
		"2009-03-17",
		"2011-09-21",
		"2014-07-08"
	],
	"Distemper": "2015-10-13"
},
"Breed": "Beagle",
"AnimalType": "Dog"
}
	]
}

Query and Scan additional details

Conditional expressions

When using the Query and Scan operations, the developer can use conditional expressions to filter the returned items. Conditional expressions can be defined in the fields Sort Key Expression (available only in the Query operation), and Filter Expression (available in the Query and Scan operations).

These expressions are similar to filtering options in a SQL query as it can be seen in the following examples:

Conditional expression example 1

Conditional expression example 2

Notice that the actual values to be used in the conditions cannot be placed in the expressions directly. Instead, a placeholder in the format "colon + name" must be used, and the value is then passed via the Expression Attribute Values field. In the example above, the following attribute values would have to be specified:

Conditional expression example 3

Conditional expression example 4 - Placeholder detail

Notice that the Placeholder field value must match exactly the names used in the conditional expressions, including the colon.

Double Braces in conditional expressions

When using Double Braces expressions embedded in conditional expressions, there is no need to specify the corresponding values through the Expression Attribute Values field. The same example above could have been specified like this:

Conditional expression example 5 - Sort Key Expression

Conditional expression example 6 - Filter Expression

In this case, the connector infers the data types and values from the JSON input message. The placeholders are automatically generated by the connector using sequential numbers. Notice that in this situation a message like the following would be expected as input:

{
	"skMinValue": 8965,
	"skMaxValue": 9165,
	"createdAtBottom": "2024-03-25T16:11:22Z",
	"deletedVal": false
}

See the AWS documentation for further reference on conditional expressions.

Paginated results

DynamoDB may paginate the results of a Query or Scan operation in two situations:

• The number of fetched items is bigger than the number established in the Limit parameter.

• The total size of the response exceeds 1MB.

In situations where a paginated response is returned, an extra lastEvaluatedKey field will be added to the output, containing the primary keys of the last item returned. These returned keys can then be used in a subsequent Query or Scan operation passing them in the Exclusive Start Key fields.

The following is an example of a paginated output:

{
	"success": true,
       	"data":[
		{...},
		{...},
		{...},
		...
	],
	"queryCount": 1000,
	"lastEvaluatedKey": {
		"partitionKeyName": 12,
		"sortKeyName" "abc",
		"indexPartitionKeyName": 123,
		"indexSortKeyName": "index-abc"
	}
}

Notice that in this case, we have a map between the key fields names and their respective values. Index key fields are omitted if the operation doesn’t have the field Index Name specified.

This is how the Exclusive Start Key fields would have to be specified for the following page iteration:

Conditional expression example 7 - Exclusive Start Key