DynamoDB Streams Trigger

Learn more about the DynamoDB Streams Trigger and how to use it in the Digibee Integration Platform.

DynamoDB Streams is a feature of the DynamoDB database that publishes near real-time events for every record modification in a DynamoDB table that has DynamoDB Streams enabled. The DynamoDB Streams Trigger captures these events and sends them to pipelines running in Digibee's infrastructure. This allows the Change Data Capture (CDC) pattern to be easily implemented with Digibee pipelines.

Parameters

Take a look at the configuration parameters of the trigger. Parameters supported by Double Braces expressions are marked with (DB).

Parameter	Description	Default value	Data type
DynamoDB Client Account	AWS account used to access the DynamoDB Streams API.	N/A	String
Table Name	Name of the DynamoDB table from which to fetch the stream of events.	N/A	String
AWS Region	Defines the AWS region where the DynamoDB is located.	us-east-1	String
Expiration	The amount of time that the event spends in the queue (in milliseconds). If the value equals 0 or is a value greater than 6h (21600000 ms), then the expiration will be 1/4 of the specified Maximum Timeout value.	600000	Integer
Maximum Timeout	Maximum time (in milliseconds) for the pipeline to process information before returning a response. Default: 30000. Limit: 900000.	30000	Integer
Allow Redelivery of Messages	If activated, the option allows messages to be delivered again if the Pipeline Engine fails.	False	Boolean

Additional information

The DynamoDB Streams Trigger does not automatically activate the Streams feature on an existing DynamoDB table. Instead, the trigger assumes that the table exists with the feature pre-configured and sends the events to the pipeline as they are, without any transformation.

See below an example of a DynamoDB Streams event:

{
   "eventID":"1",
   "eventName":"INSERT",
   "eventVersion":"1.0",
   "eventSource":"aws:dynamodb",
   "awsRegion":"us-east-1",
   "dynamodb":{
      "Keys":{
         "Id":{
            "N":"101"
         }
      },
      "NewImage":{
         "Message":{
            "S":"New item!"
         },
         "Id":{
            "N":"101"
         }
      },
      "SequenceNumber":"111",
      "SizeBytes":26,
      "StreamViewType":"NEW_AND_OLD_IMAGES"
   }
}

Note that the trigger, once deployed, always starts by consuming the latest event from the Streams. This means that the pipeline will not receive events that have been published before it is online. This prevents the execution environment from being flooded with events, which can lead to out-of-memory errors and delivery delays.