Stream DB V1 (Deprecated)

Know the component and how to use it.

Stream DB V1 makes operations through the connection with a database, streaming data to a subpipeline that processes them.
Take a look at the configuration parameters of the component:
Account: account to be used by the component.
Database URL: connection string to the database.
SQL Statement: SQL statement to be executed.
Column Name: name of the columns to be read.
Coalesce: when activated, this option controls if the data objects are coalesced into a string or if an error is generated.
Parallel Execution Of Each Iteration: occurs in parallel with the loop execution.
Fail On Error: when activated, this parameter suspends the pipeline execution only if there’s a severe occurrence in the iteration structure, disabling its complete conclusion. The “Fail On Error” parameter activation doesn’t have any connection with the errors occurred in the components used for the construction of the subpipelines (onProcess and onException).
Custom Connection Properties: specific connection properties defined by the user.
Keep Connections: if activated, the option will keep the connections to the database for 30 minutes maximum; otheriwse, it will be kept for 5 minutes only.
Advanced: advanced configurations.
Connection Test Query: SQL statement to be used before each connection is established - this is an optional parameter and must be applied to databases that don't have reliable information about the connection status.
Messages flow
Input
The component waits for a message in the following format:
{
    "parameters": {
        "name": "value"
        ...
    }
}
Message "onProcess" structure
{
    "column1":"data1", 
    "column2":"data2", 
    ...
}
Output with error
{
    "code": error_code,"error": error_message,
    "processedId": the_id_column_value
}
Output
{
    "total": 0,
    "success": 0,
    "failed": 0
}
total: total number of processed lines
success: total number of successfully processed lines
failed: total number of lines whose process failed
IMPORTANT: when the lines are correctly processed, their respective subpipelines return { "success": true } for each of them.
This component makes batch processing. To better understand the concept, click here.

Stream DB V3

Google Big Table

Last modified 5mo ago