Stream CSV Reader (Beta)

Discover more about the Stream CSV Reader connector and how to use it on the Digibee Integration Platform.

The Stream CSV Reader connector is currently in beta phase. Learn more about the Beta Program.

Stream CSV Reader reads a local CSV file row by row in a JSON structure and triggers subflows to process each line. This resource is recommended for scenarios in which large files need to be processed efficiently and at scale.

Parameters

Configure the connector using the parameters below. Fields that support Double Braces expressions are marked in the Supports DB column.

This connector supports the Alias parameter, which allows you to save its output and reference it later in the flow using Double Braces expressions. Learn more.

General tab

Parameter	Description	Type	Supports DB	Default
File Name	Name of the local CSV file to read.	String	✅	data.csv
Charset	Character encoding used to read the file.	String	❌	UTF-8
Element Identifier	In case of errors, the defined attributes are going to be sent to the onException sub-process.	String	❌	data
Parallel Execution Of Each Iteration	If enabled, each line is processed in parallel.	Boolean	❌	False
Ignore Invalid Charset	If enabled, invalid charset characters are ignored.	Boolean	❌	False
Fail On Error	If enabled, interrupts the pipeline execution when an error occurs. If disabled, execution continues, but the "success" property will be set to false.	Boolean	❌	False
Advanced	Enables advanced parameters.	Boolean	❌	False
Delimiter	Defines which delimiter to use.	String	❌	N/A
Skip	The number of lines to skip before starting to read the file.	Integer	✅	N/A
Limit	Maximum number of rows to read from the file. A value of 0 means no limit.	Integer	✅	0
Chunk Size	Number of lines to process in each batch.	Integer	✅	1
Ignore Header	If enabled, skips the first line (header) of the file.	Boolean	❌	False
Custom Headers (,)	Defines a comma-separated list of custom headers to be used instead of the original CSV headers.	String	❌	N/A
Filter Headers (,)	Defines which CSV headers should be read, provided as a comma-separated list.	String	❌	N/A

Documentation tab

Parameter	Description	Default value	Data type
Documentation	Optional field to describe the connector configuration and any relevant business rules.	N/A	String

Messages flow

Input

The connector waits for a message in the following format:

{
    "filename": "fileName"
}

Output

{
    "total": 0,
    "success": 0,
    "failed": 0
}

• total: Total number of processed rows.

• success: Total number of rows successfully processed.

• failed: Total number of rows whose processing failed.

To know whether a line has been processed correctly, the return value { "success": true } must be present for each processed line.

Additional information

• The connector throws an exception if the File Name doesn't exist or can't be read.

• The file manipulation inside a pipeline occurs in a protected way. All the files can be accessed with a temporary directory only, where each pipeline key gives access to its own files set.

• This connector makes batch processing, which means processing the data continuously and in a controlled manner in smaller batches.