Stream CSV Reader (Beta)

Discover more about the Stream CSV Reader connector and how to use it on the Digibee Integration Platform.

The Stream CSV Reader connector is currently in beta phase. Learn more about the Beta Program.

Stream CSV Reader reads a local CSV file row by row in a JSON structure and triggers subflows to process each line. This resource is recommended for scenarios in which large files need to be processed efficiently and at scale.

Parameters

Take a look at the configuration parameters for the connector. Parameters supported by Double Braces expressions are marked with (DB).

This connector supports the Alias parameter, which allows you to save its output and reference it later in the flow using Double Braces expressions. Learn more.

General tab

Parameter
Description
Default value
Data type

File Name (DB)

Name of the local CSV file to read.

data.csv

String

Charset

Character encoding used to read the file.

UTF-8

String

Headers

Custom headers to replace the file's original header (comma-separated).

A,B,C

String

Delimiter

Character used to separate values in the CSV file.

,

String

Parallel Execution Of Each Iteration

If enabled, each line is processed in parallel.

False

Boolean

Ignore Invalid Charset

If enabled, invalid charset characters are ignored.

False

Boolean

Ignore Header

If enabled, skips the first line (header) of the file.

False

Boolean

Keep Header

If enabled, processes the header line as data.

False

Boolean

Advanced

Enables advanced parameters.

False

Boolean

Metadata Only

If enabled, returns only file metadata (row count and file size) without processing data.

False

Boolean

Limit (DB)

Maximum number of rows to read from the file. A value of 0 means no limit.

0

Integer

Fail On Error

If enabled, interrupts the pipeline execution when an error occurs. If disabled, execution continues, but the "success" property will be set to false.

False

Boolean

Documentation

Parameter
Description
Default value
Data type

Documentation

Optional field to describe the connector configuration and any relevant business rules.

N/A

String

Messages flow

Input

The connector waits for a message in the following format:

{
    "filename": "fileName"
}

Output

{
    "total": 0,
    "success": 0,
    "failed": 0
}
  • total: Total number of processed rows.

  • success: Total number of rows successfully processed.

  • failed: Total number of rows whose processing failed.

Additional information

  • The connector throws an exception if the File Name doesn't exist or can't be read.

  • The file manipulation inside a pipeline occurs in a protected way. All the files can be accessed with a temporary directory only, where each pipeline key gives access to its own files set.

  • This connector makes batch processing, which means processing the data continuously and in a controlled manner in smaller batches.

Last updated

Was this helpful?