Stream Parquet File Reader

Learn more about the Stream Parquet File Reader connector and how to use it in the Digibee Integration Platform.

The Stream Parquet File Reader connector allows you to read Parquet files triggering subpipelines to process each message individually. This connector should be used for large files.

Parquet is a columnar file format designed for efficient data storage and retrieval. For more information, see the official website.

Parameters

Take a look at the configuration parameters of the connector. Parameters supported by Double Braces expressions are marked with (DB).

General Tab

Documentation Tab

A compressed Parquet file generates JSON content larger than the file itself when it is read. It is important that you checj whether the pipeline has enough memory to handle the data, as it will be stored in the pipeline's memory.

Usage examples

Reading Parquet file

  • File Name: file.parquet

  • Parallel: deactivated

Output:

{
	"total": 1000,
	"success": 1000,
	"failed": 0
}

If the lines have been processed correctly, their respective subpipelines return { "success": true } for each individual line.

Last updated