Avro File Reader

Learn more about the Avro File Reader connector and how to use it in the Digibee Integration Platform.

Avro File Reader is a Pipeline Engine v2 exclusive connector.

The Avro File Reader connector allows you to read Avro files.

Avro is a popular data serialization framework used within the Hadoop Big Data ecosystem, which is known for its schema evolution support and compactness. For more information, see the official website.

Parameters

Take a look at the configuration parameters of the connector. Parameters supported by Double Braces expressions are marked with (DB).

General tab

Parameter	Description	Default value	Data type
File Name (DB)	The file name of the Avro file to be read.	{{ message.fileName }}	String
Check File Size	If the option is active, the specified Maximum File Size is checked. In case of a higher size, an error will be thrown.	False	Boolean
Maximum File Size	Specifies the maximum size allowed (in bytes) of the file to be read.	N/A	Integer
Fail On Error	If the option is active, the execution of the pipeline with an error will be interrupted. Otherwise, the pipeline execution proceeds, but the result will show a false value for the "success" property.	False	Boolean

Documentation tab

Parameter	Description	Default value	Data type
Documentation	Section for documenting any necessary information about the connector configuration and business rules.	N/A	String

Note that a compressed Avro file produces a JSON content that is larger than the file itself when it’s read. Therefore, it’s important to validate whether the pipeline has enough memory to handle the data, as it will be stored in the pipeline's memory.

Usage examples

Reading file

Reading an Avro file without checking the file size:

• File Name: file.avro

• Check File Size: deactivated

Output:

{
  "data": [
    {
      "name": "Aquiles",
      "phoneNumbers": [
        "11 99999-9999",
        "11 93333-3333"
      ],
      "active": true,
      "address": "St. Example",
      "score": 71.3,
      "details": "Some details"
    }
  ],
  "fileName": "file.avro",
  "total": 1
}

Reading file - Checking file size

Reading an Avro file checking its size being higher than the Maximum File Size:

• File Name: file.avro

• Check File Size: activated

• Maximum File Size: 5000000

Output:

{
  "data": [
    {
      "name": "Aquiles",
      "phoneNumbers": [
        "11 99999-9999",
        "11 93333-3333"
      ],
      "active": true,
      "address": "St. Example",
      "score": 71.3,
      "details": "Some details"
    }
  ],
  "fileName": "file.avro",
  "total": 1
}