Avro File Writer

Learn more about the Avro File Writer connector and how to use it in the Digibee Integration Platform.

Avro File Writer is a Pipeline Engine v2 exclusive connector.

The Avro File Writer connector allows you to write Avro files based on Avro schemas.

Avro is a popular data serialization framework used within the Hadoop Big Data ecosystem, known for its schema evolution support and compactness. For more information, see the official website.

Parameters

Take a look at the configuration parameters of the connector. Parameters supported by Double Braces expressions are marked with (DB).

General tab

Parameter

Description

Default value

Data type

File Name (DB)

The file name of the Avro file to be written.

file.avro

String

Data (DB)

The data to be written to the Avro file.

It only accepts JSON objects or arrays of objects.

String

Schema (DB)

The Avro Schema to be used for writing the file.

It only accepts schemas with RECORD type as the root data type.

N/A

String

Data From File

If active, the data to be written to the Avro file must come from other Avro files and not from a JSON payload.

False

Boolean

Files

Defines the Avro files containing the data to be written to the final Avro file.

This option is available only when Data From File is active.

N/A

Options of Data From File

File Name (DB)

The name of the Avro file that contains the data.

This option is available only when Data From File is active.

N/A

String

Infer Schema From File

If the option is active, the Avro Schema will be inferred from the first Avro file defined in the Files parameter.

False

Boolean

File Exists Policy

Defines which behavior to be followed when a file with the same name (File Name parameter) already exists in the current pipeline execution.

You can select the following options: Append (append data to an existing file), Overwrite (overwrite the existing file), or Fail (execution interrupted with an error if the file already exists).

Append

String

Fail On Error

If the option is active, the execution of the pipeline with an error will be interrupted. Otherwise, the pipeline execution proceeds, but the result will show a false value for the "success" property.

False

Boolean

Advanced tab

Parameter

Description

Default value

Data type

Compression Codec

The compression codec to be used when compressing the Avro file.

Options:

Uncompressed
DEFLATE
BZIP2

Uncompressed

String

Compression Level

The level of compression to be applied when compressing the Avro file. Options: 1-9.

This option is only available when Compression Codec is set as DEFLATE.

Integer

Documentation tab

Parameter

Description

Default value

Data type

Documentation

Section for documenting any necessary information about the connector configuration and business rules.

N/A

String

Note that performance differences can occur when writing compressed and uncompressed Avro files. Since compression requires greater memory and processing consumption, it’s important to validate the limits that the pipeline should support when applying it.

Usage examples

File from JSON object

Writing an Avro File based on a JSON object payload:

File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite

Data example:

{
  "data": {
    "name": "Aquiles",
    "phoneNumbers": [
      "11 99999-9999",
      "11 93333-3333"
    ],
    "active": true,
    "address": "St. Example",
    "score": 71.3,
    "details": "Some details"
  }
}

Schema example:

{
  "schema": {
    "type": "record",
    "name": "Record",
    "fields": [
        {
            "name": "name",
            "type": "string"
        },
        {
            "name": "phoneNumbers",
            "type": {
                "type": "array",
                "items": "string"
            }
        },
        {
            "name": "active",
            "type": "boolean"
        },
        {
            "name": "address",
            "type": "string"
        },
        {
            "name": "score",
            "type": "double"
        },
        {
            "name": "details",
            "type": [
                "string",
                "null"
            ]
        }
    ]
  }
}

Output:

{
  "success": true,
  "fileName": "file.avro"
}

File from JSON array of objects

Writing an Avro File based on a JSON array of objects payload:

File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite

Data example:

{
  "data": [ 
    {
      "name": "Aquiles",
      "phoneNumbers": [
        "11 99999-9999",
        "11 93333-3333"
      ],
      "active": true,
      "address": "St. Example",
      "score": 71.3,
      "details": "Some details"
    },
    {
      "name": "Vitor",
      "phoneNumbers": [
        "11 97777-7777"
      ],
      "active": false,
      "address": "St. Example 2",
      "score": 80.0,
      "details": null
    }
  ]
}

Schema example:

{
  "schema": {
    "type": "record",
    "name": "Record",
    "fields": [
        {
            "name": "name",
            "type": "string"
        },
        {
            "name": "phoneNumbers",
            "type": {
                "type": "array",
                "items": "string"
            }
        },
        {
            "name": "active",
            "type": "boolean"
        },
        {
            "name": "address",
            "type": "string"
        },
        {
            "name": "score",
            "type": "double"
        },
        {
            "name": "details",
            "type": [
                "string",
                "null"
            ]
        }
    ]
  }
}

Output:

{
  "success": true,
  "fileName": "file.avro"
}

Uncompressed Avro file

Writing an uncompressed Avro File:

File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite
Compression Codec: Uncompressed

Output:

{
  "success": true,
  "fileName": "file.avro"
}

Compressed Avro file

Writing a compressed Avro File:

File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite
Compression Codec: BZIP2

Output:

{
  "success": true,
  "fileName": "file.avro"
}

File Exists Policy as Fail

Writing an Avro File with the same name of an existent file in the pipeline file directory:

File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Fail

Output:

{
  "success": false,
  "message": "Something went wrong while trying to execute the Avro Writer connector",
  "error": "com.digibee.pipelineengine.exception.PipelineEngineRuntimeException: Avro file file.avro already exists."
}

Writing file from another Avro file - Explicit schema

Writing an Avro File with the data to be written coming from other Avro files instead of from a JSON payload, using a Schema explicit configuration:

File Name: file.avro
Data From File: activated
Files:
- File Name: {{ message.existingAvroFile }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite

Output:

{
  "success": true,
  "fileName": "file.avro"
}

Writing file from another Avro file - Infer Schema

Writing an Avro File with the data to be written coming from other Avro files instead of from a JSON payload, inferring the schema from the file:

File Name: file.avro
Data From File: activated
Files:
- File Name: {{ message.existingAvroFile }}
Infer Schema: activated
File Exists Policy: Overwrite

Output:

{
  "success": true,
  "fileName": "file.avro"
}

PreviousAvro File Reader NextCSV to Excel

Last updated 7 months ago