Avro File Writer
Learn more about the Avro File Writer connector and how to use it in the Digibee Integration Platform.
Avro File Writer is a Pipeline Engine v2 exclusive connector.
The Avro File Writer connector allows you to write Avro files based on Avro schemas.
Avro is a popular data serialization framework used within the Hadoop Big Data ecosystem, known for its schema evolution support and compactness. For more information, see the official website.
Parameters
Take a look at the configuration parameters of the connector. Parameters supported by Double Braces expressions are marked with (DB)
.
General tab
Advanced tab
Documentation tab
Note that performance differences can occur when writing compressed and uncompressed Avro files. Since compression requires greater memory and processing consumption, it’s important to validate the limits that the pipeline should support when applying it.
Usage examples
File from JSON object
Writing an Avro File based on a JSON object payload:
File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite
Data example:
Schema example:
Output:
File from JSON array of objects
Writing an Avro File based on a JSON array of objects payload:
File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite
Data example:
Schema example:
Output:
Uncompressed Avro file
Writing an uncompressed Avro File:
File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite
Compression Codec: Uncompressed
Output:
Compressed Avro file
Writing a compressed Avro File:
File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite
Compression Codec: BZIP2
Output:
File Exists Policy as Fail
Writing an Avro File with the same name of an existent file in the pipeline file directory:
File Name: file.avro
Data: {{ message.data }}
Schema: {{ message.schema }}
File Exists Policy: Fail
Output:
Writing file from another Avro file - Explicit schema
Writing an Avro File with the data to be written coming from other Avro files instead of from a JSON payload, using a Schema explicit configuration:
File Name: file.avro
Data From File: activated
Files:
File Name: {{ message.existingAvroFile }}
Schema: {{ message.schema }}
File Exists Policy: Overwrite
Output:
Writing file from another Avro file - Infer Schema
Writing an Avro File with the data to be written coming from other Avro files instead of from a JSON payload, inferring the schema from the file:
File Name: file.avro
Data From File: activated
Files:
File Name: {{ message.existingAvroFile }}
Infer Schema: activated
File Exists Policy: Overwrite
Output:
Last updated