# Parquet File Writer

{% hint style="info" %}
**Parquet File Writer** is a Pipeline Engine v2 exclusive connector.
{% endhint %}

The **Parquet File Writer** connector allows you to read Parquet files based on Avro files.

Parquet is a columnar file format designed for efficient data storage and retrieval. Further information can be found [on the official website](https://parquet.apache.org/).

## **Parameters**

Take a look at the configuration parameters of the connector. Parameters supported by [Double Braces expressions](https://docs.digibee.com/documentation/connectors-and-triggers/double-braces/overview) are marked with `(DB)`.

### **General tab**

<table data-full-width="true"><thead><tr><th>Parameter</th><th>Description</th><th>Default value</th><th>Data type</th></tr></thead><tbody><tr><td><strong>Parquet File Name</strong> <code>(DB)</code></td><td>The file name of the Parquet file to be written.</td><td>file.parquet</td><td>String</td></tr><tr><td><strong>Avro File Name</strong> <code>(DB)</code></td><td><p>The file name of the Avro file that contains the data to be written to the Parquet file.</p><p>Only Avro files within schemas with the type <code>RECORD</code> as the root data type are accepted.</p></td><td>file.avro</td><td>String</td></tr><tr><td><strong>File Exists Policy</strong></td><td><p>Defines which behavior to follow when a file with the same name (<strong>Parquet File Name</strong> parameter) already exists in the current pipeline execution.</p><p>You can select the following options: <strong>Overwrite</strong> (overwrite the existing file) or <strong>Fail</strong> ( interrupt execution with an error if the file already exists).</p></td><td>Overwrite</td><td>String</td></tr><tr><td><strong>Fail On Error</strong></td><td>If the option is active, the execution of the pipeline with an error will be interrupted. Otherwise, the pipeline execution proceeds, but the result will show a false value for the <code>"success"</code> property.</td><td>False</td><td>Boolean</td></tr></tbody></table>

### **Advanced tab**

<table data-full-width="true"><thead><tr><th>Parameter</th><th>Description</th><th>Default value</th><th>Data type</th></tr></thead><tbody><tr><td><strong>Dictionary Encoding</strong></td><td>Defines if dictionary encoding for columns must be enabled.</td><td>False</td><td>Boolean</td></tr><tr><td><strong>Compression Codec</strong></td><td><p>The compression codec to be used when compressing the Parquet file.</p><p>Options:</p><ul><li><strong>Uncompressed</strong></li><li><strong>Snappy</strong></li><li><strong>GZIP</strong></li><li><strong>LZ4</strong></li><li><strong>LZ4 Raw</strong></li></ul></td><td>Uncompressed</td><td>String</td></tr><tr><td><strong>Row Group Size</strong></td><td>Defines the size of row groups in the Parquet file.</td><td>134217728</td><td>Integer</td></tr><tr><td><strong>Page Size</strong></td><td>Defines the size of pages in the Parquet file.</td><td>1048576</td><td>Integer</td></tr></tbody></table>

### **Documentation tab**

<table data-full-width="true"><thead><tr><th>Parameter</th><th>Description</th><th>Default value</th><th>Data type</th></tr></thead><tbody><tr><td><strong>Documentation</strong></td><td>Section for documenting any necessary information about the connector configuration and business rules.</td><td>N/A</td><td>String</td></tr></tbody></table>

## Important information

The **Parquet File Writer** connector can only generate Parquet files based on Avro files. It's not possible to create them directly from a JSON payload.

Despite this limitation, the Digibee Integration Platform provides a way to generate Avro files using the **Avro File Writer** connector, apart from the fact that the **Parquet File Writer** connector can handle Avro files generated from another source outside the platform.

When writing a Parquet file using the connector, Avro files containing the data types `BINARY` and `FIXED` are treated as binary data. When reading the generated file with the **Parquet File Reader** connector, the data for these types is displayed in base64 format.

You should also note that performance differences can occur when writing compressed and uncompressed Parquet files. Since compression requires more memory and processing, it's important to validate the limits supported by the pipeline when you apply it.

## **Usage examples**

### **Uncompressed Parquet file**

Writing an uncompressed Parquet File based on an Avro file:

* **Parquet File Name:** file.parquet
* **Avro File Name:** file.avro
* **File Exists Policy:** Overwrite
* **Compression Codec:** Uncompressed

**Example of Avro file content in JSON format:**

```
{
  "name": "Aquiles",
  "phoneNumbers": [
    "11 99999-9999",
    "11 93333-3333"
  ],
  "active": true,
  "address": "St. Example",
  "score": 71.3,
  "details": "Some details"
}

```

**Output:**

```
{
  "success": true,
  "fileName": "file.parquet"
}

```

### **Compressed Parquet file**

Writing a compressed Parquet File based on an Avro file:

* **Parquet File Name:** file.parquet
* **Avro File Name:** file.avro
* **File Exists Policy:** Overwrite
* **Compression Codec:** Snappy

**Example of Avro file content in JSON format:**

```
{
  "name": "Aquiles",
  "phoneNumbers": [
    "11 99999-9999",
    "11 93333-3333"
  ],
  "active": true,
  "address": "St. Example",
  "score": 71.3,
  "details": "Some details"
}

```

**Output:**

```
{
  "success": true,
  "fileName": "file.parquet"
}

```

### **File Exists Policy as Fail**

Writing a Parquet File with the same name of an existent file in the pipeline file directory:

* **Parquet File Name:** file.parquet
* **Avro File Name:** file.avro
* **File Exists Policy:** Fail

**Output:**

{% code overflow="wrap" %}

```
{
  "success": false,
  "message": "Something went wrong while trying to execute the Parquet Writer connector",
  "error": "com.digibee.pipelineengine.exception.PipelineEngineRuntimeException: Parquet file file.parquet already exists."
}

```

{% endcode %}