S3 Storage

Discover more about the S3 Storage connector and how to use it on the Digibee Integration Platform.

S3 Storage connects to AWS S3 and performs the following operations on the storage: List, Download, Upload, Delete, and Move.

Parameters

Configure the connector using the parameters below. Fields that support Double Braces expressions are marked in the Supports DB column.

Parameter

Description

Data Type

Supports DB

Default value

Visible when

Use Dynamic Account

If enabled, the connector uses the account dynamically via the Store Account connector; otherwise, it uses the account statically as configured in Accounts on the Platform.

Boolean

❌

false

—

Scoped

If enabled, isolates the stored account from other sub-processes. Not supported for accounts used in headers or body. To learn more, read the Dynamic Accounts documentation.

Boolean

❌

false

Use Dynamic Account is enabled

Account Name

Name of the account defined in the Store Account connector.

String

✅

N/A

Use Dynamic Account is enabled

Account

Account used by the connector. Must be an AWS V4 account type, configured in Accounts on the Platform. It's necessary to provide the client ID and secret key from the AWS console.

String

❌

N/A

Use Dynamic Account is disabled

Operation

Defines the operation to be performed: List, Download, Upload, Delete, or Move.

String

❌

Upload

—

Region

AWS region where the S3 bucket is located.

String

❌

South America (Sao Paulo)

—

Bucket Name

Name of the S3 bucket.

String

✅

digibee-amazon-s3-connector-test

—

Bucket Name - Move

Name of the bucket from which the file is to be moved.

String

✅

N/A

Operation is Move

File Name

Name or full path of the local file to be used in the operation (e.g., tmp/processed/file.txt).

String

✅

file.csv

Operation is Download or Upload

Remote File Name

Name or full path of the S3 file to be used in the operation (e.g., tmp/processed/file.txt).

String

✅

test.csv

Operation is Download, Upload, Delete, or Move

Remote File Name - Move

New name to be assigned to the remote file after it is moved.

String

✅

N/A

Operation is Move

Remote Directory

Remote directory in S3 to be used in a Download, Upload, or Delete operation.

String

❌

upload/

—

Remote Directory - Move

Name of the remote directory containing the file to be moved.

String

❌

N/A

Operation is Move

Page Size

Number of items to be returned per page. Default is 1000. To retrieve more items than the page size, use the nextToken returned in the response with the Next Page Token parameter.

Integer

❌

1000

Operation is List

Next Page Token

Token used to request the next page of results. The next page returns the number of items defined in Page Size.

String

❌

N/A

Operation is List

Generate Download Link

If enabled, generates a public download link for the file.

Boolean

❌

false

Operation is Upload

Expiration Timestamp (in ms)

Expiration time for the download link, in milliseconds. Provide the current timestamp plus the desired duration (e.g., current timestamp + 600000 for a 10-minute link). If not provided, the link expires 15 minutes after the current timestamp.

Number

❌

N/A

Operation is Upload and Generate Download Link is enabled

Include user metadata on upload

If enabled, attaches the metadata configured in Metadata or Metadata file to the object in S3. If disabled, the object is uploaded without user metadata, even if Metadata or Metadata file are configured.

Boolean

❌

false

Operation is Upload

Metadata (JSON)

JSON object containing custom key-value pairs to be attached to the object in S3. When configured, the connector returns the metadata in the step response. Ignored when Metadata file resolves to a non-empty path.

JSON Object

✅

N/A

Operation is Upload and Include user metadata on upload is enabled

Metadata file path

Relative path to a JSON file in the pipeline's working directory (e.g., orders-report.csv.metadata.json). When this field resolves to a non-empty path, the connector reads the file's content and uses it as the object metadata, ignoring any value set in Metadata. If the field is absent, blank, or resolves to an empty value, the connector falls back to Metadata.

String

✅

N/A

Operation is Upload and Include user metadata on upload is enabled

Path-style access

If enabled, uses path-style URL format to access S3. Useful when connecting to S3-compatible storage that requires path-style access, such as a custom endpoint.

Boolean

❌

false

—

Fail on error

If enabled, interrupts pipeline execution when an error occurs. Otherwise, execution proceeds and the result returns false for the "success" property.

Boolean

❌

false

—

Custom Endpoint

If enabled, allows the use of a custom endpoint URL for S3.

Boolean

❌

false

—

Endpoint URL

Custom endpoint URL.

String

✅

N/A

Custom Endpoint is enabled

File manipulation inside a pipeline is performed in a protected area. All files are accessible only through a temporary directory, and each pipeline key provides access to its own set of files.

Messages flow

Input

The connector only requires an input message if a field is configured with a Double Braces expression. Otherwise, no specific input is expected; simply configure the fields for the selected operation.

Output

The connector returns a JSON object containing the result of the operation. The fields returned vary by operation.

List

{
  "success": true,
  "content": [
    {
      "bucketName": "digibee-amazon-s3-connector-test",
      "key": "list/test.csv",
      "size": 9,
      "lastModified": 1596139663000,
      "storageClass": "STANDARD",
      "owner": null,
      "etag": "59587d0fd956dee6905d423bfda2acaf"
    }
  ],
  "count": 1,
  "nextToken": "1kWwy…..."
}

Output fields:

success: returns true if the operation succeeded, or false if it failed.
content: array containing the file information.
bucketName: name of the bucket.
key: directory path and file name.
size: size of the file.
lastModified: date of the last modification made to the file.
storageClass: storage class configured in S3.
owner: owner of the file.
etag: entity tag, a hash generated by S3 for the file.
count: number of objects returned.
nextToken: token used to paginate the remaining items. Returned only when there are more objects to list.

Download

{
  "success": true,
  "fileName": "test.file",
  "remoteDirectory": "pagination_folder/",
  "remoteFileName": "c4b88b6b-83bb-42b0-9de6-0371389db585.csv",
  "bucketName": "digibee-amazon-s3-connector-test"
}

Output fields:

success: returns true if the operation succeeded, or false if it failed.
fileName: name of the file downloaded to the pipeline directory.
remoteDirectory: name of the remote directory in S3.
remoteFileName: name of the remote file in S3.
bucketName: name of the S3 bucket.

Upload

{
  "success": true,
  "fileName": "test.file",
  "remoteDirectory": "pagination_folder/",
  "remoteFileName": "test.file",
  "urlGenerated": "https://digibee-amazon-s3-connector-test.s3.sa-east-1.amazonaws.com/pagination_folder/test.file?....",
  "bucketName": "digibee-amazon-s3-connector-test"
}

When Metadata or Metadata file is configured and Include user metadata is enabled, the response also includes the metadata attached to the object:

{
  "success": true,
  "fileName": "test.file",
  "remoteDirectory": "pagination_folder/",
  "remoteFileName": "test.file",
  "urlGenerated": "https://digibee-amazon-s3-connector-test.s3.sa-east-1.amazonaws.com/pagination_folder/test.file?....",
  "bucketName": "digibee-amazon-s3-connector-test",
  "metadata": {
    "source": "orders-pipeline",
    "environment": "production"
  }
}

Output fields:

success: returns true if the operation succeeded, or false if it failed.
fileName: name of the local file uploaded from the pipeline directory.
remoteDirectory: name of the remote directory in S3.
remoteFileName: name of the remote file in S3.
bucketName: name of the S3 bucket.
urlGenerated: public download link for the file. Returned only when Generate Download Link is enabled.
metadata: key-value pairs attached to the object as user metadata. Returned only when metadata is applied.

Move

{
  "success": true,
  "remoteDirectory": "pagination_folder/",
  "remoteFileName": "c4b88b6b-83bb-42b0-9de6-0371389db585.csv",
  "remoteFileNameMove": "abc.file",
  "remoteDirectoryMove": "list/",
  "bucketName": "digibee-amazon-s3-connector-test",
  "bucketNameMove": "digibee-amazon-s3-connector-test"
}

Output fields:

success: returns true if the operation succeeded, or false if it failed.
remoteDirectory: name of the original remote directory in S3.
remoteFileName: name of the original remote file in S3.
bucketName: name of the original S3 bucket.
bucketNameMove: name of the destination bucket.
remoteDirectoryMove: name of the destination remote directory.
remoteFileNameMove: new name assigned to the file after it is moved.

Delete

{
  "success": true,
  "remoteDirectory": "list/",
  "remoteFileName": "abc.file",
  "bucketName": "digibee-amazon-s3-connector-test"
}

Output fields:

success: returns true if the operation succeeded, or false if it failed.
remoteDirectory: name of the remote directory in S3.
remoteFileName: name of the remote file deleted from S3.
bucketName: name of the S3 bucket.

Error

{
  "success": false,
  "message": "Could no issue the operation: download in the AWS S3 Storage",
  "error": "com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; ...)"
}

Output fields:

success: returns false when the operation fails.
message: error description provided by the connector.
error: error message returned by the S3 server.

PreviousAWS NextSQS

Last updated 4 days ago

Was this helpful?