How to use Data Streaming with Splunk

Learn how to stream your data automatically to Splunk

Only users with the data-streaming-manager role can create, update, read, and delete data streams. To learn more, see the Roles documentation.

Enabling the Data Streaming feature allows you to export or consume data through external tools and systems of your choice automatically. Instead of using an API Key, Splunk uses a HEC Token (HTTP Event Collector Token), designed specifically for high-volume and real-time data ingestion.

Follow the steps below to learn how to use Data Streaming with Splunk.

Generating a Splunk HEC Token

Step 1: Enable the HTTP Event Collector

Before creating a token, ensure the Global Settings on your Splunk account allow data ingestion.

1. Log in to your Splunk instance as an Administrator.

2. Go to Settings > Data Inputs.

3. Click on HTTP Event Collector.

4. Click Global Settings.

5. Set All Tokens to Enabled.

6. Click Save.

Step 2: Create a new token

1. In the HTTP Event Collector screen, click New Token.

   • Name: Give it a clear name (e.g., Digibee_Data_Streaming).

   • Source name override: Leave blank (unless you have specific routing needs).

2. Click Next.

Input Settings

• Source Type: Select _json.

• App Context: Usually Search & Reporting.

• Index: Choose the index where you want the data to be stored. For example: "your_index_name" If you don't select an index here, the data might not be searchable later.

• Click Review, then Submit.

Splunk will display a long alphanumeric string called Token Value. For example: a1b2c3d4-e5f6-7890-abcd-ef1234567890. Copy the token value to paste it on the Digibee Integration Platform.

Configure the Digibee Integration Platform

1. On the Digibee Platform, click Settings on the top right corner of the page.

2. In the Settings menu, click Data streaming.

3. Click Create.

4. On Vendor, select Splunk.

5. On Streaming type, select Log or Completed Executions.

6. Configure the connections details:

   1. Endpoint URL: Full URL of Splunk HEC. For example: https://prd-p-1234.splunkcloud.com:8088/services/collector, where 8088 represents the port used by Splunk.

   2. Authentication Token: The Token Value you just copied from Splunk.

7. Click Save.

Visualize the Data Streaming on Splunk

Follow the steps below to learn how to visualize data streaming on Splunk:

Step 1: Find your Data

1. Open the Search & Reporting app in Splunk.

2. In the search bar, type a query to find your Digibee logs. If you followed the previous setup, use the name of the index you specified:

index=main" sourcetype="_json"

1. Click the magnifying glass icon to run the search. You should see a list of JSON events.

Step 2: Extract Fields with spath

To turn a specific log field (like status or executionTime) into a chart, Splunk uses the spath command.

• Example: To see a count of executions by status, run:

index="main" | spath path=status | stats count by status

Step 3: Create the Visualization

1. Once you run a search that returns statistics, click the Visualization tab.

2. By default, Splunk might show a Bar Chart. You can click the chart type icon on the left to change it to a Pie Chart, Line Chart, or Single Value.

3. Adjust the Format settings to change colors or labels.

Step 4: Save to a Dashboard

To keep these visualizations visible:

1. Click Save As (top right of the search screen).

2. Select Dashboard Panel.

   • Dashboard Title: Give it a name like Digibee Integration Monitoring.

   • Panel Title: Name this specific chart (e.g., Executions by Status).

3. Click Save, then View Dashboard.

After applying the new configuration, it may take a few seconds for the data to become available due to the update processing time. This interval may vary, but it typically doesn’t exceed a few minutes. If the data are not visible after 10 minutes, please contact the Digibee Support team.