Tutorial: File Connections

Jump to Section

What Does This Tutorial Cover?

This tutorial will walk you through the basics of working within File based Connections. A Connection represents a pathway to either a data source or a target, serving as the starting point for all use cases.

Tutorial Prerequisites
Tutorial Preparation
Tutorial Instructions
Tutorial Summary

Tutorial Prerequisites

You have completed the initial Installation of the Intelligence Hub
You have applied your Intelligence Hub license to your environment
It is recommended to read the Getting Started Series before completing this tutorial exercise
It is recommended to complete “Tutorial: Connections”

Tutorial Preparation

Enable Intelligence Hub MQTT broker
- In the left-hand navigation panel, navigate to Manage, and click Settings
- Under the MQTT Broker section enable the broker, if ports 1885 and 1886 are being utilized on your Intelligence Hub server, update to ports of your choosing, otherwise accept the defaults and click save

Import the required Connections

In the left-hand navigation panel, navigate to Manage, and click Project
Within the Import screen, ensure Full Project is off (otherwise your existing project will be overwritten)
Change the Import Type to JSON and paste in the following with the Project box and click the import button

{
	"productInfo": {
		"company": "HighByte",
		"product": "IntelligenceHub",
		"version": "3.2.0",
		"build": "2023.8.24.982",
		"stage": "Alpha"
	},
	"project": {
		"version": 5,
		"connections": [
			{
				"name": "Tutorial_File",
				"uri": "file://Tutorial_File",
				"tags": [
					"Share"
				],
				"writes": {
					"flattenModeledValues": false
				},
				"storeForward": {
					"enabled": false,
					"maxEntries": 100,
					"waitOnFailureInterval": {
						"duration": 1,
						"units": "Seconds"
					}
				},
				"settings": {
					"communicationProtocol": {
						"type": "FILE_PROTOCOL"
					},
					"directory": "/files",
					"processedDirectory": "/files/processed",
					"errorDirectory": "/files/error"
				}
			},
			{
				"name": "Tutorial_MQTT",
				"uri": "mqtt://0.0.0.0:1885",
				"tags": [
					"Share"
				],
				"writes": {
					"flattenModeledValues": false
				},
				"storeForward": {
					"enabled": false,
					"maxEntries": 100,
					"waitOnFailureInterval": {
						"duration": 1,
						"units": "Seconds"
					}
				},
				"settings": {
					"connectionTimeoutSeconds": 10,
					"keepAliveSeconds": 60,
					"requestTimeoutMS": 5000,
					"cleanSession": true,
					"ssl": false,
					"redundantBrokers": [],
					"inputDiscovery": ""
				}
			}
		],
		"inputs": [
			{
				"name": "Tutorial_Read_Wildcard",
				"connection": "Tutorial_File",
				"type": "file",
				"qualifier": {
					"fileName": "(.+)",
					"includeMetadata": false
				},
				"cacheLifetime": {
					"enabled": false
				}
			}
		],
		"outputs": [],
		"modeling": {
			"models": [],
			"instances": []
		},
		"flows": [],
		"conditions": [],
		"functions": [],
		"tags": [],
		"pipelines": []
	},
	"network": {
		"groups": [],
		"hubs": []
	}
}

Expand Collapse

Update the imported Connections as required
- Navigate to Configure and click Connections, Click Tutorial_MQTT and Update the MQTT settings as required based on the prior preparation step #1
- Navigate to Configure and click Connections, Click Tutorial_File and Update the File Directory to a local directory on the Intelligence Hub server. This tutorial will not utilize the processed or error directory.
Import the source Files
- Right click and save the following three images to the local File directory from the previous step
- Save the files as cnc_1001.jpg, cnc_1002.jpg and cnc_1003.jpg

Setup UNS Client
- In the left-hand navigation panel, navigate to Tools and right click UNS client and open Link in New Tab
- Enter login information
- For Connection select Tutorial_MQTT
- For Subscribed Topics remove the default wildcard entry # and Subscribe to Topic Tutorial/#
- Click Add
- Click Connect and confirm UNS client says “Connected to Tutorial_MQTT”
- Return to the previous tab

Tutorial Instructions

Login to your Intelligence Hub environment
Review the source data
- Navigate to Connections and select Tutorial_File
- Navigate to Inputs
- Select Tutorial_Read_Wildcard
  - Take note of the file name structure, this is the default new input for File/CSV connections. Consider it a wildcard that will read the whole directory without any additional logic
- Within the right hand References Panel, set type to “Input”
- Set Connections to Tutorial_File
- Find the Input you opened and take note of the object’s shape
- Now enable “Include Metadata” within the Input details
- Click Test Input and review the results from the Results and Reference panels
- Take note we now have the File Name available as additional metadata and the object shape has changed within the Reference panel
Create a new Input
- Take note the existing Tutorial_Read_Wildcard input returns the first file in the directory alphabetically
- Click clone in the upper right of the Input view
- Enter Tutorial_Read_ByFileType as the new Input name
- Update the File Name to: (.+jpg)
- Click submit
- Click Test Input and review the results, you should notice the _filename and value have changed based on the new File Name expression
  - Note if you only have the 3 source image files in this directory, the input will return the same value.
Create an Output for our File
- Navigate to Connections and select Tutorial_MQTT
- Navigate to Outputs
- Click New Output and enter the following and click create
```
Name: Tutorial_File_Output
Topic: Tutorial/File
Retain: On
```
Create a flow to send the File to our UNS
- Navigate to Flows
- Click New Flow
- Enter “Tutorial_File_Flow” for name and click next
- Within the Sources & Target view, use the References Panel and set Type to Input
- Set Connections to Tutorial_File
- Drag your root object “Tutorial_Read_ByFileType” to the Sources panel
- Change Type to Output
- Set Connections to Tutorial_MQTT
- Drag your Tutorial_File_Output to the Targets panel
- Click next
- Change your interval to 10 seconds
- Click enable and click submit
- Review results in your UNS client

Typically a file use case involves multiple deloyments of the Intelligence Hub to securely move Files from one network to another. For the sake of simplicity we will simulate the consumption of a file from the UNS from a single deployment.

Read the File value from our UNS
- Navigate to Connections and select Tutorial_MQTT
- Click Inputs
- Click New Input and enter the following:
```
Name: Tutorial_File_Input
Topic: Tutorial/File
```
- Click save
- Test Input to review the data
  - Take note, the “value” attribute is the binary string of the File. We will need to ensure we specify this attribute in our File output.
Create our File output
- Navigate to Connections
- Select Tutorial_File
- Select Outputs
- Click New Output, enter the following and click Save:
```
Name: Tutorial_File_Output
File Name: Tutorial_fileoutput.jpg
Payload Reference: {{this.value}}
Base64 Decoding: On
```
Create a Flow to read the data from MQTT and write out a new file - note we will only output 1 of the 3 files
- Navigate to Flows
- Click New Flow
- Enter name as “Tutorial_FileReadFromUNS”
- Click next
- Within the Sources & Target view, use the References Panel and set Type to Input
- Set Connections to Tutorial_MQTT
- Drag your root object Tutorial_File_Input to the Sources panel
- Set Type to Output
- Set Connections to “Tutorial_File”
- Drag your object “Tutorial_File_Output” to the Target panel
- Click next
- Change your interval to 10 seconds
- Click enable and click submit
Review the results
- Navigate to your File directory to view the outputted file
- Alternatively you could create an input and see our newly created file
  - Navigate to Connections
  - Select Tutorial_File
  - Inputs and click New Input
  - Enter the following
    Name: Tutorial_FileReadFromMQTT File Name: Tutorial_fileoutput.jpg
  - Click save and test the input to view results

Tutorial Summary

This exercise focused on the basics of working with a File Connection. You should now be more familiar with how to Input and Output raw files using the Intelligence Hub. We covered the basic principals of using regex to build our file names in the File Connection. These same basic principals can be used with CSV and Parquet Connections. This exercise also covered how to move files to other target sources (Such as MQTT) and read and decode the files to additional locations.

Tutorial: Models and Instances