High Availability

Overview

High Availability (HA) mode allows the Intelligence Hub to run across multiple nodes, ensuring continuous operation and failover. Only one node is active at a time; standby nodes are ready to take over if the primary node fails. This is managed through heartbeats and file synchronization.

All HA coordination is managed through a PostgreSQL database. Nodes do not communicate directly with each other; instead, each node interacts independently with the PostgreSQL server for heartbeat management, leadership election, and data synchronization.

• Heartbeats: Each node periodically signals its status to the PostgreSQL database. If the primary node stops sending heartbeats, a standby node automatically takes over.
• File Synchronization: Configuration files in appData, which includes Connection and Pipeline State Extension caches, are regularly synced through the PostgreSQL database to ensure a standby node can seamlessly assume the primary role.

The primary and standby hubs can be configured using either the Command Line or Environment Variables.

Command Line

Below are details on the command line arguments to configure and start HA. The primary commands are as follows:

• create: Initializes the failover service, syncs initial configuration and state files, and sets the node ID as the preferred primary node.
  • The preferred primary node is always selected as primary when available.
• start: Starts a node in HA mode, joining the failover group.
• help: Displays usage information.

The basic command syntax looks as follows, with the arguments detailed below. It’s best to run these commands from the runtime directory.

Windows:

java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain [options] <create|start|help>

Linux/macOS:

java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain [options] <create|start|help>

Command Line Setup

To set up a primary and standby hub using the command line, follow the steps below.

1. Install or have available a PostgreSQL server.

2. Install the primary and standby hubs.

3. On the primary hub, run the create command to initialize the database tables used for synchronization and assign this node as the preferred primary.

   Windows:

   java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain create -n <primary-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
   

   Linux/macOS:

   java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain create -n <primary-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
   

4. Copy the intelligencehub-certificatestore.pkcs12 file from the appData of the primary hub to the appData of the standby hub so the standby hub can decrypt the secrets in the synchronized configuration.

5. Start the primary hub with the start command, using the same node ID that was used in create.

   Windows:

   java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain start -n <primary-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
   

   Linux/macOS:

   java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain start -n <primary-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
   

6. Start the standby hub with the start command, using a new unique node ID.

   Windows:

   java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain start -n <standby-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
   

   Linux/macOS:

   java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain start -n <standby-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
   

Environment Variables

HA mode is enabled by setting three environment variables before starting the hub.

Where the variables are set depends on how the hub runs. See Windows Service, Linux systemd Service, or Docker for specifics. The examples in those sections configure the standby hub. For the primary hub, use its own node ID and HA_INIT until the cluster is initialized.

Environment Variables Setup

To set up a primary and standby hub using environment variables, follow the steps below.

1. Install or have available a PostgreSQL server.

2. Install the primary and standby hubs.

3. On the primary and standby hubs, set the environment variables.

4. Start the primary hub to initialize the database tables used for synchronization, push the initial configuration and state, assign this node as the preferred primary, and then run the node.

5. Change START_MODE to HA_JOIN on the primary hub so that a later restart does not reinitialize the cluster.

6. Copy the intelligencehub-certificatestore.pkcs12 file from the appData of the primary hub to the appData of the standby hub so the standby hub can decrypt the secrets in the synchronized configuration. The certificates in this file are synchronized, but the mechanisms used to encrypt and decrypt secrets are not.

7. Start the standby hub.

Windows Service

The HighByte Intelligence Hub Runtime service starts the hub by calling start-windows.bat, so the environment variables must be available to the service process.

Define the three variables as system variables under System Properties, Environment Variables, or add an env element for each variable inside the <service> element of runtime\service\service-installer-windows.xml, which limits the variables to the service:

<env name="START_MODE" value="HA_JOIN"/>
<env name="HA_NODE_ID" value="<standby-name>"/>
<env name="HA_POSTGRES_URI" value="jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&amp;password=<password>"/>

Restart the HighByte Intelligence Hub Runtime service to apply changes to the env elements.

Linux systemd Service

Add the environment variables inline to the [Service] section of the unit file:

Environment=START_MODE=HA_JOIN
Environment=HA_NODE_ID=<standby-name>
Environment=HA_POSTGRES_URI=jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>

Alternatively, point to a separate file that holds them, which can be permission restricted:

EnvironmentFile=/etc/highbyte/ha.env

The /etc/highbyte/ha.env file contains one variable per line:

START_MODE=HA_JOIN
HA_NODE_ID=<standby-name>
HA_POSTGRES_URI=jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>

Reload the unit and restart the service to apply the change:

sudo systemctl daemon-reload
sudo systemctl restart highbyte.service

Docker

When running the Intelligence Hub in Docker, set the environment variables in the container configuration.

docker run -p 45245:45245 -p 1885:1885 -p 8885:8885 -e ACCEPT_EULA=Y --name highbyte \
  --mount source=hb_vol,target=/usr/local/highbyte/appData \
  --restart=unless-stopped \
  -e START_MODE=HA_JOIN \
  -e HA_NODE_ID=<standby-name> \
  -e HA_POSTGRES_URI="jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>" \
  highbyte:X.X.X

The variables are part of the container configuration, so they apply every time the container starts. Include a restart policy such as --restart=unless-stopped so the container comes back after a failure or a host reboot.

To copy intelligencehub-certificatestore.pkcs12 out of the primary hub’s container, run the following on the primary host:

docker cp highbyte:/usr/local/highbyte/appData/intelligencehub-certificatestore.pkcs12 <folder>

To copy the file into a new volume for the standby hub, run the following on the standby host before its first start. It runs the copy with the hub image so the file is owned by the user the container runs as:

docker run --rm --entrypoint cp \
  --mount source=hb_vol,target=/usr/local/highbyte/appData \
  --mount type=bind,source=<folder>,target=/src \
  highbyte:X.X.X /src/intelligencehub-certificatestore.pkcs12 /usr/local/highbyte/appData/

Failover Behavior

Heartbeat and Lease Management

The HA system uses a heartbeat mechanism to monitor node health and manage leadership:

• Heartbeat interval: Every 5 seconds. If a manual Heartbeat Expiration Period is set, the interval is half that value, rounded down to the nearest second.
• Heartbeat expiration period: 10 seconds
• Typical failover window: 5–15 seconds by default. With a custom Heartbeat Expiration Period, the window ranges from the heartbeat interval to the heartbeat interval plus the expiration period.
  • Best case: ~heartbeat interval (standby detects failure on its next heartbeat)
  • Worst case: ~heartbeat interval + expiration period (standby waits for next heartbeat after primary heartbeat expires)

The primary node renews its heartbeat in the PostgreSQL database at the heartbeat interval (5 seconds by default). If a primary node fails to check in, the heartbeat expires after the configured Heartbeat Expiration Period (10 seconds by default). Standby nodes also send heartbeats at the same interval, and the first standby to send a heartbeat after the primary’s heartbeat expires is elected as the new primary by the database.

File Synchronization

The following files are maintained and synchronized across all nodes in the HA cluster:

Configuration Files:

• intelligencehub-configuration.json - Project configuration
• intelligencehub-settings.json - System settings
• intelligencehub-systemvariables.json - System variables
• intelligencehub-remoteconfig.json - Remote configuration settings
• intelligencehub-users.json - User accounts, roles, and API keys
• intelligencehub-secrets.json - Encrypted secrets
• intelligencehub-certificatestore.pkcs12 - Only certificates and private keys are maintained. The mechanisms used for encrypting and decrypting secrets will remain on the local instance. To decrypt secrets across multiple HA instances, copy the pkcs12 file of the primary to each instance.
• intelligencehub-identityproviders.json - Identity provider configurations
• intelligencehub.license - License file

Data Stores:

• intelligencehub-cache.db - Cache store
• intelligencehub-state.db - State store

Configuration and data stores are synchronized between nodes to ensure a standby can seamlessly take over:

• Primary mode sync interval: Half the Heartbeat Expiration Period, rounded down to the nearest second (5 seconds by default). Pushes changes to the database.
• Standby mode sync interval: Six times the Heartbeat Expiration Period (60 seconds by default). Pulls changes from the database.
• Sync operations:
  • Deployment files are compared by hash before transfer. Files are synced when there is a change.
  • Cache and state stores only push the most recent 10,000 values per sync attempt
• Data retention:
  • Data from the Data Stores are retained on the PostgreSQL database for up to 24 hours
  • Each transaction row is tracked relative to its upload time
  • Transactions that haven’t been updated and are older than 24 hours are automatically deleted
  • To miss data, more than 10,000 unique transactions per 5 seconds would need to occur in the cache or state stores

Promotion Process

When a Standby Becomes Primary

1. Pre-promotion sync: Standby performs one final pull to ensure data consistency
2. Runtime startup: Starts the main runtime with synchronized data
3. Heartbeat confirmation: New primary renews heartbeat at the heartbeat interval

When a Primary Node Fails

1. Heartbeat expiration: After the Heartbeat Expiration Period without renewal, the heartbeat expires
2. Leadership acquisition: Next heartbeat from a standby (within one heartbeat interval) acquires leadership
3. Automatic promotion: Standby node detects acquisition and promotes to primary
4. Service continuity: Total downtime falls within the typical failover window

When the Preferred Primary Comes Back Online

The system supports setting a preferred primary node configuration:

• When the preferred primary node comes online, it will take over from the current primary
• This happens during the next heartbeat cycle (within one heartbeat interval)
• The transition follows the same promotion process (sync → promote → start)

Graceful Shutdown

Primary Node Shutdown

1. Leadership release: Immediately expires the heartbeat (sets expiration to the Heartbeat Expiration Period in the past)
2. Runtime stop: Cleanly shuts down the active runtime
3. Connection cleanup: Closes HA database connection
4. Fast failover: Because the heartbeat expires immediately, standby nodes can acquire leadership on their next heartbeat (within one heartbeat interval)

Database

Currently, only PostgreSQL is supported for the database.