High Availability

Overview

High Availability (HA) mode allows the Intelligence Hub to run across multiple nodes, ensuring continuous operation and failover. Exactly one node is the primary at a time; the rest stand by, ready to take over if the primary node fails. This is managed through heartbeats and file synchronization.

How the standby nodes behave while they wait is set for the whole cluster with the Standby Mode. A passive standby, the default, sits idle and processes no data until it is promoted. An active standby runs its pipelines alongside the primary and holds its writes until it is promoted, which shortens the gap in data on failover.

All HA coordination is managed through a shared database, either PostgreSQL or Microsoft SQL Server. See Database. Nodes do not communicate directly with each other; instead, each node interacts independently with the database for heartbeat management, leadership election, and data synchronization.

  • Heartbeats: Each node periodically signals its status to the database. If the primary node stops sending heartbeats, a standby node automatically takes over.
  • File Synchronization: Configuration files in appData, which includes Connection and Pipeline State Extension caches, are regularly synced through the database to ensure a standby node can seamlessly assume the primary role.

The primary and standby hubs can be configured using either the Command Line or Environment Variables.

Command Line

Below are details on the command line arguments to configure and start HA. The primary commands are as follows:

  • create: Initializes the failover service, syncs initial configuration and state files, and, unless -p false is set or the standby mode is active, sets the node ID as the preferred primary node.
    • The preferred primary node is always selected as primary when available. This is the default in passive standby mode and can be turned off. An active cluster has no preferred primary. See Preferred Primary.
  • start: Starts a node in HA mode, joining the failover group.
  • help: Displays usage information.

The basic command syntax looks as follows, with the arguments detailed below. It’s best to run these commands from the runtime directory.

Windows:

java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain [options] <create|start|help>

Linux/macOS:

java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain [options] <create|start|help>
OptionDescriptionExample Value
-j, --jdbcURIJDBC URI of the HA database. The URI prefix selects the database type. Wrap the URI in double quotes, since shells treat ; and & as command separators. See Database."jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
"jdbc:sqlserver://<db-host>:1433;databaseName=<db-name>;user=<username>;password=<password>"
env:URI
-n, --nodeIdUnique node ID for the HA instance
-e, --heartbeatExpirationSet the Heartbeat expiration period. Defaults to 10s. Minimum value of 2. create only-e 30
env:HBE
-x, --syncExcludeComma-separated list of artifacts that are not synchronized between nodes: project, variables, secrets, state. Defaults to synchronizing everything. See Sync Exclusions. create only-x secrets,variables
env:HA_SYNC_EXCLUDE
-s, --standbyModeHow every node behaves while it is standing by: passive or active. Case-insensitive. Defaults to passive. See Standby Modes. create only-s active
env:HA_STANDBY_MODE
-p, --preferredPrimaryWhether the node running create is the preferred primary: true or false. Case-insensitive. Defaults to true in passive standby mode and false in active, which does not allow true. See Preferred Primary. create only-p false
env:HA_PREFERRED_PRIMARY

Command Line Setup

To set up a primary and standby hub using the command line, follow the steps below.

  1. Install or have available a supported database server.

  2. Install the primary and standby hubs.

  3. On the primary hub, run the create command to initialize the database tables used for synchronization and, by default, assign this node as the preferred primary.

    Windows:

    java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain create -n <primary-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
    

    Linux/macOS:

    java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain create -n <primary-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
    
  4. Copy the intelligencehub-certificatestore.pkcs12 file from the appData of the primary hub to the appData of the standby hub so the standby hub can decrypt the secrets in the synchronized configuration.

  5. Start the primary hub with the start command, using the same node ID that was used in create.

    Windows:

    java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain start -n <primary-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
    

    Linux/macOS:

    java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain start -n <primary-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
    
  6. Start the standby hub with the start command, using a new unique node ID.

    Windows:

    java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain start -n <standby-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
    

    Linux/macOS:

    java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain start -n <standby-name> -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"
    

Environment Variables

HA mode is enabled by setting three environment variables before starting the hub.

Where the variables are set depends on how the hub runs. See Windows Service, Linux systemd Service, or Docker for specifics. The examples in those sections configure the standby hub. For the primary hub, use its own node ID and HA_INIT until the cluster is initialized.

Environment VariableDescription
START_MODEControls how the node joins the HA cluster.
HA_INIT: Initializes a new HA cluster and starts the node as the primary. Set this on the primary hub only.
HA_JOIN: Joins an existing HA cluster. Set this on the standby hub, and on the primary hub once the cluster is initialized.
HA_NODE_IDUnique node ID for this HA instance
HA_DATABASE_URIJDBC URI of the HA database. The URI prefix selects the database type. See Database.
HA_POSTGRES_URIDeprecated. The name used for HA_DATABASE_URI before version 4.6. Still read when HA_DATABASE_URI is not set, so existing configurations keep working, but move the value to HA_DATABASE_URI.
HA_SYNC_EXCLUDEOptional. Comma-separated list of artifacts that are not synchronized between nodes: project, variables, secrets, state. For example, HA_SYNC_EXCLUDE=secrets,variables. Defaults to synchronizing everything. Read only when START_MODE is HA_INIT, and ignored with HA_JOIN. See Sync Exclusions.
HA_STANDBY_MODEOptional. How every node behaves while it is standing by: passive or active. Defaults to passive. Read only when START_MODE is HA_INIT, and ignored with HA_JOIN. See Standby Modes.
HA_PREFERRED_PRIMARYOptional. Whether this node is the preferred primary: true or false. Defaults to true in passive standby mode and false in active, which does not allow true. Read only when START_MODE is HA_INIT, and ignored with HA_JOIN. See Preferred Primary.

Environment Variables Setup

To set up a primary and standby hub using environment variables, follow the steps below.

  1. Install or have available a supported database server.

  2. Install the primary and standby hubs.

  3. On the primary and standby hubs, set the environment variables.

  4. Start the primary hub to initialize the database tables used for synchronization, push the initial configuration and state, assign this node as the preferred primary by default, and then run the node.

  5. Change START_MODE to HA_JOIN on the primary hub so that a later restart does not reinitialize the cluster.

  6. Copy the intelligencehub-certificatestore.pkcs12 file from the appData of the primary hub to the appData of the standby hub so the standby hub can decrypt the secrets in the synchronized configuration. The certificates in this file are synchronized, but the mechanisms used to encrypt and decrypt secrets are not.

  7. Start the standby hub.

Windows Service

The HighByte Intelligence Hub Runtime service starts the hub by calling start-windows.bat, so the environment variables must be available to the service process.

Define the three variables as system variables under System Properties, Environment Variables, or add an env element for each variable inside the <service> element of runtime\service\service-installer-windows.xml, which limits the variables to the service:

<env name="START_MODE" value="HA_JOIN"/>
<env name="HA_NODE_ID" value="<standby-name>"/>
<env name="HA_DATABASE_URI" value="jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&amp;password=<password>"/>

Restart the HighByte Intelligence Hub Runtime service to apply changes to the env elements.

Linux systemd Service

Add the environment variables inline to the [Service] section of the unit file:

Environment=START_MODE=HA_JOIN
Environment=HA_NODE_ID=<standby-name>
Environment=HA_DATABASE_URI=jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>

Alternatively, point to a separate file that holds them, which can be permission restricted:

EnvironmentFile=/etc/highbyte/ha.env

The /etc/highbyte/ha.env file contains one variable per line:

START_MODE=HA_JOIN
HA_NODE_ID=<standby-name>
HA_DATABASE_URI=jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>

Reload the unit and restart the service to apply the change:

sudo systemctl daemon-reload
sudo systemctl restart highbyte.service

Docker

When running the Intelligence Hub in Docker, set the environment variables in the container configuration.

docker run -p 45245:45245 -p 1885:1885 -p 8885:8885 -e ACCEPT_EULA=Y --name highbyte \
  --mount source=hb_vol,target=/usr/local/highbyte/appData \
  --restart=unless-stopped \
  -e START_MODE=HA_JOIN \
  -e HA_NODE_ID=<standby-name> \
  -e HA_DATABASE_URI="jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>" \
  highbyte:X.X.X

The variables are part of the container configuration, so they apply every time the container starts. Include a restart policy such as --restart=unless-stopped so the container comes back after a failure or a host reboot.

To copy intelligencehub-certificatestore.pkcs12 out of the primary hub’s container, run the following on the primary host:

docker cp highbyte:/usr/local/highbyte/appData/intelligencehub-certificatestore.pkcs12 <folder>

To copy the file into a new volume for the standby hub, run the following on the standby host before its first start. It runs the copy with the hub image so the file is owned by the user the container runs as:

docker run --rm --entrypoint cp \
  --mount source=hb_vol,target=/usr/local/highbyte/appData \
  --mount type=bind,source=<folder>,target=/src \
  highbyte:X.X.X /src/intelligencehub-certificatestore.pkcs12 /usr/local/highbyte/appData/

Standby Modes

The standby mode controls what a node does while it is standing by. It applies to the whole cluster, so every node behaves the same way while it is not the primary.

passiveactive
Data processing while standing byNone. Connections are closed and pipelines do not run.Full. Connections are open and pipelines run, but writes to targets are held rather than sent.
Gap in data on failoverThe whole failover window, plus the time it takes the promoted node to connect and start its pipelines.None for a failover that completes within the hold window, since the promoted node replays the writes it held. A promotion delayed beyond that window still loses the oldest writes.
Load on sourcesOnly the primary reads from sources.Every node reads from sources, so a source serves one set of requests per node.
Duplicate writesNone.Expected around a failover. See Write Buffering.
Web interface and REST API while standing byLimited to health checks and /v1/info. Every other request returns 503.Available.
Preferred primaryDefaults to the node that ran create.Not allowed. See Preferred Primary.

Set the mode with the -s option on the create command, or with HA_STANDBY_MODE and START_MODE=HA_INIT:

Windows:

java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain create -n <primary-name> -s active -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"

Linux/macOS:

java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain create -n <primary-name> -s active -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"

The same selection using environment variables:

START_MODE=HA_INIT
HA_NODE_ID=<primary-name>
HA_DATABASE_URI=jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>
HA_STANDBY_MODE=active

Write Buffering

An active standby runs its pipelines, but it does not write to targets while it is standing by. Each connection holds the writes it would have sent, and when the node is promoted it sends the held writes before the new ones, so the writes the failed primary never got to send are delivered by the node that takes over.

  • How long writes are held: twice the Heartbeat Expiration Period, which is 20 seconds with the default period of 10 seconds. This covers a failover of typical length, but it is a fixed window rather than a promise: a promotion delayed longer than the window, such as when no standby is ready to take over, still leaves a gap. Time spent inside a promotion does not count against the window.
  • Cleanup: held writes older than the window are deleted every time a connection holds another write, so the buffer stays bounded by the rate of writes rather than growing with uptime.
  • Where they are held: in each connection’s write store file, in the store and forward directory, whether or not Store and Forward is enabled for the connection and separately from its pending writes. They are discarded when the node starts, when it goes back to standing by, and once replayed.
  • On promotion: the node stops aging out held writes, performs its final pull, starts its data plane, then each connection replays the writes held within the window, oldest first, ahead of any new writes.

Customizing How Long Writes Are Held

The window is always twice the Heartbeat Expiration Period, so it is adjusted with the -e option on the create command rather than with a setting of its own.

Heartbeat Expiration PeriodHeartbeat intervalTypical failover windowWrites held for
2 seconds (minimum)1 second1–3 seconds4 seconds
10 seconds (default)5 seconds5–15 seconds20 seconds
30 seconds15 seconds15–45 seconds60 seconds

A shorter period fails over sooner and holds fewer writes, which also means fewer duplicates, at the cost of being quicker to treat a slow or briefly unreachable primary as failed. A longer period tolerates more delay but widens both the gap the buffer has to cover and the number of writes held in memory and on disk.

Readiness in Active Mode

In an active cluster a node can only acquire the lease once its data plane is ready, so a node that is still starting or reloading configuration does not become the primary and lose the data it would have been responsible for. Readiness gates acquiring the role, not keeping it: a node that already holds a live lease goes on renewing it while it is not ready, so a primary that is reloading stays the primary. If the lease expires and no node is ready, the cluster has no primary until one becomes ready. A node cannot take back even its own expired lease until it reports ready.

A ready active standby therefore returns 200 from /v2/readyz even though it is not the primary. See Health Checks for routing traffic in this mode.

Preferred Primary

The preferred primary is the node that takes the primary role back whenever it is running, even if another node currently holds it. It is set on the node that runs create, with -p or HA_PREFERRED_PRIMARY, and stored with the HA instance.

SettingBehavior
-p trueThe node that ran create is the preferred primary. When it is running, it holds the primary role, and it takes the role back on its next heartbeat after a restart. This is the default in passive standby mode.
-p falseThe cluster has no preferred primary. The primary role stays with whichever node holds it until that node fails or shuts down, and a recovered node stands by instead of taking the role back. This is the default in active standby mode, which does not allow -p true.

Setting -p false avoids a second failover when the original primary comes back: the first failover is unavoidable, but the failback it would otherwise perform is not. In active standby mode there is no preferred primary at all, since each failover replays held writes and so produces another set of duplicates.

To create a passive cluster with no preferred primary, add -s passive -p false to the create command shown in Standby Modes.

Failover Behavior

Heartbeat and Lease Management

The HA system uses a heartbeat mechanism to monitor node health and manage leadership:

  • Heartbeat interval: Every 5 seconds. If a manual Heartbeat Expiration Period is set, the interval is half that value, rounded down to the nearest second.
  • Heartbeat expiration period: 10 seconds
  • Typical failover window: 5–15 seconds by default. With a custom Heartbeat Expiration Period, the window ranges from the heartbeat interval to the heartbeat interval plus the expiration period.
    • Best case: ~heartbeat interval (standby detects failure on its next heartbeat)
    • Worst case: ~heartbeat interval + expiration period (standby waits for next heartbeat after primary heartbeat expires)

The primary node renews its heartbeat in the database at the heartbeat interval (5 seconds by default). If a primary node fails to check in, the heartbeat expires after the configured Heartbeat Expiration Period (10 seconds by default). Standby nodes also send heartbeats at the same interval, and the first standby to send a heartbeat after the primary’s heartbeat expires is elected as the new primary by the database.

File Synchronization

The following files are maintained and synchronized across all nodes in the HA cluster. To keep specific artifacts out of the synchronization, see Sync Exclusions.

Configuration Files:

  • intelligencehub-configuration.json - Project configuration
  • intelligencehub-settings.json - System settings
  • intelligencehub-systemvariables.json - System variables
  • intelligencehub-remoteconfig.json - Remote configuration settings
  • intelligencehub-users.json - User accounts, roles, and API keys
  • intelligencehub-secrets.json - Encrypted secrets
  • intelligencehub-certificatestore.pkcs12 - Only certificates and private keys are maintained. The mechanisms used for encrypting and decrypting secrets will remain on the local instance. To decrypt secrets across multiple HA instances, copy the pkcs12 file of the primary to each instance.
  • intelligencehub-identityproviders.json - Identity provider configurations
  • intelligencehub.license - License file

Data Stores:

  • intelligencehub-cache.db - Cache store
  • intelligencehub-state.db - State store

Configuration and data stores are synchronized between nodes to ensure a standby can seamlessly take over:

  • Primary mode sync interval: Half the Heartbeat Expiration Period, rounded down to the nearest second (5 seconds by default). Pushes changes to the database.
  • Standby mode sync interval: Six times the Heartbeat Expiration Period (60 seconds by default). Pulls changes from the database.
  • Sync operations:
    • Deployment files are compared by hash before transfer. Files are synced when there is a change.
    • Cache and state stores only push the most recent 10,000 values per sync attempt
  • Data retention:
    • Data from the Data Stores are retained in the database for up to 24 hours
    • Each transaction row is tracked relative to its upload time
    • Transactions that haven’t been updated and are older than 24 hours are automatically deleted
    • To miss data, more than 10,000 unique transactions per 5 seconds would need to occur in the cache or state stores

Sync Exclusions

By default, every artifact listed above is synchronized. Use the -x / --syncExclude option on the create command, or the HA_SYNC_EXCLUDE environment variable with START_MODE=HA_INIT, to keep specific artifacts out of the synchronization. This is useful when each node needs to hold values of its own, such as a node-specific set of system variables or node-specific credentials, while still sharing the rest of the configuration.

The value is a comma-separated, case-insensitive list of the following artifacts:

ArtifactWhat is excluded
projectThe project in intelligencehub-configuration.json: models, connections, inputs, outputs, flows, pipelines, and the rest of the project configuration. This includes any credential entered directly into a connection field, which is encrypted in place within the connection’s settings. The network settings stored alongside the project are always synchronized, even when project is excluded.
variablesSystem variables in intelligencehub-systemvariables.json
secretsThe named secrets in intelligencehub-secrets.json. This covers only secrets defined under Secrets and referenced by the project, not credentials entered directly into a connection field.
stateThe cache store and state store (intelligencehub-cache.db and intelligencehub-state.db)

For example, to keep the named secrets and system variables local to each node while the project is shared:

Windows:

java -cp "intelligencehub-runtime.jar;lib/*" com.highbyte.intelligencehub.runtime.HAMain create -n <primary-name> -x secrets,variables -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"

Linux/macOS:

java -cp "intelligencehub-runtime.jar:lib/*" com.highbyte.intelligencehub.runtime.HAMain create -n <primary-name> -x secrets,variables -j "jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>"

The same selection using the environment variable:

START_MODE=HA_INIT
HA_NODE_ID=<primary-name>
HA_DATABASE_URI=jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>
HA_SYNC_EXCLUDE=secrets,variables

Promotion Process

When a Standby Becomes Primary

  1. Pre-promotion sync: Standby performs one final pull to ensure data consistency
  2. Runtime startup: Starts the main runtime with synchronized data. An active standby is already running it, so only the writes are released
  3. Write replay: In active standby mode, each connection sends the writes it held within the hold window, oldest first, before any new writes. See Write Buffering
  4. Heartbeat confirmation: New primary renews heartbeat at the heartbeat interval

When a Primary Node Fails

  1. Heartbeat expiration: After the Heartbeat Expiration Period without renewal, the heartbeat expires
  2. Leadership acquisition: Next heartbeat from a standby (within one heartbeat interval) acquires leadership. In active standby mode, only a node that reports ready can acquire leadership
  3. Automatic promotion: Standby node detects acquisition and promotes to primary
  4. Service continuity: Total downtime falls within the typical failover window. In active standby mode, the promoted node replays its held writes. See Write Buffering

When the Preferred Primary Comes Back Online

A cluster with a preferred primary fails back to that node:

  • When the preferred primary node comes online, it will take over from the current primary
  • This happens during the next heartbeat cycle (within one heartbeat interval)
  • The transition follows the same promotion process (sync → promote → start)

A cluster created with -p false, which is the default in active standby mode, has no preferred primary. A recovered node stands by, and the primary role stays with the node that holds it.

Graceful Shutdown

Primary Node Shutdown

  1. Leadership release: Immediately expires the heartbeat (sets expiration to the Heartbeat Expiration Period in the past)
  2. Runtime stop: Cleanly shuts down the active runtime
  3. Connection cleanup: Closes HA database connection
  4. Fast failover: Because the heartbeat expires immediately, standby nodes can acquire leadership on their next heartbeat (within one heartbeat interval)

Database

The HA database can be PostgreSQL or Microsoft SQL Server. The prefix of the JDBC URI passed with -j or HA_DATABASE_URI selects which one is used, and every node in a cluster must use the same database.

DatabaseMinimum versionJDBC URI
PostgreSQL10jdbc:postgresql://<db-host>:5432/<db-name>?user=<username>&password=<password>
Microsoft SQL Server2017jdbc:sqlserver://<db-host>:1433;databaseName=<db-name>;user=<username>;password=<password>

The minimum version is the oldest release HA runs on. Older releases lack SQL that High Availability relies on.

The database itself must already exist. create builds an orchestration schema inside it, with the tables and routines HA uses, so the database user needs permission to create and drop a schema and the tables, sequences, functions, or stored procedures within it. After create, the user needs permission to read and write those tables and run those routines.

Microsoft SQL Server

Recent versions of the Microsoft JDBC driver encrypt the connection by default and verify the server’s certificate. If the server’s certificate is not trusted by the Java runtime of the hub, the connection fails. Either install a trusted certificate on the server, or add trustServerCertificate=true to the URI to skip the check:

jdbc:sqlserver://<db-host>:1433;databaseName=<db-name>;user=<username>;password=<password>;encrypt=true;trustServerCertificate=true

Node IDs are compared exactly, including case, as they are on PostgreSQL, so node1 and NODE1 are different nodes. Leading and trailing spaces are removed from node IDs on every database.

Windows Authentication

To connect with Windows authentication instead of a SQL Server login, set integratedSecurity=true in the URI and leave out the user name and password:

jdbc:sqlserver://<db-host>:1433;databaseName=<db-name>;integratedSecurity=true;encrypt=true

Windows authentication requires the Microsoft authentication DLL in the runtime directory of the hub. Follow steps 3 through 6 of MSSQL Windows Authentication to download the DLL that matches the hub’s JDBC driver and copy it into place. The DLL is not included with the hub.