OpenTelemetry

The start scripts demonstrate how the Intelligence Hub can be used with the OpenTelemetry Java Agent. Modification may be required for deploying in a Production environment.

Using OpenTelemetry with HighByte Intelligence Hub

The HighByte Intelligence Hub supports running with the OpenTelemetry Java Agent out-of-the-box. Intelligence Hub metrics, logs, and JVM exporters are currently supported.

OpenTelemetry is enabled by setting the OTEL_COLLECTOR_ENDPOINT environment variable before starting the hub. You can also override or augment agent properties using OTEL_AGENT_PROPERTIES_EXTRA. Exported metrics and logs are automatically tagged with resource attributes identifying the host and instance they came from.

Environment Variable	Description	Example Value
`OTEL_COLLECTOR_ENDPOINT`	Enables the OpenTelemetry Java Agent and sets the collector endpoint	`http://<collector-host>:4318`
`OTEL_AGENT_PROPERTIES_EXTRA`	Optional. Override or augment OTel properties passed to the Java Agent	`-Dotel.exporter.otlp.protocol=grpc`

Once the variable is set, start the hub using the appropriate script for your platform.

Windows

.\start-windows.bat

Linux

./start-linux.sh

macOS

./start-macos.sh

Docker

Pass the environment variables using -e flags on the docker run command. Because there is no local collector running inside the container, a remote endpoint is required.

docker run -p 45245:45245 -p 1885:1885 -p 8885:8885 --name highbyte \
  --mount source=hb_vol,target=/usr/local/highbyte/appData \
  -e OTEL_COLLECTOR_ENDPOINT=http://<collector-host>:4318 \
  highbyte:X.X.X

Resource Attributes

All exported metrics and logs are tagged with OpenTelemetry resource attributes that identify where the telemetry originated. This is especially useful when running more than one Intelligence Hub instance (for example, in High Availability or Kubernetes), so you can tell which host or pod each metric came from.

The following attributes are added by default:

Attribute	Description
`service.name`	Always `HighByteIntelligenceHub`.
`service.instance.id`	Unique identifier for this instance. Uses `HA_NODE_ID` when running in High Availability mode, otherwise the hostname.
`host.name`	Host name, detected by the OpenTelemetry Java Agent.
`host.id`	Machine identifier, detected by the OpenTelemetry Java Agent.
`container.name`	Defaults to the hostname. Set the `CONTAINER_NAME` environment variable to override.

Kubernetes Metadata

When running in Kubernetes, additional k8s.* attributes can be included by exposing the following environment variables to the container,typically via the Kubernetes Downward API . Each attribute is added only when its environment variable is set.

Environment Variable	Resource Attribute	Downward API Field
`K8S_NAMESPACE_NAME`	`k8s.namespace.name`	`metadata.namespace`
`K8S_POD_NAME`	`k8s.pod.name`	`metadata.name`
`K8S_POD_UID`	`k8s.pod.uid`	`metadata.uid`
`K8S_POD_IP`	`k8s.pod.ip`	`status.podIP`
`K8S_NODE_NAME`	`k8s.node.name`	`spec.nodeName`
`K8S_CLUSTER_NAME`	`k8s.cluster.name`	Set manually
`K8S_CONTAINER_NAME`	`k8s.container.name`	Set manually
`K8S_DEPLOYMENT_NAME`	`k8s.deployment.name`	Set manually (e.g. from a label)
`K8S_STATEFULSET_NAME`	`k8s.statefulset.name`	Set manually (e.g. from a label)
`K8S_DAEMONSET_NAME`	`k8s.daemonset.name`	Set manually (e.g. from a label)

Example pod spec wiring the Downward API:

env:
  - name: K8S_NAMESPACE_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
  - name: K8S_POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: K8S_POD_UID
    valueFrom: { fieldRef: { fieldPath: metadata.uid } }
  - name: K8S_POD_IP
    valueFrom: { fieldRef: { fieldPath: status.podIP } }
  - name: K8S_NODE_NAME
    valueFrom: { fieldRef: { fieldPath: spec.nodeName } }

Application Logs

All INFO, WARN, and ERROR event log messages are exported to OpenTelemetry. Logs are exported with the following attributes:

Attribute Name	Description
highbyte.hub.eventlog.source	Name of the component generating the event log message.
highbyte.hub.eventlog.type	Corresponding to the HighByte message type; INFO, WARN, and ERROR

Connection Metrics

The following metrics are exported for monitoring connections in HighByte Intelligence Hub:

Read Metrics

Metric Name	Type	Description	Attributes
`highbyte.hub.connection.stats.read.pending.size`	Histogram	Number of pending reads for a connection across all inputs	`highbyte.hub.connection.name`
`highbyte.hub.connection.stats.read.count`	Counter	Total number of reads executed for a connection’s input	`highbyte.hub.connection.name` `highbyte.hub.connection.input.name`
`highbyte.hub.connection.stats.read.error.count`	Counter	Total number of read errors for a connection’s input	`highbyte.hub.connection.name` `highbyte.hub.connection.input.name`
`highbyte.hub.connection.stats.read.duration`	Histogram	Read duration for a connection’s input in seconds	`highbyte.hub.connection.name` `highbyte.hub.connection.input.name`

Write Metrics

Metric Name	Type	Description	Attributes
`highbyte.hub.connection.stats.write.pending.size`	Histogram	Number of pending writes for a connection across all outputs	`highbyte.hub.connection.name`
`highbyte.hub.connection.stats.write.dropped.count`	Counter	Number of dropped writes for a connection across all outputs	`highbyte.hub.connection.name`
`highbyte.hub.connection.stats.write.count`	Counter	Total number of writes for a connection’s output	`highbyte.hub.connection.name` `highbyte.hub.connection.output.name`
`highbyte.hub.connection.stats.write.error.count`	Counter	Total number of write errors for a connection’s output	`highbyte.hub.connection.name` `highbyte.hub.connection.output.name`
`highbyte.hub.connection.stats.write.duration`	Histogram	Write duration for a connection’s output in seconds	`highbyte.hub.connection.name` `highbyte.hub.connection.output.name`

Pipeline Metrics

The following metrics are exported for monitoring pipelines in HighByte Intelligence Hub:

Trigger Metrics

Metric Name	Type	Description	Attributes
`highbyte.hub.pipeline.stats.trigger.count`	Counter	Total number of triggers for a pipeline’s trigger	`pipeline.name` `pipeline.trigger.name`
`highbyte.hub.pipeline.stats.trigger.error.count`	Counter	Total number of errors for a pipeline’s trigger	`pipeline.name` `pipeline.trigger.name`

Execution Metrics

Metric Name	Type	Description	Attributes
`highbyte.hub.pipeline.stats.run.count`	Counter	Total number of runs for a pipeline	`pipeline.name`
`highbyte.hub.pipeline.stats.write.pending.size`	Histogram	Number of queued writes for a pipeline	`pipeline.name`
`highbyte.hub.pipeline.stats.wait.duration`	Histogram	Wait duration for a pipeline in seconds	`pipeline.name`
`highbyte.hub.pipeline.stats.execution.duration`	Histogram	Execution duration for a pipeline in seconds	`pipeline.name`
`highbyte.hub.pipeline.stats.error.count`	Counter	Total number of errors for a pipeline’s stage	`pipeline.name` `pipeline.stage.name`

See JVM Metrics for more information about JVM metrics generated by the Open Telemetry Java Agent.