Filter

Jump to Section

The Filter stage filters data from the event. This makes it easy to move parts of the event from the value.

For example, consider data with a SiteId attribute. SiteId should be in the MQTT topic but not in the payload. A filter stage can move the SiteId field from the event value to metadata, and the output can reference metadata to include the SiteId in the MQTT topic.

The Filter stage works on complex data as well as Arrays of complex data. In the above example, if an array of objects with a SiteId attribute is passed to the filter, the SiteId attribute for each array object is filtered.

Filter List

Specify a list of filters to be applied to the payload in parallel. There are five types of filter: Exact Path, Starts With, Ends With, Contains, and REGEX.

Filter Option

The Filter stage can be set to either include or exclude keys that match the filter set. When set to include, keys are included in the payload if they match at least one of the specified filters. Likewise, when set to exclude, keys are excluded from the payload when they match at least one of the specified filters. The include and exclude option will apply to all specified filters within a filter stage, so for a combination of include and excluding, multiple filter stages in series are required.

Retain As Metadata

When enabled, anything removed from the event value is included in the event metadata under the key specified in Metadata Key.

In the above SiteId example, the MQTT topic output would be {{event.metadata.filter.siteID}}.

Metadata Key

Specifies the name of the key used to retain the filtered out data in the metadata object.

Filter Settings

Filter Text

Specify the text that the filter will use for comparison (e.g. setting the filter text to _id on a “Starts With” filter will match all keys that start with _id).

Recursive

Specifies whether the filter should be applied to only the root level of the event object, or to every level. When enabled, if a key is matched by a filter, all of its descendants will be included / excluded regardless of if they match the filter. Additionally, all keys of which the matched key is a descendant will also be included in order to maintain structure. Exact Path filters cannot be set to recursive.

Filter Types

Exact Path

Matches keys with the same full path in the format ‘Parent.Child.GrandChild’. For example with an Exact Path filter of ‘Parent.Child.GrandChild’:

json

 {
    "Parent": {
        "Child": {
            "GrandChild": "included",
            "GrandChild_Sibling": "excluded"
        },
        "Child_Sibling": "excluded"
    },
    "Parent_Sibling": "excluded"
 }

Starts With

Matches keys that start with the provided filter text

Ends With

Matches keys that end with the provided filter text

Contains

Matches keys which contain the provided filter text

REGEX

Matches keys which satisfy a provided regular expression. For example, with the filter text “gr(a|e)y” the filter will match keys containing either “gray” or “grey”. While this filter may be used for any regular expression, it is important to note that because the filter may run against every key in the event, particularly complicated expressions or particularly large payloads may inversely affect the performance stage and slow the pipeline.

Flatten