Best Practices for Log Filtering

This guide provides overall best practices for your use of Log Filtering. It describes how best to utilize log filters for sensors, connectors, and log sources.

Understanding the Types of Filters

There are two types of filters can be defined to filter ingested data:

  • Traffic filter — a traffic filter can be applied to mirror traffic on the sensor. The sensor does not report the Interflow that matches the filter to the platform.

  • Log filter — a log filter can be applied to the sensor profile to discard logs to the sensor or it can be applied to the connector configuration to discard logs pulled from a connector. In the sensor profile, the log filter can be used for both the Log Forwarder and the Windows agent.

This document describes the best practice for log filters.

Creating a Log Filter

To create a log filter, choose System | Collection | Log Filters. A table for managing log filters is displayed. Click Create to add a new filter or click the pencil icon in the table to edit and existing filter.

You can refer to Managing Log Filters.

Note the following:

  • Log filters can be created for All Tenants for one specific tenant. Once it is created for All Tenants, any tenant can apply this log filter to its sensor profiles or connectors.

  • Most of the time you want to apply a filter on one type of logs, for example, the Cisco ASA logs. Select the Log Source from the dropdown list and the corresponding real value is filled in. The real value can also be found from the msg_origin.source in the JSON record in Investigate | Threat Hunting.

  • Filters can also be created without a Log Source, which means the filter can be applied to all types of logs. That is why the Log Source is (Optional).

  • You can define multiple filter rules with different field names and field values. The maximum number of Conditions (rules) per log filter is 10.

  • Field names can only be from the Log Forwarder after parsing/normalization. Any DP enriched field names cannot be used in rules.

  • Within one log filter, all rules have AND logic. Only when all conditions match, is the filter matched.

  • There are different operators for different data types:

    • date: is, is not, contains, does not contain, field exists, field does not exist

    • ip: matches, does not match, field exists, field does not exist. For the IP data type, the value can be an IP address subnet, for example, 192.168.1.0/24.

    • number: is, is not, greater than, less than, greater than or equal to, less than or equal to, field exists, field does not exist

    • string: is, is not, contains, does not contain, field exists, field does not exist

  • For certain operators, for example, is, is not, contains, does not contain, matches, and does not match, the value can be a list. The logic is OR between these values. For example, appid_name might be test1 OR test2.

  • For the in range operator, use a combination of the greater than, less than, greater than or equal to, less than or equal to operators, so it is clear that it is inclusive or non-inclusive.

Applying a Log Filter

Once the log filter is created, it can be used for a sensor profile (Log Forwarder or Windows event), and/or a connector. You can apply multiple log filters to one sensor profile or one connector. The logic between these filters is OR. Any logs matching one of the filters is considered a match.

Filter order in the list does not impact the filtering result but may have an impact on the performance. Always order the filter by its matched possibility. For example, if filter1 can match 50% of the logs and filter2 can match 10% of the logs, put filter1 ahead of filter2 so that most of the logs to be dropped are checking filter1 without checking filter2.

Applying a Log Filter to Log Forwarder

You can apply a log filter to the Log Forwarder in both the Standard Sensor Profile and Modular Sensor Profile pages.

The maximum number of log filters that can be applied to one sensor profile is 10.

Applying a Log Filter to Windows Event

You can apply a log filter to Windows Events in the Windows tab of the Standard Sensor Profile page.

Note the following:

  • A maximum of 100 filters can be configured in the exclude list.

  • A maximum of 100 filters can be configured in the include list.

  • The log filter only applies to any Windows event not FIM event.

  • When both the exclude and include lists are empty, filtering is disabled.

  • When both the exclude list and include list are defined, logs are checked against the exclude list first. If it matches the exclude list, the log is dropped, otherwise it continues matching the include list. If it matches the include list, the log is passed, otherwise the log is dropped.

  • For a log filter to match event logs on the Windows agent sensor, do not specify a Log Source.

Applying a Log Filter to a Connector

You can apply a log filter to a connector directly on the connector configuration page.

The maximum number of log filters that can be applied to one connector is 10.

When the Function is Collect, you can apply log filters to a connector. The configuration page does not display the Log Filters field if the Function is only Respond.