Best Practices for Log Filtering

This guide provides overall best practices for your use of Log Filtering. It describes how best to utilize log filters for sensors, connectors, and log sources.

Understanding the Types of Filters

There are two types of filters can be defined to filter ingested data:

  • Traffic filter – A traffic filter can be applied to mirror traffic on the sensor. The sensor does not report the Interflow that matches the filter to the platform.

  • Log filter – A log filter can be applied to the sensor profile to discard logs to the sensor or it can be applied to the connector configuration to discard logs pulled from a connector. In the sensor profile, the log filter can be used for both the Log Forwarder and the Windows Server Sensor.

This document provides some best practices for log filters.

Creating a Log Filter

To create a log filter, navigate to System | DATA SOURCE MANAGEMENT | Data Filters | Log Filters.

A table for managing log filters appears.

Select Create to add a new filter or select the pencil icon in the table to edit an existing filter.

Refer to Managing Log Filters for details.

Note the following:

  • Log filters can be created for All Tenants or for one specific tenant. Once a log filter is created for All Tenants, any tenant can apply the log filter to its sensor profiles or connectors.

  • Most of the time you will want to apply a filter on one type of logs (for example, Cisco ASA logs). Select the Log Source from the drop-down list and the corresponding real value is filled in. The real value can also be found from the msg_origin.source in the JSON record in Threat Hunting.

  • Filters can also be created without a Log Source, which means the filter can be applied to all types of logs. That is why the Log Source is (Optional).

  • You can define multiple filter rules with different field names and field values. The maximum number of Conditions (rules) per log filter is 10.

  • Field names can only be from the Log Forwarder after parsing/normalization. Any DP-enriched field names cannot be used in rules.

  • Within one log filter, all rules have AND logic. This means that the filter is only matched when all conditions match.

  • There are different operators for different data types:

    • date: is, is not, contains, does not contain, field exists, field does not exist

    • ip: matches, does not match, field exists, field does not exist. For the IP data type, the value can be an IP address subnet (for example, 192.168.1.0/24).

    • number: is, is not, greater than, less than, greater than or equal to, less than or equal to, field exists, field does not exist

    • string: is, is not, contains, does not contain, field exists, field does not exist

  • For the following operators, the value can be a list – is, is not, contains, does not contain, matches, and does not match. The logic is OR between these values. For example, appid_name might be test1 OR test2.

  • For the in range operator, use a combination of the greater than, less than, greater than or equal to, less than or equal to operators, so it is clear that it is inclusive or non-inclusive.

Applying a Log Filter

Once the log filter is created, it can be used for a sensor profile (Log Forwarder or Windows event), and/or a connector. You can apply multiple log filters to one sensor profile or one connector. The logic between these filters is OR. Any logs matching one of the filters is considered a match.

Filter order in the list does not impact the filtering result but may have an impact on the performance. Always order the filter by its matched possibility. For example, if filter1 can match 50% of the logs and filter2 can match 10% of the logs, put filter1 ahead of filter2 so that most of the logs to be dropped are checking filter1 without checking filter2.

Applying a Log Filter to Log Forwarder

You can apply a log filter to the Log Forwarder in both the Standard Sensor Profile and Modular Sensor Profile pages (System | DATA SOURCE MANAGEMENT | Sensors | Sensor Profiles).

The maximum number of log filters that can be applied to one sensor profile is 10.

Applying a Log Filter to Windows Event

You can apply a log filter to Windows Events in the Windows tab of the Standard Sensor Profile page.

Note the following:

  • A maximum of 100 filters can be configured in the exclude list.

  • A maximum of 100 filters can be configured in the include list.

  • The log filter only applies to any Windows event and not FIM events.

  • When both the exclude and include lists are empty, filtering is disabled.

  • When both the exclude list and include list are defined, logs are checked against the exclude list first. If it matches the exclude list, the log is dropped, otherwise it continues matching the include list. If it matches the include list, the log is passed, otherwise the log is dropped.

  • For a log filter to match event logs on the Windows Server Sensor, do not specify a Log Source.

Applying a Log Filter to a Connector

You can apply a log filter to a connector directly on the connector configuration page (System | INTEGRATIONS | Connectors and then select either Create or the Edit icon).

The maximum number of log filters that can be applied to one connector is 10.

When the Function is Collect, you can apply log filters to a connector. The configuration page does not display the Log Filters field if the Function is only Respond.