Ingesting Logs

If you have a log license for your sensor, you can configure the sensor to ingest logs from devices on your network. Keep in mind the following:

  • Logs that include a security event are sent to the security index.

  • Logs that include source or destination IP addresses or ports are sent to the traffic index.

  • All other logs are sent to the syslog index.

  • Stellar Cyber limits the number of log sources you can point at a given sensor as follows:

    • Sensors with 8 GB or less of memory accept a maximum of 100 log sources.

    • Sensors with more than 8 GB of memory accept a maximum of 200 log sources.

    Log sources beyond the limit are aggregated under the special source IP address of 0.0.0.0.

To configure log ingestion:

  1. In general, configure your source device to send logs to the Stellar Cyber sensor with the following information:

    • The IP address of the sensor as the destination for the logs

    • The port on which to send the logs if the device uses a standard format

    • The port on which to send the logs if the device uses a vendor specific format

    • Formatting for the logs, which should correspond to the formatting defined for the port selected

    (See example configuration for SentinelOne).

    This release includes the ability to relay logs sent to the generic 514/6514 syslog ports on the sensor to vendor-specific parser ports. Using this approach helps you minimize the ports you need to open in your firewall. See Specific Port Destinations or Port Relay? for details.

  2. Configure any firewalls between the device and the sensor to open the ports the logs are sent over.

  3. Authorize the sensor as a log forwarder on the DP:

    1. Click System | Sensors. The Sensor List appears.

    2. From the Manage drop-down, choose Authorization. The Authorize Sensors screen appears. Authorize is already selected for the Sensor Bandwidth Authorization.

    3. Select the sensor to authorize in the Select Sensor and License to Authorize box.

    4. Select the license.

    5. Click the Add arrow. The sensor moves to the Selected Sensors box.

    6. In the Selected Sensors box, click the checkbox for the Log Forwarder.

    7. Click Submit.

  4. Verify that the log events are being received:

    1. On the DP, click Investigate | Threat Hunting. The Interflow Search page appears.

    2. Select Syslog from the Index drop-down. Syslog events are displayed.

    3. Check the table at the bottom of the page for events from your device.

If you're ingesting Windows logs, you can configure a central Windows computer to collect the logs and forward them to the sensor.

Specific Port Destinations or Port Relay?

It's a best practice in Stellar Cyber to send logs to their vendor-specific parsers, when available. In previous releases, this was accomplished by referring to the list of supported vendor-specific ports, pointing your log sources to that port on the sensor IP address, and opening the port in your firewall.

This approach is still available and can be used. As an alternative, however, you can configure your sensors to accept log traffic on the generic syslog ports of 514 (non-TLS) or 6514 (TLS) and relay that traffic to vendor-specific ports internally based on the source traffic's IP address.

You do this differently depending on the release your sensors are running:

  • For sensors running 4.3.5, you configure port relay in the sensor CLI using the instructions below.

  • For sensors running 4.3.6+, you configure port relay in the System | Log Sources page. CLI configuration is deprecated and only the Log Sources page is used.

    Port relay rules configured in the CLI must follow specific migration steps to work correctly in 4.3.6+. See Migration Steps for details.

Configuring Port Relay in the CLI

You configure the port relay feature for sensors running 4.3.5 using the set logforwarder device-ip command in the sensor CLI. The procedure is as follows:

  1. Find the IP address of your log source.

  2. Use the Log Parser Portstopic to find the parser port for your log source.

  3. Connect to the sensor CLI.

  4. Use the set logforwarder device-ip command to make an entry on the sensor for your log source and the corresponding destination port. The syntax is as follows:

    set logforwarder device-ip <IP Address> parser-port <Integer> ingestion-port <514|6514 default=514>

    So, for example, if you are sending Azure MFA logs from 10.33.5.5 to the sensor, you could either send them directly to port 5528 as you did in previous releases, or you could send them to the standard syslog port of 514 and use the following command on the sensor to relay them internally to 5528:

    set logforwarder device-ip 10.33.5.5 parser-port 5528

    This command tells the sensor to relay logs received on port 514 (the default, which is why it is not explicitly specified in the command above) from 10.33.5.5 to the vendor-specific parser port of 5528 for Azure MFA.

    You can also use the ingestion-port argument if you want to listen for a source on the generic TLS syslog port instead of the default of 514. For example, for Netfilter logs sent from 10.31.2.2, you would use the following command to relay them from 6514 to their vendor-specific parser port of 5544:

    set logforwarder device-ip 10.31.2.2 parser-port 5544 ingestion-port 6514

Notes on Using the Port Relay Feature

Keep in mind the following tips when using the port relay feature:

  • Keep in mind that the sending log source must be on the same subnet as the receiving sensor. There must be no proxy capable of changing the log source IP between the sending log source and the receiving sensor.

  • When you create a port relay entry, the sensor listens for both UDP and TCP traffic from the specified source. You can see this with the show logforwarder port-ingestion command. For example:

  • The show logforwarder port-ingestion command is also a useful tool for troubleshooting port relay entries. You can see packet and byte counts for relayed traffic and determine whether traffic is reaching the sensor.

  • You can remove port relay entries using unset logforwarder device-ip <IP Address>.

  • The CLI warns you if you try to add an unsupported parser port. It still adds the unsupported port but lists it in the show logforwarder port-ingestion output as inactive.

Timezones and Log Ingestion

Stellar Cyber converts all log timestamps to UTC during log ingestion. Timezones are handled as follows:

  • If a log includes the timezone, Stellar Cyber preserves that timezone during the conversion to UTC.

  • If a log does not include a timezone, Stellar Cyber uses the timezone of the receiving sensor.

During installation, the timezone for sensors are automatically set to UTC+0. Since the logs for some security products may only include the local time without a timezone, Stellar Cyber recommends that you set the sensor timezone to the same timezone as your security product.

Keeping Tabs on Log Ingestion

Sensors send statistics to the DP quantifying the data being sent and its source once per minute. This data is stored in the Sensor Monitoring (aella-ade) index on the DP. You can use the Threat Monitoring interface to review this data and add it to charts and dashboards. For example:

  1. Log in to Stellar Cyber.

  2. Navigate to Investigate | Threat Hunting.

  3. Click the Indices dropdown and make sure that the Sensor Monitoring index is selected.

  4. Scroll down to the Document table, enter msgtype:39 in the Search field, and press Enter to filter on just this msgtype. This is the msgtype that quantifies log ingestion by sensor.

  5. The table updates to show all entries passing the global filters at the top of the display for Message Type 39. You can check the Stage Output Msg Origin Source column to see where the data came from. In the case of log file ingestion, this column shows the actual log file source (for example, Checkpoint, F5, dlink, and so on).

Migrating 4.3.5 Port Relay Settings to 4.3.6+

The following CLI commands used to configure Port Relay settings in 4.3.5 no longer work in 4.3.6+:

  • set logforwarder device-ip

  • unset logforwarder device-ip

Use the following procedure to migrate a sensor's port relay configuration from the4.3.5 CLI to the Log Sources page in the user interface.

Start this procedure before you upgrade the sensor.

  1. Open a CLI connection to the sensor and issue the show logforwarder port-ingestion command.

    The command returns the current IP forwarding configuration of the sensor.

  2. Save the output of the show logforwarder port-ingestion command so you can use it to create the same configuration in the Log Sources page after you upgrade the sensor.

  3. Upgrade the DP to 4.3.6 or later.

  4. Navigate to the Log Sources page and use the instructions in Configuring Log Sources to create new entries for each of the forwarding rules returned by the show logforwarder port-ingestion command.

  5. Upgrade the sensor to the new version. The port relay rules in the Log Sources page are pushed to the sensor.