Adding an ElasticSearch Data Sink

You must have Root scope to use this feature.

You can add an ElasticSearch Data Sink from the System | Data Processor | Data Sinks page using the following procedure:

Before you add an ElasticSearch Data Sink, ensure that your ElasticSearch allows the DP IP address.

Another Stellar Cyber data processor can serve as an ElasticSearch Data Sink.

  1. Click System | Data Processor | Data Sinks. The Data Sink list appears.

  2. Click Create. The Setup Data Sink screen appears.

  3. Enter the Name of your new Data Sink. This field does not support multibyte characters.

  4. Choose elasticsearch for the Type.

    Additional fields appear in the Setup Data Sink screen:

  5. Enter the Host. This is the FQDN or IP address of the ElasticSearch server.
  6. Enter the Port Number. The default is 9300.
  7. Enter the ElasticSearch Cluster Name. This field does not support multibyte characters.

  8. Select the types of data to send to the Data Sink by toggling the following checkboxes:

    • Raw Data – Raw data received from sensors, log analysis, and connectors after normalization and enrichment has occurred and before the data is stored in the Data Lake.

    • Alerts – Security anomalies identified by Stellar Cyber using machine learning and third-party threat-intelligence feeds, reported in the Alerts interface, and stored in the aella-ser-* index.

    • Assets – MAC addresses, IP addresses, and routers identified by Stellar Cyber based on network traffic, log analysis, and imported asset feeds and stored in the aella-assets-* index.

    • Users – Users identified by Stellar based on network traffic and log analysis and stored in the aella-users-* index.

    Alerts, assets, and users are also known as derived data because Stellar Cyber extrapolates them from raw data.

  9. Click Next.

    The Advanced (Optional) page appears.

  10. Check the Authentication box if the specified ElasticSearch server requires it and specify the Security Port, Username, and Password.

  11. Use the Batch Window (seconds) and Batch Size fields to specify how data is written to the sink.

    • The Batch Window specifies the maximum amount of time that can elapse before data is written to the Data Sink.

    • The Batch Size specifies the maximum number of records that can accumulate before they are sent to the Data Sink. You can specify either 0 (disabled) or a number of records between 100 and 10,000.

    Stellar Cyber batches data to the Data Sink depending on whichever of these parameters is reached first.

    So, for example, consider a Data Sink with a Batch Window of 30 seconds and a Batch Size of 300 records:

    • If at the end of the Batch Window of 30 seconds, Stellar Cyber has 125 records, it sends them to the data sink. The Batch Window was reached before the Batch Size.

    • If at the end of 10 seconds, Stellar Cyber has 300 records, it send the 300 records to the Data Sink. The Batch Size was reached before the Batch Window.

    These options are primarily useful for data sink types that charge you by the API call (for example, AWS S3 and Azure). Instead of sending records as they are received, you can use these options to batch the records, minimizing both API calls and their associated costs for Data Sinks in the public cloud.

    By default, these options are disabled for ElasticSearch Data Sinks, which are not typically subject to the same pricing restraints as Data Sinks in the public cloud.

  12. You can use the Filter options to Exclude or Include specific Message Classes for the Data Sink. By default, Filter is set to None. If you check either Exclude or Include, an additional Message Class field appears where you can specify the message classes to use as part of the filter. For example:

    You can find the available message classes to use as filters by searching your Interflow in the Investigate | Threat Hunting | Interflow Search page. Search for the msg_class field in any index to see the prominent message classes in your data.

  13. Click Next to review the Data Sink configuration. Use the Back button to correct any errors you notice. When you are satisfied with the sink's configuration, click Submit to add it to the DP.

The new Data Sink is added to the list.