Working with Data Management
You must have Root scope to use this feature.
The System | Data Processor | Data Management page is your home base for data backup, import, and restore procedures. Separate tabs in the Data Management page let you perform the following tasks:
|
Tab |
Description |
|---|---|
| Data Sink Import |
Use this tab to import specific data from a Data Sink for temporary analysis needs. You can filter the import on a combination of Tenants, Indices, and a date range. Imported data stays in the DP until you delete the import task from the Data Sink Import page. At that point, the imported data is subject to normal disk cleanup and is removed according to your retention group settings. |
| Data Sink Restore |
Use this tab to restore data from the hot tier in a Data Sink. You can filter the import on a combination of Tenants, Indices, and a date range. In contrast to data sink imports, data sink restores are typically performed in disaster recovery scenarios. Restored data is immediately subject to normal disk cleanup and is removed if it is no longer in the hot tier, as specified by your retention group settings. |
| Snapshot Backup/Restore |
Use this tab to manage snapshot-based backups of data and configuration. You can create snapshot-based backups and restore from them in this tab. Keep in mind that restores are for data that is in the hot tier, as specified by your Retention Group settings, and are typically performed in disaster recovery scenarios. The Snapshot Backup/Restore tab was named Backup/Restore prior to the 4.3.1 release. |
| Snapshot Storage Configuration |
Add and manage external storage resources that can be used for backups, imports and exports, cold storage, and cold standby. The Snapshot Storage Configuration tab was named External Storage Configuration prior to the 4.3.1 release. |
| Snapshot Import |
Use this tab to import snapshot-based data from cold storage to a forensic mode DP for analysis. The Snapshot Import tab was named Cold Storage Imports prior to the 4.3.1 release. |
| Retention Groups | Use this tab to define retention times for different data types in both the hot and cold tiers. You can assign different retention groups to different tenants, allowing you to customize a data retention strategy that works best for different tenants' needs. |
| Import/Export Indices | Use this tab to export or import data in human-readable format from the DP. This option is typically used in deployments without access to external storage. Stellar Cyber recommends that you use either the Data Sink or Snapshot approaches for most common import/export needs. |
| Advanced | Migrate (as a unit) all Stellar Cyber data from one system to another, or clear data for a tenant. |
Data Management Concepts
This section explains key data management concepts in Stellar Cyber, including:
-
The differences between snapshots and data sinks and which use cases are best suited for each.
-
The differences between imports and restores
Understanding Snapshots and Data Sinks
Stellar Cyber provides two types of data backups – snapshots and data sinks:
-
Snapshots are the traditional backup tool for Stellar Cyber and existed in releases prior to 4.3.7 under different names. Snapshots provide binary backups of the ElasticSearch cluster to external storage configured in the Snapshot Storage Configuration tab (NFS, AWS S3, Azure, or OCI). Snapshots can be restored more quickly than data stored in data sinks, but are much less granular. Because of this, snapshots typically have higher storage costs than data sinks.
-
Data Sinks existed in previous Stellar Cyber versions but were refactored in 4.3.0 to provide granular storage in human-readable format (JSON) that lets you use detailed filters to store and retrieve just the data you're most interested in, saving on storage costs (refer to About the Data Sink Framework). The tradeoff is that, byte for byte, data sink imports/restores are much slower than snapshot imports/restores.
Understanding Imports and Restores
Stellar Cyber provides the ability to perform both imports and restores of data, both from data sinks and from snapshot storage. In general, imports are for temporary analysis of old data while restores are for disaster recovery. However, there are differences between the data sink and snapshot implementations of the two techniques:
|
Technique |
Snapshot |
Data Sink |
|---|---|---|
| Import |
|
|
| Restore |
|
|
Use Case Recommendations
You can perform imports and restores on both snapshots and data sinks. Because of the tradeoffs with these two storage techniques, however, in general:
-
Imports are best suited to data sinks because you can store and retrieve just the data you need, saving on costs and improving the efficiency of your analysis. In addition, there is no need to switch to forensic mode to perform the analysis.
-
Restores are best suited to snapshots because you can bring back hot data more quickly than the data sink approach. In a disaster recovery situation, it's crucial to restore data as quickly as possible.
In both cases, however, you can only import and restore from whatever type(s) of stored data you've configured. You won't, for example, be able to import from a Data Sink if you haven't configured and enabled one.
Simultaneous Import/Restore?
The rules for simultaneous imports/restores are as follows:
-
You can run imports and restores from Data Sinks simultaneously. However, the import task is given higher priority.
-
You cannot run Data Sink imports/restores simultaneously with Snapshot imports/restores.
-
Snapshot imports/restores cannot be run simultaneously with one another.
The table below summarizes these rules:
|
Simultaneous execution? |
Snapshot Import |
Snapshot Restore |
Data Sink Import |
Data Sink Restore |
|---|---|---|---|---|
| Snapshot Import | Yes | |||
| Snapshot Restore | Yes | |||
| Data Sink Import | Yes | Yes | ||
| Data Sink Restore | Yes | Yes |
