Working with Data Management

You must have Root scope to use this feature.

The System | Data Processor | Data Management page is your home base for data backup, import, and restore procedures. Separate tabs in the Data Management page let you perform the following tasks:

Tab

Description

Data Sink Import

Use this tab to import specific data from a Data Sink for temporary analysis needs. You can filter the import on a combination of Tenants, Indices, and a date range.

Imported data stays in the DP until you delete the import task from the Data Sink Import page. At that point, the imported data is subject to normal disk cleanup and is removed according to your retention group settings.

Data Sink Restore

Use this tab to restore data from the hot tier in a Data Sink. You can filter the import on a combination of Tenants, Indices, and a date range.

In contrast to data sink imports, data sink restores are typically performed in disaster recovery scenarios. Restored data is immediately subject to normal disk cleanup and is removed if it is no longer in the hot tier, as specified by your retention group settings.

Snapshot Backup/Restore

Use this tab to manage snapshot-based backups of data and configuration. You can create snapshot-based backups and restore from them in this tab.

Keep in mind that restores are for data that is in the hot tier, as specified by your Retention Group settings, and are typically performed in disaster recovery scenarios.

The Snapshot Backup/Restore tab was named Backup/Restore prior to the 4.3.1 release.

Snapshot Storage Configuration

Add and manage external storage resources that can be used for backups, imports and exports, cold storage, and cold standby.

The Snapshot Storage Configuration tab was named External Storage Configuration prior to the 4.3.1 release.

Snapshot Import

Use this tab to import snapshot-based data from cold storage to a forensic mode DP for analysis.

The Snapshot Import tab was named Cold Storage Imports prior to the 4.3.1 release.

Retention Groups Use this tab to define retention times for different data types in both the hot and cold tiers. You can assign different retention groups to different tenants, allowing you to customize a data retention strategy that works best for different tenants' needs.
Import/Export Indices Use this tab to export or import data in human-readable format from the DP. This option is typically used in deployments without access to external storage. Stellar Cyber recommends that you use either the Data Sink or Snapshot approaches for most common import/export needs.
Advanced Migrate (as a unit) all Stellar Cyber data from one system to another, or clear data for a tenant.

Data Management Concepts

This section explains key data management concepts in Stellar Cyber, including:

  • The differences between snapshots and data sinks and which use cases are best suited for each.

  • The differences between imports and restores

Understanding Snapshots and Data Sinks

Stellar Cyber provides two types of data backups – snapshots and data sinks:

  • Snapshots are the traditional backup tool for Stellar Cyber and existed in releases prior to 5.2.0 under different names. Snapshots provide binary backups of the ElasticSearch cluster to external storage configured in the Snapshot Storage Configuration tab (NFS, AWS S3, Azure, or OCI). Snapshots can be restored more quickly than data stored in data sinks, but are much less granular. Because of this, snapshots typically have higher storage costs than data sinks.

  • Data Sinks existed in previous Stellar Cyber versions but were refactored in 4.3.0 to provide granular storage in human-readable format (JSON) that lets you use detailed filters to store and retrieve just the data you're most interested in, saving on storage costs (refer to About the Data Sink Framework). The tradeoff is that, byte for byte, data sink imports/restores are much slower than snapshot imports/restores.

Understanding Imports and Restores

Stellar Cyber provides the ability to perform both imports and restores of data, both from data sinks and from snapshot storage. In general, imports are for temporary analysis of old data while restores are for disaster recovery. However, there are differences between the data sink and snapshot implementations of the two techniques:

Technique

Snapshot

Data Sink

Import
  • Data Sink configured and enabled in Data Sinks page.

  • Use Data Sink Imports page to import specified data from Data Sink to active DP.

  • Data is sinked constantly to enabled Data Sinks. Retention Groups not used to move data to Data Sinks.

  • Imported data stays in DP until import task is deleted from Data Sink Imports page.

  • Forensic mode not used.

  • More efficient than snapshot imports.

Restore
  • Data Sink configured and enabled in Data Sinks page.

  • Use Data Sink Restore page to restore specified data from Data Sink to active DP.

  • Only difference with Data Sink import is that restored data is immediately subject to removal based on normal disk cleanup and retention group settings.

Use Case Recommendations

You can perform imports and restores on both snapshots and data sinks. Because of the tradeoffs with these two storage techniques, however, in general:

  • Imports are best suited to data sinks because you can store and retrieve just the data you need, saving on costs and improving the efficiency of your analysis. In addition, there is no need to switch to forensic mode to perform the analysis.

  • Restores are best suited to snapshots because you can bring back hot data more quickly than the data sink approach. In a disaster recovery situation, it's crucial to restore data as quickly as possible.

In both cases, however, you can only import and restore from whatever type(s) of stored data you've configured. You won't, for example, be able to import from a Data Sink if you haven't configured and enabled one.

Simultaneous Import/Restore?

The rules for simultaneous imports/restores are as follows:

  • You can run imports and restores from Data Sinks simultaneously. However, the import task is given higher priority.

  • You cannot run Data Sink imports/restores simultaneously with Snapshot imports/restores.

  • Snapshot imports/restores cannot be run simultaneously with one another.

The table below summarizes these rules:

Simultaneous execution?

Snapshot Import

Snapshot Restore

Data Sink Import

Data Sink Restore

Snapshot Import Yes      
Snapshot Restore   Yes    
Data Sink Import     Yes Yes
Data Sink Restore     Yes Yes