Scaling Up with Worker Nodes

You can scale a DP deployment in AWS by adding additional Data Lake and Data Analyzer worker nodes to achieve higher data ingestion rates, greater storage capacity, and longer data retention in warm storage.

In general, Stellar Cyber recommends adding DL-worker and DA-worker nodes in pairs to prevent data loss. For example, consider the following simple deployment with a single pair of DL and DA worker nodes:

Adding a pair of worker nodes consists of the following major steps:

1. Launch and configure the Worker instances in AWS.

2. Configure the Worker nodes in the CLI as resources.

3. Configure the Worker node in the user interface, converting their resource roles to the correct worker type (DL or DA).

4. Repeat the previous steps to add pairs of DL/DA worker nodes.

Once your Data Lake scales up to six or more Data Lake workers, Stellar Cyber recommends that you create a Data Lake Coordinating Node to improve performance. Refer to Scaling Up the Data Lake with Coordinating Nodes for details.