Auto Scaling

For each cluster you can optionally enable auto scaling, which allows you to define one or more policies to automatically increase or decrease the capacity of the cluster when certain pre-defined conditions are met.

When creating an auto scaling policy, you define:

For example:

Scaling Event Scaling Policy
If CPU usage is more than 70% for 5 minutes (averaged across all cluster nodes). Add 3 compute nodes
If HDFS usage is less than 10% for 15 minutes. Remove 5 worker nodes
If Memory usage is more than 50% for 10 minutes (averaged across all cluster nodes). Add 3 compute nodes

In addition, you can configure cluster auto scaling settings that are valid for all auto scaling policies attached to a given cluster. These settings include:

By default, the cooldown period lasts 30 minutes and the cluster size is between 3 and 100 nodes.

Enabling Cluster Auto Scaling

By default, cluster autoscaling is disabled. You can enable it, and then define auto scaling policies and configure auto scaling settings:

Viewing and Managing Cluster Auto Scaling

For any running cluster, you can view or manage auto scaling from the AUTO SCALING tab on the cluster details page:

Defining An Auto Scaling Policy

You can add one or more auto scaling policies for your cluster either when creating the cluster or when the cluster is already running.

In order to define an auto scaling policy, you need to provide the following information:

Property Description
Policy Name A unique name for your policy (min 5, max 20, must start with a letter, accepted chars: a-z, 0-9, -)
Scaling Definition Metric The scaling metric definition to watch
Metric Value A value from 1 to 100 (percent); When the metric defined above reaches this value, the scaling policy will be executed
Duration The amount of time (in minutes) the metric value needs to be more/less than defined above
Action Select Add or Remove
Nodes The number of nodes to add or remove
Group The node type (worker or compute) to add or remove
Scaling Metric Definitions

The following scaling metric definitions (or "scaling definitions" for short) can be used to define scaling events:

Metric Definition Description
CPU usage Measures the CPU usage average across all nodes in the cluster.
Memory usage Measures the Memory usage average across all nodes in the cluster.
HDFS usage Measure the HDFS usage for the cluster.

Configuring Cluster Auto Scaling Settings

You can configure auto scaling settings for your cluster either when creating the cluster or when the cluster is already running.

In order to configure the auto scaling settings for your cluster, you need to provide the following information:

Setting Description Default Value
Cooldown Time After an auto scaling event occurs, the amount of time to wait before enforcing another scaling policy. 30 minutes
Cluster Size Limit The minimum and maximum size allowed for the cluster (worker + compute nodes). Auto scaling policies cannot scale the cluster above or below this size. Min 3 nodes, max 100 nodes

Cluster Auto Scaling via CLI

You can also enable autoscaling, add autoscaling policies, and configure cluster auto scaling settings using the CLI. For more information, refer to Configuring Auto Scaling for Your Cluster.