clearml-docs/aws_autoscaler.md at afe0c9d1e24b13f231bc662419d1dae03ddfd62b

mirror of https://github.com/clearml/clearml-docs synced 2025-01-31 22:48:40 +00:00

2021-05-14 02:48:51 +03:00

10 KiB

Raw Blame History

title
ClearML AWS Autoscaler Service

The ClearML AWS autoscaler optimizes AWS EC2 instance scaling according to the instance types used, and the budget configured.

In the budget, set the maximum number of each instance type to spin for experiments awaiting execution in a specific queue. Configure multiple instance types per queue, and multiple queues. The ClearML AWS autoscaler will spin down idle instances based on the maximum idle time and the polling interval configurations.

Running the ClearML AWS autoscaler

The ClearML AWS autoscaler can execute in ClearML services mode, and is configurable.

Run ClearML AWS autoscaler in one of these ways:

In the ClearML Web UI.
- The autoscaler is pre-loaded in the ClearML Server and its status is Draft (editable).
- Set the instance types and configure the budget in the ClearML Web UI, and then enqueue the Task to the services queues.
By running the aws_autoscaler.py script.
- Run script locally or as a service.
- When executed, a Task is created, named AWS Auto-Scaler that associated with the DevOps project.

Running using the ClearML Web UI

Edit the parameters for the instance types, edit budget configuration by editing the Task, and then enqueue the Task to run in ClearML Agent services mode.

Open the ClearML Web UI > Projects page > DevOps project > AWS Auto-Scaler Task.
Set the AWS and Git credentials, parameters for idle AWS EC2 instances, and a worker prefix.
- In the CONFIGURATIONS tab > HYPER PARAMETERS > Args > hover > EDIT.
  - cloud_credentials_key - AWS access key.
  - cloud_credentials_region - AWS region.
  - cloud_credentials_secret - AWS access secret.
  - cloud_provider - AWS.
  - default_docker_image - The default Docker image to use for the AWS EC2 instance.
  - git_pass - Git password.
  - git_user - Git username.
  - max_idle_time_min - The maximum time an AWS EC2 instance can be idle before the ClearML AWS autoscaler spins it down.
  - polling_interval_time_min - How often the ClearML AWS autoscaler checks for idle instances.
  - workers_prefix
Configure the budget.
- In CONFIGURATION OBJECTS > General > hover > EDIT. Edit the resource_configurations dictionary:
```
  resource_configurations {
      <resource-name> {
        instance_type = "<instance_type>"
        is_spot = <boolean>
        availability_zone = "<AWS-region>"
        ami_id = "<AMI-ID>"
        ebs_device_name = "<EBS-device-name>"
        ebs_volume_size = <EBS-size-in-GB>
        ebs_volume_type = "<EBS-vol-type>"
      }
  }
  queues {
      <queue-name> = [["<resource-name>", <max-instances-of-resource-name>]]
  }
  extra_clearml_conf = "<ClearML-config-file>"
  extra_vm_bash_script = "<bash-script>"
```
  - <resource-name> - The name assigned to each resource (AWS EC2 instance type). Used in the budget.
  - queues - The ClearML AWS autoscaler will optimize scaling for experiments awaiting execution in these queues.
  - <queue-name> - A specific queue.
  - <max-instances-of-resource-name> - The maximum number of instances of the specified resource-name to spin up.
  - is_spot - If true, then use a spot instance. If false, then use a reserved instance.
  - extra_clearml_conf - A ClearML configuration file to use for executing experiments in ClearML Agent.
  - extra_vm_bash_script - A bash script to execute when creating an instance, before ClearML Agent executes.
  View a screenshot
Set the Task to run in ClearML Agent services mode.
1. In HYPER PARAMETERS > Args > hover > EDIT.
2. Change the remote parameter to true.
View a screenshot
Click SAVE.
In the experiments table, right click the AWS Auto-Scaler Task > Enqueue > services queue > ENQUEUE.

Running using the script

The aws_autoscaler.py script includes a wizard which prompts for instance details and budget configuration.

The script can run in two ways:

Configure and enqueue.
Enqueue with an existing configuration.

To configure and enqueue:

Use the run command line option:

python aws_autoscaler.py --run

When the script runs, a configuration wizard prompts for all required information.

View the configuration wizard steps

The setup wizard begins. Enter the AWS credentials and AWS region name.

AWS Autoscaler setup wizard
---------------------------
Follow the wizard to configure your AWS auto-scaler service.
Once completed, you will be able to view and change the configuration in the clearml-server web UI.
It means there is no need to worry about typos or mistakes :)

Enter AWS Access Key ID : 
Enter AWS Secret Access Key : 
Enter AWS region name [us-east-1b]:

Enter Git credentials. These are required by ClearML Agent to set up a Task execution environment in an AWS EC2 instance.

GIT credentials:
Enter GIT username for repository cloning (leave blank for SSH key authentication): []
Enter password for user '&lt;username>':

The wizard reports the Git credentials it will use.

Git repository cloning will be using user=*************** password=***********

Enter the default Docker image and parameters to use.

Enter default docker image/parameters to use [nvidia/cuda:10.1-runtime-ubuntu18.04]:

For each AWS EC2 instance type that will be used in the budget, do the following:

Choose the instance type
Choose whether to use spot instances
Select an AMI
Define the Amazon EBS volume

Select as many instance types as needed.

Configure the machine types for the auto-scaler:
------------------------------------------------
Select Amazon instance type ['g4dn.4xlarge']:
Use spot instances? [y/N]: y
Select availability zone ['us-east-1b']:
Select the Amazon Machine Image id ['ami-07c95cafbb788face']:
Enter the Amazon EBS device ['/dev/xvda']:
Enter the Amazon EBS volume size (in GiB) [100]:
Enter the Amazon EBS volume type ['gp2']:

Name the instance type that was configured. Later in the configuration, use this name to create the budget.

Select a name for this instance type (used in the budget section) For example 'aws4gpu':

The wizard prompts whether to select another instance type.

Define another instance type? [y/N]:

Before ClearML Agent executes, enter any bash script to run on newly created instances.

Enter any pre-execution bash script to be executed on the newly created instances []:

Configure the AWS autoscaler budget. For each queue that will be used in the budget, select the queue and the maximum number of each instance type, which the ClearML AWS autoscaler can spin up to execute experiments awaiting execution in that queue.

Define the machines budget:
-----------------------------

Select a queue name (for example: 'aws_4gpu_machines') : 
Select a instance type to attach to the queue ['aws-g4dn.xlarge', 'aws-g4dn.8xlarge', 'aws-g4dn.16xlarge']:
Enter maximum number of 'aws-g4dn.xlarge' instances to spin simultaneously (example: 3) :

If needed, add another instance type to the same queue. The previous step repeats.
```
Do you wish to add another instance type to queue? [y/N]:         
```

The ClearML AWS autoscalar polls instances, and if instances have been idle for the maximum idle time that was specified, the autoscaler spins them down.

Enter maximum idle time for the auto-scaler to spin down an instance (in minutes) [15]:
Enter instances polling interval for the auto-scaler (in minutes) [5]:

The configuration is complete. ClearML initializes the Task AWS Auto-Scaler, the service begins, and the script prints a hyperlink to the Task's log.

CLEARML Task: created new task id=d0ee5309a9a3471d8802f2561da60dfa
CLEARML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
CLEARML results page: https://app.clearml-master.hosted.allegro.ai/projects/142a598b5d234bebb37a57d692f5689f/experiments/d0ee5309a9a3471d8802f2561da60dfa/output/log
Running AWS auto-scaler as a service
Execution log https://app.clearml-master.hosted.allegro.ai/projects/142a598b5d234bebb37a57d692f5689f/experiments/d0ee5309a9a3471d8802f2561da60dfa/output/log

To enqueue with an existing configuration:

Use the remote command line option:

python aws_autoscaler.py --remote

When the script runs, it allows you to create a new configuration.

10 KiB Raw Blame History