2022-08-03 09:15:15 +00:00
---
title: GCP Autoscaler
---
:::info Pro Plan Offering
The ClearML GCP Autoscaler App is available under the ClearML Pro plan
:::
2023-01-25 11:25:29 +00:00
The GCP Autoscaler Application optimizes GCP VM instance usage according to a user defined instance budget: Define your
2022-08-03 09:15:15 +00:00
budget by specifying the type and amount of available compute resources.
Each resource type is associated with a ClearML [queue ](../../fundamentals/agents_and_queues.md#what-is-a-queue ) whose
status determines the need for instances of that resource type (i.e. spin up new instances if there are pending jobs on
the queue).
When running, the autoscaler periodically polls your GCP cluster. The autoscaler automatically deletes idle VM instances
based on a specified maximum idle time, or spins up new VM instances when there aren't enough to execute pending tasks
in a queue (until reaching the defined maximum number of instances). You can add an init script, which will be executed
when each VM instance is spun up.
2023-05-07 09:41:43 +00:00
For more information about how autoscalers work, see [Autoscalers Overview ](../../cloud_autoscaling/autoscaling_overview.md#autoscaler-applications ).
2022-08-03 09:15:15 +00:00
## Autoscaler Instance Configuration
2022-09-15 13:10:11 +00:00
* **Import Configuration** - Import an app instance configuration file. This will fill the configuration wizard with the
values from the file, which can be modified before launching the app instance
2022-08-03 09:15:15 +00:00
* **GCP Configuration**
* GCP Project ID - Project used for spinning up VM instances
* GCP Zone - The GCP zone where the VM instances will be spun up. See [Regions and zones ](https://cloud.google.com/compute/docs/regions-zones )
2022-09-28 09:21:19 +00:00
* GCP Credentials - Credentials with which the autoscaler can access your GCP account for spinning VM instances
up/down. See [Generating GCP Credentials ](#generating-gcp-credentials ).
2022-08-03 09:15:15 +00:00
* **Git Configuration** - Git credentials with which the ClearML Agents running on your VM instances will access your
repositories to retrieve the code for their jobs
* Git User
* Git Password / Personal Access Token
2023-04-16 09:32:48 +00:00
* **Base Docker Image** (optional) - Default Docker image in which the ClearML Agent will run. Provide a Docker stored in a
2022-08-03 09:15:15 +00:00
Docker artifactory so VM instances can automatically fetch it
* **Compute Resources**
2022-09-15 13:10:11 +00:00
* Resource Name - Assign a name to the resource type. This name will appear in the Autoscaler dashboard
2022-08-03 09:15:15 +00:00
* GCP Machine Type - See list of [machine types ](https://cloud.google.com/compute/docs/machine-types )
* Run in CPU mode - Select to have the autoscaler utilize only CPU VM instances
* GPU Type - See list of [supported GPUs by instance ](https://cloud.google.com/compute/docs/gpus )
2022-09-15 13:10:11 +00:00
* Use Preemptible Instance - Choose whether VM instances of this type will be [preemptible ](https://cloud.google.com/compute/docs/instances/preemptible )
2022-08-03 09:15:15 +00:00
* Max Number of Instances - Maximum number of concurrent running VM instances of this type allowed
* Monitored Queue - Queue associated with this VM instance type. The tasks enqueued to this queue will be executed on VM instances of this type
2023-04-16 09:32:48 +00:00
* Machine Image (optional) - The GCP machine image to launch
* Disc Size (in GB) (optional)
2022-08-03 09:15:15 +00:00
* \+ Add Item - Define another resource type
2023-04-16 09:32:48 +00:00
* **Autoscaler Instance Name** (optional) - Name for the Autoscaler instance. This will appear in the instance list
* **Max Idle Time** (optional) - Maximum time in minutes that a VM instance can be idle before the autoscaler spins it down
* **Workers Prefix** (optional) - A Prefix added to workers’
* **Polling Interval** (optional) - Time period in minutes at which the designated queue is polled for new tasks
* **Init Script** (optional) - A bash script to execute after launching the VM instance
* **Additional ClearML Configuration** (optional) - A ClearML configuration file to use by the ClearML Agent when executing your experiments
2022-09-15 13:10:11 +00:00
* **Export Configuration** - Export the app instance configuration as a JSON file, which you can later import to create
a new instance with the same configuration.
2022-08-03 09:15:15 +00:00
:::note Enterprise Feature
2023-05-24 13:41:23 +00:00
You can utilize the [configuration vault ](../../webapp/webapp_profile.md#configuration-vault ) to configure GCP
credentials for the Autoscaler in the following format:
2022-08-03 09:15:15 +00:00
```
auto_scaler.v1 {
gcp {
gcp_credentials: """
{
"type": "service_account",
...
}
"""
}
}
```
:::
## Dashboard
2022-11-08 11:49:34 +00:00
Once an autoscaler is launched, the autoscaler's dashboard provides information about available VM instances and their
2022-08-03 09:15:15 +00:00
status.
The autoscaler dashboard shows:
* Number of Idle Instances
* Queues and the resource type associated with them
* Number of current running instances
* Console: the application log containing everything printed to stdout and stderr appears in the console log. The log
shows polling results of the autoscaler’
instances being spun up/down.
2022-09-28 09:21:19 +00:00
2023-05-17 08:38:28 +00:00
:::tip EMBEDDING CLEARML VISUALIZATION
You can embed plots from the app instance dashboard into [ClearML Reports ](../webapp_reports.md ). These visualizations
are updated live as the app instance(s) updates. The Enterprise Plan and Hosted Service support embedding resources in
external tools (e.g. Notion). Hover over the plot and click < img src = "/docs/latest/icons/ico-plotly-embed-code.svg" alt = "Embed code" className = "icon size-md space-sm" / >
to copy the embed code, and navigate to a report to paste the embed code.
:::
2022-09-28 09:21:19 +00:00
## Generating GCP Credentials
The autoscaler app accesses your GCP account with the credentials you provide.
You will need to create a service account with the required access privileges. Then generate credential keys for that
account to configure the autoscaler app:
1. In your GCP account, in the project of your choice, go to **APIs & Services** > **Credentials**
1. Click ** + CREATE CREDENTIALS** and choose the **Service account** option
1. In the **Create service account** window that is opened, fill out the service account details
1. Assign the `Service Account User` and `Compute Admin` roles to your service account
1. Complete creating the account
1. In the **API & Services** > **Credentials** page, under **Service Accounts** , click on the service account you just
created and go to its **KEYS** tab
2023-02-21 10:59:29 +00:00
1. Click **ADD KEY** and create a key in JSON format. Copy the contents of the JSON file.
2022-09-28 09:21:19 +00:00
2023-02-21 10:59:29 +00:00
1. Go to the GCP Autoscaler wizard ** >** open the **GCP Configuration** panel ** >** click *Edit* in the
**GCP Credentials** field.
Paste the contents of the JSON file from the previous step into the **GCP Credentials** popup.
2022-09-28 09:21:19 +00:00
