mirror of
https://github.com/clearml/clearml-docs
synced 2025-06-14 18:39:55 +00:00
Refactor ClearML Agent page
This commit is contained in:
alnoam
parent
c1b9b2e129
commit
94525b12c9
@ -135,7 +135,10 @@ In case a `clearml.conf` file already exists, add a few ClearML Agent specific c
|
||||
* Mac - `$HOME/clearml.conf`
|
||||
* Windows - `\User\<username>\clearml.conf`
|
||||
|
||||
1. After the `api` section, add the following `agent` section:
|
||||
1. After the `api` section, add your `agent` section
|
||||
<details className="cml-expansion-panel configuration">
|
||||
<summary className="cml-expansion-panel-summary">View sample agent section</summary>
|
||||
<div className="cml-expansion-panel-content">
|
||||
|
||||
agent {
|
||||
# Set GIT user/pass credentials (if user/pass are set, GIT protocol will be set to https)
|
||||
@ -285,12 +288,12 @@ In case a `clearml.conf` file already exists, add a few ClearML Agent specific c
|
||||
# cuda_version: 10.1
|
||||
# cudnn_version: 7.6
|
||||
}
|
||||
|
||||
</div></details>
|
||||
1. Save the configuration.
|
||||
|
||||
## Execution
|
||||
|
||||
### Simple Execution
|
||||
### Spinning up an Agent
|
||||
|
||||
#### Executing an Agent
|
||||
To execute an agent, listening to a queue, run:
|
||||
@ -370,40 +373,6 @@ clearml-agent daemon --queue default --foreground
|
||||
```
|
||||
|
||||
|
||||
## Building Docker Containers
|
||||
|
||||
### Task Container
|
||||
|
||||
Build a Docker container that when launched executes a specific experiment, or a clone (copy) of that experiment.
|
||||
|
||||
- Build a Docker container that at launch will execute a specific Task.
|
||||
```bash
|
||||
clearml-agent build --id <task-id> --docker --target <new-docker-name> --entry-point reuse_task
|
||||
```
|
||||
- Build a Docker container that at launch will clone a Task specified by Task ID, and will execute the newly cloned Task.
|
||||
```bash
|
||||
clearml-agent build --id <task-id> --docker --target <new-docker-name> --entry-point clone_task
|
||||
```
|
||||
- Run built Docker by executing:
|
||||
```bash
|
||||
docker run <new-docker-name>
|
||||
```
|
||||
|
||||
### Base Docker Container
|
||||
|
||||
Build a Docker container according to the execution environment of a specific Task.
|
||||
|
||||
```bash
|
||||
clearml-agent build --id <task-id> --docker --target <new-docker-name>
|
||||
```
|
||||
|
||||
|
||||
It's possible to add the Docker container as the base Docker image to a Task (experiment), using one of the following methods:
|
||||
|
||||
- Using the **ClearML Web UI** - See [Base Docker image](webapp/webapp_exp_tuning.md#base-docker-image) on the "Tuning
|
||||
Experiments" page.
|
||||
- In the **ClearML** configuration file - Use the **ClearML** configuration file [agent.default_docker](configs/clearml_conf.md#agentdefault_docker)
|
||||
options.
|
||||
|
||||
## Execution Environments
|
||||
|
||||
@ -490,46 +459,6 @@ venvs_cache: {
|
||||
},
|
||||
```
|
||||
|
||||
## Services Mode
|
||||
The ClearML Agent Services Mode executes an Agent that can execute multiple Tasks. This is useful for Tasks that are mostly
|
||||
idling, such as periodic cleanup services, or a [pipeline controller](references/sdk/automation_controller_pipelinecontroller.md).
|
||||
|
||||
Launch a service Task like any other Task, by enqueuing it to the appropriate queue.
|
||||
|
||||
:::note
|
||||
The default `clearml-server` configuration already runs a single `clearml-agent` in services mode that listens to the “services” queue.
|
||||
:::
|
||||
|
||||
To run a `clearml-agent` in services mode, run:
|
||||
```bash
|
||||
clearml-agent daemon --services-mode --queue services --create-queue --docker <docker_name> --cpu-only
|
||||
```
|
||||
:::note
|
||||
`services-mode` currently only supports Docker mode. Each service spins on its own Docker image.
|
||||
:::
|
||||
|
||||
:::warning
|
||||
Do not enqueue training or inference Tasks into the services queue. They will put an unnecessary load on the server.
|
||||
:::
|
||||
|
||||
### Setting Server Credentials
|
||||
|
||||
Self hosted [ClearML Server](deploying_clearml/clearml_server.md) comes by default with a services queue.
|
||||
By default, the server is open and does not require username and password, but it can be [password protected](deploying_clearml/clearml_server_security#user-access-security).
|
||||
In case it is password protected the services agent will need to be configured with server credentials (associated with a user).
|
||||
|
||||
To do that, set these environment variables on the ClearML Server machine with the appropriate credentials:
|
||||
```
|
||||
CLEARML_API_ACCESS_KEY
|
||||
CLEARML_API_SECRET_KEY
|
||||
```
|
||||
|
||||
## Google Colab
|
||||
|
||||
ClearML Agent can run on a [google colab](https://colab.research.google.com/) instance. This helps users to leverage
|
||||
compute resources provided by google colab and send experiments for execution on it. <br/>
|
||||
Check out [this](guides/ide/google_colab.md) tutorial on how to run a ClearML Agent on Google Colab!
|
||||
|
||||
## Dynamic GPU Allocation
|
||||
|
||||
:::important
|
||||
@ -575,6 +504,82 @@ Notice that a minimum and maximum value of GPUs was specified for the `opportuni
|
||||
will pull a Task from the `opportunistic` queue and allocate up to 4 GPUs based on availability (i.e. GPUs not currently
|
||||
being used by other agents).
|
||||
|
||||
## Services Mode
|
||||
The ClearML Agent Services Mode executes an Agent that can execute multiple Tasks. This is useful for Tasks that are mostly
|
||||
idling, such as periodic cleanup services, or a [pipeline controller](references/sdk/automation_controller_pipelinecontroller.md).
|
||||
|
||||
Launch a service Task like any other Task, by enqueuing it to the appropriate queue.
|
||||
|
||||
:::note
|
||||
The default `clearml-server` configuration already runs a single `clearml-agent` in services mode that listens to the “services” queue.
|
||||
:::
|
||||
|
||||
To run a `clearml-agent` in services mode, run:
|
||||
```bash
|
||||
clearml-agent daemon --services-mode --queue services --create-queue --docker <docker_name> --cpu-only
|
||||
```
|
||||
:::note
|
||||
`services-mode` currently only supports Docker mode. Each service spins on its own Docker image.
|
||||
:::
|
||||
|
||||
:::warning
|
||||
Do not enqueue training or inference Tasks into the services queue. They will put an unnecessary load on the server.
|
||||
:::
|
||||
|
||||
### Setting Server Credentials
|
||||
|
||||
Self hosted [ClearML Server](deploying_clearml/clearml_server.md) comes by default with a services queue.
|
||||
By default, the server is open and does not require username and password, but it can be [password protected](deploying_clearml/clearml_server_security#user-access-security).
|
||||
In case it is password protected the services agent will need to be configured with server credentials (associated with a user).
|
||||
|
||||
To do that, set these environment variables on the ClearML Server machine with the appropriate credentials:
|
||||
```
|
||||
CLEARML_API_ACCESS_KEY
|
||||
CLEARML_API_SECRET_KEY
|
||||
```
|
||||
|
||||
## Exporting a Task into a Standalone Docker Container
|
||||
|
||||
### Task Container
|
||||
|
||||
Build a Docker container that when launched executes a specific experiment, or a clone (copy) of that experiment.
|
||||
|
||||
- Build a Docker container that at launch will execute a specific Task.
|
||||
```bash
|
||||
clearml-agent build --id <task-id> --docker --target <new-docker-name> --entry-point reuse_task
|
||||
```
|
||||
- Build a Docker container that at launch will clone a Task specified by Task ID, and will execute the newly cloned Task.
|
||||
```bash
|
||||
clearml-agent build --id <task-id> --docker --target <new-docker-name> --entry-point clone_task
|
||||
```
|
||||
- Run built Docker by executing:
|
||||
```bash
|
||||
docker run <new-docker-name>
|
||||
```
|
||||
|
||||
### Base Docker Container
|
||||
|
||||
Build a Docker container according to the execution environment of a specific Task.
|
||||
|
||||
```bash
|
||||
clearml-agent build --id <task-id> --docker --target <new-docker-name>
|
||||
```
|
||||
|
||||
|
||||
It's possible to add the Docker container as the base Docker image to a Task (experiment), using one of the following methods:
|
||||
|
||||
- Using the **ClearML Web UI** - See [Base Docker image](webapp/webapp_exp_tuning.md#base-docker-image) on the "Tuning
|
||||
Experiments" page.
|
||||
- In the **ClearML** configuration file - Use the **ClearML** configuration file [agent.default_docker](configs/clearml_conf.md#agentdefault_docker)
|
||||
options.
|
||||
|
||||
## Google Colab
|
||||
|
||||
ClearML Agent can run on a [google colab](https://colab.research.google.com/) instance. This helps users to leverage
|
||||
compute resources provided by google colab and send experiments for execution on it. <br/>
|
||||
Check out [this](guides/ide/google_colab.md) tutorial on how to run a ClearML Agent on Google Colab!
|
||||
|
||||
|
||||
## Scheduling working hours
|
||||
|
||||
:::important
|
||||
|
||||
Loading…
Reference in New Issue
Block a user