ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-02-07 13:21:46 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Clarify ClearML Agent modes (#685)

Browse Source
This commit is contained in:
pollfly 2023-10-03 10:44:37 +03:00 committed by GitHub
parent ffb1d51072
commit 61806b5178
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 35 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
docs/clearml_agent.md
View File

@ -340,40 +340,37 @@ Run a `clearml-agent` daemon in foreground mode, sending all output to the conso
clearml-agent daemon --queue default --foreground
```
## Execution Environments
ClearML Agent supports executing tasks in multiple environments.
ClearML Agent has two primary execution modes: [Virtual Environment Mode](#virtual-environment-mode) and [Docker Mode](#docker-mode).
### PIP Mode
By default, ClearML Agent works in PIP Mode, in which it uses [pip](https://en.wikipedia.org/wiki/Pip_(package_manager))
as the package manager. When ClearML runs, it will create a virtual environment
(or [reuse an existing one](clearml_agent.md#virtual-environment-reuse)).
Task dependencies (Python packages) will be installed in the virtual environment.
### Virtual Environment Mode
### Conda Mode
This mode is similar to the PIP mode but uses [Conda](https://docs.conda.io/en/latest/) as the package
manager. To enable Conda mode, edit the `clearml.conf` file, and modify the `type: pip` to `type: conda` in the “package_manager” section.
If extra conda channels are needed, look for “conda_channels” under “package_manager”, and add the missing channel.
In Virtual Environment Mode, the agent creates a virtual environment for the experiment, installs the required Python
packages based on the task specification, clones the code repository, applies the uncommitted changes and finally
executes the code while monitoring it. This mode uses smart caching so packages and environments can be reused over
multiple tasks (see [Virtual Environment Reuse](#virtual-environment-reuse)).
### Poetry Mode
This mode is similar to the PIP mode but uses [Poetry](https://python-poetry.org/) as the package manager.
To enable Poetry mode, edit the `clearml.conf` file, and modify the `type: pip` to `type: poetry` in the “package_manager”
section.
ClearML Agent supports working with one of the following package managers:
* [`pip`](https://en.wikipedia.org/wiki/Pip_(package_manager)) (default)
* [`conda`](https://docs.conda.io/en/latest/)
* [`poetry`](https://python-poetry.org/)
To change the package manager used by the agent, edit the [`package_manager.type`](configs/clearml_conf.md#agentpackagemanager)
field in the of the `clearml.conf`. If extra channels are needed for `conda`, add the missing channels in the
`package_manager.conda_channels` field in the `clearml.conf`.
:::note Using Poetry with Pyenv
Some versions of poetry (using `install-poetry.py`) do not respect `pyenv global`.
If you are using pyenv to control the environment where you use ClearML Agent, you can:
* Use poetry v1.2 and above (which [fixes this issue](https://github.com/python-poetry/poetry/issues/5077))
* Use poetry v1.2 and above (which fixes [this issue](https://github.com/python-poetry/poetry/issues/5077))
* Install poetry with the deprecated `get-poetry.py` installer
:::
### Docker Mode
:::note
Docker Mode is only supported in linux.<br/>
Docker Mode requires docker service v19.03 or higher installed.
:::note notes
* Docker Mode is only supported in linux.
* Docker Mode requires docker service v19.03 or higher installed.
:::
When executing the ClearML Agent in Docker mode, it will:
@ -401,7 +398,7 @@ clearml-agent daemon --queue <execution_queue_to_pull_from> --docker [optional d
```
To use the current `clearml-agent` version in the Docker container, instead of the latest `clearml-agent` version that is
automatically installed, run:
automatically installed, pass the `--force-current-version` flag:
```bash
clearml-agent daemon --queue default --docker --force-current-version
```

30
docs/fundamentals/agents_and_queues.md
View File

@ -77,21 +77,23 @@ reuse machines without the need for any dedicated containers or images.
Agents can be deployed bare-metal, with multiple instances allocating
specific GPUs to the agents. They can also be deployed as dockers in a Kubernetes cluster.
The Agent has three running modes:
- Docker Mode: The agent spins a docker image based on the Task’s definition then inside the docker the agent will clone
the specified repository/code, apply the original execution’s uncommitted changes, install the required python packages
and start executing the code while monitoring it.
- Virtual Environment Mode: The agent creates a new virtual environment for the experiment, installs the required python
packages based on the Task specification, clones the code repository, applies the uncommitted changes and finally
executes the code while monitoring it.
- Conda Environment Mode: Similar to the Virtual Environment mode, only instead of using pip, it uses conda install and
pip combination. Notice this mode is quite brittle due to the Conda package version support table.
The Agent supports the following running modes:
* **Virtual Environment Mode** - The agent creates a new virtual environment for the experiment, installs the required
python packages based on the Task specification, clones the code repository, applies the uncommitted changes and
finally executes the code while monitoring it. This mode uses smart caching so packages and environments can be reused
over multiple tasks (see [Virtual Environment Reuse](../clearml_agent.md#virtual-environment-reuse)).
:::tip Agents and virtual environments
An agent that runs in Virtual Environment Mode or Conda Environment Mode needs to create virtual environments, and
it can't do that when running from a virtual environment. You can run agents in these modes with
system Python.
:::
ClearML Agent supports using the following package managers: `pip` (default), `conda`, `poetry`.
:::tip Agents and virtual environments
An agent that runs in Virtual Environment Mode needs to create virtual environments, and
it can't do that when running from a virtual environment. You can run agents in these modes with
system Python.
:::
* **Docker Mode** - The agent spins a Docker image based on the Task's definitions. Inside the Docker the agent clones
the specified repository/code, applies the original execution's uncommitted changes, sets up the Python environment and
required packages, and starts executing the code while monitoring it.
## Services Mode