mirror of
https://github.com/clearml/clearml-docs
synced 2025-04-15 05:04:31 +00:00
Terminology update (#1035)
This commit is contained in:
pollfly
GitHub
parent
4667c3d194
commit
085841ab0d
@ -41,7 +41,7 @@ error, you are good to go.
|
||||
clearml-session
|
||||
```
|
||||
|
||||
You can add flags to set a Docker image, the remote SSH port, JupyterLab/VS Code versions, and more. See [CLI options](#command-line-options)
|
||||
You can add flags to set a container image, the remote SSH port, JupyterLab/VS Code versions, and more. See [CLI options](#command-line-options)
|
||||
for all configuration options.
|
||||
|
||||
`clearml-session` creates a new [Task](../fundamentals/task.md) that is responsible for setting up the SSH and
|
||||
@ -57,8 +57,8 @@ error, you are good to go.
|
||||
launches it.
|
||||
|
||||
1. Once the agent finishes the initial setup of the interactive Task, the local `cleaml-session` connects to the host
|
||||
machine via SSH, and tunnels both SSH and IDE over the SSH connection. If a Docker is specified, the
|
||||
IDE environment runs inside the Docker.
|
||||
machine via SSH, and tunnels both SSH and IDE over the SSH connection. If a container is specified, the
|
||||
IDE environment runs inside of it.
|
||||
|
||||
1. The CLI outputs access links to the remote JupyterLab and VS Code sessions:
|
||||
|
||||
@ -106,8 +106,8 @@ To connect to an existing session:
|
||||
1. Click on the JupyterLab / VS Code link that is outputted, or connect directly to the SSH session
|
||||
|
||||
## Features
|
||||
### Running in Docker
|
||||
To run a session inside a Docker container, use the `--docker` flag and enter the docker image to use in the interactive
|
||||
### Running in a Container
|
||||
To run a session inside a container, use the `--docker` flag and enter the image to use in the interactive
|
||||
session.
|
||||
|
||||
### Kubernetes Support
|
||||
@ -224,8 +224,8 @@ clearml-session --continue-session <session_id> --store-workspace ~/workspace
|
||||
| `--disable-fingerprint-check` | If set, bypass the remote SSH server fingerprint verification process | `none` |
|
||||
| `--disable-session-cleanup` | If `True`, previous interactive sessions are not deleted | `false`|
|
||||
| `--disable-store-defaults` | If set, do not store current setup as new default configuration| `none`|
|
||||
| `--docker`| Select the docker image to use in the interactive session |`nvidia/cuda:11.6.2-runtime-ubuntu20.04` or previously used docker image|
|
||||
| `--docker-args` | Add additional arguments for the docker image to use in the interactive session | `none` or the previously used docker-args |
|
||||
| `--docker`| Select the image to use in the interactive session |`nvidia/cuda:11.6.2-runtime-ubuntu20.04` or previously used image|
|
||||
| `--docker-args` | Add additional arguments for the docker image to use in the interactive session | `none` or the previously used `docker-args` |
|
||||
| `--force_dropbear`| Force using `dropbear` instead of SSHd |`none`|
|
||||
| `--git-credentials` | If `True`, local `.git-credentials` file is sent to the interactive session.| `false`|
|
||||
| `--init-script` | Specify a BASH init script file to be executed when the interactive session is being set up | `none` or previously entered BASH script |
|
||||
|
||||
@ -18,7 +18,7 @@ line arguments, Python module dependencies, and a requirements.txt file!
|
||||
## How Does ClearML Task Work?
|
||||
|
||||
1. Execute `clearml-task`, specifying the ClearML target project and task name, along with your script (and repository / commit / branch).
|
||||
Optionally, specify an execution queue and Docker image to use.
|
||||
Optionally, specify an execution queue and container image to use.
|
||||
1. `clearml-task` does its magic! It creates a new [ClearML Task](../fundamentals/task.md),
|
||||
and, if so directed, enqueues it for execution by a ClearML Agent.
|
||||
1. While the Task is running on the remote machine, all its console outputs are logged in real-time, alongside your
|
||||
@ -26,9 +26,9 @@ line arguments, Python module dependencies, and a requirements.txt file!
|
||||
(a link to your task details page in the ClearML Web UI is printed as ClearML Task creates the task).
|
||||
|
||||
## Execution Configuration
|
||||
### Docker
|
||||
Specify a Docker container to run the code in with the `--docker <docker_image>` option.
|
||||
The ClearML Agent pulls it from Docker Hub or a Docker artifactory automatically.
|
||||
### Container
|
||||
Specify a container to run the code in with the `--docker <image>` option.
|
||||
The ClearML Agent pulls it from Docker Hub or a container artifactory automatically.
|
||||
|
||||
### Package Dependencies
|
||||
`clearml-task` automatically finds the `requirements.txt` file in remote repositories.
|
||||
@ -61,8 +61,8 @@ errors in identifying the correct default branch.
|
||||
| `--branch` | Select repository branch / tag. By default, latest commit from the master branch | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--commit` | Select commit ID to use. By default, latest commit, or local commit ID when using local repository | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--cwd` | Working directory to launch the script from. Relative to repo root or local `--folder` | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--docker` | Select the Docker image to use in the remote task | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--docker_bash_setup_script` | Add a bash script to be executed inside the Docker before setting up the task's environment | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--docker` | Select the container image to use in the remote task | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--docker_bash_setup_script` | Add a bash script to be executed inside the container before setting up the task's environment | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--docker_args` | Add Docker arguments. Pass a single string in the following format: `--docker_args "<argument_string>"`. For example: `--docker_args "-v some_dir_1:other_dir_1 -v some_dir_2:other_dir_2"` | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--folder` | Execute the code from a local folder. Notice, it assumes a git repository already exists. Current state of the repo (commit ID and uncommitted changes) is logged and replicated on the remote machine | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--import-offline-session`| Specify the path to the offline session you want to import.| <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
|
||||
@ -29,7 +29,7 @@ The preceding diagram demonstrates a typical flow where an agent executes a task
|
||||
|
||||
1. Enqueue a task for execution on the queue.
|
||||
1. The agent pulls the task from the queue.
|
||||
1. The agent launches a docker container in which to run the task's code.
|
||||
1. The agent launches a container in which to run the task's code.
|
||||
1. The task's execution environment is set up:
|
||||
1. Execute any custom setup script configured.
|
||||
1. Install any required system packages.
|
||||
@ -39,8 +39,8 @@ The preceding diagram demonstrates a typical flow where an agent executes a task
|
||||
1. The task's script/code is executed.
|
||||
|
||||
:::note Python Version
|
||||
ClearML Agent uses the Python version available in the environment or docker in which it executes the code. It does not
|
||||
install Python, so make sure to use a docker or environment with the version you need.
|
||||
ClearML Agent uses the Python version available in the environment or container in which it executes the code. It does not
|
||||
install Python, so make sure to use a container or environment with the version you need.
|
||||
:::
|
||||
|
||||
While the agent is running, it continuously reports system metrics to the ClearML Server (these can be monitored in the
|
||||
|
||||
@ -39,8 +39,7 @@ clearml-agent build --id <task-id> --docker --target <new-docker-name>
|
||||
|
||||
You can add the Docker container as the base Docker image to a task, using one of the following methods:
|
||||
|
||||
- Using the **ClearML Web UI** - See [Base Docker image](../webapp/webapp_exp_tuning.md#base-docker-image) on the "Tuning
|
||||
Tasks" page.
|
||||
- Using the **ClearML Web UI** - See [Default Container](../webapp/webapp_exp_tuning.md#default-container).
|
||||
- In the ClearML configuration file - Use the ClearML configuration file [`agent.default_docker`](../configs/clearml_conf.md#agentdefault_docker)
|
||||
options.
|
||||
|
||||
|
||||
@ -26,7 +26,7 @@ If you are using pyenv to control the environment where you use ClearML Agent, y
|
||||
* Install poetry with the deprecated `get-poetry.py` installer
|
||||
:::
|
||||
|
||||
## Docker Mode
|
||||
## Docker Mode
|
||||
:::note notes
|
||||
* Docker Mode is only supported in Linux.
|
||||
* Docker Mode requires docker service v19.03 or higher installed.
|
||||
|
||||
@ -3,7 +3,7 @@ title: Executable Task Containers
|
||||
---
|
||||
|
||||
This tutorial demonstrates using [`clearml-agent`](../../clearml_agent.md)'s [`build`](../../clearml_agent/clearml_agent_ref.md#build)
|
||||
command to package a task into an executable container. In this example, you will build a Docker image that, when
|
||||
command to package a task into an executable container. In this example, you will build a Container image that, when
|
||||
run, will automatically execute the [keras_tensorboard.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/keras/keras_tensorboard.py)
|
||||
script.
|
||||
|
||||
@ -44,8 +44,8 @@ script.
|
||||
If the container will not make use of a GPU, add the `--cpu-only` flag.
|
||||
:::
|
||||
|
||||
This command will create a Docker container, set up with the execution environment for this task in the
|
||||
specified `--target` folder. When the Docker container is launched, it will clone the task specified with `id` and
|
||||
This command will create a container, set up with the execution environment for this task in the
|
||||
specified `--target` folder. When the container is launched, it will clone the task specified with `id` and
|
||||
execute the clone (as designated by the `--entry-point` parameter).
|
||||
|
||||
1. Run the Docker, pointing to the new container:
|
||||
|
||||
@ -3,7 +3,7 @@ title: Task Environment Containers
|
||||
---
|
||||
|
||||
This tutorial demonstrates using [`clearml-agent`](../../clearml_agent.md)'s [`build`](../../clearml_agent/clearml_agent_ref.md#build)
|
||||
command to build a Docker container replicating the execution environment of an existing task. ClearML Agents can make
|
||||
command to build a container replicating the execution environment of an existing task. ClearML Agents can make
|
||||
use of such containers to execute tasks without having to set up their environment every time.
|
||||
|
||||
A use case for this would be manual hyperparameter optimization, where a base task can be used to create a container to
|
||||
@ -36,7 +36,7 @@ be used when running optimization tasks.
|
||||
```
|
||||
This ID will be used in the following section.
|
||||
|
||||
## Building the Docker Container
|
||||
## Building the Container
|
||||
|
||||
Execute the following command to build the container. Input the ID of the task created above.
|
||||
```console
|
||||
@ -56,7 +56,7 @@ Committing docker container to: new_docker
|
||||
sha256:460453b93ct1989fd1c6637c236e544031c4d378581433fc0b961103ce206af1
|
||||
```
|
||||
|
||||
## Using the New Docker Container
|
||||
## Using the New Container
|
||||
Make use of the container you've just built by having a ClearML agent make use of it for executing a new task:
|
||||
|
||||
1. In the [ClearML Web UI](../../webapp/webapp_overview.md), go to the "examples" project, "Keras with TensorBoard
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
title: Extra Docker Shell Script
|
||||
---
|
||||
|
||||
When using `clearml-agent`, an agent recreates an entire execution environment, be it by pulling the docker container or
|
||||
When using `clearml-agent`, an agent recreates an entire execution environment, be it by pulling a container or
|
||||
installing specified packages, and then executes the code on a remote machine. The Agent takes into account required Python packages,
|
||||
but sometimes, when using a Docker container, a user may need to use additional, non-Python tools.
|
||||
|
||||
|
||||
@ -27,7 +27,7 @@ clearml-session --docker nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 --packages
|
||||
|
||||
This sets the following arguments:
|
||||
|
||||
* `--docker nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04` - Docker image
|
||||
* `--docker nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04` - Container image
|
||||
|
||||
* `--packages "clearml" "tensorflow>=2.2" "keras"` - Required Python packages
|
||||
|
||||
@ -39,7 +39,7 @@ name is `DevOps`.
|
||||
:::
|
||||
|
||||
After launching the command, the `clearml-agent` listening to the `default` queue spins a remote Jupyter environment with
|
||||
the specifications. It will automatically connect to the docker on the remote machine.
|
||||
the specifications. It will automatically connect to the container on the remote machine.
|
||||
|
||||
The console should display the session's configuration details:
|
||||
|
||||
|
||||
@ -51,8 +51,8 @@ to open the app's instance launch form.
|
||||
* Repository
|
||||
* Branch
|
||||
* Commit
|
||||
* **Docker**
|
||||
* Image - Docker image used to run the IDE in
|
||||
* **Container**
|
||||
* Image - Container image used to run the IDE in
|
||||
* Docker arguments - `docker run` arguments, as a single string
|
||||
* **Extra Packages** - Extra Python packages to be installed
|
||||
* **Queue** - The queue serviced by the ClearML Agent that will execute the Jupyter Lab session
|
||||
|
||||
@ -68,8 +68,8 @@ values from the file, which can be modified before launching the app instance
|
||||
* Repository
|
||||
* Branch
|
||||
* Commit
|
||||
* **Docker** - Input details to run the session in Docker container
|
||||
* Image - Docker image to launch
|
||||
* **Container** - Input details to run the session in Docker container
|
||||
* Image - Container image to launch
|
||||
* Docker Arguments - Additional arguments for the Docker image
|
||||
* Init Script - Bash script that is executed upon container boot (comments are supported only at the beginning of the
|
||||
line)
|
||||
|
||||
@ -51,8 +51,8 @@ to open the app's instance launch form.
|
||||
* Repository
|
||||
* Branch
|
||||
* Commit
|
||||
* **Docker**
|
||||
* Image - Docker image used to run the IDE in
|
||||
* **Container**
|
||||
* Image - container image used to run the IDE in
|
||||
* Docker arguments - `docker run` arguments, as a single string
|
||||
* **Queue** - The queue serviced by the ClearML Agent that will execute the VS Code session
|
||||
* **Maximum idle time** (hours) - Maximum time of inactivity, after which the session will shut down. Configure idleness
|
||||
|
||||
@ -27,7 +27,7 @@ Disable for default desktop scale.
|
||||
* **Block running user's scripts in the browser** - Block any user and 3rd party scripts from running anywhere in the
|
||||
WebApp. Note that if enabled, the WebApp will not display debug samples, [Hyper-Dataset frame previews](../../hyperdatasets/previews.md),
|
||||
and embedded resources in [reports](../webapp_reports.md).
|
||||
* **Hide specific container arguments** - Specify which Docker environment variable values should be hidden in logs.
|
||||
* **Hide specific container arguments** - Specify which container environment variable values should be hidden in logs.
|
||||
When printed, the variable values are replaced with `********`. By default, `CLEARML_API_SECRET_KEY`, `CLEARML_AGENT_GIT_PASS`,
|
||||
`AWS_SECRET_ACCESS_KEY`, and `AZURE_STORAGE_KEY` values are redacted. To modify the hidden container argument list, click **Edit**.
|
||||
|
||||
|
||||
@ -39,7 +39,7 @@ Web UI**, edit any of the following
|
||||
|
||||
* [Source code](#source-code)
|
||||
* [Output destination for artifacts](#output-destination)
|
||||
* [Base Docker image](#base-docker-image)
|
||||
* [Default container](#default-container)
|
||||
* [Hyperparameters](#hyperparameters) - Parameters, TensorFlow Definitions, command line options, environment variables, and user-defined properties
|
||||
|
||||
:::note
|
||||
@ -71,23 +71,22 @@ and/or Reset functions.
|
||||
|
||||
|
||||
|
||||
#### Base Docker Image
|
||||
Select a pre-configured Docker that **ClearML Agent** will use to remotely execute this task (see [Building Docker containers](../clearml_agent/clearml_agent_docker.md)).
|
||||
#### Default Container
|
||||
Select a pre-configured container that the [ClearML Agent](../clearml_agent.md) will use to remotely execute this task (see [Building Docker containers](../clearml_agent/clearml_agent_docker.md)).
|
||||
|
||||
**To add, change, or delete a base Docker image:**
|
||||
**To add, change, or delete a default container:**
|
||||
|
||||
* In **EXECUTION** **>** **AGENT CONFIGURATION** **>** **BASE DOCKER IMAGE** **>** hover **>** **EDIT** **>**
|
||||
Enter the base Docker image.
|
||||
* In **EXECUTION** **>** **CONTAINER** **>** hover **>** **EDIT** **>**
|
||||
Enter the default container image.
|
||||
|
||||
:::important
|
||||
For a ClearML Agent to execute the task in a container, the agent must be running in
|
||||
Docker mode:
|
||||
[Docker Mode](../clearml_agent/clearml_agent_execution_env.md#docker-mode):
|
||||
|
||||
```bash
|
||||
clearml-agent daemon --queue <execution_queue_to_pull_from> --docker [optional default docker image to use]
|
||||
clearml-agent daemon --queue <execution_queue_to_pull_from> --docker [optional default container image to use]
|
||||
```
|
||||
|
||||
For more information, see [Docker Mode](../clearml_agent/clearml_agent_execution_env.md#docker-mode).
|
||||
:::
|
||||
|
||||
#### Output Destination
|
||||
|
||||
Loading…
Reference in New Issue
Block a user