diff --git a/docs/clearml_agent.md b/docs/clearml_agent.md index 2885bfff..6e67a888 100644 --- a/docs/clearml_agent.md +++ b/docs/clearml_agent.md @@ -1,5 +1,5 @@ --- -title: ClearML Agent +title: ClearML Agent Overview --- **ClearML Agent** is a virtual environment and execution manager for DL / ML solutions on GPU machines. It integrates with the **ClearML Python Package** and ClearML Server to provide a full AI cluster solution.
@@ -33,7 +33,7 @@ automated workflows in one (or both) of the following ways: * By using the **ClearML Web UI** (without directly working with code), by enqueuing experiments to the queue that a **ClearML Agent** is listening to. -For more information, see [ClearML Agent Reference](references/clearml_agent_ref.md), +For more information, see [ClearML Agent Reference](clearml_agent/clearml_agent_ref.md), and [configuration options](configs/clearml_conf.md#agent-section). diff --git a/docs/clearml_agent/clearml_agent_build.md b/docs/clearml_agent/clearml_agent_build.md new file mode 100644 index 00000000..870df483 --- /dev/null +++ b/docs/clearml_agent/clearml_agent_build.md @@ -0,0 +1,141 @@ +--- +title: build +--- + +This reference page provides detailed information about ClearML Agent's `build` subcommand, which you can use to create +a worker environment without executing the experiment. + +## Syntax + +```bash +clearml-agent build [-h] --id TASK_ID [--target TARGET] + [--install-globally] + [--docker [DOCKER [DOCKER ...]]] [--force-docker] + [--python-version PYTHON_VERSION] + [--entry-point {reuse_task,clone_task}] [-O] + [--git-user GIT_USER] [--git-pass GIT_PASS] + [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}] + [--gpus GPUS] [--cpu-only] +``` + +## Arguments + +--- + +**`id`** +(*mandatory*) + +* Build a worker environment for this Task ID. + +--- + +**`cpu-only`** + +* Disable GPU access for the Docker container. + +--- + +**`docker`** + +* Docker mode. A Docker container that a worker will execute at launch. + + To specify the image name and optional arguments, either: + + * Use `--docker ` on the command line, or + * Use `--docker` on the command line, and specify the image name and arguments in the configuration file. + + Environment variable settings for Dockers: + + * `CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `gpus` flag inside the Docker container. This also allows you to execute **ClearML Agent** using Docker versions earlier than 19.03. + * `NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container. + * `CLEARML_AGENT_GIT_USER` and `CLEARML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution. + + To limit GPU visibility for Docker, set the `NVIDIA_VISIBLE_DEVICES` environment variable. + +--- + +**`entry-point`** + +* Used in conjunction with `--docker`, indicates how to run the Task specified by `task-id` on Docker startup. + + The `entry-point` options are: + + * `reuse` - Overwrite the existing Task data. + * `clone_task` - Clone the Task, and execute the cloned Task. + +--- + +**`force-docker`** + +* Force using the agent-specified docker image (either explicitly in the `--docker` argument or using the agent's + default docker image). If provided, the agent will not use any docker container information stored in the task itself + (default `False`) + + +--- + +**`git-pass`** + +* Git password for repository access. + +--- + +**`git-user`** + +* Git username for repository access. + +--- + +**`gpus`** + +* Specify the active GPUs for the Docker containers to use. These are the same GPUs set in the `NVIDIA_VISIBLE_DEVICES` environment variable. + + For example: + + * `--gpus 0` + * `--gpu 0,1,2` + * `--gpus all` + +--- + +**`h`, `help`** + +* Get help for this command. + +--- + +**`install-globally`** + +* Install the required Python packages before creating the virtual environment. Use `agent.package_manager.system_site_packages` to control the installation of the system packages. When `--docker` is used, `install-globally` is always true. + +--- + +**`log-level`** + +* SDK log level. The values are: + + * `DEBUG` + * `INFO` + * `WARN` + * `WARNING` + * `ERROR` + * `CRITICAL` + +--- + +**`python-version`** + +* Virtual environment Python version to use. + +--- + +**`O`** + +* Compile optimized pyc code (see python documentation). Repeat for more optimization. + +--- + +**`target`** + +* The target folder for the virtual environment and source code that will be used at launch. + diff --git a/docs/clearml_agent/clearml_agent_config.md b/docs/clearml_agent/clearml_agent_config.md new file mode 100644 index 00000000..937bbe5d --- /dev/null +++ b/docs/clearml_agent/clearml_agent_config.md @@ -0,0 +1,12 @@ +--- +title: config +--- + +This reference page provides detailed information about ClearML Agent's `config` subcommand, which you can use to +list your ClearML Agent configuration. + +## Syntax + +``` +clearml-agent config [-h] +``` \ No newline at end of file diff --git a/docs/clearml_agent/clearml_agent_daemon.md b/docs/clearml_agent/clearml_agent_daemon.md new file mode 100644 index 00000000..a0943c4c --- /dev/null +++ b/docs/clearml_agent/clearml_agent_daemon.md @@ -0,0 +1,223 @@ +--- +title: daemon +--- + +This reference page provides detailed information about ClearML Agent's `daemon` subcommand, which you can use to +run a worker, optionally in a Docker container, listening to a queue. + +## Syntax + +```bash +clearml-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]] [--order-fairness] + [--standalone-mode] [--services-mode [SERVICES_MODE]] + [--child-report-tags CHILD_REPORT_TAGS [CHILD_REPORT_TAGS ...]] + [--create-queue] [--detached] [--stop] [--dynamic-gpus] + [--uptime [UPTIME [UPTIME ...]]] [--downtime [DOWNTIME [DOWNTIME ...]]] + [--status] [--use-owner-token] [-O] + [--git-user GIT_USER] [--git-pass GIT_PASS] + [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}] + [--gpus GPUS] [--cpu-only] + [--docker [DOCKER [DOCKER ...]]] [--force-current-version] +``` + + +## Arguments + +--- + +**`child-report-tags`** + +List of tags to send with the status reports from the worker that executes a task. + +--- + +**`cpu-only`** + +* If running in Docker mode (see the `docker` option), disable GPU access for the Docker container or virtual environment. + +--- + +**`create-queue`** + +* If the queue name provided does not exist in the server, create and use it. + +--- + +**`detached`** + +* Run agent in the background. The `clearml-agent` command returns immediately. + +--- + +**`docker`** + +* Run in Docker mode. Execute the Task inside a Docker container. + + To specify the image name and optional arguments, either: + + * use `--docker ` on the command line, or + * use `--docker` on the command line, and specify the image name and arguments in the configuration file. + + Environment variable settings for Dockers: + + * `CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `gpus` flag inside the Docker container. This also allows you to execute **ClearML Agent** using Docker versions earlier than 19.03. + * `NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container. + * `ClearML_AGENT_GIT_USER` and `ClearML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution. + +--- + +**`downtime`** + +* Specify downtime for clearml-agent in ` ` format. + + For example, use `09-13 TUE` to set Tuesday's downtime to 09-13. + +:::info +* This feature is available under the ClearML Enterprise plan +* Make sure to have only one of uptime / downtime configuration and not both. +::: + +--- + +**`dynamic-gpus`** + +* Allow to dynamically allocate gpus based on queue properties, configure with `--queue =`. + + For example: `--dynamic-gpus --queue dual_gpus=2 single_gpu=1` + +:::info Enterprise Feature +This feature is available under the ClearML Enterprise plan +::: + + + +--- + +**`force-current-version`** + +* To use your current version of **ClearML Agent** when running in Docker mode (the `docker` argument is specified), + instead of the latest **ClearML Agent** version which is automatically installed, specify `force-current-version`. + + For example, if your current **ClearML Agent** version is `0.13.1`, but the latest version of **ClearML Agent** is + `0.13.3`, use `--force-current-version` and your Task will execute in the Docker container with **ClearML Agent** version `0.13.1`. + +--- + +**`foreground`** + +* Pipe full log to stdout/stderr. Do not use this option if running in background. + +--- + +**`git-pass`** + +* Git password for repository access. + +--- + +**`git-user`** + +* Git username for repository access. + +--- + +**`gpus`** + +* If running in Docker mode (see the `docker` option), specify the active GPUs for the Docker containers to use. + + These are the same GPUs set in the `NVIDIA_VISIBLE_DEVICES` environment variable. + + For example: + + * `--gpus 0` + * `--gpu 0,1,2` + * `--gpus all` + +--- + +**`h`, `help`** + +* Get help for this command. + +--- +**`log-level`** + +* SDK log level. The values are: + + * `DEBUG` + * `INFO` + * `WARN` + * `WARNING` + * `ERROR` + * `CRITICAL` + +--- + +**`O`** + +* Compile optimized pyc code (see python documentation). Repeat for more optimization. + +--- + +**`order-fairness`** + +* Pull from each queue in a round-robin order, instead of priority order. + +--- + +**`queue`** + +* Specify the queues which the worker is listening to. The values can be any combination of: + + * One or more queue IDs. + * One or more queue names. + * `default` indicating the default queue. + +--- + +**`services-mode`** + +* Launch multiple long-term docker services. Spin multiple, simultaneous Tasks, each in its own Docker container, on the same machine. Each Task will be registered + as a new node in the system, providing tracking and transparency capabilities. Start up and shutdown of each Docker is + verified. Use in CPU mode (`--cpu-only`), only. + +* To limit the number of simultaneous tasks run in services mode, pass the maximum number immediately after the +`--services-mode` option (e.g. `--services-mode 5`) + + +--- + +**`standalone-mode`** + +* Do not use any network connects. This assumes all requirements are pre-installed. + +--- + +**`status`** + +* Print the worker's schedule (uptime properties, server's runtime properties and listening queues) + + +--- + +**`stop`** + +* Terminate a running ClearML Agent, if other arguments are the same. If no additional arguments are provided, agents are + terminated in lexicographical order. + +--- + +**`uptime`** +* Specify uptime for clearml-agent in ` ` format. for example, use `17-20 TUE` to set Tuesday's + uptime to 17-20 + +:::info +* This feature is available under the ClearML Enterprise plan +* Make sure to have only one of uptime / downtime configuration and not both. +::: + +--- + +**`use-owner-token`** + +Generate and use the task owner's token for the execution of the task. diff --git a/docs/clearml_agent/clearml_agent_env_var.md b/docs/clearml_agent/clearml_agent_env_var.md new file mode 100644 index 00000000..9f289aff --- /dev/null +++ b/docs/clearml_agent/clearml_agent_env_var.md @@ -0,0 +1,31 @@ +--- +title: Environment Variables +--- + +This page lists the available environment variables for configuring ClearML Agent. + +:::info +ClearML's environment variables override the clearml.conf file, SDK, and [configuration vault](../webapp/webapp_profile.md#configuration-vault), +but can be overridden by command-line arguments. +::: + +|Name|Description| +|---|---| +|**CLEARML_DOCKER_IMAGE** | Default ClearML Agent docker image| +|**CLEARML_WORKER_NAME** | Sets the Worker's name| +|**CLEARML_WORKER_ID** | Sets the Worker ID| +|**CLEARML_CUDA_VERSION** | Sets the CUDA version to be used| +|**CLEARML_CUDNN_VERSION** | Sets the CUDNN version to be used| +|**CLEARML_CPU_ONLY** | Force CPU only mode| +|**CLEARML_DOCKER_SKIP_GPUS_FLAG**| Skips the GPUs flag (support for docker V18| +|**CLEARML_AGENT_DISABLE_SSH_MOUNT**| Disables the auto `.ssh` mount into the docker| +|**CLEARML_AGENT_GIT_USER** | Sets the Git user for ClearML Agent| +|**CLEARML_AGENT_GIT_PASS** | Sets the Git password for ClearML Agent| +|**CLEARML_AGENT_GIT_HOST** | Sets Git host (only sending login to this host)| +|**CLEARML_AGENT_EXEC_USER**| User for Agent executing tasks (root by default)| +|**CLEARML_AGENT_EXTRA_DOCKER_ARGS**| Overrides extra docker args configuration | +|**CLEARML_AGENT_EXTRA_PYTHON_PATH**| Sets extra python path| +|**CLEARML_AGENT_INITIAL_CONNECT_RETRY_OVERRIDE**| Overrides initial server connection behavior (true by default), allows explicit number to specify number of connect retries) | +|**CLEARML_AGENT_K8S_HOST_MOUNT / CLEARML_AGENT_DOCKER_HOST_MOUNT**| Specifies Agent's mount point for Docker / K8s| +|**CLEARML_AGENT_SKIP_PIP_VENV_INSTALL**| Skips Python virtual env installation on execute and provides a custom venv binary | +|**CLEARML_AGENT_VENV_CACHE_PATH**|Overrides venv cache folder configuration| \ No newline at end of file diff --git a/docs/clearml_agent/clearml_agent_execute.md b/docs/clearml_agent/clearml_agent_execute.md new file mode 100644 index 00000000..22c6205f --- /dev/null +++ b/docs/clearml_agent/clearml_agent_execute.md @@ -0,0 +1,130 @@ +--- +title: execute +--- + +This reference page provides detailed information about ClearML Agent's `execute` subcommand, which you can use to +build and execute an experiment without a queue. + +## Syntax + +```bash +clearml-agent execute [-h] --id TASK_ID [--log-file LOG_FILE] [--disable-monitoring] + [--full-monitoring] [--require-queue] + [--standalone-mode] [--docker [DOCKER [DOCKER ...]]] [--clone] + [-O] [--git-user GIT_USER] [--git-pass GIT_PASS] + [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}] + [--gpus GPUS] [--cpu-only] +``` + + +## Arguments + +--- + +**`id`** (*mandatory*) + +* The ID of the Task to build. + +--- + +**`clone`** + +* Clone the Task specified by `id`, and then execute that cloned Task. + +--- + +**`cpu-only`** +* Disable GPU access for the daemon, only use CPU in either docker or virtual environment. + + +--- + +**`docker`** + +* Run in Docker mode. Execute the Task inside a Docker container. + + To specify the image name and optional arguments, either: + + * use `--docker ` on the command line, or + * use `--docker` on the command line, and specify the default image name and arguments in the configuration file. + + Environment variable settings for Dockers containers: + + * `ClearML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `gpus` flag inside the Docker container. This also allows you to execute **ClearML Agent** using Docker versions earlier than 19.03. + * `NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container. + * `ClearML_AGENT_GIT_USER` and `ClearML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution. + +--- + +**`disable-monitoring`** + +* Disable logging and monitoring, except for stdout. + +--- + +**`full-monitoring`** + +* Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout. + +--- + +**`git-pass`** + +* Git password for repository access. + +--- + +**`git-user`** + +* Git username for repository access. + +--- + +**`gpus`** +* Specify active GPUs for the daemon to use (docker / virtual environment), Equivalent to setting + `NVIDIA_VISIBLE_DEVICES`. Examples: `--gpus 0` or `--gpu 0,1,2` or `--gpus all` + + +--- + +**`h`, `help`** + +* Get help for this command. + +--- + +**`log-file`** + +* The log file for Task execution output (stdout / stderr) to a text file. + +--- + +**`log-level`** + +* SDK log level. The values are: + + * `DEBUG` + * `INFO` + * `WARN` + * `WARNING` + * `ERROR` + * `CRITICAL` + +--- + +**`O`** + +* Compile optimized pyc code (see python documentation). Repeat for more optimization. + +--- + +**`require-queue`** + +* If the specified task is not queued (in any Queue), the execution will fail. (Used for 3rd party scheduler + integration, e.g. K8s, SLURM, etc.) + +--- + +**`standalone-mode`** + +* Do not use any network connects, assume everything is pre-installed diff --git a/docs/clearml_agent/clearml_agent_list.md b/docs/clearml_agent/clearml_agent_list.md new file mode 100644 index 00000000..495a9a07 --- /dev/null +++ b/docs/clearml_agent/clearml_agent_list.md @@ -0,0 +1,10 @@ +--- +title: list +--- + +This reference page provides detailed information about ClearML Agent's `list` subcommand, which you can use to list +information about all active workers. + +## Syntax + + clearml-agent list [-h] \ No newline at end of file diff --git a/docs/clearml_agent/clearml_agent_ref.md b/docs/clearml_agent/clearml_agent_ref.md new file mode 100644 index 00000000..2c663f22 --- /dev/null +++ b/docs/clearml_agent/clearml_agent_ref.md @@ -0,0 +1,13 @@ +--- +title: ClearML Agent Reference +--- + +The following reference pages provide detailed information about using ClearML Agent: + +* `clearml-agent` CLI commands: + * [build](clearml_agent_build.md) - Create a worker environment, without executing an experiment. + * [config](clearml_agent_config.md) - List your ClearML Agent configuration. + * [daemon](clearml_agent_daemon.md) - Run a worker daemon listening to a queue for Task (experiments) to execute. + * [execute](clearml_agent_execute.md) - Execute an experiment, locally without a queue. + * [list](clearml_agent_list.md) - List the current workers. +* [Environment variables](clearml_agent_env_var.md) diff --git a/docs/configs/configuring_clearml.md b/docs/configs/configuring_clearml.md index 1204537f..8c3552fe 100644 --- a/docs/configs/configuring_clearml.md +++ b/docs/configs/configuring_clearml.md @@ -17,7 +17,7 @@ from the vault are applied on top of the configuration specified in `clearml.con The different ClearML configuration methods take precedence as summarized in the following list (higher ordered methods override the lower ones): -1. Command-line arguments (e.g. [clearml-task](../apps/clearml_task.md#command-line-options), [clearml-agent](../references/clearml_agent_ref.md), +1. Command-line arguments (e.g. [clearml-task](../apps/clearml_task.md#command-line-options), [clearml-agent](../clearml_agent/clearml_agent_ref.md), [clearml-session](../apps/clearml_session.md#command-line-options), [clearml-data](../clearml_data/clearml_data_cli.md) arguments) 1. Environment variables diff --git a/docs/configs/env_vars.md b/docs/configs/env_vars.md index 290120bd..e9fc427a 100644 --- a/docs/configs/env_vars.md +++ b/docs/configs/env_vars.md @@ -2,6 +2,9 @@ title: Environment Variables --- +This page lists the available environment variables for configuring ClearML. See [here](../clearml_agent/clearml_agent_env_var.md) +for environment variables to configure ClearML Agent. + :::info ClearML's environment variables override the clearml.conf file, SDK, and [configuration vault](../webapp/webapp_profile.md#configuration-vault), but can be overridden by command-line arguments. @@ -42,24 +45,3 @@ Overrides Repository Auto-logging |**CLEARML_OFFLINE_MODE** | Sets Offline mode| |**CLEARML_NO_DEFAULT_SERVER** | Disables sending information to demo server when no HOST server is set| -## ClearML Agent Variables -|Name|Description| -|---|---| -|**CLEARML_DOCKER_IMAGE** | Default ClearML Agent docker image| -|**CLEARML_WORKER_NAME** | Sets the Worker's name| -|**CLEARML_WORKER_ID** | Sets the Worker ID| -|**CLEARML_CUDA_VERSION** | Sets the CUDA version to be used| -|**CLEARML_CUDNN_VERSION** | Sets the CUDNN version to be used| -|**CLEARML_CPU_ONLY** | Force CPU only mode| -|**CLEARML_DOCKER_SKIP_GPUS_FLAG**| Skips the GPUs flag (support for docker V18| -|**CLEARML_AGENT_DISABLE_SSH_MOUNT**| Disables the auto `.ssh` mount into the docker| -|**CLEARML_AGENT_GIT_USER** | Sets the Git user for ClearML Agent| -|**CLEARML_AGENT_GIT_PASS** | Sets the Git password for ClearML Agent| -|**CLEARML_AGENT_GIT_HOST** | Sets Git host (only sending login to this host)| -|**CLEARML_AGENT_EXEC_USER**| User for Agent executing tasks (root by default)| -|**CLEARML_AGENT_EXTRA_DOCKER_ARGS**| Overrides extra docker args configuration | -|**CLEARML_AGENT_EXTRA_PYTHON_PATH**| Sets extra python path| -|**CLEARML_AGENT_INITIAL_CONNECT_RETRY_OVERRIDE**| Overrides initial server connection behavior (true by default), allows explicit number to specify number of connect retries) | -|**CLEARML_AGENT_K8S_HOST_MOUNT / CLEARML_AGENT_DOCKER_HOST_MOUNT**| Specifies Agent's mount point for Docker / K8s| -|**CLEARML_AGENT_SKIP_PIP_VENV_INSTALL**| Skips Python virtual env installation on execute and provides a custom venv binary | -|**CLEARML_AGENT_VENV_CACHE_PATH**|Overrides venv cache folder configuration| \ No newline at end of file diff --git a/docs/deploying_clearml/clearml_config_for_clearml_server.md b/docs/deploying_clearml/clearml_config_for_clearml_server.md index 47a05499..76d5ab6a 100644 --- a/docs/deploying_clearml/clearml_config_for_clearml_server.md +++ b/docs/deploying_clearml/clearml_config_for_clearml_server.md @@ -33,7 +33,7 @@ by setting [configuration options](../configs/clearml_conf.md). CLEARML_CONFIG_FILE = MyOtherClearML.conf For more information about running experiments inside Docker containers, see [ClearML Agent Execution](../clearml_agent.md#execution) - and [ClearML Agent Reference](../references/clearml_agent_ref.md). + and [ClearML Agent Reference](../clearml_agent/clearml_agent_ref.md). diff --git a/docs/guides/ide/google_colab.md b/docs/guides/ide/google_colab.md index aa1c503a..3d9ede9c 100644 --- a/docs/guides/ide/google_colab.md +++ b/docs/guides/ide/google_colab.md @@ -53,7 +53,7 @@ and running, users can send Tasks to be executed on the Google Colab's HW. !clearml-agent daemon --queue default ``` - For additional options for running `clearml-agent`, see the [clearml-agent reference](../../references/clearml_agent_ref.md). + For additional options for running `clearml-agent`, see the [clearml-agent reference](../../clearml_agent/clearml_agent_ref.md). After cell 4 is executed, the worker should now appear in the [**Workers & Queues**](../../webapp/webapp_workers_queues.md) page of your server. Clone experiments and enqueue them to your hearts content! The `clearml-agent` will fetch diff --git a/docs/references/clearml_agent_ref.md b/docs/references/clearml_agent_ref.md deleted file mode 100644 index 86c54ca9..00000000 --- a/docs/references/clearml_agent_ref.md +++ /dev/null @@ -1,516 +0,0 @@ ---- -title: ClearML Agent Reference ---- -This reference page provides detailed information about **ClearML Agent** commands, including: - -* [build](#build) - Create a worker environment, without executing an experiment. -* [config](#config) - List your **ClearML Agent** configuration. -* [daemon](#daemon) - Run a worker daemon listening to a queue for Task (experiments) to execute. -* [execute](#execute) - Execute an experiment, locally without a queue. -* [list](#list) - List the current workers. - -## build - -Use the `build` subcommand to create a worker environment without executing the experiment. - -### Syntax - -```bash -clearml-agent build [-h] --id TASK_ID [--target TARGET] - [--install-globally] - [--docker [DOCKER [DOCKER ...]]] [--force-docker] - [--python-version PYTHON_VERSION] - [--entry-point {reuse_task,clone_task}] [-O] - [--git-user GIT_USER] [--git-pass GIT_PASS] - [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}] - [--gpus GPUS] [--cpu-only] -``` - -### Arguments - ---- - -**`id`** -(*mandatory*) - -* Build a worker environment for this Task ID. - ---- - -**`cpu-only`** - -* Disable GPU access for the Docker container. - ---- - -**`docker`** - -* Docker mode. A Docker container that a worker will execute at launch. - - To specify the image name and optional arguments, either: - - * Use `--docker ` on the command line, or - * Use `--docker` on the command line, and specify the image name and arguments in the configuration file. - - Environment variable settings for Dockers: - - * `CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `gpus` flag inside the Docker container. This also allows you to execute **ClearML Agent** using Docker versions earlier than 19.03. - * `NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container. - * `CLEARML_AGENT_GIT_USER` and `CLEARML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution. - - To limit GPU visibility for Docker, set the `NVIDIA_VISIBLE_DEVICES` environment variable. - ---- - -**`entry-point`** - -* Used in conjunction with `--docker`, indicates how to run the Task specified by `task-id` on Docker startup. - - The `entry-point` options are: - - * `reuse` - Overwrite the existing Task data. - * `clone_task` - Clone the Task, and execute the cloned Task. - ---- - -**`force-docker`** - -* Force using the agent-specified docker image (either explicitly in the `--docker` argument or using the agent's - default docker image). If provided, the agent will not use any docker container information stored in the task itself - (default `False`) - - ---- - -**`git-pass`** - -* Git password for repository access. - ---- - -**`git-user`** - -* Git username for repository access. - ---- - -**`gpus`** - -* Specify the active GPUs for the Docker containers to use. These are the same GPUs set in the `NVIDIA_VISIBLE_DEVICES` environment variable. - - For example: - - * `--gpus 0` - * `--gpu 0,1,2` - * `--gpus all` - ---- - -**`h`, `help`** - -* Get help for this command. - ---- - -**`install-globally`** - -* Install the required Python packages before creating the virtual environment. Use `agent.package_manager.system_site_packages` to control the installation of the system packages. When `--docker` is used, `install-globally` is always true. - ---- - -**`log-level`** - -* SDK log level. The values are: - - * `DEBUG` - * `INFO` - * `WARN` - * `WARNING` - * `ERROR` - * `CRITICAL` - ---- - -**`python-version`** - -* Virtual environment Python version to use. - ---- - -**`O`** - -* Compile optimized pyc code (see python documentation). Repeat for more optimization. - ---- - -**`target`** - -* The target folder for the virtual environment and source code that will be used at launch. - - -## config - -Use the `config` subcommand to list your **ClearML Agent** configuration. - -### Syntax - - clearml-agent config [-h] - - -## daemon - -Use the `daemon` subcommand to run a worker, optionally in a Docker container, listening to a queue. - -### Syntax - -```bash -clearml-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]] [--order-fairness] - [--standalone-mode] [--services-mode [SERVICES_MODE]] - [--child-report-tags CHILD_REPORT_TAGS [CHILD_REPORT_TAGS ...]] - [--create-queue] [--detached] [--stop] [--dynamic-gpus] - [--uptime [UPTIME [UPTIME ...]]] [--downtime [DOWNTIME [DOWNTIME ...]]] - [--status] [--use-owner-token] [-O] - [--git-user GIT_USER] [--git-pass GIT_PASS] - [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}] - [--gpus GPUS] [--cpu-only] - [--docker [DOCKER [DOCKER ...]]] [--force-current-version] -``` - - -### Arguments - ---- - -**`child-report-tags`** - -List of tags to send with the status reports from the worker that executes a task. - ---- - -**`cpu-only`** - -* If running in Docker mode (see the `docker` option), disable GPU access for the Docker container or virtual environment. - ---- - -**`create-queue`** - -* If the queue name provided does not exist in the server, create and use it. - ---- - -**`detached`** - -* Run agent in the background. The `clearml-agent` command returns immediately. - ---- - -**`docker`** - -* Run in Docker mode. Execute the Task inside a Docker container. - - To specify the image name and optional arguments, either: - - * use `--docker ` on the command line, or - * use `--docker` on the command line, and specify the image name and arguments in the configuration file. - - Environment variable settings for Dockers: - - * `CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `gpus` flag inside the Docker container. This also allows you to execute **ClearML Agent** using Docker versions earlier than 19.03. - * `NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container. - * `ClearML_AGENT_GIT_USER` and `ClearML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution. - ---- - -**`downtime`** - -* Specify downtime for clearml-agent in ` ` format. - - For example, use `09-13 TUE` to set Tuesday's downtime to 09-13. - -:::info -* This feature is available under the ClearML Enterprise plan -* Make sure to have only one of uptime / downtime configuration and not both. -::: - ---- - -**`dynamic-gpus`** - -* Allow to dynamically allocate gpus based on queue properties, configure with `--queue =`. - - For example: `--dynamic-gpus --queue dual_gpus=2 single_gpu=1` - -:::info Enterprise Feature -This feature is available under the ClearML Enterprise plan -::: - - - ---- - -**`force-current-version`** - -* To use your current version of **ClearML Agent** when running in Docker mode (the `docker` argument is specified), - instead of the latest **ClearML Agent** version which is automatically installed, specify `force-current-version`. - - For example, if your current **ClearML Agent** version is `0.13.1`, but the latest version of **ClearML Agent** is - `0.13.3`, use `--force-current-version` and your Task will execute in the Docker container with **ClearML Agent** version `0.13.1`. - ---- - -**`foreground`** - -* Pipe full log to stdout/stderr. Do not use this option if running in background. - ---- - -**`git-pass`** - -* Git password for repository access. - ---- - -**`git-user`** - -* Git username for repository access. - ---- - -**`gpus`** - -* If running in Docker mode (see the `docker` option), specify the active GPUs for the Docker containers to use. - - These are the same GPUs set in the `NVIDIA_VISIBLE_DEVICES` environment variable. - - For example: - - * `--gpus 0` - * `--gpu 0,1,2` - * `--gpus all` - ---- - -**`h`, `help`** - -* Get help for this command. - ---- -**`log-level`** - -* SDK log level. The values are: - - * `DEBUG` - * `INFO` - * `WARN` - * `WARNING` - * `ERROR` - * `CRITICAL` - ---- - -**`O`** - -* Compile optimized pyc code (see python documentation). Repeat for more optimization. - ---- - -**`order-fairness`** - -* Pull from each queue in a round-robin order, instead of priority order. - ---- - -**`queue`** - -* Specify the queues which the worker is listening to. The values can be any combination of: - - * One or more queue IDs. - * One or more queue names. - * `default` indicating the default queue. - ---- - -**`services-mode`** - -* Launch multiple long-term docker services. Spin multiple, simultaneous Tasks, each in its own Docker container, on the same machine. Each Task will be registered - as a new node in the system, providing tracking and transparency capabilities. Start up and shutdown of each Docker is - verified. Use in CPU mode (`--cpu-only`), only. - -* To limit the number of simultaneous tasks run in services mode, pass the maximum number immediately after the -`--services-mode` option (e.g. `--services-mode 5`) - - ---- - -**`standalone-mode`** - -* Do not use any network connects. This assumes all requirements are pre-installed. - ---- - -**`status`** - -* Print the worker's schedule (uptime properties, server's runtime properties and listening queues) - - ---- - -**`stop`** - -* Terminate a running ClearML Agent, if other arguments are the same. If no additional arguments are provided, agents are - terminated in lexicographical order. - ---- - -**`uptime`** -* Specify uptime for clearml-agent in ` ` format. for example, use `17-20 TUE` to set Tuesday's - uptime to 17-20 - -:::info -* This feature is available under the ClearML Enterprise plan -* Make sure to have only one of uptime / downtime configuration and not both. -::: - ---- - -**`use-owner-token`** - -Generate and use the task owner's token for the execution of the task. - -## execute - -Use the `execute` subcommand to build and execute an experiment without a queue. - -### Syntax - -```bash -clearml-agent execute [-h] --id TASK_ID [--log-file LOG_FILE] [--disable-monitoring] - [--full-monitoring] [--require-queue] - [--standalone-mode] [--docker [DOCKER [DOCKER ...]]] [--clone] - [-O] [--git-user GIT_USER] [--git-pass GIT_PASS] - [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}] - [--gpus GPUS] [--cpu-only] -``` - - -### Arguments - ---- - -**`id`** (*mandatory*) - -* The ID of the Task to build. - ---- - -**`clone`** - -* Clone the Task specified by `id`, and then execute that cloned Task. - ---- - -**`cpu-only`** -* Disable GPU access for the daemon, only use CPU in either docker or virtual environment. - - ---- - -**`docker`** - -* Run in Docker mode. Execute the Task inside a Docker container. - - To specify the image name and optional arguments, either: - - * use `--docker ` on the command line, or - * use `--docker` on the command line, and specify the default image name and arguments in the configuration file. - - Environment variable settings for Dockers containers: - - * `ClearML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `gpus` flag inside the Docker container. This also allows you to execute **ClearML Agent** using Docker versions earlier than 19.03. - * `NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container. - * `ClearML_AGENT_GIT_USER` and `ClearML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution. - ---- - -**`disable-monitoring`** - -* Disable logging and monitoring, except for stdout. - ---- - -**`full-monitoring`** - -* Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout. - ---- - -**`git-pass`** - -* Git password for repository access. - ---- - -**`git-user`** - -* Git username for repository access. - ---- - -**`gpus`** -* Specify active GPUs for the daemon to use (docker / virtual environment), Equivalent to setting - `NVIDIA_VISIBLE_DEVICES`. Examples: `--gpus 0` or `--gpu 0,1,2` or `--gpus all` - - ---- - -**`h`, `help`** - -* Get help for this command. - ---- - -**`log-file`** - -* The log file for Task execution output (stdout / stderr) to a text file. - ---- - -**`log-level`** - -* SDK log level. The values are: - - * `DEBUG` - * `INFO` - * `WARN` - * `WARNING` - * `ERROR` - * `CRITICAL` - ---- - -**`O`** - -* Compile optimized pyc code (see python documentation). Repeat for more optimization. - ---- - -**`require-queue`** - -* If the specified task is not queued (in any Queue), the execution will fail. (Used for 3rd party scheduler - integration, e.g. K8s, SLURM, etc.) - ---- - -**`standalone-mode`** - -* Do not use any network connects, assume everything is pre-installed - - -## list - -Use the `list` subcommand to list information about all workers - -### Syntax - - clearml-agent list [-h] \ No newline at end of file diff --git a/docusaurus.config.js b/docusaurus.config.js index c2a10fbb..c9f1f4e7 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -88,10 +88,6 @@ module.exports = { label: 'Server API', to: '/docs/references/api/index', }, - { - label: 'Agent Reference', - to: '/docs/references/clearml_agent_ref', - }, ], }, { diff --git a/sidebars.js b/sidebars.js index 3323931a..8e8e19eb 100644 --- a/sidebars.js +++ b/sidebars.js @@ -16,7 +16,19 @@ module.exports = { {'ClearML Fundamentals': ['fundamentals/projects', 'fundamentals/task', 'fundamentals/hyperparameters', 'fundamentals/artifacts', 'fundamentals/logger', 'fundamentals/agents_and_queues', 'fundamentals/hpo']}, {'ClearML SDK': ['clearml_sdk/clearml_sdk', 'clearml_sdk/task_sdk', 'clearml_sdk/model_sdk', 'clearml_sdk/apiclient_sdk']}, - 'clearml_agent', + {'ClearML Agent': + [ + 'clearml_agent', + {'Reference': + [ + 'clearml_agent/clearml_agent_ref', 'clearml_agent/clearml_agent_build', + 'clearml_agent/clearml_agent_config', 'clearml_agent/clearml_agent_daemon', + 'clearml_agent/clearml_agent_execute', 'clearml_agent/clearml_agent_list', + 'clearml_agent/clearml_agent_env_var' + ] + } + ] + }, {'ClearML Pipelines':['pipelines/pipelines', {"Building Pipelines": ['pipelines/pipelines_sdk_tasks', 'pipelines/pipelines_sdk_function_decorators']