mirror of
https://github.com/clearml/clearml-docs
synced 2025-03-12 15:08:55 +00:00
Edit ClearML Task doc and tutorial (#172)
This commit is contained in:
pollfly
GitHub
parent
9727045603
commit
a7e9347bc2
@ -2,38 +2,46 @@
|
||||
title: ClearML Task
|
||||
---
|
||||
|
||||
ClearML Task is ClearML's Zero Code Integration Module. Using only the command line and **zero** additional lines of code,
|
||||
you can easily track your work and integrate ClearML with your existing code.
|
||||
Using only the command line and **zero** additional lines of code, easily track your work and integrate ClearML with your
|
||||
existing code.
|
||||
|
||||
`clearml-task` automatically integrates ClearML into any script or **any** python repository. `clearml-task` has the option
|
||||
to send the task to a queue, where a ClearML Agent assigned to the queue fetches the task and executes it on a
|
||||
remote or local machine. It's even possible to provide command line arguments and provide Python module dependencies and requirements.txt file!
|
||||
`clearml-task` automatically imports any script or Python repository into ClearML. `clearml-task` lets
|
||||
you enqueue your code for execution by an available [ClearML Agent](../clearml_agent.md). You can even provide command
|
||||
line arguments, Python module dependencies, and a requirements.txt file!
|
||||
|
||||
## What Is ClearML Task For?
|
||||
* Launching off-the-shelf code on a remote machine with dedicated resources (e.g. GPU)
|
||||
* Running [hyperparameter optimization](../fundamentals/hpo.md) on a codebase that is still not in ClearML
|
||||
* Creating a pipeline from an assortment of scripts, that you need to turn into ClearML tasks
|
||||
* Running some code on a remote machine, either using an on-prem cluster or on the cloud
|
||||
|
||||
## How Does ClearML Task Work?
|
||||
|
||||
1. Execute `clearml-task`, pointing it to your script or repository, and optionally an execution queue.
|
||||
1. `clearml-task` does its magic! It creates a new experiment on the [ClearML Server](../deploying_clearml/clearml_server.md),
|
||||
and, if a queue was specified, it sends the experiment to the queue to be fetched and executed by a **ClearML Agent**.
|
||||
1. The command line provides you with a link to your task's page in the ClearML web UI,
|
||||
where you can view the task's details.
|
||||
1. Execute `clearml-task`, specifying the ClearML target project and task name, along with your script (and repository / commit / branch).
|
||||
Optionally, specify an execution queue and Docker image to use.
|
||||
1. `clearml-task` does its magic! It creates a new task on the [ClearML Server](../deploying_clearml/clearml_server.md),
|
||||
and, if so directed, enqueues it for execution by a ClearML Agent.
|
||||
1. While the Task is running on the remote machine, all its console outputs are logged in real-time, alongside your
|
||||
TensorBoard and matplotlib. You can track your script’s progress and results in the [ClearML Web UI](../webapp/webapp_overview.md)
|
||||
(a link to your task details page in the ClearML Web UI is printed as ClearML Task creates the task).
|
||||
|
||||
## Features and Options
|
||||
## Execution Configuration
|
||||
### Docker
|
||||
Specify a docker container to run the code in by with the `--docker <docker_image>` flag.
|
||||
The ClearML Agent will pull it from dockerhub or a docker artifactory automatically.
|
||||
Specify a Docker container to run the code in with the `--docker <docker_image>` option.
|
||||
The ClearML Agent pulls it from Docker Hub or a Docker artifactory automatically.
|
||||
|
||||
### Package Dependencies
|
||||
If the local script requires packages to be installed, or the remote repository doesn't have a requirements.txt file,
|
||||
specify manually the required python packages using `--packages "<package_name>"`, for example `--packages "keras" "tensorflow>2.2"`.
|
||||
`clearml-task` automatically finds the requirements.txt file in remote repositories.
|
||||
|
||||
If a local script requires certain packages, or the remote repository doesn't have a `requirements.txt` file,
|
||||
manually specify the required Python packages using `--packages "<package_name>"`, for example `--packages "keras" "tensorflow>2.2"`.
|
||||
|
||||
### Queue
|
||||
Tasks are passed to ClearML Agents via [Queues](../fundamentals/agents_and_queues.md). Specify a queue to enqueue the task to.
|
||||
If a queue isn't chosen in the `clearml-task` command, the task will not be executed; it will be left in draft mode,
|
||||
and can be enqueued at a later point.
|
||||
Tasks are passed to ClearML Agents via [Queues](../fundamentals/agents_and_queues.md). Specify a queue in which to enqueue
|
||||
the task. If you don't input a queue, the task is created in *draft* status, and you can enqueue it at a later point.
|
||||
|
||||
### Branch and Working Directory
|
||||
To specify a specific branch and commit ID to be executed, pass
|
||||
`--branch <branch_name> --commit <commit_id>` flags.
|
||||
To specify your code's branch and commit ID, pass the `--branch <branch_name> --commit <commit_id>` options.
|
||||
If unspecified, `clearml-task` will use the latest commit from the master branch.
|
||||
|
||||
### Command Line Options
|
||||
@ -43,27 +51,62 @@ If unspecified, `clearml-task` will use the latest commit from the master branch
|
||||
|Name | Description| Optional |
|
||||
|---|----|---|
|
||||
| `--version` | Display the `clearml-task` utility version | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--project`| Set the project name for the task (Required, unless using `--base-task-id`) | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--name` | Select a name for the remote task | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--project`| Set the project name for the task (required, unless using `--base-task-id`). If the named project does not exist, it is created on-the-fly | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--name` | Set a target name for the new task | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--repo` | URL of remote repository. Example: `--repo https://github.com/allegroai/clearml.git` | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--branch` | Select specific repository branch / tag. By default, latest commit from the master branch | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--commit` | Select specific commit ID to use. By default, latest commit, or local commit ID when using local repository | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--folder` | Remotely execute the code in a local folder. Notice! It assumes a git repository already exists. Current state of the repo (commit ID and uncommitted changes) is logged and will be replicated on the remote machine | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--script` | Entry point script for the remote execution. When used in tandem with `--repo`, the script should be a relative path inside the repository. For example: `--script source/train.py`. When used with `--folder`, it supports a direct path to a file inside the local repository itself, for example: `--script ~/project/source/train.py` | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--branch` | Select repository branch / tag. By default, latest commit from the master branch | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--commit` | Select commit ID to use. By default, latest commit, or local commit ID when using local repository | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--folder` | Execute the code from a local folder. Notice, it assumes a git repository already exists. Current state of the repo (commit ID and uncommitted changes) is logged and replicated on the remote machine | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--script` | Entry point script for the remote execution. When used with `--repo`, input the script's relative path inside the repository. For example: `--script source/train.py`. When used with `--folder`, it supports a direct path to a file inside the local repository itself, for example: `--script ~/project/source/train.py` | <img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" /> |
|
||||
| `--cwd` | Working directory to launch the script from. Relative to repo root or local `--folder` | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--args` | Arguments to pass to the remote task, list of `<argument>=<value>` strings. Currently only argparse arguments are supported. Example: `--args lr=0.003 batch_size=64` | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--queue` | Select task's execution queue. If not provided, a task will be created but it will not be launched | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--args` | Arguments to pass to the remote task, list of `<argument>=<value>` strings. Currently only argparse arguments are supported | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--queue` | Select a task's execution queue. If not provided, a task is created but not launched | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--requirements` | Specify `requirements.txt` file to install when setting the session. By default, the` requirements.txt` from the repository will be used | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--packages` | Manually specify a list of required packages. Example: `--packages "tqdm>=2.1" "scikit-learn"` | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--docker` | Select the docker image to use in the remote task | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--docker_args` | Add docker arguments, pass a single string | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--docker_bash_setup_script` | Add bash script to be executed inside the docker before setting up the task's environment | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--output-uri` | Set the task `output_uri`, upload destination for task models and artifacts (Optional) | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--docker` | Select the Docker image to use in the remote task | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--docker_args` | Add Docker arguments, pass a single string | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--docker_bash_setup_script` | Add a bash script to be executed inside the Docker before setting up the task's environment | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--output-uri` | Set the task `output_uri`, upload destination for task models and artifacts | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--task-type` | Set the task type. Optional values: training, testing, inference, data_processing, application, monitor, controller, optimizer, service, qc, custom | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--skip-task-init` | If set, `Task.init()` call is not added to the entry point, and is assumed to be called within the script. Default: Add `Task.init()` call to entry point script | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--skip-task-init` | If set, `Task.init()` call is not added to the entry point, and is assumed to be called within the script | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
| `--base-task-id` | Use a pre-existing task in the system, instead of a local repo / script. Essentially clones an existing task and overrides arguments / requirements | <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" /> |
|
||||
|
||||
</div>
|
||||
|
||||
## Tutorial
|
||||
Learn how to use the `clearml-task` feature [here](../guides/clearml-task/clearml_task_tutorial.md).
|
||||
## Usage
|
||||
These commands demonstrate a few useful use cases for `clearml-task`
|
||||
|
||||
### Executing Code from a Remote Repository
|
||||
|
||||
```bash
|
||||
clearml-task --project examples --name remote_test --repo https://github.com/allegroai/events.git --script /webinar-0620/keras_mnist.py --args batch_size=64 epochs=1 --queue default
|
||||
```
|
||||
|
||||
The `keras_mnist.py` script from the [events](https://github.com/allegroai/events) GitHub repository is imported as a
|
||||
ClearML task named `remote_test` in the `examples` project. Its command line arguments `batch_size` and `epochs` values
|
||||
are set, and the task is enqueued for execution on the `default` queue.
|
||||
|
||||
### Executing a Local Script
|
||||
Using `clearml-task` to execute a local script is very similar to using it with a [remote repo](#executing-code-from-a-remote-repository).
|
||||
|
||||
```bash
|
||||
clearml-task --project examples --name local_test --script keras_mnist.py --requirements requirements.txt --args epochs=1 --queue default
|
||||
```
|
||||
|
||||
The `keras_mnist.py` script on the user's local machine is imported as a ClearML task named `local_test` in the `examples` project.
|
||||
|
||||
Its Python requirements are taken from the local `requiremnts.txt` file, and its `epochs` command line argument value is set.
|
||||
|
||||
The task is enqueued for execution on the `default` queue.
|
||||
|
||||
|
||||
### Pushing a Script to the Server
|
||||
|
||||
```bash
|
||||
clearml-task --project examples --name no_execute --script keras_mnist.py --requirements requirements.txt --args epochs=1
|
||||
```
|
||||
|
||||
The `keras_mnist.py` script on the user's local machine is imported as a ClearML task named `no_execute` in the `examples` project.
|
||||
|
||||
Its Python requirements are taken from the local `requiremnts.txt` file, and its `epochs` command line argument value is set.
|
||||
The task is created in a *draft* status (i.e. can be modified in the [WebApp UI](../webapp/webapp_overview.md) and later be enqueued).
|
||||
|
||||
@ -2,16 +2,14 @@
|
||||
title: ClearML Task Tutorial
|
||||
---
|
||||
|
||||
In this tutorial, you will use `clearml-task` to execute this [script](https://github.com/allegroai/events/blob/master/webinar-0620/keras_mnist.py)
|
||||
on a remote or local machine, from the remote repository and from a local script.
|
||||
In this tutorial, you will use `clearml-task` to execute [a script](https://github.com/allegroai/events/blob/master/webinar-0620/keras_mnist.py)
|
||||
on a remote or local machine, from a remote repository and your local machine.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- `clearml` Python package installed
|
||||
|
||||
- `clearml-agent` running on at least one machine (to execute the experiment) and assigned to listen to default queue
|
||||
|
||||
- [allegroai/events](https://github.com/allegroai/events) repository cloned (for local script execution)
|
||||
- `clearml-agent` running on at least one machine (to execute the experiment), configured to listen to default queue
|
||||
|
||||
### Executing Code from a Remote Repository
|
||||
|
||||
@ -19,58 +17,51 @@ on a remote or local machine, from the remote repository and from a local script
|
||||
clearml-task --project keras_examples --name remote_test --repo https://github.com/allegroai/events.git --script /webinar-0620/keras_mnist.py --args batch_size=64 epochs=1 --queue default
|
||||
```
|
||||
|
||||
Provide `clearml-task` with the following arguments:
|
||||
This sets the following arguments:
|
||||
|
||||
1. `--project keras_examples --name remote_test` - The project and experiment name.
|
||||
If the project entered doesn't exist, a new project will be created with the selected name.
|
||||
|
||||
1. `--repo https://github.com/allegroai/events.git` - The chosen repository's URL.
|
||||
By default, `clearml-task` will use the latest commit from the master branch.
|
||||
|
||||
1. `--script /webinar-0620/keras_mnist.py` - The script to be executed.
|
||||
* `--project keras_examples --name remote_test` - The project and experiment names
|
||||
* `--repo https://github.com/allegroai/events.git` - The repository's URL. By default, `clearml-task` uses the latest
|
||||
commit from the master branch
|
||||
* `--script /webinar-0620/keras_mnist.py` - The script to be executed
|
||||
* `--args batch_size=64 epochs=1` - Arguments passed to the script. This uses the `argparse` object to get CLI parameters
|
||||
* `--queue default` - Selected queue to send the experiment to
|
||||
|
||||
1. `--args batch_size=64 epochs=1` - Arguments passed to the script.
|
||||
This uses the `argparse` object to get CLI parameters.
|
||||
:::note Adding Requirements
|
||||
`clearml-task` automatically finds the requirements.txt file in remote repositories.
|
||||
If a remote repo does not have such a file, make sure to either add one with all the required Python packages,
|
||||
or add the `--packages "<package_name>"` option to the command (for example: `--packages "tqdm>=2.1" "scikit-learn"`).
|
||||
:::
|
||||
|
||||
1. `--queue default` - Selected queue to send the experiment to.
|
||||
|
||||
|
||||
Now `clearml-task` does the rest of the heavy-lifting!
|
||||
|
||||
* It creates a new Task on the [ClearML Server](../../deploying_clearml/clearml_server.md).
|
||||
|
||||
* Then, the Task is enqueued in the selected execution queue, where it will be executed by an available
|
||||
`clearml-agent` assigned to that queue.
|
||||
Now `clearml-task` does all the heavy-lifting!
|
||||
1. It creates a new Task on the [ClearML Server](../../deploying_clearml/clearml_server.md).
|
||||
1. `clearml-task` enqueues the task in the selected execution queue, where a [ClearML Agent](../../clearml_agent.md)
|
||||
assigned to that queue executes the task.
|
||||
|
||||
Your output should look something like this:
|
||||
Your output should look something like this:
|
||||
|
||||
```console
|
||||
New task created id=2f96ee95b05d4693b360d0fcbb26b727
|
||||
Task id=2f96ee95b05d4693b360d0fcbb26b727 sent for execution on queue default
|
||||
Execution log at: https://app.community.clear.ml/projects/552d5399112d47029c146d5248570295/experiments/2f96ee95b05d4693b360d0fcbb26b727/output/log
|
||||
```
|
||||
|
||||
:::note Adding Requirements
|
||||
`clearml-task` automatically finds the requirements.txt file in remote repositories.
|
||||
If a remote repo does not have such a file, make sure to either add one with all the required Python packages,
|
||||
or add the `--packages "<package_name>"` flag to the command (for example: `--packages "tqdm>=2.1" "scikit-learn"`).
|
||||
:::
|
||||
<br />
|
||||
|
||||
|
||||
### Executing a Local Script
|
||||
Using `clearml-task` to execute a local script is very similar to using it with a remote repo.
|
||||
|
||||
For this example, we will be using a local version of this [script](https://github.com/allegroai/events/blob/master/webinar-0620/keras_mnist.py).
|
||||
1. Go to the root folder of the cloned [allegroai/events](https://github.com/allegroai/events) repository
|
||||
1. Run `clearml-task` by executing:
|
||||
For this example, use a local version of [this script](https://github.com/allegroai/events/blob/master/webinar-0620/keras_mnist.py).
|
||||
1. Clone the [allegroai/events](https://github.com/allegroai/events) repository
|
||||
1. Go to the root folder of the cloned repository
|
||||
1. Run the following command:
|
||||
|
||||
``` bash
|
||||
clearml-task --project keras --name local_test --script webinar-0620/keras_mnist.py --requirements webinar-0620/requirements.txt --args epochs=1 --queue default
|
||||
```
|
||||
|
||||
Notice that the command is almost identical to executing code from a git repository. The only differences are:
|
||||
* `--script webinar-0620/keras_mnist.py` - Pointing `clearml-task` to a local script.
|
||||
* `--requirements webinar-0620/requirements.txt` - Manually specifying a *requirements.txt* file.
|
||||
This sets the following arguments:
|
||||
* `--project keras --name local_test` - The project and experiment names
|
||||
* `--script /webinar-0620/keras_mnist.py` - The local script to be executed
|
||||
* `-requirements webinar-0620/requirements.txt` - The local python package requirements file
|
||||
* `--args batch_size=64 epochs=1` - Arguments passed to the script. This uses the argparse object to capture CLI parameters
|
||||
* `--queue default` - Selected queue to send the experiment to
|
||||
|
||||
After executing `clearml-task`, a Task will be created according to the parameters entered. The Task will
|
||||
be sent to a queue for execution.
|
||||
`clearml-task` creates a task according to the input parameters, and sends the task to the `default` queue for execution.
|
||||
@ -19,7 +19,7 @@ module.exports = {
|
||||
'clearml_agent',
|
||||
{'ClearML Data': ['clearml_data/clearml_data', 'clearml_data/clearml_data_cli', 'clearml_data/clearml_data_sdk', 'clearml_data/best_practices',
|
||||
{'Workflows': ['clearml_data/data_management_examples/workflows', 'clearml_data/data_management_examples/data_man_simple', 'clearml_data/data_management_examples/data_man_folder_sync', 'clearml_data/data_management_examples/data_man_cifar_classification', 'clearml_data/data_management_examples/data_man_python']},]},
|
||||
{'Applications': ['apps/clearml_session', 'apps/clearml_task']},
|
||||
{'CLI Tools': ['apps/clearml_session', 'apps/clearml_task']},
|
||||
{'Integrations': ['integrations/libraries', 'integrations/storage']},
|
||||
|
||||
{'WebApp': ['webapp/webapp_overview', 'webapp/webapp_home',
|
||||
|
||||
Loading…
Reference in New Issue
Block a user