ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-06-26 18:17:44 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Edit ClearML Task doc and tutorial (#172)

Browse Source
This commit is contained in:
pollfly
2022-01-23 11:02:38 +02:00
committed by GitHub
parent 9727045603
commit a7e9347bc2
3 changed files with 111 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files

73
docs/guides/clearml-task/clearml_task_tutorial.md
View File

@@ -2,16 +2,14 @@
title: ClearML Task Tutorial
---
In this tutorial, you will use `clearml-task` to execute this [script](https://github.com/allegroai/events/blob/master/webinar-0620/keras_mnist.py)
on a remote or local machine, from the remote repository and from a local script.
In this tutorial, you will use `clearml-task` to execute [a script](https://github.com/allegroai/events/blob/master/webinar-0620/keras_mnist.py)
on a remote or local machine, from a remote repository and your local machine.
### Prerequisites
- `clearml` Python package installed
- `clearml-agent` running on at least one machine (to execute the experiment) and assigned to listen to default queue
- [allegroai/events](https://github.com/allegroai/events) repository cloned (for local script execution)
- `clearml-agent` running on at least one machine (to execute the experiment), configured to listen to default queue
### Executing Code from a Remote Repository
@@ -19,58 +17,51 @@ on a remote or local machine, from the remote repository and from a local script
clearml-task --project keras_examples --name remote_test --repo https://github.com/allegroai/events.git --script /webinar-0620/keras_mnist.py --args batch_size=64 epochs=1 --queue default
```
Provide `clearml-task` with the following arguments:
This sets the following arguments:
1. `--project keras_examples --name remote_test` - The project and experiment name.
If the project entered doesn't exist, a new project will be created with the selected name.
1. `--repo https://github.com/allegroai/events.git` - The chosen repository's URL.
By default, `clearml-task` will use the latest commit from the master branch.
1. `--script /webinar-0620/keras_mnist.py` - The script to be executed.
* `--project keras_examples --name remote_test` - The project and experiment names
* `--repo https://github.com/allegroai/events.git` - The repository's URL. By default, `clearml-task` uses the latest
commit from the master branch
* `--script /webinar-0620/keras_mnist.py` - The script to be executed
* `--args batch_size=64 epochs=1` - Arguments passed to the script. This uses the `argparse` object to get CLI parameters
* `--queue default` - Selected queue to send the experiment to
1. `--args batch_size=64 epochs=1` - Arguments passed to the script.
This uses the `argparse` object to get CLI parameters.
:::note Adding Requirements
`clearml-task` automatically finds the requirements.txt file in remote repositories.
If a remote repo does not have such a file, make sure to either add one with all the required Python packages,
or add the `--packages "<package_name>"` option to the command (for example: `--packages "tqdm>=2.1" "scikit-learn"`).
:::
1. `--queue default` - Selected queue to send the experiment to.
Now `clearml-task` does the rest of the heavy-lifting!
* It creates a new Task on the [ClearML Server](../../deploying_clearml/clearml_server.md).
* Then, the Task is enqueued in the selected execution queue, where it will be executed by an available
`clearml-agent` assigned to that queue.
Now `clearml-task` does all the heavy-lifting!
1. It creates a new Task on the [ClearML Server](../../deploying_clearml/clearml_server.md).
1. `clearml-task` enqueues the task in the selected execution queue, where a [ClearML Agent](../../clearml_agent.md)
assigned to that queue executes the task.
Your output should look something like this:
Your output should look something like this:
```console
New task created id=2f96ee95b05d4693b360d0fcbb26b727
Task id=2f96ee95b05d4693b360d0fcbb26b727 sent for execution on queue default
Execution log at: https://app.community.clear.ml/projects/552d5399112d47029c146d5248570295/experiments/2f96ee95b05d4693b360d0fcbb26b727/output/log
```
:::note Adding Requirements
`clearml-task` automatically finds the requirements.txt file in remote repositories.
If a remote repo does not have such a file, make sure to either add one with all the required Python packages,
or add the `--packages "<package_name>"` flag to the command (for example: `--packages "tqdm>=2.1" "scikit-learn"`).
:::
<br />
### Executing a Local Script
Using `clearml-task` to execute a local script is very similar to using it with a remote repo.
For this example, we will be using a local version of this [script](https://github.com/allegroai/events/blob/master/webinar-0620/keras_mnist.py).
1. Go to the root folder of the cloned [allegroai/events](https://github.com/allegroai/events) repository
1. Run `clearml-task` by executing:
For this example, use a local version of [this script](https://github.com/allegroai/events/blob/master/webinar-0620/keras_mnist.py).
1. Clone the [allegroai/events](https://github.com/allegroai/events) repository
1. Go to the root folder of the cloned repository
1. Run the following command:
``` bash
clearml-task --project keras --name local_test --script webinar-0620/keras_mnist.py --requirements webinar-0620/requirements.txt --args epochs=1 --queue default
```
Notice that the command is almost identical to executing code from a git repository. The only differences are:
* `--script webinar-0620/keras_mnist.py` - Pointing `clearml-task` to a local script.
* `--requirements webinar-0620/requirements.txt` - Manually specifying a *requirements.txt* file.
This sets the following arguments:
* `--project keras --name local_test` - The project and experiment names
* `--script /webinar-0620/keras_mnist.py` - The local script to be executed
* `-requirements webinar-0620/requirements.txt` - The local python package requirements file
* `--args batch_size=64 epochs=1` - Arguments passed to the script. This uses the argparse object to capture CLI parameters
* `--queue default` - Selected queue to send the experiment to
After executing `clearml-task`, a Task will be created according to the parameters entered. The Task will
be sent to a queue for execution.
`clearml-task` creates a task according to the input parameters, and sends the task to the `default` queue for execution.