From d7101e0f0e2153a7e83b87766f5d612cdd97d108 Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Thu, 19 Aug 2021 09:44:54 +0300 Subject: [PATCH 1/2] Delete redundant phrase (#42) --- docs/guides/frameworks/pytorch/pytorch_mnist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/frameworks/pytorch/pytorch_mnist.md b/docs/guides/frameworks/pytorch/pytorch_mnist.md index c8b75191..d2dc9353 100644 --- a/docs/guides/frameworks/pytorch/pytorch_mnist.md +++ b/docs/guides/frameworks/pytorch/pytorch_mnist.md @@ -9,7 +9,7 @@ The example script does the following: * Trains a simple deep neural network on the PyTorch built-in [MNIST](https://pytorch.org/docs/stable/torchvision/datasets.html#mnist) dataset. * Uses **ClearML** automatic logging. -* Calls the [Logger.report_scalar](../../../references/sdk/logger.md#report_scalar) method to demonstrate explicit reporting and explicit reporting, +* Calls the [Logger.report_scalar](../../../references/sdk/logger.md#report_scalar) method to demonstrate explicit reporting, which allows adding customized reporting to the code. * Creates an experiment named `pytorch mnist train`, which is associated with the `examples` project. From d5942a025cbeb152e3f831d74e5a1cfc50a246ba Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Thu, 19 Aug 2021 09:46:40 +0300 Subject: [PATCH 2/2] Add clearml-task command line arguments (#41) --- docs/apps/clearml_task.md | 43 ++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/docs/apps/clearml_task.md b/docs/apps/clearml_task.md index b8aeb43b..458e84c2 100644 --- a/docs/apps/clearml_task.md +++ b/docs/apps/clearml_task.md @@ -6,7 +6,7 @@ ClearML Task is ClearML's Zero Code Integration Module. Using only the command l you can easily track your work and integrate ClearML with your existing code. `clearml-task` automatically integrates ClearML into any script or **any** python repository. `clearml-task` has the option -to send the Task to a queue, where a **ClearML Agent** listening to the queue will fetch the Task it and executes it on a +to send the task to a queue, where a **ClearML Agent** listening to the queue will fetch the task and execute it on a remote or local machine. It's even possible to provide command line arguments and provide Python module dependencies and requirements.txt file! ## How Does ClearML Task Work? @@ -14,8 +14,8 @@ remote or local machine. It's even possible to provide command line arguments an 1. Execute `clearml-task`, pointing it to your script or repository, and optionally an execution queue. 1. `clearml-task` does its magic! It creates a new experiment on the [ClearML Server](../deploying_clearml/clearml_server.md), and, if a queue was specified, it sends the experiment to the queue to be fetched and executed by a **ClearML Agent**. -1. The command line will provide you with a link to your Task's page in the ClearML web UI, - where you will be able to view the Task's details. +1. The command line will provide you with a link to your task's page in the ClearML web UI, + where you will be able to view the task's details. ## Features and Options ### Docker @@ -24,12 +24,12 @@ The ClearML Agent will pull it from dockerhub or a docker artifactory automatica ### Package Dependencies If the local script requires packages to be installed installed or the remote repository doesn't have a requirements.txt file, -specify manually the required python packages using
+specify manually the required python packages using
`--packages ""`, for example `--packages "keras" "tensorflow>2.2"`. ### Queue -Tasks are passed to ClearML Agents via [Queues](../fundamentals/agents_and_queues.md). Specify a queue to enqueue the Task to. -If a queue isn't chosen in the `clearml-task` command, the Task will not be executed; it will be left in draft mode, +Tasks are passed to ClearML Agents via [Queues](../fundamentals/agents_and_queues.md). Specify a queue to enqueue the task to. +If a queue isn't chosen in the `clearml-task` command, the task will not be executed; it will be left in draft mode, and can be enqueued at a later point. ### Branch and Working Directory @@ -37,5 +37,34 @@ A specific branch and commit ID, other than latest commit in master, to be execu `--branch --commit ` flags. If unspecified, `clearml-task` will use the latest commit from the master branch. -Learn how to use the `clearml-task` feature [here](../guides/clearml-task/clearml_task_tutorial.md). +### Command line options +
+ +|Name | Description| Optional | +|---|----|---| +| `--version` | Display the `clearml-task` utility version | Yes | +| `--project`| Set the project name for the task (Required, unless using `--base-task-id`) | No | +| `--name` | Select a name for the remote task | No | +| `--repo` | URL of remote repository. Example: `--repo https://github.com/allegroai/clearml.git` | Yes | +| `--branch` | Select specific repository branch / tag. By default, latest commit from the master branch | Yes | +| `--commit` | Select specific commit ID to use. By default, latest commit, or local commit ID when using local repository | Yes | +| `--folder` | Remotely execute the code in a local folder. Notice! It assumes a git repository already exists. Current state of the repo (commit ID and uncommitted changes) is logged and will be replicated on the remote machine | Yes | +| `--script` | Entry point script for the remote execution. When used in tandem with `--repo`, the script should be a relative path inside the repository. For example: `--script source/train.py`. When used with `--folder`, it supports a direct path to a file inside the local repository itself, for example: `--script ~/project/source/train.py` | No | +| `--cwd` | Working directory to launch the script from. Relative to repo root or local `--folder` | Yes | +| `--args` | Arguments to pass to the remote task, list of `=` strings. Currently only argparse arguments are supported. Example: `--args lr=0.003 batch_size=64` | Yes | +| `--queue` | Select task's execution queue. If not provided, a task will be created but it will not be launched | Yes | +| `--requirements` | Specify `requirements.txt` file to install when setting the session. By default, the` requirements.txt` from the repository will be used | Yes | +| `--packages` | Manually specify a list of required packages. Example: `--packages "tqdm>=2.1" "scikit-learn"` | Yes | +| `--docker` | Select the docker image to use in the remote task | Yes | +| `--docker_args` | Add docker arguments, pass a single string | Yes | +| `--docker_bash_setup_script` | Add bash script to be executed inside the docker before setting up the task's environment | Yes | +| `--output-uri` | Set the task `output_uri`, upload destination for task models and artifacts (Optional) | Yes | +| `--task-type` | Set the task type. Optional values: training, testing, inference, data_processing, application, monitor, controller, optimizer, service, qc, custom | Yes | +| `--skip-task-init` | If set, `Task.init()` call is not added to the entry point, and is assumed to be called within the script. Default: Add `Task.init()` call to entry point script | Yes | +| `--base-task-id` | Use a pre-existing task in the system, instead of a local repo / script. Essentially clones an existing task and overrides arguments / requirements | Yes | + +
+ +## Tutorial +Learn how to use the `clearml-task` feature [here](../guides/clearml-task/clearml_task_tutorial.md).