Small edits (#476)

This commit is contained in:
pollfly 2023-02-16 12:17:53 +02:00 committed by GitHub
parent 5458f8036b
commit 2cf096f7ec
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
27 changed files with 64 additions and 64 deletions

View File

@ -45,7 +45,7 @@ to automatically sync local configurations with a remote session.
## How it Works ## How it Works
ClearML allows you to leverage a resource (e.g. GPU or CPU machine) by utilizing the [ClearML Agent](../clearml_agent.md). ClearML lets you leverage a resource (e.g. GPU or CPU machine) by utilizing the [ClearML Agent](../clearml_agent.md).
A ClearML Agent runs on a target machine, and ClearML Session instructs it to execute the Jupyter / VS Code A ClearML Agent runs on a target machine, and ClearML Session instructs it to execute the Jupyter / VS Code
server to develop remotely. server to develop remotely.
After entering a `clearml-session` command with all specifications: After entering a `clearml-session` command with all specifications:

View File

@ -36,7 +36,7 @@ clearml-agent build [-h] --id TASK_ID [--target TARGET]
|---|----|---| |---|----|---|
|`--id`| Build a worker environment for this Task ID.|<img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" />| |`--id`| Build a worker environment for this Task ID.|<img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" />|
|`--cpu-only`| Disable GPU access for the Docker container.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--cpu-only`| Disable GPU access for the Docker container.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--docker`| Run agent in Docker mode. Specify a Docker container that a worker will execute at launch. To specify the image name and optional arguments, use one of the following: <ul><li>`--docker <image_name> <args>` on the command line</li><li>`--docker` on the command line, and specify the default image name and arguments in the configuration file.</li></ul> Environment variable settings for Docker containers: <ul><li>`CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `--gpus` flag inside the Docker container. This also allows you to execute ClearML Agent using Docker versions earlier than 19.03.</li><li>`NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container.</li><li> `CLEARML_AGENT_GIT_USER` and `CLEARML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution.</li></ul> To limit GPU visibility for Docker, set the `NVIDIA_VISIBLE_DEVICES` environment variable.| <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--docker`| Run agent in Docker mode. Specify a Docker container that a worker will execute at launch. To specify the image name and optional arguments, use one of the following: <ul><li>`--docker <image_name> <args>` on the command line</li><li>`--docker` on the command line, and specify the default image name and arguments in the configuration file.</li></ul> Environment variable settings for Docker containers: <ul><li>`CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `--gpus` flag inside the Docker container. This also lets you execute ClearML Agent using Docker versions earlier than 19.03.</li><li>`NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container.</li><li> `CLEARML_AGENT_GIT_USER` and `CLEARML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution.</li></ul> To limit GPU visibility for Docker, set the `NVIDIA_VISIBLE_DEVICES` environment variable.| <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--entry-point`| Used in conjunction with `--docker`, indicates how to run the Task specified by `--task-id` on Docker startup. The `--entry-point` options are: <ul><li>`reuse` - Overwrite the existing Task data.</li><li>`clone_task` - Clone the Task, and execute the cloned Task.</li></ul>|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--entry-point`| Used in conjunction with `--docker`, indicates how to run the Task specified by `--task-id` on Docker startup. The `--entry-point` options are: <ul><li>`reuse` - Overwrite the existing Task data.</li><li>`clone_task` - Clone the Task, and execute the cloned Task.</li></ul>|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--force-docker`| Force using the agent-specified docker image (either explicitly in the `--docker` argument or using the agent's default docker image). If provided, the agent will not use any docker container information stored in the task itself (default `False`)|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--force-docker`| Force using the agent-specified docker image (either explicitly in the `--docker` argument or using the agent's default docker image). If provided, the agent will not use any docker container information stored in the task itself (default `False`)|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--git-pass`| Git password for repository access.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--git-pass`| Git password for repository access.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
@ -86,7 +86,7 @@ clearml-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]] [--order-
|`--cpu-only`| If running in Docker mode (see the `--docker` option), disable GPU access for the Docker container or virtual environment.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--cpu-only`| If running in Docker mode (see the `--docker` option), disable GPU access for the Docker container or virtual environment.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--create-queue`| If the queue name provided with `--queue` does not exist in the server, create it on-the-fly and use it.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--create-queue`| If the queue name provided with `--queue` does not exist in the server, create it on-the-fly and use it.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--detached`| Run agent in the background. The `clearml-agent` command returns immediately.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--detached`| Run agent in the background. The `clearml-agent` command returns immediately.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--docker`| Run in Docker mode. Execute the Task inside a Docker container. To specify the image name and optional arguments, either: <ul><li> Use `--docker <image_name> <args>` on the command line, or </li><li>Use `--docker` on the command line, and specify the default image name and arguments in the configuration file.</li></ul> Environment variable settings for Docker containers: <ul><li>`CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `--gpus` flag inside the Docker container. This also allows you to execute ClearML Agent using Docker versions earlier than 19.03.</li><li>`NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container.</li><li> `CLEARML_AGENT_GIT_USER` and `CLEARML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution.</li></ul>|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--docker`| Run in Docker mode. Execute the Task inside a Docker container. To specify the image name and optional arguments, either: <ul><li> Use `--docker <image_name> <args>` on the command line, or </li><li>Use `--docker` on the command line, and specify the default image name and arguments in the configuration file.</li></ul> Environment variable settings for Docker containers: <ul><li>`CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `--gpus` flag inside the Docker container. This also lets you execute ClearML Agent using Docker versions earlier than 19.03.</li><li>`NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container.</li><li> `CLEARML_AGENT_GIT_USER` and `CLEARML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution.</li></ul>|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--downtime`| Specify downtime for clearml-agent in `<hours> <days>` format. For example, use `09-13 TUE` to set Tuesday's downtime to 09-13. <br/><br/>NOTE: <ul><li>This feature is available under the ClearML Enterprise plan</li><li>Make sure to have only one of uptime / downtime configuration and not both.</li></ul> |<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--downtime`| Specify downtime for clearml-agent in `<hours> <days>` format. For example, use `09-13 TUE` to set Tuesday's downtime to 09-13. <br/><br/>NOTE: <ul><li>This feature is available under the ClearML Enterprise plan</li><li>Make sure to have only one of uptime / downtime configuration and not both.</li></ul> |<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--dynamic-gpus`| Allow to dynamically allocate GPUs based on queue properties, configure with `--queue <queue_name>=<num_gpus>`. For example: `--dynamic-gpus --queue dual_gpus=2 single_gpu=1` <br/><br/>NOTE: This feature is available under the ClearML Enterprise plan|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--dynamic-gpus`| Allow to dynamically allocate GPUs based on queue properties, configure with `--queue <queue_name>=<num_gpus>`. For example: `--dynamic-gpus --queue dual_gpus=2 single_gpu=1` <br/><br/>NOTE: This feature is available under the ClearML Enterprise plan|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--force-current-version`| To use your current version of ClearML Agent when running in Docker mode (the `--docker` argument is specified), instead of the latest ClearML Agent version which is automatically installed, specify `force-current-version`. <br/><br/> For example, if your current ClearML Agent version is `0.13.1`, but the latest version of ClearML Agent is `0.13.3`, use `--force-current-version` and your Task will execute in the Docker container with ClearML Agent version `0.13.1`.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--force-current-version`| To use your current version of ClearML Agent when running in Docker mode (the `--docker` argument is specified), instead of the latest ClearML Agent version which is automatically installed, specify `force-current-version`. <br/><br/> For example, if your current ClearML Agent version is `0.13.1`, but the latest version of ClearML Agent is `0.13.3`, use `--force-current-version` and your Task will execute in the Docker container with ClearML Agent version `0.13.1`.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
@ -128,7 +128,7 @@ clearml-agent execute [-h] --id TASK_ID [--log-file LOG_FILE] [--disable-monitor
|`--id`| The ID of the Task to build|<img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" />| |`--id`| The ID of the Task to build|<img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" />|
|`--clone`| Clone the Task specified by `--id`, and then execute that cloned Task.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--clone`| Clone the Task specified by `--id`, and then execute that cloned Task.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--cpu-only`| Disable GPU access for the daemon, only use CPU in either docker or virtual environment.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--cpu-only`| Disable GPU access for the daemon, only use CPU in either docker or virtual environment.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--docker`| Run in Docker mode. Execute the Task inside a Docker container. To specify the image name and optional arguments, use one of the following: <ul><li>`--docker <image_name> <args>` on the command line</li><li>`--docker` on the command line, and specify the default image name and arguments in the configuration file.</li></ul> Environment variable settings for Dockers containers: <ul><li>`CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `--gpus` flag inside the Docker container. This also allows you to execute ClearML Agent using Docker versions earlier than 19.03.</li><li>`NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container.</li><li> `CLEARML_AGENT_GIT_USER` and `CLEARML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution.</li></ul>| <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--docker`| Run in Docker mode. Execute the Task inside a Docker container. To specify the image name and optional arguments, use one of the following: <ul><li>`--docker <image_name> <args>` on the command line</li><li>`--docker` on the command line, and specify the default image name and arguments in the configuration file.</li></ul> Environment variable settings for Dockers containers: <ul><li>`CLEARML_DOCKER_SKIP_GPUS_FLAG` - Ignore the `--gpus` flag inside the Docker container. This also lets you execute ClearML Agent using Docker versions earlier than 19.03.</li><li>`NVIDIA_VISIBLE_DEVICES` - Limit GPU visibility for the Docker container.</li><li> `CLEARML_AGENT_GIT_USER` and `CLEARML_AGENT_GIT_PASS` - Pass these credentials to the Docker container at execution.</li></ul>| <img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--disable-monitoring`| Disable logging and monitoring, except for stdout.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--disable-monitoring`| Disable logging and monitoring, except for stdout.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--full-monitoring`| Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--full-monitoring`| Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--git-pass`| Git password for repository access.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />| |`--git-pass`| Git password for repository access.|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|

View File

@ -41,7 +41,7 @@ guide for more info!
## Using ClearML Data ## Using ClearML Data
ClearML Data offers two interfaces: ClearML Data supports two interfaces:
- `clearml-data` - A CLI utility for creating, uploading, and managing datasets. See [CLI](clearml_data_cli.md) for a reference of `clearml-data` commands. - `clearml-data` - A CLI utility for creating, uploading, and managing datasets. See [CLI](clearml_data_cli.md) for a reference of `clearml-data` commands.
- `clearml.Dataset` - A python interface for creating, retrieving, managing, and using datasets. See [SDK](clearml_data_sdk.md) for an overview of the basic methods of the `Dataset` module. - `clearml.Dataset` - A python interface for creating, retrieving, managing, and using datasets. See [SDK](clearml_data_sdk.md) for an overview of the basic methods of the `Dataset` module.

View File

@ -2,13 +2,13 @@
title: Dataset Management with CLI and SDK title: Dataset Management with CLI and SDK
--- ---
In this tutorial, we are going to manage the CIFAR dataset with `clearml-data` CLI, and then use ClearML's [`Dataset`](../../references/sdk/dataset.md) In this tutorial, you are going to manage the CIFAR dataset with `clearml-data` CLI, and then use ClearML's [`Dataset`](../../references/sdk/dataset.md)
class to ingest the data. class to ingest the data.
## Creating the Dataset ## Creating the Dataset
### Downloading the Data ### Downloading the Data
Before we can register the CIFAR dataset with `clearml-data`, we need to obtain a local copy of it. Before registering the CIFAR dataset with `clearml-data`, you need to obtain a local copy of it.
Execute this python script to download the data Execute this python script to download the data
```python ```python
@ -43,7 +43,7 @@ New dataset created id=ee1c35f60f384e65bc800f42f0aca5ec
Where `ee1c35f60f384e65bc800f42f0aca5ec` is the dataset ID. Where `ee1c35f60f384e65bc800f42f0aca5ec` is the dataset ID.
## Adding Files ## Adding Files
Add the files we just downloaded to the dataset: Add the files that were just downloaded to the dataset:
``` ```
clearml-data add --files <dataset_path> clearml-data add --files <dataset_path>
@ -72,7 +72,7 @@ In the panel's **CONTENT** tab, you can see a table summarizing version contents
## Using the Dataset ## Using the Dataset
Now that we have a new dataset registered, we can consume it. Now that you have a new dataset registered, you can consume it.
The [data_ingestion.py](https://github.com/allegroai/clearml/blob/master/examples/datasets/data_ingestion.py) example The [data_ingestion.py](https://github.com/allegroai/clearml/blob/master/examples/datasets/data_ingestion.py) example
script demonstrates using the dataset within Python code. script demonstrates using the dataset within Python code.
@ -102,7 +102,7 @@ hyperparameters. Passing `alias=<dataset_alias_string>` stores the datasets I
`dataset_alias_string` parameter in the experiment's **CONFIGURATION > HYPERPARAMETERS > Datasets** section. This way `dataset_alias_string` parameter in the experiment's **CONFIGURATION > HYPERPARAMETERS > Datasets** section. This way
you can easily track which dataset the task is using. you can easily track which dataset the task is using.
The Dataset's [`get_local_copy`](../../references/sdk/dataset.md#get_local_copy) method will return a path to the cached, The Dataset's [`get_local_copy`](../../references/sdk/dataset.md#get_local_copy) method returns a path to the cached,
downloaded dataset. Then we provide the path to PyTorch's dataset object. downloaded dataset. Then the dataset path is input to PyTorch's `datasets` object.
The script then trains a neural network to classify images using the dataset created above. The script then trains a neural network to classify images using the dataset created above.

View File

@ -78,4 +78,4 @@ Upload completed (742 bytes)
Dataset closed and finalized Dataset closed and finalized
``` ```
We can see that 2 files were added or modified, just as we expected! See that 2 files were added or modified, just as expected!

View File

@ -18,7 +18,7 @@ demonstrates how to do the following:
### Downloading the Data ### Downloading the Data
We first need to obtain a local copy of the CIFAR dataset. You first need to obtain a local copy of the CIFAR dataset.
```python ```python
from clearml import StorageManager from clearml import StorageManager
@ -79,7 +79,7 @@ In the panel's **CONTENT** tab, you can see a table summarizing version contents
## Data Ingestion ## Data Ingestion
Now that we have a new dataset registered, we can consume it! Now that a new dataset is registered, you can consume it!
The [data_ingestion.py](https://github.com/allegroai/clearml/blob/master/examples/datasets/data_ingestion.py) script The [data_ingestion.py](https://github.com/allegroai/clearml/blob/master/examples/datasets/data_ingestion.py) script
demonstrates data ingestion using the dataset created in the first script. demonstrates data ingestion using the dataset created in the first script.

View File

@ -48,7 +48,7 @@ to captures all files and sub-folders:
:::note :::note
After creating a dataset, we don't have to specify its ID when running commands, such as *add*, *remove* or *list* After creating a dataset, its ID doesn't need to be specified when running commands, such as `add`, `remove`, or `list`
::: :::
3. Close the dataset - this command uploads the files. By default, the files are uploaded to the file server, but 3. Close the dataset - this command uploads the files. By default, the files are uploaded to the file server, but
@ -151,7 +151,7 @@ You'll need to input the Dataset ID you received when created the dataset above
clearml-data list --id 8b68686a4af040d081027ba3cf6bbca6 clearml-data list --id 8b68686a4af040d081027ba3cf6bbca6
``` ```
And we see that our changes have been made! `new_data.txt` has been added, and `dancing.jpg` has been removed. And see that the changes have been made! `new_data.txt` has been added, and `dancing.jpg` has been removed.
``` ```
file name | size | hash file name | size | hash

View File

@ -2,7 +2,7 @@
title: Tutorial title: Tutorial
--- ---
In this tutorial, we will go over the model lifecycle -- from training to serving -- in the following steps: In this tutorial, you will go over the model lifecycle -- from training to serving -- in the following steps:
* Training a model using the [sklearn example script](https://github.com/allegroai/clearml-serving/blob/main/examples/sklearn/train_model.py) * Training a model using the [sklearn example script](https://github.com/allegroai/clearml-serving/blob/main/examples/sklearn/train_model.py)
* Serving the model using **ClearML Serving** * Serving the model using **ClearML Serving**
* Spinning the inference container * Spinning the inference container

View File

@ -5,7 +5,7 @@ title: Linux and macOS
Deploy the ClearML Server in Linux or macOS using the pre-built Docker image. Deploy the ClearML Server in Linux or macOS using the pre-built Docker image.
For ClearML docker images, including previous versions, see [https://hub.docker.com/r/allegroai/clearml](https://hub.docker.com/r/allegroai/clearml). For ClearML docker images, including previous versions, see [https://hub.docker.com/r/allegroai/clearml](https://hub.docker.com/r/allegroai/clearml).
However, pulling the ClearML Docker image directly is not required. We provide a docker-compose YAML file that does this. However, pulling the ClearML Docker image directly is not required. ClearML provides a docker-compose YAML file that does this.
The docker-compose file is included in the instructions on this page. The docker-compose file is included in the instructions on this page.
For information about upgrading ClearML Server in Linux or macOS, see [here](upgrade_server_linux_mac.md) For information about upgrading ClearML Server in Linux or macOS, see [here](upgrade_server_linux_mac.md)

View File

@ -364,7 +364,7 @@ Your firewall may be preventing the connection. Try one of the following solutio
**How do I modify experiment names once they have been created?** **How do I modify experiment names once they have been created?**
An experiment's name is a user-controlled property, which can be accessed via the `Task.name` variable. This allows you to use meaningful naming schemes for easily filtering and comparing of experiments. An experiment's name is a user-controlled property, which can be accessed via the `Task.name` variable. This lets you use meaningful naming schemes for easily filtering and comparing of experiments.
For example, to distinguish between different experiments, you can append the task ID to the task name: For example, to distinguish between different experiments, you can append the task ID to the task name:
```python ```python
@ -469,7 +469,7 @@ After thirty minutes, it remains unchanged.
**Can I control what ClearML automatically logs?** <a id="controlling_logging"></a> **Can I control what ClearML automatically logs?** <a id="controlling_logging"></a>
Yes! ClearML allows you to control automatic logging for `stdout`, `stderr`, and frameworks when initializing a Task Yes! ClearML lets you control automatic logging for `stdout`, `stderr`, and frameworks when initializing a Task
by calling the [`Task.init`](references/sdk/task.md#taskinit) method. by calling the [`Task.init`](references/sdk/task.md#taskinit) method.
To control a Task's framework logging, use the `auto_connect_frameworks` parameter. Turn off all automatic logging by setting the To control a Task's framework logging, use the `auto_connect_frameworks` parameter. Turn off all automatic logging by setting the
@ -629,7 +629,7 @@ experiment info panel > EXECUTION tab.
**I read there is a feature for centralized model storage. How do I use it?** <a id="centralized-model-storage"></a> **I read there is a feature for centralized model storage. How do I use it?** <a id="centralized-model-storage"></a>
When calling [Task.init](references/sdk/task.md#taskinit), When calling [Task.init](references/sdk/task.md#taskinit),
providing the `output_uri` parameter allows you to specify the location in which model checkpoints (snapshots) will be stored. providing the `output_uri` parameter lets you specify the location in which model checkpoints (snapshots) will be stored.
For example, to store model checkpoints (snapshots) in `/mnt/shared/folder`: For example, to store model checkpoints (snapshots) in `/mnt/shared/folder`:

View File

@ -86,8 +86,8 @@ method. The `set_parameters_as_dict` method updates parameters while the `set_pa
ClearML does not automatically track changes to explicitly set parameters. ClearML does not automatically track changes to explicitly set parameters.
### User Properties ### User Properties
User properties do not impact tasks execution and so can be modified at any stage. They offer the convenience of setting User properties do not impact tasks execution and can be modified at any stage. They are convenient for setting
helpful values which then be displayed in the [experiment table](../webapp/webapp_exp_table.md) (i.e. customize columns), helpful values which are displayed in the [experiment table](../webapp/webapp_exp_table.md) (i.e. customize columns),
making it easier to search / filter experiments. Add user properties to an experiment with the making it easier to search / filter experiments. Add user properties to an experiment with the
[`Task.set_user_properties`](../references/sdk/task.md#set_user_properties) method. [`Task.set_user_properties`](../references/sdk/task.md#set_user_properties) method.

View File

@ -22,19 +22,19 @@ During early stages of model development, while code is still being modified hea
the model and ensure that you choose a model that makes sense, and the training procedure works. Can be used to provide initial models for testing. the model and ensure that you choose a model that makes sense, and the training procedure works. Can be used to provide initial models for testing.
The abovementioned setups might be folded into each other and that's great! If you have a GPU machine for each researcher, that's awesome! The abovementioned setups might be folded into each other and that's great! If you have a GPU machine for each researcher, that's awesome!
The goal of this phase is to get a code, dataset and environment setup, so we can start digging to find the best model! The goal of this phase is to get a code, dataset, and environment setup, so you can start digging to find the best model!
- [ClearML SDK](../../clearml_sdk/clearml_sdk.md) should be integrated into your code (check out our [getting started](ds_first_steps.md)). - [ClearML SDK](../../clearml_sdk/clearml_sdk.md) should be integrated into your code (check out our [getting started](ds_first_steps.md)).
This helps visualizing the results and tracking progress. This helps visualizing the results and tracking progress.
- [ClearML Agent](../../clearml_agent.md) helps moving your work to other machines without the hassle of rebuilding the environment every time, - [ClearML Agent](../../clearml_agent.md) helps moving your work to other machines without the hassle of rebuilding the environment every time,
while also creating an easy queue interface that easily allows you to just drop your experiments to be executed one by one while also creating an easy queue interface that easily lets you just drop your experiments to be executed one by one
(great for ensuring that the GPUs are churning during the weekend). (great for ensuring that the GPUs are churning during the weekend).
- [ClearML Session](../../apps/clearml_session.md) helps with developing on remote machines, just like you'd develop on you local laptop! - [ClearML Session](../../apps/clearml_session.md) helps with developing on remote machines, just like you'd develop on you local laptop!
## Train Remotely ## Train Remotely
In this phase, we scale our training efforts, and try to come up with the best code / parameter / data combination that In this phase, you scale your training efforts, and try to come up with the best code / parameter / data combination that
yields the best performing model for our task! yields the best performing model for your task!
- The real training (usually) should **not** be executed on your development machine. - The real training (usually) should **not** be executed on your development machine.
- Training sessions should be launched and monitored from a web UI. - Training sessions should be launched and monitored from a web UI.
@ -55,8 +55,8 @@ that we need.
## Track EVERYTHING ## Track EVERYTHING
We believe that you should track everything! From obscure parameters to weird metrics, it's impossible to know what will end up Track everything--from obscure parameters to weird metrics, it's impossible to know what will end up
improving our results later on! improving your results later on!
- Make sure experiments are reproducible! ClearML logs code, parameters, environment in a single, easily searchable place. - Make sure experiments are reproducible! ClearML logs code, parameters, environment in a single, easily searchable place.
- Development is not linear. Configuration / Parameters should not be stored in your git, as - Development is not linear. Configuration / Parameters should not be stored in your git, as

View File

@ -2,7 +2,7 @@
title: Next Steps title: Next Steps
--- ---
So, we've already [installed ClearML's python package](ds_first_steps.md) and ran our first experiment! So, you've already [installed ClearML's python package](ds_first_steps.md) and run your first experiment!
Now, we'll learn how to track Hyperparameters, Artifacts and Metrics! Now, we'll learn how to track Hyperparameters, Artifacts and Metrics!
@ -19,7 +19,7 @@ or project & name combination. It's also possible to query tasks based on their
prev_task = Task.get_task(task_id='123456deadbeef') prev_task = Task.get_task(task_id='123456deadbeef')
``` ```
Once we have a Task object we can query the state of the Task, get its Model, scalars, parameters, etc. Once you have a Task object you can query the state of the Task, get its model, scalars, parameters, etc.
## Log Hyperparameters ## Log Hyperparameters
@ -40,7 +40,7 @@ Check [this](../../fundamentals/hyperparameters.md) out for all Hyperparameter l
## Log Artifacts ## Log Artifacts
ClearML allows you to easily store the output products of an experiment - Model snapshot / weights file, a preprocessing of your data, feature representation of data and more! ClearML lets you easily store the output products of an experiment - Model snapshot / weights file, a preprocessing of your data, feature representation of data and more!
Essentially, artifacts are files (or python objects) uploaded from a script and are stored alongside the Task. Essentially, artifacts are files (or python objects) uploaded from a script and are stored alongside the Task.
These Artifacts can be easily accessed by the web UI or programmatically. These Artifacts can be easily accessed by the web UI or programmatically.
@ -56,12 +56,12 @@ Uploading a local file containing the preprocessed results of the data:
task.upload_artifact('/path/to/preprocess_data.csv', name='data') task.upload_artifact('/path/to/preprocess_data.csv', name='data')
``` ```
We can also upload an entire folder with all its content by passing the folder (the folder will be zipped and uploaded as a single zip file). You can also upload an entire folder with all its content by passing the folder (the folder will be zipped and uploaded as a single zip file).
```python ```python
task.upload_artifact('/path/to/folder/', name='folder') task.upload_artifact('/path/to/folder/', name='folder')
``` ```
Lastly, we can upload an instance of an object; Numpy/Pandas/PIL Images are supported with npz/csv.gz/jpg formats accordingly. Lastly, you can upload an instance of an object; Numpy/Pandas/PIL Images are supported with npz/csv.gz/jpg formats accordingly.
If the object type is unknown ClearML pickles it and uploads the pickle file. If the object type is unknown ClearML pickles it and uploads the pickle file.
```python ```python
@ -128,11 +128,11 @@ local_weights_path = last_snapshot.get_local_copy()
Like before we have to get the instance of the Task training the original weights files, then we can query the task for its output models (a list of snapshots), and get the latest snapshot. Like before we have to get the instance of the Task training the original weights files, then we can query the task for its output models (a list of snapshots), and get the latest snapshot.
:::note :::note
Using TensorFlow, the snapshots are stored in a folder, meaning the `local_weights_path` will point to a folder containing our requested snapshot. Using TensorFlow, the snapshots are stored in a folder, meaning the `local_weights_path` will point to a folder containing your requested snapshot.
::: :::
As with Artifacts, all models are cached, meaning the next time we run this code, no model needs to be downloaded. As with Artifacts, all models are cached, meaning the next time we run this code, no model needs to be downloaded.
Once one of the frameworks will load the weights file, the running Task will be automatically updated with “Input Model” pointing directly to the original training Tasks Model. Once one of the frameworks will load the weights file, the running Task will be automatically updated with “Input Model” pointing directly to the original training Tasks Model.
This feature allows you to easily get a full genealogy of every trained and used model by your system! This feature lets you easily get a full genealogy of every trained and used model by your system!
## Log Metrics ## Log Metrics

View File

@ -17,11 +17,11 @@ If you are afraid of clutter, use the archive option, and set up your own [clean
These metrics can later be part of your own in-house monitoring solution, don't let good data go to waste :) These metrics can later be part of your own in-house monitoring solution, don't let good data go to waste :)
## Clone Tasks ## Clone Tasks
In order to define a Task in ClearML we have two options Define a ClearML Task with one of the following options:
- Run the actual code with `Task.init` call. This will create and auto-populate the Task in CleaML (including Git Repo / Python Packages / Command line etc.). - Run the actual code with `Task.init` call. This will create and auto-populate the Task in CleaML (including Git Repo / Python Packages / Command line etc.).
- Register local / remote code repository with `clearml-task`. See [details](../../apps/clearml_task.md). - Register local / remote code repository with `clearml-task`. See [details](../../apps/clearml_task.md).
Once we have a Task in ClearML, we can clone and edit its definitions in the UI, then launch it on one of our nodes with [ClearML Agent](../../clearml_agent.md). Once you have a Task in ClearML, you can clone and edit its definitions in the UI, then launch it on one of your nodes with [ClearML Agent](../../clearml_agent.md).
## Advanced Automation ## Advanced Automation
- Create daily / weekly cron jobs for retraining best performing models on. - Create daily / weekly cron jobs for retraining best performing models on.

View File

@ -164,7 +164,7 @@ and [pipeline](../../pipelines/pipelines.md) solutions.
Logging models into the model repository is the easiest way to integrate the development process directly with production. Logging models into the model repository is the easiest way to integrate the development process directly with production.
Any model stored by a supported framework (Keras / TensorFlow / PyTorch / Joblib etc.) will be automatically logged into ClearML. Any model stored by a supported framework (Keras / TensorFlow / PyTorch / Joblib etc.) will be automatically logged into ClearML.
ClearML also offers methods to explicitly log models. Models can be automatically stored on a preferred storage medium ClearML also supports methods to explicitly log models. Models can be automatically stored on a preferred storage medium
(s3 bucket, google storage, etc.). (s3 bucket, google storage, etc.).
#### Log Metrics #### Log Metrics
@ -208,7 +208,7 @@ tasks = Task.get_tasks(
Data is probably one of the biggest factors that determines the success of a project. Associating a models data with Data is probably one of the biggest factors that determines the success of a project. Associating a models data with
the model's configuration, code, and results (such as accuracy) is key to deducing meaningful insights into model behavior. the model's configuration, code, and results (such as accuracy) is key to deducing meaningful insights into model behavior.
[ClearML Data](../../clearml_data/clearml_data.md) allows you to version your data, so it's never lost, fetch it from every [ClearML Data](../../clearml_data/clearml_data.md) lets you version your data, so it's never lost, fetch it from every
machine with minimal code changes, and associate data to experiment results. machine with minimal code changes, and associate data to experiment results.
Logging data can be done via command line, or programmatically. If any preprocessing code is involved, ClearML logs it Logging data can be done via command line, or programmatically. If any preprocessing code is involved, ClearML logs it

View File

@ -16,19 +16,19 @@ The sections below describe the following scenarios:
## Building Tasks ## Building Tasks
### Dataset Creation ### Dataset Creation
Let's assume we have some code that extracts data from a production database into a local folder. Let's assume you have some code that extracts data from a production database into a local folder.
Our goal is to create an immutable copy of the data to be used by further steps: Your goal is to create an immutable copy of the data to be used by further steps:
```bash ```bash
clearml-data create --project data --name dataset clearml-data create --project data --name dataset
clearml-data sync --folder ./from_production clearml-data sync --folder ./from_production
``` ```
We could also add a tag `latest` to the Dataset, marking it as the latest version. You can add a tag `latest` to the Dataset, marking it as the latest version.
### Preprocessing Data ### Preprocessing Data
The second step is to preprocess the data. First we need to access it, then we want to modify it, The second step is to preprocess the data. First access the data, then modify it,
and lastly we want to create a new version of the data. and lastly create a new version of the data.
```python ```python
# create a task for the data processing part # create a task for the data processing part
@ -59,10 +59,10 @@ dataset.tags = []
new_dataset.tags = ['latest'] new_dataset.tags = ['latest']
``` ```
We passed the `parents` argument when we created v2 of the Dataset, which inherits all the parent's version content. The new dataset inherits the contents of the datasets specified in `Dataset.create`'s `parents` argument.
This not only helps trace back dataset changes with full genealogy, but also makes our storage more efficient, This not only helps trace back dataset changes with full genealogy, but also makes the storage more efficient,
since it only stores the changed and / or added files from the parent versions. since it only stores the changed and / or added files from the parent versions.
When we access the Dataset, it automatically merges the files from all parent versions When you access the Dataset, it automatically merges the files from all parent versions
in a fully automatic and transparent process, as if the files were always part of the requested Dataset. in a fully automatic and transparent process, as if the files were always part of the requested Dataset.
### Training ### Training

View File

@ -2,13 +2,13 @@
title: Dataset Management with CLI and SDK title: Dataset Management with CLI and SDK
--- ---
In this tutorial, we are going to manage the CIFAR dataset with `clearml-data` CLI, and then use ClearML's [`Dataset`](../../references/sdk/dataset.md) In this tutorial, you are going to manage the CIFAR dataset with `clearml-data` CLI, and then use ClearML's [`Dataset`](../../references/sdk/dataset.md)
class to ingest the data. class to ingest the data.
## Creating the Dataset ## Creating the Dataset
### Downloading the Data ### Downloading the Data
Before we can register the CIFAR dataset with `clearml-data`, we need to obtain a local copy of it. Before registering the CIFAR dataset with `clearml-data`, you need to obtain a local copy of it.
Execute this python script to download the data Execute this python script to download the data
```python ```python
@ -43,7 +43,7 @@ New dataset created id=ee1c35f60f384e65bc800f42f0aca5ec
Where `ee1c35f60f384e65bc800f42f0aca5ec` is the dataset ID. Where `ee1c35f60f384e65bc800f42f0aca5ec` is the dataset ID.
## Adding Files ## Adding Files
Add the files we just downloaded to the dataset: Add the files that were just downloaded to the dataset:
``` ```
clearml-data add --files <dataset_path> clearml-data add --files <dataset_path>
@ -72,7 +72,7 @@ In the panel's **CONTENT** tab, you can see a table summarizing version contents
## Using the Dataset ## Using the Dataset
Now that we have a new dataset registered, we can consume it. Now that a new dataset is registered, you can consume it.
The [data_ingestion.py](https://github.com/allegroai/clearml/blob/master/examples/datasets/data_ingestion.py) example The [data_ingestion.py](https://github.com/allegroai/clearml/blob/master/examples/datasets/data_ingestion.py) example
script demonstrates using the dataset within Python code. script demonstrates using the dataset within Python code.
@ -103,6 +103,6 @@ hyperparameters. Passing `alias=<dataset_alias_string>` stores the datasets I
you can easily track which dataset the task is using. you can easily track which dataset the task is using.
The Dataset's [`get_local_copy`](../../references/sdk/dataset.md#get_local_copy) method will return a path to the cached, The Dataset's [`get_local_copy`](../../references/sdk/dataset.md#get_local_copy) method will return a path to the cached,
downloaded dataset. Then we provide the path to PyTorch's dataset object. downloaded dataset. Then the dataset path is input to PyTorch's `datasets` object.
The script then trains a neural network to classify images using the dataset created above. The script then trains a neural network to classify images using the dataset created above.

View File

@ -18,7 +18,7 @@ demonstrates how to do the following:
### Downloading the Data ### Downloading the Data
We first need to obtain a local copy of the CIFAR dataset. You first need to obtain a local copy of the CIFAR dataset.
```python ```python
from clearml import StorageManager from clearml import StorageManager
@ -79,7 +79,7 @@ In the panel's **CONTENT** tab, you can see a table summarizing version contents
## Data Ingestion ## Data Ingestion
Now that we have a new dataset registered, we can consume it! Now that a new dataset is registered, you can consume it!
The [data_ingestion.py](https://github.com/allegroai/clearml/blob/master/examples/datasets/data_ingestion.py) script The [data_ingestion.py](https://github.com/allegroai/clearml/blob/master/examples/datasets/data_ingestion.py) script
demonstrates data ingestion using the dataset created in the first script. demonstrates data ingestion using the dataset created in the first script.

View File

@ -62,7 +62,7 @@ ClearML automatically tracks images logged to TensorboardLogger. They appear in
## Ignite ClearMLLogger ## Ignite ClearMLLogger
PyTorch Ignite also offers a dedicated `ClearMLLogger` handler to log metrics, text, model / optimizer parameters, plots, and model PyTorch Ignite also supports a dedicated `ClearMLLogger` handler to log metrics, text, model / optimizer parameters, plots, and model
checkpoints during training and validation. checkpoints during training and validation.
For more information, see the [PyTorch Ignite ClearMLLogger](pytorch_ignite_mnist.md) For more information, see the [PyTorch Ignite ClearMLLogger](pytorch_ignite_mnist.md)

View File

@ -5,7 +5,7 @@ title: PyTorch Ignite ClearMLLogger
The `ignite` repository contains the [mnist_with_clearml_logger.py](https://github.com/pytorch/ignite/blob/master/examples/contrib/mnist/mnist_with_clearml_logger.py) The `ignite` repository contains the [mnist_with_clearml_logger.py](https://github.com/pytorch/ignite/blob/master/examples/contrib/mnist/mnist_with_clearml_logger.py)
example script that uses [ignite](https://github.com/pytorch/ignite) and integrates **ClearMLLogger** and its [helper handlers](https://pytorch.org/ignite/generated/ignite.contrib.handlers.clearml_logger.html). example script that uses [ignite](https://github.com/pytorch/ignite) and integrates **ClearMLLogger** and its [helper handlers](https://pytorch.org/ignite/generated/ignite.contrib.handlers.clearml_logger.html).
PyTorch Ignite offers a `ClearMLLogger` handler to log metrics, text, model / optimizer parameters, plots, and model PyTorch Ignite supports a `ClearMLLogger` handler to log metrics, text, model / optimizer parameters, plots, and model
checkpoints during training and validation. checkpoints during training and validation.
The example script does the following: The example script does the following:

View File

@ -2,7 +2,7 @@
title: ClearML Agent on Google Colab title: ClearML Agent on Google Colab
--- ---
[Google Colab](https://colab.research.google.com) is a common development environment for data scientists. It offers a convenient IDE as well as [Google Colab](https://colab.research.google.com) is a common development environment for data scientists. It supports a convenient IDE as well as
compute provided by google. compute provided by google.
Users can transform a Google Colab instance into an available resource in ClearML using [ClearML Agent](../../clearml_agent.md). Users can transform a Google Colab instance into an available resource in ClearML using [ClearML Agent](../../clearml_agent.md).

View File

@ -51,7 +51,7 @@ task.register_artifact('train', df, metadata={'counting': 'legs', 'max legs': 69
``` ```
By changing the artifact, and calling the [Task.get_registered_artifacts](../../references/sdk/task.md#get_registered_artifacts) By changing the artifact, and calling the [Task.get_registered_artifacts](../../references/sdk/task.md#get_registered_artifacts)
method to retrieve it, we can see that ClearML tracked the change. method to retrieve it, you can see that ClearML tracked the change.
```python ```python
# change the artifact object # change the artifact object

View File

@ -11,7 +11,7 @@ filtering logic.
## Frame Viewer ## Frame Viewer
Frame viewer allows you to view and edit annotations which can be FrameGroup objects (Regions of Interest) and FrameGroup Frame viewer lets you view and edit annotations which can be FrameGroup objects (Regions of Interest) and FrameGroup
labels applied to the entire frame not a region of the frame, the frame details (see [frames](../frames.md)), labels applied to the entire frame not a region of the frame, the frame details (see [frames](../frames.md)),
frame metadata, the raw data source URI, as well as providing navigation and viewing tools. frame metadata, the raw data source URI, as well as providing navigation and viewing tools.

View File

@ -179,7 +179,7 @@ google.storage {
## Storage Manager ## Storage Manager
ClearML offers the [StorageManager](../references/sdk/storage.md) class to manage downloading, uploading, and caching of ClearML provides the [StorageManager](../references/sdk/storage.md) class to manage downloading, uploading, and caching of
content directly from code. content directly from code.
See [Storage Examples](../guides/storage/examples_storagehelper.md). See [Storage Examples](../guides/storage/examples_storagehelper.md).

View File

@ -50,7 +50,7 @@ Each pipeline step can log additional artifacts and metrics on the step task wit
Additionally, pipeline steps can directly report metrics or upload artifacts / models to the pipeline using these Additionally, pipeline steps can directly report metrics or upload artifacts / models to the pipeline using these
PipelineController and PipelineDecorator class methods: `get_logger`, `upload_model`, `upload_artifact`. PipelineController and PipelineDecorator class methods: `get_logger`, `upload_model`, `upload_artifact`.
The pipeline controller also offers automation for logging step metrics / artifacts / models on the pipeline task itself. The pipeline controller also supports automation for logging step metrics / artifacts / models on the pipeline task itself.
Each pipeline step can specify metrics / artifacts / models to also automatically log to the pipeline Task. The idea is Each pipeline step can specify metrics / artifacts / models to also automatically log to the pipeline Task. The idea is
that pipeline steps report metrics internally while the pipeline automatically collects them into a unified view on the that pipeline steps report metrics internally while the pipeline automatically collects them into a unified view on the
pipeline Task. To enable the automatic logging, use the `monitor_metrics`, `monitor_artifacts`, `monitor_models` arguments pipeline Task. To enable the automatic logging, use the `monitor_metrics`, `monitor_artifacts`, `monitor_models` arguments

View File

@ -108,7 +108,7 @@ def step_one(pickle_data_url: str, extra: int = 43):
* `repo_branch` (Optional) - Specify the remote repository branch (Ignored, if local repo path is used) * `repo_branch` (Optional) - Specify the remote repository branch (Ignored, if local repo path is used)
* `repo_commit` (Optional) - Specify the repository commit ID (Ignored, if local repo path is used) * `repo_commit` (Optional) - Specify the repository commit ID (Ignored, if local repo path is used)
* `helper_functions` (Optional) - A list of helper functions to make available for the standalone pipeline step. By default, the pipeline step function has no access to any of the other functions, by specifying additional functions here, the remote pipeline step could call the additional functions. * `helper_functions` (Optional) - A list of helper functions to make available for the standalone pipeline step. By default, the pipeline step function has no access to any of the other functions, by specifying additional functions here, the remote pipeline step could call the additional functions.
Example, assuming we have two functions, `parse_data()` and `load_data()`: `[parse_data, load_data]` Example, assuming you have two functions, `parse_data()` and `load_data()`: `[parse_data, load_data]`
* `parents` Optional list of parent steps in the pipeline. The current step in the pipeline will be sent for execution only after all the parent steps have been executed successfully. * `parents` Optional list of parent steps in the pipeline. The current step in the pipeline will be sent for execution only after all the parent steps have been executed successfully.
Additionally, you can enable automatic logging of a steps metrics / artifacts / models to the pipeline task using the Additionally, you can enable automatic logging of a steps metrics / artifacts / models to the pipeline task using the

View File

@ -175,7 +175,7 @@ def step_created_callback(
pass pass
``` ```
A `post_execute_callback` function is called when a step is completed. It allows you to modify the steps status after completion. A `post_execute_callback` function is called when a step is completed. It lets you modify the steps status after completion.
```python ```python
def step_completed_callback( def step_completed_callback(