ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-06-26 18:17:44 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Rewrite fundamentals sections (#252)

Browse Source
This commit is contained in:
pollfly
2022-05-18 11:49:31 +03:00
committed by GitHub
parent a9b5bd9697
commit 1a46cbdb93
24 changed files with 1045 additions and 660 deletions
Download Patch File Download Diff File Expand all files Collapse all files

238
docs/fundamentals/artifacts.md
View File

@@ -1,193 +1,77 @@
---
title: Artifacts & Models
title: Models
---
ClearML allows easy storage of experiments' output products as **artifacts** that can later be accessed easily
and used, through the web UI or programmatically.
ClearML supports tracking, updating, and visualizing models.
A few examples of artifacts are:
* Model snapshot / weights file
* Data preprocessing
* Feature representation of data
* and more!
Models are stored in ClearML as experiment artifacts, but unlike other artifacts that are dependent on their creating
task, models are independent entities with their own unique ID. Models can be accessed directly with a model object or
indirectly via their creating task. This property makes Models a standalone entry that can be used as an artifactory
interface.
## Artifacts
### Logging Artifacts
To log any type of artifact to a Task, use the [`upload_artifact`](../references/sdk/task.md#upload_artifact) method.
For example:
## Automatically Logging Models
* Upload a local file containing the preprocessing results of the data.
```python
task.upload_artifact(name='data', artifact_object='/path/to/preprocess_data.csv')
```
* Upload an entire folder with all its content by passing the folder, which will be zipped and uploaded as a single
zip file:
```python
task.upload_artifact(name='folder', artifact_object='/path/to/folder/')
```
* Serialize and upload a Python object. ClearML automatically chooses the file format based on the object’s type, or you
can explicitly specify the format as follows:
* dict - `.json` (default), `.yaml`
* pandas.DataFrame - `.csv.gz` (default), `.parquet`, `.feather`, `.pickle`
* numpy.ndarray - `.npz` (default), `.csv.gz`
* PIL.Image - Any PIL-supported extensions (default `.png`)
Once integrated into code, ClearML automatically logs and tracks models and any snapshots created by the following
frameworks:
- Tensorflow (see [code example](../guides/frameworks/tensorflow/tensorflow_mnist.md))
- Keras (see [code example](../guides/frameworks/keras/keras_tensorboard.md))
- Pytorch (see [code example](../guides/frameworks/pytorch/pytorch_mnist.md))
- scikit-learn (only using joblib) (see [code example](../guides/frameworks/scikit-learn/sklearn_joblib_example.md))
- XGBoost (only using joblib) (see [code example](../guides/frameworks/xgboost/xgboost_sample.md))
- FastAI (see [code example](../guides/frameworks/fastai/fastai_with_tensorboard.md))
- MegEngine (see [code example](../guides/frameworks/megengine/megengine_mnist.md))
- CatBoost (see [code example](../guides/frameworks/catboost/catboost.md))
When a supported framework loads a weights file, the running task will be automatically updated, with its input model
pointing directly to the original training task's model.
## Manually Logging Models
### Output Models
ClearML stores training results as output models. The `OutputModel` object is instantiated with a task object as an
argument (see [`task`](../references/sdk/model_outputmodel.md) parameter), so it's automatically registered as the Task’s
output model. Since OutputModel objects are connected to tasks, the models are traceable in experiments.
Output models are read-write so weights can be updated throughout training. Additionally, users can specify a model's
network design and label enumeration. Once an output model is registered, it can be used as the input model for another
experiment.
The snapshots of manually uploaded models aren't automatically captured, but ClearML provides methods to update them
through a `Task` or `OutputModel` object.
### Input Models
ClearML provides flexibility for explicitly connecting input models and experimentation, including:
* Importing pre-trained models from external sources such as Amazon AWS, GIT repositories, PyTorch, and TensorFlow.
* Using standalone models already registered in ClearML by previously run experiments.
* Defining your own input models in scripts
For example:
```python
person_dict = {'name': 'Erik', 'age': 30}
# upload as JSON artifact
task.upload_artifact(name='person dictionary json', artifact_object=person_dict)
# upload as YAML artifact
task.upload_artifact(
name='person dictionary yaml',
artifact_object=person_dict,
extension_name="yaml"
)
```
## Setting Upload Destination
See more details in the artifacts [example](../guides/reporting/artifacts.md).
* ClearML automatically captures the storage path of Models created by supported frameworks. By default, it stores the
local path they are saved to.
* Upload destinations can be specified explicitly on a per OutputModel or per experiment basis. Alternatively, the upload
destination of all OutputModels can be specified in the ClearML [configuration file](../configs/clearml_conf.md).
### Using Artifacts
To access a Task's artifact in order to use it:
1. Get the Task that created the artifact (see more details on [querying](task.md#querying--searching-tasks)
Tasks).
## WebApp Interface
1. Retrieve all the Task's artifacts with the *artifact* property, which is essentially a dictionary,
where the key is the artifact name, and the value is the artifact itself.
1. Access a specific artifact using one of the following methods:
- Access files by calling `get_local_copy()`, which caches the files for later use and returns a path to the cached
file
- Access object artifacts by using the `get()` method that returns the Python object.
The code below demonstrates how to access a file artifact using the previously generated preprocessed data:
```python
# get instance of Task that created artifact, using Task ID
preprocess_task = Task.get_task(task_id='the_preprocessing_task_id')
# access artifact
local_csv = preprocess_task.artifacts['data'].get_local_copy()
```
In the ClearML's web UI, model information can be located through a project's Model Table or through the model's creating
task.
See more details in the using artifacts [example](../guides/reporting/using_artifacts.md).
Models associated with a task appear in the task's **ARTIFACTS** tab. To see further model details, including design,
label enumeration, and general information, click the model name, which is a hyperlink to the
[model's detail page](../webapp/webapp_model_viewing.md).
### List of Supported Artifacts
Models can also be accessed through their associated project's [Model Table](../webapp/webapp_model_table.md), where all
the models associated with a project are listed.
- Numpy array (as npz file)
- Pandas dataframe
- PIL (converted to jpg)
- Files and folders
- Python objects (pickled)
![WebApp Model](../img/examples_model_update_model.png)
## Models
Models are a special kind of artifact and, unlike regular artifacts, which can only be accessed with the creating Task's ID,
Models are entities with their own unique ID that can be accessed directly or via the creating task.
## SDK Interface
This property makes Models a standalone entry that can be used as an artifactory interface.
See [the Models SDK interface](../clearml_sdk/model_sdk.md) for an overview for using the most basic Pythonic methods of the model
classes. See a detailed list of all available methods in the [Model](../references/sdk/model_model.md), [OutputModel](../references/sdk/model_outputmodel.md), and [InputModel](../references/sdk/model_inputmodel.md)
reference pages.
### Automatic Model Logging
When models are saved using certain frameworks (for instance, by calling the `torch.save()` method), ClearML automatically
logs the models and all snapshot paths.
![image](../img/examples_model_update_model.png)
See automatic model logging examples:
* [TF](../guides/frameworks/tensorflow/tensorflow_mnist.md)
* [PyTorch](../guides/frameworks/pytorch/pytorch_mnist.md)
* [Keras](../guides/frameworks/keras/keras_tensorboard.md)
* [Scikit-Learn](../guides/frameworks/scikit-learn/sklearn_joblib_example.md)
* [XGBoost](../guides/frameworks/xgboost/xgboost_sample.md)
* [FastAI](../guides/frameworks/fastai/fastai_with_tensorboard.md)
### Manual Model Logging
To manually log a model, create an instance of OutputModel class:
```python
from clearml import OutputModel, Task
# Instantiate a Task
task = Task.init(project_name="myProject", task_name="myTask")
# Instantiate an OutputModel, with a Task object argument
output_model = OutputModel(task=task, framework="PyTorch")
```
The OutputModel object is always connected to a Task object as it's instantiated with a Task object as an argument.
It is, therefore, automatically registered as the Task’s output model.
The snapshots of manually uploaded models aren't automatically captured, but there are two methods
to update an output model.
#### Updating Via Task Object
Using the [Task.update_output_model](../references/sdk/task.md#update_output_model) method:
```python
task.update_output_model(model_path='path/to/model')
```
It's possible to modify the following parameters:
* Weights file / folder - Uploads the files specified with the `model_path`.
If a remote storage is provided (S3 / GS / Https etc...), it saves the URL.
* Model Metadata - Model name, description, iteration number of model, and tags.
#### Updating Via Model Object
Using the [OutputModel.update_weights](../references/sdk/model_outputmodel.md#update_weights) method:
```python
output_model.update_weights()
```
* Specify either the name of a locally stored weights file to upload (`weights_filename`), or the URI of a storage destination
for model weight upload (`registered_uri`).
* Model Metadata - Model description and iteration number.
See [Model Configuration](../guides/reporting/model_config.md) example.
### Using Models
Loading a previously trained model is quite similar to loading artifacts.
```python
prev_task = Task.get_task(task_id='the_training_task')
last_snapshot = prev_task.models['output'][-1]
local_weights_path = last_snapshot.get_local_copy()
```
1. Get the instance of the Task that created the original weights files
2. Query the Task on its output models (a list of snapshots)
3. Get the latest snapshot (if using Tensorflow, the snapshots are stored in a folder, so the `local_weights_path` will point to a folder containing the requested snapshot).
Notice that if one of the frameworks will load the weights file, the running Task will automatically update, with
"Input Model" pointing directly to the original training Task's model. With this feature, it's easy to get a full genealogy
of every trained and used model in our system!
Loading framework models appear under the "Input Models" section, under the Artifacts tab in the ClearML UI.
### Setting Upload Destination
ClearML automatically captures the storage path of Models created by frameworks such as TF, Pytorch, and scikit-learn. By default,
it stores the local loading path they are saved to.
To automatically store all created models by a specific experiment, modify the `Task.init()` function as such:
```python
task = Task.init(project_name='examples', task_name='storing model', output_uri='s3://my_models/')
```
To automatically store all created models from all experiments in a certain storage medium, edit the `clearml.conf` (see
[ClearML Configuration Reference](../configs/clearml_conf.md#sdkdevelopment)) and set `sdk.developmenmt.default_output_uri` to the desired
storage (see [Storage](../integrations/storage.md)).
This is especially helpful when using [clearml-agent](../clearml_agent.md) to execute code.
### List of Supported Frameworks
- TensorFlow
- Keras
- PyTorch
- PyTorch Ignite
- PyTorch Lightning
- scikit-learn (only using joblib)
- XGBoost (only using joblib)
- AutoKeras
- FastAI
- LightGBM
- MegEngine
- CatBoost

262
docs/fundamentals/hyperparameters.md
View File

@@ -2,217 +2,117 @@
title: Hyperparameters
---
Hyperparameters are the configuration options given for a script.
ClearML logs hyperparameters used in experiments from multiple different sources.
Hyperparameters are a script's configuration options. Since hyperparameters can have substantial impact on
model performance, it is crucial to efficiently track and manage them.
In ClearML, parameters are split into 3 sections:
- User Properties - Modifiable section that can be edited post execution.
- Hyperparameters - Individual parameters for configuration.
- Configuration Objects - Usually configuration files (Json / YAML) or python objects.
These sections are further broken down into sub-sections (General / Args / TF_Define) for convenience.
ClearML supports tracking and managing hyperparameters in each experiment and provides a dedicated [hyperparameter
optimization module](hpo.md). With ClearML's logging and tracking capabilities, experiments can be reproduced, and their
hyperparameters and results can be saved and compared, which is key to understanding model behavior.
![image](../img/hyperparameters_sections.png)
ClearML lets you easily try out different hyperparameter values without changing your original code. ClearML’s [execution
agent](../clearml_agent.md) will override the original values with any new ones you specify through the web UI (see
[Configuration](../webapp/webapp_exp_tuning.md#configuration) in the Tuning Experiments page). It's also possible to
programmatically set experiment parameters.
## Command Line Parsing
ClearML captures any command line parameters passed when invoking code that uses standard python packages such as
argparse, [click](https://click.palletsprojects.com), or [Python Fire](https://github.com/google/python-fire). This happens automatically with no additional code required beyond
initializing ClearML.
## Tracking Hyperparameters
Hyperparameters can be added from anywhere in your code, and ClearML provides multiple ways to obtain them! ClearML logs
and tracks hyperparameters of various types, supporting automatic logging and explicit reporting.
### Argparse Example
### Automatic Logging
Once a ClearML Task has been [initialized](../references/sdk/task.md#taskinit) in a script, ClearML automatically captures and tracks
the following types of parameters:
* Command line parsing - command line parameters passed when invoking code that uses standard python packages, including:
* [click](https://click.palletsprojects.com) - see code example [here](https://github.com/allegroai/clearml/blob/master/examples/frameworks/click/click_multi_cmd.py).
* [argparse](https://docs.python.org/3/library/argparse.html) - see code example [here](../guides/frameworks/pytorch/pytorch_tensorboardx.md).
* [Python Fire](https://github.com/google/python-fire) - see code examples [here](https://github.com/allegroai/clearml/tree/master/examples/frameworks/fire).
* [LightningCLI](https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_cli.html) - see code example [here](https://github.com/allegroai/clearml/blob/master/examples/frameworks/jsonargparse/pytorch_lightning_cli.py).
* TensorFlow Definitions (`absl-py`). See examples of ClearML's automatic logging of TF Defines:
* [TensorFlow MNIST](../guides/frameworks/tensorflow/tensorflow_mnist.md)
* [TensorBoard PR Curve](../guides/frameworks/tensorflow/tensorboard_pr_curve.md)
* [Hydra](https://github.com/facebookresearch/hydra) - ClearML logs the `Omegaconf` which holds all the configuration files,
as well as values overridden during runtime. See code example [here](https://github.com/allegroai/clearml/blob/master/examples/frameworks/hydra/hydra_example.py).
:::tip Disabling Automatic Logging
Automatic logging can be disabled. See this [FAQ](../faq.md#controlling_logging).
:::
```python
from clearml import Task
import argparse
### Environment Variables
parser = argparse.ArgumentParser(description="Script Argparser")
parser.add_argument("-lr", default=0.001, help="Initial learning rate")
parser.add_argument("-epochs", default= 10, help="Total number of epochs")
args = parser.parse_args()
task = Task.init(project_name="examples",task_name="argparser logging")
```
See another argparse logging example [here](../guides/reporting/hyper_parameters.md).
### Click Example
```python
from clearml import Task
import click
task = Task.init(project_name='examples', task_name='click single command')
@click.command()
@click.option('--count', default=1, help='Number of greetings.')
@click.option('--name', prompt='Your name',
help='The person to greet.')
def hello(count, name):
for x in range(count):
click.echo("Hello {}!".format(name))
hello()
```
See another code example [here](https://github.com/allegroai/clearml/blob/master/examples/frameworks/click/click_multi_cmd.py).
## Connecting Objects
Users can directly connect objects, such as dictionaries or even custom classes, to Tasks.
All class members will be automatically fetched and logged by ClearML.
* Connecting a class:
```python
class person:
def __init__(self, name, age):
self.name = name
self.age = age
me = person('Erik',5)
task = Task.init(project_name='examples',task_name='argparser')
task.connect(me)
```
See connecting configuration objects example [here](../guides/reporting/hyper_parameters.md).
* Connecting a dictionary:
```python
task = Task.init(project_name='examples', task_name='dictionary logging')
params_dictionary = {'epochs': 3, 'lr': 0.4}
task.connect(params_dictionary)
```
## User Properties
User properties are an editable key / value store, which enables adding information to an experiment,
making it easier to search / filter. User properties are like parameters that can also be added after a Task's execution, which
can also be added to an [experiment table](../webapp/webapp_exp_table.md) (i.e. customize columns).
For example:
```python
task.set_user_properties({"name": "backbone",
"description": "network type",
"value": "great"})
```
The above example will connect to the Task a user property named "backbone", with the description "network type", and
the value of "great".
## Environment Variables
:::important
:::important Experiment Reproducibility
Relying on environment variables makes an experiment not fully reproducible, since ClearML Agent can't reproduce them at
runtime.
:::
Environment variables can be logged by modifying the [clearml.conf](../configs/clearml_conf.md) file. Modify the *log_os_environments*
Environment variables can be logged by modifying the [clearml.conf](../configs/clearml_conf.md) file. Modify the `log_os_environments`
parameter specifying parameters to log.
`log_os_environments: ["AWS_*", "CUDA_VERSION"]`
```editorconfig
log_os_environments: ["AWS_*", "CUDA_VERSION"]`
```
It's also possible to specify environment variables using the CLEARML_LOG_ENVIRONMENT variable.
It's also possible to specify environment variables using the `CLEARML_LOG_ENVIRONMENT` variable.
:::note
The CLEARML_LOG_ENVIRONMENT always overrides the clearml.conf file.
:::note Overriding clearml.conf
The `CLEARML_LOG_ENVIRONMENT` always overrides the `clearml.conf` file.
:::
## TF Defines
When a script that has integrated ClearML is executed, the environment variables listed in `clearml.conf` or specified by
the `CLEARML_LOG_ENVIRONMENT` variable are logged by ClearML.
ClearML automatically captures TensorFlow definitions, which are used as configuration files for Tensorflow.
### Explicit Logging
See examples of ClearML's automatic logging of TF Defines:
* [TensorFlow MNIST](../guides/frameworks/tensorflow/tensorflow_mnist.md)
* [TensorBoard PR Curve](../guides/frameworks/tensorflow/tensorboard_pr_curve.md)
To augment its automatic logging, ClearML supports explicitly logging parameters. ClearML provides methods to directly
connect Python objects and configuration objects, as well as manually set and update task parameters.
## Hydra
#### Connecting Python Objects
Users can directly connect Python objects, such as dictionaries and custom classes, to tasks, using the
[Task.connect](../references/sdk/task.md#connect) method. Once objects are connected to a task, all object elements
(e.g. class members, dictionary key-values pairs) are automatically logged by ClearML. Additionally, ClearML tracks these
values as they change through your code.
[Hydra](https://github.com/facebookresearch/hydra) is a module developed by FaceBook AI Research to manage experiments'
parameters. Hydra offers the best of both worlds, managing configurations with files while making parameters overridable at runtime.
When connecting objects to ClearML, users can directly access and modify an object's elements (e.g. a specific key-value
pair in a parameter dictionary).
ClearML logs the Omegaconf which holds all the configuration files, as well as overridden values.
#### Connecting Configuration Objects
Configuration objects are dictionaries or configuration files connected to the task using the
[Task.connect_configuration](../references/sdk/task.md#connect_configuration) method. With this method, configuration
objects are saved as blobs i.e. ClearML is not aware of their internal structure.
Check out the [example code](https://github.com/allegroai/clearml/blob/master/examples/frameworks/hydra/hydra_example.py),
which demonstrates the creation of a configuration object to which configuration values can be added and overridden using the
command line.
#### Setting and Updating Parameters
ClearML provides methods to set and update task parameters manually. Use the [Task.set_parameters](../references/sdk/task.md#set_parameters)
method to define parameters manually. To update the parameters in an experiment, use the [Task.set_parameters_as_dict](../references/sdk/task.md#set_parameters_as_dict)
method. The `set_parameters_as_dict` method updates parameters while the `set_parameters` method overrides the parameters.
## Configuration Objects
ClearML does not automatically track changes to explicitly set parameters.
Configuration objects are dictionaries or configuration files connected to the Task. Unlike Hyperparameters, these are saved as a whole and not
divided into individual parameters.
### User Properties
User properties do not impact tasks execution and so can be modified at any stage. They offer the convenience of setting
helpful values which then be displayed in the [experiment table](../webapp/webapp_exp_table.md) (i.e. customize columns),
making it easier to search / filter experiments. Add user properties to an experiment with the
[Task.set_user_properties](../references/sdk/task.md#set_user_properties) method.
To connect a configuration dictionary:
```python
model_config_dict = {
'value': 13.37,
'dict': {'sub_value': 'string', 'sub_integer': 11},
'list_of_ints': [1, 2, 3, 4],
}
model_config_dict = task.connect_configuration(name='dictionary', configuration=model_config_dict)
```
### Accessing Parameters
ClearML provides methods to directly access a task’s logged parameters.
To connect a configuration file:
```python
config_file_yaml = task.connect_configuration(name="yaml file", configuration='path/to/configuration/file.yaml', )
```
To get all of a task's parameters and properties (hyperparameters, configuration objects, and user properties), use the
[Task.get_parameters](../references/sdk/task.md#get_parameters) method, which will return a dictionary with the parameters,
including their sub-sections (see [WebApp sections](#webapp-access) below).
Configuration objects can be split into categories in the Configuration section.
The "name" argument, is the name of the section that the object will go into. If a section name is not specified, the default section is *General*.
## WebApp Interface
![Configuration objects](../img/fundamentals_hyperparameters_config_objects.png)
Configurations can be viewed in web UI experiment pages, in the **CONFIGURATIONS** panel.
## Manual Parameter Access
The configuration panel is split into three sections according to type:
- **User Properties** - Modifiable section that can be edited post-execution.
- **Hyperparameters** - Individual parameters for configuration
- **Configuration Objects** - Usually configuration files (Json / YAML) or Python objects.
These sections are further broken down into sub-sections based on how the parameters were logged (General / Args / TF_Define / Environment).
### Manual Parameter Input
![Task hyperparameters sections](../img/hyperparameters_sections.png)
In addition to connecting a dictionary or a class to log hyperparameters, users can also use the `set_parameters` method
to define parameters manually. Parameters are inputted as dictionaries.
## SDK Interface
Additionally, parameters can be categorized, and each category appears in its own section in the hyperparameter tab of the web UI.
Specify a section by putting its name before the parameter, for example `'Args/epochs': 'value'` - 'epochs' will go into the
'Args' section. If a section isn't specified, the parameter will go into the *General* section by default.
Calling the `set_parameter` method will set a single parameter.
```python
task = Task.init(project_name='examples', task_name='parameters')
# override parameters with provided dictionary
task.set_parameters({'Args/epochs':7, 'lr': 0.5})
# setting a single parameter
task.set_parameter(name='decay',value=0.001)
```
:::warning
The *set_parameters* method will override any parameters already logged.
:::
### Adding Parameters
To update the parameters in an experiment, use the `set_parameters_as_dict` method . Arguments and values are inputted as a dictionary.
Like in the `set_parameters` method, the dictionary can be nested, so the parameter's section can be specified.
```python
task = Task.task_get(task_id='123456789')
# add parameters
task.set_parameters_as_dict({'my_args/lr':0.3, 'epochs':10})
```
### Accessing Parameters
To get all Task's parameters, use the `get_parameters()` method, which will return a dictionary with the parameters, including
their section.
```python
task = Task.get_task(project_name='examples', task_name='parameters')
# will print a flattened dictionary of the 'section/parameter': 'value' pairs. {'Args/epochs': '7', 'General/lr': '0.5'}
print(task.get_parameters())
```
See the [Configuration section](../clearml_sdk/task_sdk.md#configuration) of the Task SDK page for an overview of basic Pythonic
methods for working with hyperparameters.

2
docs/fundamentals/logger.md
View File

@@ -71,7 +71,7 @@ logger = task.get_logger()
### Media Reporting
ClearML also supports reporting media (such as audio, video and images) for every iteration.
This section is mostly used for debugging. It's recommended to use [artifacts](artifacts.md#artifacts) for storing script
This section is mostly used for debugging. It's recommended to use [artifacts](task.md#artifacts) for storing script
outputs that would be used later on.
Only the last X results of each title / series are saved to prevent overloading the server.

4
docs/fundamentals/projects.md
View File

@@ -2,7 +2,7 @@
title: Projects
---
Projects are contextual containers for [tasks](task.md) and [models](artifacts.md#models) (as well as [dataviews](../hyperdatasets/dataviews.md)
Projects are contextual containers for [tasks](task.md) and [models](artifacts.md) (as well as [dataviews](../hyperdatasets/dataviews.md)
when Hyper-Datasets are enabled), providing a logical structure similar to file system folders.
An often useful method is to categorize components into projects according to models or objectives.
Grouping into projects helps in identifying tasks, models, and dataviews when queried.
@@ -27,7 +27,7 @@ models, and dataviews, can be viewed in the project's [experiments table](../web
### Creating Subprojects
When [initializing a task](task.md#task-creation), its project needs to be specified. If the project entered does not exist, it will be created.
When [initializing a task](../clearml_sdk/task_sdk.md#task-creation), its project needs to be specified. If the project entered does not exist, it will be created.
Projects can contain subprojects, just like folders can contain sub-folders. Input into the `project_name`
parameter a target project path. The project path should follow the project tree hierarchy, in which the project and
subprojects are slash (`/`) delimited.

305
docs/fundamentals/task.md
View File

@@ -1,5 +1,5 @@
---
title: Task / Experiment
title: Tasks
---
**ClearML Task** lies at the heart of ClearML's experiment manager.
@@ -20,7 +20,7 @@ you can work with tasks in Offline Mode, in which all information is saved in a
In the UI and code, tasks are grouped into [projects](projects.md), which are logical entities similar to folders. Users can decide
how to group tasks, though different models or objectives are usually grouped into different projects.
Tasks that are in the system can be accessed and utilized with code. To [access a task](#accessing-tasks), it can be identified either by a
Tasks that are in the system can be accessed and utilized with code. To [access a task](../clearml_sdk/task_sdk.md#accessing-tasks), it can be identified either by a
project name & task name combination or by a unique ID.
It's possible to copy ([clone](../webapp/webapp_exp_reproducing.md)) a task multiple times and to modify it for re-execution.
@@ -33,7 +33,7 @@ The sections of **ClearML Task** are made up of the information that a task capt
execution details and execution outputs. This information is used for tracking
and visualizing results, reproducing, tuning, and comparing experiments, and executing workflows.
The captured [code execution information](../webapp/webapp_exp_track_visual.md#execution-details) includes:
The captured [code execution information](../webapp/webapp_exp_track_visual.md#execution) includes:
* Git information
* Uncommitted code modifications
* Python environment
@@ -42,17 +42,53 @@ The captured [code execution information](../webapp/webapp_exp_track_visual.md#e
The captured [execution output](../webapp/webapp_exp_track_visual.md#experiment-results) includes:
* [Console output](../webapp/webapp_exp_track_visual.md#console)
* [Scalars](../webapp/webapp_exp_track_visual.md#scalars)
* [Plots](../webapp/webapp_exp_track_visual.md#other-plots)
* [Plots](../webapp/webapp_exp_track_visual.md#plots)
* [Debug samples](../webapp/webapp_exp_track_visual.md#debug-samples)
* [Models](artifacts.md#models)
* [Models](artifacts.md)
To view a more in depth description of each task section, see [Tracking Experiments and Visualizing Results](../webapp/webapp_exp_track_visual.md).
## Task Types
### Artifacts
ClearML allows easy storage of experiments' output products as artifacts that can later be accessed easily and used,
through the [web UI](../webapp/webapp_overview.md) or programmatically.
ClearML provides methods to easily track files generated throughout your experiments’ execution such as:
- Numpy objects
- Pandas DataFrames
- PIL
- Files and folders
- Python objects
- and more!
Most importantly, ClearML also logs experiments’ input and output models as well as interim model snapshots (see
[Models](artifacts.md)).
#### Logging Artifacts
ClearML provides an explicit logging interface that supports manually reporting a variety of artifacts. Any type of
artifact can be logged to a task using the [`Task.upload_artifact`](../references/sdk/task.md#upload_artifacts) method.
See more details in the [Artifacts Reporting example](../guides/reporting/artifacts.md).
ClearML can be configured to upload artifacts to any of the supported types of storage, which include local and shared
folders, AWS S3 buckets, Google Cloud Storage, and Azure Storage. For more information, see [Storage](../integrations/storage.md).
:::note Debug Sample Storage
Debug samples are handled differently, see [`Logger.set_default_upload_destination`](../references/sdk/logger.md#set_default_upload_destination)
:::
#### Accessing Artifacts
Artifacts that have been logged can be accessed by other tasks [through the task](../clearml_sdk/task_sdk.md#accessing-tasks)
they are attached to, and then retrieving the artifact with one of its following methods:
* `get_local_copy()` - caches the files for later use and returns a path to the cached file.
* `get()` - use for Python objects. The method that returns the Python object.
See more details in the [Using Artifacts example](https://github.com/allegroai/clearml/blob/master/examples/reporting/using_artifacts_example.py).
## Task Types
Tasks have a *type* attribute, which denotes their purpose (Training / Testing / Data processing). This helps to further
organize projects and ensure tasks are easy to [search and find](#querying--searching-tasks). The default task type is *training*.
Available task types are:
organize projects and ensure tasks are easy to [search and find](../clearml_sdk/task_sdk.md#querying--searching-tasks).
The default task type is *training*. Available task types are:
- Experimentation
- *training*, *testing*, *inference*
@@ -67,7 +103,7 @@ Available task types are:
## Task Lifecycle
ClearML Tasks are created in one of the following methods:
* Manually running code that is instrumented with the ClearML SDK and invokes `Task.init()`.
* Manually running code that is instrumented with the ClearML SDK and invokes [`Task.init`](../references/sdk/task.md#taskinit).
* Cloning an existing task.
* Creating a task via CLI using [clearml-task](../apps/clearml_task.md).
@@ -104,11 +140,11 @@ The above diagram demonstrates how a previously run task can be used as a baseli
## Task States
The state of a Task represents its stage in the Task lifecycle. It indicates whether the Task is read-write (editable) or
The state of a task represents its stage in the task lifecycle. It indicates whether the task is read-write (editable) or
read-only. For each state, a state transition indicates which actions can be performed on an experiment, and the new state
after performing an action.
The following table describes the Task states and state transitions.
The following table describes the task states and state transitions.
| State | Description / Usage | State Transition |
|---|---|---|
@@ -120,249 +156,8 @@ The following table describes the Task states and state transitions.
| *Aborted* | The experiment ran, and was manually or programmatically terminated. | The same as *Completed*. |
| *Published* | The experiment is read-only. Publish an experiment to prevent changes to its inputs and outputs. | A *Published* experiment cannot be reset. If it is cloned, the state of the newly cloned experiment becomes *Draft*. |
## SDK Interface
## Usage
### Task Creation
`Task.init()` is the main method used to create Tasks in ClearML. It will create a Task, and populate it with:
* A link to the running git repository (including commit ID and local uncommitted changes)
* Python packages used (i.e. directly imported Python packages, and the versions available on the machine)
* Argparse arguments (default and specific to the current execution)
* Reports to Tensorboard & Matplotlib and model checkpoints.
:::note
ClearML object (e.g. task, project) names are required to be at least 3 characters long
:::
```python
from clearml import Task
task = Task.init(
project_name='example', # project name of at least 3 characters
task_name='task template', # task name of at least 3 characters
task_type=None,
tags=None,
reuse_last_task_id=True,
continue_last_task=False,
output_uri=None,
auto_connect_arg_parser=True,
auto_connect_frameworks=True,
auto_resource_monitoring=True,
auto_connect_streams=True,
)
```
When a Task is initialized, it automatically captures parameters and outputs from supported frameworks. To control what ClearML
automatically logs, see this [FAQ](../faq.md#controlling_logging).
Once a Task is created, the Task object can be accessed from anywhere in the code by calling [`Task.current_task`](../references/sdk/task.md#taskcurrent_task).
If multiple Tasks need to be created in the same process (for example, for logging multiple manual runs),
make sure to close a Task, before initializing a new one. To close a task simply call `task.close`
(see example [here](../guides/advanced/multiple_tasks_single_process.md)).
When initializing a Task, its project needs to be specified. If the project entered does not exist, it will be created.
Projects can be divided into subprojects, just like folders are broken into sub-folders.
For example:
```python
Task.init(project_name='main_project/sub_project', task_name='test')
```
Nesting projects works on multiple levels. For example: `project_name=main_project/sub_project/sub_sub_project`
#### Task Reuse
Every `Task.init` call will create a new Task for the current execution.
In order to mitigate the clutter that a multitude of debugging Tasks might create, a Task will be reused if:
* The last time it was executed (on this machine) was under 72 hours ago (configurable, see
`sdk.development.task_reuse_time_window_in_hours` in the [`sdk.development` section](../configs/clearml_conf.md#sdkdevelopment) of
the ClearML configuration reference)
* The previous Task execution did not have any artifacts / models
It's possible to always create a new Task by passing `reuse_last_task_id=False`.
See full `Task.init` documentation [here](../references/sdk/task.md#taskinit).
### Empty Task Creation
A Task can also be created without the need to execute the code itself.
Unlike the runtime detections, all the environment and configuration details needs to be provided explicitly.
For example:
```python
task = Task.create(
project_name='example',
task_name='task template',
repo='https://github.com/allegroai/clearml.git',
branch='master',
script='examples/reporting/html_reporting.py',
working_directory='.',
docker=None,
)
```
See [`Task.create`](../references/sdk/task.md#taskcreate) in the Python SDK reference.
### Tracking Task Progress
Track a task’s progress by setting the task progress property using the [`Task.set_progress`](../references/sdk/task.md#set_progress) method.
Set a task’s progress to a numeric value between 0 - 100. Access the task’s current progress, using the
[`Task.get_progress`](../references/sdk/task.md#get_progress) method.
```python
task = Task.init(project_name="examples", task_name="Track experiment progress")
task.set_progress(0)
# task doing stuff
task.set_progress(50)
print(task.get_progress())
# task doing more stuff
task.set_progress(100)
```
While the task is running, the WebApp will show the task’s progress indication in the experiment table, next to the
task’s status. If a task failed or was aborted, you can view how much progress it had made.
<div class="max-w-50">
![Experiment table progress indication](../img/fundamentals_task_progress.png)
</div>
Additionally, you can view a task’s progress in its [INFO](../webapp/webapp_exp_track_visual.md#general-information) tab
in the WebApp.
### Accessing Tasks
A Task can be identified by its project and name, and by a unique identifier (UUID string). The name and project of
a Task can be changed after an experiment has been executed, but its ID can't be changed.
:::tip Locating Task IDs
To locate a task ID, go to the task's info panel in the [WebApp](../webapp/webapp_overview.md). In the top of the panel,
to the right of the task name, click `ID` and the task ID appears
:::
Programmatically, Task objects can be retrieved by querying the system based on either the Task ID or a project and name
combination. If a project / name combination is used, and multiple Tasks have the exact same name, the function will return
the *last modified Task*.
For example:
* Accessing a Task object with a Task ID:
```python
a_task = Task.get_task(task_id='123456deadbeef')
```
* Accessing a Task with a project / name:
```python
a_task = Task.get_task(project_name='examples', task_name='artifacts')
```
Once a Task object is obtained, it's possible to query the state of the Task, reported scalars, etc.
The Task's outputs, such as artifacts and models, can also be retrieved.
### Querying / Searching Tasks
Searching and filtering Tasks can be done via the [web UI](../webapp/webapp_overview.md), but also programmatically.
Input search parameters into the `Task.get_tasks` method, which returns a list of Task objects that match the search.
For example:
```python
task_list = Task.get_tasks(
task_ids=None, # type Optional[Sequence[str]]
project_name=None, # Optional[str]
task_name=None, # Optional[str]
task_filter=None # Optional[Dict]
)
```
It's possible to also filter Tasks by passing filtering rules to `task_filter`.
For example:
```python
task_filter={
# only Tasks with tag `included_tag` and without tag `excluded_tag`
'tags': ['included_tag', '-excluded_tag'],
# filter out archived Tasks
'system_tags': ['-archived'],
# only completed & published Tasks
'status': ['completed', 'published'],
# only training type Tasks
'type': ['training'],
# match text in Task comment or task name
'search_text': 'reg_exp_text'
}
```
### Cloning & Executing Tasks
Once a Task object is created, it can be a copied (cloned). `Task.clone` returns a copy of the original Task (`source_task`).
By default, the cloned Task is added to the same project as the original, and it's called "Clone Of ORIGINAL_NAME", but
the name / project / comment of the cloned Task can be directly overridden.
```python
cloned = Task.clone(
source_task=task, # type: Optional[Union[Task, str]]
# override default name
name='newly created task', # type: Optional[str]
comment=None, # type: Optional[str]
# insert cloned Task into a different project
project=None, # type: Optional[str]
)
```
A cloned Task starts in [draft](#task-states) mode, so its Task configurations can be edited (see
[Task.set_parameters](../references/sdk/task.md#set_parameters)).
Once a Task is modified, launch it by pushing it into an execution queue, then a [ClearML Agent](../clearml_agent.md) will pull
it from the queue and execute the Task.
```python
Task.enqueue(
task=task, # type: Union[Task, str]
queue_name='default', # type: Optional[str]
queue_id=None # type: Optional[str]
)
```
See enqueue [example](https://github.com/allegroai/clearml/blob/master/examples/automation/programmatic_orchestration.py).
### Advanced Remote Execution
A compelling workflow is:
1. Running code on the development machine for a few iterations, or just setting up the environment.
1. Moving the execution to a beefier remote machine for the actual training.
For example, to stop the current manual execution, and then re-run it on a remote machine, simply add the following
function call to the code:
```python
task.execute_remotely(
queue_name='default', # type: Optional[str]
clone=False, # type: bool
exit_process=True # type: bool
)
```
Once the function is called on the machine, it will stop the local process and enqueue the current Task into the *default*
queue. From there, an agent will be able to pick it up and launch it.
See the [Remote Execution](https://github.com/allegroai/clearml/blob/master/examples/advanced/execute_remotely_example.py) example.
#### Remote Function Execution
A specific function can also be launched on a remote machine with `create_function_task`.
For example:
```python
def run_me_remotely(some_argument):
print(some_argument)
a_func_task = task.create_function_task(
func=run_me_remotely, # type: Callable
func_name='func_id_run_me_remotely', # type:Optional[str]
task_name='a func task', # type:Optional[str]
# everything below will be passed directly to our function as arguments
some_argument=123
)
```
Arguments passed to the function will be automatically logged under the `Function` section in the Hyperparameters tab.
Like any other arguments, they can be changed from the UI or programmatically.
:::note
Function Tasks must be created from within a regular Task, created by calling `Task.init()`
:::
See [the task SDK interface](../clearml_sdk/task_sdk.md) for an overview for using the most basic Pythonic methods of the `Task`class.
See the [Task reference page](../references/sdk/task.md) for a complete list of available list.