Small edits (#420)

This commit is contained in:
pollfly 2022-12-27 16:01:47 +02:00 committed by GitHub
parent 0addbc3549
commit 439d86a46b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
34 changed files with 81 additions and 57 deletions

View File

@ -126,7 +126,7 @@ auto_connect_frameworks={'tensorboard': {'report_hparams': False}}
Every `Task.init` call will create a new task for the current execution. Every `Task.init` call will create a new task for the current execution.
In order to mitigate the clutter that a multitude of debugging tasks might create, a task will be reused if: In order to mitigate the clutter that a multitude of debugging tasks might create, a task will be reused if:
* The last time it was executed (on this machine) was under 72 hours ago (configurable, see * The last time it was executed (on this machine) was under 72 hours ago (configurable, see
[`sdk.development.task_reuse_time_window_in_hours`](../configs/clearml_conf.md#task_reuse) of [`sdk.development.task_reuse_time_window_in_hours`](../configs/clearml_conf.md#task_reuse) in
the ClearML configuration reference) the ClearML configuration reference)
* The previous task execution did not have any artifacts / models * The previous task execution did not have any artifacts / models

View File

@ -46,7 +46,7 @@ solution.
* **Serving Service Task** - Control plane object storing configuration on all the endpoints. Support multiple separated * **Serving Service Task** - Control plane object storing configuration on all the endpoints. Support multiple separated
instance, deployed on multiple clusters. instance, deployed on multiple clusters.
* **Inference Services** - Inference containers, performing model serving pre/post processing. Also supports CPU model * **Inference Services** - Inference containers, performing model serving pre/post-processing. Also supports CPU model
inferencing. inferencing.
* **Serving Engine Services** - Inference engine containers (e.g. Nvidia Triton, TorchServe etc.) used by the Inference * **Serving Engine Services** - Inference engine containers (e.g. Nvidia Triton, TorchServe etc.) used by the Inference

View File

@ -72,7 +72,7 @@ The following page goes over how to set up and upgrade `clearml-serving`.
``` ```
:::note :::note
Any model that registers with Triton engine will run the pre/post processing code on the Inference service container, Any model that registers with Triton engine will run the pre/post-processing code on the Inference service container,
and the model inference itself will be executed on the Triton Engine container. and the model inference itself will be executed on the Triton Engine container.
::: :::

View File

@ -414,7 +414,7 @@ match_rules: [
**`agent.package_manager`** (*dict*) **`agent.package_manager`** (*dict*)
* Dictionary containing the options for the Python package manager. The currently supported package managers are pip, conda, * Dictionary containing the options for the Python package manager. The currently supported package managers are pip, conda,
and, if the repository contains a poetry.lock file, poetry. and, if the repository contains a `poetry.lock` file, poetry.
--- ---

View File

@ -90,7 +90,7 @@ optimization.
optimizer = HyperParameterOptimizer( optimizer = HyperParameterOptimizer(
# specifying the task to be optimized, task must be in system already so it can be cloned # specifying the task to be optimized, task must be in system already so it can be cloned
base_task_id=TEMPLATE_TASK_ID, base_task_id=TEMPLATE_TASK_ID,
# setting the hyper-parameters to optimize # setting the hyperparameters to optimize
hyper_parameters=[ hyper_parameters=[
UniformIntegerParameterRange('number_of_epochs', min_value=2, max_value=12, step_size=2), UniformIntegerParameterRange('number_of_epochs', min_value=2, max_value=12, step_size=2),
UniformIntegerParameterRange('batch_size', min_value=2, max_value=16, step_size=2), UniformIntegerParameterRange('batch_size', min_value=2, max_value=16, step_size=2),

View File

@ -7,11 +7,11 @@ title: Tasks
A Task is a single code execution session, which can represent an experiment, a step in a workflow, a workflow controller, A Task is a single code execution session, which can represent an experiment, a step in a workflow, a workflow controller,
or any custom implementation you choose. or any custom implementation you choose.
To transform an existing script into a **ClearML Task**, one must call the [Task.init()](../references/sdk/task.md#taskinit) method To transform an existing script into a **ClearML Task**, one must call the [`Task.init()`](../references/sdk/task.md#taskinit) method
and specify a task name and its project. This creates a Task object that automatically captures code execution and specify a task name and its project. This creates a Task object that automatically captures code execution
information as well as execution outputs. information as well as execution outputs.
All the information captured by a task is by default uploaded to the [ClearML Server](../deploying_clearml/clearml_server.md) All the information captured by a task is by default uploaded to the [ClearML Server](../deploying_clearml/clearml_server.md),
and it can be visualized in the [ClearML WebApp](../webapp/webapp_overview.md) (UI). ClearML can also be configured to upload and it can be visualized in the [ClearML WebApp](../webapp/webapp_overview.md) (UI). ClearML can also be configured to upload
model checkpoints, artifacts, and charts to cloud storage (see [Storage](../integrations/storage.md)). Additionally, model checkpoints, artifacts, and charts to cloud storage (see [Storage](../integrations/storage.md)). Additionally,
you can work with tasks in Offline Mode, in which all information is saved in a local folder (see you can work with tasks in Offline Mode, in which all information is saved in a local folder (see
@ -110,7 +110,7 @@ Available task types are:
* *controller* - A task that lays out the logic for other tasks interactions, manual or automatic (e.g. a pipeline * *controller* - A task that lays out the logic for other tasks interactions, manual or automatic (e.g. a pipeline
controller) controller)
* *optimizer* - A specific type of controller for optimization tasks (e.g. [hyperparameter optimization](hpo.md)) * *optimizer* - A specific type of controller for optimization tasks (e.g. [hyperparameter optimization](hpo.md))
* *service* - Long lasting or recurring service (e.g. server cleanup, auto ingress, sync services etc) * *service* - Long lasting or recurring service (e.g. server cleanup, auto ingress, sync services etc.)
* *monitor* - A specific type of service for monitoring * *monitor* - A specific type of service for monitoring
* *application* - A task implementing custom applicative logic, like [auto-scaler](../guides/services/aws_autoscaler.md) * *application* - A task implementing custom applicative logic, like [auto-scaler](../guides/services/aws_autoscaler.md)
or [clearml-session](../apps/clearml_session.md) or [clearml-session](../apps/clearml_session.md)

View File

@ -132,8 +132,8 @@ Now, [command-line arguments](../../fundamentals/hyperparameters.md#tracking-hyp
Sit back, relax, and watch your models converge :) or continue to see what else can be done with ClearML [here](ds_second_steps.md). Sit back, relax, and watch your models converge :) or continue to see what else can be done with ClearML [here](ds_second_steps.md).
## Youtube Playlist ## YouTube Playlist
Or watch the Youtube Getting Started Playlist on our Youtube Channel! Or watch the YouTube Getting Started Playlist on our YouTube Channel!
[![Watch the video](https://img.youtube.com/vi/bjWwZAzDxTY/hqdefault.jpg)](https://www.youtube.com/watch?v=bjWwZAzDxTY&list=PLMdIlCuMqSTnoC45ME5_JnsJX0zWqDdlO&index=2) [![Watch the video](https://img.youtube.com/vi/bjWwZAzDxTY/hqdefault.jpg)](https://www.youtube.com/watch?v=bjWwZAzDxTY&list=PLMdIlCuMqSTnoC45ME5_JnsJX0zWqDdlO&index=2)

View File

@ -181,8 +181,8 @@ or check these pages out:
- Improve your experiments with [HyperParameter Optimization](../../fundamentals/hpo.md) - Improve your experiments with [HyperParameter Optimization](../../fundamentals/hpo.md)
- Check out ClearML's integrations to [external libraries](../../integrations/libraries.md). - Check out ClearML's integrations to [external libraries](../../integrations/libraries.md).
## Youtube Playlist ## YouTube Playlist
All these tips and tricks are also covered by our Youtube Getting Started series, go check it out :) All these tips and tricks are also covered by our YouTube Getting Started series, go check it out :)
[![Watch the video](https://img.youtube.com/vi/kyOfwVg05EM/hqdefault.jpg)](https://www.youtube.com/watch?v=kyOfwVg05EM&list=PLMdIlCuMqSTnoC45ME5_JnsJX0zWqDdlO&index=3) [![Watch the video](https://img.youtube.com/vi/kyOfwVg05EM/hqdefault.jpg)](https://www.youtube.com/watch?v=kyOfwVg05EM&list=PLMdIlCuMqSTnoC45ME5_JnsJX0zWqDdlO&index=3)

View File

@ -11,7 +11,7 @@ If you are afraid of clutter, use the archive option, and set up your own [clean
- Track the code base. There is no reason not to add metrics to any process in your workflow, even if it is not directly ML. Visibility is key to iterative improvement of your code / workflow. - Track the code base. There is no reason not to add metrics to any process in your workflow, even if it is not directly ML. Visibility is key to iterative improvement of your code / workflow.
- Create per-project [leaderboards](../../guides/ui/building_leader_board.md) based on custom columns - Create per-project [leaderboards](../../guides/ui/building_leader_board.md) based on custom columns
(hyper parameters and performance accuracy), and bookmark them (full URL will always reproduce the same view & table). (hyperparameters and performance accuracy), and bookmark them (full URL will always reproduce the same view & table).
- Share experiments with your colleagues and team-leaders. - Share experiments with your colleagues and team-leaders.
Invite more people to see how your project is progressing, and suggest they add metric reporting for their own. Invite more people to see how your project is progressing, and suggest they add metric reporting for their own.
These metrics can later be part of your own in-house monitoring solution, don't let good data go to waste :) These metrics can later be part of your own in-house monitoring solution, don't let good data go to waste :)

View File

@ -64,7 +64,7 @@ Cloning a task duplicates the tasks configuration, but not its outputs.
**To clone an experiment in the ClearML WebApp:** **To clone an experiment in the ClearML WebApp:**
1. Click on any project card to open its [experiments table](../../webapp/webapp_exp_table.md) 1. Click on any project card to open its [experiments table](../../webapp/webapp_exp_table.md)
1. Right click one of the experiments on the table 1. Right-click one of the experiments on the table
1. Click **Clone** in the context menu, which will open a **CLONE EXPERIMENT** window. 1. Click **Clone** in the context menu, which will open a **CLONE EXPERIMENT** window.
1. Click **CLONE** in the window. 1. Click **CLONE** in the window.
@ -76,7 +76,7 @@ Docker container image to be used, or change the hyperparameters and configurati
Once you have set up an experiment, it is now time to execute it. Once you have set up an experiment, it is now time to execute it.
**To execute an experiment through the ClearML WebApp:** **To execute an experiment through the ClearML WebApp:**
1. Right click your draft experiment (the context menu is also available through the <img src="/docs/latest/icons/ico-bars-menu.svg" alt="Menu" className="icon size-md space-sm" /> 1. Right-click your draft experiment (the context menu is also available through the <img src="/docs/latest/icons/ico-bars-menu.svg" alt="Menu" className="icon size-md space-sm" />
button on the top right of the experiments info panel) button on the top right of the experiments info panel)
1. Click **ENQUEUE,** which will open the **ENQUEUE EXPERIMENT** window 1. Click **ENQUEUE,** which will open the **ENQUEUE EXPERIMENT** window
1. In the window, select `default` in the queue menu 1. In the window, select `default` in the queue menu

View File

@ -27,7 +27,7 @@ clearml-data sync --folder ./from_production
We could also add a Tag `latest` to the Dataset, marking it as the latest version. We could also add a Tag `latest` to the Dataset, marking it as the latest version.
### Preprocessing Data ### Preprocessing Data
The second step is to preprocess the date. First we need to access it, then we want to modify it The second step is to preprocess the date. First we need to access it, then we want to modify it,
and lastly we want to create a new version of the data. and lastly we want to create a new version of the data.
```python ```python

View File

@ -15,7 +15,7 @@ which always returns the main Task.
## Hyperparameters ## Hyperparameters
ClearML automatically logs the command line options defined with `argparse`. A parameter dictionary is logged by ClearML automatically logs the command line options defined with `argparse`. A parameter dictionary is logged by
connecting it to the Task using a call to the [Task.connect](../../references/sdk/task.md#connect) method. connecting it to the Task using a call to the [`Task.connect`](../../references/sdk/task.md#connect) method.
```python ```python
additional_parameters = { additional_parameters = {

View File

@ -38,7 +38,7 @@ The example calls Matplotlib methods to log debug sample images. They appear in
## Hyperparameters ## Hyperparameters
ClearML automatically logs TensorFlow Definitions. A parameter dictionary is logged by connecting it to the Task, by ClearML automatically logs TensorFlow Definitions. A parameter dictionary is logged by connecting it to the Task, by
calling the [Task.connect](../../../references/sdk/task.md#connect) method. calling the [`Task.connect`](../../../references/sdk/task.md#connect) method.
```python ```python
task_params = {'num_scatter_samples': 60, 'sin_max_value': 20, 'sin_steps': 30} task_params = {'num_scatter_samples': 60, 'sin_max_value': 20, 'sin_steps': 30}

View File

@ -53,7 +53,7 @@ Text printed to the console for training progress, as well as all other console
## Configuration Objects ## Configuration Objects
In the experiment code, a configuration dictionary is connected to the Task by calling the [Task.connect](../../../references/sdk/task.md#connect) In the experiment code, a configuration dictionary is connected to the Task by calling the [`Task.connect`](../../../references/sdk/task.md#connect)
method. method.
```python ```python

View File

@ -33,9 +33,15 @@ By doubling clicking a thumbnail, you can view a spectrogram plot in the image v
ClearML automatically logs TensorFlow Definitions. A parameter dictionary is logged by connecting it to the Task using ClearML automatically logs TensorFlow Definitions. A parameter dictionary is logged by connecting it to the Task using
a call to the [`Task.connect`](../../../../../references/sdk/task.md#connect) method. a call to the [`Task.connect`](../../../../../references/sdk/task.md#connect) method.
configuration_dict = {'number_of_epochs': 10, 'batch_size': 4, 'dropout': 0.25, 'base_lr': 0.001} ```python
configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml configuration_dict = {
'number_of_epochs': 10,
'batch_size': 4,
'dropout': 0.25,
'base_lr': 0.001
}
configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml
```
Parameter dictionaries appear in **CONFIGURATION** **>** **HYPER PARAMETERS** **>** **General**. Parameter dictionaries appear in **CONFIGURATION** **>** **HYPER PARAMETERS** **>** **General**.
![image](../../../../../img/examples_audio_classification_UrbanSound8K_01.png) ![image](../../../../../img/examples_audio_classification_UrbanSound8K_01.png)

View File

@ -27,7 +27,7 @@ optimizer task's **CONFIGURATION** **>** **HYPER PARAMETERS**.
```python ```python
optimizer = HyperParameterOptimizer( optimizer = HyperParameterOptimizer(
base_task_id=TEMPLATE_TASK_ID, # This is the experiment we want to optimize base_task_id=TEMPLATE_TASK_ID, # This is the experiment we want to optimize
# here we define the hyper-parameters to optimize # here we define the hyperparameters to optimize
hyper_parameters=[ hyper_parameters=[
UniformIntegerParameterRange('number_of_epochs', min_value=2, max_value=12, step_size=2), UniformIntegerParameterRange('number_of_epochs', min_value=2, max_value=12, step_size=2),
UniformIntegerParameterRange('batch_size', min_value=2, max_value=16, step_size=2), UniformIntegerParameterRange('batch_size', min_value=2, max_value=16, step_size=2),

View File

@ -26,22 +26,28 @@ method.
For example, the raw data is read into a Pandas DataFrame named `train_set`, and the `head` of the DataFrame is reported. For example, the raw data is read into a Pandas DataFrame named `train_set`, and the `head` of the DataFrame is reported.
train_set = pd.read_csv(Path(path_to_ShelterAnimal) / 'train.csv') ```python
Logger.current_logger().report_table(title='ClearMLet - raw',series='pandas DataFrame',iteration=0, table_plot=train_set.head()) train_set = pd.read_csv(Path(path_to_ShelterAnimal) / 'train.csv')
Logger.current_logger().report_table(
title='ClearMLet - raw',series='pandas DataFrame',iteration=0, table_plot=train_set.head()
)
```
The tables appear in **PLOTS**. The tables appear in **PLOTS**.
![image](../../../../../img/download_and_preprocessing_07.png) ![image](../../../../../img/download_and_preprocessing_07.png)
## Hyperparameters ## Hyperparameters
A parameter dictionary is logged by connecting it to the Task using a call to the [Task.connect](../../../../../references/sdk/task.md#connect) A parameter dictionary is logged by connecting it to the Task using a call to the [`Task.connect`](../../../../../references/sdk/task.md#connect)
method. method.
logger = task.get_logger() ```python
configuration_dict = {'test_size': 0.1, 'split_random_state': 0} logger = task.get_logger()
configuration_dict = task.connect(configuration_dict) configuration_dict = {'test_size': 0.1, 'split_random_state': 0}
configuration_dict = task.connect(configuration_dict)
```
Parameter dictionaries appear in the **General** subsection. Parameter dictionaries appear in the **General** subsection.
![image](../../../../../img/download_and_preprocessing_01.png) ![image](../../../../../img/download_and_preprocessing_01.png)

View File

@ -50,7 +50,7 @@ The single scalar plot for loss appears in **SCALARS**.
ClearML automatically logs the command line options defined using `argparse`. ClearML automatically logs the command line options defined using `argparse`.
A parameter dictionary is logged by connecting it to the Task using a call to the [Task.connect](../../../references/sdk/task.md#connect) A parameter dictionary is logged by connecting it to the Task using a call to the [`Task.connect`](../../../references/sdk/task.md#connect)
method. method.
```python ```python

View File

@ -8,6 +8,6 @@ slug: /guides
To help learn and use ClearML, we provide example scripts that demonstrate how to use ClearML's various features. To help learn and use ClearML, we provide example scripts that demonstrate how to use ClearML's various features.
Examples scripts are in the [examples](https://github.com/allegroai/clearml/tree/master/examples) folder of the GitHub `clearml` Examples scripts are in the [examples](https://github.com/allegroai/clearml/tree/master/examples) folder of the GitHub `clearml`
repository. They are also pre-loaded in the **ClearML Server**: repository. They are also preloaded in the **ClearML Server**:
Each examples folder in the GitHub ``clearml`` repository contains a ``requirements.txt`` file for example scripts in that folder. Each examples folder in the GitHub ``clearml`` repository contains a ``requirements.txt`` file for example scripts in that folder.

View File

@ -37,7 +37,7 @@ experiment runs. Some possible destinations include:
* Google Cloud Storage * Google Cloud Storage
* Azure Storage. * Azure Storage.
Specify the output location in the `output_uri` parameter of the [Task.init](../../references/sdk/task.md#taskinit) method. Specify the output location in the `output_uri` parameter of the [`Task.init`](../../references/sdk/task.md#taskinit) method.
In this tutorial, we specify a local folder destination. In this tutorial, we specify a local folder destination.
In `pytorch_mnist_tutorial.py`, change the code from: In `pytorch_mnist_tutorial.py`, change the code from:

View File

@ -40,7 +40,7 @@ ClearML automatically logs TensorFlow Definitions, whether they are defined befo
flags.DEFINE_string('echo', None, 'Text to echo.') flags.DEFINE_string('echo', None, 'Text to echo.')
flags.DEFINE_string('another_str', 'My string', 'A string', module_name='test') flags.DEFINE_string('another_str', 'My string', 'A string', module_name='test')
task = Task.init(project_name='examples', task_name='hyper-parameters example') task = Task.init(project_name='examples', task_name='hyperparameters example')
flags.DEFINE_integer('echo3', 3, 'Text to echo.') flags.DEFINE_integer('echo3', 3, 'Text to echo.')
@ -54,7 +54,7 @@ TensorFlow Definitions appear in **HYPER PARAMETERS** **>** **TF_DEFINE**.
## Parameter Dictionaries ## Parameter Dictionaries
Connect a parameter dictionary to a Task by calling the [Task.connect](../../references/sdk/task.md#connect) Connect a parameter dictionary to a Task by calling the [`Task.connect`](../../references/sdk/task.md#connect)
method, and ClearML logs the parameters. ClearML also tracks changes to the parameters. method, and ClearML logs the parameters. ClearML also tracks changes to the parameters.
```python ```python

View File

@ -53,6 +53,6 @@ ClearML reports these images as debug samples in the **ClearML Web UI**, under t
![image](../../img/examples_reporting_07.png) ![image](../../img/examples_reporting_07.png)
Double click a thumbnail, and the image viewer opens. Double-click a thumbnail, and the image viewer opens.
![image](../../img/examples_reporting_07a.png) ![image](../../img/examples_reporting_07a.png)

View File

@ -38,7 +38,7 @@ Logger.current_logger().report_media(
) )
``` ```
The reported audio can be viewed in the **DEBUG SAMPLES** tab. Double click a thumbnail, and the audio player opens. The reported audio can be viewed in the **DEBUG SAMPLES** tab. Double-click a thumbnail, and the audio player opens.
![image](../../img/examples_reporting_08.png) ![image](../../img/examples_reporting_08.png)
@ -55,6 +55,6 @@ Logger.current_logger().report_media(
) )
``` ```
The reported video can be viewed in the **DEBUG SAMPLES** tab. Double click a thumbnail, and the video player opens. The reported video can be viewed in the **DEBUG SAMPLES** tab. Double-click a thumbnail, and the video player opens.
![image](../../img/examples_reporting_09.png) ![image](../../img/examples_reporting_09.png)

View File

@ -75,7 +75,7 @@ The script supports the following additional command line options:
Mutually exclusive to `exclude_users`. Mutually exclusive to `exclude_users`.
* `exclude_users` - Only report tasks that were NOT initiated by these users (usernames and user IDs are accepted). * `exclude_users` - Only report tasks that were NOT initiated by these users (usernames and user IDs are accepted).
Mutually exclusive to `include_users`. Mutually exclusive to `include_users`.
* `verbose` - If `True`, will increase verbosity of messages (such as when when tasks are polled but filtered away). * `verbose` - If `True`, will increase verbosity of messages (such as when tasks are polled but filtered away).
## Configuration ## Configuration

View File

@ -21,10 +21,12 @@ class. The storage examples include:
To download a ZIP file from storage to the `global` cache context, call the [StorageManager.get_local_copy](../../references/sdk/storage.md#storagemanagerget_local_copy) To download a ZIP file from storage to the `global` cache context, call the [StorageManager.get_local_copy](../../references/sdk/storage.md#storagemanagerget_local_copy)
method, and specify the destination location as the `remote_url` argument: method, and specify the destination location as the `remote_url` argument:
# create a StorageManager instance ```python
manager = StorageManager() # create a StorageManager instance
manager = StorageManager()
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.zip") manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.zip")
```
:::note :::note
Zip and tar.gz files will be automatically extracted to cache. This can be controlled with the`extract_archive` flag. Zip and tar.gz files will be automatically extracted to cache. This can be controlled with the`extract_archive` flag.
@ -32,11 +34,15 @@ Zip and tar.gz files will be automatically extracted to cache. This can be contr
To download a file to a specific context in cache, specify the name of the context as the `cache_context` argument: To download a file to a specific context in cache, specify the name of the context as the `cache_context` argument:
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", cache_context="test") ```python
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", cache_context="test")
```
To download a non-compressed file, set the `extract_archive` argument to `False`. To download a non-compressed file, set the `extract_archive` argument to `False`.
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", extract_archive=False) ```python
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", extract_archive=False)
```
By default, the `StorageManager` reports its download progress to the console every 5MB. You can change this using the By default, the `StorageManager` reports its download progress to the console every 5MB. You can change this using the
[`StorageManager.set_report_download_chunk_size`](../../references/sdk/storage.md#storagemanagerset_report_download_chunk_size) [`StorageManager.set_report_download_chunk_size`](../../references/sdk/storage.md#storagemanagerset_report_download_chunk_size)
@ -48,7 +54,11 @@ To upload a file to storage, call the [StorageManager.upload_file](../../referen
method. Specify the full path of the local file as the `local_file` argument, and the remote URL as the `remote_url` method. Specify the full path of the local file as the `local_file` argument, and the remote URL as the `remote_url`
argument. argument.
manager.upload_file(local_file="/mnt/data/also_file.ext", remote_url="s3://MyBucket/MyFolder") ```python
manager.upload_file(
local_file="/mnt/data/also_file.ext", remote_url="s3://MyBucket/MyFolder"
)
```
Use the `retries parameter` to set the number of times file upload should be retried in case of failure. Use the `retries parameter` to set the number of times file upload should be retried in case of failure.
@ -63,4 +73,6 @@ To set a limit on the number of files cached, call the [StorageManager.set_cache
method and specify the `cache_file_limit` argument as the maximum number of files. This does not limit the cache size, method and specify the `cache_file_limit` argument as the maximum number of files. This does not limit the cache size,
only the number of files. only the number of files.
new_cache_limit = manager.set_cache_file_limit(cache_file_limit=100) ```python
new_cache_limit = manager.set_cache_file_limit(cache_file_limit=100)
```

View File

@ -495,7 +495,7 @@ myDataView.add_mapping_rule(
### Accessing Frames ### Accessing Frames
Dataview objects can be retrieved by the Dataview ID or name using the [DataView.get](../references/hyperdataset/dataview.md#dataviewget) Dataview objects can be retrieved by the Dataview ID or name using the [`DataView.get`](../references/hyperdataset/dataview.md#dataviewget)
class method. class method.
```python ```python

View File

@ -67,7 +67,7 @@ Access these actions with the context menu in any of the following ways:
| ClearML Action | Description | | ClearML Action | Description |
|---|---| |---|---|
| Details | View Dataview details, including input datasets, label mapping, augmentation operations, and iteration control. Can also be accessed by double clicking a Dataview in the Dataviews table. | | Details | View Dataview details, including input datasets, label mapping, augmentation operations, and iteration control. Can also be accessed by double-clicking a Dataview in the Dataviews table. |
| Archive | To more easily work with active Dataviews, move a Dataview to the archive, removing it from the active Dataview table. | | Archive | To more easily work with active Dataviews, move a Dataview to the archive, removing it from the active Dataview table. |
| Restore | Action available in the archive. Restore a Dataview to the active Dataviews table. | | Restore | Action available in the archive. Restore a Dataview to the active Dataviews table. |
| Clone | Make an exact copy of a Dataview that is editable. | | Clone | Make an exact copy of a Dataview that is editable. |

View File

@ -87,7 +87,7 @@ if there is a change in the pipeline code. If there is no change, the pipeline r
### Tracking Pipeline Progress ### Tracking Pipeline Progress
ClearML automatically tracks a pipelines progress percentage: the number of pipeline steps completed out of the total ClearML automatically tracks a pipelines progress percentage: the number of pipeline steps completed out of the total
number of steps. For example, if a pipeline consists of 4 steps, after the first step completes, ClearML automatically number of steps. For example, if a pipeline consists of 4 steps, after the first step completes, ClearML automatically
sets its progress value to 25. Once a pipeline has started to run but is yet to successfully finish, , the WebApp will sets its progress value to 25. Once a pipeline has started to run but is yet to successfully finish, the WebApp will
show the pipelines progress indication in the pipeline runs table, next to the runs status. show the pipelines progress indication in the pipeline runs table, next to the runs status.
## Examples ## Examples

View File

@ -157,8 +157,8 @@ arguments.
#### pre_execute_callback & post_execute_callback #### pre_execute_callback & post_execute_callback
Callbacks can be utilized to control pipeline execution flow. Callbacks can be utilized to control pipeline execution flow.
A `pre_execute_callback` function is called when the step is created and before it is sent for execution. This allows a A `pre_execute_callback` function is called when the step is created, and before it is sent for execution. This allows a
user to modify the task before launch. Use node.job to access the [ClearmlJob](../references/sdk/automation_job_clearmljob.md) user to modify the task before launch. Use `node.job` to access the [ClearmlJob](../references/sdk/automation_job_clearmljob.md)
object, or node.job.task to directly access the Task object. Parameters are the configuration arguments passed to the object, or node.job.task to directly access the Task object. Parameters are the configuration arguments passed to the
ClearmlJob. ClearmlJob.

View File

@ -100,7 +100,7 @@ Access these actions with the context menu in any of the following ways:
| Action | Description | States Valid for the Action | State Transition | | Action | Description | States Valid for the Action | State Transition |
|---|---|---|---| |---|---|---|---|
| Details | View pipeline details. Can also be accessed by double clicking a run in the pipeline runs table. | Any state | None | | Details | View pipeline details. Can also be accessed by double-clicking a run in the pipeline runs table. | Any state | None |
| Run | Create a new pipeline run. Configure and enqueue it for execution. See [Create Run](#create-run). | Any State | *Pending* | | Run | Create a new pipeline run. Configure and enqueue it for execution. See [Create Run](#create-run). | Any State | *Pending* |
| Abort | Manually stop / cancel a run. | *Running* / *Pending* | *Aborted* | | Abort | Manually stop / cancel a run. | *Running* / *Pending* | *Aborted* |
| Continue | Rerun with the same parameters. | *Aborted* | *Pending* | | Continue | Rerun with the same parameters. | *Aborted* | *Pending* |

View File

@ -33,7 +33,7 @@ When archiving an experiment:
* Restore an experiment or model from either the: * Restore an experiment or model from either the:
* Experiments or models table - Right click the experiment or model **>** **Restore**. * Experiments or models table - Right-click the experiment or model **>** **Restore**.
* Info panel or full screen details view - Click <img src="/docs/latest/icons/ico-bars-menu.svg" alt="Bars menu" className="icon size-sm space-sm" /> * Info panel or full screen details view - Click <img src="/docs/latest/icons/ico-bars-menu.svg" alt="Bars menu" className="icon size-sm space-sm" />
(menu) **>** **Restore from Archive**. (menu) **>** **Restore from Archive**.

View File

@ -33,7 +33,7 @@ Experiments can also be modified and then executed remotely, see [Tuning Experim
The experiment's status becomes *Draft*. The experiment's status becomes *Draft*.
1. Enqueue the experiment for execution. Right click the experiment **>** **Enqueue** **>** Select a queue **>** **ENQUEUE**. 1. Enqueue the experiment for execution. Right-click the experiment **>** **Enqueue** **>** Select a queue **>** **ENQUEUE**.
The experiment's status becomes *Pending*. When a worker fetches the Task (experiment), the status becomes *Running*. The experiment's status becomes *Pending*. When a worker fetches the Task (experiment), the status becomes *Running*.
The experiment can now be tracked and its results visualized. The experiment can now be tracked and its results visualized.

View File

@ -137,7 +137,7 @@ Access these actions with the context menu in any of the following ways:
| Action | Description | States Valid for the Action | State Transition | | Action | Description | States Valid for the Action | State Transition |
|---|---|---|---| |---|---|---|---|
| Details | Open the experiment's [info panel](webapp_exp_track_visual.md#info-panel) (keeps the experiments list in view). Can also be accessed by double clicking an experiment in the experiments table. | Any state | None | | Details | Open the experiment's [info panel](webapp_exp_track_visual.md#info-panel) (keeps the experiments list in view). Can also be accessed by double-clicking an experiment in the experiments table. | Any state | None |
| View Full Screen | View experiment details in [full screen](webapp_exp_track_visual.md#full-screen-details-view). | Any state | None | | View Full Screen | View experiment details in [full screen](webapp_exp_track_visual.md#full-screen-details-view). | Any state | None |
| Manage Queue | If an experiment is *Pending* in a queue, view the utilization of that queue, manage that queue (remove experiments and change the order of experiments), and view information about the worker(s) listening to the queue. See the [Workers and Queues](webapp_workers_queues.md) page. | *Enqueued* | None | | Manage Queue | If an experiment is *Pending* in a queue, view the utilization of that queue, manage that queue (remove experiments and change the order of experiments), and view information about the worker(s) listening to the queue. See the [Workers and Queues](webapp_workers_queues.md) page. | *Enqueued* | None |
| View Worker | If an experiment is *Running*, view resource utilization, worker details, and queues to which a worker is listening. | *Running* | None | | View Worker | If an experiment is *Running*, view resource utilization, worker details, and queues to which a worker is listening. | *Running* | None |

View File

@ -26,7 +26,7 @@ Tune experiments and edit an experiment's execution details, then execute the tu
1. Edit the experiment. See [modifying experiments](#modifying-experiments). 1. Edit the experiment. See [modifying experiments](#modifying-experiments).
1. Enqueue the experiment for execution. Right click the experiment **>** **Enqueue** **>** Select a queue **>** 1. Enqueue the experiment for execution. Right-click the experiment **>** **Enqueue** **>** Select a queue **>**
**ENQUEUE**. **ENQUEUE**.
The experiment's status becomes *Pending*. When the worker assigned to the queue fetches the Task (experiment), the The experiment's status becomes *Pending*. When the worker assigned to the queue fetches the Task (experiment), the