ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-06-26 18:17:44 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge branch 'main' of https://github.com/allegroai/clearml-docs into importpro

Browse Source
This commit is contained in:
revital
2025-02-26 13:42:52 +02:00
parent eb512641b1 fd04894361
commit 8e4381a361
50 changed files with 211 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/main.yml vendored
View File

@@ -28,4 +28,4 @@ jobs:
# Runs a single command using the runners shell
- name: Run a one-line script
run: |
grep -r -Eoh "(https?://github.com/[a-zA-Z0-9./?=_%:-]*)" $GITHUB_WORKSPACE | sort -u | grep -v "://github.com/allegroai/clearml-docs" | xargs -n 1 sh -c 'curl --output /dev/null --silent --head --fail $0 || curl --output /dev/null --silent --head --fail --write-out "%{url_effective}: %{http_code}\n" $0'
grep -r -Eoh "(https?://github.com/[a-zA-Z0-9./?=_%:-]*)" $GITHUB_WORKSPACE | sort -u | grep -v "://github.com/clearml/clearml-docs" | xargs -n 1 sh -c 'curl --output /dev/null --silent --head --fail $0 || curl --output /dev/null --silent --head --fail --write-out "%{url_effective}: %{http_code}\n" $0'

2
.github/workflows/test.yml vendored
View File

@@ -25,7 +25,7 @@ jobs:
env:
INCOMING_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
with:
text: Link Checker failure in github.com/allegroai/clearl-docs
text: Link Checker failure in github.com/clearml/clearl-docs
blocks: |
[
{"type": "section", "text": {"type": "mrkdwn", "text": "Testing!"}}

2
docs/apps/clearml_param_search.md
View File

@@ -34,7 +34,7 @@ of the optimization results in table and graph forms.
|`--objective-metric-sign`| Optimization target, whether to maximize or minimize the value of the objective metric specified. Possible values: "min", "max", "min_global", "max_global". For more information, see [Optimization Objective](#optimization-objective). |<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--objective-metric-title`| Objective metric title to maximize/minimize (e.g. 'validation').|<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--optimization-time-limit`|The maximum time (minutes) for the optimization to run. The default is `None`, indicating no time limit.|<img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" />|
|`--optimizer-class`|The optimizer to use. Possible values are: OptimizerOptuna (default), OptimizerBOHB, GridSearch, RandomSearch. For more information, see [Supported Optimizers](../hpo.md#supported-optimizers). |<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--optimizer-class`|The optimizer to use. Possible values are: OptimizerOptuna (default), OptimizerBOHB, GridSearch, RandomSearch. For more information, see [Supported Optimizers](../clearml_sdk/hpo_sdk.md#supported-optimizers). |<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--params-search`|Parameters space for optimization. See more information in [Specifying the Parameter Space](#specifying-the-parameter-space). |<img src="/docs/latest/icons/ico-optional-yes.svg" alt="Yes" className="icon size-md center-md" />|
|`--params-override`|Additional parameters of the base task to override for this parameter search. Use the following JSON format for each parameter: `{"name": "param_name", "value": <new_value>}`. Windows users, see [JSON format note](#json_note).|<img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" />|
|`--pool-period-min`|The time between two consecutive polls (minutes).|<img src="/docs/latest/icons/ico-optional-no.svg" alt="No" className="icon size-md center-md" />|

2
docs/apps/clearml_task.md
View File

@@ -11,7 +11,7 @@ line arguments, Python module dependencies, and a requirements.txt file!
## What Is ClearML Task For?
* Launching off-the-shelf code on a remote machine with dedicated resources (e.g. GPU)
* Running [hyperparameter optimization](../hpo.md) on a codebase that is still not in ClearML
* Running [hyperparameter optimization](../getting_started/hpo.md) on a codebase that is still not in ClearML
* Creating a pipeline from an assortment of scripts, that you need to turn into ClearML tasks
* Running some code on a remote machine, either using an on-prem cluster or on the cloud

7
docs/clearml_data/best_practices.md → docs/best_practices/data_best_practices.md
View File

@@ -9,7 +9,8 @@ See [Hyper-Datasets](../hyperdatasets/overview.md) for ClearML's advanced querya
The following are some recommendations for using ClearML Data.
![Dataset UI gif](../img/gif/dataset.gif)
![Dataset UI gif](../img/gif/dataset.gif#light-mode-only)
![Dataset UI gif](../img/gif/dataset_dark.gif#dark-mode-only)
## Versioning Datasets
@@ -25,7 +26,7 @@ version contents ready to be updated.
Organize the datasets according to use-cases and use tags. This makes managing multiple datasets and
accessing the most updated datasets for different use-cases easier.
Like any ClearML tasks, datasets can be organized into [projects (and subprojects)](../fundamentals/projects.md#creating-subprojects).
Like any ClearML tasks, datasets can be organized into [projects (and subprojects)](../fundamentals/projects.md#creating-projects-and-subprojects).
Additionally, when creating a dataset, tags can be applied to the dataset, which will make searching for the dataset easier.
Organizing your datasets into projects by use-case makes it easier to access the most recent dataset version for that use-case.
@@ -55,5 +56,5 @@ serves as a dataset's single point of truth, you can schedule a script which use
will update the dataset based on the modifications made to the folder. This way, there is no need to manually modify a dataset.
This functionality will also track the modifications made to a folder.
See the sync function with the [CLI](clearml_data_cli.md#sync) or [SDK](clearml_data_sdk.md#syncing-local-storage)
See the sync function with the [CLI](../clearml_data/clearml_data_cli.md#sync) or [SDK](../clearml_data/clearml_data_sdk.md#syncing-local-storage)
interface.

14
docs/getting_started/ds/best_practices.md → docs/best_practices/data_scientist_best_practices.md
View File

@@ -24,12 +24,12 @@ During early stages of model development, while code is still being modified hea
These setups can be folded into each other and that's great! If you have a GPU machine for each researcher, that's awesome!
The goal of this phase is to get a code, dataset, and environment set up, so you can start digging to find the best model!
- [ClearML SDK](../../clearml_sdk/clearml_sdk.md) should be integrated into your code (check out [ClearML Setup](../../clearml_sdk/clearml_sdk_setup.md)).
- [ClearML SDK](../clearml_sdk/clearml_sdk.md) should be integrated into your code (check out [ClearML Setup](../clearml_sdk/clearml_sdk_setup.md)).
This helps visualizing the results and tracking progress.
- [ClearML Agent](../../clearml_agent.md) helps moving your work to other machines without the hassle of rebuilding the environment every time,
- [ClearML Agent](../clearml_agent.md) helps moving your work to other machines without the hassle of rebuilding the environment every time,
while also creating an easy queue interface that easily lets you drop your tasks to be executed one by one
(great for ensuring that the GPUs are churning during the weekend).
- [ClearML Session](../../apps/clearml_session.md) helps with developing on remote machines, in the same way that you'd develop on your local laptop!
- [ClearML Session](../apps/clearml_session.md) helps with developing on remote machines, in the same way that you'd develop on your local laptop!
## Train Remotely
@@ -43,12 +43,12 @@ yields the best performing model for your task!
Visualization and comparison dashboards keep your sanity at bay! At this stage you usually have a docker container with all the binaries
that you need.
- [ClearML SDK](../../clearml_sdk/clearml_sdk.md) ensures that all the metrics, parameters and Models are automatically logged and can later be
accessed, [compared](../../webapp/webapp_exp_comparing.md) and [tracked](../../webapp/webapp_exp_track_visual.md).
- [ClearML Agent](../../clearml_agent.md) does the heavy lifting. It reproduces the execution environment, clones your code,
- [ClearML SDK](../clearml_sdk/clearml_sdk.md) ensures that all the metrics, parameters and Models are automatically logged and can later be
accessed, [compared](../webapp/webapp_exp_comparing.md) and [tracked](../webapp/webapp_exp_track_visual.md).
- [ClearML Agent](../clearml_agent.md) does the heavy lifting. It reproduces the execution environment, clones your code,
applies code patches, manages parameters (including overriding them on the fly), executes the code, and queues multiple tasks.
It can even [build](../../clearml_agent/clearml_agent_docker_exec#exporting-a-task-into-a-standalone-docker-container) the docker container for you!
- [ClearML Pipelines](../../pipelines/pipelines.md) ensure that steps run in the same order,
- [ClearML Pipelines](../pipelines/pipelines.md) ensure that steps run in the same order,
programmatically chaining tasks together, while giving an overview of the execution pipeline's status.
**Your entire environment should magically be able to run on any machine, without you working hard.**

14
docs/getting_started/mlops/mlops_best_practices.md → docs/best_practices/mlops_best_practices.md
View File

@@ -7,10 +7,10 @@ From training models to data processing to deploying to production.
## Development - Preparing for Automation
Basically, track everything. There is nothing that is not worth having visibility to.
If you are afraid of clutter, use the archive option, and set up your own [cleanup service](../../guides/services/cleanup_service.md).
If you are afraid of clutter, use the archive option, and set up your own [cleanup service](../guides/services/cleanup_service.md).
- Track the code base. There is no reason not to add metrics to any process in your workflow, even if it is not directly ML. Visibility is key to iterative improvement of your code / workflow.
- Create per-project [leaderboards](../../guides/ui/building_leader_board.md) based on custom columns
- Create per-project [leaderboards](../guides/ui/building_leader_board.md) based on custom columns
(hyperparameters and performance accuracy), and bookmark them (full URL will always reproduce the same view and table).
- Share tasks with your colleagues and team-leaders.
Invite more people to see how your project is progressing, and suggest they add metric reporting for their own.
@@ -19,22 +19,22 @@ If you are afraid of clutter, use the archive option, and set up your own [clean
## Clone Tasks
Define a ClearML Task with one of the following options:
- Run the actual code with the `Task.init()` call. This will create and auto-populate the Task in CleaML (including Git Repo / Python Packages / Command line etc.).
- Register local / remote code repository with `clearml-task`. See [details](../../apps/clearml_task.md).
- Register local / remote code repository with `clearml-task`. See [details](../apps/clearml_task.md).
Once you have a Task in ClearML, you can clone and edit its definitions in the UI, then launch it on one of your nodes with [ClearML Agent](../../clearml_agent.md).
Once you have a Task in ClearML, you can clone and edit its definitions in the UI, then launch it on one of your nodes with [ClearML Agent](../clearml_agent.md).
## Advanced Automation
- Create daily / weekly cron jobs for retraining best performing models on.
- Create data monitoring & scheduling and launch inference jobs to test performance on any new coming dataset.
- Once there are two or more tasks that run after another, group them together into a [pipeline](../../pipelines/pipelines.md).
- Once there are two or more tasks that run after another, group them together into a [pipeline](../pipelines/pipelines.md).
## Manage Your Data
Use [ClearML Data](../../clearml_data/clearml_data.md) to version your data, then link it to running tasks for easy reproduction.
Use [ClearML Data](../clearml_data/clearml_data.md) to version your data, then link it to running tasks for easy reproduction.
Make datasets machine agnostic (i.e. store original dataset in a shared storage location, e.g. shared-folder / S3 / Gs / Azure).
ClearML Data supports efficient Dataset storage and caching, differentiable and compressed.
## Scale Your Work
Use [ClearML Agent](../../clearml_agent.md) to scale work. Install the agent machines (remote or local) and manage
Use [ClearML Agent](../clearml_agent.md) to scale work. Install the agent machines (remote or local) and manage
training workload with it.
Improve team collaboration by transparent resource monitoring, always know what is running where.

2
docs/clearml_agent/clearml_agent_setup.md
View File

@@ -27,7 +27,7 @@ it can't do that when running from a virtual environment.
clearml-agent init
```
The setup wizard prompts for ClearML credentials (see [here](../webapp/settings/webapp_settings_profile.md#clearml-credentials) about obtaining credentials).
The setup wizard prompts for ClearML credentials (see [here](../webapp/settings/webapp_settings_profile.md#clearml-api-credentials) about obtaining credentials).
```
Please create new clearml credentials through the settings page in your `clearml-server` web app,
or create a free account at https://app.clear.ml/settings/webapp-configuration

2
docs/clearml_data/clearml_data.md
View File

@@ -46,7 +46,7 @@ ClearML Data supports two interfaces:
- `clearml-data` - A CLI utility for creating, uploading, and managing datasets. See [CLI](clearml_data_cli.md) for a reference of `clearml-data` commands.
- `clearml.Dataset` - A Python interface for creating, retrieving, managing, and using datasets. See [SDK](clearml_data_sdk.md) for an overview of the basic methods of the `Dataset` module.
For an overview of recommendations for ClearML Data workflows and practices, see [Best Practices](best_practices.md).
For an overview of recommendations for ClearML Data workflows and practices, see [Best Practices](../best_practices/data_best_practices.md).
## Dataset Version States
The following table displays the possible states for a dataset version.

2
docs/clearml_sdk/clearml_sdk_setup.md
View File

@@ -85,7 +85,7 @@ pip install clearml
CLEARML setup completed successfully.
```
Now you can integrate ClearML into your code! Continue [here](../clearml_sdk/clearml_sdk_setup#auto-log-experiment).
Now you can integrate ClearML into your code! Continue [here](../getting_started/auto_log_exp.md).
### Jupyter Notebook
To use ClearML with Jupyter Notebook, you need to configure ClearML Server access credentials for your notebook.

2
docs/cloud_autoscaling/autoscaling_overview.md
View File

@@ -71,7 +71,7 @@ execute the tasks in the GPU queue.
#### Docker
Every task a cloud instance pulls will be run inside a docker container. When setting up an autoscaler app instance,
you can specify a default container to run the tasks inside. If the task has its own container configured, it will
override the autoscaler’s default docker image (see [Base Docker Image](../clearml_agent/clearml_agent_docker_exec#base-docker-container)).
override the autoscaler’s default docker image (see [Base Container](../getting_started/clearml_agent_base_docker.md#base-container)).
#### Git Configuration
If your code is saved in a private repository, you can add your Git credentials so the ClearML Agents running on your

98
docs/deploying_clearml/enterprise_deploy/sso_multi_tenant_login.md Normal file
View File

@@ -0,0 +1,98 @@
---
title: Multi-Tenant Login Mode
---
In a multi-tenant setup, each external tenant can be represented by an SSO client defined in the customer Identity provider
(Keycloak). Each ClearML tenant can be associated with a particular external tenant. Currently, only one
ClearML tenant can be associated with a particular external tenant
## Setup IdP/SSO Client in Identity Provider
1. Add the following URL to "Valid redirect URIs": `<clearml_webapp_address>/callback_<client_id>`
2. Add the following URLs to "Valid post logout redirect URIs":
```
<clearml_webapp_address>/login
<clearml_webapp_address>/login/<external tenant ID>
```
3. Make sure the external tenant ID and groups are returned as claims for a each user
## Configure ClearML to use Multi-Tenant Mode
Set the following environment variables in the ClearML enterprise helm chart under the `apiserver` section:
* To turn on the multi-tenant login mode:
```
- name: CLEARML__services__login__sso__tenant_login
value: "true"
```
* To hide any global IdP/SSO configuration that's not associated with a specific ClearML tenant:
```
- name: CLEARML__services__login__sso__allow_settings_providers
value: "false"
```
Enable `onlyPasswordLogin` by setting the following environment variable in the helm chart under the `webserver` section:
```
- name: WEBSERVER__onlyPasswordLogin`
value: “true”`
```
## Setup IdP for a ClearML Tenant
To set an IdP client for a ClearML tenant, you’ll need to set the ClearML tenant settings and define an identity provider:
1. Call the following API to set the ClearML tenant settings:
```
curl $APISERVER_URL/system.update_company_sso_config -H "Content-Type: application/json" -u $APISERVER_KEY:$APISERVER_SECRET -d'{
"company": "<company_id>",
"sso": {
"tenant": "<external tenant ID>",
"group_mapping": {
"IDP group name1": "Clearml group name1",
"IDP group name2": "Clearml group name2"
},
"admin_groups": ["IDP admin group name1", "IDP admin group name2"]
}}'
```
2. Call the following API to define the ClearML tenant identity provider:
```
curl $APISERVER_URL/sso.save_provider_configuration -H "Content-Type: application/json" -u $APISERVER_KEY:$APISERVER_SECRET -d'{
"provider": "keycloak",
"company": "<company_id>",
"configuration": {
"id": "<some unique id here, you can use company_id>",
"display_name": "<The text that you want to see on the login button>",
"client_id": "<client_id from IDP>",
"client_secret": "<client secret from IDP>",
"authorization_endpoint": "<authorization_endpoint from IDP OpenID configuration>",
"token_endpoint": "<token_endpoint from IDP OpenID configuration>",
"revocation_endpoint": "<revocation_endpoint from IDP OpenID configuration>",
"end_session_endpoint": "<end_session_endpoint from IDP OpenID configuration>",
"logout_from_provider": true,
"claim_tenant": "tenant_key",
"claim_name": "name",
"group_enabled": true,
"claim_groups": "ad_groups_trusted",
"group_prohibit_user_login_if_not_in_group": true
}}'
```
The above configuration assumes the following:
* On logout from ClearML, the user is also logged out from the Identity Provider
* External tenant ID for the user is returned under the `tenant_key` claim
* User display name is returned under the `name` claim
* User groups list is returned under the `ad_groups_trusted` claim
* Group integration is turned on and a user will be allowed to log in if any of the groups s/he belongs to in the
IdP exists under the corresponding ClearML tenant (this is after group name translation is done according to the ClearML tenant settings)
## Webapp Login
When running in multi-tenant login mode, a user belonging to some external tenant should use the following link to log in:
```
<clearml_webapp_address>/login/<external tenant ID>
```

2
docs/fundamentals/agents_and_queues.md
View File

@@ -17,7 +17,7 @@ from installing required packages to setting environment variables,
all leading to executing the code (supporting both virtual environment or flexible docker container configurations).
The agent also supports overriding parameter values on-the-fly without code modification, thus enabling no-code experimentation (this is also the foundation on which
ClearML [Hyperparameter Optimization](../hpo.md) is implemented).
ClearML [Hyperparameter Optimization](../getting_started/hpo.md) is implemented).
An agent can be associated with specific GPUs, enabling workload distribution. For example, on a machine with 8 GPUs you
can allocate several GPUs to an agent and use the rest for a different workload, even through another agent (see [Dynamic GPU Allocation](../clearml_agent/clearml_agent_dynamic_gpus.md)).

2
docs/fundamentals/hyperparameters.md
View File

@@ -6,7 +6,7 @@ Hyperparameters are a script's configuration options. Since hyperparameters can
model performance, it is crucial to efficiently track and manage them.
ClearML supports tracking and managing hyperparameters in each task and provides a dedicated [hyperparameter
optimization module](../hpo.md). With ClearML's logging and tracking capabilities, tasks can be reproduced, and their
optimization module](../getting_started/hpo.md). With ClearML's logging and tracking capabilities, tasks can be reproduced, and their
hyperparameters and results can be saved and compared, which is key to understanding model behavior.
ClearML lets you easily try out different hyperparameter values without changing your original code. ClearML's [execution

2
docs/fundamentals/task.md
View File

@@ -124,7 +124,7 @@ Available task types are:
* *inference* - Model inference job (e.g. offline / batch model execution)
* *controller* - A task that lays out the logic for other tasks' interactions, manual or automatic (e.g. a pipeline
controller)
* *optimizer* - A specific type of controller for optimization tasks (e.g. [hyperparameter optimization](../hpo.md))
* *optimizer* - A specific type of controller for optimization tasks (e.g. [hyperparameter optimization](../getting_started/hpo.md))
* *service* - Long lasting or recurring service (e.g. server cleanup, auto ingress, sync services etc.)
* *monitor* - A specific type of service for monitoring
* *application* - A task implementing custom applicative logic, like [autoscaler](../guides/services/aws_autoscaler.md)

2
docs/getting_started/architecture.md
View File

@@ -2,7 +2,7 @@
title: ClearML Modules
---
- [**ClearML Python Package**](auto_log_exp#install-clearml) (`clearml`) for integrating ClearML into your existing code-base.
- [**ClearML Python Package**](../clearml_sdk/clearml_sdk_setup.md) (`clearml`) for integrating ClearML into your existing code-base.
- [**ClearML Server**](../deploying_clearml/clearml_server.md) (`clearml-server`) for storing task, model, and workflow data, and supporting the Web UI experiment manager. It is also the control plane for the MLOps.
- [**ClearML Agent**](../clearml_agent.md) (`clearml-agent`), the MLOps orchestration agent. Enabling task and workflow reproducibility, and scalability.
- [**ClearML Data**](../clearml_data/clearml_data.md) (`clearml-data`) data management and versioning on top of file-systems/object-storage.

0
docs/clearml_agent/clearml_agent_base_docker.md → docs/getting_started/clearml_agent_base_docker.md
View File

0
docs/clearml_agent/clearml_agent_docker_exec.md → docs/getting_started/clearml_agent_docker_exec.md
View File

0
docs/clearml_agent/clearml_agent_scheduling.md → docs/getting_started/clearml_agent_scheduling.md
View File

8
docs/hpo.md → docs/getting_started/hpo.md
View File

@@ -13,7 +13,7 @@ solutions to boost hyperparameter optimization efficiency.
## Workflow
![Hyperparameter optimization diagram](img/hpo_diagram.png)
![Hyperparameter optimization diagram](../img/hpo_diagram.png)
The preceding diagram demonstrates the typical flow of hyperparameter optimization where the parameters of a base task are optimized:
@@ -27,8 +27,8 @@ The preceding diagram demonstrates the typical flow of hyperparameter optimizati
## ClearML Solutions
ClearML offers three solutions for hyperparameter optimization:
* [GUI application](webapp/applications/apps_hpo.md): The Hyperparameter Optimization app allows you to run and manage the optimization tasks
* [GUI application](../webapp/applications/apps_hpo.md): The Hyperparameter Optimization app allows you to run and manage the optimization tasks
directly from the web interface--no code necessary (available under the ClearML Pro plan).
* [Command-Line Interface (CLI)](apps/clearml_param_search.md): The `clearml-param-search` CLI tool enables you to configure and launch the optimization process from your terminal.
* [Python Interface](clearml_sdk/hpo_sdk.md): The `HyperParameterOptimizer` class within the ClearML SDK allows you to
* [Command-Line Interface (CLI)](../apps/clearml_param_search.md): The `clearml-param-search` CLI tool enables you to configure and launch the optimization process from your terminal.
* [Python Interface](../clearml_sdk/hpo_sdk.md): The `HyperParameterOptimizer` class within the ClearML SDK allows you to
configure and launch optimization tasks, and seamlessly integrate them in your existing model training tasks.

2
docs/getting_started/main.md
View File

@@ -112,7 +112,7 @@ alert you whenever your model improves in accuracy)
- Automatically scale cloud instances according to your resource needs with ClearML's
[AWS Autoscaler](../webapp/applications/apps_aws_autoscaler.md) and [GCP Autoscaler](../webapp/applications/apps_gcp_autoscaler.md)
GUI applications
- Run [hyperparameter optimization](../hpo.md)
- Run [hyperparameter optimization](hpo.md)
- Build [pipelines](../pipelines/pipelines.md) from code
- Much more!

BIN
docs/img/gif/dataset.gif
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 388 KiB

After

Width:  |  Height:  |  Size: 372 KiB

BIN
docs/img/gif/dataset_dark.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 MiB

BIN
docs/img/gif/infra_control_plane.gif
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 606 KiB

After

Width:  |  Height:  |  Size: 615 KiB

BIN
docs/img/gif/infra_control_plane_dark.gif
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 360 KiB

After

Width:  |  Height:  |  Size: 359 KiB

BIN
docs/img/gif/integrations_yolov5.gif
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 2.4 MiB

After

Width:  |  Height:  |  Size: 12 MiB

BIN
docs/img/gif/integrations_yolov5_dark.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 MiB

3
docs/integrations/autokeras.md
View File

@@ -95,7 +95,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:

5
docs/integrations/catboost.md
View File

@@ -93,7 +93,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:
@@ -117,5 +118,5 @@ task.execute_remotely(queue_name='default', exit_process=True)
## Hyperparameter Optimization
Use ClearML's [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../hpo.md)
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../getting_started/hpo.md)
for more information.

3
docs/integrations/fastai.md
View File

@@ -92,7 +92,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:

5
docs/integrations/keras.md
View File

@@ -105,7 +105,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:
@@ -129,5 +130,5 @@ task.execute_remotely(queue_name='default', exit_process=True)
## Hyperparameter Optimization
Use ClearML's [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../hpo.md)
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../getting_started/hpo.md)
for more information.

5
docs/integrations/lightgbm.md
View File

@@ -94,7 +94,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:
@@ -118,5 +119,5 @@ task.execute_remotely(queue_name='default', exit_process=True)
## Hyperparameter Optimization
Use ClearML's [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../hpo.md)
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../getting_started/hpo.md)
for more information.

5
docs/integrations/megengine.md
View File

@@ -90,7 +90,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:
@@ -114,5 +115,5 @@ task.execute_remotely(queue_name='default', exit_process=True)
## Hyperparameter Optimization
Use ClearML's [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../hpo.md)
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../getting_started/hpo.md)
for more information.

2
docs/integrations/optuna.md
View File

@@ -2,7 +2,7 @@
title: Optuna
---
[Optuna](https://optuna.readthedocs.io/en/latest) is a [hyperparameter optimization](../hpo.md) framework,
[Optuna](https://optuna.readthedocs.io/en/latest) is a [hyperparameter optimization](../getting_started/hpo.md) framework,
which makes use of different samplers such as grid search, random, bayesian, and evolutionary algorithms. You can integrate
Optuna into ClearML's automated hyperparameter optimization.

3
docs/integrations/pytorch.md
View File

@@ -114,7 +114,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:

5
docs/integrations/pytorch_lightning.md
View File

@@ -120,7 +120,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:
@@ -144,6 +145,6 @@ task.execute_remotely(queue_name='default', exit_process=True)
## Hyperparameter Optimization
Use ClearML's [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../hpo.md)
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../getting_started/hpo.md)
for more information.

3
docs/integrations/scikit_learn.md
View File

@@ -96,7 +96,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:

3
docs/integrations/tao.md
View File

@@ -113,7 +113,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:

5
docs/integrations/tensorflow.md
View File

@@ -107,7 +107,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:
@@ -131,5 +132,5 @@ task.execute_remotely(queue_name='default', exit_process=True)
## Hyperparameter Optimization
Use ClearML's [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../hpo.md)
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../getting_started/hpo.md)
for more information.

5
docs/integrations/transformers.md
View File

@@ -78,7 +78,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:
@@ -90,5 +91,5 @@ The ClearML Agent executing the task will use the new values to [override any ha
## Hyperparameter Optimization
Use ClearML's [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../hpo.md)
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../getting_started/hpo.md)
for more information.

5
docs/integrations/xgboost.md
View File

@@ -120,7 +120,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:
@@ -144,5 +145,5 @@ task.execute_remotely(queue_name='default', exit_process=True)
## Hyperparameter Optimization
Use ClearML's [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../hpo.md)
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../getting_started/hpo.md)
for more information.

5
docs/integrations/yolov5.md
View File

@@ -7,7 +7,7 @@ built in logger:
* Track every YOLOv5 training run in ClearML
* Version and easily access your custom training data with [ClearML Data](../clearml_data/clearml_data.md)
* Remotely train and monitor your YOLOv5 training runs using [ClearML Agent](../clearml_agent.md)
* Get the very best mAP using ClearML [Hyperparameter Optimization](../hpo.md)
* Get the very best mAP using ClearML [Hyperparameter Optimization](../getting_started/hpo.md)
* Turn your newly trained YOLOv5 model into an API with just a few commands using [ClearML Serving](../clearml_serving/clearml_serving.md)
## Setup
@@ -169,7 +169,8 @@ and shuts down instances as needed, according to a resource budget that you set.
### Cloning, Editing, and Enqueuing
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task
with the new configuration on a remote machine:

3
docs/integrations/yolov8.md
View File

@@ -166,4 +166,5 @@ with the new configuration on a remote machine:
The ClearML Agent executing the task will use the new values to [override any hard coded values](../clearml_agent.md).
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)

2
docs/webapp/webapp_exp_track_visual.md
View File

@@ -93,7 +93,7 @@ using to set up an environment (`pip` or `conda`) are available. Select which re
### Container
The Container section list the following information:
* Image - a pre-configured container that ClearML Agent will use to remotely execute this task (see [Building Docker containers](../clearml_agent/clearml_agent_docker_exec))
* Image - a pre-configured container that ClearML Agent will use to remotely execute this task (see [Building Docker containers](../getting_started/clearml_agent_docker_exec.md))
* Arguments - add container arguments
* Setup shell script - a bash script to be executed inside the container before setting up the task's environment

2
docs/webapp/webapp_exp_tuning.md
View File

@@ -72,7 +72,7 @@ and/or Reset functions.
#### Default Container
Select a pre-configured container that the [ClearML Agent](../clearml_agent.md) will use to remotely execute this task (see [Building Docker containers](../clearml_agent/clearml_agent_docker_exec)).
Select a pre-configured container that the [ClearML Agent](../clearml_agent.md) will use to remotely execute this task (see [Building Docker containers](../getting_started/clearml_agent_docker_exec.md)).
**To add, change, or delete a default container:**

3
docs/webapp/webapp_model_comparing.md
View File

@@ -46,8 +46,7 @@ models update. The Enterprise Plan and Hosted Service support embedding resource
The comparison tabs provides the following views:
* [Side-by-side textual comparison](#side-by-side-textual-comparison)
* [Tabular scalar comparison](#tabular-scalar-comparison)
* [Merged plot comparison](#plot-comparison)
* [Side-by-side graphic comparison](#graphic-comparison)
* [Plot comparison](#plot-comparison)
### Side-by-side Textual Comparison

37
docusaurus.config.js
View File

@@ -68,7 +68,7 @@ module.exports = {
},
announcementBar: {
id: 'supportus',
content: 'If you ❤️ ️<b>ClearML</b>, ⭐️ us on <a target="_blank" rel="noopener noreferrer" href="https://github.com/allegroai/clearml">GitHub</a>!',
content: 'If you ❤️ ️<b>ClearML</b>, ⭐️ us on <a target="_blank" rel="noopener noreferrer" href="https://github.com/clearml/clearml">GitHub</a>!',
isCloseable: true,
},
navbar: {
@@ -85,48 +85,61 @@ module.exports = {
to: '/docs/',
label: 'Overview',
position: 'left',
activeBaseRegex: '^/docs/latest/docs/(fundamentals/agents_and_queues|hyper_datasets|clearml_agent(/(clearml_agent_dynamic_gpus|clearml_agent_fractional_gpus)?|)?|cloud_autoscaling/autoscaling_overview|remote_session|model_registry|deploying_clearml/enterprise_deploy/appgw|build_interactive_models|deploying_models|custom_apps)?$',
},
{
to: '/docs/deploying_clearml/clearml_server',
to: '/docs/clearml_sdk/clearml_sdk_setup',
label: 'Setup',
position: 'left'
position: 'left',
activeBaseRegex: '^/docs/latest/docs/(deploying_clearml(?!/enterprise_deploy/appgw(/.*)?$)(/.*)?$|clearml_sdk/clearml_sdk_setup|user_management(/.*)?|clearml_agent/(clearml_agent_setup|clearml_agent_deployment_bare_metal|clearml_agent_deployment_k8s|clearml_agent_deployment_slurm|clearml_agent_execution_env|clearml_agent_env_caching|clearml_agent_services_mode)|integrations/storage)/?$',
},
{
to: '/docs/getting_started/auto_log_exp',
label: 'Using ClearML',
position: 'left'
position: 'left',
activeBaseRegex: '^/docs/latest/docs/(getting_started(?!/video_tutorials(/.*)?)|clearml_serving|apps/clearml_session)(/.*)?$',
},
{
label: 'Developer Center',
position: 'left', // or 'right'
to: '/docs/fundamentals/projects',
activeBaseRegex: '^/docs/latest/docs/(fundamentals(?!/agents_and_queues)(/.*)?|configs/configuring_clearml|getting_started/video_tutorials(/.*)?|clearml_sdk(?!/clearml_sdk_setup)(/.*)?|pipelines(/.*)?|hyperdatasets(/.*)?|clearml_data(/.*)?|hyperdatasets(/webapp)(/.*)?|references(/.*)?|webapp(/.*)?|clearml_agent/(clearml_agent_ref|clearml_agent_env_var)(/.*)?|configs/(clearml_conf|env_vars)(/.*)?|apps/(clearml_task|clearml_param_search)(/.*)?|best_practices(/.*)?|guides(/.*)?|integrations(/.*)?|faq|release_notes(/.*)?)$',
activeClassName: 'navbar__link--active',
items: [
{
label: 'ClearML Basics',
to: '/docs/fundamentals/projects',
activeBaseRegex: '^/docs/latest/docs/(fundamentals|getting_started/video_tutorials|clearml_sdk(/(?!clearml_sdk_setup).*|(?=/))?|pipelines|clearml_data|hyperdatasets/(?!webapp/).*)(/.*)?$',
},
{
label: 'References',
to: '/docs/references/sdk/task',
activeBaseRegex: '^/docs/latest/docs/(references/|webapp/.*|hyperdatasets/webapp/.*|clearml_agent/(clearml_agent_ref|clearml_agent_env_var)|configs/(clearml_conf|env_vars)|apps/(clearml_task|clearml_param_search))(/.*)?$',
},
{
label: 'Best Practices',
to: '/docs/getting_started/ds/best_practices'
to: 'docs/best_practices/data_scientist_best_practices',
activeBaseRegex: '^/docs/latest/docs/best_practices/'
},
{
label: 'Tutorials',
to: '/docs/guides',
activeBaseRegex: '^/docs/latest/docs/guides',
},
{
label: 'Integrations',
to: '/docs/integrations'
to: '/docs/integrations',
activeBaseRegex: '^/docs/latest/docs/integrations(?!/storage)',
},
{
label: 'FAQ',
to: '/docs/faq',
activeBaseRegex: '^/docs/latest/docs/faq$',
},
{
label: 'Release Notes',
to: '/docs/release_notes/clearml_server/open_source/ver_2_0',
activeBaseRegex: '^/docs/latest/docs/release_notes/',
},
],
@@ -134,7 +147,7 @@ module.exports = {
{
label: 'Community Resources',
position: 'left', // or 'right'
to: '/docs/community',
to: '/docs/latest/docs/community',
},
{
href: 'https://joinslack.clear.ml',
@@ -155,7 +168,7 @@ module.exports = {
'aria-label': 'Twitter',
},
{
href: 'https://github.com/allegroai/clearml',
href: 'https://github.com/clearml/clearml',
position: 'right',
className: 'header-ico header-ico--github',
'aria-label': 'GitHub repository',
@@ -202,7 +215,7 @@ module.exports = {
},
{
label: 'GitHub',
href: 'https://github.com/allegroai/clearml',
href: 'https://github.com/clearml/clearml',
},
],
},
@@ -220,13 +233,13 @@ module.exports = {
// Please change this to your repo.
breadcrumbs: false,
editUrl:
'https://github.com/allegroai/clearml-docs/edit/main/',
'https://github.com/clearml/clearml-docs/edit/main/',
},
// API: {
// sidebarPath: require.resolve('./sidebars.js'),
// // Please change this to your repo.
// editUrl:
// 'https://github.com/allegroai/clearml-docs/edit/main/',
// 'https://github.com/clearml/clearml-docs/edit/main/',
// },
blog: {
blogTitle: 'ClearML Tutorials',
@@ -236,7 +249,7 @@ module.exports = {
showReadingTime: true,
// Please change this to your repo.
editUrl:
'https://github.com/allegroai/clearml-docs/edit/main/tutorials/',
'https://github.com/clearml/clearml-docs/edit/main/tutorials/',
},
theme: {
customCss: require.resolve('./src/css/custom.css'),

2
package-lock.json generated
View File

@@ -15,7 +15,7 @@
"@docusaurus/plugin-google-analytics": "^3.6.1",
"@docusaurus/plugin-google-gtag": "^3.6.1",
"@docusaurus/preset-classic": "^3.6.1",
"@easyops-cn/docusaurus-search-local": "^0.48.0",
"@easyops-cn/docusaurus-search-local": "^0.48.5",
"@mdx-js/react": "^3.0.0",
"clsx": "^1.1.1",
"joi": "^17.4.0",

2
package.json
View File

@@ -23,7 +23,7 @@
"@docusaurus/plugin-google-analytics": "^3.6.1",
"@docusaurus/plugin-google-gtag": "^3.6.1",
"@docusaurus/preset-classic": "^3.6.1",
"@easyops-cn/docusaurus-search-local": "^0.48.0",
"@easyops-cn/docusaurus-search-local": "^0.48.5",
"@mdx-js/react": "^3.0.0",
"clsx": "^1.1.1",
"medium-zoom": "^1.0.6",

15
sidebars.js
View File

@@ -60,10 +60,10 @@ module.exports = {
'getting_started/data_management',
'getting_started/remote_execution',
'getting_started/building_pipelines',
'hpo',
'clearml_agent/clearml_agent_docker_exec',
'clearml_agent/clearml_agent_base_docker',
'clearml_agent/clearml_agent_scheduling',
'getting_started/hpo',
'getting_started/clearml_agent_docker_exec',
'getting_started/clearml_agent_base_docker',
'getting_started/clearml_agent_scheduling',
{"Deploying Model Endpoints": [
{
type: 'category',
@@ -673,6 +673,7 @@ module.exports = {
label: 'Identity Provider Integration',
link: {type: 'doc', id: 'user_management/identity_providers'},
items: [
'deploying_clearml/enterprise_deploy/sso_multi_tenant_login',
'deploying_clearml/enterprise_deploy/sso_saml_k8s',
'deploying_clearml/enterprise_deploy/sso_keycloak',
'deploying_clearml/enterprise_deploy/sso_active_directory'
@@ -690,17 +691,17 @@ module.exports = {
{
type: 'doc',
label: 'Data Scientists',
id: 'getting_started/ds/best_practices'
id: 'best_practices/data_scientist_best_practices'
},
{
type: 'doc',
label: 'MLOps and LLMOps',
id: 'getting_started/mlops/mlops_best_practices'
id: 'best_practices/mlops_best_practices'
},
{
type: 'doc',
label: 'Data Management',
id: 'clearml_data/best_practices'
id: 'best_practices/data_best_practices'
},
],
},