Merge branch 'main' of https://github.com/allegroai/clearml-docs into images_3

This commit is contained in:
revital 2025-03-17 13:45:23 +02:00
commit 611a5bef77
12 changed files with 185 additions and 93 deletions

View File

@ -27,6 +27,7 @@ but can be overridden by command-line arguments.
|**CLEARML_AGENT_DOCKER_ARGS_HIDE_ENV** | Hide Docker environment variables containing secrets when printing out the Docker command. When printed, the variable values will be replaced by `********`. See [`agent.hide_docker_command_env_vars`](../configs/clearml_conf.md#hide_docker) |
|**CLEARML_AGENT_DISABLE_SSH_MOUNT** | Disables the auto `.ssh` mount into the docker |
|**CLEARML_AGENT_FORCE_CODE_DIR**| Allows overriding the remote execution code directory to bypass repository cloning and use a repo already available where the remote agent is running. |
|**CLEARML_AGENT_FORCE_UV**| If set to `1`, force the agent to use UV as the package manager. Overrides the default manager set in the [clearml.conf](../configs/clearml_conf.md) under `agent.package_manager.type` |
|**CLEARML_AGENT_FORCE_EXEC_SCRIPT**| Allows overriding the remote execution script to bypass repository cloning and execute code already available where the remote agent is running. Use `module:file.py` format to specify a module and a script to execute (e.g. `.:main.py` to run `main.py` from the working dir)|
|**CLEARML_AGENT_FORCE_TASK_INIT**| If set to `1`, ClearML Agent adds `Task.init()` to scripts that do not have the call, creating a Task to capture code execution information and output, which is then sent to the ClearML Server. If set to `0` and the script does not include `Task.init()`, the agent will capture only the output streams and console output, without tracking code execution details, metrics, or models. |
|**CLEARML_AGENT_FORCE_SYSTEM_SITE_PACKAGES** | If set to `1`, overrides default [`agent.package_manager.system_site_packages: true`](../configs/clearml_conf.md#system_site_packages) behavior when running tasks in containers (docker mode and k8s-glue)|

View File

@ -13,6 +13,7 @@ multiple tasks (see [Virtual Environment Reuse](clearml_agent_env_caching.md#vir
ClearML Agent supports working with one of the following package managers:
* [`pip`](https://en.wikipedia.org/wiki/Pip_(package_manager)) (default)
* [`conda`](https://docs.conda.io/en/latest/)
* [`uv`](https://docs.astral.sh/uv/)
* [`poetry`](https://python-poetry.org/)
To change the package manager used by the agent, edit the [`package_manager.type`](../configs/clearml_conf.md#agentpackage_manager)

View File

@ -15,7 +15,7 @@ The following page goes over how to set up and upgrade `clearml-serving`.
[free hosted service](https://app.clear.ml)
1. Connect `clearml` SDK to the server, see instructions [here](../clearml_sdk/clearml_sdk_setup#install-clearml)
1. Install clearml-serving CLI:
1. Install the `clearml-serving` CLI:
```bash
pip3 install clearml-serving
@ -27,21 +27,22 @@ The following page goes over how to set up and upgrade `clearml-serving`.
clearml-serving create --name "serving example"
```
The new serving service UID should be printed
This command prints the Serving Service UID:
```console
New Serving Service created: id=aa11bb22aa11bb22
```
Write down the Serving Service UID
Copy the Serving Service UID (e.g., `aa11bb22aa11bb22`), as you will need it in the next steps.
1. Clone the `clearml-serving` repository:
```bash
git clone https://github.com/clearml/clearml-serving.git
```
1. Edit the environment variables file (docker/example.env) with your clearml-server credentials and Serving Service UID.
For example, you should have something like
1. Edit the environment variables file (`docker/example.env`) with your `clearml-server` API credentials and Serving Service UID.
For example:
```bash
cat docker/example.env
```
@ -55,31 +56,30 @@ The following page goes over how to set up and upgrade `clearml-serving`.
CLEARML_SERVING_TASK_ID="<serving_service_id_here>"
```
1. Spin up the `clearml-serving` containers with `docker-compose` (or if running on Kubernetes, use the helm chart)
1. Spin up the `clearml-serving` containers with `docker-compose` (or if running on Kubernetes, use the helm chart):
```bash
cd docker && docker-compose --env-file example.env -f docker-compose.yml up
```
If you need Triton support (keras/pytorch/onnx etc.), use the triton docker-compose file
If you need Triton support (Keras/PyTorch/ONNX etc.), use the triton `docker-compose` file:
```bash
cd docker && docker-compose --env-file example.env -f docker-compose-triton.yml up
```
If running on a GPU instance with Triton support (keras/pytorch/onnx etc.), use the triton gpu docker-compose file:
If running on a GPU instance with Triton support (Keras/PyTorch/ONNX etc.), use the triton gpu docker-compose file:
```bash
cd docker && docker-compose --env-file example.env -f docker-compose-triton-gpu.yml up
```
:::note
Any model that registers with Triton engine will run the pre/post-processing code on the Inference service container,
Any model that registers with Triton engine will run the pre/post-processing code in the Inference service container,
and the model inference itself will be executed on the Triton Engine container.
:::
## Advanced Setup - S3/GS/Azure Access (Optional)
To add access credentials and allow the inference containers to download models from your S3/GS/Azure object-storage,
add the respective environment variables to your env files (example.env). For further details, see
[Configuring Storage](../integrations/storage.md#configuring-storage).
To enable inference containers to download models from S3, Google Cloud Storage (GS), or Azure,
add access credentials in the respective environment variables to your env files (`example.env`):
```
AWS_ACCESS_KEY_ID
@ -92,14 +92,21 @@ AZURE_STORAGE_ACCOUNT
AZURE_STORAGE_KEY
```
For further details, see [Configuring Storage](../integrations/storage.md#configuring-storage).
## Upgrading ClearML Serving
**Upgrading to v1.1**
1. Take down the serving containers (`docker-compose` or k8s)
1. Update the `clearml-serving` CLI `pip3 install -U clearml-serving`
1. Shut down the serving containers (`docker-compose` or k8s)
1. Update the `clearml-serving` CLI:
```
pip3 install -U clearml-serving
```
1. Re-add a single existing endpoint with `clearml-serving model add ...` (press yes when asked). It will upgrade the
`clearml-serving` session definitions
`clearml-serving` session definitions.
1. Pull the latest serving containers (`docker-compose pull ...` or k8s)
1. Re-spin serving containers (`docker-compose` or k8s)

View File

@ -515,8 +515,12 @@ These settings define which Docker image and arguments should be used unless [ex
**`agent.package_manager`** (*dict*)
* Dictionary containing the options for the Python package manager. The currently supported package managers are pip, conda,
and, if the repository contains a `poetry.lock` file, poetry.
* Dictionary containing the options for the Python package manager.
* The currently supported package managers are
* pip
* conda
* uv, if the root repository contains a `uv.lock` or `pyproject.toml` file
* poetry, if the repository contains a `poetry.lock` or `pyproject.toml` file
---
@ -661,13 +665,38 @@ Torch Nightly builds are ephemeral and are deleted from time to time.
* `pip`
* `conda`
* `poetry`
* `uv`
* If `pip` or `conda` are used, the agent installs the required packages based on the "Python Packages" section of the
Task. If the "Python Packages" section is empty, it will revert to using `requirements.txt` from the repository's root
directory. If `poetry` is selected, and the root repository contains `poetry.lock` or `pyproject.toml`, the "Python
directory.
* If `poetry` is selected, and the root repository contains `poetry.lock` or `pyproject.toml`, the "Python
Packages" section is ignored, and `poetry` is used. If `poetry` is selected and no lock file is found, it reverts to
`pip` package manager behaviour.
* If `uv` is selected, and the root repository contains `uv.lock` or `pyproject.toml`, the "Python
Packages" section is ignored, and `uv` is used. If `uv` is selected and no lock file is found, it reverts to
`pip` package manager behaviour.
---
**`agent.package_manager.uv_files_from_repo_working_dir`** (*bool*)
* If set to `true`, the agent will look for the `uv.lock` or `pyproject.toml` file in the provided directory path instead of
the repository's root directory.
---
**`agent.package_manager.uv_sync_extra_args`** (*list*)
* List extra command-line arguments to pass when using `uv`.
---
**`agent.package_manager.uv_version`** (*string*)
* The `uv` version requirements. For example, `">0.4"`, `"==0.4"`, `""` (empty string will install the latest version).
<br/>
#### agent.pip_download_cache

View File

@ -6,9 +6,22 @@ ClearML provides a comprehensive set of monitoring tools to help effectively tra
These tools offer both high-level overviews and detailed insights into task execution, resource
utilization, and project performance.
## Offerings
### Project Dashboard
## Project Overview
A project's **OVERVIEW** tab in the UI presents a general picture of a project:
* Metric Snapshot A graphical representation of selected metric values across project tasks, offering a quick assessment of progress.
* Task Status Tracking When a single metric variant is selected for the snapshot, task status is color-coded (e.g.,
Completed, Aborted, Published, Failed) for better visibility.
Use the Metric Snapshot to track project progress and identify trends in task performance.
For more information, see [Project Overview](../webapp/webapp_project_overview.md).
![Project Overview](../img/webapp_project_overview.png#light-mode-only)
![Project Overview](../img/webapp_project_overview_dark.png#dark-mode-only)
## Project Dashboard
:::info Pro Plan Offering
The Project Dashboard app is available under the ClearML Pro plan.
@ -28,16 +41,22 @@ For more information, see [Project Dashboard](../webapp/applications/apps_dashbo
![Project Dashboard](../img/apps_dashboard.png#light-mode-only)
![Project Dashboard](../img/apps_dashboard_dark.png#dark-mode-only)
### Project Overview
## Task Monitoring
A project's **OVERVIEW** tab in the UI presents a general picture of a project:
* Metric Snapshot A graphical representation of selected metric values across project tasks, offering a quick assessment of progress.
* Task Status Tracking When a single metric variant is selected for the snapshot, task status is color-coded (e.g.,
Completed, Aborted, Published, Failed) for better visibility.
ClearML provides task monitoring capabilities through the [`clearml.automation.Monitor`](https://github.com/clearml/clearml/blob/master/clearml/automation/monitor.py)
class. With this class you can implement monitoring workflows such as:
Use the Metric Snapshot to track project progress and identify trends in task performance.
* Send notifications via Slack or other channels
* Trigger automated responses based on specific task conditions
For more information, see [Project Overview](../webapp/webapp_project_overview.md).
For a practical example, see the [Slack Alerts Example](../guides/services/slack_alerts.md), which demonstrates how to:
* Track task status (completion, failure, etc.)
* Send notifications to a specified Slack channel
* Retrieve task details such as status, console logs, and links to the ClearML Web UI
You can also configure filters for task types and projects to reduce unnecessary notifications.
![Slack Alerts](../img/examples_slack_alerts.png#light-mode-only)
![Slack Alerts](../img/examples_slack_alerts_dark.png#dark-mode-only)
![Project Overview](../img/webapp_project_overview.png#light-mode-only)
![Project Overview](../img/webapp_project_overview_dark.png#dark-mode-only)

View File

@ -6,11 +6,12 @@ The [Slack alerts example](https://github.com/clearml/clearml/blob/master/exampl
demonstrates how to use the `clearml.automation.monitor` class to implement a service that monitors the completion and
failure of tasks, and posts alert messages on a Slack channel.
![Slack alert example](../../img/examples_slack_alerts.png)
![Slack alert example](../../img/examples_slack_alerts.png#light-mode-only)
![Slack alert example](../../img/examples_slack_alerts_dark.png#dark-mode-only)
## Creating a Slack Bot
## Creating a Slackbot
Before configuring and running the Slack alert service, create a Slack Bot (**ClearML Bot**).
Before configuring and running the Slack alert service, create a Slackbot (**ClearML Bot**).
:::important
The Slack API token and channel you create are required to configure the Slack alert service.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 231 KiB

After

Width:  |  Height:  |  Size: 459 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 458 KiB

View File

@ -77,12 +77,12 @@ cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents:
and shuts down instances as needed, according to a resource budget that you set.
### Reproducing Tasks
### Reproducing Task Runs
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif#light-mode-only)
![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5_dark.gif#dark-mode-only)
Use ClearML's web interface to reproduce tasks and edit their details, like hyperparameters or input models, then execute the tasks
Use ClearML's web interface to reproduce task runs and edit their details, like hyperparameters or input models, then execute the tasks
with the new configuration on a remote machine.
When ClearML is integrated into a script, it captures and stores configurations, such as hyperparameters

View File

@ -3,6 +3,33 @@ title: Version 3.24
---
### Enterprise Server 3.24.2
**New Features**
* Add support for additional billing event formats
* Improve login to multi-tenant service
**Bug Fixes**
* Security fixes when exporting data to CSV
* Fix access permissions to UI Applications
### Enterprise Server 3.24.1
**New Features**
* Add service for presigning AWS S3 URLs
* Add support for additional billing event formats
* Add support for user-customized UI themes
* Add configuration to disable adding users to the workspace via the UI
* Add on-demand refresh to UI applications
* Add grouped same-event view to UI "Latest Task Events"
**Bug Fixes**
* Fix downloaded CSV file of UI “Latest Task Events” missing some events
* Fix access permissions to UI Reports
* Fix configuration modal of UI application instance displays incorrect values
* Fix UI Hyper-Dataset frame viewer navigation controls not displaying
### Enterprise Server 3.24.0
**Default Behavior Change: Access Rules Enabled**

34
package-lock.json generated
View File

@ -737,11 +737,13 @@
}
},
"node_modules/@babel/helpers": {
"version": "7.26.7",
"version": "7.26.10",
"resolved": "https://registry.npmjs.org/@babel/helpers/-/helpers-7.26.10.tgz",
"integrity": "sha512-UPYc3SauzZ3JGgj87GgZ89JVdC5dj0AoetR5Bw6wj4niittNyFh6+eOGonYvJ1ao6B8lEa3Q3klS7ADZ53bc5g==",
"license": "MIT",
"dependencies": {
"@babel/template": "^7.25.9",
"@babel/types": "^7.26.7"
"@babel/template": "^7.26.9",
"@babel/types": "^7.26.10"
},
"engines": {
"node": ">=6.9.0"
@ -826,10 +828,12 @@
}
},
"node_modules/@babel/parser": {
"version": "7.26.8",
"version": "7.26.10",
"resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.26.10.tgz",
"integrity": "sha512-6aQR2zGE/QFi8JpDLjUZEPYOs7+mhKXm86VaKFiLP35JQwQb6bwUE+XbvkH0EptsYhbNBSUGaUBLKqxH1xSgsA==",
"license": "MIT",
"dependencies": {
"@babel/types": "^7.26.8"
"@babel/types": "^7.26.10"
},
"bin": {
"parser": "bin/babel-parser.js"
@ -1939,7 +1943,9 @@
}
},
"node_modules/@babel/runtime": {
"version": "7.26.7",
"version": "7.26.10",
"resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.26.10.tgz",
"integrity": "sha512-2WJMeRQPHKSPemqk/awGrAiuFfzBmOIPXKizAsVhWH9YJqLZ0H+HS4c8loHGgW6utJ3E/ejXQUsiGaQy2NZ9Fw==",
"license": "MIT",
"dependencies": {
"regenerator-runtime": "^0.14.0"
@ -1960,12 +1966,14 @@
}
},
"node_modules/@babel/template": {
"version": "7.26.8",
"version": "7.26.9",
"resolved": "https://registry.npmjs.org/@babel/template/-/template-7.26.9.tgz",
"integrity": "sha512-qyRplbeIpNZhmzOysF/wFMuP9sctmh2cFzRAZOn1YapxBsE1i9bJIY586R/WBLfLcmcBlM8ROBiQURnnNy+zfA==",
"license": "MIT",
"dependencies": {
"@babel/code-frame": "^7.26.2",
"@babel/parser": "^7.26.8",
"@babel/types": "^7.26.8"
"@babel/parser": "^7.26.9",
"@babel/types": "^7.26.9"
},
"engines": {
"node": ">=6.9.0"
@ -1988,7 +1996,9 @@
}
},
"node_modules/@babel/types": {
"version": "7.26.8",
"version": "7.26.10",
"resolved": "https://registry.npmjs.org/@babel/types/-/types-7.26.10.tgz",
"integrity": "sha512-emqcG3vHrpxUKTrxcblR36dcrcoRDvKmnL/dCL6ZsHaShW80qxCAcNhzQZrpeM765VzEos+xOi4s+r4IXzTwdQ==",
"license": "MIT",
"dependencies": {
"@babel/helper-string-parser": "^7.25.9",
@ -15567,7 +15577,9 @@
}
},
"node_modules/prismjs": {
"version": "1.29.0",
"version": "1.30.0",
"resolved": "https://registry.npmjs.org/prismjs/-/prismjs-1.30.0.tgz",
"integrity": "sha512-DEvV2ZF2r2/63V+tK8hQvrR2ZGn10srHbXviTlcv7Kpzw8jWiNTqbVgjO3IY8RxrrOUF8VPMQQFysYYYv0YZxw==",
"license": "MIT",
"engines": {
"node": ">=6"

View File

@ -634,60 +634,55 @@ module.exports = {
]},
]
},
/* {'Getting Started': [
'getting_started/architecture',
]},*/
{
'Enterprise Server': {
'Deployment Options': [
'deploying_clearml/enterprise_deploy/multi_tenant_k8s',
'deploying_clearml/enterprise_deploy/vpc_aws',
'deploying_clearml/enterprise_deploy/on_prem_ubuntu',
],
'Maintenance and Migration': [
'deploying_clearml/enterprise_deploy/import_projects',
'deploying_clearml/enterprise_deploy/change_artifact_links',
'deploying_clearml/enterprise_deploy/delete_tenant',
]
}
},
{
type: 'category',
collapsible: true,
collapsed: true,
label: 'ClearML Application Gateway',
label: 'Enterprise Server',
items: [
'deploying_clearml/enterprise_deploy/appgw_install_compose',
'deploying_clearml/enterprise_deploy/appgw_install_k8s',
]
},
'deploying_clearml/enterprise_deploy/custom_billing',
{
'UI Applications': [
'deploying_clearml/enterprise_deploy/app_install_ubuntu_on_prem',
'deploying_clearml/enterprise_deploy/app_install_ex_server',
'deploying_clearml/enterprise_deploy/app_custom',
]
},
{
'User Management': [
'user_management/user_groups',
'user_management/access_rules',
'user_management/admin_vaults',
{
type: 'category',
collapsible: true,
collapsed: true,
label: 'Identity Provider Integration',
link: {type: 'doc', id: 'user_management/identity_providers'},
items: [
'deploying_clearml/enterprise_deploy/sso_multi_tenant_login',
'deploying_clearml/enterprise_deploy/sso_saml_k8s',
'deploying_clearml/enterprise_deploy/sso_keycloak',
'deploying_clearml/enterprise_deploy/sso_active_directory'
{'Deployment Options': [
'deploying_clearml/enterprise_deploy/multi_tenant_k8s',
'deploying_clearml/enterprise_deploy/vpc_aws',
'deploying_clearml/enterprise_deploy/on_prem_ubuntu',
]
},
{'Maintenance and Migration': [
'deploying_clearml/enterprise_deploy/import_projects',
'deploying_clearml/enterprise_deploy/change_artifact_links',
'deploying_clearml/enterprise_deploy/delete_tenant',
]
},
{'ClearML Application Gateway': [
'deploying_clearml/enterprise_deploy/appgw_install_compose',
'deploying_clearml/enterprise_deploy/appgw_install_k8s',
]
},
'deploying_clearml/enterprise_deploy/custom_billing',
{'UI Applications': [
'deploying_clearml/enterprise_deploy/app_install_ubuntu_on_prem',
'deploying_clearml/enterprise_deploy/app_install_ex_server',
'deploying_clearml/enterprise_deploy/app_custom',
]
},
{'User Management': [
'user_management/user_groups',
'user_management/access_rules',
'user_management/admin_vaults',
{
type: 'category',
collapsible: true,
collapsed: true,
label: 'Identity Provider Integration',
link: {type: 'doc', id: 'user_management/identity_providers'},
items: [
'deploying_clearml/enterprise_deploy/sso_multi_tenant_login',
'deploying_clearml/enterprise_deploy/sso_saml_k8s',
'deploying_clearml/enterprise_deploy/sso_keycloak',
'deploying_clearml/enterprise_deploy/sso_active_directory'
]
},
]
},
]
},
],