diff --git a/docs/hyperdatasets/webapp/webapp_datasets_frames.md b/docs/hyperdatasets/webapp/webapp_datasets_frames.md index 35f16346..a4eb01e3 100644 --- a/docs/hyperdatasets/webapp/webapp_datasets_frames.md +++ b/docs/hyperdatasets/webapp/webapp_datasets_frames.md @@ -181,7 +181,7 @@ or the arrow keys on the keyboard). Closing the frame editor will prompt you to |edit metadata|Edit metadata|Hover over an annotation in the list and click the icon to open the edit window. Input the metadata dictionary in JSON format. This metadata is specific to the selected annotation, not the entire frame.| |Lock annotation|Lock / Unlock annotation |Click the button on a specific annotation to make it uneditable. You can also click the button on top of the annotations list to lock all annotations in the frame.| |Trash|Delete annotation|Click the annotation or bounded area in the frame and then click the button to delete the annotation.| -|Eye Show All|Show/hide all annotations |Click the button to view the frame without annotations. When annotations are hidden, they can't be modified. | +|Eye Show All|Show/hide annotations |Click the button on a specific annotation to hide it. You can also click the button on top of the annotations list to hide all annotations. Hidden annotations can't be modified. | ||Delete label |In the relevant annotation, click **x** on the label you want to remove.| ### Frame Labels diff --git a/docs/hyperdatasets/webapp/webapp_datasets_versioning.md b/docs/hyperdatasets/webapp/webapp_datasets_versioning.md index d8096012..b099c443 100644 --- a/docs/hyperdatasets/webapp/webapp_datasets_versioning.md +++ b/docs/hyperdatasets/webapp/webapp_datasets_versioning.md @@ -248,9 +248,15 @@ according to the previously configured setting. #### Preview Source When using multi-source FrameGroups, users can choose which of the FrameGroups' sources will be displayed as the preview. -Select a source from the **PREVIEW SOURCE** menu. +Select a source from the **Current Source** menu. Choose the `Default preview source` option to present the first available source for each frame (sources are retrieved in ASCIIbetical order). +Choose the `All sources` option to present all the FrameGroup’s sources in a grid. In this view, the annotation panel +shows annotations grouped by their respective sources. Additionally, annotation tools (e.g. create/delete/modify +annotations) are not available in this view. + +![All sources preview](../../img/hyperdatasets/preview_all_sources.png) + :::note Unavailable Source If a FrameGroup doesn't have the selected preview source, the preview displays the "Source not available" message. ::: diff --git a/docs/img/examples_catboost_scalars.png b/docs/img/examples_catboost_scalars.png index 6a04370b..28a53003 100644 Binary files a/docs/img/examples_catboost_scalars.png and b/docs/img/examples_catboost_scalars.png differ diff --git a/docs/img/examples_cifar_scalars.png b/docs/img/examples_cifar_scalars.png index 34fb9e36..cfa3f951 100644 Binary files a/docs/img/examples_cifar_scalars.png and b/docs/img/examples_cifar_scalars.png differ diff --git a/docs/img/examples_lightgbm_scalars.png b/docs/img/examples_lightgbm_scalars.png index 004749ac..009e2163 100644 Binary files a/docs/img/examples_lightgbm_scalars.png and b/docs/img/examples_lightgbm_scalars.png differ diff --git a/docs/img/examples_megengine_mnist_scalars.png b/docs/img/examples_megengine_mnist_scalars.png index 359cbd92..91f8717d 100644 Binary files a/docs/img/examples_megengine_mnist_scalars.png and b/docs/img/examples_megengine_mnist_scalars.png differ diff --git a/docs/img/examples_pytorch_lightning_scalars.png b/docs/img/examples_pytorch_lightning_scalars.png index 48e1a989..0aba01ef 100644 Binary files a/docs/img/examples_pytorch_lightning_scalars.png and b/docs/img/examples_pytorch_lightning_scalars.png differ diff --git a/docs/img/examples_xgboost_metric_scalars.png b/docs/img/examples_xgboost_metric_scalars.png index fcb32145..678ff8d2 100644 Binary files a/docs/img/examples_xgboost_metric_scalars.png and b/docs/img/examples_xgboost_metric_scalars.png differ diff --git a/docs/img/fundamentals_hpo_scalars.png b/docs/img/fundamentals_hpo_scalars.png index a4a06fe7..5e515b22 100644 Binary files a/docs/img/fundamentals_hpo_scalars.png and b/docs/img/fundamentals_hpo_scalars.png differ diff --git a/docs/img/hyperdatasets/preview_all_sources.png b/docs/img/hyperdatasets/preview_all_sources.png new file mode 100644 index 00000000..c69fc400 Binary files /dev/null and b/docs/img/hyperdatasets/preview_all_sources.png differ diff --git a/docs/img/integration_openmmlab_scalars.png b/docs/img/integration_openmmlab_scalars.png index cf10c50a..707c0007 100644 Binary files a/docs/img/integration_openmmlab_scalars.png and b/docs/img/integration_openmmlab_scalars.png differ diff --git a/docs/img/integrations_transformers_scalars.png b/docs/img/integrations_transformers_scalars.png index 98e39f27..4ebc75a6 100644 Binary files a/docs/img/integrations_transformers_scalars.png and b/docs/img/integrations_transformers_scalars.png differ diff --git a/docs/img/monai_clearml_scalars.png b/docs/img/monai_clearml_scalars.png index 11f4cdd9..88a40248 100644 Binary files a/docs/img/monai_clearml_scalars.png and b/docs/img/monai_clearml_scalars.png differ diff --git a/docs/img/pipelines_comparison_plots.png b/docs/img/pipelines_comparison_plots.png new file mode 100644 index 00000000..e543ccdc Binary files /dev/null and b/docs/img/pipelines_comparison_plots.png differ diff --git a/docs/img/pipelines_comparison_plots_merged.png b/docs/img/pipelines_comparison_plots_merged.png new file mode 100644 index 00000000..38bc31c3 Binary files /dev/null and b/docs/img/pipelines_comparison_plots_merged.png differ diff --git a/docs/img/pipelines_comparison_scalars.png b/docs/img/pipelines_comparison_scalars.png new file mode 100644 index 00000000..63bc22c5 Binary files /dev/null and b/docs/img/pipelines_comparison_scalars.png differ diff --git a/docs/img/pipelines_comparison_single_scalar.png b/docs/img/pipelines_comparison_single_scalar.png new file mode 100644 index 00000000..10b13fc5 Binary files /dev/null and b/docs/img/pipelines_comparison_single_scalar.png differ diff --git a/docs/img/settings_admin_vaults.png b/docs/img/settings_admin_vaults.png index 10300de4..4449e255 100644 Binary files a/docs/img/settings_admin_vaults.png and b/docs/img/settings_admin_vaults.png differ diff --git a/docs/img/settings_service_account_config_vault.png b/docs/img/settings_service_account_config_vault.png new file mode 100644 index 00000000..59a19468 Binary files /dev/null and b/docs/img/settings_service_account_config_vault.png differ diff --git a/docs/img/webapp_info_audit_log.png b/docs/img/webapp_info_audit_log.png new file mode 100644 index 00000000..ab8df3a7 Binary files /dev/null and b/docs/img/webapp_info_audit_log.png differ diff --git a/docs/img/webapp_model_endpoints_active_table.png b/docs/img/webapp_model_endpoints_active_table.png new file mode 100644 index 00000000..7afd4104 Binary files /dev/null and b/docs/img/webapp_model_endpoints_active_table.png differ diff --git a/docs/img/webapp_model_endpoints_details.png b/docs/img/webapp_model_endpoints_details.png new file mode 100644 index 00000000..46cdf3f0 Binary files /dev/null and b/docs/img/webapp_model_endpoints_details.png differ diff --git a/docs/img/webapp_model_endpoints_loading_table.png b/docs/img/webapp_model_endpoints_loading_table.png new file mode 100644 index 00000000..cc933a97 Binary files /dev/null and b/docs/img/webapp_model_endpoints_loading_table.png differ diff --git a/docs/img/webapp_model_endpoints_monitor.png b/docs/img/webapp_model_endpoints_monitor.png new file mode 100644 index 00000000..6b00b713 Binary files /dev/null and b/docs/img/webapp_model_endpoints_monitor.png differ diff --git a/docs/img/webapp_model_scalars.png b/docs/img/webapp_model_scalars.png index 28fb8e93..911380d1 100644 Binary files a/docs/img/webapp_model_scalars.png and b/docs/img/webapp_model_scalars.png differ diff --git a/docs/img/webapp_project_card.png b/docs/img/webapp_project_card.png index eabf5cc3..97c69804 100644 Binary files a/docs/img/webapp_project_card.png and b/docs/img/webapp_project_card.png differ diff --git a/docs/img/webapp_projects_context_menu.png b/docs/img/webapp_projects_context_menu.png index e3d10a3f..8f24d301 100644 Binary files a/docs/img/webapp_projects_context_menu.png and b/docs/img/webapp_projects_context_menu.png differ diff --git a/docs/img/webapp_settings_storage_credentials.png b/docs/img/webapp_settings_storage_credentials.png new file mode 100644 index 00000000..cacc99cc Binary files /dev/null and b/docs/img/webapp_settings_storage_credentials.png differ diff --git a/docs/img/webapp_sub_project_card.png b/docs/img/webapp_sub_project_card.png index ac3284b1..7c348364 100644 Binary files a/docs/img/webapp_sub_project_card.png and b/docs/img/webapp_sub_project_card.png differ diff --git a/docs/img/webapp_tracking_33.png b/docs/img/webapp_tracking_33.png index cb20d623..fd5f74f7 100644 Binary files a/docs/img/webapp_tracking_33.png and b/docs/img/webapp_tracking_33.png differ diff --git a/docs/img/webapp_tracking_34.png b/docs/img/webapp_tracking_34.png index d8316509..84f5c437 100644 Binary files a/docs/img/webapp_tracking_34.png and b/docs/img/webapp_tracking_34.png differ diff --git a/docs/img/webapp_tracking_34a.png b/docs/img/webapp_tracking_34a.png new file mode 100644 index 00000000..87f033f2 Binary files /dev/null and b/docs/img/webapp_tracking_34a.png differ diff --git a/docs/release_notes/clearml_server/enterprise/ver_3_23.md b/docs/release_notes/clearml_server/enterprise/ver_3_23.md new file mode 100644 index 00000000..d902253b --- /dev/null +++ b/docs/release_notes/clearml_server/enterprise/ver_3_23.md @@ -0,0 +1,60 @@ +--- +title: Version 3.23 +--- + +### Enterprise Server 3.23.0 + +**Default Behavior Change for Object Access (Access Rules)** +* Users can only access objects as explicitly specified by the access rules applied to them. If users can no longer +access objects (Projects, Tasks, Datasets etc.) they had previously created, verify access rules exist that provide the +user access to these objects or their containing project. + +**New Features** +* New ClearML Model dashboard: View all live model endpoints in a single location, complete with real time metrics reporting. +* New UI pipeline run table comparative view: compare plots and scalars of selected pipeline runs +* New UI setting for configuring cloud storage credentials with which ClearML can clean up cloud storage artifacts on task deletion. +* Add UI re-enqueue of failed tasks +* Add configuration vaults for service accounts +* Add administrator vaults for UI storage credential to allow centralized configuration of storage credentials for UI access for designated user groups +* Add "All sources" view to UI Hyper-Dataset frame viewer +* Add task recent event log to UI task information +* Add UI experiment scalar results table view +* Add show/hide specific frame annotations in UI Hyper-Dataset frame viewer +* Add UI edit of default Project default output destination + +**Bug Fixes** +* Fix broken download links to artifacts stored in Azure ([ClearML Server GitHub issue #247](https://github.com/allegroai/clearml-server/issues/247)) +* Fix Cross-site Scripting (XSS) vulnerability (CWE-79) +* Fix UI experiment textual comparison diff showing contextual diffs as different contents ([ClearML GitHub issue #646](https://github.com/allegroai/clearml/issues/646)) +* Fix Model API calls fail when its creating task has been deleted ([ClearML GitHub issue #1299](https://github.com/allegroai/clearml/issues/1299)) +* Fix UI plots display "Iteration 0" when it is the only reported iteration ([ClearML GitHub issue #1267](https://github.com/allegroai/clearml/issues/1267)) +* Fix scalar series starting with angle bracket (`<`) causes UI scalar plot legend to display raw html ([ClearML GitHub issue #1292](https://github.com/allegroai/clearml/issues/1292)) +* Fix auto-refresh modifies UI experiment debug sample view ([ClearML GitHub issue #1529](https://github.com/allegroai/clearml/issues/1529)) +* Fix Hydra parameters not displaying correctly in UI experiment comparison parallel coordinates plot +* Fix UI pipeline DAG display to be "bottoms-up" so all final steps appear in the bottom of the diagram ([ClearML Web GitHub PR #86](https://github.com/allegroai/clearml-web/pull/86)) +* Fix UI experiment comparison does not overlay box plots and histograms ([ClearML GitHub issue #1298](https://github.com/allegroai/clearml/issues/1298)) +* Fix modifying UI experiment input models does not work +* Fix UI Reports image upload not working +* Fix active masks labels not displaying in UI annotation task frame viewer +* Fix UI Hyper-Dataset version status does not update when published +* Fix UI model hidden plot selection does not persist upon refresh +* Fix deleting large number of experiments via UI not working properly +* Fix experiment name legend not displayed in UI experiment plot comparison +* Fix archiving pipeline run does not abort pipeline step tasks +* Fix "Restore" and "Delete" buttons missing from UI experiment archive action bar +* Fix UI Resource Configuration allows same-named profile-pool links +* Fix UI experiment debug sample viewer sometimes displays incorrect sample +* Fix UI Resource Policy creation policy modal pops up unnecessarily +* Fix UI Resource Policy creation modal does not display newly created user groups as an assignment option +* Fix UI applications do not display indication that running instance limit has been reached +* Fix UI Settings' Configuration section does not display indication that the number of credentials limit has been reached +* Fix iteration number not displayed in titles of confusion matrix and table plots in UI experiment comparison +* Fix "Create new" option not displaying when inputting project name in UI Report creation modal +* Fix running experiment that is archived continues running +* Fix adding second resource policy to resource group does not assign the policy +* Fix UI experiment requires Git information when repository isn't set +* Fix pipeline run version not set when re-executed via the UI +* Fix UI DataView successfully created with identical IDs in label enumeration +* Fix UI DataView created successfully when "Limit Frames" repetition option is selected, but the maximum number of frames is not set +* Fix moving a enqueued experiment to a new UI queue results in error +* Fix metric and hyperparameter group string not searchable in UI table diff --git a/docs/webapp/applications/apps_overview.md b/docs/webapp/applications/apps_overview.md index 2fec9609..0bbba367 100644 --- a/docs/webapp/applications/apps_overview.md +++ b/docs/webapp/applications/apps_overview.md @@ -50,6 +50,8 @@ Each application's page is split into two sections: * App Instance List - Launch new app instances and view previously launched instances. Click on an instance to view its dashboard. Hover over it to access the [app instance actions](#app-instance-actions). * App Instance Dashboard - The main section of the app page: displays the selected app instance's status and results. + Use the search bar Magnifying glass + to quickly find an app instance by name. ![App format](../../img/apps_format_overview.png) diff --git a/docs/webapp/pipelines/webapp_pipeline_table.md b/docs/webapp/pipelines/webapp_pipeline_table.md index fbce73f6..0b788cc8 100644 --- a/docs/webapp/pipelines/webapp_pipeline_table.md +++ b/docs/webapp/pipelines/webapp_pipeline_table.md @@ -6,11 +6,14 @@ The pipeline runs table is a [customizable](#customizing-the-runs-table) list of view a run's details, and manage runs (create, continue, or abort). The runs table's auto-refresh allows users to continually monitor run progress. -View the runs table in table view Table view -or in details view Details view, -using the buttons on the top left of the page. Use the table view for a comparative view of your runs according to -columns of interest. Use the details view to access a selected run's details, while keeping the pipeline runs list in view. -Details view can also be accessed by double-clicking a specific pipeline run in the table view to open its details view. +View the runs table in table view Table view, +details view Details view, +or comparison view Comparison view +using the buttons on the top left of the page. Use the details view to access a selected run's details, while keeping +the run list in view. Details view can also be accessed by double-clicking a specific pipeline run in the table view to +open its details view. Use the [comparison view](#comparing-runs) to compare your pipeline run's scalar and plot results. +This view compares the scalars/plots of currently selected pipeline runs. If no runs are selected, The first 100 visible +runs in the table are compared. You can archive pipeline runs so the runs table doesn't get too cluttered. Click **OPEN ARCHIVE** on the top of the table to open the archive and view all archived runs. From the archive, you can restore @@ -146,3 +149,39 @@ selecting items beyond the items currently on-screen: * **None** - Clear selection * **Filtered** - Select all runs in the project that match the current active table filters +## Comparing Runs +The comparison view compares pipeline run scalar and plot results. When selected, the view presents a comparison of all +[selected runs](#selecting-multiple-runs). If no runs are selected, the first 100 visible runs in the table are compared. + +In the dropdown menu, select to view **Scalars** or **Plots**. + +**Scalars** shows pipeline run scalar results as time series line graphs. + +![Scalar line graphs](../../img/pipelines_comparison_scalars.png) + +All single value scalars are plotted into a single clustered bar chart under the "Summary" title, where each cluster +represents a reported metric, and each bar in the cluster represents an experiment. + +![Single scalar comparison](../../img/pipelines_comparison_single_scalar.png) + +Click Setting Gear to customize which +metrics to view. + +In the **Scalars** view, click Tuning to access [scalar plot tools](../webapp_exp_track_visual.md#scalar-plot-tools). + +**Plots** shows the last reported iteration sample of each metric/variant combination per compared run. + +Line, scatter, box, and bar graphs are compared by overlaying each metric/variant from all compared runs' into a single +comparative plot. + +![Merged plots](../../img/pipelines_comparison_plots_merged.png) + +Other plot types are displayed separately for each run. + +![Side-by-side plots](../../img/pipelines_comparison_plots.png) + + +### Run Details Comparison +For a more in depth comparison of pipeline runs, select the runs to compare and click **Compare** in the batch action +bar. In the run comparison pages, you can compare details, hyperparameters, scalars, plots, and debug samples. For more +information, see [Comparing Experiments](../webapp_exp_comparing.md). diff --git a/docs/webapp/settings/webapp_settings_admin_vaults.md b/docs/webapp/settings/webapp_settings_admin_vaults.md index 81b5d5cc..f7fb486e 100644 --- a/docs/webapp/settings/webapp_settings_admin_vaults.md +++ b/docs/webapp/settings/webapp_settings_admin_vaults.md @@ -7,10 +7,17 @@ This feature is available under the ClearML Enterprise plan. ::: Administrators can define multiple [configuration vaults](webapp_settings_profile.md#configuration-vault) which will each be applied to designated -[user groups](webapp_settings_users.md). Use configuration vaults to extend and/or override entries in the local ClearML [configuration file](../../configs/clearml_conf.md) -where a ClearML task is executed. Configuration vault values will be applied to tasks run by members of the designated user groups. +[user groups](webapp_settings_users.md). There are two types of vaults: +* [Client configuration (Agent/SDK/CLI)](#client-configuration-agentsdkcli) +* [UI storage credentials](#ui-storage-credentials) -To apply its contents, a vault should be enabled. New entries will extend the configuration in the local ClearML [configuration file](../../configs/clearml_conf.md). +To apply its contents, a vault should be enabled in the [Administrator Vault Table](#administrator-vault-table). + +## Client Configuration (Agent/SDK/CLI) +Client configuration vaults extend and/or override entries in the local ClearML [configuration file](../../configs/clearml_conf.md) +where a task is executed. Vault values will be applied to tasks run by members of the designated user groups. + +New entries will extend the configuration in the local ClearML [configuration file](../../configs/clearml_conf.md). Most existing configuration file entries will be overridden by the vault values. :::info @@ -23,24 +30,41 @@ The following configuration values are machine and/or agent specific, so they ca * `agent.debug` ::: -**To create a vault:** +**To create a Client configuration vault:** 1. Click **+ Add Vault** 1. Fill in vault details: 1. Vault name - Name that appears in the Administrator Vaults table 1. User Group - Specify the User Group that the vault affects - 1. Format - Specify the configuration format: HOCON / JSON / YAML. -1. Fill in the configuration values (click Info -to view configuration file reference). To import and existing configuration file, click Import. + 1. Target - Vault type. Select `Client (Agent/SDK/UI)` + 1. Format - Specify the configuration format: HOCON / JSON / YAML. + 1. Fill in the configuration values (click Info + to view configuration file reference). To import an existing configuration file, click Import. 1. Click **Save** +## UI Storage Credentials +UI storage credential vaults configure UI access to cloud storage credentials for a designated group of users. + +**To create a vault:** + +1. Click **+ Add Vault** +1. Fill in vault details: + 1. Vault name - Name that appears in the Administrator Vaults table + 1. User Group - Specify the User Group that the vault affects + 1. Target - Select `UI storage credentials` + 1. \+ Add access keys - Enter storage credentials (see [Browser Cloud Storage Access](webapp_settings_profile.md#browser-cloud-storage-access)) +1. Click **Save** + +## Administrator Vault Table + The **Administrator Vaults** table lists all currently defined vaults, and the following details: * Active - Toggle to enable / disable the vault * Name - Vault name +* Target - Type of vault: `Client (Agent/SDK/CLI)` or `UI storage credentials` * Group - User groups to apply this vault to * ID - Vault ID (click to copy) * Vault Content - Vault content summary * Update - Last update time -Hover over a vault in the table to Download, Edit, or Delete a vault. +Hover over a vault in the table to **Download**, **Edit**, or **Delete** a vault. ![Admin vaults](../../img/settings_admin_vaults.png) \ No newline at end of file diff --git a/docs/webapp/settings/webapp_settings_overview.md b/docs/webapp/settings/webapp_settings_overview.md index c77c5ae7..d44fe05f 100644 --- a/docs/webapp/settings/webapp_settings_overview.md +++ b/docs/webapp/settings/webapp_settings_overview.md @@ -24,4 +24,6 @@ The Settings page consists of the following sections: * [Identity Providers](webapp_settings_id_providers.md) (ClearML Enterprise Server) - Manage server identity providers * [Resource Configuration](webapp_settings_resource_configs.md) (ClearML Enterprise Server) - Define the available resources and the way in which they will be allocated to different workloads - * [Usage & Billing](webapp_settings_usage_billing.md) (ClearML Hosted Service) - View current usage information and billing details \ No newline at end of file + * [Usage & Billing](webapp_settings_usage_billing.md) (ClearML Hosted Service) - View current usage information and billing details + * [Storage Credentials](webapp_settings_storage_credentials.md) (ClearML Enterprise Server) - Configure storage provider access credentials to + enable ClearML to delete artifacts stored in cloud storage when tasks and models are deleted \ No newline at end of file diff --git a/docs/webapp/settings/webapp_settings_profile.md b/docs/webapp/settings/webapp_settings_profile.md index c5e7523d..c8b664d2 100644 --- a/docs/webapp/settings/webapp_settings_profile.md +++ b/docs/webapp/settings/webapp_settings_profile.md @@ -62,7 +62,7 @@ to switch to. ![Workspace configuration page](../../img/settings_workspace_configuration.png) -### ClearML Credentials +### ClearML App Credentials Generate ClearML credentials, made up of an access and secret key pair, and insert them into your [configuration file](../../configs/clearml_conf.md) or Jupyter Notebook to grant the ClearML SDK and the ClearML Agent API access to the server. @@ -91,6 +91,20 @@ these credentials cannot be recovered. **To revoke ClearML credentials:** hover over the desired credentials, and click Trash can +## AI Application Gateway Tokens + +:::important Enterprise Feature +This feature is available under the ClearML Enterprise plan. +::: + +The AI Application Gateway enables external access to ClearML tasks and applications. The gateway is configured with an +endpoint or external address (ingress), accessible from outside ClearML. + +Generate tokens providing API access to the AI Application Gateway endpoints: + +1. Click **Generate a Token** +1. Under `Expiration`, enter the number of days the token should remain valid +1. Click `Generate`, which creates a token and copies it to your clipboard ### Changing Your Workspace Name To change the name of your own workspace, click **Edit workspace name** Edit Pencil diff --git a/docs/webapp/settings/webapp_settings_storage_credentials.md b/docs/webapp/settings/webapp_settings_storage_credentials.md new file mode 100644 index 00000000..1435c5d5 --- /dev/null +++ b/docs/webapp/settings/webapp_settings_storage_credentials.md @@ -0,0 +1,55 @@ +--- +title: Storage Credentials +--- + +:::important Enterprise Feature +This feature is available under the ClearML Enterprise plan. +::: + +To enable ClearML to delete task artifacts stored in cloud storage when a task is deleted, configure access credentials for your storage provider: +* [Google Cloud Storage](#google-cloud-storage) +* [AWS S3 Storage](#aws-s3-storage) +* [Azure](#azure) + +![Storage Credentials page](../../img/webapp_settings_storage_credentials.png) + +## Google Cloud Storage +Set up credentials for Google Cloud buckets: +* Default credentials - These credentials apply to all GCS buckets unless bucket-specific credentials are set. + * Project - Default Google Cloud Storage project + * Credentials JSON +* Bucket specific credentials: + * Bucket + * Project + * Credentials JSON + +## AWS S3 Storage +Set up credentials for S3 protocol storage (i.e. AWS S3, MinIO, etc.): +* Default credentials - These credentials apply to all buckets unless bucket-specific credentials are set: + * Access Key - Default access key for the storage service. + * Secret - Default secret access key. + * Access token - Session key for temporary credentials (if applicable). + * Region - Default region for all unspecified buckets. + * Credentials Chain - If selected, use boto3 default [credentials search](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) + (i.e. look for credentials in environment variables, credential files, and instance metadata services). +* Bucket Specific Credentials: + * Bucket - Name of the specific bucket. + * Region - Region for the bucket. + * Host - For non-AWS endpoints, the host URL and port number of the specific bucket. Note that port specification + is *always* needed (e.g. `my-minio-host:9000`), even for standard ports like 433 for HTTPS (e.g. `my-minio-host:433`) + * Secure Host - Select in order to enable TLS. + * Verify SSL certificate - Select to enable SSL verification for secure hosts. + * Access key - Access key for the bucket. + * Secret - Secret key for the bucket. + * Access token - The session key for your bucket. This is only needed when you are using temporary credentials. + * Use Credentials chain - If selected, use boto3 default [credentials search](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) + (i.e. looks for credentials in environment variables, credential files, and instance metadata services). + +## Azure +Set up credentials for Azure storage containers: +* Account name - Azure storage account name. +* Account key - Azure storage account key. +* Container name - Name of the specific container. + + + diff --git a/docs/webapp/settings/webapp_settings_users.md b/docs/webapp/settings/webapp_settings_users.md index b1e94d9f..215d2eaa 100644 --- a/docs/webapp/settings/webapp_settings_users.md +++ b/docs/webapp/settings/webapp_settings_users.md @@ -109,6 +109,32 @@ To revoke a set of credentials: 1. In the editing panel, hover of the relevant credential's row 2. Click the Trash can button +### Service Account Configuration Vault +Use a service account’s configuration vault to store ClearML configuration entries that can extend the ClearML +[configuration file](../../configs/clearml_conf.md) of any ClearML Agents or ClearML SDK running with the service account's +credentials. + +Vault entries will extend the configuration in the ClearML [configuration file](../../configs/clearml_conf.md) if they +don't yet exist, and override values for those already present in the file. + +Fill in values using any of ClearML supported configuration formats: HOCON / JSON / YAML. + +To edit vault contents: +1. Click on the relevant service account to open its details panel +2. Click **EDIT** on the configuration vault +3. Insert / edit the configurations in the vault +4. Press **OK** + +To apply vault contents: +* Click the toggle atop the vault to enable / disable the configurations +* Once enabled, the configurations will be merged to the configuration file during ClearML and ClearML Agent usage + +In addition to the service account-specific configuration vault, [administrator vaults](#administrator-vaults) can also +be applied to service accounts. See all the vaults applied to the account in the **Applied administrator vaults** below +the configuration vault. + +![Service Account Config Vault](../../img/settings_service_account_config_vault.png) + ### Deleting Service Account Deleting a service account will revoke its credentials, causing agents using the account's credentials to fail. Tasks and associated artifacts logged to your workspace by a service account will remain in your workspace. diff --git a/docs/webapp/webapp_exp_comparing.md b/docs/webapp/webapp_exp_comparing.md index 1e86e3f5..f401c631 100644 --- a/docs/webapp/webapp_exp_comparing.md +++ b/docs/webapp/webapp_exp_comparing.md @@ -145,7 +145,7 @@ The **Scalars** (Graph view) and **Plots** tabs compare experiments' plots. The **Scalars** tab displays scalar values as time series line charts. The **Plots** tab compares the last reported iteration sample of each metric/variant combination per compared experiment. -Line, scatter, and bar graphs are compared by overlaying each metric/variant from all compared experiments' into a single +Line, scatter, box, and bar graphs are compared by overlaying each metric/variant from all compared experiments' into a single comparative plot. Use **Group by** to select how to group plots: diff --git a/docs/webapp/webapp_exp_table.md b/docs/webapp/webapp_exp_table.md index c2a9b219..2410e7fe 100644 --- a/docs/webapp/webapp_exp_table.md +++ b/docs/webapp/webapp_exp_table.md @@ -223,11 +223,12 @@ to open the context menu | Archive | Move experiment to the project's archive. If it is shared (ClearML Hosted Service only), the experiment becomes private. | Any state | *Pending* to *Draft* | | Restore |Action available in the archive. Restore an experiment to the active experiments table.| Any State | None | | Delete | Action available in the archive. Delete an experiment, which will also remove all their logs, results, artifacts and debug samples. | Any State | N/A | -| Enqueue | Add an experiment to a queue for a worker or workers (listening to the queue) to execute. | *Draft* | *Pending* | +| Enqueue | Add an experiment to a queue for a worker or workers (listening to the queue) to execute. | *Draft*, *Aborted* | *Pending* | | Dequeue | Remove an experiment from a queue. | *Pending* | *Draft* | | Reset | Delete the log and output from a previous run of an experiment (for example, before rerunning it). | *Completed*, *Aborted*, or *Failed* | *Draft* | | Abort | Manually terminate a *Running* experiment. | *Running* | *Aborted* | | Abort All Children | Manually terminate all *Running* experiments which have this task as a parent | *Running* or *Aborted* | None for parent experiment, *Aborted* for child experiments | +| Retry | Enqueue a failed experiment in order to rerun it. Make sure you have resolved the external problem which previously prevented the experiment’s completion. | *Failed* | *Pending* | | Publish | Publish an experiment to prevent changes to its tracking data, inputs, and outputs. Published experiments and their models are read-only. *Published* experiments cannot be enqueued, but they can be cloned, and their clones can be edited, tuned, and enqueued. | *Completed*, *Aborted*, or *Failed*. | *Published* | | Add Tag | Tag experiments with color-coded labels to assist you in organizing your work. See [tagging experiments](webapp_exp_track_visual.md#tagging-experiments). | Any state | None | | Clone | Make an exact, editable copy of an experiment (for example, to reproduce an experiment, but keep the original). | *Draft* | Newly Cloned Experiment is *Draft* | @@ -289,7 +290,7 @@ Click Tune. + +![Task audit log](../img/webapp_info_audit_log.png) + +:::note Limited persistency +ClearML maintains a system-wide, large but strict limit for task history items. Once the limit is reached, the oldest entries are purged to make room for fresh entries. +::: + +### Description +Add descriptive text to the experiment in the **Description** section. To modify the description, hover over the +description box and click **Edit**. + +### Experiment Details +The **Experiment Details** section lists information describing the experiment: -General experiment details appear in the **INFO** tab. This includes information describing the stored experiment: -* Experiment description (editable, hover over the description box and click **Edit**) * The parent experiment * Project name * Creation, start, and last update dates and times @@ -262,19 +293,21 @@ is downloadable. To view the end of the log, click **Jump to end**. All scalars that ClearML automatically logs, as well as those explicitly reported in code, appear in **SCALARS**. -Scalar series are presented in a line chart. To see the series for a metric in high resolution, -view it in full screen mode by hovering over the graph and clicking Maximize plot icon. +Scalar series can be displayed in [graph view](#graph-view) (default) or in [metric values view](#metric-values-view): + +#### Graph View +Scalar graph view (Graph view) +shows scalar series plotted as a time series line chart. The series are sub-sampled for +display efficiency. For high resolution, view a series in full screen mode by hovering over the graph and clicking Maximize plot icon. + +Single value scalars (see [`Logger.report_single_value`](../references/sdk/logger.md#report_single_value)) are shown in +a `Summary` table. :::info Full Screen Refresh Scalar graphs in full screen mode do not auto-refresh. Click Refresh to update the graph. ::: -Reported single value scalars are aggregated into a table plot displaying scalar names and values (see [`Logger.report_single_value`](../references/sdk/logger.md#report_single_value)). - - -#### Scalar Plot Tools - Use the scalar tools to improve analysis of scalar metrics. In the info panel, click Settings gear to use the tools. In the full screen details view, the tools are on the left side of the window. The tools include: * **Group by** - Select one of the following: @@ -310,6 +343,29 @@ are retrieved when the report is displayed allowing your reports to show the lat See additional [plot controls](#plot-controls) below. +#### Metric Values View + +The metric values view (Table view) +shows a table summary of your metrics with a row per metric/variant: + +* First - The metric/variant series' initial value +* Last - The metric/variant series' last value +* Min - The metric/variant series' minimum value +* Max - The metric/variant series' maximum value +* Mean - The metric/variant series' mean value + +If all the values of a specific metric/variant are the same, the row will display a Same values sign. + +![Plots tab](../img/webapp_tracking_34a.png) + +Choose which metrics to view using one of the following out the options: +* Use the quick filter bar (Magnifying glass) +to only show metrics/variants whose name fit a partial-string match +* Use the filter menu (Filter) to +select which metrics to view by clicking their show/hide button (Eye Show). +Click **Hide/Show all**, to quickly hide/show all metrics. + + ### Plots Non-time-series plots appear in **PLOTS**. These include data generated by libraries, visualization tools, and explicitly reported using the ClearML Logger. These may include 2D and 3D plots, tables (Pandas and CSV files), and diff --git a/docs/webapp/webapp_exp_tuning.md b/docs/webapp/webapp_exp_tuning.md index 19aff7d6..b0a7943c 100644 --- a/docs/webapp/webapp_exp_tuning.md +++ b/docs/webapp/webapp_exp_tuning.md @@ -56,16 +56,18 @@ User parameters are editable in any experiment, except experiments whose status #### Source Code -Select source code by changing any of the following: +Modify code execution by changing any of the following: -* Repository, commit (select by ID, tag name, or choose the last commit in the branch), script, and/or working directory. -* Installed Python packages and/or versions - Edit or clear (remove) them all. -* Uncommitted changes - Edit or clear (remove) them all. +* Repository, commit (select by ID, tag name, or choose the last commit in the branch), script, working directory, +and/or binary. +* The Python packages to be installed and/or their versions - Edit the package list, or clear it to have the ClearML +Agent either not install any packages or use an existing repo `requirements.txt` file. If the experiment is based on a +run in which the packages used were eventually different to the ones originally specified, you can easily or reset the +packages to originally recorded values ("Original Pip"). +* Uncommitted changes - Edit or clear all. -**To select different source code:** - -* In the **EXECUTION** tab, hover over a section **>** **EDIT** or (**DISCARD DIFFS** for **UNCOMMITTED CHANGES**) **>** - edit **>** **SAVE**. +**To modify the source code**, hover over the relevant sections in the **EXECUTION** tab to access Edit, Clear/Discard, +and/or Reset functions. diff --git a/docs/webapp/webapp_model_comparing.md b/docs/webapp/webapp_model_comparing.md index 45dee757..1289ba32 100644 --- a/docs/webapp/webapp_model_comparing.md +++ b/docs/webapp/webapp_model_comparing.md @@ -83,7 +83,7 @@ The **Scalars** (Graph view) and **Plots** tabs display plots attached to the mo scalar values as time series line charts. The **Plots** tab compares the last reported iteration sample of each metric/variant combination per compared model. -Line, scatter, and bar graphs are compared by a single plot per metric/variant into which the plots of all compared +Line, scatter, box, and bar graphs are compared by a single plot per metric/variant into which the plots of all compared models are combined. ![Merged plots](../img/webapp_compare_models_merge_plots.png) diff --git a/docs/webapp/webapp_model_endpoints.md b/docs/webapp/webapp_model_endpoints.md new file mode 100644 index 00000000..737edd9f --- /dev/null +++ b/docs/webapp/webapp_model_endpoints.md @@ -0,0 +1,93 @@ +--- +title: Model Endpoints +--- + +:::important Enterprise Feature +This feature is available under the ClearML Enterprise plan. +::: + +The Model Endpoints table lists all currently live (active, and being brought up) model endpoints, allowing you to view +endpoint details and monitor status over time. Whenever you deploy a model through the [ClearML Deploy UI applications](applications/apps_overview.md#deploy), +it will be listed in the table. + +View the model endpoints in table view Table view +or in details view Details view +using the buttons on the top left of the page. Use the table view for a comparative view of your endpoints according to +columns of interest. Use the details view to access a selected endpoint's details. Details view can also be accessed by +double-clicking a specific endpoint in the table view to open its details view. + +Download the model endpoint table as a CSV file by clicking **Download** Download. + +## Endpoint Tables + +Active Endpoints are displayed in the **Active** tab. The table provides the following information: +* Endpoint - Endpoint name +* Model - Model Name +* URL - Endpoint URL +* \# Instances - Number of model instances +* Uptime - Longest duration that any of the model instances has been running +* \# Requests - Total number of requests to the endpoint +* Requests / MIN (avg) - Average request rate in the last minute +* Latency (avg) - Average endpoint response latency + +![Active endpoints](../img/webapp_model_endpoints_active_table.png) + +The **Loading** tab shows endpoints that are being set up, but are not yet active. The table provides the following +information: + +* Instance ID - The model deployment application instance ID +* Model - Model Name +* Uptime - Time since this endpoint has started setting up +* Preprocess artifact - Preprocessing code used for the endpoint +* Input type - Model matrix input type (e.g. uint8, float32, int16, float16) +* Input size - Model matrix input size + +![Loading endpoints](../img/webapp_model_endpoints_loading_table.png) + +You can apply column filters by clicking Filter, +and sort endpoints by clicking Sort order on the relevant column. + +## Active Endpoint Details +Clicking on a model endpoint opens it in detailed view. + +### Details + +The **Details** tab displays the model endpoint information: +* Endpoint Name +* Endpoint URL +* Model name (click to go to ClearML or HuggingFace model page) +* Uptime - Duration of longest running endpoint instance +* Preprocess artifact - Preprocessing code used for the endpoint +* Input type - Model matrix input type (e.g. uint8, float32, int16, float16) +* Input size - Model matrix input size +* Model instances list - Model instances servicing the endpoint. The following information is displayed for each instance: + * Instance ID - Click instance ID to go to the model endpoint's ClearML Application instance dashboard . + * Uptime + * Number of requests - Total count + * Request rate - Average requests per minute + * CPU Count + * GPU Count + * Latency - Average request latency in the last minute + +![Endpoints details](../img/webapp_model_endpoints_details.png) + +### Monitor +The Monitor tab displays the endpoint's operational metrics and resource usage over time graphs: +* Total number of requests +* Average requests/min +* Average latency +* Machine utilization metrics: + * CPU and GPU Usage + * Memory Usage + * Video Memory Usage + * Network Usage + +![Endpoints monitor](../img/webapp_model_endpoints_monitor.png) + +The graphs' time span can be controlled through the menu at its top right corner. +Click Eye Show to control which +plots to display. + +For example, to display specific plots, click **HIDE ALL**, and then click Eye Show +on each plot you want to view. + diff --git a/docs/webapp/webapp_overview.md b/docs/webapp/webapp_overview.md index 2580c8f4..0133b20d 100644 --- a/docs/webapp/webapp_overview.md +++ b/docs/webapp/webapp_overview.md @@ -35,6 +35,9 @@ The WebApp's sidebar provides access to the following modules:
  • Pipelines Pipelines - View and manage your pipelines.
    • +
  • + Model endpoints Model Endpoints - Monitor your live model endpoints (available in the ClearML Enterprise plan). +
  • Reports Reports - View and manage your reports.
    • @@ -42,7 +45,7 @@ The WebApp's sidebar provides access to the following modules: Workers and Queues Orchestration - Autoscale, monitor, and manage your resource usage and workers queues.
  • - ClearML Apps Applications - ClearML's GUI applications for no-code workflow execution. + ClearML Apps Applications - ClearML's GUI applications for no-code workflow execution (available in the ClearML Pro and Enterprise plans).
    • diff --git a/docs/webapp/webapp_projects_page.md b/docs/webapp/webapp_projects_page.md index f3799b03..0576f089 100644 --- a/docs/webapp/webapp_projects_page.md +++ b/docs/webapp/webapp_projects_page.md @@ -21,12 +21,23 @@ page when relevant). Project folders display summarized project information: +
    + ![Project card](../img/webapp_project_card.png) -When relevant, the folder has a tab which displays the number of subprojects in the project. Click the tab to view a list of -subprojects. Click on a subproject's name to navigate to it. +
    -![Subproject tab](../img/webapp_sub_project_card.png) +* When the project contains subprojects, the folder shows the number of subprojects within the project as an additional + tab. Click the tab to view a list of subprojects, and click on a subproject's name to navigate to it. + +
    + + ![Subproject tab](../img/webapp_sub_project_card.png) + +
    + +* When a project's `default_output_destination` is set, the folder displays the Info + indicator. Hover over the indicator to view the default output destination. @@ -47,7 +58,7 @@ of a project folder to open its context menu and access the following project ac ![Project context menu](../img/webapp_projects_context_menu.png) -* **Rename** - Rename the project. +* **Edit** - Rename the project and/or modify its default output destination. * **New Project** - Create a new project (by default a subproject). * **Move to** - Move the project into another project. If the target project does not exist, it is created on-the-fly. * **Delete** - Delete the project. To delete a project, all of its contents (i.e. any pipelines/reports/datasets) must @@ -64,4 +75,11 @@ action. For more information see [Custom UI Context Menu Actions](../deploying_c To create a project, click the **+ NEW PROJECT** button in the top right of the page or in a project's context menu, which will open a **New Project** modal. -![New project modal](../img/webapp_projects_new_project.png) \ No newline at end of file +![New project modal](../img/webapp_projects_new_project.png) + +* Project name +* Create in - Where the project should be created, either as a top-level project (create in `Projects root`) or as a + subproject of another project +* Description +* Default output destination - The storage location where model checkpoints (snapshots) and artifacts will be uploaded + when tasks in this project that do not have an explicit output destination specified are executed diff --git a/sidebars.js b/sidebars.js index 94bd75a8..76c5cd98 100644 --- a/sidebars.js +++ b/sidebars.js @@ -138,6 +138,7 @@ module.exports = { 'webapp/pipelines/webapp_pipeline_page', 'webapp/pipelines/webapp_pipeline_table', 'webapp/pipelines/webapp_pipeline_viewing' ] }, + 'webapp/webapp_model_endpoints', 'webapp/webapp_reports', { 'Orchestration': [ @@ -185,6 +186,7 @@ module.exports = { 'webapp/settings/webapp_settings_admin_vaults', 'webapp/settings/webapp_settings_users', 'webapp/settings/webapp_settings_access_rules', 'webapp/settings/webapp_settings_id_providers', 'webapp/settings/webapp_settings_resource_configs', 'webapp/settings/webapp_settings_usage_billing', + 'webapp/settings/webapp_settings_storage_credentials' ] }, ] @@ -296,8 +298,8 @@ module.exports = { { 'Enterprise': [ - 'release_notes/clearml_server/enterprise/ver_3_22', 'release_notes/clearml_server/enterprise/ver_3_21', - 'release_notes/clearml_server/enterprise/ver_3_20', + 'release_notes/clearml_server/enterprise/ver_3_23', 'release_notes/clearml_server/enterprise/ver_3_22', + 'release_notes/clearml_server/enterprise/ver_3_21', 'release_notes/clearml_server/enterprise/ver_3_20', ] } ]}, diff --git a/static/icons/ico-equal-outline.svg b/static/icons/ico-equal-outline.svg new file mode 100644 index 00000000..8c132362 --- /dev/null +++ b/static/icons/ico-equal-outline.svg @@ -0,0 +1,3 @@ + + + \ No newline at end of file diff --git a/static/icons/ico-model-endpoints.svg b/static/icons/ico-model-endpoints.svg new file mode 100644 index 00000000..7d937b0c --- /dev/null +++ b/static/icons/ico-model-endpoints.svg @@ -0,0 +1,4 @@ + + + +