Update docs (#948)

docs/hyperdatasets/webapp/webapp_datasets_frames.md

@@ -181,7 +181,7 @@ or the arrow keys on the keyboard). Closing the frame editor will prompt you to
 |<img src="/docs/latest/icons/ico-metadata.svg" alt="edit metadata" className="icon size-md space-sm" />|Edit metadata|Hover over an annotation in the list and click the icon to open the edit window. Input the metadata dictionary in JSON format. This metadata is specific to the selected annotation, not the entire frame.|
 |<img src="/docs/latest/icons/ico-lock-open.svg" alt="Lock annotation" className="icon size-md space-sm" />|Lock / Unlock annotation |Click the button on a specific annotation to make it uneditable. You can also click the button on top of the annotations list to lock all annotations in the frame.|
 |<img src="/docs/latest/icons/ico-trash.svg" alt="Trash" className="icon size-md space-sm" />|Delete annotation|Click the annotation or bounded area in the frame and then click the button to delete the annotation.|
-|<img src="/docs/latest/icons/ico-show.svg" alt="Eye Show All" className="icon size-md space-sm" />|Show/hide all annotations |Click the button to view the frame without annotations. When annotations are hidden, they can't be modified. |
+|<img src="/docs/latest/icons/ico-show.svg" alt="Eye Show All" className="icon size-md space-sm" />|Show/hide annotations |Click the button on a specific annotation to hide it. You can also click the button on top of the annotations list to hide all annotations. Hidden annotations can't be modified. |
 ||Delete label |In the relevant annotation, click **x** on the label you want to remove.| 
 
 ### Frame Labels

docs/hyperdatasets/webapp/webapp_datasets_versioning.md

@@ -248,9 +248,15 @@ according to the previously configured setting.
 #### Preview Source
 When using multi-source FrameGroups, users can choose which of the FrameGroups' sources will be displayed as the preview. 
 
-Select a source from the **PREVIEW SOURCE** menu.
+Select a source from the **Current Source** menu.
 Choose the `Default preview source` option to present the first available source for each frame (sources are retrieved in ASCIIbetical order).
 
+Choose the `All sources` option to present all the FrameGroup’s sources in a grid. In this view, the annotation panel 
+shows annotations grouped by their respective sources. Additionally, annotation tools  (e.g. create/delete/modify 
+annotations) are not available in this view.    
+
+![All sources preview](../../img/hyperdatasets/preview_all_sources.png)
+
 :::note Unavailable Source
 If a FrameGroup doesn't have the selected preview source, the preview displays the "Source not available" message.
 ::: 

docs/release_notes/clearml_server/enterprise/ver_3_23.md

@@ -0,0 +1,60 @@
+---
+title: Version 3.23
+---
+
+### Enterprise Server 3.23.0
+
+**Default Behavior Change for Object Access (Access Rules)**
+* Users can only access objects as explicitly specified by the access rules applied to them. If users can no longer 
+access objects (Projects, Tasks, Datasets etc.) they had previously created, verify access rules exist that provide the 
+user access to these objects or their containing project.
+
+**New Features**
+* New ClearML Model dashboard: View all live model endpoints in a single location, complete with real time metrics reporting.
+* New UI pipeline run table comparative view: compare plots and scalars of selected pipeline runs
+* New UI setting for configuring cloud storage credentials with which ClearML can clean up cloud storage artifacts on task deletion. 
+* Add UI re-enqueue of failed tasks
+* Add configuration vaults for service accounts 
+* Add administrator vaults for UI storage credential to allow centralized configuration of storage credentials for UI access for designated user groups 
+* Add "All sources" view to UI Hyper-Dataset frame viewer
+* Add task recent event log to UI task information 
+* Add UI experiment scalar results table view
+* Add show/hide specific frame annotations in UI Hyper-Dataset frame viewer
+* Add UI edit of default Project default output destination
+
+**Bug Fixes**
+* Fix broken download links to artifacts stored in Azure ([ClearML Server GitHub issue #247](https://github.com/allegroai/clearml-server/issues/247))
+* Fix Cross-site Scripting (XSS) vulnerability (CWE-79)
+* Fix UI experiment textual comparison diff showing contextual diffs as different contents ([ClearML GitHub issue #646](https://github.com/allegroai/clearml/issues/646))
+* Fix Model API calls fail when its creating task has been deleted ([ClearML GitHub issue #1299](https://github.com/allegroai/clearml/issues/1299))
+* Fix UI plots display "Iteration 0" when it is the only reported iteration ([ClearML GitHub issue #1267](https://github.com/allegroai/clearml/issues/1267))
+* Fix scalar series starting with angle bracket (`<`) causes UI scalar plot legend to display raw html ([ClearML GitHub issue #1292](https://github.com/allegroai/clearml/issues/1292)) 
+* Fix auto-refresh modifies UI experiment debug sample view ([ClearML GitHub issue #1529](https://github.com/allegroai/clearml/issues/1529)) 
+* Fix Hydra parameters not displaying correctly in UI experiment comparison parallel coordinates plot
+* Fix UI pipeline DAG display to be "bottoms-up" so all final steps appear in the bottom of the diagram ([ClearML Web GitHub PR #86](https://github.com/allegroai/clearml-web/pull/86))
+* Fix UI experiment comparison does not overlay box plots and histograms ([ClearML GitHub issue #1298](https://github.com/allegroai/clearml/issues/1298))
+* Fix modifying UI experiment input models does not work
+* Fix UI Reports image upload not working
+* Fix active masks labels not displaying in UI annotation task frame viewer 
+* Fix UI Hyper-Dataset version status does not update when published
+* Fix UI model hidden plot selection does not persist upon refresh
+* Fix deleting large number of experiments via UI not working properly
+* Fix experiment name legend not displayed in UI experiment plot comparison 
+* Fix archiving pipeline run does not abort pipeline step tasks
+* Fix "Restore" and "Delete" buttons missing from UI experiment archive action bar
+* Fix UI Resource Configuration allows same-named profile-pool links
+* Fix UI experiment debug sample viewer sometimes displays incorrect sample
+* Fix UI Resource Policy creation policy modal pops up unnecessarily
+* Fix UI Resource Policy creation modal does not display newly created user groups as an assignment option
+* Fix UI applications do not display indication that running instance limit has been reached
+* Fix UI Settings' Configuration section does not display indication that the number of credentials limit has been reached
+* Fix iteration number not displayed in titles of confusion matrix and table plots in UI experiment comparison
+* Fix "Create new" option not displaying when inputting project name in UI Report creation modal
+* Fix running experiment that is archived continues running
+* Fix adding second resource policy to resource group does not assign the policy
+* Fix UI experiment requires Git information when repository isn't set
+* Fix pipeline run version not set when re-executed via the UI
+* Fix UI DataView successfully created with identical IDs in label enumeration
+* Fix UI DataView created successfully when "Limit Frames" repetition option is selected, but the maximum number of frames is not set
+* Fix moving a enqueued experiment to a new UI queue results in error 
+* Fix metric and hyperparameter group string not searchable in UI table 

docs/webapp/applications/apps_overview.md

@@ -50,6 +50,8 @@ Each application's page is split into two sections:
 * App Instance List - Launch new app instances and view previously launched instances. Click on an instance to view its 
   dashboard. Hover over it to access the [app instance actions](#app-instance-actions).
 * App Instance Dashboard - The main section of the app page: displays the selected app instance's status and results.
+  Use the search bar <img src="/docs/latest/icons/ico-search.svg" alt="Magnifying glass" className="icon size-md space-sm" /> 
+  to quickly find an app instance by name.
 
 ![App format](../../img/apps_format_overview.png)
 

docs/webapp/pipelines/webapp_pipeline_table.md

@@ -6,11 +6,14 @@ The pipeline runs table is a [customizable](#customizing-the-runs-table) list of
 view a run's details, and manage runs (create, continue, or abort). The runs table's auto-refresh allows users 
 to continually monitor run progress.
 
-View the runs table in table view <img src="/docs/latest/icons/ico-table-view.svg" alt="Table view" className="icon size-md space-sm" /> 
-or in details view <img src="/docs/latest/icons/ico-split-view.svg" alt="Details view" className="icon size-md space-sm" />, 
-using the buttons on the top left of the page. Use the table view for a comparative view of your runs according to 
-columns of interest. Use the details view to access a selected run's details, while keeping the pipeline runs list in view. 
-Details view can also be accessed by double-clicking a specific pipeline run in the table view to open its details view. 
+View the runs table in table view <img src="/docs/latest/icons/ico-table-view.svg" alt="Table view" className="icon size-md space-sm" />, 
+details view <img src="/docs/latest/icons/ico-split-view.svg" alt="Details view" className="icon size-md space-sm" />, 
+or comparison view <img src="/docs/latest/icons/ico-charts-view.svg" alt="Comparison view" className="icon size-md space-sm" />
+using the buttons on the top left of the page. Use the details view to access a selected run's details, while keeping 
+the run list in view. Details view can also be accessed by double-clicking a specific pipeline run in the table view to 
+open its details view. Use the [comparison view](#comparing-runs) to compare your pipeline run's scalar and plot results. 
+This view compares the scalars/plots  of currently selected pipeline runs. If no runs are selected, The first 100 visible 
+runs in the table are compared.
 
 You can archive pipeline runs so the runs table doesn't get too cluttered. Click **OPEN ARCHIVE** on the top of the 
 table to open the archive and view all archived runs. From the archive, you can restore 
@@ -146,3 +149,39 @@ selecting items beyond the items currently on-screen:
 * **None** - Clear selection
 * **Filtered** - Select all runs in the project that match the current active table filters 
 
+## Comparing Runs
+The comparison view compares pipeline run scalar and plot results. When selected, the view presents a comparison of all 
+[selected runs](#selecting-multiple-runs). If no runs are selected, the first 100 visible runs in the table are compared.
+
+In the dropdown menu, select to view **Scalars** or **Plots**.
+
+**Scalars** shows pipeline run scalar results as time series line graphs. 
+
+![Scalar line graphs](../../img/pipelines_comparison_scalars.png)
+
+All single value scalars are plotted into a single clustered bar chart under the "Summary" title, where each cluster 
+represents a reported metric, and each bar in the cluster represents an experiment.
+
+![Single scalar comparison](../../img/pipelines_comparison_single_scalar.png)
+
+Click <img src="/docs/latest/icons/ico-settings.svg" alt="Setting Gear" className="icon size-md" /> to customize which 
+metrics to view.
+
+In the **Scalars** view, click <img src="/docs/latest/icons/ico-tune.svg" alt="Tuning" className="icon size-md" /> to access [scalar plot tools](../webapp_exp_track_visual.md#scalar-plot-tools).
+
+**Plots** shows the last reported iteration sample of each metric/variant combination per compared run. 
+
+Line, scatter, box, and bar graphs are compared by overlaying each metric/variant from all compared runs' into a single 
+comparative plot.
+
+![Merged plots](../../img/pipelines_comparison_plots_merged.png)
+
+Other plot types are displayed separately for each run.
+
+![Side-by-side plots](../../img/pipelines_comparison_plots.png)
+
+
+### Run Details Comparison
+For a more in depth comparison of pipeline runs, select the runs to compare and click **Compare** in the batch action 
+bar. In the run comparison pages, you can compare details, hyperparameters, scalars, plots, and debug samples. For more 
+information, see [Comparing Experiments](../webapp_exp_comparing.md).

docs/webapp/settings/webapp_settings_admin_vaults.md

@@ -7,10 +7,17 @@ This feature is available under the ClearML Enterprise plan.
 :::
 
 Administrators can define multiple [configuration vaults](webapp_settings_profile.md#configuration-vault) which will each be applied to designated 
-[user groups](webapp_settings_users.md). Use configuration vaults to extend and/or override entries in the local ClearML [configuration file](../../configs/clearml_conf.md)
-where a ClearML task is executed. Configuration vault values will be applied to tasks run by members of the designated user groups. 
+[user groups](webapp_settings_users.md). There are two types of vaults: 
+* [Client configuration (Agent/SDK/CLI)](#client-configuration-agentsdkcli)
+* [UI storage credentials](#ui-storage-credentials)   
 
-To apply its contents, a vault should be enabled. New entries will extend the configuration in the local ClearML [configuration file](../../configs/clearml_conf.md). 
+To apply its contents, a vault should be enabled in the [Administrator Vault Table](#administrator-vault-table). 
+
+## Client Configuration (Agent/SDK/CLI)
+Client configuration vaults extend and/or override entries in the local ClearML [configuration file](../../configs/clearml_conf.md)
+where a task is executed. Vault values will be applied to tasks run by members of the designated user groups. 
+
+New entries will extend the configuration in the local ClearML [configuration file](../../configs/clearml_conf.md). 
 Most existing configuration file entries will be overridden by the vault values.
 
 :::info 
@@ -23,24 +30,41 @@ The following configuration values are machine and/or agent specific, so they ca
 * `agent.debug`
 :::
 
-**To create a vault:**
+**To create a Client configuration vault:**
 1. Click **+ Add Vault**
 1. Fill in vault details:
    1. Vault name - Name that appears in the Administrator Vaults table
    1. User Group - Specify the User Group that the vault affects
-   1. Format - Specify the configuration format: HOCON / JSON / YAML.
-1. Fill in the configuration values (click <img src="/docs/latest/icons/ico-info.svg" alt="Info" className="icon size-md space-sm" /> 
-to view configuration file reference). To import and existing configuration file, click <img src="/docs/latest/icons/ico-import.svg" alt="Import" className="icon size-md space-sm" />. 
+   1. Target - Vault type. Select `Client (Agent/SDK/UI)`
+   1. Format - Specify the configuration format: HOCON / JSON / YAML. 
+   1. Fill in the configuration values (click <img src="/docs/latest/icons/ico-info.svg" alt="Info" className="icon size-md space-sm" /> 
+   to view configuration file reference). To import an existing configuration file, click <img src="/docs/latest/icons/ico-import.svg" alt="Import" className="icon size-md space-sm" />. 
 1. Click **Save** 
 
+## UI Storage Credentials   
+UI storage credential vaults configure UI access to cloud storage credentials for a designated group of users. 
+
+**To create a vault:**
+
+1. Click **+ Add Vault**
+1. Fill in vault details:
+   1. Vault name - Name that appears in the Administrator Vaults table
+   1. User Group - Specify the User Group that the vault affects
+   1. Target - Select `UI storage credentials` 
+   1. \+ Add access keys - Enter storage credentials (see [Browser Cloud Storage Access](webapp_settings_profile.md#browser-cloud-storage-access))
+1. Click **Save**
+
+## Administrator Vault Table
+
 The **Administrator Vaults** table lists all currently defined vaults, and the following details:
 * Active - Toggle to enable / disable the vault
 * Name - Vault name
+* Target - Type of vault: `Client (Agent/SDK/CLI)` or `UI storage credentials` 
 * Group - User groups to apply this vault to 
 * ID - Vault ID (click to copy)
 * Vault Content - Vault content summary
 * Update - Last update time
 
-Hover over a vault in the table to Download, Edit, or Delete a vault.  
+Hover over a vault in the table to **Download**, **Edit**, or **Delete** a vault.  
 
 ![Admin vaults](../../img/settings_admin_vaults.png)

docs/webapp/settings/webapp_settings_overview.md

@@ -24,4 +24,6 @@ The Settings page consists of the following sections:
   * [Identity Providers](webapp_settings_id_providers.md) (ClearML Enterprise Server) - Manage server identity providers
   * [Resource Configuration](webapp_settings_resource_configs.md) (ClearML Enterprise Server) - Define the available resources and the way in which they 
   will be allocated to different workloads 
-  * [Usage & Billing](webapp_settings_usage_billing.md) (ClearML Hosted Service) - View current usage information and billing details 
+  * [Usage & Billing](webapp_settings_usage_billing.md) (ClearML Hosted Service) - View current usage information and billing details
+  * [Storage Credentials](webapp_settings_storage_credentials.md) (ClearML Enterprise Server) - Configure storage provider access credentials to 
+  enable ClearML to delete artifacts stored in cloud storage when tasks and models are deleted

docs/webapp/settings/webapp_settings_profile.md

@@ -62,7 +62,7 @@ to switch to.
  
 ![Workspace configuration page](../../img/settings_workspace_configuration.png)
 
-### ClearML Credentials
+### ClearML App Credentials
 
 Generate ClearML credentials, made up of an access and secret key pair, and insert them into your [configuration file](../../configs/clearml_conf.md) 
 or Jupyter Notebook to grant the ClearML SDK and the ClearML Agent API access to the server. 
@@ -91,6 +91,20 @@ these credentials cannot be recovered.
 
 **To revoke ClearML credentials:** hover over the desired credentials, and click <img src="/docs/latest/icons/ico-trash.svg" alt="Trash can" className="icon size-md" />
 
+## AI Application Gateway Tokens 
+
+:::important Enterprise Feature
+This feature is available under the ClearML Enterprise plan.
+:::
+
+The AI Application Gateway enables external access to ClearML tasks and applications. The gateway is configured with an 
+endpoint or external address (ingress), accessible from outside ClearML.
+
+Generate tokens providing API access to the AI Application Gateway endpoints:
+
+1. Click **Generate a Token**
+1. Under `Expiration`, enter the number of days the token should remain valid 
+1. Click `Generate`, which creates a token and copies it to your clipboard 
 
 ### Changing Your Workspace Name
 To change the name of your own workspace, click **Edit workspace name**  <img src="/docs/latest/icons/ico-edit.svg" alt="Edit Pencil" className="icon size-md" /> 

docs/webapp/settings/webapp_settings_storage_credentials.md

@@ -0,0 +1,55 @@
+---
+title: Storage Credentials 
+---
+
+:::important Enterprise Feature 
+This feature is available under the ClearML Enterprise plan.
+:::
+
+To enable ClearML to delete task artifacts stored in cloud storage when a task is deleted, configure access credentials for your storage provider:
+* [Google Cloud Storage](#google-cloud-storage)
+* [AWS S3 Storage](#aws-s3-storage)
+* [Azure](#azure)
+
+![Storage Credentials page](../../img/webapp_settings_storage_credentials.png)
+
+## Google Cloud Storage
+Set up credentials for Google Cloud buckets: 
+* Default credentials - These credentials apply to all GCS buckets unless bucket-specific credentials are set.
+  * Project - Default Google Cloud Storage project
+  * Credentials JSON
+* Bucket specific credentials:
+  * Bucket 
+  * Project
+  * Credentials JSON
+
+## AWS S3 Storage 
+Set up credentials for S3 protocol storage (i.e. AWS S3, MinIO, etc.): 
+* Default credentials - These credentials apply to all buckets unless bucket-specific credentials are set:
+  * Access Key - Default access key for the storage service.
+  * Secret -  Default secret access key.
+  * Access token - Session key for temporary credentials (if applicable).
+  * Region -  Default region for all unspecified buckets.
+  * Credentials Chain - If selected, use boto3 default [credentials search](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) 
+  (i.e. look for credentials in environment variables, credential files, and instance metadata services).
+* Bucket Specific Credentials: 
+  * Bucket - Name of the specific bucket.
+  * Region - Region for the bucket.
+  * Host - For non-AWS endpoints, the host URL and port number of the specific bucket. Note that port specification 
+  is *always* needed (e.g. `my-minio-host:9000`), even for standard ports like 433 for HTTPS (e.g. `my-minio-host:433`) 
+  * Secure Host - Select in order to enable TLS. 
+  * Verify SSL certificate - Select to enable SSL verification for secure hosts.
+  * Access key - Access key for the bucket.
+  * Secret - Secret key for the bucket.
+  * Access token - The session key for your bucket. This is only needed when you are using temporary credentials.
+  * Use Credentials chain - If selected, use boto3 default [credentials search](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) 
+  (i.e. looks for credentials in environment variables, credential files, and instance metadata services).
+
+## Azure
+Set up credentials for Azure storage containers: 
+* Account name - Azure storage account name.
+* Account key - Azure storage account key.
+* Container name - Name of the specific container.
+
+
+

docs/webapp/settings/webapp_settings_users.md

@@ -109,6 +109,32 @@ To revoke a set of credentials:
 1. In the editing panel, hover of the relevant credential's row
 2. Click the <img src="/docs/latest/icons/ico-trash.svg" alt="Trash can" className="icon size-md" /> button
 
+### Service Account Configuration Vault
+Use a service account’s configuration vault to store ClearML configuration entries that can extend the ClearML 
+[configuration file](../../configs/clearml_conf.md) of any ClearML Agents or ClearML SDK running with the service account's 
+credentials. 
+
+Vault entries will extend the configuration in the ClearML [configuration file](../../configs/clearml_conf.md) if they 
+don't yet exist, and override values for those already present in the file.
+
+Fill in values using any of ClearML supported configuration formats: HOCON / JSON / YAML.
+
+To edit vault contents:
+1. Click on the relevant service account to open its details panel 
+2. Click **EDIT** on the configuration vault 
+3. Insert / edit the configurations in the vault
+4. Press **OK**
+
+To apply vault contents:
+* Click the toggle atop the vault to enable / disable the configurations
+* Once enabled, the configurations will be merged to the configuration file during ClearML and ClearML Agent usage
+
+In addition to the service account-specific configuration vault, [administrator vaults](#administrator-vaults) can also 
+be applied to service accounts. See all the vaults applied to the account in the **Applied administrator vaults** below 
+the configuration vault.   
+
+![Service Account Config Vault](../../img/settings_service_account_config_vault.png)
+
 ### Deleting Service Account
 Deleting a service account will revoke its credentials, causing agents using the account's credentials to fail. 
 Tasks and associated artifacts logged to your workspace by a service account will remain in your workspace.

docs/webapp/webapp_exp_comparing.md

@@ -145,7 +145,7 @@ The **Scalars** (Graph view) and **Plots** tabs compare experiments' plots.
 The **Scalars** tab displays scalar values as time series line charts. The **Plots** tab compares the last reported 
 iteration sample of each metric/variant combination per compared experiment.
 
-Line, scatter, and bar graphs are compared by overlaying each metric/variant from all compared experiments' into a single
+Line, scatter, box, and bar graphs are compared by overlaying each metric/variant from all compared experiments' into a single
 comparative plot. 
 
 Use **Group by** to select how to group plots:

docs/webapp/webapp_exp_table.md

@@ -223,11 +223,12 @@ to open the context menu
 | Archive | Move experiment to the project's archive. If it is shared (ClearML Hosted Service only), the experiment becomes private. | Any state |  *Pending* to *Draft*  |
 | Restore |Action available in the archive. Restore an experiment to the active experiments table.| Any State | None |
 | Delete |  Action available in the archive. Delete an experiment, which will also remove all their logs, results, artifacts and debug samples. | Any State | N/A |
-| Enqueue | Add an experiment to a queue for a worker or workers (listening to the queue) to execute. | *Draft* | *Pending* |
+| Enqueue | Add an experiment to a queue for a worker or workers (listening to the queue) to execute. | *Draft*, *Aborted* | *Pending* |
 | Dequeue | Remove an experiment from a queue. | *Pending* | *Draft* |
 | Reset  | Delete the log and output from a previous run of an experiment (for example, before rerunning it). | *Completed*, *Aborted*, or *Failed* | *Draft* |
 | Abort | Manually terminate a *Running* experiment. | *Running* | *Aborted* |
 | Abort All Children | Manually terminate all *Running* experiments which have this task as a parent | *Running* or *Aborted* | None for parent experiment, *Aborted* for child experiments |
+| Retry | Enqueue a failed experiment in order to rerun it. Make sure you have resolved the external problem which previously prevented the experiment’s completion. | *Failed* | *Pending* |
 | Publish | Publish an experiment to prevent changes to its tracking data, inputs, and outputs. Published experiments and their models are read-only. *Published* experiments cannot be enqueued, but they can be cloned, and their clones can be edited, tuned, and enqueued. | *Completed*, *Aborted*, or *Failed*.  | *Published* |
 | Add Tag | Tag experiments with color-coded labels to assist you in organizing your work. See [tagging experiments](webapp_exp_track_visual.md#tagging-experiments). | Any state |  None  |
 | Clone | Make an exact, editable copy of an experiment (for example, to reproduce an experiment, but keep the original). | *Draft* | Newly Cloned Experiment is *Draft* |
@@ -289,7 +290,7 @@ Click <img src="/docs/latest/icons/ico-tune.svg" alt="Tune" className="icon size
 **Plots** shows the last reported iteration sample of each metric/variant combination per compared 
 experiment. 
 
-Line, scatter, and bar graphs are compared by overlaying each metric/variant from all compared experiments' into a 
+Line, scatter, box, and bar graphs are compared by overlaying each metric/variant from all compared experiments' into a 
 single comparative plot. 
 
 ![Merged comparison plots](../img/webapp_compare_view_4.png)

docs/webapp/webapp_exp_track_visual.md

@@ -8,7 +8,7 @@ including:
 * [Execution details](#execution) - Code, the base Docker image used for [ClearML Agent](../clearml_agent.md), output destination for artifacts, and the logging level.
 * [Configuration](#configuration) - Hyperparameters, user properties, and configuration objects.
 * [Artifacts](#artifacts) - Input model, output model, model snapshot locations, other artifacts.
-* [General information](#general-information) - Information about the experiment, for example: the experiment start, create, and last update times and dates, user creating the experiment, and its description.
+* [Info](#info) - Extended experiment information, such as the experiment start, create, and last update times and dates, user creating the experiment, and its description.
 * [Console](#console) - stdout, stderr, output to the console from libraries, and ClearML explicit reporting.
 * [Scalars](#scalars) - Metric plots.
 * [Plots](#plots) - Other plots and data, for example: Matplotlib, Plotly, and ClearML explicit reporting.
@@ -216,10 +216,41 @@ In the **ARTIFACTS** tab **>** **OTHER** **>** Select an artifact **>** Either:
 
 ![Other artifacts section](../img/webapp_tracking_30.png)
 
-## General Information
+## Info
+
+The **INFO** tab shows extended experiment information:
+* [Latest experiment events log](#latest-events-log)
+* [Experiment description](#description)
+* [Experiment details](#experiment-details)
+
+### Latest Events Log 
+
+:::important Enterprise Feature
+This feature is available under the ClearML Enterprise plan
+:::
+
+The Enterprise Server also displays a detailed history of  task activity: 
+* Task action (e.g. status changes, project move, etc.)
+* Action time
+* Acting user
+* Action source (i.e. ClearML Agent, SDK, or UI)
+* Action source version 
+
+To download the task history as a CSV file, hover over the log and click <img src="/docs/latest/icons/ico-download.svg" alt="Download" className="icon size-md space-sm" />.   
+
+![Task audit log](../img/webapp_info_audit_log.png)
+
+:::note Limited persistency
+ClearML maintains a system-wide, large but strict limit for task history items. Once the limit is reached, the oldest entries are purged to make room for fresh entries.
+:::
+
+### Description 
+Add descriptive text to the experiment in the **Description** section. To modify the description, hover over the 
+description box and click **Edit**.
+
+### Experiment Details
+The **Experiment Details** section lists information describing the experiment:
 
-General experiment details appear in the **INFO** tab. This includes information describing the stored experiment:
-* Experiment description (editable, hover over the description box and click **Edit**)
 * The parent experiment
 * Project name
 * Creation, start, and last update dates and times
@@ -262,19 +293,21 @@ is downloadable. To view the end of the log, click **Jump to end**.
 All scalars that ClearML automatically logs, as well as those explicitly reported in code, appear in
 **SCALARS**. 
 
-Scalar series are presented in a line chart. To see the series for a metric in high resolution, 
-view it in full screen mode by hovering over the graph and clicking <img src="/docs/latest/icons/ico-maximize.svg" alt="Maximize plot icon" className="icon size-sm space-sm" />.
+Scalar series can be displayed in [graph view](#graph-view) (default) or in [metric values view](#metric-values-view):
+
+#### Graph View 
+Scalar graph view (<img src="/docs/latest/icons/ico-charts-view.svg" alt="Graph view" className="icon size-md space-sm" />) 
+shows scalar series plotted as a time series line chart. The series are sub-sampled for 
+display efficiency. For high resolution, view a series in full screen mode by hovering over the graph and clicking <img src="/docs/latest/icons/ico-maximize.svg" alt="Maximize plot icon" className="icon size-sm space-sm" />.
+
+Single value scalars (see [`Logger.report_single_value`](../references/sdk/logger.md#report_single_value)) are shown in 
+a `Summary` table.
 
 :::info Full Screen Refresh
 Scalar graphs in full screen mode do not auto-refresh. Click <img src="/docs/latest/icons/ico-reset.svg" alt="Refresh" className="icon size-md space-sm medium-zoom-image" /> 
 to update the graph. 
 :::
 
-Reported single value scalars are aggregated into a table plot displaying scalar names and values (see [`Logger.report_single_value`](../references/sdk/logger.md#report_single_value)). 
-
-
-#### Scalar Plot Tools
-
 Use the scalar tools to improve analysis of scalar metrics. In the info panel, click <img src="/docs/latest/icons/ico-settings.svg" alt="Settings gear" className="icon size-md space-sm" /> to use the tools. In the full screen details view, the tools
 are on the left side of the window. The tools include:
 * **Group by** - Select one of the following:
@@ -310,6 +343,29 @@ are retrieved when the report is displayed allowing your reports to show the lat
 
 See additional [plot controls](#plot-controls) below.
 
+#### Metric Values View
+
+The metric values view (<img src="/docs/latest/icons/ico-table-view.svg" alt="Table view" className="icon size-md space-sm" />)
+shows a table summary of your metrics with a row per metric/variant:
+
+* First - The metric/variant series' initial value
+* Last - The metric/variant series' last value
+* Min -  The metric/variant series' minimum value
+* Max - The metric/variant series' maximum value
+* Mean - The metric/variant series' mean value
+
+If all the values of a specific metric/variant are the same, the row will display a <img src="/docs/latest/icons/ico-equal-outline.svg" alt="Same values" className="icon size-md space-sm" /> sign.
+
+![Plots tab](../img/webapp_tracking_34a.png)
+
+Choose which metrics to view using one of the following out the options:
+* Use the quick filter bar (<img src="/docs/latest/icons/ico-search.svg" alt="Magnifying glass" className="icon size-md space-sm" />) 
+to only show metrics/variants whose name fit a partial-string match 
+* Use the filter menu (<img src="/docs/latest/icons/ico-filter-off.svg" alt="Filter" className="icon size-md" />) to 
+select which metrics to view by clicking their show/hide button (<img src="/docs/latest/icons/ico-show.svg" alt="Eye Show" className="icon size-md space-sm" />). 
+Click **Hide/Show all**, to quickly hide/show all metrics.
+
+
 ### Plots
 Non-time-series plots appear in **PLOTS**. These include data generated by libraries, visualization tools, and 
 explicitly reported using the ClearML Logger. These may include 2D and 3D plots, tables (Pandas and CSV files), and 

docs/webapp/webapp_exp_tuning.md

@@ -56,16 +56,18 @@ User parameters are editable in any experiment, except experiments whose status
 
 #### Source Code
 
-Select source code by changing any of the following:
+Modify code execution by changing any of the following:
 
-* Repository, commit (select by ID, tag name, or choose the last commit in the branch), script, and/or working directory.
-* Installed Python packages and/or versions - Edit or clear (remove) them all.
-* Uncommitted changes - Edit or clear (remove) them all.
+* Repository, commit (select by ID, tag name, or choose the last commit in the branch), script, working directory, 
+and/or binary.
+* The Python packages to be installed and/or their versions - Edit the package list, or clear it to have the ClearML 
+Agent either not install any packages or use an existing repo `requirements.txt` file. If the experiment is based on a 
+run in which the packages used were eventually different to the ones originally specified, you can easily or reset the 
+packages to originally recorded values ("Original Pip").
+* Uncommitted changes - Edit or clear all.
 
-**To select different source code:**
-
-* In the **EXECUTION** tab, hover over a section **>** **EDIT** or (**DISCARD DIFFS** for **UNCOMMITTED CHANGES**) **>**
-  edit **>** **SAVE**.
+**To modify the source code**, hover over the relevant sections in the **EXECUTION** tab to access Edit, Clear/Discard, 
+and/or Reset functions.
 
 
 

docs/webapp/webapp_model_comparing.md

@@ -83,7 +83,7 @@ The **Scalars** (Graph view) and **Plots** tabs display plots attached to the mo
 scalar values as time series line charts. The **Plots** tab compares the last reported iteration sample of each 
 metric/variant combination per compared model. 
 
-Line, scatter, and bar graphs are compared by a single plot per metric/variant into which the plots of all compared 
+Line, scatter, box, and bar graphs are compared by a single plot per metric/variant into which the plots of all compared 
 models are combined.
 
 ![Merged plots](../img/webapp_compare_models_merge_plots.png)

docs/webapp/webapp_model_endpoints.md

@@ -0,0 +1,93 @@
+---
+title: Model Endpoints 
+---
+
+:::important Enterprise Feature
+This feature is available under the ClearML Enterprise plan.
+:::
+
+The Model Endpoints table lists all currently live (active, and being brought up) model endpoints, allowing you to view 
+endpoint details and monitor status over time. Whenever you deploy a model through the [ClearML Deploy UI applications](applications/apps_overview.md#deploy), 
+it will be listed in the table. 
+
+View the model endpoints in table view <img src="/docs/latest/icons/ico-table-view.svg" alt="Table view" className="icon size-md space-sm" /> 
+or in details view <img src="/docs/latest/icons/ico-split-view.svg" alt="Details view" className="icon size-md space-sm" /> 
+using the buttons on the top left of the page. Use the table view for a comparative view of your endpoints according to 
+columns of interest. Use the details view to access a selected endpoint's details. Details view can also be accessed by 
+double-clicking a specific endpoint in the table view to open its details view.
+
+Download the model endpoint table as a CSV file by clicking **Download** <img src="/docs/latest/icons/ico-download.svg" alt="Download" className="icon size-sm space-sm" />.
+
+## Endpoint Tables
+
+Active Endpoints are displayed in the **Active** tab. The table provides the following information: 
+* Endpoint - Endpoint name
+* Model  - Model Name
+* URL - Endpoint URL
+* \# Instances - Number of model instances
+* Uptime - Longest duration that any of the model instances has been running
+* \# Requests - Total number of requests to the endpoint
+* Requests / MIN (avg) - Average request rate in the last minute
+* Latency (avg) - Average endpoint response latency
+
+![Active endpoints](../img/webapp_model_endpoints_active_table.png)
+
+The **Loading** tab shows endpoints that are being set up, but are not yet active. The table provides the following 
+information:
+
+* Instance ID - The model deployment application instance ID
+* Model - Model Name
+* Uptime - Time since this endpoint has started setting up
+* Preprocess artifact - Preprocessing code used for the endpoint 
+* Input type - Model matrix input type  (e.g. uint8, float32, int16, float16)
+* Input size - Model matrix input size 
+
+![Loading endpoints](../img/webapp_model_endpoints_loading_table.png)
+
+You can apply column filters by clicking <img src="/docs/latest/icons/ico-filter-off.svg" alt="Filter" className="icon size-md" />, 
+and sort endpoints by clicking <img src="/docs/latest/icons/ico-sort-off.svg" alt="Sort order" className="icon size-md space-sm" /> on the relevant column.
+
+## Active Endpoint Details 
+Clicking on a model endpoint opens it in detailed view.
+
+### Details
+
+The **Details** tab displays the model endpoint information: 
+* Endpoint Name
+* Endpoint URL
+* Model name (click to go to ClearML or HuggingFace model page)
+* Uptime - Duration of longest running endpoint instance
+* Preprocess artifact - Preprocessing code used for the endpoint 
+* Input type - Model matrix input type  (e.g. uint8, float32, int16, float16)
+* Input size - Model matrix input size 
+* Model instances list - Model instances servicing the endpoint. The following information is displayed for each instance:
+  * Instance ID - Click instance ID to go to the model endpoint's ClearML Application instance dashboard .
+  * Uptime
+  * Number of requests - Total count
+  * Request rate - Average requests per minute
+  * CPU Count
+  * GPU Count 
+  * Latency - Average request latency in the last minute
+
+![Endpoints details](../img/webapp_model_endpoints_details.png)
+
+### Monitor
+The Monitor tab displays the endpoint's operational metrics and resource usage over time graphs: 
+* Total number of requests 
+* Average requests/min 
+* Average latency 
+* Machine utilization metrics:
+  * CPU and GPU Usage
+  * Memory Usage 
+  * Video Memory Usage
+  * Network Usage
+
+![Endpoints monitor](../img/webapp_model_endpoints_monitor.png)
+
+The graphs' time span can be controlled through the menu at its top right corner. 
+Click <img src="/docs/latest/icons/ico-show.svg" alt="Eye Show" className="icon size-md space-sm" /> to control which 
+plots to display. 
+
+For example, to display specific plots, click **HIDE ALL**, and then click <img src="/docs/latest/icons/ico-show.svg" alt="Eye Show" className="icon size-md space-sm" /> 
+on each plot you want to view.
+

docs/webapp/webapp_overview.md

@@ -35,6 +35,9 @@ The WebApp's sidebar provides access to the following modules:
   <li>
     <img src="/docs/latest/icons/ico-pipelines.svg" alt="Pipelines" className="icon size-md space-sm" /> <a href="pipelines/webapp_pipeline_page">Pipelines</a> - View and manage your pipelines.
   </li>
+  <li>
+    <img src="/docs/latest/icons/ico-model-endpoints.svg" alt="Model endpoints" className="icon size-md space-sm" /> <a href="pipelines/webapp_pipeline_page">Model Endpoints</a> - Monitor your live model endpoints (available in the ClearML Enterprise plan).
+  </li>
   <li>
     <img src="/docs/latest/icons/ico-reports.svg" alt="Reports" className="icon size-md space-sm" /> <a href="webapp_reports">Reports</a> - View and manage your reports.
   </li>
@@ -42,7 +45,7 @@ The WebApp's sidebar provides access to the following modules:
     <img src="/docs/latest/icons/ico-workers.svg" alt="Workers and Queues" className="icon size-md space-sm" /> <a href="webapp_workers_queues">Orchestration</a> - Autoscale, monitor, and manage your resource usage and workers queues.
   </li>
   <li>
-    <img src="/docs/latest/icons/ico-applications.svg" alt="ClearML Apps" className="icon size-md space-sm" /> <a href="applications/apps_overview">Applications</a> - ClearML's GUI applications for no-code workflow execution.
+    <img src="/docs/latest/icons/ico-applications.svg" alt="ClearML Apps" className="icon size-md space-sm" /> <a href="applications/apps_overview">Applications</a> - ClearML's GUI applications for no-code workflow execution (available in the ClearML Pro and Enterprise plans).
   </li>
 </ul>
 

docs/webapp/webapp_projects_page.md

@@ -21,12 +21,23 @@ page when relevant).
 
 Project folders display summarized project information:  
 
+<div class="max-w-50">
+
 ![Project card](../img/webapp_project_card.png)
 
-When relevant, the folder has a tab which displays the number of subprojects in the project. Click the tab to view a list of 
-subprojects. Click on a subproject's name to navigate to it.  
+</div>
 
-![Subproject tab](../img/webapp_sub_project_card.png)
+* When the project contains subprojects, the folder shows the number of subprojects within the project as an additional 
+  tab. Click the tab to view a list of subprojects, and click on a subproject's name to navigate to it.
+
+  <div class="max-w-50">
+  
+  ![Subproject tab](../img/webapp_sub_project_card.png)
+
+  </div>
+  
+* When a project's `default_output_destination` is set, the folder displays the <img src="/docs/latest/icons/ico-info.svg" alt="Info" className="icon size-md space-sm" /> 
+  indicator. Hover over the indicator to view the default output destination.
 
 <Collapsible title="Hidden Projects" type="configuration">
 
@@ -47,7 +58,7 @@ of a project folder to open its context menu and access the following project ac
 
 ![Project context menu](../img/webapp_projects_context_menu.png)
 
-* **Rename** - Rename the project.
+* **Edit** - Rename the project and/or modify its default output destination.
 * **New Project** - Create a new project (by default a subproject). 
 * **Move to** - Move the project into another project. If the target project does not exist, it is created on-the-fly.
 * **Delete** - Delete the project. To delete a project, all of its contents (i.e. any pipelines/reports/datasets) must
@@ -64,4 +75,11 @@ action. For more information see [Custom UI Context Menu Actions](../deploying_c
 To create a project, click the **+ NEW PROJECT** button in the top right of the page or in a project's context menu, 
 which will open a **New Project** modal. 
 
-![New project modal](../img/webapp_projects_new_project.png)
+![New project modal](../img/webapp_projects_new_project.png)
+
+* Project name
+* Create in - Where the project should be created, either as a top-level project (create in `Projects root`) or as a 
+  subproject of another project
+* Description
+* Default output destination - The storage location where model checkpoints (snapshots) and artifacts will be uploaded 
+  when tasks in this project that do not have an explicit output destination specified are executed

sidebars.js

@@ -138,6 +138,7 @@ module.exports = {
                     'webapp/pipelines/webapp_pipeline_page', 'webapp/pipelines/webapp_pipeline_table', 'webapp/pipelines/webapp_pipeline_viewing'
                 ]
             },
+            'webapp/webapp_model_endpoints',
             'webapp/webapp_reports',
             {
                 'Orchestration': [
@@ -185,6 +186,7 @@ module.exports = {
                     'webapp/settings/webapp_settings_admin_vaults', 'webapp/settings/webapp_settings_users',
                     'webapp/settings/webapp_settings_access_rules', 'webapp/settings/webapp_settings_id_providers',
                     'webapp/settings/webapp_settings_resource_configs', 'webapp/settings/webapp_settings_usage_billing',
+                    'webapp/settings/webapp_settings_storage_credentials'
                  ]
             },
             ]
@@ -296,8 +298,8 @@ module.exports = {
             {
                 'Enterprise':
                         [
-                           'release_notes/clearml_server/enterprise/ver_3_22', 'release_notes/clearml_server/enterprise/ver_3_21',
-                           'release_notes/clearml_server/enterprise/ver_3_20',
+                           'release_notes/clearml_server/enterprise/ver_3_23', 'release_notes/clearml_server/enterprise/ver_3_22',
+                           'release_notes/clearml_server/enterprise/ver_3_21', 'release_notes/clearml_server/enterprise/ver_3_20',
                         ]
             }
         ]},

static/icons/ico-equal-outline.svg

@@ -0,0 +1,3 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24">
+<path d="m8,9h8v2h-8v-2Zm0,6h8v-2h-8v2Zm14-3c0,5.52-4.48,10-10,10S2,17.52,2,12,6.48,2,12,2s10,4.48,10,10Zm-2,0c0-4.41-3.59-8-8-8S4,7.59,4,12s3.59,8,8,8,8-3.59,8-8Z" fill="#8f9dc9"/>
+</svg>

static/icons/ico-model-endpoints.svg

@@ -0,0 +1,4 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24.001" viewBox="0 0 24 24.001" >
+    <path data-name="al-ico-model-endpoints" d="m15 19 3 3h-2v1a1 1 0 0 1-2 0v-1h-2zM11.933 7.85a1.705 1.705 0 0 0 0 3.409 1.705 1.705 0 0 0 0-3.409M10 21v-1a1 1 0 0 0-2 0v1H6l3 3 3-3zm13.992-9.878A6.349 6.349 0 0 1 17.737 17H15.97l-.97-.942-.979.942H10a1 1 0 0 0-2 0H4.385a4.5 4.5 0 0 1-4.367-4.011 4.2 4.2 0 0 1 2.5-4.234 4.153 4.153 0 0 1 5.128-5.008A5.989 5.989 0 0 1 19.09 4.91a5.993 5.993 0 0 1 4.9 6.212M17 8.885l-1.555-.556-.223-.555.667-1.445-.778-.777-1.444.666L13.111 6l-.556-1.56h-1.11L10.889 6l-.556.222-1.444-.67-.778.777.667 1.445-.222.555L7 8.885V10l1.555.556.223.555-.667 1.445.778.778 1.445-.667.555.222.556 1.555h1.11l.556-1.555c.222-.111.334-.111.556-.222l1.444.666.778-.777-.667-1.445.223-.555L17 10z" style="fill:#8f9dc9"/>
+    <path data-name="al-ico-model-endpoints" transform="translate(0 .001)" style="fill:none" d="M0 0h24v24H0z"/>
+</svg>