diff --git a/docs/hyperdatasets/webapp/webapp_datasets_versioning.md b/docs/hyperdatasets/webapp/webapp_datasets_versioning.md index 58dd876d..737f2f19 100644 --- a/docs/hyperdatasets/webapp/webapp_datasets_versioning.md +++ b/docs/hyperdatasets/webapp/webapp_datasets_versioning.md @@ -5,8 +5,6 @@ title: Dataset Versions Use the Dataset versioning WebApp (UI) features for viewing, creating, modifying, and deleting [Dataset versions](../dataset.md#dataset-versioning). -From the [Datasets page](webapp_datasets.md), click on one of the Datasets in order to see and work with its versions. - ## Dataset Version History The WebApp (UI) will present your dataset version structure in one of two ways, depending on the structure implemented. diff --git a/docs/img/webapp_module_icons.png b/docs/img/webapp_module_icons.png index 617b0ebb..26896dbf 100644 Binary files a/docs/img/webapp_module_icons.png and b/docs/img/webapp_module_icons.png differ diff --git a/docs/img/webapp_pipeline_DAG.png b/docs/img/webapp_pipeline_DAG.png new file mode 100644 index 00000000..3d21db07 Binary files /dev/null and b/docs/img/webapp_pipeline_DAG.png differ diff --git a/docs/img/webapp_pipeline_card.png b/docs/img/webapp_pipeline_card.png new file mode 100644 index 00000000..9e5a2734 Binary files /dev/null and b/docs/img/webapp_pipeline_card.png differ diff --git a/docs/img/webapp_pipeline_context_menu.png b/docs/img/webapp_pipeline_context_menu.png new file mode 100644 index 00000000..bcf7b2af Binary files /dev/null and b/docs/img/webapp_pipeline_context_menu.png differ diff --git a/docs/img/webapp_pipeline_new_run.png b/docs/img/webapp_pipeline_new_run.png new file mode 100644 index 00000000..c9514e88 Binary files /dev/null and b/docs/img/webapp_pipeline_new_run.png differ diff --git a/docs/img/webapp_pipeline_node.png b/docs/img/webapp_pipeline_node.png new file mode 100644 index 00000000..5f8bc346 Binary files /dev/null and b/docs/img/webapp_pipeline_node.png differ diff --git a/docs/img/webapp_pipeline_run_info.png b/docs/img/webapp_pipeline_run_info.png new file mode 100644 index 00000000..3f0e811d Binary files /dev/null and b/docs/img/webapp_pipeline_run_info.png differ diff --git a/docs/img/webapp_pipeline_runs_table.png b/docs/img/webapp_pipeline_runs_table.png new file mode 100644 index 00000000..696b1a49 Binary files /dev/null and b/docs/img/webapp_pipeline_runs_table.png differ diff --git a/docs/img/webapp_pipeline_step_code.png b/docs/img/webapp_pipeline_step_code.png new file mode 100644 index 00000000..9781f8c8 Binary files /dev/null and b/docs/img/webapp_pipeline_step_code.png differ diff --git a/docs/img/webapp_pipeline_step_console.png b/docs/img/webapp_pipeline_step_console.png new file mode 100644 index 00000000..f66f52ac Binary files /dev/null and b/docs/img/webapp_pipeline_step_console.png differ diff --git a/docs/img/webapp_pipeline_step_info.png b/docs/img/webapp_pipeline_step_info.png new file mode 100644 index 00000000..ba7b0290 Binary files /dev/null and b/docs/img/webapp_pipeline_step_info.png differ diff --git a/docs/img/webapp_pipeline_table.png b/docs/img/webapp_pipeline_table.png new file mode 100644 index 00000000..5f596b16 Binary files /dev/null and b/docs/img/webapp_pipeline_table.png differ diff --git a/docs/img/webapp_pipeline_task_info.png b/docs/img/webapp_pipeline_task_info.png new file mode 100644 index 00000000..9ee52fad Binary files /dev/null and b/docs/img/webapp_pipeline_task_info.png differ diff --git a/docs/img/webapp_pipelines_context_menu.png b/docs/img/webapp_pipelines_context_menu.png new file mode 100644 index 00000000..9e49e482 Binary files /dev/null and b/docs/img/webapp_pipelines_context_menu.png differ diff --git a/docs/webapp/pipelines/webapp_pipeline_page.md b/docs/webapp/pipelines/webapp_pipeline_page.md new file mode 100644 index 00000000..e242b48f --- /dev/null +++ b/docs/webapp/pipelines/webapp_pipeline_page.md @@ -0,0 +1,34 @@ +--- +title: Pipelines Page +--- + +Use the **Pipelines** Page to navigate between and manage pipelines. The page shows execution summaries for all +[ClearML Pipelines](../../fundamentals/pipelines.md). + +Click on a pipeline card to navigate to its [Pipeline Runs Table](webapp_pipeline_table.md), where you can view the +pipeline structure, configuration and outputs for all of the pipeline’s runs, as well as create new runs. + +![Pipelines page](../../img/webapp_pipeline_table.png) + + +## Pipeline Cards + +The pipeline cards display summarized pipeline information: + +![Project card](../../img/webapp_pipeline_card.png) + +* Pipeline name +* Time since the pipeline’s most recent run +* Run summary - Number of *Running*/*Pending*/*Completed*/*Failed* runs +* Tags + + +### Pipeline Actions + +Click Menu on the top right +of a pipeline card to open its context menu and access pipeline actions. + +![Project context menu](../../img/webapp_pipeline_context_menu.png) + +* Rename - Change the pipeline’s name +* Add Tag - Add label to the pipeline to help easily classify groups of pipelines. diff --git a/docs/webapp/pipelines/webapp_pipeline_table.md b/docs/webapp/pipelines/webapp_pipeline_table.md new file mode 100644 index 00000000..bd463b97 --- /dev/null +++ b/docs/webapp/pipelines/webapp_pipeline_table.md @@ -0,0 +1,123 @@ +--- +title: The Pipeline Runs Table +--- + +The pipeline runs table is a [customizable](#customizing-the-runs-table) list of the pipeline’s runs. From the table, +you can view a run’s details, and manage runs (create, continue, or abort). The runs table's auto-refresh allows users +to continually monitor run progress. + +![Pipeline runs table](../../img/webapp_pipeline_runs_table.png) + +## Run Table Columns + +The models table contains the following columns: + +| Column | Description | Type | +|---|---|---| +| **RUN** | Pipeline run identifier | String | +| **VERSION** | The pipeline version number. Corresponds to the [PipelineController](../../references/sdk/automation_controller_pipelinecontroller.md#class-pipelinecontroller) ’s and [PipelineDecorator](../../references/sdk/automation_controller_pipelinecontroller.md#class-automationcontrollerpipelinedecorator)’s `version` parameter | Version string | +| **TAGS** | Descriptive, user-defined, color-coded tags assigned to run. | Tag | +| **STATUS** | Pipeline run's status. See a list of the [task states and state transitions](../../fundamentals/task.md#task-states). | String | +| **USER** | User who created the run. | String | +| **STARTED** | Elapsed time since the run started. To view the date and time of start, hover over the elapsed time. | Date-time | +| **UPDATED** | Elapsed time since the last update to the run. To view the date and time of update, hover over the elapsed time. | Date-time | +| **RUN TIME** | The current / total running time of the run. | Time | +| **_Metrics_** | Add metrics column (last, minimum, and / or maximum values). Available options depend upon the runs in the table. | Varies according to runs in table | +| **_Hyperparameters_** | Add hyperparameters. Available options depend upon the runs in the table. | Varies according to runs in table | + +## Customizing the Runs Table + +Customize the table using any of the following: +* Dynamic column ordering - Drag a column title to a different position. +* Show / hide columns - Click Setting Gear + to view and select columns to show. Click **Metric** and **Hyper Parameter** to add the respective custom columns +* [Filter columns](#filtering-columns) +* Sort columns +* Resize columns - Drag the column separator to change the width of that column. Double click the column separator for + automatic fit. + +Changes are persistent (cached in the browser) and represented in the URL, so customized settings can be saved in a +browser bookmark and shared with other ClearML users. + +:::note Float Values Display +By default, the runs table displays rounded up float values. Hover over a float to view its precise value in the +tooltip that appears. To view all precise values in a column, hover over a float and click Expand. +::: + +### Filtering Columns + +Filters can be applied by clicking Filter +on a column, and the relevant filter appears. + +There are a few types of filters: +* Value set - Choose which values to include from a list of all values in the column +* Numerical ranges - Insert minimum and / or maximum value +* Date ranges - Insert starting and / or ending date and time +* Tags - Choose which tags to filter by from a list of all tags used in the column. + * Filter by multiple tag values using the **ANY** or **ALL** options, which correspond to the logical "AND" and "OR" respectively. These + options appear on the top of the tag list. + * Filter by the absence of a tag (logical "NOT") by clicking its checkbox twice. An `X` will appear in the tag's checkbox. + +Once a filter is applied to a column, its filter icon will appear with a highlighted dot on its top right (Filter on ). + +To clear all active filters, click Clear filters +in the top right corner of the table. + +:::note +The following table customizations are saved on a per pipeline basis: +* Columns order +* Column width +* Active sort order +* Active filters +* Custom columns +::: + +## Create Run +To launch a new run for a pipeline, click **+ NEW RUN** on the top left of the page. This opens a **NEW RUN** modal, where +you can set the run’s parameters. By default, the fields are pre-filled with the last run’s values. + +Click **Advanced configurations** to change the run’s execution queue. + +![New run modal](../../img/webapp_pipeline_new_run.png) + +After clicking **RUN**, the new pipeline run is enqueued in the specified queue, and the run is added to the pipeline run table. + + +## Run Actions + +The following table describes the actions that can be done from the run table, including the [states](../../fundamentals/task.md#task-states) +that allow each operation. + +| Action | Description | States Valid for the Action | State Transition | +|---|---|---|---| +| Archive | To more easily work with active runs, archive a run. See [Archiving](../webapp_archiving.md). | Any state | None | +| Restore | Action available in the archive. Restore a run to the active pipeline runs table. | Any state | None | +| Run | Create a new pipeline run. Configure and enqueue it for execution. See [Create Run](#create-run). | Any State | *Pending* | +| Abort | Manually stop / cancel a run. | *Running* / *Pending* | *Aborted* | +| Continue | Rerun with the same parameters | *Aborted* | *Pending* | +| Tags | User-defined labels added to runs for grouping and organization. | Any state | None | + +These actions can be accessed with the context menu, when right-clicking a run. + +![pipeline run context menu](../../img/webapp_pipelines_context_menu.png) + +Most of the actions mentioned in the chart above can be performed on multiple runs at once. +[Select multiple runs](#selecting-multiple-runs), then use either the context menu, or the bar that appears at the bottom +of the page, to perform operations on the selected runs. Actions can be performed only on the runs that match the action criteria +(for example, only *Aborted* pipeline runs can be continued). The context menu shows the number of runs that can be +affected by each action. The same information can be found in the bottom menu, in a tooltip that appears when hovering +over an action icon. + +## Selecting Multiple Runs + +Select multiple runs by clicking the checkbox on the left of each relevant run. Clear any existing selection +by clicking the checkbox in the top left corner of the table. + +Click the checkbox in the top left corner of the table to select all items currently visible. + +An extended bulk selection tool is available through the down arrow next to the checkbox in the top left corner, enabling +selecting items beyond the items currently on-screen: +* **All** - Select all runs in the pipeline +* **None** - Clear selection +* **Filtered** - Select all runs in the project that match the current active table filters + diff --git a/docs/webapp/pipelines/webapp_pipeline_viewing.md b/docs/webapp/pipelines/webapp_pipeline_viewing.md new file mode 100644 index 00000000..971138f7 --- /dev/null +++ b/docs/webapp/pipelines/webapp_pipeline_viewing.md @@ -0,0 +1,60 @@ +--- +title: Pipeline Run Details +--- + +The run details panel shows the pipeline’s structure and the execution status of every step, as well as the run’s +configuration parameters and output. + +![Pipeline structure](../../img/webapp_pipeline_DAG.png) + +Each step shows: + +![Pipeline step info](../../img/webapp_pipeline_node.png) + +* Step name +* Step status +* Step execution time +* Step log button - Hover over the step and click console + to view the step’s [console log and code](#run-and-step-log) + +While the pipeline is running, the steps’ details and colors are updated. + +## Run and Step Details +### Run and Step Info + +On the right side of the pipeline run panel, view the **RUN INFO** which shows: +* Run Parameters +* Reported Metrics +* Produced Artifacts +* Output Models + +![Run info](../../img/webapp_pipeline_run_info.png) + +To view a run’s complete information, click **Full details**, which will open the pipeline’s controller [task page](../webapp_exp_track_visual.md). +View each list’s complete details in the pipeline task’s corresponding tabs: +* **PARAMETERS** list > **CONFIGURATION** tab +* **METRICS** list > **SCALARS** tab +* **ARTIFACTS** and **MODELS** lists > **ARTIFACTS** tab + +![Pipeline task info](../../img/webapp_pipeline_task_info.png) + +To view a specific step’s information, click the step on the execution graph, and the info panel displays its **STEP INFO**. +The panel displays the step’s name, task type, and status, as well as its parameters, metrics, artifacts, and models. + +![Step info](../../img/webapp_pipeline_step_info.png) + +To return to viewing the run’s information, click the pipeline graph, outside any of the steps. + +### Run and Step Log + +Click on **DETAILS** on the top left of the info panel to view a run’s full console log. The log contains everything printed +to stdout and stderr. + +![Step console](../../img/webapp_pipeline_step_console.png) + +To view a step’s console log, click **DETAILS** and then click on a step. + +You can also view the selected step’s code. On the top right of the console panel, click **Console**, then click **Code** +in the menu. + +![Step code](../../img/webapp_pipeline_step_code.png) \ No newline at end of file diff --git a/docs/webapp/webapp_overview.md b/docs/webapp/webapp_overview.md index 71eb9462..2f58270e 100644 --- a/docs/webapp/webapp_overview.md +++ b/docs/webapp/webapp_overview.md @@ -23,7 +23,8 @@ The ClearML Web UI is composed of the following pages: * [Compare experiments](webapp_exp_comparing.md) * [Share experiments and their models](webapp_exp_sharing.md) with other ClearML hosted service users * [Archive experiments and models](webapp_archiving.md) - + +* [Pipelines](pipelines/webapp_pipeline_page.md) - View and manage your pipelines. * [Workers and Queues](webapp_workers_queues.md) - The resource monitoring and queues management page. * [Settings](webapp_profile.md) (available through the profile menu Profile button) - diff --git a/sidebars.js b/sidebars.js index b43f46ab..b58a5bc7 100644 --- a/sidebars.js +++ b/sidebars.js @@ -37,6 +37,11 @@ module.exports = { 'webapp/webapp_archiving' ] }, + { + 'Pipelines':[ + 'webapp/pipelines/webapp_pipeline_page', 'webapp/pipelines/webapp_pipeline_table', 'webapp/pipelines/webapp_pipeline_viewing' + ] + }, 'webapp/webapp_workers_queues', 'webapp/webapp_profile'] }, {'Configurations': ['configs/configuring_clearml', 'configs/clearml_conf', 'configs/env_vars']}, diff --git a/static/icons/ico-console.svg b/static/icons/ico-console.svg new file mode 100644 index 00000000..fac885bf --- /dev/null +++ b/static/icons/ico-console.svg @@ -0,0 +1,4 @@ + + + +