mirror of
https://github.com/clearml/clearml-docs
synced 2025-01-31 22:48:40 +00:00
eb91aaa361
* add info for clear filter, float behavior, and tag exclusion * add data examples to guides * fix typo * fix filtering wording, and float admonition title * add abort all children action * dataset metadata initial * fix expand icon * initial new exp comparison window * initial scalar full screen * fix typo * edit dataset version metada * edit exp comparison and add missing alts to icons * add info about dataset-level datasets * add info about full screen mode * HOCON >>> JSON * HOCON >>> JSON * dataset card edits * edit fullscreen scalar mode * edit fullscreen scalar plots * edit pipeline example based on code fixes * full screen scalar edit * add context navigation * Add experiment selection info * experiment selection edit * custom ui plugin * datasets and versioning * fix link
83 lines
3.5 KiB
Markdown
83 lines
3.5 KiB
Markdown
---
|
|
title: Workers and Queues
|
|
---
|
|
|
|
With the **Workers and Queues** page, users can:
|
|
|
|
* Monitor resources (CPU and GPU, memory, video memory, and network usage) used by the experiments / Tasks that workers
|
|
execute
|
|
* View workers and the queues they listen to
|
|
* Create and rename queues; delete empty queues; monitor queue utilization
|
|
* Reorder, move, and remove experiments from queues
|
|
|
|
|
|
|
|
## Resources Utilization
|
|
|
|
**To monitor resource utilization:**
|
|
|
|
1. In the **WORKERS** tab, click a worker. The chart refreshes showing resource utilization over time for that worker. The
|
|
worker **INFO** slides open, showing information about the worker:
|
|
* Name
|
|
* Current experiment
|
|
* Current runtime
|
|
* Last iteration
|
|
* Last update time.
|
|
|
|
1. Select a metric and time frame:
|
|
|
|
1. In the list of resources (top left side), select **CPU and GPU Usage**, **Memory Usage**, **Video Memory Usage**, or **Network Usage**.
|
|
|
|
1. In the period list (top right side), select **3 Hours**, **6 Hours**, **12 Hours**, **1 Day**, **1 Week**, or **1 Month**.
|
|
|
|
|
|
|
|
|
|
|
|
## Worker Utilization
|
|
|
|
Optimize worker use by monitoring worker utilization in the **Workers** tab.
|
|
|
|
**To monitor worker utilization:**
|
|
|
|
* Open the **Workers** tab in the **Workers & Queues** page. The worker utilization chart
|
|
appears. Hover over any data point and see average workers and total workers.
|
|
|
|
|
|
|
|
## Queue Utilization
|
|
|
|
**To monitor all queues:**
|
|
|
|
* Open the **Queues** tab in the **Workers & Queues** page. The queue utilization chart appears and shows
|
|
average wait time (seconds) and number of experiments queued for all queues.
|
|
* Hover over any data point and see average wait time and number of experiments.
|
|
|
|
|
|
|
|
**To monitor a queue:**
|
|
|
|
1. In the queues list (below the plot on the left), click a queue.
|
|
1. The chart refreshes, showing metrics for the selected queue. The info panel slides open with two tabs:
|
|
1. To see the enqueued experiments on the queue, click the **EXPERIMENTS** tab.
|
|
2. To view information about the workers listening to the queue, click the **WORKERS** tab.
|
|
|
|
|
|
|
|
## Queue Management
|
|
|
|
In the **Queues** tab, do any of the following:
|
|
|
|
* Create a queue - Click **+ NEW QUEUE** (top right) **>** Type a queue name **>** **CREATE**.
|
|
* Do any of the following by right-clicking a queue in the queues list (lower right):
|
|
* Delete a queue - Click **Delete**.
|
|
* Rename a queue - Click **RENAME** > Type a queue name **>** **RENAME**, or click **DELETE**.
|
|
* Custom action - The ClearML Enterprise Server provides a mechanism to define your own custom actions, which will
|
|
appear in the context menu. See [Custom UI Context Menu Actions](../deploying_clearml/clearml_server_config.md#custom-ui-context-menu-actions).
|
|
* Do any of the following by right-clicking an experiment in a queue's **EXPERIMENTS** tab (lower right):
|
|
* Reorder experiments in a queue - Drag an experiment to a new position in the queue, or click <img src="/docs/latest/icons/ico-dots-v-menu.svg" alt="Menu button" className="icon size-md space-sm" />
|
|
(menu) and then select **Move to top** or **Move to bottom**.
|
|
* Move to a new queue - Click <img src="/docs/latest/icons/ico-dots-v-menu.svg" alt="Menu button" className="icon size-md space-sm" /> (menu) **>** **Move to queue...** **>** Select a queue **>** **ENQUEUE**.
|
|
* Remove an experiment - Click <img src="/docs/latest/icons/ico-dots-v-menu.svg" alt="Menu button" className="icon size-md space-sm" /> (menu) **>** **Move to queue...** **>** Select a queue **>** **ENQUEUE**.
|
|
|