2021-05-13 23:48:51 +00:00
|
|
|
|
---
|
|
|
|
|
title: Workers and Queues
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
With the **Workers and Queues** page, users can:
|
|
|
|
|
|
2021-05-23 20:17:12 +00:00
|
|
|
|
* Monitor resources (CPU and GPU, memory, video memory, and network usage) used by the experiments / Tasks that workers
|
2021-05-13 23:48:51 +00:00
|
|
|
|
execute
|
|
|
|
|
* View workers and the queues they listen to
|
|
|
|
|
* Create and rename queues; delete empty queues; monitor queue utilization
|
|
|
|
|
* Reorder, move, and remove experiments from queues
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-09-02 07:48:37 +00:00
|
|
|
|
## Resources Utilization
|
2021-05-13 23:48:51 +00:00
|
|
|
|
|
|
|
|
|
**To monitor resource utilization:**
|
|
|
|
|
|
2021-05-23 20:17:12 +00:00
|
|
|
|
1. In the **WORKERS** tab, click a worker. The chart refreshes showing resource utilization over time for that worker. The
|
|
|
|
|
worker **INFO** slides open, showing information about the worker:
|
|
|
|
|
* Name
|
2021-05-13 23:48:51 +00:00
|
|
|
|
* Current experiment
|
|
|
|
|
* Current runtime
|
2021-05-23 20:17:12 +00:00
|
|
|
|
* Last iteration
|
2021-05-13 23:48:51 +00:00
|
|
|
|
* Last update time.
|
|
|
|
|
|
|
|
|
|
1. Select a metric and time frame:
|
|
|
|
|
|
|
|
|
|
1. In the list of resources (top left side), select **CPU and GPU Usage**, **Memory Usage**, **Video Memory Usage**, or **Network Usage**.
|
2021-05-23 20:17:12 +00:00
|
|
|
|
|
2021-05-13 23:48:51 +00:00
|
|
|
|
1. In the period list (top right side), select **3 Hours**, **6 Hours**, **12 Hours**, **1 Day**, **1 Week**, or **1 Month**.
|
2021-05-23 20:17:12 +00:00
|
|
|
|
|
2022-02-13 08:29:12 +00:00
|
|
|
|
![Worker management](../img/agents_queues_resource_management.png)
|
2021-05-13 23:48:51 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-09-02 07:48:37 +00:00
|
|
|
|
## Worker Utilization
|
2021-05-13 23:48:51 +00:00
|
|
|
|
|
|
|
|
|
Optimize worker use by monitoring worker utilization in the **Workers** tab.
|
|
|
|
|
|
|
|
|
|
**To monitor worker utilization:**
|
|
|
|
|
|
2021-05-23 20:17:12 +00:00
|
|
|
|
* Open the **Workers** tab in the **Workers & Queues** page. The worker utilization chart
|
2021-05-13 23:48:51 +00:00
|
|
|
|
appears. Hover over any data point and see average workers and total workers.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-09-02 07:48:37 +00:00
|
|
|
|
## Queue Utilization
|
2021-05-13 23:48:51 +00:00
|
|
|
|
|
|
|
|
|
**To monitor all queues:**
|
|
|
|
|
|
2021-05-23 20:17:12 +00:00
|
|
|
|
* Open the **Queues** tab in the **Workers & Queues** page. The queue utilization chart appears and shows
|
|
|
|
|
average wait time (seconds) and number of experiments queued for all queues.
|
2021-05-13 23:48:51 +00:00
|
|
|
|
* Hover over any data point and see average wait time and number of experiments.
|
|
|
|
|
|
|
|
|
|
![image](../img/4100.png)
|
|
|
|
|
|
|
|
|
|
**To monitor a queue:**
|
|
|
|
|
|
2021-05-23 20:17:12 +00:00
|
|
|
|
1. In the queues list (below the plot on the left), click a queue.
|
2021-05-13 23:48:51 +00:00
|
|
|
|
1. The chart refreshes, showing metrics for the selected queue. The info panel slides open with two tabs:
|
|
|
|
|
1. To see the enqueued experiments on the queue, click the **EXPERIMENTS** tab.
|
|
|
|
|
2. To view information about the workers listening to the queue, click the **WORKERS** tab.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2021-09-02 07:48:37 +00:00
|
|
|
|
## Queue Management
|
2021-05-13 23:48:51 +00:00
|
|
|
|
|
2022-03-06 11:00:46 +00:00
|
|
|
|
The **Queues** tab lets you manage your queues and their contents:
|
|
|
|
|
|
|
|
|
|
* Create a queue - Click **+ NEW QUEUE** (top right)
|
|
|
|
|
* Right-click on a queue in the queues list to modify the queue:
|
|
|
|
|
|
|
|
|
|
![Queue context menu](../img/webapp_workers_queues_context.png)
|
|
|
|
|
|
|
|
|
|
* Delete - Delete the queue. Any pending tasks will be dequeued.
|
|
|
|
|
* Rename - Change the queue’s name
|
|
|
|
|
* Clear - Remove all pending tasks from the queue
|
2022-01-10 09:45:29 +00:00
|
|
|
|
* Custom action - The ClearML Enterprise Server provides a mechanism to define your own custom actions, which will
|
2022-03-06 11:00:46 +00:00
|
|
|
|
appear in the context menu. See [Custom UI Context Menu Actions](../deploying_clearml/clearml_server_config.md#custom-ui-context-menu-actions)
|
|
|
|
|
* Click on a queue to select it and see its task and worker information. Click on a
|
|
|
|
|
task’s menu button <img src="/docs/latest/icons/ico-dots-v-menu.svg" alt="Menu button" className="icon size-md space-sm" />
|
|
|
|
|
in the **EXPERIMENTS** tab to reorganize your queue.
|
|
|
|
|
|
|
|
|
|
![Queue experiment's menu](../img/workers_queues_experiment_actions.png)
|
|
|
|
|
|
|
|
|
|
* Move a task to the top or bottom of the queue
|
|
|
|
|
* Move the task to a different queue
|
|
|
|
|
* Dequeue the task
|
|
|
|
|
|
|
|
|
|
You can also reorder experiments in a queue by dragging an experiment to a new position in the queue
|