Add metric encoding information (#578)

This commit is contained in:
pollfly 2023-05-31 12:37:20 +03:00 committed by GitHub
parent 8d46338267
commit 7cd3a46930
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 31 additions and 12 deletions

View File

@ -261,10 +261,33 @@ task_filter={
# only training type tasks # only training type tasks
'type': ['training'], 'type': ['training'],
# match text in task comment or task name # match text in task comment or task name
'search_text': 'reg_exp_text' 'search_text': 'reg_exp_text',
# order return task lists by their update time in ascending order
'order_by': ['last_update']
} }
``` ```
:::tip Order tasks by metrics
You can order the returned tasks by performance in a specific metric with `'order_by': [last_metrics.<md5-encoded-metric-title>.<md5-encoded-metric-variant>.<value_type>]`.
* `<md5-encoded-metric-title>` and `<md5-encoded-metric-variant>` - MD5 encoded metric and variant names. In Python, you
can encode the strings with `hashlib.md5(str("<metric_name_string>").encode("utf-8")).hexdigest()`
* `<value_type>` - Specify which metric values to use. The options are: `value` (last value), `min_value`, or `max_value`
Use the `-` prefix to order the results in descending order.
```python
title = hashlib.md5(str("testing").encode("utf-8")).hexdigest()
series = hashlib.md5(str("epoch_accuracy").encode("utf-8")).hexdigest()
tasks = Task.get_tasks(
project_name='Example Project',
# order tasks by metric performance in descending order
task_filter={'order_by': [f'-last_metrics.{title}.{series}.max_value']}
)
```
:::
See [`Task.get_tasks`](../references/sdk/task.md#taskget_tasks) for all `task_filter` options. See [`Task.get_tasks`](../references/sdk/task.md#taskget_tasks) for all `task_filter` options.
### Tag Filters ### Tag Filters

View File

@ -94,7 +94,7 @@ resources will be displayed. See [Dynamic Queries](#dynamic-queries) below.
* `single` (single-scalar values table) * `single` (single-scalar values table)
* `sample` (debug sample) * `sample` (debug sample)
* `parcoords` (hyperparameter comparison plots) - for this option, you need to also specify the following parameters: * `parcoords` (hyperparameter comparison plots) - for this option, you need to also specify the following parameters:
* `metrics` - Unique metric/variant ID formatted like `metric_id.variant_id` (find with your browser's inspect. See note [below](#event_id)) * `metrics` - Unique metric/variant ID formatted like `metric_id.variant_id` (see note [below](#event_id))
* `variants` - Parameters to include in the plot (write in following format `<section_name>.<parameter_1>&<section_name>.<parameter_2>`) * `variants` - Parameters to include in the plot (write in following format `<section_name>.<parameter_1>&<section_name>.<parameter_2>`)
* `value_type` - Specify which metric values to use. The options are: * `value_type` - Specify which metric values to use. The options are:
* `min_value` * `min_value`
@ -144,22 +144,18 @@ used.
``` ```
A list of tags that the experiment should contain is specified in the `tags` argument. You can also specify tags that A list of tags that the experiment should contain is specified in the `tags` argument. You can also specify tags that
exclude experiments. See tag filter syntax examples [here](../clearml_sdk/task_sdk.md#tag-filters). exclude experiments. See tag filter syntax examples [here](../clearml_sdk/task_sdk.md#tag-filters).
* Request the `training/accuracy` scalar plot of the 5 experiments with the best accuracy scores * Request the `training/accuracy` scalar plot of the 5 experiments with the best accuracy scores (see Metric/Variant IDs note [below](#event_id)):
``` ```
src="<web_server>?objectType=task&type=scalar&metrics=training&variants=accuracy&project=4043a1657f374e9298649c6ba72ad233&page_size=5&page=0&order_by[]=-last_metrics.<metric_event_id>.<variant_event_id>.value" src="<web_server>/widgets/?objectType=task&type=scalar&metrics=training&variants=accuracy&project=4043a1657f374e9298649c6ba72ad233&page_size=5&page=0&order_by[]=-last_metrics.<metric_id>.<variant_id>.value"
``` ```
<a id="event_id"></a> <a id="event_id"></a>
:::tip Event IDs :::tip Metric/Variant IDs
The `tasks.get_all` and `models.get_all` API calls' parameters sometimes need event IDs, instead of names. To find event IDs: Metric names need to be MD5 encoded for parallel coordinate plots and for ordering query results by metric
1. Go to the relevant Experiments/Model table > Open the **Developer Tools** window (inspect) > click **Network**. performance. You can encode the strings in Python with `hashlib.md5(str("<metric_string>").encode("utf-8")).hexdigest()`,
1. Execute the action you want the embed code to do (e.g. sort by update time, sort by accuracy). and use the returned MD5 hash in your query.
1. Click on the API call `tasks.get_all_ex`/`models.get_all_ex` that appears in the **Network** tab.
1. Click on the **Payload** panel.
1. Click on the relevant parameter to see the relevant event's ID. For example, if you sorted by experiment accuracy,
you will see the metric's event ID under the `order_by` parameter.
::: :::