ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-04-20 06:19:37 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Add multi-objective optimization info (#844)

Browse Source
This commit is contained in:
pollfly 2024-05-21 11:30:46 +03:00 committed by GitHub
parent 865e46e538
commit ce58c8e2d4
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 90 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
docs/fundamentals/hpo.md
View File

@ -23,7 +23,7 @@ compare results.
The preceding diagram demonstrates the typical flow of hyperparameter optimization where the parameters of a base task are optimized:
1. Configure an Optimization Task with a base task whose parameters will be optimized, and a set of parameter values to
1. Configure an Optimization Task with a base task whose parameters will be optimized, optimization targets, and a set of parameter values to
test
1. Clone the base task. Each clone's parameter is overridden with a value from the optimization task
1. Enqueue each clone for execution by a ClearML Agent
@ -118,6 +118,19 @@ optimization.
in the task header.
:::
:::tip Multi-objective Optimization
If you are using the Optuna framework (see [Supported Optimizers](#supported-optimizers)), you can list multiple optimization objectives.
When doing so, make sure the `objective_metric_title`, `objective_metric_series`, and `objective_metric_sign` lists
are the same length. Each title will be matched to its respective series and sign.
For example, the code below sets two objectives: to minimize the `validation/loss` metric and to maximize the `validation/accuracy` metric.
```python
objective_metric_title=["validation", "validation"]
objective_metric_series=["loss", "accuracy"]
objective_metric_sign=["min", "max"]
```
:::
## Optimizer Execution Options
The `HyperParameterOptimizer` provides options to launch the optimization tasks locally or through a ClearML [queue](agents_and_queues.md#what-is-a-queue).

126
docs/guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt.md
View File

@ -114,67 +114,85 @@ if not args['template_task_id']:
## Creating the Optimizer Object
Initialize an [`automation.HyperParameterOptimizer`](../../../references/sdk/hpo_optimization_hyperparameteroptimizer.md)
object, setting the optimization parameters, beginning with the ID of the experiment to optimize.
object, setting the following optimization parameters:
```python
an_optimizer = HyperParameterOptimizer(
# This is the experiment we want to optimize
base_task_id=args['template_task_id'],
```
* ID of a ClearML task to optimize. This task will be cloned, and each clone will sample a different set of hyperparameters values:
Set the hyperparameter ranges to sample, instantiating them as ClearML automation objects using [`automation.UniformIntegerParameterRange`](../../../references/sdk/hpo_parameters_uniformintegerparameterrange.md)
and [`automation.DiscreteParameterRange`](../../../references/sdk/hpo_parameters_discreteparameterrange.md).
```python
an_optimizer = HyperParameterOptimizer(
# This is the experiment we want to optimize
base_task_id=args['template_task_id'],
```
```python
hyper_parameters=[
UniformIntegerParameterRange('layer_1', min_value=128, max_value=512, step_size=128),
UniformIntegerParameterRange('layer_2', min_value=128, max_value=512, step_size=128),
DiscreteParameterRange('batch_size', values=[96, 128, 160]),
DiscreteParameterRange('epochs', values=[30]),
],
```
* Hyperparameter ranges to sample, instantiating them as ClearML automation objects using [`automation.UniformIntegerParameterRange`](../../../references/sdk/hpo_parameters_uniformintegerparameterrange.md)
and [`automation.DiscreteParameterRange`](../../../references/sdk/hpo_parameters_discreteparameterrange.md):
Set the metric to optimize and the optimization objective.
```python
hyper_parameters=[
UniformIntegerParameterRange('layer_1', min_value=128, max_value=512, step_size=128),
UniformIntegerParameterRange('layer_2', min_value=128, max_value=512, step_size=128),
DiscreteParameterRange('batch_size', values=[96, 128, 160]),
DiscreteParameterRange('epochs', values=[30]),
],
```
```python
objective_metric_title='val_acc',
objective_metric_series='val_acc',
objective_metric_sign='max',
```
* Metric to optimize and the optimization objective:
Set the number of concurrent Tasks.
```python
max_number_of_concurrent_tasks=2,
```
Set the optimization strategy, see [Set the search strategy for optimization](#set-the-search-strategy-for-optimization).
```python
optimizer_class=aSearchStrategy,
```
Specify the queue to use for remote execution. This is overridden if the optimizer runs as a service.
```python
execution_queue='1xGPU',
```
Specify the remaining parameters, including the time limit per Task (minutes), period for checking the optimization (minutes), maximum number of jobs to launch, minimum and maximum number of iterations for each Task.
```python
# Optional: Limit the execution time of a single experiment, in minutes.
# (this is optional, and if using OptimizerBOHB, it is ignored)
time_limit_per_job=10.,
# Check the experiments every 6 seconds is way too often, we should probably set it to 5 min,
# assuming a single experiment is usually hours...
pool_period_min=0.1,
# set the maximum number of jobs to launch for the optimization, default (None) unlimited
# If OptimizerBOHB is used, it defined the maximum budget in terms of full jobs
# basically the cumulative number of iterations will not exceed total_max_jobs * max_iteration_per_job
total_max_jobs=10,
# This is only applicable for OptimizerBOHB and ignore by the rest
# set the minimum number of iterations for an experiment, before early stopping
min_iteration_per_job=10,
# Set the maximum number of iterations for an experiment to execute
# (This is optional, unless using OptimizerBOHB where this is a must)
max_iteration_per_job=30,
```python
objective_metric_title='val_acc',
objective_metric_series='val_acc',
objective_metric_sign='max',
```
) # done creating HyperParameterOptimizer
```
:::tip Multi-objective Optimization
If you are using the Optuna framework (see [Set the Search Strategy for Optimization](#set-the-search-strategy-for-optimization)),
you can list multiple optimization objectives. When doing so, make sure the `objective_metric_title`,
`objective_metric_series`, and `objective_metric_sign` lists are
the same length. Each title will be matched to its respective series and sign.
For example, the code below sets two objectives: to minimize the `validation/loss` metric and to maximize the
`validation/accuracy` metric:
```python
objective_metric_title=["validation", "validation"]
objective_metric_series=["loss", "accuracy"]
objective_metric_sign=["min", "max"]
```
:::
* Number of concurrent Tasks:
```python
max_number_of_concurrent_tasks=2,
```
* Optimization strategy (see [Set the search strategy for optimization](#set-the-search-strategy-for-optimization)):
```python
optimizer_class=aSearchStrategy,
```
* Queue to use for remote execution. This is overridden if the optimizer runs as a service.
```python
execution_queue='1xGPU',
```
* Remaining parameters, including the time limit per Task (minutes), period for checking the optimization (minutes),
maximum number of jobs to launch, minimum and maximum number of iterations for each Task:
```python
# Optional: Limit the execution time of a single experiment, in minutes.
# (this is optional, and if using OptimizerBOHB, it is ignored)
time_limit_per_job=10.,
# Check the experiments every 6 seconds is way too often, we should probably set it to 5 min,
# assuming a single experiment is usually hours...
pool_period_min=0.1,
# set the maximum number of jobs to launch for the optimization, default (None) unlimited
# If OptimizerBOHB is used, it defined the maximum budget in terms of full jobs
# basically the cumulative number of iterations will not exceed total_max_jobs * max_iteration_per_job
total_max_jobs=10,
# This is only applicable for OptimizerBOHB and ignore by the rest
# set the minimum number of iterations for an experiment, before early stopping
min_iteration_per_job=10,
# Set the maximum number of iterations for an experiment to execute
# (This is optional, unless using OptimizerBOHB where this is a must)
max_iteration_per_job=30,
) # done creating HyperParameterOptimizer
```
## Running as a Service