mirror of
https://github.com/clearml/clearml-docs
synced 2025-01-31 22:48:40 +00:00
a8802194b9
* add hpo diagram * edit around hpo diagram * small edits hpo, put tutorial under own heading * fix incorrect wording
114 lines
5.3 KiB
Markdown
114 lines
5.3 KiB
Markdown
---
|
||
title: Hyperparameter Optimization
|
||
---
|
||
|
||
## What is HyperParameter Optimization?
|
||
Hyperparameters are variables that directly control the behaviors of training algorithms, and have a significant effect on
|
||
the performance of the resulting machine learning models. Finding the hyperparameter values that yield the best
|
||
performing models can be complicated. Manually adjusting hyperparameters over the course of many training trials can be
|
||
slow and tedious. Luckily, hyperparameter optimization can be automated and boosted using ClearML's
|
||
**`HyperParameterOptimizer`** class.
|
||
|
||
## ClearML's HyperParameter Optimization
|
||
|
||
ClearML provides the `HyperParameterOptimizer` class, which takes care of the entire optimization process for users in
|
||
with a simple interface.
|
||
|
||
ClearML's approach to hyperparameter optimization is scalable, easy to set up and to manage, and it makes it easy to
|
||
compare results.
|
||
|
||
### Workflow
|
||
|
||
|
||
|
||
The diagram above demonstrates the typical flow of hyperparameter optimization where the parameters of a base task are optimized:
|
||
|
||
1. Configure an Optimization Task with a base task whose parameters will be optimized, and a set of parameter values to
|
||
test
|
||
1. Clone the base task. Each clone's parameter is overridden with a value from the optimization task
|
||
1. Enqueue each clone for execution by a ClearML Agent
|
||
1. The Optimization Task records and monitors the cloned tasks' configuration and execution details, and returns a
|
||
summary of the optimization results
|
||
|
||
|
||
|
||
|
||
### Supported Optimizers
|
||
|
||
The `HyperParameterOptimizer` class contains **ClearML**’s hyperparameter optimization modules. Its modular design enables
|
||
using different optimizers, including existing software frameworks, enabling simple, accurate, and fast hyperparameter
|
||
optimization.
|
||
|
||
* **Optuna** - `automation.optuna.optuna.OptimizerOptuna`. Optuna is the default optimizer in ClearML. It makes use of
|
||
different samplers such as grid search, random, bayesian, and evolutionary algorithms.
|
||
For more information, see the [Optuna](https://optuna.readthedocs.io/en/latest/)
|
||
documentation.
|
||
* **BOHB** - `automation.hpbandster.bandster.OptimizerBOHB`. BOHB performs robust and efficient hyperparameter optimization
|
||
at scale by combining the speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian Optimization.
|
||
For more information about HpBandSter BOHB, see the [HpBandSter](https://automl.github.io/HpBandSter/build/html/index.html)
|
||
documentation.
|
||
* **Random** uniform sampling of hyperparameters - `automation.optimization.RandomSearch`.
|
||
* **Full grid** sampling strategy of every hyperparameter combination - `Grid search automation.optimization.GridSearch`.
|
||
* **Custom** - `automation.optimization.SearchStrategy` - Use a custom class and inherit from the ClearML automation base strategy class
|
||
|
||
|
||
## Defining a hyperparameter optimization search example
|
||
|
||
1. Import ClearML's automation modules:
|
||
|
||
```python
|
||
from clearml.automation import UniformParameterRange, UniformIntegerParameterRange
|
||
from clearml.automation import HyperParameterOptimizer
|
||
from clearml.automation.optuna import OptimizerOptuna
|
||
```
|
||
1. Initialize the Task, which will be stored in ClearML Server when the code runs. After the code runs at least once,
|
||
it can be reproduced, and the parameters can be tuned:
|
||
```python
|
||
from clearml import Task
|
||
|
||
task = Task.init(project_name='Hyper-Parameter Optimization',
|
||
task_name='Automatic Hyper-Parameter Optimization',
|
||
task_type=Task.TaskTypes.optimizer,
|
||
reuse_last_task_id=False)
|
||
|
||
```
|
||
|
||
1. Define the optimization configuration and resources budget:
|
||
```python
|
||
optimizer = HyperParameterOptimizer(
|
||
# specifying the task to be optimized, task must be in system already so it can be cloned
|
||
base_task_id=TEMPLATE_TASK_ID,
|
||
# setting the hyper-parameters to optimize
|
||
hyper_parameters=[
|
||
UniformIntegerParameterRange('number_of_epochs', min_value=2, max_value=12, step_size=2),
|
||
UniformIntegerParameterRange('batch_size', min_value=2, max_value=16, step_size=2),
|
||
UniformParameterRange('dropout', min_value=0, max_value=0.5, step_size=0.05),
|
||
UniformParameterRange('base_lr', min_value=0.00025, max_value=0.01, step_size=0.00025),
|
||
],
|
||
# setting the objective metric we want to maximize/minimize
|
||
objective_metric_title='accuracy',
|
||
objective_metric_series='total',
|
||
objective_metric_sign='max',
|
||
|
||
# setting optimizer
|
||
optimizer_class=OptimizerOptuna,
|
||
|
||
# configuring optimization parameters
|
||
execution_queue='default',
|
||
max_number_of_concurrent_tasks=2,
|
||
optimization_time_limit=60.,
|
||
compute_time_limit=120,
|
||
total_max_jobs=20,
|
||
min_iteration_per_job=15000,
|
||
max_iteration_per_job=150000,
|
||
)
|
||
```
|
||
<br/><br/>
|
||
|
||
For more information about `HyperParameterOptimizer` and supported optimization modules, see the [HyperParameterOptimizer class reference](../references/sdk/hpo_optimization_hyperparameteroptimizer.md).
|
||
|
||
## Tutorial
|
||
|
||
Check out the [Hyperparameter Optimization](../guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt.md) tutorial for a step-by-step guide.
|
||
|