mirror of
https://github.com/clearml/clearml-docs
synced 2025-01-31 14:37:18 +00:00
156 lines
7.5 KiB
Markdown
156 lines
7.5 KiB
Markdown
---
|
||
title: Hyperparameter Optimization
|
||
---
|
||
|
||
## What is Hyperparameter Optimization?
|
||
Hyperparameters are variables that directly control the behaviors of training algorithms, and have a significant effect on
|
||
the performance of the resulting machine learning models. Finding the hyperparameter values that yield the best
|
||
performing models can be complicated. Manually adjusting hyperparameters over the course of many training trials can be
|
||
slow and tedious. Luckily, you can automate and boost hyperparameter optimization (HPO) with ClearML's
|
||
[**`HyperParameterOptimizer`**](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class.
|
||
|
||
## ClearML's Hyperparameter Optimization
|
||
|
||
ClearML provides the `HyperParameterOptimizer` class, which takes care of the entire optimization process for users
|
||
with a simple interface.
|
||
|
||
ClearML's approach to hyperparameter optimization is scalable, easy to set up and to manage, and it makes it easy to
|
||
compare results.
|
||
|
||
### Workflow
|
||
|
||
|
||
|
||
The preceding diagram demonstrates the typical flow of hyperparameter optimization where the parameters of a base task are optimized:
|
||
|
||
1. Configure an Optimization Task with a base task whose parameters will be optimized, and a set of parameter values to
|
||
test
|
||
1. Clone the base task. Each clone's parameter is overridden with a value from the optimization task
|
||
1. Enqueue each clone for execution by a ClearML Agent
|
||
1. The Optimization Task records and monitors the cloned tasks' configuration and execution details, and returns a
|
||
summary of the optimization results in tabular and parallel coordinate formats, and in a scalar plot.
|
||
|
||
|
||
|
||
|
||
<Collapsible title="Parallel coordinate and scalar plots" type="screenshot">
|
||
|
||
|
||
|
||
|
||
|
||
</Collapsible>
|
||
|
||
### Supported Optimizers
|
||
|
||
The `HyperParameterOptimizer` class contains ClearML’s hyperparameter optimization modules. Its modular design enables
|
||
using different optimizers, including existing software frameworks, enabling simple, accurate, and fast hyperparameter
|
||
optimization.
|
||
|
||
* **Optuna** - [`automation.optuna.OptimizerOptuna`](../references/sdk/hpo_optuna_optuna_optimizeroptuna.md). Optuna is the default optimizer in ClearML. It makes use of
|
||
different samplers such as grid search, random, bayesian, and evolutionary algorithms.
|
||
For more information, see the [Optuna](https://optuna.readthedocs.io/en/latest/)
|
||
documentation.
|
||
* **BOHB** - [`automation.hpbandster.OptimizerBOHB`](../references/sdk/hpo_hpbandster_bandster_optimizerbohb.md). BOHB performs robust and efficient hyperparameter optimization
|
||
at scale by combining the speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian Optimization.
|
||
For more information about HpBandSter BOHB, see the [HpBandSter](https://automl.github.io/HpBandSter/build/html/index.html)
|
||
documentation and a [code example](../guides/frameworks/pytorch/notebooks/image/hyperparameter_search.md).
|
||
* **Random** uniform sampling of hyperparameters - [`automation.RandomSearch`](../references/sdk/hpo_optimization_randomsearch.md).
|
||
* **Full grid** sampling strategy of every hyperparameter combination - [`automation.GridSearch`](../references/sdk/hpo_optimization_gridsearch.md).
|
||
* **Custom** - [`automation.optimization.SearchStrategy`](https://github.com/allegroai/clearml/blob/master/clearml/automation/optimization.py#L268) - Use a custom class and inherit from the ClearML automation base strategy class
|
||
|
||
|
||
## Defining a Hyperparameter Optimization Search Example
|
||
|
||
1. Import ClearML's automation modules:
|
||
|
||
```python
|
||
from clearml.automation import UniformParameterRange, UniformIntegerParameterRange
|
||
from clearml.automation import HyperParameterOptimizer
|
||
from clearml.automation.optuna import OptimizerOptuna
|
||
```
|
||
1. Initialize the Task, which will be stored in ClearML Server when the code runs. After the code runs at least once,
|
||
it can be reproduced, and the parameters can be tuned:
|
||
```python
|
||
from clearml import Task
|
||
|
||
task = Task.init(
|
||
project_name='Hyper-Parameter Optimization',
|
||
task_name='Automatic Hyper-Parameter Optimization',
|
||
task_type=Task.TaskTypes.optimizer,
|
||
reuse_last_task_id=False
|
||
)
|
||
```
|
||
|
||
1. Define the optimization configuration and resources budget:
|
||
```python
|
||
optimizer = HyperParameterOptimizer(
|
||
# specifying the task to be optimized, task must be in system already so it can be cloned
|
||
base_task_id=TEMPLATE_TASK_ID,
|
||
# setting the hyperparameters to optimize
|
||
hyper_parameters=[
|
||
UniformIntegerParameterRange('number_of_epochs', min_value=2, max_value=12, step_size=2),
|
||
UniformIntegerParameterRange('batch_size', min_value=2, max_value=16, step_size=2),
|
||
UniformParameterRange('dropout', min_value=0, max_value=0.5, step_size=0.05),
|
||
UniformParameterRange('base_lr', min_value=0.00025, max_value=0.01, step_size=0.00025),
|
||
],
|
||
# setting the objective metric we want to maximize/minimize
|
||
objective_metric_title='accuracy',
|
||
objective_metric_series='total',
|
||
objective_metric_sign='max',
|
||
|
||
# setting optimizer
|
||
optimizer_class=OptimizerOptuna,
|
||
|
||
# configuring optimization parameters
|
||
execution_queue='default',
|
||
max_number_of_concurrent_tasks=2,
|
||
optimization_time_limit=60.,
|
||
compute_time_limit=120,
|
||
total_max_jobs=20,
|
||
min_iteration_per_job=15000,
|
||
max_iteration_per_job=150000,
|
||
)
|
||
```
|
||
|
||
:::tip Locating Task ID
|
||
To locate the base task's ID, go to the task's info panel in the [WebApp](../webapp/webapp_overview.md). The ID appears
|
||
in the task header.
|
||
:::
|
||
|
||
|
||
## Optimizer Execution Options
|
||
The `HyperParameterOptimizer` provides options to launch the optimization tasks locally or through a ClearML [queue](agents_and_queues.md#what-is-a-queue).
|
||
Start a `HyperParameterOptimizer` instance using either [`HyperParameterOptimizer.start()`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md#start)
|
||
or [`HyperParameterOptimizer.start_locally()`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md#start_locally).
|
||
Both methods run the optimizer controller locally. `start()` launches the base task clones through a queue
|
||
specified when instantiating the controller, while `start_locally()` runs the tasks locally.
|
||
|
||
:::tip Remote Execution
|
||
You can also launch the optimizer controller through a queue by using [`Task.execute_remotely()`](../references/sdk/task.md#execute_remotely)
|
||
before starting the optimizer.
|
||
:::
|
||
|
||
|
||
## Tutorial
|
||
|
||
Check out the [Hyperparameter Optimization tutorial](../guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt.md) for a step-by-step guide.
|
||
|
||
## SDK Reference
|
||
|
||
For detailed information, see the complete [HyperParameterOptimizer SDK reference page](../references/sdk/hpo_optimization_hyperparameteroptimizer.md).
|
||
|
||
## CLI
|
||
|
||
ClearML also provides `clearml-param-search`, a CLI utility for managing the hyperparameter optimization process. See
|
||
[ClearML Param Search](../apps/clearml_param_search.md) for more information.
|
||
|
||
## UI Application
|
||
|
||
:::info Pro Plan Offering
|
||
The ClearML HPO App is available under the ClearML Pro plan
|
||
:::
|
||
|
||
ClearML provides the [Hyperparameter Optimization GUI application](../webapp/applications/apps_hpo.md) for launching and
|
||
managing the hyperparameter optimization process.
|