From ce58c8e2d4376675c74c376f855d5a897ef7fa69 Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Tue, 21 May 2024 11:30:46 +0300 Subject: [PATCH] Add multi-objective optimization info (#844) --- docs/fundamentals/hpo.md | 15 +- .../examples_hyperparam_opt.md | 134 ++++++++++-------- 2 files changed, 90 insertions(+), 59 deletions(-) diff --git a/docs/fundamentals/hpo.md b/docs/fundamentals/hpo.md index ebcf8970..5100df40 100644 --- a/docs/fundamentals/hpo.md +++ b/docs/fundamentals/hpo.md @@ -23,7 +23,7 @@ compare results. The preceding diagram demonstrates the typical flow of hyperparameter optimization where the parameters of a base task are optimized: -1. Configure an Optimization Task with a base task whose parameters will be optimized, and a set of parameter values to +1. Configure an Optimization Task with a base task whose parameters will be optimized, optimization targets, and a set of parameter values to test 1. Clone the base task. Each clone's parameter is overridden with a value from the optimization task 1. Enqueue each clone for execution by a ClearML Agent @@ -118,6 +118,19 @@ optimization. in the task header. ::: + :::tip Multi-objective Optimization + If you are using the Optuna framework (see [Supported Optimizers](#supported-optimizers)), you can list multiple optimization objectives. + When doing so, make sure the `objective_metric_title`, `objective_metric_series`, and `objective_metric_sign` lists + are the same length. Each title will be matched to its respective series and sign. + + For example, the code below sets two objectives: to minimize the `validation/loss` metric and to maximize the `validation/accuracy` metric. + ```python + objective_metric_title=["validation", "validation"] + objective_metric_series=["loss", "accuracy"] + objective_metric_sign=["min", "max"] + ``` + ::: + ## Optimizer Execution Options The `HyperParameterOptimizer` provides options to launch the optimization tasks locally or through a ClearML [queue](agents_and_queues.md#what-is-a-queue). diff --git a/docs/guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt.md b/docs/guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt.md index 07ac211a..e77658d9 100644 --- a/docs/guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt.md +++ b/docs/guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt.md @@ -114,67 +114,85 @@ if not args['template_task_id']: ## Creating the Optimizer Object Initialize an [`automation.HyperParameterOptimizer`](../../../references/sdk/hpo_optimization_hyperparameteroptimizer.md) -object, setting the optimization parameters, beginning with the ID of the experiment to optimize. +object, setting the following optimization parameters: -```python -an_optimizer = HyperParameterOptimizer( - # This is the experiment we want to optimize - base_task_id=args['template_task_id'], -``` +* ID of a ClearML task to optimize. This task will be cloned, and each clone will sample a different set of hyperparameters values: -Set the hyperparameter ranges to sample, instantiating them as ClearML automation objects using [`automation.UniformIntegerParameterRange`](../../../references/sdk/hpo_parameters_uniformintegerparameterrange.md) -and [`automation.DiscreteParameterRange`](../../../references/sdk/hpo_parameters_discreteparameterrange.md). - -```python - hyper_parameters=[ - UniformIntegerParameterRange('layer_1', min_value=128, max_value=512, step_size=128), - UniformIntegerParameterRange('layer_2', min_value=128, max_value=512, step_size=128), - DiscreteParameterRange('batch_size', values=[96, 128, 160]), - DiscreteParameterRange('epochs', values=[30]), - ], -``` + ```python + an_optimizer = HyperParameterOptimizer( + # This is the experiment we want to optimize + base_task_id=args['template_task_id'], + ``` + +* Hyperparameter ranges to sample, instantiating them as ClearML automation objects using [`automation.UniformIntegerParameterRange`](../../../references/sdk/hpo_parameters_uniformintegerparameterrange.md) + and [`automation.DiscreteParameterRange`](../../../references/sdk/hpo_parameters_discreteparameterrange.md): + + ```python + hyper_parameters=[ + UniformIntegerParameterRange('layer_1', min_value=128, max_value=512, step_size=128), + UniformIntegerParameterRange('layer_2', min_value=128, max_value=512, step_size=128), + DiscreteParameterRange('batch_size', values=[96, 128, 160]), + DiscreteParameterRange('epochs', values=[30]), + ], + ``` + +* Metric to optimize and the optimization objective: + + ```python + objective_metric_title='val_acc', + objective_metric_series='val_acc', + objective_metric_sign='max', + ``` + + :::tip Multi-objective Optimization + If you are using the Optuna framework (see [Set the Search Strategy for Optimization](#set-the-search-strategy-for-optimization)), + you can list multiple optimization objectives. When doing so, make sure the `objective_metric_title`, + `objective_metric_series`, and `objective_metric_sign` lists are + the same length. Each title will be matched to its respective series and sign. + + For example, the code below sets two objectives: to minimize the `validation/loss` metric and to maximize the + `validation/accuracy` metric: + ```python + objective_metric_title=["validation", "validation"] + objective_metric_series=["loss", "accuracy"] + objective_metric_sign=["min", "max"] + ``` + ::: + + * Number of concurrent Tasks: + ```python + max_number_of_concurrent_tasks=2, + ``` + * Optimization strategy (see [Set the search strategy for optimization](#set-the-search-strategy-for-optimization)): + ```python + optimizer_class=aSearchStrategy, + ``` + * Queue to use for remote execution. This is overridden if the optimizer runs as a service. + ```python + execution_queue='1xGPU', + ``` + * Remaining parameters, including the time limit per Task (minutes), period for checking the optimization (minutes), + maximum number of jobs to launch, minimum and maximum number of iterations for each Task: + ```python + # Optional: Limit the execution time of a single experiment, in minutes. + # (this is optional, and if using OptimizerBOHB, it is ignored) + time_limit_per_job=10., + # Check the experiments every 6 seconds is way too often, we should probably set it to 5 min, + # assuming a single experiment is usually hours... + pool_period_min=0.1, + # set the maximum number of jobs to launch for the optimization, default (None) unlimited + # If OptimizerBOHB is used, it defined the maximum budget in terms of full jobs + # basically the cumulative number of iterations will not exceed total_max_jobs * max_iteration_per_job + total_max_jobs=10, + # This is only applicable for OptimizerBOHB and ignore by the rest + # set the minimum number of iterations for an experiment, before early stopping + min_iteration_per_job=10, + # Set the maximum number of iterations for an experiment to execute + # (This is optional, unless using OptimizerBOHB where this is a must) + max_iteration_per_job=30, -Set the metric to optimize and the optimization objective. - -```python - objective_metric_title='val_acc', - objective_metric_series='val_acc', - objective_metric_sign='max', -``` - -Set the number of concurrent Tasks. -```python - max_number_of_concurrent_tasks=2, -``` -Set the optimization strategy, see [Set the search strategy for optimization](#set-the-search-strategy-for-optimization). -```python - optimizer_class=aSearchStrategy, -``` -Specify the queue to use for remote execution. This is overridden if the optimizer runs as a service. -```python - execution_queue='1xGPU', -``` -Specify the remaining parameters, including the time limit per Task (minutes), period for checking the optimization (minutes), maximum number of jobs to launch, minimum and maximum number of iterations for each Task. -```python - # Optional: Limit the execution time of a single experiment, in minutes. - # (this is optional, and if using OptimizerBOHB, it is ignored) - time_limit_per_job=10., - # Check the experiments every 6 seconds is way too often, we should probably set it to 5 min, - # assuming a single experiment is usually hours... - pool_period_min=0.1, - # set the maximum number of jobs to launch for the optimization, default (None) unlimited - # If OptimizerBOHB is used, it defined the maximum budget in terms of full jobs - # basically the cumulative number of iterations will not exceed total_max_jobs * max_iteration_per_job - total_max_jobs=10, - # This is only applicable for OptimizerBOHB and ignore by the rest - # set the minimum number of iterations for an experiment, before early stopping - min_iteration_per_job=10, - # Set the maximum number of iterations for an experiment to execute - # (This is optional, unless using OptimizerBOHB where this is a must) - max_iteration_per_job=30, - -) # done creating HyperParameterOptimizer -``` + ) # done creating HyperParameterOptimizer + ``` ## Running as a Service