diff --git a/docs/guides/frameworks/pytorch ignite/integration_pytorch_ignite.md b/docs/guides/frameworks/pytorch ignite/integration_pytorch_ignite.md index a42b4684..609d7980 100644 --- a/docs/guides/frameworks/pytorch ignite/integration_pytorch_ignite.md +++ b/docs/guides/frameworks/pytorch ignite/integration_pytorch_ignite.md @@ -1,182 +1,70 @@ --- -title: PyTorch Ignite Integration +title: PyTorch Ignite TensorboardLogger --- -Integrate **ClearML** into code using [ignite](https://github.com/pytorch/ignite). -Use ignite's `ClearMLLogger`, and the handlers that can be attached to it. See ignite's [handler](https://github.com/pytorch/ignite/blob/master/ignite/contrib/handlers/trains_logger.py). -:::note -If you are not already using **ClearML**, see our [Getting Started](/getting_started/ds/ds_first_steps.md). -::: +The [cifar_ignite.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/ignite/cifar_ignite.py) example +script integrates ClearML into code that uses [PyTorch Ignite](https://github.com/pytorch/ignite). -## Ignite ClearMLLogger +The example script does the following: +* Trains a neural network on the CIFAR10 dataset for image classification. +* Creates a [ClearML Task](../../../fundamentals/task.md) named `image classification CIFAR10`, which is associated with + the `examples` project. +* Calls the [`Task.connect`](../../../references/sdk/task.md#connect) method to track experiment configuration. +* Uses `ignite`'s `TensorboardLogger` and attaches handlers to it. See [`TensorboardLogger`](https://github.com/pytorch/ignite/blob/master/ignite/contrib/handlers/tensorboard_logger.py). -Integrate **ClearML** with the following steps: -1. Create an Ignite `ClearMLLogger` object. - -1. When the code runs, it connects to the **ClearML** backend, and creates a Task (experiment) in **ClearML**. - ```python - from ignite.contrib.handlers.clearml_logger import * +ClearML's automatic logging captures information and outputs logged with `TensorboardLogger`. - clearml_logger = ClearMLLogger(project_name="examples", task_name="ignite") - ``` -1. Later in the code, attach any of the **ClearML** handlers to the `ClearMLLogger` object. +## Hyperparameters + +Parameters are explicitly reported to ClearML using the [`Task.connect`](../../../references/sdk/task.md#connect) method. + +```python +params = {'number_of_epochs': 20, 'batch_size': 64, 'dropout': 0.25, 'base_lr': 0.001, 'momentum': 0.9, 'loss_report': 100} +params = task.connect(params) # enabling configuration override by clearml +``` +The hyperparameter configurations can be viewed in the WebApp in the experiment's **CONFIGURATION** tab. + + + +## Ignite TensorboardLogger + +`TensorboardLogger` is a handler to log metrics, parameters, and gradients when training a model. When ClearML is integrated +into a script which uses `TensorboardLogger`, all information logged through the handler is automatically captured by ClearML. - For example, attach the `OutputHandler` and log training loss at each iteration: - ```python - clearml_logger.attach(trainer, - log_handler=OutputHandler(tag="training", - output_transform=lambda loss: {"loss": loss}), - event_name=Events.ITERATION_COMPLETED) - ``` - -### ClearMLLogger Parameters +## Scalars -The following are the `ClearMLLogger` method parameters: +ClearML automatically captures scalars logged through `TensorboardLogger`. -* `project_name` (optional[str]) – The name of the project in which the experiment will be created. If the project does not exist, it is created. If `project_name` is `None`, the repository name becomes the project name. -* `task_name` (optional[str]) – The name of Task (experiment). If `task_name` is `None`, the Python experiment script’s file name becomes the Task name. -* `task_type` (optional[str]) – The name of the experiment. +View the scalars in the experiment's page in the **ClearML Web UI**, in **RESULTS** **>** **SCALARS**. - The `task_type` values include: - - * `TaskTypes.training` (default) - * `TaskTypes.train` - * `TaskTypes.testing` - * `TaskTypes.inference` - -* `report_freq` (optional[int]) – The histogram processing frequency (handles histogram values every X calls to the handler). Affects `GradsHistHandler` and `WeightsHistHandler`. Default value is `100`. -* `histogram_update_freq_multiplier` (optional[int]) – The histogram report frequency (report first X histograms and once every X reports afterwards). Default value is `10`. -* `histogram_granularity` (optional[int]): Optional. Histogram sampling granularity. Default is `50`. + -<a name="visualizing" class="tr_top_negative"></a> - -## Logging - -### Ignite Engine Output and / or Metrics - -To log scalars, Ignite engine's output and / or metrics, use the `OutputHandler`. - -* Log training loss at each iteration: -```python -# Attach the logger to the trainer to log training loss at each iteration -clearml_logger.attach(trainer, - log_handler=OutputHandler(tag="training", - output_transform=lambda loss: {"loss": loss}), - event_name=Events.ITERATION_COMPLETED) -``` - -* Log metrics for training: - -```python -# Attach the logger to the evaluator on the training dataset and log NLL, Accuracy metrics after each epoch -# We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch -# of the `trainer` instead of `train_evaluator`. -clearml_logger.attach(train_evaluator, - log_handler=OutputHandler(tag="training", - metric_names=["nll", "accuracy"], - global_step_transform=global_step_from_engine(trainer)), - event_name=Events.EPOCH_COMPLETED) -``` - -* Log metrics for validation: - -```python -# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after -# each epoch. We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch of the -# `trainer` instead of `evaluator`. -clearml_logger.attach(evaluator, - log_handler=OutputHandler(tag="validation", - metric_names=["nll", "accuracy"], - global_step_transform=global_step_from_engine(trainer)), - event_name=Events.EPOCH_COMPLETED) -``` - -### Optimizer Parameters - -To log optimizer parameters, use `OptimizerParamsHandler`: -```python -# Attach the logger to the trainer to log optimizer's parameters, e.g., learning rate at each iteration -clearml_logger.attach(trainer, - log_handler=OptimizerParamsHandler(optimizer), - event_name=Events.ITERATION_STARTED) -``` - -### Model Weights - -To log model weights as scalars, use `WeightsScalarHandler`: - -```python -# Attach the logger to the trainer to log model's weights norm after each iteration -clearml_logger.attach(trainer, - log_handler=WeightsScalarHandler(model, reduction=torch.norm), - event_name=Events.ITERATION_COMPLETED) -``` - -To log model weights as histograms, use `WeightsHistHandler`: - -```python -# Attach the logger to the trainer to log model's weights norm after each iteration -clearml_logger.attach(trainer, - log_handler=WeightsHistHandler(model), - event_name=Events.ITERATION_COMPLETED) -``` - ## Model Snapshots -To save input snapshots as **ClearML** artifacts, use `ClearMLSaver`: +**ClearML** automatically captures the model logged with Torch, and saves it as an artifact. -```python -to_save = {"model": model} - -handler = Checkpoint(to_save, ClearMLSaver(clearml_logger), n_saved=1, - score_function=lambda e: 123, score_name="acc", - filename_prefix="best", - global_step_transform=global_step_from_engine(trainer)) - -validation_evaluator.add_event_handler(Events.EVENT_COMPLETED, handler) -``` +View saved snapshots in the experiment's **ARTIFACTS** tab. -## Visualizing Experiment Results - -When the code with an ignite `ClearMLLogger` object and attached [handlers](https://github.com/pytorch/ignite/blob/master/ignite/contrib/handlers/trains_logger.py) -runs, the experiment results can be visualized in the **ClearML Web UI**. - -The `ignite` repository contains an MNIST ClearMLLogger example, [mnist_with_clearml_logger.py](https://github.com/pytorch/ignite/blob/master/examples/contrib/mnist/mnist_with_clearml_logger.py). - -Run this code and visualize the experiment results in the **ClearML Web UI**. - -### Scalars - -View the scalars, including training and validation metrics, in the experiment's page in the **ClearML Web UI**, under -**RESULTS** **>** **SCALARS**. - - - - -### Model Snapshots - -To save model snapshots, use `ClearMLServer`. - - -```python -handler = Checkpoint( - {"model": model}, - ClearMLSaver(clearml_logger, dirname="~/.clearml/cache/"), - n_saved=1, - score_function=lambda e: 123, - score_name="acc", - filename_prefix="best", - global_step_transform=global_step_from_engine(trainer), - ) -``` - -<br/> - -View saved snapshots in the **ARTIFACTS** tab. - - + To view the model, in the **ARTIFACTS** tab, click the model name (or download it). - + + + +## Debug Samples + +ClearML automatically tracks images logged to TensorboardLogger. They appear in **RESULTS** **>** **DEBUG SAMPLES**. + + + + +## Ignite ClearMLLogger + +PyTorch Ignite also offers a dedicated `ClearMLLogger` handler to log metrics, text, model / optimizer parameters, plots, and model +checkpoints during training and validation. + +For more information, see the [PyTorch Ignite ClearMLLogger](https://pytorch.org/ignite/generated/ignite.contrib.handlers.clearml_logger.html) +example. + diff --git a/docs/guides/frameworks/pytorch ignite/pytorch_ignite_mnist.md b/docs/guides/frameworks/pytorch ignite/pytorch_ignite_mnist.md new file mode 100644 index 00000000..adea92e2 --- /dev/null +++ b/docs/guides/frameworks/pytorch ignite/pytorch_ignite_mnist.md @@ -0,0 +1,162 @@ +--- +title: PyTorch Ignite ClearMLLogger +--- + +The `ignite` repository contains the [mnist_with_clearml_logger.py](https://github.com/pytorch/ignite/blob/master/examples/contrib/mnist/mnist_with_clearml_logger.py) +example script that uses [ignite](https://github.com/pytorch/ignite) and integrates **ClearMLLogger** and its helper [handlers](https://github.com/pytorch/ignite/blob/master/ignite/contrib/handlers/clearml_logger.py). + +PyTorch Ignite offers a `ClearMLLogger` handler to log metrics, text, model / optimizer parameters, plots, and model +checkpoints during training and validation. + +The example script does the following: +* Trains a model to classify images from the MNIST dataset. +* Creates a [ClearML Task](../../../fundamentals/task.md) named `ignite`, which is associated with the `examples` + project. ClearMLLogger connects to ClearML so everything which is logged through it and its handlers + is automatically captured by ClearML. +* Uses the following ClearMLLogger helper handlers: + * **ClearMLSaver** - Saves input snapshots as ClearML artifacts. + * **GradsHistHandler** and **WeightsHistHandler** - Logs the model's gradients and weights respectively as histograms. + * **GradsScalarHandler** and **WeightsScalarHandler** - Logs gradients and weights respectively as scalars. + + +## ClearMLLogger + +Integrate ClearML with the following steps: +1. Create a `ClearMLLogger` object. When the code runs, it connects to the ClearML backend, and creates a task in ClearML + (see ClearMLLogger's parameters [below](#parameters)). + + ```python + from ignite.contrib.handlers.clearml_logger import ClearMLLogger + + clearml_logger = ClearMLLogger(project_name="examples", task_name="ignite") + ``` + +1. Attach helper handlers to the `ClearMLLogger` object. + + For example, attach the `OutputHandler` to log training loss at each iteration: + ```python + clearml_logger.attach(trainer, + log_handler=OutputHandler(tag="training", + output_transform=lambda loss: {"loss": loss}), + event_name=Events.ITERATION_COMPLETED) + ``` + +### Parameters +The following are the `ClearMLLogger` parameters: +* `project_name` - The name of the project in which the experiment will be created. +* `task_name` – The name of task. +* `task_type` – The type of experiment (see [task types](../../../fundamentals/task.md#task-types)). +* `report_freq` – The histogram processing frequency (handles histogram values every X calls to the handler). Affects + `GradsHistHandler` and `WeightsHistHandler`. Default value is 100. +* `histogram_update_freq_multiplier` – The histogram report frequency (report first X histograms and once every X + reports afterwards). Default value is 10. +* `histogram_granularity` - Histogram sampling granularity. Default is 50. + +### Logging +To log scalars, ignite engine's output and / or metrics, use the `OutputHandler`. + +* Log training loss at each iteration: +```python +clearml_logger.attach(trainer, + log_handler=OutputHandler(tag="training", + output_transform=lambda loss: {"loss": loss}), + event_name=Events.ITERATION_COMPLETED) +``` + +* Log metrics for training: + +```python +clearml_logger.attach(train_evaluator, + log_handler=OutputHandler(tag="training", + metric_names=["nll", "accuracy"], + global_step_transform=global_step_from_engine(trainer)), + event_name=Events.EPOCH_COMPLETED) +``` + +* Log metrics for validation: + +```python +clearml_logger.attach(evaluator, + log_handler=OutputHandler(tag="validation", + metric_names=["nll", "accuracy"], + global_step_transform=global_step_from_engine(trainer)), + event_name=Events.EPOCH_COMPLETED) +``` + +To log optimizer parameters, use the `attach_opt_params_handler` method: +```python +# Attach the logger to the trainer to log optimizer's parameters, e.g., learning rate at each iteration +clearml_logger.attach_opt_params_handler( + trainer, event_name=Events.ITERATION_COMPLETED(every=100), optimizer=optimizer +``` + +### Model Weights + +To log model weights as scalars, use `WeightsScalarHandler`: + +```python +from ignite.contrib.handlers.clearml_logger import WeightsScalarHandler + +clearml_logger.attach(trainer, + log_handler=WeightsScalarHandler(model, reduction=torch.norm), + event_name=Events.ITERATION_COMPLETED) +``` + +To log model weights as histograms, use `WeightsHistHandler`: + +```python +from ignite.contrib.handlers.clearml_logger import WeightsHistHandler + +clearml_logger.attach(trainer, + log_handler=WeightsHistHandler(model), + event_name=Events.ITERATION_COMPLETED) +``` + + +### Model Snapshots + +To save model checkpoints as ClearML artifacts, use `ClearMLSaver`: + +```python +from ignite.handlers import Checkpoint +from ignite.contrib.handlers.clearml_logger import ClearMLSaver + +handler = Checkpoint( + {"model": model}, + ClearMLSaver(), + n_saved=1, + score_function=lambda e: e.state.metrics["accuracy"], + score_name="val_acc", + filename_prefix="best", + global_step_transform=global_step_from_engine(trainer), + ) + +validation_evaluator.add_event_handler(Events.EPOCH_COMPLETED, handler) +``` + + +## Visualizing Experiment Results + +When the code runs, the experiment results can be viewed in the [ClearML Web UI](../../../webapp/webapp_overview.md). + +### Scalars + +View the scalars, including training and validation metrics, in the experiment's page in the ClearML Web UI, under +**RESULTS** **>** **SCALARS**. + + + +<br/> + + + +### Model Snapshots + + +View saved snapshots in the **ARTIFACTS** tab. + + + +To view model details, in the **ARTIFACTS** tab, click the model name (or download it). + + diff --git a/docs/img/examples_cifar_artifacts.png b/docs/img/examples_cifar_artifacts.png new file mode 100644 index 00000000..01a8ee2c Binary files /dev/null and b/docs/img/examples_cifar_artifacts.png differ diff --git a/docs/img/examples_cifar_model.png b/docs/img/examples_cifar_model.png new file mode 100644 index 00000000..0f2a6d3b Binary files /dev/null and b/docs/img/examples_cifar_model.png differ diff --git a/docs/img/examples_cifar_scalars.png b/docs/img/examples_cifar_scalars.png new file mode 100644 index 00000000..691b76a7 Binary files /dev/null and b/docs/img/examples_cifar_scalars.png differ diff --git a/docs/img/examples_integration_pytorch_ignite_config.png b/docs/img/examples_integration_pytorch_ignite_config.png new file mode 100644 index 00000000..e25a682f Binary files /dev/null and b/docs/img/examples_integration_pytorch_ignite_config.png differ diff --git a/docs/img/examples_integration_pytorch_ignite_debug.png b/docs/img/examples_integration_pytorch_ignite_debug.png new file mode 100644 index 00000000..d994f39d Binary files /dev/null and b/docs/img/examples_integration_pytorch_ignite_debug.png differ diff --git a/sidebars.js b/sidebars.js index c94ccc9b..7a1a01e3 100644 --- a/sidebars.js +++ b/sidebars.js @@ -71,11 +71,11 @@ module.exports = { }, {'LightGBM': ['guides/frameworks/lightgbm/lightgbm_example']}, {'Matplotlib': ['guides/frameworks/matplotlib/matplotlib_example']}, - {'Pytorch': + {'PyTorch': ['guides/frameworks/pytorch/pytorch_distributed_example', 'guides/frameworks/pytorch/pytorch_matplotlib', 'guides/frameworks/pytorch/pytorch_mnist', 'guides/frameworks/pytorch/pytorch_tensorboard', 'guides/frameworks/pytorch/pytorch_tensorboardx', 'guides/frameworks/pytorch/tensorboard_toy_pytorch', - {'Pytorch Notebooks': [ + {'PyTorch Notebooks': [ {'Audio': ['guides/frameworks/pytorch/notebooks/audio/audio_classification_UrbanSound8K', 'guides/frameworks/pytorch/notebooks/audio/audio_preprocessing_example']}, {'Image': ['guides/frameworks/pytorch/notebooks/image/hyperparameter_search', 'guides/frameworks/pytorch/notebooks/image/image_classification_CIFAR10']}, {'Table': ['guides/frameworks/pytorch/notebooks/table/download_and_preprocessing', 'guides/frameworks/pytorch/notebooks/table/tabular_training_pipeline']}, @@ -83,8 +83,8 @@ module.exports = { } ] }, - {'Pytorch Ignite': ['guides/frameworks/pytorch ignite/integration_pytorch_ignite']}, - {'Pytorch Lightning': ['guides/frameworks/pytorch_lightning/pytorch_lightning_example']}, + {'PyTorch Ignite': ['guides/frameworks/pytorch ignite/integration_pytorch_ignite', 'guides/frameworks/pytorch ignite/pytorch_ignite_mnist']}, + {'PyTorch Lightning': ['guides/frameworks/pytorch_lightning/pytorch_lightning_example']}, {'Scikit-Learn': ['guides/frameworks/scikit-learn/sklearn_joblib_example', 'guides/frameworks/scikit-learn/sklearn_matplotlib_example']}, {'TensorBoardX': ['guides/frameworks/tensorboardx/tensorboardx', "guides/frameworks/tensorboardx/video_tensorboardx"]}, {