diff --git a/docs/guides/frameworks/pytorch_ignite/integration_pytorch_ignite.md b/docs/guides/frameworks/pytorch_ignite/integration_pytorch_ignite.md index 69d54a94..1726a97e 100644 --- a/docs/guides/frameworks/pytorch_ignite/integration_pytorch_ignite.md +++ b/docs/guides/frameworks/pytorch_ignite/integration_pytorch_ignite.md @@ -1,6 +1,5 @@ --- title: PyTorch Ignite TensorboardLogger -displayed_sidebar: mainSidebar --- The [cifar_ignite.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/ignite/cifar_ignite.py) example diff --git a/docs/guides/frameworks/pytorch_ignite/pytorch_ignite_mnist.md b/docs/guides/frameworks/pytorch_ignite/pytorch_ignite_mnist.md index adc39cad..ea6aab2f 100644 --- a/docs/guides/frameworks/pytorch_ignite/pytorch_ignite_mnist.md +++ b/docs/guides/frameworks/pytorch_ignite/pytorch_ignite_mnist.md @@ -1,6 +1,5 @@ --- title: PyTorch Ignite ClearMLLogger -displayed_sidebar: mainSidebar --- The `ignite` repository contains the [mnist_with_clearml_logger.py](https://github.com/pytorch/ignite/blob/master/examples/contrib/mnist/mnist_with_clearml_logger.py) diff --git a/docs/guides/frameworks/tensorflow/tensorflow_mnist.md b/docs/guides/frameworks/tensorflow/tensorflow_mnist.md index 108d3b52..b13aa133 100644 --- a/docs/guides/frameworks/tensorflow/tensorflow_mnist.md +++ b/docs/guides/frameworks/tensorflow/tensorflow_mnist.md @@ -1,6 +1,5 @@ --- -title: TensorFlow -displayed_sidebar: mainSidebar +title: TensorFlow MNIST --- The [tensorflow_mnist.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorflow_mnist.py) diff --git a/docs/img/gif/tensorflow.gif b/docs/img/gif/tensorflow.gif new file mode 100644 index 00000000..72770880 Binary files /dev/null and b/docs/img/gif/tensorflow.gif differ diff --git a/docs/integrations/ignite.md b/docs/integrations/ignite.md new file mode 100644 index 00000000..eb2bbaa3 --- /dev/null +++ b/docs/integrations/ignite.md @@ -0,0 +1,146 @@ +--- +title: PyTorch Ignite +--- + +[PyTorch Ignite](https://pytorch.org/ignite/index.html) is a library for training and evaluating neural networks in +PyTorch. You can integrate ClearML into your code using Ignite’s built-in loggers: [TensorboardLogger](#tensorboardlogger) +and [ClearMLLogger](#clearmllogger). + +## TensorboardLogger + +ClearML integrates seamlessly with TensorboardLogger, and automatically captures all information logged through the +handler: metrics, parameters, images, and gradients. + +All you have to do is add two lines of code to your script: + +```python +from clearml import Task +task = Task.init(task_name="", project_name="") +``` + +This will create a [ClearML Task](../fundamentals/task.md) that captures your script's information, including Git details, +uncommitted code, python environment, all information logged through `TensorboardLogger`, and more. + +Visualize all the captured information in the experiment's page in ClearML's [WebApp](#webapp) + +See a code example [here](https://github.com/allegroai/clearml/blob/master/examples/frameworks/ignite/cifar_ignite.py). + +## ClearMLLogger +PyTorch Ignite supports a ClearML Logger to log metrics, text, model/optimizer parameters, plots, and model checkpoints +during training and validation. + +Integrate ClearML with the following steps: +1. Create a `ClearMLLogger` object: + + ```python + from ignite.contrib.handlers.clearml_logger import * + + clearml_logger = ClearMLLogger(task_name="ignite", project_name="examples") + ``` + + This creates a [ClearML Task](../fundamentals/task.md) called `ignite` in the `examples` project, which captures your + script's information, including Git details, uncommitted code, python environment. + + You can also pass the following parameters to the `ClearMLLogger` object: + * `task_type` – The type of experiment (see [task types](../fundamentals/task.md#task-types)). + * `report_freq` – The histogram processing frequency (handles histogram values every X calls to the handler). Affects + `GradsHistHandler` and `WeightsHistHandler` (default: 100). + * `histogram_update_freq_multiplier` – The histogram report frequency (report first X histograms and once every X + reports afterwards) (default: 10). + * `histogram_granularity` - Histogram sampling granularity (default: 50). + +1. Attach the C`learMLLogger` to output handlers to log metrics: + + ```python + # Attach the logger to the trainer to log training loss + clearml_logger.attach_output_handler( + trainer, + event_name=Events.ITERATION_COMPLETED(every=100), + tag="training", + output_transform=lambda loss: {"batchloss": loss}, + ) + + # Attach the logger to log loss and accuracy for both training and validation + for tag, evaluator in [("training metrics", train_evaluator), ("validation metrics", validation_evaluator)]: + clearml_logger.attach_output_handler( + evaluator, + event_name=Events.EPOCH_COMPLETED, + tag=tag, + metric_names=["loss", "accuracy"], + global_step_transform=global_step_from_engine(trainer), + ) + ``` + +1. Attach the ClearMLLogger object to helper handlers to log experiment outputs. Ignite supports the following helper handlers for ClearML: + + * **ClearMLSaver** - Saves input snapshots as ClearML artifacts. + * **GradsHistHandler** and **WeightsHistHandler** - Logs the model's gradients and weights respectively as histograms. + * **GradsScalarHandler** and **WeightsScalarHandler** - Logs gradients and weights respectively as scalars. + * **OptimizerParamsHandler** - Logs optimizer parameters + + ```python + # Attach the logger to the trainer to log model's weights norm + clearml_logger.attach( + trainer, log_handler=WeightsScalarHandler(model), event_name=Events.ITERATION_COMPLETED(every=100) + ) + + # Attach the logger to the trainer to log model's weights as a histogram + clearml_logger.attach(trainer, log_handler=WeightsHistHandler(model), event_name=Events.EPOCH_COMPLETED(every=100)) + + # Attach the logger to the trainer to log model’s gradients as scalars + clearml_logger.attach( + trainer, log_handler=GradsScalarHandler(model), event_name=Events.ITERATION_COMPLETED(every=100) + ) + + #Attach the logger to the trainer to log model's gradients as a histogram + clearml_logger.attach(trainer, log_handler=GradsHistHandler(model), event_name=Events.EPOCH_COMPLETED(every=100)) + + handler = Checkpoint( + {"model": model}, + ClearMLSaver(), + n_saved=1, + score_function=lambda e: e.state.metrics["accuracy"], + score_name="val_acc", + filename_prefix="best", + global_step_transform=global_step_from_engine(trainer), + ) + validation_evaluator.add_event_handler(Events.EPOCH_COMPLETED, handler) + + # Attach the logger to the trainer to log optimizer's parameters, e.g. learning rate at each iteration + clearml_logger.attach( + trainer, + log_handler=OptimizerParamsHandler(optimizer), + event_name=Events.ITERATION_STARTED + ) + ``` + +Visualize all the captured information in the experiment's page in ClearML's [WebApp](#webapp). + +For more information, see the [ignite documentation](https://pytorch.org/ignite/generated/ignite.contrib.handlers.clearml_logger.html). + +See code example [here](https://github.com/pytorch/ignite/blob/master/examples/contrib/mnist/mnist_with_clearml_logger.py) + +## WebApp + +All the experiment information that ClearML captures can be viewed in the [WebApp](../webapp/webapp_overview.md): + +### Models + +View saved model snapshots in the **ARTIFACTS** tab. + +![Model snapshots](../img/ignite_artifact.png) + +### Scalars + +View the scalars in the experiment's **SCALARS** tab. + +![Scalars](../img/examples_cifar_scalars.png) + + +### Debug Samples + +ClearML automatically tracks images logged to `TensorboardLogger`. They appear in the experiment's **DEBUG SAMPLES**. + + +![Debug Samples](../img/examples_integration_pytorch_ignite_debug.png) + diff --git a/docs/integrations/tensorflow.md b/docs/integrations/tensorflow.md new file mode 100644 index 00000000..2f8b6a4b --- /dev/null +++ b/docs/integrations/tensorflow.md @@ -0,0 +1,134 @@ +--- +title: TensorFlow +--- + +:::tip +If you are not already using ClearML, see [Getting Started](../getting_started/ds/ds_first_steps.md) for setup +instructions. +::: + +ClearML integrates with [TensorFlow](https://www.tensorflow.org/) out-of-the-box, automatically logging its models, +definitions, scalars, as well as TensorBoard outputs. + +All you have to do is simply add two lines of code to your TensorFlow script: + +```python +from clearml import Task +task = Task.init(task_name="", project_name="") +``` + +And that’s it! This creates a [ClearML Task](../fundamentals/task.md) which captures: +* Source code and uncommitted changes +* Installed packages +* TensorFlow definitions +* TensorFlow model files +* [TensorBoard](https://www.tensorflow.org/tensorboard) outputs (see example [here](https://clear.ml/docs/latest/docs/guides/frameworks/tensorflow/tensorboard_toy/)) +* Scalars (loss, learning rates) +* Console output +* General details such as machine details, runtime, creation date etc. +* And more + +You can view all the task details in the [WebApp](../webapp/webapp_overview.md). + +![WebApp Gif](../img/gif/tensorflow.gif) + +## Automatic Logging Control +By default, when ClearML is integrated into your TensorFlow script, it captures TensorFlow definitions, models, and +scalars. But, you may want to have more control over what your experiment logs. + +To control a task's framework logging, use the `auto_connect_frameworks` parameter of [`Task.init()`](../references/sdk/task.md#taskinit). +Completely disable all automatic logging by setting the parameter to `False`. For finer grained control of logged +frameworks, input a dictionary, with framework-boolean pairs. + +For example: + +```python +auto_connect_frameworks={ + 'matplotlib': True, 'tensorflow': False, 'tensorboard': False, 'pytorch': True, + 'xgboost': False, 'scikit': True, 'fastai': True, 'lightgbm': False, + 'hydra': True, 'detect_repository': True, 'tfdefines': True, 'joblib': True, + 'megengine': True, 'jsonargparse': True, 'catboost': True +} +``` + +You can also input wildcards as dictionary values, so ClearML will log a model created by a framework only if its local +path matches at least one wildcard. + +For example, in the code below, ClearML will log TensorFlow models only if their paths have the `.pt` extension. The +unspecified frameworks' values default to true so all their models are automatically logged. + +```python +auto_connect_frameworks={'tensorflow' : '*.pt'} +``` + +## Manual Logging +To augment its automatic logging, ClearML also provides an explicit logging interface. + +See more information about explicitly logging information to a ClearML Task: +* [Models](../clearml_sdk/model_sdk.md#manually-logging-models) +* [Configuration](../clearml_sdk/task_sdk.md#configuration) (e.g. parameters, configuration files) +* [Artifacts](../clearml_sdk/task_sdk.md#artifacts) (e.g. output files or python objects created by a task) +* [Scalars](../clearml_sdk/task_sdk.md#scalars) +* [Text/Plots/Debug Samples](../fundamentals/logger.md#manual-reporting) + +See [Explicit Reporting Tutorial](../guides/reporting/explicit_reporting.md). + +## Examples + +Take a look at ClearML’s TensorFlow examples. The examples use TensorFlow and ClearML in different configurations with +additional tools, like Abseil and TensorBoard: + +* [TensorFlow MNIST](../guides/frameworks/tensorflow/tensorflow_mnist.md) - Demonstrates ClearML's automatic logging of +model checkpoints, TensorFlow definitions, and scalars logged using TensorFlow methods +* [TensorBoard PR Curve](../guides/frameworks/tensorflow/tensorboard_pr_curve.md) - Demonstrates ClearML’s automatic +logging of TensorBoard output and TensorFlow definitions. +* [TensorBoard Toy](../guides/frameworks/tensorflow/tensorboard_toy.md) - Demonstrates ClearML’s automatic logging of +TensorBoard scalars, histograms, images, and text, as well as all console output and TensorFlow Definitions. +* [Absl flags](https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/absl_flags.py) - Demonstrates +ClearML’s automatic logging of parameters defined using `absl.flags` + +## Remote Execution +ClearML logs all the information required to reproduce an experiment on a different machine (installed packages, +uncommitted changes etc.). The [ClearML Agent](../clearml_agent) listens to designated queues and when a task is enqueued, +the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the +experiment manager. + +Deploy a ClearML Agent onto any machine (e.g. a cloud VM, a local GPU machine, your own laptop) by simply running the +following command on it: + +```commandline +clearml-agent daemon --queue [--docker] +``` + +Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md), to help you manage cloud workloads in the +cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up +and shuts down instances as needed, according to a resource budget that you set. + +### Cloning, Editing, and Enqueuing + +![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif) + +Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task +with the new configuration on a remote machine: + +* Clone the experiment +* Edit the hyperparameters and/or other details +* Enqueue the task + +The ClearML Agent executing the task will use the new values to [override any hard coded values](../clearml_agent). + +### Executing a Task Remotely + +You can set a task to be executed remotely programmatically by adding [`Task.execute_remotely()`](../references/sdk/task.md#execute_remotely) +to your script. This method stops the current local execution of the task, and then enqueues it to a specified queue to +re-run it on a remote machine. + +```python +# If executed locally, process will terminate, and a copy will be executed by an agent instead +task.execute_remotely(queue_name='default', exit_process=True) +``` + +## Hyperparameter Optimization +Use ClearML’s [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find +the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../fundamentals/hpo.md) +for more information. diff --git a/sidebars.js b/sidebars.js index 2e832a2c..d343a9c5 100644 --- a/sidebars.js +++ b/sidebars.js @@ -66,9 +66,9 @@ module.exports = { 'guides/frameworks/lightgbm/lightgbm_example', 'guides/frameworks/matplotlib/matplotlib_example', 'guides/frameworks/megengine/megengine_mnist', 'integrations/openmmv', 'integrations/optuna', 'integrations/python_fire', 'guides/frameworks/pytorch/pytorch_mnist', - {'PyTorch Ignite':['guides/frameworks/pytorch_ignite/integration_pytorch_ignite', 'guides/frameworks/pytorch_ignite/pytorch_ignite_mnist']}, + 'integrations/ignite', 'guides/frameworks/pytorch_lightning/pytorch_lightning_example', 'guides/frameworks/scikit-learn/sklearn_joblib_example', - 'guides/frameworks/pytorch/pytorch_tensorboard', 'guides/frameworks/tensorboardx/tensorboardx', 'guides/frameworks/tensorflow/tensorflow_mnist', + 'guides/frameworks/pytorch/pytorch_tensorboard', 'guides/frameworks/tensorboardx/tensorboardx', 'integrations/tensorflow', 'integrations/seaborn', 'integrations/xgboost', 'integrations/yolov5', 'integrations/yolov8' ] },