diff --git a/docs/fundamentals/agents_and_queues.md b/docs/fundamentals/agents_and_queues.md index 6deff963..e202d315 100644 --- a/docs/fundamentals/agents_and_queues.md +++ b/docs/fundamentals/agents_and_queues.md @@ -95,9 +95,9 @@ Some tasks, mainly control (Like a pipeline controller) or services (Like an arc This is where the `services-modes` comes into play. An agent running in services-mode will spin multiple tasks at the same time, each Task will register itself as a sub-agent (visible in the workers Tab in the UI). Some examples for suitable tasks are: -- [Pipeline controller](https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_controller.py) - Implementing the pipeline scheduling and logic -- [Hyper-Parameter Optimization](https://github.com/allegroai/clearml/blob/master/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py) - Implementing an active selection of experiments -- [Control Service](https://github.com/allegroai/clearml/blob/master/examples/services/aws-autoscaler/aws_autoscaler.py) - AWS Autoscaler for example -- [External services](https://github.com/allegroai/clearml/blob/master/examples/services/monitoring/slack_alerts.py) - Such as Slack integration alert service +- [Pipeline controller](../guides/pipeline/pipeline_controller.md) - Implementing the pipeline scheduling and logic +- [Hyper-Parameter Optimization](../guides/optimization/hyper-parameter-optimization/examples_hyperparam_opt.md) - Implementing an active selection of experiments +- [Control Service](../guides/services/aws_autoscaler.md) - AWS Autoscaler for example +- [External services](../guides/services/slack_alerts.md) - Such as Slack integration alert service By default, [ClearML Server](../deploying_clearml/clearml_server.md) comes with an Agent running on the machine that runs it. It also comes with a Services queue. diff --git a/docs/fundamentals/artifacts.md b/docs/fundamentals/artifacts.md index 9ba0de5c..276990cb 100644 --- a/docs/fundamentals/artifacts.md +++ b/docs/fundamentals/artifacts.md @@ -76,10 +76,14 @@ logs the models and all snapshot paths. ![image](../img/fundamentals_artifacts_logging_models.png) -See model storage examples, [TF](https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorflow_mnist.py), -[PyTorch](https://github.com/allegroai/clearml/blob/master/examples/frameworks/pytorch/pytorch_mnist.py), -[Keras](https://github.com/allegroai/clearml/blob/master/examples/frameworks/keras/keras_tensorboard.py), -[Scikit-Learn](https://github.com/allegroai/clearml/blob/master/examples/frameworks/scikit-learn/sklearn_joblib_example.py). +See automatic model logging examples: +* [TF](../guides/frameworks/tensorflow/tensorflow_mnist.md) +* [PyTorch](../guides/frameworks/pytorch/pytorch_mnist.md) +* [Keras](../guides/frameworks/keras/keras_tensorboard.md) +* [Scikit-Learn](../guides/frameworks/scikit-learn/sklearn_joblib_example.md) +* [XGBoost](../guides/frameworks/xgboost/xgboost_sample.md) +* [FastAI](../guides/frameworks/fastai/fastai_with_tensorboard.md) + ### Manual Model Logging @@ -121,6 +125,7 @@ output_model.update_weights() for model weight upload (`registered_uri`). * Model Metadata - Model description and iteration number. +See [Model Configuration](../guides/reporting/model_config.md) example. ### Using Models diff --git a/docs/fundamentals/hpo.md b/docs/fundamentals/hpo.md index 2668f4d1..ba042d9b 100644 --- a/docs/fundamentals/hpo.md +++ b/docs/fundamentals/hpo.md @@ -57,7 +57,7 @@ optimization. * **BOHB** - `automation.hpbandster.bandster.OptimizerBOHB`. BOHB performs robust and efficient hyperparameter optimization at scale by combining the speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian Optimization. For more information about HpBandSter BOHB, see the [HpBandSter](https://automl.github.io/HpBandSter/build/html/index.html) - documentation. + documentation and a [code example](../guides/frameworks/pytorch/notebooks/image/hyperparameter_search.md). * **Random** uniform sampling of hyperparameters - `automation.optimization.RandomSearch`. * **Full grid** sampling strategy of every hyperparameter combination - `Grid search automation.optimization.GridSearch`. * **Custom** - `automation.optimization.SearchStrategy` - Use a custom class and inherit from the ClearML automation base strategy class diff --git a/docs/fundamentals/hyperparameters.md b/docs/fundamentals/hyperparameters.md index bba2dc11..528b7317 100644 --- a/docs/fundamentals/hyperparameters.md +++ b/docs/fundamentals/hyperparameters.md @@ -34,6 +34,8 @@ args = parser.parse_args() task = Task.init(project_name="examples",task_name="argparser logging") ``` +See another argparse logging example [here](../guides/reporting/hyper_parameters.md). + ### Click Example ```python @@ -55,7 +57,6 @@ hello() See another code example [here](https://github.com/allegroai/clearml/blob/master/examples/frameworks/click/click_multi_cmd.py). - ## Connecting Objects Users can directly connect objects, such as dictionaries or even custom classes, to Tasks. @@ -75,6 +76,8 @@ task = Task.init(project_name='examples',task_name='argparser') task.connect(me) ``` +See connecting configuration objects example [here](../guides/reporting/hyper_parameters.md). + * Connecting a dictionary: ```python @@ -121,7 +124,11 @@ The CLEARML_LOG_ENVIRONMENT always overrides the clearml.conf file. ## TF Defines -ClearML automatically captures TFDefine files, which are used as configuration files for Tensorflow. +ClearML automatically captures TensorFlow definitions, which are used as configuration files for Tensorflow. + +See examples of ClearML's automatic logging of TF Defines: +* [TensorFlow MNIST](../guides/frameworks/tensorflow/tensorflow_mnist.md) +* [TensorBoard PR Curve](../guides/frameworks/tensorflow/tensorboard_pr_curve.md) ## Hydra diff --git a/docs/fundamentals/logger.md b/docs/fundamentals/logger.md index a1237279..dce5e6d1 100644 --- a/docs/fundamentals/logger.md +++ b/docs/fundamentals/logger.md @@ -18,7 +18,7 @@ In ClearML, there are four types of reports: ## Automatic Reporting -ClearML automatically captures metrics reported to tools, such as Tensorboard and Matplotlib, with no additional code +ClearML automatically captures metrics reported to tools, such as TensorBoard and Matplotlib, with no additional code necessary. In addition, ClearML will capture and log everything written to standard output, from debug messages to errors to @@ -28,10 +28,28 @@ GPU, CPU, Memory and Network information is also automatically captured. ![image](../img/fundamentals_logger_cpu_monitoring.png) -### Supported packages -- [Tensorboard](https://www.tensorflow.org/tensorboard) -- [TensorboardX](https://github.com/lanpa/tensorboardX) -- [matplotlib](https://matplotlib.org/) +### Supported Packages +- [TensorBoard](https://www.tensorflow.org/tensorboard) +- [TensorBoardX](https://github.com/lanpa/tensorboardX) +- [Matplotlib](https://matplotlib.org/) + +### Automatic Reporting Examples +Check out some of ClearML's automatic reporting examples for supported packages: +* TensorBoard + * [TensorBoard PR Curve](../guides/frameworks/tensorflow/tensorboard_pr_curve.md) - logging TensorBoard outputs and + TensorFlow flags + * [TensorBoard Toy](../guides/frameworks/tensorflow/tensorboard_toy.md) - logging TensorBoard histograms, scalars, images, text, and + TensorFlow flags + * [Tensorboard with PyTorch](../guides/frameworks/pytorch/pytorch_tensorboard.md) - logging TensorBoard scalars, debug samples, and text integrated into + code that uses PyTorch +* [TensorBoardX](../guides/frameworks/tensorboardx/tensorboardx.md) - logging TensorBoardX scalars, debug + samples, and text in code using PyTorch +* Matplotlib + * [Matplotlib Script Example](../guides/frameworks/matplotlib/matplotlib_example.md) and [Jupyter Notebook](../guides/frameworks/matplotlib/allegro_clearml_matplotlib_example.md) - + logging scatter diagrams plotted with Matplotlib + * [Matplotlib with PyTorch](../guides/frameworks/pytorch/pytorch_matplotlib.md) - logging debug images shown + by Matplotlib + ## Manual Reporting @@ -46,9 +64,7 @@ The object used for reporting metrics is called **logger** and is obtained by ca logger = task.get_logger() ``` -Check out all the available object types that can be reported in the example [here](../guides/reporting/scalar_reporting.md). - -#### Media reporting +### Media Reporting ClearML also supports reporting media (such as audio, video and images) for every iteration. This section is mostly used for debugging. It's recommended to use [artifacts](artifacts.md#artifacts) for storing script @@ -59,4 +75,26 @@ See details in [Logger.report_media](../references/sdk/logger.md#report_media). ![image](../img/fundamentals_logger_reported_images.png) -Check out the Media Reporting [example](../guides/reporting/media_reporting). \ No newline at end of file +### Explicit Reporting Examples + +Check out ClearML's explicit reporting examples for various types of results: +- [Text](../guides/reporting/text_reporting.md) +- [Scalars](../guides/reporting/scalar_reporting.md) +- Plots + - [2d plots](../guides/reporting/scatter_hist_confusion_mat_reporting.md) + - Histograms + - Confusion matrices + - Scatter plots + - [3d plots](../guides/reporting/3d_plots_reporting.md) + - Surface plots + - Scatter plots + - [Tables](../guides/reporting/pandas_reporting.md) + - Pandas DataFrames + - CSV file + - [Matplotlib figures](../guides/reporting/manual_matplotlib_reporting.md) + - [Plotly figures](../guides/reporting/plotly_reporting.md) +- Debug Samples + - [Images](../guides/reporting/image_reporting.md) + - [HTML](../guides/reporting/html_reporting.md) + - [Media - images, audio, video](../guides/reporting/media_reporting.md) +- Explicit reporting in Jupyter Notebook [example](../guides/reporting/clearml_logging_example.md) diff --git a/docs/fundamentals/pipelines.md b/docs/fundamentals/pipelines.md index 3a14852e..c1293b25 100644 --- a/docs/fundamentals/pipelines.md +++ b/docs/fundamentals/pipelines.md @@ -66,4 +66,8 @@ Custom pipelines usually involve cloning template tasks, modifying their paramet them to queues (for execution by [agents](../clearml_agent.md)). It's possible to create custom logic that controls inputs (e.g. overriding hyperparameters and artifacts) and acts upon task outputs. -See an example of a custom pipeline [here](../guides/automation/task_piping.md). +See examples of custom pipelines: + * [Task Piping](../guides/automation/task_piping.md) + * [Manual Random Parameter Search](../guides/automation/manual_random_param_search_example.md) + + diff --git a/docs/guides/storage/examples_storagehelper.md b/docs/guides/storage/examples_storagehelper.md index 9075b859..aab58efa 100644 --- a/docs/guides/storage/examples_storagehelper.md +++ b/docs/guides/storage/examples_storagehelper.md @@ -5,9 +5,9 @@ title: Storage Examples This page describes storage examples using the [StorageManager](../../references/sdk/storage.md) class. The storage examples include: -* [Downloading a file](#downloading_storagemanager) - Get an object from storage. -* [Uploading a file](#uploading_storagemanager) - Upload an object. -* [Setting cache limits](#cache) - Set the maximum number of objects. +* [Downloading a file](#downloading-a-file) - Get an object from storage. +* [Uploading a file](#uploading-a-file) - Upload an object. +* [Setting cache limits](#setting-cache-limits) - Set the maximum number of objects. :::note `StorageManager` supports http(s), S3, Google Cloud Storage, Azure, and file system folders. @@ -26,6 +26,10 @@ method, and specify the destination location as the `remote_url` argument: manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.zip") +:::note +Zip and tar.gz files will be automatically extracted to cache. This can be controlled with the`extract_archive` flag. +::: + To download a file to a specific context in cache, specify the name of the context as the `cache_context` argument: manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", cache_context="test") @@ -34,7 +38,6 @@ To download a non-compressed file, set the `extract_archive` argument to `False` manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", extract_archive=False) - ### Uploading a file diff --git a/docs/integrations/storage.md b/docs/integrations/storage.md index b4a312ca..4996e7d8 100644 --- a/docs/integrations/storage.md +++ b/docs/integrations/storage.md @@ -99,31 +99,11 @@ It's also possible to specify credentials for a specific bucket. ## Storage Manager -ClearML Offers a package to manage downloading, uploading and caching of content directly from code. +ClearML offers the [StorageManager](../references/sdk/storage.md) class to manage downloading, uploading, and caching of +content directly from code. -### Uploading files -To upload a file using storage manager, just run the following line specifying the path to a local file or folder, and the -remote destination. -```python -from clearml import StorageManager +See [Storage Examples](../guides/storage/examples_storagehelper.md). -StorageManager.upload_file(local_file='path_to_file',remote_url='s3://my_bucket') -``` - - -### Downloading files -To download files into cache, run the following line, specifying the remote destination's URL. -```python -StorageManager.get_local_copy(remote_url='s3://my_bucket/path_to_file') -``` - -:::note -Zip and tar.gz files will be automatically extracted to cache. This can be controlled with the`extract_archive` flag. -::: - -### Controling cache file limit -It's possible to control the maximum cache size by limiting the number of files it stores. -This is done by calling the ```StorageManager.set_cache_file_limit()``` method. ## Caching ClearML also manages a cache of all downloaded content so nothing is duplicated, and code won't need to download the same diff --git a/sidebars.js b/sidebars.js index fb57f75b..947b68c9 100644 --- a/sidebars.js +++ b/sidebars.js @@ -87,7 +87,7 @@ module.exports = { ] }, {'Scikit-Learn': ['guides/frameworks/scikit-learn/sklearn_joblib_example', 'guides/frameworks/scikit-learn/sklearn_matplotlib_example']}, - {'TensorboardX': ['guides/frameworks/tensorboardx/tensorboardx']}, + {'TensorBoardX': ['guides/frameworks/tensorboardx/tensorboardx']}, { 'Tensorflow': ['guides/frameworks/tensorflow/tensorboard_pr_curve', 'guides/frameworks/tensorflow/tensorboard_toy', 'guides/frameworks/tensorflow/tensorflow_mnist', 'guides/frameworks/tensorflow/integration_keras_tuner']