From ddb2d2e9a07b47aeb62536b9b227bcb8fcfb8995 Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Thu, 10 Aug 2023 11:49:05 +0300 Subject: [PATCH 1/3] Rewrite Keras integration page (#641) --- .../frameworks/keras/keras_tensorboard.md | 3 +- docs/integrations/keras.md | 132 ++++++++++++++++++ sidebars.js | 2 +- 3 files changed, 134 insertions(+), 3 deletions(-) create mode 100644 docs/integrations/keras.md diff --git a/docs/guides/frameworks/keras/keras_tensorboard.md b/docs/guides/frameworks/keras/keras_tensorboard.md index 83989f8c..0e5c7ae1 100644 --- a/docs/guides/frameworks/keras/keras_tensorboard.md +++ b/docs/guides/frameworks/keras/keras_tensorboard.md @@ -1,6 +1,5 @@ --- -title: Keras -displayed_sidebar: mainSidebar +title: Keras with TensorBoard --- The example below demonstrates the integration of ClearML into code which uses Keras and TensorBoard. diff --git a/docs/integrations/keras.md b/docs/integrations/keras.md new file mode 100644 index 00000000..19a8fe6b --- /dev/null +++ b/docs/integrations/keras.md @@ -0,0 +1,132 @@ +--- +title: Keras +--- + +:::tip +If you are not already using ClearML, see [Getting Started](../getting_started/ds/ds_first_steps.md) for setup +instructions. +::: + +ClearML integrates with [Keras](https://keras.io/) out-of-the-box, automatically logging its models, scalars, +TensorFlow definitions, and TensorBoard outputs. + +All you have to do is simply add two lines of code to your Keras script: + +```python +from clearml import Task +task = Task.init(task_name="", project_name="") +``` + +And that’s it! This creates a [ClearML Task](../fundamentals/task.md) which captures: +* Source code and uncommitted changes +* Installed packages +* Keras models +* Scalars (e.g. accuracy, loss) +* TensorFlow definitions +* [TensorBoard](https://www.tensorflow.org/tensorboard) outputs +* Console output +* General details such as machine details, runtime, creation date etc. +* And more + +You can view all the task details in the [WebApp](../webapp/webapp_exp_track_visual.md). + +![WebApp Gif](../img/gif/tensorflow.gif) + +## Automatic Logging Control +By default, when ClearML is integrated into your Keras script, it captures Keras models and scalars, as well as TensorFlow +definitions and TensorBoard outputs. But, you may want to have more control over what your experiment logs. + +To control a task's framework logging, use the `auto_connect_frameworks` parameter of [`Task.init()`](../references/sdk/task.md#taskinit). +Completely disable all automatic logging by setting the parameter to `False`. For finer grained control of logged +frameworks, input a dictionary, with framework-boolean pairs. + +For example: + +```python +auto_connect_frameworks={ + 'tensorflow': False, 'tensorboard': False, 'matplotlib': True, 'pytorch': True, + 'xgboost': False, 'scikit': True, 'fastai': True, 'lightgbm': False, + 'hydra': True, 'detect_repository': True, 'tfdefines': True, 'joblib': True, + 'megengine': True, 'jsonargparse': True, 'catboost': True +} +``` + +To control Keras logging, use the `tensorflow` and `tensorboard` keys. + +You can also input wildcards as dictionary values, so ClearML will log a model created by a framework only if its local +path matches at least one wildcard. + +For example, in the code below, ClearML will log TensorFlow (and/or keras) models only if their paths have the `.keras` extension. The +unspecified frameworks' values default to true so all their models are automatically logged. + +```python +auto_connect_frameworks={'tensorflow' : '*.keras'} +``` + +## Manual Logging +To augment its automatic logging, ClearML also provides an explicit logging interface. + +See more information about explicitly logging information to a ClearML Task: +* [Models](../clearml_sdk/model_sdk.md#manually-logging-models) +* [Configuration](../clearml_sdk/task_sdk.md#configuration) (e.g. parameters, configuration files) +* [Artifacts](../clearml_sdk/task_sdk.md#artifacts) (e.g. output files or python objects created by a task) +* [Scalars](../clearml_sdk/task_sdk.md#scalars) +* [Text/Plots/Debug Samples](../fundamentals/logger.md#manual-reporting) + +See [Explicit Reporting Tutorial](../guides/reporting/explicit_reporting.md). + +## Examples + +Take a look at ClearML’s Keras examples. The examples use Keras and ClearML in different configurations with +additional tools like TensorBoard and Matplotlib: +* [Keras with Tensorboard](../guides/frameworks/keras/keras_tensorboard.md) - Demonstrates ClearML logging a Keras model, +and plots and scalars logged to TensorBoard +* [Keras with Matplotlib](../guides/frameworks/keras/jupyter.md) - Demonstrates ClearML logging a Keras model, Matplotlib plots, +and debug samples, plots, and scalars logged to TensorBoard + + +## Remote Execution +ClearML logs all the information required to reproduce an experiment on a different machine (installed packages, +uncommitted changes etc.). The [ClearML Agent](../clearml_agent) listens to designated queues and when a task is enqueued, +the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the +experiment manager. + +Deploy a ClearML Agent onto any machine (e.g. a cloud VM, a local GPU machine, your own laptop) by simply running the +following command on it: + +```commandline +clearml-agent daemon --queue [--docker] +``` + +Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md), to help you manage cloud workloads in the +cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up +and shuts down instances as needed, according to a resource budget that you set. + +### Cloning, Editing, and Enqueuing + +![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif) + +Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task +with the new configuration on a remote machine: + +* Clone the experiment +* Edit the hyperparameters and/or other details +* Enqueue the task + +The ClearML Agent executing the task will use the new values to [override any hard coded values](../clearml_agent). + +### Executing a Task Remotely + +You can set a task to be executed remotely programmatically by adding [`Task.execute_remotely()`](../references/sdk/task.md#execute_remotely) +to your script. This method stops the current local execution of the task, and then enqueues it to a specified queue to +re-run it on a remote machine. + +```python +# If executed locally, process will terminate, and a copy will be executed by an agent instead +task.execute_remotely(queue_name='default', exit_process=True) +``` + +## Hyperparameter Optimization +Use ClearML’s [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find +the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../fundamentals/hpo.md) +for more information. diff --git a/sidebars.js b/sidebars.js index f2ab616a..c5629d69 100644 --- a/sidebars.js +++ b/sidebars.js @@ -62,7 +62,7 @@ module.exports = { 'guides/frameworks/autokeras/integration_autokeras', 'integrations/catboost', 'integrations/click', 'guides/frameworks/fastai/fastai_with_tensorboard', 'integrations/hydra', - 'guides/frameworks/keras/keras_tensorboard', 'guides/frameworks/tensorflow/integration_keras_tuner', + 'integrations/keras', 'guides/frameworks/tensorflow/integration_keras_tuner', 'integrations/lightgbm', 'integrations/matplotlib', 'integrations/megengine', 'integrations/openmmv', 'integrations/optuna', 'integrations/python_fire', 'integrations/pytorch', From 1d5647ebb18fe310327f6ae2c4996e11596009c6 Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Thu, 10 Aug 2023 11:49:58 +0300 Subject: [PATCH 2/3] Update AWS autoscaler conf options (#643) --- docs/webapp/applications/apps_aws_autoscaler.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/webapp/applications/apps_aws_autoscaler.md b/docs/webapp/applications/apps_aws_autoscaler.md index 961e4b75..9829c7e6 100644 --- a/docs/webapp/applications/apps_aws_autoscaler.md +++ b/docs/webapp/applications/apps_aws_autoscaler.md @@ -42,6 +42,7 @@ For more information about how autoscalers work, see [Autoscalers Overview](../. * **Compute Resources** * Resource Name - Assign a name to the resource type. This name will appear in the Autoscaler dashboard * EC2 Instance Type - See [Instance Types](https://aws.amazon.com/ec2/instance-types) for full list of types + * Run in CPU mode - Check box to run with CPU only * Use Spot Instance - Check box to use a spot instance. Else, a reserved instance is used * Regular Instance Rollback Timeout - Controls when the autoscaler will revert to starting a regular instance after failing to start a spot instance. It will first attempt to start a spot, and then wait and retry again and again. Once the time it waited exceeded the Regular Instance Rollback Timeout, the autoscaler will try to start a regular instance instead. This is for a specific attempt, where starting a spot fails and an alternative instance needs to be started. * Spot Instance Blackout Period - Specifies a blackout period after failing to start a spot instance. This is related to future attempts: after failing to start a spot instance, all requests to start additional spot instances will be converted to attempts to start regular instances, as a way of "easing" the spot requests load on the cloud provider and not creating a "DOS" situation in the cloud account which might cause the provider to refuse creating spots for a longer period. @@ -55,13 +56,14 @@ For more information about how autoscalers work, see [Autoscalers Overview](../. commas * EBS Device (optional) - Disk mount point * EBS Volume Size (optional) - Disk size (GB) - * EBS Volume Type (optional) - See [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) + * EBS Volume Type (optional) - See [Amazon EBS volume types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) for full list of types * Instance Key Pair (optional) - AWS key pair that is provided to the spun EC2 instances for connecting to them via - SSH. Provide the Key Pair's name, as was created in AWS. See [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) + SSH. Provide the Key Pair's name, as was created in AWS. See [Amazon EC2 key pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) for more details * Security Group ID (optional) - Comma separated list of AWS VPC Security Group IDs to attach to the launched instance. Read more [here](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) + * VPC Subnet ID - The subnet ID For the created instance. If more than one ID is provided, the instance will be started in the first available subnet. For more information, see [What is Amazon VPC?](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) * \+ Add Item - Define another resource type * **IAM Instance Profile** (optional) - Set an IAM instance profile for all instances spun by the Autoscaler * Arn - Amazon Resource Name specifying the instance profile From 3c8a31b61f1e355b16d0dac0907fb7c74248ef7b Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Thu, 10 Aug 2023 11:55:09 +0300 Subject: [PATCH 3/3] Rewrite AutoKeras integration page (#642) --- .../autokeras/autokeras_imdb_example.md | 2 +- .../autokeras/integration_autokeras.md | 42 ------- docs/integrations/autokeras.md | 117 ++++++++++++++++++ sidebars.js | 4 +- 4 files changed, 120 insertions(+), 45 deletions(-) delete mode 100644 docs/guides/frameworks/autokeras/integration_autokeras.md create mode 100644 docs/integrations/autokeras.md diff --git a/docs/guides/frameworks/autokeras/autokeras_imdb_example.md b/docs/guides/frameworks/autokeras/autokeras_imdb_example.md index 8111f86a..49f5341b 100644 --- a/docs/guides/frameworks/autokeras/autokeras_imdb_example.md +++ b/docs/guides/frameworks/autokeras/autokeras_imdb_example.md @@ -1,5 +1,5 @@ --- -title: AutoKeras IMDB +title: AutoKeras --- The [autokeras_imdb_example.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/autokeras/autokeras_imdb_example.py) example script demonstrates the integration of ClearML into code, which uses [autokeras](https://github.com/keras-team/autokeras). diff --git a/docs/guides/frameworks/autokeras/integration_autokeras.md b/docs/guides/frameworks/autokeras/integration_autokeras.md deleted file mode 100644 index 666e94c8..00000000 --- a/docs/guides/frameworks/autokeras/integration_autokeras.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: AutoKeras -displayed_sidebar: mainSidebar ---- -Integrate ClearML into code that uses [autokeras](https://github.com/keras-team/autokeras). Initialize a ClearML -Task in a code, and ClearML automatically logs scalars, plots, and images reported to TensorBoard, Matplotlib, Plotly, -and Seaborn, and all other automatic logging, and explicit reporting added to the code (see [Logging](../../../fundamentals/logger.md)). - -ClearML allows to: - -* Visualize experiment results in the **ClearML Web UI**. -* Track and upload models. -* Track model performance and create tracking leaderboards. -* Rerun experiments, reproduce experiments on any target machine, and tune experiments. -* Compare experiments. - -See the [AutoKeras](autokeras_imdb_example.md) example, which shows ClearML automatically logging: -* Scalars -* Hyperparameters -* The console log -* Models. - -Once these are logged, they can be visualized in the **ClearML Web UI**. - -:::note -If you are not already using ClearML, see [Getting Started](/getting_started/ds/best_practices.md). -::: - -## Adding ClearML to Code - -Add two lines of code: -```python -from clearml import Task -task = Task.init(project_name="myProject", task_name="myExperiment") -``` - -When the code runs, it initializes a Task in ClearML Server. A hyperlink to the experiment's log is output to the console. - - CLEARML Task: created new task id=c1f1dc6cf2ee4ec88cd1f6184344ca4e - CLEARML results page: https://app.clear.ml/projects/1c7a45633c554b8294fa6dcc3b1f2d4d/experiments/c1f1dc6cf2ee4ec88cd1f6184344ca4e/output/log - -Later in the code, define callbacks using TensorBoard, and ClearML logs TensorBoard scalars, histograms, and images. \ No newline at end of file diff --git a/docs/integrations/autokeras.md b/docs/integrations/autokeras.md new file mode 100644 index 00000000..eac70e08 --- /dev/null +++ b/docs/integrations/autokeras.md @@ -0,0 +1,117 @@ +--- +title: AutoKeras +--- + +:::tip +If you are not already using ClearML, see [Getting Started](../getting_started/ds/ds_first_steps.md) for setup +instructions. +::: + +ClearML integrates seamlessly with [AutoKeras](https://autokeras.com/), automatically logging its models and scalars. + +All you have to do is simply add two lines of code to your AutoKeras script: + +```python +from clearml import Task +task = Task.init(task_name="", project_name="") +``` + +And that’s it! This creates a [ClearML Task](../fundamentals/task.md) which captures: +* Source code and uncommitted changes +* Installed packages +* AutoKeras model files +* Scalars (loss, learning rates) +* Console output +* General details such as machine details, runtime, creation date etc. +* Hyperparameters created with standard python packages (e.g. argparse, click, Python Fire, etc.) +* And more + +You can view all the task details in the [WebApp](../webapp/webapp_exp_track_visual.md). + +See an example of AutoKeras and ClearML in action [here](../guides/frameworks/autokeras/autokeras_imdb_example.md). + +![Experiment scalars](../img/examples_keras_14.png) + +## Automatic Logging Control +By default, when ClearML is integrated into your AutoKeras script, it captures AutoKeras models and scalars, as well as TensorFlow +definitions and TensorBoard outputs. But, you may want to have more control over what your experiment logs. + +To control a task's framework logging, use the `auto_connect_frameworks` parameter of [`Task.init()`](../references/sdk/task.md#taskinit). +Completely disable all automatic logging by setting the parameter to `False`. For finer grained control of logged +frameworks, input a dictionary, with framework-boolean pairs. + +For example: + +```python +auto_connect_frameworks={ + 'tensorflow': False, 'tensorboard': False, 'pytorch': True, + 'xgboost': False, 'scikit': True, 'fastai': True, 'lightgbm': False, + 'hydra': True, 'detect_repository': True, 'tfdefines': True, 'joblib': True, + 'megengine': True, 'jsonargparse': True, 'catboost': False +} +``` + +To control AutoKeras logging, use the `tensorflow` and `tensorboard` keys. + +You can also input wildcards as dictionary values, so ClearML will log a model created by a framework only if its local +path matches at least one wildcard. + +For example, in the code below, ClearML will log TensorFlow models only if their paths have the `.h5` extension. The +unspecified frameworks' values default to true so all their models are automatically logged. + +```python +auto_connect_frameworks={'tensorflow' : '*.h5'} +``` + +## Manual Logging +To augment its automatic logging, ClearML also provides an explicit logging interface. + +See more information about explicitly logging information to a ClearML Task: +* [Models](../clearml_sdk/model_sdk.md#manually-logging-models) +* [Configuration](../clearml_sdk/task_sdk.md#configuration) (e.g. parameters, configuration files) +* [Artifacts](../clearml_sdk/task_sdk.md#artifacts) (e.g. output files or python objects created by a task) +* [Scalars](../clearml_sdk/task_sdk.md#scalars) +* [Text/Plots/Debug Samples](../fundamentals/logger.md#manual-reporting) + +See [Explicit Reporting Tutorial](../guides/reporting/explicit_reporting.md). + +## Remote Execution +ClearML logs all the information required to reproduce an experiment on a different machine (installed packages, +uncommitted changes etc.). The [ClearML Agent](../clearml_agent) listens to designated queues and when a task is enqueued, +the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the +experiment manager. + +Deploy a ClearML Agent onto any machine (e.g. a cloud VM, a local GPU machine, your own laptop) by simply running the +following command on it: + +```commandline +clearml-agent daemon --queue [--docker] +``` + +Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md), to help you manage cloud workloads in the +cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up +and shuts down instances as needed, according to a resource budget that you set. + +### Cloning, Editing, and Enqueuing + +![Cloning, editing, enqueuing gif](../img/gif/integrations_yolov5.gif) + +Use ClearML's web interface to edit task details, like configuration parameters or input models, then execute the task +with the new configuration on a remote machine: + +* Clone the experiment +* Edit the hyperparameters and/or other details +* Enqueue the task + +The ClearML Agent executing the task will use the new values to [override any hard coded values](../clearml_agent). + +### Executing a Task Remotely + +You can set a task to be executed remotely programmatically by adding [`Task.execute_remotely()`](../references/sdk/task.md#execute_remotely) +to your script. This method stops the current local execution of the task, and then enqueues it to a specified queue to +re-run it on a remote machine. + +```python +# If executed locally, process will terminate, and a copy will be executed by an agent instead +task.execute_remotely(queue_name='default', exit_process=True) +``` diff --git a/sidebars.js b/sidebars.js index c5629d69..17ddfd2b 100644 --- a/sidebars.js +++ b/sidebars.js @@ -59,7 +59,7 @@ module.exports = { {'ClearML Serving':['clearml_serving/clearml_serving', 'clearml_serving/clearml_serving_setup', 'clearml_serving/clearml_serving_cli', 'clearml_serving/clearml_serving_tutorial']}, {'CLI Tools': ['apps/clearml_session', 'apps/clearml_task', 'apps/clearml_param_search']}, {'Integrations': [ - 'guides/frameworks/autokeras/integration_autokeras', + 'integrations/autokeras', 'integrations/catboost', 'integrations/click', 'guides/frameworks/fastai/fastai_with_tensorboard', 'integrations/hydra', 'integrations/keras', 'guides/frameworks/tensorflow/integration_keras_tuner', @@ -147,7 +147,7 @@ module.exports = { {'Distributed': ['guides/distributed/distributed_pytorch_example', 'guides/distributed/subprocess_example']}, {'Docker': ['guides/docker/extra_docker_shell_script']}, {'Frameworks': [ - {'Autokeras': ['guides/frameworks/autokeras/integration_autokeras', 'guides/frameworks/autokeras/autokeras_imdb_example']}, + 'guides/frameworks/autokeras/autokeras_imdb_example', 'guides/frameworks/catboost/catboost', 'guides/frameworks/fastai/fastai_with_tensorboard', {'Keras': ['guides/frameworks/keras/jupyter', 'guides/frameworks/keras/keras_tensorboard']},