Small edits (#177)

This commit is contained in:
pollfly 2022-01-30 12:54:40 +02:00 committed by GitHub
parent 1667d6b64e
commit 020cf9a819
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
13 changed files with 35 additions and 47 deletions

View File

@ -128,7 +128,7 @@ Install ClearML Agent as a system Python package and not in a Python virtual env
* Mac - `$HOME/clearml.conf` * Mac - `$HOME/clearml.conf`
* Windows - `\User\<username>\clearml.conf` * Windows - `\User\<username>\clearml.conf`
1. Optionally, configure **ClearML** options for **ClearML Agent** (default docker, package manager, etc.). See the [ClearML Configuration Reference](configs/clearml_conf.md). 1. Optionally, configure ClearML options for **ClearML Agent** (default docker, package manager, etc.). See the [ClearML Configuration Reference](configs/clearml_conf.md).
:::note :::note
The ClearML Enterprise server provides a [configuration vault](webapp/webapp_profile.md#configuration-vault), the contents The ClearML Enterprise server provides a [configuration vault](webapp/webapp_profile.md#configuration-vault), the contents
@ -142,7 +142,7 @@ In case a `clearml.conf` file already exists, add a few ClearML Agent specific c
**Adding ClearML Agent to a ClearML configuration file:** **Adding ClearML Agent to a ClearML configuration file:**
1. Open the **ClearML** configuration file for editing. Depending upon the operating system, it is: 1. Open the ClearML configuration file for editing. Depending upon the operating system, it is:
* Linux - `~/clearml.conf` * Linux - `~/clearml.conf`
* Mac - `$HOME/clearml.conf` * Mac - `$HOME/clearml.conf`
* Windows - `\User\<username>\clearml.conf` * Windows - `\User\<username>\clearml.conf`
@ -384,7 +384,7 @@ A single agent can listen to multiple queues. The priority is set by their order
```bash ```bash
clearml-agent daemon --detached --queue high_q low_q --gpus 0 clearml-agent daemon --detached --queue high_q low_q --gpus 0
``` ```
This ensures the agent first tries to pull a Task from the `hiqh_q` queue, and only if it is empty, the agent will try to pull This ensures the agent first tries to pull a Task from the `high_q` queue, and only if it is empty, the agent will try to pull
from the `low_q` queue. from the `low_q` queue.
To make sure an agent pulls from all queues equally, add the `--order-fairness` flag. To make sure an agent pulls from all queues equally, add the `--order-fairness` flag.
@ -392,7 +392,7 @@ To make sure an agent pulls from all queues equally, add the `--order-fairness`
clearml-agent daemon --detached --queue group_a group_b --order-fairness --gpus 0 clearml-agent daemon --detached --queue group_a group_b --order-fairness --gpus 0
``` ```
It will make sure the agent will pull from the `group_a` queue, then from `group_b`, then back to `group_a`, etc. This ensures It will make sure the agent will pull from the `group_a` queue, then from `group_b`, then back to `group_a`, etc. This ensures
that `group A` or `group_b` will not be able to starve one another of resources. that `group_a` or `group_b` will not be able to starve one another of resources.
### Explicit Task Execution ### Explicit Task Execution
@ -468,7 +468,7 @@ When executing the ClearML Agent in Docker mode, it will:
ClearML Agent uses the provided default Docker container, which can be overridden from the UI. ClearML Agent uses the provided default Docker container, which can be overridden from the UI.
All ClearML Agent flags (Such as `--gpus` and `--foreground`) are applicable to Docker mode as well. All ClearML Agent flags (such as `--gpus` and `--foreground`) are applicable to Docker mode as well.
To execute ClearML Agent in Docker mode, run: To execute ClearML Agent in Docker mode, run:
```bash ```bash

View File

@ -14,15 +14,15 @@ and in the AWS community AMI catalog.
* [ClearML Server community AMIs](#clearml-server-aws-community-amis) - Configured by default without authentication to allow quick access and onboarding. * [ClearML Server community AMIs](#clearml-server-aws-community-amis) - Configured by default without authentication to allow quick access and onboarding.
After deploying either type of AMI, configure the **ClearML Server** instance to provide the authentication scheme that After deploying either type of AMI, configure the ClearML Server instance to provide the authentication scheme that
best matches the workflow. best matches the workflow.
For information about upgrading a **ClearML Server** in an AWS instance, see [here](upgrade_server_aws_ec2_ami.md). For information about upgrading a ClearML Server in an AWS instance, see [here](upgrade_server_aws_ec2_ami.md).
:::important :::important
If **ClearML Server** is being reinstalled, we recommend clearing browser cookies for **ClearML Server**. For example, If ClearML Server is being reinstalled, we recommend clearing browser cookies for ClearML Server. For example,
for Firefox, go to Developer Tools > Storage > Cookies, and for Chrome, go to Developer Tools > Application > Cookies, for Firefox, go to Developer Tools > Storage > Cookies, and for Chrome, go to Developer Tools > Application > Cookies,
and delete all cookies under the **ClearML Server** URL. and delete all cookies under the ClearML Server URL.
::: :::
## Launching ## Launching

View File

@ -23,6 +23,7 @@ and delete all cookies under the ClearML Server URL.
## Prerequisites ## Prerequisites
For Linux users only: For Linux users only:
* Linux distribution must support Docker. For more information, see this [explanation](https://docs.docker.com/engine/install/) in the Docker documentation. * Linux distribution must support Docker. For more information, see this [explanation](https://docs.docker.com/engine/install/) in the Docker documentation.
@ -123,7 +124,7 @@ instructions in the [Security](clearml_server_security.md) page.
sudo curl https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml sudo curl https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml
1. For Linux only, configure the **ClearML Agent Services**. If `CLEARML_HOST_IP` is not provided, then ClearML Agent Services uses the external public address of the **ClearML Server**. If `CLEARML_AGENT_GIT_USER` / `CLEARML_AGENT_GIT_PASS` are not provided, then **ClearML Agent Services** can't access any private repositories for running service tasks. 1. For Linux only, configure the **ClearML Agent Services**. If `CLEARML_HOST_IP` is not provided, then ClearML Agent Services uses the external public address of the ClearML Server. If `CLEARML_AGENT_GIT_USER` / `CLEARML_AGENT_GIT_PASS` are not provided, then ClearML Agent Services can't access any private repositories for running service tasks.
export CLEARML_HOST_IP=server_host_ip_here export CLEARML_HOST_IP=server_host_ip_here
export CLEARML_AGENT_GIT_USER=git_username_here export CLEARML_AGENT_GIT_USER=git_username_here

View File

@ -717,7 +717,7 @@ Yes! You can run ClearML in Jupyter Notebooks using either of the following:
1. Use the [Task.set_credentials](references/sdk/task.md#taskset_credentials) 1. Use the [Task.set_credentials](references/sdk/task.md#taskset_credentials)
method to specify the host, port, access key and secret key (see step 1). method to specify the host, port, access key and secret key (see step 1).
```python ```python
# Set your credentials using the trains apiserver URI and port, access_key, and secret_key. # Set your credentials using the clearml apiserver URI and port, access_key, and secret_key.
Task.set_credentials(host='http://localhost:8008',key='<access_key>', secret='<secret_key>') Task.set_credentials(host='http://localhost:8008',key='<access_key>', secret='<secret_key>')
``` ```

View File

@ -46,7 +46,7 @@ The single scalar plot for loss appears in **RESULTS** **>** **SCALARS**.
## Hyperparameters ## Hyperparameters
**ClearML** automatically logs the command line options defined using `argparse`. ClearML automatically logs the command line options defined using `argparse`.
A parameter dictionary is logged by connecting it to the Task using a call to the [Task.connect](../../../references/sdk/task.md#connect) A parameter dictionary is logged by connecting it to the Task using a call to the [Task.connect](../../../references/sdk/task.md#connect)
method. method.

View File

@ -6,10 +6,11 @@ The [pytorch_matplotlib.py](https://github.com/allegroai/clearml/blob/master/exa
example demonstrates the integration of **ClearML** into code that uses PyTorch and Matplotlib. example demonstrates the integration of **ClearML** into code that uses PyTorch and Matplotlib.
The example does the following: The example does the following:
1. The script calls Matplotlib methods to show images, each with a different title. * Creates an experiment named `pytorch with matplotlib example`, in the `examples` project.
1. **ClearML** automatically logs the images as debug samples. * The script calls Matplotlib methods to show images, each with a different title.
1. When the script runs, it creates an experiment named `pytorch with matplotlib example`, which is associated with the * ClearML automatically logs the images as debug samples.
`examples` project.
## Debug Samples
The images shown in the example script's `imshow` function appear according to metric in **RESULTS** **>** **DEBUG SAMPLES**. The images shown in the example script's `imshow` function appear according to metric in **RESULTS** **>** **DEBUG SAMPLES**.

View File

@ -8,7 +8,7 @@ example demonstrates the integration of ClearML into code that uses PyTorch and
The example does the following: The example does the following:
* Trains a simple deep neural network on the PyTorch built-in [MNIST](https://pytorch.org/vision/stable/datasets.html#mnist) * Trains a simple deep neural network on the PyTorch built-in [MNIST](https://pytorch.org/vision/stable/datasets.html#mnist)
dataset. dataset.
* Creates an experiment named `pytorch with tensorboardX`, which is associated with the `examples` project. * Creates an experiment named `pytorch with tensorboardX` in the `examples` project.
* ClearML automatically captures scalars and text logged using the TensorBoardX `SummaryWriter` object, and * ClearML automatically captures scalars and text logged using the TensorBoardX `SummaryWriter` object, and
the model created by PyTorch. the model created by PyTorch.

View File

@ -3,7 +3,7 @@ title: PyTorch TensorBoard Toy
--- ---
The [tensorboard_toy_pytorch.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/pytorch/tensorboard_toy_pytorch.py) The [tensorboard_toy_pytorch.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/pytorch/tensorboard_toy_pytorch.py)
example demonstrates the integration of **ClearML** into code, which creates a TensorBoard `SummaryWriter` object to log example demonstrates the integration of ClearML into code, which creates a TensorBoard `SummaryWriter` object to log
debug sample images. When the script runs, it creates an experiment named `pytorch tensorboard toy example`, which is debug sample images. When the script runs, it creates an experiment named `pytorch tensorboard toy example`, which is
associated with the `examples` project. associated with the `examples` project.
@ -16,6 +16,6 @@ The debug sample images appear according to metric, in the experiment page in th
## Hyperparameters ## Hyperparameters
**ClearML** automatically logs TensorFlow Definitions. They appear in **CONFIGURATIONS** **>** **HYPER PARAMETERS** **>** **TF_DEFINE**. ClearML automatically logs TensorFlow Definitions. They appear in **CONFIGURATIONS** **>** **HYPER PARAMETERS** **>** **TF_DEFINE**.
![image](../../../img/examples_tensorboard_toy_pytorch_00.png) ![image](../../../img/examples_tensorboard_toy_pytorch_00.png)

View File

@ -5,7 +5,7 @@ title: scikit-learn with Joblib
The [sklearn_joblib_example.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/scikit-learn/sklearn_joblib_example.py) The [sklearn_joblib_example.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/scikit-learn/sklearn_joblib_example.py)
demonstrates the integration of ClearML into code that uses `scikit-learn` and `joblib` to store a model and model snapshots, demonstrates the integration of ClearML into code that uses `scikit-learn` and `joblib` to store a model and model snapshots,
and `matplotlib` to create a scatter diagram. When the script runs, it creates an experiment named and `matplotlib` to create a scatter diagram. When the script runs, it creates an experiment named
`scikit-learn joblib examplescikit-learn joblib example`, which is associated with the `examples` project. `scikit-learn joblib example`, which is associated with the `examples` project.
## Plots ## Plots

View File

@ -3,13 +3,13 @@ title: scikit-learn with Matplotlib
--- ---
The [sklearn_matplotlib_example.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/scikit-learn/sklearn_matplotlib_example.py) The [sklearn_matplotlib_example.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/scikit-learn/sklearn_matplotlib_example.py)
script demonstrates the integration of **ClearML** into code that uses `scikit-learn` and `matplotlib`. script demonstrates the integration of ClearML into code that uses `scikit-learn` and `matplotlib`.
The example does the following: The example does the following:
* Uses `scikit-learn` to determine cross-validated training and test scores. * Uses `scikit-learn` to determine cross-validated training and test scores.
* Uses `matplotlib` to plot the learning curves. * Uses `matplotlib` to plot the learning curves.
* Through ClearML, automatically logs the scatter diagrams for the learning curves. * ClearML automatically logs the scatter diagrams for the learning curves.
* Creates an experiment named `scikit-learn matplotlib example` which is associated with the `examples` project. * Creates an experiment named `scikit-learn matplotlib example` in the `examples` project.
## Plots ## Plots

View File

@ -3,16 +3,16 @@ title: TensorBoard PR Curve
--- ---
The [tensorboard_pr_curve.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorboard_pr_curve.py) The [tensorboard_pr_curve.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorboard_pr_curve.py)
example demonstrates the integration of **ClearML** into code that uses TensorFlow and TensorBoard. example demonstrates the integration of ClearML into code that uses TensorFlow and TensorBoard.
The example script does the following: The example script does the following:
* Creates an experiment named `tensorboard pr_curve` in the `examples` project.
* Creates three classes, R, G, and B, and generates colors within the RGB space from normal distributions. The true * Creates three classes, R, G, and B, and generates colors within the RGB space from normal distributions. The true
label of each random color is associated with the normal distribution that generated it. label of each random color is associated with the normal distribution that generated it.
* Computes the probability that each color belongs to the class, using three other normal distributions. * Computes the probability that each color belongs to the class, using three other normal distributions.
* Generate PR curves using those probabilities. * Generate PR curves using those probabilities.
* Creates a summary per class using [tensorboard.plugins.pr_curve.summary](https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/pr_curve/summary.py), * Creates a summary per class using [tensorboard.plugins.pr_curve.summary](https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/pr_curve/summary.py),
* Automatically logs the TensorBoard output, TensorFlow Definitions, and output to the console, using **ClearML**. * ClearML automatically captures TensorBoard output, TensorFlow Definitions, and output to the console
* When the script runs, Creates an experiment named `tensorboard pr_curve`, which is associated with the `examples` project.
## Plots ## Plots
@ -27,7 +27,7 @@ In the **ClearML Web UI**, the PR Curve summaries appear in the experiment's pag
## Hyperparameters ## Hyperparameters
**ClearML** automatically logs TensorFlow Definitions. They appear in **CONFIGURATIONS** **>** **HYPER PARAMETERS** **>** **TF_DEFINE**. ClearML automatically logs TensorFlow Definitions. They appear in **CONFIGURATIONS** **>** **HYPER PARAMETERS** **>** **TF_DEFINE**.
![image](../../../img/examples_tensorboard_pr_curve_04.png) ![image](../../../img/examples_tensorboard_pr_curve_04.png)

View File

@ -3,22 +3,15 @@ title: TensorBoard Toy
--- ---
The [tensorboard_toy.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorboard_toy.py) The [tensorboard_toy.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorboard_toy.py)
example demonstrates **ClearML**'s automatic logging of TensorBoard scalars, histograms, images, and text, as well as example demonstrates ClearML's automatic logging of TensorBoard scalars, histograms, images, and text, as well as
all other console output and TensorFlow Definitions. all other console output and TensorFlow Definitions.
The script uses `tf.summary.create_file_writer` with the following: When the script runs, it creates an experiment named `tensorboard toy example` in the `examples`
* `tf.summary.histogram`
* `tf.summary.scalar`
* `tf.summary.text`
* `tf.summary.image`
When the script runs, it creates an experiment named `tensorboard toy example`, which is associated with the `examples`
project. project.
## Scalars ## Scalars
The `tf.summary.scalar` output appears in the experiment's page in the **ClearML web UI** under **RESULTS** **>** The `tf.summary.scalar` output appears in the ClearML web UI, in the experiment's **RESULTS** **>**
**SCALARS**. Resource utilization plots, which are titled **:monitor: machine**, also appear in the **SCALARS** tab. **SCALARS**. Resource utilization plots, which are titled **:monitor: machine**, also appear in the **SCALARS** tab.
![image](../../../img/examples_tensorboard_toy_03.png) ![image](../../../img/examples_tensorboard_toy_03.png)
@ -31,13 +24,13 @@ The `tf.summary.histogram` output appears in **RESULTS** **>** **PLOTS**.
## Debug Samples ## Debug Samples
**ClearML** automatically tracks images and text output to TensorFlow. They appear in **RESULTS** **>** **DEBUG SAMPLES**. ClearML automatically tracks images and text output to TensorFlow. They appear in **RESULTS** **>** **DEBUG SAMPLES**.
![image](../../../img/examples_tensorboard_toy_05.png) ![image](../../../img/examples_tensorboard_toy_05.png)
## Hyperparameters ## Hyperparameters
**ClearML** automatically logs TensorFlow Definitions. They appear in **CONFIGURATIONS** **>** **HYPER PARAMETERS** **>** ClearML automatically logs TensorFlow Definitions. They appear in **CONFIGURATIONS** **>** **HYPER PARAMETERS** **>**
**TF_DEFINE**. **TF_DEFINE**.
![image](../../../img/examples_tensorboard_toy_01.png) ![image](../../../img/examples_tensorboard_toy_01.png)

View File

@ -6,14 +6,7 @@ The [tensorflow_mnist.py](https://github.com/allegroai/clearml/blob/master/examp
example demonstrates the integration of ClearML into code that uses TensorFlow and Keras to train a neural network on example demonstrates the integration of ClearML into code that uses TensorFlow and Keras to train a neural network on
the Keras built-in [MNIST](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/mnist) handwritten digits dataset. the Keras built-in [MNIST](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/mnist) handwritten digits dataset.
The script builds a TensorFlow Keras model, and trains and tests it with the following: When the script runs, it creates an experiment named `Tensorflow v2 mnist with summaries` in the `examples` project.
* Loss objective function - [tf.keras.metrics.SparseCategoricalCrossentropy](https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy)
* Accuracy metric - [tf.keras.metrics.SparseCategoricalAccuracy](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/SparseCategoricalAccuracy)
* Model checkpointing - [tf.clearml.Checkpoint](https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint?hl=ca) and [tf.train.CheckpointManager](https://www.tensorflow.org/api_docs/python/tf/train/CheckpointManager?hl=ca)
When the script runs, it creates an experiment named `Tensorflow v2 mnist with summaries`, which is associated with the
`examples` project.
## Scalars ## Scalars