diff --git a/docs/clearml_agent/clearml_agent_env_var.md b/docs/clearml_agent/clearml_agent_env_var.md index ae1c8746..f8dd6bd2 100644 --- a/docs/clearml_agent/clearml_agent_env_var.md +++ b/docs/clearml_agent/clearml_agent_env_var.md @@ -22,7 +22,7 @@ but can be overridden by command-line arguments. |**CLEARML_CUDA_VERSION** | Sets the CUDA version to be used | |**CLEARML_CUDNN_VERSION** | Sets the CUDNN version to be used | |**CLEARML_CPU_ONLY** | Force CPU only mode | -|**CLEARML_DOCKER_SKIP_GPUS_FLAG** | Skips the GPUs flag (support for docker V18 | +|**CLEARML_DOCKER_SKIP_GPUS_FLAG** | Skips the GPUs flag (support for docker V18) | |**CLEARML_AGENT_DOCKER_ARGS_HIDE_ENV** | Hide Docker environment variables containing secrets when printing out the Docker command. When printed, the variable values will be replaced by `********`. See [`agent.hide_docker_command_env_vars`](../configs/clearml_conf.md#hide_docker) | |**CLEARML_AGENT_DISABLE_SSH_MOUNT** | Disables the auto `.ssh` mount into the docker | |**CLEARML_AGENT_FORCE_CODE_DIR**| Allows overriding the remote execution code directory to bypass repository cloning and use a repo already available where the remote agent is running. | diff --git a/docs/clearml_data/data_management_examples/data_man_cifar_classification.md b/docs/clearml_data/data_management_examples/data_man_cifar_classification.md index d27e1acd..59a65eac 100644 --- a/docs/clearml_data/data_management_examples/data_man_cifar_classification.md +++ b/docs/clearml_data/data_management_examples/data_man_cifar_classification.md @@ -10,7 +10,7 @@ class to ingest the data. ### Downloading the Data Before registering the CIFAR dataset with `clearml-data`, you need to obtain a local copy of it. -Execute this python script to download the data +Execute this python script to download the data: ```python from clearml import StorageManager diff --git a/docs/configs/clearml_conf.md b/docs/configs/clearml_conf.md index 15e76932..e753f49a 100644 --- a/docs/configs/clearml_conf.md +++ b/docs/configs/clearml_conf.md @@ -128,7 +128,7 @@ Use with care! This might introduce security risks by allowing access to keys/se --- -**`docker_args_extra_precedes_task`** (*bool*) +**`agent.docker_args_extra_precedes_task`** (*bool*) * Allow the arguments specified in `agent.extra_docker_arguments` to override task level docker arguments, in the case that the same argument is passed in both. If set to `False`, a task's docker arguments will override the `extra_docker_arguments`. @@ -1039,6 +1039,8 @@ and limitations on bucket naming. --- + + **`sdk.development.log_os_environments`** (*[string]*) * Log specific environment variables. OS environments are listed in the UI, under an experiment's diff --git a/docs/faq.md b/docs/faq.md index 3ca5e9b4..2c0c6728 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -855,7 +855,7 @@ export CLEARML_API_HOST="http://localhost:8008" #### How can I track OS environment variables with experiments? You can set environment variables to track in an experiment by specifying them in the `sdk.development.log_os_environments` -field of the [`clearml.conf`](configs/clearml_conf.md) file: +field of the [`clearml.conf`](configs/clearml_conf.md#log_env_var) file: ```editorconfig log_os_environments: ["AWS_*", "CUDA_VERSION"] diff --git a/docs/fundamentals/agents_and_queues.md b/docs/fundamentals/agents_and_queues.md index 1a0a7b34..7cf98b33 100644 --- a/docs/fundamentals/agents_and_queues.md +++ b/docs/fundamentals/agents_and_queues.md @@ -1,5 +1,5 @@ --- -title: Workers & Queues +title: Workers and Queues --- Two major components of MLOps/LLMOps are experiment reproducibility, and the ability to scale work to multiple machines. ClearML workers, diff --git a/docs/fundamentals/hyperparameters.md b/docs/fundamentals/hyperparameters.md index b7eff96b..e021076a 100644 --- a/docs/fundamentals/hyperparameters.md +++ b/docs/fundamentals/hyperparameters.md @@ -61,7 +61,7 @@ You can also specify environment variables using the `CLEARML_LOG_ENVIRONMENT` e export CLEARML_LOG_ENVIRONMENT=PWD,PYTHONPATH ``` -* No environment variables +* No environment variables: ``` export CLEARML_LOG_ENVIRONMENT= ``` diff --git a/docs/guides/frameworks/megengine/megengine_mnist.md b/docs/guides/frameworks/megengine/megengine_mnist.md index 6c3493fa..87048ec2 100644 --- a/docs/guides/frameworks/megengine/megengine_mnist.md +++ b/docs/guides/frameworks/megengine/megengine_mnist.md @@ -50,5 +50,6 @@ The model info panel contains the model details, including: ## Console All console output during the script's execution appears in the experiment's **CONSOLE** page. + ![Console tab](../../../img/examples_megengine_console.png) diff --git a/docs/guides/frameworks/pytorch/pytorch_mnist.md b/docs/guides/frameworks/pytorch/pytorch_mnist.md index e65a1b71..00b26850 100644 --- a/docs/guides/frameworks/pytorch/pytorch_mnist.md +++ b/docs/guides/frameworks/pytorch/pytorch_mnist.md @@ -9,7 +9,7 @@ The example script does the following: * Trains a simple deep neural network on the PyTorch built-in [MNIST](https://pytorch.org/vision/stable/datasets.html#mnist) dataset. * Creates an experiment named `pytorch mnist train` in the `examples` project. -* ClearML automatically logs `argparse` command line options, and models (and their snapshots) created by PyTorch +* ClearML automatically logs `argparse` command line options, and models (and their snapshots) created by PyTorch. * Additional metrics are logged by calling [`Logger.report_scalar()`](../../../references/sdk/logger.md#report_scalar). ## Scalars diff --git a/docs/guides/frameworks/tensorflow/tensorboard_pr_curve.md b/docs/guides/frameworks/tensorflow/tensorboard_pr_curve.md index acf7458a..c09cc68a 100644 --- a/docs/guides/frameworks/tensorflow/tensorboard_pr_curve.md +++ b/docs/guides/frameworks/tensorflow/tensorboard_pr_curve.md @@ -11,8 +11,8 @@ The example script does the following: label of each random color is associated with the normal distribution that generated it. * Computes the probability that each color belongs to the class, using three other normal distributions. * Generate PR curves using those probabilities. -* Creates a summary per class using [tensorboard.plugins.pr_curve.summary](https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/pr_curve/summary.py), -* ClearML automatically captures TensorBoard output, TensorFlow Definitions, and output to the console +* Creates a summary per class using [tensorboard.plugins.pr_curve.summary](https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/pr_curve/summary.py). +* ClearML automatically captures TensorBoard output, TensorFlow Definitions, and output to the console. ## Plots diff --git a/docs/guides/ide/remote_jupyter_tutorial.md b/docs/guides/ide/remote_jupyter_tutorial.md index a79f846a..121ed568 100644 --- a/docs/guides/ide/remote_jupyter_tutorial.md +++ b/docs/guides/ide/remote_jupyter_tutorial.md @@ -85,7 +85,7 @@ Jupyter Lab URL: http://localhost:8878/?token=ff7e5e8b9e5493a01b1a72530d18181320 VSCode server available at http://localhost:8898/ ``` -Click on the JupyterLab link, which will open the remote session +Click on the JupyterLab link, which will open the remote session. Now, let's execute some code in the remote session! diff --git a/docs/guides/reporting/media_reporting.md b/docs/guides/reporting/media_reporting.md index 130fcbb6..eed88097 100644 --- a/docs/guides/reporting/media_reporting.md +++ b/docs/guides/reporting/media_reporting.md @@ -3,15 +3,16 @@ title: Media Reporting --- The [media_reporting.py](https://github.com/allegroai/clearml/blob/master/examples/reporting/media_reporting.py) example -demonstrates reporting (uploading) images, audio, and video. Use the [Logger.report_media](../../references/sdk/logger.md#report_media) -method to upload from: +demonstrates reporting (uploading) images, audio, and video. Use [`Logger.report_media()`](../../references/sdk/logger.md#report_media) +to upload from: * Local path * BytesIO stream * URL of media already uploaded to some storage -ClearML uploads media to the bucket specified in the ClearML configuration file or ClearML can be configured for image storage, see [Logger.set_default_upload_destination](../../references/sdk/logger.md#set_default_upload_destination) -(storage for [artifacts](../../clearml_sdk/task_sdk.md#setting-upload-destination) is different). Set credentials for storage in the ClearML -[configuration file](../../configs/clearml_conf.md). +ClearML uploads media to the bucket specified in the ClearML configuration file. You can configure ClearML for image +storage using [`Logger.set_default_upload_destination()`](../../references/sdk/logger.md#set_default_upload_destination) +(note that [artifact storage](../../clearml_sdk/task_sdk.md#setting-upload-destination) is handled differently). +Set the storage credentials in the [clearml.conf file](../../configs/clearml_conf.md#sdk-section). ClearML reports media in the **ClearML Web UI** **>** experiment details **>** **DEBUG SAMPLES** tab. @@ -21,8 +22,7 @@ project. ## Reporting (Uploading) Media from a Source by URL -Report by calling the [Logger.report_media](../../references/sdk/logger.md#report_media) -method using the `url` parameter. +Report by using the `url` parameter of [`Logger.report_media()`](../../references/sdk/logger.md#report_media): ```python # report video, an already uploaded video media (url) @@ -45,7 +45,7 @@ The reported audio can be viewed in the **DEBUG SAMPLES** tab. Click a thumbnail ## Reporting (Uploading) Media from a Local File -Use the `local_path` parameter. +Report by using the `local_path` parameter of [`Logger.report_media()`](../../references/sdk/logger.md#report_media): ```python # report audio, report local media audio file diff --git a/docs/guides/reporting/text_reporting.md b/docs/guides/reporting/text_reporting.md index ca83222a..e75f985d 100644 --- a/docs/guides/reporting/text_reporting.md +++ b/docs/guides/reporting/text_reporting.md @@ -3,14 +3,15 @@ title: Text Reporting --- The [text_reporting.py](https://github.com/allegroai/clearml/blob/master/examples/reporting/text_reporting.py) script -demonstrates reporting explicit text, by calling the [Logger.report_text](../../references/sdk/logger.md#report_text) -method. +demonstrates reporting explicit text by calling [`Logger.report_text()`](../../references/sdk/logger.md#report_text). -ClearML reports these tables in the **ClearML Web UI**, experiment details, **CONSOLE** tab. +ClearML reports the text in the **ClearML Web UI**, in the experiment's **CONSOLE** tab. When the script runs, it creates an experiment named `text reporting` in the `examples` project. - # report text - Logger.current_logger().report_text("hello, this is plain text") +```python +# report text +Logger.current_logger().report_text("hello, this is plain text") +``` ![image](../../img/examples_reporting_text.png) \ No newline at end of file diff --git a/docs/guides/ui/tuning_exp.md b/docs/guides/ui/tuning_exp.md index 5155f7b1..fcb7f89a 100644 --- a/docs/guides/ui/tuning_exp.md +++ b/docs/guides/ui/tuning_exp.md @@ -16,7 +16,9 @@ example script. In the `examples/frameworks/pytorch` directory, run the experiment script: - python pytorch_mnist.py +```commandline +python pytorch_mnist.py +``` ## Step 2: Clone the Experiment @@ -42,15 +44,12 @@ To demonstrate tuning, change two hyperparameter values. ## Step 4: Run a Worker Daemon Listening to a Queue -To execute the cloned experiment, use a worker that can run a worker daemon listening to a queue. +To execute the cloned experiment, use a [ClearML Agent](../../fundamentals/agents_and_queues.md). -:::note -For more information about workers, worker daemons, and queues, see [Agents and queues](../../fundamentals/agents_and_queues.md). -::: - -Run the worker daemon on the local development machine. +Run the agent on the local development machine: 1. Open a terminal session. 1. Run the following `clearml-agent` command which runs a worker daemon listening to the `default` queue: + ``` clearml-agent daemon --queue default ``` @@ -119,5 +118,4 @@ To compare the original and tuned experiments: ## Next Steps -* For more information about editing experiments, see [modify experiments](../../webapp/webapp_exp_tuning.md#modifying-experiments) - in the User Interface section. \ No newline at end of file +* For more information about editing experiments, see [modifying experiments](../../webapp/webapp_exp_tuning.md#modifying-experiments). \ No newline at end of file diff --git a/docs/webapp/webapp_exp_track_visual.md b/docs/webapp/webapp_exp_track_visual.md index d8a2ffc3..1952e625 100644 --- a/docs/webapp/webapp_exp_track_visual.md +++ b/docs/webapp/webapp_exp_track_visual.md @@ -132,7 +132,7 @@ parameter's line, and the type, description, and default value appear, if they w #### Environment Variables -If environment variables were listed in the `CLEARML_LOG_ENVIRONMENT` environment variable or the `sdk.development.log_os_environments` +If environment variables were listed in the `CLEARML_LOG_ENVIRONMENT` environment variable or the [`sdk.development.log_os_environments`](../configs/clearml_conf.md#log_env_var) field of the `clearml.conf` file, the **Environment** group displays the listed environment variables (see [this FAQ](../faq.md#track-env-vars)). :::note diff --git a/docs/webapp/webapp_exp_tuning.md b/docs/webapp/webapp_exp_tuning.md index 6c4f055d..19aff7d6 100644 --- a/docs/webapp/webapp_exp_tuning.md +++ b/docs/webapp/webapp_exp_tuning.md @@ -127,7 +127,7 @@ Add, change, or delete hyperparameters, which are organized in the **ClearML Web * **General** - Parameter dictionaries (from code, connected to the Task by calling [`Task.connect()`](../references/sdk/task.md#connect)). * Environment variables - Tracked if variables were listed in the `CLEARML_LOG_ENVIRONMENT` environment variable -or the `sdk.development.log_os_environments` field of the `clearml.conf` file (see this [FAQ](../faq.md#track-env-vars)). +or the [`sdk.development.log_os_environments`](../configs/clearml_conf.md#log_env_var) field of the `clearml.conf` file (see this [FAQ](../faq.md#track-env-vars)). * Custom named parameter groups (see the `name` parameter in [`Task.connect`](../references/sdk/task.md#connect)). @@ -178,7 +178,7 @@ model in the **MODELS** tab. **To edit a model's configuration or label enumeration:** 1. Click the model name hyperlink. The model details appear in the **MODELS** tab. -1. Edit the model configuration or label enumeration. +1. Edit the model configuration or label enumeration: * Model configuration - In the **NETWORK** tab **>** Hover and click **EDIT**. **>** CLick **EDIT** or **CLEAR** (to remove the configuration).