From 61b9b482f8f9397dd0fcef950034fe1ecd549614 Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Sun, 4 Sep 2022 11:45:24 +0300 Subject: [PATCH] Add clearml-agent build tutorials (#319) --- docs/clearml_agent.md | 11 ++- .../executable_exp_containers.md | 59 +++++++++++++ .../exp_environment_containers.md | 83 +++++++++++++++++++ sidebars.js | 1 + 4 files changed, 151 insertions(+), 3 deletions(-) create mode 100644 docs/guides/clearml_agent/executable_exp_containers.md create mode 100644 docs/guides/clearml_agent/exp_environment_containers.md diff --git a/docs/clearml_agent.md b/docs/clearml_agent.md index 6e67a888..61906c6f 100644 --- a/docs/clearml_agent.md +++ b/docs/clearml_agent.md @@ -457,22 +457,27 @@ Build a Docker container that when launched executes a specific experiment, or a docker run ``` +Check out [this tutorial](guides/clearml_agent/executable_exp_containers.md) for building executable experiment +containers. + ### Base Docker Container -Build a Docker container according to the execution environment of a specific Task. +Build a Docker container according to the execution environment of a specific task. ```bash clearml-agent build --id --docker --target ``` - -It's possible to add the Docker container as the base Docker image to a Task (experiment), using one of the following methods: +It's possible to add the Docker container as the base Docker image to a task (experiment), using one of the following methods: - Using the **ClearML Web UI** - See [Base Docker image](webapp/webapp_exp_tuning.md#base-docker-image) on the "Tuning Experiments" page. - In the ClearML configuration file - Use the ClearML configuration file [agent.default_docker](configs/clearml_conf.md#agentdefault_docker) options. +Check out [this tutorial](guides/clearml_agent/exp_environment_containers.md) for building a Docker container +replicating the execution environment of an existing task. + ## Google Colab ClearML Agent can run on a [Google Colab](https://colab.research.google.com/) instance. This helps users to leverage diff --git a/docs/guides/clearml_agent/executable_exp_containers.md b/docs/guides/clearml_agent/executable_exp_containers.md new file mode 100644 index 00000000..0d0ac67f --- /dev/null +++ b/docs/guides/clearml_agent/executable_exp_containers.md @@ -0,0 +1,59 @@ +--- +title: Executable Experiment Containers +--- + +This tutorial demonstrates using [`clearml-agent`](../../clearml_agent.md)’s [`build`](../../clearml_agent/clearml_agent_build.md) +command to package an experiment into an executable container. In this example, you will build a Docker image that, when +run, will automatically execute the [keras_tensorboard.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/keras/keras_tensorboard.py) +script. + +## Prerequisites +* [`clearml-agent`](../../clearml_agent.md#installation) installed and configured +* [`clearml`](../../getting_started/ds/ds_first_steps.md#install-clearml) installed and configured +* [clearml](https://github.com/allegroai/clearml) repo cloned (`git clone https://github.com/allegroai/clearml.git`) + +## Creating the ClearML Experiment +1. Set up the experiment’s execution environment: + + ```console + cd clearml/examples/frameworks/keras + pip install -r requirements.txt + ``` + +1. Run the experiment: + + ```console + python keras_tensorboard.py + ``` + This creates a ClearML task called "Keras with TensorBoard example" in the "examples" project. + + Note the task ID in the console output when running the script above: + + ```console + ClearML Task: created new task id= + ``` + This ID will be used in the following section. + +## Building and Launching a Containerized Task +1. Execute the following command to build the container. Input the ID of the task created above. + ```console + clearml-agent build --id --docker --target new-docker --entry-point clone_task + ``` + + :::tip + If the container will not make use of a GPU, add the `--cpu-only` flag + ::: + + This command will create a Docker container, set up with the execution environment for this experiment in the + specified `--target` folder. When the Docker container is launched, it will clone the task specified with `id` and + execute the clone (as designated by the `--entry-point` parameter). + +1. Run the Docker, pointing to the new container: + + ```console + docker run new-docker + ``` + + The task will be executed inside the container. Task details can be viewed in the [ClearML Web UI](../../webapp/webapp_overview.md). + +For additional ClearML Agent options, see the [ClearML Agent reference page](../../clearml_agent/clearml_agent_ref.md). diff --git a/docs/guides/clearml_agent/exp_environment_containers.md b/docs/guides/clearml_agent/exp_environment_containers.md new file mode 100644 index 00000000..943b6b55 --- /dev/null +++ b/docs/guides/clearml_agent/exp_environment_containers.md @@ -0,0 +1,83 @@ +--- +title: Experiment Environment Containers +--- + +This tutorial demonstrates using [`clearml-agent`](../../clearml_agent.md)’s [`build`](../../clearml_agent/clearml_agent_build.md) +command to build a Docker container replicating the execution environment of an existing task. ClearML Agents can make +use of such containers to execute tasks without having to set up their environment every time. + +A use case for this would be manual hyperparameter optimization, where a base task can be used to create a container to +be used when running optimization tasks. + +## Prerequisites +* [`clearml-agent`](../../clearml_agent.md#installation) installed and configured +* [`clearml`](../../getting_started/ds/ds_first_steps.md#install-clearml) installed and configured +* [clearml](https://github.com/allegroai/clearml) repo cloned (`git clone https://github.com/allegroai/clearml.git`) + +## Creating the ClearML Experiment +1. Set up the experiment’s execution environment: + + ```console + cd clearml/examples/frameworks/keras + pip install -r requirements.txt + ``` + +1. Run the experiment: + + ```console + python keras_tensorboard.py + ``` + This creates a ClearML task called "Keras with TensorBoard example" in the "examples" project. + + Note the task ID in the console output when running the script above: + + ```console + ClearML Task: created new task id= + ``` + This ID will be used in the following section. + +## Building the Docker Container + +Execute the following command to build the container. Input the ID of the task created above. +```console +clearml-agent build --id --docker --target new_docker +``` + +:::tip +If the container will not make use of a GPU, add the `--cpu-only` flag +::: + +This will create a container with the specified task’s execution environment in the `--target` folder. +When the Docker build completes, the console output shows: + +```console +Docker build done +Committing docker container to: new_docker +sha256:460453b93ct1989fd1c6637c236e544031c4d378581433fc0b961103ce206af1 +``` + +## Using the New Docker Container +Make use of the container you've just built by having a ClearML agent make use of it for executing a new experiment: + +1. In the [ClearML Web UI](../../webapp/webapp_overview.md), go to the "examples" project, "Keras with TensorBoard + example" task (the one executed [above](#creating-the-clearml-experiment)). +1. [Clone](../../webapp/webapp_exp_reproducing.md) the experiment. +1. In the cloned experiment, go to the **EXECUTION** tab **>** **CONTAINER** section. Under **IMAGE**, insert the name + of the new Docker image, `new_docker`. See [Tuning Experiments](../../webapp/webapp_exp_tuning.md) for more task + modification options. +1. Enqueue the cloned experiment to the `default` queue. +1. Launch a `clearml-agent` in [Docker Mode](../../clearml_agent.md#docker-mode) and assign it to the `default` queue: + ```console + clearml-agent daemon --docker --queue default + ``` + + :::tip + If the agent will not make use of a GPU, add the `--cpu-only` flag + ::: + + This agent will pull the enqueued task and run it using the `new_docker` image to create the execution environment. + In the task’s **CONSOLE** tab, one of the first logs should be: + + ```console + Executing: ['docker', 'run', ..., 'CLEARML_DOCKER_IMAGE=new_docker', ...]. + ``` \ No newline at end of file diff --git a/sidebars.js b/sidebars.js index 8e8e19eb..d4ad1671 100644 --- a/sidebars.js +++ b/sidebars.js @@ -106,6 +106,7 @@ module.exports = { {'Advanced': ['guides/advanced/execute_remotely', 'guides/advanced/multiple_tasks_single_process']}, {'Automation': ['guides/automation/manual_random_param_search_example', 'guides/automation/task_piping']}, {'ClearML Task': ['guides/clearml-task/clearml_task_tutorial']}, + {'ClearML Agent': ['guides/clearml_agent/executable_exp_containers', 'guides/clearml_agent/exp_environment_containers']}, {'Datasets': ['guides/datasets/data_man_cifar_classification', 'guides/datasets/data_man_python']}, {'Distributed': ['guides/distributed/distributed_pytorch_example', 'guides/distributed/subprocess_example']}, {'Docker': ['guides/docker/extra_docker_shell_script']},