From b5ee71c21c098e19c0e418381ea28ae4c073c64b Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Sun, 15 May 2022 15:26:38 +0300 Subject: [PATCH] Add clearml-task import offline session option (#251) --- docs/apps/clearml_task.md | 1 + docs/faq.md | 12 ++++++++---- docs/guides/set_offline.md | 37 +++++++++++++++++++++++++------------ 3 files changed, 34 insertions(+), 16 deletions(-) diff --git a/docs/apps/clearml_task.md b/docs/apps/clearml_task.md index 3da76e07..647b7540 100644 --- a/docs/apps/clearml_task.md +++ b/docs/apps/clearml_task.md @@ -75,6 +75,7 @@ errors in identifying the correct default branch. | `--task-type` | Set the task type. Optional values: training, testing, inference, data_processing, application, monitor, controller, optimizer, service, qc, custom | Yes | | `--skip-task-init` | If set, `Task.init()` call is not added to the entry point, and is assumed to be called within the script | Yes | | `--base-task-id` | Use a pre-existing task in the system, instead of a local repo / script. Essentially clones an existing task and overrides arguments / requirements | Yes | +| `--import-offline-session`| Specify the path to the offline session you want to import.| Yes | diff --git a/docs/faq.md b/docs/faq.md index 2d474658..ae3b48fd 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -490,12 +490,16 @@ See [`Task.init`](references/sdk/task.md#taskinit). **Can I run ClearML Task while working offline?** Yes! You can use ClearML's Offline Mode, in which all the data and logs that a task captures from the code are stored in a -local folder +local folder. Before initializing a task, use the [Task.set_offline](references/sdk/task.md#taskset_offline) -class method and set the `offline_mode` argument to `True`. When executed, this returns the Task ID and a path to the local -session folder. In order to upload to the ClearML Server the local execution data that the Task captured offline, use the -[Task.import_offline_session](references/sdk/task.md#taskimport_offline_session) method. See [Storing Task Data Offline](guides/set_offline.md). +class method and set the `offline_mode` argument to `True`. When executed, this returns the Task ID and a path to the +session folder. In order to upload to the ClearML Server the execution data that the Task captured offline, do one of the +following: +* Use the `import-offline-session ` option of the [clearml-task](apps/clearml_task.md) CLI +* Use the [Task.import_offline_session](references/sdk/task.md#taskimport_offline_session) method. + +See [Storing Task Data Offline](guides/set_offline.md). diff --git a/docs/guides/set_offline.md b/docs/guides/set_offline.md index b5c072b5..96724fcd 100644 --- a/docs/guides/set_offline.md +++ b/docs/guides/set_offline.md @@ -21,7 +21,7 @@ task = Task.init(project_name="examples", task_name="my_task") # Rest of code is executed. All data is logged locally and not onto the server ``` -The method returns the Task ID and a path to the local session folder: +The method returns the Task ID and a path to the session folder: ```console ClearML Task: created new task id=offline-372657bb04444c25a31bc6af86552cc9 @@ -33,25 +33,38 @@ ClearML Task: Offline session stored in /home/user/.clearml/cache/offline/b78684 All the information captured by the Task is saved locally. Once the task script finishes execution, it's zipped. The session's zip folder's location is `~/.clearml/cache/offline/.zip`. -## Uploading Local Session +## Uploading Session Data -In order to upload to the ClearML Server the local execution data that the Task captured offline, use the -[Task.import_offline_session](../references/sdk/task.md#taskimport_offline_session) method. This method will upload the -Task's full execution details and outputs. +Upload the session's execution data that the Task captured offline to the ClearML Server using one of the following: -```python -from clearml import Task -Task.import_offline_session(session_folder_zip="path/to/session/.clearml/cache/offline/b786845decb14eecadf2be24affc7418.zip") -``` +* `clearml-task` CLI + + ```bash + clearml-task --import-offline-session "/home/user/.clearml/cache/offline/b786845decb14eecadf2be24affc7418.zip" + ``` -In the `session_folder_zip` argument, insert the path to the zip folder containing the session. + Pass the path to the zip folder containing the session with the `--import-offline-session` parameter. + +* [Task.import_offline_session](../references/sdk/task.md#taskimport_offline_session) method. -This method returns a link to the Task's results page on the ClearML Server: + ```python + from clearml import Task + + Task.import_offline_session( + session_folder_zip="/home/user/.clearml/cache/offline/b786845decb14eecadf2be24affc7418.zip" + ) + ``` + + In the `session_folder_zip` argument, insert the path to the zip folder containing the session. + + +Both options will upload the Task's full execution details and outputs and return a link to the Task's results page on +the ClearML Server: ```console ClearML: Importing offline session from /home/user/.clearml/cache/offline/b786845decb14eecadf2be24affc7418.zip -ClearML results page: https://app.community.clear.ml/projects/4043a1657f374e9298649c6ba72ad233/experiments/bb8b0f6fa0f94536a0d27fb55f02d3a5/output/log +ClearML results page: https://app.clear.ml/projects/4043a1657f374e9298649c6ba72ad233/experiments/bb8b0f6fa0f94536a0d27fb55f02d3a5/output/log ``` The session details can be viewed in the ClearML WebApp, in the "my_task" experiment of the "examples"