Move ClearML Session (#756)

This commit is contained in:
pollfly 2024-01-15 15:04:32 +02:00 committed by GitHub
parent 13aad52fd5
commit 4340f59943
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 35 additions and 16 deletions

View File

@ -1,5 +1,5 @@
--- ---
title: ClearML Session title: Remote Session
--- ---
Machine Learning and Deep Learning development is sometimes more challenging than traditional software development. If Machine Learning and Deep Learning development is sometimes more challenging than traditional software development. If
@ -11,7 +11,19 @@ in the UI and send it for long-term training on a remote machine.
**If you are not that lucky**, this section is for you :) **If you are not that lucky**, this section is for you :)
## What Does ClearML Session Do? ClearML provides tools that allow you to launch remote sessions and to execute code on a remote machine that better
meets resource needs:
* [Clearml Session CLI](#clearml-session-cli) - Launch a JupyterLab and VS Code session on a remote machine
* [GUI Applications](#gui-applications) (available under ClearML Enterprise Plan)
* [JupyterLab](../webapp/applications/apps_jupyter_lab.md) - Launch a JupyterLab session on a remote machine
* [VS Code](../webapp/applications/apps_vscode.md) - Launch a VS Code session on a remote machine
:::info Remote PyCharm
You can also work with PyCharm in a remote session over SSH. Use the [PyCharm Plugin](../guides/ide/integration_pycharm.md)
to automatically sync local configurations with a remote session.
:::
## ClearML Session CLI
`clearml-session` is a feature that allows to launch a session of JupyterLab and VS Code, and to execute code on a remote `clearml-session` is a feature that allows to launch a session of JupyterLab and VS Code, and to execute code on a remote
machine that better meets resource needs. This feature provides local links to access JupyterLab and VS Code on a machine that better meets resource needs. This feature provides local links to access JupyterLab and VS Code on a
remote machine over a secure and encrypted SSH connection. By default, the JupyterLab and remote machine over a secure and encrypted SSH connection. By default, the JupyterLab and
@ -31,12 +43,6 @@ VS Code remote sessions use ports 8878 and 8898 respectively.
</Collapsible> </Collapsible>
:::info Remote PyCharm
You can also work with PyCharm in a remote session over SSH. Use the [PyCharm Plugin](../guides/ide/integration_pycharm.md)
to automatically sync local configurations with a remote session.
:::
## How to Run ClearML Session
### Prerequisites ### Prerequisites
* `clearml` installed and configured. See [Getting Started](../getting_started/ds/ds_first_steps.md) for details. * `clearml` installed and configured. See [Getting Started](../getting_started/ds/ds_first_steps.md) for details.
@ -120,24 +126,24 @@ To connect to an existing session:
1. Enter the following command: `clearml-session --attach <session_id>`. 1. Enter the following command: `clearml-session --attach <session_id>`.
1. Click on the JupyterLab / VS Code link that is outputted, or connect directly to the SSH session 1. Click on the JupyterLab / VS Code link that is outputted, or connect directly to the SSH session
## Features ### Features
### Running in Docker #### Running in Docker
To run a session inside a Docker container, use the `--docker` flag and enter the docker image to use in the interactive To run a session inside a Docker container, use the `--docker` flag and enter the docker image to use in the interactive
session. session.
### Installing Requirements #### Installing Requirements
`clearml-session` can install required Python packages when setting up the remote environment. `clearml-session` can install required Python packages when setting up the remote environment.
Specify requirements in one of the following ways: Specify requirements in one of the following ways:
* Attach a `requirement.txt` file to the command using `--requirements </file/location.txt>`. * Attach a `requirement.txt` file to the command using `--requirements </file/location.txt>`.
* Manually specify packages using `--packages "<package_name>"` * Manually specify packages using `--packages "<package_name>"`
(for example `--packages "keras" "clearml"`), and they'll be automatically installed. (for example `--packages "keras" "clearml"`), and they'll be automatically installed.
### Accessing a Git Repository #### Accessing a Git Repository
To access a git repository remotely, add a `--git-credentials` flag and set it to `true`, so the local `.git-credentials` To access a git repository remotely, add a `--git-credentials` flag and set it to `true`, so the local `.git-credentials`
file is sent to the interactive session. This is helpful if working on private git repositories, and it allows for seamless file is sent to the interactive session. This is helpful if working on private git repositories, and it allows for seamless
cloning and tracking of git references, including untracked changes. cloning and tracking of git references, including untracked changes.
### Starting a Debugging Session #### Starting a Debugging Session
You can debug previously executed experiments registered in the ClearML system on a remote interactive session. You can debug previously executed experiments registered in the ClearML system on a remote interactive session.
Input into `clearml-session` the ID of a Task to debug, then `clearml-session` clones the experiment's git repository and Input into `clearml-session` the ID of a Task to debug, then `clearml-session` clones the experiment's git repository and
replicates the environment on a remote machine. Then the code can be interactively executed and debugged on JupyterLab / VS Code. replicates the environment on a remote machine. Then the code can be interactively executed and debugged on JupyterLab / VS Code.
@ -153,7 +159,7 @@ The Task must be connected to a git repository, since currently single script de
1. In JupyterLab / VS Code, access the experiment's repository in the `environment/task_repository` folder. 1. In JupyterLab / VS Code, access the experiment's repository in the `environment/task_repository` folder.
## Command Line Options ### Command Line Options
<div className="tbl-cmd"> <div className="tbl-cmd">
@ -193,4 +199,16 @@ The Task must be connected to a git repository, since currently single script de
| `--verbose` | Increase verbosity of logging | `none` | | `--verbose` | Increase verbosity of logging | `none` |
| `--yes`, `-y`| Automatic yes to prompts; assume "yes" as answer to all prompts and run non-interactively |N/A| | `--yes`, `-y`| Automatic yes to prompts; assume "yes" as answer to all prompts and run non-interactively |N/A|
</div> </div>
## GUI Applications
:::important Enterprise Feature
This feature is available under the ClearML Enterprise plan
:::
The ClearML Enterprise Server provides GUI applications for setting up remote sessions in VS Code and JupyterLab. These
apps provide local links to access JupyterLab or VS Code on a remote machine over a secure and encrypted SSH connection,
letting you use the IDE as if you're running on the target machine itself
For more information, see [JupyterLab](../webapp/applications/apps_jupyter_lab.md) and/or [VS Code](../webapp/applications/apps_vscode.md).

View File

@ -56,8 +56,9 @@ module.exports = {
{'ClearML Data': ['clearml_data/clearml_data', 'clearml_data/clearml_data_cli', 'clearml_data/clearml_data_sdk', 'clearml_data/best_practices', {'ClearML Data': ['clearml_data/clearml_data', 'clearml_data/clearml_data_cli', 'clearml_data/clearml_data_sdk', 'clearml_data/best_practices',
{'Workflows': ['clearml_data/data_management_examples/workflows', 'clearml_data/data_management_examples/data_man_simple', 'clearml_data/data_management_examples/data_man_folder_sync', 'clearml_data/data_management_examples/data_man_cifar_classification', 'clearml_data/data_management_examples/data_man_python']},]}, {'Workflows': ['clearml_data/data_management_examples/workflows', 'clearml_data/data_management_examples/data_man_simple', 'clearml_data/data_management_examples/data_man_folder_sync', 'clearml_data/data_management_examples/data_man_cifar_classification', 'clearml_data/data_management_examples/data_man_python']},]},
'model_registry', 'model_registry',
'apps/clearml_session',
{'ClearML Serving':['clearml_serving/clearml_serving', 'clearml_serving/clearml_serving_setup', 'clearml_serving/clearml_serving_cli', 'clearml_serving/clearml_serving_tutorial']}, {'ClearML Serving':['clearml_serving/clearml_serving', 'clearml_serving/clearml_serving_setup', 'clearml_serving/clearml_serving_cli', 'clearml_serving/clearml_serving_tutorial']},
{'CLI Tools': ['apps/clearml_session', 'apps/clearml_task', 'apps/clearml_param_search']}, {'CLI Tools': ['apps/clearml_task', 'apps/clearml_param_search']},
{'Integrations': [ {'Integrations': [
'integrations/autokeras', 'integrations/autokeras',
'integrations/catboost', 'integrations/click', 'integrations/fastai', 'integrations/transformers', 'integrations/catboost', 'integrations/click', 'integrations/fastai', 'integrations/transformers',