diff --git a/docs/apps/clearml_session.md b/docs/apps/clearml_session.md index 527362d3..d440b91b 100644 --- a/docs/apps/clearml_session.md +++ b/docs/apps/clearml_session.md @@ -6,35 +6,56 @@ Machine Learning and Deep Learning development is sometimes more challenging tha you are working on an average laptop or computer, and you have a sizeable dataset that requires significant computation, your local machine may not be able to provide you with the resources for an effective workflow. -If you can run and debug your code on your own machine, congrats you are lucky! Continue doing that, then clone your code +If you can run and debug your code on your own machine, congrats you are lucky! Continue doing that, then clone your code in the UI and send it for long-term training on a remote machine. **If you are not that lucky**, this section is for you :) -## What does Clearml Session do? +## What does ClearML Session do? `clearml-session` is a feature that allows to launch a session of Jupyterlab and VS Code, and to execute code on a remote machine that better meets resource needs. With this feature, local links are provided, which can be used to access -JupyterLab and VSCode on a remote machine over a secure and encrypted SSH connection. +JupyterLab and VS Code on a remote machine over a secure and encrypted SSH connection. -![image](../img/clearml_session_jupyter.png) +
+Jupyter-Lab Window +
+ +![image](../img/session_jupyter.png) + +
+
+ +
+ +
+VS Code Window +
+ +![image](../img/session_vs_code.png) + +
+
+ + ## How it Works ClearML allows to leverage a resource (e.g. GPU or CPU machine) by utilizing the [ClearML Agent](../clearml_agent). -A ClearML Agent will be executed on target machine, and ClearML Session will instruct it to execute the Jupyter \ VSCode server to develop remotely. -After entering a `clearml-session` command with all -specifications: +A ClearML Agent will run on a target machine, and ClearML Session will instruct it to execute the Jupyter \ VS Code +server to develop remotely. +After entering a `clearml-session` command with all specifications: 1. `clearml-session` creates a new [Task](../fundamentals/task.md) that is responsible for setting up the SSH and - JupyterLab / VSCode environment, according to your specifications, on the host machine. + JupyterLab / VS Code environment according to your specifications on the host machine. - 1. The Task is enqueued to the queue ClearML Agent listens to and then executed by it. It will download the appropriate server and execute it. + 1. The Task is enqueued, and a ClearML Agent pulls and executes it. The agent downloads the appropriate server and + launches it. - 1. Once the Agent finishes the initial setup of the interactive Task, the local `cleaml-session` connects to the host + 1. Once the agent finishes the initial setup of the interactive Task, the local `cleaml-session` connects to the host machine via SSH, and tunnels both SSH and JupyterLab over the SSH connection. If a specific Docker was specified, the JupyterLab environment will run inside the Docker. - 1. The CLI outputs access links to the remote JupyterLab and VSCode sessions: + 1. The CLI outputs access links to the remote JupyterLab and VS Code sessions: ```console Interactive session is running: @@ -52,16 +73,16 @@ specifications: To run a session inside a Docker container, use the `--docker` flag and enter the docker image to use in the interactive session. -### Passing requirements -`clearml-session` can download required Python packages. -A `requirement.txt` file can be attached to the command using `--requirements `. +### Installing requirements +`clearml-session` can install required Python packages when setting up the remote environment. A `requirement.txt` file +can be attached to the command using `--requirements `. Alternatively, packages can be manually specified, using `--packages ""` -(for example `--packages "keras" "clearml"`) and they'll be automatically installed. +(for example `--packages "keras" "clearml"`), and they'll be automatically installed. -### Passing Git credentials -To send local .git-credentials file to the interactive session, add a `--git-credentials` flag and set it to `True`. -This is helpful if working on private git repositories and allows for seemless cloning and tracking of git references, -including untracked changes. +### Accessing a git repository +To access a git repository remotely, add a `--git-credentials` flag and set it to `true`, so the local .git-credentials +file will be sent to the interactive session. This is helpful if working on private git repositories, and it allows for seamless +cloning and tracking of git references, including untracked changes. ### Re-launching and shutting down sessions If a `clearml-session` was launched locally and is still running on a remote machine, users can easily reconnect to it. @@ -72,7 +93,7 @@ to an existing session will show up: Connect to active session id=c7302b564aa945408aaa40ac5c69399c [Y]/n?` ``` -If multiple sessions were launched from a local machine and are still active, choose the session to reconnect to: +If multiple sessions were launched from a local machine and are still active, choose the desired session: ```console Active sessions: @@ -84,19 +105,21 @@ Connect to session [0-1] or 'N' to skip To shut down a remote session, which will free the `clearml-agent` and close the CLI, enter "Shutdown". If a session is shutdown, there is no option to reconnect to it. -### Connecting to existing session -If a `clearml-session` is running remotely, it's possible to continue working on the session from any machine. Starting a -session initializes a Task with a unique ID in the ClearML Server. To connect to an existing session: -1. Go to the ClearML UI, find the interactive session Task (by default it's in project "DevOps"). -1. Click on the ID button to the right of the Task name, and copy the unique ID. +### Connecting to an existing session +If a `clearml-session` is running remotely, it's possible to continue working on the session from any machine. +When `clearml-session` is launched, it initializes a task with a unique ID in the ClearML Server. + +To connect to an existing session: +1. Go to the web UI, find the interactive session task (by default, it's in project "DevOps"). +1. Click on the ID button to the right of the task name, and copy the unique ID. 1. Enter the following command: `clearml-session --attach `. -1. Click on the JupyterLab / VSCode link that is outputted, or connect directly to the SSH session +1. Click on the JupyterLab / VS Code link that is outputted, or connect directly to the SSH session ### Starting a debugging session Previously executed experiments in the ClearML system can be debugged on a remote interactive session. Input into `clearml-session` the ID of a Task to debug, then `clearml-session` clones the experiment's git repository and -replicates the environment on a remote machine. Then the code can be interactively executed and debugged on JupyterLab / VSCode. +replicates the environment on a remote machine. Then the code can be interactively executed and debugged on JupyterLab / VS Code. :::note The Task must be connected to a git repository, since currently single script debugging is not supported. @@ -105,5 +128,29 @@ The Task must be connected to a git repository, since currently single script de 1. In the **ClearML web UI**, find the experiment (Task) that needs debugging. 1. Click on the ID button next to the Task name, and copy the unique ID. 1. Enter the following command: `clearml-session --debugging-session ` -1. Click on the JupyterLab / VSCode link, or connect directly to the SSH session. -1. In JupyterLab / VSCode, access the experiment's repository in the `environment/task_repository` folder. +1. Click on the JupyterLab / VS Code link, or connect directly to the SSH session. +1. In JupyterLab / VS Code, access the experiment's repository in the `environment/task_repository` folder. + + +### Command line options + +
+ +| Command line options | Description | Default value | +|-----|---|---| +| `--jupyter-lab` | Download a Jupyter-Lab environment | `true` | +| `--vscode-server` | Download a VSCode environment | `true` | +| `--public-ip` | Register the public IP of the remote machine (if you are running the session on a public cloud) | Session runs on the machine whose agent is executing the session| +| `--init-script` | Specify a BASH init script file to be executed when the interactive session is being set up | `none` or previously entered BASH script | +| `--user-folder` | Specify the path for the session's remote base folder for the session | Home folder(`~/`) or previously entered user folder path | +| `--config-file` | Specify a path to another configuration file for `clearml-session` to store its previous state | `.clearml_session.json` or previously entered configuration file | +| `--remote-gateway` | Specify a gateway IP to pass to the interactive session, if an external address needs to be accessed | `none`| +| `--base-task-id` | Pass the ID of a task that will become the base task, so the session will use its configurations | `none` or the previously entered base task | +| `--disable-keepalive` | Disable transparent proxy that keep sockets alive to maintain the connection to the remote resource | `false` | +| `--queue-excluded-tag` | The queue option list will exclude queues with specified tags. See the `tags` parameter in the [queues.create](../references/api/endpoints#post-queuescreate) API call | `none` | +| `--queue-include-tag` | The queue option list will include only queues with specified tags. See the `tags` parameter in the [queues.create](../references/api/endpoints#post-queuescreate) API call | `none` | +| `--skip-docker-network` | Pass the `--network host` flag to the Docker that is launching the remote session. See [Networking using the host network](https://docs.docker.com/network/network-tutorial-host/) | `false`| +| `--username`| Set your own SSH username for the interactive session | `root` or a previously used username | +| `--password`| Set your own SSH password for the interactive session | A randomly generated password or a previously used one | + +
\ No newline at end of file diff --git a/docs/img/session_jupyter.png b/docs/img/session_jupyter.png new file mode 100644 index 00000000..4a1515e7 Binary files /dev/null and b/docs/img/session_jupyter.png differ diff --git a/docs/img/session_vs_code.png b/docs/img/session_vs_code.png new file mode 100644 index 00000000..cf0ddf82 Binary files /dev/null and b/docs/img/session_vs_code.png differ diff --git a/src/css/custom.css b/src/css/custom.css index 25226f49..30fa7f94 100644 --- a/src/css/custom.css +++ b/src/css/custom.css @@ -492,6 +492,17 @@ html[data-theme="light"] .footer__link-item[href*="stackoverflow"] { border-bottom-right-radius: var(--ifm-card-border-radius); } +.markdown table code { + font-size: 12px; +} + +@media only screen and (min-width: 1400px) { + .tbl-cmd tbody tr td:first-child code { + white-space: nowrap; + } +} + + html[data-theme="dark"] .markdown thead th { color: var(--ifm-color-gray-600); }