mirror of
https://github.com/clearml/clearml-docs
synced 2025-02-12 07:25:44 +00:00
Add ClearML session command line options (#18)
This commit is contained in:
parent
b0e148a3fe
commit
63ed502115
@ -11,26 +11,47 @@ in the UI and send it for long-term training on a remote machine.
|
|||||||
|
|
||||||
**If you are not that lucky**, this section is for you :)
|
**If you are not that lucky**, this section is for you :)
|
||||||
|
|
||||||
## What does Clearml Session do?
|
## What does ClearML Session do?
|
||||||
`clearml-session` is a feature that allows to launch a session of Jupyterlab and VS Code, and to execute code on a remote
|
`clearml-session` is a feature that allows to launch a session of Jupyterlab and VS Code, and to execute code on a remote
|
||||||
machine that better meets resource needs. With this feature, local links are provided, which can be used to access
|
machine that better meets resource needs. With this feature, local links are provided, which can be used to access
|
||||||
JupyterLab and VS Code on a remote machine over a secure and encrypted SSH connection.
|
JupyterLab and VS Code on a remote machine over a secure and encrypted SSH connection.
|
||||||
|
|
||||||
![image](../img/clearml_session_jupyter.png)
|
<details className="cml-expansion-panel screenshot">
|
||||||
|
<summary className="cml-expansion-panel-summary">Jupyter-Lab Window</summary>
|
||||||
|
<div className="cml-expansion-panel-content">
|
||||||
|
|
||||||
|
![image](../img/session_jupyter.png)
|
||||||
|
|
||||||
|
</div>
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<br/>
|
||||||
|
|
||||||
|
<details className="cml-expansion-panel screenshot">
|
||||||
|
<summary className="cml-expansion-panel-summary">VS Code Window</summary>
|
||||||
|
<div className="cml-expansion-panel-content">
|
||||||
|
|
||||||
|
![image](../img/session_vs_code.png)
|
||||||
|
|
||||||
|
</div>
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<!--![image](../img/clearml_session_jupyter.png)-->
|
||||||
|
|
||||||
## How it Works
|
## How it Works
|
||||||
|
|
||||||
ClearML allows to leverage a resource (e.g. GPU or CPU machine) by utilizing the [ClearML Agent](../clearml_agent).
|
ClearML allows to leverage a resource (e.g. GPU or CPU machine) by utilizing the [ClearML Agent](../clearml_agent).
|
||||||
A ClearML Agent will be executed on target machine, and ClearML Session will instruct it to execute the Jupyter \ VSCode server to develop remotely.
|
A ClearML Agent will run on a target machine, and ClearML Session will instruct it to execute the Jupyter \ VS Code
|
||||||
After entering a `clearml-session` command with all
|
server to develop remotely.
|
||||||
specifications:
|
After entering a `clearml-session` command with all specifications:
|
||||||
|
|
||||||
1. `clearml-session` creates a new [Task](../fundamentals/task.md) that is responsible for setting up the SSH and
|
1. `clearml-session` creates a new [Task](../fundamentals/task.md) that is responsible for setting up the SSH and
|
||||||
JupyterLab / VSCode environment, according to your specifications, on the host machine.
|
JupyterLab / VS Code environment according to your specifications on the host machine.
|
||||||
|
|
||||||
1. The Task is enqueued to the queue ClearML Agent listens to and then executed by it. It will download the appropriate server and execute it.
|
1. The Task is enqueued, and a ClearML Agent pulls and executes it. The agent downloads the appropriate server and
|
||||||
|
launches it.
|
||||||
|
|
||||||
1. Once the Agent finishes the initial setup of the interactive Task, the local `cleaml-session` connects to the host
|
1. Once the agent finishes the initial setup of the interactive Task, the local `cleaml-session` connects to the host
|
||||||
machine via SSH, and tunnels both SSH and JupyterLab over the SSH connection. If a specific Docker was specified, the
|
machine via SSH, and tunnels both SSH and JupyterLab over the SSH connection. If a specific Docker was specified, the
|
||||||
JupyterLab environment will run inside the Docker.
|
JupyterLab environment will run inside the Docker.
|
||||||
|
|
||||||
@ -52,16 +73,16 @@ specifications:
|
|||||||
To run a session inside a Docker container, use the `--docker` flag and enter the docker image to use in the interactive
|
To run a session inside a Docker container, use the `--docker` flag and enter the docker image to use in the interactive
|
||||||
session.
|
session.
|
||||||
|
|
||||||
### Passing requirements
|
### Installing requirements
|
||||||
`clearml-session` can download required Python packages.
|
`clearml-session` can install required Python packages when setting up the remote environment. A `requirement.txt` file
|
||||||
A `requirement.txt` file can be attached to the command using `--requirements </file/location.txt>`.
|
can be attached to the command using `--requirements </file/location.txt>`.
|
||||||
Alternatively, packages can be manually specified, using `--packages "<package_name>"`
|
Alternatively, packages can be manually specified, using `--packages "<package_name>"`
|
||||||
(for example `--packages "keras" "clearml"`) and they'll be automatically installed.
|
(for example `--packages "keras" "clearml"`), and they'll be automatically installed.
|
||||||
|
|
||||||
### Passing Git credentials
|
### Accessing a git repository
|
||||||
To send local .git-credentials file to the interactive session, add a `--git-credentials` flag and set it to `True`.
|
To access a git repository remotely, add a `--git-credentials` flag and set it to `true`, so the local .git-credentials
|
||||||
This is helpful if working on private git repositories and allows for seemless cloning and tracking of git references,
|
file will be sent to the interactive session. This is helpful if working on private git repositories, and it allows for seamless
|
||||||
including untracked changes.
|
cloning and tracking of git references, including untracked changes.
|
||||||
|
|
||||||
### Re-launching and shutting down sessions
|
### Re-launching and shutting down sessions
|
||||||
If a `clearml-session` was launched locally and is still running on a remote machine, users can easily reconnect to it.
|
If a `clearml-session` was launched locally and is still running on a remote machine, users can easily reconnect to it.
|
||||||
@ -72,7 +93,7 @@ to an existing session will show up:
|
|||||||
Connect to active session id=c7302b564aa945408aaa40ac5c69399c [Y]/n?`
|
Connect to active session id=c7302b564aa945408aaa40ac5c69399c [Y]/n?`
|
||||||
```
|
```
|
||||||
|
|
||||||
If multiple sessions were launched from a local machine and are still active, choose the session to reconnect to:
|
If multiple sessions were launched from a local machine and are still active, choose the desired session:
|
||||||
|
|
||||||
```console
|
```console
|
||||||
Active sessions:
|
Active sessions:
|
||||||
@ -84,11 +105,13 @@ Connect to session [0-1] or 'N' to skip
|
|||||||
To shut down a remote session, which will free the `clearml-agent` and close the CLI, enter "Shutdown". If a session
|
To shut down a remote session, which will free the `clearml-agent` and close the CLI, enter "Shutdown". If a session
|
||||||
is shutdown, there is no option to reconnect to it.
|
is shutdown, there is no option to reconnect to it.
|
||||||
|
|
||||||
### Connecting to existing session
|
### Connecting to an existing session
|
||||||
If a `clearml-session` is running remotely, it's possible to continue working on the session from any machine. Starting a
|
If a `clearml-session` is running remotely, it's possible to continue working on the session from any machine.
|
||||||
session initializes a Task with a unique ID in the ClearML Server. To connect to an existing session:
|
When `clearml-session` is launched, it initializes a task with a unique ID in the ClearML Server.
|
||||||
1. Go to the ClearML UI, find the interactive session Task (by default it's in project "DevOps").
|
|
||||||
1. Click on the ID button to the right of the Task name, and copy the unique ID.
|
To connect to an existing session:
|
||||||
|
1. Go to the web UI, find the interactive session task (by default, it's in project "DevOps").
|
||||||
|
1. Click on the ID button to the right of the task name, and copy the unique ID.
|
||||||
1. Enter the following command: `clearml-session --attach <session_id>`.
|
1. Enter the following command: `clearml-session --attach <session_id>`.
|
||||||
1. Click on the JupyterLab / VS Code link that is outputted, or connect directly to the SSH session
|
1. Click on the JupyterLab / VS Code link that is outputted, or connect directly to the SSH session
|
||||||
|
|
||||||
@ -107,3 +130,27 @@ The Task must be connected to a git repository, since currently single script de
|
|||||||
1. Enter the following command: `clearml-session --debugging-session <experiment_id_here>`
|
1. Enter the following command: `clearml-session --debugging-session <experiment_id_here>`
|
||||||
1. Click on the JupyterLab / VS Code link, or connect directly to the SSH session.
|
1. Click on the JupyterLab / VS Code link, or connect directly to the SSH session.
|
||||||
1. In JupyterLab / VS Code, access the experiment's repository in the `environment/task_repository` folder.
|
1. In JupyterLab / VS Code, access the experiment's repository in the `environment/task_repository` folder.
|
||||||
|
|
||||||
|
|
||||||
|
### Command line options
|
||||||
|
|
||||||
|
<div className="tbl-cmd">
|
||||||
|
|
||||||
|
| Command line options | Description | Default value |
|
||||||
|
|-----|---|---|
|
||||||
|
| `--jupyter-lab` | Download a Jupyter-Lab environment | `true` |
|
||||||
|
| `--vscode-server` | Download a VSCode environment | `true` |
|
||||||
|
| `--public-ip` | Register the public IP of the remote machine (if you are running the session on a public cloud) | Session runs on the machine whose agent is executing the session|
|
||||||
|
| `--init-script` | Specify a BASH init script file to be executed when the interactive session is being set up | `none` or previously entered BASH script |
|
||||||
|
| `--user-folder` | Specify the path for the session's remote base folder for the session | Home folder(`~/`) or previously entered user folder path |
|
||||||
|
| `--config-file` | Specify a path to another configuration file for `clearml-session` to store its previous state | `.clearml_session.json` or previously entered configuration file |
|
||||||
|
| `--remote-gateway` | Specify a gateway IP to pass to the interactive session, if an external address needs to be accessed | `none`|
|
||||||
|
| `--base-task-id` | Pass the ID of a task that will become the base task, so the session will use its configurations | `none` or the previously entered base task |
|
||||||
|
| `--disable-keepalive` | Disable transparent proxy that keep sockets alive to maintain the connection to the remote resource | `false` |
|
||||||
|
| `--queue-excluded-tag` | The queue option list will exclude queues with specified tags. See the `tags` parameter in the [queues.create](../references/api/endpoints#post-queuescreate) API call | `none` |
|
||||||
|
| `--queue-include-tag` | The queue option list will include only queues with specified tags. See the `tags` parameter in the [queues.create](../references/api/endpoints#post-queuescreate) API call | `none` |
|
||||||
|
| `--skip-docker-network` | Pass the `--network host` flag to the Docker that is launching the remote session. See [Networking using the host network](https://docs.docker.com/network/network-tutorial-host/) | `false`|
|
||||||
|
| `--username`| Set your own SSH username for the interactive session | `root` or a previously used username |
|
||||||
|
| `--password`| Set your own SSH password for the interactive session | A randomly generated password or a previously used one |
|
||||||
|
|
||||||
|
</div>
|
BIN
docs/img/session_jupyter.png
Normal file
BIN
docs/img/session_jupyter.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 50 KiB |
BIN
docs/img/session_vs_code.png
Normal file
BIN
docs/img/session_vs_code.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 48 KiB |
@ -492,6 +492,17 @@ html[data-theme="light"] .footer__link-item[href*="stackoverflow"] {
|
|||||||
border-bottom-right-radius: var(--ifm-card-border-radius);
|
border-bottom-right-radius: var(--ifm-card-border-radius);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.markdown table code {
|
||||||
|
font-size: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
@media only screen and (min-width: 1400px) {
|
||||||
|
.tbl-cmd tbody tr td:first-child code {
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
html[data-theme="dark"] .markdown thead th {
|
html[data-theme="dark"] .markdown thead th {
|
||||||
color: var(--ifm-color-gray-600);
|
color: var(--ifm-color-gray-600);
|
||||||
}
|
}
|
||||||
|
Loading…
Reference in New Issue
Block a user