mirror of
https://github.com/clearml/clearml-docs
synced 2025-06-26 18:17:44 +00:00
Add air-gapped env deployment
This commit is contained in:
revital
parent
85fc719674
commit
9867b840c1
171
docs/deploying_clearml/enterprise_deploy/air_gapped_env.md
Normal file
171
docs/deploying_clearml/enterprise_deploy/air_gapped_env.md
Normal file
@ -0,0 +1,171 @@
|
||||
---
|
||||
title: Air-Gapped Environments
|
||||
---
|
||||
|
||||
This guide covers how to deploy and operate ClearML in **air-gapped environments**—environments with restricted or no
|
||||
internet access. It includes steps for locally hosting all required resources and configuring ClearML components accordingly.
|
||||
|
||||
The guide covers:
|
||||
|
||||
- Hosting dependencies for ClearML Applications
|
||||
- Configuring application containers with offline resources
|
||||
- Configuring Kubernetes deployments using private registries and image pull secrets
|
||||
|
||||
## Preparing ClearML Applications for Air-Gapped Use
|
||||
|
||||
Various application dependencies that are auto-downloaded from the internet can be locally hosted and configured for offline access.
|
||||
|
||||
### Hosting Required Python Packages
|
||||
|
||||
Ensure the following Python packages are locally hosted in your PyPI proxy or Python packages artifactory, and
|
||||
available using a local URL. Or, if you are going to use custom images, make sure they are installed.
|
||||
|
||||
```requirements
|
||||
jupyter
|
||||
jupyterlab>4,<4.4
|
||||
traitlets
|
||||
mitmproxy<10.2
|
||||
werkzeug>2,<3.0 ; python_version < '3.9'
|
||||
clearml>=1.9
|
||||
clearml_session==0.16.0
|
||||
tqdm
|
||||
boto3>=1.9
|
||||
pylint
|
||||
clearml-agent
|
||||
```
|
||||
|
||||
If hosting the previous Python packages locally, make sure to set `PIP_EXTRA_INDEX_URL=<LOCAL_REPO_URL>` for containers
|
||||
running ClearML tasks. The is an example in Kubernetes using the ClearML Agent helm values override:
|
||||
|
||||
```yaml
|
||||
agentk8sglue:
|
||||
queues:
|
||||
myQueue:
|
||||
templateOverrides:
|
||||
env:
|
||||
- name: PIP_EXTRA_INDEX_URL
|
||||
value: "<LOCAL_REPO_URL>"
|
||||
```
|
||||
|
||||
|
||||
Application environment variables (see [below](#app-specific-offline-resources)) can be set using any of the following:
|
||||
|
||||
- [ClearML Administrator Vault](../../webapp/settings/webapp_settings_admin_vaults.md) configuration: Set `agent.extra_docker_arguments`
|
||||
|
||||
- Agent config on VMs or bare-metal: Set `agent.extra_docker_arguments`
|
||||
- Kubernetes ClearML Agent deployments: Set the `basePodTemplate`
|
||||
|
||||
Ensure that:
|
||||
|
||||
* All containers and pods are configured to pull images from your private container registry
|
||||
* Custom images include Python 3
|
||||
|
||||
|
||||
### App-Specific Offline Resources
|
||||
|
||||
#### VSCode Application
|
||||
|
||||
For the ClearML VSCode Application to work offline, provide the following to all containers started by the ClearML Agent
|
||||
running GPU workloads:
|
||||
|
||||
- **VSCode Server debian package**: Set `CLEARML_SESSION_VSCODE_SERVER_DEB=<PATH_TO_DEB_FILE>`. Package can be found [here](https://github.com/coder/code-server/releases/download/v4.96.2/code-server_4.96.2_amd64.deb)
|
||||
(version number can be updated, see https://github.com/coder/code-server/releases).
|
||||
- **VSCode Python extension**: Set `CLEARML_SESSION_VSCODE_PY_EXT=<PATH_TO_EXTENSION_FILE>` pointing to the Visual Studio
|
||||
marketplace. Package can be found [here](https://marketplace.visualstudio.com/_apis/public/gallery/publishers/ms-python/vsextensions/python/2022.12.0/vspackage)
|
||||
(version number can be updated, see the [Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=ms-python.python)).
|
||||
|
||||
Example in Kubernetes using the ClearML Agent helm values override:
|
||||
|
||||
```yaml
|
||||
agentk8sglue:
|
||||
queues:
|
||||
myQueue:
|
||||
templateOverrides:
|
||||
env:
|
||||
- name: CLEARML_SESSION_VSCODE_SERVER_DEB
|
||||
value: "<PATH_TO_DEB_FILE>"
|
||||
- name: CLEARML_SESSION_VSCODE_PY_EXT
|
||||
value: "<PATH_TO_EXTENSION_FILE>"
|
||||
```
|
||||
|
||||
#### SSH Session Application
|
||||
|
||||
For air-gapped SSH applications using the DropBear server (required for non-privileged containers):
|
||||
* Download and host the [DropBear executable](https://github.com/allegroai/dropbear/releases/download/DROPBEAR_CLEARML_2023.02/dropbearmulti).
|
||||
* - Set `CLEARML_DROPBEAR_EXEC=<PATH_TO_EXECUTABLE>` in all containers started by the ClearML Agent running GPU workloads
|
||||
|
||||
|
||||
### Convert Image Registry
|
||||
|
||||
[ClearML Application](extra_configs/apps.md) installation requires running the `convert_image_registry.py` script
|
||||
included in the package. Images that need to be mirrored will be listed in the script output of the same script. Mirror these images
|
||||
to your private registry before proceeding
|
||||
with the upload of application packages.
|
||||
|
||||
## Kubernetes Environments
|
||||
|
||||
### Use a Custom imagePullSecret
|
||||
|
||||
To ensure Kubernetes workloads (ClearML Agents, Server, and App Gateway) use your private registry, configure `imagePullSecrets`
|
||||
in the appropriate Helm override files.
|
||||
|
||||
* To use a custom defined `imagePullSecret` for a **ClearML Agent** and the tasks Pods it creates, configure the following
|
||||
in your `clearml-agent-values.override.yaml` file.
|
||||
|
||||
```yaml
|
||||
imageCredentials:
|
||||
extraImagePullSecrets:
|
||||
- name: "<IMAGE_PULL_SECRET_NAME>"
|
||||
```
|
||||
|
||||
* To use a custom defined `imagePullSecret` for the **ClearML Server**, configure the following in your `clearml-values.override.yaml` file.
|
||||
|
||||
```yaml
|
||||
imageCredentials:
|
||||
existingImagePullSecrets:
|
||||
- name: "<IMAGE_PULL_SECRET_NAME>"
|
||||
```
|
||||
|
||||
* To use a custom defined `imagePullSecret` for the **ClearML App Gateway**, configure the following in your `clearml-app-gateway-values.override.yaml` file.
|
||||
|
||||
```yaml
|
||||
imageCredentials:
|
||||
existingImagePullSecrets:
|
||||
- name: "<IMAGE_PULL_SECRET_NAME>"
|
||||
```
|
||||
|
||||
### Create a Custom imagePullSecret
|
||||
|
||||
To create a registry secret in Kubernetes, use the following command example. The secret needs to be created in the namespace where it will be used.
|
||||
|
||||
```bash
|
||||
kubectl create secret docker-registry -n <NAMESPACE> <SECRET_NAME> \
|
||||
--docker-server=<REPO_URL> \
|
||||
--docker-username=<USERNAME> \
|
||||
--docker-password=<PASSWORD> \
|
||||
--docker-email=<EMAIL_OR_EMPTY_STRING>
|
||||
```
|
||||
|
||||
### List Images Used in a ClearML Helm Chart
|
||||
|
||||
To see all container images used by a ClearML Helm chart:
|
||||
|
||||
```bash
|
||||
helm template <CHART_NAME> | yq '..|.image? | select(.)' | sort -u
|
||||
```
|
||||
|
||||
:::
|
||||
This requires the `helm` and `yq` commands to be installed.
|
||||
:::
|
||||
|
||||
## Webserver
|
||||
|
||||
When using a private registry, this configuration will make the Webserver reference the correct extra index URL for
|
||||
Enterprise packages.
|
||||
|
||||
In Kubernetes:
|
||||
|
||||
```yaml
|
||||
clearml:
|
||||
extraIndexUrl: "<YOUR_REPO_URL>"
|
||||
```
|
||||
@ -1,115 +0,0 @@
|
||||
---
|
||||
title: Air-Gapped Environments
|
||||
---
|
||||
|
||||
## ClearML Applications
|
||||
|
||||
Various application dependencies that are auto-downloaded from the internet can be locally hosted and specified to the
|
||||
applications.
|
||||
|
||||
When environment variables should be provided to applications (see below), these can be set using one of the following:
|
||||
|
||||
- ClearML Administrator Vault setting the `agent.extra_docker_arguments` setting
|
||||
- Pre-configuring the specific agents configuration (when using agents installed on VMs or bare-metal machines) using the `agent.extra_docker_arguments`
|
||||
- Preconfiguring the `basePodTemplate` in Kubernetes ClearML Agent deployments
|
||||
|
||||
Also, make sure of the following:
|
||||
|
||||
- All containers/pods should be configured to use your local registry.
|
||||
- If you are going to use custom images, make sure python 3 is installed.
|
||||
- Make sure the following python packages are locally hosted in your pypi proxy or python packages artifactory, and available using some local URL. Or, if you are going to use custom images, make sure they are installed.
|
||||
|
||||
```requirements
|
||||
jupyter
|
||||
jupyterlab>4,<4.4
|
||||
traitlets
|
||||
mitmproxy<10.2
|
||||
werkzeug>2,<3.0 ; python_version < '3.9'
|
||||
clearml>=1.9
|
||||
clearml_session==0.16.0
|
||||
tqdm
|
||||
boto3>=1.9
|
||||
pylint
|
||||
clearml-agent
|
||||
```
|
||||
|
||||
- If hosting the previous python packages locally, make sure to set `PIP_EXTRA_INDEX_URL=<LOCAL_REPO_URL>` for containers running ClearML tasks. Following is an example in Kubernetes using the ClearML Agent helm values override:
|
||||
|
||||
```yaml
|
||||
agentk8sglue:
|
||||
queues:
|
||||
myQueue:
|
||||
templateOverrides:
|
||||
env:
|
||||
- name: PIP_EXTRA_INDEX_URL
|
||||
value: "<LOCAL_REPO_URL>"
|
||||
```
|
||||
|
||||
### VSCode App Resources
|
||||
|
||||
Provide the environment variables mentioned below to all containers started by the ClearML Agent running GPU workloads:
|
||||
|
||||
- **VSCode Server debian package**, set using the `CLEARML_SESSION_VSCODE_SERVER_DEB=<PATH_TO_DEB_FILE>` environment variable. Package can be found [here](https://github.com/coder/code-server/releases/download/v4.96.2/code-server_4.96.2_amd64.deb) (version number can be updated, see https://github.com/coder/code-server/releases).
|
||||
- **VSCode Python extension**, set using the `CLEARML_SESSION_VSCODE_PY_EXT=<PATH_TO_EXTENSION_FILE>` environment variable, pointing to the Visual Studio marketplace. Package can be found at https://marketplace.visualstudio.com/_apis/public/gallery/publishers/ms-python/vsextensions/python/2022.12.0/vspackage (version number can be updated, see https://marketplace.visualstudio.com/items?itemName=ms-python.python).
|
||||
|
||||
Example in Kubernetes using the ClearML Agent helm values override:
|
||||
|
||||
```yaml
|
||||
agentk8sglue:
|
||||
queues:
|
||||
myQueue:
|
||||
templateOverrides:
|
||||
env:
|
||||
- name: CLEARML_SESSION_VSCODE_SERVER_DEB
|
||||
value: "<PATH_TO_DEB_FILE>"
|
||||
- name: CLEARML_SESSION_VSCODE_PY_EXT
|
||||
value: "<PATH_TO_EXTENSION_FILE>"
|
||||
```
|
||||
|
||||
### SSH App Resources
|
||||
|
||||
If choosing to use the DropBear server instead of the SSH server (required for non-privileged containers), make sure the following packages are locally hosted and available using some local URL.
|
||||
|
||||
Provide the environment variables mentioned below to all containers started by the ClearML Agent running GPU workloads:
|
||||
|
||||
- DropBear ssh server, set using the `CLEARML_DROPBEAR_EXEC` environment variable. Package can be found [here](https://github.com/allegroai/dropbear/releases/download/DROPBEAR_CLEARML_2023.02/dropbearmulti).
|
||||
|
||||
## Kubernetes Environments
|
||||
|
||||
### Use a Custom imagePullSecret
|
||||
|
||||
To use a custom defined `imagePullSecret` for a **ClearML Agent** and the tasks Pods it creates, configure the following in your `clearml-agent-values.override.yaml` file.
|
||||
|
||||
```yaml
|
||||
imageCredentials:
|
||||
extraImagePullSecrets:
|
||||
- name: "<IMAGE_PULL_SECRET_NAME>"
|
||||
```
|
||||
|
||||
To use a custom defined `imagePullSecret` for the **ClearML Server**, configure the following in your `clearml-values.override.yaml` file.
|
||||
|
||||
```yaml
|
||||
imageCredentials:
|
||||
existingImagePullSecrets:
|
||||
- name: "<IMAGE_PULL_SECRET_NAME>"
|
||||
```
|
||||
|
||||
To use a custom defined `imagePullSecret` for the **ClearML App Gateway**, configure the following in your `clearml-app-gateway-values.override.yaml` file.
|
||||
|
||||
```yaml
|
||||
imageCredentials:
|
||||
existingImagePullSecrets:
|
||||
- name: "<IMAGE_PULL_SECRET_NAME>"
|
||||
```
|
||||
|
||||
### Create a Custom imagePullSecret
|
||||
|
||||
To create a registry secret in Kubernetes, you can use the following command example. The secret needs to be created in the namespace where it will be used.
|
||||
|
||||
```bash
|
||||
kubectl create secret docker-registry -n <NAMESPACE> <SECRET_NAME> \
|
||||
--docker-server=<REPO_URL> \
|
||||
--docker-username=<USERNAME> \
|
||||
--docker-password=<PASSWORD> \
|
||||
--docker-email=<EMAIL_OR_EMPTY_STRING>
|
||||
```
|
||||
@ -654,6 +654,7 @@ module.exports = {
|
||||
'deploying_clearml/enterprise_deploy/multi_tenant_k8s',
|
||||
'deploying_clearml/enterprise_deploy/vpc_aws',
|
||||
'deploying_clearml/enterprise_deploy/on_prem_ubuntu',
|
||||
'deploying_clearml/enterprise_deploy/air_gapped_env',
|
||||
]
|
||||
},
|
||||
{'Maintenance and Migration': [
|
||||
|
||||
Loading…
Reference in New Issue
Block a user