ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-03-23 06:18:04 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Edit AI App Gateway instructions
Some checks failed
CI / build (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Noam Wasersprung 2025-03-19 10:32:59 +02:00 committed by GitHub
commit acf5bc08d1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
9 changed files with 160 additions and 121 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
docs/deploying_clearml/enterprise_deploy/appgw.md
View File

@ -30,12 +30,12 @@ their instances:
* [Embedding Model Deployment](../../webapp/applications/apps_embed_model_deployment.md) * [Embedding Model Deployment](../../webapp/applications/apps_embed_model_deployment.md)
* [Llama.cpp Model Deployment](../../webapp/applications/apps_llama_deployment.md) * [Llama.cpp Model Deployment](../../webapp/applications/apps_llama_deployment.md)
The AI Application Gateway is provided through an additional component to the ClearML Server deployment: The ClearML Task Traffic Router. The AI Application Gateway requires an additional component to the ClearML Server deployment: the **ClearML App Gateway Router**.
If your ClearML Deployment does not have the Task Traffic Router properly installed, these application instances may not be accessible. If your ClearML Deployment does not have the App Gateway Router properly installed, these application instances may not be accessible.
#### Installation #### Installation
The Task Traffic Router supports two deployment options: The App Gateway Router supports two deployment options:
* [Docker Compose](appgw_install_compose.md) * [Docker Compose](appgw_install_compose.md)
* [Kubernetes](appgw_install_k8s.md) * [Kubernetes](appgw_install_k8s.md)

149
docs/deploying_clearml/enterprise_deploy/appgw_install_compose.md
View File

@ -40,77 +40,72 @@ This is an example of the `docker-compose` file you will need:
``` ```
version: '3.5' version: '3.5'
services: services:
task_traffic_webserver: task_traffic_webserver:
image: allegroai/task-traffic-router-webserver:${TASK-TRAFFIC-ROUTER-WEBSERVER-TAG} image: clearml/ai-gateway-proxy:${PROXY_TAG:?err}
ports: network_mode: "host"
- "80:8080" restart: unless-stopped
restart: unless-stopped container_name: task_traffic_webserver
container_name: task_traffic_webserver volumes:
volumes: - ./task_traffic_router/config/nginx:/etc/nginx/conf.d:ro
- ./task_traffic_router/config/nginx:/etc/nginx/conf.d:ro - ./task_traffic_router/config/lua:/usr/local/openresty/nginx/lua:ro
- ./task_traffic_router/config/lua:/usr/local/openresty/nginx/lua:ro task_traffic_router:
task_traffic_router: image: clearml/ai-gateway-router:${ROUTER_TAG:?err}
image: allegroai/task-traffic-router:${TASK-TRAFFIC-ROUTER-TAG} restart: unless-stopped
restart: unless-stopped container_name: task_traffic_router
container_name: task_traffic_router volumes:
volumes: - /var/run/docker.sock:/var/run/docker.sock
- /var/run/docker.sock:/var/run/docker.sock - ./task_traffic_router/config/nginx:/etc/nginx/conf.d:rw
- ./task_traffic_router/config/nginx:/etc/nginx/conf.d:rw - ./task_traffic_router/config/lua:/usr/local/openresty/nginx/lua:rw
- ./task_traffic_router/config/lua:/usr/local/openresty/nginx/lua:rw environment:
environment: - ROUTER_NAME=${ROUTER_NAME:?err}
- LOGGER_LEVEL=INFO - ROUTER__WEBSERVER__SERVER_PORT=${ROUTER__WEBSERVER__SERVER_PORT:?err}
- CLEARML_API_HOST=${CLEARML_API_HOST:?err} - ROUTER_URL=${ROUTER_URL:?err}
- CLEARML_API_ACCESS_KEY=${CLEARML_API_ACCESS_KEY:?err} - CLEARML_API_HOST=${CLEARML_API_HOST:?err}
- CLEARML_API_SECRET_KEY=${CLEARML_API_SECRET_KEY:?err} - CLEARML_API_ACCESS_KEY=${CLEARML_API_ACCESS_KEY:?err}
- ROUTER_URL=${ROUTER_URL:?err} - CLEARML_API_SECRET_KEY=${CLEARML_API_SECRET_KEY:?err}
- ROUTER_NAME=${ROUTER_NAME:?err} - AUTH_COOKIE_NAME=${AUTH_COOKIE_NAME:?err}
- AUTH_ENABLED=${AUTH_ENABLED:?err} - AUTH_SECURE_ENABLED=${AUTH_SECURE_ENABLED}
- SSL_VERIFY=${SSL_VERIFY:?err} - TCP_ROUTER_ADDRESS=${TCP_ROUTER_ADDRESS}
- AUTH_COOKIE_NAME=${AUTH_COOKIE_NAME:?err} - TCP_PORT_START=${TCP_PORT_START}
- AUTH_BASE64_JWKS_KEY=${AUTH_BASE64_JWKS_KEY:?err} - TCP_PORT_END=${TCP_PORT_END}
- LISTEN_QUEUE_NAME=${LISTEN_QUEUE_NAME}
- EXTRA_BASH_COMMAND=${EXTRA_BASH_COMMAND}
- TCP_ROUTER_ADDRESS=${TCP_ROUTER_ADDRESS}
- TCP_PORT_START=${TCP_PORT_START}
- TCP_PORT_END=${TCP_PORT_END}
``` ```
Create a *runtime.env* file containing the following entries: Create a `runtime.env` file containing the following entries:
``` ```
TASK-TRAFFIC-ROUTER-WEBSERVER-TAG= PROXY_TAG=
TASK-TRAFFIC-ROUTER-TAG= ROUTER_TAG=
CLEARML_API_HOST=https://api. ROUTER_NAME=main-router
ROUTER__WEBSERVER__SERVER_PORT=8010
ROUTER_URL=
CLEARML_API_HOST=
CLEARML_API_ACCESS_KEY= CLEARML_API_ACCESS_KEY=
CLEARML_API_SECRET_KEY= CLEARML_API_SECRET_KEY=
ROUTER_URL=
ROUTER_NAME=main-router
AUTH_ENABLED=true
SSL_VERIFY=true
AUTH_COOKIE_NAME= AUTH_COOKIE_NAME=
AUTH_BASE64_JWKS_KEY= AUTH_SECURE_ENABLED=true
LISTEN_QUEUE_NAME=
EXTRA_BASH_COMMAND=
TCP_ROUTER_ADDRESS= TCP_ROUTER_ADDRESS=
TCP_PORT_START= TCP_PORT_START=
TCP_PORT_END= TCP_PORT_END=
``` ```
Edit it according to the following guidelines: Edit it according to the following guidelines:
* `PROXY_TAG`: AI Application Gateway proxy tag. The Docker image tag for the proxy component, which needs to be
* `CLEARML_API_HOST`: URL usually starting with `https://api.` specified during installation. This tag is provided by ClearML to ensure compatibility with the recommended version.
* `CLEARML_API_ACCESS_KEY`: ClearML server api key * `ROUTER_TAG`: App Gateway Router tag. The Docker image tag for the router component. It defines the specific version
* `CLEARML_API_SECRET_KEY`: ClearML server secret key to be installed and is provided by ClearML as part of the setup process.
* `ROUTER_URL`: URL for this router that was previously configured in the load balancer starting with `https://` * `ROUTER_NAME`: In the case of [multiple routers on the same tenant](#multiple-router-in-the-same-tenant), each router
* `ROUTER_NAME`: Unique name for this router needs to have a unique name.
* `AUTH_ENABLED`: Enable or disable http calls authentication when the router is communicating with the ClearML server * `ROUTER__WEBSERVER__SERVER_PORT`: Webserver port. The default port is 8080, but it can be adjusted to meet specific network requirements.
* `SSL_VERIFY`: Enable or disable SSL certificate validation when the router is communicating with the ClearML server * `ROUTER_URL`: External address to access the router. This can be the IP address or DNS of the node where the router
* `AUTH_COOKIE_NAME`: Cookie name used by the ClearML server to store the ClearML authentication cookie. This can usually be found in the `value_prefix` key starting with `allegro_token` in `envoy.yaml` file in the ClearML server installation (`/opt/allegro/config/envoy/envoy.yaml`) (see below) is running, or the address of a load balancer if the router operates behind a proxy/load balancer. This URL is used
* `AUTH_SECURE_ENABLED`: Enable the Set-Cookie `secure` parameter to access AI workload applications (e.g. remote IDE, model deployment, etc.), so it must be reachable and resolvable for them.
* `AUTH_BASE64_JWKS_KEY`: Value form `k` key in the `jwks.json` file in the ClearML server installation * `CLEARML_API_HOST`: ClearML API server URL starting with `https://api.`
* `LISTEN_QUEUE_NAME`: (*optional*) Name of queue to check for tasks (if none, every task is checked) * `CLEARML_API_ACCESS_KEY`: ClearML server API key.
* `EXTRA_BASH_COMMAND`: Command to be launched before starting the router * `CLEARML_API_SECRET_KEY`: ClearML server secret key.
* `AUTH_COOKIE_NAME`: Cookie used by the ClearML server to store the ClearML authentication cookie. This can usually be
found in the `envoy.yaml` file in the ClearML server installation (`/opt/allegro/config/envoy/envoy.yaml`), under the
`value_prefix` key starting with `allegro_token`
* `AUTH_SECURE_ENABLED`: Enable the Set-Cookie `secure` parameter. Set to `false` in case services are exposed with `http`.
* `TCP_ROUTER_ADDRESS`: Router external address, can be an IP or the host machine or a load balancer hostname, depends on network configuration * `TCP_ROUTER_ADDRESS`: Router external address, can be an IP or the host machine or a load balancer hostname, depends on network configuration
* `TCP_PORT_START`: Start port for the TCP Session feature * `TCP_PORT_START`: Start port for the TCP Session feature
* `TCP_PORT_END`: End port for the TCP Session feature * `TCP_PORT_END`: End port for the TCP Session feature
@ -121,12 +116,42 @@ Run the following command to start the router:
sudo docker compose --env-file runtime.env up -d sudo docker compose --env-file runtime.env up -d
``` ```
:::note How to find my jwkskey ### Advanced Configuration
The *JSON Web Key Set* (*JWKS*) is a set of keys containing the public keys used to verify any JSON Web Token (JWT). #### Using Open HTTP
In a `docker-compose` server installation, this can be found in the `CLEARML__secure__auth__token_secret` env var in the apiserver server component. To deploy the App Gateway Router on open HTTP (without a certificate), set the `AUTH_SECURE_ENABLED` entry
to `false` in the `runtime.env` file.
::: #### Multiple Router in the Same Tenant
If you have workloads running in separate networks that cannot communicate with each other, you need to deploy multiple
routers, one for each isolated environment. Each router will only process tasks from designated queues, ensuring that
tasks are correctly routed to agents within the same network.
For example:
* If Agent A and Agent B are in separate networks, each must have its own router to receive tasks.
* Router A will handle tasks from Agent A’s queues. Router B will handle tasks from Agent B’s queues.
To achieve this, each router must be configured with:
* A unique `ROUTER_NAME`
* A distinct set of queues defined in `LISTEN_QUEUE_NAME`.
##### Example Configuration
Each router's `runtime.env` file should include:
* Router A:
```
ROUTER_NAME=router-a
LISTEN_QUEUE_NAME=queue1,queue2
```
* Router B:
```
ROUTER_NAME=router-b
LISTEN_QUEUE_NAME=queue3,queue4
```
Make sure `LISTEN_QUEUE_NAME` is set in the [`docker-compose` environment variables](#docker-compose-file) for each router instance.

105
docs/deploying_clearml/enterprise_deploy/appgw_install_k8s.md
View File

@ -3,17 +3,26 @@ title: Kubernetes Deployment
--- ---
:::important Enterprise Feature :::important Enterprise Feature
The Application Gateway is available under the ClearML Enterprise plan. The AI Application Gateway is available under the ClearML Enterprise plan.
:::
This guide details the installation of the ClearML App Gateway Router.
The App Gateway Router enables access to your AI workload applications (e.g. remote IDEs like VSCode and Jupyter, model API interface, etc.).
It acts as a proxy, identifying ClearML Tasks running within its [K8s namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)
and making them available for network access.
:::important
The App Gateway Router must be installed in the same K8s namespace as a dedicated ClearML Agent.
It can only configure access for ClearML Tasks within its own namespace.
::: :::
This guide details the installation of the ClearML AI Application Gateway, specifically the ClearML Task Router Component.
## Requirements ## Requirements
* Kubernetes cluster: `>= 1.21.0-0 < 1.32.0-0` * Kubernetes cluster: `>= 1.21.0-0 < 1.32.0-0`
* Helm installed and configured * Helm installed and configured
* Helm token to access `allegroai` helm-chart repo * Helm token to access `clearml` helm-chart repo
* Credentials for `allegroai` docker repo * Credentials for `clearml` docker repo
* A valid ClearML Server installation * A valid ClearML Server installation
## Optional for HTTPS ## Optional for HTTPS
@ -26,62 +35,55 @@ This guide details the installation of the ClearML AI Application Gateway, speci
### Login ### Login
``` ```
helm repo add allegroai-enterprise \ helm repo add clearml-enterprise \
https://raw.githubusercontent.com/clearml/clearml-enterprise-helm-charts/gh-pages \ https://raw.githubusercontent.com/clearml/clearml-enterprise-helm-charts/gh-pages \
--username <GITHUB_TOKEN> \ --username <GITHUB_TOKEN> \
--password <GITHUB_TOKEN> --password <GITHUB_TOKEN>
``` ```
Replace `<GITHUB_TOKEN>` with your valid GitHub token that has access to the ClearML Enterprise Helm charts repository.
### Prepare Values ### Prepare Values
Before installing the TTR, create a `helm-override` files named `task-traffic-router.values-override.yaml`: Before installing the App Gateway Router, create a Helm override file:
``` ```
imageCredentials: imageCredentials:
password: "<DOCKERHUB_TOKEN>" password: ""
clearml: clearml:
apiServerKey: "" apiServerKey: ""
apiServerSecret: "" apiServerSecret: ""
apiServerUrlReference: "https://api." apiServerUrlReference: ""
jwksKey: "" authCookieName: ""
authCookieName: "" sslVerify: true
ingress: ingress:
enabled: true enabled: true
hostName: "task-router.dev" hostName: ""
tcpSession: tcpSession:
routerAddress: "" routerAddress: ""
portRange: service:
start: type: LoadBalancer
end: portRange:
start:
end:
``` ```
Edit it accordingly to these guidelines: Configuration options:
* `clearml.apiServerUrlReference`: URL usually starting with `https://api.` * `imageCredentials.password`: ClearML DockerHub Access Token.
* `clearml.apiServerKey`: ClearML server api key * `clearml.apiServerKey`: ClearML server API key.
* `clearml.apiServerSecret`: ClearML server secret key * `clearml.apiServerSecret`: ClearML server secret key.
* `ingress.hostName`: URL of router we configured previously for load balancer starting with `https://` * `clearml.apiServerUrlReference`: ClearML API server URL starting with `https://api.`.
* `clearml.sslVerify`: Enable or disable SSL certificate validation on apiserver calls check * `clearml.authCookieName`: Cookie used by the ClearML server to store the ClearML authentication cookie.
* `clearml.authCookieName`: Value from `value_prefix` key starting with `allegro_token` in `envoy.yaml` file in ClearML server installation. * `clearml.sslVerify`: Enable or disable SSL certificate validation on `apiserver` calls check.
* `clearml.jwksKey`: Value form `k` key in `jwks.json` file in ClearML server installation (see below) * `ingress.hostName`: Hostname of router used by the ingress controller to access it.
* `tcpSession.routerAddress`: Router external address can be an IP or the host machine or a load balancer hostname, depends on the network configuration * `tcpSession.routerAddress`: The external router address (can be an IP, hostname, or load balancer address) depending on your network setup. Ensure this address is accessible for TCP connections.
* `tcpSession.portRange.start`: Start port for the TCP Session feature * `tcpSession.service.type`: Service type used to expose TCP functionality, default is `NodePort`.
* `tcpSession.portRange.end`: End port for the TCP Session feature * `tcpSession.portRange.start`: Start port for the TCP Session feature.
* `tcpSession.portRange.end`: End port for the TCP Session feature.
:::note How to find my jwkskey
The *JSON Web Key Set* (*JWKS*) is a set of keys containing the public keys used to verify any JSON Web Token (JWT).
```
kubectl -n clearml get secret clearml-conf \
-o jsonpath='{.data.secure_auth_token_secret}' \
| base64 -d && echo
```
:::
The whole list of supported configuration is available with the command: The full list of supported configuration is available with the command:
``` ```
helm show readme allegroai-enterprise/clearml-enterprise-task-traffic-router helm show readme allegroai-enterprise/clearml-enterprise-task-traffic-router
@ -94,9 +96,22 @@ To install the TTR component via Helm use the following command:
``` ```
helm upgrade --install \ helm upgrade --install \
<RELEASE_NAME> \ <RELEASE_NAME> \
-n <NAME_SPACE> \ -n <WORKLOAD_NAMESPACE> \
allegroai-enterprise/clearml-enterprise-task-traffic-router \ allegroai-enterprise/clearml-enterprise-task-traffic-router \
--version <CURRENT CHART VERSION> \ --version <CHART_VERSION> \
-f task-traffic-router.values-override.yaml -f override.yaml
``` ```
Replace the placeholders with the following values:
* `<RELEASE_NAME>` - Unique name for the App Gateway Router within the K8s namespace. This is a required parameter in
Helm, which identifies a specific installation of the chart. The release name also defines the router’s name and
appears in the UI within AI workload application URLs (e.g. Remote IDE URLs). This can be customized to support multiple installations within the same
namespace by assigning different release names.
* `<WORKLOAD_NAMESPACE>` - [Kubernetes Namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)
where workloads will be executed. This namespace must be shared between a dedicated ClearML Agent and an App
Gateway Router. The agent is responsible for monitoring its assigned task queues and spawning workloads within this
namespace. The router monitors the same namespace for AI workloads (e.g. remote IDE applications). The router has a
namespace-limited scope, meaning it can only detect and manage tasks within its
assigned namespace.
* `<CHART_VERSION>` - Version recommended by the ClearML Support Team.

11
docs/deploying_clearml/enterprise_deploy/multi_tenant_k8s.md
View File

@ -513,31 +513,30 @@ Create a `NetworkPolicy` in the tenant namespace with the following configuratio
- podSelector: {} - podSelector: {}
``` ```
### Install Task Traffic Router Chart ### Install the App Gateway Router Chart
Install the [Task Traffic Router](appgw.md) in your Kubernetes cluster, allowing it to manage and route tasks: Install the App Gateway Router in your Kubernetes cluster, allowing it to manage and route tasks:
1. Prepare the `overrides.yaml` file with the following content: 1. Prepare the `overrides.yaml` file with the following content:
``` ```
imageCredentials: imageCredentials:
password: "<allegroaienterprise_DockerHub_TOKEN>" password: "<clearmlenterprise_DockerHub_TOKEN>"
clearml: clearml:
apiServerUrlReference: "<http://clearml-enterprise-apiserver.clearml:8008>" apiServerUrlReference: "<http://clearml-enterprise-apiserver.clearml:8008>"
apiserverKey: "<TENANT_KEY>" apiserverKey: "<TENANT_KEY>"
apiserverSecret: "<TENANT_SECRET>" apiserverSecret: "<TENANT_SECRET>"
jwksKey: "<JWKS_KEY>"
ingress: ingress:
enabled: true enabled: true
hostName: "<unique url in same domain as apiserver/webserver>" hostName: "<unique url in same domain as apiserver/webserver>"
``` ```
2. Install Task Traffic Router in the specified tenant namespace: 2. Install App Gateway Router in the specified tenant namespace:
``` ```
helm install -n <TENANT_NAMESPACE> \\ helm install -n <TENANT_NAMESPACE> \\
clearml-ttr \\ clearml-ttr \\
allegroai-enterprise/clearml-task-traffic-router \\ clearml-enterprise/clearml-task-traffic-router \\
--create-namespace \\ --create-namespace \\
-f overrides.yaml -f overrides.yaml
``` ```

2
docs/webapp/applications/apps_embed_model_deployment.md
View File

@ -13,7 +13,7 @@ running, it serves your embedding model through a secure, publicly accessible ne
endpoint activity and shuts down if the model remains inactive for a specified maximum idle time. endpoint activity and shuts down if the model remains inactive for a specified maximum idle time.
:::info AI Application Gateway :::info AI Application Gateway
The Embedding Model Deployment app makes use of the ClearML Traffic Router which implements a secure, authenticated The Embedding Model Deployment app makes use of the App Gateway Router which implements a secure, authenticated
network endpoint for the model. network endpoint for the model.
If the ClearML AI application Gateway is not available, the model endpoint might not be accessible. If the ClearML AI application Gateway is not available, the model endpoint might not be accessible.

2
docs/webapp/applications/apps_gradio.md
View File

@ -16,7 +16,7 @@ The Gradio launcher monitors the Gradio app activity and shuts down if it is ina
<a id="traffic_router"/> <a id="traffic_router"/>
:::important AI Application Gateway :::important AI Application Gateway
The Gradio Launcher relies on the ClearML Traffic Router which implements user authentication, and redirects requests The Gradio Launcher relies on the ClearML App Gateway Router which implements user authentication, and redirects requests
to the IP/port served by the Gradio app. to the IP/port served by the Gradio app.
If the ClearML AI application Gateway is not available, the Gradio app might not be accessible. If the ClearML AI application Gateway is not available, the Gradio app might not be accessible.

2
docs/webapp/applications/apps_llama_deployment.md
View File

@ -12,7 +12,7 @@ running, it serves your model through a secure, publicly accessible network endp
and shuts down if the model remains inactive for a specified maximum idle time. and shuts down if the model remains inactive for a specified maximum idle time.
:::important AI Application Gateway :::important AI Application Gateway
The llama.cpp Model Deployment app makes use of the ClearML Traffic Router which implements a secure, authenticated The llama.cpp Model Deployment app makes use of the App Gateway Router which implements a secure, authenticated
network endpoint for the model. network endpoint for the model.
If the ClearML AI application Gateway is not available, the model endpoint might not be accessible. If the ClearML AI application Gateway is not available, the model endpoint might not be accessible.

2
docs/webapp/applications/apps_model_deployment.md
View File

@ -13,7 +13,7 @@ it serves your model through a secure, publicly accessible network endpoint. The
shuts down if the model remains inactive for a specified maximum idle time. shuts down if the model remains inactive for a specified maximum idle time.
:::info AI Application Gateway :::info AI Application Gateway
The vLLM Model Deployment app makes use of the ClearML Traffic Router which implements a secure, authenticated The vLLM Model Deployment app makes use of the App Gateway Router which implements a secure, authenticated
network endpoint for the model. network endpoint for the model.
If the ClearML AI application Gateway is not available, the model endpoint might not be accessible. If the ClearML AI application Gateway is not available, the model endpoint might not be accessible.

2
docs/webapp/applications/apps_streamlit.md
View File

@ -17,7 +17,7 @@ time.
<a id="traffic_router"/> <a id="traffic_router"/>
:::important AI Application Gateway :::important AI Application Gateway
The Streamlit Launcher relies on the ClearML Traffic Router which implements user authentication, and redirects requests The Streamlit Launcher relies on the ClearML App Gateway Router which implements user authentication, and redirects requests
to the IP/port served by the Streamlit app. to the IP/port served by the Streamlit app.
If the ClearML AI application Gateway is not available, the Streamlit app might not be accessible. If the ClearML AI application Gateway is not available, the Streamlit app might not be accessible.