From 5f246a9d843a80a643702bea332903d850778087 Mon Sep 17 00:00:00 2001 From: revital Date: Mon, 17 Mar 2025 07:35:38 +0200 Subject: [PATCH] Edits --- .../enterprise_deploy/appgw.md | 6 +- .../appgw_install_compose.md | 78 +++++++++---------- .../enterprise_deploy/appgw_install_k8s.md | 34 +++++--- .../enterprise_deploy/multi_tenant_k8s.md | 6 +- .../apps_embed_model_deployment.md | 2 +- docs/webapp/applications/apps_gradio.md | 2 +- .../applications/apps_llama_deployment.md | 2 +- .../applications/apps_model_deployment.md | 2 +- docs/webapp/applications/apps_streamlit.md | 2 +- 9 files changed, 72 insertions(+), 62 deletions(-) diff --git a/docs/deploying_clearml/enterprise_deploy/appgw.md b/docs/deploying_clearml/enterprise_deploy/appgw.md index f556c23c..647c575a 100644 --- a/docs/deploying_clearml/enterprise_deploy/appgw.md +++ b/docs/deploying_clearml/enterprise_deploy/appgw.md @@ -30,12 +30,12 @@ their instances: * [Embedding Model Deployment](../../webapp/applications/apps_embed_model_deployment.md) * [Llama.cpp Model Deployment](../../webapp/applications/apps_llama_deployment.md) -The AI Application Gateway requires an additional component to the ClearML Server deployment: **The ClearML AI App Gateway Router**. -If your ClearML Deployment does not have the AI App Gateway Router properly installed, these application instances may not be accessible. +The AI Application Gateway requires an additional component to the ClearML Server deployment: the **ClearML App Gateway Router**. +If your ClearML Deployment does not have the App Gateway Router properly installed, these application instances may not be accessible. #### Installation -The AI Application Gateway Router supports two deployment options: +The App Gateway Router supports two deployment options: * [Docker Compose](appgw_install_compose.md) * [Kubernetes](appgw_install_k8s.md) diff --git a/docs/deploying_clearml/enterprise_deploy/appgw_install_compose.md b/docs/deploying_clearml/enterprise_deploy/appgw_install_compose.md index b9da9362..8d902803 100644 --- a/docs/deploying_clearml/enterprise_deploy/appgw_install_compose.md +++ b/docs/deploying_clearml/enterprise_deploy/appgw_install_compose.md @@ -89,15 +89,21 @@ TCP_PORT_END= ``` Edit it according to the following guidelines: -* `PROXY_TAG`: AI Application Gateway proxy tag. -* `ROUTER_TAG`: AI Application Gateway Router tag. -* `ROUTER_NAME`: Unique name for this router, needed in case of [multiple routers on the same tenant](#install-multiple-routers-for-the-same-tenant). -* `ROUTER__WEBSERVER__SERVER_PORT`: Webserver port. Default is 8080 but can be set differently based on network needs. -* `ROUTER_URL`: URL for this router that was previously configured in the load balancer starting with `https://`. -* `CLEARML_API_HOST`: ClearML API server URL usually starting with `https://api.` +* `PROXY_TAG`: AI Application Gateway proxy tag. The Docker image tag for the proxy component, which needs to be + specified during installation. This tag is provided by ClearML to ensure compatibility with the recommended version. +* `ROUTER_TAG`: App Gateway Router tag. The Docker image tag for the router component. It defines the specific version + to be installed and is provided by ClearML as part of the setup process. +* `ROUTER_NAME`: Each router needs to have a unique name across ClearML server tenant (in case of [multiple routers on the same tenant](#multiple-router-in-the-same-tenant)). +* `ROUTER__WEBSERVER__SERVER_PORT`: Webserver port. The default port is 8080, but it can be adjusted to meet specific network requirements. +* `ROUTER_URL`: External address to access the router. This can be the IP address or DNS of the node where the router + is running, or the address of a load balancer if the router operates behind a proxy/load balancer. Clients use this URL + to access AI workload applications (e.g. remote IDE, model deployment, etc.), so it must be reachable and resolvable for them. +* `CLEARML_API_HOST`: ClearML API server URL starting with `https://api.` * `CLEARML_API_ACCESS_KEY`: ClearML server API key. * `CLEARML_API_SECRET_KEY`: ClearML server secret key. -* `AUTH_COOKIE_NAME`: Cookie name used by the ClearML server to store the ClearML authentication cookie. This can usually be found in the `value_prefix` key starting with `allegro_token` in `envoy.yaml` file in the ClearML server installation (`/opt/allegro/config/envoy/envoy.yaml`) +* `AUTH_COOKIE_NAME`: Cookie used by the ClearML server to store the ClearML authentication cookie. This can usually be + found in the `envoy.yaml` file in the ClearML server installation (`/opt/allegro/config/envoy/envoy.yaml`), under the + `value_prefix` key starting with `allegro_token` * `AUTH_SECURE_ENABLED`: Enable the Set-Cookie `secure` parameter. Set to `false` in case services are exposed with `http`. * `TCP_ROUTER_ADDRESS`: Router external address, can be an IP or the host machine or a load balancer hostname, depends on network configuration * `TCP_PORT_START`: Start port for the TCP Session feature @@ -111,46 +117,40 @@ sudo docker compose --env-file runtime.env up -d ### Advanced Configuration -#### Running without Certificates -When running on `docker-compose` with an HTTP interface without certificates, set the following entry in the -`runtime.env`: +#### Using Open HTTP -``` -AUTH_SECURE_ENABLED=false -``` +To deploy the App Gateway Router on open HTTP (without a certificate), set the `AUTH_SECURE_ENABLED` entry +to `false` in the `runtime.env` file. -#### Install Multiple Routers for the Same Tenant -To deploy multiple routers within the same tenant, you must configure each router to handle specific workloads. +#### Multiple Router in the Same Tenant -Using this setting, each router will only route tasks that originated from its assigned queues. This -is important in case you have multiple networks with different agents. For example: -* Tasks started by Agent A can only be reached by Router A (within the same network), but cannot be reached by Router B -* Agent B will handle a separate set of tasks which can only be reached by Router B - -The assumption in this case is that Agent A and Agent B will service different queues, and routers must be configured to -route tasks based on these queue definitions. - -Each router in the same tenant must have: -* A unique `ROUTER_NAME` -* Distinct set of queues listed in `LISTEN_QUEUE_NAME` +If you have workloads running in separate networks that cannot communicate with each other, you need to deploy multiple +routers, one for each isolated environment. Each router will only process tasks from designated queues, ensuring that +tasks are correctly routed to agents within the same network. For example: -* **Router-A** `runtime.env` +* If Agent A and Agent B are in separate networks, each must have its own router to receive tasks. +* Router A will handle tasks from Agent A’s queues. Router B will handle tasks from Agent B’s queues. + +To achieve this, each router must be configured with: +* A unique `ROUTER_NAME` +* A distinct set of queues defined in `LISTEN_QUEUE_NAME`. + +##### Example Configuration +Each router's `runtime.env` file should include: + +* Router A: ``` - ROUTER_NAME=router-a - LISTEN_QUEUE_NAME=queue1,queue2 + ROUTER_NAME=router-a + LISTEN_QUEUE_NAME=queue1,queue2 ``` -* **Router-B** `runtime.env` +* Router B: ``` - ROUTER_NAME=router-b - LISTEN_QUEUE_NAME=queue3,queue4 - ```` - -Ensure that `LISTEN_QUEUE_NAME` is included in the [`docker-compose` environment variables](#docker-compose-file) for each router -instance. - - - + ROUTER_NAME=router-b + LISTEN_QUEUE_NAME=queue3,queue4 + ``` + +Make sure `LISTEN_QUEUE_NAME` is set in the [`docker-compose` environment variables](#docker-compose-file) for each router instance. diff --git a/docs/deploying_clearml/enterprise_deploy/appgw_install_k8s.md b/docs/deploying_clearml/enterprise_deploy/appgw_install_k8s.md index 8626af2d..de17d6e4 100644 --- a/docs/deploying_clearml/enterprise_deploy/appgw_install_k8s.md +++ b/docs/deploying_clearml/enterprise_deploy/appgw_install_k8s.md @@ -6,13 +6,14 @@ title: Kubernetes Deployment The AI Application Gateway is available under the ClearML Enterprise plan. ::: -This guide details the installation of the ClearML AI App Gateway Router. -The AI App Gateway Router enables access to session-based applications like VSCode and Jupyter. -It acts as a proxy, discovering ClearML Tasks running within its namespace and configuring them for user access. +This guide details the installation of the ClearML App Gateway Router. +The App Gateway Router enables access to your AI workloads (e.g. remote IDEs like VSCode and Jupyter, model API interface, etc.). +It acts as a proxy, identifying ClearML Tasks running within its [K8s namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) +and making them available for network access. :::important -The AI App Gateway Router must be installed in the same namespace as a dedicated ClearML Agent. -It can only discover ClearML Tasks within its own namespace. +The App Gateway Router must be installed in the same K8s namespace as a dedicated ClearML Agent. +It can only configure access for ClearML Tasks within its own namespace. ::: @@ -44,7 +45,7 @@ Replace `` with your valid GitHub token that has access to the Cle ### Prepare Values -Before installing the AI App Gateway Router, create a Helm override file: +Before installing the App Gateway Router, create a Helm override file: ``` imageCredentials: @@ -67,13 +68,13 @@ tcpSession: end: ``` -Edit it according to these guidelines: +Configuration options: * `imageCredentials.password`: ClearML DockerHub Access Token. * `clearml.apiServerKey`: ClearML server API key. * `clearml.apiServerSecret`: ClearML server secret key. -* `clearml.apiServerUrlReference`: ClearML API server URL usually starting with `https://api.`. -* `clearml.authCookieName`: Cookie name used by the ClearML server to store the ClearML authentication cookie. +* `clearml.apiServerUrlReference`: ClearML API server URL starting with `https://api.`. +* `clearml.authCookieName`: Cookie used by the ClearML server to store the ClearML authentication cookie. * `clearml.sslVerify`: Enable or disable SSL certificate validation on `apiserver` calls check. * `ingress.hostName`: Hostname of router used by the ingress controller to access it. * `tcpSession.routerAddress`: The external router address (can be an IP, hostname, or load balancer address) depending on your network setup. Ensure this address is accessible for TCP connections. @@ -82,7 +83,7 @@ Edit it according to these guidelines: * `tcpSession.portRange.end`: End port for the TCP Session feature. -The whole list of supported configuration is available with the command: +The full list of supported configuration is available with the command: ``` helm show readme allegroai-enterprise/clearml-enterprise-task-traffic-router @@ -103,6 +104,15 @@ allegroai-enterprise/clearml-enterprise-task-traffic-router \ Replace the placeholders with the following values: -* `` - Unique name for the AI App Gateway Router within the namespace. This name will appear in the UI and be used for the redirection URL. -* `` - Namespace that will be shared with a dedicated ClearML Agent. +* `` - Unique name for the App Gateway Router within the K8s namespace. This is a required parameter in + Helm, which identifies a specific installation of the chart. The release name also defines the router’s name and + appears in the UI within session app URLs in the IDE. While we typically recommend using a fixed string (e.g. + `clearml-enterprise` or `clearml-agent`), it can be customized to support multiple installations within the same + namespace by assigning different release names. +* `` - [Kubernetes Namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) + where workloads will be executed. This namespace must be shared between a dedicated ClearML Agent and an App + Gateway Router. The agent is responsible for monitoring its assigned task queues and spawning workloads within this + namespace (unless otherwise configured). Meanwhile, the router monitors the same namespace for AI workloads, such as + session-based tasks. The router has a namespace-limited scope, meaning it can only detect and manage tasks within its + assigned namespace. * `` - Version recommended by the ClearML Support Team. \ No newline at end of file diff --git a/docs/deploying_clearml/enterprise_deploy/multi_tenant_k8s.md b/docs/deploying_clearml/enterprise_deploy/multi_tenant_k8s.md index cd205eee..60650bf2 100644 --- a/docs/deploying_clearml/enterprise_deploy/multi_tenant_k8s.md +++ b/docs/deploying_clearml/enterprise_deploy/multi_tenant_k8s.md @@ -513,9 +513,9 @@ Create a `NetworkPolicy` in the tenant namespace with the following configuratio - podSelector: {} ``` -### Install AI App Gateway Router Chart +### Install the App Gateway Router Chart -Install the AI App Gateway Router in your Kubernetes cluster, allowing it to manage and route tasks: +Install the App Gateway Router in your Kubernetes cluster, allowing it to manage and route tasks: 1. Prepare the `overrides.yaml` file with the following content: @@ -531,7 +531,7 @@ Install the AI App Gateway Router in your Kubernetes cluster, allowing it to man hostName: "" ``` -2. Install AI App Gateway Router in the specified tenant namespace: +2. Install App Gateway Router in the specified tenant namespace: ``` helm install -n \\ diff --git a/docs/webapp/applications/apps_embed_model_deployment.md b/docs/webapp/applications/apps_embed_model_deployment.md index 4743ee24..40d57048 100644 --- a/docs/webapp/applications/apps_embed_model_deployment.md +++ b/docs/webapp/applications/apps_embed_model_deployment.md @@ -13,7 +13,7 @@ running, it serves your embedding model through a secure, publicly accessible ne endpoint activity and shuts down if the model remains inactive for a specified maximum idle time. :::info AI Application Gateway -The Embedding Model Deployment app makes use of the AI App Gateway Router which implements a secure, authenticated +The Embedding Model Deployment app makes use of the App Gateway Router which implements a secure, authenticated network endpoint for the model. If the ClearML AI application Gateway is not available, the model endpoint might not be accessible. diff --git a/docs/webapp/applications/apps_gradio.md b/docs/webapp/applications/apps_gradio.md index 59ba538c..e8bb9a54 100644 --- a/docs/webapp/applications/apps_gradio.md +++ b/docs/webapp/applications/apps_gradio.md @@ -16,7 +16,7 @@ The Gradio launcher monitors the Gradio app activity and shuts down if it is ina :::important AI Application Gateway -The Gradio Launcher relies on the ClearML AI App Gateway Router which implements user authentication, and redirects requests +The Gradio Launcher relies on the ClearML App Gateway Router which implements user authentication, and redirects requests to the IP/port served by the Gradio app. If the ClearML AI application Gateway is not available, the Gradio app might not be accessible. diff --git a/docs/webapp/applications/apps_llama_deployment.md b/docs/webapp/applications/apps_llama_deployment.md index 93548c2b..9cc5d37d 100644 --- a/docs/webapp/applications/apps_llama_deployment.md +++ b/docs/webapp/applications/apps_llama_deployment.md @@ -12,7 +12,7 @@ running, it serves your model through a secure, publicly accessible network endp and shuts down if the model remains inactive for a specified maximum idle time. :::important AI Application Gateway -The llama.cpp Model Deployment app makes use of the AI App Gateway Router which implements a secure, authenticated +The llama.cpp Model Deployment app makes use of the App Gateway Router which implements a secure, authenticated network endpoint for the model. If the ClearML AI application Gateway is not available, the model endpoint might not be accessible. diff --git a/docs/webapp/applications/apps_model_deployment.md b/docs/webapp/applications/apps_model_deployment.md index 7b043fd7..0936db39 100644 --- a/docs/webapp/applications/apps_model_deployment.md +++ b/docs/webapp/applications/apps_model_deployment.md @@ -13,7 +13,7 @@ it serves your model through a secure, publicly accessible network endpoint. The shuts down if the model remains inactive for a specified maximum idle time. :::info AI Application Gateway -The vLLM Model Deployment app makes use of the AI App Gateway Router which implements a secure, authenticated +The vLLM Model Deployment app makes use of the App Gateway Router which implements a secure, authenticated network endpoint for the model. If the ClearML AI application Gateway is not available, the model endpoint might not be accessible. diff --git a/docs/webapp/applications/apps_streamlit.md b/docs/webapp/applications/apps_streamlit.md index f93905c5..199e28e0 100644 --- a/docs/webapp/applications/apps_streamlit.md +++ b/docs/webapp/applications/apps_streamlit.md @@ -17,7 +17,7 @@ time. :::important AI Application Gateway -The Streamlit Launcher relies on the ClearML AI App Gateway Router which implements user authentication, and redirects requests +The Streamlit Launcher relies on the ClearML App Gateway Router which implements user authentication, and redirects requests to the IP/port served by the Streamlit app. If the ClearML AI application Gateway is not available, the Streamlit app might not be accessible.