mirror of
https://github.com/clearml/clearml-docs
synced 2025-04-10 07:56:45 +00:00
Update server Kubernetes deployment/upgrade instructions (#231)
This commit is contained in:
pollfly
GitHub
parent
80c5200069
commit
65d8b1ce9f
docs/deploying_clearml
@ -58,6 +58,6 @@ When necessary, upgrade your ClearML Server on any of the available formats:
|
|||||||
* [Google Cloud Platform](upgrade_server_gcp.md)
|
* [Google Cloud Platform](upgrade_server_gcp.md)
|
||||||
* [Linux or MacOS](upgrade_server_linux_mac.md)
|
* [Linux or MacOS](upgrade_server_linux_mac.md)
|
||||||
* [Windows 10](upgrade_server_win.md)
|
* [Windows 10](upgrade_server_win.md)
|
||||||
* [Kubernetes using Helm](upgrade_server_kubernetes_helm.md)
|
* [Kubernetes](upgrade_server_kubernetes_helm.md)
|
||||||
|
|
||||||
If you are using v0.15 or Older, [upgrade to ClearML Server](clearml_server_es7_migration.md).
|
If you are using v0.15 or Older, [upgrade to ClearML Server](clearml_server_es7_migration.md).
|
||||||
@ -2,146 +2,33 @@
|
|||||||
title: Kubernetes
|
title: Kubernetes
|
||||||
---
|
---
|
||||||
|
|
||||||
:::important
|
To upgrade an existing ClearML Server Kubernetes deployment, see [here](upgrade_server_kubernetes_helm.md).
|
||||||
**This documentation page is being updated.** Please see the new and improved **[ClearML Helm Charts repository](https://github.com/allegroai/clearml-helm-charts)**
|
|
||||||
for the most updated instructions.
|
|
||||||
:::
|
|
||||||
|
|
||||||
:::warning
|
:::info
|
||||||
If **ClearML Server** is being reinstalled, we recommend clearing browser cookies for **ClearML Server**. For example,
|
If ClearML Server is being reinstalled, we recommend clearing browser cookies for ClearML Server. For example,
|
||||||
for Firefox, go to Developer Tools > Storage > Cookies, and for Chrome, go to Developer Tools > Application > Cookies,
|
for Firefox, go to Developer Tools > Storage > Cookies, and for Chrome, go to Developer Tools > Application > Cookies,
|
||||||
and delete all cookies under the **ClearML Server** URL.
|
and delete all cookies under the ClearML Server URL.
|
||||||
:::
|
:::
|
||||||
|
|
||||||
For information about upgrading **ClearML Server** in Kubernetes Clusters using Help, see [here](upgrade_server_kubernetes_helm.md).
|
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
* A Kubernetes cluster.
|
* Set up a Kubernetes cluster - For setting up Kubernetes on various platforms refer to the Kubernetes [getting started guide](https://kubernetes.io/docs/setup).
|
||||||
* `kubectl` installed and configured (see [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) in the Kubernetes documentation).
|
* Set up a single node LOCAL Kubernetes on laptop / desktop - For setting up Kubernetes on your laptop/desktop, we suggest [kind](https://kind.sigs.k8s.io).
|
||||||
* `helm` is installed (see [Installing Helm](https://helm.sh/docs/using_helm.html#installing-helm) in the Helm documentation).
|
* Install `helm` - Helm is a tool for managing Kubernetes charts. Charts are packages of pre-configured Kubernetes resources.
|
||||||
* One node labeled `app=clearml`.
|
To install Helm, refer to the [Helm installation guide](https://helm.sh/docs/using_helm.html#installing-helm) in the Helm documentation.
|
||||||
|
Ensure that the `helm` binary is in the PATH of your shell.
|
||||||
|
|
||||||
:::warning
|
## Deployment
|
||||||
ClearML Server deployment uses node storage. If more than one node is labeled as ``app=clearml``, and the server is later
|
|
||||||
redeployed or updated, then **ClearML Server** may not locate all the data.
|
|
||||||
:::
|
|
||||||
|
|
||||||
## Deploying
|
You will create a multi-node Kubernetes cluster using Helm, and then install ClearML in your cluster. For deployment
|
||||||
|
instructions with up-to-date Helms charts, see the [clearml-helm-charts repository](https://github.com/allegroai/clearml-helm-charts/tree/main/charts/clearml#local-environment).
|
||||||
|
|
||||||
:::warning
|
:::warning Server Access
|
||||||
By default, **ClearML Server** launches with unrestricted access. To restrict **ClearML Server** access, follow the
|
By default, ClearML Server launches with unrestricted access. To restrict ClearML Server access, follow the
|
||||||
instructions in the [Security](clearml_server_security.md) page.
|
instructions in the [Security](clearml_server_security.md) page.
|
||||||
:::
|
:::
|
||||||
|
|
||||||
|
|
||||||
### Step 1: Modify Elasticsearch Default Values in the Docker Configuration File
|
|
||||||
|
|
||||||
Before deploying **ClearML Server** in a Kubernetes cluster, modify several Elasticsearch settings in the Docker configuration.
|
|
||||||
For more information, see [Install Elasticsearch with Docker](https://www.elastic.co/guide/en/elasticsearch/reference/master/docker.html#_notes_for_production_use_and_defaults)
|
|
||||||
in the Elasticsearch documentation and [Daemon configuration file](https://docs.docker.com/config/daemon/) in the Docker documentation.
|
|
||||||
|
|
||||||
**To modify Elasticsearch default values in the Docker configuration file:**
|
|
||||||
|
|
||||||
1. Connect to the node in the Kubernetes cluster labeled `app=clearml`.
|
|
||||||
1. Create or edit (if one exists) the `/etc/docker/daemon.json` file, and add or modify the `defaults-ulimits` section as
|
|
||||||
the following example shows:
|
|
||||||
|
|
||||||
{
|
|
||||||
"default-ulimits": {
|
|
||||||
"nofile": {
|
|
||||||
"name": "nofile",
|
|
||||||
"hard": 65536,
|
|
||||||
"soft": 1024
|
|
||||||
},
|
|
||||||
"memlock":
|
|
||||||
{
|
|
||||||
"name": "memlock",
|
|
||||||
"soft": -1,
|
|
||||||
"hard": -1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
1. Elasticsearch requires that the `vm.max_map_count` kernel setting, which is the maximum number of memory map areas a
|
|
||||||
process can use, be set to at least `262144`.
|
|
||||||
|
|
||||||
For CentOS 7, Ubuntu 16.04, Mint 18.3, Ubuntu 18.04 and Mint 19.x, use the following commands to set `vm.max_map_count`:
|
|
||||||
|
|
||||||
echo "vm.max_map_count=262144" > /tmp/99-clearml.conf
|
|
||||||
sudo mv /tmp/99-clearml.conf /etc/sysctl.d/99-clearml.conf
|
|
||||||
sudo sysctl -w vm.max_map_count=262144
|
|
||||||
|
|
||||||
1. Restart docker:
|
|
||||||
|
|
||||||
sudo service docker restart
|
|
||||||
|
|
||||||
### Step 2. Deploy ClearML Server in the Kubernetes Using Helm
|
|
||||||
|
|
||||||
After modifying several Elasticsearch settings in the Docker configuration (see Step 1 above), deploy **ClearML Server**.
|
|
||||||
|
|
||||||
**To deploy ClearML Server in Kubernetes using Helm:**
|
|
||||||
|
|
||||||
1. Add the clearml-server repository to Helm:
|
|
||||||
|
|
||||||
helm repo add allegroai https://allegroai.github.io/clearml-helm-charts
|
|
||||||
|
|
||||||
1. Confirm the clearml repository is now in Helm:
|
|
||||||
|
|
||||||
helm search repo allegroai
|
|
||||||
|
|
||||||
The helm search results must include `allegroai/clearml`.
|
|
||||||
|
|
||||||
1. Install `clearml` on your cluster:
|
|
||||||
|
|
||||||
helm install clearml-server allegroai/clearml -n clearml --create-namespace
|
|
||||||
|
|
||||||
A clearml `namespace` is created in the cluster and clearml-server is deployed in it.
|
|
||||||
|
|
||||||
## Port Mapping
|
|
||||||
|
|
||||||
After **ClearML Server** is deployed, the services expose the following node ports:
|
|
||||||
|
|
||||||
* API server on `30008`.
|
|
||||||
* Web server on `30080`.
|
|
||||||
* File server on `30081`.
|
|
||||||
|
|
||||||
The node ports map to the following container ports:
|
|
||||||
|
|
||||||
* `30080` maps to `clearml-webserver` container on port `8080`
|
|
||||||
* `30008` maps to `clearml-apiserver` container on port `8008`
|
|
||||||
* `30081` maps to `clearml-fileserver` container on port `8081`
|
|
||||||
|
|
||||||
:::important
|
|
||||||
We recommend using the container ports (``8080``, ``8008``, and ``8081``), or a load balancer (see the next section, [Accessing ClearML Server](#accessing-clearml-server)).
|
|
||||||
:::
|
|
||||||
|
|
||||||
## Accessing ClearML Server
|
|
||||||
|
|
||||||
**To access ClearML Server:**
|
|
||||||
|
|
||||||
* Create a load balancer and domain with records pointing to **ClearML Server** using the following rules, which **ClearML**
|
|
||||||
uses to translate domain names:
|
|
||||||
|
|
||||||
* The record to access the **ClearML Web UI**:
|
|
||||||
|
|
||||||
*app.<your domain name>.*
|
|
||||||
|
|
||||||
For example, `clearml.app.mydomainname.com` points to your node on port `30080`.
|
|
||||||
|
|
||||||
* The record to access the ClearML API:
|
|
||||||
|
|
||||||
*api.<your domain name>.*
|
|
||||||
|
|
||||||
For example, `clearml.api.mydomainname.com` points to your node on port `30008`.
|
|
||||||
|
|
||||||
* The record to access the ClearML file server:
|
|
||||||
|
|
||||||
*files.<your domain name>.*
|
|
||||||
|
|
||||||
For example, `clearmlfiles.mydomainname.com` points to your node on port `30081`.
|
|
||||||
|
|
||||||
|
|
||||||
## Next Step
|
## Next Step
|
||||||
|
|
||||||
* [Configuring ClearML for ClearML Server](clearml_config_for_clearml_server.md).
|
* [Configuring ClearML for ClearML Server](clearml_config_for_clearml_server.md).
|
||||||
|
|||||||
@ -2,26 +2,23 @@
|
|||||||
title: Kubernetes
|
title: Kubernetes
|
||||||
---
|
---
|
||||||
|
|
||||||
:::important
|
|
||||||
**This documentation page is being updated.** Please see the new and improved **[ClearML Helm Charts repository](https://github.com/allegroai/clearml-helm-charts)**
|
**To update to the latest version of the Helms chart,** execute the following:
|
||||||
for the most updated instructions.
|
|
||||||
|
```bash
|
||||||
|
helm repo update
|
||||||
|
helm upgrade clearml allegroai/clearml
|
||||||
|
```
|
||||||
|
|
||||||
|
**To change the values in an existing installation,** execute the following:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm upgrade clearml allegroai/clearml --version <CURRENT CHART VERSION> -f custom_values.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
See the [clearml-helm-charts repository](https://github.com/allegroai/clearml-helm-charts/tree/main/charts/clearml#local-environment)
|
||||||
|
to view the up-to-date charts.
|
||||||
|
|
||||||
|
:::tip
|
||||||
|
When changing values, make sure to set the chart version (`--version`) to avoid a chart update. We recommend keeping separate procedures between version and value updates to separate potential concerns.
|
||||||
:::
|
:::
|
||||||
|
|
||||||
:::note
|
|
||||||
We strongly encourage to keep the **ClearML Server** up to date, by upgrading to the current release.
|
|
||||||
:::
|
|
||||||
|
|
||||||
1. Upgrade using new or upgraded values.yaml
|
|
||||||
|
|
||||||
helm upgrade clearml-server allegroai/clearml-server-chart -f new-values.yaml
|
|
||||||
|
|
||||||
1. If **ClearML Server** was previously deployed, first delete old deployments using the following command:
|
|
||||||
|
|
||||||
helm delete --purge clearml-server
|
|
||||||
|
|
||||||
1. If upgrading from Trains Server version 0.15 or older, a data migration is required before continuing this upgrade.
|
|
||||||
See instructions [here](clearml_server_es7_migration.md).
|
|
||||||
|
|
||||||
1. Upgrade deployment to match repository version.
|
|
||||||
|
|
||||||
helm upgrade clearml-server allegroai/clearml-server-chart
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user