# ClearML Ecosystem for Kubernetes

![Version: 4.1.0](https://img.shields.io/badge/Version-4.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.6.0](https://img.shields.io/badge/AppVersion-1.6.0-informational?style=flat-square)

MLOps platform

**Homepage:** <https://clear.ml>

## Maintainers

| Name | Email | Url |
| ---- | ------ | --- |
| valeriano-manassero |  | <https://github.com/valeriano-manassero> |

## Introduction

The **clearml-server** is the backend service infrastructure for [ClearML](https://github.com/allegroai/clearml).
It allows multiple users to collaborate and manage their experiments.

**clearml-server** contains the following components:

* The ClearML Web-App, a single-page UI for experiment management and browsing
* RESTful API for:
    * Documenting and logging experiment information, statistics and results
    * Querying experiments history, logs and results
* Locally-hosted file server for storing images and models making them easily accessible using the Web-App

## Local environment

For development/evaluation it's possible to use [kind](https://kind.sigs.k8s.io).
After installation, following commands will create a complete ClearML insatllation:

```
cat <<EOF | kind create cluster --config=-                                                                  ─╯
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  # API server's default nodePort is 30008. If you customize it in helm values by
  # `apiserver.service.nodePort`, `containerPort` should match it
  - containerPort: 30008
    hostPort: 30008
    listenAddress: "127.0.0.1"
    protocol: TCP
  # Web server's default nodePort is 30080. If you customize it in helm values by
  # `webserver.service.nodePort`, `containerPort` should match it
  - containerPort: 30080
    hostPort: 30080
    listenAddress: "127.0.0.1"
    protocol: TCP
  # File server's default nodePort is 30081. If you customize it in helm values by
  # `fileserver.service.nodePort`, `containerPort` should match it
  - containerPort: 30081
    hostPort: 30081
    listenAddress: "127.0.0.1"
    protocol: TCP
  extraMounts:
  - hostPath: /tmp/clearml-kind/
    containerPath: /var/local-path-provisioner
EOF

helm install clearml allegroai/clearml
```

After deployment, the services will be exposed on localhost on the following ports:

* API server on `30008`
* Web server on `30080`
* File server on `30081`

Data persisted in every Kubernetes volume by ClearML will be accessible in /tmp/clearml-kind folder on the host.

## Production cluster environment

In a production environment it's suggested to install an ingress controller and verify that is working correctly.
During ClearML deployment enable `ingress` section of chart values.
This will create 3 ingress rules:

* `app.<your domain name>`
* `files.<your domain name>`
* `api.<your domain name>`

(*for example, `app.clearml.mydomainname.com`, `files.clearml.mydomainname.com` and `api.clearml.mydomainname.com`*)

Just pointing the domain records to the IP where ingress controller is responding will complete the deployment process.

## Upgrades/ Values upgrades

Updating to latest version of this chart can be done in two steps:

```
helm repo update
helm upgrade clearml allegroai/clearml
```

Changing values on existing installation can be done with:

```
helm upgrade clearml allegroai/clearml --version <CURRENT CHART VERSION> -f custom_values.yaml
```

Please note: updating values only should always be done setting explicit chart version to avoid a possible chart update.
Keeping separate updates procedures between version and values can be a good practice to seprate potential concerns.

## Additional Configuration for ClearML Server

You can also configure the **clearml-server** for:

* fixed users (users with credentials)
* non-responsive experiment watchdog settings

For detailed instructions, see the [Optional Configuration](https://github.com/allegroai/clearml-server#optional-configuration) section in the **clearml-server** repository README file.

## Source Code

* <https://github.com/allegroai/clearml-helm-charts>
* <https://github.com/allegroai/clearml>

## Requirements

| Repository | Name | Version |
|------------|------|---------|
| file://../../dependency_charts/elasticsearch | elasticsearch | 7.16.2 |
| file://../../dependency_charts/mongodb | mongodb | 10.3.4 |
| file://../../dependency_charts/redis | redis | 10.9.0 |

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| apiserver.additionalConfigs | object | `{}` | additional configurations that can be used by api server; check examples in values.yaml file |
| apiserver.affinity | object | `{}` |  |
| apiserver.authCookiesMaxAge | int | `864000` | Amount of seconds the authorization cookie will last in user browser |
| apiserver.configDir | string | `"/opt/clearml/config"` |  |
| apiserver.extraEnvs | list | `[]` |  |
| apiserver.image.pullPolicy | string | `"IfNotPresent"` |  |
| apiserver.image.repository | string | `"allegroai/clearml"` |  |
| apiserver.image.tag | string | `"1.6.0"` |  |
| apiserver.livenessDelay | int | `60` |  |
| apiserver.nodeSelector | object | `{}` |  |
| apiserver.podAnnotations | object | `{}` |  |
| apiserver.prepopulateArtifactsPath | string | `"/mnt/fileserver"` |  |
| apiserver.prepopulateEnabled | string | `"true"` |  |
| apiserver.prepopulateZipFiles | string | `"/opt/clearml/db-pre-populate"` |  |
| apiserver.readinessDelay | int | `60` |  |
| apiserver.replicaCount | int | `1` |  |
| apiserver.resources | object | `{}` |  |
| apiserver.service.nodePort | int | `30008` | If service.type set to NodePort, this will be set to service's nodePort field. If service.type is set to others, this field will be ignored |
| apiserver.service.port | int | `8008` |  |
| apiserver.service.type | string | `"NodePort"` | This will set to service's spec.type field |
| apiserver.tolerations | list | `[]` |  |
| clearml | object | `{"defaultCompany":"d1bd92a3b039400cbafc60a7a5b1e52b"}` | ClearMl generic configurations |
| elasticsearch.clusterHealthCheckParams | string | `"wait_for_status=yellow&timeout=1s"` |  |
| elasticsearch.clusterName | string | `"clearml-elastic"` |  |
| elasticsearch.enabled | bool | `true` |  |
| elasticsearch.esConfig."elasticsearch.yml" | string | `"xpack.security.enabled: false\n"` |  |
| elasticsearch.esJavaOpts | string | `"-Xmx2g -Xms2g"` |  |
| elasticsearch.extraEnvs[0].name | string | `"bootstrap.memory_lock"` |  |
| elasticsearch.extraEnvs[0].value | string | `"false"` |  |
| elasticsearch.extraEnvs[1].name | string | `"cluster.routing.allocation.node_initial_primaries_recoveries"` |  |
| elasticsearch.extraEnvs[1].value | string | `"500"` |  |
| elasticsearch.extraEnvs[2].name | string | `"cluster.routing.allocation.disk.watermark.low"` |  |
| elasticsearch.extraEnvs[2].value | string | `"500mb"` |  |
| elasticsearch.extraEnvs[3].name | string | `"cluster.routing.allocation.disk.watermark.high"` |  |
| elasticsearch.extraEnvs[3].value | string | `"500mb"` |  |
| elasticsearch.extraEnvs[4].name | string | `"cluster.routing.allocation.disk.watermark.flood_stage"` |  |
| elasticsearch.extraEnvs[4].value | string | `"500mb"` |  |
| elasticsearch.extraEnvs[5].name | string | `"http.compression_level"` |  |
| elasticsearch.extraEnvs[5].value | string | `"7"` |  |
| elasticsearch.extraEnvs[6].name | string | `"reindex.remote.whitelist"` |  |
| elasticsearch.extraEnvs[6].value | string | `"*.*"` |  |
| elasticsearch.extraEnvs[7].name | string | `"xpack.monitoring.enabled"` |  |
| elasticsearch.extraEnvs[7].value | string | `"false"` |  |
| elasticsearch.extraEnvs[8].name | string | `"xpack.security.enabled"` |  |
| elasticsearch.extraEnvs[8].value | string | `"false"` |  |
| elasticsearch.httpPort | int | `9200` |  |
| elasticsearch.minimumMasterNodes | int | `1` |  |
| elasticsearch.persistence.enabled | bool | `true` |  |
| elasticsearch.replicas | int | `1` |  |
| elasticsearch.resources.limits.memory | string | `"4Gi"` |  |
| elasticsearch.resources.requests.memory | string | `"4Gi"` |  |
| elasticsearch.roles.data | string | `"true"` |  |
| elasticsearch.roles.ingest | string | `"true"` |  |
| elasticsearch.roles.master | string | `"true"` |  |
| elasticsearch.roles.remote_cluster_client | string | `"true"` |  |
| elasticsearch.volumeClaimTemplate.accessModes[0] | string | `"ReadWriteOnce"` |  |
| elasticsearch.volumeClaimTemplate.resources.requests.storage | string | `"50Gi"` |  |
| externalServices.elasticsearchHost | string | `""` | Existing ElasticSearch Hostname to use if elasticsearch.enabled is false |
| externalServices.elasticsearchPort | int | `9200` | Existing ElasticSearch Port to use if elasticsearch.enabled is false |
| externalServices.mongodbHost | string | `""` | Existing MongoDB Hostname to use if elasticsearch.enabled is false |
| externalServices.mongodbPort | int | `27017` | Existing MongoDB Port to use if elasticsearch.enabled is false |
| externalServices.redisHost | string | `""` | Existing Redis Hostname to use if elasticsearch.enabled is false |
| externalServices.redisPort | int | `6379` | Existing Redis Port to use if elasticsearch.enabled is false |
| fileserver.affinity | object | `{}` |  |
| fileserver.extraEnvs | list | `[]` |  |
| fileserver.image.pullPolicy | string | `"IfNotPresent"` |  |
| fileserver.image.repository | string | `"allegroai/clearml"` |  |
| fileserver.image.tag | string | `"1.6.0"` |  |
| fileserver.nodeSelector | object | `{}` |  |
| fileserver.podAnnotations | object | `{}` |  |
| fileserver.replicaCount | int | `1` |  |
| fileserver.resources | object | `{}` |  |
| fileserver.service.nodePort | int | `30081` | If service.type set to NodePort, this will be set to service's nodePort field. If service.type is set to others, this field will be ignored |
| fileserver.service.port | int | `8081` |  |
| fileserver.service.type | string | `"NodePort"` | This will set to service's spec.type field |
| fileserver.storage.data.class | string | `""` |  |
| fileserver.storage.data.size | string | `"50Gi"` |  |
| fileserver.tolerations | list | `[]` |  |
| imageCredentials | object | `{"email":"someone@host.com","enabled":false,"existingSecret":"","password":"pwd","registry":"docker.io","username":"someone"}` | Private image registry configuration |
| imageCredentials.email | string | `"someone@host.com"` | Email |
| imageCredentials.enabled | bool | `false` | Use private authentication mode |
| imageCredentials.existingSecret | string | `""` | If this is set, chart will not generate a secret but will use what is defined here |
| imageCredentials.password | string | `"pwd"` | Registry password |
| imageCredentials.registry | string | `"docker.io"` | Registry name |
| imageCredentials.username | string | `"someone"` | Registry username |
| ingress.annotations | object | `{}` |  |
| ingress.api.annotations | object | `{}` |  |
| ingress.api.enabled | bool | `false` |  |
| ingress.api.hostName | string | `"api.clearml.127-0-0-1.nip.io"` |  |
| ingress.api.path | string | `"/"` |  |
| ingress.api.tlsSecretName | string | `""` |  |
| ingress.app.annotations | object | `{}` |  |
| ingress.app.enabled | bool | `false` |  |
| ingress.app.hostName | string | `"app.clearml.127-0-0-1.nip.io"` |  |
| ingress.app.path | string | `"/"` |  |
| ingress.app.tlsSecretName | string | `""` |  |
| ingress.files.annotations | object | `{}` |  |
| ingress.files.enabled | bool | `false` |  |
| ingress.files.hostName | string | `"files.clearml.127-0-0-1.nip.io"` |  |
| ingress.files.path | string | `"/"` |  |
| ingress.files.tlsSecretName | string | `""` |  |
| ingress.name | string | `"clearml-server-ingress"` |  |
| mongodb.architecture | string | `"standalone"` |  |
| mongodb.auth.enabled | bool | `false` |  |
| mongodb.enabled | bool | `true` |  |
| mongodb.persistence.accessModes[0] | string | `"ReadWriteOnce"` |  |
| mongodb.persistence.enabled | bool | `true` |  |
| mongodb.persistence.size | string | `"50Gi"` |  |
| mongodb.replicaCount | int | `1` |  |
| mongodb.service.name | string | `"{{ .Release.Name }}-mongodb"` |  |
| mongodb.service.port | int | `27017` |  |
| mongodb.service.portName | string | `"mongo-service"` |  |
| mongodb.service.type | string | `"ClusterIP"` |  |
| redis.cluster.enabled | bool | `false` |  |
| redis.databaseNumber | int | `0` |  |
| redis.enabled | bool | `true` |  |
| redis.master.name | string | `"{{ .Release.Name }}-redis-master"` |  |
| redis.master.persistence.accessModes[0] | string | `"ReadWriteOnce"` |  |
| redis.master.persistence.enabled | bool | `true` |  |
| redis.master.persistence.size | string | `"5Gi"` |  |
| redis.master.port | int | `6379` |  |
| redis.usePassword | bool | `false` |  |
| secret.authToken | string | `"1SCf0ov3Nm544Td2oZ0gXSrsNx5XhMWdVlKz1tOgcx158bD5RV"` | Set for auth_token field |
| secret.credentials.apiserver.accessKey | string | `"5442F3443MJMORWZA3ZH"` | Set for apiserver_key field |
| secret.credentials.apiserver.secretKey | string | `"BxapIRo9ZINi8x25CRxz8Wdmr2pQjzuWVB4PNASZqCtTyWgWVQ"` | Set for apiserver_secret field |
| secret.credentials.tests.accessKey | string | `"ENP39EQM4SLACGD5FXB7"` | Set for tests_user_key field |
| secret.credentials.tests.secretKey | string | `"lPcm0imbcBZ8mwgO7tpadutiS3gnJD05x9j7afwXPS35IKbpiQ"` | Set for tests_user_secret field |
| secret.httpSession | string | `"9Tw20RbhJ1bLBiHEOWXvhplKGUbTgLzAtwFN2oLQvWwS0uRpD5"` | Set for http_session field |
| webserver.additionalConfigs | object | `{}` |  |
| webserver.affinity | object | `{}` |  |
| webserver.extraEnvs | list | `[]` |  |
| webserver.image.pullPolicy | string | `"IfNotPresent"` |  |
| webserver.image.repository | string | `"allegroai/clearml"` |  |
| webserver.image.tag | string | `"1.6.0"` |  |
| webserver.nodeSelector | object | `{}` |  |
| webserver.podAnnotations | object | `{}` |  |
| webserver.replicaCount | int | `1` |  |
| webserver.resources | object | `{}` |  |
| webserver.service.nodePort | int | `30080` | If service.type set to NodePort, this will be set to service's nodePort field. If service.type is set to others, this field will be ignored |
| webserver.service.port | int | `80` |  |
| webserver.service.type | string | `"NodePort"` | This will set to service's spec.type field |
| webserver.tolerations | list | `[]` |  |