This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::
This page describes the **ClearML Server** [deployment](#clearml-server-deployment-configuration) and [feature](#clearml-server-feature-configurations) configurations. Namely, it contains instructions on how to configure **ClearML Server** for:
* [Sub-domains and load balancers](#sub-domains-and-load-balancers) - An AWS load balancing example.
* [Opening Elasticsearch, MongoDB, and Redis for External Access](#opening-elasticsearch-mongodb-and-redis-for-external-access).
* [Web login authentication](#web-login-authentication) - Create and manage users and passwords.
* [Using hashed passwords](#using-hashed-passwords) - Option to use hashed passwords instead of plain-text passwords
* [Non-responsive Task watchdog](#non-responsive-task-watchdog) - For inactive experiments.
For all configuration options, see the [ClearML Configuration Reference](../configs/clearml_conf.md) page.
:::important
We recommend using the latest version of **ClearML Server**.
:::
## ClearML Server deployment configuration
**ClearML Server** supports two deployment configurations: single IP (domain) and sub-domains.
### Single IP (domain) configuration
Single IP (domain) with the following open ports:
* Web application on port `8080`
* API service on port `8008`
* File storage service on port `8081`
### Sub-domain configuration
Sub-domain configuration with default http/s ports (`80` or `443`):
* Web application on sub-domain: `app.*.*`
* API service on sub-domain: `api.*.*`
* File storage service on sub-domain: `files.*.*`
When [configuring sub-domains](#sub-domains-and-load-balancers) for **ClearML Server**, they will map to the **ClearML Server**'s
internally configured ports for the Dockers. As a result, **ClearML Server** Dockers remain accessible if, for example,
some type of port forwarding is implemented.
:::important
``app``, ``api``, and ``files`` as the sub-domain labels must be used.
:::
For example, a domain is called `mydomain.com`, and a sub-domain named `clearml.mydomain.com` is created, use the following:
*`app.clearml.mydomain.com` (web server)
*`api.clearml.mydomain.com` (API server)
*`files.clearml.mydomain.com` (file server)
Accessing the **ClearML Web UI** with `app.clearml.mydomain.com` will automatically send API requests to `api.clearml.mydomain.com`.
## ClearML Server Feature Configurations
**ClearML Server** features can be configured using either configuration files or environment variables.
### Configuration files
The **ClearML Server** uses the following configuration files:
*`apiserver.conf`
*`hosts.conf`
*`logging.conf`
*`secure.conf`
*`services.conf`
When starting up, the **ClearML Server** will look for these configuration files, in the `/opt/clearml/config` directory
(this path can be modified using the `CLEARML_CONFIG_DIR` environment variable).
The default configuration files are in the [clearml-server](https://github.com/allegroai/clearml-server/tree/master/apiserver/config/default) repository.
:::note
Within the default structure, the `services.conf` file is represented by a subdirectory with service-specific `.conf` files.
If `services.conf` is used to configure the server, any setting related to a file under the `services` subdirectory can
simply be represented by a key within the `services.conf` file.
For example, to override `multi_task_histogram_limit` that appears in the `default/services/tasks.conf`, the `services.conf` file should contain:
```
tasks {
"multi_task_histogram_limit": <new-value>
}
```
:::
### Environment variables
The **ClearML Server** supports several fixed environment variables that affect its behavior,
as well as dynamic environment variable that can be used to override any configuration file setting.
#### Fixed environment variables
General
*`CLEARML_CONFIG_DIR` allows overriding the default directory where the server looks for configuration files. Multiple directories can be specified (in the same format used for specifying the system's `PATH` env var)
Database service overrides:
*`CLEARML_MONGODB_SERVICE_HOST` allows overriding the hostname for the MongoDB service
*`CLEARML_MONGODB_SERVICE_PORT` allows overriding the port for the MongoDB service
*`CLEARML_ELASTIC_SERVICE_HOST` allows overriding the hostname for the ElasticSearch service
*`CLEARML_ELASTIC_SERVICE_PORT` allows overriding the port for the ElasticSearch service
*`CLEARML_REDIS_SERVICE_HOST` allows overriding the hostname for the Redis service
*`CLEARML_REDIS_SERVICE_PORT` allows overriding the port for the Redis service
#### Dynamic environment variables
Dynamic environment variables can be used to override any configuration setting that appears in the configuration files.
The environment variable's name should be `CLEARML__<configuration-path>`, where `<configuration-path>` represents the full path
to the configuration field being set, including the configuration file name. Elements of the configuration path
should be separated by `__` (double underscore).
For example, given the default `secure.conf` file contents:
```
...
credentials {
apiserver {
role: "system"
user_key: "defualt-key"
user_secret: "default-secret"
}
...
}
```
the default secret for the system's apiserver component can be overridden by setting the following environment variable:
* Since configuration fields may contain JSON-parsable values, make sure to always quote strings (otherwise the server
might fail to parse them)
* In order to comply with environment variables standards, it is also recommended to use only upper-case characters in
environment variable keys. For this reason, ClearML Server will always convert the configuration path specified in the
dynamic environment variable's key to lower-case before overriding configuration values with the environment variable value.
:::
## Configuration procedures
### Sub-domains and load balancers
To illustrate this configuration, we provide the following example based on AWS load balancing:
1. In the **ClearML Server**`/opt/clearml/config/apiserver.conf` file, add the following `auth.cookies` section:
auth {
cookies {
httponly: true
secure: true
domain: ".clearml.mydomain.com"
max_age: 99999999999
}
}
1. Use the following load balancer configuration:
* Listeners:
* Optional: HTTP listener, that redirects all traffic to HTTPS.
* HTTPS listener for `app.` forwarded to `AppTargetGroup`
* HTTPS listener for `api.` forwarded to `ApiTargetGroup`
* HTTPS listener for `files.` forwarded to `FilesTargetGroup`
* Target groups:
*`AppTargetGroup`: HTTP based target group, port `8080`
*`ApiTargetGroup`: HTTP based target group, port `8008`
*`FilesTargetGroup`: HTTP based target group, port `8081`
* Security and routing:
* Load balancer: make sure the load balancers are able to receive traffic from the relevant IP addresses (Security
groups and Subnets definitions).
* Instances: make sure the load balancers are able to access the instances, using the relevant ports (Security
groups definitions).
1. Restart **ClearML Server**.
### Opening Elasticsearch, MongoDB, and Redis for external access
For improved security, the ports for **ClearML Server** Elasticsearch, MongoDB, and Redis servers are not exposed by default;
they are only open internally in the docker network. If external access is needed, open these ports (but make sure to
understand the security risks involved with doing so).
:::warning
Opening the ports for Elasticsearch, MongoDB, and Redis for external access may pose a security concern and is not recommended
unless you know what you're doing. Network security measures, such as firewall configuration, should be considered when
opening ports for external access.
:::
To open external access to the Elasticsearch, MongoDB, and Redis ports:
1. Shutdown **ClearML Server**. Execute the following command (which assumes the configuration file is in the environment path).
docker-compose down
1. Edit the `docker-compose.yml` file as follows:
* In the `elasticsearch` section, add the two lines:
ports:
- "9200:9200"
* In the `mongo` section, add the two lines:
ports:
- "27017:27017"
* In the `redis` section, add the two lines:
ports:
- "6379:6379"
1. Startup **ClearML Server**.
docker-compose -f docker-compose.yml pull
docker-compose -f docker-compose.yml up -d
### Web Login Authentication
Web login authentication can be configured in the **ClearML Server** in order to permit only users provided
with credentials to access the **ClearML** system. Those credentials are a username and password.
Without web login authentication, **ClearML Server** does not restrict access (by default).
**To add web login authentication to the ClearML Server:**
1. In **ClearML Server**`/opt/clearml/config/apiserver.conf`, add the `auth.fixed_users` section and specify the users.
For example:
auth {
# Fixed users login credentials
# No other user will be able to login
fixed_users {
enabled: true
pass_hashed: false
users: [
{
username: "jane"
password: "12345678"
name: "Jane Doe"
},
{
username: "john"
password: "12345678"
name: "John Doe"
},
]
}
}
1. Restart **ClearML Server**.
### Using hashed passwords
You can also use hashed passwords instead of plain-text passwords. To do that:
- Set `pass_hashed: true`
- Use a base64-encoded hashed password in the `password` field instead of a plain-text password. Assuming Jane's plain-text password is `123456`, use the following bash command to generate the base64-encoded hashed password:
