mirror of
https://github.com/clearml/clearml-server
synced 2025-06-26 23:15:47 +00:00
Update readme and documentation
This commit is contained in:
allegroai
parent
1a6da301a8
commit
69f9b424c7
217
README.md
217
README.md
@ -11,23 +11,18 @@
|
||||
|
||||
The **trains-server** is the backend service infrastructure for [TRAINS](https://github.com/allegroai/trains).
|
||||
It allows multiple users to collaborate and manage their experiments.
|
||||
By default, TRAINS is set up to work with the TRAINS demo server, which is open to anyone and resets periodically.
|
||||
In order to host your own server, you will need to install **trains-server** and point TRAINS to it.
|
||||
By default, **TRAINS** is set up to work with the **TRAINS** demo server, which is open to anyone and resets periodically.
|
||||
In order to host your own server, you will need to launch **trains-server** and point **TRAINS** to it.
|
||||
|
||||
**trains-server** contains the following components:
|
||||
|
||||
* The TRAINS Web-App, a single-page UI for experiment management and browsing
|
||||
* The **TRAINS** Web-App, a single-page UI for experiment management and browsing
|
||||
* RESTful API for:
|
||||
* Documenting and logging experiment information, statistics and results
|
||||
* Querying experiments history, logs and results
|
||||
* Locally-hosted file server for storing images and models making them easily accessible using the Web-App
|
||||
|
||||
You can quickly setup your **trains-server** using:
|
||||
- [Docker Installation](#installation)
|
||||
- Pre-built Amazon [AWS image](#aws)
|
||||
- [Kubernetes Helm](https://github.com/allegroai/trains-server-helm#trains-server-for-kubernetes-clusters-using-helm)
|
||||
or manual [Kubernetes installation](https://github.com/allegroai/trains-server-k8s#trains-server-for-kubernetes-clusters)
|
||||
|
||||
You can quickly [deploy](#launching-trains-server) your **trains-server** using Docker, AWS EC2 AMI, or Kubernetes.
|
||||
|
||||
## System design
|
||||
|
||||
@ -44,136 +39,42 @@ You can quickly setup your **trains-server** using:
|
||||
- Web application on sub-domain: app.\*.\*
|
||||
- API service on sub-domain: api.\*.\*
|
||||
- File storage service on sub-domain: files.\*.\*
|
||||
|
||||
## Launching trains-server
|
||||
|
||||
## Install / Upgrade - AWS <a name="aws"></a>
|
||||
### Prerequisites
|
||||
|
||||
Use one of our pre-installed Amazon Machine Images for easy deployment in AWS.
|
||||
|
||||
For details and instructions, see [TRAINS-server: AWS pre-installed images](docs/install_aws.md).
|
||||
|
||||
## Docker Installation - Linux, macOS, and Windows <a name="installation"></a>
|
||||
|
||||
Use our pre-built Docker image for easy deployment in Linux and macOS. <br>
|
||||
For [Windows](https://github.com/allegroai/trains-server/blob/master/docs/faq.md#docker_compose_win10), please see detailed docker-compose installation instructions on our [FAQ](https://github.com/allegroai/trains-server/blob/master/docs/faq.md#docker_compose_win10).<br>
|
||||
Latest docker images can be found [here](https://hub.docker.com/r/allegroai/trains).
|
||||
|
||||
1. Setup Docker (docker-compose installation details: [Ubuntu](docs/faq.md#ubuntu) / [macOS](docs/faq.md#mac-osx))
|
||||
|
||||
<details>
|
||||
<summary>Make sure ports 8080/8081/8008 are available for the TRAINS-server services:</summary>
|
||||
The ports 8080/8081/8008 must be available for the **trains-server** services.
|
||||
|
||||
For example, to see if port `8080` is in use:
|
||||
For example, to see if port `8080` is in use:
|
||||
|
||||
```bash
|
||||
$ sudo lsof -Pn -i4 | grep :8080 | grep LISTEN
|
||||
```
|
||||
* Linux or Mac OS X:
|
||||
|
||||
sudo lsof -Pn -i4 | grep :8080 | grep LISTEN
|
||||
|
||||
* Windows:
|
||||
|
||||
netstat -an |find /i "8080"
|
||||
|
||||
### Launching
|
||||
|
||||
</details>
|
||||
|
||||
Increase vm.max_map_count for `ElasticSearch` docker
|
||||
Launch **trains-server** in any of the following formats:
|
||||
|
||||
- Linux
|
||||
```bash
|
||||
$ echo "vm.max_map_count=262144" > /tmp/99-trains.conf
|
||||
$ sudo mv /tmp/99-trains.conf /etc/sysctl.d/99-trains.conf
|
||||
$ sudo sysctl -w vm.max_map_count=262144
|
||||
$ sudo service docker restart
|
||||
```
|
||||
|
||||
- macOS
|
||||
```bash
|
||||
$ screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
|
||||
$ sysctl -w vm.max_map_count=262144
|
||||
```
|
||||
- Pre-built [AWS EC2 AMI](https://github.com/allegroai/trains-server/blob/master/docs/install_aws.md)
|
||||
- Pre-built Docker Image
|
||||
- [Linux](https://github.com/allegroai/trains-server/blob/master/docs/install_linux_mac.md)
|
||||
- [Mac OS X](https://github.com/allegroai/trains-server/blob/master/docs/install_linux_mac.md)
|
||||
- [Windows 10](https://github.com/allegroai/trains-server/blob/master/docs/install_win.md)
|
||||
- Kubernetes
|
||||
- [Kubernetes Helm](https://github.com/allegroai/trains-server-helm#prerequisites)
|
||||
- Manual [Kubernetes installation](https://github.com/allegroai/trains-server-k8s#prerequisites)
|
||||
|
||||
1. Create local directories for the databases and storage.
|
||||
## Connecting TRAINS to your trains-server
|
||||
|
||||
```bash
|
||||
$ sudo mkdir -p /opt/trains/data/elastic
|
||||
$ sudo mkdir -p /opt/trains/data/mongo/db
|
||||
$ sudo mkdir -p /opt/trains/data/mongo/configdb
|
||||
$ sudo mkdir -p /opt/trains/data/redis
|
||||
$ sudo mkdir -p /opt/trains/logs
|
||||
$ sudo mkdir -p /opt/trains/data/fileserver
|
||||
$ sudo mkdir -p /opt/trains/config
|
||||
```
|
||||
|
||||
Set folder permissions
|
||||
|
||||
- Linux
|
||||
```bash
|
||||
$ sudo chown -R 1000:1000 /opt/trains
|
||||
```
|
||||
- macOS
|
||||
```bash
|
||||
$ sudo chown -R $(whoami):staff /opt/trains
|
||||
```
|
||||
|
||||
1. Download the `docker-compose.yml` file, either download [manually](https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose.yml) or execute:
|
||||
|
||||
```bash
|
||||
$ curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose.yml -o docker-compose.yml
|
||||
```
|
||||
|
||||
1. Launch the Docker containers <a name="launch-docker"></a>
|
||||
|
||||
```bash
|
||||
$ docker-compose -f docker-compose.yml up
|
||||
```
|
||||
|
||||
1. Your server is now running on [http://localhost:8080](http://localhost:8080) and the following ports are available:
|
||||
|
||||
* Web server on port `8080`
|
||||
* API server on port `8008`
|
||||
* File server on port `8081`
|
||||
|
||||
**\* If something went wrong along the way, check our FAQ: [Docker Setup](docs/docker_setup.md#setup-docker), [Ubuntu Support](docs/faq.md#ubuntu), [macOS Support](docs/faq.md#mac-osx)**
|
||||
|
||||
## Optional Configuration
|
||||
|
||||
The **trains-server** default configuration can be easily overridden using external configuration files. By default, the server will look for these files in `/opt/trains/config`.
|
||||
|
||||
In order to apply the new configuration, you must restart the server (see [Restarting trains-server](#restart-server)).
|
||||
|
||||
### Adding Web Login Authentication
|
||||
|
||||
By default anyone can login to the **trains-server** Web-App.
|
||||
You can configure the **trains-server** to allow only a specific set of users to access the system.
|
||||
|
||||
Enable this feature by placing `apiserver.conf` file under `/opt/trains/config`.
|
||||
|
||||
Sample `apiserver.conf` configuration file can be found [here](https://github.com/allegroai/trains-server/blob/master/docs/apiserver.conf)
|
||||
|
||||
To apply the changes, you must [restart the *trains-server*](#restart-server).
|
||||
|
||||
### Configuring the Non-Responsive Experiments Watchdog
|
||||
|
||||
The non-responsive experiment watchdog, monitors experiments that were not updated for a given period of time,
|
||||
and marks them as `aborted`. The watchdog is always active with a default of 7200 seconds (2 hours) of inactivity threshold.
|
||||
|
||||
To change the watchdog's timeouts, place a `services.conf` file under `/opt/trains/config`.
|
||||
|
||||
Sample watchdog `services.conf` configuration file can be found [here](https://github.com/allegroai/trains-server/blob/master/docs/services.conf)
|
||||
|
||||
To apply the changes, you must [restart the *trains-server*](#restart-server).
|
||||
|
||||
### Restarting trains-server <a name="restart-server"></a>
|
||||
|
||||
To restart the **trains-server**, you must first stop the containers, and then restart them.
|
||||
```bash
|
||||
$ docker-compose down
|
||||
$ docker-compose -f docker-compose.yml up
|
||||
```
|
||||
|
||||
|
||||
## Configuring **TRAINS** client
|
||||
|
||||
Once you have installed the **trains-server**, make sure to configure **TRAINS** [client](https://github.com/allegroai/trains)
|
||||
to use your locally installed server (and not the demo server).
|
||||
|
||||
- Run the `trains-init` command for an interactive setup
|
||||
|
||||
- Or manually edit `~/trains.conf` file, making sure the `api_server` value is configured correctly, for example:
|
||||
By default, the **TRAINS** client is set up to work with the [**TRAINS** demo server](https://demoapp.trains.allegro.ai/).
|
||||
To have the **TRAINS** client use your **trains-server** instead:
|
||||
- Run the `trains-init` command for an interactive setup.
|
||||
- Or manually edit `~/trains.conf` file, making sure the server settings (`api_server`, `web_server`, `file_server`) are configured correctly, for example:
|
||||
|
||||
api {
|
||||
# API server on port 8008
|
||||
@ -186,26 +87,42 @@ to use your locally installed server (and not the demo server).
|
||||
files_server: "http://localhost:8081"
|
||||
}
|
||||
|
||||
* Notice that if you setup **trains-server** in a sub-domain configuration, there is no need to specify a port number,
|
||||
**Note**: If you have set up **trains-server** in a sub-domain configuration, then there is no need to specify a port number,
|
||||
it will be inferred from the http/s scheme.
|
||||
|
||||
See [Installing and Configuring TRAINS](https://github.com/allegroai/trains#configuration) for more details.
|
||||
After launching the **trains-server** and configuring the **TRAINS** client to use the **trains-server**,
|
||||
you can [use](https://github.com/allegroai/trains#using-trains) **TRAINS** in your experiments and view them in your **trains-server** web server,
|
||||
for example http://localhost:8080.
|
||||
For more information about the TRAINS client, see [**TRAINS**](https://github.com/allegroai/trains).
|
||||
|
||||
## What next?
|
||||
## Advanced Functionality
|
||||
|
||||
Now that the **trains-server** is installed, and TRAINS is configured to use it,
|
||||
you can [use](https://github.com/allegroai/trains#using-trains) TRAINS in your experiments and view them in the web server,
|
||||
for example http://localhost:8080
|
||||
**trains-server** provides a few additional useful features, which can be manually enabled:
|
||||
|
||||
* [Web login authentication](https://github.com/allegroai/trains-server/blob/master/docs/faq.md#web-auth)
|
||||
* [Non-responsive experiments watchdog](https://github.com/allegroai/trains-server/blob/master/docs/faq.md#watchdog-the-non-responsive-task-watchdog-settings)
|
||||
|
||||
## Restarting trains-server
|
||||
|
||||
To restart the **trains-server**, you must first stop the containers, and then restart them.
|
||||
|
||||
```bash
|
||||
docker-compose down
|
||||
docker-compose -f docker-compose-unified.yml up
|
||||
```
|
||||
|
||||
## Upgrading <a name="upgrade"></a>
|
||||
|
||||
We are constantly updating, improving and adding to the **trains-server**.
|
||||
New releases will include new pre-built Docker images.
|
||||
When we release a new version and include a new pre-built Docker image for it, upgrade as follows:
|
||||
**trains-server** releases are also reflected in the [docker compose configuration file](https://github.com/allegroai/trains-server/blob/master/docker-compose-unified.yml).
|
||||
We strongly encourage you to keep your **trains-server** up to date, by keeping up with the current release.
|
||||
|
||||
**Note**: The following upgrade instructions use the Linux OS as an example.
|
||||
|
||||
To upgrade your existing **trains-server** deployment:
|
||||
|
||||
1. Shut down the docker containers
|
||||
```bash
|
||||
$ docker-compose down
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
1. We highly recommend backing up your data directory before upgrading.
|
||||
@ -213,7 +130,7 @@ When we release a new version and include a new pre-built Docker image for it, u
|
||||
Assuming your data directory is `/opt/trains`, to archive all data into `~/trains_backup.tgz` execute:
|
||||
|
||||
```bash
|
||||
$ sudo tar czvf ~/trains_backup.tgz /opt/trains/data
|
||||
sudo tar czvf ~/trains_backup.tgz /opt/trains/data
|
||||
```
|
||||
|
||||
<details>
|
||||
@ -221,24 +138,24 @@ When we release a new version and include a new pre-built Docker image for it, u
|
||||
|
||||
To restore this example backup, execute:
|
||||
```bash
|
||||
$ sudo rm -R /opt/trains/data
|
||||
$ sudo tar -xzf ~/trains_backup.tgz -C /opt/trains/data
|
||||
sudo rm -R /opt/trains/data
|
||||
sudo tar -xzf ~/trains_backup.tgz -C /opt/trains/data
|
||||
```
|
||||
</details>
|
||||
|
||||
1. Download the latest `docker-compose.yml` file, either [manually](https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose.yml) or execute:
|
||||
1. Download the latest `docker-compose-unified.yml` file.
|
||||
|
||||
```bash
|
||||
$ curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose.yml -o docker-compose.yml
|
||||
curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose-unified.yml -o docker-compose-unified.yml
|
||||
```
|
||||
|
||||
1. Spin up the docker containers, it will automatically pull the latest trains-server build
|
||||
1. Spin up the docker containers, it will automatically pull the latest **trains-server** build
|
||||
```bash
|
||||
$ docker-compose -f docker-compose.yml pull
|
||||
$ docker-compose -f docker-compose.yml up
|
||||
docker-compose -f docker-compose-unified.yml pull
|
||||
docker-compose -f docker-compose-unified.yml up
|
||||
```
|
||||
|
||||
**\* If something went wrong along the way, check our FAQ: [Docker Upgrade](docs/docker_setup.md#common-docker-upgrade-errors)**
|
||||
**\* If something went wrong along the way, check our FAQ: [Common Docker Upgrade Errors](https://github.com/allegroai/trains-server/blob/master/docs/faq.md#common-docker-upgrade-errors).**
|
||||
|
||||
|
||||
## Community & Support
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
auth {
|
||||
# Fixed users login credetials
|
||||
# Fixed users login credentials
|
||||
# No other user will be able to login
|
||||
fixed_users {
|
||||
enabled: true
|
||||
|
||||
@ -1,166 +0,0 @@
|
||||
# TRAINS-server: Using Docker Pre-Built Images
|
||||
|
||||
The pre-built Docker image for the **trains-server** is the quickest way to get started with your own **TRAINS** server.
|
||||
|
||||
You can also build the entire **trains-server** architecture using the code available in the [trains-server](https://github.com/allegroai/trains-server) repository.
|
||||
|
||||
**Note**: We tested this pre-built Docker image with Linux, only. For Windows users, we recommend installing the pre-built image on a Linux virtual machine.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* You must be logged in as a user with sudo privileges
|
||||
* Use `bash` for all command-line instructions in this installation
|
||||
|
||||
## Setup Docker
|
||||
|
||||
### Step 1: Install Docker CE
|
||||
|
||||
You must first install Docker. For instructions about installing Docker, see [Supported platforms](https://docs.docker.com/install//#support) in the Docker documentation.
|
||||
|
||||
For example, to [install in Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/) / Mint (x86_64/amd64):
|
||||
|
||||
```bash
|
||||
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
|
||||
. /etc/os-release
|
||||
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $UBUNTU_CODENAME stable"
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y docker-ce
|
||||
```
|
||||
|
||||
### Step 2: Set the Maximum Number of Memory Map Areas
|
||||
|
||||
Elastic requires that the `vm.max_map_count` kernel setting, which is the maximum number of memory map areas a process can use, is set to at least 262144.
|
||||
|
||||
For CentOS 7, Ubuntu 16.04, Mint 18.3, Ubuntu 18.04 and Mint 19.x, we tested the following commands to set `vm.max_map_count`:
|
||||
|
||||
```bash
|
||||
echo "vm.max_map_count=262144" > /tmp/99-trains.conf
|
||||
sudo mv /tmp/99-trains.conf /etc/sysctl.d/99-trains.conf
|
||||
sudo sysctl -w vm.max_map_count=262144
|
||||
```
|
||||
|
||||
For information about setting this parameter on other systems, see the [elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode) documentation.
|
||||
|
||||
### Step 3: Restart the Docker daemon
|
||||
|
||||
Restart the Docker daemon.
|
||||
|
||||
```bash
|
||||
sudo service docker restart
|
||||
```
|
||||
|
||||
### Step 4: Choose a Data Directory
|
||||
|
||||
Choose a directory on your system in which all data maintained by the **trains-server** is stored.
|
||||
Create this directory, and set its owner and group to `uid` 1000. The data stored in this directory includes the database, uploaded files and logs.
|
||||
|
||||
For example, if your data directory is `/opt/trains`, then use the following command:
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /opt/trains/data/elastic
|
||||
sudo mkdir -p /opt/trains/data/mongo/db
|
||||
sudo mkdir -p /opt/trains/data/mongo/configdb
|
||||
sudo mkdir -p /opt/trains/data/redis
|
||||
sudo mkdir -p /opt/trains/logs
|
||||
sudo mkdir -p /opt/trains/data/fileserver
|
||||
sudo mkdir -p /opt/trains/config
|
||||
|
||||
sudo chown -R 1000:1000 /opt/trains
|
||||
```
|
||||
|
||||
## TRAINS-server: Manually Launching Docker Containers <a name="launch"></a>
|
||||
|
||||
You can manually launch the Docker containers using the following commands.
|
||||
|
||||
If your data directory is not `/opt/trains`, then in the five `docker run` commands below, you must replace all occurrences of `/opt/trains` with your data directory path.
|
||||
|
||||
1. Launch the **trains-elastic** Docker container.
|
||||
|
||||
sudo docker run -d --restart="always" --name="trains-elastic" -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -e "bootstrap.memory_lock=true" -e "cluster.name=trains" -e "discovery.zen.minimum_master_nodes=1" -e "node.name=trains" -e "script.inline=true" -e "script.update=true" -e "thread_pool.bulk.queue_size=2000" -e "thread_pool.search.queue_size=10000" -e "xpack.security.enabled=false" -e "xpack.monitoring.enabled=false" -e "cluster.routing.allocation.node_initial_primaries_recoveries=500" -e "node.ingest=true" -e "http.compression_level=7" -e "reindex.remote.whitelist=*.*" -e "script.painless.regex.enabled=true" --network="host" -v /opt/trains/data/elastic:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:5.6.16
|
||||
|
||||
1. Launch the **trains-mongo** Docker container.
|
||||
|
||||
sudo docker run -d --restart="always" --name="trains-mongo" -v /opt/trains/data/mongo/db:/data/db -v /opt/trains/data/mongo/configdb:/data/configdb --network="host" mongo:3.6.5
|
||||
|
||||
1. Launch the **trains-redis** Docker container.
|
||||
|
||||
sudo docker run -d --restart="always" --name="trains-redis" -v /opt/trains/data/redis:/data --network="host" redis:5.0
|
||||
|
||||
1. Launch the **trains-fileserver** Docker container.
|
||||
|
||||
sudo docker run -d --restart="always" --name="trains-fileserver" --network="host" -v /opt/trains/logs:/var/log/trains -v /opt/trains/data/fileserver:/mnt/fileserver allegroai/trains:latest fileserver
|
||||
|
||||
1. Launch the **trains-apiserver** Docker container.
|
||||
|
||||
sudo docker run -d --restart="always" --name="trains-apiserver" --network="host" -v /opt/trains/logs:/var/log/trains -v /opt/trains/config:/opt/trains/config allegroai/trains:latest apiserver
|
||||
|
||||
1. Launch the **trains-webserver** Docker container.
|
||||
|
||||
sudo docker run -d --restart="always" --name="trains-webserver" -p 8080:80 allegroai/trains:latest webserver
|
||||
|
||||
1. Your server is now running on [http://localhost:8080](http://localhost:8080) and the following ports are available:
|
||||
|
||||
* API server on port `8008`
|
||||
* Web server on port `8080`
|
||||
* File server on port `8081`
|
||||
|
||||
## Manually Upgrading TRAINS-server Containers <a name="upgrade"></a>
|
||||
|
||||
We are constantly updating, improving and adding to the **trains-server**.
|
||||
New releases will include new pre-built Docker images.
|
||||
When we release a new version and include a new pre-built Docker image for it, upgrade as follows:
|
||||
|
||||
1. Shut down and remove each of your Docker instances using the following commands:
|
||||
|
||||
```bash
|
||||
$ sudo docker stop <docker-name>
|
||||
$ sudo docker rm -v <docker-name>
|
||||
```
|
||||
|
||||
The Docker names are (see [Launching Docker Containers](#launch-docker)):
|
||||
|
||||
* `trains-elastic`
|
||||
* `trains-mongo`
|
||||
* `trains-redis`
|
||||
* `trains-fileserver`
|
||||
* `trains-apiserver`
|
||||
* `trains-webserver`
|
||||
|
||||
2. We highly recommend backing up your data directory!. A simple way to do that is using `tar`:
|
||||
|
||||
For example, if your data directory is `/opt/trains`, use the following command:
|
||||
|
||||
```bash
|
||||
$ sudo tar czvf ~/trains_backup.tgz /opt/trains/data
|
||||
```
|
||||
This backups all data to an archive in your home directory.
|
||||
|
||||
To restore this example backup, use the following command:
|
||||
```bash
|
||||
$ sudo rm -R /opt/trains/data
|
||||
$ sudo tar -xzf ~/trains_backup.tgz -C /opt/trains/data
|
||||
```
|
||||
|
||||
3. Pull the new **trains-server** docker image using the following command:
|
||||
|
||||
```bash
|
||||
$ sudo docker pull allegroai/trains:latest
|
||||
```
|
||||
|
||||
If you wish to pull a different version, replace `latest` with the required version number, for example:
|
||||
```bash
|
||||
$ sudo docker pull allegroai/trains:0.11.0
|
||||
```
|
||||
|
||||
4. Launch the newly released Docker image (see [Launching Docker Containers](#trains-server-manually-launching-docker-containers-)).
|
||||
|
||||
|
||||
#### Common Docker Upgrade Errors
|
||||
|
||||
* In case of a docker error: "... The container name "/trains-???" is already in use by ..."
|
||||
Try removing deprecated images with:
|
||||
```bash
|
||||
$ docker rm -f $(docker ps -a -q)
|
||||
```
|
||||
|
||||
382
docs/faq.md
382
docs/faq.md
@ -1,107 +1,42 @@
|
||||
# TRAINS-server FAQ
|
||||
# trains-server FAQ
|
||||
|
||||
* [Deploying trains-server on Kubernetes clusters](#kubernetes)
|
||||
Launching **trains-server**
|
||||
|
||||
* [Creating a Helm Chart for trains-server Kubernetes deployment](#helm)
|
||||
* How do I launch **trains-server** on:
|
||||
|
||||
* [Running trains-server on Mac OS X](#mac-osx)
|
||||
|
||||
* [Running trains-server on Windows 10](#docker_compose_win10)
|
||||
|
||||
* [Installing trains-server on stand alone Linux Ubuntu systems ](#ubuntu)
|
||||
|
||||
* [Resolving port conflicts preventing fixed users mode authentication and login](#port-conflict)
|
||||
|
||||
* [Configuring trains-server for sub-domains and load balancers](#sub-domains)
|
||||
|
||||
|
||||
### Deploying trains-server on Kubernetes clusters <a name="kubernetes"></a>
|
||||
|
||||
**trains-server** supports Kubernetes. See [trains-server-k8s](https://github.com/allegroai/trains-server-k8s)
|
||||
which contains the YAML files describing the required services and detailed instructions for deploying
|
||||
**trains-server** to a Kubernetes clusters.
|
||||
|
||||
### Creating a Helm Chart for trains-server Kubernetes deployment <a name="helm"></a>
|
||||
|
||||
**trains-server** supports creating a Helm chart for Kubernetes deployment. See [trains-server-helm](https://github.com/allegroai/trains-server-helm)
|
||||
which you can use to create a Helm chart for **trains-server** and contains detailed instructions for deploying
|
||||
**trains-server** to a Kubernetes clusters using Helm.
|
||||
|
||||
### Running trains-server on Mac OS X <a name="mac-osx"></a>
|
||||
|
||||
To install and configure **trains-server** on Mac OS X, follow the steps below.
|
||||
|
||||
1. Install [docker for OS X](https://docs.docker.com/docker-for-mac/install/).
|
||||
|
||||
1. Configure [Docker](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode).
|
||||
|
||||
$ screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
|
||||
$ sysctl -w vm.max_map_count=262144
|
||||
|
||||
1. Create local directories for the databases and storage.
|
||||
|
||||
$ sudo mkdir -p /opt/trains/data/elastic
|
||||
$ sudo mkdir -p /opt/trains/data/mongo/db
|
||||
$ sudo mkdir -p /opt/trains/data/mongo/configdb
|
||||
$ sudo mkdir -p /opt/trains/data/redis
|
||||
$ sudo mkdir -p /opt/trains/logs
|
||||
$ sudo mkdir -p /opt/trains/config
|
||||
$ sudo mkdir -p /opt/trains/data/fileserver
|
||||
$ sudo chown -R $(whoami):staff /opt/trains
|
||||
|
||||
1. Open the Docker app, select **Preferences**, and then on the **File Sharing** tab, add `/opt/trains`.
|
||||
|
||||
1. Clone the [trains-server](https://github.com/allegroai/trains-server) repository and change directories to the new **trains-server** directory.
|
||||
|
||||
$ git clone https://github.com/allegroai/trains-server.git
|
||||
$ cd trains-server
|
||||
|
||||
1. Run `docker-compose` with the unified docker image.
|
||||
|
||||
$ docker-compose -f docker-compose-unified.yml up
|
||||
|
||||
Your server is now running on [http://localhost:8080](http://localhost:8080)
|
||||
|
||||
### Running trains-server on Windows 10 <a name="docker_compose_win10"></a>
|
||||
|
||||
You can run **trains-server** on Windows 10 using Docker Desktop for Windows (see the Docker [System Requirements](https://docs.docker.com/docker-for-windows/install/#system-requirements)).
|
||||
|
||||
To run **trains-server** on Windows 10, follow the steps below.
|
||||
|
||||
1. Install the Docker Desktop for Windows application by either:
|
||||
|
||||
* Following the [Install Docker Desktop on Windows](https://docs.docker.com/docker-for-windows/install/) instructions.
|
||||
* Running the Docker installation [wizard](https://hub.docker.com/?overlay=onboarding).
|
||||
|
||||
1. Increase the memory allocation in Docker Desktop to `4GB`.
|
||||
|
||||
1. In your Windows notification area (system tray), right click the Docker icon.
|
||||
* [Stand alone Linux Ubuntu systems?](#ubuntu)
|
||||
|
||||
1. Click *Settings*, *Advanced*, and then set the memory to at least `4096`.
|
||||
* [Mac OS X?](#mac-osx)
|
||||
|
||||
1. Click *Apply*.
|
||||
* [Windows 10?](#docker_compose_win10)
|
||||
|
||||
1. Create local directories for data and logs. Open PowerShell and execute the following commands:
|
||||
* [How do I restart trains-server?](#restart)
|
||||
|
||||
mkdir c:\opt\trains\logs
|
||||
mkdir c:\opt\trains\config
|
||||
mkdir c:\opt\trains\data
|
||||
mkdir c:\opt\trains\data\elastic
|
||||
mkdir c:\opt\trains\data\redis
|
||||
mkdir c:\opt\trains\data\fileserver
|
||||
Kubernetes
|
||||
|
||||
1. Save the **trains-server** docker-compose YAML file [docker-compose-win10.yml](https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose-win10.yml) as `c:\opt\trains\docker-compose.yml`.
|
||||
* [Can I deploy trains-server on Kubernetes clusters?](#kubernetes)
|
||||
|
||||
1. Run `docker-compose`. In PowerShell, execute the following commands:
|
||||
* [Can I create a Helm Chart for trains-server Kubernetes deployment?](#helm)
|
||||
|
||||
cd c:\opt\trains\
|
||||
docker-compose up
|
||||
Configuration
|
||||
|
||||
Your server is now running on [http://localhost:8080](http://localhost:8080)
|
||||
* [How do I configure trains-server for sub-domains and load balancers?](#sub-domains)
|
||||
|
||||
### Installing trains-server on stand alone Linux Ubuntu systems <a name="ubuntu"></a>
|
||||
* [Can I add web login authentication to trains-server?](#web-auth)
|
||||
|
||||
To install **trains-server** on a stand alone Linux Ubuntu, follow the steps belows.
|
||||
* [Can I modify the non-responsive experiment watchdog settings?](#watchdog)
|
||||
|
||||
Troubleshooting
|
||||
|
||||
* [How do I fix Docker upgrade errors?](#common-docker-upgrade-errors)
|
||||
|
||||
* [Why is web login authentication not working?](#port-conflict)
|
||||
|
||||
## Launching **trains-server**
|
||||
|
||||
### How do I launch trains-server on stand alone Linux Ubuntu systems? <a name="ubuntu"></a>
|
||||
|
||||
To launch **trains-server** on a stand alone Linux Ubuntu:
|
||||
|
||||
1. Install [docker for Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/).
|
||||
|
||||
@ -114,79 +49,125 @@ To install **trains-server** on a stand alone Linux Ubuntu, follow the steps bel
|
||||
|
||||
**WARNING**: This clears all existing **TRAINS** databases.
|
||||
|
||||
$ sudo rm -R /opt/trains/
|
||||
sudo rm -R /opt/trains/
|
||||
|
||||
1. Create local directories for the databases and storage.
|
||||
|
||||
$ sudo mkdir -p /opt/trains/data/elastic
|
||||
$ sudo mkdir -p /opt/trains/data/mongo/db
|
||||
$ sudo mkdir -p /opt/trains/data/mongo/configdb
|
||||
$ sudo mkdir -p /opt/trains/logs
|
||||
$ sudo mkdir -p /opt/trains/config
|
||||
$ sudo mkdir -p /opt/trains/data/fileserver
|
||||
$ sudo chown -R 1000:1000 /opt/trains
|
||||
sudo mkdir -p /opt/trains/data/elastic
|
||||
sudo mkdir -p /opt/trains/data/mongo/db
|
||||
sudo mkdir -p /opt/trains/data/mongo/configdb
|
||||
sudo mkdir -p /opt/trains/logs
|
||||
sudo mkdir -p /opt/trains/config
|
||||
sudo mkdir -p /opt/trains/data/fileserver
|
||||
sudo chown -R 1000:1000 /opt/trains
|
||||
|
||||
1. Clone the [trains-server](https://github.com/allegroai/trains-server) repository and change directories to the new **trains-server** directory.
|
||||
|
||||
$ git clone https://github.com/allegroai/trains-server.git
|
||||
$ cd trains-server
|
||||
git clone https://github.com/allegroai/trains-server.git
|
||||
cd trains-server
|
||||
|
||||
1. Run `docker-compose`
|
||||
|
||||
$ /usr/local/bin/docker-compose -f docker-compose.yml up
|
||||
/usr/local/bin/docker-compose -f docker-compose.yml up
|
||||
|
||||
Your server is now running on [http://localhost:8080](http://localhost:8080)
|
||||
|
||||
### How do I launch trains-server on Mac OS X? <a name="mac-osx"></a>
|
||||
|
||||
To launch **trains-server** on Mac OS X:
|
||||
|
||||
1. Install [docker for OS X](https://docs.docker.com/docker-for-mac/install/).
|
||||
|
||||
1. Configure [Docker](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode).
|
||||
|
||||
screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
|
||||
sysctl -w vm.max_map_count=262144
|
||||
|
||||
1. Create local directories for the databases and storage.
|
||||
|
||||
sudo mkdir -p /opt/trains/data/elastic
|
||||
sudo mkdir -p /opt/trains/data/mongo/db
|
||||
sudo mkdir -p /opt/trains/data/mongo/configdb
|
||||
sudo mkdir -p /opt/trains/data/redis
|
||||
sudo mkdir -p /opt/trains/logs
|
||||
sudo mkdir -p /opt/trains/config
|
||||
sudo mkdir -p /opt/trains/data/fileserver
|
||||
sudo chown -R $(whoami):staff /opt/trains
|
||||
|
||||
1. Open the Docker app, select **Preferences**, and then on the **File Sharing** tab, add `/opt/trains`.
|
||||
|
||||
1. Clone the [trains-server](https://github.com/allegroai/trains-server) repository and change directories to the new **trains-server** directory.
|
||||
|
||||
git clone https://github.com/allegroai/trains-server.git
|
||||
cd trains-server
|
||||
|
||||
1. Run `docker-compose` with the unified docker image.
|
||||
|
||||
docker-compose -f docker-compose-unified.yml up
|
||||
|
||||
Your server is now running on [http://localhost:8080](http://localhost:8080)
|
||||
|
||||
### Resolving port conflicts preventing fixed users mode authentication and login <a name="port-conflict"></a>
|
||||
### How do I launch trains-server on Windows 10? <a name="docker_compose_win10"></a>
|
||||
|
||||
A port conflict may occur between the **trains-server** MongoDB and Elastic instances and other
|
||||
instances running on your system. **trains-server** uses the following default ports which may be in conflict with other instances:
|
||||
You can run **trains-server** on Windows 10 using Docker Desktop for Windows (see the Docker [System Requirements](https://docs.docker.com/docker-for-windows/install/#system-requirements)).
|
||||
|
||||
* MongoDB port `27017`
|
||||
* Elastic port `9200`
|
||||
To launch **trains-server** on Windows 10:
|
||||
|
||||
You can check for port conflicts in the logs in `/opt/trains/log`.
|
||||
1. Install the Docker Desktop for Windows application by either:
|
||||
|
||||
If a port conflict occurs, first change the port in your **trains-server** `/opt/trains/server/config/default/hosts.conf` file to the new port and then
|
||||
run the `docker run` command with the `port` option specifying the new port to restart the **trains-server** instance.
|
||||
* following the [Install Docker Desktop on Windows](https://docs.docker.com/docker-for-windows/install/) instructions.
|
||||
* running the Docker installation [wizard](https://hub.docker.com/?overlay=onboarding).
|
||||
|
||||
For example, to resolve a MongoDB port conflict change port `27017` to `27018`:
|
||||
1. Increase the memory allocation in Docker Desktop to `4GB`.
|
||||
|
||||
1. Modify `/opt/trains/server/config/default/hosts.conf` changing the ports in the `mongo` section:
|
||||
1. In your Windows notification area (system tray), right click the Docker icon.
|
||||
|
||||
1. Click *Settings*, *Advanced*, and then set the memory to at least `4096`.
|
||||
|
||||
1. Click *Apply*.
|
||||
|
||||
elastic {
|
||||
events {
|
||||
hosts: [{host: "127.0.0.1", port: 9200}]
|
||||
args {
|
||||
timeout: 60
|
||||
dead_timeout: 10
|
||||
max_retries: 5
|
||||
retry_on_timeout: true
|
||||
}
|
||||
index_version: "1"
|
||||
}
|
||||
}
|
||||
1. Create local directories for data and logs. Open PowerShell and execute the following commands:
|
||||
|
||||
mongo {
|
||||
backend {
|
||||
host: "mongodb://127.0.0.1:27018/backend"
|
||||
}
|
||||
auth {
|
||||
host: "mongodb://127.0.0.1:27018/auth"
|
||||
}
|
||||
}
|
||||
cd c:
|
||||
mkdir c:\opt\trains\data
|
||||
mkdir c:\opt\trains\logs
|
||||
|
||||
2. Start the **trains-server** MongoDB container using `--port 27018`.
|
||||
1. Download the **trains-server** docker-compose YAML file [docker-compose-win10.yml](https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose-win10.yml) as `c:\opt\trains\docker-compose.yml`.
|
||||
|
||||
sudo docker run -d --restart="always" --name="trains-mongo" -v /opt/trains/data/mongo/db:/data/db -v /opt/trains/data/mongo/configdb:/data/configdb --network="host" mongo:3.6.5 mongod --port 27018
|
||||
1. Run `docker-compose`. In PowerShell, execute the following commands:
|
||||
|
||||
In a future version of **trains-server**, to start the API server, environment variables will be available to use instead of modifying the configuration file (instead of Step 1 above).
|
||||
The environment variables will be available to set different ports for both MongoDB and Elastic instances:
|
||||
docker-compose -f up docker-compose-win10.yml
|
||||
|
||||
* `MONGODB_SERVICE_PORT` (e.g., `MONGODB_SERVICE_PORT=27018`)
|
||||
* `ELASTIC_SERVICE_POST` (e.g., `ELASTIC_SERVICE_POST=9201`)
|
||||
Your server is now running on [http://localhost:8080](http://localhost:8080)
|
||||
|
||||
### Configuring trains-server for sub-domains and load balancers <a name="sub-domains"></a>
|
||||
### How do I restart trains-server? <a name="restart"></a>
|
||||
|
||||
Restart *trains-server* by first stopping the Docker containers and then restarting them.
|
||||
|
||||
```bash
|
||||
docker-compose down
|
||||
docker-compose up -f docker-compose-unified.yml
|
||||
```
|
||||
|
||||
**Note**: If you are using a different docker-compose YAML file, specify that file.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
### Can I deploy trains-server on Kubernetes clusters? <a name="kubernetes"></a>
|
||||
|
||||
**trains-server** supports Kubernetes. See [trains-server-k8s](https://github.com/allegroai/trains-server-k8s)
|
||||
which contains the YAML files describing the required services and detailed instructions for deploying
|
||||
**trains-server** to a Kubernetes clusters.
|
||||
|
||||
### Can I create a Helm Chart for trains-server Kubernetes deployment? <a name="helm"></a>
|
||||
|
||||
**trains-server** supports creating a Helm chart for Kubernetes deployment. See [trains-server-helm](https://github.com/allegroai/trains-server-helm)
|
||||
which you can use to create a Helm chart for **trains-server** and contains detailed instructions for deploying
|
||||
**trains-server** to a Kubernetes clusters using Helm.
|
||||
|
||||
## Configuration
|
||||
|
||||
### How do I configure trains-server for sub-domains and load balancers? <a name="sub-domains"></a>
|
||||
|
||||
You can configure **trains-server** for sub-domains and a load balancer.
|
||||
|
||||
@ -222,3 +203,126 @@ For example, if your domain is `trains.mydomain.com` and your sub-domains are `a
|
||||
|
||||
1. Run the Docker containers with our updated `docker run` commands (see [Launching Docker Containers](#https://github.com/allegroai/trains-server#launching-docker-containers)).
|
||||
|
||||
### Can I add web login authentication to trains-server? <a name="web-auth"></a>
|
||||
|
||||
By default, anyone can login to the **trains-server** Web-App.
|
||||
You can configure the **trains-server** to allow only a specific set of users to access the system.
|
||||
|
||||
To add web login authentication to **trains-server**:
|
||||
|
||||
1. If you are not using the current **trains-server** version, then [upgrade](https://github.com/allegroai/trains-server#upgrade).
|
||||
|
||||
1. In `/opt/trains/config/apiserver.conf`, add the `auth` section and in it specify the users, for example:
|
||||
|
||||
**Note**: A sample `apiserver.conf` configuration file is also available [here](https://github.com/allegroai/trains-server/blob/master/docs/apiserver.conf).
|
||||
|
||||
auth {
|
||||
# Fixed users login credentials
|
||||
# No other user will be able to login
|
||||
fixed_users {
|
||||
enabled: true
|
||||
users: [
|
||||
{
|
||||
username: "jane"
|
||||
password: "12345678"
|
||||
name: "Jane Doe"
|
||||
},
|
||||
{
|
||||
username: "john"
|
||||
password: "12345678"
|
||||
name: "John Doe"
|
||||
},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
1. Restart **trains-server** (see the [Restarting trains-server](#restart) FAQ).
|
||||
|
||||
### Can I modify the experiment watchdog settings? <a name="watchdog"></a>
|
||||
|
||||
The non-responsive experiment watchdog monitors experiments that were not updated for a specified period of time
|
||||
and marks them as `aborted`. The watchdog is always active.
|
||||
|
||||
You can modify the following settings for the watchdog:
|
||||
|
||||
* the time threshold (in seconds) of experiment inactivity (default value is 7200 seconds (2 hours))
|
||||
* the time interval (in seconds) between watchdog cycles
|
||||
|
||||
To change the watchdog's settings:
|
||||
|
||||
1. In `/opt/trains/config`, add the `services.conf` file and in it specify the watchdog settings, for example:
|
||||
|
||||
**Note**: A sample watchdog `services.conf` configuration file is also available [here](https://github.com/allegroai/trains-server/blob/master/docs/services.conf).
|
||||
|
||||
tasks {
|
||||
non_responsive_tasks_watchdog {
|
||||
# In-progress tasks that haven't been updated for at least 'value' seconds will be stopped by the watchdog
|
||||
threshold_sec: 7200
|
||||
|
||||
# Watchdog will sleep for this number of seconds after each cycle
|
||||
watch_interval_sec: 900
|
||||
}
|
||||
}
|
||||
|
||||
1. Restart **trains-server** (see the [Restarting trains-server](#restart) FAQ).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### How do I fix Docker upgrade errors? <a name="common-docker-upgrade-errors"></a>
|
||||
|
||||
To resolve the Docker error "... The container name "/trains-???" is already in use by ...", try removing deprecated images:
|
||||
|
||||
docker rm -f $(docker ps -a -q)
|
||||
|
||||
### Why is web login authentication not working?
|
||||
|
||||
A port conflict between the **trains-server** MongoDB and / or Elastic instances, and other
|
||||
instances running on your system may prevent web login authentication
|
||||
from working correctly.
|
||||
|
||||
**trains-server** uses the following default ports which may be in conflict with other instances:
|
||||
|
||||
* MongoDB port `27017`
|
||||
* Elastic port `9200`
|
||||
|
||||
You can check for port conflicts in the logs in `/opt/trains/log`.
|
||||
|
||||
If a port conflict occurs, change the MongoDB and / or Elastic ports in the `docker-compose-unified.yml`,
|
||||
and then run the Docker compose commands to restart the **trains-server** instance.
|
||||
|
||||
To change the MongoDB and / or Elastic ports for **trains-server**:
|
||||
|
||||
1. Edit the `docker-compose-unified.yml` file.
|
||||
|
||||
1. In the `services/trainsserver/environment` section, add the following environment variable(s):
|
||||
|
||||
* For MongoDB:
|
||||
|
||||
MONGODB_SERVICE_PORT: <new-mongodb-port>
|
||||
|
||||
* For Elastic:
|
||||
|
||||
ELASTIC_SERVICE_PORT: <new-elasticsearch-port>
|
||||
|
||||
For example:
|
||||
|
||||
MONGODB_SERVICE_PORT: 27018
|
||||
ELASTIC_SERVICE_PORT: 9201
|
||||
|
||||
1. For MongoDB, in the `services/mongo/ports` section, expose the new MongoDB port:
|
||||
|
||||
<new-mongodb-port>:27017
|
||||
|
||||
For example:
|
||||
|
||||
20718:27017
|
||||
|
||||
1. For Elastic, in the `services/elasticsearch/ports` section, expose the new Elastic port:
|
||||
|
||||
<new-elsticsearch-port>:9200
|
||||
|
||||
For example:
|
||||
|
||||
9201:9200
|
||||
|
||||
2. Restart **trains-server** (see the [Restarting trains-server](#restart) FAQ).
|
||||
@ -1,32 +1,36 @@
|
||||
# **TRAINS-server**: AWS pre-installed images
|
||||
# Deploying **trains-server** on AWS
|
||||
|
||||
In order to easily deploy **trains-server** on AWS, we created the following Amazon Machine Images (AMIs).
|
||||
To deploy **trains-server** on AWS, use one of the Amazon Machine Images (AMIs) listed in the [Released versions](#released-versions) section of this page.
|
||||
|
||||
Service port numbers on these AMIs are:
|
||||
- Web: 8080
|
||||
- API: 8008
|
||||
- File Server: 8081
|
||||
|
||||
- Web: `8080`
|
||||
- API: `8008`
|
||||
- File Server: `8081`
|
||||
|
||||
Persistent storage configuration:
|
||||
- MongoDB: /opt/trains/data/mongo/
|
||||
- ElasticSearch: /opt/trains/data/elastic/
|
||||
- File Server: /mnt/fileserver/
|
||||
|
||||
Instructions on launching a custom AMI from the EC2 console can be found [here](https://aws.amazon.com/premiumsupport/knowledge-center/launch-instance-custom-ami/)
|
||||
and a detailed version [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/launching-instance.html).
|
||||
- MongoDB: `/opt/trains/data/mongo/`
|
||||
- ElasticSearch: `/opt/trains/data/elastic/`
|
||||
- File Server: `/mnt/fileserver/`
|
||||
|
||||
## Installing
|
||||
|
||||
The minimum recommended instance type is **t3a.large**
|
||||
We provide AMIs per region for each released version of **trains-server**, see [Released versions](#released-versions) on this page.
|
||||
|
||||
For instructions on launching a custom AMI from the EC2 console, see the [AWS Knowledge Center](https://aws.amazon.com/premiumsupport/knowledge-center/launch-instance-custom-ami/) or detailed instructions in the [AWS Documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/launching-instance.html).
|
||||
|
||||
The minimum recommended amount of RAM is 8GB. For example, **t3.large** or **t3a.large** would have the minimum recommended amount of resources.
|
||||
|
||||
## Upgrading
|
||||
|
||||
In order to upgrade **trains-server** on an existing EC2 instance based on one of these AMIs, SSH into the instance and follow the [upgrade instructions](../README.md#upgrade) for **trains-server**.
|
||||
To upgrade **trains-server** on an existing EC2 instance based on one of these AMIs, SSH into the instance and follow the [upgrade instructions](../README.md#upgrade) for **trains-server**.
|
||||
|
||||
### Upgrading AMI's to v0.12
|
||||
**Including the automatically updated AMI**
|
||||
|
||||
Version 0.12 introduced an additional REDIS docker to the trains-server setup.
|
||||
This upgrade includes the automatically updated AMI in Version 0.12. It also includes an additional REDIS docker to the **trains-server** setup.
|
||||
|
||||
AMI upgrading instructions:
|
||||
To upgrade the AMI:
|
||||
|
||||
1. SSH to the EC2 machine running one of the `Latest Version AMI's`
|
||||
2. Execute the following bash commands
|
||||
@ -44,47 +48,49 @@ AMI upgrading instructions:
|
||||
|
||||
## Released versions
|
||||
|
||||
The following sections provide a list containing AMI Image ID per region for each released **trains-server** version.
|
||||
The following sections contain lists of AMI Image IDs, per region, for each released **trains-server** version.
|
||||
|
||||
### Latest Version AMI <a name="autoupdate"></a>
|
||||
**For easier upgrades: The following AMI automatically update to the latest release every reboot**
|
||||
### Latest version AMI - v0.12.1 (auto update)<a name="autoupdate"></a>
|
||||
|
||||
* **eu-north-1** : ami-055909c1b9471451d
|
||||
* **ap-south-1** : ami-0476123cc77226faf
|
||||
* **eu-west-3** : ami-01df7d35ab63cca70
|
||||
* **eu-west-2** : ami-00e8004c11fd0228e
|
||||
* **eu-west-1** : ami-04293fbba6d3acad1
|
||||
* **ap-northeast-2** : ami-004331f9c5eb13e94
|
||||
* **ap-northeast-1** : ami-08cc80e2049b30e61
|
||||
* **sa-east-1** : ami-06d814a0b6ffa3153
|
||||
* **ca-central-1** : ami-069210ff757e9c1b7
|
||||
* **ap-southeast-1** : ami-0d12cc70d6e9c0f39
|
||||
* **ap-southeast-2** : ami-0b4615aa76c055267
|
||||
* **eu-central-1** : ami-06537f431e52e4763
|
||||
* **us-east-2** : ami-0c3cfbcb8e72ecfc5
|
||||
* **us-west-1** : ami-0d83de031b83b6880
|
||||
* **us-west-2** : ami-06968633c4f7187c4
|
||||
* **us-east-1** : ami-07ff2f5f7ef99e8f6
|
||||
For easier upgrades, the following AMIs automatically update to the latest release every reboot:
|
||||
|
||||
### v0.12.1
|
||||
* **eu-north-1** : ami-003118a8103286d84
|
||||
* **ap-south-1** : ami-02dfe86baa48e096f
|
||||
* **eu-west-3** : ami-0cc1f01267d2a780d
|
||||
* **eu-west-2** : ami-0e4c8332e5ce09585
|
||||
* **eu-west-1** : ami-03459a2f0b0a3b1ab
|
||||
* **ap-northeast-2** : ami-08f6c2aed3a53f24c
|
||||
* **ap-northeast-1** : ami-0b798eab95a7c5435
|
||||
* **sa-east-1** : ami-0d3ee166c09f0d1b2
|
||||
* **ca-central-1** : ami-00a758c56bd63acd5
|
||||
* **ap-southeast-1** : ami-0be64d4988cd03fbb
|
||||
* **ap-southeast-2** : ami-02087310d43a63f31
|
||||
* **eu-central-1** : ami-097bbefeac0c74225
|
||||
* **us-east-2** : ami-07eda256712b90f4d
|
||||
* **us-west-1** : ami-02ef2b55cbd01c7df
|
||||
* **us-west-2** : ami-037c6176ef4735360
|
||||
* **us-east-1** : ami-08715c20c0e3f1c15
|
||||
* **eu-north-1** : ami-055909c1b9471451d
|
||||
* **ap-south-1** : ami-0476123cc77226faf
|
||||
* **eu-west-3** : ami-01df7d35ab63cca70
|
||||
* **eu-west-2** : ami-00e8004c11fd0228e
|
||||
* **eu-west-1** : ami-04293fbba6d3acad1
|
||||
* **ap-northeast-2** : ami-004331f9c5eb13e94
|
||||
* **ap-northeast-1** : ami-08cc80e2049b30e61
|
||||
* **sa-east-1** : ami-06d814a0b6ffa3153
|
||||
* **ca-central-1** : ami-069210ff757e9c1b7
|
||||
* **ap-southeast-1** : ami-0d12cc70d6e9c0f39
|
||||
* **ap-southeast-2** : ami-0b4615aa76c055267
|
||||
* **eu-central-1** : ami-06537f431e52e4763
|
||||
* **us-east-2** : ami-0c3cfbcb8e72ecfc5
|
||||
* **us-west-1** : ami-0d83de031b83b6880
|
||||
* **us-west-2** : ami-06968633c4f7187c4
|
||||
* **us-east-1** : ami-07ff2f5f7ef99e8f6
|
||||
|
||||
### v0.12.1 (static update)
|
||||
* **eu-north-1** : ami-003118a8103286d84
|
||||
* **ap-south-1** : ami-02dfe86baa48e096f
|
||||
* **eu-west-3** : ami-0cc1f01267d2a780d
|
||||
* **eu-west-2** : ami-0e4c8332e5ce09585
|
||||
* **eu-west-1** : ami-03459a2f0b0a3b1ab
|
||||
* **ap-northeast-2** : ami-08f6c2aed3a53f24c
|
||||
* **ap-northeast-1** : ami-0b798eab95a7c5435
|
||||
* **sa-east-1** : ami-0d3ee166c09f0d1b2
|
||||
* **ca-central-1** : ami-00a758c56bd63acd5
|
||||
* **ap-southeast-1** : ami-0be64d4988cd03fbb
|
||||
* **ap-southeast-2** : ami-02087310d43a63f31
|
||||
* **eu-central-1** : ami-097bbefeac0c74225
|
||||
* **us-east-2** : ami-07eda256712b90f4d
|
||||
* **us-west-1** : ami-02ef2b55cbd01c7df
|
||||
* **us-west-2** : ami-037c6176ef4735360
|
||||
* **us-east-1** : ami-08715c20c0e3f1c15
|
||||
|
||||
### v0.12.0 (static update)
|
||||
|
||||
### v0.12.0
|
||||
* **eu-north-1** : ami-03ff8ab48cd43e77e
|
||||
* **ap-south-1** : ami-079c1a41ff836487c
|
||||
* **eu-west-3** : ami-0121ef0398ae87ab0
|
||||
@ -102,7 +108,8 @@ The following sections provide a list containing AMI Image ID per region for eac
|
||||
* **us-west-2** : ami-0018d5a7e58966848
|
||||
* **us-east-1** : ami-08f24178fc14a84d2
|
||||
|
||||
### v0.11.0
|
||||
### v0.11.0 (static update)
|
||||
|
||||
* **eu-north-1** : ami-0cbe338f058018c97
|
||||
* **ap-south-1** : ami-06d72ff894f7a5e5d
|
||||
* **eu-west-3** : ami-00f2a45d67df2d2f3
|
||||
@ -120,7 +127,8 @@ The following sections provide a list containing AMI Image ID per region for eac
|
||||
* **us-west-2** : ami-0e384b6f78bf96ebe
|
||||
* **us-east-1** : ami-0a7b46f907d5d9c4a
|
||||
|
||||
### v0.10.1
|
||||
### v0.10.1 (static update)
|
||||
|
||||
* **eu-north-1** : ami-09937ec4d18350c32
|
||||
* **ap-south-1** : ami-089d6ba7541ec4c7f
|
||||
* **eu-west-3** : ami-0accb1a94bdd5c5c1
|
||||
@ -138,7 +146,8 @@ The following sections provide a list containing AMI Image ID per region for eac
|
||||
* **us-west-2** : ami-0d1cb8ba7de246ff0
|
||||
* **us-east-1** : ami-049ccba6abdb40cba
|
||||
|
||||
### v0.10.0
|
||||
### v0.10.0 (static update)
|
||||
|
||||
* **eu-north-1** : ami-05ba33c763877e54e
|
||||
* **ap-south-1** : ami-0529eec569161cae5
|
||||
* **eu-west-3** : ami-03cb9396f63e26ff6
|
||||
@ -157,7 +166,7 @@ The following sections provide a list containing AMI Image ID per region for eac
|
||||
* **us-west-2** : ami-04a522ecb2250fb44
|
||||
* **us-east-1** : ami-0a66ddbd50959f91e
|
||||
|
||||
### v0.9.0
|
||||
### v0.9.0 (static update)
|
||||
|
||||
* **us-east-1** : ami-0991ad536ecbacdac
|
||||
* **eu-north-1** : ami-07cbcdff501b14afe
|
||||
@ -175,3 +184,7 @@ The following sections provide a list containing AMI Image ID per region for eac
|
||||
* **us-east-2** : ami-03b01914b07428488
|
||||
* **us-west-1** : ami-0cf4768e9d47ed076
|
||||
* **us-west-2** : ami-0b145f37da31eb9fb
|
||||
|
||||
## Next Step
|
||||
|
||||
Configure the [TRAINS client for trains-server](https://github.com/allegroai/trains-server/blob/master/README#configuring-the-trains-client-for-trains-server).
|
||||
97
docs/install_linux_mac.md
Normal file
97
docs/install_linux_mac.md
Normal file
@ -0,0 +1,97 @@
|
||||
# Launching the **trains-server** Docker in Linux or Mac OS X
|
||||
|
||||
For Linux or Mac OS X, use our pre-built Docker image for easy deployment. The latest Docker images can be found [here](https://hub.docker.com/r/allegroai/trains).
|
||||
|
||||
For Linux users:
|
||||
|
||||
* You must be logged in as a user with sudo privileges.
|
||||
* Use `bash` for all command-line instructions in this installation.
|
||||
|
||||
To launch **trains-server** on Linux or Mac OS X:
|
||||
|
||||
1. Install Docker.
|
||||
|
||||
* Linux - see [Docker for Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/).
|
||||
* Mac OS X - see [Docker for OS X](https://docs.docker.com/docker-for-mac/install/).
|
||||
|
||||
1. Verify the Docker CE installation. Execute the command:
|
||||
|
||||
sudo docker run hello-world
|
||||
|
||||
The expected is output is:
|
||||
|
||||
Hello from Docker!
|
||||
This message shows that your installation appears to be working correctly.
|
||||
To generate this message, Docker took the following steps:
|
||||
|
||||
1. The Docker client contacted the Docker daemon.
|
||||
2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64)
|
||||
3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
|
||||
4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.
|
||||
|
||||
1. For Linux only, install `docker-compose`. Execute the following commands (for more information, see [Install Docker Compose](https://docs.docker.com/compose/install/) in the Docker documentation):
|
||||
|
||||
sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
|
||||
sudo chmod +x /usr/local/bin/docker-compose
|
||||
|
||||
1. Increase `vm.max_map_count` for ElasticSearch docker.
|
||||
|
||||
Linux:
|
||||
|
||||
echo "vm.max_map_count=262144" > /tmp/99-trains.conf
|
||||
sudo mv /tmp/99-trains.conf /etc/sysctl.d/99-trains.conf
|
||||
sudo sysctl -w vm.max_map_count=262144
|
||||
sudo service docker restart
|
||||
|
||||
Mac OS X:
|
||||
|
||||
screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
|
||||
sysctl -w vm.max_map_count=262144
|
||||
|
||||
|
||||
1. Remove any previous installation of **trains-server**.
|
||||
|
||||
**WARNING**: This clears all existing **TRAINS** databases.
|
||||
|
||||
sudo rm -R /opt/trains/
|
||||
|
||||
1. Create local directories for the databases and storage.
|
||||
|
||||
sudo mkdir -p /opt/trains/data/elastic
|
||||
sudo mkdir -p /opt/trains/data/mongo/db
|
||||
sudo mkdir -p /opt/trains/data/mongo/configdb
|
||||
sudo mkdir -p /opt/trains/data/redis
|
||||
sudo mkdir -p /opt/trains/logs
|
||||
sudo mkdir -p /opt/trains/config
|
||||
sudo mkdir -p /opt/trains/data/fileserver
|
||||
|
||||
1. For Mac OS X only, open the Docker app, select **Preferences**, and then on the **File Sharing** tab, add `/opt/trains`.
|
||||
|
||||
1. Grant access to the Dockers.
|
||||
|
||||
Linux:
|
||||
|
||||
sudo chown -R 1000:1000 /opt/trains
|
||||
|
||||
Mac OS X:
|
||||
|
||||
sudo chown -R $(whoami):staff /opt/trains
|
||||
|
||||
1. Download the **trains-server** docker-compose YAML file.
|
||||
|
||||
cd /opt/trains
|
||||
curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose-unified.yml -o docker-compose-unified.yml
|
||||
|
||||
1. Run `docker-compose` with the downloaded configuration file.
|
||||
|
||||
sudo docker-compose -f docker-compose-unified.yml up
|
||||
|
||||
Your server is now running on [http://localhost:8080](http://localhost:8080) and the following ports are available:
|
||||
|
||||
* Web server on port `8080`
|
||||
* API server on port `8008`
|
||||
* File server on port `8081`
|
||||
|
||||
## Next Step
|
||||
|
||||
Configure the [TRAINS client for trains-server](https://github.com/allegroai/trains-server/blob/master/README#configuring-the-trains-client-for-trains-server).
|
||||
50
docs/install_win.md
Normal file
50
docs/install_win.md
Normal file
@ -0,0 +1,50 @@
|
||||
# Launching the **trains-server** Docker in Windows 10
|
||||
|
||||
For Windows, we recommend launching our pre-built Docker image on a Linux virtual machine.
|
||||
However, you can launch **trains-server** on Windows 10 using Docker Desktop for Windows (see the Docker [System Requirements](https://docs.docker.com/docker-for-windows/install/#system-requirements)).
|
||||
|
||||
To launch **trains-server** on Windows 10:
|
||||
|
||||
1. Install the Docker Desktop for Windows application by either:
|
||||
|
||||
* Following the [Install Docker Desktop on Windows](https://docs.docker.com/docker-for-windows/install/) instructions.
|
||||
* Running the Docker installation [wizard](https://hub.docker.com/?overlay=onboarding).
|
||||
|
||||
1. Increase the memory allocation in Docker Desktop to `4GB`.
|
||||
|
||||
1. In your Windows notification area (system tray), right click the Docker icon.
|
||||
|
||||
1. Click *Settings*, *Advanced*, and then set the memory to at least `4096`.
|
||||
|
||||
1. Click *Apply*.
|
||||
|
||||
1. Remove any previous installation of **trains-server**.
|
||||
|
||||
**WARNING**: This clears all existing **TRAINS** databases.
|
||||
|
||||
rmdir c:\opt\trains /s
|
||||
|
||||
1. Create local directories for data and logs. Open PowerShell and execute the following commands:
|
||||
|
||||
cd c:
|
||||
mkdir c:\opt\trains\data
|
||||
mkdir c:\opt\trains\logs
|
||||
|
||||
1. Save the **trains-server** docker-compose YAML file.
|
||||
|
||||
cd c:\opt\trains
|
||||
curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose-win10.yml -o docker-compose-win10.yml
|
||||
|
||||
1. Run `docker-compose`. In PowerShell, execute the following commands:
|
||||
|
||||
docker-compose -f docker-compose-win10.yml up
|
||||
|
||||
Your server is now running on [http://localhost:8080](http://localhost:8080) and the following ports are available:
|
||||
|
||||
* Web server on port `8080`
|
||||
* API server on port `8008`
|
||||
* File server on port `8081`
|
||||
|
||||
## Next Step
|
||||
|
||||
Configure the [TRAINS client for trains-server](https://github.com/allegroai/trains-server/blob/master/README#configuring-the-trains-client-for-trains-server).
|
||||
Loading…
Reference in New Issue
Block a user