Add Server Upgrade instructions (#184)

This commit is contained in:
pollfly 2022-02-13 18:54:59 +02:00 committed by GitHub
parent 57e44edcfe
commit e4a7448935
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
16 changed files with 159 additions and 146 deletions

View File

@ -58,6 +58,6 @@ When necessary, upgrade your ClearML Server on any of the available formats:
* [Google Cloud Platform](upgrade_server_gcp.md)
* [Linux or MacOS](upgrade_server_linux_mac.md)
* [Windows 10](upgrade_server_win.md)
* [Kubernetes using Helm](upgrade_server_kubernetes_helm.md).
* [Kubernetes using Helm](upgrade_server_kubernetes_helm.md)
If you are using v0.15 or Older, [upgrade to ClearML Server](clearml_server_es7_migration.md).

View File

@ -2,10 +2,6 @@
title: AWS EC2 AMIs
---
:::important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::
Deployment of **ClearML Server** on AWS is easily performed using AWS AMIs, which are available in the AWS Marketplace catalog
and in the AWS community AMI catalog.

View File

@ -2,8 +2,9 @@
title: Upgrading Server from v0.15 or Older to ClearML Server
---
:::important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::important updating to ClearML Server v1.2+
If you are upgrading your ClearML Server to version 1.2 or newer, you will need to migrate your database contents to be compatible with the new version,
after performing the migration instructions below.
:::
In v0.16, the Elasticsearch subsystem of **Trains Server** was upgraded from version 5.6 to version 7.6. This change necessitates
@ -135,13 +136,7 @@ and **ClearML Server** needs to be installed.
kubectl get jobs -n upgrade-elastic
### Finishing Up
To finish up:
1. Verify the data migration
1. Conclude the upgrade.
#### Step 1. Verifying the Data Migration
### Verifying the Data Migration
Upon successful completion, the migration script renames the original **Trains Server** directory, which contains the now
migrated data, and prints a completion message:
@ -160,68 +155,14 @@ For help in resolving migration issues, check the **allegro-clearml** [Slack Cha
[GitHub Issues](https://github.com/allegroai/clearml-server/issues), and the **ClearML Server** sections of the [FAQ](../faq.md).
:::
#### Step 2. Completing the Installation
### Upgrading to ClearML Server v.1.2 or Newer
If you are upgrading your ClearML Server to version 1.2 or newer, you will need to migrate your database contents to be
compatible with the new version. See instructions [here](clearml_server_mongo44_migration.md). Otherwise, continue the instructions below.
After verifying the data migration completed successfully, conclude the **ClearML Server** installation process.
### Completing the Installation
##### Linux or macOS
For Linux or macOS, conclude with the steps in this section. For other deployment formats, see [below](#other-deployment-formats).
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">Important: Upgrading from v0.14 or older</summary>
<div className="cml-expansion-panel-content">
For Linux only, if upgrading from **Trains Server** v0.14 or older, configure the **ClearML Agent Services**.
* If ``CLEARML_HOST_IP`` is not provided, then **ClearML Agent Services** will use the external public address of the
**ClearML Server**.
* If ``CLEARML_AGENT_GIT_USER`` / ``CLEARML_AGENT_GIT_PASS`` are not provided, then **ClearML Agent Services** will
not be able to access any private repositories for running service tasks.
export CLEARML_HOST_IP=server_host_ip_here
export CLEARML_AGENT_GIT_USER=git_username_here
export CLEARML_AGENT_GIT_PASS=git_password_here
:::note
For backwards compatibility, the environment variables ``TRAINS_HOST_IP``, ``TRAINS_AGENT_GIT_USER``, and ``TRAINS_AGENT_GIT_PASS`` are supported.
:::
</div>
</details>
1. We recommend backing up data and, if the configuration folder is not empty, backing up the configuration.
For example, if the data and configuration folders are in `/opt/trains`, then archive all data into `~/trains_backup_data.tgz`,
and the configuration into `~/trains_backup_config.tgz`:
sudo tar czvf ~/trains_backup_data.tgz -C /opt/trains/data .
sudo tar czvf ~/trains_backup_config.tgz -C /opt/trains/config .
1. Rename `/opt/trains` and its subdirectories to `/opt/clearml`.
sudo mv /opt/trains /opt/clearml
1. Download the latest `docker-compose.yml` file.
curl https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml
1. Startup **ClearML Server**. This automatically pulls the latest **ClearML Server** build.
docker-compose -f /opt/clearml/docker-compose.yml pull
docker-compose -f /opt/clearml/docker-compose.yml up -d
If issues arise during the upgrade, see the FAQ page, [How do I fix Docker upgrade errors?](../faq.md#common-docker-upgrade-errors).
##### Other Deployment Formats
To conclude the upgrade for deployment formats other than Linux, follow their upgrade instructions:
After verifying the data migration completed successfully, continue upgrading your server:
* [AWS EC2 AMIs](upgrade_server_aws_ec2_ami.md)
* [Google Cloud Platform custom images](upgrade_server_gcp.md)
* [Linux and macOS](upgrade_server_linux_mac.md)
* [Windows](upgrade_server_win.md)
* [Kubernetes](upgrade_server_kubernetes.md)
* [Kubernetes Using Helm](upgrade_server_kubernetes_helm.md).

View File

@ -2,10 +2,6 @@
title: Google Cloud Platform
---
:::important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::
Deploy **ClearML Server** on the Google Cloud Platform (GCP) using one of the pre-built GCP Custom Images. **ClearML**
provides custom images for each released version of **ClearML Server**. For a list of the pre-built custom images, see
[ClearML Server GCP Custom Image](#clearml-server-gcp-custom-image).

View File

@ -1,9 +0,0 @@
---
title: Kubernetes
---
:::important
Kubernetes support is being replaced by Kubernetes using Helm. Please see the new and improved [ClearML Helm Charts repository](https://github.com/allegroai/clearml-helm-charts)
for the most updated instructions
:::

View File

@ -1,5 +1,5 @@
---
title: Kubernetes Using Helm
title: Kubernetes
---
:::important
@ -7,10 +7,6 @@ title: Kubernetes Using Helm
for the most updated instructions.
:::
:::note
This documentation page applies to deploying your own open source **ClearML Server**. It does not apply to **ClearML Hosted Service** users.
:::
:::warning
If **ClearML Server** is being reinstalled, we recommend clearing browser cookies for **ClearML Server**. For example,
for Firefox, go to Developer Tools > Storage > Cookies, and for Chrome, go to Developer Tools > Application > Cookies,

View File

@ -2,10 +2,6 @@
title: Linux and macOS
---
:::important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::
Deploy the **ClearML Server** in Linux or macOS using the pre-built Docker image.
For ClearML docker images, including previous versions, see [https://hub.docker.com/r/allegroai/clearml](https://hub.docker.com/r/allegroai/clearml).

View File

@ -0,0 +1,128 @@
---
title: Upgrading Server from v1.1 or Older
---
:::important
If your server is v0.15 or older, you must first go through the [v0.16 upgrade procedure](clearml_server_es7_migration.md)
and continue here once successfully completed.
:::
In v1.2, the MongoDB subsystem of ClearML Server has been upgraded from version 3.6 to version 4.4. This change
necessitates the migration of the database contents to be compatible with the new version.
:::note Kubernetes installations
[ClearMLs helm chart](https://github.com/allegroai/clearml-helm-charts/tree/main/charts/clearml) is already running
mongodb version 4.4. If your ClearML server had been deployed with this chart (with the default mongodb bitnami chart) -
You can stop reading here, as no migration is required.
:::
The migration process is comprised of the following:
* Backing up your data
* Creating a copy of the data which will be migrated for use with the new version
* Sequentially running data migration for MongoDB: 3.6->4.0, 4.0->4.2, 4.0->4.4
At the end of the process, your data will be available in the new location, ready to be used with ClearML Server v1.2.
Once ClearML Server v1.2 is up and running, and you are satisfied that your data is intact, you can safely delete the
old copy of your data.
## Prerequisites
* Read/write permissions for the default ClearML Server data directory `/opt/clearml/data` and its subdirectories, or,
if a different location had been used, permissions for that directory and its subdirectories.
* Minimum free disk space of at least three times the size of the data.
* Python version >=2.7 or >=3.6, and Python accessible from the command-line as python
## Before You Begin
To avoid data corruption, shut down your ClearML server before applying the migration procedure:
* Linux and macOS
```bash
docker-compose -f /opt/clearml/docker-compose.yml down
```
* Windows
```bash
docker-compose -f c:\opt\clearml\docker-compose-win10.yml down
```
## Migrating the Data
The following are the instructions for migrating your ClearML data from mongodb 3.6 to mongodb 4.4.
The commands are provided for a Linux environment, and assume ClearML is installed in its default paths.
If you are using a different OS, replace the shell and docker commands to the ones appropriate for your OS. If youve
installed ClearML under a different path, replace the file locations accordingly.
:::info automating your migration
A migration script is available to automatically run this process for all supported OSs.
[Download the script](https://github.com/allegroai/clearml-server/releases/download/1.2.0/clearml-server-1.2.0-migration.py) and run it on your ClearML server.
Run `clearml-server-1.2.0-migration.py help` to see execution options.
Note the script will create a backup archive of your data in the original directory.
:::
1. Backup the source data folder:
```bash
sudo tar -zcf mongodb_upgrade.gz -C /opt/clearml/data/mongo .
```
1. Copy the data to the new folder and set permissions (permissions are required on Linux):
```bash
sudo mkdir /opt/clearml/data/mongo_4
sudo tar -xzf mongodb_upgrade.gz -C /opt/clearml/data/mongo_4
sudo chown -R 1000:1000 /opt/clearml/data/mongo_4
```
1. Run the mongo 4.0 container:
```bash
sudo docker run -id -v /opt/clearml/data/mongo_4/db:/data/db -v /opt/clearml/data/mongo_4/configdb:/data/configdb --name mongodb_upgrade mongo:4.0.27
```
1. Upgrade data to mongo 4.0:
```bash
sudo docker exec mongodb_upgrade bash -c 'mongo --eval "db.adminCommand({setFeatureCompatibilityVersion:\"4.0\"})"'
```
This command should return an `{ "ok" : 1 }` response.
1. Stop the mongo 4.0 container
```bash
sudo docker stop mongodb_upgrade && sudo docker rm mongodb_upgrade
```
1. Run the mongo 4.2 container:
```bash
sudo docker run -id -v /opt/clearml/data/mongo_4/db:/data/db -v /opt/clearml/data/mongo_4/configdb:/data/configdb --name mongodb_upgrade mongo:4.2.16
```
1. Upgrade data to mongo 4.2:
```bash
sudo docker exec mongodb_upgrade bash -c 'mongo --eval "db.adminCommand({setFeatureCompatibilityVersion:\"4.2\"})"'
```
This command should return an `{ "ok" : 1 }` response.
1. Stop the mongo 4.2 container
```bash
sudo docker stop mongodb_upgrade && sudo docker rm mongodb_upgrade
```
1. Run the mongo 4.4 container:
```bash
sudo docker run -id -v /opt/clearml/data/mongo_4/db:/data/db -v /opt/clearml/data/mongo_4/configdb:/data/configdb --name mongodb_upgrade mongo:4.4.9
```
1. Upgrade data to mongo 4.4:
```bash
sudo docker exec mongodb_upgrade bash -c 'mongo --eval "db.adminCommand({setFeatureCompatibilityVersion:\"4.4\"})"'
```
This command should return an `{ "ok" : 1 }` response.
1. Stop the mongo 4.4
```bash
sudo docker stop mongodb_upgrade && sudo docker rm mongodb_upgrade
```
1. Continue upgrading your ClearML Server:
* [AWS EC2 AMIs](upgrade_server_aws_ec2_ami.md)
* [Google Cloud Platform custom images](upgrade_server_gcp.md)
* [Linux and macOS](upgrade_server_linux_mac.md)
* [Windows](upgrade_server_win.md)

View File

@ -2,10 +2,6 @@
title: Windows 10
---
:::important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::
For Windows, we recommend launching the pre-built Docker image on a Linux virtual machine (see [Deploying ClearML Server: Linux or macOS](clearml_server_linux_mac.md)).
However, **ClearML Server** can be launched on Windows 10, using Docker Desktop for Windows (see the Docker [System Requirements](https://docs.docker.com/docker-for-windows/install/#system-requirements)).

View File

@ -2,10 +2,6 @@
title: AWS EC2 AMIs
---
:::important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::
:::note
For upgrade purposes, the terms **Trains Server** and **ClearML Server** are interchangeable.
:::
@ -34,9 +30,9 @@ Some legacy **Trains Server** AMIs provided an auto-upgrade on restart capabilit
1. We recommend [backing up your data](clearml_server_aws_ec2_ami.md#backing-up-and-restoring-data-and-configuration) and,
if your configuration folder is not empty, backing up your configuration.
1. If upgrading from **Trains Server** version 0.15 or older, a data migration is required before upgrading,
First follow these [data migration instructions](clearml_server_es7_migration.md), and then continue this upgrade.
1. If upgrading from ClearML Server version older than 1.2, you need to migrate your data before upgrading your server. See instructions [here](clearml_server_mongo44_migration.md).
If upgrading from Trains Server version 0.15 or older, a data migration is required before continuing this upgrade. See instructions [here](clearml_server_es7_migration.md).
1. If upgrading from **Trains Server** to **ClearML Server**, rename `/opt/trains` to `/opt/clearml`.
1. Download the latest `docker-compose.yml` file. Execute the following command:

View File

@ -2,19 +2,16 @@
title: Google Cloud Platform
---
:::important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::
**To upgrade ClearML Server Docker deployment:**
1. Shut down the docker containers with the following command:
docker-compose -f docker-compose.yml down
1. If upgrading from ClearML Server version older than 1.2, you need to migrate your data before upgrading your server. See instructions [here](clearml_server_mongo44_migration.md).
1. If upgrading from **Trains Server** version 0.15 or older to **ClearML Server**, do the following:
1. A data migration is required before upgrading. First follow these [data migration instructions](clearml_server_es7_migration.md),
1. Follow these [data migration instructions](clearml_server_es7_migration.md),
and then continue this upgrade.
1. Rename `/opt/trains` and its subdirectories to `/opt/clearml`.

View File

@ -1,10 +0,0 @@
---
title: Kubernetes
---
:::important
Kubernetes support is being replaced by Kubernetes using Helm. Please see the new and improved [ClearML Helm Charts repository](https://github.com/allegroai/clearml-helm-charts)
for the most updated instructions.
:::

View File

@ -1,5 +1,5 @@
---
title: Kubernetes Helm
title: Kubernetes
---
:::important
@ -8,9 +8,7 @@ for the most updated instructions.
:::
:::note
* This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
* We strongly encourage to keep the **ClearML Server** up to date, by upgrading to the current release.
We strongly encourage to keep the **ClearML Server** up to date, by upgrading to the current release.
:::
1. Upgrade using new or upgraded values.yaml
@ -21,8 +19,8 @@ for the most updated instructions.
helm delete --purge clearml-server
1. If upgrading from **Trains Server** version 0.15 or older to **ClearML Server**, a data migration is required before
upgrading. First follow these [data migration instructions](clearml_server_es7_migration.md), and then continue this upgrade.
1. If upgrading from Trains Server version 0.15 or older, a data migration is required before continuing this upgrade.
See instructions [here](clearml_server_es7_migration.md).
1. Upgrade deployment to match repository version.

View File

@ -2,12 +2,6 @@
title: Linux or macOS
---
:::important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::
<br/>
<details className="cml-expansion-panel info">
<summary className="cml-expansion-panel-summary">Important: Upgrading from v0.14 or older</summary>
<div class="cml-expansion-panel-content">
@ -35,12 +29,13 @@ For backwards compatibility, the environment variables ``TRAINS_HOST_IP``, ``TRA
**To upgrade ClearML Server Docker deployment:**
1. Shutdown **ClearML Server**. Execute the following command (which assumes the configuration file is in the environment path).
1. Shutdown ClearML Server. Execute the following command (which assumes the configuration file is in the environment path).
docker-compose -f docker-compose.yml down
1. If upgrading from **Trains Server** version 0.15 or older to **ClearML Server**, a data migration is required before
upgrading. First follow these [data migration instructions](clearml_server_es7_migration.md), and then continue this upgrade.
1. If upgrading from ClearML Server version older than 1.2, you need to migrate your data before upgrading your server. See instructions [here](clearml_server_mongo44_migration.md).
1. If upgrading from **Trains Server** version 0.15 or older, a data migration is required before continuing this upgrade. See instructions [here](clearml_server_es7_migration.md).
1. We recommend [backing up data](clearml_server_linux_mac.md#backing-up-and-restoring-data-and-configuration) and, if the configuration folder is
not empty, backing up the configuration.

View File

@ -2,10 +2,6 @@
title: Windows
---
:::important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
:::
**To upgrade ClearML Server Docker deployment:**
1. Shut down the docker containers.
@ -20,8 +16,9 @@ This documentation page applies to deploying your own open source ClearML Server
docker-compose -f c:\opt\trains\docker-compose-win10.yml down
1. If upgrading from **Trains Server** version 0.15 or older to **ClearML Server**, a data migration is required before
upgrading. First follow these [data migration instructions](clearml_server_es7_migration.md), and then continue this upgrade.
1. If upgrading from ClearML Server version older than 1.2, you need to migrate your data before upgrading your server. See instructions [here](clearml_server_mongo44_migration.md).
1. If upgrading from **Trains Server** version 0.15 or older, a data migration is required before continuing this upgrade. See instructions [here](clearml_server_es7_migration.md).
1. We recommend backing up data and, if the configuration folder is not empty, backing up the configuration.

View File

@ -44,14 +44,14 @@ module.exports = {
{'ClearML Server': ['deploying_clearml/clearml_server',
{
'Deploying ClearML Server': ['deploying_clearml/clearml_server_aws_ec2_ami', 'deploying_clearml/clearml_server_gcp',
'deploying_clearml/clearml_server_linux_mac', 'deploying_clearml/clearml_server_win', 'deploying_clearml/clearml_server_kubernetes',
'deploying_clearml/clearml_server_linux_mac', 'deploying_clearml/clearml_server_win',
'deploying_clearml/clearml_server_kubernetes_helm']
},
{
'Upgrading ClearML Server':['deploying_clearml/upgrade_server_aws_ec2_ami','deploying_clearml/upgrade_server_gcp',
'deploying_clearml/upgrade_server_linux_mac', 'deploying_clearml/upgrade_server_win',
'deploying_clearml/upgrade_server_kubernetes', 'deploying_clearml/upgrade_server_kubernetes_helm',
'deploying_clearml/clearml_server_es7_migration']
'deploying_clearml/upgrade_server_kubernetes_helm',
'deploying_clearml/clearml_server_es7_migration', 'deploying_clearml/clearml_server_mongo44_migration']
},
'deploying_clearml/clearml_server_config', 'deploying_clearml/clearml_config_for_clearml_server', 'deploying_clearml/clearml_server_security'
]},