diff --git a/docs/deploying_clearml/clearml_server.md b/docs/deploying_clearml/clearml_server.md index 011bf269..b7e68910 100644 --- a/docs/deploying_clearml/clearml_server.md +++ b/docs/deploying_clearml/clearml_server.md @@ -58,6 +58,6 @@ When necessary, upgrade your ClearML Server on any of the available formats: * [Google Cloud Platform](upgrade_server_gcp.md) * [Linux or MacOS](upgrade_server_linux_mac.md) * [Windows 10](upgrade_server_win.md) -* [Kubernetes using Helm](upgrade_server_kubernetes_helm.md). +* [Kubernetes using Helm](upgrade_server_kubernetes_helm.md) If you are using v0.15 or Older, [upgrade to ClearML Server](clearml_server_es7_migration.md). \ No newline at end of file diff --git a/docs/deploying_clearml/clearml_server_aws_ec2_ami.md b/docs/deploying_clearml/clearml_server_aws_ec2_ami.md index bfc7f9ca..2fd69221 100644 --- a/docs/deploying_clearml/clearml_server_aws_ec2_ami.md +++ b/docs/deploying_clearml/clearml_server_aws_ec2_ami.md @@ -2,10 +2,6 @@ title: AWS EC2 AMIs --- -:::important -This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. -::: - Deployment of **ClearML Server** on AWS is easily performed using AWS AMIs, which are available in the AWS Marketplace catalog and in the AWS community AMI catalog. diff --git a/docs/deploying_clearml/clearml_server_es7_migration.md b/docs/deploying_clearml/clearml_server_es7_migration.md index cf2a5c29..6ff6208f 100644 --- a/docs/deploying_clearml/clearml_server_es7_migration.md +++ b/docs/deploying_clearml/clearml_server_es7_migration.md @@ -2,8 +2,9 @@ title: Upgrading Server from v0.15 or Older to ClearML Server --- -:::important -This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. +:::important updating to ClearML Server v1.2+ +If you are upgrading your ClearML Server to version 1.2 or newer, you will need to migrate your database contents to be compatible with the new version, +after performing the migration instructions below. ::: In v0.16, the Elasticsearch subsystem of **Trains Server** was upgraded from version 5.6 to version 7.6. This change necessitates @@ -135,13 +136,7 @@ and **ClearML Server** needs to be installed. kubectl get jobs -n upgrade-elastic -### Finishing Up - -To finish up: -1. Verify the data migration -1. Conclude the upgrade. - -#### Step 1. Verifying the Data Migration +### Verifying the Data Migration Upon successful completion, the migration script renames the original **Trains Server** directory, which contains the now migrated data, and prints a completion message: @@ -160,68 +155,14 @@ For help in resolving migration issues, check the **allegro-clearml** [Slack Cha [GitHub Issues](https://github.com/allegroai/clearml-server/issues), and the **ClearML Server** sections of the [FAQ](../faq.md). ::: -#### Step 2. Completing the Installation +### Upgrading to ClearML Server v.1.2 or Newer +If you are upgrading your ClearML Server to version 1.2 or newer, you will need to migrate your database contents to be +compatible with the new version. See instructions [here](clearml_server_mongo44_migration.md). Otherwise, continue the instructions below. -After verifying the data migration completed successfully, conclude the **ClearML Server** installation process. +### Completing the Installation -##### Linux or macOS - -For Linux or macOS, conclude with the steps in this section. For other deployment formats, see [below](#other-deployment-formats). - -
-Important: Upgrading from v0.14 or older -
- For Linux only, if upgrading from **Trains Server** v0.14 or older, configure the **ClearML Agent Services**. - - * If ``CLEARML_HOST_IP`` is not provided, then **ClearML Agent Services** will use the external public address of the - **ClearML Server**. - * If ``CLEARML_AGENT_GIT_USER`` / ``CLEARML_AGENT_GIT_PASS`` are not provided, then **ClearML Agent Services** will - not be able to access any private repositories for running service tasks. - - - export CLEARML_HOST_IP=server_host_ip_here - export CLEARML_AGENT_GIT_USER=git_username_here - export CLEARML_AGENT_GIT_PASS=git_password_here - - -:::note -For backwards compatibility, the environment variables ``TRAINS_HOST_IP``, ``TRAINS_AGENT_GIT_USER``, and ``TRAINS_AGENT_GIT_PASS`` are supported. -::: - -
-
- - -1. We recommend backing up data and, if the configuration folder is not empty, backing up the configuration. - - For example, if the data and configuration folders are in `/opt/trains`, then archive all data into `~/trains_backup_data.tgz`, - and the configuration into `~/trains_backup_config.tgz`: - - sudo tar czvf ~/trains_backup_data.tgz -C /opt/trains/data . - sudo tar czvf ~/trains_backup_config.tgz -C /opt/trains/config . - -1. Rename `/opt/trains` and its subdirectories to `/opt/clearml`. - - sudo mv /opt/trains /opt/clearml - -1. Download the latest `docker-compose.yml` file. - - curl https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml - -1. Startup **ClearML Server**. This automatically pulls the latest **ClearML Server** build. - - docker-compose -f /opt/clearml/docker-compose.yml pull - docker-compose -f /opt/clearml/docker-compose.yml up -d - -If issues arise during the upgrade, see the FAQ page, [How do I fix Docker upgrade errors?](../faq.md#common-docker-upgrade-errors). - -##### Other Deployment Formats - -To conclude the upgrade for deployment formats other than Linux, follow their upgrade instructions: - +After verifying the data migration completed successfully, continue upgrading your server: * [AWS EC2 AMIs](upgrade_server_aws_ec2_ami.md) * [Google Cloud Platform custom images](upgrade_server_gcp.md) * [Linux and macOS](upgrade_server_linux_mac.md) * [Windows](upgrade_server_win.md) -* [Kubernetes](upgrade_server_kubernetes.md) -* [Kubernetes Using Helm](upgrade_server_kubernetes_helm.md). \ No newline at end of file diff --git a/docs/deploying_clearml/clearml_server_gcp.md b/docs/deploying_clearml/clearml_server_gcp.md index 536ed7aa..2a29f675 100644 --- a/docs/deploying_clearml/clearml_server_gcp.md +++ b/docs/deploying_clearml/clearml_server_gcp.md @@ -2,10 +2,6 @@ title: Google Cloud Platform --- -:::important -This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. -::: - Deploy **ClearML Server** on the Google Cloud Platform (GCP) using one of the pre-built GCP Custom Images. **ClearML** provides custom images for each released version of **ClearML Server**. For a list of the pre-built custom images, see [ClearML Server GCP Custom Image](#clearml-server-gcp-custom-image). diff --git a/docs/deploying_clearml/clearml_server_kubernetes.md b/docs/deploying_clearml/clearml_server_kubernetes.md deleted file mode 100644 index 0a744301..00000000 --- a/docs/deploying_clearml/clearml_server_kubernetes.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: Kubernetes ---- - -:::important -Kubernetes support is being replaced by Kubernetes using Helm. Please see the new and improved [ClearML Helm Charts repository](https://github.com/allegroai/clearml-helm-charts) -for the most updated instructions -::: - diff --git a/docs/deploying_clearml/clearml_server_kubernetes_helm.md b/docs/deploying_clearml/clearml_server_kubernetes_helm.md index b5b1913f..17df5ee0 100644 --- a/docs/deploying_clearml/clearml_server_kubernetes_helm.md +++ b/docs/deploying_clearml/clearml_server_kubernetes_helm.md @@ -1,5 +1,5 @@ --- -title: Kubernetes Using Helm +title: Kubernetes --- :::important @@ -7,10 +7,6 @@ title: Kubernetes Using Helm for the most updated instructions. ::: -:::note -This documentation page applies to deploying your own open source **ClearML Server**. It does not apply to **ClearML Hosted Service** users. -::: - :::warning If **ClearML Server** is being reinstalled, we recommend clearing browser cookies for **ClearML Server**. For example, for Firefox, go to Developer Tools > Storage > Cookies, and for Chrome, go to Developer Tools > Application > Cookies, diff --git a/docs/deploying_clearml/clearml_server_linux_mac.md b/docs/deploying_clearml/clearml_server_linux_mac.md index 884d7ac7..7473652b 100644 --- a/docs/deploying_clearml/clearml_server_linux_mac.md +++ b/docs/deploying_clearml/clearml_server_linux_mac.md @@ -2,10 +2,6 @@ title: Linux and macOS --- -:::important -This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. -::: - Deploy the **ClearML Server** in Linux or macOS using the pre-built Docker image. For ClearML docker images, including previous versions, see [https://hub.docker.com/r/allegroai/clearml](https://hub.docker.com/r/allegroai/clearml). diff --git a/docs/deploying_clearml/clearml_server_mongo44_migration.md b/docs/deploying_clearml/clearml_server_mongo44_migration.md new file mode 100644 index 00000000..5fd60ebc --- /dev/null +++ b/docs/deploying_clearml/clearml_server_mongo44_migration.md @@ -0,0 +1,128 @@ +--- +title: Upgrading Server from v1.1 or Older +--- + +:::important +If your server is v0.15 or older, you must first go through the [v0.16 upgrade procedure](clearml_server_es7_migration.md) +and continue here once successfully completed. +::: + +In v1.2, the MongoDB subsystem of ClearML Server has been upgraded from version 3.6 to version 4.4. This change +necessitates the migration of the database contents to be compatible with the new version. + +:::note Kubernetes installations +[ClearML’s helm chart](https://github.com/allegroai/clearml-helm-charts/tree/main/charts/clearml) is already running +mongodb version 4.4. If your ClearML server had been deployed with this chart (with the default mongodb bitnami chart) - +You can stop reading here, as no migration is required. +::: + +The migration process is comprised of the following: +* Backing up your data +* Creating a copy of the data which will be migrated for use with the new version +* Sequentially running data migration for MongoDB: 3.6->4.0, 4.0->4.2, 4.0->4.4 + +At the end of the process, your data will be available in the new location, ready to be used with ClearML Server v1.2. + +Once ClearML Server v1.2 is up and running, and you are satisfied that your data is intact, you can safely delete the +old copy of your data. + +## Prerequisites +* Read/write permissions for the default ClearML Server data directory `/opt/clearml/data` and its subdirectories, or, + if a different location had been used, permissions for that directory and its subdirectories. +* Minimum free disk space of at least three times the size of the data. +* Python version >=2.7 or >=3.6, and Python accessible from the command-line as python + +## Before You Begin +To avoid data corruption, shut down your ClearML server before applying the migration procedure: +* Linux and macOS + + ```bash + docker-compose -f /opt/clearml/docker-compose.yml down + ``` +* Windows + + ```bash + docker-compose -f c:\opt\clearml\docker-compose-win10.yml down + ``` + +## Migrating the Data +The following are the instructions for migrating your ClearML data from mongodb 3.6 to mongodb 4.4. + +The commands are provided for a Linux environment, and assume ClearML is installed in its default paths. + +If you are using a different OS, replace the shell and docker commands to the ones appropriate for your OS. If you’ve +installed ClearML under a different path, replace the file locations accordingly. + +:::info automating your migration +A migration script is available to automatically run this process for all supported OSs. +[Download the script](https://github.com/allegroai/clearml-server/releases/download/1.2.0/clearml-server-1.2.0-migration.py) and run it on your ClearML server. +Run `clearml-server-1.2.0-migration.py –help` to see execution options. +Note the script will create a backup archive of your data in the original directory. +::: + +1. Backup the source data folder: + + ```bash + sudo tar -zcf mongodb_upgrade.gz -C /opt/clearml/data/mongo . + ``` +1. Copy the data to the new folder and set permissions (permissions are required on Linux): + + ```bash + sudo mkdir /opt/clearml/data/mongo_4 + sudo tar -xzf mongodb_upgrade.gz -C /opt/clearml/data/mongo_4 + sudo chown -R 1000:1000 /opt/clearml/data/mongo_4 + ``` +1. Run the mongo 4.0 container: + + ```bash + sudo docker run -id -v /opt/clearml/data/mongo_4/db:/data/db -v /opt/clearml/data/mongo_4/configdb:/data/configdb --name mongodb_upgrade mongo:4.0.27 + ``` +1. Upgrade data to mongo 4.0: + + ```bash + sudo docker exec mongodb_upgrade bash -c 'mongo --eval "db.adminCommand({setFeatureCompatibilityVersion:\"4.0\"})"' + ``` + This command should return an `{ "ok" : 1 }` response. +1. Stop the mongo 4.0 container + + ```bash + sudo docker stop mongodb_upgrade && sudo docker rm mongodb_upgrade + ``` +1. Run the mongo 4.2 container: + + ```bash + sudo docker run -id -v /opt/clearml/data/mongo_4/db:/data/db -v /opt/clearml/data/mongo_4/configdb:/data/configdb --name mongodb_upgrade mongo:4.2.16 + ``` +1. Upgrade data to mongo 4.2: + + ```bash + sudo docker exec mongodb_upgrade bash -c 'mongo --eval "db.adminCommand({setFeatureCompatibilityVersion:\"4.2\"})"' + ``` + This command should return an `{ "ok" : 1 }` response. +1. Stop the mongo 4.2 container + + ```bash + sudo docker stop mongodb_upgrade && sudo docker rm mongodb_upgrade + ``` +1. Run the mongo 4.4 container: + + ```bash + sudo docker run -id -v /opt/clearml/data/mongo_4/db:/data/db -v /opt/clearml/data/mongo_4/configdb:/data/configdb --name mongodb_upgrade mongo:4.4.9 + ``` +1. Upgrade data to mongo 4.4: + + ```bash + sudo docker exec mongodb_upgrade bash -c 'mongo --eval "db.adminCommand({setFeatureCompatibilityVersion:\"4.4\"})"' + ``` + This command should return an `{ "ok" : 1 }` response. +1. Stop the mongo 4.4 + + ```bash + sudo docker stop mongodb_upgrade && sudo docker rm mongodb_upgrade + ``` +1. Continue upgrading your ClearML Server: + * [AWS EC2 AMIs](upgrade_server_aws_ec2_ami.md) + * [Google Cloud Platform custom images](upgrade_server_gcp.md) + * [Linux and macOS](upgrade_server_linux_mac.md) + * [Windows](upgrade_server_win.md) + \ No newline at end of file diff --git a/docs/deploying_clearml/clearml_server_win.md b/docs/deploying_clearml/clearml_server_win.md index 3b83956e..0789faf0 100644 --- a/docs/deploying_clearml/clearml_server_win.md +++ b/docs/deploying_clearml/clearml_server_win.md @@ -2,10 +2,6 @@ title: Windows 10 --- -:::important -This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. -::: - For Windows, we recommend launching the pre-built Docker image on a Linux virtual machine (see [Deploying ClearML Server: Linux or macOS](clearml_server_linux_mac.md)). However, **ClearML Server** can be launched on Windows 10, using Docker Desktop for Windows (see the Docker [System Requirements](https://docs.docker.com/docker-for-windows/install/#system-requirements)). diff --git a/docs/deploying_clearml/upgrade_server_aws_ec2_ami.md b/docs/deploying_clearml/upgrade_server_aws_ec2_ami.md index d40bd30a..6f8ea3b7 100644 --- a/docs/deploying_clearml/upgrade_server_aws_ec2_ami.md +++ b/docs/deploying_clearml/upgrade_server_aws_ec2_ami.md @@ -2,10 +2,6 @@ title: AWS EC2 AMIs --- -:::important -This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. -::: - :::note For upgrade purposes, the terms **Trains Server** and **ClearML Server** are interchangeable. ::: @@ -34,9 +30,9 @@ Some legacy **Trains Server** AMIs provided an auto-upgrade on restart capabilit 1. We recommend [backing up your data](clearml_server_aws_ec2_ami.md#backing-up-and-restoring-data-and-configuration) and, if your configuration folder is not empty, backing up your configuration. -1. If upgrading from **Trains Server** version 0.15 or older, a data migration is required before upgrading, - First follow these [data migration instructions](clearml_server_es7_migration.md), and then continue this upgrade. - +1. If upgrading from ClearML Server version older than 1.2, you need to migrate your data before upgrading your server. See instructions [here](clearml_server_mongo44_migration.md). +If upgrading from Trains Server version 0.15 or older, a data migration is required before continuing this upgrade. See instructions [here](clearml_server_es7_migration.md). + 1. If upgrading from **Trains Server** to **ClearML Server**, rename `/opt/trains` to `/opt/clearml`. 1. Download the latest `docker-compose.yml` file. Execute the following command: diff --git a/docs/deploying_clearml/upgrade_server_gcp.md b/docs/deploying_clearml/upgrade_server_gcp.md index a47e21f4..b077f8b6 100644 --- a/docs/deploying_clearml/upgrade_server_gcp.md +++ b/docs/deploying_clearml/upgrade_server_gcp.md @@ -2,19 +2,16 @@ title: Google Cloud Platform --- -:::important -This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. -::: - **To upgrade ClearML Server Docker deployment:** 1. Shut down the docker containers with the following command: docker-compose -f docker-compose.yml down - + +1. If upgrading from ClearML Server version older than 1.2, you need to migrate your data before upgrading your server. See instructions [here](clearml_server_mongo44_migration.md). 1. If upgrading from **Trains Server** version 0.15 or older to **ClearML Server**, do the following: - 1. A data migration is required before upgrading. First follow these [data migration instructions](clearml_server_es7_migration.md), + 1. Follow these [data migration instructions](clearml_server_es7_migration.md), and then continue this upgrade. 1. Rename `/opt/trains` and its subdirectories to `/opt/clearml`. diff --git a/docs/deploying_clearml/upgrade_server_kubernetes.md b/docs/deploying_clearml/upgrade_server_kubernetes.md deleted file mode 100644 index 2cdfe0bd..00000000 --- a/docs/deploying_clearml/upgrade_server_kubernetes.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: Kubernetes ---- - - -:::important -Kubernetes support is being replaced by Kubernetes using Helm. Please see the new and improved [ClearML Helm Charts repository](https://github.com/allegroai/clearml-helm-charts) -for the most updated instructions. -::: - diff --git a/docs/deploying_clearml/upgrade_server_kubernetes_helm.md b/docs/deploying_clearml/upgrade_server_kubernetes_helm.md index e7eb5bbe..264cde46 100644 --- a/docs/deploying_clearml/upgrade_server_kubernetes_helm.md +++ b/docs/deploying_clearml/upgrade_server_kubernetes_helm.md @@ -1,5 +1,5 @@ --- -title: Kubernetes Helm +title: Kubernetes --- :::important @@ -8,9 +8,7 @@ for the most updated instructions. ::: :::note -* This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. - -* We strongly encourage to keep the **ClearML Server** up to date, by upgrading to the current release. +We strongly encourage to keep the **ClearML Server** up to date, by upgrading to the current release. ::: 1. Upgrade using new or upgraded values.yaml @@ -21,8 +19,8 @@ for the most updated instructions. helm delete --purge clearml-server -1. If upgrading from **Trains Server** version 0.15 or older to **ClearML Server**, a data migration is required before - upgrading. First follow these [data migration instructions](clearml_server_es7_migration.md), and then continue this upgrade. +1. If upgrading from Trains Server version 0.15 or older, a data migration is required before continuing this upgrade. + See instructions [here](clearml_server_es7_migration.md). 1. Upgrade deployment to match repository version. diff --git a/docs/deploying_clearml/upgrade_server_linux_mac.md b/docs/deploying_clearml/upgrade_server_linux_mac.md index d3200faa..97dc414d 100644 --- a/docs/deploying_clearml/upgrade_server_linux_mac.md +++ b/docs/deploying_clearml/upgrade_server_linux_mac.md @@ -2,12 +2,6 @@ title: Linux or macOS --- -:::important -This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. -::: - -
-
Important: Upgrading from v0.14 or older
@@ -35,12 +29,13 @@ For backwards compatibility, the environment variables ``TRAINS_HOST_IP``, ``TRA **To upgrade ClearML Server Docker deployment:** -1. Shutdown **ClearML Server**. Execute the following command (which assumes the configuration file is in the environment path). +1. Shutdown ClearML Server. Execute the following command (which assumes the configuration file is in the environment path). docker-compose -f docker-compose.yml down -1. If upgrading from **Trains Server** version 0.15 or older to **ClearML Server**, a data migration is required before - upgrading. First follow these [data migration instructions](clearml_server_es7_migration.md), and then continue this upgrade. +1. If upgrading from ClearML Server version older than 1.2, you need to migrate your data before upgrading your server. See instructions [here](clearml_server_mongo44_migration.md). + +1. If upgrading from **Trains Server** version 0.15 or older, a data migration is required before continuing this upgrade. See instructions [here](clearml_server_es7_migration.md). 1. We recommend [backing up data](clearml_server_linux_mac.md#backing-up-and-restoring-data-and-configuration) and, if the configuration folder is not empty, backing up the configuration. diff --git a/docs/deploying_clearml/upgrade_server_win.md b/docs/deploying_clearml/upgrade_server_win.md index ea7a0d93..9c9a9a8d 100644 --- a/docs/deploying_clearml/upgrade_server_win.md +++ b/docs/deploying_clearml/upgrade_server_win.md @@ -2,10 +2,6 @@ title: Windows --- -:::important -This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users. -::: - **To upgrade ClearML Server Docker deployment:** 1. Shut down the docker containers. @@ -20,8 +16,9 @@ This documentation page applies to deploying your own open source ClearML Server docker-compose -f c:\opt\trains\docker-compose-win10.yml down -1. If upgrading from **Trains Server** version 0.15 or older to **ClearML Server**, a data migration is required before - upgrading. First follow these [data migration instructions](clearml_server_es7_migration.md), and then continue this upgrade. +1. If upgrading from ClearML Server version older than 1.2, you need to migrate your data before upgrading your server. See instructions [here](clearml_server_mongo44_migration.md). + +1. If upgrading from **Trains Server** version 0.15 or older, a data migration is required before continuing this upgrade. See instructions [here](clearml_server_es7_migration.md). 1. We recommend backing up data and, if the configuration folder is not empty, backing up the configuration. diff --git a/sidebars.js b/sidebars.js index a92bb48e..55a35175 100644 --- a/sidebars.js +++ b/sidebars.js @@ -44,14 +44,14 @@ module.exports = { {'ClearML Server': ['deploying_clearml/clearml_server', { 'Deploying ClearML Server': ['deploying_clearml/clearml_server_aws_ec2_ami', 'deploying_clearml/clearml_server_gcp', - 'deploying_clearml/clearml_server_linux_mac', 'deploying_clearml/clearml_server_win', 'deploying_clearml/clearml_server_kubernetes', + 'deploying_clearml/clearml_server_linux_mac', 'deploying_clearml/clearml_server_win', 'deploying_clearml/clearml_server_kubernetes_helm'] }, { 'Upgrading ClearML Server':['deploying_clearml/upgrade_server_aws_ec2_ami','deploying_clearml/upgrade_server_gcp', 'deploying_clearml/upgrade_server_linux_mac', 'deploying_clearml/upgrade_server_win', - 'deploying_clearml/upgrade_server_kubernetes', 'deploying_clearml/upgrade_server_kubernetes_helm', - 'deploying_clearml/clearml_server_es7_migration'] + 'deploying_clearml/upgrade_server_kubernetes_helm', + 'deploying_clearml/clearml_server_es7_migration', 'deploying_clearml/clearml_server_mongo44_migration'] }, 'deploying_clearml/clearml_server_config', 'deploying_clearml/clearml_config_for_clearml_server', 'deploying_clearml/clearml_server_security' ]},