--- title: Exporting and Importing ClearML Projects --- When migrating from a ClearML Open Server to a ClearML Enterprise Server, you may need to transfer projects. This is done using the `data_tool.py` script. This utility is available in the `apiserver` Docker image, and can be used for exporting and importing ClearML project data for both open source and Enterprise versions. This guide covers the following: * Exporting data from Open Source and Enterprise servers * Importing data into an Enterprise server * Handling the artifacts stored on the file server. :::note Export instructions differ for ClearML open and Enterprise servers. Make sure you follow the guidelines that match your server type. ::: ## Exporting Data The export process is done by running the ***data_tool*** script that generates a zip file containing project and task data. This file should then be copied to the server on which the import will run. Note that artifacts stored in the ClearML ***file server*** should be copied manually if required (see [Handling Artifacts](#handling-artifacts)). ### Exporting Data from ClearML Open Servers #### Preparation * Make sure the `apiserver` is at least Open Source server version 1.12.0. * Note that any `pending` or `running` tasks will not be exported. If you wish to export them, make sure to stop/dequeue them before exporting. #### Running the Data Tool Execute the data tool within the `apiserver` container. Open a bash session inside the `apiserver` container of the server: * In docker-compose: ```commandline sudo docker exec -it clearml-apiserver /bin/bash ``` * In Kubernetes: ```commandline kubectl exec -it -n -- bash ``` #### Export Commands **To export specific projects:** ```commandline python3 -m apiserver.data_tool export --projects --statuses created stopped published failed completed --output .zip ``` As a result, you should get a `.zip` file that contains all the data from the specified projects and their children. **To export all the projects:** ```commandline python3 -m apiserver.data_tool export \ --all \ --statuses created stopped published failed completed \ --output .zip ``` #### Optional Parameters * `--experiments ` - If not specified then all experiments from the specified projects are exported * `--statuses ` - Export tasks of specific statuses. If the parameter is omitted, only `published` tasks are exported * `--no-events` - Do not export task events, i.e. logs and metrics (scalar, plots, debug samples). Make sure to copy the generated zip file containing the exported data. ### Exporting Data from ClearML Enterprise Servers #### Preparation * Make sure the `apiserver` is at least Enterprise Server version 3.18.0. * Note that any `pending` or `running` tasks will not be exported. If you wish to export them, make sure to stop/dequeue before exporting. #### Running the Data Tool Execute the data tool from within the `apiserver` docker container. Open a bash session inside the `apiserver` container of the server: * In `docker-compose`: ```commandline sudo docker exec -it allegro-apiserver /bin/bash ``` * In Kubernetes: ```commandline kubectl exec -it -n -- bash ``` #### Export Commands **To export specific projects:** ```commandline PYTHONPATH=/opt/seematics/apiserver/trains-server-repo python3 data_tool.py \ export \ --projects \ --statuses created stopped published failed completed \ --output .zip ``` As a result, you should get `.zip` file that contains all the data from the specified projects and their children. **To export all the projects:** ```commandline PYTHONPATH=/opt/seematics/apiserver/trains-server-repo python3 data_tool.py \ export \ --all \ --statuses created stopped published failed completed \ --output .zip ``` #### Optional Parameters * `--experiments ` - If not specified then all experiments from the specified projects are exported * `--statuses ` - Can be used to allow exporting tasks of specific statuses. If the parameter is omitted, only `published` tasks are exported. * `--no-events` - Do not export task events, i.e. logs, and metrics (scalar, plots, debug samples). Make sure to copy the generated zip file containing the exported data. ## Importing Data This section explains how to import the exported data into a ClearML Enterprise server. ### Preparation * It is highly recommended to back up the ClearML databases before importing data, as import injects data into the databases, and can't be undone. * Make sure you are working with `apiserver` version 3.22.3 or higher. * Make the zip file accessible from within the `apiserver` container by copying the exported data to the `apiserver` container or to a folder on the host, which the `apiserver` is mounted to. ### Usage The data tool should be executed from within the `apiserver` docker container. 1. Open a bash session inside the `apiserver` container of the server: * In `docker-compose`: ```commandline sudo docker exec -it allegro-apiserver /bin/bash ``` * In Kubernetes: ```commandline kubectl exec -it -n -- bash ``` 1. Run the data tool script in *import* mode: ```commandline PYTHONPATH=/opt/seematics/apiserver/trains-server-repo python3 data_tool.py \ import \ \ --company \ --user ``` * `company_id`- The default company ID used in the target deployment. Inside the `apiserver` container you can usually get it from the environment variable `CLEARML__APISERVER__DEFAULT_COMPANY`. If you do not specify the `--company` parameter then all the data will be imported as `Examples` (read-only) * `user_id` - The ID of the user in the target deployment who will become the owner of the imported data ## Handling Artifacts ***Artifacts*** refers to any content which the ClearML server holds references to. This can include: * Dataset or Hyper-Dataset frame URLs * ClearML artifact URLs * Model snapshots * Debug samples Artifacts may be stored in any external storage (e.g., AWS S3, minio, Google Cloud Storage) or in the ClearML file server. * If the artifacts are **not** stored in the ClearML file server, they do not need to be moved during the export/import process, as the URLs registered in ClearML entities pointing to these artifacts will not change. * If the artifacts are stored in the ClearML file server, then the file server content must also be moved, and the URLs in the ClearML databases must point to the new location. See instructions [below](#exporting-file-server-data-for-clearml-open-server). ### Exporting File Server Data for ClearML Open Server Data in the file server is organized by project. For each project, all data references by entities in that project is stored in a folder bearing the name of the project. This folder can be located in: ``` /opt/clearml/data/fileserver/ ``` The entire projects' folders content should be copied to the target server (see [Importing Fileserver Data](#importing-file-server-data)). ### Exporting File Server Data for ClearML Enterprise Server Data in the file server is organized by tenant and project. For each project, all data references by entities in that project is stored in a folder bearing the name of the project. This folder can be located in: ``` /opt/allegro/data/fileserver// ``` The entire projects' folders content should be copied to the target server (see [Importing Fileserver Data](#importing-file-server-data)). ## Importing File Server Data ### Copying the Data Place the exported projects' folder(s) content into the target file server's storage in the following folder: ``` /opt/allegro/data/fileserver// ``` ### Fixing Registered URLs Since URLs pointing to the file server contain the file server's address, these need to be changed to the address of the new file server. Note that this is not required if the new file server is replacing the old file server and can be accessed using the same exact address. Once the projects' data has been copied to the target server, and the projects themselves were imported, see [Changing CleaML Artifacts Links](change_artifact_links.md) for information on how to fix the URLs.