Add FAQs about server migrating (#106)

This commit is contained in:
pollfly 2021-11-07 10:59:25 +02:00 committed by GitHub
parent 4e65045f3b
commit 0488a5b470
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -14,6 +14,8 @@ title: FAQ
* [Can I store the model configuration file as well?](#store-model-configuration)
* [I am training multiple models at the same time, but I only see one of them. What happened?](#only-last-model-appears)
* [Can I log input and output models manually?](#manually-log-models)
* [Models are not accessible from the UI after I migrated ClearML Server to a new domain. How do I fix this?](#migrate_server_models)
* [Models are not accessible from the UI after I moved them (different bucket / server). How do I fix this?](#relocate_models)
**Experiments**
@ -44,6 +46,7 @@ title: FAQ
* [Is there something ClearML can do about uncommitted code running?](#help-uncommitted-code)
* [I read there is a feature for centralized model storage. How do I use it?](#centralized-model-storage)
* [When using PyCharm to remotely debug a machine, the Git repo is not detected. Do you have a solution?](#pycharm-remote-debug-detect-git)
* [Debug images and / or artifacts are not loading in the UI after I migrated ClearML Server to a new domain. How do I fix this?](#migrate_server_debug)
**Remote Debugging (ClearML PyCharm Plugin)**
@ -201,6 +204,68 @@ method to manually connect a model weights file.
For more information about models, see [InputModel](references/sdk/model_inputmodel.md)
and [OutputModel](references/sdk/model_outputmodel.md) classes.
<br/>
**Models are not accessible the UI after I migrated ClearML Server to a new domain. How do I fix this?** <a id="migrate_server_models"></a>
This can happen if your models were uploaded to the ClearML files server, since the value registered was their full URL
at the time of registration (e.g. `https://files.<OLD_DOMAIN>/path/to/model`).
To fix this, the registered URL of each model needs to be replaced with its current URL.
To replace the URL of each model, execute the following commands:
1. Open bash in the mongo DB docker container:
```bash
sudo docker exec -it clearml-mongo /bin/bash
```
1. Inside the docker shell, create the following script. Make sure to replace `<old-bucket-name>` and `<new-bucket-name>`,
as well as the URL protocol if you aren't using `s3`.
```bash
cat <<EOT >> script.js
db.model.find({uri:{$regex:/^s3/}}).forEach(function(e,i) {
e.uri = e.uri.replace("s3://<old-bucket-name>/","s3://<new-bucket-name>/");
db.model.save(e);});
EOT
```
1. Run the script against the backend DB:
```bash
mongo backend script.js
```
<br/>
**Models are not accessible from the UI after I moved them (different bucket / server). How do I fix this?** <a id="relocate_models"></a>
This can happen if your models were uploaded to the ClearML files server, since the value registered was their full URL
at the time of registration (e.g. `https://files.<OLD_DOMAIN>/path/to/model`).
To fix this, the registered URL of each model needs to be replaced with its current URL:
1. Open bash in the mongo DB Docker container:
```bash
sudo docker exec -it clearml-mongo /bin/bash
```
1. Inside the Docker shell, create the following script. Make sure to replace `<old-bucket-name>` and `<new-bucket-name>`, as well as the URL protocol prefixes if you aren't using S3.
```bash
cat <<EOT >> script.js
db.model.find({uri:{$regex:/^s3/}}).forEach(function(e,i) {
e.uri = e.uri.replace("s3://<old-bucket-name>/","s3://<new-bucket-name>/");
db.model.save(e);});
EOT
```
1. Run the script against the backend DB:
```bash
mongo backend script.js
```
## Experiments
**I noticed I keep getting the message "warning: uncommitted code". What does it mean?** <a id="uncommitted-code-warning"></a>
@ -521,6 +586,56 @@ see [ClearML Configuration Reference](configs/clearml_conf.md).
Yes! Since this is such a common occurrence, we created a PyCharm plugin that allows a remote debugger to grab your local
repository / commit ID. For detailed information about using the plugin, see the [ClearML PyCharm Plugin](guides/ide/integration_pycharm.md).
<br/>
**Debug images and/or artifacts are not loading in the UI after I migrated ClearML Server to a new domain. How do I fix this?** <a id="migrate_server_debug"></a>
This can happen if your debug images and / or artifacts were uploaded to the ClearML file server, since the value
registered was their full URL at the time of registration (e.g. `https://files.<OLD_DOMAIN>/path/to/artifact`).
To fix this, the registered URL of each debug image and / or artifact needs to be replaced with its current URL.
* For **debug images**, use the following command. Make sure to insert the old domain and the new domain that will replace it
```bash
curl --header "Content-Type: application/json" \
--request POST \
--data '{
"script": {
"source": "ctx._source.url = ctx._source.url.replace('https://files.<OLD_DOMAIN>', 'https://files.<NEW_DOMAIN>')",
"lang": "painless"
},
"query": {
"match_all": {}
}
}' \
```
* For **artifacts**, you can do the following:
1. Open bash in the mongo DB docker container:
```bash
sudo docker exec -it clearml-mongo /bin/bash
```
1. Inside the docker shell, create the following script. Make sure to replace `<old-bucket-name>` and `<new-bucket-name>`,
as well as the URL protocol prefixes if you aren't using `s3`.
```bash
cat <<EOT >> script.js
db.model.find({uri:{$regex:/^s3/}}).forEach(function(e,i) {
e.uri = e.uri.replace("s3://<old-bucket-name>/","s3://<new-bucket-name>/");
db.model.save(e);});
EOT
```
1. Run the script against the backend DB:
```bash
mongo backend script.js
```
## Jupyter
**I am using Jupyter Notebook. Is this supported?** <a id="jupyter-notebook"></a>
@ -880,6 +995,7 @@ api {
```
## ClearML Agent
**How can I execute ClearML Agent without installing packages each time?** <a className="tr_top_negative" id="system_site_packages"></a>