ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-03-03 10:42:51 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Exemplify configuration file environment variable access (#90)

Browse Source
This commit is contained in:
pollfly 2021-10-21 09:44:30 +03:00 committed by GitHub
parent 63e3ff1f9c
commit d05555b01a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 44 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
docs/integrations/storage.md
View File

@ -18,6 +18,8 @@ Once uploading an object to a storage medium, each machine that uses the object
Configuration for storage is done by editing the [clearml.conf](../configs/clearml_conf.md). Configuration for storage is done by editing the [clearml.conf](../configs/clearml_conf.md).
The ClearML configuration file uses [HOCON](https://github.com/lightbend/config/blob/main/HOCON.md) format, which supports runtime environment variable access.
### Configuring AWS S3 ### Configuring AWS S3
Modify these parts of the clearml.conf file and add the key, secret, and region of the s3 bucket. Modify these parts of the clearml.conf file and add the key, secret, and region of the s3 bucket.
@ -49,6 +51,20 @@ aws {
} }
``` ```
AWS's S3 access parameters can be specified by referencing the standard environment variables if already defined.
For example:
```
s3 {
...
# default, used for any bucket not specified below
key: "${AWS_ACCESS_KEY_ID}"
secret: "${AWS_SECRET_ACCESS_KEY}"
region: "${AWS_DEFAULT_REGION}"
...
}
```
ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration: ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration:
``` ```
# { # {
@ -60,6 +76,7 @@ ClearML also supports [MinIO](https://github.com/minio/minio) by adding this con
# } # }
``` ```
### Configuring Azure ### Configuring Azure
To configure Azure blob storage specify the account name and key. To configure Azure blob storage specify the account name and key.
@ -75,6 +92,20 @@ To configure Azure blob storage specify the account name and key.
} }
``` ```
Azure's storage access parameters can be specified by referencing the standard environment variables if already defined.
For example:
```
...
containers: [
{
account_name: "${AZURE_STORAGE_ACCOUNT}"
account_key: "${AZURE_STORAGE_KEY}"
# container_name:
}
]
```
### Configuring Google Storage ### Configuring Google Storage
To configure Google Storage, specify the project and the path to the credentials json file. To configure Google Storage, specify the project and the path to the credentials json file.
It's also possible to specify credentials for a specific bucket. It's also possible to specify credentials for a specific bucket.
@ -98,6 +129,19 @@ It's also possible to specify credentials for a specific bucket.
} }
``` ```
GCP's storage access parameters can be specified by referencing the standard environment variables if already defined.
```
...
credentials = [
{
bucket: "my-bucket"
...
credentials_json: "${GOOGLE_APPLICATION_CREDENTIALS}"
}
```
## Storage Manager ## Storage Manager
ClearML offers the [StorageManager](../references/sdk/storage.md) class to manage downloading, uploading, and caching of ClearML offers the [StorageManager](../references/sdk/storage.md) class to manage downloading, uploading, and caching of