Fix storage examples (#811)

This commit is contained in:
pollfly 2024-03-26 10:10:37 +02:00 committed by GitHub
parent ec1f4d069f
commit 57be45d2a8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -45,7 +45,8 @@ You can specify additional [ExtraArgs](https://boto3.amazonaws.com/v1/documentat
to pass to boto3 when uploading files. You can set this on a per-bucket basis.
```
aws {
sdk {
aws {
s3 {
# S3 credentials, used for read/write access by various SDK elements
@ -72,6 +73,7 @@ aws {
pool_connections: 512
max_multipart_concurrency: 16
}
}
}
```
@ -79,19 +81,22 @@ AWS's S3 access parameters can be specified by referencing the standard environm
For example:
```
aws {
sdk {
aws {
s3 {
# default, used for any bucket not specified below
key: ${AWS_ACCESS_KEY_ID}
secret: ${AWS_SECRET_ACCESS_KEY}
region: ${AWS_DEFAULT_REGION}
}
}
}
```
ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration:
```
aws {
sdk {
aws {
s3 {
# default, used for any bucket not specified below
key: ""
@ -109,6 +114,7 @@ aws {
}
]
}
}
}
```
@ -121,7 +127,8 @@ To enable TLS, pass `secure: true`.
To configure Azure blob storage specify the account name and key.
```
azure.storage {
sdk {
azure.storage {
containers: [
{
account_name: ""
@ -129,6 +136,7 @@ azure.storage {
# container_name:
}
]
}
}
```
@ -136,7 +144,8 @@ Azure's storage access parameters can be specified by referencing the standard e
For example:
```
azure.storage {
sdk {
azure.storage {
containers: [
{
account_name: ${AZURE_STORAGE_ACCOUNT}
@ -144,6 +153,7 @@ azure.storage {
# container_name:
}
]
}
}
```
@ -154,7 +164,8 @@ It's also possible to specify credentials for a specific bucket in the `google.s
configuration provided in the `google.storage` section is applied to any bucket without a bucket-specific configuration.
```
google.storage {
sdk {
google.storage {
# Default project and credentials file
# Will be used when no bucket configuration is found
project: "clearml"
@ -169,13 +180,15 @@ google.storage {
credentials_json: "/path/to/credentials.json"
},
]
}
}
```
GCP's storage access parameters can be specified by referencing the standard environment variables if already defined.
```
google.storage {
sdk {
google.storage {
credentials = [
{
bucket: ""
@ -184,6 +197,7 @@ google.storage {
credentials_json: ${GOOGLE_APPLICATION_CREDENTIALS}
},
]
}
}
```
@ -208,8 +222,8 @@ substitution allows for registering the data into `clearml-data` once, and then
To enable path substitution, modify the clearml.conf file and configure:
```bash
sdk{
storage{
sdk {
storage {
path_substitution = [
# Replace registered links with local prefixes,
# Solve mapping issues, and allow for external resource caching.
@ -233,7 +247,8 @@ piece twice!
Configure cache location by modifying the [clearml.conf](../configs/clearml_conf.md) file:
```
storage {
sdk {
storage {
cache {
# Defaults to <system_temp_folder>/clearml_cache
default_base_dir: "~/.clearml/cache"
@ -245,6 +260,7 @@ storage {
# Objects are specified in glob format, available for url and content_type.
{ url: "file://*" } # file-urls are always directly referenced
]
}
}
```