Fix storage examples (#811)

This commit is contained in:
pollfly 2024-03-26 10:10:37 +02:00 committed by GitHub
parent ec1f4d069f
commit 57be45d2a8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -45,6 +45,7 @@ You can specify additional [ExtraArgs](https://boto3.amazonaws.com/v1/documentat
to pass to boto3 when uploading files. You can set this on a per-bucket basis.
```
sdk {
aws {
s3 {
# S3 credentials, used for read/write access by various SDK elements
@ -73,12 +74,14 @@ aws {
max_multipart_concurrency: 16
}
}
}
```
AWS's S3 access parameters can be specified by referencing the standard environment variables if already defined.
For example:
```
sdk {
aws {
s3 {
# default, used for any bucket not specified below
@ -87,10 +90,12 @@ aws {
region: ${AWS_DEFAULT_REGION}
}
}
}
```
ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration:
```
sdk {
aws {
s3 {
# default, used for any bucket not specified below
@ -110,6 +115,7 @@ aws {
]
}
}
}
```
:::info non-AWS Endpoints
@ -121,6 +127,7 @@ To enable TLS, pass `secure: true`.
To configure Azure blob storage specify the account name and key.
```
sdk {
azure.storage {
containers: [
{
@ -130,12 +137,14 @@ azure.storage {
}
]
}
}
```
Azure's storage access parameters can be specified by referencing the standard environment variables if already defined.
For example:
```
sdk {
azure.storage {
containers: [
{
@ -145,6 +154,7 @@ azure.storage {
}
]
}
}
```
### Configuring Google Storage
@ -154,6 +164,7 @@ It's also possible to specify credentials for a specific bucket in the `google.s
configuration provided in the `google.storage` section is applied to any bucket without a bucket-specific configuration.
```
sdk {
google.storage {
# Default project and credentials file
# Will be used when no bucket configuration is found
@ -170,11 +181,13 @@ google.storage {
},
]
}
}
```
GCP's storage access parameters can be specified by referencing the standard environment variables if already defined.
```
sdk {
google.storage {
credentials = [
{
@ -185,6 +198,7 @@ google.storage {
},
]
}
}
```
:::tip Direct Decoding
@ -233,6 +247,7 @@ piece twice!
Configure cache location by modifying the [clearml.conf](../configs/clearml_conf.md) file:
```
sdk {
storage {
cache {
# Defaults to <system_temp_folder>/clearml_cache
@ -246,6 +261,7 @@ storage {
{ url: "file://*" } # file-urls are always directly referenced
]
}
}
```
### Direct Access