Fix storage examples (#811)

This commit is contained in:
pollfly 2024-03-26 10:10:37 +02:00 committed by GitHub
parent ec1f4d069f
commit 57be45d2a8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -45,7 +45,8 @@ You can specify additional [ExtraArgs](https://boto3.amazonaws.com/v1/documentat
to pass to boto3 when uploading files. You can set this on a per-bucket basis. to pass to boto3 when uploading files. You can set this on a per-bucket basis.
``` ```
aws { sdk {
aws {
s3 { s3 {
# S3 credentials, used for read/write access by various SDK elements # S3 credentials, used for read/write access by various SDK elements
@ -72,6 +73,7 @@ aws {
pool_connections: 512 pool_connections: 512
max_multipart_concurrency: 16 max_multipart_concurrency: 16
} }
}
} }
``` ```
@ -79,19 +81,22 @@ AWS's S3 access parameters can be specified by referencing the standard environm
For example: For example:
``` ```
aws { sdk {
aws {
s3 { s3 {
# default, used for any bucket not specified below # default, used for any bucket not specified below
key: ${AWS_ACCESS_KEY_ID} key: ${AWS_ACCESS_KEY_ID}
secret: ${AWS_SECRET_ACCESS_KEY} secret: ${AWS_SECRET_ACCESS_KEY}
region: ${AWS_DEFAULT_REGION} region: ${AWS_DEFAULT_REGION}
} }
}
} }
``` ```
ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration: ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration:
``` ```
aws { sdk {
aws {
s3 { s3 {
# default, used for any bucket not specified below # default, used for any bucket not specified below
key: "" key: ""
@ -109,6 +114,7 @@ aws {
} }
] ]
} }
}
} }
``` ```
@ -121,7 +127,8 @@ To enable TLS, pass `secure: true`.
To configure Azure blob storage specify the account name and key. To configure Azure blob storage specify the account name and key.
``` ```
azure.storage { sdk {
azure.storage {
containers: [ containers: [
{ {
account_name: "" account_name: ""
@ -129,6 +136,7 @@ azure.storage {
# container_name: # container_name:
} }
] ]
}
} }
``` ```
@ -136,7 +144,8 @@ Azure's storage access parameters can be specified by referencing the standard e
For example: For example:
``` ```
azure.storage { sdk {
azure.storage {
containers: [ containers: [
{ {
account_name: ${AZURE_STORAGE_ACCOUNT} account_name: ${AZURE_STORAGE_ACCOUNT}
@ -144,6 +153,7 @@ azure.storage {
# container_name: # container_name:
} }
] ]
}
} }
``` ```
@ -154,7 +164,8 @@ It's also possible to specify credentials for a specific bucket in the `google.s
configuration provided in the `google.storage` section is applied to any bucket without a bucket-specific configuration. configuration provided in the `google.storage` section is applied to any bucket without a bucket-specific configuration.
``` ```
google.storage { sdk {
google.storage {
# Default project and credentials file # Default project and credentials file
# Will be used when no bucket configuration is found # Will be used when no bucket configuration is found
project: "clearml" project: "clearml"
@ -169,13 +180,15 @@ google.storage {
credentials_json: "/path/to/credentials.json" credentials_json: "/path/to/credentials.json"
}, },
] ]
}
} }
``` ```
GCP's storage access parameters can be specified by referencing the standard environment variables if already defined. GCP's storage access parameters can be specified by referencing the standard environment variables if already defined.
``` ```
google.storage { sdk {
google.storage {
credentials = [ credentials = [
{ {
bucket: "" bucket: ""
@ -184,6 +197,7 @@ google.storage {
credentials_json: ${GOOGLE_APPLICATION_CREDENTIALS} credentials_json: ${GOOGLE_APPLICATION_CREDENTIALS}
}, },
] ]
}
} }
``` ```
@ -208,8 +222,8 @@ substitution allows for registering the data into `clearml-data` once, and then
To enable path substitution, modify the clearml.conf file and configure: To enable path substitution, modify the clearml.conf file and configure:
```bash ```bash
sdk{ sdk {
storage{ storage {
path_substitution = [ path_substitution = [
# Replace registered links with local prefixes, # Replace registered links with local prefixes,
# Solve mapping issues, and allow for external resource caching. # Solve mapping issues, and allow for external resource caching.
@ -233,7 +247,8 @@ piece twice!
Configure cache location by modifying the [clearml.conf](../configs/clearml_conf.md) file: Configure cache location by modifying the [clearml.conf](../configs/clearml_conf.md) file:
``` ```
storage { sdk {
storage {
cache { cache {
# Defaults to <system_temp_folder>/clearml_cache # Defaults to <system_temp_folder>/clearml_cache
default_base_dir: "~/.clearml/cache" default_base_dir: "~/.clearml/cache"
@ -245,6 +260,7 @@ storage {
# Objects are specified in glob format, available for url and content_type. # Objects are specified in glob format, available for url and content_type.
{ url: "file://*" } # file-urls are always directly referenced { url: "file://*" } # file-urls are always directly referenced
] ]
}
} }
``` ```