ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-03-03 18:53:37 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Reformat storage.md (#127)

Browse Source
This commit is contained in:
pollfly 2021-12-02 19:56:33 +02:00 committed by GitHub
parent 49de7323ab
commit 7077afbe24
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 100 additions and 91 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

109
docs/integrations/storage.md
View File

@ -36,12 +36,12 @@ aws {
credentials: [ credentials: [
# specifies key/secret credentials to use when handling s3 urls (read or write) # specifies key/secret credentials to use when handling s3 urls (read or write)
# { {
# bucket: "my-bucket-name" bucket: "my-bucket-name"
# key: "my-access-key" key: ""
# secret: "my-secret-key" secret: ""
# verify: "/path/to/ca/bundle.crt" OR false to not verify verify: "/path/to/ca/bundle.crt" OR false to not verify
# }, },
] ]
} }
@ -56,50 +56,56 @@ AWS's S3 access parameters can be specified by referencing the standard environm
For example: For example:
``` ```
aws {
s3 { s3 {
...
# default, used for any bucket not specified below # default, used for any bucket not specified below
key: "${AWS_ACCESS_KEY_ID}" key: "${AWS_ACCESS_KEY_ID}"
secret: "${AWS_SECRET_ACCESS_KEY}" secret: "${AWS_SECRET_ACCESS_KEY}"
region: "${AWS_DEFAULT_REGION}" region: "${AWS_DEFAULT_REGION}"
... }
} }
``` ```
ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration: ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration:
``` ```
# { aws {
# host: "my-minio-host:9000" s3 {
# key: "12345678" # default, used for any bucket not specified below
# secret: "12345678" key: ""
# multipart: false secret: ""
# secure: false region: ""
# }
credentials: [
{
# This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
host: "my-minio-host:9000"
key: ""
secret: ""
multipart: false
secure: false
}
]
}
}
``` ```
Note: to force usage of a non Amazon endpoint (like the MinIO example above), port declaration is *always* needed, even if standard; this is an example [MinIO](https://github.com/minio/minio) with TLS enabled: :::info non-AWS Endpoints
``` To force usage of a non-AWS endpoint (like the MinIO example above), port declaration is *always* needed, even if standard.
# { To enable TLS, pass `secure: true`.
# host: "my-minio-host:443" :::
# key: "12345678"
# secret: "12345678"
# multipart: false
# secure: true
# }
```
### Configuring Azure ### Configuring Azure
To configure Azure blob storage specify the account name and key. To configure Azure blob storage specify the account name and key.
``` ```
azure.storage { azure.storage {
# containers: [ containers: [
# { {
# account_name: "clearml" account_name: ""
# account_key: "secret" account_key: ""
# # container_name: # container_name:
# } }
# ] ]
} }
``` ```
@ -107,7 +113,7 @@ Azure's storage access parameters can be specified by referencing the standard e
For example: For example:
``` ```
... azure.storage {
containers: [ containers: [
{ {
account_name: "${AZURE_STORAGE_ACCOUNT}" account_name: "${AZURE_STORAGE_ACCOUNT}"
@ -115,6 +121,7 @@ containers: [
# container_name: # container_name:
} }
] ]
}
``` ```
### Configuring Google Storage ### Configuring Google Storage
@ -123,34 +130,36 @@ It's also possible to specify credentials for a specific bucket.
``` ```
google.storage { google.storage {
# # Default project and credentials file # Default project and credentials file
# # Will be used when no bucket configuration is found # Will be used when no bucket configuration is found
# project: "clearml" project: "clearml"
# credentials_json: "/path/to/credentials.json" credentials_json: "/path/to/credentials.json"
# # Specific credentials per bucket and sub directory # Specific credentials per bucket and sub directory
# credentials = [ credentials = [
# { {
# bucket: "my-bucket" bucket: ""
# subdir: "path/in/bucket" # Not required subdir: "path/in/bucket" # Not required
# project: "clearml" project: ""
# credentials_json: "/path/to/credentials.json" credentials_json: "/path/to/credentials.json"
# }, },
# ] ]
} }
``` ```
GCP's storage access parameters can be specified by referencing the standard environment variables if already defined. GCP's storage access parameters can be specified by referencing the standard environment variables if already defined.
``` ```
... google.storage {
credentials = [ credentials = [
{ {
bucket: "my-bucket" bucket: ""
... subdir: "path/in/bucket" # Not required
project: ""
credentials_json: "${GOOGLE_APPLICATION_CREDENTIALS}" credentials_json: "${GOOGLE_APPLICATION_CREDENTIALS}"
},
]
} }
``` ```
## Storage Manager ## Storage Manager