ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-03-03 10:42:51 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Reformat storage.md (#127)

Browse Source
This commit is contained in:
pollfly 2021-12-02 19:56:33 +02:00 committed by GitHub
parent 49de7323ab
commit 7077afbe24
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 100 additions and 91 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

109
docs/integrations/storage.md
View File

@ -36,12 +36,12 @@ aws {
credentials: [
# specifies key/secret credentials to use when handling s3 urls (read or write)
# {
# bucket: "my-bucket-name"
# key: "my-access-key"
# secret: "my-secret-key"
# verify: "/path/to/ca/bundle.crt" OR false to not verify
# },
{
bucket: "my-bucket-name"
key: ""
secret: ""
verify: "/path/to/ca/bundle.crt" OR false to not verify
},
]
}
@ -56,50 +56,56 @@ AWS's S3 access parameters can be specified by referencing the standard environm
For example:
```
aws {
s3 {
...
# default, used for any bucket not specified below
key: "${AWS_ACCESS_KEY_ID}"
secret: "${AWS_SECRET_ACCESS_KEY}"
region: "${AWS_DEFAULT_REGION}"
...
}
}
```
ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration:
```
# {
# host: "my-minio-host:9000"
# key: "12345678"
# secret: "12345678"
# multipart: false
# secure: false
# }
aws {
s3 {
# default, used for any bucket not specified below
key: ""
secret: ""
region: ""
credentials: [
{
# This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
host: "my-minio-host:9000"
key: ""
secret: ""
multipart: false
secure: false
}
]
}
}
```
Note: to force usage of a non Amazon endpoint (like the MinIO example above), port declaration is *always* needed, even if standard; this is an example [MinIO](https://github.com/minio/minio) with TLS enabled:
```
# {
# host: "my-minio-host:443"
# key: "12345678"
# secret: "12345678"
# multipart: false
# secure: true
# }
```
:::info non-AWS Endpoints
To force usage of a non-AWS endpoint (like the MinIO example above), port declaration is *always* needed, even if standard.
To enable TLS, pass `secure: true`.
:::
### Configuring Azure
To configure Azure blob storage specify the account name and key.
```
azure.storage {
# containers: [
# {
# account_name: "clearml"
# account_key: "secret"
# # container_name:
# }
# ]
containers: [
{
account_name: ""
account_key: ""
# container_name:
}
]
}
```
@ -107,7 +113,7 @@ Azure's storage access parameters can be specified by referencing the standard e
For example:
```
...
azure.storage {
containers: [
{
account_name: "${AZURE_STORAGE_ACCOUNT}"
@ -115,6 +121,7 @@ containers: [
# container_name:
}
]
}
```
### Configuring Google Storage
@ -123,34 +130,36 @@ It's also possible to specify credentials for a specific bucket.
```
google.storage {
# # Default project and credentials file
# # Will be used when no bucket configuration is found
# project: "clearml"
# credentials_json: "/path/to/credentials.json"
# Default project and credentials file
# Will be used when no bucket configuration is found
project: "clearml"
credentials_json: "/path/to/credentials.json"
# # Specific credentials per bucket and sub directory
# credentials = [
# {
# bucket: "my-bucket"
# subdir: "path/in/bucket" # Not required
# project: "clearml"
# credentials_json: "/path/to/credentials.json"
# },
# ]
# Specific credentials per bucket and sub directory
credentials = [
{
bucket: ""
subdir: "path/in/bucket" # Not required
project: ""
credentials_json: "/path/to/credentials.json"
},
]
}
```
GCP's storage access parameters can be specified by referencing the standard environment variables if already defined.
```
...
google.storage {
credentials = [
{
bucket: "my-bucket"
...
bucket: ""
subdir: "path/in/bucket" # Not required
project: ""
credentials_json: "${GOOGLE_APPLICATION_CREDENTIALS}"
},
]
}
```
## Storage Manager