ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-01-31 06:27:22 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Edit AWS S3 storage section (#913)

Browse Source
This commit is contained in:
pollfly 2024-08-28 17:06:20 +03:00 committed by GitHub
parent 50ead21ad9
commit 187cfcbc8a
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 41 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

53
docs/integrations/storage.md
View File

@ -32,17 +32,17 @@ The ClearML configuration file uses [HOCON](https://github.com/lightbend/config/
### Configuring AWS S3 ### Configuring AWS S3
Modify these parts of the clearml.conf file and add the key, secret, and region of the S3 bucket. Modify the `sdk.aws.s3` section of the `clearml.conf` to add the key, secret, and region of the S3 bucket.
You can also give access to specific S3 buckets in the `aws.s3.credentials` section. The default configuration You can also give access to specific S3 buckets in the `sdk.aws.s3.credentials` section. The default configuration
provided in the `aws.s3` section is applied to any bucket without a bucket-specific configuration. provided in the `sdk.aws.s3` section is applied to any bucket without a bucket-specific configuration.
You can also enable using a credentials chain to let Boto3 You can also enable using a credentials chain to let Boto3
pick the right credentials. This includes picking credentials from environment variables, a credential file, and metadata service pick the right credentials. This includes picking credentials from environment variables, a credential file, and metadata service
with an IAM role configured. See [Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). with an IAM role configured. For more details, see [Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
You can specify additional [ExtraArgs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html#the-extraargs-parameter) You can specify additional [ExtraArgs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html#the-extraargs-parameter)
to pass to boto3 when uploading files. You can set this on a per-bucket basis. to pass to Boto3 when uploading files. You can set this on a per-bucket basis.
``` ```
sdk { sdk {
@ -63,7 +63,6 @@ sdk {
bucket: "my-bucket-name" bucket: "my-bucket-name"
key: "" key: ""
secret: "" secret: ""
verify: "/path/to/ca/bundle.crt" OR false to not verify
use_credentials_chain: false use_credentials_chain: false
}, },
@ -77,7 +76,7 @@ sdk {
} }
``` ```
AWS's S3 access parameters can be specified by referencing the standard environment variables if already defined. AWS S3 access parameters can be specified by referencing the standard environment variables if they are already defined.
For example: For example:
``` ```
@ -93,7 +92,10 @@ sdk {
} }
``` ```
ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration: #### Non-AWS Endpoints
ClearML supports any S3-compatible services, such as [MinIO](https://github.com/minio/minio) as well as other
cloud-based or locally deployed storage services. For non-AWS endpoints, use a configuration like this:
``` ```
sdk { sdk {
aws { aws {
@ -111,6 +113,7 @@ sdk {
secret: "" secret: ""
multipart: false multipart: false
secure: false secure: false
verify: true # OR "/path/to/ca/bundle.crt" OR "https://url/of/ca/bundle.crt" OR false to not verify
} }
] ]
} }
@ -118,10 +121,36 @@ sdk {
} }
``` ```
:::info non-AWS Endpoints To force usage of a non-AWS endpoint, port declaration is *always* needed (e.g. `host: "my-minio-host:9000"`),
To force usage of a non-AWS endpoint (like the MinIO example above), port declaration is *always* needed, even if standard. even for standard ports like `433` for HTTPS (e.g. `host: "my-minio-host:433"`).
To enable TLS, pass `secure: true`.
::: To enable TLS, pass `secure: true`. For example:
```
sdk {
aws {
s3 {
key: ""
secret: ""
region: ""
credentials: [
{
host: "my-minio-host:9000"
key: ""
secret: ""
multipart: false
secure: true
verify: true
}
]
}
}
}
```
Use the `sdk.aws.s3.credentials.verify` configuration option to control SSL certificate verification:
* By default, verify is set to `true`, meaning certificate verification is enabled
* You can provide a path or a URL to a CA bundle for custom certificate verification
### Configuring Azure ### Configuring Azure
To configure Azure blob storage specify the account name and key. To configure Azure blob storage specify the account name and key.