ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-06-08 23:57:40 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Fix storage examples (#811)

Browse Source
This commit is contained in:
pollfly 2024-03-26 10:10:37 +02:00 committed by GitHub
parent ec1f4d069f
commit 57be45d2a8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 120 additions and 104 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
docs/integrations/storage.md
View File

@ -45,6 +45,7 @@ You can specify additional [ExtraArgs](https://boto3.amazonaws.com/v1/documentat
to pass to boto3 when uploading files. You can set this on a per-bucket basis. to pass to boto3 when uploading files. You can set this on a per-bucket basis.
``` ```
sdk {
aws { aws {
s3 { s3 {
# S3 credentials, used for read/write access by various SDK elements # S3 credentials, used for read/write access by various SDK elements
@ -73,12 +74,14 @@ aws {
max_multipart_concurrency: 16 max_multipart_concurrency: 16
} }
} }
}
``` ```
AWS's S3 access parameters can be specified by referencing the standard environment variables if already defined. AWS's S3 access parameters can be specified by referencing the standard environment variables if already defined.
For example: For example:
``` ```
sdk {
aws { aws {
s3 { s3 {
# default, used for any bucket not specified below # default, used for any bucket not specified below
@ -87,10 +90,12 @@ aws {
region: ${AWS_DEFAULT_REGION} region: ${AWS_DEFAULT_REGION}
} }
} }
}
``` ```
ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration: ClearML also supports [MinIO](https://github.com/minio/minio) by adding this configuration:
``` ```
sdk {
aws { aws {
s3 { s3 {
# default, used for any bucket not specified below # default, used for any bucket not specified below
@ -110,6 +115,7 @@ aws {
] ]
} }
} }
}
``` ```
:::info non-AWS Endpoints :::info non-AWS Endpoints
@ -121,6 +127,7 @@ To enable TLS, pass `secure: true`.
To configure Azure blob storage specify the account name and key. To configure Azure blob storage specify the account name and key.
``` ```
sdk {
azure.storage { azure.storage {
containers: [ containers: [
{ {
@ -130,12 +137,14 @@ azure.storage {
} }
] ]
} }
}
``` ```
Azure's storage access parameters can be specified by referencing the standard environment variables if already defined. Azure's storage access parameters can be specified by referencing the standard environment variables if already defined.
For example: For example:
``` ```
sdk {
azure.storage { azure.storage {
containers: [ containers: [
{ {
@ -145,6 +154,7 @@ azure.storage {
} }
] ]
} }
}
``` ```
### Configuring Google Storage ### Configuring Google Storage
@ -154,6 +164,7 @@ It's also possible to specify credentials for a specific bucket in the `google.s
configuration provided in the `google.storage` section is applied to any bucket without a bucket-specific configuration. configuration provided in the `google.storage` section is applied to any bucket without a bucket-specific configuration.
``` ```
sdk {
google.storage { google.storage {
# Default project and credentials file # Default project and credentials file
# Will be used when no bucket configuration is found # Will be used when no bucket configuration is found
@ -170,11 +181,13 @@ google.storage {
}, },
] ]
} }
}
``` ```
GCP's storage access parameters can be specified by referencing the standard environment variables if already defined. GCP's storage access parameters can be specified by referencing the standard environment variables if already defined.
``` ```
sdk {
google.storage { google.storage {
credentials = [ credentials = [
{ {
@ -185,6 +198,7 @@ google.storage {
}, },
] ]
} }
}
``` ```
:::tip Direct Decoding :::tip Direct Decoding
@ -233,6 +247,7 @@ piece twice!
Configure cache location by modifying the [clearml.conf](../configs/clearml_conf.md) file: Configure cache location by modifying the [clearml.conf](../configs/clearml_conf.md) file:
``` ```
sdk {
storage { storage {
cache { cache {
# Defaults to <system_temp_folder>/clearml_cache # Defaults to <system_temp_folder>/clearml_cache
@ -246,6 +261,7 @@ storage {
{ url: "file://*" } # file-urls are always directly referenced { url: "file://*" } # file-urls are always directly referenced
] ]
} }
}
``` ```
### Direct Access ### Direct Access