diff --git a/docs/clearml_data/clearml_data_cli.md b/docs/clearml_data/clearml_data_cli.md index f9a6ddfa..ed7aad1b 100644 --- a/docs/clearml_data/clearml_data_cli.md +++ b/docs/clearml_data/clearml_data_cli.md @@ -104,7 +104,12 @@ clearml-data remove [-h] [--id ID] [--files [FILES [FILES ...]]] ## upload Upload the local dataset changes to the server. By default, it's uploaded to the [ClearML Server](../deploying_clearml/clearml_server.md). You can specify a different storage -medium by entering an upload destination, such as `s3://bucket`, `gs://`, `azure://`, `/mnt/shared/`. +medium by entering an upload destination. For example: +* A shared folder: `:/mnt/shared/folder` +* S3: `s3://bucket/folder` +* Non-AWS S3-like services (e.g. MinIO): `s3://host_addr:port/bucket` +* Google Cloud Storage: `gs://bucket-name/folder` +* Azure Storage: `azure://.blob.core.windows.net/path/to/file` ```bash clearml-data upload [-h] [--id ID] [--storage STORAGE] [--chunk-size CHUNK_SIZE] diff --git a/docs/clearml_data/clearml_data_sdk.md b/docs/clearml_data/clearml_data_sdk.md index c174c055..f6a5da30 100644 --- a/docs/clearml_data/clearml_data_sdk.md +++ b/docs/clearml_data/clearml_data_sdk.md @@ -62,7 +62,13 @@ and auto-increments the version number. ::: Use the `output_uri` parameter to specify a network storage target to upload the dataset files, and associated information -(such as previews) to (e.g. `s3://bucket/data`, `gs://bucket/data`, `azure://.blob.core.windows.net/path/to/file`, `file:///mnt/share/data`). +(such as previews) to. For example: +* A shared folder: `/mnt/share/folder` +* S3: `s3://bucket/folder` +* Non-AWS S3-like services (e.g. MinIO): `s3://host_addr:port/bucket` +* Google Cloud Storage: `gs://bucket-name/folder` +* Azure Storage: `azure://.blob.core.windows.net/path/to/file` + By default, the dataset uploads to ClearML's file server. The `output_uri` parameter of the [`Dataset.upload`](#uploading-files) method overrides this parameter's value. @@ -248,7 +254,13 @@ dataset.get_logger().report_histogram( To upload the dataset files to network storage, use the [`Dataset.upload`](../references/sdk/dataset.md#upload) method. -Use the `output_url` parameter to specify storage target, such as S3 / GS / Azure (e.g. `s3://bucket/data`, `gs://bucket/data`, `azure://.blob.core.windows.net/path/to/file`, `/mnt/share/data`). +Use the `output_url` parameter to specify storage target, such as S3 / GS / Azure. For example: +* A shared folder: `/mnt/share/folder` +* S3: `s3://bucket/folder` +* Non-AWS S3-like services (e.g. MinIO): `s3://host_addr:port/bucket` +* Google Cloud Storage: `gs://bucket-name/folder` +* Azure Storage: `azure://.blob.core.windows.net/path/to/file` + By default, the dataset uploads to ClearML's file server. This target storage overrides the `output_uri` value of the [`Dataset.create`](#creating-datasets) method. diff --git a/docs/clearml_sdk/task_sdk.md b/docs/clearml_sdk/task_sdk.md index 09120f6c..50c277fa 100644 --- a/docs/clearml_sdk/task_sdk.md +++ b/docs/clearml_sdk/task_sdk.md @@ -782,6 +782,15 @@ task = Task.init( ) ``` +:::tip Output URI Formats +Specify the model storage URI location using the relevant format: +* A shared folder: `/mnt/share/folder` +* S3: `s3://bucket/folder` +* Non-AWS S3-like services (e.g. MinIO): `s3://host_addr:port/bucket` +* Google Cloud Storage: `gs://bucket-name/folder` +* Azure Storage: `azure://.blob.core.windows.net/path/to/file` +::: + To automatically store all models created by any experiment at a specific location, edit the `clearml.conf` (see [ClearML Configuration Reference](../configs/clearml_conf.md#sdkdevelopment)) and set `sdk.developmenmt.default_output_uri` to the desired storage (see [Storage](../integrations/storage.md)). This is especially helpful when diff --git a/docs/clearml_serving/clearml_serving_cli.md b/docs/clearml_serving/clearml_serving_cli.md index f063dcb2..40545d5f 100644 --- a/docs/clearml_serving/clearml_serving_cli.md +++ b/docs/clearml_serving/clearml_serving_cli.md @@ -202,7 +202,7 @@ clearml-serving model upload [-h] --name NAME [--tags TAGS [TAGS ...]] --project |`--publish`| Publish the newly created model (change model state to "published" (i.e. locked and ready to deploy)|Yes| |`--path`|Specify a model file/folder to be uploaded and registered| Yes| |`--url`| Specify an already uploaded model url (e.g. `s3://bucket/model.bin`, `gs://bucket/model.bin`)|Yes| -|`--destination`|Specify the target destination for the model to be uploaded (e.g. `s3://bucket/folder/`, `gs://bucket/folder/`)|Yes| +|`--destination`|Specify the target destination for the model to be uploaded. For example: `s3://bucket/folder/`, `s3://host_addr:port/bucket` (for non-AWS S3-like services like MinIO), `gs://bucket-name/folder`, `azure://.blob.core.windows.net/path/to/file`|Yes| diff --git a/docs/clearml_serving/clearml_serving_tutorial.md b/docs/clearml_serving/clearml_serving_tutorial.md index d967692d..2f7a368b 100644 --- a/docs/clearml_serving/clearml_serving_tutorial.md +++ b/docs/clearml_serving/clearml_serving_tutorial.md @@ -104,7 +104,7 @@ or with the `clearml-serving` CLI. :::info Model Storage You can also provide a different storage destination for the model, such as S3/GS/Azure, by passing -`--destination="s3://bucket/folder"`, `gs://bucket/folder`, `azure://.blob.core.windows.net/path/to/file`. There is no need to provide a unique +`--destination="s3://bucket/folder"`, `s3://host_addr:port/bucket` (for non-AWS S3-like services like MinIO), `gs://bucket/folder`, `azure://.blob.core.windows.net/path/to/file`. There is no need to provide a unique path to the destination argument, the location of the model will be a unique path based on the serving service ID and the model name ::: diff --git a/docs/faq.md b/docs/faq.md index 704dc2e0..c4e92670 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -670,7 +670,13 @@ the experiment's ID. If the experiment's ID is `6ea4f0b56d994320a713aeaf13a86d9d /mnt/shared/folder/task.6ea4f0b56d994320a713aeaf13a86d9d/models/ ``` -ClearML supports other storage types for `output_uri`, including: +ClearML supports other storage types for `output_uri`: +* S3: `s3://bucket/folder` +* Non-AWS S3-like services (e.g. MinIO): `s3://host_addr:port/bucket` +* Google Cloud Storage: `gs://bucket-name/folder` +* Azure Storage: `azure://.blob.core.windows.net/path/to/file` + +For example: ```python # AWS S3 bucket task = Task.init(project_name, task_name, output_uri="s3://bucket-name/folder") diff --git a/docs/webapp/webapp_exp_tuning.md b/docs/webapp/webapp_exp_tuning.md index b897f16f..5ce02cac 100644 --- a/docs/webapp/webapp_exp_tuning.md +++ b/docs/webapp/webapp_exp_tuning.md @@ -86,6 +86,7 @@ and formats for specifying locations include: * A shared folder: `/mnt/share/folder` * S3: `s3://bucket/folder` +* Non-AWS S3-like services (e.g. MinIO): `s3://host_addr:port/bucket` * Google Cloud Storage: `gs://bucket-name/folder` * Azure Storage: `azure://.blob.core.windows.net/path/to/file`