From 21954b31bbd952c440b65d2920886a228dfdc698 Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Sun, 2 Feb 2025 13:28:26 +0200 Subject: [PATCH 1/2] Add notes about port specification for non-AWS S3 endpoints (#1025) --- docs/clearml_data/clearml_data_cli.md | 2 +- docs/clearml_data/clearml_data_sdk.md | 4 ++-- docs/clearml_sdk/task_sdk.md | 2 +- docs/clearml_serving/clearml_serving_cli.md | 2 +- docs/clearml_serving/clearml_serving_tutorial.md | 11 +++++++++-- docs/faq.md | 2 +- docs/hyperdatasets/single_frames.md | 2 +- docs/integrations/storage.md | 12 ++++++++++++ docs/webapp/webapp_exp_tuning.md | 2 +- 9 files changed, 29 insertions(+), 10 deletions(-) diff --git a/docs/clearml_data/clearml_data_cli.md b/docs/clearml_data/clearml_data_cli.md index bc205cf8..61625588 100644 --- a/docs/clearml_data/clearml_data_cli.md +++ b/docs/clearml_data/clearml_data_cli.md @@ -107,7 +107,7 @@ Upload the local dataset changes to the server. By default, it's uploaded to the medium by entering an upload destination. For example: * A shared folder: `/mnt/shared/folder` * S3: `s3://bucket/folder` -* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket` +* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket`. **Note that port specification is required**. * Google Cloud Storage: `gs://bucket-name/folder` * Azure Storage: `azure://.blob.core.windows.net/path/to/file` diff --git a/docs/clearml_data/clearml_data_sdk.md b/docs/clearml_data/clearml_data_sdk.md index 9216cb17..a2a869fc 100644 --- a/docs/clearml_data/clearml_data_sdk.md +++ b/docs/clearml_data/clearml_data_sdk.md @@ -65,7 +65,7 @@ Use the `output_uri` parameter to specify a network storage target to upload the (such as previews) to. For example: * A shared folder: `/mnt/share/folder` * S3: `s3://bucket/folder` -* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket` +* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket`. **Note that port specification is required**. * Google Cloud Storage: `gs://bucket-name/folder` * Azure Storage: `azure://.blob.core.windows.net/path/to/file` @@ -264,7 +264,7 @@ To upload the dataset files to network storage, use [`Dataset.upload()`](../refe Use the `output_url` parameter to specify storage target, such as S3 / GS / Azure. For example: * A shared folder: `/mnt/share/folder` * S3: `s3://bucket/folder` -* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket` +* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket`. **Note that port specification is required**. * Google Cloud Storage: `gs://bucket-name/folder` * Azure Storage: `azure://.blob.core.windows.net/path/to/file` diff --git a/docs/clearml_sdk/task_sdk.md b/docs/clearml_sdk/task_sdk.md index a26763f1..34c4950d 100644 --- a/docs/clearml_sdk/task_sdk.md +++ b/docs/clearml_sdk/task_sdk.md @@ -800,7 +800,7 @@ task = Task.init( Specify the model storage URI location using the relevant format: * A shared folder: `/mnt/share/folder` * S3: `s3://bucket/folder` -* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket` +* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket`. **Note that port specification is required**. * Google Cloud Storage: `gs://bucket-name/folder` * Azure Storage: `azure://.blob.core.windows.net/path/to/file` ::: diff --git a/docs/clearml_serving/clearml_serving_cli.md b/docs/clearml_serving/clearml_serving_cli.md index be377690..50050f40 100644 --- a/docs/clearml_serving/clearml_serving_cli.md +++ b/docs/clearml_serving/clearml_serving_cli.md @@ -202,7 +202,7 @@ clearml-serving model upload [-h] --name NAME [--tags TAGS [TAGS ...]] --project |`--publish`| Publish the newly created model (change model state to "published" (i.e. locked and ready to deploy)|No| |`--path`|Specify a model file/folder to be uploaded and registered| No| |`--url`| Specify an already uploaded model url (e.g. `s3://bucket/model.bin`, `gs://bucket/model.bin`)|No| -|`--destination`|Specify the target destination for the model to be uploaded. For example: `s3://bucket/folder/`, `s3://host_addr:port/bucket` (for non-AWS S3-like services like MinIO), `gs://bucket-name/folder`, `azure://.blob.core.windows.net/path/to/file`|No| +|`--destination`|Specify the target destination for the model to be uploaded. For example:
  • S3: `s3://bucket/folder`
  • Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket`. **Note that port specification is required**.
  • Google Cloud Storage: `gs://bucket-name/folder`
  • Azure Storage: `azure://.blob.core.windows.net/path/to/file`
|No| diff --git a/docs/clearml_serving/clearml_serving_tutorial.md b/docs/clearml_serving/clearml_serving_tutorial.md index d6e383d6..203f2ee5 100644 --- a/docs/clearml_serving/clearml_serving_tutorial.md +++ b/docs/clearml_serving/clearml_serving_tutorial.md @@ -107,8 +107,15 @@ or with the `clearml-serving` CLI. ``` :::info Model Storage -You can also provide a different storage destination for the model, such as S3/GS/Azure, by passing -`--destination="s3://bucket/folder"`, `s3://host_addr:port/bucket` (for non-AWS S3-like services like MinIO), `gs://bucket/folder`, `azure://.blob.core.windows.net/path/to/file`. There is no need to provide a unique +You can also provide a different storage destination for the model, such as S3/GS/Azure, by using +`--destination`. For example: + +* S3: `s3://bucket/folder` +* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket`. **Note that port specification is required**. +* Google Cloud Storage: `gs://bucket-name/folder` +* Azure Storage: `azure://.blob.core.windows.net/path/to/file` + +There is no need to provide a unique path to the destination argument, the location of the model will be a unique path based on the serving service ID and the model name. ::: diff --git a/docs/faq.md b/docs/faq.md index fcad943c..64861974 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -682,7 +682,7 @@ the experiment's ID. If the experiment's ID is `6ea4f0b56d994320a713aeaf13a86d9d ClearML supports other storage types for `output_uri`: * S3: `s3://bucket/folder` -* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket` +* Non-AWS S3-like services (such as MinIO): `s3://host_addr:port/bucket`. **Note that port specification is required**. * Google Cloud Storage: `gs://bucket-name/folder` * Azure Storage: `azure://.blob.core.windows.net/path/to/file` diff --git a/docs/hyperdatasets/single_frames.md b/docs/hyperdatasets/single_frames.md index 42ecb4ee..45915ca3 100644 --- a/docs/hyperdatasets/single_frames.md +++ b/docs/hyperdatasets/single_frames.md @@ -222,7 +222,7 @@ frame = SingleFrame( :::tip Previewing Frames in non-AWS S3-like services For the ClearML UI to be able to show frames stored in non-AWS S3-like services (e.g. MinIO), make sure the `preview_uri` link -uses the `s3://` prefix and explicitly specifies the port number in the URL (e.g. `s3://my_address.com:80/bucket/my_image.png`). +uses the `s3://` prefix and **explicitly specifies the port number** in the URL (e.g. `s3://my_address.com:80/bucket/my_image.png`). Additionally, make sure to provide cloud storage access in the WebApp [**Settings > Configuration > Web App Cloud Access**](../webapp/settings/webapp_settings_profile.md#browser-cloud-storage-access). Input `:` in the **Host** field. diff --git a/docs/integrations/storage.md b/docs/integrations/storage.md index e7d44c81..43fb9d11 100644 --- a/docs/integrations/storage.md +++ b/docs/integrations/storage.md @@ -124,6 +124,18 @@ sdk { To force usage of a non-AWS endpoint, port declaration is *always* needed (e.g. `host: "my-minio-host:9000"`), even for standard ports like `433` for HTTPS (e.g. `host: "my-minio-host:433"`). +:::important +Port specification is mandatory whenever you specify non-AWS S3 endpoint access. Use the following URI +format: `s3://://path`. + +This applies when: +* Setting output URIs for tasks (via SDK or UI) +* Registering Hyper-Dataset frames +* All fields where endpoint access is specified. +::: + + +##### TLS To enable TLS, pass `secure: true`. For example: ``` sdk { diff --git a/docs/webapp/webapp_exp_tuning.md b/docs/webapp/webapp_exp_tuning.md index a9837911..520d1113 100644 --- a/docs/webapp/webapp_exp_tuning.md +++ b/docs/webapp/webapp_exp_tuning.md @@ -97,7 +97,7 @@ and formats for specifying locations include: * A shared folder: `/mnt/share/folder` * S3: `s3://bucket/folder` -* Non-AWS S3-like services (e.g. MinIO): `s3://host_addr:port/bucket` +* Non-AWS S3-like services (e.g. MinIO): `s3://host_addr:port/bucket`. **Note that port specification is required**. * Google Cloud Storage: `gs://bucket-name/folder` * Azure Storage: `azure://.blob.core.windows.net/path/to/file` From a8f1ebce1d536bea36571a72c4c8ce0ceafd76ed Mon Sep 17 00:00:00 2001 From: pollfly <75068813+pollfly@users.noreply.github.com> Date: Sun, 2 Feb 2025 14:01:53 +0200 Subject: [PATCH 2/2] Update AWS autoscaler policy (#1024) --- docs/webapp/applications/apps_aws_autoscaler.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/webapp/applications/apps_aws_autoscaler.md b/docs/webapp/applications/apps_aws_autoscaler.md index 59bbaeb9..ada66331 100644 --- a/docs/webapp/applications/apps_aws_autoscaler.md +++ b/docs/webapp/applications/apps_aws_autoscaler.md @@ -320,6 +320,16 @@ to an IAM user, and create credentials keys for that user to configure in the au "ssm:GetParameter" ], "Resource": "arn:aws:ssm:*::parameter/aws/service/marketplace/*" + }, + { + "Sid": "AllowUsingDeeplearningAMIAliases", + "Effect": "Allow", + "Action": [ + "ssm:GetParametersByPath", + "ssm:GetParameters", + "ssm:GetParameter" + ], + "Resource": "arn:aws:ssm:*::parameter/aws/service/deeplearning/*" } ] }