diff --git a/docs/deploying_clearml/enterprise_deploy/custom_billing.md b/docs/deploying_clearml/enterprise_deploy/custom_billing.md deleted file mode 100644 index 484bff77..00000000 --- a/docs/deploying_clearml/enterprise_deploy/custom_billing.md +++ /dev/null @@ -1,122 +0,0 @@ ---- -title: Custom Billing Events ---- - -:::important Enterprise Feature -Sending custom billing events is available under the ClearML Enterprise plan. -::: - -ClearML supports sending custom events to selected Kafka topics. Event sending is triggered by API calls and -is available only for the companies with the `custom_events` settings set. - -## Enabling Custom Events in ClearML Server - -:::important Prerequisite -**Precondition**: Customer Kafka for custom events is installed and reachable from the `apiserver`. -::: - -Set the following environment variables in the ClearML Enterprise helm chart under the `apiserver.extraEnv`: - -* Enable custom events: - - ``` - - name: CLEARML__services__custom_events__enabled - value: "true" - ``` -* Mount custom message template files into `/mnt/custom_events/templates` folder in the `apiserver` container and point - the `apiserver` into it: - - ``` - - name: CLEARML__services__custom_events__template_folder - value: "/mnt/custom_events/templates" - ``` -* Configure the Kafka host for sending events: - - ``` - - name: CLEARML__hosts__kafka__custom_events__host - value: "[]" - ``` - Configure Kafka security parameters. Below is the example for SASL plaintext security: - - ``` - - name: CLEARML__SECURE__KAFKA__CUSTOM_EVENTS__security_protocol - value: "SASL_PLAINTEXT" - - name: CLEARML__SECURE__KAFKA__CUSTOM_EVENTS__sasl_mechanism - value: "SCRAM-SHA-512" - - name: CLEARML__SECURE__KAFKA__CUSTOM_EVENTS__sasl_plain_username - value: "" - - name: CLEARML__SECURE__KAFKA__CUSTOM_EVENTS__sasl_plain_password - value: "" - ``` -* Define Kafka topics for lifecycle and inventory messages: - - ``` - - name: CLEARML__services__custom_events__channels__main__topics__service_instance_lifecycle - value: "lifecycle" - - name: CLEARML__services__custom_events__channels__main__topics__service_instance_inventory - value: "inventory" - ``` -* For the desired companies set up the custom events properties required by the event message templates: - - ``` - curl $APISERVER_URL/system.update_company_custom_events_config -H "Content-Type: application/json" -u $APISERVER_KEY:$APISERVER_SECRET -d'{ - "company": "", - "fields": { - "service_instance_id": "", - "service_instance_name": "", - "service_instance_customer_tenant_name": "", - "service_instance_customer_space_name": "", - "service_instance_customer_space_id": "", - "parameters_connection_points": ["", ""] - }}' - ``` - -## Sending Custom Events to the API Server - -:::important Prerequisite -**Precondition:** Dedicated custom-events Redis instance installed and reachable from all the custom events deployments. -::: - -Environment lifecycle events are sent directly by the `apiserver`. Other event types are emitted by the following helm charts: - -* `clearml-pods-monitor-exporter` - Monitors running pods and sends container lifecycle events (should run one per cluster with a unique identifier, a UUID is required for the installation): - - ``` - # -- Universal Unique string to identify Pods Monitor instances across worker clusters. It cannot be empty. - # Uniqueness is required across different cluster installations to preserve the reported data status. - podsMonitorUUID: "" - # Interval - checkIntervalSeconds: 60 - ``` -* `clearml-pods-inventory` - Periodically sends inventory events about running pods. - - ``` - # Cron schedule - https://crontab.guru/ - cronJob: - schedule: "@daily" - ``` -* `clearml-company-inventory` - Monitors Clearml companies and sends environment inventory events. - - ``` - # Cron schedule - https://crontab.guru/ - cronJob: - schedule: "@daily" - ``` - -For every script chart add the below configuration to enable redis access and connection to the `apiserver`: - -``` -clearml: - apiServerUrlReference: "" - apiServerKey: "" - apiServerSecret: "" -redisConnection: - host: "" - port: - password: "" -``` - -See all other available options to customize the `custom-events` charts by running: -``` -helm show readme allegroai-enterprise/ -``` \ No newline at end of file diff --git a/docs/deploying_clearml/enterprise_deploy/extra_configs/custom_billing.md b/docs/deploying_clearml/enterprise_deploy/extra_configs/custom_billing.md new file mode 100644 index 00000000..a1eb82b8 --- /dev/null +++ b/docs/deploying_clearml/enterprise_deploy/extra_configs/custom_billing.md @@ -0,0 +1,122 @@ +--- +title: Custom Events +--- + +:::important Enterprise Feature +Sending custom events is available under the ClearML Enterprise plan. +::: + +ClearML supports sending custom events to selected Kafka topics. Event sending is triggered by API calls and +is available only to tenants with the `custom_events` feature configured. + +## Enabling Custom Events in ClearML Server + +:::important Prerequisite +**Precondition**: Customer Kafka for custom events is installed and reachable from the `apiserver`. +::: + +Modify the server's `clearml-values.override.yaml` file, then upgrade the chart: + +```yaml +apiserver: + extraEnvs: + # Enable custom events. + - name: CLEARML__services__custom_events__enabled + value: "true" + # Folder where the ClearML Apiserver container can find the custom message templates. + - name: CLEARML__services__custom_events__template_folder + value: "/mnt/custom_events/templates" + # Kafka host configuration for custom events. + - name: CLEARML__hosts__kafka__custom_events__host + value: "[:]" + # Kafka security parameters. Below is the example for SASL plaintext security. + - name: CLEARML__SECURE__KAFKA__CUSTOM_EVENTS__security_protocol + value: "SASL_PLAINTEXT" + - name: CLEARML__SECURE__KAFKA__CUSTOM_EVENTS__sasl_mechanism + value: "SCRAM-SHA-512" + - name: CLEARML__SECURE__KAFKA__CUSTOM_EVENTS__sasl_plain_username + value: "" + - name: CLEARML__SECURE__KAFKA__CUSTOM_EVENTS__sasl_plain_password + value: "" + # Kafka topics names for lifecycle and inventory messages. + - name: CLEARML__services__custom_events__channels__main__topics__service_instance_lifecycle + value: "lifecycle" + - name: CLEARML__services__custom_events__channels__main__topics__service_instance_inventory + value: "inventory" +``` + +### Configure Per-Tenant Custom Event Fields + +For each tenant, configure the values required by the message templates using the following API call: + +```bash +curl $APISERVER_URL/system.update_company_custom_events_config -H "Content-Type: application/json" -u $APISERVER_KEY:$APISERVER_SECRET -d'{ + "company": "", + "fields": { + "service_instance_id": "", + "service_instance_name": "", + "service_instance_customer_tenant_name": "", + "service_instance_customer_space_name": "", + "service_instance_customer_space_id": "", + "parameters_connection_points": ["", ""] +}}' +``` + + +## Sending Custom Events to the API Server + +:::important Prerequisite +**Precondition:** Dedicated custom-events Redis instance installed and reachable from all the custom events deployments. +::: + +Environment lifecycle events are sent directly by the ClearML `apiserver`. Other event types are emitted by the following +services (Helm charts in Kubernetes): + + +* `clearml-pods-monitor-exporter` - Monitors running pods (tasks) and sends lifecycle events. Should run one per cluster + with a unique identifier (a UUID is required for the installation): + +* `clearml-pods-inventory` - Periodically sends inventory events about running pods. + +* `clearml-company-inventory` - Monitors ClearML companies and sends environment inventory events. + +### Install custom-events + +The `clearml-custom-events` is the Kubernetes "Custom Events" umbrella Helm chart for ClearML. +It bundles all necessary services and sets up a dedicated Redis instance. + +#### Prepare Values + +Create a `clearml-custom-events-values.override.yaml` file with the following content (make sure to replace the ``): + +```yaml +global: + imageCredentials: + password: "" + clearml: + apiServerUrlReference: "" + apiServerKey: "" + apiServerSecret: "" + +clearml-pods-monitor-exporter: + checkIntervalSeconds: 60 + # -- *REQUIRED* Universal Unique string to identify Pods Monitor instances across worker clusters. Cannot be empty. + # Uniqueness is required across different clusters installations to preserve the reported data status. + podsMonitorUUID: "" + +clearml-pods-inventory: + cronJob: + schedule: "@daily" + +clearml-company-inventory: + cronJob: + schedule: "@daily" +``` + +#### Install + +Install the Custom Events umbrella chart: + +```bash +helm install clearml-custom-events clearml-enterprise/clearml-custom-events -f clearml-custom-events-values.override.yaml +``` \ No newline at end of file diff --git a/sidebars.js b/sidebars.js index f0684e8c..726f5049 100644 --- a/sidebars.js +++ b/sidebars.js @@ -668,7 +668,6 @@ module.exports = { 'deploying_clearml/enterprise_deploy/appgw_install_k8s', ] }, - 'deploying_clearml/enterprise_deploy/custom_billing', {'UI Applications': [ 'deploying_clearml/enterprise_deploy/app_install_ubuntu_on_prem', 'deploying_clearml/enterprise_deploy/app_install_ex_server',