ClearML/clearml-serving
1
0
Fork 0
mirror of https://github.com/clearml/clearml-serving synced 2025-02-12 07:25:04 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Allegro AI 2022-03-06 01:39:59 +02:00 committed by GitHub
parent 0fb51c42d1
commit 59c8d343ea
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 18 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
README.md
View File

@ -5,7 +5,7 @@
**ClearML Serving - Model deployment made easy** **ClearML Serving - Model deployment made easy**
## **`clearml-serving` </br> Model Serving (ML/DL) Orchestration and Repository Made Easy** ## **`clearml-serving` </br> :sparkles: Model Serving (ML/DL) Made Easy :tada:**
[![GitHub license](https://img.shields.io/github/license/allegroai/clearml-serving.svg)](https://img.shields.io/github/license/allegroai/clearml-serving.svg) [![GitHub license](https://img.shields.io/github/license/allegroai/clearml-serving.svg)](https://img.shields.io/github/license/allegroai/clearml-serving.svg)
@ -26,17 +26,17 @@ Features:
* Support Machine Learning Models (Scikit Learn, XGBoost, LightGBM) * Support Machine Learning Models (Scikit Learn, XGBoost, LightGBM)
* Support Deep Learning Models (Tensorflow, PyTorch, ONNX) * Support Deep Learning Models (Tensorflow, PyTorch, ONNX)
* Customizable RestAPI for serving (i.e. allow per model pre/post-processing for easy integration) * Customizable RestAPI for serving (i.e. allow per model pre/post-processing for easy integration)
* Flexibility * Flexible
* On-line model deployment * On-line model deployment
* On-line endpoint model/version deployment (i.e. no need to take the service down) * On-line endpoint model/version deployment (i.e. no need to take the service down)
* Per model standalone preprocessing and postprocessing python code * Per model standalone preprocessing and postprocessing python code
* Scalability * Scalable
* Multi model per container * Multi model per container
* Multi models per serving service * Multi models per serving service
* Multi-service support (fully seperated multiple serving service running independently) * Multi-service support (fully seperated multiple serving service running independently)
* Multi cluster support * Multi cluster support
* Out-of-the-box node auto-scaling based on load/usage * Out-of-the-box node auto-scaling based on load/usage
* Efficiency * Efficient
* multi-container resource utilization * multi-container resource utilization
* Support for CPU & GPU nodes * Support for CPU & GPU nodes
* Auto-batching for DL models * Auto-batching for DL models
@ -55,11 +55,7 @@ Features:
### ClearML Serving Design Principles ### ClearML Serving Design Principles
* Modular **Modular** , **Scalable** , **Flexible** , **Customizable** , **Open Source**
* Scalable
* Flexible
* Customizable
* Open Source
<a href="https://excalidraw.com/#json=v0ip945hun2SnO4HVLe0h,QKHfB04TFQLds3_4aqeBjQ"><img src="https://github.com/allegroai/clearml-serving/blob/dev/docs/design_diagram.png?raw=true" width="100%"></a> <a href="https://excalidraw.com/#json=v0ip945hun2SnO4HVLe0h,QKHfB04TFQLds3_4aqeBjQ"><img src="https://github.com/allegroai/clearml-serving/blob/dev/docs/design_diagram.png?raw=true" width="100%"></a>
@ -67,13 +63,19 @@ Features:
### Concepts ### Concepts
CLI - Secure configuration interface for on-line model upgrade/deployment on running Serving Services **CLI** - Secure configuration interface for on-line model upgrade/deployment on running Serving Services
Serving Service Task - Control plane object storing configuration on all the endpoints. Support multiple separated instance, deployed on multiple clusters.
Inference Services - Inference containers, performing model serving pre/post processing. Also support CPU model inferencing. **Serving Service Task** - Control plane object storing configuration on all the endpoints. Support multiple separated instance, deployed on multiple clusters.
Serving Engine Services - Inference engine containers (e.g. Nvidia Triton, TorchServe etc.) used by the Inference Services for heavier model inference.
Statistics Service - Single instance per Serving Service collecting and broadcasting model serving & performance statistics **Inference Services** - Inference containers, performing model serving pre/post processing. Also support CPU model inferencing.
Time-series DB - Statistics collection service used by the Statistics Service, e.g. Prometheus
Dashboard Service - Customizable dashboard-ing solution on top of the collected statistics, e.g. Grafana **Serving Engine Services** - Inference engine containers (e.g. Nvidia Triton, TorchServe etc.) used by the Inference Services for heavier model inference.
**Statistics Service** - Single instance per Serving Service collecting and broadcasting model serving & performance statistics
**Time-series DB** - Statistics collection service used by the Statistics Service, e.g. Prometheus
**Dashboards** - Customizable dashboard-ing solution on top of the collected statistics, e.g. Grafana
### prerequisites ### prerequisites