ClearML/clearml-serving
1
0
Fork 0
mirror of https://github.com/clearml/clearml-serving synced 2025-06-26 18:16:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

update readme and fix docker-compose-gpu.yml

Browse Source
This commit is contained in:
IlyaMescheryakov1402 2025-03-20 02:26:54 +03:00
parent db3d4539ee
commit 34c4a9df91
2 changed files with 4 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docker/docker-compose-gpu.yml
View File

@ -75,7 +75,7 @@ services:
clearml-serving-inference:
image: clearml-serving-inference:latest
image: allegroai/clearml-serving-inference:latest
container_name: clearml-serving-inference
restart: unless-stopped
# optimize perforamnce

9
examples/vllm/readme.md
View File

@ -11,12 +11,7 @@
clearml-serving --id <service_id> model add --model-id <model_id> --engine vllm --endpoint "test_vllm" --preprocess "examples/vllm/preprocess.py"
```
4. If you already have the `clearml-serving` docker-compose running, it might take it a minute or two to sync with the new endpoint.
Or you can run the clearml-serving container independently:
```
docker run -v ~/clearml.conf:/root/clearml.conf -p 8080:8080 -e CLEARML_SERVING_TASK_ID=<service_id> clearml-serving-inference:latest
```
4. If you already have the `clearml-serving` docker-compose running, it might take it a minute or two to sync with the new endpoint. To run docker-compose, see [docker-compose instructions](/README.md#nail_care-initial-setup), p. 8 (and use [docker-compose-gpu.yml](/docker/docker-compose-gpu.yml) file for vllm on gpu and [docker-compose.yml](/docker/docker-compose.yml) otherwise)
5. Test new endpoint (do notice the first call will trigger the model pulling, so it might take longer, from here on, it's all in memory):
@ -32,6 +27,8 @@
see [test_openai_api.py](test_openai_api.py) for more information.
6. Check metrics using grafana (You have to select Prometheus as data source, all of vLLM metrics have "vllm:" prefix). For more information, see [Model monitoring and performance metrics](/README.md#bar_chart-model-monitoring-and-performance-metrics-bell)
NOTE!
If you want to use send_request method, keep in mind that you have to pass "completions" or "chat/completions" in entrypoint (and pass model as a part of "data" parameter) and use it for non-streaming models: