clearml-docs/docs/clearml_agent/clearml_agent_env_caching.md

33 lines
1.3 KiB
Markdown
Raw Normal View History

2024-07-15 12:53:41 +00:00
---
title: Environment Caching
---
2025-02-06 15:31:11 +00:00
ClearML Agent caches virtual environments so when running tasks multiple times, there's no need to spend time reinstalling
2024-07-15 12:53:41 +00:00
pre-installed packages. To make use of the cached virtual environments, enable the virtual environment reuse mechanism.
## Virtual Environment Reuse
2025-02-06 15:31:11 +00:00
The virtual environment reuse feature may reduce task startup time dramatically.
2024-07-15 12:53:41 +00:00
By default, ClearML uses the package manager's environment caching. This means that even if no
new packages need to be installed, checking the list of packages can take a long time.
ClearML has a virtual environment reuse mechanism which, when enabled, allows using environments as-is without resolving
2025-02-06 15:31:11 +00:00
installed packages. This means that when executing multiple tasks with the same package dependencies,
2024-07-15 12:53:41 +00:00
the same environment will be used.
:::note
ClearML does not support environment reuse when using Poetry package manager
:::
2024-07-24 07:49:10 +00:00
To enable virtual environment reuse, modify the `clearml.conf` file and uncomment the `venvs_cache` section.
2024-07-15 12:53:41 +00:00
```
venvs_cache: {
# maximum number of cached venvs
max_entries: 10
# minimum required free space to allow for cache entry, disable by passing 0 or negative value
free_space_threshold_gb: 2.0
2024-07-21 12:19:32 +00:00
# uncomment to enable virtual environment caching
2024-07-15 12:53:41 +00:00
# path: ~/.clearml/venvs-cache
},
```