mirror of
https://github.com/clearml/clearml-docs
synced 2025-03-20 12:08:28 +00:00
110 lines
5.7 KiB
Markdown
110 lines
5.7 KiB
Markdown
---
|
|
title: Transformers
|
|
---
|
|
|
|
HuggingFace's [Transformers](https://huggingface.co/docs/transformers/index) is a popular deep learning framework. You can
|
|
seamlessly integrate ClearML into your Transformer's PyTorch [Trainer](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/trainer)
|
|
code using the built-in [`ClearMLCallback`](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/callback#transformers.integrations.ClearMLCallback).
|
|
ClearML automatically logs Transformer's models, parameters, scalars, and more.
|
|
|
|
All you have to do is install and set up ClearML:
|
|
|
|
1. Install the `clearml` Python package:
|
|
|
|
```commandline
|
|
pip install clearml
|
|
```
|
|
|
|
1. To keep track of your tasks and/or data, ClearML needs to communicate to a server. You have 2 server options:
|
|
* Sign up for free to the [ClearML Hosted Service](https://app.clear.ml/)
|
|
* Set up your own server, see [here](../deploying_clearml/clearml_server.md).
|
|
1. Connect the ClearML SDK to the server by creating credentials (go to the top right in the UI to **Settings > Workspace > Create new credentials**),
|
|
then execute the command below and follow the instructions:
|
|
|
|
```commandline
|
|
clearml-init
|
|
```
|
|
|
|
That's it! In every training run from now on, the ClearML task
|
|
manager will capture:
|
|
* Source code and uncommitted changes
|
|
* Hyperparameters - PyTorch trainer [parameters](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/trainer#transformers.TrainingArguments)
|
|
and TensorFlow definitions
|
|
* Installed packages
|
|
* Model files (make sure the `CLEARML_LOG_MODEL` environment variable is set to `True`)
|
|
* Scalars (loss, learning rates)
|
|
* Console output
|
|
* General details such as machine details, runtime, creation date etc.
|
|
* And more
|
|
|
|
All of this is captured into a [ClearML Task](../fundamentals/task.md). By default, a task called `Trainer` is created
|
|
in the `HuggingFace Transformers` project. To change the task's name or project, use the `CLEARML_PROJECT` and `CLEARML_TASK`
|
|
environment variables
|
|
|
|
:::tip project names
|
|
ClearML uses `/` as a delimiter for subprojects: using `example/sample` as a name will create the `sample`
|
|
task within the `example` project.
|
|
:::
|
|
|
|
To log the models created during training, set the `CLEARML_LOG_MODEL` environment variable to `True`.
|
|
|
|
You can see all the captured data in the task's page of the ClearML [WebApp](../webapp/webapp_exp_track_visual.md).
|
|
|
|
|
|
|
|
|
|
Additionally, you can view all of your Transformers runs tracked by ClearML in the [Task Table](../webapp/webapp_model_table.md).
|
|
Add custom columns to the table, such as mAP values, so you can easily sort and see what is the best performing model.
|
|
You can also select multiple tasks and directly [compare](../webapp/webapp_exp_comparing.md) them.
|
|
|
|
See an example of Transformers and ClearML in action [here](../guides/frameworks/huggingface/transformers.md).
|
|
|
|
## Remote Execution
|
|
ClearML logs all the information required to reproduce a task run on a different machine (installed packages,
|
|
uncommitted changes etc.). The [ClearML Agent](../clearml_agent.md) listens to designated queues and when a task is
|
|
enqueued, the agent pulls it, recreates its execution environment, and runs it, reporting its scalars, plots, etc. to the
|
|
task manager.
|
|
|
|
Deploy a ClearML Agent onto any machine (e.g. a cloud VM, a local GPU machine, your own laptop) by simply running
|
|
the following command on it:
|
|
|
|
```commandline
|
|
clearml-agent daemon --queue <queues_to_listen_to> [--docker]
|
|
```
|
|
|
|
Use the ClearML [Autoscalers](../cloud_autoscaling/autoscaling_overview.md) to help you manage cloud workloads in the
|
|
cloud of your choice (AWS, GCP, Azure) and automatically deploy ClearML agents: the autoscaler automatically spins up
|
|
and shuts down instances as needed, according to a resource budget that you set.
|
|
|
|
|
|
### Reproducing Task Runs
|
|
|
|
|
|
|
|
|
|
Use ClearML's web interface to reproduce task runs and edit their details, like hyperparameters or input models, then execute the tasks
|
|
with the new configuration on a remote machine.
|
|
|
|
When ClearML is integrated into a script, it captures and stores configurations, such as hyperparameters
|
|
and model settings. When executing a task, the ClearML Agent will, by default, override runtime configuration values
|
|
(such as hyperparameters and environment variables) with the values specified in the task.
|
|
|
|
However, for tasks using Transformers, the default behavior is different. By default, Transformers tasks ignore UI
|
|
overrides and use execution-time parameters (such as environment variables). This is done to prevent potential issues
|
|
with environment-specific settings when running tasks on different machines.
|
|
|
|
**To rerun a task with modified configuration:**
|
|
1. Clone the task
|
|
1. Edit the hyperparameters and/or other details.
|
|
1. In the **CONFIGURATION > HYPERPARAMETERS > Transformers** section, set both `_ignore_hparams_ui_overrides_` and `_ignore_model_config_ui_overrides_`
|
|
to `False` . This allows the task to use the new hyperparameter and model
|
|
configuration values respectively during execution.
|
|
1. Enqueue the task
|
|
|
|
The ClearML Agent executing the task will use the new values to [override any hard coded values](../clearml_agent.md).
|
|
|
|
## Hyperparameter Optimization
|
|
Use ClearML's [`HyperParameterOptimizer`](../references/sdk/hpo_optimization_hyperparameteroptimizer.md) class to find
|
|
the hyperparameter values that yield the best performing models. See [Hyperparameter Optimization](../getting_started/hpo.md)
|
|
for more information.
|