"[](https://colab.research.google.com/github/allegroai/clearml/blob/master/examples/frameworks/huggingface/transformers.ipynb)\n",
"\n",
"HuggingFace's [Transformers](https://huggingface.co/docs/transformers/index) is a popular deep learning framework. You can seamlessly integrate ClearML into your Transformer's PyTorch Trainer code using the built-in [ClearMLCallback](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/callback#transformers.integrations.ClearMLCallback). ClearML automatically logs Transformer's models, parameters, scalars, and more."
],
"metadata": {
"id": "jF2e1XVCazr3"
}
},
{
"cell_type": "markdown",
"source": [
"## Set up ClearML\n",
"1. To keep track of your experiments and/or data, ClearML needs to communicate to a server. You have 2 server options:\n",
" * Sign up for free to the [ClearML Hosted Service](https://app.clear.ml/)\n",
" * Set up your own server, see [here](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server).\n",
"1. Add your ClearML credentials below. ClearML will use these credentials to connect to your server (see instructions for generating credentials [here](https://clear.ml/docs/latest/docs/getting_started/ds/ds_first_steps/#jupyter-notebook)).\n"
"Create a trainer, in which you'll typically pass the following parameters to [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer):\n",
"1. A [PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel) or a [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module):\n",
"2. [TrainingArguments](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments) contains the model hyperparameters you can change like learning rate, batch size, and the number of epochs to train for. The default values are used if you don't specify any training arguments.\n",
"ClearML will capture all the training arguments.\n",
"\n"
],
"metadata": {
"id": "Va3iOGxa_Vra"
}
},
{
"cell_type": "code",
"source": [
"from transformers import TrainingArguments\n",
"\n",
"training_args = TrainingArguments(\n",
" output_dir=\"path/to/save/folder/\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=8,\n",
" per_device_eval_batch_size=8,\n",
" num_train_epochs=2,\n",
")\n"
],
"metadata": {
"id": "LjSvf25e_XuL"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"3. A preprocessing class like a tokenizer, image processor, feature extractor, or processor:\n",
"5. Create a function to tokenize the dataset, and then apply over the entire dataset with [map](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.map)::\n",
"6. A [DataCollatorWithPadding](https://huggingface.co/docs/transformers/main/en/main_classes/data_collator#transformers.DataCollatorWithPadding) to create a batch of examples from your dataset:\n"
"Now gather all these classes in [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer):"
],
"metadata": {
"id": "Q3WimRh3AOQ6"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "RMTBiL159_Tn"
},
"outputs": [],
"source": [
"from transformers import Trainer\n",
"\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=dataset[\"train\"],\n",
" eval_dataset=dataset[\"test\"],\n",
" tokenizer=tokenizer,\n",
" data_collator=data_collator,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1rc5xo-F9_Tn"
},
"source": [
"## Start Training\n",
"When you're ready, call [train()](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.train) to start training:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jO6FkQM_9_To"
},
"outputs": [],
"source": [
"trainer.train()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QdLmkuBF9_To"
},
"source": [
"Since `clearml` is installed, the trainer will use the [ClearMLCallback](https://huggingface.co/docs/transformers/main/en/main_classes/callback#transformers.integrations.ClearMLCallback) so a ClearML task is created, which captures your experiment's models, scalars, images, and more.\n",
"\n",
"By default, a task called `Trainer` is created in the `HuggingFace Transformers` project. To change the task’s name or project, use the `CLEARML_PROJECT` and `CLEARML_TASK` environment variables.\n",
"\n",
"The console output displays the task ID and a link to the task's page in the [ClearML web UI](https://clear.ml/docs/latest/docs/webapp/webapp_exp_track_visual). In the UI, you can visualize all the information captured by the task.\n"
