mirror of
https://github.com/clearml/clearml
synced 2025-02-12 07:35:08 +00:00
696 lines
130 KiB
Plaintext
696 lines
130 KiB
Plaintext
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"attachments": {},
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "fauyMKCEJ6x4"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"<div align=\"center\">\n",
|
||
|
|
"\n",
|
||
|
|
" <a href=\"https://clear.ml\" target=\"_blank\">\n",
|
||
|
|
" <img width=\"512\", src=\"https://github.com/allegroai/clearml/raw/master/docs/clearml-logo.svg\"></a>\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"<br>\n",
|
||
|
|
"\n",
|
||
|
|
"<h1>Notebook 1: Experiment Management</h1>\n",
|
||
|
|
"\n",
|
||
|
|
"<br>\n",
|
||
|
|
"\n",
|
||
|
|
"Hi there! This is the ClearML getting started notebook, meant to teach you the ropes. ClearML has a lot of modules that you can use, so in this notebook, we'll start with the most well-known one: <a href=\"https://app.clear.ml/projects\" target=\"_blank\">Experiment Management.</a>\n",
|
||
|
|
"\n",
|
||
|
|
"You can find out more details about the other ClearML modules and the technical specifics of each in <a href=\"https://clear.ml/docs\" target=\"_blank\">our documentation.</a>\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"<table>\n",
|
||
|
|
"<tbody>\n",
|
||
|
|
" <tr>\n",
|
||
|
|
" <td><b>Step 1: Experiment Management</b></td>\n",
|
||
|
|
" <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_1_Experiment_Management.ipynb\">\n",
|
||
|
|
" <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
|
||
|
|
"</a></td>\n",
|
||
|
|
" </tr>\n",
|
||
|
|
" <tr>\n",
|
||
|
|
" <td>Step 2: Remote Agent</td>\n",
|
||
|
|
" <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_2_Setting_Up_Agent.ipynb\">\n",
|
||
|
|
" <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
|
||
|
|
"</a></td>\n",
|
||
|
|
" </tr>\n",
|
||
|
|
" <tr>\n",
|
||
|
|
" <td>Step 3: Remote Task Execution</td>\n",
|
||
|
|
" <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_3_Remote_Execution.ipynb\">\n",
|
||
|
|
" <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
|
||
|
|
"</a></td>\n",
|
||
|
|
" </tr>\n",
|
||
|
|
"</tbody>\n",
|
||
|
|
"</table>\n",
|
||
|
|
"\n",
|
||
|
|
"</div>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "s8eUiQauOA31"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"# 📦 Setup\n",
|
||
|
|
"\n",
|
||
|
|
"Since we are using a notebook here, we're importing the special `browser_login` function, it will try to help you easily log in. If it doesn't work, don't worry, it will guide you through the steps to get it done :)\n",
|
||
|
|
"\n",
|
||
|
|
"**If it asks you to generate new credentials, keep them handy, you'll need them again in later notebooks**\n",
|
||
|
|
"\n",
|
||
|
|
"When installing ClearML in a normal python environment (not a colab notebook), you'll want to use `clearml-init` instead. It, too, will guide you through the setup.\n",
|
||
|
|
"\n",
|
||
|
|
"What we're doing here is connecting to a ClearML server, that will store all your experiment details."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"id": "IMZsEYw5J5JI"
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"%pip install --upgrade xgboost clearml\n",
|
||
|
|
"import clearml\n",
|
||
|
|
"clearml.browser_login()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "TLsgNR5PhPPN"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"# ✈️ Example: XGBoost\n",
|
||
|
|
"\n",
|
||
|
|
"Let's start simple, by adding the ClearML experiment tracker to an XGBoost training script.\n",
|
||
|
|
"\n",
|
||
|
|
"The important parts are:\n",
|
||
|
|
"\n",
|
||
|
|
"- Initializing ClearML. Always do this as a very first line if possible!\n",
|
||
|
|
"- Manually log the parameter dict (e.g. CLI commands are captured automatically)\n",
|
||
|
|
"\n",
|
||
|
|
"**⚠️ NOTE: `output_uri` in `Task.init` is an important parameter. By default it is set to `False`, meaning any registered models will NOT be uploaded to ClearML, but their info will be registered. Set this to `True` to automatically upload all model files.**"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"colab": {
|
||
|
|
"base_uri": "https://localhost:8080/"
|
||
|
|
},
|
||
|
|
"id": "CSaL3XTqhYAy",
|
||
|
|
"outputId": "6c870d67-d4f8-4c11-a356-0678b2fd9a41"
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"ClearML Task: created new task id=38ba982031b04187a59acbd7346c5010\n",
|
||
|
|
"ClearML results page: https://app.clear.ml/projects/df7858a441f94c29a1230d467f131840/experiments/38ba982031b04187a59acbd7346c5010/output/log\n",
|
||
|
|
"2023-03-06 16:06:09,239 - clearml.Task - INFO - Storing jupyter notebook directly as code\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"from sklearn.model_selection import train_test_split\n",
|
||
|
|
"from sklearn.datasets import load_iris\n",
|
||
|
|
"from clearml import Task\n",
|
||
|
|
"import xgboost as xgb\n",
|
||
|
|
"import numpy as np\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"# Always initialize ClearML before anything else. Automatic hooks will track as\n",
|
||
|
|
"# much as possible for you!\n",
|
||
|
|
"task = Task.init(\n",
|
||
|
|
" project_name=\"Getting Started\",\n",
|
||
|
|
" task_name=\"XGBoost Training\",\n",
|
||
|
|
" output_uri=True # IMPORTANT: setting this to True will upload the model\n",
|
||
|
|
" # If not set the local path of the model will be saved instead!\n",
|
||
|
|
")\n",
|
||
|
|
"\n",
|
||
|
|
"# Training data\n",
|
||
|
|
"X, y = load_iris(return_X_y=True)\n",
|
||
|
|
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
||
|
|
" X, y, test_size=0.2, random_state=100\n",
|
||
|
|
")\n",
|
||
|
|
"\n",
|
||
|
|
"dtrain = xgb.DMatrix(X_train, label=y_train)\n",
|
||
|
|
"dtest = xgb.DMatrix(X_test, label=y_test)\n",
|
||
|
|
"\n",
|
||
|
|
"# Setting the parameters\n",
|
||
|
|
"params = {\n",
|
||
|
|
" 'max_depth': 2,\n",
|
||
|
|
" 'eta': 1,\n",
|
||
|
|
" 'objective': 'reg:squarederror',\n",
|
||
|
|
" 'nthread': 4,\n",
|
||
|
|
" 'eval_metric': 'rmse',\n",
|
||
|
|
"}\n",
|
||
|
|
"# Make sure ClearML knows these parameters are our hyperparameters!\n",
|
||
|
|
"task.connect(params)\n",
|
||
|
|
"\n",
|
||
|
|
"# Train the model\n",
|
||
|
|
"bst = xgb.train(\n",
|
||
|
|
" params,\n",
|
||
|
|
" dtrain,\n",
|
||
|
|
" num_boost_round=100,\n",
|
||
|
|
" evals=[(dtrain, \"train\"), (dtest, \"test\")],\n",
|
||
|
|
" verbose_eval=0,\n",
|
||
|
|
")\n",
|
||
|
|
"\n",
|
||
|
|
"# Save the model, saving the model will automatically also register it to \n",
|
||
|
|
"# ClearML thanks to the automagic hooks\n",
|
||
|
|
"bst.save_model(\"best_model\")"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"id": "P150YtRbhYww"
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"# When a python script ends, the ClearML task is closed automatically. But in\n",
|
||
|
|
"# a notebook (that never ends), we need to manually close the task.\n",
|
||
|
|
"task.close()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "-jWVBTuXyRHd"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"## 🔹 XGBoost WebUI\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"After running the code above you should have a new ClearML project on your server called \"Getting Started\".\n",
|
||
|
|
"\n",
|
||
|
|
"Inside the task, a lot of things are tracked!\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"### Source information (git, installed packages, uncommitted changes, ...)\n",
|
||
|
|
"\n",
|
||
|
|
"Naturally, running this notebook doesn't actually give us any git information. Instead ClearML saves the execution order of the cells you've executed until it detected the `Task.close()` command.\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"### Configuration\n",
|
||
|
|
"\n",
|
||
|
|
"The configuration section holds all the values we added using the `task.connect()` call before. You can also use `task.set_parameter()` for a single value or `task.connect_configuration()` to connect an external configuration file.\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"### Artifacts\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"Artifacts are very flexible and can mean every type of file storage. Mostly this is used to track and save input and output models. In this case we get the saved XGBoost model. But when running a notebook you'll also get the original notebook here as well as an HTML preview!\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"### Scalars\n",
|
||
|
|
"\n",
|
||
|
|
"Scalars are the performance values of your models. They can either be a value for each epoch/iteration, which will display them as a plot, or a single one, which displays them as a table. In our example above the scalars where automatically grabbed from XGBoost using our integration!\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"### Plots and Debug Samples\n",
|
||
|
|
"\n",
|
||
|
|
"We haven't generated any plots or debug samples in our XGBoost example. We'll look at these a little later in the tutorial.\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"### Info + Console logs\n",
|
||
|
|
"\n",
|
||
|
|
"These sections speak for themselves: you get some additional information (like runtime or original machine hostname) as well as the original console logs.\n",
|
||
|
|
"\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "pkYjS0RTtJ6s"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"# 🚀 Example: Pytorch + Tensorboard + Matplotlib\n",
|
||
|
|
"\n",
|
||
|
|
"As you might have seen, our previous example was missing plots and debug samples, none were logged to ClearML! Luckily, XGBoost is not the only integration that ClearML has, it can also lift scalars, plots and debug samples from other frameworks you most likely use already.\n",
|
||
|
|
"\n",
|
||
|
|
"A full list of our integrations can be found [here](https://clear.ml/docs/latest/docs/integrations/libraries)."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "5iwK3p-X3WIC"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"## 🔹 Logging scalars through Tensorboard\n",
|
||
|
|
"\n",
|
||
|
|
"ClearML will detect this and also log scalars, images etc. to the ClearML experiment manager. So you don't even have to change your existing code!\n",
|
||
|
|
"\n",
|
||
|
|
""
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"colab": {
|
||
|
|
"base_uri": "https://localhost:8080/"
|
||
|
|
},
|
||
|
|
"id": "4aeacPunj_7o",
|
||
|
|
"outputId": "52a3afcf-bb5e-4be2-e345-5b2199a5358a"
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"ClearML Task: created new task id=0bfefb8b86ba44798d9abe34ba0a6ab4\n",
|
||
|
|
"ClearML results page: https://app.clear.ml/projects/df7858a441f94c29a1230d467f131840/experiments/0bfefb8b86ba44798d9abe34ba0a6ab4/output/log\n",
|
||
|
|
"Epoch 1 complete\n",
|
||
|
|
"Epoch 2 complete\n",
|
||
|
|
"Epoch 3 complete\n",
|
||
|
|
"Epoch 4 complete\n",
|
||
|
|
"Epoch 5 complete\n",
|
||
|
|
"Epoch 6 complete\n",
|
||
|
|
"Epoch 7 complete\n",
|
||
|
|
"Epoch 8 complete\n",
|
||
|
|
"Epoch 9 complete\n",
|
||
|
|
"Epoch 10 complete\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch\n",
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"import torch.optim as optim\n",
|
||
|
|
"from clearml import Task\n",
|
||
|
|
"from torch.utils.data import DataLoader\n",
|
||
|
|
"from torchvision.datasets import MNIST\n",
|
||
|
|
"from torchvision.transforms import ToTensor\n",
|
||
|
|
"from torch.utils.tensorboard import SummaryWriter\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"# Always initialize ClearML before anything else. Automatic hooks will track as\n",
|
||
|
|
"# much as possible for you (such as in this case TensorBoard logs)!\n",
|
||
|
|
"task = Task.init(project_name=\"Getting Started\", task_name=\"TB Logging\")\n",
|
||
|
|
"\n",
|
||
|
|
"# Set up TensorBoard logging\n",
|
||
|
|
"writer = SummaryWriter()\n",
|
||
|
|
"\n",
|
||
|
|
"# Load MNIST dataset\n",
|
||
|
|
"train_data = MNIST('data', train=True, download=True, transform=ToTensor())\n",
|
||
|
|
"train_loader = DataLoader(train_data, batch_size=64, shuffle=True)\n",
|
||
|
|
"\n",
|
||
|
|
"# Define model\n",
|
||
|
|
"model = nn.Sequential(\n",
|
||
|
|
" nn.Linear(784, 128),\n",
|
||
|
|
" nn.ReLU(),\n",
|
||
|
|
" nn.Linear(128, 10)\n",
|
||
|
|
")\n",
|
||
|
|
"\n",
|
||
|
|
"# Define loss and optimizer\n",
|
||
|
|
"criterion = nn.CrossEntropyLoss()\n",
|
||
|
|
"optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
|
||
|
|
"\n",
|
||
|
|
"# Train the model\n",
|
||
|
|
"for epoch in range(10):\n",
|
||
|
|
" for i, (inputs, labels) in enumerate(train_loader):\n",
|
||
|
|
" # Flatten input images\n",
|
||
|
|
" inputs = inputs.view(-1, 784)\n",
|
||
|
|
" \n",
|
||
|
|
" # Zero the gradients\n",
|
||
|
|
" optimizer.zero_grad()\n",
|
||
|
|
" \n",
|
||
|
|
" # Forward pass\n",
|
||
|
|
" outputs = model(inputs)\n",
|
||
|
|
" loss = criterion(outputs, labels)\n",
|
||
|
|
" \n",
|
||
|
|
" # Backward pass and update parameters\n",
|
||
|
|
" loss.backward()\n",
|
||
|
|
" optimizer.step()\n",
|
||
|
|
" \n",
|
||
|
|
" # Log loss to TensorBoard\n",
|
||
|
|
" # ClearML will detect this and also log the scalar to the ClearML\n",
|
||
|
|
" # experiment manager. So you don't even have to change your existing code!\n",
|
||
|
|
" writer.add_scalar('Training loss', loss.item(), epoch * len(train_loader) + i)\n",
|
||
|
|
" \n",
|
||
|
|
" print(f'Epoch {epoch + 1} complete')\n",
|
||
|
|
" \n",
|
||
|
|
"# Close TensorBoard writer\n",
|
||
|
|
"writer.close()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "fsHJZGcC3afb"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"## 🔹 Logging debug samples through matplotlib\n",
|
||
|
|
"\n",
|
||
|
|
"Whenever you use `plt.imshow()` ClearML will intercept the call and immediately log the image to the experiment manager. The images will become visible under the Debug Samples tab.\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"You can log basically any media type (images, videos, audio, ...) as a debug sample. Check [our docs](https://clear.ml/docs/latest/docs/references/sdk/logger#report_media) for more info."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"colab": {
|
||
|
|
"base_uri": "https://localhost:8080/",
|
||
|
|
"height": 281
|
||
|
|
},
|
||
|
|
"id": "bAEB04OUuI3R",
|
||
|
|
"outputId": "d9e8e484-3903-4212-9674-7490cbb192e3"
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAEICAYAAABf40E1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydeVzU1ff/X3c2ZlgERmQVEFAUAxU1MSlEzVxLS9FU3MXUytLSj6lZlpFLZZpaWn21MlPUTFwzN9x3BFNZBEFA9p1h9vf5/QHMDxQEZgYTnefjcR8w7+Xce9/LeZ+7ncOICCZMmHh24f3XBTBhwsR/i0kJmDDxjGNSAiZMPOOYlIAJE884JiVgwsQzjkkJmDDxjGNSAiZMPOOYlMATBmMshTGmYozZPbA9mjFGjLE2lb+3VP7uUe2Ytowxqvb7JGNsWrXfCxljdxljZYyxdMbYjsrtNyu3lTHGtIwxRbXfC2sp46eMsa3Gr72J/wKTEngyuQtgTNUPxpgfAPNajisAsKwhAhljEwGMB/AyEVkC6A7gGAAQ0XNEZFm5/TSAd6p+E1G4YVUx8aRjUgJPJr8BmFDt90QAv9Zy3C8AOjHGejdA5vMA/iaiJAAgoiwi2mRwSQFUWiSzGGOJjLFSxtjnjDEvxtg5xlgJYyyCMSaqPNaWMbafMZbLGCus/L91NVkejLFTlXKOMsbWV7c6GGM9K+UWMcZiGGPB1fZNYowlV557lzE2zhj1e9oxKYEnkwsAWjDGfBhjfABvAqjN/C4HEA7giwbKnMAYm8cY614p15gMANANQE8A8wFsAhAKwBWAL/6/ZcMDsBmAOwA3AHIA66rJ2QbgEoCWAD5FhfUCAGCMuQA4gArrRwrgQwC7GWOtGGMWANYCGEREVgB6Abhu5Do+lZiUwJNLlTXQH8BtABl1HLcRgBtjbNCjhBHRVgDvouJljQKQwxj7n/GKi5VEVEJENwH8C+AIESUTUTGAQwD8K8uRT0S7iaiciEpRocB6AwBjzA0VFssSIlIR0RkAkdXyCAVwkIgOEhFHRP8AuAJgcOV+DoAvY0xCRJmVZTFRDyYl8OTyG4CxACah9qYAAICIlAA+r0yPhIh+J6KXAdgAmAHgc8bYAKOUFsiu9r+8lt+WAMAYM2eMbWSMpTLGSgCcAmBTaZk4AyggovJq56ZV+98dQEhlU6CIMVYE4EUATkQkAzC6sl6ZjLEDjLEORqrbU41JCTyhEFEqKjoIBwP4s57DN6PixX6jgbLVRLQTQCwqTPXHyQcA2gMIIKIWAIIqtzMAmQCkjLHqnaCu1f5PA/AbEdlUSxZEtBwAiOhvIuoPwAlAHIAfm7oyTwMmJfBkMxVA38qvXJ0QkQbAJwDqNO8rO82GMMasGGO8yubDcwAuGrXE9WOFCsugiDEmRUW5AegU3xUAnzLGRIyxFwC8Wu3crQBeZYwNYIzxGWNixlgwY6w1Y8yBMTassm9ACaAMFc0DE/VgUgJPMESURERXGnj4H6j4ktZFCYCFAO4BKAKwEsDMynb34+RbABIAeajorDz8wP5xAF4AkI+KDsAdqHipQURpAIahoh65qLAM5qHiOeYBmAvgPiqGTnsDmNm0VXk6YCanIiaeZConNMUR0Sf1HmxCL0yWgIknCsbY85VzDHiMsYGo+PL/9V+X62lG8F8XwISJB3BERUdoSwDpqGiyRP+3RXq6abLmQKUWXwOAD+Cnqh5cEyZMPFk0iRKoHPNNQMVEl3QAlwGMIaJbRs/MhAkTBtFUzYEeAO4QUTIAMMa2o6JtV6sSsLOzozZt2jRRUUyYMAEAV69ezSOiVg9ubyol4IKaM73SAQRUP4AxNh3AdABwc3PDlSsNHQkzYcKEPjDGUmvb/p+NDhDRJiLqTkTdW7V6SDmZMGHiMdFUSiADNad7tkbdC2BMmDDxH9JUzYHLANoxxjxQ8fK/iYrFMCaeAYgIubm52LVrFwDA0tISnTp1gqenJ1q0aPEfl87EgzSJEiAiDWPsHQB/o2KI8P+a87LOuLg4bNmyBVKpFG+//TYsLCz+6yI9sXAch927dyMiIgK7d+8GAIjFYrRv3x4DBgzAW2+9BTc3N/D5xnZn8HRTWlqK3377DT4+PnjppZcgEBjx1SWi/zx169aNnmQiIyPJ0tKSOnbsSAUFBU2WT3FxMX3zzTc0b948un37Nmk0mibLqzZkMhkdO3aMQkJCaPXq1ZSZmdloGdeuXaMOHToQj8d7KFlaWtLcuXOpsLCwCUr/36LVakmj0eiSMVGr1fTZZ5+RjY0NvfPOO1ReXl5jf0lJCf3xxx+0ceNGUqlUdcoBcIVqef9MMwYbCGMMPF7T9aOWl5fjhx9+wJ9//gkvLy+89dZb+OKLL/DCCy802VezvLwcmZmZaNGiBWJjY7FhwwacPXsWeXl5OHLkCKKjo7FmzRrY2Ng0WKadnR0YY7rfNjY2EAqFKCwsRHl5OTZt2gQrKyssWLAAYrG4KaoFANBqtdBoNGCMQSgU1iiTsSAilJSU4NChQ9izZw9u3Lih2zdz5kyEhYUZpY4cx+H69esoKSlBdHQ0tm3bpmtWZWVlYevWrcjIyMC3336rn4VQm2Z43EkfS0Aul9OhQ4do7ty5lJSURFqtloqKiiglJYWSkpIoPz+fEhISKCkpiXJzcw3SzpGRkWRlZUW+vr5NYgloNBo6evQo9ezZk6Kjo0mhUNCePXuoc+fOlJOTY/T8lEolRUZG0qBBg0gqlZKfnx/Z2dmRQCAgPp+v+3IHBgY22hrgOI4iIiLI29ubIiMjKTMzk0pKSmjr1q3k7e1NPB6PXFxcaN26daRUKo1eN6KKL2dERAQ5OzuTk5MTrVixwuh5abVaunnzJnXo0IEkEkmN68bj8UgsFtPIkSMpNjbW4Lw4jqOkpCQaOnQoCQQCcnR0pJYtW5K5uTl1796dhg8fTteuXav3GcfTZAnIZDKsW7cOW7ZsgVgsxu+//w4zMzNoNBoolUpYW1vD19cXcrkcfn5+cHFxQceOHdG/f/8nsi2qVquxevVq+Pj4oEuXLgCA4cOHQyKRQCQSgYiM+iUTiUTw9PTE33//DcYYiouLQZUzR3k8HszMzKBQKPSSzRjDiBEj0KdPH9jZ/X+v6SNGjEB2djaWLFmCzMxMzJ49GzKZDB9++KHRLSytVovWrVsjLCwMERER+Oijj5Cfn49ly5ZBKBTqJZOIwHEcFAoFDh06hAsXLmD37t24d+9ercerVCr8+eefSExMxLlz52BuXpuz6IbBGIO1tTUyMzMRHh6OsLAwcBwHjUaDFi1awMzMzLDnozbN8LhTYyyBsrIyWrZsGXl6etLJkyfpnXfeIS8vLzp58iTdu3eP0tLSjN7mbGpLIC4ujlq2bEnbtm2rsb24uJgWLlxo1Dw5jqPi4mJat24d8Xg84vP5JBKJSCqVkkQiIUdHR/r666+Jz+fTgAEDjGqJlJSU0MqVK8nFxYX4fD65uLhQfHy80eQ/CMdxtHnzZrK1taVRo0aRVqvVS45Wq6W0tDRasmQJdenShYRCIYlEIhIKhcTj8cjKyoo8PDzIxcWFnJycyMnJiSQSCfF4PJJKpbRlyxbiOE7vemg0Gtq8eTO5u7vTP//8o7cc1GEJ/OcKgBqhBLRaLcXExJCXlxctW7aMZDIZpaamUmlpqUEXuT4ehxKws7OjP/74o8b2X3/9laysrOjevXtGyUer1dL9+/dpzJgxZGlpSXw+nxwdHWnUqFG0a9cuevPNN0koFJJAICAvLy/avHmz3i9OXRQXF9PatWvJ1taWeDwezZ0716jyH8xr5syZZGVlRRs2bCC1Wt1oGRzHUWZmJg0YMIBEIhHZ2NjQm2++SW+//TaNGjWKrK2tKSQkhKKjo+n27dsUGxtLsbGxtGDBAnJzcyMej0dDhgyh4uJiveuhVCopNDSUQkNDqaSkRG85dSmBZtUcUKvV+Pvvv2FmZoZXXnkF5ubmcHNz+6+L1WTY2dkZtfmSmZmJt956C2fOnIFWq4W/vz/CwsLwxhtvQCqVwtHREZGRk
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 432x288 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {
|
||
|
|
"needs_background": "light"
|
||
|
|
},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import matplotlib.pyplot as plt\n",
|
||
|
|
"import torchvision\n",
|
||
|
|
"\n",
|
||
|
|
"# Helper function to show an image\n",
|
||
|
|
"def matplotlib_imshow(img):\n",
|
||
|
|
" img = img.mean(dim=0)\n",
|
||
|
|
" img = img / 2 + 0.5 # unnormalize\n",
|
||
|
|
" npimg = img.numpy()\n",
|
||
|
|
" plt.title(\"MNIST Images\")\n",
|
||
|
|
" plt.imshow(npimg, cmap=\"Greys\")\n",
|
||
|
|
"\n",
|
||
|
|
"# get some random training images\n",
|
||
|
|
"dataiter = iter(train_loader)\n",
|
||
|
|
"images, labels = next(dataiter)\n",
|
||
|
|
"\n",
|
||
|
|
"# create grid of images\n",
|
||
|
|
"img_grid = torchvision.utils.make_grid(images)\n",
|
||
|
|
"\n",
|
||
|
|
"# show images\n",
|
||
|
|
"matplotlib_imshow(img_grid)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "pjtyBV0h5Aak"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"## 🔹 Logging plots through Matplotlib\n",
|
||
|
|
"\n",
|
||
|
|
"Similar to above, matplotlib is automatically captured, but this time, we use `plt.show()` (implicit inside `ConfusionMatrixDisplay`) which logs the result as a plot!\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"colab": {
|
||
|
|
"base_uri": "https://localhost:8080/",
|
||
|
|
"height": 313
|
||
|
|
},
|
||
|
|
"id": "GvM3VYPvxZoR",
|
||
|
|
"outputId": "b1d637a6-47b5-446c-9728-d6b720106a51"
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"text/plain": [
|
||
|
|
"<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x7efe139e0b50>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"execution_count": 24,
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "execute_result"
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAT4AAAEWCAYAAAD/x/trAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd5gV5fXHP+eW7X2XXqQKAmJDihUVURRFE000qGjsYkUldhMTsUTsouEnSRAVBcUWRUCR2CIWUASktwUWlu293Xt+f8xdWHDLLTPLLryf55ln9868c+ad984989bzFVXFYDAYDiZc+zsDBoPB0NwYx2cwGA46jOMzGAwHHcbxGQyGgw7j+AwGw0GHcXwGg+Ggwzi+ZkREYkXkAxEpFJHZEdgZKyLz7czb/kBE5orIuP2dj2AQkXtE5OX9nQ+DPRjHVw8i8gcR+V5ESkQkK/ADPcEG0xcA7YB0Vb0wXCOq+pqqjrQhP3shIsNFREXknX32HxHYvyhIO38WkVebSqeqo1R1ehj5vFxEvgz1vEhQ1UmqelVzXtPgHMbx7YOITACeBiZhOamuwBRgjA3mDwHWqGqNDbacYhcwTETS6+wbB6yx6wJiYZ49w/5DVc0W2IBkoAS4sJE00ViOcXtgexqIDhwbDmwFbgeygSzgisCxvwBVQHXgGlcCfwZerWO7G6CAJ/D5cmADUAxsBMbW2f9lnfOOA74DCgN/j6tzbBHwV+CrgJ35QEYD91ab/5eA8YF9bmAb8ACwqE7aZ4BMoAj4ATgxsP/Mfe7zpzr5eDiQj3KgV2DfVYHjLwJv17H/GPApIPXkc6/73+dYY2XRHfg8UA6fAC/sU/6XAZuBXOB+YBMwInBs93dV53saB2wBcoB769iJBaYD+cAvwERg6/5+vs22ZzNv3b0ZBsQA7zSS5l5gKHAkcAQwGLivzvH2WA60E5Zze0FEUlX1Qaxa5JuqmqCq0xrLiIjEA88Co1Q1EesH/WM96dKADwNp04EngQ/3qbH9AbgCaAtEAXc0dm3gFSwnAHAGsBzLydflO6wySANeB2aLSIyqfrzPfR5R55xLgWuARCwHU5fbgcMDzdgTscpunAY8STAEURavA98Gjv05kJ/ac/th1ezHAh3Y8x02xglAH+A04AEROSyw/0Es59gDOB24JNh7MDQPxvHtTTqQo403RccCD6lqtqruwqrJXVrneHXgeLWqfoRV6+kTZn78wAARiVXVLFVdUU+as4G1qjpDVWtUdSawCjinTpp/qeoaVS0HZmE5rAZR1a+BNBHpg+UAX6knzauqmhu45mSsmnBT9/lvVV0ROKd6H3tlWOX4JPAqcJOqbm3C3r40WBYi0hU4FnhAVatU9Uvg/TrnXgB8oKpfqmoVVg23Kaf7F1UtV9WfgJ+wXoQAvwMmqWp+4B6eDfE+DA5jHN/e5AIZIuJpJE1H9q6tbA7s221jH8dZBiSEmhFVLQV+D1wHZInIhyLSN4j81Oapbm1lRxj5mQHcCJxCPTVgEblDRH4JjFAXYNWQMpqwmdnYQVVdjNW0FywHHSqNlUVHIC/gYOvLT8e6nwPpcpu4XkPlupctmrhvQ/NjHN/e/A+oBM5rJM12rEGKWrry62ZgsJQCcXU+t697UFXnqerpWE2vVcD/BZGf2jxtCzNPtcwAbgA+2sdZEGiKTsSq2aSqagpWn5rUZr0Bm43WoERkPFbNcXvAfqg0VhZZWLXYuuXdpc7/WUDnOnmJxWoBhMNetva5jqEFYBxfHVS1EKuJ84KInCcicSLiFZFRIvJ4INlM4D4RaSMiGYH0TU7daIAfgZNEpKuIJAN31x4QkXYiMibQ11eJ1WT212PjI+DQwBQcj4j8HugH/CfMPAGgqhuBk7H6NPclEajBGgH2iMgDQFKd4zuBbqGM3IrIocDfsPrDLgUmikhjTXIRkZi6G42UhapuBr4H/iwiUSIyjL27A97CahIfJyJRWH2AQnjMAu4WkVQR6YRVcza0IIzj24dAf9UErAGLXVjNlBuBdwNJ/ob1A1oG/AwsCewL51oLgDcDtn5gb2flCuRjO5CH5YSur8dGLjAaa3AgF6umNFpVc8LJ0z62v1TV+mqz84CPsaa4bAYq2Ls5Vzs5O1dEljR1nUDXwqvAY6r6k6quBe4BZohIdAOnHYc1Olx3K6TxshiLNYCVi/WdvYn1UiHQf3oT8AZWja0Ea2S+sqn818NDWKPjG7FGj98K047BISSEQTOD4YBCRN4EVgVG3Pc9lgAUAL0Dtd9IrnM9cJGqnhyJHYN9mBqf4aBBRI4VkZ4i4hKRM7Empb9b5/g5ge6NeOAJrBr9pjCu00FEjg9cpw9WDbSxKVKGZqax0UuD4UCjPTAHa9BiK3C9qi6tc3wM1qCOYHVnXBTKPMI6RAH/wJowXYDVfJ4SQb4NNmOaugaD4aDDNHUNBsNBR4tq6iamebRNp4YG8cInZ7n9Ng2G1kgFpVRpZbjTdAA445R4zc3zBZX2h2WV81T1zEiu5wQtyvG16RTNpHcOazphiEw7tLvtNg2G1shi/TRiG7l5Pr6d1zWotO4Oa5tazbNfaFGOz2AwtHwU8Nc7l771YByfwWAICUWp1uCaui0V4/gMBkPImBqfwWA4qFAUXyufBmccn8FgCBl/k6EKWzYt1vEtn57E6lmJoNDnd8UMuLwIgBWvJPHLa4mIG7oML2PwxHyKt3p4e1QnkrtbsS3bHlnJ8Q81FUptbwYNL+K6v27H7VLmzkxj1vPtIr4Hb7SfyXPW4Y1S3B7liw9TmPFE+6ZPbIQ2Hau485ktpLSpAYWPXk3n3WltIs4rtJ4yqMXlUp77eA25WV4eGNfDFpsTntzCkBHFFOR4uPbUcOPH7k1r+86aQgGfcXwNE1gP+QyWbsPLqvpoMOflrfGyelYiY97ajsurzLuyPV1PKaMky8OWT+M4/4NtuKOgPHfP/OvErjWc/354YfFcLmX8pG3cfVEPcrK8PPfRWr6Zl8yWtTFh2aululKYeGFPKsrcuD3Kk++u47uFiaxaEh+2TV+NMPWhjqz7OY7YeB/Pf7yGJZ8nRpzX1lQGtZx3VQ6Za2OIS7Cvo33+m2m8/68M7nzGvtihre07C4bWXuNzbOWGiLixxFxGYcVEuziga9Akheu9tD2iEk+s4vJA+8EVbJofz6qZiQy8pgB3lJUuNt2eDtY+R5WxfVMUO7ZEU1PtYtF7KQw7o9AGy0JFmRsAj1dxe5VIu0bysr2s+9mKpVle6iZzXQwZHaqbOKtpWlMZAGR0qGLwaUXMfT0tcmN1WL44geJ8e+sDre87axwFqlWD2loqTi5ZGwysU9UNAQ2DNwhSojG1dzU7vo+hIt9FTbmQ+d9YSrPcFG70svP7GN6/oAMfjm3PrmVRu88p2erhnTEd+XBse3Z8F9pKjfT21ezavsdWTpbXlgcTrLfylAWreXPZCpZ+nsDqpZHXdGpp17mKngPKWbUkrunETdDayuC6v2zn5b91QP0RLUJodlrLd9YYiuILcmupOOn4OrF3cMqt1KNaJSLXBMS7vy/Os6QqUnpVM/DqAj7+Y3s+vrI96YdVIW7w+4TKQhfnzM5i8MQ8Ft7aFlWIa1vD7xdlcv572xlydx6Lbm9LVUnL+EH4/cINp/dh7DH96HNkGYf0KbfFbkycj/tf3sRLD3SkrMRti02nsLsMhowooiDHs7sW1VpoTd9Zoyj4gtxaKvs9SIGqTlXVQao6KDFtTxOjz4UlnPfOdka/nkV0kp+kbtXEt6/hkJFliECbI6oQgYp8F+4oiEm1mr0ZA6pI7FpN4UZv0HnI3eGlTceq3Z8zOlSTkxX8+cFQWuTmp68TOPaU4ohtuT3K/S9vYuGcVL6am2JD7lpXGfQ7tpShI4uYvngld7+4mSNOKGHic/tqDLUsWut3Vh/Wyo3gtpaKk45vG3uLrHQmBAGc2oGLku1uNs2Po+c5pRwyooysxVbHbeFGD/5qISbVT3meC3+gf7toi4eiTV6SujSmELk3q3+Mo1P3Ktp1qcTj9TN8TAHfzE8O+vyGS
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 432x288 with 2 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {
|
||
|
|
"needs_background": "light"
|
||
|
|
},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"# Load MNIST test dataset\n",
|
||
|
|
"test_data = MNIST('data', train=False, download=True, transform=ToTensor())\n",
|
||
|
|
"test_loader = DataLoader(test_data, batch_size=64, shuffle=False)\n",
|
||
|
|
"\n",
|
||
|
|
"# Test the model and compute confusion matrix\n",
|
||
|
|
"y_true = []\n",
|
||
|
|
"y_pred = []\n",
|
||
|
|
"model.eval()\n",
|
||
|
|
"with torch.no_grad():\n",
|
||
|
|
" for inputs, labels in test_loader:\n",
|
||
|
|
" inputs = inputs.view(-1, 784)\n",
|
||
|
|
" outputs = model(inputs)\n",
|
||
|
|
" _, predicted = torch.max(outputs, 1)\n",
|
||
|
|
" y_true.extend(labels.numpy())\n",
|
||
|
|
" y_pred.extend(predicted.numpy())\n",
|
||
|
|
"cm = confusion_matrix(y_true, y_pred)\n",
|
||
|
|
"\n",
|
||
|
|
"# Display confusion matrix\n",
|
||
|
|
"plt.title(\"Confusion Matrix Logging\")\n",
|
||
|
|
"ConfusionMatrixDisplay(cm).plot(ax=plt.gca())"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"id": "6hvbrbvEp3yb"
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"# When a python script ends, the ClearML task is closed automatically. But in\n",
|
||
|
|
"# a notebook (that never ends), we need to manually close the task.\n",
|
||
|
|
"task.close()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "nXCUDvfZ8kAs"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"# 🚁 Example: Sklearn\n",
|
||
|
|
"\n",
|
||
|
|
"As a third example, let's train an sklearn example. Here, too, ClearML will automatically capture a number of outputs, which we will describe in the code comments."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"colab": {
|
||
|
|
"base_uri": "https://localhost:8080/",
|
||
|
|
"height": 264
|
||
|
|
},
|
||
|
|
"id": "I_UH3SSZ9WQ0",
|
||
|
|
"outputId": "e6c6c58d-db9d-4dd0-edb2-3d556283ae55"
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"ClearML Task: created new task id=62fb0fc73d384181a220dfb8adb2c75a\n",
|
||
|
|
"ClearML results page: https://app.clear.ml/projects/df7858a441f94c29a1230d467f131840/experiments/62fb0fc73d384181a220dfb8adb2c75a/output/log\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAADSCAYAAACFOhYiAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAABO7UlEQVR4nO2dd1iT1xfHvzdAICETAmFvFREnQ1zgRK2Ke1u1ddZR+9PaYZdW29rWtlpttbZ11K24V92jFTeguGUIyF6yV8j5/REajMhQQVDez/O8D8l97zhvyMl773nPPYcRETg4OBoevLoWgIODo27glJ+Do4HCKT8HRwOFU34OjgYKp/wcHA0UTvk5OBoonPK/ojDGRjPGjta1HByvLpzy11MYYw8YY90rOk9Em4jI/zn6Pc0YK2CMZTPGshhjVxljHzHGDJ+hD2KMuTzr2M/KyxqnocIp/ysIY0z/BbuYQURiAJYA5gAYAeAQY4y9sHAcrwyc8r8CMMbGM8bOMcZ+YoylAZhfWvZv6XlWei659G4exhhzr6pfIsolotMAAgC0A9CntD9vxth5xtgjxlgCY2wFY4xfeu5safNrjLEcxthwxpicMXaAMZbCGMsofW3zhPyRpbONKMbY6MfOvc0Yu13a7ghjzL6icV74g+TQgVP+V4e2ACIBKAF89cQ5fwC+ABoDkAIYBiCtuh0TUQyAKwA6lRaVAPgfAAU0PwrdAEwrretbWqclEYmIaBs036O1AOwB2AHIB7ACABhjxgB+BtC7dLbRHkBo6bn+AOYBGATADMA/ALZUMg5HDcIp/6tDPBEtJyIVEeU/ca4YgBiAKwBGRLeJKOFZ+wdgAgBEdJWILpSO9QDAbwD8KmpIRGlEtJOI8ogoG5ofp8frqwG4M8YERJRARDdLy6cC+KZUXhWArwG0+u/uz1G7cMr/6hBb0QkiOgnNnfYXAMmMsdWMMckz9m8NIB0AGGONS6fuiYyxLGiUUlFRQ8aYkDH2G2MsurT+WQAyxpgeEeUCGA6Noicwxg4yxlxLm9oDWFa6vHhUOj4rlYWjluGU/9Wh0u2XRPQzEXkAcINm+j+3uh0zxmwBeEAz7QaAlQDuAGhERBJopuaVGQPnAGgCoG1p/f+m7KxUtiNE1AMaA+MdAL+Xno8FMIWIZI8dAiIKqq7sHM8Pp/yvAYwxL8ZYW8aYAYBcAAXQTLWraidkjPkB2AvgEoBDpafEALIA5JTepd95omkSAKfH3ouhWec/YoyZAPjisTGUjLH+pWv/QgA5j8m2CsDHjLFmpXWljLGhlYzDUYNwyv96IIHmbpoBIBoaY9/3ldRfwRjLhka5lgLYCaAXEf2nlO8DGAUgu7TfJ41t8wGsL52uDyvtQwAgFcAFAH8/VpcHYDY0NoV0aGwB7wAAEe0G8C2AraXLhRsAelcyDkcNwrhgHhwcDRPuzs/B0UDhlJ+Do4HCKT8HRwOFU34OjgbKi24QqRYKhYIcHBxexlAcHByPcfXq1VQiMnvauZei/A4ODrhy5crLGIqDg+MxGGPRFZ3jpv0cHA0UTvk5OBoonPJzcDRQOOXn4GigcMrPwdFA4ZSfg6OBwik/B0cDhVN+Do4GCqf8HBwNFE75OTgaKJzyc3A0UDjl5+BooHDKz8HRQOGUn4OjgcIpPwdHA4VTfg6OBgqn/BwcDRRO+Tk4GigvJYwXx8shLy8PQUFBEAgE8PHxgZ6eXl2LxFGP4ZT/NWHLli2YNmMGbBwbIT8vB6r8POzaGYg2bdrUtWgc9RRO+V8D7t69i+kz38WHv26BfWM3AMCFo/vRp28/RD+IAp/Pr2MJOeoj3Jr/NWDtunXwDRimVXwA8PHvB3MbB/z999+VtORoyHDK/xqQkfEIElPzcuVShRkyMjLqQCKOVwFO+V8Devr3wKUje1CiUmnLsjLScC3oDLp27VqHknHUZ7g1/2tAQEAAVq3+Hd9OGwnf/iORn5uNY1vXYObMmbC1ta1r8TjqKZzyvwbo6+vj4P592LRpE3bv3QehUIA/Vv2Knj171rVoHPUYRkS1Poinpydx6bo4OF4+jLGrROT5tHPcmp+Do4HCKf8rwJkzZ/DmuPHoG9Afv/zyC/Lz8+taJI7XAE756zk/LV2K4aPGgKd0glP7nli7bRc6d+3G/QBwvDCc8tdjMjIyMH/+fMxbvQO9Rr6Ndj0DMHvpOqj0DLFhw4a6Fo/jFYdT/npMUFAQXJq1gpmVjbaMMQaf3oNw+MjROpSM43WAU/56jFwuR3pKEp58IvMoJQmmJiZ1JBXH6wKn/C+JK1euoG+/AFhYWsHT2wfbtm2rso2Pjw8MeMCJwA3aH4DEmCgc27YGEye8Xdsic7zmcE4+L4GQkBD49+yFAVNm45N3PkFsxF3M/fgTpKSmYsb06RW24/F4OLBvLwIGDMTx7esgNVEg+v5tfLt4MXx8fF7iFXC8jnBOPi+BQUOGQuLcAj1HvKUtiw2/gyUzxiDuYSwMDAwqbU9EuHz5MjIzM9G2bVtIJJLaFpnjNaEyJx/uzv8SCAkJwYzRM3TKbF1cAR4PiYmJVfrfM8bg7e1dmyJyNEC4Nf9LwMHBAQ/u3tApy0hJRGF+PkxNTWt1bCJCYmIicnNza3UcjlcPTvlfAh/OfR87ln+De9eugIiQmhCH1V/MxsSJEyEUCmtt3IMHD6JRE1e4ujWD0sIS49+egJycnFobj+PVglvzvyQ2bdqEeZ9+hoyMdPAYD1OmTMFXixZCX792Vl5Xr16Ff6/emLJwGdy9OyI36xE2/jAfpoY87ArcUStjctQ/Klvzc8r/ElGr1cjIyIBYLK71uHpvjhsPMrVFnzenaMuKCvIxq48Pwq6Fcvv8Gwjcrr56Ao/Hg6mp6UsJqBn14AHsGjXVKeMbCWBl54iYmJhaH5+j/sMpfz1DrVZj06ZN+PzzzxEaGvrc/bT19ETY+TM6ZY9Sk/HwQTjc3NwqaMXRkOAe9dUjbty4Ad/OXaAGg6W9E779fglatmyJC0HnwOM92+/0e+/NgqeXNwTGYvj0DEBqwkPsWLEYM6bPgFwur6Ur4HiV4O789Qj/Xr3Rvs9g/HLkCj7/IxArDl9EfHIqpkyZUnXjJ7C1tUXQuX9hmJeKn2aNxeE/fsSH/3sXXy1aWAuSc7yKcAa/esKNGzfg4eWN1aeug29opC2/euYY1i+eh5TEhDqUjuNVhTP4vQIkJSXB0MgIBnxDnXKpiQLFRcV1JBXH6wy35q8FSkpKMGfOHOzevRtyuRy//PILOnToUGkbPz8/qIqLcetyEJp5l9U9vWcrmjRuVK1xQ0JCsH//fgiFQgwbNgx2dnYvdB0cz05YWBj27NkDQ0NDDB06FI6OjuXqlJSU4PDhw7h06RLs7OwwfPhwiMXily8sEdX64eHhQQ2FvLw8kpmYktzcggLemk7tegaQgaERTZo0qcq28+fPJyOBkAZMnEkzvllBXl16kdBYRLdu3aq0nVqtpv/NnkPmllbUb9xU6jFkDEllctq4cWNNXRZHNfh03kekkIpogJsZ9XE1I5lISKtX/6ZTJycnhzq09SJXKxMa7q6gji7mpFSYUFhYWK3IBOAKVaCX3Jq/hunevTvuxcRh4V/7tWv3m5fO4ftZ45GclAipVFpp+3379uGLBV8iPSMDLd3dsXz5z7C3t6+0zdmzZzHyzXFYsOEAjMWa/h9G3MWiiUPwICqSs+6/BK5cuYK+PbpiSVclJIaaCXV8dhE+PJWAu+GRsLCwAAB88flnOLV5FWZ7mYLHGADgaMQjXCo2x6XgazUuF7fmf4lcCbmGfmPf0THaNfPuAFOlFRYsWFBl+4CAAIRcvYLoyAjs27e3SsUHgO2BgegUMFyr+ABg49wEbh5tuUSdL4nAHTvgZ2OkVXwAsBLz4WEtwYEDB7RlO7ZsRj8nY63iA0A3Rynu3ruPhISXa9TllL+GYQxQk7pcuVpd8szP6quLHuOB1E8bUw322JeMo/bg8Xh42hyaAJ3/AY/HoH5itk3QLL9r6/tREZzy1zA+Xp7Yt+YXFD4WWjv03ClkpCZj/vz5OnUjIiIw673/o
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 288x216 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import joblib\n",
|
||
|
|
"\n",
|
||
|
|
"from sklearn import datasets\n",
|
||
|
|
"from sklearn.linear_model import LogisticRegression\n",
|
||
|
|
"from sklearn.model_selection import train_test_split\n",
|
||
|
|
"import numpy as np\n",
|
||
|
|
"import matplotlib.pyplot as plt\n",
|
||
|
|
"\n",
|
||
|
|
"from clearml import Task\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"# Connecting ClearML with the current process,\n",
|
||
|
|
"# from here on everything is logged automatically\n",
|
||
|
|
"task = Task.init(\n",
|
||
|
|
" project_name=\"Getting Started\",\n",
|
||
|
|
" task_name=\"Scikit-Learn\",\n",
|
||
|
|
" output_uri=True\n",
|
||
|
|
")\n",
|
||
|
|
"\n",
|
||
|
|
"iris = datasets.load_iris()\n",
|
||
|
|
"X = iris.data\n",
|
||
|
|
"y = iris.target\n",
|
||
|
|
"\n",
|
||
|
|
"X_train, X_test, y_train, y_test = \\\n",
|
||
|
|
" train_test_split(X, y, test_size=0.2, random_state=42)\n",
|
||
|
|
"\n",
|
||
|
|
"model = LogisticRegression(solver='liblinear', multi_class='auto')\n",
|
||
|
|
"model.fit(X_train, y_train)\n",
|
||
|
|
"\n",
|
||
|
|
"# Using joblib to save the model will automatically register it to ClearML, too!\n",
|
||
|
|
"joblib.dump(model, 'model.pkl', compress=True)\n",
|
||
|
|
"\n",
|
||
|
|
"loaded_model = joblib.load('model.pkl')\n",
|
||
|
|
"result = loaded_model.score(X_test, y_test)\n",
|
||
|
|
"x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n",
|
||
|
|
"y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n",
|
||
|
|
"h = .02 # step size in the mesh\n",
|
||
|
|
"xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
|
||
|
|
"plt.figure(1, figsize=(4, 3))\n",
|
||
|
|
"\n",
|
||
|
|
"plt.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k', cmap=plt.cm.Paired)\n",
|
||
|
|
"plt.title(\"Iris Dataset\")\n",
|
||
|
|
"plt.xlabel('Sepal length')\n",
|
||
|
|
"plt.ylabel('Sepal width')\n",
|
||
|
|
"\n",
|
||
|
|
"plt.xlim(xx.min(), xx.max())\n",
|
||
|
|
"plt.ylim(yy.min(), yy.max())\n",
|
||
|
|
"plt.xticks(())\n",
|
||
|
|
"plt.yticks(())\n",
|
||
|
|
"\n",
|
||
|
|
"# Plt.show() will trigger ClearML to log the resulting plot automatically\n",
|
||
|
|
"plt.show()\n",
|
||
|
|
"\n",
|
||
|
|
"# Always close the task when in a notebook! If using a python file, the task is\n",
|
||
|
|
"# closed automatically when the script ends.\n",
|
||
|
|
"task.close()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "AaxJ4QmErwTw"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"# ✋ Manual Logging\n",
|
||
|
|
"\n",
|
||
|
|
"Naturally, when our integrations aren't cutting it for you, you can always manually log anything you wish!\n",
|
||
|
|
"\n",
|
||
|
|
"Check our documentation for [a list of examples](https://clear.ml/docs/latest/docs/fundamentals/logger#explicit-reporting-examples) of how to manually log anything else!"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"attachments": {},
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"id": "Rn1Csnat7b5D"
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"# 🥾 Next Steps\n",
|
||
|
|
"\n",
|
||
|
|
"Now that you have the basics of the experiment manager, take a look at **running this experiment remotely**! Start by setting up a remote agent and then clone and enqueue these experiments using the ClearML orchestration component.\n",
|
||
|
|
"\n",
|
||
|
|
"The [second notebook](https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_2_Setting_Up_Agent.ipynb) in this series will get you started with this."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"attachments": {},
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"<table>\n",
|
||
|
|
"<tbody>\n",
|
||
|
|
" <tr>\n",
|
||
|
|
" <td>Step 1: Experiment Management</td>\n",
|
||
|
|
" <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_1_Experiment_Management.ipynb\">\n",
|
||
|
|
" <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
|
||
|
|
"</a></td>\n",
|
||
|
|
" </tr>\n",
|
||
|
|
" <tr>\n",
|
||
|
|
" <td><b>NEXT UP -> Step 2: Remote Execution Agent Setup</b></td>\n",
|
||
|
|
" <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_2_Setting_Up_Agent.ipynb\">\n",
|
||
|
|
" <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
|
||
|
|
"</a></td>\n",
|
||
|
|
" </tr>\n",
|
||
|
|
" <tr>\n",
|
||
|
|
" <td>Step 3: Remotely Execute Tasks</td>\n",
|
||
|
|
" <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_3_Remote_Execution.ipynb\">\n",
|
||
|
|
" <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
|
||
|
|
"</a></td>\n",
|
||
|
|
" </tr>\n",
|
||
|
|
"</tbody>\n",
|
||
|
|
"</table>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": []
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {
|
||
|
|
"accelerator": "GPU",
|
||
|
|
"colab": {
|
||
|
|
"collapsed_sections": [
|
||
|
|
"TLsgNR5PhPPN",
|
||
|
|
"pkYjS0RTtJ6s",
|
||
|
|
"5iwK3p-X3WIC",
|
||
|
|
"fsHJZGcC3afb",
|
||
|
|
"nXCUDvfZ8kAs"
|
||
|
|
],
|
||
|
|
"provenance": [],
|
||
|
|
"toc_visible": true
|
||
|
|
},
|
||
|
|
"gpuClass": "standard",
|
||
|
|
"kernelspec": {
|
||
|
|
"display_name": "Python 3",
|
||
|
|
"name": "python3"
|
||
|
|
},
|
||
|
|
"language_info": {
|
||
|
|
"name": "python"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"nbformat": 4,
|
||
|
|
"nbformat_minor": 0
|
||
|
|
}
|