ClearML/clearml
1
0
Fork 0
mirror of https://github.com/clearml/clearml synced 2025-03-03 10:42:00 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Update examples to ClearML

Browse Source
This commit is contained in:
allegroai 2020-12-30 16:53:19 +02:00
parent 0bb590b5da
commit 4c8a24f1ff
17 changed files with 86 additions and 87 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
examples/frameworks/keras/jupyter.ipynb
View File

@ -10,7 +10,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# Trains - Example of integrating plots and training on jupyter notebook. \n", "# ClearML - Example of integrating plots and training on jupyter notebook. \n",
"# In this example, simple graphs are shown, then an MNIST classifier is trained using Keras.\n", "# In this example, simple graphs are shown, then an MNIST classifier is trained using Keras.\n",
"import os\n", "import os\n",
"import tempfile\n", "import tempfile\n",
@ -35,14 +35,14 @@
"name": "stdout", "name": "stdout",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"TRAINS Task: created new task id=2f9f2f08fa90427aa51e34b839e49fb6\n", "ClearML Task: created new task id=2f9f2f08fa90427aa51e34b839e49fb6\n",
"TRAINS results page: https://demoapp.trains.allegro.ai/projects/0e152d03acf94ae4bb1f3787e293a9f5/experiments/2f9f2f08fa90427aa51e34b839e49fb6/output/log\n" "ClearML results page: https://demoapp.clearml.allegro.ai/projects/0e152d03acf94ae4bb1f3787e293a9f5/experiments/2f9f2f08fa90427aa51e34b839e49fb6/output/log\n"
] ]
} }
], ],
"source": [ "source": [
"# Connecting TRAINS\n", "# Connecting ClearML\n",
"from trains import Task\n", "from clearml import Task\n",
"task = Task.init(project_name = 'examples', task_name = 'notebook example')\n" "task = Task.init(project_name = 'examples', task_name = 'notebook example')\n"
] ]
}, },
@ -146,7 +146,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# Notice, Updating task_params is traced and updated in TRAINS\n", "# Notice, Updating task_params is traced and updated in ClearML\n",
"task_params['batch_size'] = 128\n", "task_params['batch_size'] = 128\n",
"task_params['nb_classes'] = 10\n", "task_params['nb_classes'] = 10\n",
"task_params['nb_epoch'] = 6\n", "task_params['nb_epoch'] = 6\n",

14
examples/frameworks/keras/Allegro_Trains_keras_TB_example.ipynb → examples/frameworks/keras/jupyter_keras_TB_example.ipynb
View File

@ -7,11 +7,11 @@
"id": "wFJPLbY7w7Vj" "id": "wFJPLbY7w7Vj"
}, },
"source": [ "source": [
"# Allegro Trains Keras with TensorBoard example\n", "# Allegro ClearML Keras with TensorBoard example\n",
"\n", "\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allegroai/trains/blob/master/examples/frameworks/keras/Allegro_Trains_keras_TB_example.ipynb)\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allegroai/clearml/blob/master/examples/frameworks/keras/Allegro_Trains_keras_TB_example.ipynb)\n",
"\n", "\n",
"This example introduces Trains with Keras and TensorBoard functionality, including automatic logging, models, and TensorBoard outputs. You can find more frameworks examples [here](https://github.com/allegroai/trains/tree/master/examples/frameworks).\n", "This example introduces ClearML with Keras and TensorBoard functionality, including automatic logging, models, and TensorBoard outputs. You can find more frameworks examples [here](https://github.com/allegroai/clearml/tree/master/examples/frameworks).\n",
"\n", "\n",
"Note: This example is based on the Keras `mnist_mlp.py` example.\n" "Note: This example is based on the Keras `mnist_mlp.py` example.\n"
] ]
@ -26,7 +26,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install trains\n", "!pip install clearml\n",
"!pip install tensorflow>=2.0" "!pip install tensorflow>=2.0"
] ]
}, },
@ -38,7 +38,7 @@
}, },
"source": [ "source": [
"### Create a new task.\n", "### Create a new task.\n",
"To create a new Task object, call the `Task.init` method providing it with `project_name` (the project name for the experiment) and `task_name` (the name of the experiment). When `Task.init` executes, a link to the Web UI Results page for the newly generated Task will be printed, and the Task will be updated in real time in the Trains demo server.\n", "To create a new Task object, call the `Task.init` method providing it with `project_name` (the project name for the experiment) and `task_name` (the name of the experiment). When `Task.init` executes, a link to the Web UI Results page for the newly generated Task will be printed, and the Task will be updated in real time in the ClearML demo server.\n",
"\n", "\n",
"You can read about the `Task` class in the docs [here](https://allegro.ai/docs/task.html)." "You can read about the `Task` class in the docs [here](https://allegro.ai/docs/task.html)."
] ]
@ -60,7 +60,7 @@
"import tensorflow as tf\n", "import tensorflow as tf\n",
"from tensorflow import keras\n", "from tensorflow import keras\n",
"from tensorflow.keras import utils as np_utils\n", "from tensorflow.keras import utils as np_utils\n",
"from trains import Task\n", "from clearml import Task\n",
"\n", "\n",
"# Start a new task\n", "# Start a new task\n",
"task = Task.init(project_name=\"Colab notebooks\", task_name=\"Keras with TensorBoard example\")\n" "task = Task.init(project_name=\"Colab notebooks\", task_name=\"Keras with TensorBoard example\")\n"
@ -205,7 +205,7 @@
"metadata": { "metadata": {
"colab": { "colab": {
"collapsed_sections": [], "collapsed_sections": [],
"name": "Allegro Trains keras TB example.ipynb", "name": "Allegro ClearML keras TB example.ipynb",
"provenance": [] "provenance": []
}, },
"kernelspec": { "kernelspec": {

12
examples/frameworks/keras/legacy/jupyter.ipynb
View File

@ -18,7 +18,7 @@
} }
], ],
"source": [ "source": [
"# Trains - Example of integrating plots and training on jupyter notebook. \n", "# ClearML - Example of integrating plots and training on jupyter notebook. \n",
"# In this example, simple graphs are shown, then an MNIST classifier is trained using Keras.\n", "# In this example, simple graphs are shown, then an MNIST classifier is trained using Keras.\n",
"\n", "\n",
"from keras.callbacks import TensorBoard, ModelCheckpoint\n", "from keras.callbacks import TensorBoard, ModelCheckpoint\n",
@ -41,14 +41,14 @@
"name": "stdout", "name": "stdout",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"TRAINS Task: overwriting (reusing) task id=6de40029e54c41d7a1a24a1f2dc9cad2\n", "ClearML Task: overwriting (reusing) task id=6de40029e54c41d7a1a24a1f2dc9cad2\n",
"TRAINS results page: https://demoapp.trains.allegro.ai/projects/087f765c846c4c76a7e9f3d035667d82/experiments/6de40029e54c41d7a1a24a1f2dc9cad2/output/log\n" "ClearML results page: https://demoapp.clearml.allegro.ai/projects/087f765c846c4c76a7e9f3d035667d82/experiments/6de40029e54c41d7a1a24a1f2dc9cad2/output/log\n"
] ]
} }
], ],
"source": [ "source": [
"# Connecting TRAINS\n", "# Connecting ClearML\n",
"from trains import Task\n", "from clearml import Task\n",
"task = Task.init(project_name = 'examples', task_name = 'notebook example')\n" "task = Task.init(project_name = 'examples', task_name = 'notebook example')\n"
] ]
}, },
@ -152,7 +152,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"# Notice, Updating task_params is traced and updated in TRAINS\n", "# Notice, Updating task_params is traced and updated in ClearML\n",
"task_params['batch_size'] = 128\n", "task_params['batch_size'] = 128\n",
"task_params['nb_classes'] = 10\n", "task_params['nb_classes'] = 10\n",
"task_params['nb_epoch'] = 6\n", "task_params['nb_epoch'] = 6\n",

20
examples/frameworks/matplotlib/Allegro_Trains_matplotlib_example.ipynb → examples/frameworks/matplotlib/jupyter_matplotlib_example.ipynb
View File

@ -7,13 +7,13 @@
"id": "NKas2cYws8F6" "id": "NKas2cYws8F6"
}, },
"source": [ "source": [
"# Allegro Trains matplotlib example\n", "# Allegro ClearML matplotlib example\n",
"\n", "\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allegroai/trains/blob/master/examples/frameworks/matplotlib/Allegro_Trains_matplotlib_example.ipynb)\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allegroai/clearml/blob/master/examples/frameworks/matplotlib/Allegro_Trains_matplotlib_example.ipynb)\n",
"\n", "\n",
"This example introduces Trains with matplotlib functionality. It also shows seaborn functionality. You can find more frameworks examples [here](https://github.com/allegroai/trains/tree/master/examples/frameworks).\n", "This example introduces ClearML with matplotlib functionality. It also shows seaborn functionality. You can find more frameworks examples [here](https://github.com/allegroai/clearml/tree/master/examples/frameworks).\n",
"\n", "\n",
"Note: This example is based on the Trains [matplotlib_example.py](https://github.com/allegroai/trains/blob/master/examples/frameworks/matplotlib/matplotlib_example.py) example." "Note: This example is based on the ClearML [matplotlib_example.py](https://github.com/allegroai/clearml/blob/master/examples/frameworks/matplotlib/matplotlib_example.py) example."
] ]
}, },
{ {
@ -26,7 +26,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install trains\n", "!pip install clearml\n",
"!pip install numpy\n", "!pip install numpy\n",
"!pip install seaborn" "!pip install seaborn"
] ]
@ -40,7 +40,7 @@
"source": [ "source": [
"## Create a new task.\n", "## Create a new task.\n",
"\n", "\n",
"To create a new Task object, call the `Task.init` method providing it with `project_name` (the project name for the experiment) and `task_name` (the name of the experiment). When `Task.init` executes, a link to the Web UI Results page for the newly generated Task will be printed, and the Task will be updated in real time in the Trains demo server.\n", "To create a new Task object, call the `Task.init` method providing it with `project_name` (the project name for the experiment) and `task_name` (the name of the experiment). When `Task.init` executes, a link to the Web UI Results page for the newly generated Task will be printed, and the Task will be updated in real time in the ClearML demo server.\n",
"\n", "\n",
"You can read about the `Task` class in the docs [here](https://allegro.ai/docs/task.html)." "You can read about the `Task` class in the docs [here](https://allegro.ai/docs/task.html)."
] ]
@ -59,7 +59,7 @@
"import numpy as np\n", "import numpy as np\n",
"import seaborn as sns\n", "import seaborn as sns\n",
"\n", "\n",
"from trains import Task\n", "from clearml import Task\n",
"\n", "\n",
"# Start a new task\n", "# Start a new task\n",
"task = Task.init(project_name=\"Colab notebooks\", task_name=\"Matplotlib example\")\n" "task = Task.init(project_name=\"Colab notebooks\", task_name=\"Matplotlib example\")\n"
@ -74,7 +74,7 @@
"source": [ "source": [
"## Matplotlib support\n", "## Matplotlib support\n",
"\n", "\n",
"Trains automatically logs Matplotlib plots. They appear in the Web UI Results tab.\n" "ClearML automatically logs Matplotlib plots. They appear in the Web UI Results tab.\n"
] ]
}, },
{ {
@ -121,7 +121,7 @@
"id": "yKT5UjDk6DGB" "id": "yKT5UjDk6DGB"
}, },
"source": [ "source": [
"By calling the `imshow` method, Trains automatically reports plot images in Results tab." "By calling the `imshow` method, ClearML automatically reports plot images in Results tab."
] ]
}, },
{ {
@ -212,7 +212,7 @@
"metadata": { "metadata": {
"colab": { "colab": {
"collapsed_sections": [], "collapsed_sections": [],
"name": "Allegro Trains matplotlib example.ipynb", "name": "Allegro ClearML matplotlib example.ipynb",
"provenance": [] "provenance": []
}, },
"kernelspec": { "kernelspec": {

12
examples/frameworks/pytorch/notebooks/audio/audio_classifier_UrbanSound8K.ipynb
View File

@ -15,7 +15,7 @@
"! pip install -U torchaudio==0.5.1\n", "! pip install -U torchaudio==0.5.1\n",
"! pip install -U torchvision==0.6.1\n", "! pip install -U torchvision==0.6.1\n",
"! pip install -U matplotlib==3.2.1\n", "! pip install -U matplotlib==3.2.1\n",
"! pip install -U trains>=0.16.1\n", "! pip install -U clearml>=0.16.1\n",
"! pip install -U pandas==1.0.4\n", "! pip install -U pandas==1.0.4\n",
"! pip install -U numpy==1.18.4\n", "! pip install -U numpy==1.18.4\n",
"! pip install -U tensorboard==2.2.1" "! pip install -U tensorboard==2.2.1"
@ -50,8 +50,8 @@
"from torchvision.transforms import ToTensor\n", "from torchvision.transforms import ToTensor\n",
"from torchvision import models\n", "from torchvision import models\n",
"\n", "\n",
"from trains import Task\n", "from clearml import Task\n",
"from trains.storage import StorageManager\n", "from clearml.storage import StorageManager\n",
"\n", "\n",
"%matplotlib inline" "%matplotlib inline"
] ]
@ -65,7 +65,7 @@
"task = Task.init(project_name='Audio Example', task_name='audio classification UrbanSound8K')\n", "task = Task.init(project_name='Audio Example', task_name='audio classification UrbanSound8K')\n",
"configuration_dict = {'number_of_epochs': 3, 'batch_size': 8, 'dropout': 0.3, 'base_lr': 0.005, \n", "configuration_dict = {'number_of_epochs': 3, 'batch_size': 8, 'dropout': 0.3, 'base_lr': 0.005, \n",
" 'number_of_mel_filters': 64, 'resample_freq': 22050}\n", " 'number_of_mel_filters': 64, 'resample_freq': 22050}\n",
"configuration_dict = task.connect(configuration_dict) # enabling configuration override by trains\n", "configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml\n",
"print(configuration_dict) # printing actual configuration (after override in remote mode)" "print(configuration_dict) # printing actual configuration (after override in remote mode)"
] ]
}, },
@ -81,8 +81,8 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"# Download UrbanSound8K dataset (https://urbansounddataset.weebly.com/urbansound8k.html)\n", "# Download UrbanSound8K dataset (https://urbansounddataset.weebly.com/urbansound8k.html)\n",
"# For simplicity we will use here a subset of that dataset using trains StorageManager\n", "# For simplicity we will use here a subset of that dataset using clearml StorageManager\n",
"path_to_UrbanSound8K = StorageManager.get_local_copy(\"https://allegro-datasets.s3.amazonaws.com/trains/UrbanSound8K.zip\", \n", "path_to_UrbanSound8K = StorageManager.get_local_copy(\"https://allegro-datasets.s3.amazonaws.com/clearml/UrbanSound8K.zip\", \n",
" extract_archive=True)\n", " extract_archive=True)\n",
"path_to_UrbanSound8K_csv = Path(path_to_UrbanSound8K) / 'UrbanSound8K' / 'metadata' / 'UrbanSound8K.csv'\n", "path_to_UrbanSound8K_csv = Path(path_to_UrbanSound8K) / 'UrbanSound8K' / 'metadata' / 'UrbanSound8K.csv'\n",
"path_to_UrbanSound8K_audio = Path(path_to_UrbanSound8K) / 'UrbanSound8K' / 'audio'" "path_to_UrbanSound8K_audio = Path(path_to_UrbanSound8K) / 'UrbanSound8K' / 'audio'"

6
examples/frameworks/pytorch/notebooks/audio/audio_preprocessing_example.ipynb
View File

@ -12,7 +12,7 @@
"! pip install -U torch==1.5.1\n", "! pip install -U torch==1.5.1\n",
"! pip install -U torchaudio==0.5.1\n", "! pip install -U torchaudio==0.5.1\n",
"! pip install -U matplotlib==3.2.1\n", "! pip install -U matplotlib==3.2.1\n",
"! pip install -U trains>=0.16.1\n", "! pip install -U clearml>=0.16.1\n",
"! pip install -U tensorboard==2.2.1" "! pip install -U tensorboard==2.2.1"
] ]
}, },
@ -28,7 +28,7 @@
"from torch.utils.tensorboard import SummaryWriter\n", "from torch.utils.tensorboard import SummaryWriter\n",
"import matplotlib.pyplot as plt\n", "import matplotlib.pyplot as plt\n",
"\n", "\n",
"from trains import Task\n", "from clearml import Task\n",
"\n", "\n",
"%matplotlib inline" "%matplotlib inline"
] ]
@ -41,7 +41,7 @@
"source": [ "source": [
"task = Task.init(project_name='Audio Example', task_name='data pre-processing')\n", "task = Task.init(project_name='Audio Example', task_name='data pre-processing')\n",
"configuration_dict = {'number_of_samples': 3}\n", "configuration_dict = {'number_of_samples': 3}\n",
"configuration_dict = task.connect(configuration_dict) # enabling configuration override by trains\n", "configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml\n",
"print(configuration_dict) # printing actual configuration (after override in remote mode)" "print(configuration_dict) # printing actual configuration (after override in remote mode)"
] ]
}, },

16
examples/frameworks/pytorch/notebooks/image/hyperparameter_search.ipynb
View File

@ -7,12 +7,12 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"# execute this in command line on all machines to be used as workers before initiating the hyperparamer search \n", "# execute this in command line on all machines to be used as workers before initiating the hyperparamer search \n",
"# ! pip install -U trains-agent==0.15.0\n", "# ! pip install -U clearml-agent==0.15.0\n",
"# ! trains-agent daemon --queue default\n", "# ! clearml-agent daemon --queue default\n",
"\n", "\n",
"# pip install with locked versions\n", "# pip install with locked versions\n",
"! pip install -U pandas==1.0.3\n", "! pip install -U pandas==1.0.3\n",
"! pip install -U trains>=0.16.2\n", "! pip install -U clearml>=0.16.2\n",
"! pip install -U optuna==2.0.0" "! pip install -U optuna==2.0.0"
] ]
}, },
@ -22,11 +22,11 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"from trains.automation import UniformParameterRange, UniformIntegerParameterRange\n", "from clearml.automation import UniformParameterRange, UniformIntegerParameterRange\n",
"from trains.automation import HyperParameterOptimizer\n", "from clearml.automation import HyperParameterOptimizer\n",
"from trains.automation.optuna import OptimizerOptuna\n", "from clearml.automation.optuna import OptimizerOptuna\n",
"\n", "\n",
"from trains import Task" "from clearml import Task"
] ]
}, },
{ {
@ -72,7 +72,7 @@
" objective_metric_series='total',\n", " objective_metric_series='total',\n",
" objective_metric_sign='max', # maximize or minimize the objective metric\n", " objective_metric_sign='max', # maximize or minimize the objective metric\n",
"\n", "\n",
" # setting optimizer - trains supports GridSearch, RandomSearch, OptimizerBOHB and OptimizerOptuna\n", " # setting optimizer - clearml supports GridSearch, RandomSearch, OptimizerBOHB and OptimizerOptuna\n",
" optimizer_class=OptimizerOptuna,\n", " optimizer_class=OptimizerOptuna,\n",
" \n", " \n",
" # Configuring optimization parameters\n", " # Configuring optimization parameters\n",

6
examples/frameworks/pytorch/notebooks/image/image_classification_CIFAR10.ipynb
View File

@ -15,7 +15,7 @@
"! pip install -U torch==1.5.1\n", "! pip install -U torch==1.5.1\n",
"! pip install -U torchvision==0.6.1\n", "! pip install -U torchvision==0.6.1\n",
"! pip install -U numpy==1.18.4\n", "! pip install -U numpy==1.18.4\n",
"! pip install -U trains>=0.16.1\n", "! pip install -U clearml>=0.16.1\n",
"! pip install -U tensorboard==2.2.1" "! pip install -U tensorboard==2.2.1"
] ]
}, },
@ -36,7 +36,7 @@
"import torchvision.datasets as datasets\n", "import torchvision.datasets as datasets\n",
"import torchvision.transforms as transforms\n", "import torchvision.transforms as transforms\n",
"\n", "\n",
"from trains import Task" "from clearml import Task"
] ]
}, },
{ {
@ -47,7 +47,7 @@
"source": [ "source": [
"task = Task.init(project_name='Image Example', task_name='image classification CIFAR10')\n", "task = Task.init(project_name='Image Example', task_name='image classification CIFAR10')\n",
"configuration_dict = {'number_of_epochs': 3, 'batch_size': 4, 'dropout': 0.25, 'base_lr': 0.001}\n", "configuration_dict = {'number_of_epochs': 3, 'batch_size': 4, 'dropout': 0.25, 'base_lr': 0.001}\n",
"configuration_dict = task.connect(configuration_dict) # enabling configuration override by trains\n", "configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml\n",
"print(configuration_dict) # printing actual configuration (after override in remote mode)" "print(configuration_dict) # printing actual configuration (after override in remote mode)"
] ]
}, },

6
examples/frameworks/pytorch/notebooks/table/download_and_preprocessing.ipynb
View File

@ -8,7 +8,7 @@
"source": [ "source": [
"! pip install -U pip\n", "! pip install -U pip\n",
"! pip install -U torch==1.5.1\n", "! pip install -U torch==1.5.1\n",
"! pip install -U trains>=0.15.1\n", "! pip install -U clearml>=0.15.1\n",
"! pip install -U pandas==1.0.4\n", "! pip install -U pandas==1.0.4\n",
"! pip install -U numpy==1.18.4\n", "! pip install -U numpy==1.18.4\n",
"! pip install -U pathlib2==2.3.5\n", "! pip install -U pathlib2==2.3.5\n",
@ -28,7 +28,7 @@
"import torch\n", "import torch\n",
"from datetime import datetime\n", "from datetime import datetime\n",
"from pathlib2 import Path\n", "from pathlib2 import Path\n",
"from trains import Task" "from clearml import Task"
] ]
}, },
{ {
@ -40,7 +40,7 @@
"task = Task.init(project_name='Table Example', task_name='tabular preprocessing')\n", "task = Task.init(project_name='Table Example', task_name='tabular preprocessing')\n",
"logger = task.get_logger()\n", "logger = task.get_logger()\n",
"configuration_dict = {'test_size': 0.1, 'split_random_state': 0}\n", "configuration_dict = {'test_size': 0.1, 'split_random_state': 0}\n",
"configuration_dict = task.connect(configuration_dict) # enabling configuration override by trains\n", "configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml\n",
"print(configuration_dict) # printing actual configuration (after override in remote mode)" "print(configuration_dict) # printing actual configuration (after override in remote mode)"
] ]
}, },

12
examples/frameworks/pytorch/notebooks/table/download_and_split.ipynb
View File

@ -7,7 +7,7 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"! pip install -U pip\n", "! pip install -U pip\n",
"! pip install -U trains==0.16.2rc0\n", "! pip install -U clearml==0.16.2rc0\n",
"! pip install -U pandas==1.0.4\n", "! pip install -U pandas==1.0.4\n",
"! pip install -U scikit-learn==0.23.1\n", "! pip install -U scikit-learn==0.23.1\n",
"! pip install -U pathlib2==2.3.5" "! pip install -U pathlib2==2.3.5"
@ -23,7 +23,7 @@
"from pathlib2 import Path\n", "from pathlib2 import Path\n",
"from sklearn.model_selection import train_test_split\n", "from sklearn.model_selection import train_test_split\n",
"\n", "\n",
"from trains import Task" "from clearml import Task"
] ]
}, },
{ {
@ -35,7 +35,7 @@
"task = Task.init(project_name='Tabular Example', task_name='Download and split tabular dataset')\n", "task = Task.init(project_name='Tabular Example', task_name='Download and split tabular dataset')\n",
"logger = task.get_logger()\n", "logger = task.get_logger()\n",
"configuration_dict = {'test_size': 0.1, 'split_random_state': 0}\n", "configuration_dict = {'test_size': 0.1, 'split_random_state': 0}\n",
"configuration_dict = task.connect(configuration_dict) # enabling configuration override by trains\n", "configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml\n",
"print(configuration_dict) # printing actual configuration (after override in remote mode)" "print(configuration_dict) # printing actual configuration (after override in remote mode)"
] ]
}, },
@ -54,9 +54,9 @@
"source": [ "source": [
"# Download the shelter-animal-outcomes dataset (https://www.kaggle.com/c/shelter-animal-outcomes)\n", "# Download the shelter-animal-outcomes dataset (https://www.kaggle.com/c/shelter-animal-outcomes)\n",
"# and save it to your cloud storage or your mounted local storage\n", "# and save it to your cloud storage or your mounted local storage\n",
"# If the data is on your cloud storage, you can use trains' storage manager to get a local copy of it:\n", "# If the data is on your cloud storage, you can use clearml' storage manager to get a local copy of it:\n",
"# from trains.storage import StorageManager\n", "# from clearml.storage import StorageManager\n",
"# path_to_ShelterAnimal = StorageManager.get_local_copy(\"https://allegro-datasets.s3.amazonaws.com/trains/UrbanSound8K.zip\", \n", "# path_to_ShelterAnimal = StorageManager.get_local_copy(\"https://allegro-datasets.s3.amazonaws.com/clearml/UrbanSound8K.zip\", \n",
"# extract_archive=True)\n", "# extract_archive=True)\n",
"path_to_ShelterAnimal = '/home/sam/Datasets/shelter-animal-outcomes'" "path_to_ShelterAnimal = '/home/sam/Datasets/shelter-animal-outcomes'"
] ]

6
examples/frameworks/pytorch/notebooks/table/pick_best_model.ipynb
View File

@ -7,7 +7,7 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"! pip install -U pip\n", "! pip install -U pip\n",
"! pip install -U trains==0.16.2rc0" "! pip install -U clearml==0.16.2rc0"
] ]
}, },
{ {
@ -16,7 +16,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"from trains import Task, OutputModel" "from clearml import Task, OutputModel"
] ]
}, },
{ {
@ -27,7 +27,7 @@
"source": [ "source": [
"task = Task.init(project_name='Tabular Example', task_name='pick best model')\n", "task = Task.init(project_name='Tabular Example', task_name='pick best model')\n",
"configuration_dict = {'train_tasks_ids': ['c9bff3d15309487a9e5aaa00358ff091', 'c9bff3d15309487a9e5aaa00358ff091']}\n", "configuration_dict = {'train_tasks_ids': ['c9bff3d15309487a9e5aaa00358ff091', 'c9bff3d15309487a9e5aaa00358ff091']}\n",
"configuration_dict = task.connect(configuration_dict) # enabling configuration override by trains\n", "configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml\n",
"print(configuration_dict) # printing actual configuration (after override in remote mode)" "print(configuration_dict) # printing actual configuration (after override in remote mode)"
] ]
}, },

6
examples/frameworks/pytorch/notebooks/table/preprocessing_and_encoding.ipynb
View File

@ -7,7 +7,7 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"! pip install -U pip\n", "! pip install -U pip\n",
"! pip install -U trains==0.16.2rc0\n", "! pip install -U clearml==0.16.2rc0\n",
"! pip install -U pandas==1.0.4\n", "! pip install -U pandas==1.0.4\n",
"! pip install -U numpy==1.18.4" "! pip install -U numpy==1.18.4"
] ]
@ -22,7 +22,7 @@
"import numpy as np\n", "import numpy as np\n",
"from collections import Counter\n", "from collections import Counter\n",
"\n", "\n",
"from trains import Task" "from clearml import Task"
] ]
}, },
{ {
@ -35,7 +35,7 @@
"logger = task.get_logger()\n", "logger = task.get_logger()\n",
"configuration_dict = {'data_task_id': '39fbf86fc4a341359ac6df4aa70ff91b',\n", "configuration_dict = {'data_task_id': '39fbf86fc4a341359ac6df4aa70ff91b',\n",
" 'fill_categorical_NA': True, 'fill_numerical_NA': True}\n", " 'fill_categorical_NA': True, 'fill_numerical_NA': True}\n",
"configuration_dict = task.connect(configuration_dict) # enabling configuration override by trains\n", "configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml\n",
"print(configuration_dict) # printing actual configuration (after override in remote mode)" "print(configuration_dict) # printing actual configuration (after override in remote mode)"
] ]
}, },

6
examples/frameworks/pytorch/notebooks/table/tabular_ml_pipeline.ipynb
View File

@ -8,7 +8,7 @@
"source": [ "source": [
"# pip install with locked versions\n", "# pip install with locked versions\n",
"! pip install -U pip\n", "! pip install -U pip\n",
"! pip install -U trains==0.16.2rc0" "! pip install -U clearml==0.16.2rc0"
] ]
}, },
{ {
@ -17,8 +17,8 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"from trains import Task\n", "from clearml import Task\n",
"from trains.automation.controller import PipelineController" "from clearml.automation.controller import PipelineController"
] ]
}, },
{ {

6
examples/frameworks/pytorch/notebooks/table/train_tabular_predictor.ipynb
View File

@ -8,7 +8,7 @@
"source": [ "source": [
"! pip install -U pip\n", "! pip install -U pip\n",
"! pip install -U torch==1.5.1\n", "! pip install -U torch==1.5.1\n",
"! pip install -U trains==0.16.2rc0\n", "! pip install -U clearml==0.16.2rc0\n",
"! pip install -U pandas==1.0.4\n", "! pip install -U pandas==1.0.4\n",
"! pip install -U numpy==1.18.4\n", "! pip install -U numpy==1.18.4\n",
"! pip install -U tensorboard==2.2.1" "! pip install -U tensorboard==2.2.1"
@ -29,7 +29,7 @@
"from torch.utils.data import Dataset\n", "from torch.utils.data import Dataset\n",
"from torch.utils.tensorboard import SummaryWriter\n", "from torch.utils.tensorboard import SummaryWriter\n",
"\n", "\n",
"from trains import Task" "from clearml import Task"
] ]
}, },
{ {
@ -42,7 +42,7 @@
"logger = task.get_logger()\n", "logger = task.get_logger()\n",
"configuration_dict = {'data_task_id': 'b605d76398f941e69fc91b43420151d2', \n", "configuration_dict = {'data_task_id': 'b605d76398f941e69fc91b43420151d2', \n",
" 'number_of_epochs': 15, 'batch_size': 100, 'dropout': 0.3, 'base_lr': 0.1}\n", " 'number_of_epochs': 15, 'batch_size': 100, 'dropout': 0.3, 'base_lr': 0.1}\n",
"configuration_dict = task.connect(configuration_dict) # enabling configuration override by trains\n", "configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml\n",
"print(configuration_dict) # printing actual configuration (after override in remote mode)" "print(configuration_dict) # printing actual configuration (after override in remote mode)"
] ]
}, },

6
examples/frameworks/pytorch/notebooks/text/text_classification_AG_NEWS.ipynb
View File

@ -10,7 +10,7 @@
"! pip install -U torch==1.5.0\n", "! pip install -U torch==1.5.0\n",
"! pip install -U torchtext==0.6.0\n", "! pip install -U torchtext==0.6.0\n",
"! pip install -U matplotlib==3.2.1\n", "! pip install -U matplotlib==3.2.1\n",
"! pip install -U trains>=0.15.0\n", "! pip install -U clearml>=0.15.0\n",
"! pip install -U tensorboard==2.2.1" "! pip install -U tensorboard==2.2.1"
] ]
}, },
@ -28,7 +28,7 @@
"from torchtext.datasets import text_classification\n", "from torchtext.datasets import text_classification\n",
"from torch.utils.tensorboard import SummaryWriter\n", "from torch.utils.tensorboard import SummaryWriter\n",
"\n", "\n",
"from trains import Task\n", "from clearml import Task\n",
"\n", "\n",
"%matplotlib inline" "%matplotlib inline"
] ]
@ -41,7 +41,7 @@
"source": [ "source": [
"task = Task.init(project_name='Text Example', task_name='text classifier')\n", "task = Task.init(project_name='Text Example', task_name='text classifier')\n",
"configuration_dict = {'number_of_epochs': 6, 'batch_size': 16, 'ngrams': 2, 'base_lr': 1.0}\n", "configuration_dict = {'number_of_epochs': 6, 'batch_size': 16, 'ngrams': 2, 'base_lr': 1.0}\n",
"configuration_dict = task.connect(configuration_dict) # enabling configuration override by trains\n", "configuration_dict = task.connect(configuration_dict) # enabling configuration override by clearml\n",
"print(configuration_dict) # printing actual configuration (after override in remote mode)" "print(configuration_dict) # printing actual configuration (after override in remote mode)"
] ]
}, },

26
examples/reporting/Allegro_Trains_logging_example.ipynb
View File

@ -7,13 +7,13 @@
"id": "RZiRah3QiR_G" "id": "RZiRah3QiR_G"
}, },
"source": [ "source": [
"# Allegro Trains logging example\n", "# Allegro ClearML logging example\n",
"\n", "\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allegroai/trains/blob/master/examples/reporting/Allegro_Trains_logging_example.ipynb)\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allegroai/clearml/blob/master/examples/reporting/Allegro_Trains_logging_example.ipynb)\n",
"\n", "\n",
"This example introduces Trains [Logger](https://allegro.ai/docs/logger.html) functionality. Logger is the Trains console log and metric interface.\n", "This example introduces ClearML [Logger](https://allegro.ai/docs/logger.html) functionality. Logger is the ClearML console log and metric interface.\n",
"\n", "\n",
"You can find more reporting examples [here](https://github.com/allegroai/trains/tree/master/examples/reporting)." "You can find more reporting examples [here](https://github.com/allegroai/clearml/tree/master/examples/reporting)."
] ]
}, },
{ {
@ -26,7 +26,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"!pip install trains\n", "!pip install clearml\n",
"!pip install numpy" "!pip install numpy"
] ]
}, },
@ -40,7 +40,7 @@
"### Create a new Task\n", "### Create a new Task\n",
"Create a new Task and get a Logger object for the Task.\n", "Create a new Task and get a Logger object for the Task.\n",
"\n", "\n",
"To create a new Task object, call the `Task.init` method providing it with `project_name` (the project name for the experiment) and `task_name` (the name of the experiment). When `Task.init` executes, a link to the Web UI Results page for the newly generated Task will be printed, and the Task will be updated in real time in the Trains demo server.\n", "To create a new Task object, call the `Task.init` method providing it with `project_name` (the project name for the experiment) and `task_name` (the name of the experiment). When `Task.init` executes, a link to the Web UI Results page for the newly generated Task will be printed, and the Task will be updated in real time in the ClearML demo server.\n",
"\n", "\n",
"You can read about the `Task` class in the docs [here](https://allegro.ai/docs/task.html).\n", "You can read about the `Task` class in the docs [here](https://allegro.ai/docs/task.html).\n",
"\n", "\n",
@ -59,7 +59,7 @@
"source": [ "source": [
"import numpy as np\n", "import numpy as np\n",
"\n", "\n",
"from trains import Task\n", "from clearml import Task\n",
"\n", "\n",
"# Start a new task\n", "# Start a new task\n",
"task = Task.init(project_name=\"Colab notebooks\", task_name=\"Explicit Logging\")\n", "task = Task.init(project_name=\"Colab notebooks\", task_name=\"Explicit Logging\")\n",
@ -77,7 +77,7 @@
}, },
"source": [ "source": [
"### Explicit scalar logging\n", "### Explicit scalar logging\n",
"Use the [Logger.report_scalar](https://allegro.ai/docs/logger.html#trains.logger.Logger.report_scalar) method to explicitly log scalars. Scalar plots appear in the Web UI, Results > Scalars tab." "Use the [Logger.report_scalar](https://allegro.ai/docs/logger.html#clearml.logger.Logger.report_scalar) method to explicitly log scalars. Scalar plots appear in the Web UI, Results > Scalars tab."
] ]
}, },
{ {
@ -241,7 +241,7 @@
}, },
"outputs": [], "outputs": [],
"source": [ "source": [
"from trains.storage import StorageManager\n", "from clearml.storage import StorageManager\n",
"image_local_copy = StorageManager.get_local_copy(\n", "image_local_copy = StorageManager.get_local_copy(\n",
" remote_url=\"https://pytorch.org/tutorials/_static/img/neural-style/picasso.jpg\",\n", " remote_url=\"https://pytorch.org/tutorials/_static/img/neural-style/picasso.jpg\",\n",
" name=\"picasso.jpg\"\n", " name=\"picasso.jpg\"\n",
@ -273,7 +273,7 @@
"source": [ "source": [
"#### Report images and media\n", "#### Report images and media\n",
"\n", "\n",
"Use [Logger.report_image](https://allegro.ai/docs/logger.html?highlight=report_image#trains.logger.Logger.report_image) and [Logger.report_media](https://allegro.ai/docs/logger.html?highlight=report_media#trains.logger.Logger.report_media) to report the downloaded samples. The debug samples appear in the Results > Debug Samples tab." "Use [Logger.report_image](https://allegro.ai/docs/logger.html?highlight=report_image#clearml.logger.Logger.report_image) and [Logger.report_media](https://allegro.ai/docs/logger.html?highlight=report_media#clearml.logger.Logger.report_media) to report the downloaded samples. The debug samples appear in the Results > Debug Samples tab."
] ]
}, },
{ {
@ -316,7 +316,7 @@
}, },
"source": [ "source": [
"### Explicit text logging\n", "### Explicit text logging\n",
"Use [Logger.report_text](https://allegro.ai/docs/logger.html?highlight=report_text#trains.logger.Logger.report_text) to log text message. They appear in Results > Log." "Use [Logger.report_text](https://allegro.ai/docs/logger.html?highlight=report_text#clearml.logger.Logger.report_text) to log text message. They appear in Results > Log."
] ]
}, },
{ {
@ -344,7 +344,7 @@
"\n", "\n",
"Reports are flushed in the background every couple of seconds, and at the end of the process execution.\n", "Reports are flushed in the background every couple of seconds, and at the end of the process execution.\n",
"\n", "\n",
"Or, flush the Logger by calling [Logger.flush](https://allegro.ai/docs/logger.html?highlight=report_text#trains.logger.Logger.flush)." "Or, flush the Logger by calling [Logger.flush](https://allegro.ai/docs/logger.html?highlight=report_text#clearml.logger.Logger.flush)."
] ]
}, },
{ {
@ -364,7 +364,7 @@
"metadata": { "metadata": {
"colab": { "colab": {
"collapsed_sections": [], "collapsed_sections": [],
"name": "Allegro Trains logging example.ipynb", "name": "Allegro ClearML logging example.ipynb",
"provenance": [] "provenance": []
}, },
"kernelspec": { "kernelspec": {

1
examples/reporting/manual_matplotlib_reporting.py
View File

@ -1 +0,0 @@
matplotlib_manual_reporting.py