---
title: Version 0.16
---
:::important
**Trains** is now **ClearML**.
:::


### Trains 0.16.4

**Features**

- Add Hydra support (<a href="https://github.com/allegroai/trains/issues/219" target="_blank">GitHub trains Issue 219</a>).
- Add cifar ignite example (<a href="https://github.com/allegroai/trains/issues/237" target="_blank">GitHub trains Issue 237</a>).
- Add auto extraction of `tar.gz` files when using `StorageManager` (<a href="https://github.com/allegroai/trains/issues/237" target="_blank">GitHub trains Issue 237</a>).
- Add `Task.init()` argument `auto_connect_streams` controlling stdout / stderr / logging capture (<a href="https://github.com/allegroai/trains/issues/181" target="_blank">GitHub trains Issue 181</a>).
- Add carriage return flush support using the `sdk.development.worker.console_cr_flush_period` configuration setting (<a href="https://github.com/allegroai/trains/issues/181" target="_blank">GitHub trains Issue 181</a>).
- Add `Task.create_function_task()` to allow creating a new task, using a function and arguments, to be executed remotely (<a href="https://github.com/allegroai/trains/issues/230" target="_blank">GitHub trains Issue 230</a>).
- Allow disabling SSL certificates verification using `Task.setup_upload()` argument `verify` or AWS S3 bucket configuration `verify` property (<a href="https://github.com/allegroai/trains/issues/256" target="_blank">GitHub trains Issue 256</a>).
- Add `StorageManager.get_files_server()`.
- Add `Task.get_project_id()` using project name.
- Add `project_name` argument to `Task.set_project()`.
- Add `Task.connect()` support for class / instance objects.
- Add `Task get_configuration_object()` and `Task.set_configuration_object()` for easier automation.
- Improve Auto-Scaler - allow extra configurations, key name and security group are now optional, defaults using empty strings.
- Use a built-in matplotlib convertor.
- Add reporting text as debug sample example.

**Bug Fixes**

- Fix Optuna HPO parameter serializing (<a href="https://github.com/allegroai/trains/issues/254" target="_blank">GitHub trains Issue 254</a>).
- Fix connect dictionary `''` cast to `None` (<a href="https://github.com/allegroai/trains/issues/258" target="_blank">GitHub trains Issue 258</a>).
- Fix lightgbm binding keyword argument issue (<a href="https://github.com/allegroai/trains/issues/251" target="_blank">GitHub trains Issue 251</a>).
- Fix artifact preview if artifact body is remote URI (<a href="https://github.com/allegroai/trains/issues/239" target="_blank">GitHub trains Issue 239</a>).
- Fix infinite recursion in `StorageManager` upload (<a href="https://github.com/allegroai/trains/issues/253" target="_blank">GitHub trains Issue 253</a>).
- Fix keras reusing model object only if the filename is the same (<a href="https://github.com/allegroai/trains/issues/252" target="_blank">GitHub trains Issue 252</a>).
- Fix running remotely with no configuration should not crash but output a warning (<a href="https://github.com/allegroai/trains/issues/243" target="_blank">GitHub trains Issue 243</a>).
- Fix matplotlib 3.3.3 support:
    - Fix global figure enumeration.
    - Fix binding without a title reported a single plot (`untitled 00`) instead of increasing the counter.
- Fix Python 2.7 / 3.5 support.
- Fix quote issue when reporting debug images.
- Fix replace quote safe characters in upload file to include `;=@$`.
- Fix `at_exit` called from another process should be ignored.
- Fix `Task.set_tags()` for completed / published tasks.
- Fix `Task.add_tags()` not working when running remotely.
- Fix `Task.set_user_properties()` docstring and interface.
- Fix preview with JSON (dict) artifacts did not store the artifact.
- Fix `Logger.report_text()` on task created using `Task.create()` was not supported.
- Fix initialization for torch: only call torch `get_worker_info` if torch was loaded.
- Fix flush (wait) on auxiliary task (obtained using` Task.get_task()`) should wait on all upload events.
- Fix server was not updated with the defaults from the code when running remotely and configuration section is missing.
- Fix connect dict containing `None` default values, blocked the remote execution from passing string instead of None.
- Fix `Task.upload_artifact()` argument `delete_after_upload=True` used in conjunction with `wait_for_upload=True` was not supported.


### Trains 0.16.3

**Features**

* Add LightGBM support.
* Add initial Hydra support (<a href="https://github.com/allegroai/trains/issues/219" target="_blank">GitHub trains Issue 219</a>).
* Add synchronous support for `Task.upload_artifact()`  (<a href="https://github.com/allegroai/trains/issues/231" target="_blank">GitHub trains Issue 231</a>).
* Add `sdk.development.store_code_diff_from_remote` (default `false`) to store diff from remote HEAD instead of local HEAD  (<a href="https://github.com/allegroai/trains/issues/222" target="_blank">GitHub trains Issue 222</a>).
* Add `sdk.development.detect_with_conda_freeze` (default `true`) for full conda freeze (requires trains-agent >= 16.2).
* Add user properties support in Task object.
* Add `Logger.report_table()` support for table as list of lists.
* Add support to split DAG and Table in pipeline DAG plot. Pipeline DAG single nodes are now round circles below the DAG graph..
* Add Pipeline / Optimization can be attached to any Task (not just the current task).
* Add `force_download` flag to `StorageManager.get_local_copy()`.
* Add control over the artifact preview using `Task.upload_artifact()` `preview` argument.
* Add `Logger.report_matplotlib_figure()` with examples.
* Add `Task.set_task_type()`.
* Improve AWS auto-scaler:
  * Add key pair and security groups support.
  * Add multi-line support for both extra bash script and extra `trains.conf` data.
* Update examples.

**Bug Fixes**

* Fix `Task.update_output_model()` wrong argument order  (<a href="https://github.com/allegroai/trains/issues/220" target="_blank">GitHub trains Issue 220</a>).
* Fix initializing task on argparse parse in remote mode. Do not call `Task.init()` to avoid auto connect, use `Task.get_task()` instead.
* Fix detected task cwd outside of repository root folder.
* Fix `Task.connect(dict)` to place non-existing entries on the section name instead of General.
* Fix `Task.clone()` support for trains-server < 0.16.
* Fix `StorageManager` cache extract zipped artifacts. Use modified time instead of access time for cached files.
* Fix diff command output was stripped.
* Make sure local packages with multi-files are marked as `package`.
* Fix `Task.set_base_docker()` should be skipped when running remotely.
* Fix ArgParser binding handling of string argument with boolean default value (affects PyTorch Lightning integration).
* When using `detect_with_pip_freeze` make sure that `package @ file://` lines are replaced with `package==x.y.z` as local file will probably not be available.
* Fix git packages to new pip standard `package @ git+`.
* Improve conda package naming `_` and `-` support.
* Do not add specific setuptools version to requirements (pip can't install it anyway).
* Fix image URL quoting when uploading from a file path.


### Trains 0.16.2

**Features**

* Add `Task.set_resource_monitor_iteration_timeout()` to set ResourceMonitor iteration wait duration timeout (<a href="https://github.com/allegroai/trains/issues/208" target="_blank">GitHub trains Issue 208</a>).
* Add PyTorch Lightning save/restore model binding (<a href="https://github.com/allegroai/trains/issues/212" target="_blank">GitHub trains Issue 212</a>).
* Add `git diff` for repository submodule (requires git 2.14 or above).
* Add `TrainsJob.is_completed()` and `TrainsJob.is_aborted()`.
* Add `Task.logger` property.
* Add Pipeline Controller automation and example (see [here](https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_tasks.py)).
* Add improved trace filtering capabilities in `trains.debugging.trace.trace_trains()`.
* Add default help per argument (if not provided) in ArgParser binding.
* Deprecate `Task.reporter`.
* Update PyTorch example.
* Remove warning on skipped auto-magic model logging (<a href="https://github.com/allegroai/trains/issues/206" target="_blank">GitHub trains Issue 206</a>).
* Support Keras restructuring for Network, Model and Sequential.
* Update autokeras requirements according to [https://github.com/keras-team/autokeras#installation](https://github.com/keras-team/autokeras#installation).

**Bug Fixes**

* Fix joblib auto logging models failing on compressed streams (<a href="https://github.com/allegroai/trains/issues/203" target="_blank">GitHub trains Issue 203</a>).
* Fix sending empty reports (<a href="https://github.com/allegroai/trains/issues/205" target="_blank">GitHub trains Issue 205</a>).
* Fix scatter2d sub-sampling and rounding.
* Fix plots reporting:
    * `NaN` representation (matplotlib conversion).
    * Limit the number of digits in a plot to reduce plot size (using `sdk.metrics.plot_max_num_digits` configuration value).
* Fix `Task.wait_for_status()` to reload after it ends.
* Fix thread wait Ctrl-C interrupt did not exit process.
* Improve Windows support for installed packages analysis.
* Fix auto model logging using relative path.
* Fix Hyperparameter Optimization example.
* Fix `Task.clone()` when working with TrainsServer < 0.16.0.
* Fix pandas artifact handling.
* Avoid adding `unnamed:0` column.
* Return original pandas object.
* Fix `TrainsJob` hyper-params overriding order was not guaranteed.
* Fix ArgParse auto-connect to support default function type.


### Trains 0.16.1

**Features**

* Enhance HyperParameter optimizer.

**Bug Fixes**

* Fix typing dependency for `Python<3.5` ([GitHub trains Issue 184](https://github.com/allegroai/trains/issues/184)).
* Fix git+https requirements handling, resolve top_level.txt package name (kerastuner from git was not detected).
* Fix `Task.get_reported_console_output()` for new Trains Server API v2.9.
* Fix cache handling for different partitions / drives / devices.
* Disable offline mode when running remotely (i.e. executed by Trains Agent).
* Fix artifact upload to only use file stream when not uploading a locally stored file (multipart upload is not supported on stream upload) (<a href="https://github.com/allegroai/trains/issues/189" target="_blank">GitHub trains Issue 189</a>).
* Fix double-escaped model design text when connecting OutputModel.


### Trains 0.16.0

**Features**

* Add continuing of previously executed experiments. Add `Task.init()` argument `continue_last_task` to continue a previously used Task ([GitHub Issue #160](https://github.com/allegroai/trains/issues/160)).
* Allow Task editing / creation from code. `Task.export_task/import_task/update_task()` ([GitHub Issue #128](https://github.com/allegroai/trains/issues/128)).
* Add offline mode. Use `Task.set_offline()` and `Task.import_offline_session()`:
  * Support setting offline mode via `TRAINS_OFFLINE_MODE=1` environment variable.
  * Support setting offline API version via `TRAINS_OFFLINE_MODE=2.9` environment variable.
* Automatically pickle all objects uploaded as artifacts, `task.upload_artifact()` argument `auto_pickle=True` ([GitHub Issue #153](https://github.com/allegroai/trains/issues/153)).
* Add multiple sections / groups support for Task hyperparameters, using `Task.connect()`.
* Add multiple configurations (files) using `Task.connect_configuration()`.
* Allow enabling OS environment logging using the `sdk.development.log_os_environments` configuration parameter (complements the  `TRAINS_LOG_ENVIRONMENT` environment variable).
* Add Optuna support for hyperparameter optimization controller. `OptimizerOptuna` is now the default optimizer.
* Add initial Keras-Tuner support ([GitHub Issue keras-team/keras-tuner #334](https://github.com/keras-team/keras-tuner/issues/334)).
* Add automatic FastAI logging. It is disabled if Tensorboard is loaded (assuming TensorBoardLogger will be used).
* Support Tensorboard text logging (`add_text()`) as debug samples (`.txt` files), instead of as console output.
* Allow for more standard confusion matrix reporting.  `Logger.report_confusion_matrix()` argument `yaxis_reversed` (flips the confusion matrix if `True`, default `False`) ([GitHub Issue #165](https://github.com/allegroai/trains/issues/165)).
* Add support for Trains Server 0.16.0 (API v2.9 support).
* Allow disabling Trains update message from the log using the `TRAINS_SUPPRESS_UPDATE_MESSAGE` environment variable ([GitHub Issue #157](https://github.com/allegroai/trains/issues/157)).
* Add AWS EC2 Auto-Scaler service wizard and Service.
* Improved and updated examples:
  * Add Keras Tuner CIFAR10 example.
  * Add FastAI example.
  * Update PyTorch Jupyter notebook examples ([GitHub Issue #150](https://github.com/allegroai/trains/issues/150)).
* Support global requirements detection using `pip freeze` (set `sdk.development.detect_with_pip_freeze` configuration in `trains.conf`).
* Add `Task.get_projects()` to get all projects in the system, sorted by last update time.

**Bug Fixes**

* Fix UTC to time stamp in comment ([GitHub Issue #152](https://github.com/allegroai/trains/issues/152)).
* Fix and enhance GPU monitoring:
  * Fix GPU stats on Windows machines ([GitHub Issue #177](https://github.com/allegroai/trains/issues/177)).
  * More robust GPU monitoring ([GitHub Issue #170](https://github.com/allegroai/trains/issues/170)).
* Fix filename too long bug ([GitHub trains-server Issue #49](https://github.com/allegroai/trains-server/issues/49)).
* Fix TensorFlow image logging to allow images with no width / height / color metadata ([GitHub Issue #182](https://github.com/allegroai/trains/issues/182)).
* Fix multiprocessing Pool throw exception in pool hangs execution. Call original signal handler and re-flush `stdout`.
* Fix `plotly` support for `matplotlib` 3.3.
* Add Python 2.7 support for `get_current_thread_id()`.
* Update examples requirements.
* Fix and improve signal handling.
* Fix Tensorboard 2D convolution histogram, improve histogram accuracy on very small histograms.
* Fix auto logging multiple argparse calls before `Task.init()`.
* Limit experiment Git diff logging to 500Kb. If larger than 500Kb, diff section will contain a warning and entire diff will be uploaded as an artifact named `auxiliary_git_dif`.
* Fix requirements detection:
  * Fix Trains installed from `git+`.
  * Fix when Trains is not directly imported.
  * Fix multiple `-e` packages were not detected (only the first one).
  * Fix running with Trains in `PYTHONPATH` resulted in double entry of trains.
* Fix `Task.set_base_docker()` on main task to do nothing when running remotely.