ClearML/clearml-docs
1
0
Fork 0
mirror of https://github.com/clearml/clearml-docs synced 2025-06-26 18:17:44 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Refactor Release Notes (#938)

Browse Source
This commit is contained in:
pollfly
2024-09-26 16:09:25 +03:00
committed by GitHub
parent eb36f48413
commit 8249768619
89 changed files with 3127 additions and 2660 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
docs/release_notes/sdk/open_source/ver_0_10.md Normal file
View File

@@ -0,0 +1,123 @@
---
title: Version 0.10
---
:::important
**Trains** is now **ClearML**.
:::
### Trains 0.10.7
**Features and Bug Fixes**
* Artifacts support.
* Removed apache-libcloud from requirements.
* `trains-init` now verifies credentials against the trains-server installation.
### Trains 0.10.6
**Features and Bug Fixes**
* Fix broken (v0.10.5) Keras Binding support.
### Trains 0.10.5
**Features and Bug Fixes**
* Add GPU monitoring support (add gpustat package to extras_require).
* Install with GPU monitoring support:
```
pip install trains[gpu]
```
* Move all cloud storage package requirements to extras_require. Install with specific cloud provider support:
* Microsoft Azure support:
```
pip install trains[azure]
```
* Google Storage support:
```
pip install trains[gs]
```
* Amazon S3 support:
```
pip install trains[s3]
```
* Combine Cloud support with GPU monitoring. For example, install S3 and GPU using the following command:
```
pip install trains[s3,gpu]
```
* Improve stability with intermittent network connection.
* Support upgrading trains-server while running training jobs without losing log data.
### Trains 0.10.4
**Features and Bug Fixes**
* Replace opencv-python with the more standard Pillow package.
* Improve matplotlib support (custom axis ticks).
* Improve Python package detection.
### Trains 0.10.3
**Features and Bug Fixes**
* Add scikit-learn support (load/store using joblib) ([GitHub Issue #20](https://github.com/allegroai/trains/issues/20)).
* Add xgboost support ([GitHub Issue #10](https://github.com/allegroai/trains/issues/10)).
* Add loguru support ([GitHub Issue #29](https://github.com/allegroai/trains/issues/29)).
* Add sub-domain support [trains.conf](https://github.com/allegroai/trains/blob/master/docs/trains.conf#L3) ([GitHub Issue #27](https://github.com/allegroai/trains/issues/27)).
* Fix sub-process support.
* Fix multiple TensorBoard writers ([GitHub Issue #26](https://github.com/allegroai/trains/issues/26)).
### Trains 0.10.2
**Features and Bug Fixes**
* Add Matplotlib SVG support.
* Add Seaborn support.
* Add TRAINS_LOG_ENVIRONMENT environment logging ([GitHub trains Issue 17](https://github.com/allegroai/trains/issues/17#issuecomment-507398767)).
* Add Microsoft Azure notebook support.
* Add Google Colab support.
* Fix TensorBoard RGB channel order.
### Trains 0.10.1
**Features and Bug Fixes**
* Fix Jenkins CI/CD support.
### Trains 0.10.0
* Experiment code execution detection
* Automatically create package requirements section (including used versions).
* Automatically detect and store source code uncommitted changes.
* Improve Jupyter Notebook support
* Automatically convert notebook to Python script (stored under uncommitted changes).
* Automatically update used packages in Jupyter Notebook (including used versions).
* Add resource monitoring to experiment metrics
* Sample every 500ms, averaged over 30 seconds.
* CPU, network, I/O, memory, and other resources.
* For GPU support please install gpustat.
(currently not part of the requirements due to gpustat compatibility issues with Windows).
```
pip install gpustat
```
* Automatically stop inactive experiments (default: 2 hours)
* Improve visibility
* Finer status definitions: Identify successful completion vs. user aborted.
* Experiment plot comparison: Ensure different colors for different experiments.
* Parse newline character in experiment description.
* Show experiment start time in table display.
* Add vertical guide in scalar plots.
* Move hyperparameters to the designated tab.
* "Admin" section now named "Profile".

53
docs/release_notes/sdk/open_source/ver_0_11.md Normal file
View File

@@ -0,0 +1,53 @@
---
title: Version 0.11
---
:::important
**Trains** is now **ClearML**.
:::
### Trains 0.11.3
**Features and Bug Fixes**
* Resource-Monitor will only monitor active GPU devices based on environment variables: `NVIDIA_VISIBLE_DEVICES` or `CUDA_VISIBLE_DEVICES`.
* Fix issue ([GitHub Issue #48](https://github.com/allegroai/trains/issues/48)).
### Trains 0.11.2
**Features and Bug Fixes**
* Fix Python 2.7 support.
* Improve sample code Windows support.
### Trains 0.11.1
**Features and Bug Fixes**
* Embed GPU Monitoring into **Trains** (removed gpustat dependency).
* Add initial support for TensorFlow v2.0 (tested with v2.0.0rc1).
* Add artifact upload retry on network errors (default: 3).
* Suppress urllib3 retry warnings.
* Fix Matplotlib support with Agg backend (multiple plot windows caused repeated graphs to be sent).
* Fix support for tuples in hyperparameters.
* Fix multi processing issues with different task types.
### Trains 0.11.0
**Features and Bug Fixes**
* Full artifacts support (supported by trains-server >= 0.11.0).
* Artifacts include, Pandas.DataFrame, Numpy, PIL Image, local files, and local folder / wildcard ([example](https://github.com/allegroai/trains/blob/master/examples/reporting/artifacts.py)).
* Artifacts support for folder / wildcard, selected files will be zipped and uploaded.
* Resource monitoring, remove sensor reading failure warnings.
**Breaking Changes**
* Logger `info`/`error`/`warning`/`console` functions were removed, use `Logger.report_text` (or Python logging or print instead).
* TensorBoard scalars are not grouped into one graph, but are stored on individual graphs (to match TensorBoard behavior).
To restore previous behavior call: `Logger.tensorboard_auto_group_scalars(group_scalars=True)`.

55
docs/release_notes/sdk/open_source/ver_0_12.md Normal file
View File

@@ -0,0 +1,55 @@
---
title: Version 0.12
---
:::important
**Trains** is now **ClearML**.
:::
### Trains 0.12.2
**Features and Bug Fixes**
* Add `default_output_uri` feature ([GitHub Issue #57](https://github.com/allegroai/trains/issues/57)).
* Improve `trains-init` configuration wizard.
* Improve argparser binding.
* Fix artifact support in data pipeline ([GitHub Issue #63](https://github.com/allegroai/trains/issues/63)).
* Fix threading issue while querying multiple experiments ([GitHub Issue #64](https://github.com/allegroai/trains/issues/64)).
* Fix uploading large files over slow HTTP connections.
* Fix support for Git versions < 2.
:::info Breaking Changes
Do not reuse an experiment with artifacts.
:::
### Trains 0.12.1
**Features and Bug Fixes**
* Add `Task.connect_configuration`, connect configuration file (or configuration dictionary including nesting support).
* Add `Task.connect_label_enumeration`, store model detection classes together with the model and experiment.
* Edit and override configuration file (or dictionary) content when executing with [trains-agent](https://github.com/allegroai/trains-agent).
* `Task.connect` automatically supports nested dictionaries (e.g. param / nested / key), including overriding values when
running with [trains-agent](https://github.com/allegroai/trains-agent).
* Add `Artifact.get`, download and load an artifact.
* Add `Task.get_tasks`, retrieve previous experiments, get metrics, rename, and other uses.
* Improve Windows support.
* Improve Minio support.
* Fix Python2 support.
* Fix Issue ([GitHub Issue #56](https://github.com/allegroai/trains/issues/56)).
### Trains 0.12.0
**Features and Bug Fixes**
* TRAINS Agent support (Full DevOps support).
* Add automation and orchestration ([examples/automation](https://github.com/allegroai/trains/tree/master/examples/automation)).
Supports TRAINS-server v0.12 or above.
* Add Logger x/y/z axis title for: report_surface / report_confusion_matrix / report_scatter3d / report_scatter2d / report_histogram.
* Add support for TensorFlow 2.0.
* Embed pyhocon into package.
* Fix artifacts support on Windows.
* Fix example code Windows support.

75
docs/release_notes/sdk/open_source/ver_0_13.md Normal file
View File

@@ -0,0 +1,75 @@
---
title: Version 0.13
---
:::important
**Trains** is now **ClearML**.
:::
### Trains 0.13.3
**Features and Bug Fixes**
* Add a binding for `tensorboard.summarywriter.addscalars`
* Add the `tensorboard_single_series_per_graph` method, which supports separate plots for each TensorBoard scalar.
* Add the `Task.set_base_docker` and `Task.get_base_docker` methods for the base Docker image used by **Trains Agent**.
* Add support for the standard OS environment variables to obtain default credentials for:
* AWS: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_DEFAULT_REGION`.
* Azure Storage: `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_KEY`.
* Google Cloud Storage: `GOOGLE_APPLICATION_CREDENTIALS`.
* Add the `Task.get_parameters_as_dict` and `Task.set_parameters_as_dict` methods supporting get / set of parameters from referenced Tasks (use the `Task.get_task` to get a reference).
* Make sure `Task.connect` always returns the connected instance passed to it.
* `tensorflow_gpu` takes precedence over `tensorflow` when Trains detects installed packages to record experiment dependencies.
* Remove title and series naming restrictions (allow `$` and `.`) when reporting metrics.
* Fix incorrect printouts in initialization wizard and upgrade notifications.
* Fix debug images URL for uploaded files with `%` in their name.
### Trains 0.13.2
**Features and Bug Fixes**
* Allow reporting a pre-uploaded image url in `Logger.report_image()`
using the `Logger.report_image.params.url` parameter.
* Add support for Git repositories without a `.git` suffix, for example Azure Repos.
* Improve conda support.
* Improve hyperparameters [argparser](https://docs.python.org/3/library/argparse.html) integration.
* Fix [savefig](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.savefig.html) patching in matplotlib binding.
* Fix logs, events and Jupyter Notebook flushing on exit.
### Trains 0.13.1
**Features and Bug Fixes**
* Add support for `pyplot.savefig` and `pylab.savefig` in matplotlib binding.
* Add support for SageMaker.
* Improve configuration wizard.
* Try to make sure TensorBoard is available when using torch.
* Do not store keras model network design if it cannot be serialized ([GitHub Issue #72](https://github.com/allegroai/trains/issues/72)).
* Fix matplotlib binding support.
### Trains 0.13.0
**Features and Bug Fixes**
* Add support for [trains-server](https://github.com/allegroai/trains-server) v0.13.0.
* Add support for nested (non-main) tasks.
* Add warning when automatic argument parser binding cannot be turned off.
* Add `Task.upload_artifact` support for external URLs (pre-uploaded).
* Add support for special characters in hyperparameter keys (white-spaces, `.` and `$`) ([GitHub Issue #69](https://github.com/allegroai/trains/issues/69)).
* Add support for PyTorch `.pt` model files.
* Calculate data-audit artifact uniqueness by user-criteria ([GitHub Issue #45](https://github.com/allegroai/trains/issues/45)).
* Use an environment variable for setting a default docker image ([GitHub Issue #58](https://github.com/allegroai/trains/issues/58)).
* Improve `trains-init` configuration wizard.
* Update examples for new joblib versions.
* Update jupyter example to TensorFlow 2.
* Fix task clone to copy only input artifacts.
* Fix matplotlib import binding when using `Agg` backend.
* Fix `ProxyDictPreWrite` and `ProxyDictPostWrite` so they can be pickled correctly ([GitHub Issue #72](https://github.com/allegroai/trains/issues/72)).
* Fix requests issue in Python 2.7 that can cause a deadlock when importing netrc.
* Fix argparser binding sub-parser and type casting support ([GitHub Issue #74](https://github.com/allegroai/trains/issues/74)).
* Fix argparser binding Python 2.7 unicode handling.
* Fix unsynced connected hyperparameters overridden during remote execution.

99
docs/release_notes/sdk/open_source/ver_0_14.md Normal file
View File

@@ -0,0 +1,99 @@
---
title: Version 0.14
---
:::important
**Trains** is now **ClearML**.
:::
### Trains 0.14.3
**Features**
* Add the `Logger.report_histogram` method `mode` parameter.
The values are `group` (the default), `stack`, and `relative`.
* Add [PEP610](https://www.python.org/dev/peps/pep-0610/) support for the new `pip` version 20.1 and newer Git reference
feature (see [pypa/pip#609](https://github.com/pypa/pip/issues/609)) ([GitHub Issue #62](https://github.com/allegroai/trains/issues/62)).
* Improve support for PyTorch Ignite integration.
* Support `#` and `?` in uploaded image file names.
**Bug Fixes**
* Support `ArgumentParser.parse_arg` use before initializing a Task when `auto_connect_arg_parser=False`.
* Fix `upload_object_via_stream` in the Azure storage driver.
* Fix the `Task.get_tasks` method return type.
### Trains 0.14.2
**Features**
* Add media (audio) support for both `Logger` and TensorBoard binding using `Logger.report_media` ([GitHub Issue #120](https://github.com/allegroai/trains/issues/120)).
* Add [autokeras example](https://github.com/allegroai/trains/blob/master/examples/frameworks/autokeras/autokeras_imdb_example.py) ([GitHub Issue #125](https://github.com/allegroai/trains/issues/125)).
* Add caching of extracted zip artifacts.
* Add `Task.get_tasks` filtering support.
* Add `trains.storage.StorageManager`.
* Add nicer stdout log flush.
* Add text logging to TF V1 example.
* Improve SSL retries and error handling.
* Improve shutdown behavior in case of a user abort (Ctrl-C).
* Improve repository and module detection:
* Improve resource monitor.
* Report memory usage for process (and sub-processes).
* Report GPU memory usage for process (and sub-processes).
* Stability
* Improve Jupyter support:
* Make sure `trains` is included in Jupyter requirements.
* Ignore IPython directives in converted Python script (like `%` and `!` lines).
* Update PyTorch / TensorboardX examples.
**Bug Fixes**
* Fix renaming / deleting model file right after saving will break asynchronous upload ([GitHub Issue #123](https://github.com/allegroai/trains/issues/123)).
* Fix shutdown behavior:
* Logger flush delay and termination.
* Repository / package detection termination.
* Threads termination.
* Fix text encoding `utf-8` and `pr_curve` broken in Tensorboard support.
* Fix single log request exceeds max packet size.
* Fix upload debug assets with series / title including `,` or `/` did not add iteration index to the filename.
* Fix `scikit-learn` module detection.
* Use UID in message if username cannot be retrieved.
* Fix cast in Tensorflow v2 example.
### Trains 0.14.1
**Features and Bug Fixes**
* Add OpenMPI / Slurm support, support `TRAINS_FORCE_MASTER_NODE` environment variable to allow forcefully specifying a node is master (not based on the MPI Rank).
* Add `Task.get_models` and `Task.models` for retrieving stored models for previously executed tasks
* Support multiple EventWriter in TensorFlow eager mode (TF 2.0+) ([GitHub Issue #26](https://github.com/allegroai/trains/issues/26)).
* Support reusing models.
* Fix TensorFlow not a number (`NaN`) and infinity (`Inf`)
values support ([GitHub Issue #118](https://github.com/allegroai/trains/issues/118)).
* Fix joblib binding.
* Fix metrics / files / models upload on shutdown.
* Fix support for Task init / close multiple times.
* Update `manual_reporting.py` example with `Logger.report_table` example.
* Deprecate `Task.set_model_config`, `Task.get_model_config_text`, and `Task.get_model_config_dict`.
### Trains 0.14.0
**Features and Bug Fixes**
* Add support for reporting tables (pandas.DataFrame, CSV files, URL to CSV file) using the `Logger.report_table` method.
* Add support for setting a Task's initial iteration to allow continuing previous runs using the `Task.set_initial_iteration` and `Task.get_initial_iteration` methods.
* Add multiprocess task support ([GitHub Issue #96](https://github.com/allegroai/trains/issues/96)).
* Add `trains-init` support for config file override using the `TRAINS_CONFIG_FILE` environment variable or `--file` command line argument ([GitHub Issue #105](https://github.com/allegroai/trains/issues/105)).
* Add initial slurm support.
* Allow disabling repository detection when calling `Task.init`.
* Add Jupyter Notebook server execution example.
* Update examples.
* Improve warning messages for storage errors and missing storage packages.
* Fix HTTP link quoting in stored links.

74
docs/release_notes/sdk/open_source/ver_0_15.md Normal file
View File

@@ -0,0 +1,74 @@
---
title: Version 0.15
---
:::important
**Trains** is now **ClearML**.
:::
### Trains 0.15.1
**Features**
* Add direct plotly figure reporting ([GitHub Issue #136](https://github.com/allegroai/trains/issues/136)).
* Improve matplotlib integration ([GitHub Issue #140](https://github.com/allegroai/trains/issues/140)).
* Add `Task.set_parent` ([GitHub Issue #144](https://github.com/allegroai/trains/issues/144)).
* Add `Task.wait_for_status` ([GitHub Issue #145](https://github.com/allegroai/trains/issues/145)).
* Enhance framework `save` / `load` binding callbacks ([pytorch/ignite Issue #1056](https://github.com/pytorch/ignite/issues/1056)).
* Add `TrainsJob.is_failed`.
* Add APIClient support using `trains.backend_api.session.APIClient`.
* Add URL tracking when using `StorageManager.get_local_copy` (log remote URL instead of local file copy).
* Improve and expand examples.
**Bug Fixes**
* Fix `pytorch_tensorboard.py` example ([GitHub Issue #148](https://github.com/allegroai/trains/issues/148)).
* Fix `ssh://user@domain@server.com/` in Git repository ([GitHub Issue #149](https://github.com/allegroai/trains/issues/149)).
* Fix do not call `make_deterministic` on import.
* Fix initialization wizard (allow at most two verification retries, then print error) ([GitHub trains-agent Issue #16](https://github.com/allegroai/trains-agent/issues/16)).
* Fix Google CoLab code / package detection.
* Improve pytorch-ignite integration.
* Fix TensorFlow version 2 and later histogram binding.
* Fix `Logger.tensorboard_single_series_per_graph`.
* Fix anonymous named models.
* Fix incorrect entry point detection when called from Trains wrapper (e.g. `TrainsLogger` in PyTorch Ignite / Lightning).
### Trains 0.15.0
**Features**
* Add automation support including hyperparameter optimization (see example [here](https://github.com/allegroai/clearml/blob/master/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py))
* `Task.init` `auto_connect_arg_parser` argument can accept a dictionary disabling specific keys from the argparser (Trains Slack channel [thread](https://clearml.slack.com/archives/CTK20V944/p1589355878206700))
* Allow `worker_id` override using `TRAINS_WORKER_NAME` environment variable (Trains Slack channel [thread](https://clearml.slack.com/archives/CTK20V944/p1589920581251600?thread_ts=1589920220.250100&cid=CTK20V944))
* Support layout configuration for plotly objects using `extra_layout` argument in all `Logger` reporting methods ([GitHub Issue #136](https://github.com/allegroai/trains/issues/136)).
* Add `Task.execute_remotely` to allow cloning and enqueuing a locally executed task (or stopping and re-enqueuing a remotely executed task) ([GitHub Issue #128](https://github.com/allegroai/trains/issues/128)).
* Add Parquet framework and model type.
* Support recursive model folder packaging.
* Add `Task.get_reported_console_output` and `Task.get_reported_scalars` to allow retrieval of reported output and scalar metrics.
* Add `Task.add_requirements` to force requirement package into "installed packages."
* Improve task reuse responsiveness.
* Add `raise_on_error` (default False) argument to `Model.get_local_copy` and `Artifact.get_local_copy` ([GitHub trains-agent Issue #17](https://github.com/allegroai/trains-agent/issues/17)).
* Support `Task.get_task` without project name (i.e. all projects)
* Support using the file calling `Task.init` as the task's script in case sys.argv doesn't point to a git repository
* Support detecting and remotely executing code running from a module (i.e. `-m module`)
* Add callback for framework `save` / `load` binding for better integration with pytorch / ignite ([GitHub pytorch/ignite Issue #1056](https://github.com/pytorch/ignite/issues/1056))
* Support new task types provided in Trains Server v0.15.0
* Add automation and distributed examples
* Upgrade default pip version to `<20.2`
**Bug Fixes**
* Fix `exact_match_regex` in case of empty pattern ([GitHub Issue #138](https://github.com/allegroai/trains/issues/138)).
* Address deprecation warning and newer `attrs` versions in MetricsEventAdapter ([GitHub Issue #134](https://github.com/allegroai/trains/issues/134)).
* Fix issues with plotly support (Trains Slack channel [thread](https://clearml.slack.com/archives/CTK20V944/p1589871253243600) and [thread](https://clearml.slack.com/archives/CTK20V944/p1589279340195000)).
* Fix default argument behavior to match argparse behavior.
* Fix `OutputModel` with `task=None` should use current task, if exists.
* Fix `Task.get_task()` to raise proper error on incorrect `task_id`.
* Fix `Task.enqueue()` to use an exact queue name match.
* Fix `NaN`, `Inf` and `-Inf` values display in reported table (not supported by JSON).
* Limit max requirement size to 0.5 MB.
* Fix issues with repository analysis.
* Fix `StorageManager` so it should only try to extract .zip files, `Model` should not auto extract package ([GitHub trains-agent Issue #17](https://github.com/allegroai/trains-agent/issues/17)).

203
docs/release_notes/sdk/open_source/ver_0_16.md Normal file
View File

@@ -0,0 +1,203 @@
---
title: Version 0.16
---
:::important
**Trains** is now **ClearML**.
:::
### Trains 0.16.4
**Features**
- Add Hydra support (<a href="https://github.com/allegroai/trains/issues/219" target="_blank">GitHub trains Issue 219</a>).
- Add cifar ignite example (<a href="https://github.com/allegroai/trains/issues/237" target="_blank">GitHub trains Issue 237</a>).
- Add auto extraction of `tar.gz` files when using `StorageManager` (<a href="https://github.com/allegroai/trains/issues/237" target="_blank">GitHub trains Issue 237</a>).
- Add `Task.init()` argument `auto_connect_streams` controlling stdout / stderr / logging capture (<a href="https://github.com/allegroai/trains/issues/181" target="_blank">GitHub trains Issue 181</a>).
- Add carriage return flush support using the `sdk.development.worker.console_cr_flush_period` configuration setting (<a href="https://github.com/allegroai/trains/issues/181" target="_blank">GitHub trains Issue 181</a>).
- Add `Task.create_function_task()` to allow creating a new task, using a function and arguments, to be executed remotely (<a href="https://github.com/allegroai/trains/issues/230" target="_blank">GitHub trains Issue 230</a>).
- Allow disabling SSL certificates verification using `Task.setup_upload()` argument `verify` or AWS S3 bucket configuration `verify` property (<a href="https://github.com/allegroai/trains/issues/256" target="_blank">GitHub trains Issue 256</a>).
- Add `StorageManager.get_files_server()`.
- Add `Task.get_project_id()` using project name.
- Add `project_name` argument to `Task.set_project()`.
- Add `Task.connect()` support for class / instance objects.
- Add `Task get_configuration_object()` and `Task.set_configuration_object()` for easier automation.
- Improve Auto-Scaler - allow extra configurations, key name and security group are now optional, defaults using empty strings.
- Use a built-in matplotlib convertor.
- Add reporting text as debug sample example.
**Bug Fixes**
- Fix Optuna HPO parameter serializing (<a href="https://github.com/allegroai/trains/issues/254" target="_blank">GitHub trains Issue 254</a>).
- Fix connect dictionary `''` cast to `None` (<a href="https://github.com/allegroai/trains/issues/258" target="_blank">GitHub trains Issue 258</a>).
- Fix lightgbm binding keyword argument issue (<a href="https://github.com/allegroai/trains/issues/251" target="_blank">GitHub trains Issue 251</a>).
- Fix artifact preview if artifact body is remote URI (<a href="https://github.com/allegroai/trains/issues/239" target="_blank">GitHub trains Issue 239</a>).
- Fix infinite recursion in `StorageManager` upload (<a href="https://github.com/allegroai/trains/issues/253" target="_blank">GitHub trains Issue 253</a>).
- Fix keras reusing model object only if the filename is the same (<a href="https://github.com/allegroai/trains/issues/252" target="_blank">GitHub trains Issue 252</a>).
- Fix running remotely with no configuration should not crash but output a warning (<a href="https://github.com/allegroai/trains/issues/243" target="_blank">GitHub trains Issue 243</a>).
- Fix matplotlib 3.3.3 support:
- Fix global figure enumeration.
- Fix binding without a title reported a single plot (`untitled 00`) instead of increasing the counter.
- Fix Python 2.7 / 3.5 support.
- Fix quote issue when reporting debug images.
- Fix replace quote safe characters in upload file to include `;=@$`.
- Fix `at_exit` called from another process should be ignored.
- Fix `Task.set_tags()` for completed / published tasks.
- Fix `Task.add_tags()` not working when running remotely.
- Fix `Task.set_user_properties()` docstring and interface.
- Fix preview with JSON (dict) artifacts did not store the artifact.
- Fix `Logger.report_text()` on task created using `Task.create()` was not supported.
- Fix initialization for torch: only call torch `get_worker_info` if torch was loaded.
- Fix flush (wait) on auxiliary task (obtained using` Task.get_task()`) should wait on all upload events.
- Fix server was not updated with the defaults from the code when running remotely and configuration section is missing.
- Fix connect dict containing `None` default values, blocked the remote execution from passing string instead of None.
- Fix `Task.upload_artifact()` argument `delete_after_upload=True` used in conjunction with `wait_for_upload=True` was not supported.
### Trains 0.16.3
**Features**
* Add LightGBM support.
* Add initial Hydra support (<a href="https://github.com/allegroai/trains/issues/219" target="_blank">GitHub trains Issue 219</a>).
* Add synchronous support for `Task.upload_artifact()` (<a href="https://github.com/allegroai/trains/issues/231" target="_blank">GitHub trains Issue 231</a>).
* Add `sdk.development.store_code_diff_from_remote` (default `false`) to store diff from remote HEAD instead of local HEAD (<a href="https://github.com/allegroai/trains/issues/222" target="_blank">GitHub trains Issue 222</a>).
* Add `sdk.development.detect_with_conda_freeze` (default `true`) for full conda freeze (requires trains-agent >= 16.2).
* Add user properties support in Task object.
* Add `Logger.report_table()` support for table as list of lists.
* Add support to split DAG and Table in pipeline DAG plot. Pipeline DAG single nodes are now round circles below the DAG graph..
* Add Pipeline / Optimization can be attached to any Task (not just the current task).
* Add `force_download` flag to `StorageManager.get_local_copy()`.
* Add control over the artifact preview using `Task.upload_artifact()` `preview` argument.
* Add `Logger.report_matplotlib_figure()` with examples.
* Add `Task.set_task_type()`.
* Improve AWS auto-scaler:
* Add key pair and security groups support.
* Add multi-line support for both extra bash script and extra `trains.conf` data.
* Update examples.
**Bug Fixes**
* Fix `Task.update_output_model()` wrong argument order (<a href="https://github.com/allegroai/trains/issues/220" target="_blank">GitHub trains Issue 220</a>).
* Fix initializing task on argparse parse in remote mode. Do not call `Task.init()` to avoid auto connect, use `Task.get_task()` instead.
* Fix detected task cwd outside of repository root folder.
* Fix `Task.connect(dict)` to place non-existing entries on the section name instead of General.
* Fix `Task.clone()` support for trains-server < 0.16.
* Fix `StorageManager` cache extract zipped artifacts. Use modified time instead of access time for cached files.
* Fix diff command output was stripped.
* Make sure local packages with multi-files are marked as `package`.
* Fix `Task.set_base_docker()` should be skipped when running remotely.
* Fix ArgParser binding handling of string argument with boolean default value (affects PyTorch Lightning integration).
* When using `detect_with_pip_freeze` make sure that `package @ file://` lines are replaced with `package==x.y.z` as local file will probably not be available.
* Fix git packages to new pip standard `package @ git+`.
* Improve conda package naming `_` and `-` support.
* Do not add specific setuptools version to requirements (pip can't install it anyway).
* Fix image URL quoting when uploading from a file path.
### Trains 0.16.2
**Features**
* Add `Task.set_resource_monitor_iteration_timeout()` to set ResourceMonitor iteration wait duration timeout (<a href="https://github.com/allegroai/trains/issues/208" target="_blank">GitHub trains Issue 208</a>).
* Add PyTorch Lightning save/restore model binding (<a href="https://github.com/allegroai/trains/issues/212" target="_blank">GitHub trains Issue 212</a>).
* Add `git diff` for repository submodule (requires git 2.14 or above).
* Add `TrainsJob.is_completed()` and `TrainsJob.is_aborted()`.
* Add `Task.logger` property.
* Add Pipeline Controller automation and example (see [here](https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_tasks.py)).
* Add improved trace filtering capabilities in `trains.debugging.trace.trace_trains()`.
* Add default help per argument (if not provided) in ArgParser binding.
* Deprecate `Task.reporter`.
* Update PyTorch example.
* Remove warning on skipped auto-magic model logging (<a href="https://github.com/allegroai/trains/issues/206" target="_blank">GitHub trains Issue 206</a>).
* Support Keras restructuring for Network, Model and Sequential.
* Update autokeras requirements according to [https://github.com/keras-team/autokeras#installation](https://github.com/keras-team/autokeras#installation).
**Bug Fixes**
* Fix joblib auto logging models failing on compressed streams (<a href="https://github.com/allegroai/trains/issues/203" target="_blank">GitHub trains Issue 203</a>).
* Fix sending empty reports (<a href="https://github.com/allegroai/trains/issues/205" target="_blank">GitHub trains Issue 205</a>).
* Fix scatter2d sub-sampling and rounding.
* Fix plots reporting:
* `NaN` representation (matplotlib conversion).
* Limit the number of digits in a plot to reduce plot size (using `sdk.metrics.plot_max_num_digits` configuration value).
* Fix `Task.wait_for_status()` to reload after it ends.
* Fix thread wait Ctrl-C interrupt did not exit process.
* Improve Windows support for installed packages analysis.
* Fix auto model logging using relative path.
* Fix Hyperparameter Optimization example.
* Fix `Task.clone()` when working with TrainsServer < 0.16.0.
* Fix pandas artifact handling.
* Avoid adding `unnamed:0` column.
* Return original pandas object.
* Fix `TrainsJob` hyper-params overriding order was not guaranteed.
* Fix ArgParse auto-connect to support default function type.
### Trains 0.16.1
**Features**
* Enhance HyperParameter optimizer.
**Bug Fixes**
* Fix typing dependency for `Python<3.5` ([GitHub trains Issue 184](https://github.com/allegroai/trains/issues/184)).
* Fix git+https requirements handling, resolve top_level.txt package name (kerastuner from git was not detected).
* Fix `Task.get_reported_console_output()` for new Trains Server API v2.9.
* Fix cache handling for different partitions / drives / devices.
* Disable offline mode when running remotely (i.e. executed by Trains Agent).
* Fix artifact upload to only use file stream when not uploading a locally stored file (multipart upload is not supported on stream upload) (<a href="https://github.com/allegroai/trains/issues/189" target="_blank">GitHub trains Issue 189</a>).
* Fix double-escaped model design text when connecting OutputModel.
### Trains 0.16.0
**Features**
* Add continuing of previously executed experiments. Add `Task.init()` argument `continue_last_task` to continue a previously used Task ([GitHub Issue #160](https://github.com/allegroai/trains/issues/160)).
* Allow Task editing / creation from code. `Task.export_task/import_task/update_task()` ([GitHub Issue #128](https://github.com/allegroai/trains/issues/128)).
* Add offline mode. Use `Task.set_offline()` and `Task.import_offline_session()`:
* Support setting offline mode via `TRAINS_OFFLINE_MODE=1` environment variable.
* Support setting offline API version via `TRAINS_OFFLINE_MODE=2.9` environment variable.
* Automatically pickle all objects uploaded as artifacts, `task.upload_artifact()` argument `auto_pickle=True` ([GitHub Issue #153](https://github.com/allegroai/trains/issues/153)).
* Add multiple sections / groups support for Task hyperparameters, using `Task.connect()`.
* Add multiple configurations (files) using `Task.connect_configuration()`.
* Allow enabling OS environment logging using the `sdk.development.log_os_environments` configuration parameter (complements the `TRAINS_LOG_ENVIRONMENT` environment variable).
* Add Optuna support for hyperparameter optimization controller. `OptimizerOptuna` is now the default optimizer.
* Add initial Keras-Tuner support ([GitHub Issue keras-team/keras-tuner #334](https://github.com/keras-team/keras-tuner/issues/334)).
* Add automatic FastAI logging. It is disabled if Tensorboard is loaded (assuming TensorBoardLogger will be used).
* Support Tensorboard text logging (`add_text()`) as debug samples (`.txt` files), instead of as console output.
* Allow for more standard confusion matrix reporting. `Logger.report_confusion_matrix()` argument `yaxis_reversed` (flips the confusion matrix if `True`, default `False`) ([GitHub Issue #165](https://github.com/allegroai/trains/issues/165)).
* Add support for Trains Server 0.16.0 (API v2.9 support).
* Allow disabling Trains update message from the log using the `TRAINS_SUPPRESS_UPDATE_MESSAGE` environment variable ([GitHub Issue #157](https://github.com/allegroai/trains/issues/157)).
* Add AWS EC2 Auto-Scaler service wizard and Service.
* Improved and updated examples:
* Add Keras Tuner CIFAR10 example.
* Add FastAI example.
* Update PyTorch Jupyter notebook examples ([GitHub Issue #150](https://github.com/allegroai/trains/issues/150)).
* Support global requirements detection using `pip freeze` (set `sdk.development.detect_with_pip_freeze` configuration in `trains.conf`).
* Add `Task.get_projects()` to get all projects in the system, sorted by last update time.
**Bug Fixes**
* Fix UTC to time stamp in comment ([GitHub Issue #152](https://github.com/allegroai/trains/issues/152)).
* Fix and enhance GPU monitoring:
* Fix GPU stats on Windows machines ([GitHub Issue #177](https://github.com/allegroai/trains/issues/177)).
* More robust GPU monitoring ([GitHub Issue #170](https://github.com/allegroai/trains/issues/170)).
* Fix filename too long bug ([GitHub trains-server Issue #49](https://github.com/allegroai/trains-server/issues/49)).
* Fix TensorFlow image logging to allow images with no width / height / color metadata ([GitHub Issue #182](https://github.com/allegroai/trains/issues/182)).
* Fix multiprocessing Pool throw exception in pool hangs execution. Call original signal handler and re-flush `stdout`.
* Fix `plotly` support for `matplotlib` 3.3.
* Add Python 2.7 support for `get_current_thread_id()`.
* Update examples requirements.
* Fix and improve signal handling.
* Fix Tensorboard 2D convolution histogram, improve histogram accuracy on very small histograms.
* Fix auto logging multiple argparse calls before `Task.init()`.
* Limit experiment Git diff logging to 500Kb. If larger than 500Kb, diff section will contain a warning and entire diff will be uploaded as an artifact named `auxiliary_git_dif`.
* Fix requirements detection:
* Fix Trains installed from `git+`.
* Fix when Trains is not directly imported.
* Fix multiple `-e` packages were not detected (only the first one).
* Fix running with Trains in `PYTHONPATH` resulted in double entry of trains.
* Fix `Task.set_base_docker()` on main task to do nothing when running remotely.

162
docs/release_notes/sdk/open_source/ver_0_17.md Normal file
View File

@@ -0,0 +1,162 @@
---
title: Version 0.17
---
## ClearML 0.17.5
**Features**
- Add `force_download` argument to `Artifact.get()` and `Artifact.get_local_copy()` ([clearml GitHub Issue 319](https://github.com/allegroai/clearml/issues/319)).
- Support all reporting using subprocesses instead of threads (default `sdk.development.report_use_subprocess=True`).
- Improve Datasets support:
* Add `clearml-data publish` to allow publishing a dataset task.
* Add `clearml-data sync`, which can now create, sync and close a dataset with a single command.
* Allow Dataset to be created on a running Task.
* Add `dataset_tags` argument to `Dataset.get()`.
* Add `Dataset.get_logger()`.
- Add `Task.add_requirements()` support for specifying version `<>=~` etc.
- Add `StorageManager.upload_folder()` and `StorageManager.download_folder()`.
- Add progress report logging for `StorageHelper.upload_from_stream()`, `StorageHelper.upload()` and `StorageHelper.upload_from_stream()`.
- Add jupyter auto-magic - store Jupyter Notebook as an artifact on the Task (default `sdk.development.store_jupyter_notebook_artifact=True`).
- Add upload HTML preview of Jupyter Notebook as an artifact.
- Add `PipelineController` disable clone base task option.
- Add links to Tasks in optimization summary table (not just Task IDs).
- Add support for datetime in scatter plots + matplotlib support.
- Improve plotly value type conforming.
- Improve PyTorch `DataLoader` speed ([clearml GitHub Issue 207](https://github.com/allegroai/clearml/issues/207)).
- Update Auto Scaler default values and configuration.
- Examples:
* Add Hydra example.
* Add artifacts retrieval example.
* Update various examples.
**Bug Fixes**
- Fix warning or error message if requirements parsing failed ([clearml GitHub Issue 291](https://github.com/allegroai/clearml/issues/291)).
- Fix pytorch-lighting multi node store ([clearml GitHub Issue 292](https://github.com/allegroai/clearml/issues/292)).
- Fix strip remote diff ([clearml GitHub Issue 295](https://github.com/allegroai/clearml/issues/295)).
- Fix Python package detection `sklearn` -> `scikit-learn` ([clearml GitHub Issue 296](https://github.com/allegroai/clearml/issues/296)).
- Fix argparse issues:
* Fix argparse with `[None]` in default parameter ([clearml GitHub Issue 297](https://github.com/allegroai/clearml/issues/297)).
* Fix parsing of arguments in scientific notation ([clearml GitHub Issue 313](https://github.com/allegroai/clearml/issues/313)).
* Fix argparser logging always captures defaults (Windows only, ignored cmd).
* Fix argparse `nargs` passed in command line `--nargs 1 2` should be stored as `[1, 2]` not as `['1', '2']`.
* Fix support for nonstandard argparse with default value that is not of defined type.
* Fix server updated with the argparse in remote before Task.init() is called (respect skipped args).
- Fix Dataset support:
* Fix `Dataset.remove_files()` can't find files when files are in dataset root ([clearml GitHub Issue 303](https://github.com/allegroai/clearml/issues/303)).
* Fix closing a dataset with only files removed.
* Fix Dataset generate removed / modified / added of summary table.
- Fix Hydra multi-run support ([clearml GitHub Issue 306](https://github.com/allegroai/clearml/issues/306)).
- Fix TF/TensorBoard support:
* Fix TensorBoard multiple `Task.init()` / `Task.close()` calls within the same process ([clearml GitHub Issue 312](https://github.com/allegroai/clearml/issues/312)).
* Fix TensorBoard 2+ `pr_curve`.
* Fix TF `pr_curve` should not be inverted.
* Fix TF +2.3 mixed eager mode execution summary metrics not reported.
* Fix TF bind keyboard interrupt.
* Fix TF 2.4 keras load / save model.
- Fix `clearml-task`:
* Fix error when script cannot be found.
* Fix `--docker` flag not passed.
* Fix patching local git diff.
- Fix `clearml-data`:
* Fix `clearml-data sync` requires `--name`.
* Fix missing required argument `--files` in `clearml-data remove`.
- Fix `Task.execute_remotely()` from Jupyter Notebook.
- Fix populate Task called from Jupyter Notebook (use `Task.create(packages=True)` to auto populate based on locally installed packages).
- Fix plotly plot with numpy containing `NaN` / `datetime`.
- Fix matplotlib with Agg backend (or in remote execution).
- Fix trying to upload model file as a folder (automatically package the folder).
- Fix broken packages on package `importlib` detection failed the entire requirements detection.
- Fix `Task.connect(object)` should always return the same object instance.
- Fix `Task.create()` with repo and script that exists locally.
- Fix crash in case `Logger.get_logger()` cannot get the file name.
- Fix exception at exit in Python 3.8+ on macOS / Windows.
- Fix make pipeline summary table link to Task step logs.
- Fix Hydra 1.1 support (argparse description).
- Fix close task after logger is closed.
- Fix `Task.set_base_docker()` in remote execution.
- Fix artifact preview limit to 64Kb.
- Fix JupyterLab Notebook detection.
- Fix Python 2.7 support.
### ClearML 0.17.4
**Features**
- Add `HyperParameterOptimizer` parallel coordinates ([clearml GitHub Issue 279](https://github.com/allegroai/trains/issues/279)).
- Add `Task.init()` argument `tags`.
- Change HPO parallel coordinates color scale, yellow is low.
- Change `HyperParameterOptimizer` `spawn_task_project` to `spawn_project`.
- Revert all Numpy requirement changes, use only lower limit.
**Bug Fixes**
- Fix argparse `nargs` support.
- Fix argparse with `action="append"`.
- Fix PyJWT v2.0 token parsing.
- Fix Python package detection it should not list `file://` links.
### ClearML 0.17.3
**Features**
* Add `Task.delete()` support.
* Add `Task.debug_simulate_remote_task()` to simulate task execution by ClearML Agent.
* Add warning on archived Task in pipeline ([clearml GitHub Issue 274](https://github.com/allegroai/trains/issues/274)).
* Add `Task.init(..., output_uri=True)`, which will use the default files_server as output uri.
* Make `clearml-data` CLI stateful, so it remembers last dataset ID as default dataset.
* Added `HyperParameterOptimizer.get_optimizer_top_experiments()` for querying post execution optimization pipeline.
* Add `Task.set_archived()` and `Task.get_archived()`.
* Add `Task.set_credentials()` option to store into credentials file.
* Add `clearml-data` close now auto uploads.
* Add `HyperParameterOptimizer` arguments `spawn_task_project` and `save_top_k_tasks_only`.
**Bug Fixes**
* Fix PipelineController running remotely without configuration does not execute the default code pipeline ([clearml GitHub Issue 273](https://github.com/allegroai/trains/issues/273)).
* Fix reusing task after its project was deleted ([clearml GitHub Issue 274](https://github.com/allegroai/trains/issues/274)).
* Fix `Task.archived_tag read-only` property does not work ([clearml GitHub Issue 274](https://github.com/allegroai/trains/issues/274)).
* Fix argparse support to store consistent str representation of custom objects. Avoid changing default value if remote value matches.
* Fix argparse type as function.
* Fix Dataset add single and multiple file(s).
* Fix get project name from parent dataset if not specified.
* Fix mpl exporter. Added support for legend.
* Fix model upload.
* Fix optimizer callback best experiment.
* Fix Optuna optimizer failing on tasks with None value in scalar query.
* Fix auto Python package detection installed directly from URLs.
* Fix dataset upload aborted on server watchdog.
* Fix dataset genealogy, graph and restoring data.
* Fix numpy dependency for Python versions.
### ClearML 0.17.2
**Bug Fixes**
* Fix broken `clearml-task` CLI.
### ClearML 0.17.1
**Bug Fixes**
* `clearml-task` CLI link fixed.
### ClearML 0.17.0
**Features**
* Renamed `trains` as `clearml` (see [clearml](https://github.com/allegroai/clearml) in GitHub).
* Add `clearml-data` CLI for Differential Dataset Management & Versioning (see <a href="https://github.com/allegroai/clearml/blob/master/docs/datasets.md">details</a>).
* Add `clearml-task` CLI for creating (importing) code into **ClearML** (see <a href="https://github.com/allegroai/clearml/blob/master/docs/clearml-task.md">details</a>).
* Add `Task.create`, which supports importing a code / repository into **ClearML**, including remote execution with `clearml-agent`.
**Bug Fixes**
* PyJWT v2.0 breaks the interface.

30
docs/release_notes/sdk/open_source/ver_0_9.md Normal file
View File

@@ -0,0 +1,30 @@
---
title: Version 0.9
---
:::important
**Trains** is now **ClearML**.
:::
### Trains 0.9.3
**Features and Bug Fixes**
* Improve Jupyter and inline Matplotlib support.
* Allow for insecure SSL connection to trains-server (use with care!).
* Automatically announce new **Trains** version available (at least until we hit version 1.0).
* Fix support for local git branches (i.e. no matching remote branch).
* Verify support for Official TensorFlow 1.14 release and PyTorch with TensorBoard.
### Trains 0.9.2
**Features and Bug Fixes**
* Improve support for Windows.
### Trains 0.9.1
**Features and Bug Fixes**
* Improve support for Python 2.7.

167
docs/release_notes/sdk/open_source/ver_1_0.md Normal file
View File

@@ -0,0 +1,167 @@
---
title: Version 1.0
---
### ClearML 1.0.5
**Features**
- Add Click support ([ClearML GitHub issue #386](https://github.com/allegroai/clearml/issues/386))
- Add progress bar to SHA2 generation ([ClearML GitHub issue #396](https://github.com/allegroai/clearml/issues/396))
- Add prefix to Task reported runtime info: `cpu_cores`, `gpu_driver_version` and `gpu_driver_cuda_version`
- Add support for `Logger.report_text()` explicit log-level reporting
- Add `return_full_path` argument to `StorageManager.list()`
- Support `Task.get_tasks()` passing multiple project names
- Add `TaskScheduler`
- Add `task_filter` argument to `Objective.get_top_tasks()`, allow `name` as a `task_filter` field
- Add `--output-uri` command-line option to `clearml-task`
- Add `requirements_file` argument to `Task.force_requirements_env_freeze()` to allow specifying a local requirements file
- Add support for `list` type argument in `Task.connect_configuration()` (previously only `dict` type was supported)
- Rename `TrainsTuner` to `ClearmlTuner`
- Update documentation links
**Bug Fixes**
- Fix Pandas with multi-index ([ClearML GitHub issue #399](https://github.com/allegroai/clearml/issues/399))
- Fix check permissions fail in `HTTPDriver` ([ClearML GitHub issue #394](https://github.com/allegroai/clearml/issues/394))
- Fix Dataset not setting system tag on existing `data_processing` Tasks
- Fix disable redundant resource monitoring in pipeline controller
- Fix `ClearMLJob` when both `project` and `target_project` are specified
- Fix `ClearMLJob` docker container info is not cached
- Fix no print logging after Python logging handlers are cleared
- Fix `PipelineController` callback returning `False`
- Fix internal `logging.Logger` can't be pickled (only applicable to Python 3.6 or lower)
- Wait for reported events to flush to ensure `Task.flush()` with `wait_for_uploads=True` awaits background processes
### ClearML 1.0.4
**Features**
- Add Google Colab notebook tutorial ([ClearML GitHub PR 368](https://github.com/allegroai/clearml/issues/368) and [ClearML GitHub PR 374](https://github.com/allegroai/clearml/issues/374))
- Add support for GIF images in Tensorboard ([ClearML GitHub issue 372](https://github.com/allegroai/clearml/issues/372))
- Add a tensorboardX example for `add_video` (creates GIFs in tensorboard) ([ClearML GitHub PR 372](https://github.com/allegroai/clearml/issues/372))
- Add auto scaler customizable boot bash script
- Add `Task.ignore_requirements`
- Deprecate `Logger.tensorboard_single_series_per_graph()` as it is now controlled from the UI 🙂
**Bug Fixes**
- Fix `default_output_uri` for Dataset creation ([ClearML GitHub issue 371](https://github.com/allegroai/clearml/issues/371))
- Fix `clearml-task` failing without a docker script ([ClearML GitHub issue 378](https://github.com/allegroai/clearml/issues/378))
- Fix PyTorch DDP sub-process spawn multi-process
- Fix `Task.execute_remotely()` on created Task (not initialized Task)
- Fix auto scaler custom bash script should be called last before starting agent
- Fix auto scaler spins too many instances at once then kills the idle ones (spin time is longer than poll time)
- Fix multi-process spawn context using `ProcessFork` kills sub-process before parent process ends
### ClearML 1.0.3
**Features**
- Use default `boto` credential chain if no keys are provided in the configuration file or environment variables ([ClearML GitHub PR 342](https://github.com/allegroai/clearml/issues/342))
- Support `DummyModel` configuration ([Slack Channel](https://clearml.slack.com/archives/CTK20V944/p1621469235085400))
- Add `report_matplotlib_figure(..., report_interactive=False)` allowing to upload a matplotlib as a non-interactive (high quality png) plot
- Add `Logger.matplotlib_force_report_non_interactive()`
- Remove matplotlib axis range (`plotly.js` auto-range can adjust it in real-time)
- Add object-storage support in cleanup-service
- Add `dataset_tags` argument to `Dataset.create()`
- Expose `docker_args` and `docker_bash_setup_script` in `clearml-task` CLI
- Add logging for Nvidia driver and Cuda version
- Add optional ignored packages in script requirements (currently used for `pywin32`)
- Update examples
* Increase channel result to support max of 1K channels for finding slack channel and use cursor in Slack Alerts monitoring service
* Add `csv` data sample to `data_samples`
* Remove deprecated examples
**Bug Fixes**
- Fix Hydra should not store the full resolved OmegaConf ([ClearML GitHub issue 327](https://github.com/allegroai/clearml/issues/327))
- Fix direct import of keras save/load model functions ([ClearML GitHub issue 355](https://github.com/allegroai/clearml/issues/355))
- Fix run as module ([ClearML GitHub issue 359](https://github.com/allegroai/clearml/issues/359))
- Fix Python 2.7 support ([ClearML GitHub issue 366](https://github.com/allegroai/clearml/issues/366))
- Fix `Task.add_requirements()` passing `package_version` starting with `@`, `;` or `#`
- Fix import keras from TF
- Fix support for Hydra's `run_job()` change in parameter order by passing `config` and `task_function` as keyword arguments
- Fix background upload retries with Google Storage (`gs://`)
- Fix Python 3.8 race condition in `Task.close()`
- Fix shutting down a Task immediately after creation might block
- Fix `Task.execute_remotely()` from Jupyter notebook
- Fix Jupyter Notebook inside VSCode
- Fix support for `Dataset.create()` argument `use_current_task`
- Fix `Dataset.finalize()` can hang in extreme scenario
- Protect against wrong file object type when auto-binding models
- Fix matplotlib date convertor
- Fix automation controller overrides nodes clone
### ClearML 1.0.2
**Bug Fixes**
- Fix in rare scenarios process stuck on exit, again :)
### ClearML 1.0.1
**Bug Fixes**
- Fix in rare scenarios process stuck on exit
### ClearML 1.0.0
:::info Breaking Changes
Arguments order changed in `Logger.report_line_plot()`, `Logger.report_plotly()` and `Logger.report_matplotlib_figure()` - please use keywords instead of positional arguments
:::
**Features**
* Add OS environment variable `CLEARML_DEFAULT_OUTPUT_URI` to override default `output_uri` for automatic models upload ([ClearML GitHub issue 328](https://github.com/allegroai/clearml/issues/328))
* Add `Task.force_requirements_env_freeze()` forcing `pip freeze` instead of package analysis ([ClearML GitHub issue 304](https://github.com/allegroai/clearml/issues/304))
* Improve pipeline support
* Add Task parameters override in `ClearmlJob` ([ClearML GitHub issue 214](https://github.com/allegroai/clearml/issues/214))
* Add PipelineController node skip visualization and pre/post-execution callback
* Allow enabling/disabling tensorboard auto-connect using `Task.init(auto_connect_frameworks={'tensorboard': True/False})` (default: `True`)
* Allow to continue running from previous execution iteration
* Support stopping instead of resetting in `Task.execute_remotely()` when server supports enqueueing stopped tasks
* Add Model query interface using `Model.query_models()`
* Support storing dict with `"."` in the keys using Task.connect_configuration
* Add `force` argument to `Task.mark_failed()` to allow setting Task status to `failed` regardless of Task state
* Add `docker_args` and `docker_bash_setup_script` arguments to `Task.create()`
* Change Logger plot iteration argument default to 0 (or `None`)
* Add task filtering in `Task.get_task()` using the `allow_archived` and `task_filter` arguments
* Add `Task.get_configuration_objects()` for retrieving configuration object as a blob of text in automation use-cases (instead of using `Task.connect_configuration()`)
* Improved pipeline support
* Add pipeline step caching
* Add Pipeline controller caching
* Improve pipeline plot reporting
* Add `PipelineController` into main namespace (use `from clearml import PipelineController`)
* Add jupyter notebook preview update time stamp
* Change default metric report flushing to every 5 seconds
* Improve `Task.artifacts` for safer Task multi node use cases
* Support ClearML server API v2.13
* Add Python 3.9 to the support table
* Improve documentation and examples
* Improve documentation for clearml-data, clearml-task
* Add `multiple_tasks_single_process` example script for multiple Tasks created in same code
* Add `using_artifacts_example` and `execute_remotely_example` example scripts
* Update `requirements.txt` for `KerasTuner` example
**Bug Fixes**
* Fix `\r` and `\n` in debug samples title/series ([ClearML GitHub issue 323](https://github.com/allegroai/clearml/issues/323))
* Fix Google API credentials without storage section raises exception on import ([ClearML GitHub issue 331](https://github.com/allegroai/clearml/issues/331))
* Fix Matplotlib automagic export legend not showing series names ([ClearML GitHub issue 337](https://github.com/allegroai/clearml/issues/337))
* Fix `Task.set_comment/name()` passing `None` should be equivalent to `''`
* Fix `Task.set_credentials()` uses incorrect config file when both `trains.conf` and `clearml.conf` are found
* Fix `Task.set_credetnials()` should override settings from config file
* Fix Keras h5 model storage
* Fix `Task.execute_remotely()` with sub process
* Fix Task requirements should conform to strings
* Fix reporting after task closing
* Fix remove deprecation warning with matplotlib 3.1
* Fix dataset create single root plot
* Fix `Model.task` should return task ID
* Fix code hangs when leaving process
* Fix flushing on CR timeout (default 10sec)
* Wait for subprocess when terminating
* Improve flush behaviour on `KeyboardInterrupt` exception
* Wait for subprocesses to flush when terminating
* Fix `Task.set_credentials()` with `store_conf_file=True`

245
docs/release_notes/sdk/open_source/ver_1_1.md Normal file
View File

@@ -0,0 +1,245 @@
---
title: Version 1.1
---
### ClearML 1.1.6
**Features**
- Add `Task.force_store_standalone_script()` to force storing standalone script instead of a Git repository reference ([ClearML GitHub issue #340](https://github.com/allegroai/clearml/issues/340))
- Add `Logger.set_default_debug_sample_history()` and `Logger.get_default_debug_sample_history()` to allow controlling
maximum debug samples programmatically
- Add populate now stores function arg types as part of the hyperparameters
- Add `status_message` argument to `Task.mark_stopped()`
- Change `HTTP` driver timeout and retry codes (connection timeout will now trigger a retry)
**Bug Fixes**
- Fix and upgrade the SlackMonitor ([ClearML GitHub issue #533](https://github.com/allegroai/clearml/issues/533))
- Fix network issues causing Task to stop on status change when no status change has occurred ([ClearML GitHub issue #535](https://github.com/allegroai/clearml/issues/535))
- Fix Pipeline controller function support for dict as input argument
- Fix uploading the same metric/variant from multiple processes in threading mode should create a unique file per process (since global counter is not passed between the subprocesses)
- Fix resource monitoring should only run in the main process when using threaded logging mode
- Fix fork patching so that the signal handler (`at_exit`) will be called on time
- Fix fork (process pool) hangs or drops reports when reports are at the end of the forked function in both threaded and subprocess mode reporting
- Fix multi-pipeline support
- Fix delete artifacts after upload
- Fix artifact preview has no truth value
- Fix storage cache cleanup does not remove all entries on a silent fail
- Fix always store session cache in `~/.clearml` (regardless of the cache folder)
- Fix `StorageManager.download_folder()` fails on Windows path
### ClearML 1.1.5
**Features**
* Add support for `jsonargpraser` ([ClearML GitHub issue #403](https://github.com/allegroai/clearml/issues/403))
* Add `HyperParameterOptimizer.get_top_experiments_details()` returns the hparams and metrics of the top performing
experiments of an HPO ([ClearML GitHub issue #473](https://github.com/allegroai/clearml/issues/473))
* Allow overriding initial iteration offset using environment variable (`CLEARML_SET_ITERATION_OFFSET`) or `Task.init(continue_last_task==<offset>)`
( [ClearML GitHub issue #496](https://github.com/allegroai/clearml/issues/496))
* Add better input handling for `clearml-init` in colab ([ClearML GitHub issue #515](https://github.com/allegroai/clearml/issues/515))
* Add environment variable for default request method ([ClearML GitHub issue #521](https://github.com/allegroai/clearml/issues/521))
* Add `LocalClearmlJob` as possible option for HPO ([ClearML GitHub issue #525](https://github.com/allegroai/clearml/issues/525))
* Add convenience functionality to `clearml-data` ([ClearML GitHub issue #526](https://github.com/allegroai/clearml/issues/526))
* Add support for `vscode-jupyter` ([microsoft/vscode-jupyter#8531](https://github.com/microsoft/vscode-jupyter/pull/8531))
* Improve detection of running reporting subprocess (including zombie state)
* Support controlling S3/Google Cloud Storage `_stream_download_pool_connections` using the `stream_connections` configuration
setting in `clearml.conf` (default 128)
* Add warning when losing reporting subprocess
* Add `Model.remove()` to allow removing a model from the model repository
* Add HTTP download timeout control (change default connection timeout to 30 seconds)
* Add initial setup callback to monitoring class
* Add `Task.get_reported_plots()`
* Allow `Monitor.get_query_parameters` to override defaults
* Add support for Google Cloud Storage `pool_connections` and `pool_maxsize` overrides
* Add last worker time to `AutoScaler`
* Add warning when opening an aborted Dataset
* Store multi-pipeline execution plots on the master pipeline Task
* Support pipeline return value stored on pipeline Task
* Add `PipelineDecorator.multi_instance_support`
* Add `PipelineDecorator` to `clearml` and `clearml.automation` namespaces
* Documentation and examples
* Update docstrings ([ClearML GitHub issue #501](https://github.com/allegroai/clearml/issues/501))
* Add Markdown in pipeline jupyter notebooks ([ClearML GitHub issue #502](https://github.com/allegroai/clearml/issues/502))
* Update pipeline example ([ClearML GitHub issue #494](https://github.com/allegroai/clearml/issues/494))
* Add abseil example ([ClearML GitHub issue #509](https://github.com/allegroai/clearml/issues/509))
* Change README to dark theme ([ClearML GitHub issue #513](https://github.com/allegroai/clearml/issues/513))
* Update XGBoost example ([ClearML GitHub issue #524](https://github.com/allegroai/clearml/issues/524))
* Change example name ([ClearML GitHub issue #528](https://github.com/allegroai/clearml/issues/528))
**Bug Fixes**
* Fix `TriggerScheduler` on Dataset change ([ClearML GitHub issue #491](https://github.com/allegroai/clearml/issues/491))
* Fix links in Jupyter Notebooks ([ClearML GitHub issue #505](https://github.com/allegroai/clearml/issues/505))
* Fix `pandas` delta datetime conversion ([ClearML GitHub issue #510](https://github.com/allegroai/clearml/issues/510))
* Fix `matplotlib` auto-magic detect bar graph series name ([ClearML GitHub issue #518](https://github.com/allegroai/clearml/issues/518))
* Fix path limitation on storage services (posix, object storage) when storing target artifacts by limiting length of
project name (full path) and task name used for object path ([ClearML GitHub issue #516](https://github.com/allegroai/clearml/issues/516))
* Fix multi-processing context block catching exception
* Fix Google Cloud Storage with no default project causes a crash
* Fix main process's reporting subprocess lost, switch back to thread mode
* Fix forked `StorageHelper` should use its own `ThreadExecuter`
* Fix local `StorageHelper.delete()` raising exception on non-existent file instead of returning false
* Fix `StorageHelper` rename partial file throwing errors on multiple access
* Fix resource monitor fails on permission issues (skip over parts)
* Fix reusing Task does not reset it
* Fix support `clearml` PyCharm Plugin 1.0.2 (support partial pycharm git repo sync)
* Fix `Task.reset()` force argument ineffective
* Fix PY3.5 compatibility
* Fix validation error causes infinite loop
* Fix tasks schema prevents sending null container parts
* Fix missing `CLEARML_SET_ITERATION_OFFSET` definition
* Fix `Model.get_weights_package()` returns None on error
* Fix download progress bar based on `sdk.storage.log.report_download_chunk_size_mb` configuration
* Fix Conda lists the `CudaToolkit` version installed (for the agent to reproduce)
* Fix Jupyter kernel shutdown causing nested atexit callbacks leaving Task in running state
* Fix multi-subprocess can cause Task to hand at close
* Fix TF 2.7 support (get `logdir` on with multiple TB writers)
### ClearML 1.1.4
**Bug Fixes**
- Fix duplicate keyword argument (affects `clearml-data`, `Dataset.get()`) ([ClearML GitHub issue #490](https://github.com/allegroai/clearml/issues/490))
- Fix session raises missing host error when in offline mode ([ClearML GitHub issue #489](https://github.com/allegroai/clearml/issues/489))
- Fix `Task.get_task()` does not load `output_uri` from stored Task
- Fix `Task.get_models()['input']` returns string instead of `clearml.Model`
- Fix `tf.saved_model.load()` binding for `TensorFlow>=2.0`
- Fix hyperparams with `None` value converted to empty string causes inferred type to change to `str` in consecutive `Task.connect()` calls
### ClearML 1.1.3
**Features**
- Add support for MegEngine with examples ([ClearML GitHub issue #455](https://github.com/allegroai/clearml/issues/455))
- Add `TaskTypes` to main namespace ([ClearML GitHub issue #453](https://github.com/allegroai/clearml/issues/453))
- Add `LogUnifomParameterRange` for hyperparameter optimization with Optuna ([ClearML GitHub issue #462](https://github.com/allegroai/clearml/issues/462))
- Add joblib (equivalent to scikit) to `Task.init(auto_connect_frameworks)` argument
- Log environment variables starting with `*` in `environ_bind.py` ([ClearML GitHub issue #459](https://github.com/allegroai/clearml/issues/459))
- Pipeline
- Add eager decorated pipeline execution
- Support pipeline monitoring for scalars/models/artifacts
- Add `PipelineController.upload_model()`
- Add `PipelineController.add_step(configuration_overrides)` argument allowing to override Task configuration objects
- Change `PipelineController.start_locally()` default `run_pipeline_steps_locally=False`
- Add `PipelineController.stop(mark_failed, mark_aborted)` arguments
- Add `PipelineController.run_locally` decorator
- Add `PipelineController.continue_on_fail` property
- Add `PipelineController.__init__(abort_on_failure)` argument
- Add nested pipeline components missing pipeline tags
- Add ClearmlJob state cache (refresh every second)
- Datasets
- Add `clearml-data` multi-chunk support
- Change `clearml-data` default chunk size to 512 MB
- Change `Dataset.create()` now automatically reverts to using current Task if no project/name provided
- Add `Optimizer.get_top_experiments_id_metrics_pair()` for top performing experiments
- Add support for setting default value to auto connected argparse arguments
- Add `Task.get_script()` and `Task.set_script()` for getting and setting task's script properties for execution
- Add `Task.mark_completed()` `force` and `status_message` arguments
- Add `Task.stopped()` `reason` argument
- Add `Task.query_tasks()`, `Task.get_task()` and `Task.get_tasks()` `tags` argument
**Bug Fixes**
- Fix PyJWT resiliency support
- Fix xgb train overload ([ClearML GitHub issue #456](https://github.com/allegroai/clearml/issues/456))
- Fix `http://` throws `OSError` in Windows by using `pathlib2` instead of `os` ([ClearML GitHub issue #463](https://github.com/allegroai/clearml/issues/463))
- Fix local diff should include staged commits, otherwise applying git diff fails ([ClearML GitHub issue #457](https://github.com/allegroai/clearml/issues/457))
- Fix `task.upload_artifact` non-standard dictionary will now revert to `pickle` ([ClearML GitHub issue #452](https://github.com/allegroai/clearml/issues/452))
- Fix `S3BucketConfig.is_valid()` for EC2 environments with `use_credentials_chain` ([ClearML GitHub issue #478](https://github.com/allegroai/clearml/issues/478))
- Fix audio classifier example when training with a custom dataset ([ClearML GitHub issue #484](https://github.com/allegroai/clearml/issues/484))
- Fix `clearml-task` diff was corrupted by Windows drive letter and separator ([ClearML GitHub issue #483](https://github.com/allegroai/clearml/issues/483))
- Fix TQDM "line cleanup" not using `CR` but rather arrow-up escape sequence
- Fix `task.connect(dict)` value casting - if `None` is the default value, use backend stored type
- Fix Jupyter notebook should always set Task as completed/stopped, never failed (exceptions are caught in interactive session)
- Fix Pipeline support
- Fix `LocalClearmlJob` setting failed status
- Fix pipeline stopping all running steps
- Fix nested pipeline component parent point to pipeline Task
- Fix `PipelineController.start()` should not kill the process when done
- Fix pipeline failing to create Step Task should cause the pipeline to be marked failed
- Fix nested pipeline components missing pipeline tags
- Fix images reported over history size were not sent if frequency was too high
- Fix git detectors missing git repository without origin
- Fix support for upload `LazyEvalWrapper` artifacts
- Fix duplicate task dataset tags
- Fix FileLock create target folder
- Fix crash inside forked subprocess might leave SafeQueue in a locked state, causing `task.close()` to hang
- Fix PyTorch distributed example `TimeoutSocket` issue in Windows
- Fix broken `Dataset.finalize()`
- Fix Python 3.5 compatibility
### ClearML 1.1.2
**Bug Fix**
- Fix PyJWT issue (limit dependency to `<2.2.0`)
### ClearML 1.1.1
**Bug Fixes**
- Fix `Logger.report_image()` throws warning
- Fix TensorBoard `add_image()` not being reported
### ClearML 1.1.0
:::info Breaking Changes
- New PipelineController v2 was introduced. New constructor is not backwards compatible!
- ClearML will no longer try to use the demo server by default (change this by setting the `CLEARML_NO_DEFAULT_SERVER=0` environment variable)
- `Task.completed()` was deprecated, use `Task.mark_completed()` instead
:::
**Features**
- Add Task Trigger Scheduler
- Add Task Cron Scheduler
- Add PipelineController from function
- Add PipelineDecorator (`PipelineDecorator.pipeline` and `PipelineDecorator.component` decorators for full custom pipeline logic)
- Add xgboost auto metric logging ([ClearML GitHub issue #381](https://github.com/allegroai/clearml/issues/381))
- Add `sdk.storage.log.report_upload_chunk_size_mb` and `sdk.storage.log.report_download_chunk_size_mb` configuration options to control upload/download log reporting ([ClearML GitHub issue #424](https://github.com/allegroai/clearml/issues/424))
- Add new optional `auto_connect_frameworks` argument value to `Task.init()` (e.g. `auto_connect_frameworks={'tfdefines':False}`) to allow disabling TF defines ([ClearML GitHub issue #408](https://github.com/allegroai/clearml/issues/408))
- Add support for `CLEARNL_CONFIG_VERBOSE` environment variable to allow external control over verbosity of the configuration loading process
- Add support for uploading artifacts with a list of files using `Task.upload_artifcats(name, [Path(), Path()])`
- Add missing *clearml-task* parameters `--docker_args`, `--docker_bash_setup_script` and `--output-uri`
- Change `CreateAndPopulate` will auto list packages imported but not installed locally
- Add `clearml.task.populate.create_task_from_function()` to create a Task from a function, wrapping function input arguments into hyper-parameter section as kwargs and storing function results as named artifacts
- Add support for Task serialization (e.g. for pickle)
- Add `Task.get_configuration_object_as_dict()`
- Add `docker_image` argument to `Task.set_base_docker()` (deprecate `docker_cmd`)
- Add `auto_version_bump` argument to `PipelineController`
- Add `sdk.development.detailed_import_report` configuration option to provide a detailed report of all python package imports
- Set current Task as Dataset parent when creating dataset
- Add support for deferred configuration
- Examples
- Add Pipeline v2 examples
- Add `TaskScheduler` and `TriggerScheduler` examples
- Add pipeline controller callback example
- Improve existing examples and docstrings
**Bug Fixes**
- Fix plotly plots converting `NaN` to `nan` instead of `null` ([ClearML GitHub issue #373](https://github.com/allegroai/clearml/issues/373))
- Fix deprecation warning ([ClearML GitHub issue #376](https://github.com/allegroai/clearml/issues/376))
- Fix plotly multi-index without index names ([ClearML GitHub issue #399](https://github.com/allegroai/clearml/issues/399))
- Fix click support ([ClearML GitHub issue #437](https://github.com/allegroai/clearml/issues/437))
- Fix docstring ([ClearML GitHub issue #438](https://github.com/allegroai/clearml/issues/438))
- Fix passing `task-type` to *clearml-task* ([ClearML GitHub issue #422](https://github.com/allegroai/clearml/issues/422))
- Fix `clearml-task --version` throws an error ([ClearML GitHub issue #422](https://github.com/allegroai/clearml/issues/422))
- Fix *clearml-task* ssh repository links are not detected as remote repositories ([ClearML GitHub issue #423](https://github.com/allegroai/clearml/issues/423))
- Fix `getattr` throws an exception ([ClearML GitHub issue #426](https://github.com/allegroai/clearml/issues/426))
- Fix encoding while saving notebook preview ([ClearML GitHub issue #443](https://github.com/allegroai/clearml/issues/443))
- Fix poetry toml file without requirements.txt ([ClearML GitHub issue #444](https://github.com/allegroai/clearml/issues/444))
- Fix `PY3.x` fails calling `SemLock._after_fork` with forkserver context, forking while lock is acquired ([ClearML Agent GitHub issue #73](https://github.com/allegroai/clearml-agent/issues/73))
- Fix wrong download path in `StorageManager.download_folder()`
- Fix jupyter notebook `display(...)` convert to `print(...)`
- Fix TensorFlow `add_image()` with `description='text'`
- Fix `Task.close()` should remove `current_task()` reference
- Fix `TaskScheduler` weekdays, change default `execute_immediately` to `False`
- Fix Python2 compatibility
- Fix *clearml-task* exit with error when failing to verify `output_uri` (output warning instead)
- Fix unsafe Google Storage delete object
- Fix multi-process spawning wait-for-uploads can create a deadlock in very rare cases
- Fix `task.set_parent()` fails when passing Task object
- Fix `PipelineController` skipping queued Tasks
- Remove `humanfriendly` dependency (unused)

90
docs/release_notes/sdk/open_source/ver_1_10.md Normal file
View File

@@ -0,0 +1,90 @@
---
title: Version 1.10
---
### ClearML 1.10.4
**New Features and Improvements**
* Update references to `GetAllRequest` in Task docstrings ([ClearML GitHub issue #981](https://github.com/allegroai/clearml/issues/981))
**Bug Fixes**
* Fix issue with JSON conversion when using `jsonargparse` during remote execution ([ClearML GitHub issue #1010](https://github.com/allegroai/clearml/issues/1010)
and [ClearML Agent GitHub issue #153](https://github.com/allegroai/clearml-agent/issues/153))
* Fix `ProxyDictPostWrite.update()` not triggering a write-back to the backend ([ClearML GitHub issue #985](https://github.com/allegroai/clearml/issues/985))
* Fix standard streams being patched in subtasks, which caused the streams to be reported to the wrong task
* Fix TypeError when using `Task.query_tasks()` without specifying the `task_filter` parameter
* Fix Tensorboard negative iterations, they are now zeroed
* Fix `Task.init(continue_last_task=0)` being ignored when running remotely
* Fix `urllib3` import error when using `urllib3>=2.0.0`
### ClearML 1.10.3
**Bug Fix**
* Fix mixed-up logs when using pipelines. Now pipeline controller logs won't be visible in pipeline steps' logs.
### ClearML 1.10.2
**New Features and Improvements**
* Allow task filtering with `Task.query_tasks()` using `TaskTypes` ([ClearML GitHub issue #938](https://github.com/allegroai/clearml/issues/938))
* Allow adding tags to individual pipeline steps and to the `PipelineController`
* Restrict `Dataset.create()` usage for old server API versions
* Improve docstrings
**Bug Fixes**
* Fix `Task.get_tasks` docstring ([ClearML GitHub issue #937](https://github.com/allegroai/clearml/issues/937))
* Fix `Task.get_reported_scalars()` docstring for `x_axis` parameter's `"timestamp"` value ([ClearML GitHub issue #963](https://github.com/allegroai/clearml/issues/963))
* Fix missing plots on models in Windows
* Fix Pipeline version auto-bumping
* Fix `APIClient.events.debug_images()` for latest server API version
### ClearML 1.10.1
**New Features and Improvements**
* Add cast parameter to Task.get_parameter method ([ClearML GitHub issue #951](https://github.com/allegroai/clearml/issues/951))
**Bug Fixes**
* Fix backward compatibility issue when using datetime filtering with `APIClient().models.get_all` and `APIClient().tasks.get_all` requests
* Fix docstrings explaining tag filtering on `Model` and `Task`
### ClearML 1.10.0
**Breaking Changes**
* `StorageManager.download_file()` and `StorageManager.download_folder()` from MinIO and Azure no longer include bucket name
in target local path ([ClearML GitHub issue #709](https://github.com/allegroai/clearml/issues/709))
**New Features and Improvements**
* Add support for attaching plots and scalars to models
* Add `Task.get_debug_sample()` to retrieve task's debug samples ([ClearML GitHub issue #761](https://github.com/allegroai/clearml/issues/761))
* Add custom artifact serialization for pipelines ([ClearML GitHub issue #689](https://github.com/allegroai/clearml/issues/689))
* Add table layout configuration to `Logger.report_table()` through `extra_data` parameter ([ClearML GitHub issue #796](https://github.com/allegroai/clearml/issues/796))
* Add `PipelineController.connect_configuration()` to add configuration dictionaries to a pipeline
* Add `Dataset.delete()` configurable hosted contents deletion support:
* Delete ClearML file server hosted files
* Delete external storage hosted files
* Add `force_download` argument to `Model.get_local_copy()` to force download a new copy of a model even if already in cache
* Add `PipelineController.is_successful()` criteria customization
* Add support for API v2.23
* Add `Getting Started` Notebooks ([ClearML PR #948](https://github.com/allegroai/clearml/pull/948))
* Improve `hyper_parameter` example ([ClearML PR #927](https://github.com/allegroai/clearml/pull/927))
* Correct `Task.connect()` docstring ([ClearML Docs issue #473](https://github.com/allegroai/clearml-DOCS/issues/473))
* Improve the `Task.mark_completed()` and `Task.close()` docstrings ([ClearML Docs issue #486](https://github.com/allegroai/clearml-docs/issues/486))
**Bug Fixes**
* Fix logging dependencies that use the subdirectory argument when pip installing from a git repo ([ClearML GitHub issue #946](https://github.com/allegroai/clearml/issues/946))
* Fix `Task.import_offline_session()` does not import offline models ([ClearML GitHub issue #653](https://github.com/allegroai/clearml/issues/653))
* Fix `clearml-init` incorrectly sets Web and API server ports ([ClearML Server GitHub issue #181](https://github.com/allegroai/clearml-server/issues/181) and [ClearML GitHub issue #910](https://github.com/allegroai/clearml/issues/910))
* Fix multiple models trained by the same framework are not all automatically logged ([ClearML GitHub issue #767](https://github.com/allegroai/clearml/issues/767))
* Fix parallel coordinates plot displays categorical variables unclearly ([ClearML GitHub issue #907](https://github.com/allegroai/clearml/issues/907))
* Fix runtime toggling task offline mode in the context of an open task
* Fix information not being captured from Jupyter Notebook in Sagemaker studio
* Fix `python-fire` support
* Fix mishandling nested dictionaries in `Task.connect()` in `PipelineDecorator`
* Fix pipeline version number isn't updated when `auto_version_bump` is set to `True`
* Fix `get_or_create_project` crashes when run in parallel
* Fix task auto-publishing is not backwards compatible
* Fix `task.get_user_properties(value_only=True)` raises exception
* Fix `CLEARML_VCS_DIFF` set to empty string does not cause task to log empty uncommitted changes
* Fix `silence_errors` parameter in `StorageManager.get_file_size_bytes()` not working
* Fix setting `Dataset` metadata name to a string prefixed with `data_` breaks `Dataset.get()`
* Fix Pipeline fails creating tasks from multi-argument functions

42
docs/release_notes/sdk/open_source/ver_1_11.md Normal file
View File

@@ -0,0 +1,42 @@
---
title: Version 1.11
---
### ClearML 1.11.1
**New Features and Improvements**
* Support `lightning>=2.0` ([ClearML GitHub issue #1032](https://github.com/allegroai/clearml/issues/1032))
* Add `clearml-init` support for standard HTTP and HTTPS ports in webapp in conjunction with non-default api/files server ports
([ClearML Server GitHub issue #198](https://github.com/allegroai/clearml-server/issues/198))
* Improve docstrings
**Bug Fixes**
* Fix remote execution does not log configuration values from modified `jsonargparse` configuration files
* Fix `ValueError` when setting task properties at the start of a pipeline component
* Fix OS exceptions on passing `Path` where `str` expected
* Fix infinite recursion when remotely updating a `dict` added to task with `Task.connect()`
* Fix `Task.connect()` race condition overwriting task description with connected input model
### ClearML 1.11.0
**New Features and Improvements**
* Add offline mode for ClearML Datasets ([ClearML GitHub issue #854](https://github.com/allegroai/clearml/issues/854))
* Improve performance of `Dataset.add_external_files()` ([ClearML GitHub PR #962](https://github.com/allegroai/clearml/pull/962))
* Add `Task.launch_multi_node()` for distributed experiment execution
* Add `Task.get_all_reported_scalars()` to fetch all available scalar data
* Add `Task.remove_input_models()` to disassociate input models from a task
* Improve `Dataset.list_datasets()` to include dataset version in the result
* Add support for PyTorch Lightning CLI >=2.0
* Add support for automatic Dataset content preview for JSON files
**Bug Fixes**
* Fix multiple local parallel experiments conflicting on storage access validation ([ClearML GitHub issue #897](https://github.com/allegroai/clearml/issues/897))
* Fix code running with `joblib` hangs ([ClearML GitHub issue #1009](https://github.com/allegroai/clearml/issues/1009))
* Fix comment for storage scheme `gs://` ([ClearML GitHub issue #986](https://github.com/allegroai/clearml/issues/986))
* Fix typo "Categories per column" ([ClearML GitHub PR #1021](https://github.com/allegroai/clearml/pull/1021))
* Fix downloading artifacts from GCP when the download URI includes unsafe characters
* Fix pipeline step cache is not cleared when the only change to the step is that the docker image changed
* Fix `Task.report_text()` sometimes reporting to an incorrect task when multiple tasks run simultaneously
* Fix` Task.set_offline(offline_mode=False)` raising an `UnboundLocalError`
* Fix `scikit-image` package (`skimage`) is sometimes not detected as a dependency
* Fix `PipelineDecorator` sometimes causing a race condition when starting a remote execution

49
docs/release_notes/sdk/open_source/ver_1_12.md Normal file
View File

@@ -0,0 +1,49 @@
---
title: Version 1.12
---
### ClearML 1.12.2
**New Features and Improvements**
* Add `always_create_from_code` argument to `PipelineController` (default `True`) to allow remote execution to create
the Pipeline DAG at runtime (as opposed to adhering to the existing structure stored on the task when running locally)
**Bug Fixes**
* Fix using `OutputModel` reporting and other methods fail if `OutputModel.update_weights()` is not called before that ([ClearML GitHub issue #1078](https://github.com/allegroai/clearml/issues/1078))
* Fix hash for Pandas objects artifacts
* Fix Hydra/OmegaConf `--overrides` support
* Fix Pipeline metrics not forwarded to pipeline controller at the end of a pipeline step
### ClearML 1.12.1
**New Features and Improvements**
* Raise error if dataset name is empty ([ClearML GitHub PR #1083](https://github.com/allegroai/clearml/pull/1083))
* Add support for `spawn` start method for Python `multiprocess`. This should help circumvent issues like [this](https://github.com/python/cpython/issues/84559).
**Bug Fixes**
* Fix S3 uploads fail with `LazyEvalWrapper` type error ([ClearML GitHub issue #1081](https://github.com/allegroai/clearml/issues/1081))
### ClearML 1.12.0
:::info BREAKING CHANGES
As a result of fixing [issue #1051](https://github.com/allegroai/clearml/issues/1051) and similar issues with retrieving
task data/models/artifacts/samples stored in GCS, we introduced some breaking changes. To access resources registered to
GCS from ClearML SDK versions 1.11.0 to 1.11.1, see
the instructions [here](https://github.com/allegroai/clearml/tree/master/docs/errata_breaking_change_gcs_sdk_1_11_x.md).
:::
**New Features and Improvements**
* Add `include_archive` parameter to `Dataset.list_datasets()`: include archived datasets in list ([ClearML GitHub issue #1067](https://github.com/allegroai/clearml/issues/1067))
* Add support to specify the multipart chunk size and threshold using the `aws.boto3.multipart_chunksize` and
`aws.boto3.multipart_threshold` configuration options in the clearml.conf ([ClearML GitHub issue #1058](https://github.com/allegroai/clearml/issues/1058))
* Add `PipelineController.get_pipeline()` for retrieving previously run pipelines.
**Bug Fixes**
* Fix AWS driver issues ([ClearML GitHub PR #1000](https://github.com/allegroai/clearml/pull/1000)):
* Fix credential authentication failure when attempting to use token
* Fix instantiation within VPC without AvailabilityZones
* Fix `continue_last_task=0` is ignored in pipelines run with `retry_on_failure` ([ClearML GitHub issue #1054](https://github.com/allegroai/clearml/issues/1054))
* Fix `Task.connect_configuration()` doesn't handle dictionaries with special characters
* Fix pipeline steps created with `PipelineDecorator` aren't cached
* Fix `Task.get_by_name()` doesn't return the most recent task when multiple tasks have same name
* Fix Boto3 certificate verification ignores custom self-signed certificates

65
docs/release_notes/sdk/open_source/ver_1_13.md Normal file
View File

@@ -0,0 +1,65 @@
---
title: Version 1.13
---
### ClearML v1.13.2
**New Features and Improvements**
* Support parallel dataset verification ([ClearML GitHub issue #1130](https://github.com/allegroai/clearml/issues/1130))
* Support `lightgbm` model loading from a string ([ClearML GitHub issue #1135](https://github.com/allegroai/clearml/issues/1135))
* Add support for Hydra command-line syntax for modifying `omegaconf`
* Add support for setting `output_uri` in pipeline decorators and functions
* Add more visibility when overriding `jsonargparse` arguments
* Use `StorageManager` cache when downloading models
* Support parsing GS credentials as a JSON string if they do not point to a JSON file
* Support ignoring `default_output_uri` on remote runs when `Task.init(output_uri=False)` was specified and the backend task data has no `output_uri` specified
* Add configuration files logging [example](https://github.com/allegroai/clearml/blob/e6f1cc8a5c65f7007cf6452745d09795e776a548/examples/reporting/config_files.py)
* Support raising `LoginError` when initializing
* Add Hugging Face Transformers example ([ClearML GitHub PR #1146](https://github.com/allegroai/clearml/pull/1146))
* Improve docstrings for tag filters
* Add a security policy file to the repository ([ClearML GitHub issue #1148](https://github.com/allegroai/clearml/issues/1148))
**Bug Fixes**
* Fix Google Storage subdir path in bucket URL breaks validation ([ClearML GitHub PR #1117](https://github.com/allegroai/clearml/pull/1117))
* Fix jsonschema `DeprecationWarning` in newer versions ([ClearML GitHub PR #1128](https://github.com/allegroai/clearml/pull/1128))
* Fix `task.set_script(diff="")` does not clear the uncommitted changes ([ClearML GitHub issue #1106](https://github.com/allegroai/clearml/issues/1106))
* Fix parallel dataset verification fails on older Python versions ([ClearML GitHub issue #1144](https://github.com/allegroai/clearml/issues/1144))
* Fix `attrs.__version__` deprecation warning ([ClearML GitHub issue #1145](https://github.com/allegroai/clearml/issues/1145))
* Fix `Task.get_parameters_as_dict(cast=True)` raises a YAML parser error
* Fix potential threading issues when setting `self.lock` to `None`
* Fix pipeline version is not displayed in the table view and version list
* Fix displayed pipeline run time in pipeline components is not the same as that shown in the info tab in full detail
* Fix `Model.connect()` in remote execution might result in the wrong model being connected
* Fix exception raised on `ConnectionError` (which actually indicates that a read has timed out, and should be retried)
* Fix missing configuration error not raised in certain cases
* Fix loading queues containing metadata fails due to an incorrect queue metadata scheme
* Fix `pyjwt` requirements for Python < 2.9
* Fix pipeline examples
### ClearML 1.13.1
**Bug Fixes**
* Fix Hydra overrides integration ([ClearML GitHub issue #1120](https://github.com/allegroai/clearml/issues/1120))
* Fix GPU memory used reports 0 when memory can not be queried per process (edge case)
### ClearML 1.13.0
**New Features**
* Add support for recursive list, dict, and tuple ref parsing for `PipelineController.add step()` parameter overrides ([ClearML GitHub issue #1089](https://github.com/allegroai/clearml/issues/1089))
* Update PyNVML to the latest NVIDIA version for better GPU monitoring
* Add `force_single_script_file` argument to `Task.create()` to avoid git repository auto detection
* Use `os.register_at_fork` instead of monkey patching fork for `python>3.6`
* Add support to programmatically archive and unarchive models from the model registry using the `Model.archive()` and
`Model.unarchive()` methods ([ClearML GitHub issue #1096](https://github.com/allegroai/clearml/issues/1096))
**Bug Fixes**
* Fix model reporting in TensorFlow 2.13 does not work properly ([ClearML GitHub issue #1112](https://github.com/allegroai/clearml/issues/1112))
* Fix GPU memory used reports 0 when memory can not be queried per process
* Fix FastAI binding not reporting scalars if TensorBoard is installed
* Fix JSON serialization error does not raise exception
* Fix OutputModel should prefer connected task name over filename stem
* Fix dataset zip extraction might fail when creating folders concurrently (with `Exception [Errno 17] File exists` warning)
* Fix import sys should not appear in pipeline component auto-detected packages
* Fix compatibility issue with `requests<2.28`

73
docs/release_notes/sdk/open_source/ver_1_14.md Normal file
View File

@@ -0,0 +1,73 @@
---
title: Version 1.14
---
### ClearML 1.14.4
**Bug Fixes**
* Fix AWS S3 configuration `secure` property not forwarded in `StorageHelper` ([ClearML GitHub PR #1205](https://github.com/allegroai/clearml/pull/1205))
* Fix `NVIDIA_VISIBLE_DEVICES=all` env var value will result in no GPU metrics being reported
* Fix setting `output_uri` when an AWS S3 config section with `secure=false` exists raises an error
* Fix `CacheManager` attempts to clear cache when file limit has not been reached resulting in slow unnecessary checks
### ClearML 1.14.3
**New Features and Bug Fixes**
* Add hash check for pickle files (resolves [CVE-2024-24590](https://github.com/advisories/GHSA-cpcw-9h9m-wqw9))
* Fix `clearml-task` relative paths calculation when `--cwd`, `--folder`, and `--script` are provided ([ClearML GitHub issue #1160](https://github.com/allegroai/clearml/issues/1160))
* Fix regression causing an error when `secure` is provided in AWS S3 bucket settings
### ClearML 1.14.2
**New Features and Improvements**
* Report upload/download with `tqdm` if installed
* Add support for a `PipelineController.version` property
* Add support for specifying an AWS profile when providing bucket storage credentials
* Add an `ignore_remote_overrides` argument to `Task.connect()` and `Task.connect_configuration()` allowing users to
disable overriding values when running remotely
* Add support for setting a custom certificate path using the `CLEARML_API_HOST_VERIFY_CERT` environment variable
(instead of just allowing the user to disable verification)
* Add dataset reporting example ([ClearML GitHub PR #1180](https://github.com/allegroai/clearml/pull/1180))
**Bug Fixes**
* Fix missing component callbacks on multiple step calls ([ClearML GitHub PR #1195](https://github.com/allegroai/clearml/pull/1195))
* Fix unsafe usage of `tempfile.mktemp`
* Fix potential path traversal on file download ([CVE-2024-24591](https://github.com/advisories/GHSA-m95h-p4gg-wfw3))
* Fix Python 3.12 support by not using `distutils`
* Fix `TaskScheduler` does not save local state before running remotely
* Fix pipeline should continue instead of aborting when `PipelineController.abort_on_failure` is set to `False` and a
step has been aborted
* Fix HPO crashes when its corresponding task is initialized with `continue_last_task=True`
### ClearML 1.14.1
**Bug Fixes**
* Fix created pipelines do not appear in the UI pipelines section
### ClearML 1.14.0
**New Features and Improvements**
* Add support for decorated pipeline steps ([ClearML GitHub issue #1154](https://github.com/allegroai/clearml/issues/1154))
* Improve `torch.distributed` behavior and auto-detect Huggingface accelerate distributed execution
* Improve `jsonargparse` integration: `allow_config_file_override_from_ui` argument added for enabling/disabling
overriding configuration file entries
* Add `raise_on_error` parameter to `Model.update_weights()` and `Model.update_weights_package()`
* Add HPO multi-objective optimization support by specifying multiple optimization objectives to the HyperParameterOptimizer class
* Improve pipelines from functions: libraries used by pipeline steps no longer have to be imported in the step itself
and can be imported globally instead
* Support clearing repository details in `Task.set_repo()`
* Auto-detect `tensorflow-intel` as `tensorflow`
* Add support for auto-detecting `torch` and transformers `accelerate` distributed execution
* Add tabular data multi-stage pipeline process example
**Bug Fixes**
* Fix `langchain` integration ([ClearML GitHub issues #1124](https://github.com/allegroai/clearml/issues/1124) and [ClearML GitHub issue #1126](https://github.com/allegroai/clearml/issues/1126))
* Fix using `target_project` with `TaskScheduler.add_task()` corrupts project if used with pipelines ([ClearML GitHub issue #1137](https://github.com/allegroai/clearml/issues/1137))
* Fix `PipelineController.add_function_step()` fails when adding custom decorated step ([ClearML GitHub issue #1154](https://github.com/allegroai/clearml/issues/1154))
* Fix `jsonargparse` binding does not capture parameters before `Task.init()` is called ([ClearML GitHub issue #1164](https://github.com/allegroai/clearml/issues/1164))
* Fix invalid `jsonargparse` arguments should be deleted
* Fix handling of appended parameters to Hydra defaults list
* Fix `Task.set_repo()` and `Task.set_packages()` do not wait for automatic asynchronous repository detection
* Fix active GPU filtering in resource monitor to support GPU ID and not just GPU idx
* Fix misleading error raised when using the SDK with no configuration file or an invalid configuration file
* Fix process sometimes hangs issue by improving exit and exception handlers, unregistering signal handling in child
processes except for the `BackgroundMonitor`

35
docs/release_notes/sdk/open_source/ver_1_15.md Normal file
View File

@@ -0,0 +1,35 @@
---
title: Version 1.15
---
### ClearML 1.15.1
**Bug Fixes**
* Fix autoscaler should recheck that the worker is still idle before shutting it down ([ClearML GitHub issue #1202](https://github.com/allegroai/clearml/issues/1202))
* Fix resource monitor fails to get GPU stats in some edge cases
### ClearML 1.15.0
**New Features**
* Add draft option to pipeline steps ([ClearML GitHub PR #1226](https://github.com/allegroai/clearml/pull/1226))
* Add support for custom working directory for pipelines ([ClearML GitHub issue #1194](https://github.com/allegroai/clearml/issues/1194))
* Add `Task.get_requirements()` method that returns the task's requirements
* Allow controlling the number of threads used by `StorageManager.download_folder()` using the `max_workers` argument
* Update examples dependencies
* Improve auto populate in `Task.init()`
**Bug Fixes**
* Fix UTF-8 script code encoding issue ([ClearML GitHub issue #1207](https://github.com/allegroai/clearml/issues/1207))
* Fix Colab docs ([ClearML GitHub issue #1204](https://github.com/allegroai/clearml/issues/1204))
* Fix metrics reporting with `OutputModel` while in offline mode results in an error ([ClearML GitHub issue #1172](https://github.com/allegroai/clearml/issues/1172))
* Fix task running in Google Colab doesn't properly get the notebook diff ([ClearML GitHub issue #1204](https://github.com/allegroai/clearml/issues/1204))
* Fix hydra binds break in offline mode ([ClearML GitHub issue #1215](https://github.com/allegroai/clearml/issues/1215))
* Fix HPO crashes when optimizing for single value scalars ([ClearML GitHub issue #1221](https://github.com/allegroai/clearml/issues/1221))
* Fix GPU info such as `gpu_memory` and `gpu_type` is not being collected in some cases
* Fix `clearml-data` CLI tool moves non-dataset tasks to a `.dataset` project when the respective task is not a dataset
* Fix pandas `DataFrame` artifacts with hierarchical indices get mangled by the CSV round-trip
* Fix `urllib3` sends deprecation warning when setting `ssl_version`
* Fix `Task.connect(dict)` return value is not dict-compatible
* Fix `jsonargparse` sub-command config parsing
* Fix Lightning integration crashes when a config entry contains `.` in its name
* Fix Python 3.5 compatibility

82
docs/release_notes/sdk/open_source/ver_1_16.md Normal file
View File

@@ -0,0 +1,82 @@
---
title: Version 1.16
---
### ClearML 1.16.4
**New Features**
* Add custom task binary support to `clearml-task` and `CreateAndPopulate` (allows bash script execution, requires agent version `>=1.9.0`)
* Add support for a default extension name when uploading a pandas `dataframe` artifact (see `sdk.development.artifacts.default_pandas_dataframe_extension_name`
configuration option)
* Add `verify` field support for downloadable URL instead of a file path (see `sdk.aws.s3` configuration section)
**Bug Fix**
* Fix valid model URL might be overridden by an invalid one in case the upload failed
### ClearML 1.16.3
**New Features and Improvements**
* Add `--tags` option to `clearml-task` ([ClearML GitHub issue #1284](https://github.com/allegroai/clearml/issues/1284))
* Add `retries` parameter to `StorageManager.upload_folder()` ([ClearML GitHub issue #1305](https://github.com/allegroai/clearml/issues/1305))
* Add `clearml-task` and `CreateAndPopulate` support for bash scripts, ipynb, and python modules (requires `clearml-agent v1.9+`)
* Add support for HTTP file upload progress reporting
* Add `CLEARML_MULTI_NODE_SINGLE_TASK` (values -1, 0, 1, 2) for easier multi-node single Task workloads
* Add `Model.original_task` property to models
* Change `Model.task` property to return connected task
**Bug Fixes**
* Fix Kerastuner framework and examples ([ClearML GitHub issue #1279](https://github.com/allegroai/clearml/issues/1279))
* Fix scalar logging bug with Fire ([ClearML GitHub issue #1300](https://github.com/allegroai/clearml/issues/1300))
* Fix support passing folder to `Task.get_script_info()` to get the git info
* Fix `Task.launch_multi_node()` to enforce the parent of sub-tasks to be the master node 0 task
* Fix tensorboard numpy 2.0 incompatibility breaks binding
* Fix `Task.launch_multi_node()` not supported when used via Pytorch Lightning
* Fix Jupyter notebook packages and uncommitted changes are sometimes not fetched
* Fix `"can't create new thread at interpreter shutdown"` errors (known issue with Python v3.12.0 and other versions)
* Fix injected task import in `Task.populate()`
* Fix dataset with external links will not reuse downloaded data from parents
* Fix hierarchy for pipeline nodes without args
* Fix when abort callback is set, set task status to stopped only if running locally, otherwise leave it for the Agent to set it
* Fix `jsonschema` / `referencing` import to include `TypeError` protection
* Fix Dataset offline behavior
### ClearML 1.16.2
**New Features and Improvements**
* Make dataset preview optional ([ClearML GitHub issue #1227](https://github.com/allegroai/clearml/issues/1227))
* Add `api.public_ip_ping` (default: `8.8.8.8`) and `api.public_ip_service_urls` (default: `["api.ipify.org";, "ident.me";]`)
configuration settings to detect public IP and network interface
**Bug Fixes**
* Fix python-fire integration ([ClearML GitHub issue #1268](https://github.com/allegroai/clearml/issues/1268))
* Fix path substitution for `file://` URIs (ClearML GitHub issues [#1217](https://github.com/allegroai/clearml/issues/1217)
and [#1233](https://github.com/allegroai/clearml/issues/1233))
* Fix numpy 2.0 compatibility (`np.NINF` removed)
* Fix no need to recreate reporter if forking and reporting in subprocess
* Fix forked detection mechanism
### ClearML 1.16.1
**Bug Fixes**
* Fix pipeline breaks when `continue_on_abort` is set to `true`
* Fix Pycharm Plugin Windows/Linux interoperability
### ClearML 1.16.0
**New Features**
* Add additional warning instructing on how to install in case we failed detecting a Jupyter notebook with an import error
* Add `Task.get_executed_queue()` to get the name/ID of the queue a task was executed in
* Move `Task.set_resource_monitor_iteration_timeout()` to a class method, add `wait_for_first_iteration_to_start_sec`
and `max_wait_for_first_iteration_to_start_sec` arguments (also add `sdk.development.worker.wait_for_first_iteration_to_start_sec`
and `sdk.development.worker.max_wait_for_first_iteration_to_start_sec` configuration options)
* Add support for better pipeline continue behavior including control of children using the `continue_behaviour` argument
* Add Python 3.12 support
**Bug Fixes**
* Fix pytorch-lightning patches ([ClearML GitHub issue #1249](https://github.com/allegroai/clearml/issues/1249))
* Fix parameter overrides are converted to string when using HPO ([ClearML GitHub issue #975](https://github.com/allegroai/clearml/issues/975))
* Fix FastAI performance ([ClearML GitHub issue #1234](https://github.com/allegroai/clearml/issues/1234))
* Fix MIG GPU support
* Fix AMD GPU metrics collection
* Fix Jupyter password might not be used in some protected JupyterLab instances
* Fix URL substitution was not applied to registered uploaded files when reporting an event

56
docs/release_notes/sdk/open_source/ver_1_2.md Normal file
View File

@@ -0,0 +1,56 @@
---
title: Version 1.2
---
### ClearML 1.2.1
**Bug Fixes**
- Fix HTTP download fails constructing URL ([ClearML GitHub issue #593](https://github.com/allegroai/clearml/issues/593))
### ClearML 1.2.0
**Features**
- Add fastai v2 support ([ClearML GitHub PR #571](https://github.com/allegroai/clearml/pull/571))
- Add catboost support ([ClearML GitHub PR #542](https://github.com/allegroai/clearml/pull/542))
- Add Python Fire support ([ClearML GitHub PR #550](https://github.com/allegroai/clearml/pull/550))
- Add new Azure Storage driver support ([ClearML GitHub PR #548](https://github.com/allegroai/clearml/pull/548))
- Add requirements file support in `Task.add_requirements` ([ClearML GitHub PR #575](https://github.com/allegroai/clearml/pull/575))
- Allow overriding `auto_delete_file` in `Task.update_output_model()` ([ClearML GitHub issue #554](https://github.com/allegroai/clearml/issues/554))
- Support `artifact_object` empty string
- Add `skip_zero_size_check` to `StorageManager.download_folder()`
- Add support for extra HTTP retry codes (see [here](https://github.com/allegroai/clearml/blob/2c916181b90c784fe0bd267cd67ea915e53e36e4/clearml/backend_api/config/default/api.conf#L29) or use `CLEARML_API_EXTRA_RETRY_CODES`)
- Add `Task.get_parameters()` cast back to original type
- Add callback support to `Task.delete()`
- Add autoscaler CPU-only support
- Add AWS autoscaler IAM instance profile support
- Update examples
- Edit HTML reporting examples ([ClearML GitHub PR #546](https://github.com/allegroai/clearml/pull/546))
- Add model reporting examples ([ClearML GitHub PR #553](https://github.com/allegroai/clearml/pull/553))
**Bug Fixes**
- Fix `nargs="?"` without type does not properly cast the default value ([ClearML GitHub issue #531](https://github.com/allegroai/clearml/issues/531))
- Fix using invalid configurations ([ClearML GitHub issue #544](https://github.com/allegroai/clearml/issues/544))
- Fix extra_layout not passed to report_matrix ([ClearML GitHub issue #559](https://github.com/allegroai/clearml/issues/559))
- Fix group arguments in click ([ClearML GitHub PR #561](https://github.com/allegroai/clearml/pull/561))
- Fix no warning when failing to patch argparse ([ClearML GitHub PR #576](https://github.com/allegroai/clearml/pull/576))
- Fix crash in `Dataset.upload()` when there is nothing to upload ([ClearML GitHub PR #579](https://github.com/allegroai/clearml/pull/579))
- Fix requirements, refactor and reformat examples ([ClearML GitHub PRs #567](https://github.com/allegroai/clearml/pull/567), [#573](https://github.com/allegroai/clearml/pull/573), [#582](https://github.com/allegroai/clearml/pull/582))
- Auto-scaler
- Change confusing log message
- Fix AWS tags support
- Fix instance startup script fails on any command (should only fail on the agent failing to launch)
- Fix spin down stuck machine, ignore unknown stale workers
- Fix pandas object passed as `Task.upload_artifact()` preview object
- Fix incorrect timeout used for stale workers
- Fix `clearml-task` calls `Task.init()` in the wrong place when a single local file is used
- Fix ArgumentParser `SUPPRESS` as default should be resolved at remote execution in the same way (i.e. empty string equals `SUPPRESS`)
- Upgrade six version (in case `pathlib2>2.3.7` is installed)
- Fix connected object base class members are not used
- Fix `clearml-init` changing web host after pasting full credentials
- Fix fileserver upload does not support path in URL
- Fix crash on semaphore acquire error
- Fix docs and docstrings ([ClearML GitHub PRs #558](https://github.com/allegroai/clearml/pull/558), [#560](https://github.com/allegroai/clearml/pull/560))

60
docs/release_notes/sdk/open_source/ver_1_3.md Normal file
View File

@@ -0,0 +1,60 @@
---
title: Version 1.3
---
### ClearML 1.3.2
**New Features and Improvements**
* Add support for setting reported values for `NaN` and `Inf` ([ClearML GitHub issue #604](https://github.com/allegroai/clearml/issues/604))
* Add reserved OS environments warning
* Add git credentials to colab example ([ClearML GitHub PR #621](https://github.com/allegroai/clearml/pull/621))
* Add jsonargparse support ([ClearML GitHub issue #403](https://github.com/allegroai/clearml/issues/403))
* Update autokeras example
**Bug Fixes**
* Fix sub-project separators are incorrectly quoted in generated URLs ([ClearML GitHub PR #584](https://github.com/allegroai/clearml/pull/584))
* Revert Optuna deprecation fix ([ClearML GitHub PR #613](https://github.com/allegroai/clearml/pull/613))
* Fix HPO randomly aborts running tasks before the time limit
* Fix cloud driver overwrites `agent.extra_docker_arguments`
* Fix Pipeline Controller auto-magic framework connect
* Fix unused scroll is not cleared in `Task.get_reported_plots()`
### ClearML 1.3.1
**New Features and Improvements**
* Add Python 3.10 support
**Bug Fixes**
* Update Slack SDK requirement ([ClearML GitHub issue #597](https://github.com/allegroai/clearml/issues/597))
* Fix fork after `task.close()` is called ([ClearML GitHub issue #605](https://github.com/allegroai/clearml/issues/605))
* Fix Azure storage upload ([ClearML GitHub issue #598](https://github.com/allegroai/clearml/issues/598))
* Fix offline mode crash
* Fix task delete response not checked
* Fix pipeline controller kwargs with list
* Fix `PipelineDecorator.debug_pipeline()`
* Fix PipelineDecorator example
* Fix Python 3.10 issues
* Fix handling of legacy fileserver (files.community.clear.ml)
* Fix cloud driver may use None credentials
* Fix APIClient worker raises exception when accessing .name attribute
* Fix minimum/default API version setting
### ClearML 1.3.0
**New Features and Improvements**
* Add new pipeline visualization support (requires ClearML Server v1.3)
* Support IAM Instance Profile in AWS auto-scaler
* Remove old server API versions support (pre-ClearML Server)
* Restructure FastAI examples
**Bug Fixes**
* Fix failed catboost bind on GPU ([ClearML GitHub issue #592](https://github.com/allegroai/clearml/issues/592))
* Fix Optuna `n_jobs` deprecation warning
* Fix invalid method called on `delete()` error

61
docs/release_notes/sdk/open_source/ver_1_4.md Normal file
View File

@@ -0,0 +1,61 @@
---
title: Version 1.4
---
### ClearML 1.4.1
**Bug Fixes**
* Fix Process Pool hangs at exit ([ClearML GitHub issue #674](https://github.com/allegroai/clearml/issues/674))
### ClearML 1.4.0
**New Features**
* Add OpenMMLab example ([ClearML GitHub issue #654](https://github.com/allegroai/clearml/issues/654))
* Add support for saving artifacts with different formats ([ClearML GitHub issue #634](https://github.com/allegroai/clearml/issues/634))
* Add support for setting reported values for `NaN` and `Inf` ([ClearML GitHub issue #604](https://github.com/allegroai/clearml/issues/604))
* Support more than 500 results in `Task.get_tasks()` using the `fetch_only_first_page` argument ([ClearML GitHub issue #612](https://github.com/allegroai/clearml/issues/612))
* Support links in `clearml-data` ([ClearML GitHub issue #585](https://github.com/allegroai/clearml/issues/585))
* Support deferred task initialization using `Task.init()` argument `deferred_init` (beta feature)
* Support resuming experiments when importing an Offline session
* Add `--import-offline-session` command line option to `clearml-task`
* Support automatically logging Tensorboard Hparams
* Add wildcard support for model auto-logging, see [`Task.init()`](../../../references/sdk/task.md#taskinit)
* Add support for Lightning CLI
* Support None values in `Task.connect()`
* Add `Model.project` getter/setter
* Add support for Task progress indication
* Datasets
* Improve Dataset version table
* Add warning to Dataset creation on current Task
* Examples and documentation
* Add manual seaborn logging example ([ClearML GitHub PR #628](https://github.com/allegroai/clearml/pull/628))
* Change package author
* Change pipeline example to run locally ([ClearML GitHub PR #642](https://github.com/allegroai/clearml/pull/642))
* Update PyTorch Lightning example for `pytorch-lightning>=v1.6.0` ([ClearML GitHub PR #650](https://github.com/allegroai/clearml/pull/650))
**Bug Fixes**
* Fix Keras model config serialization in `PatchKerasModelIO` ([ClearML GitHub issue #614](https://github.com/allegroai/clearml/issues/614))
* Fix `task.get_parameters_as_dict(cast=True)` casts `False` to `True` ([ClearML GitHub PR #622](https://github.com/allegroai/clearml/pull/622))
* Fix Fire integration is not compatible with typing library ([ClearML GitHub issue #610](https://github.com/allegroai/clearml/issues/610))
* Fix remote execution with `argparse` mutually exclusive groups raises "required" error even when no argument is required
* Fix Hydra tasks never fail and are only set to completed (fix handling return code)
* Fix `clearml-data` wildcard support
* Fix HPO randomly aborts running tasks before the time limit
* Fix matplotlib capture
* Fix issue with accessing images in projects containing `/`
* AutoScaler
* Fix resource name with a prefix matching a resource type may cause the auto-scaler to avoid spinning down idle instances
* Fix Idle workers should contain resource name and not instance type
* Fix backwards compatibility issue when using `abstractmethod`
* Matplotlib
* Fix uploading 3D plots with matplotlib plt shows 2D plot on task results page
* Fix wrong Histogram plotting using when matplotlib
* Fix PyTorch `ScriptModule` autobind
* Fix PyTorch auto-magic logging TorchScript models
* Fix forked process will not call `_at_exit` and flush all outstanding reports
* Fix matplotlib to plotly conversion fails on subplots (convert as image if figure has subplots)
* Fix Windows sub process might end up waiting forever for uploads to finish if subprocess is very shot lived
* Fix `StorageManager.get_local_copy()` returning None for a valid path in Windows
* Fix Jupyter notebook cannot be detected
* Fix PipelineController does not change node Task name, only pipeline step name
* Fix `Task.query_tasks()` specifying page size or page number

31
docs/release_notes/sdk/open_source/ver_1_5.md Normal file
View File

@@ -0,0 +1,31 @@
---
title: Version 1.5
---
### ClearML 1.5.0
**New Features and Improvements**
* Add support for single value metric reporting ClearML GitHub issue ([ClearML GitHub issue #400](https://github.com/allegroai/clearml/issues/400))
* Add support for specifying parameter sections in `PipelineDecorator` ([ClearML GitHub issue #629](https://github.com/allegroai/clearml/issues/629))
* Add support for parallel uploads and downloads (upload / download and zip / unzip of artifacts)
* Add support for specifying execution details (repository, branch, commit, packages, image) in `PipelineDecorator`
* Bump PyJWT version due to "*Key confusion through non-blocklisted public key formats*" vulnerability
* Add support for AWS Session Token (using boto3's `aws_session_token` argument)
**Bug Fixes**
* Fix `Task.get_projects()` retrieves only the first 500 results ([ClearML GitHub issue #612](https://github.com/allegroai/clearml/issues/612))
* Fix failure to delete artifacts stored in Azure ([ClearML GitHub issue #660](https://github.com/allegroai/clearml/issues/660))
* Fix Process Pool hangs at exit ([ClearML GitHub issue #674](https://github.com/allegroai/clearml/issues/674))
* Fix number of unpacked values when syncing a dataset ([ClearML GitHub issue #682](https://github.com/allegroai/clearml/issues/682))
* Fix FastAI DeprecationWarning ([ClearML GitHub PR #683](https://github.com/allegroai/clearml/issues/683))
* Fix `StorageManager.download_folder()` crash
* Fix pipelines can't handle `None` return value
* Fix pre-existing pipeline raises an exception
* Fix deprecation warning in the `image_reporting` example
* Fix patches are kept binded after `Task.close()` is called
* Fix running pipeline code remotely without first running it locally (i.e. no configuration on the Task)
* Fix local task execution with empty working directory
* Fix permission check fails when using local storage folder that does not exist
* Fix pipeline `add_function_step` breaks in remote execution
* Fix wrong mimetype used for any file or folder uploaded to S3 using `StorageManager`
* Add missing default `default_cache_manager_size` in configuration files

96
docs/release_notes/sdk/open_source/ver_1_6.md Normal file
View File

@@ -0,0 +1,96 @@
---
title: Version 1.6
---
### ClearML 1.6.4
**Bug Fix**
* Fix `APIClient` fails when calling `get_all` endpoints with API 2.20 (affects CLI tools such as `clearml-session`)
### ClearML 1.6.3
**New Features and Improvements**
* Add option to specify an endpoint URL when creating S3 resource service ([ClearML GitHub issue #673](https://github.com/allegroai/clearml/issues/673))
* Add support for providing `ExtraArgs` to boto3 when uploading files using the `sdk.aws.s3.extra_args` configuration option
* Add support for Server API 2.20
* Add `Task.get_num_enqueued_tasks()` to get the number of tasks enqueued in a specific queue
* Add support for updating model metadata using `Model.set_metadata()`, `Model.get_metadata()`, `Model.get_all_metadata()`,
`Model.get_all_metadata_casted()`, and `Model.set_all_metadata()`
* Add `Task.get_reported_single_value()`
* Add a retry mechanism for models and artifacts upload
* Pipelines with empty configuration takes it from code
* Add support for running pipeline steps on preemptible instances
* Datasets
* Add description to Datasets
* Add wild-card support in `clearml-data`
**Bug Fixes**
* Fix dataset download ([ClearML GitHub issue #713](https://github.com/allegroai/clearml/issues/713))
* Fix lock is not released after dataset cache is downloaded ([ClearML GitHub issue #671](https://github.com/allegroai/clearml/issues/671))
* Fix deadlock might occur when using process pool large number processes ([ClearML GitHub issue #674](https://github.com/allegroai/clearml/issues/674))
* Fix 'series' not appearing on UI when using `logger.report_table()` ([ClearML GitHub issue #684](https://github.com/allegroai/clearml/issues/684))
* Fix `Task.init()` docstring to include behavior when executing remotely ([ClearML GitHub PR #737](https://github.com/allegroai/clearml/pull/737))
* Fix `KeyError` when running remotely and no params were passed to click ([ClearML Agent GitHub issue #111](https://github.com/allegroai/clearml-agent/issues/111))
* Fix full path is stored when uploading a single artifact file
* Fix passing non-alphanumeric filename in `sdk.development.detect_with_pip_freeze`
* Fix Python 3.6 and 3.10 support
* Fix mimetype cannot be `None` when uploading to S3
* Pipelines
* Fix pipeline DAG
* Add support for pipelines with spot instances
* Fix pipeline proxy object is always resolved in main pipeline logic
* Fix pipeline steps with empty configuration should try and take it from code
* Fix wait for jobs based on local/remote pool frequency
* Fix `UniformIntegerParameterRange.to_list()` ignores min value
* Fix pipeline component returning a list of length 1
* Datasets
* Fix `Dataset.get()` does not respect `auto_create`
* Fix getting datasets fails with new ClearML Server v1.6
* Fix datasets can't be queried by project/name alone
* Fix adding child dataset to older parent dataset without stats
* Fix error when connecting an input model
* Fix deadlocks, including:
* Change thread Event/Lock to a process fork safe threading objects
* Use file lock instead of process lock to avoid future deadlocks since python process lock is not process safe
(killing a process holding a lock will Not release the lock)
* Fix `StorageManager.list()` on a local Windows path
* Fix model not created in the current project
* Fix `keras_tuner_cifar` example raises DeprecationWarning and ValueError
### ClearML 1.6.2
**Bug Fix**
* Fix format string construction sometimes causing delayed evaluation errors ([ClearML GitHub issue #706](https://github.com/allegroai/clearml/issues/706))
### ClearML 1.6.1
**Bug Fixes**
* Fix `Task.get_tasks()` fails when sending `search_hidden=False`
* Fix LightGBM example shows UserWarning
### ClearML 1.6.0
**New Features and Improvements**
* New Hyperparameter Optimization CLI `clearml-param-search`
* Improvements to ClearML Data
* Add support for a new ClearML Data UI in the ClearML WebApp
* Add `clearml-data` new options set-description and rename
* Add random seed control using `Task.set_random_seed()` allowing to set a new random seed for task initialization or
to disable it
* Improve error messages when failing to download an artifact
* Improve error messages when testing for permissions
**Bug Fixes**
* Fix axis range settings when logging plots
* Fix `Task.get_project()` to return more than 500 entries ([ClearML GitHub issue #612](https://github.com/allegroai/clearml/issues/612))
* Fix pipeline progress calculation
* Fix `StorageManager.upload_folder()` returns `None` for both successful and unsuccessful uploads
* Fix script path capturing stores a relative path and not an absolute path
* Fix HTML debug samples are saved incorrectly on S3
* Fix Hydra deprecation warning in examples
* Fix missing requirement for TensorBoardX example
**Known Issues**
* When removing an image from a Dataset, its preview image won't be removed
* Moving Datasets between projects still shows the Dataset in the old project

71
docs/release_notes/sdk/open_source/ver_1_7.md Normal file
View File

@@ -0,0 +1,71 @@
---
title: Version 1.7
---
### ClearML 1.7.2
**New Features and Improvements**
* Support running Jupyter Notebook inside a git repository (repository will be referenced without uncommitted changes
and Jupyter Notebook will be stored on plain code as uncommitted changes)
* Add Jupyter Notebook fail warning
* Allow pipeline steps to return string paths without them being treated as a folder artifact and zipped ([ClearML GitHub issue #780](https://github.com/allegroai/clearml/issues/780))
* Remove `future` from Python 3 requirements
**Bug Fixes**
* Fix exception raised when using `ThreadPool` ([ClearML GitHub issue #790](https://github.com/allegroai/clearml/issues/790))
* Fix Pyplot/Matplotlib binding reports incorrect line labels and colors ([ClearML GitHub issue #791](https://github.com/allegroai/clearml/issues/791))
* Pipelines
* Fix crash when running cloned pipeline that invokes a step twice ([ClearML GitHub issue #769](https://github.com/allegroai/clearml/issues/769))
* Fix pipeline argument becomes `None` if default value is not set
* Fix `retry_on_failure` callback does nothing when specified on `PipelineController.add_step()`
* Fix pipeline clone logic
* Jupyter Notebook
* Fix support for multiple Jupyter servers running on the same machine
* Fix issue with old/new notebook packages installed
* Fix local cache with access rules disabling partial local access
* Fix `Task.upload_artifact()` fails uploading pandas `DataFrame`
* Fix relative paths in examples ([ClearML GitHub PR #787](https://github.com/allegroai/clearml/issues/787)
### ClearML 1.7.1
**New Features and Improvements**
* Add callback option for pipeline step retry
**Bug Fixes**
* Fix Python Fire binding
* Fix Dataset failing to load helper packages should not crash
* Fix `Dataset.get_local_copy()` is allowed for a non-finalized dataset
* Fix `Task.upload_artifact()` does not upload empty lists/tuples
* Fix pipeline retry mechanism interface
* Fix `Python <3.5` compatibility
* Fix local cache warning (should be a debug message)
### ClearML 1.7.0
**New Features and Improvements**
* ClearML Data: Support providing list of links
* Upload artifacts with a custom serializer ([ClearML GitHub issue #689](https://github.com/allegroai/clearml/issues/689))
* Allow user to specify extension when using custom serializer functions (for artifacts)
* Skip server URL verification in `clearml-init` wizard process
* When calling `Dataset.get()` without "alias" field, tell user that he can use alias to log it in the UI
* Add support for mmcv for logging models
* Add support for Azure and GCP storage in `task.setup_upload()`
* Support pipeline retrying tasks which are failing on suspected non-stable failures
* Better storage (AWS, GCP) internal load balancing and configurations
* Add `Task.register_abort_callback`
**Bug Fixes**
* Allow getting datasets with non-semantic versioning ([ClearML GitHub issue #776](https://github.com/allegroai/clearml/issues/776))
* Fix interactive plots (instead of a generated png)
* Fix Python 2.7 support
* Fix clearml datasets `list` functionality
* Fix `Dataset.init` to not modify task (to `Dataset.create`)
* Fix failure with large files upload on HTTPS
* Fix 3d plots with plt shows 2d plot on task results page
* Fix uploading files with project's `default_upload_destination` ([ClearML GitHub issue #734](https://github.com/allegroai/clearml/issues/734))
* Fix broken Matplotlib reporting - using logarithmic scale breaks reporting
* Fix wildcard support in `clearml-data` CLI
* Fix `report_histogram` - does not show "horizontal" orientation ([ClearML GitHub issue 699](https://github.com/allegroai/clearml/issues/699))
* Fix table reporting - when using `logger.report_table(title, series, iteration, etc)`, the `series` arg does not appear in UI ([ClearML GitHub issue 684](https://github.com/allegroai/clearml/issues/684))
* Fix artifacts (and models) use task's original name and not new name
* Fix very long filenames from S3 can't be downloaded (with `get_local_copy()`)
* Fix overwrite of existing output models on pipeline task with `monitor_models` ([ClearML GitHub issue #757](https://github.com/allegroai/clearml/issues/757))

70
docs/release_notes/sdk/open_source/ver_1_8.md Normal file
View File

@@ -0,0 +1,70 @@
---
title: Version 1.8
---
### ClearML 1.8.3
**Bug Fixes**
* Set GCS credentials to `None` if invalid service account credentials are provided ([ClearML GitHub issue #840](https://github.com/allegroai/clearml/issues/840))
* Fix a sync issue when loading deferred configuration
### ClearML 1.8.2
**New Features and Improvements**
* Added `VCS_ENTRY_POINT` environment variable that overrides ClearML's entrypoint auto-detection
**Bug Fixes**
* Fix all parameters returned from a pipeline are considered strings
* Fix `Task.set_parameters()` does not add parameter type when parameter exists but does not have a type
### ClearML 1.8.1
**New Features and Improvements**
* Raise error on failed uploads ([ClearML GitHub issue #810](https://github.com/allegroai/clearml/issues/819))
* Add hyperdataset examples ([ClearML GitHub PR #823](https://github.com/allegroai/clearml/commit/f6b9efe54e1246adba4036c56bc6e8a0bdb99948))
* Change `report_event_flush_threshold` default to 100
* Add `ModelInfo.weights_object()` to store callback access to the actual model object being stored (valid for both
pre/post save calls, otherwise `None`)
* Support `num_workers` in dataset operation
* Support max connections setting for Azure storage using the `sdk.azure.storage.max_connection` configuration option
**Bug Fixes**
* Fix clearml logger default level cannot be changed ([ClearML GitHub issue #741](https://github.com/allegroai/clearml/issues/741))
* Fix Hydra doesn't get overridden information from ClearML ([ClearML GitHub issue #751](https://github.com/allegroai/clearml/issues/751))
* Fix `StorageManager.list(“s3://..”, with_metadata=True)` doesn't work
* Fix `ModelsList.keys()` is missing
* Fix `CLEARML_DEFERRED_TASK_INIT=1` doesn't work
* Fix default API method does not work when set in configuration
### ClearML 1.8.0
**New Features and Improvements**
* Add tarfile member sanitization to `extractall()`([ClearML GitHub PR #803](https://github.com/allegroai/clearml/pull/803))
* Add `Task.delete_artifacts()` with `raise_on_errors` argument ([ClearML GitHub issue #805](https://github.com/allegroai/clearml/issues/805))
* Add CI/CD example ([ClearML GitHub PR #815](https://github.com/allegroai/clearml/pull/815))
* Limit number of `_serialize` requests when adding list of links with `add_external_files()` ([ClearML GitHub issue #813](https://github.com/allegroai/clearml/issues/813))
* Add support for connecting Enum values as parameters
* Improve Colab integration (store entire colab, not history)
* Add `clearml.browser_login` to authenticate browser online sessions such as Colab, Jupyter Notebooks etc.
* Remove `import_bind` from stack trace of import errors
* Add `sdk.development.worker.report_event_flush_threshold` configuration option to control the number of events to trigger a report
* Return stub object from `Task.init()` if no `clearml.conf` file is found
* Improve manual model uploading example
* Remove deprecated demo server
**Bug Fixes**
* Fix passing `compression=ZIP_STORED` (or 0) to `Dataset.upload()` uses `ZIP_DEFLATED` and overrides the user-supplied
argument ([ClearML GitHub PR #812](https://github.com/allegroai/clearml/pull/812))
* Fix `unique_selector` is not applied properly on batches after the first batch. Remove default selector value since
it does not work for all event types (and we always specify it anyway)
* Fix `clearml-init` colab detection
* Fix cloning pipelines run with `start_locally()` doesn't work
* Fix if project has a default output uri there is no way to disable it in development mode (manual), allow passing
`output_uri=False` to disable it
* Fix git remote repository detection when remote is not "origin"
* Fix reported images might not all be reported when waiting to complete the task
* Fix `Dataset.get_local_copy()` deletes the source archive if it is stored locally
* Fix too many parts will cause preview to inflate Task object beyond its 16MB limit - set a total limit of 320kbs
* Fix media preview is created instead of a table preview
* Fix `task.update_output_model()` should always upload local models to a remote server
* Fix broken pip package might mess up requirements detection

80
docs/release_notes/sdk/open_source/ver_1_9.md Normal file
View File

@@ -0,0 +1,80 @@
---
title: Version 1.9
---
### ClearML 1.9.3
**Bug Fixes**
* Fix broken `Task._get_status()`, which was breaking `clearml-session` in the latest version
* Fix path substitution, making it possible to store unsubstituted URL for models ([ClearML GitHub PR #935](https://github.com/allegroai/clearml/pull/935))
### ClearML 1.9.2
**New Features and Improvements**
* Support parsing queue name when providing execution queue in pipelines code ([ClearML GitHub PR #857](https://github.com/allegroai/clearml/pull/857))
* Ignore `None` values for keys in the `click` argument parser ([ClearML GitHub issue #902](https://github.com/allegroai/clearml/issues/902))
* Improve docstrings for `Task.mark_completed()` and `Task.close()` (ClearML GitHub PRs [#920](https://github.com/allegroai/clearml/pull/920) and [#921](https://github.com/allegroai/clearml/pull/921))
* Add pre/post execution callbacks to pipeline steps through `@PipelineDecorator.component`
* Add status-change callback to pipeline steps through `PipelineController.add_step()`, `PipelineController.add_function_step()`,
and `@PipelineDecorator.component`
**Bug Fixes**
* Fix missing debug samples when reporting using TensorBoard ([ClearML GitHub issue #923](https://github.com/allegroai/clearml/issues/923))
* Fix wrong Jupyter API token during repository detection ([ClearML GitHub PR #904](https://github.com/allegroai/clearml/pull/904))
* Fix typo in the warning for very large git diffs ([ClearML GitHub PR #932](https://github.com/allegroai/clearml/pull/932))
* Fix pipelines from tasks don't propagate `parameter_override` values in `PipelineController.add_step()`
* Fix folders and files uploaded to S3 and Azure with `StorageManager.upload_file()` receive wrong MIME types
* Fix the CSV file preview in Datasets
* Fix `Task.connect_configuration()` doesn't work with non-string dictionary keys
* Fix `lightgbm_example` deprecation warning
* Fix potential race condition in `get_or_create_project()`
### ClearML 1.9.1
**New Features and Improvements**
* Add signature version to `boto3` configuration ([ClearML GitHub issue #883](https://github.com/allegroai/clearml/issues/883))
* Allow requesting custom token expiration using the `api.auth.req_token_expiration_sec` configuration setting
* Add Python 3.11 support
**Bug Fixes**
* Fix `UniformParameterRange.to_list` throws error when step size is not defined ([ClearML GitHub issue #858](https://github.com/allegroai/clearml/issues/858))
* Fix `StorageManager.list()` does not return size metadata ([ClearML GitHub issue #865](https://github.com/allegroai/clearml/issues/865))
* Fix storage with path substitutions ([ClearML GitHub issue #825](https://github.com/allegroai/clearml/issues/825))
* Fix extras in ClearML installation prevents clearml from being included in requirements ([ClearML GitHub issue #867](https://github.com/allegroai/clearml/issues/867))
* Fix metadata set on an uploaded model object is not accessible ([ClearML GitHub issue #890](https://github.com/allegroai/clearml/issues/890))
* Fix Azure storage upload not working ([ClearML GitHub issue #868](https://github.com/allegroai/clearml/issues/868))
* Fix `task.connect` list of dicts parsed incorrectly in remote
* Fix casting `None` to `int` fails uploads and permission checks
* Fix numpy 1.24 support
* Fix `clearml-data` previews are saved on file server even when `output_uri` is specified
* Fix connecting a dictionary to task sometimes raises an exception
* Fix authentication headers are not set on substituted fileserver URLs
* Fix `Task.get_project_id()` cannot find hidden projects
* Fix TriggerScheduler docstrings ([ClearML GitHub issue #881](https://github.com/allegroai/clearml/issues/881))
### ClearML 1.9.0
**New Features and Improvements**
* Add `r` prefix to `re.match()` strings ([ClearML GitHub issue #834](https://github.com/allegroai/clearml/issues/834))
* Add `path_substitution` to `clearml.conf` example file ([ClearML GitHub PR #842](https://github.com/allegroai/clearml/pull/842))
* Clarify `deferred_init` usage in `Task.init()` ([ClearML GitHub issue #855](https://github.com/allegroai/clearml/issues/855))
* Add pipeline decorator argument to control docker image ([ClearML GitHub issue #856](https://github.com/allegroai/clearml/issues/856))
* Add `StorageManager.set_report_upload_chunk_size()` and `StorageManager.set_report_download_chunk_size()` to set chunk
size for upload and download
* Add `allow_archived` argument in `Task.get_tasks()`
* Support querying model metadata in `Model.query_models()`
* Add `Dataset.set_metadata()` and `Dataset.get_metadata()`
* Add `delete_from_storage` (default `True`) to `Task.delete_artifacts()`
**Bug Fixes**
* Fix jsonargparse and pytorch lightning integration broken for remote execution ([ClearML GitHub issue #403](https://github.com/allegroai/clearml/issues/403))
* Fix error when using `TaskScheduler` with `limit_execution_time` ([ClearML GitHub issue #648](https://github.com/allegroai/clearml/issues/648))
* Fix dataset not synced if the changes are only modified files ([ClearML GitHub issue #822](https://github.com/allegroai/clearml/issues/822))
* Fix `StorageHelper.delete()` does not respect path substitutions ([ClearML GitHub issue #825](https://github.com/allegroai/clearml/issues/825))
* Fix can't write more than 2 GB to a file
* Fix `StorageManager.get_file_size_bytes()` returns `ClientError` instead of `None` for invalid S3 links
* Fix Dataset lineage view is broken with multiple dataset dependencies
* Fix `tensorflow_macos` support
* Fix crash when calling `task.flush(wait_for_uploads=True)` while executing remotely
* Fix `None` values get casted to empty strings when connecting a dictionary