allegroai
240b762a2a
Allow users to disable remote overrides when using Task.connect() or Task.connect_configuration()
2024-02-04 19:33:41 +02:00
allegroai
233f94f741
Support AWS profile when specifying bucket storage credentials
2024-02-04 19:31:30 +02:00
allegroai
122b308646
Make PipelineController.version attribute publicly accessible
2024-02-04 19:29:38 +02:00
Matteo Destro
56ba25f554
Fix missing component callbacks on multiple step calls ( #1195 )
2024-02-01 16:04:38 +02:00
allegroai
831c1394da
Fix potential path traversal on file download (CVE-2024-24591)
2024-01-30 19:28:31 +02:00
allegroai
1b77e08ec8
Support setting certificate path in the main session
2024-01-30 19:26:32 +02:00
allegroai
0a928c24ca
Fix unsafe usage of tempfile.mktemp
2024-01-30 19:20:22 +02:00
allegroai
39652cac94
Fix HPO crash when its corresponding task is initialized with continue_last_task=True
2024-01-23 17:29:05 +02:00
allegroai
0806902f8e
Black formatting
2024-01-20 15:18:30 +02:00
allegroai
e7ec688022
Handle the case where PipelineController.abort_on_failure is set to False and a step has been aborted
2024-01-18 17:12:36 +02:00
allegroai
1b80a77c92
Fix TaskScheduler does not save local state before running remotely
2024-01-18 17:09:42 +02:00
allegroai
a2d1ff18d0
Remove usage of distutils for LooseVersion
2024-01-18 17:08:46 +02:00
allegroai
6504bb4824
Remove usage of distutils for strtobool
2024-01-18 16:07:11 +02:00
allegroai
0e0763d566
Report upload/download with tqdm if installed
2024-01-18 16:06:46 +02:00
allegroai
cc4a9912bf
Fix Flake8 error
2024-01-16 11:08:29 +02:00
allegroai
b8425133bc
Version bump to v1.14.1
2024-01-12 12:37:13 +02:00
allegroai
830dc451d2
Fix created pipelines do not appear in the UI pipelines section
2024-01-12 12:32:55 +02:00
allegroai
92ab713920
Version bump to v1.14.0
2024-01-10 14:07:19 +02:00
allegroai
c5510bb06e
Refactor new code
2024-01-10 14:01:52 +02:00
allegroai
1f72b0ca66
Change pytorch distributed polling time to 250ms
2024-01-09 15:37:02 +02:00
allegroai
f7b441ab49
Add multi objective optimization to HPO
2024-01-06 12:35:49 +02:00
allegroai
d993fa75e4
Fix decorated pipeline steps
2024-01-06 12:35:06 +02:00
allegroai
801c7b4cd4
Add support for auto detecting torch and transformers accelerate distributed execution
2024-01-06 12:34:32 +02:00
allegroai
141a183235
Fix static method should not get any argument
2024-01-06 12:33:52 +02:00
allegroai
0ed5ff4123
Auto detect tensorflow-intel as tensorflow (there is no tensorflow-intel package)
2024-01-06 12:32:57 +02:00
allegroai
4bd6b20a62
Add support for decorated pipeline steps ( #1154 )
2024-01-03 22:18:39 +02:00
allegroai
cf39029cb2
Black formatting
2023-12-30 23:01:19 +02:00
allegroai
3b53ef2708
Protect signal unregistering
2023-12-26 10:09:47 +02:00
allegroai
a81ef3f922
Fix invalid jsonargs arguments should be deleted
2023-12-26 10:09:03 +02:00
allegroai
7813602ad2
Add raise_on_error flag to Model.update_weights() and Model.update_weights_package()
2023-12-24 11:37:59 +02:00
allegroai
8968753467
PEP8
2023-12-17 23:36:13 +02:00
allegroai
081f4e5a56
Version bump to v1.13.3rc0
2023-12-13 17:51:39 +02:00
allegroai
23bdbe4b87
Fix jsonargparse binding does not capture parameters before Task.init is called ( #1164 )
2023-12-13 17:50:03 +02:00
allegroai
65c6ba33e4
Add more visibility into the way we override jsonargparse arguments
2023-12-13 16:45:51 +02:00
allegroai
1076d20808
Fix using target_project with TaskScheduler.add_task() corrupts project if used with pipelines ( #1137 )
2023-12-13 16:42:31 +02:00
allegroai
8cb7d14abb
Fix process sometimes hangs issue by improving exit and exception handlers, unregistering signal handling in child processes except for the BackgroundMonitor
2023-12-13 16:41:21 +02:00
allegroai
07dcbcac08
Support raising ConfigurationError when checking for min API version (makes sure a proper error is raised when importing clearml while using an invalid configuration)
2023-12-11 10:37:35 +02:00
allegroai
73e2ecf7fd
Support clearing repository details in task.set_repo()
2023-12-07 16:31:51 +02:00
allegroai
82c28ba43b
Fix set_repo and set_packages should wait for automatic async repo detection
2023-12-05 12:52:45 +02:00
allegroai
b3edd27e8d
Remove trailing whitespace (PEP8)
2023-11-29 12:12:19 +02:00
allegroai
19bfa3cd31
Fix active GPU filtering in resource monitor to support GPU ID and not just GPU idx
2023-11-26 11:11:43 +02:00
allegroai
5911d9e6d6
Handle appends to Hydra defaults list
2023-11-21 11:05:30 +02:00
allegroai
0af630e9cf
Add ChunkedEncodingError, ContentDecodingError and StreamConsumedError to retry list
2023-11-17 09:25:15 +02:00
allegroai
65c1062675
Version bump to v1.13.2
2023-11-08 23:08:21 +02:00
allegroai
9d5c39aac3
Support ignoring default_output_uri on remote runs when output_uri=False was specified and the backend task data has no output_uri specified
2023-11-07 17:03:11 +02:00
allegroai
1e5992cd5a
Fix metadata schema is a for queues
2023-11-07 16:35:51 +02:00
allegroai
68aa2a9e9e
Fix attrs.__version__ deprecation warning ( #1145 )
2023-11-07 16:35:04 +02:00
allegroai
ecdeb2d0a2
Support raising LoginError when checking for min api version
2023-11-07 16:34:43 +02:00
allegroai
3d6ab41a7a
Explicitly request raising an error when failing to check api version
2023-11-07 15:11:06 +02:00
allegroai
f67ce9c04b
Improve docstrings for tag filters
2023-11-07 15:07:41 +02:00
allegroai
fdd2c03900
Fix missing configuration error not raised correctly
2023-11-05 21:07:33 +02:00
allegroai
32832ae46d
Fix Model.connect() in remote execution might result in the wrong model being connected
2023-11-05 21:07:33 +02:00
allegroai
e7f4497e36
Fix potential threading issues when setting self.lock to None
2023-11-05 21:07:33 +02:00
allegroai
e746b3e535
Support GS credentials as JSON string if not a JSON file reference
2023-11-05 21:07:32 +02:00
allegroai
4303664d5b
Fix parallel dataset verification fails on older versions of Python ( #1144 )
2023-11-05 21:07:32 +02:00
allegroai
d6d8aa9318
Fix various CI and tests issues
2023-11-05 21:07:32 +02:00
Quy Vu
e89b2a03a7
Fix several jsonschema DeprecationWarning ( #1128 )
...
* address DeprecationWarnings from jsonschema
* use `referencing` for Python >= 3.8
* undo unrelated change by auto linter
2023-10-30 18:30:38 +02:00
allegroai
2f233bb9fd
Fix incorrect return type annotation
2023-10-28 22:55:56 +03:00
allegroai
b2bb04899c
Use StorageManager cache when downloading models
2023-10-24 18:45:18 +03:00
allegroai
b8ceba38dc
Add more visibility when overriding jsonargparse arguments
2023-10-24 18:44:24 +03:00
allegroai
f2057febd0
Fix displayed pipeline run time is pipeline components is not the same as that shown in info tab in full detail
2023-10-24 18:43:04 +03:00
allegroai
0caa787d7d
Fix calling Task.get_parameters_as_dict(cast=True) raises a YAML parser error
2023-10-24 18:40:40 +03:00
allegroai
f964783046
Improve parallel behavior
2023-10-24 18:39:10 +03:00
allegroai
c26efb83af
Add support for Hydra command-line syntax for modifying omegaconf
2023-10-24 18:36:53 +03:00
allegroai
9142f861fd
Fix pipeline version is not displaying in the table view and in the version list
2023-10-24 18:35:14 +03:00
allegroai
97035dcf7b
Add support for setting output_uri in pipeline decorators and functions
2023-10-24 18:33:53 +03:00
allegroai
7b85555d0d
Fix task.set_script(diff="") does not clear the uncommitted changes ( #1106 )
2023-10-24 18:32:29 +03:00
allegroai
190b25a44b
Fix don't raise exception on ConnectionError (which actually indicates that a read has timed out)
2023-10-24 18:30:16 +03:00
Matteo Destro
a8ec1076da
Remove subdir path from bucket url ( #1117 )
2023-10-17 20:13:20 +03:00
bedapisl
8a834af777
Support lightgbm model loading from a string instead of file ( #1136 )
...
Co-authored-by: Bedrich Pisl <bedrich.pisl@colsys.cz>
2023-10-15 12:28:54 +03:00
allegroai
040fd671a4
Fix PEP8 issues
2023-10-13 22:17:29 +03:00
charlienewey-odin
9687ca3db1
Verify dataset in parallel ( #1131 )
2023-10-07 16:39:08 +03:00
allegroai
2a5fd56fdb
Version bump to v1.13.1
2023-09-26 00:33:44 +03:00
allegroai
b83baa6f2e
Fix Hydra overrides integration ( #1120 )
2023-09-25 22:16:54 +03:00
allegroai
96646dc46a
Fix GPU memory used reports 0 when memory can not be queried per process (edge case)
2023-09-25 22:16:02 +03:00
allegroai
e71c257290
Version bump to v1.13.0
2023-09-22 16:26:06 +03:00
allegroai
c5675733a0
Fix GPU memory used reports 0 when memory can not be queried per process
2023-09-22 16:25:50 +03:00
allegroai
0442579e23
Fix scroll_id argument missing in events.get_scalar_metric_data endpoint (mainly for use in the APIClient)
2023-09-20 15:27:27 +03:00
allegroai
9e69b09c9f
Fix PEP8 issues
2023-09-20 15:26:50 +03:00
allegroai
701582d5ff
Fix incorrect module used
2023-09-16 20:53:47 +03:00
allegroai
1820423b02
Fix keep backwards compatibility for pipeline.add_step() positional arguments
2023-09-16 20:53:30 +03:00
allegroai
1b27ed07f7
Add force_single_script_file to Task.create() to avoid git repo auto detection
2023-09-16 20:52:42 +03:00
allegroai
76076b09dd
Fix compatibility issue with requests < 2.28
2023-09-12 00:51:47 +03:00
allegroai
d131ff2a45
Fix import sys should not appear in pipeline component auto detected packages
2023-09-12 00:51:25 +03:00
allegroai
96b89d76b8
Fix dataset zip extraction might fail when creating folders concurrently
2023-09-12 00:50:23 +03:00
natephysics
d458924160
Add support for recursive list, dict, and tuple ref parsing for pipeline controller.add step() parameter override ( #1099 )
...
* feat: ✨ Added optional support for list, dicts, tuples in Pipeline parameter_overrides
* style: 🎨 Updated to pass the flake8 formatting guidelines
* docs: 📝 There was a small typo I noticed in the documentation. Two extra '
2023-09-09 22:17:49 +03:00
allegroai
c922c40d13
Fix model reporting in tensorflow 2.13 does not work properly ( #1112 )
2023-09-08 22:14:53 +03:00
pollfly
cd61efe6df
Edit dataset_path docstring ( #1105 )
2023-09-06 17:07:59 +03:00
allegroai
04fe7dffe3
Fix OutputModel should prefer connected task name over filename stem
2023-09-02 17:49:02 +03:00
allegroai
71c74f977b
Add Model archive/unarchive methods
2023-08-31 00:10:38 +03:00
allegroai
713501c611
Fix hydra overrides integration (issue #552 )
2023-08-31 00:10:21 +03:00
allegroai
6756dec06a
Json serialization error should raise exception
2023-08-31 00:09:36 +03:00
Alex Burlacu
0959974920
Fix fastAI binding not reporting scalars if tensorboard is installed
2023-08-24 13:55:05 +03:00
Alex Burlacu
0b521b00a6
Use os.register_at_fork instead of monkey patching fork for python > 3.6
2023-08-21 14:06:59 +03:00
Alex Burlacu
88d036b367
Update PyNVML to the latest NVIDIA version
2023-08-15 13:54:04 +03:00
Alex Burlacu
afe178b002
Bump version to 1.12.2
2023-08-11 19:57:07 +03:00
Alex Burlacu
83a04d438c
Adjust for pandas < 2.0
2023-08-11 19:56:08 +03:00
Alex Burlacu
d4b11dfa22
Create OutputModel base model lazily
2023-08-11 13:11:58 +03:00
Alex Burlacu
2c44bff461
Fix Hydra support both Hydra section overrides and
2023-08-11 13:09:19 +03:00
allegroai
c15f012e1b
Add PipelineController always_create_from_code=True (False is previous behavior where we deserialize always from backend when running remotely, new flow means the pipeline is always created from code)
...
Fix pipeline decorator does not read the pipeline arguments back from the backend when running remotely
2023-08-04 19:54:55 +03:00
allegroai
d26ce48dbe
Improve artifacts serialization: should now be consistent when serializing pandas object into gzip stream (csv.gz)
2023-08-04 19:08:51 +03:00
allegroai
197894735b
Make sure we work on a copy of task objects
2023-08-04 19:08:09 +03:00
allegroai
696c349488
Improve warning message on task execute_remotely
2023-08-04 19:07:35 +03:00
allegroai
b846772138
Fix pipeline monitor must be called after pipeline is completed (just in case we missed something)
2023-08-04 19:07:05 +03:00
allegroai
99c7eecbee
Fix pipeline should always use artifacts hash for checking cache hits
2023-08-04 19:06:15 +03:00
allegroai
46c6d2bf0f
Fix proxy object support for lists
2023-08-04 19:05:24 +03:00
Alex Burlacu
92511efbde
Allow setting multiprocessing start method
2023-07-31 21:34:13 +03:00
Alex Burlacu
80ef359f45
Bump version and clarify docs
2023-07-31 21:33:28 +03:00
allegroai
e467c05fb4
Fix S3 uploads fail with LazyEvalWrapper type error ( #1081 )
2023-07-30 13:04:42 +03:00
Michael Pilosov
b1f99abc96
Raise error if dataset name empty ( #1083 )
...
* raise error if dataset name empty.
hidden dataset project when name is an empty string.
if the dataset project happens to match an existing project, it can be quite jarring to have one's history of results suddenly become hidden.
if there's an intent behind this behavior, please let me know and I'd gladly change it to a warning instead.
* move check to top of init, change message
* whitespace
2023-07-27 14:54:19 +03:00
pollfly
c451589298
Edit docstrings ( #1084 )
2023-07-25 12:19:24 +03:00
Alex Burlacu
09363b0d30
Fix typo
2023-07-21 14:23:32 +03:00
Alex Burlacu
618a625b74
Bump version to 1.12.0
2023-07-19 17:33:28 +03:00
Alex Burlacu
cc875f0fbe
Fix linter issues
2023-07-19 17:28:58 +03:00
Alex Burlacu
028a835676
Deprecate Task.get_by_name
2023-07-19 02:11:40 +03:00
Alex Burlacu
75b4015fdb
Fix misconfigured boto3 bucket credentials verify
2023-07-19 02:09:49 +03:00
Alex Burlacu
c8c8a1224e
Add support for .get ing pipelines and enqueue-ing them
2023-07-18 15:03:31 +03:00
achaiah
9d57dad652
AWS credential access fix ( #1000 )
...
Fixing AWS credential access that uses a token, instantiation within VPC without AvailabilityZones and ebs volume creation. Note: EBS is attached but never mounted.
2023-07-16 15:22:10 +03:00
Alex Burlacu
f6ae803110
Add default values in .conf files for s3 multipart
2023-07-14 14:24:10 +03:00
alex-burlacu-clear-ml
bae4886412
Merge pull request #1059 from cgaudreau-ubisoft/s3-configure-multipart
...
Added boto3 parameters for multipart configurations
2023-07-14 14:13:21 +03:00
Alex Burlacu
e7edcbb813
Fix #1054 , now retring a pipeline step will continue from the correct tf epoch
2023-07-14 14:04:38 +03:00
pollfly
44faf6ef7b
Improve clearml.conf ( #1071 )
2023-07-11 10:53:19 +03:00
natephysics
29227cade1
Add include_archived to optionally exclude archived datasets from dataset.list_datasets() ( #1069 )
...
Addresses #1067
* feat(datasets.py): ✨ Added the option `include_archived` to Dataset.list_datasets() class method which, if False, will exclude archived datasets from the list. Defaults to True to maintain the current behavior of the class method.
While it's possible to filter datasets using the get() class method it wasn't possible to do the same for the list_datasets() method. This feature seems useful to include and it's a simple, non-breaking modification.
* style(datasets.py): 🚨 Updated formatting (white spaces/spaces after comma/line breaks) in accordance with the flake8 formatting checks for dataset.py.
As suggesting in the contribution guidelines, fake8 was used to validate the formatting. Several errors were found from prior commits that were fixed.
2023-07-10 20:59:49 +03:00
allegroai
5aa80267d6
Version bump to v1.11.2rc0
...
Small refactoring
2023-07-10 11:20:27 +03:00
allegroai
3f882c37b9
Fix pipeline steps are not cached when the arguments come from the pipeline controller
2023-07-10 11:18:37 +03:00
allegroai
c394046a2d
Fix Task.connect_configuration() removes trailing + sign from configuration key
2023-07-10 11:14:09 +03:00
Alex Burlacu
13f11a68be
Improve pipeline caching behaviour
2023-07-08 16:58:41 +03:00
Alex Burlacu
6dc2c8ca8c
Fix GCS quoting issue
2023-07-06 12:23:06 +03:00
Make42
5824e27ada
Correct spelling, code formatting, add deprecation warnings to logger methods ( #1052 )
...
* Correct spelling in docstrings and comments in automation controller
* Improve readability by adding line breaks in automation controller
* Add deprecation warnings for logger methods
* Add docstring text regarding deprecation of logger methods
* Format deprecation warnings for logger methods in Black style
* Correct the docstring of query_tasks
* Remove new line in 'pre_execute_callback' docstring
---------
Co-authored-by: Make42 <none@none.net>
2023-06-29 18:10:09 +03:00
AbdulHamid Merii
9e8ed16b3e
Added boto3 parameters for multipart configurations
...
- multipart threshold and chunksize
2023-06-26 12:07:40 -05:00
AbdulHamid Merii
cbd14e3d6a
Fix typo in projects docstring ( #1038 )
2023-06-25 08:05:53 +03:00
Alex Burlacu
7eb4cb46af
Bump version to 1.11.1
2023-06-21 18:33:44 +03:00
allegroai
8524874998
Fix task status comparison breaks when using a string
2023-06-21 08:54:01 +03:00
allegroai
ab3c148de2
Fix message not reloaded before appending text
2023-06-21 08:51:47 +03:00
allegroai
1da47a20f8
Add CLI client name to client header
2023-06-19 11:02:05 +03:00
Alex Burlacu
050d6d8970
Fix infinite recursion when updating connected dict remotely
2023-06-16 13:52:55 +03:00
Alex Burlacu
eede2b6155
Bump version to 1.11.1rc2
2023-06-13 13:57:41 +03:00
Alex Burlacu
4ab46dd92c
Patch Gradio root_path argument
2023-06-12 15:13:13 +03:00
allegroai
be65b6055a
Fix support for new lightning namespace
2023-06-11 14:12:16 +03:00
Andreas Weinmann
ec51f35ec7
Add support for new lightning namespace ( #1033 )
2023-06-11 14:05:39 +03:00
Paweł Kotiuk
1ccdff5e77
Support clearml-init hosts detection for WebApp with no port ( #1031 )
...
Support clearml-init hosts detection for WebApp with no port and api/files hosts use the default ports.
2023-06-06 16:52:40 +03:00
allegroai
a343fc4c26
Bump version to 1.11.1rc1
2023-06-04 10:30:30 +03:00
allegroai
13c44b0b6a
Fix lazy evaluated value should be casted to str before passed through to OS calls
2023-06-04 10:30:20 +03:00
Alex Burlacu
9ec28b47b3
Bump version to 1.11.1rc0
2023-06-01 15:13:03 +03:00
Alex Burlacu
f207e72200
Fix jsonargparse not loading new values from changed config files
2023-06-01 15:12:30 +03:00
pollfly
56d4de04e4
Edit docstrings ( #1024 )
2023-05-30 15:40:43 +03:00
pollfly
db2f899d95
Edit docstrings ( #1013 )
2023-05-28 08:48:49 +03:00
Alex Burlacu
449a4cc42d
Bump version to v1.11.0
2023-05-25 19:00:20 +03:00
Alex Burlacu
0b6cb2edb8
Add Task.get_all_reported_scalars
2023-05-25 19:00:05 +03:00
Alex Burlacu
5772a1551e
Add support for offline datasets and JSON previews
2023-05-25 18:15:33 +03:00
