Commit Graph

97 Commits

Author SHA1 Message Date
allegroai
966cd6118a Add Model and Task type-annotations 2020-05-08 22:08:48 +03:00
allegroai
f90f8f06e2 Fix Task.get_tasks() return type to Task 2020-05-08 22:07:43 +03:00
allegroai
3de95063ee Remove auto_connect_arg_parser constraint when ArgumentParser.parse_arg() is used before initializing task 2020-05-08 21:33:54 +03:00
allegroai
3d9683f290 Fix logger creation guard 2020-04-28 10:44:40 +03:00
allegroai
7dc27b9a5e Rename utilities dir 2020-04-28 10:43:47 +03:00
allegroai
302a8cbf75 Report memory usage only for the experiment process (and sub-processes) 2020-04-26 23:16:13 +03:00
allegroai
215aa32239 Detect exception in debugger session, set Task to Failed 2020-04-16 16:43:10 +03:00
allegroai
4b9c5c235c Update docstrings 2020-04-13 18:58:39 +03:00
allegroai
3ac7dbdb49 Refactor shutdown sequence 2020-04-09 13:10:29 +03:00
allegroai
d9aa83380f Stop resource monitoring before signaling task stop 2020-04-09 13:07:26 +03:00
allegroai
ab263bb59f Raise ValueError if Task.get_logger() is called after task was closed 2020-04-09 13:06:06 +03:00
allegroai
7f00e45d6c Do not recreate logger after Task was closed/exited 2020-04-09 12:59:00 +03:00
allegroai
aa737e6b5b Make sure task is marked as started in remote execution (just in case) 2020-04-09 12:53:43 +03:00
allegroai
2f395cc76b Use sub-process fork signal hooks instead of os._exit patch 2020-04-09 12:50:04 +03:00
allegroai
9a0a84a83e Do not wait for logs if we are aborting the task manually (i.e. ctrl-C) 2020-04-09 12:41:10 +03:00
allegroai
98ce0bbe43 Change TaskHandler.close() wait default to False as it should not wait for logs to flush 2020-04-09 12:39:09 +03:00
allegroai
b3c9872a3f Intercept SystemExit and do nothing so we could kill the thread 2020-04-09 12:33:16 +03:00
allegroai
337e60a376 Kill repo/package detection thread on exit 2020-04-09 12:28:57 +03:00
allegroai
b2c2002c40 Create dev task manually when constructing the Task 2020-04-09 12:27:13 +03:00
allegroai
07daf8f5e6 Fix logger sometimes getting stuck at end of experiment 2020-04-09 12:05:56 +03:00
allegroai
172ed62d41 Add Task.get_tasks() filtering support 2020-04-01 18:54:16 +03:00
allegroai
c4719f2e2f Add type annotations and fix docstrings 2020-03-23 23:26:46 +02:00
allegroai
766c8ab24f Add Task.models property 2020-03-23 23:25:55 +02:00
allegroai
0211d233d4 Deprecate Task.set_model_config(), Task.get_model_config_text() and Task.get_model_config_dict() 2020-03-23 23:25:16 +02:00
allegroai
023f1721c1 Add Task.get_models() retrieving stored models on previously executed tasks 2020-03-22 18:19:07 +02:00
allegroai
babaf9f1ce Add OpenMPI/Slurm support 2020-03-20 10:23:00 +02:00
allegroai
0adbd79975 Fix StorageHelper upload on shutdown 2020-03-20 10:20:44 +02:00
allegroai
afad6a42ea Add initial slurm support (multiple nodes sharing the same task id) 2020-03-12 18:12:16 +02:00
allegroai
b3dff9a4eb Support setting task initial iteration for continuing previous runs 2020-03-12 17:40:29 +02:00
allegroai
f3531c1af2 Allow Task.set_credentials() to override configuration file in dev mode 2020-03-12 17:22:09 +02:00
allegroai
5bc39271e3 Fix store uncommitted code configuration option 2020-03-12 17:17:39 +02:00
allegroai
2e3820603a Allow argparser override values with command line even in remote execution (essential for sub-process support) 2020-03-05 12:28:36 +02:00
allegroai
1d9e70bd8b Fix signal hooking registeration (cont.) 2020-03-05 12:26:56 +02:00
allegroai
181a0be0af Remove temporary file lock at the end of the execution or in Task.close() 2020-03-05 12:25:17 +02:00
allegroai
b0c602c832 Fix signal hooking registeration 2020-03-05 12:24:14 +02:00
allegroai
bcf97afeb9 Forking processes should not pass along the original File based Lock 2020-03-05 12:22:14 +02:00
allegroai
888c53f67d Allow disabling repository detection when calling Task.init() 2020-03-05 12:19:40 +02:00
allegroai
a2ecb2c75d Only use file based locks for main task. Secondary tasks use traditional multiprocessing lock 2020-03-05 12:10:23 +02:00
allegroai
da804ca75f Add support for Popen subprocesses with task edit protection from multiple processes 2020-03-05 12:05:12 +02:00
allegroai
e3ae4f4e26 Optimize task refresh while pulling task status in local worker and last iteration for Resource Monitoring 2020-03-05 11:40:27 +02:00
Karthikeyan Singaravelan
a97850e5b6 Import ABC from collections.abc instead of collections for Python 3.9 compatibility. 2020-03-03 21:38:03 +05:30
allegroai
cf850020fb Don't print empty line at end of process if there's no artifacts summary 2020-02-26 17:06:17 +02:00
allegroai
8972c1f005 Add Task.[get/set]_parameters_as_dict() to allow interaction with non-main task parameters (no need to connect()) 2020-02-20 18:32:12 +02:00
allegroai
b5168010e9 Make sure Task.connect() returns the same value it is passed 2020-02-18 11:26:52 +02:00
allegroai
1cc0ea6cf3 Fix logs, events and jupyter flushing on exit 2020-01-21 16:41:01 +02:00
allegroai
30eaed79ea Add warning when automatic argument parser binding cannot be turned off 2020-01-06 17:20:15 +02:00
allegroai
bc33ad0da3 Calculate data-audit artifact uniqueness by user-criteria 2020-01-06 17:19:44 +02:00
allegroai
54ae340ccb Use source task id to determine cloned task parent 2020-01-02 12:01:03 +02:00
allegroai
67d9a9e5d4 Do not reuse experiment if it has artifacts 2019-12-15 00:09:36 +02:00
allegroai
f92278750a Fix default_output_uri in remote execution mode
Fix arguments order in Task.create
2019-12-04 23:47:26 +02:00