356 Commits

Author	SHA1	Message	Date
allegroai	004f925454	ThreadPool should be terminated, not closed, otherwise it might hang	2020-04-09 12:47:38 +03:00
allegroai	9916c93ce0	Add 10sec timeout for stdout/stderr flush at end of process	2020-04-09 12:46:30 +03:00
allegroai	1718aa20d4	Add thread_waited_join waited join for Thread/Process Pools	2020-04-09 12:45:06 +03:00
allegroai	23bd6097a8	Add nicer stdout log flush	2020-04-09 12:42:45 +03:00
allegroai	9a0a84a83e	Do not wait for logs if we are aborting the task manually (i.e. ctrl-C)	2020-04-09 12:41:10 +03:00
allegroai	98ce0bbe43	Change TaskHandler.close() wait default to False as it should not wait for logs to flush	2020-04-09 12:39:09 +03:00
allegroai	b3c9872a3f	Intercept SystemExit and do nothing so we could kill the thread	2020-04-09 12:33:16 +03:00
allegroai	5ec4d80493	Disconnect stdout/stderr logger on exit	2020-04-09 12:31:43 +03:00
allegroai	de9c88bc2d	Do not try to wait for Lock	2020-04-09 12:30:42 +03:00
allegroai	337e60a376	Kill repo/package detection thread on exit	2020-04-09 12:28:57 +03:00
allegroai	b2c2002c40	Create dev task manually when constructing the Task	2020-04-09 12:27:13 +03:00
allegroai	11420adce7	Log reports at the end of the task	2020-04-09 12:24:37 +03:00
allegroai	ffedb219d5	Local modules (except trains) imported from a folder inside the git project should not be logged as "local packages", they should be ignored	2020-04-09 12:21:37 +03:00
allegroai	07daf8f5e6	Fix logger sometimes getting stuck at end of experiment	2020-04-09 12:05:56 +03:00
allegroai	e6f29428eb	Add StorageManager	2020-04-09 12:03:41 +03:00
allegroai	e1fc9b3dc8	ThreadPool should be terminated, not closed, otherwise it might hang	2020-04-09 11:39:03 +03:00
allegroai	070fd8149a	Store the version that matching the Session API so we do not reload every time	2020-04-09 11:35:51 +03:00
allegroai	a425a70fc6	Add api.ssl_error_count_verbosity and make sure SSL retries are taken care by the session	2020-04-09 11:33:55 +03:00
allegroai	101e5393d1	Fix TRAINS_VCS_ROOT path conversion	2020-04-01 19:06:30 +03:00
allegroai	41ca1a2e49	Fix requirements detection to make sure trains is detected even if we execute without actually being installed	2020-04-01 19:04:57 +03:00
allegroai	01772430d6	Ignore virtual-environment folder that might be inside the project's directory	2020-04-01 19:02:54 +03:00
allegroai	6de3d4b6fd	Ignore local modules imported from a folder inside the git project	2020-04-01 19:01:21 +03:00
allegroai	172ed62d41	Add Task.get_tasks() filtering support	2020-04-01 18:54:16 +03:00
allegroai	581edf1098	Version bump to v0.14.1	2020-03-24 20:36:57 +02:00
allegroai	c4719f2e2f	Add type annotations and fix docstrings	2020-03-23 23:26:46 +02:00
allegroai	766c8ab24f	Add Task.models property	2020-03-23 23:25:55 +02:00
allegroai	0211d233d4	Deprecate Task.set_model_config(), Task.get_model_config_text() and Task.get_model_config_dict()	2020-03-23 23:25:16 +02:00
allegroai	023f1721c1	Add Task.get_models() retrieving stored models on previously executed tasks	2020-03-22 18:19:07 +02:00
allegroai	332e9e2f63	Fix Tensorflow direct V2.1 multiple FileWriters	2020-03-22 18:17:16 +02:00
allegroai	493cce443a	Reuse Model objects if we are storing local files (reduce clutter)	2020-03-22 18:15:32 +02:00
allegroai	4e2564cd3a	Support reusing Models. Use trains.Model as general purpose registered Model.	2020-03-22 18:13:56 +02:00
allegroai	63507c82f7	Fix Model.download_model_weights() to reuse previously downloaded file	2020-03-22 18:11:30 +02:00
allegroai	477665ee33	Fix storage_uri handling in Model.update()	2020-03-22 18:05:05 +02:00
allegroai	abc9b512f7	Fix logging typos	2020-03-22 18:03:25 +02:00
allegroai	7817ef5cda	Fix joblib binding	2020-03-20 10:30:13 +02:00
allegroai	5db53ba643	Support multiple EventWriter in TensorFlow eager mode (TF 2.0+)	2020-03-20 10:29:18 +02:00
allegroai	b4050ecf25	Fix TensorFlow NaN/Inf values support	2020-03-20 10:27:52 +02:00
allegroai	babaf9f1ce	Add OpenMPI/Slurm support	2020-03-20 10:23:00 +02:00
allegroai	0adbd79975	Fix StorageHelper upload on shutdown	2020-03-20 10:20:44 +02:00
allegroai	dc915d0241	Fix support for Task init/close multiple times	2020-03-20 10:20:06 +02:00
allegroai	667ddcab88	Fix import for services that do not exist in old versions	2020-03-20 10:16:48 +02:00
allegroai	3b1d2d3258	Version bump to v0.14.0	2020-03-12 19:42:48 +02:00
allegroai	afad6a42ea	Add initial slurm support (multiple nodes sharing the same task id)	2020-03-12 18:12:16 +02:00
allegroai	5b29aa194c	Make sure artifact temporary files names are valid file names	2020-03-12 18:10:03 +02:00
allegroai	84a34428b6	Add trains-init support for config file env override (as well as argument)	2020-03-12 18:09:03 +02:00
allegroai	b3dff9a4eb	Support setting task initial iteration for continuing previous runs	2020-03-12 17:40:29 +02:00
allegroai	f3531c1af2	Allow Task.set_credentials() to override configuration file in dev mode	2020-03-12 17:22:09 +02:00
allegroai	5bc39271e3	Fix store uncommitted code configuration option	2020-03-12 17:17:39 +02:00
allegroai	461fbd9df0	Better warning messages for storage errors	2020-03-12 17:13:36 +02:00
allegroai	30cf6b4834	Fix HTTP link quoting in stored links	2020-03-12 17:04:31 +02:00