Commit Graph

499 Commits

Author SHA1 Message Date
allegroai
0a7a32f2eb Support caching of extracted zip artifacts 2020-04-26 23:19:42 +03:00
allegroai
6ff3cc0ee4 Fix duplicate suffix for artifacts 2020-04-26 23:17:51 +03:00
allegroai
302a8cbf75 Report memory usage only for the experiment process (and sub-processes) 2020-04-26 23:16:13 +03:00
allegroai
7e839204d1 Raise error in case of missing artifact directory 2020-04-26 23:14:33 +03:00
allegroai
617e5f7c1d Improve Jupyter support by ignoring IPython directives in converted python script (like % and ! lines) 2020-04-26 23:13:00 +03:00
allegroai
d9f2f01bbd Make sure trains is included in Jupyter requirements 2020-04-26 23:11:51 +03:00
allegroai
dbb3346332 Improve module requirements detection 2020-04-26 23:10:45 +03:00
allegroai
9726f782f2 Add requirement parsing optimization using entry point by default 2020-04-26 22:58:13 +03:00
allegroai
7e4fba9d10 Fix scikit-learn module detection 2020-04-26 22:54:49 +03:00
allegroai
13ce783fa3 Check for updates based on session version 2020-04-26 22:53:18 +03:00
Allegro AI
b9f5cdf8bf
Merge pull request #125 from jkhenning/master
Add autokeras example
2020-04-22 09:59:58 +03:00
Jake Henning
0a3db7186d
Update autokeras_imdb_example.py 2020-04-20 15:48:39 +03:00
Erez Schanider
d0e04b8975 Add autokeras example 2020-04-20 14:55:59 +03:00
allegroai
466b2859d9 Upload debug assets with series / title including dot or slash did not add iteration index to the filename 2020-04-16 16:49:21 +03:00
allegroai
f7b80a0da2 matplotlib with no logger should not use the last iteration reported by the resource monitor 2020-04-16 16:48:19 +03:00
allegroai
c06f72ae3a Match Torch TensorboardX example to Tensorboard 2020-04-16 16:47:03 +03:00
allegroai
cb139f2d17 Fix Tensorflow V1/V2 audio support 2020-04-16 16:46:02 +03:00
allegroai
8f1dd8ba8b Add log text to TF V1 example 2020-04-16 16:44:42 +03:00
allegroai
215aa32239 Detect exception in debugger session, set Task to Failed 2020-04-16 16:43:10 +03:00
allegroai
4eaa77dbdb Fix single log request exceeds max packet size, set limit to 1MB per request 2020-04-16 16:41:18 +03:00
allegroai
7f4b100042 Fix text encoding utf-8 and pr_curve broken in Tensorboard support 2020-04-16 16:40:14 +03:00
allegroai
4bb17ca420 Fix renaming/deleting model file right after saving will break async upload (trains/issues#123) 2020-04-13 19:03:15 +03:00
allegroai
12659307a8 Fix update_weights() to use model upload target file when passed 2020-04-13 19:00:35 +03:00
allegroai
4b9c5c235c Update docstrings 2020-04-13 18:58:39 +03:00
allegroai
648779380c Add media (audio) support for both Logger and Tensorboard bind 2020-04-09 13:14:14 +03:00
allegroai
7ac7e088a1 Add trace feature 2020-04-09 13:12:50 +03:00
allegroai
0df3d38862 Fix self references in configuration when environment variables exist 2020-04-09 13:11:21 +03:00
allegroai
3ac7dbdb49 Refactor shutdown sequence 2020-04-09 13:10:29 +03:00
allegroai
7dae058359 Optimize locking for TaskHandler, avoid lock when shutting down 2020-04-09 13:08:46 +03:00
allegroai
d9aa83380f Stop resource monitoring before signaling task stop 2020-04-09 13:07:26 +03:00
allegroai
ab263bb59f Raise ValueError if Task.get_logger() is called after task was closed 2020-04-09 13:06:06 +03:00
allegroai
3c4925d605 Fix resource monitor and check if task is valid 2020-04-09 13:00:24 +03:00
allegroai
7f00e45d6c Do not recreate logger after Task was closed/exited 2020-04-09 12:59:00 +03:00
allegroai
3f6fb5379a Revert fork patching as signal is not enough and is not called from forked processes 2020-04-09 12:57:50 +03:00
allegroai
5eb4ae6600 Use a daemon thread for the log 2020-04-09 12:56:55 +03:00
allegroai
1b901b7d13 Fix logger in case a packet was dropped before it was overwritten 2020-04-09 12:56:02 +03:00
allegroai
aa737e6b5b Make sure task is marked as started in remote execution (just in case) 2020-04-09 12:53:43 +03:00
allegroai
f61cbdeb39 Check if join was successful when waiting for std flush pool 2020-04-09 12:51:34 +03:00
allegroai
2f395cc76b Use sub-process fork signal hooks instead of os._exit patch 2020-04-09 12:50:04 +03:00
allegroai
004f925454 ThreadPool should be terminated, not closed, otherwise it might hang 2020-04-09 12:47:38 +03:00
allegroai
9916c93ce0 Add 10sec timeout for stdout/stderr flush at end of process 2020-04-09 12:46:30 +03:00
allegroai
1718aa20d4 Add thread_waited_join waited join for Thread/Process Pools 2020-04-09 12:45:06 +03:00
allegroai
23bd6097a8 Add nicer stdout log flush 2020-04-09 12:42:45 +03:00
allegroai
9a0a84a83e Do not wait for logs if we are aborting the task manually (i.e. ctrl-C) 2020-04-09 12:41:10 +03:00
allegroai
98ce0bbe43 Change TaskHandler.close() wait default to False as it should not wait for logs to flush 2020-04-09 12:39:09 +03:00
allegroai
b3c9872a3f Intercept SystemExit and do nothing so we could kill the thread 2020-04-09 12:33:16 +03:00
allegroai
5ec4d80493 Disconnect stdout/stderr logger on exit 2020-04-09 12:31:43 +03:00
allegroai
de9c88bc2d Do not try to wait for Lock 2020-04-09 12:30:42 +03:00
allegroai
337e60a376 Kill repo/package detection thread on exit 2020-04-09 12:28:57 +03:00
allegroai
b2c2002c40 Create dev task manually when constructing the Task 2020-04-09 12:27:13 +03:00