ClearML/clearml
1
0
Fork 0
mirror of https://github.com/clearml/clearml synced 2025-06-26 18:16:07 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Fix ontinuing Tasks with tensorflow sometimes result in wrong initial iteration (#762)

Browse Source
This commit is contained in:
allegroai 2022-12-13 15:36:05 +02:00
parent 3da182426f
commit 16df0794be
1 changed files with 10 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
clearml/binding/frameworks/tensorflow_bind.py
View File

@ -726,6 +726,16 @@ class EventTrainsWriter(object):
'Received event without step, assuming step = {}'.format(step)) 'Received event without step, assuming step = {}'.format(step))
else: else:
step = int(step) step = int(step)
# unlike other frameworks, tensorflow already accounts for the iteration number
# when continuing the training. we substract the smallest iteration such that we
# don't increment the step twice number
step_before = step
if EventTrainsWriter._current_task:
step -= EventTrainsWriter._current_task.get_initial_iteration()
# there can be a few metrics getting reported again, so the step can be negative
# for the first few reports
if step <= 0:
return
self._max_step = max(self._max_step, step) self._max_step = max(self._max_step, step)
if value_dicts is None: if value_dicts is None:
LoggerRoot.get_base_logger(TensorflowBinding).debug("Summary arrived without 'value'") LoggerRoot.get_base_logger(TensorflowBinding).debug("Summary arrived without 'value'")