Python – Tensorflow (1.4.1) Do Tensorboard visualizations go back in time?

I’ve created some summary actions throughout the chart as follows:

tf.summary.scalar('cross_entropy', cross_entropy)
tf.summary.scalar('accuracy', accuracy)

Of course merged and got an author:

sess = tf. InteractiveSession()
summaries = tf.summary.merge_all()
train_writer = tf.summary.FileWriter(TENSORBOARD_TRAINING_DIR, sess.graph)
tf.global_variables_initializer().run()

I write these in each training iteration:

summary, acc = sess.run([summaries, accuracy], feed_dict={...})
train_writer.add_summary(summary, i)

When I load the tensorboard, I get some strange results :

This is strange for several reasons:

1. cross_entropy The Y-axis on the chart does not have increased (or different) tick marks
2. Line plots seem to fold back or go back in time on their own

I

did check – I have some previous event files in my training summary folder:

$ ls /tmp/tv_train/
events.out.tfevents.1517210066.xxxxxxx.local    
events.out.tfevents.1517210097.xxxxxxx.local    
...
events.out.tfevents.1517210392.xxxxxxx.local

I

guess I must have restarted the training loop at some point, resulting in multiple digests logged at the (0, 1, etc.) index.

How do I attach to an old training log? Can I point the author to a specific tfevents file to “pick up where I left off”?

You can’t (easily) reopen and “attach” to an existing event file, but it’s not required.
Tensorboard displays sequential event files well, as long as the step values in the record are consistent.
When you save the summary, you specify a step value that indicates at which point on the x-axis the summary should be drawn.

The graph “goes back in time” because on each new run, you restart the pedometer from 0. To make it consistent across multiple runs, you should define an global_step variable that is saved to the checkpoint when the network is saved. That way, when you restore the network in the next training run, your global steps will start where it left off and your graph will no longer look weird.