Batch normalization leads to a huge difference between training and inference losses
I followed the instructions on the Tensorflow webpage to tf.layers.batch_ Normalization sets training to True at training time and False at inference (valid and test).
However, batch normalization always gives me a huge difference between training and effective loss, for example:
2018-09-11 09:22:34: step 993, loss 1.23001, acc 0.488638
2018-09-11 09:22:35: step 994, loss 0.969551, acc 0.567364
2018-09-11 09:22:35: step 995, loss 1.31113, acc 0.5291
2018-09-11 09:22:35: step 996, loss 1.03135, acc 0.607861
2018-09-11 09:22:35: step 997, loss 1.16031, acc 0.549255
2018-09-11 09:22:36: step 998, loss 1.42303, acc 0.454694
2018-09-11 09:22:36: step 999, loss 1.33105, acc 0.496234
2018-09-11 09:22:36: step 1000, loss 1.14326, acc 0.527387
Round 4: valid
Loading from valid, 1383 samples available
2018-09-11 09:22:36: step 1000, loss 44.3765, acc 0.000743037
2018-09-11 09:22:36: step 1000, loss 36.9143, acc 0.0100708
2018-09-11 09:22:37: step 1000, loss 35.2007, acc 0.0304909
2018-09-11 09:22:37: step 1000, loss 39.9036, acc 0.00510307
2018-09-11 09:22:37: step 1000, loss 42.2612, acc 0.000225067
2018-09-11 09:22:37: step 1000, loss 29.9964, acc 0.0230831
2018-09-11 09:22:37: step 1000, loss 28.1444, acc 0.00278473
Sometimes even worse (for another model):
2018-09-11 09:19:39: step 591, loss 0.967038, acc 0.630745
2018-09-11 09:19:40: step 592, loss 1.26836, acc 0.406095
2018-09-11 09:19:40: step 593, loss 1.33029, acc 0.536824
2018-09-11 09:19:41: step 594, loss 0.809579, acc 0.651354
2018-09-11 09:19:41: step 595, loss 1.41018, acc 0.491683
2018-09-11 09:19:42: step 596, loss 1.37515, acc 0.462998
2018-09-11 09:19:42: step 597, loss 0.972473, acc 0.663277
2018-09-11 09:19:43: step 598, loss 1.01062, acc 0.624355
2018-09-11 09:19:43: step 599, loss 1.13029, acc 0.53893
2018-09-11 09:19:44: step 600, loss 1.41601, acc 0.502889
Round 2: valid
Loading from valid, 1383 samples available
2018-09-11 09:19:44: step 600, loss 23242.2, acc 0.204348
2018-09-11 09:19:44: step 600, loss 22038, acc 0.196325
2018-09-11 09:19:44: step 600, loss 22223, acc 0.0991791
2018-09-11 09:19:44: step 600, loss 22039.2, acc 0.220871
2018-09-11 09:19:45: step 600, loss 25587.3, acc 0.155427
2018-09-11 09:19:45: step 600, loss 12617.7, acc 0.481486
2018-09-11 09:19:45: step 600, loss 17226.6, acc 0.234989
2018-09-11 09:19:45: step 600, loss 18530.3, acc 0.321573
2018-09-11 09:19:45: step 600, loss 21043.5, acc 0.157935
2018-09-11 09:19:46: step 600, loss 17232.6, acc 0.412151
2018-09-11 09:19:46: step 600, loss 28958.8, acc 0.297459
2018-09-11 09:19:46: step 600, loss 22603.7, acc 0.146518
2018-09-11 09:19:46: step 600, loss 29485.6, acc 0.266186
2018-09-11 09:19:46: step 600, loss 26039.7, acc 0.215589
The bulk normalization code I used:
def bn(inp, train_flag, name=None):
return tf.layers.batch_normalization(inp, training=train_flag, name=name)
def gn(inp, groups=32):
return tf.contrib.layers.group_norm(inp, groups=groups)
def conv(*args, padding='same', with_relu=True, with_bn=False,
train_flag=None, with_gn=False, name=None, **kwargs):
# inp, filters, kernel_size, strides
use_bias = False if with_bn else True
x = tf.layers.conv2d(*args, **kwargs, padding=padding,
kernel_initializer=xavier_initializer(),
use_bias=use_bias, name=name)
if with_bn:
bn_name = name+'/batchnorm' if name is not None else None
x = bn(x, train_flag, name=bn_name)
if with_gn: x = gn(x)
if with_relu: x = relu(x)
return x
After I remove the batch normalization layer, the huge difference between the training loss and the validation loss disappears.
The following code is used for optimization.
update_ops = tf.get_collection(tf. GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
The model is trained from scratch without transfer learning.
I followed the problem< a href="https://github.com/keras-team/keras/issues/7265" rel="noreferrer noopener nofollow" > The Batch Normalization layer gives significant difference between train and validation loss on the exact same data, and tries to reduce momentum, but it doesn’t work either.
I wonder why this is so. I would appreciate it if you could give me some advice.
Addendum: train_flag is a placeholder throughout the model.
Solution
Since you didn’t provide the
full code, and you didn’t provide a link, I need to ask the following question:
How are you feeding the train_flag ?
The correct way is to set the train_flag to tf. Placeholder。 There are other ways, but this is the easiest. You can then use a simple Python Bool to provide data for it.
If you manually set train_flag=True during training and train_flag=False during validation, this could be the source of the problem. I don’t see reuse=tf in your code. AUTO_REUSE。 This means that during validation, when you set train_flag=False, a separate layer is created that does not share weights with the previous layer used during training.
The reason why the problem disappears when you don’t use batch normalization, because in that case, there is no need to use train_flag with convolutional layers. So, it works fine.
This is my speculation based on observations.