Python – Create a new sequential model in a for loop (using Keras)

I

want to try training my model with different hyperparameters, so I set up a series of nested for loops to iterate through them.

model = None
batch_generator = None

for sequence_length in all_sequence_length:
    for label_periods in all_label_periods:
        for num_layers in all_num_layers:
            for num_units in all_num_units:
                loadFiles()
                createmodel()
                trainmodel()

The first iteration creates a model like this:

Layer (type)                 Output Shape              Param #
=================================================================
cu_dnnlstm_1 (CuDNNLSTM)     (None, 100, 75)           45300
_________________________________________________________________
dropout_1 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
cu_dnnlstm_2 (CuDNNLSTM)     (None, 100, 75)           45600
_________________________________________________________________
dropout_2 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
cu_dnnlstm_3 (CuDNNLSTM)     (None, 100, 75)           45600
_________________________________________________________________
dropout_3 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
cu_dnnlstm_4 (CuDNNLSTM)     (None, 100, 75)           45600
_________________________________________________________________
dropout_4 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
cu_dnnlstm_5 (CuDNNLSTM)     (None, 100, 75)           45600
_________________________________________________________________
dropout_5 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
dense_1 (Dense)              (None, 3)                 228
=================================================================

Then I call model.fit_generator() to train the model and it performs well. Then create the model again in the next loop iteration, with a summary as follows:

Layer (type)                 Output Shape              Param #
=================================================================
cu_dnnlstm_6 (CuDNNLSTM)     (None, 100, 75)           45300
_________________________________________________________________
dropout_6 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
cu_dnnlstm_7 (CuDNNLSTM)     (None, 100, 75)           45600
_________________________________________________________________
dropout_7 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
cu_dnnlstm_8 (CuDNNLSTM)     (None, 100, 75)           45600
_________________________________________________________________
dropout_8 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
cu_dnnlstm_9 (CuDNNLSTM)     (None, 100, 75)           45600
_________________________________________________________________
dropout_9 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
cu_dnnlstm_10 (CuDNNLSTM)     (None, 100, 75)           45600
_________________________________________________________________
dropout_10 (Dropout)          (None, 100, 75)           0
_________________________________________________________________
dense_2 (Dense)              (None, 3)                 228
=================================================================

You’ll see that the layer IDs have increased, which surprised me because I created a new sequential model for the model variables, so I expected the same summary as the first one.

I

get this error when I call model.fit_generator():

InvalidArgumentError (see above for traceback): You must feed a value
for placeholder tensor ‘cu_dnnlstm_1_input’ with dtype float and shape
[?,100,74]

You see that it expects input from cu_dnnlstm_1_input, which is input from the first iteration of the model, not input from cu_dnnlstm_6 of the second iteration. The code I used to create the model was done in a function:

def createmodel():

global model

model = Sequential()
    model.add( CuDNNLSTM(units=num_units, return_sequences=True, input_shape=(sequence_length, features_size) ) )

for _ in range(num_layers):
        model.add( Dropout(dropout_rate) )
        model.add( CuDNNLSTM(units=num_units, return_sequences=True) )

model.add( Dropout(dropout_rate) )
    model.add( CuDNNLSTM(units=num_units, return_sequences=False) )

model.add( Dropout(dropout_rate) )
    model.add( Dense(labels_size) )

model.compile(loss='mean_absolute_error', optimizer='adam')

model.summary()

The model is trained with this function:

def trainmodel():

global model

model.fit_generator(generator=batch_generator,
        epochs=num_epochs,
        steps_per_epoch=num_steps_per_epoch,
        validation_data=validation_data_tuple,
        callbacks=callbacks)

Who can find out if I was “deliberate” wrong?

I guess this is because Keras is trying to create different models on the same tensorflow graph. This is not possible because your model has a different architecture.

Try importing tensorflow:

import tensorflow as tf

And modify your loop this way:

for sequence_length in all_sequence_length:
    for label_periods in all_label_periods:
        for num_layers in all_num_layers:
            for num_units in all_num_units:
                graph = tf. Graph()
                with tf. Session(graph=graph):
                    loadFiles()
                    createmodel()
                    trainmodel()