Python – Ensemble model with different inputs (expect to see 2 arrays)

I trained 2 models.

The first model is UNet:

print(model_unet.summary())

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_4 (InputLayer)            (None, 128, 128, 1)  0                                            
__________________________________________________________________________________________________
conv2d_26 (Conv2D)              (None, 128, 128, 32) 320         input_4[0][0]                    
__________________________________________________________________________________________________
conv2d_27 (Conv2D)              (None, 128, 128, 32) 9248        conv2d_26[0][0]  
.....
.....
conv2d_44 (Conv2D)              (None, 128, 128, 1)  33          zero_padding2d_4[0][0]           
==================================================================================================
Total params: 7,846,081
Trainable params: 7,846,081
Non-trainable params: 0

This is followed by ResNet:

print(model_resnet.summary())

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_3 (InputLayer)            (None, 128, 128, 3)  0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (None, 134, 134, 3)  0           input_3[0][0]                    
....
....
conv2d_25 (Conv2D)              (None, 128, 128, 3)  99          zero_padding2d_3[0][0]           
==================================================================================================
Total params: 24,186,915
Trainable params: 24,133,795
Non-trainable params: 53,120

UNet has 1 channel (gray) and ResNet has 3 channels.

Then, I tried to create an ensemble model:

def ensemble(models, models_input):

outputs = [model(models_input[idx]) for idx, model in enumerate(models)]
    x = Average()(outputs)

model_inputs = [model for model in models_input]
    model = Model(model_inputs, x)

return model

models = [model_unet, model_resnet]
models_input = [Input((128,128,1)), Input((128,128, 3))]

ensemble_model = ensemble(models, models_input)

When I try to predict validation data:

pred_val = ensemble_model.predict(X_val)

I get the error:

Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[[0.46755977],
         [0.52268691],
         [0.52766109],
         ....

X_val.shape is : (800, 128, 128, 1)

I

think the problem is with the channel, but I don’t know how to overcome it.

If your training data is grayscale images, and considering that your ResNet model takes RGB images as input, you should ask yourself, how do you want to transition from grayscale images to RGB images? One answer is to repeat the grayscale image 3 times to get an RBG image. You can then easily define a model that has an input layer that takes your grayscale images and provides them accordingly to the model you define:

from keras import backend as K

input_image = Input(shape=(128,128,1))

unet_out = model_unet(input_image)
rgb_image = Lambda(lambda x: K.repeat_elements(x, 3, -1))(input_image)
resnet_out = model_resnet(rgb_image)

output = Average()([unet_out, resnet_out])

ensemble_model = Model(input_image, output)

You can then easily call predict: with an input array

pred_val = ensemble_model.predict(X_val)

Another option for this solution is to use the solution you used in the problem. However, you first need to convert the image from grayscale to RGB and then pass both arrays to the predict method:

X_val_rgb = np.repeat(X_val, 3, -1)

pred_val = ensemble_model.predict([X_val, X_val_rgb])