Python – Hyperparameter tuning of Keras models with large datasets

I want to perform hyperparameter tuning on my Keras model. The problem is that the dataset is relatively large, generally in training I use fit_generator batch load data from disk, but common packages such as SKlearn Gridsearch, Talos and so on only support the fit method.

I’m trying to load the entire data into memory by:

train_generator = train_datagen.flow_from_directory(
    original_dir,
    target_size=(img_height, img_width),
    batch_size=train_nb,
    class_mode='categorical')
X_train,y_train = train_generator.next()

However, when performing a grid search, the operating system terminates it because it consumes a lot of memory.
I also tried undersampling my dataset to only 25%, but it’s still too big.

Has anyone had the same experience as me? Can you share your strategy for performing hyperparameter tuning on large datasets?

Based on @dennis-ec’s answer, I tried following the SkOpt tutorial here: http://slashtutorial.com/ai/tensorflow/19_hyper-parameters/ This is a very comprehensive tutorial

In my opinion, GridSearch is not a good way to tune hyperparameters, especially in deep learning with many hyperparameters.

I would recommend Bayesian hyperparameter optimization. Here Tutorial on how to implement it using skopt. As you can see, you need to write a function to train and return the validation score you want to optimize, so the API doesn’t care if you use fit or fit_generator in keras.