Transfer Learning and CIFAR 10 dataset

Abstract

In this article we will see how using transfer learning we achieved a accuracy of 90% using the VGG16 algorithm and the CIFAR10 dataset as a base model containing a total of 50,000 training images and 10,000 test images.

Introduction

Currently it is possible to cut the time it can take to process and recognize a series of images to identify which image we are talking about. Therefore, in the world of machine learning, there is the possibility of transferring this prior knowledge made by an already trained algorithm and using it to achieve the same goal or something similar, this is known as transfer learning. In this space we will see how to use a trained model (VGG16) and how to use CIFAR10 dataset, we will achieve a validation accuracy of 90%

Remember that CIFAR10 data contains 60,000 32x32 color images in 10 classes, with 6000 images per class. There are 50,000 training images and 10,000 test images, the data set is divided into five training batches and one test batch, each with 10,000 images. The test lot contains exactly 1000 randomly selected images from each class. Training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training bundles contain exactly 5000 images of each class.

Here you can enter this dataset https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz

Materials and Methods

As we well know, transfer learning allows us to take as a base a previously trained model that shares the characteristics we need initially to be able to use it correctly and obtain good results.
In this case we will use the model VGG16 a model already pre trained in a general way and will be perfect for our case in particular, this model has some very particular characteristics and among those is its easy implementation in addition to the use of ImageNet (ILSVRC-2014) that allows us to classify images something that we will need at this time. Moreover this model VGG16 is available in Keras which is very good for our goal.

You could also experiment with other networks that were also trained with ImageNet on Keras, and it will depend on you and your time to achieve a better result than I have.

Xception
InceptionV3
ResNet50
VGG16
VGG19
MobileNet

To understand a bit how this works with the VGG16 model we have to understand that this model as well as the classification models have a structure that is composed of convolutional layers for feature extraction and the decision stage based on dense layers.

The Architecture VGG16

Once we understand in a general way the architecture and the operation of VGG16 and as it has been previously trained with ImageNet we can assume that this model is the correct one to be able to classify different images or objects by each one of its characteristics that make it unique, the following step will be to preload the VGG16 model.

The first thing we will do is to load the CIFAR10 data into our environment and then make use of it.

(x_train, y_train), (x_test, y_test) = K.datasets.cifar10.load_data()
x_train, y_train = preprocess_data(x_train, y_train)
x_test, y_test = preprocess_data(x_test, y_test)

The next thing we will do is to define our VGG16…

base_model = K.applications.vgg16.VGG16(include_top=False, weights='imagenet',                                        pooling='avg',                                        classes=y_train.shape[1])

Remember the following each of the parameters set previously determine a key aspect on the model, for example Include_top allows to include a dense neural network at the end which means that we will get a complete network (Feature extraction and decision stage) and this is something we do not want at the moment so this parameter will be indicated as False, on the other hand what we need is a model that is already pre trained so Weights will be indicated as imagenet.

The next thing we will do additional layers and dropout…

Once the model is defined we go on to determine the number of layers, remember that this step can be under trial and error. You can achieve a better performance than mine by increasing or decreasing the number of layers that you consider to determine a better result.

It is very important to avoid overfitting so it is fundamental to tell the model that to avoid this problem you should use Upsampling and dropout.

model = K.Sequential()model.add(K.layers.UpSampling2D())model.add(base_model)model.add(K.layers.Flatten())model.add(K.layers.Dense(256, activation=('relu')))model.add(K.layers.Dropout(0.5))model.add(K.layers.Dense(256, activation=('relu')))model.add(K.layers.Dropout(0.5))model.add(K.layers.Dense(10, activation=('softmax')))

Finally, once the model is defined, we compile it specifying which will be the optimization function, we will also take into account the cost or loss function and finally which will be the metric to use.

In this case, for the optimization we will use Adam and for the loss function categorical_crossentropy and for the metrics accuracy

model.compile(optimizer=K.optimizers.Adam(lr=2e-5),              loss='categorical_crossentropy', metrics=['accuracy'])

Results

Within the results we can see aspects such as loss, accuracy, loss validation and finally the validation of accuracy.

2020-09-26 16:21:00.882137: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
Train on 50000 samples, validate on 10000 samples
Epoch 1/5
50000/50000 [==============================] - 507s 101ms/step - loss: 0.6372 - acc: 0.7780 - val_loss: 0.4127 - val_acc: 0.8569
Epoch 2/5
50000/50000 [==============================] - 483s 97ms/step - loss: 0.3144 - acc: 0.8921 - val_loss: 0.3551 - val_acc: 0.8786
Epoch 3/5
50000/50000 [==============================] - 484s 97ms/step - loss: 0.2085 - acc: 0.9276 - val_loss: 0.3472 - val_acc: 0.8852
Epoch 4/5
50000/50000 [==============================] - 492s 99ms/step - loss: 0.1386 - acc: 0.9512 - val_loss: 0.2792 - val_acc: 0.9077
Epoch 5/5
50000/50000 [==============================] - 489s 98ms/step - loss: 0.0929 - acc: 0.9671 - val_loss: 0.3456 - val_acc: 0.9047

Discussion

We can see that during the process of learning the model in the epoch number 2 already has surpassed of substantial form 87% of precision, nevertheless the model continues surpassing this precision up to the epoch number 4 with a val_acc of 90% quite efficient but it happens that during the epoch number 5 we have a detriment in the validation of the precision, it is for this reason that up to the epoch number 4 it is the model that we must have as case of successful in our model. It is very important to remember that acc indicates the precision in the training set, that is to say, in the data that the model has been able to train before, while val_acc is the precision with the validation or test set, that is to say, data that the model has not seen.

remember that when the accuracy in the validation data gets worse that is the exact point where our model is starting to overfitting.

Literature Cited

Spring Research Presentation

Improving EEG-Based Emotion Classification Using Conditional Transfer Learning

Keras documentation: VGG16 and VGG19