Exercise 11: VGG-16 Network and Image Classification Task

VGG-16 Network and Image Classification Task

In this notebook, you can try out how convolutional neural networks actually work. We will use the VGG-16 network and test it in an image classification task. We have as many as 1000 classes at our disposal!
At the very beginning, we will load the standard libraries needed for further work.

#%pip install pandas

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

Specifically, from the TensorFlow library we load the pre-trained VGG-16 model. When loading this model, we need to indicate that we want to use the model that was trained on images from the ImageNet dataset. We do this by setting the `weights` argument of the `VGG16` function to `imagenet`. The loading of the model itself may take a bit of time.

#%pip install tensorflow
#%pip install --upgrade numpy

from tensorflow.keras.applications import VGG16
model = VGG16(weights='imagenet')

Next, we will check the layers that make up this network. Don't be confused if you don't understand all the details displayed. It is important to know that the input is expected to be a 224x224 pixel color image (hence the 224, 224, 3 next to the input layer) and that the output is one of 1000 classes. You can compare the output with the VGG-16 model diagram we discussed to discover more information.

VGG16 is object detection and classification algorithm which is able to classify 1000 images of 1000 different categories with 92.7% accuracy. It is one of the popular algorithms for image classification and is easy to use with transfer learning.

VGG16 Architecture

model.summary()

We can see that the number of parameters of this network is over 138 million. We will not train the network, but we will only use the pre-trained model. Therefore, it is important not to change the model parameters during operation - each one has its own contribution.
The idea is to test the network with an arbitrary image from the web. To achieve this, we will use a couple of libraries. The PIL library is the standard Python library for working with images, while we will use the urllib and io libraries to load the image.

from PIL import Image
from urllib import request
from io import BytesIO

The `load_image` function will help us fetch any image we want from the web. It is enough to provide it with the URL path.

def load_image(url_path):
    response = request.urlopen(url_path).read()
    return Image.open(BytesIO(response))

We have chosen an image of a golden retriever! By changing the path in the `image_url` variable, you can set the image you want.

image_url = 'https://images.unsplash.com/photo-1558788353-f76d92427f16?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=738&q=80' # https://unsplash.com/photos/x5oPmHmY3kQ
test_image = load_image(url_path=image_url)

Now we will display the loaded image.

plt.imshow(test_image)
plt.axis('off')
plt.show()

When choosing an image, it is important to keep in mind that the class of the object in the image must be known to the model. Since the VGG-16 model is trained on over 1.2 million images, it recognizes a lot of objects, even 1000 different ones. As it has learned about the golden retriever, we can expect a good result. If we give the model an image with an object it does not recognize, it will give us predictions of classes whose images most resemble ours. We will see at the end which object classes resemble the golden retriever.
Since the image that we need to pass to the model needs to be specially prepared, the following section covers the preparations.

We will also load a couple of libraries that will make our work easier. It is common for the community to share all accompanying functionalities that are useful in these steps when sharing models.

from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.vgg16 import preprocess_input, decode_predictions

First, we will prepare the image by setting its dimensions to 224x224 and specifying that it uses three color channels RGB.

test_image = test_image.resize((224, 224))
test_image = test_image.convert('RGB')
plt.imshow(test_image)
plt.axis('off')
plt.show()

Next, we will ensure that the image is in the appropriate matrix format.

matrix_form_test_image = image.img_to_array(test_image)
print(matrix_form_test_image.shape)

As you know, networks always work with data batches. Therefore, we will create a batch that contains our image. As you would expect, its dimension will be 1 x 224 x 224 x 3 because we have only one image in the batch.

batch = np.expand_dims(matrix_form_test_image, axis=0)
print(f'(number_of_test_images, height, width, number_of_channels) = {batch.shape}')

Next is the part that deals with numerical preprocessing of the image in the form of normalization. A special function `preprocess_input` that accompanies the model will help us with this.

test_image_batch = preprocess_input(batch)

Since we have finished the preparation, we can now use the model for the classification task. The function that will help us in this part is called `predict`.

model_predictions = model.predict(test_image_batch)

The obtained model predictions represent the probabilities of our image belonging to each of the 1000 classes.

model_predictions.shape

To extract the class to which our image belongs, we can use the `decode_predictions` function, which will return the probabilities and names of the 5 most likely classes. This will give us insight into how confident the model is during classification.

most_likely_classes = decode_predictions(model_predictions)[0]
print(most_likely_classes)

As we can see, in the first place is the class *golden retriever* with a probability of 0.8042132. To better follow the model output, we can also display it graphically.

class_names = [item[1] for item in most_likely_classes]
class_probabilities = [item[2] for item in most_likely_classes]
plt.figure(figsize=(10, 4))
plt.bar(class_names, class_probabilities, color=['teal', 'yellow', 'green', 'blue', 'orange'])
plt.title('Top 5 Most Likely Classes')
plt.ylabel('Probability')
plt.show()

We see that the model has predicted with high confidence that the image we selected is a golden retriever. Some of the other classes the model considered are other types of retrievers. Interestingly, a tennis ball also appeared in the list of results.