100 cases of deep learning - convolutional neural network (CNN) attention detection | day 17

1, Preliminary work

What should children do if they are always distracted in class? The attention detection program is arranged! Driver fatigue driving how to do, attention detection program arrangement! This paper will achieve the purpose of detecting attention through the recognition of human eye state ⏳ Add confusion matrix module.

🚀 My environment:

Locale: Python 3 six point five
compiler: jupyter notebook
Deep learning environment: tensorflow2 four point one
Data link: https://pan.baidu.com/s/1kIk75W8vLC1TkjKaA0odLw Extraction code: nolr

🚀 In depth learning newcomers must see:

🚀 Previous highlights:

🚀 From column: 100 cases of deep learning

1. Set GPU

If you are using a CPU, you can comment out this part of the code.

import tensorflow as tf

gpus = tf.config.list_physical_devices("GPU")

if gpus:
    tf.config.experimental.set_memory_growth(gpus[0], True)  #Set the amount of GPU video memory and use it on demand
    tf.config.set_visible_devices([gpus[0]],"GPU")

# Print the graphics card information and confirm that the GPU is available
print(gpus)

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

2. Import data

import matplotlib.pyplot as plt
# Support Chinese
plt.rcParams['font.sans-serif'] = ['SimHei']  # Used to display Chinese labels normally
plt.rcParams['axes.unicode_minus'] = False  # Used to display negative signs normally

import os,PIL

# Set random seeds so that the results can be reproduced as much as possible
import numpy as np
np.random.seed(1)

# Set random seeds so that the results can be reproduced as much as possible
import tensorflow as tf
tf.random.set_seed(1)

import pathlib

data_dir = "D:/jupyter notebook/DL-100-days/datasets/017_Eye_dataset"

data_dir = pathlib.Path(data_dir)

3. View data

image_count = len(list(data_dir.glob('*/*')))

print("The total number of pictures is:",image_count)

Total number of pictures: 4307

2, Data preprocessing

1. Load data

Using image_ dataset_ from_ The directory method loads the data from the disk into TF data. In dataset

batch_size = 64
img_height = 224
img_width = 224

Students with TensorFlow version 2.2.0 may encounter module 'TensorFlow keras. preprocessing' has no attribute 'image_ dataset_ from_ The error of 'directory' is reported. Just upgrade TensorFlow.

"""
about image_dataset_from_directory()Please refer to the following article for details: https://mtyjkh.blog.csdn.net/article/details/117018789
"""
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    validation_split=0.2,
    subset="training",
    seed=12,
    image_size=(img_height, img_width),
    batch_size=batch_size)

Found 4307 files belonging to 4 classes.
Using 3446 files for training.

"""
about image_dataset_from_directory()Please refer to the following article for details: https://mtyjkh.blog.csdn.net/article/details/117018789
"""
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    data_dir,
    validation_split=0.2,
    subset="validation",
    seed=12,
    image_size=(img_height, img_width),
    batch_size=batch_size)

Found 4307 files belonging to 4 classes.
Using 861 files for validation.

We can use class_names the label of the output dataset. The labels will correspond to the directory name in alphabetical order.

class_names = train_ds.class_names
print(class_names)

['close_look', 'forward_look', 'left_look', 'right_look']

2. Visual data

plt.figure(figsize=(10, 5))  # The width of the figure is 10 and the height is 5
plt.suptitle("Data display")

for images, labels in train_ds.take(1):
    for i in range(8):
        
        ax = plt.subplot(2, 4, i + 1)  
        
        ax.patch.set_facecolor('yellow')
        
        plt.imshow(images[i].numpy().astype("uint8"))
        plt.title(class_names[labels[i]])
        
        plt.axis("off")

3. Recheck the data

for image_batch, labels_batch in train_ds:
    print(image_batch.shape)
    print(labels_batch.shape)
    break

(64, 224, 224, 3)
(64,)

Image_batch is the tensor of the shape (8, 224, 224, 3). This is a batch of 8 pictures with shape of 240x240x3 (the last dimension refers to color channel RGB).
Label_batch is the tensor of shape (8,), and these labels correspond to 8 pictures

4. Configure dataset

shuffle(): scramble data. For a detailed description of this function, please refer to: https://zhuanlan.zhihu.com/p/42417456
prefetch(): prefetch data and speed up operation. For details, please refer to my previous two articles, which are explained in them.
cache(): cache data sets into memory to speed up operation

AUTOTUNE = tf.data.AUTOTUNE

train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds   = val_ds.cache().prefetch(buffer_size=AUTOTUNE)

3, Call the official network model

model = tf.keras.applications.VGG16()
# Print model information
model.summary()

Model: "vgg16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________

......   
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0         
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544 
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312  
_________________________________________________________________
predictions (Dense)          (None, 1000)              4097000   
=================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
_________________________________________________________________

4, Set dynamic learning rate

Here are the advantages and disadvantages of high learning rate and low learning rate.

High learning rate
- Advantages: 1. Speed up the learning rate. 2. It helps to jump out of the local optimal value.
- Disadvantages: 1. Model training does not converge. 2. Only using the college attendance rate can easily lead to the inaccuracy of the model.
Low learning rate
- Advantages: 1. It is conducive to model convergence and model refinement. 2. 2. Improve model accuracy.
- Disadvantages: 1. It is difficult to jump out of the local optimal value. 2. Slow convergence.

Note: the dynamic learning rate set here is exponential decay. Before each epoch starts, the learning_rate will be reset to the initial_learning_rate, and then the attenuation will start again. The calculation formula is as follows:

learning_rate = initial_learning_rate * decay_rate ^ (step / decay_steps)

# Set initial learning rate
initial_learning_rate = 1e-4

lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate, 
        decay_steps=20,      # Knock on the blackboard!!! This refers to steps, not epochs
        decay_rate=0.96,     # LR will become decay after one attenuation_ rate*lr
        staircase=True)

# Feed exponential decay learning rate into optimizer
optimizer = tf.keras.optimizers.Adam(learning_rate=lr_schedule)

5, Compile

Before preparing to train the model, you need to make some more settings. The following is added in the compilation step of the model:

Loss function (loss): used to measure the accuracy of the model during training.
optimizer: determines how the model updates based on the data it sees and its own loss function.
metrics: used to monitor training and testing steps. The following example uses the accuracy rate, that is, the ratio of images correctly classified.

model.compile(optimizer=optimizer,
              loss     ='sparse_categorical_crossentropy',
              metrics  =['accuracy'])

6, Training model

epochs = 10

history = model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=epochs
)

Epoch 1/10
54/54 [==============================] - 29s 377ms/step - loss: 1.0835 - accuracy: 0.6709 - val_loss: 0.6919 - val_accuracy: 0.7933
Epoch 2/10
54/54 [==============================] - 14s 268ms/step - loss: 0.2332 - accuracy: 0.9248 - val_loss: 0.1173 - val_accuracy: 0.9628
Epoch 3/10
54/54 [==============================] - 14s 259ms/step - loss: 0.1072 - accuracy: 0.9634 - val_loss: 0.0771 - val_accuracy: 0.9779
Epoch 4/10
54/54 [==============================] - 14s 256ms/step - loss: 0.0663 - accuracy: 0.9794 - val_loss: 0.0566 - val_accuracy: 0.9826
Epoch 5/10
54/54 [==============================] - 14s 258ms/step - loss: 0.0480 - accuracy: 0.9855 - val_loss: 0.0609 - val_accuracy: 0.9768
Epoch 6/10
54/54 [==============================] - 14s 258ms/step - loss: 0.0431 - accuracy: 0.9852 - val_loss: 0.0597 - val_accuracy: 0.9768
Epoch 7/10
54/54 [==============================] - 14s 256ms/step - loss: 0.0289 - accuracy: 0.9910 - val_loss: 0.0539 - val_accuracy: 0.9837
Epoch 8/10
54/54 [==============================] - 14s 259ms/step - loss: 0.0221 - accuracy: 0.9927 - val_loss: 0.0626 - val_accuracy: 0.9744
Epoch 9/10
54/54 [==============================] - 14s 257ms/step - loss: 0.0281 - accuracy: 0.9910 - val_loss: 0.0605 - val_accuracy: 0.9814
Epoch 10/10
54/54 [==============================] - 14s 257ms/step - loss: 0.0203 - accuracy: 0.9936 - val_loss: 0.0663 - val_accuracy: 0.9791

7, Model evaluation

1. Accuracy and Loss diagram

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs_range = range(epochs)

plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)

plt.plot(epochs_range, acc, label='Training Accuracy')
plt.plot(epochs_range, val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')

plt.subplot(1, 2, 2)
plt.plot(epochs_range, loss, label='Training Loss')
plt.plot(epochs_range, val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.show()

2. Confusion matrix

Seaborn is a drawing library. It carries out higher-level API encapsulation based on the Matplotlib core library, which allows you to easily draw more beautiful graphics. Seaborn's beauty is mainly reflected in more comfortable color matching and more delicate style of graphic elements.

from sklearn.metrics import confusion_matrix
import seaborn as sns
import pandas as pd

# Define a function for drawing confusion matrix graph
def plot_cm(labels, predictions):
    
    # Generating confusion matrix
    conf_numpy = confusion_matrix(labels, predictions)
    # Convert matrix to DataFrame
    conf_df = pd.DataFrame(conf_numpy, index=class_names ,columns=class_names)  
    
    plt.figure(figsize=(8,7))
    
    sns.heatmap(conf_df, annot=True, fmt="d", cmap="BuPu")
    
    plt.title('Confusion matrix',fontsize=15)
    plt.ylabel('True value',fontsize=14)
    plt.xlabel('Estimate',fontsize=14)

val_pre   = []
val_label = []

for images, labels in val_ds:#Here, you can take part of the validation data (. take(1)) to generate a confusion matrix
    for image, label in zip(images, labels):
        # You need to add a dimension to the picture
        img_array = tf.expand_dims(image, 0) 
        # Use the model to predict the characters in the picture
        prediction = model.predict(img_array)

        val_pre.append(class_names[np.argmax(prediction)])
        val_label.append(class_names[label])

plot_cm(val_label, val_pre)

8, Save and load model

This is the simplest method of model saving and loading

# Save model
model.save('model/17_model.h5')

# Loading model
new_model = tf.keras.models.load_model('model/17_model.h5')

9, Forecast

# Use the loaded model (new_model) to see the prediction results

plt.figure(figsize=(10, 5))  # The width of the figure is 10 and the height is 5
plt.suptitle("Display of prediction results")

for images, labels in val_ds.take(1):
    for i in range(8):
        ax = plt.subplot(2, 4, i + 1)  
        
        # display picture
        plt.imshow(images[i].numpy().astype("uint8"))
        
        # You need to add a dimension to the picture
        img_array = tf.expand_dims(images[i], 0) 
        
        # Use the model to predict the characters in the picture
        predictions = new_model.predict(img_array)
        plt.title(class_names[np.argmax(predictions)])

        plt.axis("off")

Previous highlights