1 Overview

This article introduces the two main contents of video analysis in OpenCV-Python, namely:

• 🐟Background difference method for moving object detection
• 🐟Meanshift and Camshift algorithm object tracking

PS: Video analysis also has an important content: optical flow method motion trajectory drawing, which will be introduced in future articles.

2. Principle introduction

2.1 Background difference method for moving object detection

Most of the scene captured by a static camera does not change, and this part generally does not need special attention. The dynamic objects in the corresponding video are more worthy of attention and analysis.

The background difference method is very simple in principle. It subtracts the current picture from the background, subtracts the unchanged background, and leaves the changing objects (usually moving) in the scene. Borrowing a picture from the official website is easy to understand.

 

Therefore, a key part of the algorithm is the construction of the background image, which consists of two parts: ① initialize the background and ② background update. But these two steps do not need to write their own code in OpenCV-Python, createBackgroundSubtractor() can generate the object of the related algorithm, the following two sentences are the background masks under different algorithms are MOG2 and KNN, they use the same, use The apply() method can input the picture frame by frame. From the results, the KNN effect is better.

#Choose one of the following two lines
#backSub = cv.createBackgroundSubtractorMOG2()
backSub = cv.createBackgroundSubtractorKNN()
fgMask = backSub.apply(frame)

After extracting the background mask, the extraction of the target can be done using a simple code that is:

Deobject=cv.add(frame,frame,mask=fgMask)

The relevant steps show the effect as follows:

original image	background mask	target extraction

2.2 Meanshift and Camshift algorithms track objects

Meanshift can be translated as mean shift. Camshift is called Continuously Apative Mean-Shift, which is an improvement of Meanshift.

First introduce Meanshift, which actually uses the histogram of the object to be tracked as a reference. In the next frame of each graphic, search for the area near the object. If you find an area with a great similarity to the histogram of the object, judge him. Continue tracking for that object. Therefore, in this algorithm, we need to go through several steps. As shown below:

• Select the area to track the object in the first frame
• Convert color to HSV, then generate histogram
• Find an area similar to the current histogram in the next frame and delineate that area as the object to track. This step uses cv.CamShift(probImage, window, criteria), its three parameters are histogram back projection, the window where the object is located, and the algorithm stops the criteria
• Proceed to the next frame to search for tracking.

The following introduces Camshift, which mainly solves a problem of Meanshift, that is, the window size of Meanshift is fixed. Because the objects in the video have the characteristics of near and far, so if the object moves before and after the lens, the recognition accuracy of the algorithm is not high. Camshift takes this problem into account, and it will adjust the window size when the Meanshift matching rate is low. The detailed steps are introduced in the code.

Effect comparison:

initial image	algorithm	Tracking Results
	Meanshift
	Camshift

3. Code Analysis

See the article for environment configuration OpenCv-Python image feature recognition - Matrix_CS's blog - CSDN blog

3.1 Background difference method for moving object detection

This method uses the BackgroundSubtractor in Open-CV to automatically generate and update the background, and finally uses the background mask and the original image to perform operations to extract moving objects.

from __future__ import print_function
import cv2 as cv
import argparse

'''
The code attempts to use the background difference method to complete the extraction of dynamic objects in the fixed camera.
'''
#There are two algorithms to choose from, KNN and MOG2, the following code uses KNN as an attempt
algo='KNN'
if algo == 'MOG2':
    backSub = cv.createBackgroundSubtractorMOG2()
else:
    backSub = cv.createBackgroundSubtractorKNN()
#Open a video file
capture = cv.VideoCapture(cv.samples.findFileOrKeep('vtest.avi'))
#Determine whether the video is read successfully
if not capture.isOpened():
    print('Unable to open')
    exit(0)
#Read the video frame by frame for correlation analysis
while True:
    #Read the first frame of the video
    ret, frame = capture.read()
    if frame is None:
        break
    #Use the defined backSub object, enter a new frame, and generate a background mask
    fgMask = backSub.apply(frame)
    #Add the current frame of the original video and the mask to extract the foreground object
    Object=cv.add(frame,frame,mask=fgMask)
    #Display the objects in the video. The three windows represent the original video, the background, and the moving target.
    cv.imshow('Frame', frame)
    cv.imshow('FG Mask', fgMask)
    cv.imshow('Object',Object)
    #After each frame is displayed, wait 30 milliseconds
    keyboard = cv.waitKey(30)
    #Press q to exit the program
    if keyboard == 'q' or keyboard == 27:
        break

3.2 Meanshif algorithm for object tracking

Use the Meanshif algorithm to track the object, query the surrounding images of the moving object according to the histogram features of the object to be tracked, perform matching, and return the position of the moving object.

import numpy as np
import cv2 as cv
import argparse
'''
use Meanshift Algorithm to track the specified object in the video
'''

cap = cv.VideoCapture('slow_traffic_small.mp4')
#Name the display window
cv.namedWindow('image')
#Read the first frame of the video
ret,frame = cap.read()
#Define a range as the object to track, in this case a car
x, y, w, h = 306, 194, 80, 40
track_window = (x, y, w, h)
# From the first frame, extract the pictures in the range
roi = frame[y:y+h, x:x+w]
# Convert to hsv color, because this color format is generally used for drawing histograms
hsv_roi =  cv.cvtColor(roi, cv.COLOR_BGR2HSV)
#Generate a mask to filter out colors that are not in the range
mask = cv.inRange(hsv_roi, np.array((0., 60.,32.)), np.array((180.,255.,255.)))
#Generate histogram
roi_hist = cv.calcHist([hsv_roi],[0],mask,[180],[0,180])
#Do histogram normalization
cv.normalize(roi_hist,roi_hist,0,255,cv.NORM_MINMAX)
# Set termination criteria, 10 iterations or move at least 1 pixel
term_crit = ( cv.TERM_CRITERIA_EPS | cv.TERM_CRITERIA_COUNT, 10, 1 )
while(1):
    #Read the next frame of the video
    ret, frame = cap.read()
    #ret is used to determine whether the video is read successfully
    if ret == True:
        #Keep the same color format hsv as above
        hsv = cv.cvtColor(frame, cv.COLOR_BGR2HSV)
        #This step calculates the back projection of hist for the following as input for easy reading
        dst = cv.calcBackProject([hsv],[0],roi_hist,[0,180],1)
        # Use meanShift to find matching windows
        ret, track_window = cv.meanShift(dst, track_window, term_crit)
        #The following code is to use a rectangular box to circle the recognized objects
        x,y,w,h = track_window
        img2 = cv.rectangle(frame, (x,y), (x+w,y+h), 255,2)
        cv.imshow('image',img2)
        #Wait 30ms, press ESC to exit
        k = cv.waitKey(30) & 0xff
        if k == 27:
            break
    else:
        break

3.3 Camshift algorithm to track objects

This method is an improvement of the Meanshif algorithm. The window size can be adjusted according to the movement of the object. The code is very similar to the Meanshif algorithm.

import numpy as np
import cv2 as cv
import argparse
'''
use CamShift Algorithm to track objects, the algorithm improves MeanShift，Its window size will be custom adjusted
'''
#open video
cap = cv.VideoCapture('slow_traffic_small.mp4')
#Read the first frame of the video
ret,frame = cap.read()
#Define a range as the object to track, in this case a car
x, y, w, h = 306, 194, 80, 40 # simply hardcoded the values
track_window = (x, y, w, h)
# From the first frame, extract the pictures in the range
roi = frame[y:y+h, x:x+w]
# Convert to hsv color, because this color format is generally used for drawing histograms
hsv_roi =  cv.cvtColor(roi, cv.COLOR_BGR2HSV)
#Generate a mask to filter out colors that are not in the range
mask = cv.inRange(hsv_roi, np.array((0., 60.,32.)), np.array((180.,255.,255.)))
#Generate histogram
roi_hist = cv.calcHist([hsv_roi],[0],mask,[180],[0,180])
#Do histogram normalization
cv.normalize(roi_hist,roi_hist,0,255,cv.NORM_MINMAX)
# Set termination criteria, 10 iterations or move at least 1 pixel
term_crit = ( cv.TERM_CRITERIA_EPS | cv.TERM_CRITERIA_COUNT, 10, 1 )
while(1):
    #Read the next frame of the video
    ret, frame = cap.read()
    #ret is used to determine whether the video is read successfully
    if ret == True:
        # Keep the same color format hsv as above
        hsv = cv.cvtColor(frame, cv.COLOR_BGR2HSV)
        # This step calculates the back projection of hist for the following as input for easy reading
        dst = cv.calcBackProject([hsv],[0],roi_hist,[0,180],1)
        # Use CamShift to find matching windows
        ret, track_window = cv.CamShift(dst, track_window, term_crit)
        #The following code is to use a quadrilateral to frame the recognized object
        pts = cv.boxPoints(ret)
        pts = np.int0(pts)
        img2 = cv.polylines(frame,[pts],True, 255,2)
        cv.imshow('img2',img2)
        k = cv.waitKey(30) & 0xff
        # Wait 30ms, press ESC to exit
        if k == 27:
            break
    else:
        break