This article is the fifth article in the **Introduction to Deep Learning (deep learning tutorial, DLT)** series. It mainly introduces multivariate classification. Students who want to learn deep learning or machine learning can pay attention to the official account GeodataAnalysis, and I will gradually update this series of articles.

1 Multivariate classification model representation

Multivariate classification is also called "one-to-many" or "one-to-remainder" classification, and its essence is still binary classification. It is nothing more than one class among multiple classes marked as forward class ( y = 1 y=1 y=1), and then mark all other classes as negative classes, this model is denoted as h θ ( 1 ) ( x ) h_\theta^{\left( 1 \right)}\left( x \right) hθ(1)​(x). Next, similarly we select another class labeled as the forward class ( y = 2 y=2 y=2), and then mark other classes as negative classes, and record this model as h θ ( 2 ) ( x ) h_\theta^{\left( 2 \right)}\left( x \right) hθ(2)​(x), and so on.

In the end we get a series of model abbreviations as: h θ ( i ) ( x ) = p ( y = i ∣ x ; θ ) h_\theta^{\left( i \right)}\left( x \right)=p\left( y=i|x;\theta \right) hθ(i)​(x)=p(y=i∣x;θ) where: i = ( 1 , 2 , 3.... k ) i=\left( 1,2,3....k \right) i=(1,2,3....k)

When we need to make a prediction, we run all the classifiers once, and then for each input variable, we choose the output variable with the highest probability.

Anyway, if you've learned binary classification, then we've done what we need to do, now we need to train this logistic regression classifier: h θ ( i ) ( x ) h_\theta^{\left( i \right)}\left( x \right) hθ(i)​(x), where i i i corresponds to every possible y = i y=i y=i, and finally, to make a prediction, we give input a new x x x value, use this to make predictions. All we have to do is feed into our three classifiers x x x, then we choose a h θ ( i ) ( x ) h_\theta^{\left( i \right)}\left( x \right) hθ(i)​(x) largest i i i, namely max ⁡ i   h θ ( i ) ( x ) \mathop{\max}\limits_i\,h_\theta^{\left( i \right)}\left( x \right) imax​hθ(i)​(x).

2 Encapsulating the binary classification model

Through the previous explanations, we already know that the multivariate classification model is essentially a collection of many binary classification models. Therefore, we first encapsulate the binary classification model in the previous chapter for subsequent use. The encapsulation method is similar to the model encapsulation of multiple linear regression in Chapter 3, the only difference is that a parameter is added when initializing the instance, and some transformations can be made to the data. The code is as follows:

class LogicRe():

	def __init__(self, x, y, times=1):
		self.y = y
		self.x = x
		self.times = times
		for i in range(2, times+1):
		    self.x = np.vstack((self.x, x**i))
		self.input_shape = self.x.shape
		self.parameters = np.random.rand(self.input_shape[0]+1)

	# Omit other functions, please refer to the code of multiple linear regression in Chapter 3

    def predict(self, input_x):
        x = input_x.copy()
        for i in range(2, self.times+1):
            x = np.vstack((x, input_x**i))
        x = self._normalization(x, self.normalization)
        x = np.vstack((np.ones(self.input_shape[1]), x))
        p = self.hypothesis_fun(x, self.parameters)
        return p

3 Luanwei flower data set

from sklearn import datasets

iris = datasets.load_iris()

x = iris.data.T
y = iris.target
x.shape, y.shape, np.unique(y)

((4, 150), (150,), array([0, 1, 2]))

4 Model Training and Prediction

models = []
for i in np.unique(y):
	x2, y2 = x.copy(), y.copy()
	y2[y==i] = 1
	y2[y!=i] = 0
	model = LogicRe(x2, y2, times=3)
	model.fit(epoch_size=1000, batch_size=100, 
			train_step=100, learning_rate=0.1, 
			normalization='feature')
	models.append(model)

predict_ys = []
for model in models:
    predict_y = model.predict(x)
    predict_ys.append(predict_y)
predict_y = np.array(predict_ys)
predict_y = np.argmax(predict_y, axis=0)

fig, axs = plt.subplots(1, 3, figsize=(27, 9), constrained_layout=True)

for i, (model, ax) in enumerate(zip(models, axs)):
	ax.plot(model.loss[:])
	ax.set_xlabel('Iteration number')
	ax.set_ylabel('Loss')
	ax.set_title('Class {}'.format(i+1), fontdict={'size': 30})
plt.show()

Visualization of prediction results

fig, ax = plt.subplots(1, 1, figsize=(16, 9))

ax.plot(y, 'k-', label='Real')
ax.plot(predict_y, 'r--', label='Predict')
plt.legend(fontsize='xx-large')
plt.show();