Activity address: CSDN 21-day Learning Challenge

1 Introduction

There are many simple regression problems in life, such as a significant linear relationship between two variables. We can use regression to quantify analysis problems and predict variables.

In this example, we explore the relationship between speed and braking displacement.

0	4	2
1	4	10
2	7	4
3	7	22
4	8	16

View the data

From the image, we can see that there is a linear relationship between the two variables. Next, we quantify it by building a regression model.

2 Regression Model

Our goal is to find the parameters for the following functions

        

Where x is the speed and y is the braking distance.

We need to set up a function to measure the difference between the predicted and the true values so that we know how well the final parameters we get work.

The loss function is:

n is the number of samples,For the true value of a brake,Is the predicted value.

Naturally, we need L to reach the minimum so that we can get the most accurate regression model. There are two ways to do this.

3 Two Solutions

3.1 Direct Method

The direct method can be used to find the optimal solution directly, but it needs to satisfy that the loss function is convex and the solution is analytical.

[Concavity and Convexity of Functions]

Set a function to f(x) with two points on the image, the point above chord AB is.

Then the equation of chord AB is:

amongThenThe parametric equation for chord AB is:

Because the function value of point P on the chord where f(x) is predecessor A and B is greater than that on f(x), there are:

Parameter calculation method refers to blog: least square method

Experimental section:

Draw the resulting line:

Then the function relationship is:

3.2 Iteration Method

There are limitations to optimizing loss functions directly, and it is difficult to find an optimal solution if the loss function is nonconvex. Therefore, an iterative method is proposed, similar to the back propagation algorithm of the previously learned neural network, to minimize the loss value by updating the parameters continuously and slightly. Therefore, the difference between Iteration and Direct is that the parameters can be updated by direct calculation, while the latter needs to be updated by small batch gradient descent.

The loss function is:

Is batch size

Goal Solving:​

Parameter update:For learning rate

Experimental section:

import  random

#Variable Update Function
def update_var(pre_alpha,pre_beta,y,x,lr):
    n=len(x)
    diff_alpha=np.sum(-(y-pre_beta*x-pre_alpha))/n
    diff_beta =np.sum(-x*(y-pre_beta*x-pre_alpha))/n
    new_alpha=pre_alpha-lr*diff_alpha
    new_beta=pre_beta-lr*diff_beta
    
    return (new_alpha,new_beta)

#Implement Iterative Process
def iterative_func(y,x,raw_alpha,raw_beta,lr,num,sample_num):
    alpha_list = []
    beta_list = []
    alpha = raw_alpha
    beta = raw_beta
    num_list = list(range(1, len(y)+1))
    for i in range(num):
        alpha_list.append(alpha)
        beta_list.append(beta)
        random.shuffle(num_list)
        index = num_list[:sample_num]
        alpha, beta = update_var(alpha, beta,y[index], x[index], lr)
        print("[{}]Now alpha:{},beta:{}".format(i,alpha,beta))
    return (alpha_list, beta_list)

#Random initialization of alpha and beta
raw_alpha=np.random.random()*10
raw_beta =np.random.random()*10
raw_alpha

#Setting superparameters: learning rate lr=0.005 iterations num=2000,sample_num=16
lr = 0.005
num = 10000
sample_num = 16
alpha_list, beta_list = iterative_func(y, x, raw_alpha, raw_beta,
lr, num,sample_num)

print("after {} times iteration:alpha: {}, beta:{}".format(num,alpha_list[-1], beta_list[-1]))

after 10000 times iteration:alpha: -17.791370073935074, beta:4.000650060840287

#Persistence of data from parameter iteration
import csv

var_data=zip(alpha_list,beta_list)
with open("../data/20220801_vardata.csv",'w',newline='')as f:
        csv_writer=csv.writer(f)
        csv_writer.writerows(['alpha','beta'])
        csv_writer.writerows(var_data)

plt.subplot(121)
plt.plot(alpha_list)
plt.title("alpha change process")
plt.subplot(122)
plt.plot(beta_list)
plt.title("beta change process")
plt.show()

Reference resources

(44 messages) Machine Learning Theory and Case Analysis (part2) --Regression_ GoatGui's Blog - CSDN Blog