Deep belief network (DBN)
Deep belief network (DBN) was proposed by Geoffrey Hinton in 2006. It is a generation model. By training the weights among its neurons, we can make the whole neural network generate training data according to the maximum probability. We can not only use DBN to identify features and classify data, but also use it to generate data.
After the deep belief network has been launched, it has been sought after by countless people because of its effectiveness, and it also leads the direction of machine learning.
DBN is composed of two parts, one is the bottom network and the other is the top network. The bottom network can be composed of RBM (restricted Boltzmann machine). The top layer is usually a logic layer. When training, the bottom network is trained first, and the top network is trained.
The underlying structure is as follows:
You can customize the number of layers. The h of the first layer is used as the v of the second layer, and so on.
The top layer is a logic layer, and the h of the last layer of the bottom layer is used as the input of the logic layer for training.
Build DBN structure:
# construct multi-layer for i in range(self.n_layers): # layer_size if i == 0: input_size = n_ins else: input_size = hidden_layer_sizes[i - 1] # layer_input if i == 0: layer_input = self.x else: layer_input = self.sigmoid_layers[-1].sample_h_given_v() # construct sigmoid_layer sigmoid_layer = HiddenLayer(input=layer_input, n_in=input_size, n_out=hidden_layer_sizes[i], numpy_rng=numpy_rng, activation=sigmoid) self.sigmoid_layers.append(sigmoid_layer) # construct rbm_layer rbm_layer = RBM(input=layer_input, n_visible=input_size, n_hidden=hidden_layer_sizes[i], W=sigmoid_layer.W, # W, b are shared hbias=sigmoid_layer.b) self.rbm_layers.append(rbm_layer) # layer for output using Logistic Regression self.log_layer = LogisticRegression(input=self.sigmoid_layers[-1].sample_h_given_v(), label=self.y, n_in=hidden_layer_sizes[-1], n_out=n_outs)
Train the bottom layer and pay attention to the relationship between input and output of the bottom layer:
def pretrain(self, lr=0.1, k=1, epochs=100): # pre-train layer-wise for i in range(self.n_layers): if i == 0: layer_input = self.x else: layer_input = self.sigmoid_layers[i-1].sample_h_given_v(layer_input) rbm = self.rbm_layers[i] for epoch in range(epochs): rbm.contrastive_divergence(lr=lr, k=k, input=layer_input) # cost = rbm.get_reconstruction_cross_entropy() # print >> sys.stderr, \ # 'Pre-training layer %d, epoch %d, cost ' %(i, epoch), cost
Finally, train the logic layer:
def finetune(self, lr=0.1, epochs=100): layer_input = self.sigmoid_layers[-1].sample_h_given_v() # train log_layer epoch = 0 done_looping = False while (epoch < epochs) and (not done_looping): self.log_layer.train(lr=lr, input=layer_input) # self.finetune_cost = self.log_layer.negative_log_likelihood() # print >> sys.stderr, 'Training epoch %d, cost is ' % epoch, self.finetune_cost lr *= 0.95 epoch += 1
Due to the addition of a logic layer, the use of DBN is the same as that of softmax. The predicted output results are as follows:
def predict(self, x): layer_input = x for i in range(self.n_layers): sigmoid_layer = self.sigmoid_layers[i] layer_input = sigmoid_layer.output(input=layer_input) out = self.log_layer.predict(layer_input) return out
DBN is an important direction of machine learning and a reason why machine learning is so hot. There are many knowledge points involved in this process, but no specific process is given in the derivation process. If you need to know the derivation process, you can refer to specific literature. Here, if you talk about the derivation process too much, you can't explain the application process clearly. It can be seen that although DBN is similar to previous neural networks, However, its theoretical basis is different, which creates a new direction of e-learning.