上篇文章，自己推导了一遍误差反向传播算法，从而对深度学习有了一定的认识。这期试着自己搭建一个基于python的深度学习神经网，来解决一定的问题。
需要用到：python3、numpy
数据集：MNIST数据集（一个被”嚼烂”了的数据集, 很多教程都会对它”下手”, 几乎成为一个 “典范”）
MNIST 数据集可在 http://yann.lecun.com/exdb/mnist/ 获取, 它包含了四个部分:
Training set images: train-images-idx3-ubyte.gz (9.9 MB, 解压后 47 MB, 包含 60,000 个样本)
Training set labels: train-labels-idx1-ubyte.gz (29 KB, 解压后 60 KB, 包含 60,000 个标签)
Test set images: t10k-images-idx3-ubyte.gz (1.6 MB, 解压后 7.8 MB, 包含 10,000 个样本)
Test set labels: t10k-labels-idx1-ubyte.gz (5KB, 解压后 10 KB, 包含 10,000 个标签)
MNIST 数据集来自美国国家标准与技术研究所, National Institute of Standards and Technology (NIST). 训练集 (training set) 由来自 250 个不同人手写的数字构成, 其中 50% 是高中学生, 50% 来自人口普查局 (the Census Bureau) 的工作人员. 测试集(test set) 也是同样比例的手写数字数据。
我们创建一个DataParser.py文件来解析MNIST数据：
def load_mnist(path, kind='train'):
labels_path = os.path.join(path, '%s-labels-idx1-ubyte' % kind)
images_path = os.path.join(path, '%s-images-idx3-ubyte' % kind)
with open(labels_path, 'rb')as lbfile:
magic, n = struct.unpack('>II', lbfile.read(8))
labels = np.fromfile(lbfile, dtype=np.uint8)
with open(images_path, 'rb')as imgfile:
magic, num, rows, cols = struct.unpack('>IIII', imgfile.read(16))
images = np.fromfile(imgfile, dtype=np.uint8).reshape(len(labels), 784)
return images /255, labels
除以255把图片像素值映射到[0，1]上。
手写图片的尺寸是28*28
训练思路：
我们将训练过程分为60个epochs，每个epoch里训练1000条数据，训练结束后用测试数据集来计算训练的准确率。
每次测试时，我们需要打乱测试数据，以保证测试结果更准确。
确定网络参数：
由于深度学习需要包括至少2个隐层，所以我们将网络结构定义如下：
数据层——全连接层1——隐层1——全连接层2——隐层2——输出层
数据层、全连接层1各有784个神经元（对应手写图片上的28*28个像素），全连接层2包含256个，输出层10个（对应0—9的概率）
激活函数采用Sigmoid函数，损失函数采用交叉熵损失函数（交叉熵损失函数的导数分母有一个1-x项，尔Sigmoid函数的导数分子有一个1-x项，两者相乘的时候完美解决了使用Sigmoid函数作为激活函数而产生的梯度发散问题）
下面我们就可以用python来实现这个网络了。
数据层定义如下:（数据层正向传播输出其本身，且无需反向传播）
class DataLayer:
def __init__(self):
imgs, lbls = dp.load_mnist('', kind='train')
self.x = imgs
self.y = lbls
self.pos = 0
def forward(self, index):
pos = index
xx = self.x[pos]
ret = (xx.reshape(xx.size,1), self.y[pos])
return ret
def backward(self, d):
pass
测试数据层定义如下:（使用shuffle_data每次打乱数据）
class TestLayer:
def __init__(self):
imgs, lbls = dp.load_mnist('', kind='t10k')
self.x = imgs
self.y = lbls
def shuffle_data(self):
l = len(self.x)
index = list(range(l))
np.random.shuffle(index)
self.x = self.x[index]
self.y = self.y[index]
def forward(self, index):
xx = self.x[index]
ret = (xx.reshape(xx.size,1), self.y[index])
return ret
全连接层如下：（权重除以一个值np.sqrt(l_x)，会避免产生异常数据）
向前传播forward：矩阵运算，计算z值
反向传播backward：根据接收到的误差更新权重和偏置
class FullConnect:
def __init__(self, l_x, l_y):
self.weights = np.random.randn(l_y, l_x) / np.sqrt(l_x)
self.bias = np.random.randn(l_y, 1)
self.lr = 0
def forward(self, x):
self.x = x
self.y = np.dot(self.weights, x)+self.bias
return self.y
def backward(self, d):
self.dw = np.dot(d, self.x.T)
self.db = d
self.dx = np.dot(self.weights.T, d)
self.weights -= self.lr * self.dw
self.bias -= self.lr * self.db
return self.dx
激活函数：
class Sigmoid:
def __init__(self):
pass
def sigmoid(self, x):
return 1 / (1 + np.exp(-x))
def forward(self, x):
self.x = x
self.y = self.sigmoid(x)
return self.y
def backward(self, d):
sig = self.sigmoid(self.x)
self.dx =d * sig * (1 - sig)
return self.dx # 反向传递梯度
损失函数：
class CrossEntropyLoss:
def __init__(self):
pass
def forward(self, x, label):
self.x = x
self.label = np.zeros_like(x)
self.label[label]=1.0
self.loss=np.nan_to_num(-self.label * np.log(x) -(1- self.label)*np.log(1-x))
self.loss=np.sum(self.loss) / x.shape[0]
return self.loss
def backward(self):
self.dx=(self.x-self.label)/self.x /(1- self.x)
return self.dx
确定准确度的方法为：(数组x中最大值（最大概率）的下标和标记值一样，我们认为结果准确，反之不正确)
class Accuracy:
def __init__(self):
pass
def forward(self, x, label):
b = x.max()
if x[label] > b-0.0001:
return 1
else:
return 0
然后我们将这些层进行简单的组装，就可以进行训练了：
def train():
block_size = 1000
datalayer1 = DataLayer()
datalayer2 = TestLayer()
inner_layers = []
inner_layers.append(FullConnect(784, 256))
inner_layers.append(Sigmoid())
inner_layers.append(FullConnect(256, 10))
inner_layers.append(Sigmoid())
lostlayer = CrossEntropyLoss()
accuracy = Accuracy()
epochs = 60
for layer in inner_layers:
layer.lr = 0.05
for i in range(epochs):
print('epochs: ', i)
losssum = 0
iters = 0
for j in range(block_size):
losssum = 0
iters += 1
x, label = datalayer1.forward(i * block_size +j)
for layer in inner_layers:
x = layer.forward(x)
loss = lostlayer.forward(x, label)
losssum += loss
d = lostlayer.backward()
for layer in inner_layers[::-1]:
d = layer.backward(d)
if j == block_size-1:
accu = 0
datalayer2.shuffle_data()
for k in range(10000):
x, label = datalayer2.forward(index=k)
for layer in inner_layers:
x = layer.forward(x)
accu += accuracy.forward(x, label)
print('accuracy: ', accu/10000)
我们把学习速度定位0.05，然后经过不到2分钟的训练，就能将准确率提升到96%了。输出数据我们可以看到，第一回合到第二回合性能上有了很大的提升。
epochs: 0
accuracy: 0.6473
epochs: 1
accuracy: 0.846
epochs: 2
accuracy: 0.8799
epochs: 3
accuracy: 0.8828
epochs: 4
accuracy: 0.8964
epochs: 5
accuracy: 0.8988
epochs: 6
accuracy: 0.9022
epochs: 7
accuracy: 0.8958
epochs: 8
accuracy: 0.9058
epochs: 9
accuracy: 0.9157
epochs: 10
accuracy: 0.9055
epochs: 11
accuracy: 0.915
epochs: 12
accuracy: 0.9149
epochs: 13
accuracy: 0.9114
epochs: 14
accuracy: 0.9259
epochs: 15
accuracy: 0.9226
epochs: 16
accuracy: 0.9308
epochs: 17
accuracy: 0.9304
epochs: 18
accuracy: 0.9394
epochs: 19
accuracy: 0.9323
epochs: 20
accuracy: 0.9328
epochs: 21
accuracy: 0.9356
epochs: 22
accuracy: 0.94
epochs: 23
accuracy: 0.9456
epochs: 24
accuracy: 0.9431
epochs: 25
accuracy: 0.9374
epochs: 26
accuracy: 0.9466
epochs: 27
accuracy: 0.9468
epochs: 28
accuracy: 0.9465
epochs: 29
accuracy: 0.9389
epochs: 30
accuracy: 0.9471
epochs: 31
accuracy: 0.9473
epochs: 32
accuracy: 0.9524
epochs: 33
accuracy: 0.953
epochs: 34
accuracy: 0.9501
epochs: 35
accuracy: 0.944
epochs: 36
accuracy: 0.9458
epochs: 37
accuracy: 0.9554
epochs: 38
accuracy: 0.9556
epochs: 39
accuracy: 0.9536
epochs: 40
accuracy: 0.9497
epochs: 41
accuracy: 0.9557
epochs: 42
accuracy: 0.9548
epochs: 43
accuracy: 0.9517
epochs: 44
accuracy: 0.9496
epochs: 45
accuracy: 0.9495
epochs: 46
accuracy: 0.9579
epochs: 47
accuracy: 0.9573
epochs: 48
accuracy: 0.9485
epochs: 49
accuracy: 0.9605
epochs: 50
accuracy: 0.9618
epochs: 51
accuracy: 0.9625
epochs: 52
accuracy: 0.9599
epochs: 53
accuracy: 0.9574
epochs: 54
accuracy: 0.961
epochs: 55
accuracy: 0.9621
epochs: 56
accuracy: 0.9597
epochs: 57
accuracy: 0.9565
epochs: 58
accuracy: 0.9591
epochs: 59
accuracy: 0.9604
到此，通过搜集网上各种资料，自己实现了第一个深度学习神经网络，中间不免有不合理的地方，继续加油！！！！
