前言：
文章以Andrew Ng 的 deeplearning.ai 视频课程为主线，记录Programming Assignments 的实现过程。相对于斯坦福的CS231n课程，Andrew的视频课程更加简单易懂，适合深度学习的入门者系统学习！
本次作业主要是练习正交化，使用正交化可以有效缓解数据过拟合，提高网络的性能。正交化主要有两种方式实现，Regularization和Dropout，下面我们分别对这两种方式进行实现。
1.1 Dataset
首先还是看一下这次的数据集，还是一个binary classification问题。
train_X, train_Y, test_X, test_Y = load_2D_dataset()
1.2 Non-regularized model
我们首先看一下在没有正规化的情况下，网络的训练效果：
def model(X, Y, learning_rate = 0.3, num_iterations = 30000, print_cost = True, lambd = 0, keep_prob = 1):
grads = {}
costs = []
m = X.shape[1]
layers_dims = [X.shape[0], 20, 3, 1]
parameters = initialize_parameters(layers_dims)
for i in range(0, num_iterations):
if keep_prob == 1:
a3, cache = forward_propagation(X, parameters)
elif keep_prob < 1:
a3, cache = forward_propagation_with_dropout(X, parameters, keep_prob)
# Cost function
if lambd == 0:
cost = compute_cost(a3, Y)
else:
cost = compute_cost_with_regularization(a3, Y, parameters, lambd)
assert(lambd==0 or keep_prob==1)
if lambd == 0 and keep_prob == 1:
grads = backward_propagation(X, Y, cache)
elif lambd != 0:
grads = backward_propagation_with_regularization(X, Y, cache, lambd)
elif keep_prob < 1:
grads = backward_propagation_with_dropout(X, Y, cache, keep_prob)
parameters = update_parameters(parameters, grads, learning_rate)
if print_cost and i % 10000 == 0:
print("Cost after iteration {}: {}".format(i, cost))
if print_cost and i % 1000 == 0:
costs.append(cost)
plt.plot(costs)
plt.ylabel('cost')
plt.xlabel('iterations (x1,000)')
plt.title("Learning rate =" + str(learning_rate))
plt.show()
return parameters
parameters = model(train_X, train_Y)
print ("On the training set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)
训练结果如下：
我们再看一下训练结果形成的decision boundary
plt.title("Model without regularization")
axes = plt.gca()
axes.set_xlim([-0.75,0.40])
axes.set_ylim([-0.75,0.65])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)
从decision boundary中可以看出训练的网络明显过拟合了。
1.3 L2 Regularization
L2 Regularization主要针对的是权值的L2范数，相应的在前向算法计算cost函数和后向算法中都要做相应的修改
cost函数：
def compute_cost_with_regularization(A3, Y, parameters, lambd):
m = Y.shape[1]
W1 = parameters["W1"]
W2 = parameters["W2"]
W3 = parameters["W3"]
cross_entropy_cost = compute_cost(A3, Y)
L2_regularization_cost=lambd/(2*m)*(np.sum(W1*W1)+np.sum(W2*W2)+np.sum(W3*W3))
cost = cross_entropy_cost + L2_regularization_cost
return cost
后向算法：
def backward_propagation_with_regularization(X, Y, cache, lambd):
m = X.shape[1]
(Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache
dZ3 = A3 - Y
dW3 = 1./m * np.dot(dZ3, A2.T) + lambd/m*W3
db3 = 1./m * np.sum(dZ3, axis=1, keepdims = True)
dA2 = np.dot(W3.T, dZ3)
dZ2 = np.multiply(dA2, np.int64(A2 > 0))
dW2 = 1./m * np.dot(dZ2, A1.T) + lambd/m*W2
db2 = 1./m * np.sum(dZ2, axis=1, keepdims = True)
dA1 = np.dot(W2.T, dZ2)
dZ1 = np.multiply(dA1, np.int64(A1 > 0))
dW1 = 1./m * np.dot(dZ1, X.T) + lambd/m*W1
db1 = 1./m * np.sum(dZ1, axis=1, keepdims = True)
gradients = {"dZ3": dZ3, "dW3": dW3, "db3": db3,"dA2": dA2,
"dZ2": dZ2, "dW2": dW2, "db2": db2, "dA1": dA1,
"dZ1": dZ1, "dW1": dW1, "db1": db1}
return gradients
现在我们加入L2 Regularization后对模型的训练结果如下：
parameters = model(train_X, train_Y, lambd = 0.7)
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)
我们再看一下加入L2 Regularization后形成的decision boundary
plt.title("Model with L2-regularization")
axes = plt.gca()
axes.set_xlim([-0.75,0.40])
axes.set_ylim([-0.75,0.65])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)
可以明显看到过拟合的情况得到了有效的缓解，L2 Regularization 使 decision boundary 变得smoother，但是L2 Regularization中的超参数lambd需要tune
1.3 Dropout
dropout也是正规化常用的手段，在每次迭代训练的过程中随机关闭一些神经元，这些关闭的神经元对前向过程和反向梯度过程都不make contribution，通过这样的方式来避免过拟合的情况。我们来看一下具体的代码实现：
前向过程：
def forward_propagation_with_dropout(X, parameters, keep_prob = 0.5):
np.random.seed(1)
W1 = parameters["W1"]
b1 = parameters["b1"]
W2 = parameters["W2"]
b2 = parameters["b2"]
W3 = parameters["W3"]
b3 = parameters["b3"]
Z1 = np.dot(W1, X) + b1
A1 = relu(Z1)
D1 = np.random.rand(A1.shape[0],A1.shape[1])
D1 = D1<keep_prob
A1 = A1*D1
A1/=keep_prob
Z2=np.dot(W2,A1)+b2
A2=relu(Z2)
D2=np.random.rand(A2.shape[0],A2.shape[1])
D2=D2
D2=D2<leep_prob
A2=A2*D2
A2/=keep_prob
Z3=np.dot(W3,A2)+b3
A3=sigmoid(Z3)
cache = (Z1, D1, A1, W1, b1, Z2, D2, A2, W2, b2, Z3, A3, W3, b3)
return A3, cache
反向过程：前向过程关闭的神经元在反向过程中同样关闭即可
def backward_propagation_with_dropout(X, Y, cache, keep_prob):
m=Y.shape[1]
(Z1, D1, A1, W1, b1, Z2, D2, A2, W2, b2, Z3, A3, W3, b3) = cache
dZ3=A3-Y
dW3 = 1 / m * np.dot(dZ3, A2.T)
db3 = 1 / m * np.sum(dZ3, axis=1, keepdims=True)
dA2 = np.dot(W3.T, dZ3)
dA2=dA2*D2/keep_prob
temp = np.copy(dA2)
temp[Z2 < 0] = 0
dZ2 = temp
dW2 = 1 / m * np.dot(dZ2, A1.T)
db2 = 1 / m * np.sum(dZ2, axis=1, keepdims=True)
dA1= np.dot(W2.T, dZ2)
dA1 = dA1* D1 / keep_prob
temp = np.copy(dA1)
temp[Z1 < 0] = 0
dZ1 = temp
dW1 = 1 / m * np.dot(dZ1, X.T)
db1 = 1 / m * np.sum(dZ1, axis=1, keepdims=True)
gradients = {"dZ3": dZ3, "dW3": dW3, "db3": db3, "dA2": dA2,
"dZ2": dZ2, "dW2": dW2, "db2": db2, "dA1": dA1,
"dZ1": dZ1, "dW1": dW1, "db1": db1}
return gradients
现在我们对加入Dropout后对模型的训练结果：
parameters = model(train_X, train_Y, keep_prob = 0.86, learning_rate = 0.3)
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)
我们再看一下decision boundary
注意：Dropout 仅仅使用在train阶段，而不能将其使用在test阶段
最后附上我作业的得分，表示我程序没有问题，如果觉得我的文章对您有用，请随意打赏，我将持续更新Deeplearning.ai的作业！
