Logistic Regression with SGD in 2 dimensions¶
Here are some training data points with labels, and a weight vector w initialized to all-zeros.
In [1]:
import numpy as np
# load data
Xy = np.loadtxt('synthetic.txt')
X = Xy[:, :-1]
y = Xy[:, -1]
# split into train and test
ttsplit = int(y.size*.8)
trainX = X[:ttsplit, :]
trainy = y[:ttsplit]
testX = X[ttsplit:, :]
testy = y[ttsplit:]
import matplotlib.pyplot as plt
%matplotlib inline
from demofuncs import *
plot_labeled_data(trainX, trainy)
Run a perceptron learner (the basic one without bias, etc) on this training data, and measure the accuracy on the test set.
Accuracy is measured in two ways: raw, and mean squared, which is
$$\dfrac{1}{p}\sum_{i=1}^p (y-h_\theta(x^{(i)}))^2$$In [4]:
def perceptron_train(X, y, maxiter):
num_points, num_dims = X.shape
w = np.zeros(num_dims)
for epoch in range(maxiter):
num_errors = 0
for i in range(num_points):
if y[i]*w.dot(X[i])<=0:
w += y[i]*X[i]
num_errors +=1
print 'Epoch', epoch, ':', num_errors, 'errors', w
if epoch%1==0:
plot_labeled_data(X, y, w)
if num_errors==0:
break
return w
w = perceptron_train(trainX, trainy, 20)
meansq_acc, acc = score(testX, testy, w)
print 'test accuracy: mean square=', meansq_acc, 'raw=', acc
Now learn the hyperplane with logistic regression (stochastic gradient descent, eta=0.8):
In [5]:
w = logreg_train(trainX, trainy, 20)
meansq_acc, acc = score(testX, testy, w)
print 'test accuracy: mean square=', meansq_acc, 'raw=', acc
The above experiment shows that logistic regression is better than the perceptron at finding a hyperplane that separates the test points well (where "well" means having a good mean square accuracy).
What happens when the data is not linearly separable?
In [6]:
nonlinX = np.vstack((trainX, [1, -0.5]))
nonliny = np.append(trainy, -1)
plot_labeled_data(nonlinX, nonliny, [0, 0])
In [7]:
w = perceptron_train(nonlinX, nonliny, 20)
meansq_acc, acc = score(testX, testy, w)
print 'test accuracy: mean square=', meansq_acc, 'raw=', acc
In [8]:
w = logreg_train(nonlinX, nonliny, 20)
meansq_acc, acc = score(testX, testy, w)
print 'test accuracy: mean square=', meansq_acc, 'raw=', acc
In [ ]: