Linear Methods for Classification

Linear Discriminant Analysis Classifier (LDAC)

See [Hastie09], page 106.

class mlpy.LDAC

Linear Discriminant Analysis Classifier.

Initialization.

bias()

Returns the bias. For multiclass classification this method returns a 1d numpy array where b[i] contains the coefficients of label i. For binary classification an float (b_1 - b_0) is returned.

labels()

Outputs the name of labels.

learn(x, y)

Learning method.

Parameters :
x : 2d array_like object

training data (N, P)

y : 1d array_like object integer

target values (N)

pred(t)

Does classification on test vector(s) t.

Parameters :
t : 1d (one sample) or 2d array_like object

test data ([M,] P)

Returns :
p : integer or 1d numpy array

predicted class(es)

w()

Returns the coefficients. For multiclass classification this method returns a 2d numpy array where w[i] contains the coefficients of label i. For binary classification an 1d numpy array (w_1 - w_0) is returned.

Examples

Binary classification:

>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import mlpy
>>> np.random.seed(0)
>>> mean1, cov1, n1 = [1, 5], [[1,1],[1,2]], 200  # 200 samples of class 1
>>> x1 = np.random.multivariate_normal(mean1, cov1, n1)
>>> y1 = np.ones(n1, dtype=np.int)
>>> mean2, cov2, n2 = [2.5, 2.5], [[1,0],[0,1]], 300 # 300 samples of class -1
>>> x2 = np.random.multivariate_normal(mean2, cov2, n2)
>>> y2 = -np.ones(n2, dtype=np.int)
>>> x = np.concatenate((x1, x2), axis=0) # concatenate the samples
>>> y = np.concatenate((y1, y2))
>>> ldac = mlpy.LDAC()
>>> ldac.learn(x, y)
>>> w = ldac.w()
>>> w
array([ 2.5948979  -2.58553746])
>>> b = ldac.bias()
>>> b
5.63727441841
>>> xx = np.arange(np.min(x[:,0]), np.max(x[:,0]), 0.01)
>>> yy = - (w[0] * xx + b) / w[1] # separator line
>>> fig = plt.figure(1) # plot
>>> plot1 = plt.plot(x1[:, 0], x1[:, 1], 'ob', x2[:, 0], x2[:, 1], 'or')
>>> plot2 = plt.plot(xx, yy, '--k')
>>> plt.show()
_images/lda_binary.png
>>> test = [[0, 2], [4, 2]] # test points
>>> ldac.pred(test)
array([-1, -1])
>>> ldac.labels()
array([-1,  1])

Multiclass classification:

>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import mlpy
>>> np.random.seed(0)
>>> mean1, cov1, n1 = [1, 25], [[1,1],[1,2]], 200  # 200 samples of class 0
>>> x1 = np.random.multivariate_normal(mean1, cov1, n1)
>>> y1 = np.zeros(n1, dtype=np.int)
>>> mean2, cov2, n2 = [2.5, 22.5], [[1,0],[0,1]], 300 # 300 samples of class 1
>>> x2 = np.random.multivariate_normal(mean2, cov2, n2)
>>> y2 = np.ones(n2, dtype=np.int)
>>> mean3, cov3, n3 = [5, 28], [[0.5,0],[0,0.5]], 200 # 200 samples of class 2
>>> x3 = np.random.multivariate_normal(mean3, cov3, n3)
>>> y3 = 2 * np.ones(n3, dtype=np.int)
>>> x = np.concatenate((x1, x2, x3), axis=0) # concatenate the samples
>>> y = np.concatenate((y1, y2, y3))
>>> ldac = mlpy.LDAC()
>>> ldac.learn(x, y)
>>> w = ldac.w()
>>> w # w[i]: coefficients label ldac.labels()[i]
array([[-0.30949939  4.53041257]
       [ 2.52002288  1.50501818]
       [ 4.2499381   5.90569921]])
>>> b = ldac.bias()
>>> b # b[i]: bias for label ldac.labels()[i]
array([-12.65129158  -5.7628039  -35.63605709])
>>> xx = np.arange(np.min(x[:,0]), np.max(x[:,0]), 0.01)
>>> yy1 = (xx* (w[1][0]-w[0][0]) + b[1] - b[0]) / (w[0][1]-w[1][1])
>>> yy2 = (xx* (w[2][0]-w[0][0]) + b[2] - b[0]) / (w[0][1]-w[2][1])
>>> yy3 = (xx* (w[2][0]-w[1][0]) + b[2] - b[1]) / (w[1][1]-w[2][1])
>>> fig = plt.figure(1) # plot
>>> plot1 = plt.plot(x1[:, 0], x1[:, 1], 'ob', x2[:, 0], x2[:, 1], 'or', x3[:, 0], x3[:, 1], 'og')
>>> plot2 = plt.plot(xx, yy1, '--k')
>>> plot3 = plt.plot(xx, yy2, '--k')
>>> plot4 = plt.plot(xx, yy3, '--k')
>>> plt.show()
_images/lda_multi.png
>>> test = [[6,7], [4, 2]] # test points
>>> ldac.pred(test)
array([2, 1])
>>> ldac.labels()
array([0, 1, 2])

Basic Perceptron

class mlpy.Perceptron(alpha=0.10000000000000001, thr=0.0, maxiters=1000)

Perceptron binary classifier.

The algorithm stops when the iteration error is less or equal than thr, or a predetermined number of iterations (maxiters) have been completed.

Parameters :
alpha : float, in range (0.0, 1]

learning rate

thr : float, in range [0.0, 1.0]

iteration error (e.g. thr=0.13 for error=13%)

maxiters : integer (>0)

maximum number of iterations

bias()

Returns the bias.

err()

Returns the iteration error

iters()

Returns the number of iterations

labels()

Outputs the name of labels.

learn(x, y)

Learning method.

Parameters :
x : 2d array_like object

training data (N, P)

y : 1d array_like object integer (only two classes)

target values (N)

pred(t)

Prediction method.

Parameters :
t : 1d or 2d array_like object

testing data ([M,], P)

w()

Returns the coefficients.

Examples

>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import mlpy
>>> np.random.seed(0)
>>> mean1, cov1, n1 = [1, 5], [[1,1],[1,2]], 200  # 200 samples of class 1
>>> x1 = np.random.multivariate_normal(mean1, cov1, n1)
>>> y1 = np.ones(n1, dtype=np.int)
>>> mean2, cov2, n2 = [2.5, 2.5], [[1,0],[0,1]], 300 # 300 samples of class -1
>>> x2 = np.random.multivariate_normal(mean2, cov2, n2)
>>> y2 = -np.ones(n2, dtype=np.int)
>>> x = np.concatenate((x1, x2), axis=0) # concatenate the samples
>>> y = np.concatenate((y1, y2))
>>> p = mlpy.Perceptron(alpha=0.1, thr=0.05, maxiters=100) # basic perceptron
>>> p.learn(x, y)
>>> w = p.w()
>>> w
array([-69.00185254,  46.49202132])
>>> b = p.bias()
>>> b
-59.600000000000001
>>> p.err()
0.050000000000000003
>>> p.iters()
46
>>> xx = np.arange(np.min(x[:,0]), np.max(x[:,0]), 0.01)
>>> yy = - (w[0] * xx + b) / w[1] # separator line
>>> fig = plt.figure(1) # plot
>>> plot1 = plt.plot(x1[:, 0], x1[:, 1], 'ob', x2[:, 0], x2[:, 1], 'or')
>>> plot2 = plt.plot(xx, yy, '--k')
>>> plt.show()
_images/perceptron.png
>>> test = [[0, 2], [4, 2]] # test points
>>> p.pred(test)
array([ 1, -1])
>>> p.labels()
array([-1,  1])

Elastic Net Classifier

See [Hastie09], Chapter 18, page 661.

class mlpy.ElasticNetC(lmb, eps, supp=True, tol=0.01)

Elastic Net Regularization via Iterative Soft Thresholding for classification.

See the ElasticNet class documentation.

Initialization.

Parameters :
lmb : float

regularization parameter controlling overfitting. lmb can be tuned via cross validation.

eps : float

correlation parameter preserving correlation among variables against sparsity. The solutions obtained for different values of the correlation parameter have the same prediction properties but different feature representation.

supp : bool

if True, the algorithm stops when the support of beta reached convergence. If False, the algorithm stops when the coefficients reached convergence, that is when the beta_{l}(i) - beta_{l+1}(i) > tol * beta_{l}(i) for all i.

tol : double

tolerance for convergence

bias()

Returns the bias.

labels()

Outputs the name of labels.

learn(x, y)

Compute the classification coefficients.

Parameters :
x : 2d array_like object (N x P)

matrix

y : 1d array_like object integer (N)

class labels

pred(t)

Compute the predicted labels.

Parameters :
t : 1d or 2d array_like object ([M,] P)

test data

Returns :
p : integer or 1d numpy array

predicted labels

w()

Returns the coefficients.

Example:

>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import mlpy
>>> np.random.seed(0)
>>> mean1, cov1, n1 = [1, 5], [[1,1],[1,2]], 200  # 200 samples of class 1
>>> x1 = np.random.multivariate_normal(mean1, cov1, n1)
>>> y1 = np.ones(n1, dtype=np.int)
>>> mean2, cov2, n2 = [2.5, 2.5], [[1,0],[0,1]], 300 # 300 samples of class -1
>>> x2 = np.random.multivariate_normal(mean2, cov2, n2)
>>> y2 = -np.ones(n2, dtype=np.int)
>>> x = np.concatenate((x1, x2), axis=0) # concatenate the samples
>>> y = np.concatenate((y1, y2))
>>> en = mlpy.ElasticNetC(lmb=0.01, eps=0.001)
>>> en.learn(x, y)
>>> w = en.w()
>>> w
array([-0.27733363,  0.30115026])
>>> b = en.bias()
>>> b
-0.73445916200332606
>>> en.iters()
1000
>>> xx = np.arange(np.min(x[:,0]), np.max(x[:,0]), 0.01)
>>> yy = - (w[0] * xx + b) / w[1] # separator line
>>> fig = plt.figure(1) # plot
>>> plot1 = plt.plot(x1[:, 0], x1[:, 1], 'ob', x2[:, 0], x2[:, 1], 'or')
>>> plot2 = plt.plot(xx, yy, '--k')
>>> plt.show()
_images/elasticnetc.png
>>> test = [[1, 4], [2, 2]] # test points
>>> en.pred(test)
array([ 1., -1.])

Support Vector Classification

See Large Linear Classification from [LIBLINEAR]

Diagonal Linear Discriminant Analysis (DLDA)

See [Hastie09], page 651.

class mlpy.DLDA(delta)

Diagonal Linear Discriminant Analysis classifier. The algorithm uses the procedure called Nearest Shrunken Centroids (NSC).

Initialization.

Parameters :
delta : float

regularization parameter

dprime()

Return the dprime d’_kj (C, P), where C is the number of classes.

labels()

Outputs the name of labels.

learn(x, y)

Learning method.

Parameters :
x : 2d array_like object

training data (N, P)

y : 1d array_like object integer

target values (N)

pred(t)

Does classification on test vector(s) t.

Parameters :
t : 1d (one sample) or 2d array_like object

test data ([M,] P)

Returns :
p : int or 1d numpy array

the predicted class(es) for t is returned.

prob(t)

For each sample returns C (number of classes) probability estimates.

sel()

Returns the most important features (the features that have a nonzero dprime for at least one of the classes).

Example:

>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import mlpy
>>> np.random.seed(0)
>>> mean1, cov1, n1 = [1, 5], [[1,1],[1,2]], 200  # 200 samples of class 0
>>> x1 = np.random.multivariate_normal(mean1, cov1, n1)
>>> y1 = np.zeros(n1, dtype=np.int)
>>> mean2, cov2, n2 = [2.5, 2.5], [[1,0],[0,1]], 300 # 300 samples of class 1
>>> x2 = np.random.multivariate_normal(mean2, cov2, n2)
>>> y2 = np.ones(n2, dtype=np.int)
>>> mean3, cov3, n3 = [5, 8], [[0.5,0],[0,0.5]], 200 # 200 samples of class 2
>>> x3 = np.random.multivariate_normal(mean3, cov3, n3)
>>> y3 = 2 * np.ones(n3, dtype=np.int)
>>> x = np.concatenate((x1, x2, x3), axis=0) # concatenate the samples
>>> y = np.concatenate((y1, y2, y3))
>>> da = mlpy.DLDA(delta=0.1)
>>> da.learn(x, y)
>>> xmin, xmax = x[:,0].min()-1, x[:,0].max()+1
>>> ymin, ymax = x[:,1].min()-1, x[:,1].max()+1
>>> xx, yy = np.meshgrid(np.arange(xmin, xmax, 0.1), np.arange(ymin, ymax, 0.1))
>>> xnew = np.c_[xx.ravel(), yy.ravel()]
>>> ynew = da.pred(xnew).reshape(xx.shape)
>>> fig = plt.figure(1)
>>> cmap = plt.set_cmap(plt.cm.Paired)
>>> plot1 = plt.pcolormesh(xx, yy, ynew)
>>> plot2 = plt.scatter(x[:,0], x[:,1], c=y)
>>> plt.show()
_images/dlda.png

Golub Classifier

class mlpy.Golub

Golub binary classifier described in [Golub99].

Decision function is D(x) = w (x-mu), where w is defined as w_i = (mu_i(+) - mu_i(-)) / (std_i(+) + std_i(-)) and mu id defined as (mu(+) + mu(-)) / 2.

[Golub99]T R Golub et al. Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science, 1999.

Initialization.

labels()

Outputs the name of labels.

learn(x, y)

Learning method.

Parameters :
x : 2d array_like object

training data (N, P)

y : 1d array_like object integer (only two classes)

target values (N)

pred(t)

Prediction method.

Parameters :
t : 1d or 2d array_like object

testing data ([M,], P)

w()

Returns the coefficients.

[Hastie09](1, 2, 3) T Hastie, R Tibshirani, J Friedman. The Elements of Statistical Learning. Second Edition.