ClassificationSVM
Support vector machine (SVM) for one-class and binary classification
Description
ClassificationSVM is a support vector machine (SVM) classifier for one-class and two-class learning. Trained ClassificationSVM classifiers store training data, parameter values, prior probabilities, support vectors, and algorithmic implementation information. Use these classifiers to perform tasks such as fitting a score-to-posterior-probability transformation function (see fitPosterior) and predicting labels for new data (see predict).
Creation
Create a ClassificationSVM object by using fitcsvm.
Properties
Object Functions
compact | Reduce size of support vector machine (SVM) classifier |
compareHoldout | Compare accuracies of two classification models using new data |
crossval | Cross-validate support vector machine (SVM) classifier |
discardSupportVectors | Discard support vectors for linear support vector machine (SVM) classifier |
edge | Find classification edge for support vector machine (SVM) classifier |
fitPosterior | Fit posterior probabilities for support vector machine (SVM) classifier |
incrementalLearner | Convert binary classification support vector machine (SVM) model to incremental learner |
loss | Find classification error for support vector machine (SVM) classifier |
margin | Find classification margins for support vector machine (SVM) classifier |
partialDependence | Compute partial dependence |
plotPartialDependence | Create partial dependence plot (PDP) and individual conditional expectation (ICE) plots |
predict | Classify observations using support vector machine (SVM) classifier |
resubEdge | Find classification edge for support vector machine (SVM) classifier by resubstitution |
resubLoss | Find classification loss for support vector machine (SVM) classifier by resubstitution |
resubMargin | Find classification margins for support vector machine (SVM) classifier by resubstitution |
resubPredict | Classify observations in support vector machine (SVM) classifier |
resume | Resume training support vector machine (SVM) classifier |
Examples
Train SVM Classifier
Load Fisher's iris data set. Remove the sepal lengths and widths and all observed setosa irises.
load fisheriris inds = ~strcmp(species,'setosa'); X = meas(inds,3:4); y = species(inds);
Train an SVM classifier using the processed data set.
SVMModel = fitcsvm(X,y)
SVMModel =
ClassificationSVM
ResponseName: 'Y'
CategoricalPredictors: []
ClassNames: {'versicolor' 'virginica'}
ScoreTransform: 'none'
NumObservations: 100
Alpha: [24x1 double]
Bias: -14.4149
KernelParameters: [1x1 struct]
BoxConstraints: [100x1 double]
ConvergenceInfo: [1x1 struct]
IsSupportVector: [100x1 logical]
Solver: 'SMO'
Properties, Methods
SVMModel is a trained ClassificationSVM classifier. Display the properties of SVMModel. For example, to determine the class order, use dot notation.
classOrder = SVMModel.ClassNames
classOrder = 2x1 cell
{'versicolor'}
{'virginica' }
The first class ('versicolor') is the negative class, and the second ('virginica') is the positive class. You can change the class order during training by using the 'ClassNames' name-value pair argument.
Plot a scatter diagram of the data and circle the support vectors.
sv = SVMModel.SupportVectors; figure gscatter(X(:,1),X(:,2),y) hold on plot(sv(:,1),sv(:,2),'ko','MarkerSize',10) legend('versicolor','virginica','Support Vector') hold off
The support vectors are observations that occur on or beyond their estimated class boundaries.
You can adjust the boundaries (and, therefore, the number of support vectors) by setting a box constraint during training using the 'BoxConstraint' name-value pair argument.
Train and Cross-Validate SVM Classifier
Load the ionosphere data set.
load ionosphereTrain and cross-validate an SVM classifier. Standardize the predictor data and specify the order of the classes.
rng(1); % For reproducibility CVSVMModel = fitcsvm(X,Y,'Standardize',true,... 'ClassNames',{'b','g'},'CrossVal','on')
CVSVMModel =
ClassificationPartitionedModel
CrossValidatedModel: 'SVM'
PredictorNames: {1x34 cell}
ResponseName: 'Y'
NumObservations: 351
KFold: 10
Partition: [1x1 cvpartition]
ClassNames: {'b' 'g'}
ScoreTransform: 'none'
Properties, Methods
CVSVMModel is a ClassificationPartitionedModel cross-validated SVM classifier. By default, the software implements 10-fold cross-validation.
Alternatively, you can cross-validate a trained ClassificationSVM classifier by passing it to crossval.
Inspect one of the trained folds using dot notation.
CVSVMModel.Trained{1}ans =
CompactClassificationSVM
ResponseName: 'Y'
CategoricalPredictors: []
ClassNames: {'b' 'g'}
ScoreTransform: 'none'
Alpha: [78x1 double]
Bias: -0.2209
KernelParameters: [1x1 struct]
Mu: [1x34 double]
Sigma: [1x34 double]
SupportVectors: [78x34 double]
SupportVectorLabels: [78x1 double]
Properties, Methods
Each fold is a CompactClassificationSVM classifier trained on 90% of the data.
Estimate the generalization error.
genError = kfoldLoss(CVSVMModel)
genError = 0.1168
On average, the generalization error is approximately 12%.
More About
Algorithms
For the mathematical formulation of the SVM binary classification algorithm, see Support Vector Machines for Binary Classification and Understanding Support Vector Machines.
NaN,<undefined>, empty character vector (''), empty string (""), and<missing>values indicate missing values.fitcsvmremoves entire rows of data corresponding to a missing response. When computing total weights (see the next bullets),fitcsvmignores any weight corresponding to an observation with at least one missing predictor. This action can lead to unbalanced prior probabilities in balanced-class problems. Consequently, observation box constraints might not equalBoxConstraint.fitcsvmremoves observations that have zero weight or prior probability.For two-class learning, if you specify the cost matrix (see
Cost), then the software updates the class prior probabilities p (seePrior) to pc by incorporating the penalties described in .Specifically,
fitcsvmcompletes these steps:Compute
Normalize pc* so that the updated prior probabilities sum to 1.
K is the number of classes.
Reset the cost matrix to the default
Remove observations from the training data corresponding to classes with zero prior probability.
For two-class learning,
fitcsvmnormalizes all observation weights (seeWeights) to sum to 1. The function then renormalizes the normalized weights to sum up to the updated prior probability of the class to which the observation belongs. That is, the total weight for observation j in class k iswj is the normalized weight for observation j; pc,k is the updated prior probability of class k (see previous bullet).
For two-class learning,
fitcsvmassigns a box constraint to each observation in the training data. The formula for the box constraint of observation j isn is the training sample size, C0 is the initial box constraint (see the
'BoxConstraint'name-value pair argument), and is the total weight of observation j (see previous bullet).If you set
'Standardize',trueand the'Cost','Prior', or'Weights'name-value pair argument, thenfitcsvmstandardizes the predictors using their corresponding weighted means and weighted standard deviations. That is,fitcsvmstandardizes predictor j (xj) usingxjk is observation k (row) of predictor j (column).
Assume that
pis the proportion of outliers that you expect in the training data, and that you set'OutlierFraction',p.For one-class learning, the software trains the bias term such that 100
p% of the observations in the training data have negative scores.The software implements robust learning for two-class learning. In other words, the software attempts to remove 100
p% of the observations when the optimization algorithm converges. The removed observations correspond to gradients that are large in magnitude.
If your predictor data contains categorical variables, then the software generally uses full dummy encoding for these variables. The software creates one dummy variable for each level of each categorical variable.
The
PredictorNamesproperty stores one element for each of the original predictor variable names. For example, assume that there are three predictors, one of which is a categorical variable with three levels. ThenPredictorNamesis a 1-by-3 cell array of character vectors containing the original names of the predictor variables.The
ExpandedPredictorNamesproperty stores one element for each of the predictor variables, including the dummy variables. For example, assume that there are three predictors, one of which is a categorical variable with three levels. ThenExpandedPredictorNamesis a 1-by-5 cell array of character vectors containing the names of the predictor variables and the new dummy variables.Similarly, the
Betaproperty stores one beta coefficient for each predictor, including the dummy variables.The
SupportVectorsproperty stores the predictor values for the support vectors, including the dummy variables. For example, assume that there are m support vectors and three predictors, one of which is a categorical variable with three levels. ThenSupportVectorsis an n-by-5 matrix.The
Xproperty stores the training data as originally input and does not include the dummy variables. When the input is a table,Xcontains only the columns used as predictors.
For predictors specified in a table, if any of the variables contain ordered (ordinal) categories, the software uses ordinal encoding for these variables.
For a variable with k ordered levels, the software creates k – 1 dummy variables. The jth dummy variable is –1 for levels up to j, and +1 for levels j + 1 through k.
The names of the dummy variables stored in the
ExpandedPredictorNamesproperty indicate the first level with the value +1. The software stores k – 1 additional predictor names for the dummy variables, including the names of levels 2, 3, ..., k.
All solvers implement L1 soft-margin minimization.
For one-class learning, the software estimates the Lagrange multipliers, α1,...,αn, such that
References
[1] Hastie, T., R. Tibshirani, and J. Friedman. The Elements of Statistical Learning, Second Edition. NY: Springer, 2008.
[2] Scholkopf, B., J. C. Platt, J. C. Shawe-Taylor, A. J. Smola, and R. C. Williamson. “Estimating the Support of a High-Dimensional Distribution.” Neural Comput., Vol. 13, Number 7, 2001, pp. 1443–1471.
[3] Christianini, N., and J. C. Shawe-Taylor. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge, UK: Cambridge University Press, 2000.
[4] Scholkopf, B., and A. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond, Adaptive Computation and Machine Learning. Cambridge, MA: The MIT Press, 2002.