Support Vector Machines (SVMs) are fundamentally binary classifiers; they classify data into one of two categories. However, many real-world problems involve multiple classes, necessitating techniques that extend SVMs to handle multiclass classification. This chapter explores methods for adapting SVMs to multiclass tasks and provides a practical example using Python’s Scikit-learn library.
To handle multiclass classification tasks, SVMs can be extended using one of two primary strategies:
In the One-vs-Rest approach, a single SVM model is trained per class, with the samples of that class as positive samples and all other samples as negatives. This method involves training as many classifiers as there are classes. The classifier with the highest decision function score is picked to label a new sample.
In the One-vs-One approach, an SVM model is trained for every pair of classes. If there are ( N ) classes, ( \frac{N(N-1)}{2} ) classifiers are trained. A voting strategy is then used for classification, where every classifier assigns the new sample to one of the two classes it was trained on, and the class that gets the most votes determines the sample’s label.
Both methods are supported in Scikit-learn through the SVC class automatically, which handles multiclass classification using OvO by default.
The following example demonstrates how to implement multiclass classification using the Iris dataset, which contains three classes of flowers.
from sklearn import datasets
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, accuracy_score
# Load the Iris dataset
iris = datasets.load_iris()
X = iris.data
y = iris.target
# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Standardize the features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Train the SVM model on the scaled data
svm_model = SVC(kernel='rbf', C=1.0, gamma='auto')
svm_model.fit(X_train_scaled, y_train)
# Make predictions and evaluate the model
y_pred = svm_model.predict(X_test_scaled)
print("Accuracy on test set: ", accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))
Evaluation metrics for multiclass classification can include accuracy, precision, recall, and the F1-score, calculated for each class and possibly averaged (macro or weighted) to get an overall score. Scikit-learn’s classification_report function provides a detailed analysis of the model’s performance across all classes.
Adapting SVMs for multiclass classification involves using strategies like One-vs-Rest or One-vs-One to handle multiple classes seamlessly. By leveraging these strategies, SVMs can be effectively used for a wide range of applications that require categorization into multiple categories.
This chapter has outlined the methods for extending binary SVMs to multiclass classification tasks, illustrated with a practical implementation in Scikit-learn. Understanding and applying these techniques allows SVMs to be utilized in diverse scenarios beyond binary classification, enhancing their utility in complex, real-world datasets.