What are support vector machines, and why did they matter so much?

Imagine a table covered with red and blue marbles, mostly red on one side and blue on the other, and a ruler you must lay down to split them. Many positions would work. A support vector machine picks the one that stays as far as possible from the nearest marbles on both sides, so a new marble that lands near the line is still likely to be sorted correctly.

That simple picture sits behind one of the most influential methods in machine learning. Its best-known form was published in 1995 by Corinna Cortes and Vladimir Vapnik of AT&T Bell Laboratories in Holmdel, New Jersey, in a paper called "Support-Vector Networks" in the journal Machine Learning. The journal received it in May 1993.

Where the idea came from

Vapnik had been working on the theory of learning since the 1960s. With Alexey Chervonenkis, he developed what is now called VC theory, which asks a practical question: when a model fits its training examples, how confident can we be that it will work on new ones? One answer is the "optimal hyperplane," the flat boundary with the widest margin between two classes. A hyperplane is just a line in two dimensions, a flat sheet in three, and the same idea in more dimensions.

Two pieces were still missing by the early 1990s. First, real data often cannot be split by a straight line at all. In 1992, Bernhard Boser, Isabelle Guyon and Vapnik showed a shortcut, now called the kernel trick, for getting curved boundaries cheaply. Second, real data is noisy: a few marbles always sit on the wrong side. The 1995 paper solved that.

How a support vector machine works

The paper's method rests on three ideas.

Widest margin. Among all the boundaries that separate the classes, choose the one with the largest gap to the nearest examples. Those nearest examples are the support vectors. Only they decide where the boundary goes; the rest of the data could move around, and the boundary would not change. The paper links a small share of support vectors to good performance on new data.

Soft margin. Cortes and Vapnik's contribution was to let the machine accept some training errors when perfect separation is impossible. A setting controls the trade-off between a wide margin and few mistakes. This "soft margin" version is the one used in software packages to this day.

Kernels. To draw curved boundaries, you can imagine lifting the data into a space with many more dimensions, where a flat boundary does the job. Computing in that space directly would be hopeless; the paper notes it could mean a billion dimensions. The kernel trick avoids it by comparing pairs of examples in the original space and transforming the result, which gives the same answer as working in the huge space.

Here is the marble picture as a worked example:

Situation	What the machine does
Red and blue split cleanly	Lays the boundary in the middle of the widest gap
A couple of marbles on the wrong side	Accepts those errors (soft margin) rather than twist the boundary
Red marbles form a ring around a blue cluster	Uses a kernel, which acts like a curved boundary drawn around the blue cluster
A new marble arrives	Checks which side of the boundary it falls on

Try it: Scatter ten coins and ten buttons on a table so the groups overlap slightly. Lay a pencil to separate them, then slide and turn it to make the gap to the closest coin and closest button as wide as possible. Count how many objects actually touch the edges of your gap: those are your support vectors.

The digit tests

The paper tested the method on handwritten digits, a hot problem at Bell Labs. With a US Postal Service set of 7,300 training digits and 2,000 test digits, support vector machines using polynomial kernels made about 4.2 to 4.7 percent errors. That beat LeNet1, a neural network designed especially for the task, at 5.1 percent, and human performance on the set was 2.5 percent. On a larger benchmark of 60,000 training digits built from NIST data, a single type of support vector machine reached 1.1 percent errors, without any special design for images.

Ten classifiers were trained for the ten digits, each separating one digit from all the others; the highest score won.

Why it mattered

Support vector machines arrived with three strengths that neural networks of the 1990s lacked. They came with a theory explaining why they should work on new data. Training them is a convex problem, meaning there is a single best answer and no risk of getting stuck in a poor one. And they worked well with modest amounts of data and little tuning.

They soon became a standard choice for classification: sorting text into categories, recognizing images and handwriting, and classifying proteins in biology. Many teams reached for an SVM before trying anything else. Deep neural networks, fed with much larger datasets and faster hardware, have since overtaken them on images, speech and language.

Are support vector machines still worth learning now that deep learning dominates?

Yes, for two reasons. First, they remain practical. For small or medium datasets, especially with clear numerical features, an SVM often trains in seconds, needs little tuning and performs as well as far bigger models. The common machine learning libraries include them, and they make a strong baseline to compare against before trying a neural network.

Second, the ideas travel. The margin is a clean way to think about why a model generalizes. Kernels teach you to think of data in terms of similarity between examples, which shows up again in many other methods. And the support vector idea, that a few critical examples define the decision, is a useful lens for understanding any classifier.

If you are starting out, try an SVM on a small dataset first. Seeing a simple, well-understood model do surprisingly well is a good way to understand what the larger models are adding.