From ancient myths and Turing to Transformers.
No single design has replaced the Transformer, but three research directions are being tested seriously - state-space models such as Mamba, which read text in one pass with a fixed-size memory; hybrids that…
5 min read
Models read very long inputs by making attention cheaper to compute, not by changing what attention does.
4 min read
Researchers have found that language models store recognizable concepts, called features, such as a particular landmark, a bug in code or flattery, and that these features connect into circuits that carry out…
4 min read
Language models do not read a long input evenly; research known as "Lost in the Middle" found they use information best when it sits near the start or the end, and noticeably worse when it is buried in the…
4 min read
A mixture-of-experts model splits part of each layer into several smaller sub-networks, called experts, and uses a small router to send each token to only a few of them.
4 min read
Your prompt is cut into tokens, each token becomes a list of numbers called an embedding, and a stack of Transformer layers uses attention to work out how every token relates to the others.
4 min read
ChatGPT learned everything it knows from a fixed collection of text gathered before its training, and it cannot go online to read anything newer.
4 min read
A diffusion model is an AI system that learns to make images by learning to remove noise.
4 min read
A language model is trained in two broad stages. First it reads a huge collection of text, cut into small pieces called tokens, and learns to guess the next token over and over; then it is often fine-tuned on…
4 min read
Reinforcement learning from human feedback (RLHF) is a way of training a language model with people's judgments instead of raw text alone.
5 min read
Think of three nested circles. Artificial intelligence is the widest: any technique that lets a machine do something we would call intelligent.
4 min read
BERT (Bidirectional Encoder Representations from Transformers) is a language model that Google researchers described on 11 October 2018; it learns by filling in hidden words using the context on both sides of…
4 min read
Google Neural Machine Translation (GNMT) is a deep neural network that translates whole sentences at once, announced by Google on 27 September 2016 and switched on that day for all Chinese-to-English…
4 min read
WaveNet is a neural network, introduced by DeepMind on 8 September 2016, that creates audio one tiny sample at a time instead of stitching together recorded snippets.
5 min read
On 23 March 2016 Microsoft put Tay, a chatbot meant to chat like an American teenager and learn from conversations, on Twitter.
5 min read
Between 9 and 15 March 2016, in Seoul, DeepMind's AlphaGo beat Lee Sedol, one of the greatest Go players of the previous decade, by four games to one in a five-game match watched by tens of millions of people.
5 min read
In October 2015, DeepMind's program AlphaGo beat Fan Hui, the reigning European Go champion and a professional player, five games to none in a closed-door match in London, the first time a computer had beaten…
5 min read
A residual network, or ResNet, is a neural network built from blocks that each learn only a small change to their input and then add the original input back through a shortcut.
5 min read
TensorFlow is a free software library for building and training machine learning models, especially neural networks, that Google released as open source on 9 November 2015.
4 min read
On 6 November 2014, Amazon unveiled the Echo, a cylinder-shaped speaker that stays switched on and responds when you say the wake word "Alexa," with its intelligence running in Amazon's cloud.
5 min read
Sequence-to-sequence learning is a way to train one neural network to read a sequence, such as a sentence, and a second network to write a different sequence from it, such as the translation.
5 min read
In a paper first posted on 1 September 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio let a translation network look back at every word of the source sentence and decide, for each word it writes,…
5 min read
Generative adversarial networks, introduced by Ian Goodfellow and colleagues at the Université de Montréal in a paper posted on 10 June 2014, train two neural networks against each other: a generator that…
5 min read
On 7 June 2014, at an event at the Royal Society in London, the chatbot Eugene Goostman, posing as a 13-year-old Ukrainian boy, convinced 10 of 30 judges (33 percent) that it was human in five-minute text…
4 min read
In January 2014 Google agreed to buy DeepMind, a small London artificial intelligence company known for teaching neural networks to play Atari games, as part of a push into machine learning and robotics.
4 min read
On 19 December 2013, the London startup DeepMind posted a paper showing one neural network that learned to play seven Atari 2600 games using only the screen pixels and the score, with no game-specific rules,…
4 min read
Word2vec is a method, described by Tomas Mikolov and colleagues at Google in a paper posted on 16 January 2013, that turns each word into a list of numbers (a vector) learned from the words that surround it…
4 min read
Researchers disagree. Critics such as Emily Bender argue that a model trained only on text learns the form of language but not its meaning, so its fluent output only looks understood; others point to…
4 min read
Self-attention lets each word in a sentence build its meaning from the other words around it.
5 min read
In June 2012, a Google and Stanford team showed that a neural network with about a billion connections, trained for three days on 16,000 computer cores using 10 million unlabeled frames from YouTube videos,…
4 min read
AlexNet was a deep convolutional neural network built by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton at the University of Toronto that won the 2012 ImageNet image-recognition challenge with a top-5…
4 min read
A large language model is a neural network trained on a huge amount of text to predict the next word, with billions of adjustable numbers called parameters.
4 min read
Siri was a voice assistant that Apple announced on 4 October 2011 as the headline feature of the iPhone 4S, letting people set reminders, send messages or check the weather by speaking in ordinary sentences.
4 min read
In matches broadcast on 14, 15 and 16 February 2011, IBM's Watson computer defeated Ken Jennings and Brad Rutter, the two most successful champions in the history of the US quiz show Jeopardy!.
4 min read
DeepMind is an artificial intelligence research company founded in London in 2010 by Demis Hassabis, Shane Legg and Mustafa Suleyman, with the goal of building general-purpose learning systems rather than…
5 min read
ImageNet is a huge database of photographs, each checked by people and sorted into thousands of named categories, that a Princeton team led by Li Fei-Fei presented at the CVPR computer vision conference in…
5 min read
In a paper published in the journal Neural Computation in July 2006, Geoffrey Hinton, Simon Osindero and Yee-Whye Teh showed a fast way to train neural networks with many layers: build them one layer at a…
5 min read
On 8 October 2005, Stanley, a driverless Volkswagen Touareg built by a Stanford-led team, drove itself across about 132 miles of Mojave Desert in just under seven hours and won DARPA's $2 million Grand…
5 min read
On 13 March 2004, the US Defense Advanced Research Projects Agency (DARPA) offered $1 million to any driverless vehicle that could cross about 142 to 150 miles of the Mojave Desert on its own in under ten…
4 min read
The Roomba is a small, disc-shaped vacuum cleaner that iRobot launched in the United States in September 2002 at $199.95, which drives around a room and cleans on its own.
5 min read
ASIMO is a child-sized humanoid robot that Honda presented in Tokyo in 2000, able to walk smoothly on two legs, turn while walking and climb ordinary stairs.
5 min read
LeNet-5 is a seven-layer convolutional neural network for reading handwritten digits, and MNIST is the set of 70,000 digit images used to test it, both described by Yann LeCun, Léon Bottou, Yoshua Bengio and…
5 min read
Long short-term memory (LSTM) is a design for recurrent neural networks, published by Sepp Hochreiter and Jürgen Schmidhuber in the journal Neural Computation in November 1997, that lets a network hold on to…
4 min read
On 11 May 1997 in New York, IBM's Deep Blue won the sixth and final game of a rematch against Garry Kasparov, taking the match 3.5 to 2.5 and becoming the first computer to defeat a reigning world chess…
5 min read
A.L.I.C.E. (Artificial Linguistic Internet Computer Entity) is a chatbot created by Richard Wallace, first brought to life on 23 November 1995, that replies by matching what you type against tens of thousands…
5 min read
A support vector machine sorts examples into two groups by finding the boundary that leaves the widest possible gap between them, a boundary defined by only a few key examples called support vectors.
5 min read
TD-Gammon, built by Gerald Tesauro at IBM Research in the early 1990s, was a neural network that learned backgammon by playing hundreds of thousands of games against itself, adjusting its judgment after every…
5 min read
On 8 November 1991, at the Computer Museum in Boston, ten ordinary members of the public typed conversations with hidden partners, some human and some computer programs, and ranked them from least to most…
5 min read
In 1989, Yann LeCun and six colleagues at AT&T Bell Laboratories trained a neural network with backpropagation to read handwritten ZIP code digits taken from real US mail, feeding it raw pixels instead of…
5 min read
Jabberwacky is a chatbot created by the British programmer Rollo Carpenter, who began the learning project in 1988 and put it on the web in 1997.
4 min read
The second AI winter was a slump in funding, business and confidence in artificial intelligence that ran roughly from 1987 to 1993, after a boom built on expert systems and specialized Lisp computers.
5 min read
A neural network is a large set of simple calculating units, loosely inspired by brain cells, arranged in layers that pass numbers to one another.
4 min read
Backpropagation is a method for training a neural network with several layers: it measures how wrong the output is, then works backward through the network to calculate how much each connection contributed to…
5 min read
The Terminator is James Cameron's 1984 film about a cyborg assassin sent back in time to kill Sarah Connor, whose unborn son will lead humanity's fight against Skynet, a military AI that has turned on its…
5 min read
In ordinary programming, a person writes the rules and the computer follows them.
4 min read
Neuromancer, William Gibson's first novel, published by Ace Books in July 1984, follows Case, a burned-out hacker hired for a job that turns out to be arranged by an artificial intelligence called Wintermute,…
5 min read
Cyc is a project, started by Douglas Lenat in 1984, to write down the everyday common sense that people never bother to say out loud, in a logical language a computer can reason with.
5 min read
A language model is a program that has read a great deal of text and learned to estimate which word is likely to come next after any stretch of words.
4 min read
Blade Runner is Ridley Scott's 1982 film about Rick Deckard, a police "blade runner" in a rain-soaked Los Angeles of November 2019 who must find and kill four escaped replicants, artificial people almost…
5 min read
On January 7, 1954, Georgetown University and IBM showed an IBM 701 computer in New York translating more than sixty Russian sentences into English, using a vocabulary of 250 words and just six grammar rules.
5 min read
A Hopfield network is a web of simple on/off units, all connected to each other, that stores memories as stable patterns and can rebuild a whole memory from a damaged or partial clue.
5 min read
Christopher Strachey, a schoolteacher at Harrow School in England, wrote a program to play draughts (checkers) against a human, starting on the Pilot ACE computer in 1951 and getting it working on the…
5 min read
SNARC, the Stochastic Neural Analog Reinforcement Calculator, was a learning machine built in the summer of 1951 at Harvard by two Princeton graduate students, Marvin Minsky and Dean Edmonds.
5 min read
The Fifth Generation Computer Systems project was a ten-year national program launched by Japan's Ministry of International Trade and Industry in 1982 to build parallel computers that reason with logic, as a…
5 min read
I, Robot is a 1950 book of nine linked short stories by Isaac Asimov, published by Gnome Press in New York on December 2, 1950, in which robots are built to obey the Three Laws of Robotics and the drama comes…
5 min read
In "Programming a Computer for Playing Chess," published in March 1950 after a talk in March 1949, Claude Shannon of Bell Labs explained that a computer could choose chess moves by scoring positions with an…
5 min read
The Chinese room is a thought experiment published in 1980 by Berkeley philosopher John Searle: a man who knows no Chinese follows an English rulebook to answer Chinese questions so well that outsiders think…
5 min read
Between 1948 and 1949, the neurophysiologist William Grey Walter built two small battery-powered robots, Elmer and Elsie, at the Burden Neurological Institute in Bristol, England.
5 min read
XCON, also called R1, was a rule-based expert system written by John McDermott at Carnegie-Mellon University that checked and completed customer orders for Digital Equipment Corporation's VAX computers and…
5 min read
In The Organization of Behavior (1949), the Canadian psychologist Donald Hebb proposed that when one brain cell repeatedly helps to fire another, the link between them grows stronger, and that groups of cells…
5 min read
In "Intelligent Machinery," a 1948 report for Britain's National Physical Laboratory, Alan Turing argued that machines could be made to behave intelligently and that the best route was to educate them,…
5 min read
The first AI winter was a stretch of roughly 1974 to 1980 when government funding and public enthusiasm for artificial intelligence fell sharply in the United States and Britain.
5 min read
Cybernetics is the study of control and communication in animals and machines, built around the idea of feedback: a system measures the gap between what it wants and what it is doing, and uses that gap to…
5 min read
The Lighthill report was a review of artificial intelligence written in 1972 by Cambridge mathematician Sir James Lighthill for Britain's Science Research Council and published in 1973, which concluded that…
5 min read
ENIAC (Electronic Numerical Integrator and Computer) was a room-sized, programmable electronic computer built at the University of Pennsylvania for the U.S. Army and shown to the public on February 14, 1946.
5 min read
The McCulloch-Pitts neuron is a simplified model of a brain cell, published in 1943 by the neurophysiologist Warren McCulloch and the young logician Walter Pitts: it adds up its incoming signals and "fires"…
5 min read
MYCIN was an expert system built at Stanford in the 1970s that advised doctors on serious bacterial infections, first of the blood and later meningitis, using several hundred if-then rules written with…
5 min read
The Three Laws of Robotics are rules Isaac Asimov built into the fictional robots of his stories: do not harm humans, obey humans unless that causes harm, and protect yourself unless that breaks the first two…
4 min read
PARRY was a conversation program written in 1972 by Stanford psychiatrist Kenneth Colby to imitate a person with paranoid schizophrenia, complete with suspicions, fears and a fixed story about bookies and the…
5 min read
In "On Computable Numbers," completed in 1936, Alan Turing described an imaginary machine that reads and writes symbols on a tape, and showed that one "universal" machine could carry out the work of any other…
5 min read
The robot in Metropolis is a gleaming metal "machine-human" built by the inventor Rotwang and then given the face of Maria, a young woman the workers trust, so it can turn them against each other.
5 min read
Prolog, short for the French "programmation en logique," is a programming language created in Marseille in 1972 by Alain Colmerauer and Philippe Roussel, in which you describe facts and rules and the computer…
5 min read
The word "robot" comes from R.U.R. (Rossum's Universal Robots), a play by the Czech writer Karel Čapek, published in 1920 and premiered at the National Theatre in Prague on 25 January 1921.
5 min read
El Ajedrecista ("The Chess Player") was an electromechanical machine completed around 1912 by the Spanish engineer Leonardo Torres Quevedo that could win one chess endgame, king and rook against a lone king,…
5 min read
SHRDLU was a program written by Terry Winograd at the MIT Artificial Intelligence Laboratory between 1968 and 1970 that held typed conversations in ordinary English about a small simulated world of blocks,…
4 min read
"The Machine Stops" is a short story by E. M. Forster, first published in The Oxford and Cambridge Review in November 1909.
5 min read
The frame problem, named by John McCarthy and Patrick Hayes in a 1969 paper, is the difficulty of telling a logic-based AI system what changes when an action happens without also having to tell it, one fact…
5 min read
The word "android" goes back to the 1700s, when "androides" meant human-shaped automatons, but it took its modern sense of an artificial human from the French novel L'Ève future (Tomorrow's Eve), published in…
4 min read
In Samuel Butler's 1872 satirical novel Erewhon, the people of an imaginary country have destroyed most of their machines because a learned book convinced them that machines would evolve consciousness and…
5 min read
Perceptrons, a 1969 book by MIT's Marvin Minsky and Seymour Papert, used careful mathematics to show that simple one-layer neural networks called perceptrons cannot recognize certain basic patterns, such as…
5 min read
"Darwin among the Machines" is a letter by the English writer Samuel Butler, signed "Cellarius" and published in The Press of Christchurch, New Zealand, on 13 June 1863.
4 min read
HAL 9000 is the calm, talking computer that runs the spaceship Discovery One in Stanley Kubrick's film 2001: A Space Odyssey, which premiered in Washington, D.C., on 2 April 1968 and was written by Kubrick…
5 min read
In his 1854 book The Laws of Thought, George Boole wrote reasoning as algebra, using letters for groups of things and a special arithmetic in which the only values are 0 and 1.
5 min read
In 1843 Ada Lovelace published an English translation of an Italian engineer's article on Charles Babbage's Analytical Engine, with seven notes of her own that were about three times longer than the article.
5 min read
Do Androids Dream of Electric Sheep? is a 1968 science fiction novel by Philip K.
5 min read
The Analytical Engine was a general-purpose mechanical computer that Charles Babbage designed from the mid-1830s and first described in 1837.
5 min read
The Difference Engine was a hand-cranked machine that Charles Babbage designed to calculate and print mathematical tables automatically, using nothing but repeated addition.
5 min read
The ALPAC report, published in November 1966 by a committee of the US National Academy of Sciences, concluded that machine translation was slower, poorer and more expensive than human translation and had no…
5 min read
A Transformer is a type of neural network, introduced by Google researchers in the 2017 paper "Attention Is All You Need," that processes a whole sentence at once and uses a mechanism called attention to…
4 min read
Frankenstein; or, The Modern Prometheus, published anonymously in London on 1 January 1818, tells of a young scientist, Victor Frankenstein, who brings a body to life and then abandons the intelligent,…
5 min read
Shakey was a wheeled robot built at Stanford Research Institute between 1966 and 1972, the first mobile robot that could look at its surroundings, reason about them and plan its own sequence of actions to…
5 min read
"The Sandman" is a German short story by E. T. A. Hoffmann, dated 1816, in which a student named Nathanael falls in love with Olimpia, a beautiful young woman who turns out to be a clockwork automaton.
5 min read
The Mechanical Turk was a chess-playing "automaton" that Wolfgang von Kempelen unveiled at Schönbrunn Palace in Vienna in 1770: a life-size figure in Ottoman dress seated at a cabinet, which beat most of the…
5 min read
DENDRAL was a research project begun at Stanford University in 1965 that built a computer program to work out the structure of unknown organic molecules from mass spectrometry data, the way a trained chemist…
5 min read
The Jaquet-Droz automata are three mechanical dolls, the Writer, the Draughtsman and the Musician, built between 1768 and 1774 by the Swiss watchmakers Pierre Jaquet-Droz, his son Henri-Louis and…
5 min read
ELIZA was a 1966 MIT program by Joseph Weizenbaum that held typed conversations by spotting keywords and turning the user's words into a reply, most famously in the role of a psychotherapist.
4 min read
Unimate was a 4,000-pound programmable robot arm that went to work in 1961 on a General Motors line in Ewing Township, New Jersey, lifting hot metal parts out of die-casting machines.
4 min read
The Digesting Duck was a life-size mechanical duck built by the French inventor Jacques de Vaucanson and unveiled in 1739; it flapped its wings, drank, took grain from a visitor's hand and seemed to digest it…
5 min read
The Engine is an imaginary machine in Jonathan Swift's satire Gulliver's Travels, published in London on 28 October 1726: a huge wooden frame covered with words that students shuffle by turning handles,…
5 min read
In the July 1959 issue of the IBM Journal of Research and Development, IBM engineer Arthur Samuel reported a checkers program for the IBM 704 that improved with experience until it played better than he did,…
5 min read
The term "artificial intelligence" was coined by John McCarthy, a young mathematician at Dartmouth College, in a funding proposal dated August 31, 1955 that he wrote with Marvin Minsky, Nathaniel Rochester…
4 min read
In his Dissertation on the Art of Combinations, published in Leipzig in 1666, Gottfried Wilhelm Leibniz proposed that every idea is a combination of a small set of simple ideas, the way words are built from…
5 min read
John McCarthy and Marvin Minsky started the Artificial Intelligence Project at MIT in the fall of 1958, a date often given as 1959, with backing from the Institute's Research Laboratory of Electronics and no…
5 min read
In Leviathan, published in London in 1651, the English philosopher Thomas Hobbes argued that reasoning is a kind of arithmetic, nothing more than adding and subtracting ideas expressed in words.
5 min read
The Turing test is a typed conversation game proposed by Alan Turing in 1950: if a judge chatting with a hidden machine and a hidden person cannot reliably tell which is which, the machine is said to play the…
4 min read
No. In his Discourse on the Method, published in 1637, René Descartes argued that a machine could copy the body and even utter words, but could never use language flexibly to answer whatever is said to it,…
4 min read
Lisp, short for "list processor," is a programming language John McCarthy designed in 1958 so that computers could manipulate symbols, such as words and logical statements, instead of only numbers.
5 min read
The golem of Prague is a figure of Jewish folklore: a man made of clay that, according to the legend, Rabbi Judah Loew of Prague brought to life with a sacred name to serve and protect his community in the…
5 min read
The idea of a thinking machine is far older than the computer: it runs from Greek myths of crafted servants, through medieval tales and clockwork automatons, to philosophers who suspected reasoning was a kind…
4 min read
Yes. Leonardo da Vinci's notebooks contain fragmentary designs for an armored mechanical figure, usually dated to about 1495, that could sit up, move its arms and open its visor using pulleys, cables and…
4 min read
The perceptron was a model of a brain-like learning machine that Frank Rosenblatt developed at the Cornell Aeronautical Laboratory, first simulated in 1957 and published in Psychological Review in 1958.
5 min read
The Ars Magna, or "Great Art," was a method devised by the Majorcan philosopher Ramon Llull for producing statements by combining a small set of basic concepts, using letters, tables and rotating paper…
4 min read
The General Problem Solver (GPS) was a program by Allen Newell, Cliff Shaw and Herbert Simon, begun around 1957 and first fully described in a report dated December 1958, that tried to solve many kinds of…
5 min read
Al-Jazari was a 12th-century engineer who served the Artuqid rulers of eastern Anatolia and, in 1206, completed The Book of Knowledge of Ingenious Mechanical Devices.
4 min read
Hero of Alexandria was a Greek-speaking mathematician and engineer who worked in Alexandria, Egypt, most likely in the 1st century CE.
4 min read
From about June 18 to August 17, 1956, a shifting group of roughly twenty researchers met at Dartmouth College to discuss how machines might learn and reason, with only three of them staying the whole summer.
5 min read
Pygmalion is a sculptor from Cyprus who carves a woman out of ivory, falls in love with her, and prays to Venus, who turns the statue into a living woman.
4 min read
The Logic Theorist was a computer program built in 1955–56 by Allen Newell, Herbert Simon and Cliff Shaw that found proofs for theorems in Whitehead and Russell's Principia Mathematica.
5 min read
Talos was a giant man of bronze in Greek myth, set on the island of Crete to keep strangers away by hurling rocks at their ships.
5 min read
Sign up for the weekly newsletter and receive the free guide: what changed in AI this week, one lesson, one prompt to try. Unsubscribe in one click.