# What is Google's PaLM, and why does a 540-billion-parameter model matter?

Date: 2022-04-05 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-palm/

**Answer:** PaLM, short for Pathways Language Model, is a new Google language model with 540 billion parameters, trained on 6,144 of Google's TPU chips with a new system called Pathways. Announced on 4 April 2022, it sets new records on hundreds of language, reasoning and coding tests, and some abilities, such as explaining jokes and solving multi-step math problems, appear only at its largest size.

Yesterday, Google Research introduced **PaLM**, the Pathways Language Model. With 540 billion parameters, it is roughly three times the size of OpenAI's GPT-3 and among the largest language models anyone has described. Google says it beats earlier large models on most of the tests it tried, often by a wide margin.

The size is only part of the story. PaLM is also the first big test of **Pathways**, a new Google system for spreading one training job across thousands of chips. And the research team reports something intriguing: some skills do not grow smoothly with size but seem to switch on once the model gets big enough.

## What Google announced, and who built it

On Monday, 4 April 2022, two Google Research software engineers, Sharan Narang and Aakanksha Chowdhery, described PaLM on the Google AI Blog. The full technical paper, "PaLM: Scaling Language Modeling with Pathways," appeared on arXiv today, 5 April. It lists Chowdhery first among 67 authors, with Noam Shazeer, Jacob Devlin and Jason Wei among the contributors and Jeff Dean, Slav Petrov and Douglas Eck among the project's advisors.

A **language model** is a program trained to predict the next word (more precisely, the next **token**, a word or piece of a word) in a text. Its **parameters** are the internal numbers adjusted during training; more parameters give the model more room to store patterns. Google trained three versions so it could compare them:

| Model | Parameters | Use in the paper |
|---|---|---|
| PaLM 8B | 8 billion | Small baseline |
| PaLM 62B | 62 billion | Middle size |
| PaLM 540B | 540 billion | The headline model |

All three read the same 780 billion tokens of text exactly once.

## How it was trained

PaLM is a **dense, decoder-only Transformer**. "Transformer" is the attention-based network design Google introduced in 2017; "decoder-only" means it simply continues text, like GPT-3; "dense" means every parameter takes part in every prediction, unlike some recent models that switch on only parts of themselves.

The training data, according to the paper, breaks down this way: half is multilingual social media conversation, 27% is filtered web pages, 13% English books, 5% code from GitHub, 4% Wikipedia and 1% English news. More than 100 languages appear, though only about 22% of the data is not English. Google also built a vocabulary that keeps every space (important for code) and splits numbers into single digits, so "123" becomes "1", "2", "3".

The hardware is where Pathways comes in. Google's **TPUs** (Tensor Processing Units) are its own AI chips, grouped into large clusters called **Pods**. PaLM trained on two TPU v4 Pods at once, 3,072 chips each, for 6,144 in total, which Google calls the largest TPU setup used for training so far. It reports that the chips spent 57.8% of their capacity doing useful math, a high efficiency for a job this large.

## What PaLM can do

Most results use **few-shot prompting**: the model gets a handful of worked examples in its prompt, then a new question, with no extra training. Google reports that PaLM 540B:

- beat the earlier few-shot records of models such as GPT-3, Gopher, Chinchilla and LaMDA on all but one of 29 standard English tests;
- scored better than the average person on a group of 58 tasks from BIG-bench, a new collection of more than 150 difficult tests;
- solved 58% of GSM8K, a set of grade-school math word problems, when shown eight examples that spell out the reasoning step by step, edging past the previous best of 55%;
- matched a 12-billion-parameter version of OpenAI's Codex on coding tests, even though only 5% of its training data is code.

The step-by-step trick is called **chain-of-thought prompting**: instead of showing only question and answer, each example walks through the intermediate steps. Here is the idea in miniature:

```prompt
Q: A baker makes 4 trays of 12 muffins and sells 30. How many are left?
A: 4 trays of 12 is 48 muffins. 48 minus 30 is 18. The answer is 18.

Q: A library has 5 shelves of 20 books and lends out 37. How many remain?
A:
```

A model that has learned the pattern writes out "5 shelves of 20 is 100 books…" before giving 63, and that habit makes it far more accurate on problems with several steps.

The paper also shows PaLM explaining jokes it could not have seen before. Given a joke about someone who gets excited when they smell food and goes on alert when the doorbell rings, it worked out that the joke is about dogs.

> **Try it:** Take a two-step word problem from a school textbook. Write one worked example in the chain-of-thought style above, then the new question, and paste both into any text-generating model you can use. Compare the result with the same question asked on its own, without the worked example.

## Why it matters

PaLM is strong evidence that making language models bigger still pays off. Google says gains on BIG-bench follow a steady upward line as size grows, with no sign yet of leveling off. On roughly a quarter of the BIG-bench tasks, the jump from 62 billion to 540 billion parameters was much larger than the smaller models predicted. If that pattern holds, researchers cannot fully forecast what a bigger model will be able to do by testing smaller ones.

It also shows Google's infrastructure bet working. Pathways is meant to let one model learn many tasks efficiently across huge numbers of chips, and PaLM is its first large-scale use.

Google pairs the results with caution. The paper includes a **model card** (a standard summary of what a model is for and its limits) and a **datasheet** for the training data, along with tests for bias, toxic output and memorized text. It warns that, like other models trained on web text, PaLM can reflect harmful stereotypes, and that anyone building on it should check for those risks in their own setting.

## Can you use PaLM yourself?

Not at this point. Google has published the blog post, the paper and sample outputs, but it has not released the model, offered a public demo or opened an interface for developers. The model card describes PaLM's main purpose as research, says Google uses it internally for research on text and code generation, and recommends that no product be built on it without first assessing safety and fairness.

What you can do today is read the paper's examples, which show the prompts and the model's answers, and practice the techniques it relies on. Few-shot and chain-of-thought prompting work with other large language models already available to the public, so the worked example above is a fair way to see why this research is attracting so much attention.

## Sources

- Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance: https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html
- PaLM: Scaling Language Modeling with Pathways: https://arxiv.org/abs/2204.02311v1