# What is Gemini 1.5 Pro, and what can a one-million-token context window do?

Date: 2024-02-16 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-gemini-1-5-pro/

**Answer:** Gemini 1.5 Pro is Google's new mid-size AI model, announced on February 15, 2024, that performs at about the level of its largest model, 1.0 Ultra, while using less computing power. Its headline feature is a context window of up to one million tokens, enough to take in about an hour of video, 11 hours of audio or more than 700,000 words in one prompt, offered for now to a limited group of developers and businesses.

Yesterday, February 15, 2024, Google announced **Gemini 1.5**, the next generation of the Gemini models it introduced in December. The first model in this generation is **Gemini 1.5 Pro**. It arrives one week after Google released its biggest model, 1.0 Ultra, in the paid Gemini Advanced chatbot, and about two months after the first Gemini models appeared.

Two claims stand out. Google says the new mid-size Pro model reaches roughly the quality of 1.0 Ultra while needing less computing power. And it can read far more at once than any large model released so far: up to one million tokens in a single prompt.

## What Google announced, and who can use it

The announcement came in a blog post introduced by Sundar Pichai, chief executive of Google and Alphabet, with the details from Demis Hassabis, who leads Google DeepMind. A companion post on Google's developer blog explained how to try it.

| | Gemini 1.5 Pro |
|---|---|
| Type | Mid-size **multimodal** model (it handles text, images, audio and video) |
| Standard context window | 128,000 tokens |
| Experimental context window | Up to 1,000,000 tokens, in private preview |
| Tested in research | Up to 10,000,000 tokens |
| Where | Google AI Studio (developers) and Vertex AI (Google Cloud customers) |
| Cost during the preview | Free for early testers, with slower responses expected |

Ordinary users of the Gemini chatbot do not get 1.5 Pro yet. Google says the 128,000-token version will come when the model is ready for wider release, followed by pricing tiers that scale up to one million tokens.

## What a context window is

A model's **context window** is how much it can consider at one time: your instructions, any documents you attach and the conversation so far. It is measured in **tokens**, small pieces of input. For text, a token is often a part of a word; Gemini also turns images, audio and video into tokens.

Think of it as the size of a desk. A small desk holds a few pages; you have to summarize or swap papers in and out. A very large desk lets you spread out a whole archive and look across all of it. Until this week, Google's developer blog notes, the biggest window offered by any large language model open to the public held 200,000 tokens. The original Gemini 1.0 models had 32,000.

Here is what Google says one million tokens can hold:

| Input | Roughly how much fits |
|---|---|
| Video | 1 hour |
| Audio | 11 hours |
| Code | More than 30,000 lines |
| Text | More than 700,000 words |

## What Google showed it doing

Google's demonstrations give a sense of the possibilities:

- **A long document.** Given the 402 pages of transcripts from the Apollo 11 Moon mission, the model answered questions about conversations and events scattered across the whole text. TechRadar describes the demo in which it found humorous moments between the astronauts in about 30 seconds.
- **A full film.** Shown a silent Buster Keaton film lasting 44 minutes, it described plot points and small details. In one test, a rough drawing of a water tower was enough for it to find the matching scene, about ten minutes in.
- **A big codebase.** Handed a codebase of over 100,000 lines, it suggested changes and explained how different parts fit together.
- **A new language.** With only a grammar manual for Kalamang, a language that fewer than 200 people speak, it learned to translate English into Kalamang about as well as a person studying the same material. This is called **in-context learning**: the model picks up a skill from what is in the prompt, without being retrained.

A common check for long context is the **needle in a haystack** test: hide one specific sentence somewhere in a huge block of text and ask the model to find it. Google reports that 1.5 Pro found it 99% of the time, even in blocks of one million tokens. On Google's broader set of benchmarks, 1.5 Pro beat 1.0 Pro on 87% of them and performed at a broadly similar level to 1.0 Ultra.

## How it works

Gemini 1.5 uses a **mixture-of-experts** design. Instead of running one giant network for every request, the model is divided into smaller specialist networks, and it learns to send each input only to the most relevant ones. Less of the model works on each request, which makes it cheaper to train and run. Google has published research on this technique for years, and the open Mixtral model uses a similar idea.

Google says the model went through its usual safety and ethics testing, and that it is building new tests for the risks specific to very long inputs.

## Why it matters

A huge context window changes how you can work with AI. Instead of cutting a report into pieces or building a search system to feed the model a few paragraphs at a time, you can, in principle, hand over the whole thing and ask questions across it. For video and audio, it opens tasks such as finding one moment in an hour of footage.

There are caveats. The million-token version is an experiment, limited to selected testers, and Google warns that responses will be slow. Its benchmark results come from Google itself and will need independent testing.

> **Try it:** Pick a long document you know well, such as a manual or a book chapter. Write three questions whose answers are spread across different sections. Whatever model you use now, these questions make a useful test of how well it handles long material.

## How can you start testing Gemini 1.5 Pro?

If you are a developer, sign up for the private preview through Google AI Studio, Google's free web tool for experimenting with Gemini models. Once accepted, you can upload several PDFs, a code repository or a long video directly as part of your prompt. AI Studio splits a video into thousands of still frames, without the sound, for the model to analyze. Businesses that use Google Cloud can ask their Vertex AI account team for access.

When you test, start with tasks that need the whole input: questions that connect the beginning and end of a document, or moments you can verify in a video. Expect longer waits than usual and check the answers against the source.

If you are not a developer, the free Gemini chatbot and Gemini Advanced still run on the 1.0 models. Google has not said when 1.5 Pro will reach them.

## Sources

- Introducing Gemini 1.5, Google's next-generation AI model: https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/
- Gemini 1.5: Our next-generation model, now available for Private Preview in Google AI Studio: https://developers.googleblog.com/2024/02/gemini-15-available-for-private-preview-in-google-ai-studio.html
- Google's Gemini AI can now handle bigger prompts thanks to next-gen upgrade: https://www.techradar.com/computing/artificial-intelligence/googles-gemini-ai-can-now-handle-bigger-prompts-thanks-to-next-gen-upgrade