Short answer
You can chat with Llama 3.1 405B for free in Hugging Face's HuggingChat, try the Llama 3.1 family in GroqChat, and call 405B through paid APIs such as Together AI's, without owning a single GPU. Running the same test prompt in each place is the quickest way to see what a very large open model can do for you.
Follow-up question What do hosted open models cost?
Meta's Llama 3.1 405B is the largest openly available model, and that is exactly why most people can't run it. Hugging Face estimates that its weights alone need about 810 GB of memory at full 16-bit precision, around 405 GB at 8-bit and roughly 203 GB at 4-bit, before counting the extra memory used to hold a long conversation. A single server with eight top-end H100 graphics cards has about 640 GB. That is far beyond a laptop, and expensive even for a small company.
The good news is that "open" means other companies are free to host the model and let you use it. In the nine days since the release, several have done so, which turns a 405-billion-parameter model into something you can try in a browser tab.
Three places to try it today
| Service | What you get | Cost to try | Best for |
|---|---|---|---|
| HuggingChat (huggingface.co/chat) | A chat interface with Llama 3.1 405B Instruct and 70B Instruct among the model choices | Free with a Hugging Face account | A first look at 405B with no setup |
| GroqChat | Chat with Llama 3.1 models running on Groq's own LPU chips, built for fast replies | Open to the public | Seeing how speed changes the experience |
| Together AI | Paid API access to Llama 3.1 8B, 70B and 405B, plus fine-tuning | Pay per token | Building an app or running many tests |
A few details are worth knowing. HuggingChat is itself built on open-source software, chat-ui and text-generation-inference. Groq says its 405B API is in early access for selected customers only, while its chat app and developer console are open more widely. Hugging Face also offers API endpoints for the Llama 3.1 models to its paid PRO subscribers. Hosted versions are often quantized, meaning their numbers are stored with fewer bits to save memory; Meta publishes an official 8-bit version of 405B for this purpose, and quality is close to the original.
A worked same-prompt test
Testing one prompt in several places tells you more than any leaderboard. Pick a task that resembles your real work and that has an answer you can judge. Here is one that mixes reasoning, writing and a language switch:
A community garden has 24 plots. 9 are rented for the full season at $40,
the rest for half a season at $25. Two plots are given free to a school.
1. How much rental income does the garden collect? Show the steps.
2. Write a three-sentence notice inviting new members for next season.
3. Translate the notice into French.The correct income is $685: 9 full-season plots bring $360, and 13 of the remaining 15 plots are rented for half a season, bringing $325. A model that overlooks the two free plots will answer $735. That small trap is the point of the test. Score the answers in a simple table like this one (the scores shown are illustrative):
| Model and service | Math correct? | Notice clear and the right length? | French natural? | Speed |
|---|---|---|---|---|
| 405B on HuggingChat | yes | yes | yes | moderate |
| 70B on GroqChat | no, counted the free plots | yes | yes | very fast |
Run the same prompt two or three times, since answers vary from one run to the next, and keep the wording identical across services. If the 70B model gets everything right for your task, you may not need the 405B one, and it will cost less to use.
Try it: Choose one task you do every week, write it as a single prompt with a checkable answer, and run it on Llama 3.1 405B in HuggingChat and on a smaller Llama 3.1 model elsewhere. Record which one you would actually use, and why.
What to watch for
Hosted is not the same as private. When you use a hosted model, your prompts go to that company's servers under its terms, so read the privacy policy before pasting anything sensitive. Free chat services can also change their model lists, add limits or slow down at busy times. If you need guaranteed access or control over your data, a paid API with clear terms, or eventually your own hardware, becomes the better choice.
What do hosted open models cost?
Trying them in a chat app costs nothing beyond an account. Using them through an API is billed by the token, a small chunk of text roughly three-quarters of a word. Together AI's pricing page this week lists Llama 3.1 405B Turbo at $5.00 per million tokens, with input and output counted together. The 70B model costs $0.88 per million in its Turbo version and $0.54 in its Lite version, and 8B costs between $0.10 and $0.20 depending on the version.
As an illustration, suppose you test 2,000 prompts that average 1,500 tokens each, counting the prompt and the reply. That is 3 million tokens: about $15 on 405B Turbo at Together's listed price, or about $2.64 on 70B Turbo. The gap is large enough that it pays to find the smallest model that passes your test.
Prices for open models also tend to differ between providers, because each one competes on its own hardware and software. Before you commit, compare the current price pages, check whether the provider counts input and output separately, and run your test prompt on each candidate. The model is the same Llama 3.1, but speed, limits and terms are not.
Sources
- Llama 3.1 - 405B, 70B & 8B with multilinguality and long context — Hugging Face, July 23, 2024
- Now Available on Groq: The Largest and Most Capable Openly Available Foundation Model to Date, Llama 3.1 405B — Groq, July 23, 2024
- Together Pricing — Together AI, July 27, 2024