Short answer
BloombergGPT is a 50-billion-parameter language model that Bloomberg trained from scratch on a mix of its own financial archives and general public text, described in a paper posted in March 2023. It shows what a domain model changes: on finance tasks it beat open general models of similar size, while staying about as good on everyday language tasks, but it cost a large training run and is kept for Bloomberg's own use.
Follow-up question Will every industry get its own model?
Most language models people use are general models: trained on a broad sweep of web pages, books and code so they can handle almost any topic. A domain model is built for one field, such as finance, medicine or law, by training it heavily on that field's own documents. At the end of March, Bloomberg, the financial data and news company, published the most detailed account yet of building one: BloombergGPT.
Bloomberg's argument is simple. Finance has its own vocabulary and its own tasks, from judging whether a headline is good or bad news for a company to pulling numbers out of filings. The company says general models handle many things well, but its experience with earlier specialized models convinced it they could not fully replace them.
How Bloomberg built it
The key ingredient is data. Bloomberg's analysts have collected and curated financial documents for about forty years. From that archive, the team built a dataset it calls FinPile: 363 billion tokens (the word pieces a model reads) of English financial text, including web pages about finance, news, company filings, press releases and Bloomberg's own content. It added 345 billion tokens of general public text, the kind other models train on.
The team then trained a model with 50 billion parameters, the internal numbers a model adjusts while it learns. Training used 569 billion tokens from the combined collection and ran on 512 high-end graphics processors for about 53 days. That is a serious investment, well beyond what a typical company spends on AI.
The mix was the new part. According to the paper, earlier domain models were either trained only on specialist text or made by adapting a very large general model afterward. Bloomberg trained on both kinds of text from the start, hoping to get finance skills without losing general ones.
What it does better
On public finance benchmarks, such as judging the sentiment of financial news or answering questions that involve figures from reports, BloombergGPT outperformed open models of similar size, including GPT-NeoX and OPT, by wide margins. On general language tests, it held its own or did better. Bloomberg also tested it on internal tasks closer to its real uses.
The paper's examples show the practical side. BloombergGPT can turn a plain-English request into Bloomberg Query Language (BQL), the complex code used to pull data on the Bloomberg Terminal. It can suggest short news headlines from a paragraph, a job journalists do every day. And it answers finance questions, such as who leads a given company, more reliably than the other models shown alongside it.
There is a catch for the rest of us: Bloomberg has not released the model. The paper explains that a model can leak pieces of its training data, and that data is the heart of Bloomberg's business. So BloombergGPT is a tool for Bloomberg and its customers, not something you can download or try in a chat window.
Three ways to get a "specialist"
BloombergGPT is one of three routes a business can take. Here is how they compare:
| Approach | What it means | Effort and cost | Best when |
|---|---|---|---|
| Domain model from scratch | Train a new model on your field's data plus general text (BloombergGPT) | Very high: huge dataset, weeks of training on hundreds of processors, a research team | You own a vast, unique archive and many uses for it |
| Fine-tune a general model | Further train an existing model on a set of your own examples | Moderate: OpenAI fine-tuning for GPT-3.5 Turbo, opened in August, charges by tokens trained and used | One narrow task done the same way many times, such as a fixed format or a house style |
| Prompt a general model | Give a model like GPT-4 your context, examples and rules in each request | Low: no training at all | Varied tasks, small volume, or while you are still learning what you need |
Try it: Write down one specialist task from your work, such as summarizing a type of report or tagging support tickets. Place it in the table above: does it need a vast private archive, hundreds of repeated examples, or just a good prompt with context? For most people, the honest answer is the third row, so start there.
Will every industry get its own model?
Some will, but probably not every one, and not every company. BloombergGPT shows the conditions that make a custom model worthwhile: a huge collection of high-quality text that nobody else has, many internal uses for it, and the budget for a multi-week training run. Few organizations have all three. Those that do, in fields like finance, law or medicine, have a strong reason to try.
For everyone else, cheaper routes are getting better. OpenAI says early tests show a fine-tuned GPT-3.5 Turbo can match or beat the base GPT-4 on certain narrow tasks, and the data used to fine-tune stays the customer's. Open models such as Llama 2 can also be adapted by anyone with the right skills, as Meta itself did when it built Code Llama for programming.
A likely pattern is layers rather than one model per industry: a few large general models, a small number of true domain models where unique data justifies them, and a great many fine-tuned or carefully prompted versions on top. The lesson BloombergGPT offers a smaller team is less about building a model than about data. As Bloomberg's own team put it, a model's quality depends on what you train it on. Knowing your own documents well, and keeping them clean, pays off whichever route you take.
Sources
- BloombergGPT: A Large Language Model for Finance — arXiv (Bloomberg and Johns Hopkins University authors), March 30, 2023
- Introducing BloombergGPT, Bloomberg's 50-billion parameter large language model, purpose-built from scratch for finance — Bloomberg LP, March 30, 2023
- GPT-3.5 Turbo fine-tuning and API updates — OpenAI, August 22, 2023