The idea that LLMs are "running out of training data" is pretty widespread, but does it actually hold up? Has anyone seen representatives of the cutting-edge AI labs - OpenAI, Anthropic, Meta, Mistral etc - express concern about this? The impression I've been getting is that quality turns out to be much more important than quality. This tweet from Andrej Karpathy for example: twitter.com/karpathy/status/17973131734… > Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out.

Here's one of the signals that makes me think quality matters more than quantity: twitter.com/karpathy/status/17973131734… > Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.

There's a pretty common misconception that training LLMs is about loading in as much data as possible no matter the source. That might have been true a few years ago but today the top AI labs are all focusing on quality: they're trying to find the best possible sources of high quality tokens, not randomly dumping in anything they can obtain. Andrej Karpathy said this last year: twitter.com/karpathy/status/17973131734… > Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.

>Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. >The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out. x.com/karpathy/status/17973131734497649… Context: FineWeb-Edu, which used Llama 70B to [train a classifier to] filter FineWeb for quality, rejecting >90% of pages. huggingface.co/spaces/HuggingFaceFW/blo…...

Andrej Karpathy: twitter.com/karpathy/status/17973131734… > Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it’s not even clear how prior LLMs learn anything at all. People keep assuming that AI companies want to pipe any old junk into their models, but I'm increasingly convinced that this isn't true.

Because most companies genuinely don't value training on user data in that way. It just isn't that valuable, even without the huge amount of negative publicity attached to doing that. The cutting edge AI labs are leaning much more into high quality data (licensed from the Associated Press for example) and synthetic data, which it turns out is a huge part of Claude and Microsoft's Phi series. Andrej Karpathy said: "The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all." - twitter.com/karpathy/status/17973131734…

I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be. Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement. OpenAI are licensing data from sources like the Associated Press. Andrej Karpathy said this: twitter.com/karpathy/status/17973131734… > Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.