Why are LLMs not very data efficient?

Training a large language model or neural net is famously not very data efficient. To give an idea of the scale: a fairly basic model like GPT-3 was training on something like the equivalent of 200 billion words. By contrast, a child might come across something like 100 million. That’s something like a factor of a thousand difference.

One significant factor lying behind this gap might be that we have yet to implement continual learning into our models.

That means that the models only learn in the initial training run, but don’t get to build up experience through trial and error. It is much, much more data efficient to come up with a new hypothesis, try it out and see whether it is falsified or not, rather than just try learn over the examples you find scattered across your immense training curriculum.

One counter argument might be that while an LLM can’t do trial and error itself, it can still learn offline from the many examples of human trial and error it can find. Why isn’t this enough?

The problem is that in practice firstly we normally only publish the final results of our work, not all the intermediate steps we took to get there. Secondly, and even more fundamentally, many of these intermediate thoughts are never externalised into words on a page or computer: they remain internal in our own head.

When this is not true, and we are able to build up a vast library of legible, intermediate examples, neural nets can learn much deeper thinking. This is basically the pattern we saw with reasoning models over 2025, as they were trained in depth on many, many examples of correct and incorrect reasoning in math and coding.

But for most areas of human thought we don’t have this library of examples. This examples the pattern of some of the mistakes you still see neural nets making, breaking common sense in a way that a human never would. Another way to think about this, is they are much less likely to natively track the state of the world, and all the features in it we think are important. While something might seem statistically likely on the surface to an LLM, a human would be much more likely to simulate it in real time and see the absurd consequences it comes to.