The Bottlenecks of LLMs

What are the biggest bottlenecks that prevent today’s LLM based AI models from reaching a human level of AGI, or even superintelligence?

Four of the most commonly cited bottlenecks include their lack of continual learning, that are unable to be truly creative and make new conceptual breakthroughs, that they still can be extremely brittle and make mistakes that seem very stupid to a human, and that they have still have very low data efficiency, meaning that we have to train them on orders of magnitude more information than a human child experiences.

What if, in actual fact, these are not four separate bottlenecks, but facets of the same underlying issue?

Without continual learning, you have no way to merge and refactor basic concepts and come up with something new that truly generalises. You’re stuck at very shallow levels of creativity.

While LLMs can compose their existing representational primitives in novel ways, they cannot forge new concepts or restructure the relationships between them. Their weights are fixed after training, which means the conceptual building blocks they work with—and the ways those blocks can relate to each other—are static.

Without the ability to learn, you can't do trial and error as you go, or learn off your own intermediate thoughts.¹ That makes it far harder to learn, and means you have to see many more examples before you're confident of any new pattern.

And then all three of these deficits mean that it’s hard for a LLM to build a coherent, truly multimodal world model. That increases their brittleness and their likelihood of making to us what seem odd mistakes.

We can somewhat offset these limitations with greater scaffolding, but scaffolding is not the same as building something natively into the model - and so it will always be an incomplete solution.

Footnotes

1. The exception is if you can express them purely in a technical language such as mathematics that can be formally proven, and produced in an explicit chain of thought for future models to learn form. ↩