The Advantage of Neural Nets

Throughout much of the history of the development of AI, there has been a debate between those who believed that the most powerful types of system would be centred on a rules-based architecture like a traditional computer program (“good old fashioned AI”), and those who believed that ultimately it would have run on something closer to the kind of neural net we see used in the brain.

Neural nets have some clear disadvantages. They can be computationally expensive to run, require massive amounts of training at the start before they become useful, and are much harder to interpret or deliberately iterate and improve than the explicit architecture of a program.

On the other side of the balance sheet, one counterveiling advantage of neural nets is their massively parallel design allows them to process massively complex calculations at a relatively low absolute clock speed. Neurons in the brain update at most at something like 100 times per hour. This in turn could be one reason that the human brain can operate on such a low energy budget: in most areas of physics, going faster soon runs into square law increased energy costs.

Even more important, with neural nets we have methods that let them teach themselves. Chuck in enough data, run enough compute and a neural net can discover structure by itself.

This may be highly convenient, but is it fundamental? In other words, if we wanted to and had the time, could we hand label enough rules to build a model just as powerful as a neural net? Or would this run into some fundamental barrier where this just wouldn’t work? Is it possible to build intelligence without a neural net?

One argument that neural nets are essential to intelligence is that they have a unique ability to discover dimensional structure on their own, to commensurate across dimensions and interpolate between them.

What do I mean by that?

A traditional database in a normal computer program comes come with a defined structure. Think of a table in such a database: each column is a specific type of data, such as a number, a string or a binary. In some cases if the columns are of the same type, you might be able to combine or compare them in limited ways, but there is no guarantee. If you have a column called “Apples” and another column called “Oranges”, there is no automatic way for the system to be able to sum them together.

By contrast, a neural net given the way that it has been trained, will almost certainly not treat these two concepts as completely orthogonal - but have built them up so that they share some common vectors of ‘fruitness’. Given two points in a multidimensional space there is always some answer of what lies half way between them or how they compare on any dimension you like. This in turn makes it possible to interpolate between two ideas, or add them up, or answer classification questions that annoying definitional questions that philosophers have argued about for generations (“What is a chair?”)

Even more important, it gives them the flexibility to develop and identify what features - or combinations of other dimensions - are really key and bring them to the fore. Is it is this conceptual flexibility that allows neural nets to refactor ideas, learn over time and be genuinely creative, looking at the world through a different perspective.

Traditional code and its fixed data models doesn’t really have an easy way to do that.