How do AI chatbots and large language models ‘reason’ like humans?

This could be an interesting new research paper by Tal Linzen and Paul Smolensky

"... The study provides evidence that the internal representations of neural networks implicitly realize symbolic structure. ...

For the new study, ... analyzed the internal functions of several high-profile LLMs. They found that neural networks implicitly grasp symbolic structure within the numeric lists that drive them.

“Our analysis demonstrates that, despite appearances, the vectors powering LLMs are organized in a way that is equivalent to the symbolic structures that drive much of human cognitive function,” ...

“With LLMs, we show that vectors are organized in a very particular way that gives rise to emergent symbolic structure — symbolic structure that is present at a high level despite not being apparent at a low level.” ...

The researchers found that they could replace an LLM’s entire representation-generating process with “role-filler” approximations embodying symbolic structures and the AI system’s behavior would remain largely unchanged. The finding held for small-scale neural networks trained to manipulate lists as well as for seven large-scale LLMs operating in four domains that have long been central in research on symbolic intelligence: language, arithmetic, logic, and computer coding. ..."

From the abstract:

"Modern systems in artificial intelligence (AI) somehow excel in domains for which they seem poorly suited.

Intelligence has traditionally been modeled as operating over structured combinations of symbols, such as logical formulas.

However, the strongest modern AI systems are based on neural networks, which instead represent information in continuous vectors. Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas.

How do they do it? In this work, we propose a potential answer: Despite appearances, perhaps the internal representations of neural networks implicitly realize symbolic structure.

In support of this hypothesis, we show that the vector representations of a variety of neural networks can be closely approximated with symbolic structures: we can replace the network's entire representation-generating process with a closed-form equation instantiating a symbolic structure, and the network's behavior remains largely unchanged.

This finding holds for both small-scale neural networks trained to manipulate lists as well as large language models (LLMs) operating in four domains that are central in symbolic traditions: arithmetic, logic, computer code, and language.

Further, our symbolic approximation allows us to modify an LLM's behavior in targeted ways via precise interventions on its internal representations, showing that the LLM's behavior is reliant on the symbolic structures we have identified. This work provides a potential way to reconcile longstanding symbolic conceptions of intelligence with the vector-based nature of modern AI."

How do AI chatbots ‘reason’ like humans? A Yale-led study offers clues | Yale News "The neural networks that power popular AI chatbots process data much differently than humans but capably perform tasks involving language, logic, and math. A new study sheds light on how those systems “think.”"

The Emergent Symbolic Structure of Artificial Neural Networks (preprint, open access)