In March, during an event in Seoul, I witnessed a moment that sparked debate regarding artificial intelligence’s capabilities. A program I contributed to made a seemingly bizarre move in a game of Go, placing a stone on the fifth line of the board. Observers initially suspected a programming error, but the move was intentional. Ultimately, AlphaGo triumphed over Lee Sedol, one of the world’s top Go players. This incident led many, including Lee, to question the nature of AlphaGo’s decision-making, with some suggesting that the AI demonstrated creativity. However, it is crucial to clarify that the essence of AlphaGo’s success lay not in creativity but in its sophisticated reasoning abilities—an area where current AI models, particularly large language models (LLMs), fall short.
AlphaGo operates through two interconnected systems: its policy network, which predicts human-like moves, and its search mechanism, which evaluates potential future outcomes by constructing an extensive game tree. This dual approach allows AlphaGo to assess moves based on both immediate value and long-term consequences. This is reminiscent of human cognition as defined by Daniel Kahneman’s theory, which categorizes thought into two systems: the fast, intuitive System 1 and the slower, more deliberate System 2. In contrast, LLMs function primarily as advanced predictive algorithms, generating text based on statistical patterns without a true understanding or reasoning framework. While recent enhancements in LLMs have introduced intermediate steps to improve problem-solving, these strategies still stem from the same prediction model, lacking a genuine separation between knowledge and reasoning.
Three fundamental limitations of current LLMs prevent them from achieving true reasoning capabilities. First, they do not maintain a transparent epistemic state, which is essential for tracking hypotheses and uncertainties. Second, they fail to distinguish between knowledge and the methods of manipulating that knowledge, resulting in a lack of clarity about how conclusions are reached. Third, the apparent deliberation in LLM outputs often misrepresents the underlying process, as bots may fabricate chains of thought after arriving at an answer. This is particularly concerning in high-stakes fields like medicine and engineering, where the rationale behind decisions is as critical as the decisions themselves. Given these shortcomings, I believe that a new approach to machine reasoning is necessary—one that builds on the architecture of systems like AlphaGo, incorporating a structured epistemic state to enhance the reliability and transparency of AI decision-making.
Source: Don’t be fooled—LLMs don’t reason via MIT Technology Review