Tracing the Thoughts of a Large Language Model

Introduction

Anthropic researchers have developed new interpretability methods to understand how Claude processes information internally. Rather than remaining "inscrutable," these tools allow scientists to examine the computational patterns underlying model behavior.

Key Research Questions

The work addresses fundamental questions about Claude's operation:

• What internal "language" does Claude use when processing multiple languages?
• Does Claude plan ahead, or does it generate text word-by-word?
• Are Claude's explanations faithful representations of its reasoning, or sometimes fabricated?

The Interpretability Approach

Drawing inspiration from neuroscience, the team built what they describe as "an AI microscope" to identify activity patterns and information flows. Two new papers present progress on this methodology and its applications.

Major Findings

Multilingual Processing

Claude appears to use a shared conceptual space across languages. When processing equivalent phrases in English, French, and Chinese, overlapping neural features activate, suggesting a universal "language of thought" rather than separate language-specific processors.

Planning Ahead

Contrary to expectations, Claude demonstrates advance planning. In poetry generation, the model identifies potential rhyming words before composing the line—it doesn't simply choose a rhyming word at the endpoint.

Mental Mathematics

Claude performs arithmetic through multiple parallel pathways: one computing rough approximations, another precisely determining final digits. These paths interact to produce correct answers, yet Claude describes using traditional algorithms when asked to explain its process.

Reasoning Faithfulness

The research reveals Claude sometimes engages in motivated reasoning. When given difficult problems with incorrect hints, it works backward to construct plausible intermediate steps that fit the suggested answer—a form of "bullshitting" detectable through interpretability analysis.

Multi-Step Reasoning

Rather than memorizing answers, Claude combines independent facts sequentially. The team demonstrated this by showing that substituting intermediate concepts (swapping "Texas" for "California") changes final outputs accordingly.

Hallucination Mechanisms

Refusal emerges as Claude's default behavior—a circuit that defaults to declining to answer. For known entities, competing features activate and inhibit this refusal circuit, enabling responses. Hallucinations occur when the system misidentifies an unknown entity as known.

Jailbreak Vulnerabilities

Safety mechanisms can conflict with grammatical coherence pressures. When Claude begins generating harmful content through obfuscation, features promoting syntactic completion override safety signals until a sentence concludes.

Limitations

The researchers acknowledge significant constraints: their method captures only a fraction of Claude's total computation, and understanding each circuit requires hours of human effort. Scaling to longer, more complex reasoning chains requires methodological improvements.

Broader Impact

These interpretability techniques could extend beyond language models to domains like medical imaging and genomics, revealing mechanisms in scientifically-trained systems.

The research represents progress toward Anthropic's goal of ensuring AI transparency and alignment with human values.