The Geometry of Concepts in Large Language Models

While modern artificial intelligence has achieved remarkable empirical success, establishing a rigorous "science of AI" remains an open challenge. This talk presents a mathematical foundation for AI interpretability, focusing on how large language models (LLMs) encode high-level semantic concepts within their internal representation spaces. Because LLMs fundamentally operate via softmax distributions over vocabularies, we ground our analysis in the geometry induced by the softmax function.

First, we formalize the "linear representation hypothesis"—the widely observed but previously informal idea that high-level concepts are linearly encoded as directions. By introducing a "causal inner product," we connect semantic structure to geometric structure, revealing a special property: this inner product unifies the embedding and unembedding spaces.

Next, using information geometry, we reveal the underlying geometry of the representation space to be a dually flat (Bregman) manifold. This rich duality structure allows us to rigorously link the linear representation hypothesis to the tasks of measuring (probing) and manipulating (steering) concepts. Leveraging this framework, we develop "Dual Steering" by adding concept directions within the dual space. We prove both theoretically and empirically that, given a linear probe, Dual Steering is the optimal intervention: it effectively manipulates the target concept while strictly minimizing interference with off-target concepts.