At this point, anyone claiming that LLMs are "just" language models aren't arguing in good faith. LLMs are a general purpose computing paradigm. LLMs are circuit builders, the converged parameters define pathways through the architecture that pick out specific programs. Or as Karpathy puts it, LLMs are a differentiable computer[1]. Training LLMs discovers programs that well reproduce the input sequence. Tokens can represent anything, not just words. Roughly the same architecture can generate passable images, music, or even video. [1] x.com/karpathy/status/15828073679886540…

>LLM capabilities are self-evident This is undermined by all the disagreement about what LLMs can do and/or how to characterize it. >LLM confabulations are statistical, not a consequence of missing information or damage. LLMs aren't statistical in any substantive sense. LLMs are a general purpose computing paradigm. They are circuit builders, the converged parameters define pathways through the architecture that pick out specific programs. Or as Karpathy puts it, LLMs are a differentiable computer[1]. So yes, narrative crafting in terms of leveraging available putative facts into a narrative is an apt characterization of what LLMs do. [1] x.com/karpathy/status/15828073679886540…

Original creator of the blog here. Yes, you are right. The title of the blog is a bit misleading. I originally wrote this for a course from my University and thought it was interesting so integrated it into my website. I didn't expect to get that much traffic on the blog and didn't intend to do any clickbaiting. I just wanted to extend on what Andrej karpathy mentioned in one of his tweets: twitter.com/karpathy/status/15828073679…. I think i also linked it in the blogpost. I think this view on transformers is a very powerful one and explains in the best way why the transformer is build that way. This also allows us to reason about certain improvements for the transformer. For example weight tying some layers to make it easier to map to algorithms that have a lot of repeated operation. The Goal of the blog was not to use the transformer as a general purpose differentiable computer but to convince you that the transformer already IS a general purpose differentiable computer. Maybe i will create another blog where i train the transformer on simple algorithmic problems like sorting or copying data, just like in the differentiable turing machine, or even show how you can hard code the transformer weights to solve them. What do you think would be a better title for the blog?

Original creator of the blog here. I originally wrote this for a course from my University and thought it was interesting so integrated it into my website. So i didn't want to "solve" anything. I just wanted to extend on what Andrej karpathy mentioned in one of his tweets: twitter.com/karpathy/status/15828073679…. I think this view on transformers is a very powerful one and explains in the best way why the transformer is build that way. This also allows us to reason about certain improvements for the transformer. This was mostly an educational blog

This overly reductive description of LLMs misses the forest for the trees. LLMs are circuit builders, the converged parameters pick out specific paths through the network that define programs. In other words, LLMs are differentiable computers[1]. Analogous to how a CPU is configured by the program state to execute arbitrary programs, the parameters of a converged LLM configure the high level matmul sequences towards a wide range of information dynamics. Statistics has little relevance to LLM operation. The statistics of the training corpus imparts constraints on the converged circuit dynamics, but otherwise has no representation internally to the LLM. [1] x.com/karpathy/status/15828073679886540…

we've discovered some kind of differentiable computer[1] and as with all computers, people have their own interests and hobbies they use them for. but unlike computers, everyone pitches their interest or hobby as being the only one that matters. [1] x.com/karpathy/status/15828073679886540…

LLMs aren't language models, but are a general purpose computing paradigm. LLMs are circuit builders, the converged parameters define pathways through the architecture that pick out specific programs. Or as Karpathy puts it, LLMs are a differentiable computer[1]. Training LLMs discovers programs that well reproduce the input sequence. Roughly the same architecture can generate passable images, music, or even video. It's not that language generation is all there is to AGI, but that to sufficiently model text that is about the wide range of human experiences, we need to model those experiences. LLMs model the world to varying degrees, and perhaps in the limit of unbounded training data, they can model the human's perspective in it as well. [1] x.com/karpathy/status/15828073679886540…

LLMs are a general purpose computing paradigm. LLMs are circuit builders, the converged parameters define pathways through the architecture that pick out specific programs. Or as Karpathy puts it, LLMs are a differentiable computer[1]. Training LLMs discovers programs that well reproduce the input sequence. Roughly the same architecture can generate passable images, music, or even video. The sequence of matrix multiplications are the high level constraint on the space of programs discoverable. But the specific parameters discovered are what determines the specifics of information flow through the network and hence what program is defined. The complexity of the trained network is emergent, meaning the internal complexity far surpasses that of the course-grained description of the high level matmul sequences. LLMs are not just matmuls and logits. [1] x.com/karpathy/status/15828073679886540…