The thing is, it's not gibberish. A sufficiently small language model can be understood by humans: twitter.com/karpathy/status/16451156225… The explanation is perfectly sensical, just too complex for humans to understand as the model scales up. The thing you're looking for - a reductive explanation of the weights of a ANN that's easy to fit in your head, does not exist. If it were simple enough to satisfy your demands, it wouldn't work at all.

I know how language models work. It's funny how you paint this obvious in retrospect. Guess Jan Leike doesn't know how statistical language models work. mobile.twitter.com/janleike/status/1625…...

Jan Leike's point is a bit more subtle there. He's not surprised it can learn translation with a relatively small number of training examples, but surprised that it seems to know some instruction with no examples. And just a few Tweets later speculates it's likely because there are examples in the training data, and the success of the model to follow instructions in a non-English language from "nothing" is proportional to the presence of those examples in the training data [0]. I don't think he would disagree with anything in my statement. 0. mobile.twitter.com/janleike/status/1625…...

Karpathy discusses viewing GPT as a finite-state Markov chain in this Twitter thread (and especially in the intro to the linked Colab notebook) : twitter.com/karpathy/status/16451156225…