Relatedly, Karpathy had an insightful post awhile back on perils of tokenization twitter.com/karpathy/status/16579492345… (I cited it in my Excel tutorial on BPE youtube.com/watch?v=PvZN3-WqAOI&t=1916s )

Karpathy himself has raised some of these issues with BPE twitter.com/karpathy/status/16579492345… There is research about getting rid of tokens, some of which is in that thread

The point isn't that it doesn't work at all, but that it doesn't work as well as other approaches that we have. Which is evidenced by the fact that all the best-performing models on the market use tokenization. It's not a secret that tokenization is fundamentally a hack, and that ideally we'll get rid of it eventually one way or another (twitter.com/karpathy/status/16579492345…). And in principle, you can compensate for the deficiencies of byte-level tokenization with larger models and larger contexts. But what this means in practice is that a model that has the same level of intelligence (on most tasks; obviously, there are some specific tasks, like say counting characters in a word, where tokenization is detrimental to intelligence) takes a lot more resources to train, hence why we aren't seeing more of those.

Relevant tweet (imo) from Andrej Karpathy - twitter.com/karpathy/status/16579492345…

How many examples like that do you think it's seen? You can't given an example of something that is in effect a trick to get character level tokenization and then expect it to do well when it's seen practically zero of such data in it's training set. Nobody who suggests methods like character or byte level 'tokenization' suggests a model trained on current tokenization schemes should be able to do what you are suggesting. They are suggesting actually train it on characters or bytes. You say all this as though I'm suggesting something novel. I'm not. Appealing to authority is kinda lame, but maybe see Andrej's take: x.com/karpathy/status/16579492345352110…

Is Karpathy lying saying that word tokenization brings such problems that can be seen in many LLMs? twitter.com/karpathy/status/16579492345… I'm not arguing that you can't use single chars just that many of the issues parent discussed are caused by this.