I'd recommend following author of this thread for technical analysis of chatgpt and other models twitter.com/goodside/status/15988746742…...

Likely it made it up, since GPT is based on tokens, not words: - platform.openai.com/tokenizer That said, all the words listed in the list you provide are treated as tokens. “The” and “the” would not have the same token IDs, but are the same word. “reshape” tokens are [resh][ape] and the token IDs are [3447, 1758]. “Reshape” tokens are [Res][h][ape] and the token IDs are [4965, 71, 1758]. Notice though the “ape” token is the same, since the characters are the same. Not aware of any character sets that are identical, but based on context have different token IDs. ____________ One possible solution to test this would be to ask ChatGPT to list the first 100 token IDs from its training corpus — then if confirm IDs match the IDs match the tokens on the URL listed above, then cross reference the order of the words it originally provided. That said, it’s fair to assume if you did that, the data would contain factual errors or be unrelated gibberish. ____________ Reason I say it’s likely made up is that to my knowledge ChatGPT lacks the abilities of introspection and assimilation. That is, it’s unable to inspect its current state or assimilate its own output beyond a very limited memory window of few thousand tokens. twitter.com/goodside/status/15988746742… ____________ Related questions are here: - news.ycombinator.com/item?id=34677573

ChatGPT has a published context window of 4096 tokens. Although, I saw someone on Twitter saying the real figure, based on experiments, was closer to 8192 tokens. [0] Still, that’s an obvious roadblock to “understanding” large code bases - large code bases are too big to fit in its “short-term memory”, and at runtime its “long-term memory” is effectively read-only. Some possible approaches: (A) wait for future models that are planned to have much longer contexts (B) fine tune a model on this specific code base, so the code base is part of the training data not the prompt (C) Break the problem up into multiple invocations of the model. Feed each source file in separately and ask it to give a brief plain text summary of each. Then concatenate those summaries and ask it questions about it. Still probably not going to perform that well, but likely better than just giving it a large code base directly Another issue is that, even the best of us make mistakes sometimes, but then we try the answer and see it doesn’t work (compilation error, we remembered the name of the class wrong because there is no class by that name in the source code, etc). OOTB, ChatGPT has no access to compilers/etc so it can’t validate its answers. If one gave it access to an external system for doing that, it would likely perform better. [0] mobile.twitter.com/goodside/status/1598…...

It does not appear to be that simple: twitter.com/goodside/status/15988746742…