Important to note though, that even if they say it has 200k context, there's a good chance it won't be able to use all of it. For example, here is an experiment on long context recall for Claude and GPT4: twitter.com/GregKamradt/status/17270181… twitter.com/GregKamradt/status/17223867… Even though claude claims 200k tokens context, perfect memory stops at about 19k tokens. I'd personally expect the Yi-34b models to perform even worse at the task, though I have not tested it.

Emergent awareness is cool, but it seems Claude Opus has imperfect recall at the 25k token mark, and has lossy recall well before the half-payload mark (100k)? For reference, GPT4-Turbo has perfect recall to 64k and drops pretty badly past that until 128k: twitter.com/GregKamradt/status/17223867…