> The tricky part is coming up with a reasonable set of "add noise" transformations. Yes, as well as dealing with a variable-length window. When generating images with diffusion, one specifies the image ahead-of-time. When generating text with diffusion, it's a bit more open-ended. How long do we want this paragraph to go? Well, that depends on what goes into it -- so how do we adjust for that? Do we use a hierarchical tree-structure approach? Chunk it and do a chain of overlapping segments that are all of fixed-length (could possibly be combined with a transformer model)? Hard to say what would finally work in the end, but I think this is the sort of thing that YLC is talking about when he encourages students to look beyond LLMs. [1] * [1] x.com/ylecun/status/1793326904692428907

x.com/ylecun/status/1742563244949078369 This one. He deliberately writes wrong information. Even he is right in the future, 0 reason to follow his word.