From the Hacker News thread
An excellent X thread from Yann LeCun saying similar: x.com/ylecun/status/1791890883425570823
Account
@ylecun
- Ideas
- 4
- Recommendations
- 6
- HN citations
- 14
AI4
View @ylecun on XFrom the Hacker News thread
Well said. I especially appreciate the distinction you're making between balancing liberty with liberty-destroying ideology. This is the kind of difficult nuance that I rarely see in these discussions. One one side there's the free speech absolutists, whose arguments tend to ignore the fact that unmoderated propaganda, and hate speech tends to be more addictive, and spread ignorance faster then fact-checking can fix it. The consequences of this sort of callous attitude are literally genocide[1]. On the other hand, there's the 'cancel-culture mobs' (for lack of a better term) which are now censoring regular speech that disagrees, or appears to disagrees, or isn't sufficiently subservient to their opinions. Just yesterday I was sadly reading this depressing thread where Yann LeCun was run off twitter[2] for explaining how bias (in the social science sense) can be traced back to various steps in the ML pipeline (in this case, mainly a feature of the dataset itself, but also the choice of errors, bias vs variance, etc). The inability to admit nuance is the only thing I can think both these groups share, and maybe what needs to be emphasized more. [1] nytimes.com/2018/11/06/technology/myanm…... [2] twitter.com/ylecun/status/1274782757907…
From the Hacker News thread
> But the other side doesn't see it that way, specifically not the "something more" part. It's "just math" all the way down, in our brains as well. The "emergent phenomena" are not undefined in this sense - they're (obviously in LLMs) also math, it's just that we don't understand it yet due to the sheer complexity of the resulting system. But that's not at all unusual - humans build useful things that we don't fully understand all the time (just look at physics of various processes). You might not, but you don't have to look far in these very comments to be met with woo and mysticism on the emergent phenomena side, either. > The obvious explanation is that, between the limited amount of information that can be extracted from 2D photos that the NN is trained on, and the limit on the complexity of world modeling that NN of a particular size can fit, its model of the world is just not particularly accurate. I think the more obvious solution is that it's not modelling the world, because... why would it be? It seems obvious to me that this is a significantly more complex way to handle their given task than every other explanation that does not require them create a model of the world. > Pragmatically, at some point, you have to pick the most likely (usually this means the simplest) explanation to move forward. I agree completely with this, which is why the idea that this is modelling the world is absolutely bizarre to me, particularly when our understanding of their training is that it's all weighting for pixel nearest neighbors via ranking how well it does at denoising 2D images. It's not like de-rendering is something new, either. ShaderMap has existed since the mid 2000s and could get similar results from 2D images without any AI/ML, and we have other models that can generate it without anyone suggesting they model the world, e.g. arxiv.org/abs/2201.02279 People like LeCun don't think that even models where the stated goal is world simulation are able to do it, e.g. sora, either - twitter.com/ylecun/status/1758740106955… - so it seems even more unlikely when that isn't the goal that this has happened due to some emergent phenomenon.
From the Hacker News thread
> The tricky part is coming up with a reasonable set of "add noise" transformations. Yes, as well as dealing with a variable-length window. When generating images with diffusion, one specifies the image ahead-of-time. When generating text with diffusion, it's a bit more open-ended. How long do we want this paragraph to go? Well, that depends on what goes into it -- so how do we adjust for that? Do we use a hierarchical tree-structure approach? Chunk it and do a chain of overlapping segments that are all of fixed-length (could possibly be combined with a transformer model)? Hard to say what would finally work in the end, but I think this is the sort of thing that YLC is talking about when he encourages students to look beyond LLMs. [1] * [1] x.com/ylecun/status/1793326904692428907