Interestingly this point was indicated by Karpathy last summer that RLHF is barely RL. He said it would be very difficult to apply pure reinforcement learning on open-domains. This is why RLHF are a shortcut to fill this gap but still because the reward model is trained on human vibes checks the LLM could easily game the RM by giving out misleading responses or gaming the system. Importantly the barrier is that open domains are too complex and too undefined to have a clear reward function. But if someone cracks that — meaning they create a way for AI to self-optimize in these messy, subjective spaces — it'll completely revolutionize LLMs through pure RL. Here's the link of the tweet: x.com/karpathy/status/18212772649963522…

> So RLHF is the secret sauce behind modern LLMs? Karpathy wrote[^0]: " RL is powerful. RLHF is not. […] And yet, RLHF is a net helpful step of building an LLM Assistant. I think there's a few subtle reasons but my favorite one to point to is that through it, the LLM Assistant benefits from the generator-discriminator gap. That is, for many problem types, it is a significantly easier task for a human labeler to select the best of few candidate answers, instead of writing the ideal answer from scratch. […] No production-grade actual RL on an LLM has so far been convincingly achieved and demonstrated in an open domain, at scale. " --- [^0]: x.com/karpathy/status/18212772649963522…

well, as yann lecun said : "Adversarial training, RLHF, and input-space contrastive methods have limited performance. Why? Because input spaces are BIG. There are just too many ways to be wrong" [1] A way to solve the problem is projecting onto latent space and then try and discriminate/predict the best action down there. There's much less feature correlation down in latent space than in your observation space. [2] [1]:x.com/ylecun/status/1803696298068971992 [2]: openreview.net/pdf?id=BZ5a1r-kVsf

But RL algorithms do implement things like curiosity to drive exploration?? arxiv.org/pdf/1810.12894. Thinking to arbitrary depth sounds like Monte Carlo tree search? Which is often implemented in conjunction with RL. And working memory I think is a matter of the architecture you use in conjunction with RL, agree that transformers aren't very helpful for this. I think what you call 'trial and error', is what I intuitively think of RL as doing. AlphaProof runs an RL algorithm during training, AND at inference time. When given an olympiad problem, it generates many variations on that problem, tries to solve them, and then uses RL to effectively finetune itself on the particular problem currently being solved. Note again that this process is done at inference time, not just training. And AlphaProof uses an LLM to generate the Lean proofs, and uses RL to train this LLM. So it kinda strikes me as a type error to say that DeepMind have somehow abandoned RL in favour of LLMs? Note this Demis tweet x.com/demishassabis/status/181659656839… where it seems like he is saying that they are going to combine some of this RL stuff with the main gemini models.

x.com/karpathy/status/18212772649963522…