Allow me to offer you this twitter thread: twitter.com/chaosprime/status/160789517…

You might be interested in this thread by a DeepMind’s research scientist. mobile.twitter.com/AndrewLampinen/statu…...

This is true. See twitter.com/cHHillee/status/16357903308… GPT-4 can't solve the easy, newer problems at CodeForces, but gets all the older hard problems.

Equally there’s evidence that GPT-4 only excels on tests that happened to appear in it train data (presumably via accidental contamination). As has been demonstrated by it strong ability to solve tests written before its data cutoff date (Sep 2021), but struggles with equivalent tests written after that date. twitter.com/cHHillee/status/16357903308… towardsdatascience.com/the-decontaminat…... aisnakeoil.substack.com/p/gpt-4-and-pro…...

Ok, so Sam Altman disagreed with AI doomers, great, but the point is still generally valid, for a couple of reasons: 1. What about Elon Musk and hundreds of other AI investors? It's in their interest to overhype AI, while temporarily slowing down competition by spreading singularity fears. 2. OpenAI released the GPT4 report where they claim better performance of their model than it's in reality [1]. [1] twitter.com/cHHillee/status/16357903308…

Their "own" ideas? Let me remind you that OpenAI released a report purposefully suggesting that GPT4 has relatively high IQ, passes a lot of college-level tests, and solves coding problems. Then it was revealed that there was training data contamination that led to good results in such tests [1], but GPT4 marketing received 10000 more attention than truth anyway. The popular belief is that using LLMs will give you a professional competitive advantage. Also, when we talk about the achievements of LLMs, then we anthropomorphize, but when we talk about their failures, then we don't anthropomorphize, i.e., "AI cannot lie"? Don't you see human biases drive AI hype? In my opinion, people clearly are confused and misled by marketing and this isn't the first time it's happening. For instance, people were confused for 40+ about global warming, among others due to greenwashing campaigns [2]. Is it ok to mislead in ads? Are we supposed to purposefully take advantage of others by keeping them confused to gain a competitive advantage? [1] twitter.com/cHHillee/status/16357903308… [2] en.wikipedia.org/wiki/Global_Climate_Co…

twitter.com/cHHillee/status/16357903308…

“Reflection-Based GPT-4 Agent is State-of-the-Art on Code Gen Iteratively refines code, shifting “accuracy bottleneck” from correct code gen to correct test gen HumanEval accuracy: -Reflexion-based GPT-4 88% -GPT-4 67.0% -CodeT 65.8% -PaLM 26.2%” with link to code in the Tweet: mobile.twitter.com/johnjnay/status/1639…... 21% improvement after adding a feedback loop and self-reflection to GPT-4, which just went public 12 days ago. (The approach is based on a preprint published 4 days ago.) Human coders often need a feedback loop and self-reflection to properly “generate” code for problems novel to them as well. ----- A larger question: Are we hurling ourselves toward a (near) future of unaligned AGI with self-improvement capabilities?

Someone asked GPT-4 to build a complete app from scratch. It's now on the store. He seems to apply good prompt engineering techniques. Screenshots, video demo, and process here: mobile.twitter.com/mortenjust/status/16…... It seems plausible to me now that a junior developer position would be hard to find in 2-3 years (I thought it would be ~5 years).

I feel like we're missing the forest for the trees here. I'm not really asking it to add two numbers. I'm asking it to apply a simple set of logical rules on two symbols with an operation '+'. Of course it can generate code to add two numbers, because the code to add two numbers exists in its training data. What transformers seem to be good at is generalizing over the training data. Where they seem to perform poorly is data outside of the training set. Binary numbers are interesting because you can add a bit of entropy just by increasing the size of your binary number, so it's very easy to generate examples outside of the training data. You also see this, albeit less directly, when trying to ask it to generate code that wasn't part of its training data. E.g. twitter.com/cHHillee/status/16357903308…

Presented with no comment :) twitter.com/chhillee/status/16357903308…...