They're very aware of that. I expect GPT-4 will improve on it, especially given the rumours about it being a big improvement over GPT-3/chatGPT. twitter.com/sama/status/160173129579241…

I'll advocate against it in its current form. I've seen lots of people saying they have been using ChatGPT heavily and have "learned a lot from it". Based on my own experimentation I'm certain these people have swallowed a large number of falsehoods. Even I swallowed one despite actively looking for incorrect statements and posting its responses to friends in a discussion about its capabilities (my friends also swallowed it). It said something so mundanely plausible it didn't even occur to me to look it up until the next day, at which point I found it was a complete fabrication. Even OpenAI's CEO, Sam Altman, says [1] people shouldn't be relying on it for factual queries. It may come to be useful, but currently it's borderline dangerous. [1] twitter.com/sama/status/160173129579241…...

> If you’re looking for factual answers - then you need to verify what comes out of these models. And let’s be honest you should be doing this with any source of information I've seen a lot of statements like this - equating, often flippantly, ChatGPT's (lack of a) relationship with truth to the fact that humans make mistakes, stackoverflow answers are sometimes wrong, etc. It's really not the same thing, though, neither quantitatively (ChatGPT is "wrong" much more often than existing information sources) nor qualitatively (ChatGPT is "wrong" in a different way to existing sources - it is optimised for plausibility and freely mixes correct and incorrect statements in ways highly likely to trip up even the wary). If I google the definition of a word and I see the top result contains a definition visible on the results page and sourced from Merriam-Webster, then no I generally don't need to double check it or even follow the link. I have reasons to believe both that the information did come from that source and that the source is unlikely to be fabricated. If I ask ChatGPT I have no reason to believe anything it says. Here's what I got from ChatGPT for "What is Gell-Mann amnesia?" > Gell-Mann amnesia is a term coined by science writer Michael Crichton to describe the phenomenon where people tend to forget information that contradicts their preconceptions or beliefs. The term is named after physicist Murray Gell-Mann, who won the 1969 Nobel Prize in Physics for his work on the theory of elementary particles. > According to Crichton, Gell-Mann amnesia occurs when people encounter information that contradicts their beliefs, but rather than updating their beliefs to reflect the new information, they simply forget the information and continue to hold onto their preconceptions. This type of cognitive bias can be a significant barrier to learning and can lead to flawed decision-making. The first paragraph of this contains a lot of completely correct factual information: the term was coined by Michael Crichton and named after Murray Gell-Mann, who did win a Nobel prize in that year for that reason. However, the actual statement of what Gell-Mann amnesia means in both paragraphs is completely wrong but utterly plausible. (The real meaning of Gell-Mann amnesia is extremely relevant to the ways some people are currently interacting with ChatGPT.) For this output to be any use to me, I would have to individually Google (or rather Kagi in my case) every individual thing it has said. If I don't I will swallow complete fabrications. Not merely occasionally, but in my experience most times I use it. I've seen people on hn saying they are doing hundreds of ChatGPT "queries" (they are not queries) a day and "learning a lot" from it. I cannot believe these people are spending anywhere near the amount of effort required to properly validate the results, because it would take more effort than just googling the info in the first place. Even Sam Altman says not to use to use ChatGPT as a source of factual information: twitter.com/sama/status/160173129579241…...

> Whether they apply "reasoning" or "extremely multidimensional synthesis across hundreds of thousands of existing solutions" is a question of semantics. I see where you're coming from - if it's good enough that we can't distinguish it, then does any difference really matter? I submit it's fundamentally different. This is essentially the Chinese Room thought experiment [1] or a nice similar metaphor with an octopus from Section 4 of this paper [2]. The trouble is not in its ability but human's interpretation it. Humans see an "avocado chair" and think this AI can invent art and concepts. Producing combinations of existing concepts is not "that hard". Even for combinations that have never existed before. Meanwhile, it's failing at basic tasks: you can find plenty of examples of it failing basic logic, anything with math or arithmetic, a lot of ethics/bias concerns stemming from the training data, etc. I think when we look forward to an AI that "reasons" this is not what anybody would mean. Current AIs are bullshit engines. They are very impressive, and probably even useful. They are a milestone. But they are not reasoning in any meaningful way. And if you look at the math behind them there's really no reason to think they would. So I guess given a methodology that seemingly shouldn't produce reasoning ability, and no evidence that it has so far, sure, maybe scale will magically unlock it. There's always a chance I guess. But it doesn't really seem too sensible. See Sam Altman's take as well on Twitter here [3] [1] en.wikipedia.org/wiki/Chinese_room [2] aclanthology.org/2020.acl-main.463.pdf [3] twitter.com/sama/status/160173129579241…