Really? I have been doing research on language models in medical diagnostics even before GPT-2, and found that when trained and applied in certain ways, language models (even much smaller than GPT-3!) are very good at diagnosis predictions, they can compete with much more complex symptom checkers at that.
I applied for GPT-3 access on the next day since the application form was available, described my research and experience in detail, but there was no reply.
Now, they gave access to these people at nabla, and they just asked a bunch of stupid questions using top-k random sampling to generate answers and claimed that this debunks something. This study debunks nothing and proves nothing, it is stupid and only done to get some hype from GPT-3 popularity.
Ok, I am sorry for being rude, but I am really upset because I spent years working on this problem using whatever computational resources I could get and obtained some interesting results, and based on these I think that GPT-3 should be capable to do amazing things for diagnostics when used properly. Why won't OpenAI give access to a researcher who wants to do some serious but a bit mundane work, but gives it to people who use it to create hype?
Until then, we shouldn't just assert that it's false either.
GPT-3 is coming quite close to being able to write novel convincing research papers. With GPT-4 and GPT-5 on the horizon I don't see why language models couldn't be significantly involved in winning a Nobel prize in the future. At least I wouldn't categorically exclude that possibility.
This is a rational argument, and I see it a lot, but it does not probe deep enough. GPT-3 use cases "fall apart at the lightest real world case" because our expectations are wrong. For the vast majority of computing history, computers have been used to do things deterministically and accurately. Next-gen AI does not operate in this way. It encroaches on the human part of work where we expect and allow for people to get it wrong.
GPT-3 does shockingly well at classification tasks with basically 0 training/prompting (outside of the base model). And it works for an incredibly broad set of use cases.
But using QA and generation are much harder to judge because we can't say (in general) whether generated text is "correct"
While I disagree with the authors completely, comparing GPT-3 to a working plane is also wrong.
GPT-3 can produce human-sounding pieces of text that don't have any meaning. I am quite certain that it will prove a dead end in the advancement of NLP, simply because trying to learn human writing by matching lots of text is unlikely to create an accurate model of the world that you could use to produce meaningful communication.
The article is a critical view on GPT-3. Fair. It is well known that Gary Marcus is not a fan of the GPT kind of systems. And he does make some valid points.
If you want to look at a better balanced view it actually helps to look at all their prompts [1].
That said, I think it's more of a hype that GPT-3 is moving towards AGI. The actual GPT-3 paper says "Language Models are Few-Shot Learners"[2]. So it's actually surprising that no one has actually done a real analysis of this. Are they really few shot learners? My experiments seem to suggest otherwise.
But for sure, GPT-3 is the best general purpose natural language system out there in the world. I don't think anyone can say otherwise.
I think that's overly dismissive of it. GPT-3 for me represents the very beginnings of breaking out of the paradigm of a specialized machine learning system for each task. GPT-3 is alternatively a (surprisingly good and consistent) machine translating system, a (disappointingly mediocre) calculator, a (very uneven with flashes of brilliance) chatbot, etc. All without special engineering to do any of those things. I can't think of any other system I have access to that does that.
Regardless of whether you think GPT-3 represents a track towards true AGI, this is a huge advance! Even OpenAI's API for it is astounding compared to other APIs. I can't think of any other API that amounts to "provide an English language description of your task" and returns an answer. Like I said, the results are still quite uneven, but the fact that it's not extraordinarily outlandish to provide such an API is absolutely mind-boggling to me.
I don't think I could have ever imagined such an API existing even just 5 years ago!
This is way way beyond even the same qualitative thing as ELIZA.
That's interesting, GPT-3 can do classification too? Or did I misunderstood and you meant your engineers used classification to build a language model that didn't perform as well as GPT-3 (which is less surprising indeed) ?
Respectfully I don't think you read my comment. GPT3 != ChatGPT. ChatGPT is built on GPT-3 and is not breaking new ground. GPT3 is 3 years old and was breaking new ground in 2020 but Meta/Google/DeepMind all have LLM of their own which could be turned into a Chat-Something.
That's the point LeCunn is making. He's not out there negating that the paper you linked was ground-breaking, he's saying that converting that model into ChatGPT was not ground-breaking from an academic standpoint.
It think it is all right except for the A.I. part.
GPT-3 is taking a graph-structured object ("language" inclusive of syntax and semantics) over a variable-length discrete domain and crushing it into a high-dimensional vector in a continuous euclidean space. That's like fitting the 3-d spherical earth onto a 2-d map; any way you do it you do violence to the map.
I think systems like GPT-3 are approaching an asymptote. You could put 10x the resources in and get 10% better results, another 10x and get 1% better results, something like that.
You might do better with multi-task learning oriented towards specific useful functions (e.g. "is this period the end of a sentence?") but the training problem for GPT-3 is by no means sufficient for text understanding.
GPT-3 fascinates people for various reasons, one of them being almost good enough at language, lacking understanding, faking it, and being the butt of a joke.
If GPT-3 were a person with similar language skills and people blogged about that person, mocking it's output, the way we do with GPT-3, people would find that cringeworthy. Neurotypicals welcome it as one of their own, and aspies envy it because it can pass better than they can.
At $2 a page it can replace richmansplainers such as Graham and Thiel who never listen. It's not a solution for folks like like Phillip Greenspun who read the comments on their blogs.
For that matter, it may very well model the mindlessness of corporate America: if you accept GPT-3 you prove you will see the Emperor's clothes no matter how buck naked he is. AT&T executives had a perfectly good mobile phone business: what possessed them to buy a failing satellite TV business? Could GPT-3 replace that "thinking" at $2 a page? Such a bargain.
I smile every time I hear this argument. To me it’s obviously false and head in the sand. There was a moment, with GPT-2 where it felt exactly right, then with GPT-3 where you could see a glimmer of something emerging. Now GPT-4 has something real going on.
A solid example is the tool use. Package up a brand new, never existed before or seen in the training data plug-in, give a few paragraphs of human readable documentation, and suddenly GPT-4 can interact with other systems, databases, and models. It doesn’t have 10,000 examples of interacting with this novel plugin to predict next word off of, it just has a short English description, enough for a human to be able to use and understand the same plugin effectively.
There’s a reason this is knocking the socks off many smart folks. It’s upending a lot of the sacred cows of human intelligence.
So the idea is that statistical language modelling is not enough. You need a model based on logic too for "real" artificial intelligence. I wonder what the evidence for this claim is? Because the inferences and reasoning GPT3 is already capable of is incredible and beats most expert systems that I know of. And GPT4 is around the corner, Stable Diffusion was published like only a few months ago. I don't see why not more compute, more training data, and better network architectures couldn't lead to leaps and bounds of model improvements. At least for a few more years.
Absolutely, because GPT-3 is a super hot rod text prediction algorithm but has strictly zero consciousness or concept of what it's writing about. It's merely using statistics to do more complex versions of stuff like, when words a, b, c, d, and e are used together, words f, g, h, and j are typically used as well. Take this simple kind of idea and push it to the current max et voilà, GPT-3.
GPT3's strength is on language generation, so using *GLUE for evaluating it (where encoder type models are just better) and claiming to have 99.9% less parameters is sensationalism.
Yeah I mean technically GPT 3 was already out for a while and looked impressive, but it was still only accesible to like 10 people and it was difficult to tell if it fell flat outside prearranged demo examples.
In fact given what we know now, it likely wasn't nearly as good as it looked. So was hard to say at that point if this whole language model thing will pan out or become yet another useless learning approach that's unfeasible to apply to anything.
Wouldn't the empirical success of GPT-3 in simple programming tasks itself be evidence against this interpretation?
Furthermore, GPT-3 is only a language model because it is trained on textual data. Transformer architectures simply map sequences to other sequences. It doesn't particularly matter what those sequences represent. GPT-2 has been used to complete images, for example: https://openai.com/blog/image-gpt/
Argh. The hype around GPT is incredible and if I had a nickel for every nonsense claim I'd heard over the past few weeks I'd have $10--which isn't a lot but its enough to be annoying.
GPT-4 is a generative algorithm and large language model. It performs no reasoning, and instead predicts the next token in language. This works okay on average, because most data on the internet has some type of coherency to it, but there is a major gap between "reasoning" and "next token prediction."
Proof: Link to my paper (written back in 2019) and a bit less technical article. http://www.dialog-21.ru/media/4632/tarasovdplusetal-069.pdf https://www.linkedin.com/pulse/language-models-multi-purpose...
I applied for GPT-3 access on the next day since the application form was available, described my research and experience in detail, but there was no reply.
Now, they gave access to these people at nabla, and they just asked a bunch of stupid questions using top-k random sampling to generate answers and claimed that this debunks something. This study debunks nothing and proves nothing, it is stupid and only done to get some hype from GPT-3 popularity.
Ok, I am sorry for being rude, but I am really upset because I spent years working on this problem using whatever computational resources I could get and obtained some interesting results, and based on these I think that GPT-3 should be capable to do amazing things for diagnostics when used properly. Why won't OpenAI give access to a researcher who wants to do some serious but a bit mundane work, but gives it to people who use it to create hype?
reply