GPT-4 is a magnitude larger and not a magnitude better. Even before that, GPT-3 was not a particularly high watermark (compared to T5 and BERT) and GPT-2 was famously so expensive to run that it ran up a 6-figure monthly cloud spend just for inferencing. Lord knows what GPT-4 costs at-scale, but I'm not convinced it's cost-competitive with the alternatives.
They’re extrapolating from the performance of GPT-3.5. It’s speculative, but not anecdotal. GPT has improved rapidly over time, so it's not a huge leap to predict that GPT-4 will be even better.
GPT4 is also much more expensive to use, even at the probably-running-at-a-loss pricing. Accuracy per dollar and/or joule doesn't seem to be increasing too much.
I pay for it because I can afford it, but honestly I haven't been super impressed.
GPT-4 is definitely better at some tasks and moderately less prone to hallucination, but it botched the first couple scripting tasks I gave it, so it didn't exactly wow me. The main difference I've observed is that it gives wordier responses than GPT-3.5.
That said, I might have a different opinion if usage of GPT-4 weren't so severely rate-limited. Currently, you can only send 25 requests every 3 hours, so it's hard to experiment with it as freely as I want.
GPT-4 still confidently makes up sources for wrong answers and throws subtle mistakes (the obvious mistakes aren't as big a nuisance) into output.
This isn't to say gpt-4 isn't cool or impressive or a development to watch and learn about and be excited about, but I frequently see criticism dismissed as "you must be using 3.5" while I find 4 still costs more time than it would have potentially saved.
Every so often I go back to GPT-3.5 for a simpler task I think it might be able to handle (and which I either want faster or cheaper), and am always disappointed. GPT-3.5 is way better than GPT-3, and GPT-4 is way better than GPT-3.5.
GPT4 is more difficult to measure I think. The value I get from GPT4 is in the details it gets right on very obscure, complex questions. I'm not sure benchmarks are capturing how far GPT4 is ahead of other models. For simple stuff it's not that much better than 3.5.
In the llm-assisted search spaces I'm involved in, a lot of folks are trying to build solutions based on fine tuning and support software surrounding 3.5, which is economical for a massive userbase, using 4 only as a testing judge for quality control.
In my experience so far, GPT-4o seems to sit somewhere between the capability of GPT-3.5 and GPT-4.
I'm working on an app that relies more on GPT-4's reasoning abilities than inference speed. For my use case, GPT-4o seems to do worse than GPT-4 Turbo on reasoning tasks. For me this seems like a step-up from GPT-3.5 but not from GPT-4 Turbo.
At half the cost and significantly faster inference speed, I'm sure this is a good tradeoff for other use cases though.
Ultra benchmarked around the original release of GPT-4, not the current model. My understanding is that was fairly accurate — it's close to current GPT-4 but not quite equal. However, close-to-GPT-4 but 4x cheaper and 10x context length would be very impressive and IMO useful.
reply