The prices on OpenAI's website are listed per million tokens, the cheaper model can read a typical book for less than the cost of that book in paperback, second hand, during a closing down discount sale, and even the fancy model is still in range of a cheap second-hand paperback (just not during a closing-down sale) for reading and just about the price of a new cheap paperback for writing — cheap enough it might as well be free.

Plus, they're an intern at everything all at once. You don't get to hire someone who has a late-degree-level grasp of programming (in every major programming language at the same time) and genetics and law and physics and medicine and who has a business-level grasp of all the most common written languages on Earth — there is no such human.

(If and when we can make an AI, be it an LLM or otherwise, which can act at the level of a senior in whichever field it was trained on, even if that training takes a gigawatt-year to compute per field, and even if inference costs 100 kW just to reach human output speed, it's going to seriously mess up the whole world economy; right now it's mainly messing up the economic incentives to hire people fresh from university while boosting the amount of simple stuff that actually gets done because normal people and not just business can now also afford to hire the weird intern).

> and might even grow to good hires.

Even if they do, will you be the one to hire them?

Also, ChatGPT has so far existed for about as long as I spent on my (paid) year in industry that formed part of my degree. It wasn't called an internship, but it basically was. In that time, the models have grown and improved significantly, suggesting (but not proving, because induction doesn't do that) that they will continue to grow and improve.

I see LLM models not as the end of society, just as new powerful productivity tools.

To the extent that they may come up with novel ideas, they have no ability to compare them against the true state of the world. This is not exactly a limitation of them per se that could be overcome with more computation, so much as just a structural fact about them; they have no loop where they can form a hypothesis, test it, and adjust based on data. It simply doesn't exist.

Which is part of why I keep saying that while I'm less impressed than everyone else is with LLMs, the future AIs that will incorporate them but not simply be an LLM is going to really knock people's socks off. Pretty much all the things people trying to convince LLMs to do that they really can't do are going to work in that generation. I have no idea if that generation is six months or six years away but I wouldn't bet much more than a few years.

It always strikes me as obtuse when I hear about the job-displacement potential of AI within software engineering or 'programming', and I try to understand where this chorus is coming from. To me it's clear that there is this specific animosity towards engineering and related skill-based professions within the business world, and a ravenous desire to replace the skillset en masse (no doubt because of how expensive it is).

It's important to recognise why good SWEs are so expensive (emphasis on good), and I think it in large part simply boils down to how integral the discipline is to just about every facet of the economy, the value creation potential that it has (demand), and the fact that to do it well is challenging, cognitively discerning and demands constant upskilling/learning of complex technical concepts (supply).

There is a big difference between being a 'coder' who can print hello world, and performing at the level which is required to maintain a high-paying role at the likes of big-tech. In any corner of the world outside of the population-skewed tech hubs, you could walk into a room full of people (the masses), everyone of which would be a consumer of tech products in some way, and almost none of which can/will perform the technical roles required to build them.

With the AI tools, I've found chatGPT particularly useful as a supplement to reading technical documentation due to the conversational format, and copilot as a shotgun-approach which sometimes sparks inspiration and gets me to a solution faster. However, the achilles heel of both is the distinct lack of comprehension for what is being output, especially as they lack longitudinal awareness of the desired output. To me this is best illustrated when trying to solve complex leetcode-style questions which are novel or deviate from existing solutions and require the solver to think through the problem at a macro/conceptual scale before authoring a line-by-line solution.

Is it possible that AI can overcome these limitations? Probably. It remains to be seen how far the current GPT paradigms can be pushed, or if further breakthroughs are needed to reach AGI performance (or at there very least, a version of intelligence which more closely resembles human capabilities). It could be as close as the OpenAI optimism suggests, or much further out as experts like Andrew Ng say.

One thing I feel more confident about however, is that when the time comes that it can truely replace a highly-competent SWE, it can also replace the overwhelming majority of professionals, including traditional middle-managers and their ineffable 'leadership' skillsets. I do not see a version of the future where engineering is uniquely displaced while all other professions remained unaffected. I think it's much more likely that engineering is amongst the last to go.

Ultimately, I think the collective attitude should be that we are all in this together, and we have to seriously architect a socio-economic future for humanity where we can all thrive in a world where the concept of employment is a relic of the distant past.

What I can say is, I think there’s a very important disagreement here and it divides nerds into two camps. The first think LLMs can reason, the second don’t.

It’s very important to resolve this debate, because if the former are correct then we are likely very close to AGI historically speaking (<10 years). If not, then this is just a stepwise improvement and we will now plateaux until the next level of sophistication of model or computer power etc is achieved.

I think a lot of very smart people are in the second camp. But they are biased by their overestimation of human cognition. And that bias might be causing them to misjudge the most important innovation in history. An innovation that will certainly be more impactful than the steam engine and may be more dangerous than the atomic bomb.

We should really resolve this argument asap so we can all either breathe a sigh of relief or start taking the situation very very seriously.

Recall that the problem with programming isn’t generating more code. Completing a fragment of code by analyzing millions of similar examples is a matter of the practical application of statistics and linear algebra. And a crap ton of hardware that depends on a brittle supply chain, hundreds of humans exploited by relaxed labour laws, and access to a large enough source of constant energy.

All of that and LLMs still cannot write an elegant proof or know that what they’re building could be more easily written as a shell script with their time better spent on more important tasks.

In my view it’s not an algorithm that’s coming for my job. It’s capitalists who want more profits without having to pay me to do the work when they could exploit a machine learning model instead. It will take their poor, ill defined specifications without complaint and generate something that is mostly good enough and it won’t ask for a raise or respect? Sold!

The hype will die down on LLMs (slowly... all those ML researchers need to justify the sunk-cost of specializing into a very niche, albeit sales-friendly, field).

If it's any consolation, I doubt we'll see progress towards a "technological singularity" until the current crop of career scientists retire into the dirt -- or there is a fundamental change in resource allocation allowing other, more creative types to start experimenting with building out a probabilistic model for:

Human language in -> determination of "what to do"/workflow to run -> run

We're seeing a few startups in this area, but I haven't seen anyone create any useful agent.

(at which point we should maybe be speaking of Applied Computation rather than Computer Science? then again, at systems institutions that Rubicon has already been crossed...)

I find it very easy to solve problems, but tedious to broadly apply solutions across domains. I'm also very sloppy as a programmer, letting my mind wander further into the future problems to the determent of the current task. Having an LLM buddy to codify my thoughts, regularize them and apply them with high precision would make me MUCH more productive.

In the end, it may be that LLMs are simply better programmers than 99.999% of people. But there will always be need for specialists to bridge between the LLM and some other domain and programmers of today will be that bridge.

And if not... then AGI will have eaten us all up to make paper clips anyway.

Despise my bullish sentiment on AI (I won't be surprised by LLMs being PhD level in everything in 2 years), it is entirely possible that the LLM approach has practical limits that mean it will always be a jack of all trades and master of none, that mastery can only be achieved by specialised AI such as AlphaZero etc. that the LLMs can call out to and which themselves are hard to create in new domains.

This could in turn cause another AI winter, even as the current state of the art models are turned into fully open-sourced commodities that can run locally on cellphones.

There will be a lot of wasted effort in pursuit of this unreachable goal (with LLM technology), an effort better spent elsewhere, like solving cancer or climate change, and stealing young and naive people’s minds away from these problems.

Now, people seem to miss or ignore that fact, and I think is a very risky path.

I'd say that ca. 8/10 founders who pitched to me an idea leveraging LLMs completely missed that limitation. An example would be something like using LLMs as a replacement for therapy.

> How this is dismissed because it’s not 100% perfect (might at add, “yet”) is beyond me.

Again, I'm not dismissing them, but the current tech behind GPT or LLaMA has no concept of "correctness". These models don't understand what they're saying and this is not a trivial issue to fix.

> I struggle to see how even the current GPT-4 is any worse than your average human.

Where, how, and what do you mean by worse? I'm pretty sure there are cases where I'd agree with you, but this is a very broad statement.

Regarding computing technology, I think the positive feedback you're describing happened with chip design and vlsi stuff, eg. better computers help design the next generation of chips or help lead to materials breakthroughs. I'm willing to believe LLMs have a macro effect on knowledge work in a similar way search engines, but as you said, it remains to be seen whether the models can feed back into their own development. From what I can tell, gpu speed and efficiency along with better data sets are the most important inputs for these things. Maybe synthetic data works out, who knows.

This is an enormous extrapolation from what the LLMs are currently capable of. There has been enormous progress, but the horizon seems pretty clear here: these models are incapable of abstract reasoning, they are incapable of producing anything novel, and they are often confidently wrong. These problems are not incidental, they are inherent. It cannot really abstractly because its "brain" is just connections between language, which human thought is not reducible to. It can't reason produce anything really novel because it requires whatever question you ask to resemble something already in its training set in some way, and it will be confidently wrong because it doesn't understand what it is saying, it relies on trusting that the language in its training set is factual, plus manual human verification.

Given these limits, I really fail to see how this is going to replace intellectual labor in any meaningful sense.