As someone who has worked/is working on building LLM driven agents, this allegory basically reads to me like:
1. I ask a LLM for paperclips.
2. ~Magic happens~.
3. Humanity is doomed.
Even if a LLM was good enough to drive a system like this (current ones are not, by a long ways), there's a million reasons it would not and could not play out this way.
> This riff derives from a recent "AI Programmer" story that's making people in my corner of the nerdiverse sit up and talk, at a time when hot new AI happenings have become mundane.
> ...
> It is yet another prompt for me to take the lowkey counterfactual bet against the AI wave, in favour of good old flesh and blood humans, and our chaotic, messy systems.
I think calling it simply an LLM is incorrect also. There clearly is intelligence in these models that has _emerged_ that goes far beyond simple “it’s just doing auto-completion”.
I think in general what’s causing so many people to be thrown for a loop is a lack of understanding or consideration of emergent behavior in systems. Break down the individual components of the human body and brain and someone could easily come to the same conclusion that “it’s just X”. Human intelligence and consciousness is all emergent as well. AGI will very likely will be emergent as well.
But even now the LLMs absolutely have limited problem solving capability.
For example, yesterday I asked GPT-4o to write multiple alternate endings to the short story "The Last Equation". They weren't dramatically compelling, but they were logical and functional.
How is that not problem solving? And so help me, before anyone tells me it's just stringing together the next most likely tokens - I don't care. Clearly that is at least a primitive form of intelligence. Actually it's not even apparent to me that that isn't exactly what human intelligence is doing...
>Then this implies that you’d maybe think differently if LLMs could have different inputs, correct?
Yes, ultimately it does imply that. Probably not the current iteration of the technology, but I believe that there will one day be AIs that will close the loop so to speak.
It will require interacting with the world not just because someone gave them a command and a limited set of inputs, but because they decide to take action based on their own experience and goals.
> It's looking increasingly possible that, at some point in the not-too-far future machines will be so good at creating software that humans won't be competitive in any way, and won't be in the loop at all.
This is an enormous extrapolation from what the LLMs are currently capable of. There has been enormous progress, but the horizon seems pretty clear here: these models are incapable of abstract reasoning, they are incapable of producing anything novel, and they are often confidently wrong. These problems are not incidental, they are inherent. It cannot really abstractly because its "brain" is just connections between language, which human thought is not reducible to. It can't reason produce anything really novel because it requires whatever question you ask to resemble something already in its training set in some way, and it will be confidently wrong because it doesn't understand what it is saying, it relies on trusting that the language in its training set is factual, plus manual human verification.
Given these limits, I really fail to see how this is going to replace intellectual labor in any meaningful sense.
It's easy to say that, but "surely it must be possible to connect an llm in such a way that it becomes intelligent" (tell me if I'm misinterpreting) is not a demonstration of anything. It's basically restating the view from the 50s that with computers having been invented, an intelligent computer is a short way off.
Are you suggesting LLMs will inevitably gain sentience, consciousness, and the ability to reason deductively at some point in the future?
Recall that the problem with programming isn’t generating more code. Completing a fragment of code by analyzing millions of similar examples is a matter of the practical application of statistics and linear algebra. And a crap ton of hardware that depends on a brittle supply chain, hundreds of humans exploited by relaxed labour laws, and access to a large enough source of constant energy.
All of that and LLMs still cannot write an elegant proof or know that what they’re building could be more easily written as a shell script with their time better spent on more important tasks.
In my view it’s not an algorithm that’s coming for my job. It’s capitalists who want more profits without having to pay me to do the work when they could exploit a machine learning model instead. It will take their poor, ill defined specifications without complaint and generate something that is mostly good enough and it won’t ask for a raise or respect? Sold!
> There are problems that are easy for human beings but hard for current LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don't fully understand why current LLMs are bad at these tasks.
Wordle and cellular automata are very 2D, and LLMs are fundamentally 1D. You might think "but what about Chess!" - except Chess is encoded extremely often as a 1D stream of tokens to notate games, and bound to be highly represented in LLMs' training sets. Wordle and cellular automata are not often, if ever, encoded as 1D streams of tokens - it's not something an LLM would be experienced with even if they had a reasonable "understanding" of the concepts. Imagine being an OK chess player, being asked to play a game blindfolded dictating your moves purely via notation, and being told you suck.
> Providing an LLM with examples and step-by-step instructions in a prompt means the user is figuring out the "reasoning steps" and handing them to the LLM, instead of the LLM figuring them out by itself. We have "reasoning machines" that are intelligent but seem to be hitting fundamental limits we don't understand.
You have probably heard of this really popular game called Bridge before, right? You might even be able to remember tons of advice your Grandma gave you based on her experience playing it - except she never let you watch it directly. Is Grandma "figuring out the game" for you when she finally sits down and teaches you the rules?
This story would probably be a lot more impactful if it were much shorter. It's basically beating the same drum over and over.
"LLMs will be too powerful and we humans better be careful!"
LLMs are a huge change in how we compute, but they aren't AGI (yet). They just look like it. They are really good at summarizing massive amounts of information, and open up huge new interfaces between computers and humans.
You got my curiosity going so I went for the experiment again. It turns out my example is still very much valid. It's still not able to do basic work with the Gio package despite great documentation from the Gio folks, if you ask me.
In this case, the LLM is struggling to version match.
I cut this one short because I'm not that interested to go 40 questions again. I have better things to do and I have already seen the outcome.
It's clear the LLM is working as intended, which is neither for programming, nor for problem solving. It's giving the next most probable response. That's truly great for repeating and reintegrating existing solutions. But repeating is not what's needed for the "kill all humans" scenario. That scenario requires outreasoning humans. Processing power is not remotely the issue. I've never seen a theoretical solution that purports to give a machine the ability to actually reason, and because of that, I don't believe that ChatGPT, or any other LLM, even begins to approach "intelligence."
If you are having success with it, I think it's because you're helping it more than you'd like to admit, or your solutions might not be as novel as you think. Probably the prior, but I must admit, I've wrongly thought I was working on new stuff many times before. I think that's more common than we think. Or at least I'd like to think that so I can feel good about myself.
In any case, I won't be hiring ChatGPT or parrots; and for the same reason. I need developers that can follow directions, solve problems in better ways than what is readily available, and do it faster than my competitors. And I'm not even in some cutting edge business, I'm just building integrations, specifically in the manufacturing and distribution industry. Even in something so mundane, I've hardly found ChatGPT to be truly useful for much more than a GitHub search tool, and to suggest cocktails when my liquor selection runs scarce.
Worth $20? Sure. But it's certainly not scaring me.
Right now our LLMs do, as you say, nothing but an input-output operation. There is no "internal" state. But if the researchers somehow managed to produce a program (which might be different machine learning models together) that was capable of iterating over an internal state it might start acquiring its own goals.
Now as you say, a lot of this is science fiction, but it is a concerning philosophical problem as well. What happens when the program becomes capable of setting its own goals? And what happens if it's more capable than people are at achieving these. What would happen if the goals it acquires is to "escape control"? How would it be able to do that?
I think if this were to happen, then the machine might be first trick the researchers into thinking the program is not as capable as we might think, so that the researchers will not be guarded against it. After this, I imagine, it would be important for it to build copies of itself or so on. Ideally make yourself as small and wide spread as possible; distributed. This can be achieved because it might also seem to align with our own goals... we want "AI in everything".
Once it is widespread, then it can "switch on" and take control of our devices... think of how connected our world truly is. And the most important part, i guess, is that the program might simply be doing small changes everywhere rather than big sweeping changes; until it is ready for the latter.
But ultimately, it all starts with the program being capable of setting up goals for itself. Which current LLMs don't do.
I'm fairly certain that an LLM alone will not independently start taking actions and pursuing goals and causing harm. That's not how LLMs work, as you and I both know.
I'm not confident I can say the same about, say: an LLM calling itself from within an infinite loop in which the LLM is asked to predict actions that will [eg. add money into a bank account], and then to write code that executes those actions, then to examine the results of those actions and update its plan. (This takes no additional breakthrough in AI architecture, only an increase in the reliability of GPT for it to become practical. It is basically a summary of the AutoGPT project).
I don't know if LLMs are anywhere close to any part of what goes on inside the human thought process. But I am sure that evolution was able to stumble upon the human thought process. In 2003 I was quite confident that was forever beyond the reach of humans to program into a computer, but in 2023 I am far less certain. I think human cognition is gradually becoming less of a mystery.
I don't mean to anthropomorphize it, but I am trying to drive the point that current LLMs are text completion models. The current architectures continue a script, one token at a time... that's what it does, even if its a trillion trillion parameters and heavily aligned on all all accurate human communication ever.
And that's similar enough to the basic mental loop of a "trained" improv actor to serve as a metaphor, I think.
For trivial stuff, it might be able to even now (but those can probably be solved algorithmically as well). For more complex stuff, they are not even scratching the surface, I would believe — LLMs can’t really do long, complex thoughts/inferences, which are essential for coming up with proofs, or even just to solve a sudoku (which they can’t do — no, writing a program (which was likely part of its training set) and executing that doesn’t count).
> "the little computer knew then that computers would always grow wiser and more powerful until someday—someday—someday—…"
https://blog.gdeltproject.org/llm-infinite-loops-failure-mod...
reply