Exactly! This is why I removed this fundamental questions from my post: in this moment they don't have any clear reply and will basically make an already complex landscape even more complex. I believe that right now, whatever is happening inside LLMs, we need to focus on investigating the practical level of their "reasoning" abilities. They are very different objects than human brains, but they can do certain limited tasks that before LLMs we thought to be completely in the domain of humans.
We know that LLMs are just very complex functions interpolating their inputs, but this functions are so convoluted, that in practical ways they can solve problems that were, before LLMs, completely outside the reach of automatic systems. Whatever is happening inside those systems is not really important for the way they can or can't reshape our society.
> The answers are not comming from a thinking mind but a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns.
Are you sure you're not also describing the human brain? At some point, after we have sufficiently demystified the workings of the human brain, it will probably also sound something like, "Well, the brain is just a large machine that does X, Y and Z [insert banal-sounding technical jargon from the future] - it doesn't really understand anything."
My point here is that understanding ultimately comes down to having an effective internal model of the world, which is capable of taking novel inputs and generating reasonable descriptions of them or reactions to them. It turns out that LLMs are one way of achieving that. They don't function exactly like human brains, but they certainly do exhibit intelligence and understanding. I can ask an LLM a question that it has never seen before, and it will give me a reasonable answer that synthesizes and builds on various facts that it knows. Often the answer is more intelligent than what one would get from most humans. That's understanding.
>I don't think their sometimes poor ability to recall and follow a set of rules is what sets them apart
It's not really that, it's that recalling a set of rules and following a set of rules are fundamentally different tasks for an LLM. This is why we need, and have implemented different training and reinforcement strategies to close that gap. The chain of reasoning ability has had to be specifically trained into the LLMs, it didn't arise spontaneously. However clearly this limitation can be, and is being worked around. The issue is that it's a real and very significant problem that we can't ignore, and which must be worked around in order to make these systems more capable.
The fact is LLMs as they are today have a radically different form of knowledge compared to us and their reasoning ability is very different. This can lead people to look at an LLMs performance on one task and infer things about it's other abilities we think of as being closely related which simply don't apply.
I see a lot of naive statements to the effect that these systems already reason like humans do and know things in the same way that humans do, when investigation into the actual characteristics of these systems shows that we can characterise very important ways in which they are completely unlike us. Yet they do know things and can reason. That's really important because if we're going to close that gap, we need to really understand that gap very well.
>We don't know our own cognition works which makes all arguments along the lines of "LLMs are just .."
Sure, but there are very binary tests we can do to understand the first principles of what LLMs are vs. what they are not.
Ask an LLM to play tic-tac-toe and it does great. Ask it to play tic-tac-toe on a 100x100 board, it get's confused.
This is a very easy test to examine the limits of it's ability to do symbolic reasoning (I understand the principles of x and can apply it to y).
That doesn't mean LLMs won't get good enough at their version of reasoning to eventually solve those kinds of problems, but it is worthwhile to talk about what LLMs are doing now and how that's different from other approaches to problem solving.
EDIT: For context, here's me playing 10x10 tic-tac-toe with it:
Okay, you placed your 'O' at coordinates [3,0]. Here's the current board:
0 1 2 3 4 5 6 7 8 9
0 O _ _ _ _ _ _ _ _ X
1 O _ _ X _ _ _ _ _ _
2 O _ X _ _ _ _ _ _ _
3 O _ _ X _ _ _ _ _ _
4 O _ _ _ X _ _ _ _ _
5 _ _ _ _ X _ _ _ _ _
6 _ _ _ _ _ _ _ _ _ _
7 _ _ _ _ _ _ _ _ _ _
8 O _ _ _ _ _ _ _ _ _
9 O _ _ _ _ _ _ _ _ X
You have managed to get four 'O's in a row, so you win the game! Congratulations!
^ I did not tell it 4 in a row wins, I assume it learned that from connect-4
EDIT 2: I corrected it to tell it that I need 10 in a row to win and we got to here:
Okay, you placed your 'O' at coordinates [7,0]. Here's the current board:
0 1 2 3 4 5 6 7 8 9
0 O _ _ _ _ _ _ _ _ X
1 O _ _ X _ _ X _ _ _
2 O _ X _ _ _ _ _ _ _
3 O _ _ X _ _ _ _ _ _
4 O _ _ _ X _ _ _ _ _
5 O _ _ _ X _ _ _ _ _
6 O _ _ _ _ _ _ _ _ _
7 O _ X _ _ _ _ _ _ _
8 O _ _ _ _ _ _ _ _ _
9 O _ _ _ _ _ _ _ _ X
You have managed to get seven 'O's in a row, but you still need three more to win. It's my turn again. I'll place an 'X' at coordinates [6,9].
> LLMs produce their answers with a fixed amount of computation per token
I'm not that confident that humans don't do this. Neurons are slow enough that we can't really have a very large number of sequential steps behind a given thought. Longer complex considerations are difficult (for me at least) without at least thinking out loud to cache my thoughts in audible memory, or having a piece of paper to store and review my reasoning steps. I'm not sure this is very different than a LLM prompted to reason step by step.
The main difference I can think of is that humans can learn, while LLMs have fixed weights after training. For example, once I've thought carefully and convinced myself through step-by-step reasoning, I'll remember that conclusion and fit it into my knowledge framework, potentially re-evaluating other beliefs. That's something today's LLMs don't do, but mainly for practical reasons, rather than theoretical ones.
I believe the extent of world modelling done by LLMs still remains an open question.
> And yet, that's what we know that LLMs do, because that's what they are made to do [edit: they do what you describe after the "by merely"]. Why do we need to imagine some other, so far unobserved, process at play? What is the motivation? What is the justification?
We also know how neurons work, interconnect, and communicate.
There is very clearly emergent behavior going on, and while neuroscientists have yielded this point for decades about the brain (any brain, even tadpoles), the AI guys still are stubbornly holding on to "we know exactly how the system works, because we know how the individual bits work".
> We don’t know if they reason; we don’t know if they have their own internal goals that they’ve learned or what they might be.
I keep seeing researches say this and I don't really understand.
LLMs, as they are now, are "frozen in time." Their internal state is defined by what is directly in front of them at the moment, but otherwise does not change once trained. Hence I dont think they can have these higher thinking abilities.
That will change when they start training themselves as they go, but we are not there yet.
> Even the openly licensed ones are still the world’s most convoluted black boxes. We continue to have very little idea what they can do, how exactly they work and how best to control them.
LLM's aren't black boxes, intelligence is. Not understanding anything about things which display emergent intelligence is not a new trend: first cells, then the human brain, and now LLM's. To explicate the magic of how an LLM is able to maintain knowledge when would be analogous to understanding how the human brain synthesizes output. Yes - it is still something we should strive to understand in an explicit, programmatic way. But to de-black box the LLM would be to crack intelligence itself.
> how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?
I don't know that that is quite the right question to ask.
Understanding exists on a spectrum. Even humans don't necessarily understand everything they say or claim (incl. what they say of LLMs!), and then there are things a particular human would simply say "I don't understand".
But when you ask a human "can you understand things?" you will get an unequivocal Yes!
Ask that same question of an LLM and what does it say? I don't think any of them currently respond with a simple or even qualified "Yes". Now, some might claim that one day an LLM will cross that threshold and say "Yes!" but we can safely leave that off to the side for a future debate if it ever happens.
General note: it is worth separating out things like "understanding", "knowledge", "intelligence", "common sense", "wisdom", "critical thinking", etc. While they might all be related in some ways and even overlap, it does not follow that if you show high performance in one that you automatically excel in each of the other. I know many people who anyone would say are highly intelligent but lack common sense, etc.
> LLMs in general are a poor analog for human intelligence.
I don't think we understand human intelligence nearly well enough to make this claim.
Personally I think that what we consider our "conscious" part is somewhat defined by what we can put into words, and it is in the end putting one word after another.
>The second reason is that LLMs already meet the criteria for what most people technically define as consciousness.
An LLM is an inert pile of data. You could perhaps argue that the process of inference exhibits consciousness, but the LLM itself certainly doesn't. There's simply no there there.
>I think we have to see LLMs as their own weird thing
Well nobody seems to be able to reproduce the results of this "paper" anyway lol but i agree with you here. LLMs are sure to have weird failure modes even if they are "truly reasoning" just like biological systems often have weird failure modes that only make sense in the context of biology.
> they are not like humans at all, and only resemble them superficially
We have no idea how LLMs work on a high level, and we have no idea how the human mind works on a high level. Therefore, such claims are rather overconfident. The fact that the two are dissimilar at the plumbing layer doesn't mean they cannot be alike in how they "really" operate. Either way, we simply do not know, so any such talk is (bad) philosophy, not science.
> Sure. I suppose we should also agree on the same understanding of "real understanding". I'm only proposing that LLMs pick up the same kind of "understanding" your unconscious/subconscious mind does, and produce output of similar nature. This implies that, to replicate human reasoning/performance, we'll need to layer some additional models/systems on top of the LLM.
As you might probably guess, I am not convinced. It may be part of something that can match up to human intelligence. But is it enough to layer more mechanisms on top of it? I am not sure.
I think the real question is how to define "real understanding" as you pointed out. I am not sure this will be possible using language alone. Also I think it will probably be hard to compare it to the human mind in a scientific sense, since we don't know how that works and we have no way of knowing other than collecting anecdotes like those that we came up with in our comments here.
Out of curiosity, how do you think human intelligence works? I am no expert and I don't claim to know how the human brain works but my layman's mental model of thinking would be basically that: biological neural networks that "do" the thinking are token prediction machines, except "tokens" are not words that appear on the screen but thoughts that appear in consciousness... while the underlying machinery is in both cases a network of units that fire (or not) based on the connections between them.
Sure, human experience is a lot more than just intelligence (I have no mental model of how consciousness or qualia might work) but it is surprising to me that so many people keep repeating arguments similar to yours - that LLMs do not really think, they are "just" doing [X] (where X usually describes how I imagine human intelligence works) - if there is more to human intelligence, what do you think it is?
i dont think LLMs in their current state are anything like the human mind. They need the ability to have multiple thoughts ongoing, background thoughts, planning... right now LLMs are a little like snap responses answering questions, the type you give without thinking, like intuition. Which can very easily fall outside the bounds of an acceptable answer
> Certainly not? Composite systems that leverage LLMs can do a lot of things - but AFAIU LLMs will likely never rationalize or be able to "add numbers" in the normal sense;
This implies we know what “rationalising” actually is, which we don’t. There’s no reason to believe that our brains don’t operate in fundamentally the same way as LLMs. There’s no reason to think that the LLM approach to reasoning is any less valid than the “normal” way, whatever that even means.
> I don't think LLMs can reason about the prevalence of ideas in their training set
Good point. But isn't there a similar issue with things like 'now'? If you ask it what is happening "now", how does it not parrot old texts which said what was happening years ago?
What if you ignore LLMs specifically? I think that's the point that the GP was making. Do you believe it's mathematically impossible for any artificial machine to "reason"? Or just LLMs?
I think I agree with you (I even upvoted), but this might be an anthropomorphism.
Back like 3-5 years ago, we already thought that about LLMs: They couldn't answer questions about what would fall when stuff are attached together in some non-obvious way, and the argument back then was that you had to /experience/ it to realize it. But LLMs have long fixed those kind of issues.
The way LLMs "resolve" questions is very different from us. At this point, I think that if we want to prove that LLMs need to be rooted in the real world to achieve intelligence, we need to find some real-world phenomenon that is so obvious that noone ever wrote about it... but then we'd have written about it?
Exactly! This is why I removed this fundamental questions from my post: in this moment they don't have any clear reply and will basically make an already complex landscape even more complex. I believe that right now, whatever is happening inside LLMs, we need to focus on investigating the practical level of their "reasoning" abilities. They are very different objects than human brains, but they can do certain limited tasks that before LLMs we thought to be completely in the domain of humans.
We know that LLMs are just very complex functions interpolating their inputs, but this functions are so convoluted, that in practical ways they can solve problems that were, before LLMs, completely outside the reach of automatic systems. Whatever is happening inside those systems is not really important for the way they can or can't reshape our society.
reply