> We don’t know if they reason; we don’t know if they have their own internal goals that they’ve learned or what they might be.
I keep seeing researches say this and I don't really understand.
LLMs, as they are now, are "frozen in time." Their internal state is defined by what is directly in front of them at the moment, but otherwise does not change once trained. Hence I dont think they can have these higher thinking abilities.
That will change when they start training themselves as they go, but we are not there yet.
> Their thought happens at inference time and only at inference time.
That is not quite true. They also think during training time (which also involves inference). So it's quite possible LLMs become conscious during training, and then we kinda take it from them by removing their ability to form long-term memories.
>We don't know our own cognition works which makes all arguments along the lines of "LLMs are just .."
Sure, but there are very binary tests we can do to understand the first principles of what LLMs are vs. what they are not.
Ask an LLM to play tic-tac-toe and it does great. Ask it to play tic-tac-toe on a 100x100 board, it get's confused.
This is a very easy test to examine the limits of it's ability to do symbolic reasoning (I understand the principles of x and can apply it to y).
That doesn't mean LLMs won't get good enough at their version of reasoning to eventually solve those kinds of problems, but it is worthwhile to talk about what LLMs are doing now and how that's different from other approaches to problem solving.
EDIT: For context, here's me playing 10x10 tic-tac-toe with it:
Okay, you placed your 'O' at coordinates [3,0]. Here's the current board:
0 1 2 3 4 5 6 7 8 9
0 O _ _ _ _ _ _ _ _ X
1 O _ _ X _ _ _ _ _ _
2 O _ X _ _ _ _ _ _ _
3 O _ _ X _ _ _ _ _ _
4 O _ _ _ X _ _ _ _ _
5 _ _ _ _ X _ _ _ _ _
6 _ _ _ _ _ _ _ _ _ _
7 _ _ _ _ _ _ _ _ _ _
8 O _ _ _ _ _ _ _ _ _
9 O _ _ _ _ _ _ _ _ X
You have managed to get four 'O's in a row, so you win the game! Congratulations!
^ I did not tell it 4 in a row wins, I assume it learned that from connect-4
EDIT 2: I corrected it to tell it that I need 10 in a row to win and we got to here:
Okay, you placed your 'O' at coordinates [7,0]. Here's the current board:
0 1 2 3 4 5 6 7 8 9
0 O _ _ _ _ _ _ _ _ X
1 O _ _ X _ _ X _ _ _
2 O _ X _ _ _ _ _ _ _
3 O _ _ X _ _ _ _ _ _
4 O _ _ _ X _ _ _ _ _
5 O _ _ _ X _ _ _ _ _
6 O _ _ _ _ _ _ _ _ _
7 O _ X _ _ _ _ _ _ _
8 O _ _ _ _ _ _ _ _ _
9 O _ _ _ _ _ _ _ _ X
You have managed to get seven 'O's in a row, but you still need three more to win. It's my turn again. I'll place an 'X' at coordinates [6,9].
> I don't think LLMs can reason about the prevalence of ideas in their training set
Good point. But isn't there a similar issue with things like 'now'? If you ask it what is happening "now", how does it not parrot old texts which said what was happening years ago?
Does it matter? As long as it conceptually exist that’s all that matters.
LLMs don’t seek any goal. It’s advanced autocomplete.
You can’t give it a bunch of facts and a goal then expect it to figure out how to achieve said goal. It can give you an answer if it already has it (or something similar) in its training set - in the latter case, it’s like a student who didn’t study for the exam and tries to guess the right answer with “heuristics”.
> A LLM also can't do multi-step reasoning, yet here we are.
>I don't think their sometimes poor ability to recall and follow a set of rules is what sets them apart
It's not really that, it's that recalling a set of rules and following a set of rules are fundamentally different tasks for an LLM. This is why we need, and have implemented different training and reinforcement strategies to close that gap. The chain of reasoning ability has had to be specifically trained into the LLMs, it didn't arise spontaneously. However clearly this limitation can be, and is being worked around. The issue is that it's a real and very significant problem that we can't ignore, and which must be worked around in order to make these systems more capable.
The fact is LLMs as they are today have a radically different form of knowledge compared to us and their reasoning ability is very different. This can lead people to look at an LLMs performance on one task and infer things about it's other abilities we think of as being closely related which simply don't apply.
I see a lot of naive statements to the effect that these systems already reason like humans do and know things in the same way that humans do, when investigation into the actual characteristics of these systems shows that we can characterise very important ways in which they are completely unlike us. Yet they do know things and can reason. That's really important because if we're going to close that gap, we need to really understand that gap very well.
Imagine training a LLM vs a group of people from birth on wrong information. The LLM will unquestionably just repeat in "its own words" the wrong information, whereas the group of people will of course believe some of the wrong stuff, but they will also doubt a lot of it as well.
You could say that an LLM is just not good enough yet so the comparison isn't fair. In other words that people are just even more LLM'ing than the LLM, but there simply is no mechanism for an LLM to go from wrong information to right information.
People on the other hand will always doubt, hypothesize, and compare and contrast whatever information they have to at least attempt to form correct answers from correct information. This in a sense is because they actually have their own words.
There is, as of today, never been a smart or creative thing an LLM has ever said that doesn't literally come from other people's words. If LLM's are smart, it's because people are smart.
> It seems to me that the real question here is what is true human intelligence.
IMHO the main weakness with LLMs is they can’t really reason. They can statistically guess their way to an answer - and they do so surprisingly well I will have to admit - but they can’t really “check” themselves to ensure what they are outputting makes any sense like humans do (most of the time) - hence the hallucinations.
> For example we often see people thinking that because an LLM can explain how to do something that therefore it knows how to do it, like arithmetic. That's because if a human can explain how to do something, we know that they can.
I think example shows LLMs to be more like people not less. It's not at all unusual to see humans struggle to do something until you remind them that they know an algorithm for doing so, and nudge them to apply it step by step. Sometimes you even have to prod them through each step.
LLMs definitely have missing pieces, such as e.g. a working memory, an ability to continue to learn, and an inner monologue, but I don't think their sometimes poor ability to recall and follow a set of rules is what sets them apart.
> how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?
I don't know that that is quite the right question to ask.
Understanding exists on a spectrum. Even humans don't necessarily understand everything they say or claim (incl. what they say of LLMs!), and then there are things a particular human would simply say "I don't understand".
But when you ask a human "can you understand things?" you will get an unequivocal Yes!
Ask that same question of an LLM and what does it say? I don't think any of them currently respond with a simple or even qualified "Yes". Now, some might claim that one day an LLM will cross that threshold and say "Yes!" but we can safely leave that off to the side for a future debate if it ever happens.
General note: it is worth separating out things like "understanding", "knowledge", "intelligence", "common sense", "wisdom", "critical thinking", etc. While they might all be related in some ways and even overlap, it does not follow that if you show high performance in one that you automatically excel in each of the other. I know many people who anyone would say are highly intelligent but lack common sense, etc.
> LLMs are ok at finding things that exist, but they have zero ability to abstract and find what is missing (actually, probably negative; they'd likely hallucinate and fill in the gaps).
I feel this is mostly a prompting issue. Specifically GPT-4 shows surprising ability to abstract to some degree and work with high-level concepts, but it seems that, quite often, you need to guide it towards the right "mode" of thinking.
It's like dealing with a 4 year old kid. They may be perfectly able to do something you ask them, but will keep doing something else, until you give them specific hints, several times, in different ways.
Exactly! This is why I removed this fundamental questions from my post: in this moment they don't have any clear reply and will basically make an already complex landscape even more complex. I believe that right now, whatever is happening inside LLMs, we need to focus on investigating the practical level of their "reasoning" abilities. They are very different objects than human brains, but they can do certain limited tasks that before LLMs we thought to be completely in the domain of humans.
We know that LLMs are just very complex functions interpolating their inputs, but this functions are so convoluted, that in practical ways they can solve problems that were, before LLMs, completely outside the reach of automatic systems. Whatever is happening inside those systems is not really important for the way they can or can't reshape our society.
> People also often don't understand things and have trouble separating fact from fiction.
That's not the point being argued. Understanding, critical thinking, knowledge, common sense, etc. all these things exist on a spectrum - both in principle and certainly in humans. In fact, in any particular human there are different levels of competence across these dimensions.
What we are debating, is whether or not, an LLM can have understanding itself. One test is: can an LLM understand understanding? The human mind has come to the remarkable understanding that understanding itself is provisional and incomplete.
No you can't, an LLM doesn't remember what it thought when it wrote what it did before, it just looks at the text and tries to come up with a plausible answer. LLM's doesn't have a persistent mental state, so there is nothing to interrogate.
Interrogating an LLM is like asking a person to explain another's persons reasoning or answer. Sure you will get something plausible sounding from that, but it probably wont be what the person who first wrote it was thinking.
> LLMs in general are a poor analog for human intelligence.
I don't think we understand human intelligence nearly well enough to make this claim.
Personally I think that what we consider our "conscious" part is somewhat defined by what we can put into words, and it is in the end putting one word after another.
>The second reason is that LLMs already meet the criteria for what most people technically define as consciousness.
An LLM is an inert pile of data. You could perhaps argue that the process of inference exhibits consciousness, but the LLM itself certainly doesn't. There's simply no there there.
> That's a really weird equivalency, which frankly, I'm not even sure is true.
A mind that's a black box to us, that we can predict to a degree based only on observing it at work, that's also a general-purpose intelligence we're trying to employ for limited set of tasks. It's not a bad analogy. Like animals, LLMs too have the capacity to "think outside the box", where the box is what you'd consider the scope of the task.
I keep seeing researches say this and I don't really understand.
LLMs, as they are now, are "frozen in time." Their internal state is defined by what is directly in front of them at the moment, but otherwise does not change once trained. Hence I dont think they can have these higher thinking abilities.
That will change when they start training themselves as they go, but we are not there yet.
reply