The central claim is that a machine which answers exactly the same thing a human would answer given the same input does not have understanding, while the human does.

This claim is religious, not scientific. In this worldview, "understanding" is a property of humans which can't be observed but exists nonetheless. It's like claiming humans have a soul.

I'm not sure that's the case. Simple chat programs can do a pretty good job simulating a human interlocutor. For a 'full' simulation, which we need by definition, we'd need something much more sophisticated (able to reason about all sorts of abstract and concrete things), but conceivably the solution might be very different from brain-simulation.

The thought experiment can easily be adjusted to close the door on my objection here: rather than a man in a room with an enormous card index, we have a pretty accurate real-time computer simulation of some specific person. That way the computational problem is defined to be equivalent to something we consider conscious. Laboriously computing that simulation by hand (presumably not in real time but instead over millennia) would change the substrate, but not the computational problem. If we're ok with there being an outer host consciousness and an inner hosted consciousness, the thought experiment poses no problem.

Of course, this isn't the position I started at, but it makes some sense that the real meaning of the thought experiment change as we adjust the simulated process. If our man were looking up the best moves to play tic-tac-toe, it would be plainly obvious that we're looking at competence without comprehension. If he's instead simulating the full workings of a human brain, the situation is different. The foreign language problem is somewhere between these extremes.

> It's a good part about artificial algorithms that they are transparent and we can show that they aren't only effective in conversation, but have everything to be had behind that conversation. It's due to the latter. Simply put, the algorithm isn't GPT, but AGI.

I'm not sure I quite follow you here. I agree that the depth and detail of the simulation is an important factor.

> What's new is that the life of virtual people is shown at length and discussed questions as to what identity those people should have, what worldview, religion, philosophy, pride, dignity, justice.

In contrast to Ex Machina that had the computer as a sociopathic villain with only surface level feigning of normal human emotion and motivation.

While we're vaguely on the topic, homomorphic crypto also puts a spin on things. We know it's possible for a host computer to be entirely 'unaware' of what's going on in the VM that it's running, in a cryptographic sense. Related to this, I've long thought that there's a sticky 'interpretation problem' with consciousness (perhaps philosophers have another term for it) that people rarely talk about.

If you run a brain simulator inside a homomorphically encrypted system, such that no one else will ever know what you're running in there, does that impact whether we treat it as conscious? Part of it is that the simulated brain isn't hooked up to any real-world sensors or actuators, but that's just like any old brain in a jar. Philosophically pedestrian. This goes far beyond that. Someone could inspect the physical computer, and they'd have no idea what was really running on it. They'd just see a pseudorandom stream of states. If there's consciousness inside the VM, it's only there with respect to the homomorphic crypto key!

If we allow that to count as consciousness, we've opened the door to all sorts of computations counting as consciousness, if only we knew the key. We can take this further: we can always invent a correspondence such that any sequence of states maps to a computation stream that we would identify as yielding consciousness. This looks like some kind of absurd endgame of panpsychism, but here we are.

Is there an alternative? I'm increasingly of the opinion that it seems like a non-starter to try to deny that transistor-based computers could ever be the substrate of consciousness. Short of that, where else is there to go?

</sidetrack>

I think what it means to understand language is to be able to generate and react to language to accomplish a wide range of goals.

Neural nets are clearly not capable of understanding language as well as humans by this definition, but they're somewhere on the spectrum between rocks and humans, and getting closer to the human side every day.

I can't help but think that arguments that algorithms simply don't or can't understand language at all are appealing to some kind of Cartesian dualism that separates entities that have mind from those are merely mechanical. If that's your metaphysics, then you're going to continue to find that no particular mechanical system really understands language, all the way up to (and maybe beyond) the point where mechanical systems can use and react to language in fully all situations that humans can.

If the AI was just taught the relationship between the letters of each language and never the translation between them, the only way to explain the concept would be to understand them no?

[1] http://web.stanford.edu/class/cs224n/

I really don't see it like that—it has very little memory, it has no ability to introspect before "choosing" what to say, no awareness of the concept of the coherency of statements (i.e. whether or not it's saying things that directly contradict its training), seems to have little sense of non-pattern-driven computation beyond what token patterns can encode at a surface level (e.g. of course it knows 1 + 1 = 2, but does it recognize odd notation/can it recognize and analyze arbitrary statements? of course not). I fully grant it is compelling evidence we can replicate many brain-like processes with software neural nets, but that's an entirely different thing than raising it to a level of thought or consciousness or self-awareness (which I argue is necessary in order to appropriately issue coherent statements, as perspective is a necessary thing to address even when attempting to make factual statements), but it strikes me as a lot closer to an analogy for a potential constituent component of a mind rather than a mind per se.

Yes, there are humans who bullshit and bloviate and say things they don't really mean or understand. But that's worlds away from not even having a distinct consciousness that can "understand" anything.

To the best of my knowledge, none of our current AIs have anything that we could meaningfully call consciousness or understanding. The vast majority of them have a set of static "trained parameters" that dictate how they are likely to respond to various stimuli. These parameters are going to be tuned for particular kinds of stimuli. They have no continuous state, no continuous input, and no continuous learning. Every time you submit something to GPT-3, it is just taking that discrete input, sticking it into its neural network, and blindly spitting out what comes out the other end.

I don't know the details of how LaMDA is set up, but I'd guess that it's very similar: it gets discrete inputs and produces outputs that look very much like what a human would say based on them, but it has no internal experience of them. It's not "making decisions", it doesn't have a mood or thoughts or emotions of its own, because that's not how our current generation of AIs work at a fundamental level.

In terms of consciousness and sentience, these AIs aren't just not on the same level as a human: they're not even on the same level as an animal. Even the most basic animal, one that's got no theory of mind or sense of self or anything, is still operating in a continuous feedback loop of input, processing, output, and most animals have enough of what we understand as a mind to be able to continuously learn, and have an internal life of some sort. Watch your dog or cat, and you will be able to see that they have emotions and thoughts, even if they are not thoughts on the same level as our own (most of the time).

AIs have none of this. The entire idea of a current-gen AI "having a clue what it's talking about" is just meaningless, because they have nothing with which to understand it.

Perhaps by having a dataset of symbols that corresponds to symbols that you similarly understand how they can be put together, as well by having a model for how to respond to inputs given some prompts.

You can fill a SQL database with different kinds of apples and their prices.

You can "ask" the price of a Golden Delicious apple, and the DBMS responds--intelligently, with the "right" answer given the question asked in semantic business language.

How could the DBMS do that, if it didn't "understand" the data?

The answer is, the machine system contained a system of symbols and a model for how to give you want you want. But the system didn't utilize any kind of first principles in its "understanding".

I hypothesize there is a "true sense of understanding" or "understanding before anything else" that machines can mimic but that--currently--only humans are good at (even with LLM advances). At least, I haven't seen any evidence to the contrary.

As an aside, if there really were an LLM that could understand, reason, "think" (so that it "understood" with an "inherent representation" to "encode an understanding of our world"), then it would, by all likelihood, be spouting some disruptive-seeming output (in the best sense possible--as in, output supportive of paradigm shifts from status quo positions in place for no reason other than momentum) across domains, and I'm just not seeing that. (Sure, the machine might be coerced to APPEAR to do such things, but that would be an illusion, and not doing it on its own--not true intelligence analysis.)

Your attempt to trivialize it doesn't make any sense. It's like watching someone try to trivialize the moon landing. "Oh all we did was put a bunch of people in some metal cylinder then light the tail end on fire. Boom simple propulsion! and then we're off to the moon! You don't need any intelligence to do that!"

>I'm saying that it "understands" your query only insofar as its words can be tied to the web of associations it's memorized. The impressive part (to me) is that some of its concepts can act as facades for other concepts: it can insert arbitrary information into an HTML document, a poem, a shell session, a five-paragraph essay, etc.

You realize the human brain CAN only be the sum of it's own knowledge. That means anything creative we produce anything at all that comes from the human brain is DONE by associating different things together. Even the concept of understanding MUST be done this way simply because the human brain can only create thoughts by transforming it's own knowledge.

YOU yourself are a web of associations. That's all you are. That's all I am. The difference is we have different types of associations we can use. We have context of a three dimensional world with sound, sight and emotion. chatGPT must do all of the same thing with only textual knowledge and a more simple neural network so it's more limited. But the concept is the same. YOU "understand" things through "association" also because there is simply no other way to "understand" anything.

If this is what you mean by "reasoning by analogy" then I hate to tell you this, but "reasoning by analogy" is "reasoning" in itself. There's really no form of reasoning beyond associating things you already know. Think about it.

>But none of this shows that it can relate ideas in ways more complex than the superficial, and follow the underlying patterns that don't immediately fall out from the syntax. For instance, it's probably been trained on millions of algebra problems, but in my experience it still tends to produce outputs that look vaguely plausible but are mathematically nonsensical. If it remembers a common method that looks kinda right, then it will always prefer that to an uncommon method.

See here's the thing. Some stupid math problem it got wrong doesn't change the fact that the feat performed in this article is ALREADY more challenging then MANY math problems. You're dismissing all the problems it got right.

The other thing is, I feel it knows math as well as some D student in highschool. Are you saying the D student in highschool can't understand anything? No. So you really can't use this logic to dismiss LLMs because PLENTY of people don't know math well either, and you'd have to dismiss them as sentient beings if you followed your own reasoning to the logical conclusion.

>I mean, it's not utterly impossible that GPT-4 comes along and humbles all the naysayers like myself with its frightening powers of intellect, but I won't be holding my breath just yet.

What's impossible here is to flip your bias. You and others like you will still be naysaying LLMs even after they take your job. Like software bugs, these AIs will always have some flaws or weaknesses along some dimension of it's intelligence and your bias will lead you to magnify that weakness (like how you're currently magnifying chatGPT's weakness in math). Then you'll completely dismiss the fact that chatGPT taking over your job as some trivial "word association" phenomenon. There's no need to hold your breath when you wield control of your own perception of reality and perceive only what you want to perceive.

Literally any feat of human intelligence or artificial intelligence can literally be turned into a "word association" phenomenon using the same game you're running here.

Yes, every time you see something that for human obviously doesn't make sense it makes you dismiss it. You would look at that output differently though if you were talking with a child. Just like a child can miss some information making it say something ridiculous it may miss some patterns connections.

But have you ever observed carefully how we connect patterns and make sentences? Our highly sophisticated discussions and reasoning is just pattern matching. Then most prominent patterns ordered in time also known as consciousness.

Watch hackernews comments and look how after somebody used a rare adjective or cluster of words more commenters tend to use it without even paying conscious attention to that.

Long story short, give it a try and see what examples of what people already did with it even in it's limited form.

To me you are looking at an early computer and saying that it's not doing anything that a bunch of people with calculators couldn't do.

There is a representational / symbolic divide in much of AI. Unfortunately some people think symbolic processes are somehow more "true" or "real" than representational processes. This leads them to say things like "A full understanding of an utterance or a question requires understanding the one and only one thought that a speaker is trying to convey." As if there were such a thing in a jello-like mass of briefly sparking cells.

But symbols are super useful, you're processing them now to understand my thoughts. They are so useful that you see them constantly in human innovations. Math, logic, art, literature ... everywhere! So what is a more useful way to understand the relationship between our gooey protoplasm and the beauty of say the pythagorean theorem? It's the same way to understand the relationship between analog and digital computation. The latter is built on the former.

Your digital (symbolic, supposedly pure) world is built on differences in ranges of voltages in physical circuits. (Or ranges of magnetic polarization, or ranges of light frequencies). We tame a continuously variable world by defining boundaries on continuous ranges and then pretend those are pure symbolic values.

This is the problem the author is wrestling with. They haven't differentiated the computational substrate from the computation. Can we build systems that "understand" using probabilistic function approximation? Sure! That's how we work. But the program running on that messy substrate isn't there yet and that's what's got the author in a tizzy.