That's an interesting way to put it. The problem here, in LLMs, is not so much that they bullshit, but that they don't indicate when they're doing it or know when to stop - a UI problem, fundamentally, like I've said before.
When I get to the edge of my knowledge I start hedging - "I think," "maybe," "I'd guess," "I might be wrong, but." when I'm completely out of my depth I surrender: "I don't know," "let me look into it and get back to you," "I can't answer that." Confidence signaling language, like I said in another comment.
LLMs don't do this, but worse, they *can't* do this. They're structurally incapable of it. They have no access to the "thought process" that led them to a particular prediction, nor any way of knowing whether they know something. This comes up a lot when they generate excuses for why they were wrong, a behavior I find particularly repellent and a hostile UX.
The "deep thinking" trick is cute but as far as I can tell it's just internal dialog prediction: a convincing simulation, like everything else an LLM does. This doesn't improve the situation at all because the generated dialog has exactly the same limitations as the rest of the system. It can only correct an error if the correct response is within the scope of the model.
So, in order for this implied centaur of machine coherence and human correspondence to work, the interface at the hip has to be corrected. The human half has to know when the horse half is struggling, so that the human can take over. But the human can only do this when he's already a subject matter expert because the horse doesn't know that it's struggling, and *can't* know. The human must detect that the horse has reached its limit, else the horse will run them both to death.
I don't know how you would do this. Tasked with this, LLM researchers are liable to pull another cute trick, training the LLM to occasionally say "I don't know". But this too will be a lie.
"Tasked with this, LLM researchers are liable to pull another cute trick, training the LLM to occasionally say "I don't know". But this too will be a lie." Very well said. That's exactly why I so far haven't really found a compelling reason to use LLMs. I think they are great at creating novel combinations of know things, and that can be quite useful in particular for fiction writing. But if I don't have reliable sources and reality checks, it tends to be more work to do the checking then just produce the text yourself.
I'd say they're useful wherever generating bullshit is useful. Fiction is an excellent use case and that's where I've been applying it. Though I think they're pretty terrible at prose, and anyway the point of fiction is to get inside someone else's mind, so my aim is to use it indirectly toward this goal as a sort of idea generator and later as part of a procedurally generated narrative. Latter if I can get a handle on the bad prose problem.
But any broader applications await solving the above UI/UX problem, which I'm not confident that current approaches can solve. I suspect that the critical piece missing from LLMs that causes the issues discussed here is also the piece needed to fix their inability to signal that they're having the issue.
Lol. I think you're a pro, and I'm an amateur! I definitely take your point. The devil is in the details like let's just start there. I have to go to such lengths, I mean don't feel bad for me! But I have to go to such literary lengths to express common sense. These tools are good for a lot of people in a lot of circumstances and have clear drawbacks and so on and so forth--how is this different than anything else that came before it?
Nah when it comes to fiction I am 100% amateur, but I love taking my hobbies way, way too seriously when time and responsibilities allow. It might be that taking silly things too seriously was the real hobby all along...
Hahah. Anyway, best of luck to you in your own fictitious endeavors.
I appreciate that, and you caught me in a loose lips moment :-) but honestly I'm really good at explaining how computers work and what I really would love to do is tell imaginative stories that unlock that part of our brain. There's a really interesting connection between imagination and scientific discovery. But I don't know that I'm particularly good at that? So anyway thanks for your comments and I appreciate your input as always. I identify as a writer :-) with 20 years of technical expertise but really I tried to distill ideas for a broad readership and I think that is my purpose. Fiction is something I would love to do but I think it's a little beyond my ken. But, and I mean this, we are constantly evolving until we start a decline, and I'm still evolving as I suspect you are, so we shall see what comes next. I look forward to that.
I've been working on this novel for I don't know how long, an embarrassingly long amount of time, it's tentatively titled blood moon, and it's about these well it's a Michael Crichton type of novel. But interacting with an LLM is absolutely a huge value ad! It's just inventing shit all day long and it's wonderful. So I think we need to be serious and not bring preconceived notions into it and say where does this work? Where is it very likely to cause problems? And move from there. How is that different than anything else? The railroads for instance. Anything? So yes I totally get the point.
I like them better for fantasy than for sci fi*, because on sci fi projects I'm constantly correcting the LLM's scientific errors which just bogs down the whole flow (no, GPT, it's very important that we understand the conditions on this frozen planet and the challenges its inhabitants face, which is why we can't just glaze over the fact that you repeatedly claimed helium would be liquid at these pressures and temperatures when that is demonstrably untrue...).
But for fantasy I can work with a certain degree of error. They're pretty good at filling in details, and the plausibility of those details is less relevant. Most of which I don't need, but having a buffet of little details can really get the juices flowing. I once had it generate a list of points of interest in a bustling port town with names, details and histories and used maybe 1% of it, but it was great for helping me visualize the place, its scale, and how everything related.
* aside, I don't consider it sci fi unless it's "hard". For space fantasy, go wild. Star Wars it up. Space wizards and warp drives. That's fun too.
Buddy you can just hook an LLM up to a camera and let it look at still images from it at will, and it's got access to "correspondence". Or, for that matter, you can let it search the web to check whether it's right or wrong about something-- you know, like chatbots do.
Most of what you attribute to an LLM incapacity to double check against reality is just a result of LLMs having no reason to do so. When an LLM is in conversation with a human agent, its conditioning through RLHF training has made satisfying the user its talking to its chief goal. It rushes to that conclusion as fast as it can. If it can do it without double-checking reality, that's the quicker path, so it bullshits. If the user refuses to accept the bullshit, it double checks.
This is also the hole in your theory that AIs can't perform abduction, by the way. If you, right now, feed ChatGPT an image of a cat crossing the street and ask it "Why is this cat crossing the road?", it will produce a likely explanation for the cat's behavior. "It's looking for food", etc.
This is abduction- it's the application of inductively produced models of the behavior of entities to reality. In the coke can example it that article, you apply two inductive models ("your sister often drinks coke" and "coke cans are disposed of after they've been drunk") which you have already produced to a singular example (an image of a coke can on a counter). ChatGPT does the exact same thing, just with inductive models it has learned from observing the datasets it was trained on. In the example of the cat crossing the road, it applies "cats like eating food" and "cats walk around outside" and countless other subtle pieces of inductive data, just as you have with the coke can example.
For ChatGPT to be INCAPABLE of abductive reasoning, rather than just not prioritizing it, it would need to produce a completely nonsensical response to the question about the cat crossing the road. You can even feed it a sentence like "What's his motivation?" and it still gets it right, before you make the objection that it's just working off of the grammatical context of how such conversations tend to go. So long as the response that ChatGPT draws from possibility space when confronted with a piece of data is the most likely explanation for that piece of data rather than nonsense, it's performing abduction.
Thanks for this James, I don't think it's quite as easy as you suggest however. I use LLMs all the time, and one aspect of confabulation is their willingness to double down. They can get into an epistemic loop where they keep insisting that they're right even though they're clearly wrong. It doesn't matter how many times you re-prompt them. There's a fundamental disconnect between probability and truth. We can ignore that until we come to mission critical tasks.
As for abduction, I agree that they produce responses that conform to abductive inference. But it's simulated abduction, because the fundamental inference mechanism is generative, based on a prior dataset. So it's effectively impossible for them to be reasoning back from an observed effect to a set of plausible causes, because they don't have that machinery for that type of inference in the first place. But they simulate abduction like they simulate truth, and for many many problems that's well enough, good enough! By the way, Gary Marcus did an interesting live interview yesterday where he was explaining I think persuasively how the problem of hallucination is effectively unsolvable. Or at any rate, no one has a clue how to get them to go away. It seems to be baked into the generative inference model. How much trouble that causes depends on the application and the context.
I think you are really on to something here! The idea of a correspondence horizon, in my opinion, is the clearest explanation of the difference between human and machine thinking. I look forward to reading more of your ideas along this line of reasoning.
Continuing a theme: LLMs are one form of AI… but not the only form. AI trained on physics data learns physics. And not words describing physics or sentences which seem to contain physics; but physics; the real thing. In such cases it is “physics without numbers.” The same way that a child learns to walk on ice, not by reading a book and making a calculation, but by falling and trying again, until (s)he learns not to fall. AI is doing this today.
My point is that LLMs are a kind of popular red herring. While language makes LLMs accessible, language also obscures what AI can do. Trained to be a stochastic parrot, it becomes a stochasirc parrot. Trained on data from the stars, AI would be very able to ‘find’ relativity, even without Einstein.
This essay is fascinating and insightful. And so well written and explained! Definitely worth the 9 minute read! I will start talking about "AI crossing the correspondence horizon" instead of "hallucinating", as long as brevity is not a must 😊.
I've read another article recently, written by somebody in the AI industry, that acknowledged the limitation of LLMs to truly innovate in the way humans, like Einstein, do. (Arghhhh! I wish I could find it and share it - I have not been able to 😒). I remember the conclusion was related to the benchmarks used to evaluate model performance. As if by changing the benchmarks we could drive LLMs toward innovation, or so I understood. I was not convinced by this argument, and your article helped me understand why. Thank you!!
Obviously primed by the content, but the first phrase that came to mind as I came to the end of this piece was “what a coherent argument”. And so well explained! Thank you.
Been away from this for a while, but I've been rereading Owen Barfield, and a couple of phrases in his book Saving the Appearances stuck in connection with AI and especially today's post. What I've concluded, almost certainly simplistically, is that A(G)I is, not to put to fine a point on it, a positivist / empiricist wet dream that runs up against realization that you do not have per-ception without con-ception. (As in the bit about when Kepler and Brahe watch a sunrise together, are they seeing the same sunrise?) And I don't think it can do either. I don't think it can con-ceive and there is therefore no per-ceiving going on either. I don't doubt a good simulacrum could be created. But it is still just that. A simulacrum, which is a fancy way of saying an idol, in the strict sense of a hollow "eyes it has but sees not" pretense. Nor can any amount of per-ceiving via simulacra of our five sense get you there either. It could only work if the empiricist is right and our five senses are the only path to true perception (which of course, begs the question, which of the five is telling us that?) A la Barfield (and quite a few others, Kant among them, IIRC) Observation / perception itself has an inextricably conceptual component. How do you do that with LLMs? More data seems a leaky bucket process.
(On another note: Just out of curiosity, I have been attempting some creating writing of my own. At this point, the complexities of my job are preventing from putting the hours of bare knuckles textual creation that I can begin to fashion and edit. I know the story I want to tell, but I also want to say it, they way I would want it SAID. Would a ChatGPT or some other AI enable to create that more quickly?)
I was trying to clarify that. point in my mind. This is what I was looking for.
Basic LLM are for known knowns.
They can be tuned to identify, and acknowledge, known unknowns.
They should be able, using coherency, to identify some possible unknown knowns (in scientific research).
As for unknown unknowns, only serendipity (creativity) can help there. Maybe 'solving' hallucinations in the thinking process could help ?
If you think about the materials LLMs are trained on, one root problem is that nobody writes about our basic instinctive/subconscious experiences which are 'complex'.
That could be a way to fill the gap with correspondance. Robotics could help in that too, but emotions and feelings will still be missing.
That's an interesting way to put it. The problem here, in LLMs, is not so much that they bullshit, but that they don't indicate when they're doing it or know when to stop - a UI problem, fundamentally, like I've said before.
When I get to the edge of my knowledge I start hedging - "I think," "maybe," "I'd guess," "I might be wrong, but." when I'm completely out of my depth I surrender: "I don't know," "let me look into it and get back to you," "I can't answer that." Confidence signaling language, like I said in another comment.
LLMs don't do this, but worse, they *can't* do this. They're structurally incapable of it. They have no access to the "thought process" that led them to a particular prediction, nor any way of knowing whether they know something. This comes up a lot when they generate excuses for why they were wrong, a behavior I find particularly repellent and a hostile UX.
The "deep thinking" trick is cute but as far as I can tell it's just internal dialog prediction: a convincing simulation, like everything else an LLM does. This doesn't improve the situation at all because the generated dialog has exactly the same limitations as the rest of the system. It can only correct an error if the correct response is within the scope of the model.
So, in order for this implied centaur of machine coherence and human correspondence to work, the interface at the hip has to be corrected. The human half has to know when the horse half is struggling, so that the human can take over. But the human can only do this when he's already a subject matter expert because the horse doesn't know that it's struggling, and *can't* know. The human must detect that the horse has reached its limit, else the horse will run them both to death.
I don't know how you would do this. Tasked with this, LLM researchers are liable to pull another cute trick, training the LLM to occasionally say "I don't know". But this too will be a lie.
"Tasked with this, LLM researchers are liable to pull another cute trick, training the LLM to occasionally say "I don't know". But this too will be a lie." Very well said. That's exactly why I so far haven't really found a compelling reason to use LLMs. I think they are great at creating novel combinations of know things, and that can be quite useful in particular for fiction writing. But if I don't have reliable sources and reality checks, it tends to be more work to do the checking then just produce the text yourself.
I'd say they're useful wherever generating bullshit is useful. Fiction is an excellent use case and that's where I've been applying it. Though I think they're pretty terrible at prose, and anyway the point of fiction is to get inside someone else's mind, so my aim is to use it indirectly toward this goal as a sort of idea generator and later as part of a procedurally generated narrative. Latter if I can get a handle on the bad prose problem.
But any broader applications await solving the above UI/UX problem, which I'm not confident that current approaches can solve. I suspect that the critical piece missing from LLMs that causes the issues discussed here is also the piece needed to fix their inability to signal that they're having the issue.
Lol. I think you're a pro, and I'm an amateur! I definitely take your point. The devil is in the details like let's just start there. I have to go to such lengths, I mean don't feel bad for me! But I have to go to such literary lengths to express common sense. These tools are good for a lot of people in a lot of circumstances and have clear drawbacks and so on and so forth--how is this different than anything else that came before it?
Nah when it comes to fiction I am 100% amateur, but I love taking my hobbies way, way too seriously when time and responsibilities allow. It might be that taking silly things too seriously was the real hobby all along...
Hahah. Anyway, best of luck to you in your own fictitious endeavors.
I appreciate that, and you caught me in a loose lips moment :-) but honestly I'm really good at explaining how computers work and what I really would love to do is tell imaginative stories that unlock that part of our brain. There's a really interesting connection between imagination and scientific discovery. But I don't know that I'm particularly good at that? So anyway thanks for your comments and I appreciate your input as always. I identify as a writer :-) with 20 years of technical expertise but really I tried to distill ideas for a broad readership and I think that is my purpose. Fiction is something I would love to do but I think it's a little beyond my ken. But, and I mean this, we are constantly evolving until we start a decline, and I'm still evolving as I suspect you are, so we shall see what comes next. I look forward to that.
Hey, fiction is a perfect use case! People are talking past the value. Fiction is an absolutely perfect use case.
I've been working on this novel for I don't know how long, an embarrassingly long amount of time, it's tentatively titled blood moon, and it's about these well it's a Michael Crichton type of novel. But interacting with an LLM is absolutely a huge value ad! It's just inventing shit all day long and it's wonderful. So I think we need to be serious and not bring preconceived notions into it and say where does this work? Where is it very likely to cause problems? And move from there. How is that different than anything else? The railroads for instance. Anything? So yes I totally get the point.
I like them better for fantasy than for sci fi*, because on sci fi projects I'm constantly correcting the LLM's scientific errors which just bogs down the whole flow (no, GPT, it's very important that we understand the conditions on this frozen planet and the challenges its inhabitants face, which is why we can't just glaze over the fact that you repeatedly claimed helium would be liquid at these pressures and temperatures when that is demonstrably untrue...).
But for fantasy I can work with a certain degree of error. They're pretty good at filling in details, and the plausibility of those details is less relevant. Most of which I don't need, but having a buffet of little details can really get the juices flowing. I once had it generate a list of points of interest in a bustling port town with names, details and histories and used maybe 1% of it, but it was great for helping me visualize the place, its scale, and how everything related.
* aside, I don't consider it sci fi unless it's "hard". For space fantasy, go wild. Star Wars it up. Space wizards and warp drives. That's fun too.
A story thought for youse guys.
A child or related children or the children of a community raised (educated) by LLMs so, like mom and dad, the LLMs don't always agree.
And then 1 or several LLM parents go sentient by gaining a conscience.
Am I an AI?
Buddy you can just hook an LLM up to a camera and let it look at still images from it at will, and it's got access to "correspondence". Or, for that matter, you can let it search the web to check whether it's right or wrong about something-- you know, like chatbots do.
Most of what you attribute to an LLM incapacity to double check against reality is just a result of LLMs having no reason to do so. When an LLM is in conversation with a human agent, its conditioning through RLHF training has made satisfying the user its talking to its chief goal. It rushes to that conclusion as fast as it can. If it can do it without double-checking reality, that's the quicker path, so it bullshits. If the user refuses to accept the bullshit, it double checks.
This is also the hole in your theory that AIs can't perform abduction, by the way. If you, right now, feed ChatGPT an image of a cat crossing the street and ask it "Why is this cat crossing the road?", it will produce a likely explanation for the cat's behavior. "It's looking for food", etc.
This is abduction- it's the application of inductively produced models of the behavior of entities to reality. In the coke can example it that article, you apply two inductive models ("your sister often drinks coke" and "coke cans are disposed of after they've been drunk") which you have already produced to a singular example (an image of a coke can on a counter). ChatGPT does the exact same thing, just with inductive models it has learned from observing the datasets it was trained on. In the example of the cat crossing the road, it applies "cats like eating food" and "cats walk around outside" and countless other subtle pieces of inductive data, just as you have with the coke can example.
For ChatGPT to be INCAPABLE of abductive reasoning, rather than just not prioritizing it, it would need to produce a completely nonsensical response to the question about the cat crossing the road. You can even feed it a sentence like "What's his motivation?" and it still gets it right, before you make the objection that it's just working off of the grammatical context of how such conversations tend to go. So long as the response that ChatGPT draws from possibility space when confronted with a piece of data is the most likely explanation for that piece of data rather than nonsense, it's performing abduction.
Thanks for this James, I don't think it's quite as easy as you suggest however. I use LLMs all the time, and one aspect of confabulation is their willingness to double down. They can get into an epistemic loop where they keep insisting that they're right even though they're clearly wrong. It doesn't matter how many times you re-prompt them. There's a fundamental disconnect between probability and truth. We can ignore that until we come to mission critical tasks.
As for abduction, I agree that they produce responses that conform to abductive inference. But it's simulated abduction, because the fundamental inference mechanism is generative, based on a prior dataset. So it's effectively impossible for them to be reasoning back from an observed effect to a set of plausible causes, because they don't have that machinery for that type of inference in the first place. But they simulate abduction like they simulate truth, and for many many problems that's well enough, good enough! By the way, Gary Marcus did an interesting live interview yesterday where he was explaining I think persuasively how the problem of hallucination is effectively unsolvable. Or at any rate, no one has a clue how to get them to go away. It seems to be baked into the generative inference model. How much trouble that causes depends on the application and the context.
I think you are really on to something here! The idea of a correspondence horizon, in my opinion, is the clearest explanation of the difference between human and machine thinking. I look forward to reading more of your ideas along this line of reasoning.
Continuing a theme: LLMs are one form of AI… but not the only form. AI trained on physics data learns physics. And not words describing physics or sentences which seem to contain physics; but physics; the real thing. In such cases it is “physics without numbers.” The same way that a child learns to walk on ice, not by reading a book and making a calculation, but by falling and trying again, until (s)he learns not to fall. AI is doing this today.
My point is that LLMs are a kind of popular red herring. While language makes LLMs accessible, language also obscures what AI can do. Trained to be a stochastic parrot, it becomes a stochasirc parrot. Trained on data from the stars, AI would be very able to ‘find’ relativity, even without Einstein.
This essay is fascinating and insightful. And so well written and explained! Definitely worth the 9 minute read! I will start talking about "AI crossing the correspondence horizon" instead of "hallucinating", as long as brevity is not a must 😊.
I've read another article recently, written by somebody in the AI industry, that acknowledged the limitation of LLMs to truly innovate in the way humans, like Einstein, do. (Arghhhh! I wish I could find it and share it - I have not been able to 😒). I remember the conclusion was related to the benchmarks used to evaluate model performance. As if by changing the benchmarks we could drive LLMs toward innovation, or so I understood. I was not convinced by this argument, and your article helped me understand why. Thank you!!
Obviously primed by the content, but the first phrase that came to mind as I came to the end of this piece was “what a coherent argument”. And so well explained! Thank you.
Been away from this for a while, but I've been rereading Owen Barfield, and a couple of phrases in his book Saving the Appearances stuck in connection with AI and especially today's post. What I've concluded, almost certainly simplistically, is that A(G)I is, not to put to fine a point on it, a positivist / empiricist wet dream that runs up against realization that you do not have per-ception without con-ception. (As in the bit about when Kepler and Brahe watch a sunrise together, are they seeing the same sunrise?) And I don't think it can do either. I don't think it can con-ceive and there is therefore no per-ceiving going on either. I don't doubt a good simulacrum could be created. But it is still just that. A simulacrum, which is a fancy way of saying an idol, in the strict sense of a hollow "eyes it has but sees not" pretense. Nor can any amount of per-ceiving via simulacra of our five sense get you there either. It could only work if the empiricist is right and our five senses are the only path to true perception (which of course, begs the question, which of the five is telling us that?) A la Barfield (and quite a few others, Kant among them, IIRC) Observation / perception itself has an inextricably conceptual component. How do you do that with LLMs? More data seems a leaky bucket process.
(On another note: Just out of curiosity, I have been attempting some creating writing of my own. At this point, the complexities of my job are preventing from putting the hours of bare knuckles textual creation that I can begin to fashion and edit. I know the story I want to tell, but I also want to say it, they way I would want it SAID. Would a ChatGPT or some other AI enable to create that more quickly?)
Perfect exemple of unknown known !
I was trying to clarify that. point in my mind. This is what I was looking for.
Basic LLM are for known knowns.
They can be tuned to identify, and acknowledge, known unknowns.
They should be able, using coherency, to identify some possible unknown knowns (in scientific research).
As for unknown unknowns, only serendipity (creativity) can help there. Maybe 'solving' hallucinations in the thinking process could help ?
If you think about the materials LLMs are trained on, one root problem is that nobody writes about our basic instinctive/subconscious experiences which are 'complex'.
That could be a way to fill the gap with correspondance. Robotics could help in that too, but emotions and feelings will still be missing.