Error detection and correction in this context is the question of determining whether a statement is truthful without resort to a true oracle (because if we had a true oracle it would be moot).
This is the hardest problem in logic and philosophy. It remains unsolved. We have systems, like the scientific method, to help us reduce our error rate, but they are slow - much too slow to apply in real time to LLMs - and not 100% reliable.
LLMs further confound our informal systems for detecting error: intuition about about the other's areas of expertise, linguistic and body language tells indicating that he's bullshitting or less confident in a statement. An LLM can be accidentally correct or incorrect about a claim on the *same subject* at different points *in the same conversation* and speak with equal confidence and equal degrees of competence signaling language in both instances.
This is less like a mythical oracle and more like a mythical demon, which might tell you the truth most of the time to gain your confidence and then lie strategically to sabotage you. Except of course the LLM has no such strategy, it's just sometimes wrong and sometimes right with no discernable pattern or frequency.
Anyway, cutting that rant short(ish), my point is: LLMs cannot be treated as flawed oracles. This is a terrible way to think of them and ML in general. We are not equipped to use them this way.
In some contexts it's probably fine, e.g. using computer vision or data/textual analysis to narrow down possible candidates for expert human inspection. Where false positives *and* false negatives are low consequence and every positive is independently analyzed by a qualified human.
But an unqualified/inexpert human is absolutely helpless in evaluating claims from any machine learning system. They *must not be presented as sources of truth*. They should not be presented as search engines, let alone knowledge bases. This will go badly and is going badly.
On other thing here about impact: you can gauge impact in different scenarios, roughly. In legal, for instance, catastrophic errors would involve cases like "Tow vs. Wade," which doesn't exist. Getting a date wrong might in some circumstances be catastrophic and in others minor. But we do have an idea how certain flaws "screw us" and in what way we're likely to be screwed. It's not deterministic, but it's not that way with humans. I agree that humans have epistemic virtues LLMs can never have, but in real world scenarios LLMs are paired with humans. Say the firm Dewey, Cheatum & Howe must generate a thousand page summary. Consider two cases. (1) A team of junior attorneys does it over the course of weeks and (2): A team of junior attorneys and an LLM does it in one day, with the proviso that the content is subject to LLM error. What's the cost to the firm for (1) vs. (2)? In both cases we can assume that even Dewey, Cheatum and Howe can't afford grievous errors. But much of the grunt work has been done by the LLM in the case of (2) and the junior attorneys now need to keyword search and sanity check in other ways the critical parts. My point is not that this will always be better but that over time we use tools in a way that reduces our manual efforts, and LLMs in the end are not likely to be fundamentally different.
Getting off topic, but I think it's funny that the example you came up with involves generating copious amounts of text that most likely nobody cares about.
I wonder if the better solution is something that reduces the need for thousands of pages of legalese in the first place. And this crops up everywhere in all sorts of business cases for LLMs: let them generate all that tedious text for you! Never write a TPS report again. Nobody reads them anyway, and nobody is honest in them, so who cares if it's wrong.
Might be we need to back up and ask, "why are we doing this in the first place, and how might machine learning, or any other software architecture, eliminate the task instead of automating it?"
Nice comment, Fukitol. If I wanted to be impish I'd point out that I never defined what I mean by "flawed" and so technically could get a nice exposition of LLMs out of "flawed oracle." You could, for instance, just take your critique and put that into what I mean by flawed (like "really flawed"). The sense of oracle here is the common sense notion, that you ask some computer program something and you get an answer. I'm not treating this as I would in computational complexity theory. But I take your point. One other quick point, which you caught: since there are no intentions, there are no demons.
Substantively, I'm working on practical strategies for error reduction, and I've noted many times the problem of impact--not all errors have the same impact. You're of course totally right that our lack of access to errors and their impact is profoundly troubling. I've written about this as well. It too bothers me deeply.
Back to practical computer science stuff, take a prompt - response pair we would classify as a request for explanation:
Prompt: Why was Napoleon in London in 1812? In pragmatics and rhetorical theory this is known as a PRESUPPOSITION because it assumes the truth of the proposition that Napoleon was in fact in London in 1812 (he was not).
It turns out that if you have the LLM itself classify prompt response pairs using a controlled vocabular like Rhetorical Structure Theory or Segmented Discourse Representation Theory (using JSON or XML or what have you), so that
<presupposition>the question</presupposition> you have many more cards to play for getting a reduced rate of errors. The LLM itself, looking at the tag when presented with this example again, would be extremely unlikely to give a confabulated response. I have not quantified this but it seems true given me messing around with such examples for a while. The LLMs ability to generate correct tags of its own prompt - response pairs seems to be quite high. Again not quantified but it doesn't seem to ever get them wrong when given the specification in a prior prompt.
Another way out of the epistemic nightmare of LLMs is to have them tag all their prompt responses this way in which case you can run some rules engine over the tagged pairs. For instance, if you have a list of court cases with a <date_constraint>1980-1989</>, so that you don't want to see any cases not in the 1980s, a Python rule-based engine could easily catch these errors and either flag them for the human user or generate a new prompt that forces the LLM to recheck (or both).
I am not yet sure that this approach will be doable, but it does illustrate my broader point that we will develop this or other strategies as well as ones that no one has yet thought of to deal with LLM hallucinations.
My point here is not that you're wrong about anything you said, but that this is an open area of investigation rather than a throw up your hands, let's try to "get rid" of LLMs.
Also, and this is really a separate discussion, but it's not clear to me that the nasty variety of errors we get from LLMs is much different than a part catastrophically failing with some small frequency. ALL engineering has errors, and sometimes these errors are absolutely catastrophic (as in: everyone dies). A failure point and an error are related but no quite the same, which is why I say this may be a separate discussion. But it's worth thinking about. LLMs present a challenge for the somewhat ironic reason that they work so well most of the time. That means that when an error comes, we may not expect it or detect it.
Re: tagging the user prompt before feeding it back to the LLM for inference, I had a similar idea that I've yet to implement, so it's encouraging to hear that you've had some success with it.
I'm only working on structured interactive fiction, so it's mostly acceptable when the LLM produces a bad response (worst case, the user has a rollback feature if it comes up with something truly terrible or nonsensical). But as I get more into the structure side of it (an underlying game engine that tracks world state) it will be more important that it doesn't invent things, e.g. generating dialog for creatures who can't speak. I also need it to reliably pick up on user input that has a game-mechanic consequence, e.g. attacking or trading, thus the idea of analyzing and tagging the prompt before feeding it back to inference.
Anyhow, when I say low consequence errors, fiction seems like an obvious use case where it's okay for the LLM to be incorrect, so that's where I decided to put my energy. Though even in fiction there has to be internal consistency, which is a problem I'd say I have made good progress with.
Re: catastrophic failures, the problem as I see it is that we can normally tell when a machine fails. The car doesn't start, the program crashes, something explodes. With LLMs there is no apparent difference in behavior between an error and a correct response from the user POV. In your example, when the LLM tells the user that Napoleon was in London for a diplomatic conference in 1812, and then confabulates an elaborate narrative about what he did there, who he spoke to, and how the conference concluded... this doesn't look any different from when it answers a historical question correctly.
So the user has to already know the answer to the question he asks in order to detect that something has gone wrong, and if he knows this he's just fooling around, he's not depending on the machine for anything (and so the failure is low consequence).
Obviously other kinds of machine failure can do irreversible damage, and very rarely it's just a freak accident and not due to neglect, misuse or flawed design. But the idea of a system whose main mode of failure is invisible to the intended user really rubs the wrong way.
One approach established in critical air transport is parallel dissimilarity and voting. Off the top of my head, for reliability we might have 3 parallel LLMs trained exclusively on sources in English, Russian and Mandarin respectively.
Here is a follow-on question to your idea that this is the end of fundamental innovation in machine learning. Is this also the end of fundamental innovation in automation technologies?
A lot of our civilizational problems and economic efficiencies are due to social factors that revolve around people's values and the qualities those values engender in human behavior and the build environment. Automation technologies can't really solve these problems. For example, you could built a fantastic infrastructure using driverless electric cars to transport people anywhere quicker and more efficiently than our current system, but you'd have to shred property rights, tear down a bunch of buildings in urban areas and substantially re-imagine land use across the board. Elon Musk isn't Robert Moses.
I would suggest that, perhaps, new innovations will come from a radically different intellectual framework than the underpinnings of machine learning. It won't be a flavor of computer science.
To underscore this, I think that’s a really interesting question: are we reaching the end of automation? I don’t have a ready-made answer for that. I have to think about that more. My knee-jerk reaction is automation will continue to get better, but it’s going to follow the same trajectory of tweaks followed by smaller tweaks, and so on. But I have not thought about it. So I don’t want to extend too far in that direction.
I don't want to imply that we won't continue to improve various automated technologies or processes. I have no doubt that we will. Nor will we abandon much of the automation that has already been developed. But my question is whether or not we'll a) devise any new fundamental approaches to automation, and b) spend so much capital and cultural attention on developing new (and generally more complex) means of automation.
The case with ML is a bit different from automation but it's an interesting question. The case for ML is specific to the fact that ML is data-driven and data-hungry, so large models seem to be the end game. Check the Notes, by the way as I just wrote about this.
I totally agree with this (in fact that's what I'm saying): I would suggest that, perhaps, new innovations will come from a radically different intellectual framework than the underpinnings of machine learning
I'm less sanguine that it won't be a flavor of computer science, but of course no one yet knows. Computer science is a much bigger tent than ML.
My view is that machine learning is the apotheosis of the information theory revolution that happened after World War II. And the practical goal of the technologies that have come out of this revolution has been automation, more than anything else. For example, some of the industrial process complexity you described in your previous article stems from seeing automation as an end-in-itself, rather than just a component of some holistic set of social procedures that should be made more productive, efficient, valuable. So, the end of innovation for machine learning technology might really be the end of the line for this whole framework as a leading-edge cultural project.
I should note that I think the stagnation we're seeing in fundamental physics and other physical sciences, as well as in the humanities, is not unconnected.
For a long time, I've held the view that cultural paradigms can produce phenomena at different times and places and in different media that are qualitatively related, but not necessarily causally connected. This is most salient in abstract systems, like paradigms of knowledge, but it can seep into every area of culture. Kuhn is an obvious point of reference here. He describes “scientific paradigms” where there is an animating core idea to scientific research programs that make certain kinds of conclusions possible while excluding consideration of other lines of inquiry. However, these paradigms don’t predicate specific outcomes and the people working within them don’t have to coordinate with one another closely. Something else to read in this vein is Foucault’s "The Order of Things," which is a great book. He looks at how the paradigm of European systems of knowledge shifted from the late 18th century to the early 19th century not just in one field but simultaneously in three. An even more grandiose version of this approach is presented in Spengler’s "The Decline of Western Civilization." Spengler describes, at great length, how “Great Cultures” develop an animating idea across all of their aspects (art, literature, law, politics, historical events, etc.) in a roughly 1000-year time span.
I think something similar has taken hold of scientific culture in the industrialized world since WWII. We’ve decided to place the idea of “information” at the metaphysical center of research into foundational issues. What is the brain doing? It is processing information. How do genes work? They store information, which is then retrieved by certain intracellular processes, and duplicated in reproduction. How do the mathematical structures codified in physics operate in the physical world? Particles embody information and the laws of physics are the result of information shared through force interactions. How does culture propagate? Through the information carried by memes. Etc. It infects all of our thinking these days. People did not see the world this way 100 years ago, even though some of the precursors to information revolution were well established. We barely notice it because we’re completely ensconced in the “information” paradigm. This really hit home to me when I read "The Information" by James Gleick, which is an excellent history of the subject up through 2012, when he published the book. I’m sure you know this history pretty well.
My theory is that information is about how correspondences are established among representations with respect to what, in common, is being represented. If I represent a tree with the word “tree” and also draw a picture of a tree, “information” is the referent that these two representations hold in common among the two distinct media, oral language and 2D abstract visual representation. Information theory has succeeded wildly at developing technologies within this theme, but problems start when people ignore that these ideas about information are only valid AMONG REPRESENTATIONS. Information doesn’t function as a phenomenological category outside cognition. Somehow we’ve forgotten the dictum, “the map is not the territory.”
The practical value we’ve discovered with intellectual advances in the information paradigm have largely to do with automation and convenience. If you can set up systems that cycle rapidly among representations in different media by controlling these correspondences through conventional functions (i.e., “encoding” them), you can automate representational processes among media without human mediation, and this was really hard to prior to the development of information theory. You can also shorten the processing time involved within the human brain by externalizing some of the work of converting representations across media via computers and other types of information systems.
I see “machine learning” as the apotheosis of the information revolution because, more than any other theories or applications of information theory, it is the most extreme abstraction of our implicit ideas about information from human cognition. “AGI” is the platonic idea of information processing without any human involvement whatsoever. So, if we’ve reached the end of fundamental innovation in machine learning, chances are that there won’t be a more purely abstract version of “information” as a metaphysical category that can be sought after. New major advances in human knowledge will only be found by ditching “information” as THE central metaphysical category (and we can do this while still retaining the parts of information theory that are useful and instructive).
In my view, at the heart of human (and also animal, maybe even plant?) intelligence is the capacity to build mental models of the world. Essentially recreating constantly something like Plato's world of pure ideas. But ML doesn't do that. It uses prefabricated models, that are simply so complex that they can fit any problem. But there is no new model. That's the difference between human and machine intelligence.
This is right (machine learning has limits, and we can see them) and also missing the crucial thing: it has limits because it isn't intelligent, alive, embodied, evolved or adapted to live in the world. It doesn't understand anything: what is a cat, what is a number, which of these images in trhe CAPTCHA is the wrong way round. This might be a simple dispute over 'well, we can compensate for that' - or not - but for the absurd/terrifying energy appetite of these two kinds of 'intelligence': a factor of (at Stuart Russell's estimate) about a million to one. Meanwhile the tech industry disputes whether DeepSeek's compute is or is not eqivalent to GPT4's. Deckchairs on the Titanic, people!
I need to try to read all of this properly, will take some time. A comment and a question.
Since 1992, I have been involved in studying how children interpret "word problems" (see Verschaffel, Greer, & de Corte, 2000). I am peripherally involved now in a study of how LLMs handle them. For example: John lives 5 miles from school, Mary 3 miles from school. How far apart do John and Mary live? Confident answer: 2 miles. But no doubt that will be fixed by the next version.
Comment: in 2007 I happened to be in Leuven when Penrose received an honorary degree. He talked about the limitations of computability and invoked quantum theory as necessary. Have we any more Godels on the horizon? What about an incompleteness theory of the brain (it is incapable of understanding how itself works)?
I'm a late developer, just turned 80. Any hints would be welcome.
Great article again! I love all your articles. They give good perspective on the current happenings in the AI technology world, contrasting it with the alternative possibilities which remain unexplored. I agree with the position put forth here that the current LLM based systems can be seen as the pinnacle of BIG data-driven ML paradigm. It is almost the End-Game for this paradigm. Data-Driven ML have self-actualized. In one of his recent Substack post Gary Marcus calls it a transition from Narrow AI to Broad-Shallow-AI (BSI). The claims of AGI are completely misplaced. What we need is Broad Deep Understanding. The future would be interesting. We are on the cusp of a major change in direction again. For the first time ML community is starting to talk about reasoning. And that's one sure sign that ML paradigm is in its end-game mode. Death of Traditional ML; Integration of Learning with Reasoning will be the future... Long Live AI! ... Thanks for another wise article!
Erik, for a non-technical person, could you elaborate on “The End of Machine Learning Progress?” Which parts of the technology would that affect, and why would you call it here? Are the areas that will improve non-ML?
If you have a minute, please educate me. I'd love to learn a bit more on this.
Sure, what I mean by "the end of machine learning" is not that we won't keep using ML everywhere. Clearly, we will. I mean that since machine learning is data-driven and data-hungry, in the rough sense that it takes as input data and that the more data input the better the model "trained," the end point of that seems to be a very large model. That is, we've reached the end of machine learning when we reach the very large models that show the best performance. We've maxed out what machine learning IS, with LLMs, and there's really no conceivable way to go further with fundamental innovations. It's tweaks and then smaller tweaks until it's permanently plateaued. So it's silly to say that LLMs are a flawed design (by the way, the reader who said this, I like, it just gave me this idea), because they are the end point of the assumptions of ML and thus are the last expression of ML. We'll still have all the nuts and bolts of data science and data science is, again, I think forever part of the modern world, but ML as a reach toward greater and greater "intelligence" "has reached its end game. That's why I think it's historic, and most of the criticism (and even praise) misses this point. Does that help?
What’s your thinking on the angle that the ML tech as it stands and is being tweaked will be helping with progress in research and science? Will it play a part in spawning or refining something else?
Error detection and correction in this context is the question of determining whether a statement is truthful without resort to a true oracle (because if we had a true oracle it would be moot).
This is the hardest problem in logic and philosophy. It remains unsolved. We have systems, like the scientific method, to help us reduce our error rate, but they are slow - much too slow to apply in real time to LLMs - and not 100% reliable.
LLMs further confound our informal systems for detecting error: intuition about about the other's areas of expertise, linguistic and body language tells indicating that he's bullshitting or less confident in a statement. An LLM can be accidentally correct or incorrect about a claim on the *same subject* at different points *in the same conversation* and speak with equal confidence and equal degrees of competence signaling language in both instances.
This is less like a mythical oracle and more like a mythical demon, which might tell you the truth most of the time to gain your confidence and then lie strategically to sabotage you. Except of course the LLM has no such strategy, it's just sometimes wrong and sometimes right with no discernable pattern or frequency.
Anyway, cutting that rant short(ish), my point is: LLMs cannot be treated as flawed oracles. This is a terrible way to think of them and ML in general. We are not equipped to use them this way.
In some contexts it's probably fine, e.g. using computer vision or data/textual analysis to narrow down possible candidates for expert human inspection. Where false positives *and* false negatives are low consequence and every positive is independently analyzed by a qualified human.
But an unqualified/inexpert human is absolutely helpless in evaluating claims from any machine learning system. They *must not be presented as sources of truth*. They should not be presented as search engines, let alone knowledge bases. This will go badly and is going badly.
I couldn’t agree more. Why are we automating the equivalent of TPS reports. That’s brilliant man. Love it.
On other thing here about impact: you can gauge impact in different scenarios, roughly. In legal, for instance, catastrophic errors would involve cases like "Tow vs. Wade," which doesn't exist. Getting a date wrong might in some circumstances be catastrophic and in others minor. But we do have an idea how certain flaws "screw us" and in what way we're likely to be screwed. It's not deterministic, but it's not that way with humans. I agree that humans have epistemic virtues LLMs can never have, but in real world scenarios LLMs are paired with humans. Say the firm Dewey, Cheatum & Howe must generate a thousand page summary. Consider two cases. (1) A team of junior attorneys does it over the course of weeks and (2): A team of junior attorneys and an LLM does it in one day, with the proviso that the content is subject to LLM error. What's the cost to the firm for (1) vs. (2)? In both cases we can assume that even Dewey, Cheatum and Howe can't afford grievous errors. But much of the grunt work has been done by the LLM in the case of (2) and the junior attorneys now need to keyword search and sanity check in other ways the critical parts. My point is not that this will always be better but that over time we use tools in a way that reduces our manual efforts, and LLMs in the end are not likely to be fundamentally different.
Getting off topic, but I think it's funny that the example you came up with involves generating copious amounts of text that most likely nobody cares about.
I wonder if the better solution is something that reduces the need for thousands of pages of legalese in the first place. And this crops up everywhere in all sorts of business cases for LLMs: let them generate all that tedious text for you! Never write a TPS report again. Nobody reads them anyway, and nobody is honest in them, so who cares if it's wrong.
Might be we need to back up and ask, "why are we doing this in the first place, and how might machine learning, or any other software architecture, eliminate the task instead of automating it?"
Nice comment, Fukitol. If I wanted to be impish I'd point out that I never defined what I mean by "flawed" and so technically could get a nice exposition of LLMs out of "flawed oracle." You could, for instance, just take your critique and put that into what I mean by flawed (like "really flawed"). The sense of oracle here is the common sense notion, that you ask some computer program something and you get an answer. I'm not treating this as I would in computational complexity theory. But I take your point. One other quick point, which you caught: since there are no intentions, there are no demons.
Substantively, I'm working on practical strategies for error reduction, and I've noted many times the problem of impact--not all errors have the same impact. You're of course totally right that our lack of access to errors and their impact is profoundly troubling. I've written about this as well. It too bothers me deeply.
Back to practical computer science stuff, take a prompt - response pair we would classify as a request for explanation:
Prompt: Why was Napoleon in London in 1812? In pragmatics and rhetorical theory this is known as a PRESUPPOSITION because it assumes the truth of the proposition that Napoleon was in fact in London in 1812 (he was not).
It turns out that if you have the LLM itself classify prompt response pairs using a controlled vocabular like Rhetorical Structure Theory or Segmented Discourse Representation Theory (using JSON or XML or what have you), so that
<presupposition>the question</presupposition> you have many more cards to play for getting a reduced rate of errors. The LLM itself, looking at the tag when presented with this example again, would be extremely unlikely to give a confabulated response. I have not quantified this but it seems true given me messing around with such examples for a while. The LLMs ability to generate correct tags of its own prompt - response pairs seems to be quite high. Again not quantified but it doesn't seem to ever get them wrong when given the specification in a prior prompt.
Another way out of the epistemic nightmare of LLMs is to have them tag all their prompt responses this way in which case you can run some rules engine over the tagged pairs. For instance, if you have a list of court cases with a <date_constraint>1980-1989</>, so that you don't want to see any cases not in the 1980s, a Python rule-based engine could easily catch these errors and either flag them for the human user or generate a new prompt that forces the LLM to recheck (or both).
I am not yet sure that this approach will be doable, but it does illustrate my broader point that we will develop this or other strategies as well as ones that no one has yet thought of to deal with LLM hallucinations.
My point here is not that you're wrong about anything you said, but that this is an open area of investigation rather than a throw up your hands, let's try to "get rid" of LLMs.
Also, and this is really a separate discussion, but it's not clear to me that the nasty variety of errors we get from LLMs is much different than a part catastrophically failing with some small frequency. ALL engineering has errors, and sometimes these errors are absolutely catastrophic (as in: everyone dies). A failure point and an error are related but no quite the same, which is why I say this may be a separate discussion. But it's worth thinking about. LLMs present a challenge for the somewhat ironic reason that they work so well most of the time. That means that when an error comes, we may not expect it or detect it.
Nice comment, thank you.
Re: tagging the user prompt before feeding it back to the LLM for inference, I had a similar idea that I've yet to implement, so it's encouraging to hear that you've had some success with it.
I'm only working on structured interactive fiction, so it's mostly acceptable when the LLM produces a bad response (worst case, the user has a rollback feature if it comes up with something truly terrible or nonsensical). But as I get more into the structure side of it (an underlying game engine that tracks world state) it will be more important that it doesn't invent things, e.g. generating dialog for creatures who can't speak. I also need it to reliably pick up on user input that has a game-mechanic consequence, e.g. attacking or trading, thus the idea of analyzing and tagging the prompt before feeding it back to inference.
Anyhow, when I say low consequence errors, fiction seems like an obvious use case where it's okay for the LLM to be incorrect, so that's where I decided to put my energy. Though even in fiction there has to be internal consistency, which is a problem I'd say I have made good progress with.
Re: catastrophic failures, the problem as I see it is that we can normally tell when a machine fails. The car doesn't start, the program crashes, something explodes. With LLMs there is no apparent difference in behavior between an error and a correct response from the user POV. In your example, when the LLM tells the user that Napoleon was in London for a diplomatic conference in 1812, and then confabulates an elaborate narrative about what he did there, who he spoke to, and how the conference concluded... this doesn't look any different from when it answers a historical question correctly.
So the user has to already know the answer to the question he asks in order to detect that something has gone wrong, and if he knows this he's just fooling around, he's not depending on the machine for anything (and so the failure is low consequence).
Obviously other kinds of machine failure can do irreversible damage, and very rarely it's just a freak accident and not due to neglect, misuse or flawed design. But the idea of a system whose main mode of failure is invisible to the intended user really rubs the wrong way.
One approach established in critical air transport is parallel dissimilarity and voting. Off the top of my head, for reliability we might have 3 parallel LLMs trained exclusively on sources in English, Russian and Mandarin respectively.
Sorry, reliability => error detection/correction
Here is a follow-on question to your idea that this is the end of fundamental innovation in machine learning. Is this also the end of fundamental innovation in automation technologies?
A lot of our civilizational problems and economic efficiencies are due to social factors that revolve around people's values and the qualities those values engender in human behavior and the build environment. Automation technologies can't really solve these problems. For example, you could built a fantastic infrastructure using driverless electric cars to transport people anywhere quicker and more efficiently than our current system, but you'd have to shred property rights, tear down a bunch of buildings in urban areas and substantially re-imagine land use across the board. Elon Musk isn't Robert Moses.
I would suggest that, perhaps, new innovations will come from a radically different intellectual framework than the underpinnings of machine learning. It won't be a flavor of computer science.
To underscore this, I think that’s a really interesting question: are we reaching the end of automation? I don’t have a ready-made answer for that. I have to think about that more. My knee-jerk reaction is automation will continue to get better, but it’s going to follow the same trajectory of tweaks followed by smaller tweaks, and so on. But I have not thought about it. So I don’t want to extend too far in that direction.
I don't want to imply that we won't continue to improve various automated technologies or processes. I have no doubt that we will. Nor will we abandon much of the automation that has already been developed. But my question is whether or not we'll a) devise any new fundamental approaches to automation, and b) spend so much capital and cultural attention on developing new (and generally more complex) means of automation.
Hi Jeffrey,
The case with ML is a bit different from automation but it's an interesting question. The case for ML is specific to the fact that ML is data-driven and data-hungry, so large models seem to be the end game. Check the Notes, by the way as I just wrote about this.
I totally agree with this (in fact that's what I'm saying): I would suggest that, perhaps, new innovations will come from a radically different intellectual framework than the underpinnings of machine learning
I'm less sanguine that it won't be a flavor of computer science, but of course no one yet knows. Computer science is a much bigger tent than ML.
My view is that machine learning is the apotheosis of the information theory revolution that happened after World War II. And the practical goal of the technologies that have come out of this revolution has been automation, more than anything else. For example, some of the industrial process complexity you described in your previous article stems from seeing automation as an end-in-itself, rather than just a component of some holistic set of social procedures that should be made more productive, efficient, valuable. So, the end of innovation for machine learning technology might really be the end of the line for this whole framework as a leading-edge cultural project.
I should note that I think the stagnation we're seeing in fundamental physics and other physical sciences, as well as in the humanities, is not unconnected.
I think there’s a lot of teeth in this. Do you have something interesting as a reference? Something to read?
There is a lot packed into these few sentences.
For a long time, I've held the view that cultural paradigms can produce phenomena at different times and places and in different media that are qualitatively related, but not necessarily causally connected. This is most salient in abstract systems, like paradigms of knowledge, but it can seep into every area of culture. Kuhn is an obvious point of reference here. He describes “scientific paradigms” where there is an animating core idea to scientific research programs that make certain kinds of conclusions possible while excluding consideration of other lines of inquiry. However, these paradigms don’t predicate specific outcomes and the people working within them don’t have to coordinate with one another closely. Something else to read in this vein is Foucault’s "The Order of Things," which is a great book. He looks at how the paradigm of European systems of knowledge shifted from the late 18th century to the early 19th century not just in one field but simultaneously in three. An even more grandiose version of this approach is presented in Spengler’s "The Decline of Western Civilization." Spengler describes, at great length, how “Great Cultures” develop an animating idea across all of their aspects (art, literature, law, politics, historical events, etc.) in a roughly 1000-year time span.
I think something similar has taken hold of scientific culture in the industrialized world since WWII. We’ve decided to place the idea of “information” at the metaphysical center of research into foundational issues. What is the brain doing? It is processing information. How do genes work? They store information, which is then retrieved by certain intracellular processes, and duplicated in reproduction. How do the mathematical structures codified in physics operate in the physical world? Particles embody information and the laws of physics are the result of information shared through force interactions. How does culture propagate? Through the information carried by memes. Etc. It infects all of our thinking these days. People did not see the world this way 100 years ago, even though some of the precursors to information revolution were well established. We barely notice it because we’re completely ensconced in the “information” paradigm. This really hit home to me when I read "The Information" by James Gleick, which is an excellent history of the subject up through 2012, when he published the book. I’m sure you know this history pretty well.
My theory is that information is about how correspondences are established among representations with respect to what, in common, is being represented. If I represent a tree with the word “tree” and also draw a picture of a tree, “information” is the referent that these two representations hold in common among the two distinct media, oral language and 2D abstract visual representation. Information theory has succeeded wildly at developing technologies within this theme, but problems start when people ignore that these ideas about information are only valid AMONG REPRESENTATIONS. Information doesn’t function as a phenomenological category outside cognition. Somehow we’ve forgotten the dictum, “the map is not the territory.”
The practical value we’ve discovered with intellectual advances in the information paradigm have largely to do with automation and convenience. If you can set up systems that cycle rapidly among representations in different media by controlling these correspondences through conventional functions (i.e., “encoding” them), you can automate representational processes among media without human mediation, and this was really hard to prior to the development of information theory. You can also shorten the processing time involved within the human brain by externalizing some of the work of converting representations across media via computers and other types of information systems.
I see “machine learning” as the apotheosis of the information revolution because, more than any other theories or applications of information theory, it is the most extreme abstraction of our implicit ideas about information from human cognition. “AGI” is the platonic idea of information processing without any human involvement whatsoever. So, if we’ve reached the end of fundamental innovation in machine learning, chances are that there won’t be a more purely abstract version of “information” as a metaphysical category that can be sought after. New major advances in human knowledge will only be found by ditching “information” as THE central metaphysical category (and we can do this while still retaining the parts of information theory that are useful and instructive).
In my view, at the heart of human (and also animal, maybe even plant?) intelligence is the capacity to build mental models of the world. Essentially recreating constantly something like Plato's world of pure ideas. But ML doesn't do that. It uses prefabricated models, that are simply so complex that they can fit any problem. But there is no new model. That's the difference between human and machine intelligence.
This is right (machine learning has limits, and we can see them) and also missing the crucial thing: it has limits because it isn't intelligent, alive, embodied, evolved or adapted to live in the world. It doesn't understand anything: what is a cat, what is a number, which of these images in trhe CAPTCHA is the wrong way round. This might be a simple dispute over 'well, we can compensate for that' - or not - but for the absurd/terrifying energy appetite of these two kinds of 'intelligence': a factor of (at Stuart Russell's estimate) about a million to one. Meanwhile the tech industry disputes whether DeepSeek's compute is or is not eqivalent to GPT4's. Deckchairs on the Titanic, people!
I need to try to read all of this properly, will take some time. A comment and a question.
Since 1992, I have been involved in studying how children interpret "word problems" (see Verschaffel, Greer, & de Corte, 2000). I am peripherally involved now in a study of how LLMs handle them. For example: John lives 5 miles from school, Mary 3 miles from school. How far apart do John and Mary live? Confident answer: 2 miles. But no doubt that will be fixed by the next version.
Comment: in 2007 I happened to be in Leuven when Penrose received an honorary degree. He talked about the limitations of computability and invoked quantum theory as necessary. Have we any more Godels on the horizon? What about an incompleteness theory of the brain (it is incapable of understanding how itself works)?
I'm a late developer, just turned 80. Any hints would be welcome.
I found Ochigame's MIT dissertation stimulating.
Brian Greer brian1060ne@yahoo.com
Great article again! I love all your articles. They give good perspective on the current happenings in the AI technology world, contrasting it with the alternative possibilities which remain unexplored. I agree with the position put forth here that the current LLM based systems can be seen as the pinnacle of BIG data-driven ML paradigm. It is almost the End-Game for this paradigm. Data-Driven ML have self-actualized. In one of his recent Substack post Gary Marcus calls it a transition from Narrow AI to Broad-Shallow-AI (BSI). The claims of AGI are completely misplaced. What we need is Broad Deep Understanding. The future would be interesting. We are on the cusp of a major change in direction again. For the first time ML community is starting to talk about reasoning. And that's one sure sign that ML paradigm is in its end-game mode. Death of Traditional ML; Integration of Learning with Reasoning will be the future... Long Live AI! ... Thanks for another wise article!
Very much appreciated, thank you.
Erik, for a non-technical person, could you elaborate on “The End of Machine Learning Progress?” Which parts of the technology would that affect, and why would you call it here? Are the areas that will improve non-ML?
If you have a minute, please educate me. I'd love to learn a bit more on this.
Hi Nico,
Sure, what I mean by "the end of machine learning" is not that we won't keep using ML everywhere. Clearly, we will. I mean that since machine learning is data-driven and data-hungry, in the rough sense that it takes as input data and that the more data input the better the model "trained," the end point of that seems to be a very large model. That is, we've reached the end of machine learning when we reach the very large models that show the best performance. We've maxed out what machine learning IS, with LLMs, and there's really no conceivable way to go further with fundamental innovations. It's tweaks and then smaller tweaks until it's permanently plateaued. So it's silly to say that LLMs are a flawed design (by the way, the reader who said this, I like, it just gave me this idea), because they are the end point of the assumptions of ML and thus are the last expression of ML. We'll still have all the nuts and bolts of data science and data science is, again, I think forever part of the modern world, but ML as a reach toward greater and greater "intelligence" "has reached its end game. That's why I think it's historic, and most of the criticism (and even praise) misses this point. Does that help?
Does help, thanks.
What’s your thinking on the angle that the ML tech as it stands and is being tweaked will be helping with progress in research and science? Will it play a part in spawning or refining something else?