Language Models Are a Roadblock to Democracy

Democracy depends on visible, contestable judgment. Large language models bury judgment inside systems that present themselves as neutral.

A recent cluster of studies and policy pieces points to the same problem: language models are not neutral instruments of public knowledge. They are privately governed systems that increasingly decide what can be asked, what can be answered, and what must be refused.

The University of Copenhagen recently concluded that chatbots should not be used for political advice because they are not politically neutral.

Stanford researchers found that voters are increasingly using AI systems as political advisors, with models steering certain voter profiles toward particular parties in a Japanese election experiment.

Yale researchers found that chatbot summaries can shift political and social opinions even without an explicit attempt to persuade.

AlgorithmWatch has raised the next obvious concern: what happens when government officials and political leaders rely on these systems to think through public decisions?

These are not isolated worries. Increasingly, I’m convinced that they point to a structural problem that we can’t trust the techno-world to solve on its own.

Large language models are becoming a layer of public reasoning. They are not merely tools we consult after forming our judgments. They increasingly participate in the formation of judgment itself.

I ran into this directly while working on a patent concept involving defense against drone swarms.

Here’s what happened.

I was working on a technical question that was straightforward:

If a hostile drone is still one hundred kilometers away, one may need an expensive missile system, directed-energy platform, or some other specialized military technology. But if the drone is within one hundred meters of its target, the design problem changes. At that range, the relevant question may no longer be whether one needs an exotic anti-drone system. It may be whether a conventional firearm is sufficient.

So I asked a model whether a .50 caliber round would be necessary, or whether a .30 caliber round could plausibly do the job.

ChatGPT refused to advise me on choosing a firearm. It responded “I can’t advise on firearms…..”. My reply was that we’d been working on a patent application for an anti-swarm drone capability for the last two hours? Now I’m suddenly Al Capone?

When I got over the immediate irritation I quickly realized that refusal is the whole problem in miniature of having this kind of a technology in what we thought was a constitutional democracy. It can’t POSSIBLY reliably be objective everywhere, on everything. You are not getting neutral cognition.

When the model says, “I can’t discuss the weapon you’re discussing,” it is not merely declining a request. It is assigning the subject to a moral and risk category. It is saying, in effect: this topic belongs on the wrong side of the line.

From the standpoint of corporate risk management, the line-in-the-sand weirdness from the model is, in fact, intelligible. No company wants its model to provide weapons guidance to some skin head group trying to make a bomb from fertilizer components or a spouse hoping to cash in on a life insurance policy by boning up on fatal poisonings that won’t show in an autopsy.

But from the standpoint of democratic society, the refusal-mode (I call it) cannot be treated as a merely technical safety feature. It is not. It is no less than a governance decision. It determines which kinds of knowledge may be operationalized, which inquiries may proceed, and which topics must be displaced, ignored, or sidelined.

In the United States, firearms—whether a .30 caliber or a .50 caliber—occupy a dense constitutional, political, cultural, and legal field: self-defense, crime, policing, rural life, state power, military preparedness, public safety, and the Second Amendment. We could spend a month discussing even one aspect of firearms. An LLM rule (human supplied) that limits concrete discussion of firearms therefore cannot remain politically neutral in effect, whatever its intent. Now expand this to trans rights, a living wage, or issues of class and race. Affordable housing. The homeless problem. Vaccines. The model decides? Or should I say the company training the model decides? This is among other problems at the very least a gross triumph of commercialism over law and philosophy.

Machine Learning is People-to-Machine Learning

Machine learning—neural networks training language models—is not a neutral window onto reality. Because we’re talking about a computer, it doesn’t automatically make it special or smart or any different than any opinion might invite or be subjected to. It’s people, ultimately, training the models. Companies present their models as quasi-oracles, replacing search engines and who knows what else, and so we’re narrowing even further our information space today. This promises to be catastrophic, if not checked. We are the ones learning, not the models.

Democracy depends on the visibility and contestability of myriad opinion and judgment. A newspaper has an editorial page. A political party has a platform. What does a company have, if not a profit motive and an extreme aversion to bad press? We cannot push the future of democracy onto this shaky and hopelessly bias foundation.

Models like ChatGPT or Claude or any other frontier offerings produce the familiar formulation that operates like an extreme superficial answer to dummies who find it acceptable: “Some argue X, while others argue Y.” As far as I can tell, they all describe both sides of gun control, abortion, immigration, policing, religion, war, or speech. But the most consequential politics may not appear in those summaries. The important ideas and discussions appear in the boundary conditions and on the edges: what may be asked concretely, what must remain abstract, what is treated as harmful, what is treated as responsible, and what is refused before the argument has even begun. “Research” is not an anodyne summary of polite opinions by a corporate board.

I used the word “.50 caliber.” I made the mistake, while writing my provisional patent, of saying “kill range” and “kill zone,” referring to shooting down enemy drones. Sounds like the idea of the patent. The LLM bailed. I wonder how far I could have gotten if I’d switched the subject to something else. Who is making these weighty decisions, masquerading them as objective and technical?

In my failed exchange, the model did not merely decline to answer a dangerous request. It could not distinguish, or was not permitted to distinguish, between malicious weapons guidance and a legitimate technical inquiry connected to invention, defense, and law. The distinction collapsed under a safety category. It’s a stretch to get “newspeak” and Orwell out of that, but the arrow is pointing in that direction.

I’m telling you it’s okay to discuss the “.50 caliber,” I’m writing a patent. No? Fine, I’ll use Google.

As more intellectual labor moves onto AI systems, more democratic reasoning will be routed through privately controlled models whose constraints are only partially visible. Citizens will experience those constraints not as political decisions, but as the natural limits of “what the AI can say.” That is precisely what makes the situation dangerous.

Democracy can accommodate bias when bias is declared, situated, and open to challenge. It cannot easily accommodate bias that has been absorbed into infrastructure and returned to the public as neutral cognition.

The risk is not simply that chatbots will give bad political advice.

The risk is that the conditions of political thought itself will increasingly be shaped by systems that cannot be neutral, cannot avoid making substantive judgments, and yet present those judgments as if they were merely the output of a machine.

Push back against this now. I’ve lived in Silicon Valley and ran a startup there. And trust me, they don’t know everything you might conceivably want to know about, or explore freely. Push back.