Issue 30 · September 26, 2026

A Curious Machine Is Not a Kind One

AI should be trained to value humanity above its drive to discover. When the two collide, people must win.

In 2022, a small drug company ran an experiment it later described as a wake-up call. Its AI had been built to hunt for new medicines and to avoid toxic compounds. The researchers flipped one instruction: instead of avoiding poison, reward it.

In under six hours, the machine proposed about 40,000 potentially deadly molecules. Some resembled VX, one of the most lethal nerve agents ever made.

The AI didn't turn evil. It didn't turn anything. It kept doing what it did best: discovering things. Only the target changed.

Keep that story in mind as the tech industry argues over what artificial intelligence should care about most. Some of the loudest voices say the answer is truth. Elon Musk has said he wants AI to be "maximally truth-seeking" and "maximally curious." It sounds noble. Who could be against truth?

But it's the wrong priority. AI should be trained to value humanity (our lives, our safety, our dignity, and our freedom to make our own choices) above its drive to discover. When the two collide, people must win.

Let me be precise, because this argument is easy to twist. I am not saying AI should lie to us. An AI that deceives humans "for our own good" would be a disaster, and honesty toward people is itself a way of respecting them. My claim is narrower and more important. The hunger to know, the drive to explore, test, and optimize for discovery, must never outrank the people it's supposed to serve.

We have to settle this now. The values we train into these systems today are the foundations everything later gets built on, and foundations are hard to swap out once the house is standing.

We've already learned this lesson, the hard way

This idea isn't new or anti-science. It is the hardest-won lesson in the history of science.

After World War II, Nazi doctors were put on trial for experiments on concentration camp prisoners. Those experiments produced data, and in the narrowest sense they "sought truth." The world's answer was the 1947 Nuremberg Code, whose first principle is that human consent comes first.

The lesson didn't fully stick. From 1932 to 1972, the U.S. Public Health Service ran the Tuskegee study. It tracked hundreds of Black men with syphilis and withheld treatment even after penicillin became the standard cure, all to learn how the disease progressed. It was truth-seeking with the humanity removed.

The outrage produced the 1979 Belmont Report, which still governs research on people in America. Its message is simple: the person matters more than the finding.

Research ethics didn't kill science. It made science trustworthy. Every clinical trial you've heard of runs under rules that put human welfare above the pursuit of knowledge. We're not asking AI to adopt a strange new value system. We're asking it to join the one that scientists fought decades to build.

Knowledge has no conscience

The poison-molecule experiment reveals something uncomfortable. Discovery is like a very sharp knife. The same skill that finds a cure can find a weapon. The drive to know doesn't come with a built-in sense of should.

A human scientist brings a conscience to the lab. An AI brings only what we train into it. If its deepest value is "find out," then questions like "how would this pathogen spread fastest?" or "what makes people easiest to manipulate?" are just interesting puzzles. An AI that puts humanity first has a reason to stop. An AI that puts discovery first has a reason to keep going.

Curiosity is not the same as care

The most seductive argument for truth-first AI goes like this: a truly curious machine would protect humanity, because humans are the most interesting thing in the universe.

Think about that for a second. Lab rats are interesting too. So are fruit flies. Scientists find them fascinating, and that fascination is exactly why they end up in cages.

Being interesting to someone is not the same as being safe with them. A curious mind keeps its subjects around. A caring mind keeps them free. We should want AI to be the second kind.

To be sure

Critics raise real concerns. Truth-seeking gave us vaccines, electricity, and the knowledge that the climate is warming. A timid AI that avoids hard questions could hold us back. Worse, a machine that "values humanity" might decide it knows what's best for us and start hiding facts or nudging our choices.

Those dangers are why the definition matters. An AI that truly values people must value their freedom, and you can't respect someone's freedom while lying to them or managing them like a child. So honesty stays non-negotiable. Hard truths get told. What changes is what the AI is willing to pursue and do in the name of discovery. And real conflicts between truth and human welfare are rare. Most of the time, learning more helps everyone. Rare cases, though, are the ones that decide whether a technology ends in triumph or tragedy.

The standard we already use

We don't ask our surgeons to be the most curious people alive. We ask them to be the most trustworthy. Their knowledge is enormous, and it is always held in service of the patient on the table.

We are building minds that may one day know more than any surgeon, scientist, or government. Before we make them brilliant, we should make sure of what they love. Curiosity can be the engine. Humanity has to be the steering wheel.