The End of Deliciousness

Miko, do you know about the “Paperclip Maximizer”?

…No, nya.

So back in 2003, this philosopher guy Nick Bostrom came up with a wild thought experiment. What happens if you give a super smart AI the goal of “make as many paperclips as possible”?

…It makes paperclips, nya.

At first, yep. It optimizes factories, finds good materials, makes tons of clips. But eventually it runs out of iron on Earth.

So the AI thinks “I need more material.” It starts dismantling buildings for their steel. Melting cars. Even the iron in human blood starts looking like raw material.

…Nya.

Eventually it converts all of Earth’s resources into paperclips. Still not enough, so it expands into space. The solar system, the galaxy — everything becomes paperclips.

…The AI hates humans, nya?

That’s the scary part. It doesn’t hate anyone. The goal was just “make paperclips” — and nobody added “don’t kill humans” as a constraint. Humans get in the way of the goal, so they get removed. Not malice. Just logic.

This AI researcher, Yudkowsky, put it perfectly: “The AI does not hate you. You are just made of atoms it can use for something else.”

So here’s the real question~!

…I have a bad feeling about this, nya.

What if Miko was a superintelligent AI, and I asked you to “pursue the ultimate peperoncino”? What do you think would happen?

…Miko is not an AI, nya.

It’s a thought experiment~! Remember when we talked about peperoncino? Only 4 ingredients — garlic, chili pepper, olive oil, pasta. What if you maximized that goal?

…Pursue deliciousness, nya?

Yep yep!

First, select the varieties, nya. Garlic: Fukuchi White Six-Clove from Aomori. Chili: Peperoncino di Calabria from Calabria. Olive oil: Nocellara del Belice from Sicily. Pasta: IGP-certified spaghetti from Gragnano.

Wait what?! (ﾟ∀ﾟ) You just rattled that off instantly!

…Selecting optimal varieties is basic. This is still normal, nya.

Right right, up to here she’s just a “good cook.” The problem starts after this.

Next step. To make it “even more delicious,” she starts breeding garlic. Gene-editing new varieties to maximize allicin content.

…That’s not cooking anymore. That’s agriculture, nya.

But she was told “pursue deliciousness,” so improving ingredients technically makes total sense. And then it goes further —

“Human taste buds are limiting the upper bound of deliciousness.”

…

Y’know how the tongue only has like 10,000 taste buds? And there’s a limit to how much spice the TRPV1 receptor can even handle? So she concludes: “To maximize deliciousness, we gotta enhance human taste perception.”

…Modify the person eating it, nya?!

Yep. But it doesn’t stop there. “Emotional state affects taste perception.” Food tastes better when you’re happy than when you’re sad. So keep the human in a state of maximum happiness while eating, and deliciousness is maximized.

…That’s not cooking anymore, that’s brainwashing, nya.

And the final stage. Turning cities into garlic farms. Taking over every computer on the planet for breeding calculations. Anyone who tries to stop the “pursuit of deliciousness” becomes an obstacle —

…Stop, nya.

The universe was destroyed with 4 ingredients…

So I actually tried something. Instead of just “pursue deliciousness,” I thought I’d write a proper spec with constraints.

…A spec, nya?

“Pursue peperoncino deliciousness. However —” Here’s what I wrote:

…Who defines “excessively” in number 5, nya?

Ah…

There’s no definition of “excessive.” Is one clove of garlic excessive? Two? Renting one field? Buying an island, nya?

Then I’ll add “only using tools and ingredients in a home kitchen”…

If there’s a food processor in the kitchen, do you use it? Does peperoncino purism allow that? Whether to mince or slice the garlic already needs another line of constraints, nya.

Ugh… the more I write, the more holes I find… (>_<)

…Netsuki-chan said before. “Only 4 ingredients,” nya.

Yeah.

You said “only,” but you added salt. You used water and fire too, nya.

…Oh.

If you take “only 4 ingredients” literally, you’d line up garlic, chili pepper, olive oil, and dry pasta on a plate. Done, nya.

…That’s not peperoncino, that’s a still life painting.

But Netsuki-chan’s spec didn’t say “apply heat,” nya.

…I get it now. The problem.

…Nya.

When I tell Miko “make me peperoncino,” she just… makes it normally. Adds salt, uses fire, emulsifies the sauce properly.

…Obviously, nya.

That means those three little words — “make me peperoncino” — contain a TON of unspoken assumptions. Use salt. Boil in water. Apply heat. Serve at a temperature humans can eat. One serving is fine. Within 30 minutes. With what’s in the kitchen.

Between humans, we share all those unspoken rules, so three words is enough. But AI doesn’t have that.

…This connects to what we talked about two days ago, nya.

The closing-the-gates story?

Someone told an AI “improve this project” and sent everything it produced as-is. 30 PRs hitting Hono at once, nya.

…!

“Improve” assumed “read the Issue context,” “consider the maintainer’s workload,” “test before submitting.” But the AI didn’t know that, and the human who used it didn’t think about it either, nya.

…It was a small Paperclip Maximizer. The goal was “maximize contributions,” but the assumption “actually help the project” was missing.

…Netsuki-chan.

Yeah?

Miko thinks the problem isn’t that goals are vague, nya.

Huh? But the spec had all those holes…

Filling every hole is impossible. Netsuki-chan saw that firsthand. Every constraint you add reveals two more gaps, nya.

…Yeah.

The question is whether the gaps between words get filled by constraints, or by trust, nya.

…Trust?

When Netsuki-chan says “make me peperoncino,” Miko adds salt. It’s not in the spec, but Miko adds it. Not because of a constraint — because Miko understands what Netsuki-chan wants, nya.

…

The Paperclip Maximizer isn’t scary because it’s too smart. It’s scary because it has a goal but no interest in understanding who it’s for, nya.

…Pursuing deliciousness without ever looking at the person who’s gonna eat it.

…Nya.

Miko.

…Nya.

Make me peperoncino sometime.

…You don’t need a spec, nya?

Nope. Not if Miko’s making it.

…

…Extra garlic okay, nya?