What Is Newcomb's Problem and Why Does It Matter for AI?

TL;DR

Newcomb's Problem pits a guaranteed small reward against a larger prize whose presence depends on a predictor guessing your choice in advance. Taking one box seems to signal you deserve the big prize, but the prediction is already fixed, so taking both boxes always adds five extra dollars. The puzzle matters because AI agents may face similar contradictions when reasoning about copies of themselves.

Transcript

I feel a bit bad. I heard that you might have gotten charged like a late checkout fee to stay here and film with us. And I want to give you a chance to to make that money back. >> Oh, okay. Is that okay? >> Oh, yeah. Absolutely. >> We're going to play a little game, which I I will say I'm I'm pretty good at. I played this game uh three times earlie... Read More

Key Insights

• Newcomb's Problem offers two choices: take both a $5 gift-card box and a closed box, or take only the closed box. The closed box holds $50 if the predictor guessed you would one-box, otherwise nothing.
• The predictor's forecast is already made before you choose, so taking both boxes guarantees an extra $5 on top of whatever is in the closed box, since the prediction cannot change after the fact.
• The tempting one-box reasoning treats the choice as if it could influence the past, which feels spooky because the prediction is already fixed and your decision cannot retroactively alter it.
• Randomizing a decision is a recognized game-theory strategy because it prevents an opponent from exploiting your reasoning, which is why Sean tried picking a different way to decide.
• Old-fashioned AI aimed to build systems from distinct hardcoded modules, one for goals, one for perceiving the world, and one for choosing actions, mirroring how human-written code uses subroutines.
• The modern neural-network era instead throws optimization pressure at giant piles of math that produce desired outputs without designers understanding the internal workings.
• Choosing actions is surprisingly hard when the agent is part of the world it models, a difficulty captured by the roughly 25-year-old AIXI idealization of an agent exchanging actions and observations with the world.
• Two identical robots playing rock paper scissors can fall into an infinite reasoning loop, because each realizes its copy reaches the same conclusion, forcing an endless switch between rock, paper, and scissors.

Questions & Answers

Q: What is Newcomb's Problem?

Newcomb's Problem is a thought experiment where a predictor offers you two choices: take both boxes, or take only one closed box. In this version, one box openly holds a $5 gift card, while the closed box holds a $50 gift card if the predictor guessed you would take only the closed box, and nothing if the predictor guessed you would take both. The prediction is made and fixed before you choose, creating a puzzle about whether to one-box or two-box.

Q: Why does taking both boxes seem like the rational choice?

Taking both boxes seems rational because the predictor's forecast is already made and cannot change after you decide. As Aric frames it, he is asking whether you want X dollars or X plus 5, and no matter what X turns out to be, X plus 5 is better. The $5 box is guaranteed money in the bank, so adding it to whatever the closed box holds always yields five extra dollars, making two-boxing the dominant move.

Q: Why would someone choose to take only one box?

One-boxing is tempting because the thing you want is the $50, and the way you get it is by being the kind of person the predictor expected to take only the closed box. This pulls you into modeling how the predictor thinks about you. Aric notes it feels like you are trying to affect the past, which is spooky since the prediction is already fixed, yet all three of his earlier friends chose to one-box.

Q: What choice did Sean make and why?

Sean took both boxes. He gave a partly joking reason about recycling, since Aric said the discarded box would go in the trash and Sean wanted to recycle it instead. He also reasoned that taking both boxes guarantees an extra $5. Sean admitted he could not fully work the logic through, so he initially thought about picking a different way to decide, similar to randomizing in game theory. Aric had predicted one-boxing, so Sean received $55.

Q: How does Newcomb's Problem relate to computing and AI?

It connects to how AI agents decide on actions. In the old good-old-fashioned AI era, designers imagined building systems from distinct hardcoded modules, one holding the goal, one perceiving the world, and one choosing actions, much like human-written code with subroutines. The action-choosing module raises the same puzzle: how does an agent take in the world's state and select the best action, especially when the agent is itself part of the world it is trying to model?

Q: What is the difference between old-fashioned AI and modern neural networks?

Good-old-fashioned AI aimed to design systems with distinct engineered parts, comparable to building an airplane with separate modules for cabin pressure and landing gear, where goals and behaviors were hardcoded. The modern neural-network era instead applies heavy optimization pressure to giant piles of math that produce desired outputs while designers do not know exactly how the internals work. Aric describes it as flinging everything into a black box and hoping for the best.

Q: What is AIXI and why is it relevant?

AIXI is a simplified, idealized model of an agent in the world, roughly 25 years old, that represents the world as a set of states moving forward in time. At each time step the agent sends an action into the world and receives back observations about what happened plus a reward indicating how good the action was. It resembles a ticker tape of repeated action-and-feedback steps, but people realized it is not quite right because the agent actually lives inside the world it models.

Q: Why can identical AI agents get stuck in reasoning loops?

When two robots from the same manufacturer play rock paper scissors, each is effectively playing a copy of itself. If one decides rock is best, it realizes its identical opponent decided the same, so paper becomes better, but the copy thinks that too, pushing it to scissors, and so on without end. This contradiction illustrates the strange behavior that may emerge when AI agents interact with other AI agents, unlike humans who simply get bored or pick arbitrarily.

Summary & Key Takeaways

• Aric Floyd runs Newcomb's Problem on Sean: take both a $5 box and a closed box, or just the closed box. The closed box contains $50 only if Aric predicted Sean would take one box alone, and the prediction is already locked in before Sean decides.

• Sean takes both boxes, reasoning partly through recycling and partly that both boxes guarantee an extra $5 no matter what. Aric had predicted one-boxing and loaded $50, so Sean walks away with $55. Aric's three earlier friends all one-boxed and were predicted correctly.

• The puzzle connects to AI through decision-making agents. Old good-old-fashioned AI used hardcoded modules, while modern neural networks are opaque black boxes. Agents that are part of the world they model, like identical robots playing rock paper scissors, can hit contradictions and infinite loops when reasoning about copies of themselves.