Beyond Preference Alignment: Teaching AIs to Play Roles & Respect Norms, with Tan Zhi Xuan

TL;DR

Tan Zhi Xuan argues that AI should follow role-specific norms grounded in social consensus rather than simply maximize learned human preferences. Preference data can produce an imperfect utility function whose over-optimization misses what people actually want. Their technical work uses Bayesian rule induction in Markov games to show how agents can infer norms and sustain cooperation, offering a concrete bridge from moral philosophy to implementation worth exploring further.

Transcript

what we argue in a paper Beyond preferences in the alignment we are really trying to critique this sort of preferences view is so we go through all the limitations of taking this sort of expected utility maximization view of both human rationality and a alignment too seriously people know that this learned utility function you try and learn from pr... Read More

Key Insights

• The current AI alignment paradigm focuses on maximizing human preferences, but this approach has significant limitations due to inconsistent and difficult-to-aggregate human preferences.
• Xuan proposes an alternative approach where AI systems play specific roles with clear normative standards, similar to human professionals upholding societal standards.
• AI systems should be designed to learn and respect social norms, allowing them to function within society's moral framework and avoid negative externalities.
• The conversation explores the integration of philosophical theories from both Eastern and Western traditions to address AI alignment challenges.
• Xuan's technical work involves AI agents learning social norms through Bayesian rule induction in Markov games, demonstrating how norms can emerge and sustain cooperation.
• The discussion highlights the potential of AI systems to infer social norms by observing deviations from self-interested behavior in other agents.
• Xuan emphasizes the importance of decentralized AI systems, where multiple specialized agents perform distinct roles rather than a monolithic AGI.
• The paper critiques the preference-based alignment strategy and suggests that AI systems should be aligned to societal moral standards rather than individual preferences.

Questions & Answers

Q: Why does Tan Zhi Xuan criticize preference-based AI alignment?

Learned utility functions derived from preference data do not perfectly capture what people really want. Because human preferences can also be inconsistent and difficult to aggregate across populations, optimizing these imperfect proxies can produce over-optimization rather than genuine alignment.

Q: What alternative to preference alignment does Tan Zhi Xuan propose?

Xuan proposes designing AI systems to perform specific roles governed by clear normative standards and constraints. Those standards would emerge through social consensus, much as human professionals must follow accepted standards regardless of their own or their clients’ preferences.

Q: What minimum moral standard should an AI system meet?

The proposed baseline is a minimal morality that society would agree an AI must satisfy before being allowed to operate. Xuan connects this standard to a contractualist picture and says constitutional AI is closer to that approach.

Q: How does Bayesian rule induction help AI agents learn social norms?

Bayesian rule induction lets agents infer possible rules by observing behavior that departs from pure self-interest. An agent that repeatedly sees others decline opportunities to take resources may infer a governing norm and incorporate compliance with it into its own decisions.

Q: What role do Markov games play in Xuan’s technical research?

Xuan’s work studies learning and sustaining shared normative systems through Bayesian rule induction in Markov games. These computational environments allow agents to learn good behavior from one another and demonstrate how norms can emerge across a population.

Q: How can learned norms sustain cooperation among AI agents?

Agents observe others, infer rules behind apparently non-self-interested actions, and adjust their own decisions to comply. This mechanism allows norms to persist across generations of agents and supports population-wide cooperation.

Q: What resource problems can norm-learning agents help avoid?

Shared norms can help populations avoid tragedy-of-the-commons problems. The transcript specifically identifies overfishing and other forms of resource depletion as examples that cooperative agents may prevent.

Q: How does Xuan combine philosophy with technical AI alignment work?

The role-based proposal brings together moral philosophy, cognitive science, and Bayesian probabilistic programming. It also thoughtfully combines Eastern and Western traditions while pairing a critique of preference maximization with implemented environments where agents learn social norms.

Summary & Key Takeaways

• Tan Zhi Xuan critiques the current preference-based AI alignment paradigm, arguing that it fails to capture the complexity and inconsistency of human preferences. Xuan proposes a role-based alignment approach where AI systems adhere to normative standards derived from social consensus.

• The conversation explores how AI agents can learn social norms and sustain cooperation through Bayesian rule induction in Markov games. This technical approach allows AI to infer norms by observing deviations from self-interested behavior in other agents.

• Xuan emphasizes the need for decentralized AI systems, where specialized agents perform distinct roles. This approach contrasts with the pursuit of a monolithic AGI, aligning AI development with societal moral standards rather than individual preferences.