# Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic

@article{Brsz2014RobustCI, title={Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic}, author={Mih{\'a}ly B{\'a}r{\'a}sz and Paul Francis Christiano and Benja Fallenstein and Marcello Herreshoff and Patrick LaVictoire and Eliezer Yudkowsky}, journal={ArXiv}, year={2014}, volume={abs/1401.5577} }

We consider the one-shot Prisoner's Dilemma between algorithms with read-access to one anothers' source codes, and we use the modal logic of provability to build agents that can achieve mutual cooperation in a manner that is robust, in that cooperation does not require exact equality of the agents' source code, and unexploitable, meaning that such an agent never cooperates when its opponent defects. We construct a general framework for such "modal agents", and study their properties.

#### Topics from this paper

#### 18 Citations

Robust program equilibrium

- Computer Science
- Theory and Decision
- 2018

It is argued that this program is similar to the tit for tat strategy for the iterated prisoner’s dilemma and generalizes this approach of turning strategies for the repeated version of a game into programs for the one-shot version of an game to other two-player games and proves that the resulting programs inherit properties of the underlying strategy. Expand

Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents

- Computer Science, Mathematics
- ArXiv
- 2016

This paper introduces an effective version of Lob's theorem which is applicable given such bounded resources and has powerful implications for the game theory of bounded agents who are able to write proofs about themselves and one another. Expand

Game-Theoretic Models of Moral and Other-Regarding Agents (extended abstract)

- Computer Science
- TARK
- 2021

This work investigates Kantian equilibria in finite normal form games, a class of non-Nashian, morally motivated courses of action that was recently proposed in the economics literature, and proposes some general, intuitive, computationally tractable, otherregarding equilibrium that interpolates between purely self-regarding and Kantian behavior. Expand

Cooperative and Competitive Reasoning: From Games to Revolutions

- 2018

I develop a game theoretic model where players use two different reasoning processes in strategic situations: cooperative and competitive. Players always consider cooperating at first: if they… Expand

Tiling Agents for Self-Modifying AI , and the Löbian Obstacle *

- 2013

We model self-modification in AI by introducing “tiling” agents whose decision systems will approve the construction of highly similar agents, creating a repeating pattern (including similarity of… Expand

Safe Pareto Improvements for Delegated Game Playing

- Computer Science
- AAMAS
- 2021

It is proved that the notion of safe Pareto improvements is closely related to a notion of outcome correspondence between games and is also shown that under some specific assumptions about how the representatives play games, finding safe Paringo improvements is NP-complete. Expand

Open Problems in Cooperative AI

- Computer Science
- ArXiv
- 2020

This research integrates ongoing work on multi-agent systems, game theory and social choice, human-machine interaction and alignment, natural-language processing, and the construction of social tools and platforms into Cooperative AI, which is an independent bet about the productivity of specific kinds of conversations that involve these and other areas. Expand

Problem Class Dominance in Predictive Dilemmas

- Political Science
- 2014

One decision procedure dominates a given one if it performs well on the entire class of problems the given decision procedure performs well on, and then goes on to perform well on other problems that… Expand

A NOTE ON IDENTIFYING VULNERABLE MORAL PROPENSITIES

- 2014

There are a variety of processes that steer the future; that is, they move it toward certain states and away from others dynamically, with changing behaviors in response to changing conditions. Our… Expand

Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda

- Political Science
- 2017

In this chapter, we discuss a host of technical problems that we think AI scientists could work on to ensure that the creation of smarter-than-human machine intelligence has a positive impact.… Expand

#### References

SHOWING 1-10 OF 26 REFERENCES

Program equilibrium—a program reasoning approach

- Mathematics, Computer Science
- Int. J. Game Theory
- 2013

This paper investigates an approach to program equilibrium in which richer conditions are allowed, based on model checking—one of the most successful approaches to reasoning about programs, and introduces a decision-tree model of strategies, which may be conditioned on strategies of others. Expand

Program equilibrium

- Computer Science
- Games Econ. Behav.
- 2004

It is shown that in a program equilibrium of the one-shot prisoners dilemma mutual cooperation is obtained, and more generally, the set of program equilibrium payoffs of a game coincides with theSet of feasible and individually rational payoff of it. Expand

A commitment folk theorem

- Economics, Computer Science
- Games Econ. Behav.
- 2010

This paper proposes a model of conditional commitments that unifies earlier models while avoiding circularities that often arise in such models. Expand

Modeling rational players I

- Computer Science
- 1987

This paper focuses on the non-cooperative analysis of games with some dynamic structure in which the choice of one move or another during the play of the game may convey valuable information to the other players. Expand

Modeling Rational Players: Part I

- Economics
- 1987

Game theory has proved a useful tool in the study of simple economic models. However, numerous foundational issues remain unresolved. The situation is particularly confusing in respect of the… Expand

Rationality, Computability, and Nash Equilibrium

- Economics
- 1992

Suppose two agents play a game, each using a computable algorithm to decide what to do, these algorithms being common knowledge. The author shows that it is possible to act rationally provided he… Expand

The evolution of cooperation.

- Sociology, Medicine
- Science
- 1981

A model is developed based on the concept of an evolutionarily stable strategy in the context of the Prisoner's Dilemma game to show how cooperation based on reciprocity can get started in an asocial world, can thrive while interacting with a wide range of other strategies, and can resist invasion once fully established. Expand

Some notes on Church's thesis and the theory of games

- Mathematics
- 1990

This paper considers games in normal form played by Turing Machines. The machines are fed as input all the relevent information and then are required to play the game. Some ‘impossibility’ results… Expand

Strong mediated equilibrium

- Computer Science
- Artif. Intell.
- 2006

It is shown that desired behaviors, which are stable against deviations by coalitions, can be obtained using mediators in a rich class of settings. Expand

Definable and Contractible Contracts

- Mathematics
- 2012

This paper analyzes Bayesian normal form games in which players write contracts that condition their actions on the contracts of the other players. These contracts are required to be representable in… Expand