The AI Economist

Optimal Economic Policy Design via Two-level Deep Reinforcement Learning
Stephan Zheng et al., 2021

  • Economic policy shapes how people work, trade, and share the benefits of production.
  • Analytical models need simplifying assumptions. Real-world policy experiments are difficult, and historical data cover only policies already tried.
  • Could a simulation with learning agents become a tool for designing economic policy?

The research idea

Learn economic behavior and economic policy together.

Agents learn to pursue their own interests. A planner learns rules that improve a chosen social objective.

Two learners, different objectives

Economic agentsSocial planner
ChooseHow to gather, trade, and buildIncome tax rates
Learn to improveOwn utility from income and effortA measure of social welfare
Respond toResources, other agents, and taxesThe economy agents create

When the planner changes taxes, it changes the incentives the agents are learning under.

A simulated economy for testing the idea

Agents collect resources, trade, and build houses. Skill and resource access give them different opportunities.

Making two-level learning work

  1. Learn to earn.Agents first learn useful behavior without taxes. Labor costs increase gradually.
  2. Learn under changing incentives.Introduce taxes and expose agents to varied tax schedules.
  3. Keep both levels adapting.Exploration regularization supports gradual learning by agents and planner.

Early exploration can be costly. The training procedure helps agents keep exploring long enough to learn.

A check against economic theory

In a one-step economy, each agent chooses labor independently.

Figure 3a: the AI Economist and Saez achieve similar utilitarian welfare in a one-step economy; their difference is not statistically significant. Both exceed the free-market and US Federal baselines.
Figure 3b: learned and Saez marginal tax schedules closely track one another across income brackets. The legend identifies Free Market, US Federal, Saez Formula, and AI Economist.

The learned policy closely matches the Saez solution in both welfare and tax rates.

Zheng et al. (2021), Figure 3a–b. Five training seeds; error bars show standard error. N.S. = not significant.

Better policy in the studied economies

Four-agent OpenQuadrant. Separate experiments optimize each welfare objective.

>8% higher utilitarian welfare
Figure 3c, four-agent panel: AI Economist has the highest inverse-income-weighted utility, ahead of Saez, US Federal, and Free Market, with standard-error bars and significance annotations.
>12% higher equality × productivity
Figure 3d, four-agent panel: AI Economist has the highest equality-times-productivity score, ahead of Saez, US Federal, and Free Market. Bar colors use the same legend as panel c.
Free MarketUS FederalSaez FormulaAI Economist

Paper results, Figure 3c–d (four-agent panels). Gains compare with Saez. Ten training seeds; error bars show standard error.

What would you investigate?

Appendix

Implementing the AI Economy

World · Worker · Agent · Learning
Market · Tax and welfare · Two-level learning

1

The world: where resources appear

Change coverage and the seed. Harvest resources, then watch them regrow as time advances automatically.

2

The worker: gather and build for yourself

Select the world to move with the arrow keys. Gather wood and stone, then press B to build on clear ground.

3

The agent: let a policy choose the action

Agents move according to a policy, a probability distribution for taking an action given the observed state. Step through a recorded PPO policy.

4

Learning: experience changes the policy

Policies evolve through experience to increase return. Watch how 4 versions differ after varying levels of training on episodes of length 120.

5

Deep reinforcement learning: how learning works under the hood

CNNs and LSTMs compute action probabilities towards maximizing future reward as well as what that expected future reward would be. PPO compares the estimated and actual reward sequence to improve both networks.

6

The market: people can buy and sell resources

Watch how a market lets workers buy and sell resources to reduce labor costs of crossing the map.

7

Learning produces division of labor with a market

Four workers on the full 25 × 25 map. Compare their learned activities with trading off or on, using the same skills and training budget.

8

Tax: collect income tax, then share it equally

Every 100 steps, workers pay tax on their income. Follow the coin into the government pool and back out as an equal rebate.

9

Tax: different rules, different outcomes

Choose two tax policies to watch on the full map. Compare all four on the right: how do production and its distribution change?

10

Two-level learning: using a curriculum to smooth simultaneous training

Workers adapt as the planner learns taxes. Compare no curriculum with gradual taxes and entropy in recorded episodes.