In game theory, generalists sometimes win out over specialists in reinforcement learning

This could be an interesting new paper by Zico Kolter.

"... it does offer new insights into so-called imperfect-information games that involve two contestants facing off in a “zero-sum” competition, where one player’s gain means the other player’s loss. ...

The focus of the new work is on algorithms that could be used to train neural networks to participate in imperfect-information games. The assumption, long-held in the field, was that algorithms grounded in principles of game theory would, in this setting, clearly outcompete a general-purpose variety of algorithms called policy gradient methods, which came into use for decision-making in the 1990s. The term “policy” in this context basically means strategy, whereas “gradient” refers to a path that leads in the direction of greatest change — to the top (or bottom) of a hill, for example. Policy gradient methods are being used to train neural networks to make decisions that move — in small, sequential steps — toward a particular goal (like reaching a summit, metaphorically speaking), with continual adjustments and course corrections made along the way to bring the agent closer to the intended destination. ..."

From the abstract:

"In the past decade, motivated by the putative failure of naive self-play deep reinforcement learning (DRL) in adversarial imperfect-information games, researchers have developed numerous DRL algorithms based on fictitious play (FP), double oracle (DO), and counterfactual regret minimization (CFR).

In light of recent results of the magnetic mirror descent algorithm, we hypothesize that simpler generic policy gradient methods like PPO are competitive with or superior to these FP-, DO-, and CFR-based DRL approaches. To facilitate the resolution of this hypothesis, we implement and release the first broadly accessible exact exploitability computations for five large games.

Using these games, we conduct the largest-ever exploitability comparison of DRL algorithms for imperfect-information games. Over 7000 training runs, we find that FP-, DO-, and CFR-based approaches fail to outperform generic policy gradient methods."

In game theory, generalists sometimes win out over specialists | MIT News | Massachusetts Institute of Technology "Researchers show that for certain kinds of games, an overlooked class of algorithms performs much better than expected."

Reevaluating Policy Gradient Methods for Imperfect-Information Games