[Master Thesis] Diversity and Novelty as Objectives in Poker

Transcript

1. Diversity and Novelty as Objectives in Poker Jéssica Pauli de C. Bonson Supervisor: Dr. Malcolm I. Heywood Co-Supervisor: Dr. Andrew R. McIntyre
2. Motivation • Evolutionary algorithms can lead to efficient solutions without a predefined design • Downsides: ◦ prone to early convergence ◦ may be deceived by a non-informative or deceptive fitness function 2
3. Deceptive tasks 3
4. Diversity applied to Deceptive Tasks 4
5. Texas Hold’em Poker 5
6. Methodology • Evolve agents with various combinations of diversity maintenance methods and fitness-based evolution • Compare them in ten scenarios for diverse types of hands and opponents • Analyze effects on diversity, performance and behaviors 6
7. SBB architecture 7
8. Methodology: Fitness Function • The performance of a player is measured by the average chips won per hand 8
9. Methodology: Opponents • Static opponents • Bayesian opponents • Hall of Fame opponents 9
10. Methodology: Diversity • Diversity maintenance methods: ◦ bid diversity ◦ genotypic diversity ◦ behavioral diversity ◦ novelty search 10
11. Methodology: Inputs • Groups ◦ Game State ◦ Opponent Model 11
12. Experiments • Opponent Complexity Group • Degrees of Diversity Group • Diversity Models Group • Analysis of the behaviors 12
13. Results 13
14. Opponent Complexity Group 14
15. Degrees of Diversity Group 15
16. Results: Diversity Models Group • Comparisons using the cumulative plots ◦ Diversity and Performance ◦ 9 models in 10 scenarios ◦ Tests: Friedman, Bonferroni-Dunn, and Nemenyi 16
17. Results: Diversity Models Group • Most models were able to improve the diversity of the agents • Two models translated diversity into performance 17
18. Results: Diversity Models Group • The results indicate that novelty search alone does not work well for Texas Hold'em Poker • The model with novelty search and fitness was significantly better than the one without fitness 18
19. Results: Analysis of Behaviors • Formulated the hypothesis that novelty would incentive bluffing • The model with only novelty search bluffed as much as 3 models, and significantly more than 5 models 19
20. Playing Styles (Balanced and Unbalanced) 20
21. Cumulative Score vs Cumulative Hands Won 21
22. Conclusions • Diversity maintenance methods were able to improve diversity and performance • Novelty search alone was not enough to improve neither diversity nor performance • Diversity was useful mainly to increase the exploitation of chips per hand 22
23. Future Work • Find a way to deploy a subset of the agents • Further test diversity and novelty on a more ambiguous and complex version of Poker 23
24. Thank you! 24
25. Extras 25
26. Motivation • How to deal with deceptive tasks? ◦ Diversity maintenance ▪ Genotypic diversity ▪ Behavioral diversity ▪ Novelty search 26
27. Background: Inputs • Game State Inputs ◦ Hand Strength, Effective Potential, Pot Odds, Betting Position, Round • Opponent Model Inputs ◦ Last Action, Overall Long-term Aggressiveness, Overall Short-term Aggressiveness, Hand Aggressiveness, Tight/Loose, Passive/Aggressive, Bluffing, Chips, Self Overall Short-term Aggressiveness 27
28. Background: Hands • Each point corresponds to a poker hand. • Training points are balanced in nine categories, per hand strength. • Real-world hands: 60% weak, 30% intermediate, 10% strong. 28
29. Differences from Previous Work • The main differences between the work developed by Alberta's group and this research ◦ evolve a diverse group of capable agents ◦ agents evolve their strategies from scratch ◦ agents work as teams of programs ◦ it is not possible to use simulations ◦ use poker as a domain, not as the goal 29
30. Overall flow of the classic Genetic Algorithm 30
31. Most used inputs per teams 31
32. • Training chart? Too noisy • Tested before in other tasks • Why not tournament? To focus on diversity • Normalized between 0.0 and 10.0 ◦ Better for SBB due to previous work results 32 Possible Questions
33. Bluffing • Behavior ◦ teams play less hands to avoid losing chips due to weaker hands ◦ they also increase their bluffing, to exploit the opponent's weaker hands • The teams are using their opponent modeling inputs to find when the opponent seems to have weaker hands, and then bluff 33