Building an MMORPG - Here be monsters

In this session, Yan will share some of his experiences of building a successful MMORPG for a social audience and insights into some of the technical challenges that his team has had to overcome along the way.

Transcript

2. Name : Yan Cui Industry : Social Gaming
12. Homestead
13. Homestead
14. Farming
24. Catching Monsters!
36. In April..
37. • 200,000 unique players • 300,000 hours of game play • 100,000,000 server requests
38. • 45,000,000 items foraged • 1,000,000 butterflies caught • 4,000,000 seeds planted • 4,000,000 monsters caught
39. • 5,700,000 items bought • 24,000,000 items sold • 280,000 hours of time potion used • 526,000,000+ km travelled • around the earth 13,000 times...
40. • Bulk of the game developed by • 1 server developer • 2 flash developers • 1 producer and 2 game designers • Several artists
41. We have had to find ways to solve problems effectively and efficiently
43. • States can be BIG • Grow with content
45. • 1000s of items on homestead • Each item has some state
47. • Storage units add 1000s more items • Players like to collect things ;-)
50. • A lot of states to track on character • 100s of items in backpack • 100s of quests/achievements • Players like to collect things ;-)
51. • Many actions requires both states at once • Operations are expensive • Best players get the worst latency
52. Stateless Server Database Client
53. • Stateless server approach + Easy to scale up and down + Scaling can be dynamic based on load + Every layer can scale independently
54. • Stateless server approach – Serialization/deserialization is heavy on CPU – Heavy load (1:1 read write ratio) on databases – High bandwidth usage – High latency
55. • Stateless server approach – Inefficient – Expensive
56. Server Client Session 1 Session 2
57. • Traffic is bursty • Keep state on server during bursts = Less serialization & deserialization = Less load on database = Fewer server and database nodes =
58. Server Client Database
59. • Stateful server approach + 500% improvement on efficiency + 60% reduction in average latency + Fewer game servers + Fewer database nodes
60. But...
61. • Server affinity – Need to ensure server affinity during one burst – Need to route calls to affined server
62. • Load balancing – Some players are more active than others – Need to avoid hot spots
63. • Server hogging – Session lengths vary greatly – Need to avoid players hogging a server – Need to be able to scale down!
64. • Persist state after short inactivity • Move player to another server after persistence
66. • Scaling down – Need to gracefully drain player states – Need to be able to stop accepting new players
67. • Stateful server approach – Complexity + But worthwhile! + Others avoid scaling down to reduce complexity
68. • Progress tied to most actions • Avoid scripting • Data driven
69. Caught a Gnome
70. EXP Item Gold Quest Progress Caught a Gnome
71. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete
72. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest
73. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest
74. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest Achievement Progress
75. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest Achievement Progress Achievement Complete
76. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest Achievement Progress Achievement Complete
77. Level Up Quest Progress EXP Item Gold Caught a Gnome Quest Complete New Quest Achievement Progress Achievement Complete
78. And that’s not all...
79. • 100+ actions available in the game • Most can be tied to quests/achievements • Many yields rewards
80. • Triggered from different abstraction layers • Trapping controller • Level controller • ...
81. • Non-functional requirements • Analytics • Partner reporting • ...
83. • Message broker pattern • Something happened, it’s a FACT • Caught a Gnome • Got some EXP • ...
84. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting
85. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting Ignore Process Process Process Process Ignore
86. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold
87. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold
88. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold Process Ignore Ignore Ignore Process Ignore
89. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold Level Up
90. Caught Gnome Trapping Queue Levelling Quests Achievements Analytics Partner Reporting EXP Item Gold Level Up
91. • Message broker approach + Simple + Flexible + Extensible
92. • Message broker approach – Requires large number of different facts + F# to the rescue!
94. • OK for small number of DU cases • 100+ actions available in the game
96. • Players love quests
97. • Need regular content releases • Game is heavily data driven + No developer involvement for content changes • Tech team needs to be enablers
98. • Producers need ability to • Plan changes weeks in advance • Quickly see and validate changes before release
99. • Game designers need ability to • Work on changes in isolation • Quickly iterate and test changes in isolation
100. • QA need ability to • Quickly switch between different changes to test • Smoke-test changes in live environment without affecting real players
101. • Server team needs ability to • Version data changes • Rollback/hotfix data changes with no downtime
102. We version control our code, why not our data?
103. • Git + Git flow • Servers allow multiple data branches and switching between them
104. • What do our players like to do? • Every action in the game is recorded
105. Feature Analyze
106. • Analytics is a key decision-making tool • A/B testing
107. • 150,000,000+ events recorded in April • Several GBs of analytics data a day • Fast growing!
108. • Google BigQuery + Cheap + Fast + Scalable + SQL-like query syntax
109. • Google BigQuery – OLAP – Append-only
110. • What is our code doing?
111. • Monitoring • Not the same as profiling • Should be always ON • Difficult to set up in large distributed environment
112. What to monitor?
113. • Method execution time • e.g. IO, CPU intensive • Method execution count • e.g. service entry points • Error count • ...
114. • Amazon CloudWatch + Cheap + Supports alarms and notifications + Visualization tool
115. PostSharp
117. PostSharp Amazon CloudWatch + =
122. • Metrics are aggregated at instance level • Metrics gatherer has high concurrency requirement • F# agents to the rescue!
123. Thank You! @theburningmonk
124. Here Be Monsters http://apps.facebook.com/herebemonsters AOP and pseudo real-time monitoring with CloudWatch http://bit.ly/11GL3SQ http://bit.ly/10fe98N Building social games with a .Net stack http://bit.ly/ZKtHWb http://bit.ly/10CtyY9 F# Agents http://bit.ly/ZmXuEP http://bit.ly/17kjbam F# Deep Dives http://bit.ly/XZe10E