Share
Embed
Transcript
65 slides
-
This is a conversation about ideas, it is not a
prescription for how you have to do it - Chris Kelly, @amateurhuman (paraphrased) -
15 second bio • Chief Software Architect @ Kabam •
We make mobile and web games • #1 Top Grossing iOS app of 2012 Kingdoms of Camelot • Of course we’re hiring –30– -
config/database.yml production: adapter: mysql2 host: partytimedb1.colo database: partytime_production username: partytime
password: d34db33f timeout: 5000 pool: 10 encoding: utf8 reconnect: true Ways to interact with external systems -
config/locales/es.yml es: hello_world: “hola mundo” ice_cream: “helado” soccer: “fútbol” football:
“fútbol americano” Data that will be used elsewhere in your system, like translations -
app/models/post.rb class Post < ActiveRecord::Base @@per_page = 30 end Constants
and presentation logic sprinkled all over the place; magic numbers This is probably one of the most common magic numbers, and yes that is in completely the wrong place. -
app/models/quest.rb class Quest < ActiveRecord::Base GOLD_COINS_REWARD = 50 end And
here is a business logic constant; how many coins will you get for completing a quest? -
config/envs/production.rb Partytime::Application.configure config.cache_classes = true config.assets.compile = false config.assets.digest =
true end Hm, this one is starting to look like code to me. -
Configuration is... • everything you want it to be •
for our purposes: values that you may want to share, or change Revel in this gloriously ambiguous description -
Configuration, ahoy • 15+ di erent types of buildings •
10+ levels for each building • 8 di erent requirements per level • 1,200+ values Shared-world strategy game -
Configuration, ahoy • Research • Knights • Battles • NPCs
• Items • Quests • etc... -
Getting serious • Technical QA • Playtesting QA • Deploying
new configurations just as rigorously as new code • Bad config can be just as damaging as bad code Technical QA - did we break anything from an engineering sense? Playtesting QA - did we break anything from a game design sense? Anything become OP or get nerfed? we have millions of players, the stakes are high. -
Agenda • Configuration context • Baking configurations • Deploying configurations
• Applications • Further exploration Configuration context - hwo do we make our configurations more easily relevant to an environment, a datacenter, a region, a machine? Baking configurations - how can we make this a serious build process? Deploying configurations - we put a lot of care into deploying code, can we do the same for config? Not all of these are necessarily in use on this game, but they are here in my presentation for you to think about. So let’s dive in to this amazing agenda, we could do all this -
...and, sometimes, a prize! the actual code for handling the
click is left as an exercise for the viewer. -
#1 class PuppyClick COINS_PER_CLICK = 10 ODDS_OF_WINNING_PRIZE = 0.1 PRIZES
= %w{yo-yo candy} ... end -
#2: values out of source # config/game.yml game: coins_per_click: 10
odds_of_winning_prize: 0.1 prizes: - ‘candy’ - ‘yo-yo’ -
#3: respect Rails.env development: coins_per_click: 10 odds_of_winning_prize: 1 prizes: -
‘candy’ - ‘yo-yo’ test: coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: - ‘candy’ on dev i always want a prize -
#4: composite config core.yml coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: -
‘candy’ - ‘yo-yo’ development.yml odds_of_winning_prize: 1 test.yml prizes: - ‘screen_test_256_colors’ -
#4: composite config core = Yaml.load ‘config/core.yml’ env = Yaml.load
“config/#{Rails.env}.yml” config = core.deep_merge(env) composite config introduces precedence -
#4: composite config core.yml coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: -
‘candy’ - ‘yo-yo’ development.yml odds_of_winning_prize: 1.0 test.yml prizes: - ‘screen_test_256_colors’ -
#4: composite config > Rails.env => “development” > config[:coins_per_click] =>
10 # from core.yml > config[:odds_of_winning_prize] => 1.0 # from development.yml -
#5: more context • datacenters • AWS regions • roles
• machine names -
#5: more context @config = {} files = [ “config/core.yml”,
“config/envs/#{Rails.env}.yml”, “config/regions/#{AWS_REGION}.yml”, “config/hosts/#{HOSTNAME}.yml” ] files.each do |filename| context = Yaml.load filename @config.deep_merge(context) end -
#6: tags • apply a config based on ENV •
canary or A/B new code • assign role-specific configuration • debugging -
#6: tags tags = ENV[‘TAGS’].split(‘,’) # load all other configs
tags.each do |tag| filename = “config/tags/#{tag}.yml” context = Yaml.load filename @config.deep_merge(context) end -
rails_config • Settings.coins_per_click • Settings[:coins_per_click] • reloading of settings files
• composite configurations • developer-local settings • http://github.com/railsjedi/ rails_config Seemingly the marquee feature for any Rails config library, method notation! -
And yet $ vim $ rake spec $ git add
-u $ git commit -m “...” $ git push origin master $ cap deploy -
Reset • Our settings are aware of their context and
environment! • But still in the source repo • Still tied to a commit and deploy • Still inaccessible to other apps • Still inaccessible to non-engineers -
Requirements • Decouple generation process entirely from app/repo • Use
code deployment principles • Build an artifact (like we used to) • Version and store • Deploy What if we could generate a configuration instead? -
The power of the internet • A webapp!!! • Single,
enforceable process • Access controls, individual auditing • Bounds/sanity checking • Incorporate your particular corporate rules/insanity Anecdote about sanity checking (putting 100000 in the yml file) * sanity checking, etc is outside the bounds of your app but good for generating configs! -
Wrap an app around this dim name key value core
core coins_per_click 10 core core odds_of_winning_prize 0.1 core core prizes [‘candy’, ‘yo-yo’] env dev odds_of_winning_prize 1.0 env test prizes [‘screen_test_256_colors’] Decomposed our configuration data into a table... not normalized at all of course. Yes this works even better with a document store... -
Dump directly from DB? • slow • dogpile • harder
to replicate than static http asset • unnecessary load on your most precious resource • may be OK for your use case -
Build process • transform config values into a single Yaml
doc - building an artifact • take note of: builder, build #, timestamp, changelog • include checksum to prevent alteration, well-intentioned or otherwise -
New values dim name key value core core coins_per_click 20
core core odds_of_winning_prize 0.1 core core prizes [‘candy’, ‘yo-yo’] env dev odds_of_winning_prize 1.0 env test prizes [‘screen_test_256_colors’] -
Our new Yaml doc config: coins_per_click: 20 odds_of_winning_prize: 0.1 prizes:
- ‘candy’ - ‘yo-yo’ environments: development: odds_of_winning_prize: 1 test: prizes: - ‘screen_test_256_colors’ -
Artifact • r8471-double_coins_promo- bac8318fcd59afa3912f3c9.yml • http://config/artifacts/8471.yml production.yml - an eventual
promotion! this is how we get a config there -
Promotion • Map a revision to an environment or other
dimension • http://config/artifacts/8471.yml • http://config/envs/playtest.yml • Promotion can be end-user accessible production.yml - an eventual promotion! this is how we get a config there -
Reset • Our settings are aware of their context and
environment! • Settings are outside the repo! • Accessible to other apps! • Changes accessible to non-engineers, but not deploys • How do we best deploy config changes? -
Release process • run test/CI suites against config • run
QA against config • run playtest against config • deploy to production -
Good... # config/environment.rb [ “envs/#{Rails.env}.yml”, “regions/#{AWS_REGION}.yml”, “hosts/#{HOSTNAME}.yml” ].each do |filename|
yaml = Yaml.parse(Http.get “http:// config/#{filename}”) (@config ||= {}).deep_merge!(yaml) end Still have to restart to see changes -
But I don’t want to restart • Spawn a thread
to poll for updates in the background • Celluloid timer is perfect for this • Doesn’t work with GIL (MRI) • Pretty gross multi-process (Unicorn) • Don’t forget to be threadsafe -
Timer class ConfigFetcher include Celluloid def initialize(sources) refresh @timer =
after(30) { refresh; @timer.reset } end def refresh # poll and assemble config... end end -
Better... > Config.get(‘coins_per_click’) => 10 # ... deploy new config
to production, wait for refresh ... > Config.get(‘coins_per_click’) => 20 -
Best Config.on_change ‘coins_per_click’ do |new_value| Logger.warn “Coins changed to #
{new_value}” # break cache end -
Why poll and not push? • What happens if things
go wrong? • Failed push means app carries on with old config • Responsibility in the wrong place • If poll fails, app can shut itself down -
Reset • config is out of source • non-engineers can
build and deploy • config can be shared across apps • update config live without restarts • test configs before production deploy • recorded change control -
Other applications • rapid A/B testing • I18n and translation
management • feature flags -
Bonus non-Ruby content • Archaius: Netflix’s Java composite configuration library
• https://github.com/Netflix/archaius -
Bonus non-Ruby content • ZooKeeper: cluster synchronization service (strongly consistent)
• Nodes with watches • http://zookeeper.apache.org/ -
Bonus non-Ruby content • jones: Python config app • ZooKeeper-backed
• Composite configurations (views) • https://github.com/mwhooker/jones