Building a Resilient API with open source

Transcript

188 slides

! Building a Resilient API with open source Wynn Netherland
GitHub
! Hello, my name is Wynn.
None
 Folks call me @pengwynn.
! I work @ GitHub.
! GitHub makes software for people who make software.
! I work mostly on the GitHub API.
! I make software for computers who work for people
who make software and other stuﬀ. — me, at Thanksgiving ” ”
! Building a Resilient API with open source I want
to talk about
! re·sil·ient /ri'zilyənt/ able to recoil or spring back into
shape after stretching, bending, or compressing "
! Stretching
! # LOAD #
! Bending
! Feature changes
! % Bug ﬁxes
! Compressing
! Feature removal
! Internal refactoring
! Infrastructure changes
! Change happens.
! Who 301'd my cheese?
! measure change & minimize change ' handle change (
At GitHub, we so we can in order to
! Measuring change &
! Use data to inform decisions.
! How often is this feature used?
! Who is using this feature?
! ”If an API method is removed in the woods,
and nobody is there to call it, does it still respond with 410?” ! — unknown
! /graph me -1hour @api.status.codes !
! ) Chatops
! Hubot *
! hubot.github.com
! /graph me -1hr @api.queries.serialized !
! /graph me -3hr @api.events.delivered !
! The Graph Store +
! The Graph Store + There's a graph for that™
! The Graph Store + There's a graph for that™
…if not, make one.
! Built on Graphite
!
!
! , Instrumentation
! # counter GitHub.stats.increment("the-thing.the-action") ! # gauge GitHub.stats.gauge("the-thing.the-gauge", thing.current_reading) !
# timing GitHub.stats.time("the-thing.the-operation") { the_operation } -
! github.com/graphite-project
! . Logging
! /splunk -3m @production "timeline.json" ! pengwynn: ! 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd
local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" request_id="AD003EA0:5BD8:4B8AE05:53BEC973" remote_address=192.168.1.1 request_method=get path_info="/timeline.json" content_length=375 user_agent="@gitlost account on twitter, Python urllib2" accept=nil language=nil status=410 at=finish elapsed=0.005 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" user=nil repo=nil route=nil zone=UTC referrer=nil requested_at="2014-07-10 17:12:19 UTC" url="https://github.com/timeline.json" controller=EventsController action=index request_id="AD003EA0:5BD8:4B8AE05:53BEC973" dbconn=write serializer=json at=start 2014-07-10T10:12:19-07:00 github-fe117-cp1-prd local7:info unicorn_rails[12813]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00"
! /splunk -3m @production "timeline.json" ! pengwynn: ! 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd
local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" request_id="AD003EA0:5BD8:4B8AE05:53BEC973" remote_address=192.168.1.1 request_method=get path_info="/timeline.json" content_length=375 user_agent="@gitlost account on twitter, Python urllib2" accept=nil language=nil status=410 at=finish elapsed=0.005 2014-07-10T10:12:19-07:00 github-fe126-cp1-prd local7:info unicorn_rails[11990]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00" user=nil repo=nil route=nil zone=UTC referrer=nil requested_at="2014-07-10 17:12:19 UTC" url="https://github.com/timeline.json" controller=EventsController action=index request_id="AD003EA0:5BD8:4B8AE05:53BEC973" dbconn=write serializer=json at=start 2014-07-10T10:12:19-07:00 github-fe117-cp1-prd local7:info unicorn_rails[12813]: app=github env=production enterprise=false now="2014-07-10T10:12:19-07:00"
! /splunk -3m @production "timeline.json" | stats count by status,
avg(elapsed) ! pengwynn: ┌──────┬────────────┐ │status│avg(elapsed)│ ├──────┼────────────┤ │200 │0.067745 │ ├──────┼────────────┤ │410 │0.005597 │ └──────┴────────────┘
! key=value logging
! / app=github env=development enterprise=false now="2014-07-21T11:49:45-07:00" request_id=f80765a8-8aa9-4e92-b32b-01300193631a remote_address=127.0.0.1 request_method=get path_info="/"
content_length=1866 user_agent=curl/7.30.0 accept=*/* language=nil status=200 at=finish elapsed=0.110
! / app=github env=development enterprise=false now="2014-07-21T11:49:45-07:00" request_id=f80765a8-8aa9-4e92-b32b-01300193631a remote_address=127.0.0.1 request_method=get path_info="/"
content_length=1866 user_agent=curl/7.30.0 accept=*/* language=nil status=200 at=finish elapsed=0.110
! curl https://api.github.com/ -I | grep Request-Id ❯ / !
X-GitHub-Request-Id: 4317CCE2:6434:21DCB73:53CD6239
! curl https://api.github.com/ -I | grep Request-Id ❯ / !
X-GitHub-Request-Id: 4317CCE2:6434:21DCB73:53CD6239 A single identifier to trace requests through logging.
! /splunk -3m @production request_id=f80765a8-8aa9-4e92- b32b-01300193631a !
! github.com/asenchi/scrolls
! github.com/technoweenie/grohl If Golang is your thang.
! Logstash, Splunk*, Loggly* *SaaS solutions, not open source
! Minimizing change '
! Isolation
! tree -d -L 2 github ❯ / github ├──
app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ...
app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ... Sinatra app inside a Rails app
app │ ├── api │ ├── assets │ ├── controllers │ ├── helpers │ ├── mailers │ ├── models │ ├── view_models │ └── views ├── bin ├── config │ ├── inititalizers ... Sinatra app inside a Rails app
! Shared business logic data access logging instrumentation plumbing Independent
workﬂows output
! Testing ❯ testrb test/integration/api/repos_test.rb Loading suite in dotcom mode
... Run options: # Running tests: Finished tests in 30.638101s, 4.7979 tests/s, 64.7886 assertions/s. 147 tests, 1985 assertions, 0 failures, 0 errors, 0 skips ruby -v: ruby 2.1.0p0-github (development) [x86_64-darwin13.0]
! Rack::Test
! brynary/rack-test github.com/
! # test/integration/api/root_test.rb ! test "provides hypermedia URLs to top-level
resources" do get "/" data = JSON.parse(last_response.body) assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -
resources" do get "/" data = JSON.parse(last_response.body) assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -
resources" do data = api :get, "/" ! assert_equal 200, last_response.status assert_equal "https://#{GitHub.api_host_name}/user", data["current_user_url"] end -
! JSON Schema
! json-schema.org
! Validating output.
! / # Running tests: ! [1/5] Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 = 2.06
s 1) Failure: Api::RateLimitTest#test_OAuth_app_with_custom_rate_limit_L63 [/Users/ wynn/github/github/test/integration/api/rate_limit_test.rb:67]: {"resources"=>{"core"=>{}, "search"=>{}}, "rate"=>{}} The property '#/rate' did not contain a required property of 'limit' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'remaining' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1# The property '#/rate' did not contain a required property of 'reset' in schema c1ac8566-70fa-5c34-b050-0cfa6801c1c1#
! Validating input.
! / # update a label $ curl -ni http://api.github.dev/repos/defunkt/dotjs/labels/bug
\ -X PATCH -H "Content-Type: application/json" \ -d '{ "name":[], "color":42 } ! HTTP/1.1 422 Unprocessable Entity ! { "message": "Invalid request.\n\nExpected data to be of type \"string \"; value was: [].\nExpected data to be of type \"string\"; value was: 42.", "documentation_url": "https://developer.github.com/v3/issues/labels/ #update-a-label" }
! / # update a label $ curl -ni http://api.github.dev/repos/defunkt/dotjs/labels/bug
\ -X PATCH -H "Content-Type: application/json" \ -d '{ "name":[], "color":42 } ! HTTP/1.1 422 Unprocessable Entity ! { "message": "Invalid request.\n\nExpected data to be of type \"string \"; value was: [].\nExpected data to be of type \"string\"; value was: 42.", "documentation_url": "https://developer.github.com/v3/issues/labels/ #update-a-label" }
! MiniTest
! Custom assertions
! test "can create a pull request" do auth_as @forker
! data = api :post, @pub_pulls_url, {}, :input => { :title => "new pr", :body => "this is an amazing change", :base => "master", :head => "#{@forker}:ahead" }.to_json assert_equal 201, last_response.status assert_schema :v3, :pull_request, data end -
! test "can create a pull request" do auth_as @forker
! data = api :post, @pub_pulls_url, {}, :input => { :title => "new pr", :body => "this is an amazing change", :base => "master", :head => "#{@forker}:ahead" }.to_json assert_equal 201, last_response.status assert_schema :v3, :pull_request, data end -
! test "can retrieve pull requests" do auth_as @owner !
data = api :get, @pub_pulls_url assert_equal 200, last_response.status ! assert_schema_for_collection \ :v3, :pull_request_simple, data end -
! test "can retrieve pull requests" do auth_as @owner !
data = api :get, @pub_pulls_url assert_equal 200, last_response.status ! assert_schema_for_collection \ :v3, :pull_request_simple, data end -
! Even more assertions
! test "authz list commits for pull request" do !
assert_authz :get, @priv_commits_url do |check| check.disallows_with :not_found check.authorized_users @owner, @collaborator check.unauthorized_users :anon, @rando check.minimum_scopes :repo end end -
! 5 lines = dozens of AuthZ assertions.
! But what about those critical code paths?
! What if you don’t trust your tests?
! Science 0
! github/dat-science github.com/
! @jbarnette
! @rick
!
!
! Safely observe alternate code paths for non-mutating methods that
return a value.
! require "dat/science" ! class MyApp::Widget def allows?(user) experiment =
Dat::Science::Experiment.new \ "widget-permissions" do |e| ! e.control { old_method } # old way e.candidate { new_method } # new way end ! experiment.run end end -
Runs candidate before control 50% of the time Measures the
duration of both behaviors Compares the results of both behaviors Swallows any exceptions raised by the candidate behavior Publishes ﬁndings for tracking and reporting
! Fancy a DSL instead?
! require "dat/science" ! class MyApp::Widget def allows?(user) science "widget-permissions"
do |e| e.control { model.check_user(user).valid? } e.candidate { user.can? :read, model } end end end -
! require "dat/science" ! class MyApp::Widget def allows?(user) science "widget-permissions"
do |e| e.control { model.check_user(user).valid? } e.candidate { user.can? :read, model } end end end -
! Create and run.
! require "dat/science" ! class Blog::Post def permalink(user) science "permalink"
do |e| e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -
do |e| e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -
! Ramping up experiments.
! require "dat/science" ! module Myapp class Experiment < Dat::Science::Experiment
def enabled? rand(100) < 10 # Run 10% of the time end end end -
! github.com/jnunemaker/ﬂipper!
def enabled? MyApp.flipper[name].enabled? end end end -
! Publishing results.
def publish MyApp.instrument "science.#{event}", payload end end end -
def publish MyApp.instrument "science.#{event}", payload end end end -
None
None
None
! Adding custom context.
do |e| e.context :post => post e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -
do |e| e.context :post => post e.control { post[:permalink] } e.candidate { post.slug } e.comparator {|a, b| a.downcase == b.downcase } end end end -
Mad science.
1 Don't try this at home
! module GitHub class Experiment < Dat::Science::Experiment class << self
# Public: enable mad science mode: returns the candidate values by default # instead of the control. def mad_science=(val) @mad_science = val end ! # Internal: whether or not to always run the candidate instead of the # control. def candidate_instead_of_control? @mad_science end end … end -
! module GitHub class Experiment < Dat::Science::Experiment class << self
# Public: enable mad science mode: returns the candidate values by default # instead of the control. def mad_science=(val) @mad_science = val end ! # Internal: whether or not to always run the candidate instead of the # control. def candidate_instead_of_control? @mad_science end end … end -
! Analyzing results.
! github.com/github/dat-analysis
! Build an analyzer to poke on results.
! require 'dat/analysis' ! module MyApp class Analysis < Dat::Analysis
def read Redis.rpop "dat-science.#{experiment_name}.results" end ! def count Redis.llen "dat-science.#{experiment_name}.results" end ! def cook(raw_result) return nil unless raw_result JSON.parse(raw_result) end end end -
! / irb> a = MyApp::Analysis.new('widget-permissions') => #<MyApp::Analysis:0x007fae4a0101f8 …> !
! !
irb> a.fetch => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } ! !
irb> a.fetch => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } ! irb> a.result => {"experiment"=>"widget-permissions", "user"=>{ ... } .... } !
! / irb> a.result.control => {"duration"=>12.307, "exception"=>nil, "value"=>false} ! !
!
! / irb> a.result.control => {"duration"=>12.307, "exception"=>nil, "value"=>false} ! irb>
a.result.candidate => {"duration"=>12.366999999999, "exception"=>nil, "value"=>true} !
! Handling change (
! New formats
! Features evolve. URLs don’t have to.
! / [ "[email protected]", "[email protected]" ] curl -n http://api.github.com/user/emails ❯
! / [ "[email protected]", "[email protected]" ] curl -n http://api.github.com/user/emails ❯
! / curl -n http://api.github.com/user/emails \ -H "Accept: application/vnd.github.v3" [
{ "email": "[email protected]", "primary": false, "verified": true }, { "email": "[email protected]", "primary": false, "verified": true } ] ❯
! Media types
! curl https://api.github.com/users/technoweenie -I / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3
! curl https://api.github.com/users/technoweenie -I curl https://api.github.com/users/technoweenie -I \ -H "Accept:
application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json
application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json curl https://api.github.com/users/technoweenie -I \ -H "Accept: application/vnd.github.beta.full+json" HTTP/1.1 200 OK X-GitHub-Media-Type: github.beta; param=full; format=json
application/vnd.github.full+json" / HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3 HTTP/1.1 200 OK X-GitHub-Media-Type: github.v3; param=full; format=json curl https://api.github.com/users/technoweenie -I \ -H "Accept: application/vnd.github.beta.full+json" HTTP/1.1 200 OK X-GitHub-Media-Type: github.beta; param=full; format=json
! @jasonrudolph “Beta like you mean it.”
! Media types as beta code words.
! curl https://api.github.com/the/new/shiny / HTTP/1.1 415 Unsupported Media Type !
{ "message": "Unsupported 'Accept' header: github.v3." }
{ "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK
{ "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK
{ "message": "Unsupported 'Accept' header: github.v3." } curl https://api.github.com/the/new/shiny \ -H "Accept: application/vnd.github.new-shiny-preview" HTTP/1.1 200 OK acknowledge API is a preview release
! Client testing
None
! octokit.github.com
! github.com/octokit/octokit.rb
! github.com/rspec/rspec
! describe Octokit::Client do ! describe ".rate_limit" do it "makes
a response if there is no last response" do client = Octokit::Client.new VCR.use_cassette "rate_limit" do rate = client.rate_limit ! expect(rate.limit).to be_kind_of Fixnum expect(rate.remaining).to be_kind_of Fixnum end end # .rate_limit end … -
! Record live API requests to play back later. Is
it live, or is it Memorex®
! github.com/vcr/vcr
! Environment smoke tests
! / rm -rf spec/cassettes ❯ ~/.coral/repos/octokit.rb@octokit master ❯ ~/.coral/repos/octokit.rb@octokit
master*
! / rm -rf spec/cassettes ❯ ~/.coral/repos/octokit.rb@octokit master OCTOKIT_API_ENDPOINT="https://rails5.github.com" rspec
❯ ~/.coral/repos/octokit.rb@octokit master* ................................................. ..............F.................................. ........................FFF...................... ...................F............................. ........F........................................ ................................................. .....................F........F..................
! measure change & minimize change ' handle change (
At GitHub, we so we can in order to
! One more thing.
! github.com/interagent
! 410 Gone