Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling Instagram at AirBnB OpenAir 2015

Scaling Instagram at AirBnB OpenAir 2015

Mike Krieger

June 04, 2015
Tweet

More Decks by Mike Krieger

Other Decks in Technology

Transcript

  1. unicorn & proxygen & scribe & thrift nginx & cassandra

    & memcached & postgres & rabbitmq & django
  2. CAROUSEL ADS ADS >  fab  -­‐z20  djangos  update_git    

    >  fab  -­‐z20  djangos  restart_django
  3. CAROUSEL ADS ADS >  fab  -­‐z20  djangos  rollout:server   ...doing

     fresh  git  fetch     ...zipping  up  origin/master   ...uploading  to  S3   ...pulling  down  zip   ...unpacking  zip   ...mapping  'current'  symlink   ...restarting  Django
  4. CAROUSEL ADS ADS “did you remember to roll to a

    canary?” “don't roll to the workers with a -z of > 40!” “did you tail the error logs?” “did you catch that new tier we deployed?”
  5. CAROUSEL ADS ADS >  fab  -­‐z20  djangos  rollout:server   ...grabbing

     lock  from  Sauron   ...doing  fresh  git  fetch     ...zipping  up  origin/master   ...uploading  to  S3   ...pulling  down  zip  to  canary  1   ...unpacking  zip  on  canary  1   ...mapping  'current'  symlink  on  canary  1   ...restarting  Django  on  canary  1
  6. CAROUSEL ADS ADS ...tailing  error  logs  on  canary  1  

    ...ok,  200  responses  are  even   ...deploying  to  async  worker  1   ...measuring  success  rate  on  worker  1   ...looks  good,  deploying  widely
  7. ADS

  8. CAROUSEL ADS ADS CELEBRITY_OVERRIDES  =  {      'taylor  swift':

     19151555,      'taylorswift':  19151555,      'justinbieber':  6860189,      'justin  bieber':  6860189   } ACTUAL CODE :(
  9. CAROUSEL ADS ADS curl  -­‐XPUT  'http://solr/update/json'  -­‐d  '{    

         {"add":                {"doc":  {                  "username"  :  "justinbieber",                  "followed_by":  12345678              }          }   }'
  10. CAROUSEL ADS ADS -­‐  CELEBRITY_OVERRIDES  =  {   -­‐  

     'taylor  swift':  19151555,   -­‐    'taylorswift':  19151555,   -­‐    'justin  bieber':  68680189   -­‐  }
  11. CAROUSEL ADS ADS curl  -­‐XPUT  'http://es:9200/users/user/6860189'  -­‐d  '{    

         "username"  :  "justinbieber",          "followed_by":  12345678   }'
  12. CAROUSEL ADS ADS //  people  who  I  follow  named  Justin

      (and  (term  justin*)            (term  followedby:4))   //  people  followed  by  the  people  I  follow,  named  Justin   (and  (term  justin*)            (apply  followedby:(term  followedby:4))   //  people  named  Justin,  prioritizing  the  people  I  follow   (weak-­‐and  (term  followedby:4  :optional-­‐hits  2)                      (term  justin*))
  13. CAROUSEL ADS ADS //  photos  I  haven't  liked,  but  the

     people  I  follow  liked   (difference          (or  likedby:friendA  likedby:friendB  …)                    likedby:4   )
  14. ADS

  15. CAROUSEL ADS ADS //  photos  I  haven't  liked  yet,  liked

     by  people  whose  photos   I  already  liked   (difference          (apply  liker:              (extract  owner:  liker:4))          liker:4)