Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Instagram Under the Hood

Carl Meyer
November 04, 2016

Instagram Under the Hood

Django Under the Hood 2016

Carl Meyer

November 04, 2016
Tweet

Other Decks in Technology

Transcript

  1. class VerticalPartitionRouter(object): DB_FOR_MODEL = { 'likes.like': 'likes', 'comments.comment': 'comments', 'media.media':

    'media', } def _db_for(self, model_or_obj): label = model_or_obj._meta.label_lower return self.DB_FOR_MODEL.get(label, 'default') def db_for_read(self, model, **hints): return self._db_for(model) def db_for_write(self, model, **hints): return self._db_for(model) def allow_relation(self, obj1, obj2, **hints): return self._db_for(obj_1) == self._db_for(obj_2)
  2. class ShardedObject(object): def insert(self, shard_on_id, from_table, values): shard, db =

    get_conn_for_shard_key(shard_on_id) cursor = db.cursor() placeholders = ','.join( [("%%(%s)s" % key) for key in values.keys()]) columns = ','.join(values.keys()) insert_statement = ( "INSERT INTO idb%s.%s (%s) VALUES (%s)" % (shard, from_table, columns, placeholders) ) cursor.execute(insert_statement, values) db.commit()
  3. 138726300013410905 SHARDED UNIQUE IDS TIMESTAMP SHARD ID SEQUENCE CREATE OR

    REPLACE FUNCTION insta5.next_id... CREATE TABLE insta5.our_table ( "id" bigint NOT NULL DEFAULT insta5.next_id(), ...rest of table schema... )
  4. TAO DATA MODEL Jan follows Pat. Pat posts a photo.

    Jan authors a comment on the photo. Pat likes the comment. Jan Pat Follows Followed by "Contemplative cat!" Comment on Has comment Posted Posted by Authored Authored by Liked by Likes
  5. OUR (MONKEY) PATCHES 40 1 Don't recompile URL regexes for

    every active language. 2 Don't try to load translations from an app with no locale directory. 3Unlazified settings!
  6. from django.conf import settings def force_unlazified_settings(): for key in dir(settings):

    settings.__dict__[key] = getattr(settings, key) UNLAZY ALL THE SETTINGS!
  7. COUNTING CPU INSTRUCTIONS WITH PERF struct perf_event_attr pe; pe.type =

    PERF_TYPE_HARDWARE; pe.config = PERF_COUNT_HW_INSTRUCTIONS; fd = perf_event_open(&pe, 0, -1, -1, 0); ioctl(fd, PERF_EVENT_IOC_ENABLE); // code whose CPU instructions you want to measure ioctl(fd, PERF_EVENT_IOC_DISABLE); read(fd, &count, sizeof(long long));
  8. DYNOSTATS class DynostatsMiddleware(object): def process_request(self, req): req.dynostats_enabled = ( 1

    == random.randint(1, settings.DYNO_SAMPLE_RATE)) if req.dynostats_enabled: # uses Linux perf library req.dyno_start_cpu_instr = get_cpu_instructions() # use clock_gettime from librt req.dyno_start_wall_time = get_real_wall_time() req.dyno_start_cpu_time = get_process_cpu_time() # uses /proc/<pid>/statm req.dyno_start_rss_mem = get_process_rss_mem() def process_response(self, req, response): if req.dynostats_enabled: # get end values, send to scribe w/ req details return response
  9. CPROFILE class ProfilerMiddleware(object): def process_request(self, req): req.cprofile_enabled = ( 1

    == randint(1, settings.CPROFILE_SAMPLE_RATE)) if req.cprofile_enabled: req.profiler = cProfile.Profile() req.profiler.enable() def process_response(self, req, response): if req.cprofile_enabled: req.profiler.disable() req.profiler.create_stats() send_to_scribe(msgpack.dumps(profiler.stats))
  10. import cProfile import resource def get_cpu_instr(): # use perf to

    get CPU instructions cpu_profiler = cProfile.Profile(timer=get_cpu_instr) def get_rss_mem(): return resource.getrusage( resource.RUSAGE_SELF).ru_maxrss mem_profiler = cProfile.Profile(timer=get_rss_mem) CUSTOM CPROFILE TIMERS
  11. FIXING EFFICIENCY REGRESSIONS - Fixing the obvious. - Don't do

    useless work. - Cache things that don't change. - Change a .py to a .pyx: Cython. - Rewrite in C.
  12. — Mike Krieger “THE PIECES WERE PLUGGABLE ENOUGH... EVEN WITH

    OUR OWN ORM WE COULD USE MOST
 OF THE REST OF DJANGO.”
  13. AN INCOMPLETE LIST OF THE DJANGO WE RELY ON -

    HTTP stack - requests and responses - contrib.sessions - contrib.auth - middleware - url routing - settings - forms - i18n - contrib.gis - django.utils - cache backends - HTTP decorators - CSRF - signals - management commands
  14. PHOTOS database by RockIcon, smiley by Vandana Agrawal, server by

    Alexander Skowalsky, from Noun Project https://www.flickr.com/photos/yashh/2834704689 https://unsplash.com/photos/KEXUeZIev10 https://unsplash.com/photos/pd4lo70LdbI https://unsplash.com/photos/jh2KTqHLMjE https://www.flickr.com/photos/johnsonderman/15144843722 https://www.flickr.com/photos/kennethreitz/5521545772/ https://www.instagram.com/p/mNj4L3OTzj/ https://www.flickr.com/photos/67926342@N08/6175870684 https://www.flickr.com/photos/lytfyre/6489338411 https://unsplash.com/photos/4fQAMZNaGUo https://unsplash.com/photos/glHJybGNt1M https://www.flickr.com/photos/nedrichards/51132692 https://www.flickr.com/photos/sophistechate/2913053678 https://www.flickr.com/photos/elviskennedy/6784123582 https://unsplash.com/photos/HkTMcmlMOUQ