Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons learned defying Joel Spolsky with Django

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →

Lessons learned defying Joel Spolsky with Django

Talk given at the Copenhagen Django meet up about our experiences porting Iconfinder from a LAMP stack to a de facto Python web stack.

Nick Bruun

April 24, 2013

Transcript

71 slides
  1. “… the sin le worst strate ic mistake that any

    software company can make: They decided to rewrite the code from scratch.” That’s Joel!
  2. September 2012 LAMP stack Linux, Apache, MySQL, PHP and memcached

    (dv) (-ish) 16 cores, 16 GiB RAM, death star
  3. April 2013 3 M monthly visitors 3 M requests per

    day 30 GiB data in Post reSQL
  4. April 2013 09 PM Mon 22 03 AM 06 AM

    09 AM 12 PM 03 PM 0 ms 200 ms 400 ms 600 ms 800 ms 83ms avera e response time 329ms 99th percentile response time
  5. Watch it like a boss # In postgresql.conf: log_destination =

    'csvlog' log_directory = 'pg_log' logging_collector = on log_filename = 'postgres-%Y-%m-%d_%H%M%S' log_rotation_age = 1d log_rotation_size = 1GB log_min_duration_statement = 0ms log_checkpoints = on log_connections = on log_disconnections = on log_lock_waits = on log_temp_files = 0 dump every statement and its execution time
  6. Watch it like a boss $ tail -f "/usr/local/var/postgres/pg_log/ $(ls

    -1t /usr/local/var/postgres/pg_log/ | head -1)" Latest log file Follow the tail of the log file
  7. Did I say stupid? SomeModel.objects.filter( other = None ) SomeModel.objects.filter(

    other_id = None ) SELECT "myapp_somemodel"."id", "myapp_somemodel"."arbitrary", "myapp_somemodel"."other_id" FROM "myapp_somemodel" LEFT OUTER JOIN "myapp_othermodel" ON ( "myapp_somemodel"."other_id" = "myapp_othermodel"."id" ) WHERE "myapp_othermodel"."id" IS NULL SELECT "myapp_somemodel"."id", "myapp_somemodel"."arbitrary", "myapp_somemodel"."other_id" FROM "myapp_somemodel" WHERE "myapp_somemodel"."other_id" IS NULL
  8. ORMs are leaky abstractions count_by_name = {m.name: m.other_set.count() for m

    in SomeModel.objects.all()} count_by_names = {m['name']: m['num_other'] for m in SomeModel.objects.all().annotate( num_other=Count('other') ).values('name', 'num_other')} If we don’t want to waste resources like this: The essence of SQL leaks throu h:
  9. Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization
  10. Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization 4. Fine rained cachin
  11. Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization 4. Fine rained cachin 5. Throw money at it
  12. Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization 4. Fine rained cachin 5. Throw money at it 6. Go to step 5
  13. “… the sin le worst strate ic mistake that any

    software company can make: They decided to rewrite the code from scratch.” 87.91 % of time spent renderin
  14. “… the sin le worst strate ic mistake that any

    software company can make: They decided to rewrite the code from scratch.” 29.16 % of time spent escapin HTML
  15. Enqueuin tasks in transactions Request Back round worker Be in

    transaction Create m odel Add task to queue
  16. Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue
  17. Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue Get m odel
  18. Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue Get m odel raises M odel.DoesNotExist
  19. Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue Com m it transaction Get m odel raises M odel.DoesNotExist
  20. With djan o-celery-transactions Request Back round worker Be in transaction

    Create m odel Locally add task to queue Com m it transaction
  21. With djan o-celery-transactions Request Back round worker Be in transaction

    Create m odel Locally add task to queue Com m it transaction Actually add task to queue
  22. Dequeue task from queue With djan o-celery-transactions Request Back round

    worker Be in transaction Create m odel Locally add task to queue Com m it transaction Actually add task to queue
  23. Dequeue task from queue With djan o-celery-transactions Request Back round

    worker Be in transaction Create m odel Locally add task to queue Com m it transaction Get m odel Actually add task to queue
  24. Dequeue task from queue With djan o-celery-transactions Request Back round

    worker Be in transaction Create m odel Locally add task to queue Com m it transaction Get m odel Actually add task to queue Finish task

More Decks by Nick Bruun

Other Decks in Programming