Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons learned defying Joel Spolsky with Django

Lessons learned defying Joel Spolsky with Django

Talk given at the Copenhagen Django meet up about our experiences porting Iconfinder from a LAMP stack to a de facto Python web stack.

Nick Bruun

April 24, 2013
Tweet

More Decks by Nick Bruun

Other Decks in Programming

Transcript

  1. “… the sin le worst strate ic mistake that any

    software company can make: They decided to rewrite the code from scratch.” That’s Joel!
  2. September 2012 LAMP stack Linux, Apache, MySQL, PHP and memcached

    (dv) (-ish) 16 cores, 16 GiB RAM, death star
  3. April 2013 3 M monthly visitors 3 M requests per

    day 30 GiB data in Post reSQL
  4. April 2013 09 PM Mon 22 03 AM 06 AM

    09 AM 12 PM 03 PM 0 ms 200 ms 400 ms 600 ms 800 ms 83ms avera e response time 329ms 99th percentile response time
  5. Watch it like a boss # In postgresql.conf: log_destination =

    'csvlog' log_directory = 'pg_log' logging_collector = on log_filename = 'postgres-%Y-%m-%d_%H%M%S' log_rotation_age = 1d log_rotation_size = 1GB log_min_duration_statement = 0ms log_checkpoints = on log_connections = on log_disconnections = on log_lock_waits = on log_temp_files = 0 dump every statement and its execution time
  6. Watch it like a boss $ tail -f "/usr/local/var/postgres/pg_log/ $(ls

    -1t /usr/local/var/postgres/pg_log/ | head -1)" Latest log file Follow the tail of the log file
  7. Did I say stupid? SomeModel.objects.filter( other = None ) SomeModel.objects.filter(

    other_id = None ) SELECT "myapp_somemodel"."id", "myapp_somemodel"."arbitrary", "myapp_somemodel"."other_id" FROM "myapp_somemodel" LEFT OUTER JOIN "myapp_othermodel" ON ( "myapp_somemodel"."other_id" = "myapp_othermodel"."id" ) WHERE "myapp_othermodel"."id" IS NULL SELECT "myapp_somemodel"."id", "myapp_somemodel"."arbitrary", "myapp_somemodel"."other_id" FROM "myapp_somemodel" WHERE "myapp_somemodel"."other_id" IS NULL
  8. ORMs are leaky abstractions count_by_name = {m.name: m.other_set.count() for m

    in SomeModel.objects.all()} count_by_names = {m['name']: m['num_other'] for m in SomeModel.objects.all().annotate( num_other=Count('other') ).values('name', 'num_other')} If we don’t want to waste resources like this: The essence of SQL leaks throu h:
  9. Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization
  10. Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization 4. Fine rained cachin
  11. Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization 4. Fine rained cachin 5. Throw money at it
  12. Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization 4. Fine rained cachin 5. Throw money at it 6. Go to step 5
  13. “… the sin le worst strate ic mistake that any

    software company can make: They decided to rewrite the code from scratch.” 87.91 % of time spent renderin
  14. “… the sin le worst strate ic mistake that any

    software company can make: They decided to rewrite the code from scratch.” 29.16 % of time spent escapin HTML
  15. Enqueuin tasks in transactions Request Back round worker Be in

    transaction Create m odel Add task to queue
  16. Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue
  17. Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue Get m odel
  18. Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue Get m odel raises M odel.DoesNotExist
  19. Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue Com m it transaction Get m odel raises M odel.DoesNotExist
  20. With djan o-celery-transactions Request Back round worker Be in transaction

    Create m odel Locally add task to queue Com m it transaction
  21. With djan o-celery-transactions Request Back round worker Be in transaction

    Create m odel Locally add task to queue Com m it transaction Actually add task to queue
  22. Dequeue task from queue With djan o-celery-transactions Request Back round

    worker Be in transaction Create m odel Locally add task to queue Com m it transaction Actually add task to queue
  23. Dequeue task from queue With djan o-celery-transactions Request Back round

    worker Be in transaction Create m odel Locally add task to queue Com m it transaction Get m odel Actually add task to queue
  24. Dequeue task from queue With djan o-celery-transactions Request Back round

    worker Be in transaction Create m odel Locally add task to queue Com m it transaction Get m odel Actually add task to queue Finish task