Upgrade to Pro — share decks privately, control downloads, hide ads and more …

La historia de todo lo que pudo salir mal... pe...

Jorge Bastida
November 24, 2013

La historia de todo lo que pudo salir mal... pero salió bien

PyconES talk about lot's of things we have learn while migrating streetlife from mongodb to postgresql and from flask to django.

Jorge Bastida

November 24, 2013
Tweet

More Decks by Jorge Bastida

Other Decks in Technology

Transcript

  1. de todo lo que pudo salir mal... salió bien. La

    historia... pero @jorgebastida Jorge Bastida
  2. de todo lo que pudo salir mal... salió bien. La

    historia... pero @jorgebastida Jorge Bastida
  3. $ fab -l Available commands: deploy Deploy all the things!

    purge_varnish Purge varnish cache. start_maintenance_mode Start maintenance mode. stop_maintenance_mode Stop maintenance mode. track_deploy Track deploy. stop_workers Stop celery workers. start_workers Start celery workers. track_upgrade Track upgrade. restart_webserver Restart web server. restore_db Restore database. run_puppet Run Puppet. server_stats Get server stats. get_celery_logs Get celery logs from web servers. get_log Merge and dowload a log file. pg_create_db Create a database. pg_create_role Create a role. pg_drop_cluster Drop Postgresqls cluster pg_drop_db Drop a database. pg_grant_db_to_role Grant db roles. start_webserver Start web server. stop_webserver Restart the web server. ... fat-fab
  4. migrate_users #100 celery canvas ftw #200 #300 #400 #500 #600

    #n migrate_messages #100 #200 #300 #400 #500 #600 ... #n ... migrate_pages #100 #200 #300 #400 #500 #600 #n ... migrate_sales #100 #200 #300 #400 #500 #600 #n ... migrate_friends #100 #200 #300 #400 #500 #600 #n ... migrate_mail #100 #200 #300 #400 #500 #600 #n ... migrate_ ... #100 #200 #300 #400 #500 #600 #n ... #1 #2 #3 fab migrate <backup>
  5. class BaseEmail(TemplatedHTMLEmailMessageView): unsubscribable = False mail_list = None stats_group =

    'other' bcc_ratio = 0.01 @classmethod def delay(cls, **kwargs): mailer.delay(cls.__module__, cls.__name__, **kwargs) def confirm(self, context): return True ...
  6. class UserBaseEmail(BaseEmail): def destination(self, context): return context['user'].email def confirm(self, context):

    checks = [super(UserBaseEmail, self).confirm(context)] checks.append(not context['user'].author.closed_account) checks.append(context['user'].valid_email) return all(checks) ActivationEmail.delay(user_id=user.id)
  7. @celery.task(base=EmailTask) def mailer(module, name, **kwargs): email_cls = mailer.get_cls(module, name) try:

    email = email_cls(**kwargs) email.send() except BotoServerError, error: ... except AbortedEmailError, error: log.error(...) graphite.incr('sl.emails.{0}.aborted'.format(email_cls.stats_group)) else: log.info(...) graphite.incr('sl.emails.{0}.sent'.format(email_cls.stats_group))
  8. graphite server carbon graphite statsd udp 8125 diamond tcp 80

    db server diamond logs web server diamond app logs ☁
  9. graphite server carbon graphite statsd udp 8125 diamond tcp 80

    db server diamond logs web server diamond app logs dev env fab deploy ☁
  10. class BaseTask(Task): abstract = True def __call__(self, *args, **kwargs): graphite.incr('sl.tasks.{0}.start'.format(self.__class__.__name__))

    with graphite.timer('sl.tasks.{0}'.format(self.__class__.__name__)): return super(BaseTask, self).__call__(*args, **kwargs) def on_failure(self, *args, **kwargs): graphite.incr('sl.tasks.{0}.failed'.format(self.__class__.__name__)) def on_success(self, *args, **kwargs): graphite.incr('sl.tasks.{0}.succeeded'.format(self.__class__.__name__)) statsd
  11. class BaseTask(Task): abstract = True def __call__(self, *args, **kwargs): graphite.incr('sl.tasks.{0}.start'.format(self.__class__.__name__))

    with graphite.timer('sl.tasks.{0}'.format(self.__class__.__name__)): return super(BaseTask, self).__call__(*args, **kwargs) def on_failure(self, *args, **kwargs): graphite.incr('sl.tasks.{0}.failed'.format(self.__class__.__name__)) def on_success(self, *args, **kwargs): graphite.incr('sl.tasks.{0}.succeeded'.format(self.__class__.__name__)) statsd
  12. class BaseTask(Task): abstract = True def __call__(self, *args, **kwargs): graphite.incr('sl.tasks.{0}.start'.format(self.__class__.__name__))

    with graphite.timer('sl.tasks.{0}'.format(self.__class__.__name__)): return super(BaseTask, self).__call__(*args, **kwargs) def on_failure(self, *args, **kwargs): graphite.incr('sl.tasks.{0}.failed'.format(self.__class__.__name__)) def on_success(self, *args, **kwargs): graphite.incr('sl.tasks.{0}.succeeded'.format(self.__class__.__name__)) statsd
  13. Stats class UserStats(rediscache.CacheableBaseModel): user = models.OneToOneField('users.User', related_name='stats', primary_key=True) # Regenerable

    messages = models.IntegerField(default=0) comments = models.IntegerField(default=0) ... # Buffers last_email_click = models.DateTimeField(null=True) last_seen = models.DateTimeField(null=True) ... django Model
  14. Stats class UserCache(rediscache.ModelCache): model = UserStats def regenerate(self, user): self.messages

    = Message.objects.filter(...) self.comments = Comments.objects.filter(...) self.save() cache “Model” Cache
  15. Stats Cache User >>> UserCache.regenerate_all() every day >>> user =

    User.objects.get(...) >>> user.cache <users.models.UserCache at 0x44946d0> >>> user.cache.messages 23 .cache every 30m >>> UserCache.flush_all() #1 #2 #3 #4 #n #1 #2 #3 #4 #n only required ones.
  16. Stats Cache User >>> UserCache.regenerate_all() every day >>> user =

    User.objects.get(...) >>> user.cache <users.models.UserCache at 0x44946d0> >>> user.cache.messages 23 .cache every 30m >>> UserCache.flush_all() #1 #2 #3 #4 #n #1 #2 #3 #4 #n only required ones.
  17.  Planificación ⚙ Puppet ⚙ Staging Migración ✉ E-mails Metrics

    Redis Testing  Varnish Backups Fabric celery django-mail-views django-redis lifeordeath wal-e graphite