ticketea internals

Talk at PyCON ES 2014 (Spanish)

Transcript

1. internals @ticketeaEng
2. ticketea ¿Quiénes somos? @esanchezm @adiazher @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg @loquedigairune @Leidan @RafaRM20
4. ticketea Al Principio… ✤ Primera web, usando .NET ✤ Reescritura en PHP y Yii ✤ Reescritura del frontal en PHP (framework propio)
6. ticketea Actualmente… ✤ De 4 personas a 12, en un año y medio ✤ Sin perfil de sistemas/dba puro ✤ Sistemas en AWS ✤ Hemos llegado a varios países
7. Cambio de Estrategia
8. No ha sido un camino fácil…
9. odin
10. ticketea Herramienta de informes ✤ Business Intelligence ✤ Exportación de informes (PDF/XLSX) ✤ Compatibilidad de sesiones con PHP
12. ticketea Compatibilidad con lo anterior… class SessionStore(CacheStore): def load(self): pass def save(self, must_create=False): pass @property def cache_key(self): pass def decode(self, session_data): pass def encode(self, session_dict): pass
13. ticketea SessionStore -> CacheStore def load(self): try: session_data = self.decode( self._cache.get(self.cache_key, None) ) except Exception: session_data = None if session_data is not None and 'user' in session_data: try: session_data[SESSION_KEY] = session_data['user']['id'] except KeyError: pass return session_data self.create() return {}
14. ticketea SessionStore -> CacheStore def save(self, must_create=False): if must_create: func = self._cache.add else: func = self._cache.set result = func( self.cache_key, self.encode(self._get_session(no_load=must_create)), self.get_expiry_age() ) if must_create and not result: raise CreateError
15. ticketea SessionStore -> CacheStore @property def cache_key(self): return "memc.sess.key." + self._get_or_create_session_key() def decode(self, session_data): if session_data is not None: return PHPUnserialize().session_decode(session_data) return session_data def encode(self, session_dict): return PHPSerialize().session_encode(session_dict)
16. ticketea Usando el admin sin modelos… # __init__.py from . import admin_models if 'django.contrib.admin.models' in sys.modules: del sys.modules['django.contrib.admin.models'] sys.modules['django.contrib.admin.models'] = admin_models from django.contrib.admin.sites import AdminSite AdminSite.check_dependencies = nothing from django.core.management.base import BaseCommand BaseCommand.validate = lambda x, *args, **kwargs: None
17. pylongtable
19. ticketea
20. Python Era Begins
21. forseti
22. ticketea ¿Qué es forseti? ✤ Gestionar grupos de auto-escalado ✤ Generación de AMIs ✤ Despliegue de nuevo código ✤ Gestión de alarmas
23. ticketea ¿Qué es forseti? FORSETI BOTO BOTO-CORE EC2AutoScaleConfig ELBalancer CloudWatchMetricAlarm EC2AutoScaleGroup EC2AutoScalePolicy GoldenEC2Image EC2Instance
24. ticketea RELEASE THE KRAKEN
25. frontend
26. ticketea Arquitectura de sistemas API Load Balancer API API API { Frontend Frontend Frontend Frontend Load Balancer DB { Usuario
27. ticketea Arquitectura del frontend supervisor chaussette
28. ticketea Nuestro “django” Modelos Contrib auth ModelForms DB Templates Sesiones CBVs NO SÍ
29. ticketea Resultado ✤ La planificación fue un desastre: 2x tiempo. ✤ Mejor logging y reporting de errores. ✤ Tiempo de respuesta reducido a la mitad. ✤ Cobertura de tests > 85%
30. ticketea django-configurations settings ├── base.py ├── bundles │ ├── crispy.py │ ├── gunicorn.py │ ├── pipeline.py │ ├── raven.py │ ├── statsd.py │ └── tkt.py ├── development.py ├── production.py └── testing.py
31. ticketea django-configurations class Base(CrispyForms, Settings): DJANGO_APPS = [] BUNDLE_APPS = [] TKT_APPS = [] @property def INSTALLED_APPS(self): return self.DJANGO_APPS + self.BUNDLE_APPS + self.TKT_APPS @property def DATABASES(self): return { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'db_name', 'USER': 'db_user', 'PASSWORD': self.DB_PASSWORD, 'HOST': 'localhost', } }
32. ticketea django-configurations class Production(Gunicorn, Base): DB_PASSWORD = values.Value() class Development(Base): DB_PASSWORD = 'pass'
33. ticketea django-configurations class CrispyForms(object): def __init__(self): super(CrispyForms, self).__init__() self.BUNDLE_APPS += ( 'crispy_forms', )
34. ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api)
35. ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api)
36. ticketea Uso del api client api.user(6077).event(1).update({ 'name': name, 'description': description })
37. ticketea Creando un endpoint… class User(CRUDEndpoint): def __init__(self, pk): super(User, self).__init__(pk) self.add_endpoint(Event) def activate(self, activation_key, email): return self._http_client.request.post( '%s/activate' % self.url, data={ 'activation_key': activation_key, 'email': email } )
38. ticketea Internals del api-client class RestCRUDEndpoint(object): . . . def add_endpoint(self, klass): url_endpoint = klass.__name__.lower() python_name = from_camel_to_snake(url_endpoint) list_name = plural_name(python_name) create_name = 'create_' + python_name klass.endpoint = '%s/%s' % (self.url, url_endpoint) klass._http_client = self._http_client setattr(self, python_name, klass) setattr( self, create_name, self._create_object(url_endpoint) )
39. ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api) TicketeaApiMiddleware TicketeaApiException
40. ticketea Inyectando el api-client en el request class TicketeaApiMiddleware(object): def process_request(self, request): if not hasattr(request, 'user'): raise ImproperlyConfigured('error') if isinstance(request.user, AnonymousUser): self._login_as_guest(request) request.api = create_api_client(...) def process_response(self, request, response): # Check if the token was refreshed return response
41. ticketea Manejando errores del API class TicketeaApiExceptions(object): def process_exception(self, request, exception): if isinstance(exception, ApiException): if exception.code == self.REQUEST_TOKEN_EXPIRED: return self._manage_expired_token(request) elif isinstance(exception, ForbiddenException): raise PermissionDenied elif isinstance(exception, NotFoundException): raise Http404
42. ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api) ApiDispatchMixin TktTemplateView
43. ticketea Mixins class ApiDispatchMixin(object): def dispatch(self, *args, **kwargs): if hasattr(self, 'get_api_data'): result = self.get_api_data() if isinstance(result, HttpResponse): return result return super(ApiDispatchMixin, self).dispatch( *args, **kwargs )
44. ticketea Mixins # FAIL class AccessKwargsMixin(object): def get_context_data(self, **kwargs): kw = super(AccessKwargsMixin, self) \ .get_context_data(**kwargs) kw['url_kwargs'] = self.kwargs return kw
45. ticketea Nuestras vistas genéricas class TktView( LoginRequiredMixin, UserInjectorMixin, RoleRequiredMixin, ApiDispatchMixin, View ) class TktTemplateView( LoginRequiredMixin, UserInjectorMixin, RoleRequiredMixin, ApiDispatchMixin, TemplateView )
46. ticketea Clase base de nuestros FormView class TktFormView(FormView): def post(self, request, *args, **kwargs): form_class = self.get_form_class() form = self.get_form(form_class) if form.is_valid(): validate_and_commit = getattr(self, 'validate_and_commit', lambda x: True) validate_result = validate_and_commit(form) if validate_result is None: raise ValueError('validate_and_commit must return something') if isinstance(validate_result, HttpResponse): return validate_result elif validate_result: return self.form_valid(form) else: return self.form_invalid(form) else: messages.error(request, _('there has been an error')) return self.form_invalid(form)
47. ticketea Clase base de nuestros Form @decorator def handle_api_errors(form_save_method, self, *args, **kwargs): try: save_method = form_save_method(self, *args, **kwargs) return save_method if save_method is not None else True except ApiException as e: self.fill_form_errors(e.attrs) return False class TktForm(forms.Form): api_field_names = {} def fill_form_errors(self, errors): for field, errno in errors.items(): real_field_name = self.api_field_names.get(field, field) error_description = API_ERRORS_NAMES.get(errno) self._errors[real_field_name] = self.error_class( [error_description] )
48. ticketea Ejemplo de formulario class AccountInfoForm(TktForm): email = forms.EmailField() def __init__(self, *args, **kwargs): super(AccountInfoForm, self).__init__(*args, **kwargs) self.helper = AccountInfoHelper() @handle_api_errors def save(self, user): user.update(**self.cleaned_data)
49. ticketea Ejemplo de vista class AccountInfoView(TktFormView): form_class = AccountInfoForm template_name = 'users_config/users/profile.html' min_role = 'organizer' def get_api_data(self): self.user = self.request.api.user(self.kwargs['id']) def validate_and_commit(self, form): committed = form.save(self.user) if committed: messages.success(self.request, _('OK')) else: messages.error(self.request, _('KO')) return committed def get_success_url(self): return reverse('profile_url')
50. ticketea third-party apps ✤ django-crispy-forms ✤ django-braces ✤ django-configurations ✤ django-pipeline ✤ django-secure ✤ django-debug-toolbar ✤ werkzeug ✤ HTTPretty
51. heracles
52. ticketea Arquitectura RabbitMQ API (PHP) Celery Celery Celery {
53. ticketea Tarea base class BaseTask(Task): abstract = True ignore_result = True max_retries = 5 def retry_time(self, retries): RETRIES = [10, 15, 20, 30] try: return RETRIES[retries] except IndexError: return RETRIES[-1] def retry(self, *args, **kwargs): retries = self.request.retries + 1 max_retries = kwargs.get('max_retries', self.max_retries) if retries > max_retries * 0.3: logger.warning(...) kwargs['countdown'] = self.retry_time(retries - 1) return super(BaseTask, self).retry(*args, **kwargs)
54. ticketea Tarea base class BaseTask(Task): def on_success(self, *args, **kwargs): statsd.incr('tasks.%s' % self.name) def on_failure(self, exc, task_id, args, kwargs, einfo): logger.warning(...) def __call__(self, *args, **kwargs): with transaction.atomic(): try: return super(BaseTask, self).__call__( *args, **kwargs ) except Exception as exc: if isinstance(exc, exceptions.Retry): raise exc else: logger.exception('Unhandled exception') self.retry(exc=exc)
55. no hay solución perfecta
56. thor
57. ticketea Objetivos ✤ Pasar a sistemas distribuídos ✤ Permitir el funcionamiento degradado del sistema ✤ Poder desarrollar cada proyecto a una velocidad distinta ✤ Poder ajustar la infraestructura usada de cada proyecto ✤ Reducir puntos únicos de fallo
58. ticketea Arquitectura RabbitMQ API (PHP) Celery Celery Celery { THOR DBLS Tasks
59. ticketea Arquitectura RabbitMQ API (PHP) Celery Celery Celery { THOR DBLS Tasks Views X
60. Más importante que tener un equipo A…
61. … es tener un plan B
62. graphing && logging
63. ticketea Herramientas usadas ✤ sentry ✤ graphite ✤ grafana ✤ rsyslog ✤ logstash ✤ cabot
64. ticketea Sabina, Noviembre 2014
65. ticketea PREGUNTAS
66. @ticketeaEng GRACIAS