RESTful APIs with Tastypie

PyCon 2012 talk - An overview of Tastypie, a REST framework for Django.

Transcript

1. RESTful APIs with Tastypie
2. ...or...
3. How I learned to stop worrying and love the JSON
4. Who am I? • Daniel Lindsley
5. Who am I? • Daniel Lindsley • Consulting & OSS as Toast Driven
6. Who am I? • Daniel Lindsley • Consulting & OSS as Toast Driven • Primary author of Tastypie
7. What is Tastypie? • A REST framework for Django
8. What is Tastypie? • A REST framework for Django • Designed for extension
9. What is Tastypie? • A REST framework for Django • Designed for extension • Supports both Model & non-Model data
10. What is Tastypie? • A REST framework for Django • Designed for extension • Supports both Model & non-Model data • http://tastypieapi.org/
11. Philosophy
12. Make good use of HTTP
13. Make good use of HTTP Try to be “of the internet” & use the REST methods/status codes properly
14. Graceful Degradation
15. Graceful Degradation Try to keep backward compatibility & give users a gradual upgrade path
16. Flexible serialization
17. Flexible serialization Not everyone wants JSON
18. Wait. Scratch that.
19. Flexible serialization Not everyone wants JSON
20. Flexible EVERYTHING
21. Flexible EVERYTHING Customizability is a core feature
22. Data can round-trip
23. Data can round-trip Anything you can GET, you should be able to POST/PUT
24. Reasonable defaults
25. Reasonable defaults But easy to extend
26. URIs everywhere!
27. URIs everywhere! Make HATEOAS a reality
28. HATEOAS?
29. Gesundheit!
30. HATEOAS • “Hypermedia As The Engine Of Application State” • Basically, the user shouldn’t have to know anything in advance • All about explore-ability • Deep linking • http://en.wikipedia.org/wiki/HATEOAS
31. So what about Tastypie?
32. Pie?
33. Pie? Pie? Pie? Pie? Pie? Pie? Pie? Pie?
34. He can’t haz.
35. He can’t haz. But you can!
36. Tastypie • Builds on top of Django & plays nicely with other apps • Full GET/POST/PUT/DELETE/PATCH • Any data source (not just Models) • Designed to be extended
37. Tastypie (cont.) • Supports a variety of serialization formats • JSON • XML • YAML • bplist • Easy to add more
38. Tastypie (cont.) • HATEOAS by default (you’ll see soon) • Lots of hooks for customization • Well-tested • Decently documented
39. The Setup
40. Installation pip install django-tastypie
41. Installation (cont.) INSTALLED_APPS += [‘tastypie’]
42. Installation (cont.) $ ./manage.py syncdb
43. Done.
44. Let’s add an API for django.contrib.auth
45. High-Level 1. Code goes in our apps, not Django
46. High-Level 1. Code goes in our apps, not Django 2. Define the resource for User
47. High-Level 1. Code goes in our apps, not Django 2. Define the resource for User 3. Hook up that resource in your URLconf
48. 1. Don’t fork Django! No, seriously. You don’t need to. Done.
49. 2a. Setup # Assuming we’re in your project directory... $ cd <myapp> # Substitute your app_name here. $ mkdir api $ touch api/__init__.py $ touch api/resources.py # Done!
50. 2b. User Resource from django.contrib.auth.models import User from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: queryset = User.objects.all()
51. 3. URLconf # In your ``ROOT_URLCONF``... # Stuff then... from tastypie.api import Api from <myapp>.api.resources import UserResource v1_api = Api() v1_api.register(UserResource()) urlpatterns = patterns(‘’, (r’^api/’, include(v1_api.urls), # Then the usual... )
52. HTTP 201
53. Hit your new API. Curl: http://localhost:8000/api/v1/ Browser: http://localhost:8000/api/v1/?format=json
54. Demo-time.
55. What’s there? • /api/v1/ - A list of all available resources • /api/v1/user/ - A list of all users • /api/v1/user/2/ - A specific user • /api/v1/user/schema/ - A definition of what an individual user consists of • /api/v1/user/multiple/1;4;5/ - Get those three users as one request
56. What’s there? (cont.) • All serialization formats available* • curl -H “Accept: application/xml” http://localhost:8000/api/v1/ user/ • http://localhost:8000/api/v1/user/2/? format=yaml * Provided lxml, PyYAML, biplist are installed.
57. What’s there? (cont.) • Serialization format negotiated by either Accepts header or the ”? format=json” GET param • Pagination by default • Everyone has full read-only GET access
58. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser
59. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser • Ability to filter
60. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser • Ability to filter • Authentication / Authorization
61. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser • Ability to filter • Authentication / Authorization • Caching (disabled by default)
62. What’s not there? (Yet) • Leaking sensitive information! • email/password/is_staff/is_superuser • Ability to filter • Authentication / Authorization • Caching (disabled by default) • Throttling (disabled by default)
63. Fix data leaks from django.contrib.auth.models import User from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’]
64. Authentication “Are you a recognized user?”
65. Add BASIC Auth from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: # What was there before... authentication = BasicAuthentication()
66. Add filtering from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.resources import ModelResource, ALL class UserResource(ModelResource): class Meta: # What was there before... filtering = { ‘username’: ALL, ‘date_joined’: [‘range’, ‘gt’, ‘gte’, ‘lt’, ‘lte’], }
67. Filtering • Using GET params, we can now filter out what we want. • Examples: • curl http://localhost:8000/api/v1/user/? username__startswith=a • curl http://localhost:8000/api/v1/user/? date_joined__gte=2011-12-01
68. Authorization “Are you allowed to perform that action?”
69. Add authorization from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.authorization import DjangoAuthorization from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: # What was there before... authorization = DjangoAuthorization()
70. Add caching* from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.authorization import DjangoAuthorization from tastypie.cache import SimpleCache from tastypie.resources import ModelResource class UserResource(ModelResource): class Meta: # What was there before... cache = SimpleCache() * We’ll talk more about caching later.
71. Add throttling from django.contrib.auth.models import User from tastypie.authentication import BasicAuthentication from tastypie.authorization import DjangoAuthorization from tastypie.cache import SimpleCache from tastypie.resources import ModelResource from tastypie.throttle import CacheDBThrottle class UserResource(ModelResource): class Meta: # What was there before... throttle = CacheDBThrottle()
72. Whew.
73. What’s there now? • Everything we had before
74. What’s there now? • Everything we had before • Full GET/POST/PUT/DELETE/PATCH access
75. What’s there now? • Everything we had before • Full GET/POST/PUT/DELETE/PATCH access • Only registered users can use the API & only perform actions on objects they’re allowed to
76. What’s there now? (cont.) • Object-level caching (GET detail)
77. What’s there now? (cont.) • Object-level caching (GET detail) • Logged throttling that limits users to 150 reqs per hour
78. What’s there now? (cont.) • Object-level caching (GET detail) • Logged throttling that limits users to 150 reqs per hour • The ability to filter the content
79. HTTP 302
80. Extensibility
81. Why classes?
82. Why classes? Not because I’m OO-crazy. It makes extending behavior trivial.
83. Why so many classes?
84. Why so many classes? Composition > Inheritance
85. Why so many methods?
86. Why so many methods? Hooks, hooks, hooks. Also makes delegating to composition behaviors easy.
87. Extensibility • Tastypie tries to use reasonable defaults: • You probably want JSON • You probably want full POST/PUT/ DELETE by default • You probably want to use the Model’s default manager unfiltered
88. But.
89. YMMV, so let’s make that possible.
90. Extensibility • Plug in custom classes/instances for things like: • Serialization
91. Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication
92. Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication • Authorization
93. Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication • Authorization • Pagination
94. Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication • Authorization • Pagination • Caching
95. Extensibility • Plug in custom classes/instances for things like: • Serialization • Authentication • Authorization • Pagination • Caching • Throttling
96. Extensibility • Resource has lots of methods, many of which are pretty granular • Override or extend as meets your needs
97. Customize serialization • As an example, let’s customize serialization
98. Customize serialization • As an example, let’s customize serialization • Supports JSON, XML, YAML, bplist by default
99. Customize serialization • As an example, let’s customize serialization • Supports JSON, XML, YAML, bplist by default • Let’s disable everything but JSON & XML, then add a custom type
100. Just JSON & XML, please from django.contrib.auth.models import User from tastypie.resources import ModelResource from tastypie.serialization import Serializer class UserResource(ModelResource): class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’] serializer = Serializer(formats=[‘json’, ‘xml’])
101. Hm, now what format is missing?
102. HTML
103. HTML serialization from django.shortcuts import render_to_response from tastypie.serialization import Serializer class TemplateSerializer(Serializer): formats = Serializer.formats + [‘html’] def to_html(self, data): template_name = ‘api/api_detail.html’ if ‘objects’ in data: template_name = ‘api/api_list.html’ return render_to_response(template_name, data)
104. HTML serialization (cont.) # ಠ_ಠ import cgi from stringio import StringIO class TemplateSerializer(Serializer): # ... def from_html(self, content): form = cgi.FieldStorage(fp=StringIO(content)) data = {} for key in form: data[key] = form[key].value return data
105. Now use it from django.contrib.auth.models import User from tastypie.resources import ModelResource from myapp.api.serializers import TemplateSerializer class UserResource(ModelResource): class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’] serializer = TemplateSerializer(formats=[‘json’, ‘xml’, ‘html’])
106. Fields
107. Fields • You haven’t seen it yet, but just like a ModelForm, you can control all the exposed fields on a Resource/ ModelResource
108. Fields • You haven’t seen it yet, but just like a ModelForm, you can control all the exposed fields on a Resource/ ModelResource • Just like Django, you use a declarative syntax
109. Fields from django.contrib.auth.models import User from tastypie import fields from tastypie.resources import ModelResource class UserResource(ModelResource): # Provided they take no args, even callables work! full_name = fields.CharField(‘get_full_name’, blank=True) class Meta: queryset = User.objects.all() excludes = [‘email’, ‘password’, ‘is_staff’, ‘is_superuser’]
110. Fields (cont.) • Also like ModelForm, you can control how data gets prepared for presentation (dehydrate) or accepted from the user (hydrate)
111. Fields (cont.) • Also like ModelForm, you can control how data gets prepared for presentation (dehydrate) or accepted from the user (hydrate) • Happens automatically on fields with attribute=... set
112. Fields (cont.) • Also like ModelForm, you can control how data gets prepared for presentation (dehydrate) or accepted from the user (hydrate) • Happens automatically on fields with attribute=... set • Can provide methods for non-simple access
113. Fields (cont.) class UserResource(ModelResource): # Assuming all the same bits as before. full_name = fields.CharField(blank=True) def dehydrate_full_name(self, bundle): return bundle.obj.get_full_name() def hydrate_full_name(self, bundle): name_bits = bundle.data.get(‘full_name’,’’).split() bundle.obj.first_name = name_bits[0] bundle.obj.last_name = ‘ ‘.join(name_bits[1:]) return bundle
114. Fields (cont.) • This just scratches the surface of what dehydrate/hydrate can do • ModelResource uses introspection & just creates the fields for you
115. Caching
116. Caching • The SimpleCache combined with Resource.cached_obj_get caches SINGLE objects only!
117. Caching • The SimpleCache combined with Resource.cached_obj_get caches SINGLE objects only! • Doesn’t cache the serialized output
118. Caching • The SimpleCache combined with Resource.cached_obj_get caches SINGLE objects only! • Doesn’t cache the serialized output • Doesn’t cache the list view
119. Why? • More complex behaviors get opinionated fast
120. Why? • More complex behaviors get opinionated fast • Tastypie would rather be general & give you the tools to build what you need
121. Why? • More complex behaviors get opinionated fast • Tastypie would rather be general & give you the tools to build what you need • Filters & serialization formats make it complex
122. Why? • More complex behaviors get opinionated fast • Tastypie would rather be general & give you the tools to build what you need • Filters & serialization formats make it complex • Besides...
123. What you actually want is Varnish.
124. Varnish! • https://www.varnish-cache.org/
125. Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C
126. Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C • Already caches by URI/headers
127. Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C • Already caches by URI/headers • Way faster than the Django request/ response cycle
128. Varnish! • https://www.varnish-cache.org/ • Super-fast caching reverse proxy in C • Already caches by URI/headers • Way faster than the Django request/ response cycle • POST/PUT/DELETE just pass through
129. Varnish! (cont.) • So put Varnish in front of your API (& perhaps the rest of your site) & win in the general case
130. Varnish! (cont.) • So put Varnish in front of your API (& perhaps the rest of your site) & win in the general case • Additionally, use Tastypie’s internal caching to further speed up Varnish cache-misses
131. Varnish! (cont.) • So put Varnish in front of your API (& perhaps the rest of your site) & win in the general case • Additionally, use Tastypie’s internal caching to further speed up Varnish cache-misses • Easy to extend Resource to add in more caching
132. Varnish! (cont.) • So put Varnish in front of your API (& perhaps the rest of your site) & win in the general case • Additionally, use Tastypie’s internal caching to further speed up Varnish cache-misses • Easy to extend Resource to add in more caching • If you get to that point, you’re already serving way more load than I ever have
133. Long story short: You should be using Varnish.
134. Data source Not Just Models!
135. Source dive? • If you hit up tastypie/resources.py, you’ll find something interesting
136. Source dive? • If you hit up tastypie/resources.py, you’ll find something interesting • ModelResource is just a relatively thin (~300 lines) wrapper on top of Resource (~1200 lines)
137. Source dive? • If you hit up tastypie/resources.py, you’ll find something interesting • ModelResource is just a relatively thin (~300 lines) wrapper on top of Resource (~1200 lines) • Just the ORM/Model bits
138. That’s right, virtually everything in Tastypie is available to non-ORM setups.
139. The Mighty Resource • By subclassing from Resource & overriding at least 3 (up to 9 for CRUD) methods, you can hook up any data source • For giggles, let’s hook up Solr! • For brevity, we’ll do GET-only
140. SolrResource import pysolr from tastypie import fields from tastypie.resources import Resource # Wrap the response dict to be object-like. class SolrObject(object): def __init__(self, initial=None): self.__dict__[‘_data’] = initial or {} def __getattr__(self, key): return self._data.get(key, None)
141. SolrResource (cont.) class SolrResource(Resource): id = fields.CharField(attribute=’id’) title = fields.CharField(attribute=‘text’) class Meta: resource_name = ‘solr’ object_class = SolrObject
142. SolrResource (cont.) class SolrResource(Resource): # ... def get_resource_uri(self, bundle_or_obj): # Super-naïve. return ‘/’.join([self._meta.api_name, self._meta.resource_name, bundle_or_obj.id]) def get_object_list(self, request, **kwargs): query = kwargs.get(‘query’, None) or request.GET.get(‘q’, ‘*:*’) solr = pysolr.Solr(‘http://localhost:8963/solr’) return [SolrObject(initial=res) for res in solr.search(query)])
143. SolrResource (cont.) class SolrResource(Resource): # ... def obj_get_list(self, request=None, **kwargs): return self.get_object_list(request) def obj_get(self, request=None, **kwargs): return self.get_object_list(request, {‘query’: ‘id:%s’ % kwargs[‘pk’])
144. The Mighty Resource • Takes some work but still does a lot for you • Docs have a more complete example based on Riak • See also django-tastypie-nonrel
145. HTTP 418
146. Now, let’s talk about a personal failure.
147. Man, this seems super-useful. Wish I could use it with...
148. Man, this seems super-useful. Wish I could use it with... Flask
149. Man, this seems super-useful. Wish I could use it with... Flask Pyramid
150. Man, this seems super-useful. Wish I could use it with... Flask Pyramid Itty
151. Man, this seems super-useful. Wish I could use it with... Flask Pyramid Itty Straight WSGI
152. Piecrust The extraction that failed
153. Piecrust • Seems like an awesome, helpful idea. Let’s do it! • Late 2011, I tried extracting Tastypie to work everywhere • Tastypie would just become a light shim on top with Django conveniences • https://github.com/toastdriven/piecrust
154. Piecrust • Finished the extraction, not the tests/ docs
155. Piecrust • Finished the extraction, not the tests/ docs • Close to functional, but...
156. Piecrust • Finished the extraction, not the tests/ docs • Close to functional, but... • Failed in terms of complexity & lack of standardization
157. Piecrust • Complex areas: • Non-standard request/response objects • Non-standard URL setups • Data storage layer • Relying on Django makes these easy(- ier)
158. Piecrust • Just implementing for Flask took a couple hundred lines alone & it was incomplete • Paradox of choice • Code is there if you want to play
159. Thanks!
160. Images in main theme by Julia Elman @juliaelman http://juliaelman.com/
161. I’m Daniel Lindsley of Toast Driven @toastdriven http://toastdriven.com/