Scaleconf 2016: The dirty little secrets of building large, highly available, scalable HTTP APIs

When you first start building an API for a new product you mostly focus on getting an MVP ready, with the goal of shipping as soon as possible so you can get feedback from customers. If you are lucky enough, your product will be successful and you will have to start worrying about things like authentication, authorization, documentation, validation, rate limiting, geo-redundancy, and no downtime deployments. In this talk I will go over some real life examples of our experience evolving our APIs at Auth0 and some of the tools we use for that.

Transcript

1. The dirty little secrets of building large, highly available, scalable HTTP APIs by @dschenkelman
2. Why? var express = require('express'); var app = express(); app.get('/', function (req, res) { res.send('Hello World!'); }); var server = app.listen(3000, function () { var host = server.address().address; var port = server.address().port; console.log('Example app listening at http://%s:%s', host, port); });
3. Developers
4. Evolution
5. Pillars
6. Validation
7. enabled: true
8. enabled: 1
9. enabled: 1.0
10. enabled: "false"
11. JSON Schemas "enabled": { "type": "boolean" }
12. Errors for developers { "statusCode": 400, "error": "Bad Request", "message": "Payload validation error: 'Expected type boolean but found type string' on property enabled (<code>true</code> if the rule is enabled, <code>false</code> otherwise).", "errorCode": "invalid_body" }
13. ratify const ZSchemaErrors = require('z-schema-errors'); const errorReporters = ['headers', 'query', 'path', 'payload'].reduce((current, part) => { current[part] = ZSchemaErrors.init({/*...*/}); return current; }, {}); plugins.push({ register: require('ratify'), options: { errorReporters: errorReporters, /*...*/ } });
14. Documentation
15. Old API Explorer
16. Auto-generate docs
17. Swagger
18. Leverage validation "enabled": { "type": "boolean", "description": "<code>true</code> if the connection is enabled, <code>false</code> (default) otherwise" }
19. Customize
20. AuthN & AuthZ
21. Token Exchange API Client API Server 1. Credentials 2. API Token 3. Requests
22. Granular Security
23. Decentralized Issuance
24. Expiration Control
25. Debuggability
26. JSON Web Token JWT
27. Authenticate For all endpoints
28. Authorize Per endpoint
29. Route { method: 'GET', path: '/api/v2/clients/{id}', config: { auth: { strategy: 'jwt', scope: ['read:clients', 'read:client_keys'] }, } }
30. Blacklisting
31. T-shirt time
32. Stress tests
33. Devops Stress
34. What to do…
35. Rate Limiting
36. Token Bucket
37. limitd #port to listen on port: 9001 #db path db: /var/limitd/database #define the bucket types buckets: customers: size: 10 per_second: 10
38. 429 X-RateLimit-Limit Maximum amount X-RateLimit-Remaining How many there are left X-RateLimit-Reset When will bucket be full again
39. patova plugins.push({ register: require('patova'), options: { event: 'onPostAuth', // when to perform the limit check type: 'tenant', // bucket address: env.LIMITD_SERVER, extractKey: function(request, reply, done){ const key = request.auth.credentials.__tenant; done(null, key); } } });
40. By Example
41. Admit the problem
42. Response times
43. Flame graphs
44. Sunburst
45. Password Hashing app.use('/login', (req, res) => { // fetch the user by req.username from db db.users.findOne({ email: req.body.email }, (err, user) => { ... // compare bcrypt for req.password to db hash const success = bcrypt.compare(req.body.password, user.passwordHash); res.send(success ? 200 : 401); }); });
46. Faster Hash
47. Caching
48. Scale up
49. Multiple IdP instances ELB IdP IdP IdP IdP
50. BaaS app.use('/login', (req, res) => { // fetch the user by req.username from db db.users.findOne({ email: req.body.email }, (err, user) => { ... // compare bcrypt for req.password to db hash baas.compare(req.body.password, user.passwordHash).then((e success) => { res.send(err || !success ? 401 : 200); }); }); });
51. Auto Scaling Identity provider ELB BaaS BaaS BaaS BaaS
52. Cost comparison Price / (1M req) #req per sec / vCPU t2-micro $0.36 10.00 t2-medium $0.76 9.50 c4-large $1.53 10.00 c3-8xlarge $1.64 8.88
53. Fail gracefully
54. Geo Redundancy
55. Data Center Failure
56. Natural Disasters
57. Cloud Provider Failure
58. Our Setup
59. Our Setup App Instances MongoDB Elastic Search App Instances MongoDB Elastic Search MongoDB
60. The switch
61. Changes
62. Experiments
63. feature-change const feature_change = require('feature-change'); var options = { expected: function(cb){ mongo_search(mongo_opts, cb); }, actual: function(cb){ es_search(es_opts, cb); }, logAction: function(current_result, new_result){ // invoked when there is a difference in the results // (useful for logging) } }; feature_change(options, function(err, result){ // this is the original callback you were using for mongo // err and result always come from mongo_search });
64. Iron out differences
65. Wrap up
66. Shipping is important
67. Remember: evolve
68. Validation & Docs • http://json-schema.org • http://swagger.io • https://auth0.com/docs/api/v2 (example of branded auto-generated docs) • https://github.com/mac-/ratify
69. AuthN & AuthZ • https://tools.ietf.org/html/rfc7519 • http://jwt.io/ • https://auth0.com/blog/2014/12/02/using-json- web-tokens-as-api-keys/ • https://auth0.com/blog/2015/03/10/blacklist-json- web-token-api-keys/ • https://github.com/auth0/hapi-auth-jwt
70. Rate limiting • http://tools.ietf.org/html/rfc6585 • https://github.com/limitd • https://github.com/dschenkelman/patova
71. BaaS • http://security.stackexchange.com/a/83382 • http://www.brendangregg.com/FlameGraphs/ cpuflamegraphs.html • https://github.com/thlorenz/v8-perf/issues/4 • https://github.com/auth0/node-baas • http://docs.aws.amazon.com/AutoScaling/latest/ DeveloperGuide/US_SetUpASLBApp.html
72. Geo Redundancy • http://highscalability.com/blog/2014/12/1/auth0- architecture-running-in-multiple-cloud-providers- and-r.html • https://auth0.com/availability-trust • http://docs.mongodb.org/master/tutorial/deploy- geographically-distributed-replica-set/
73. Feature Changes • http://zachholman.com/talk/move-fast-break- nothing/ • https://auth0.com/blog/2015/10/27/feature- changes-at-auth0/ • https://github.com/dschenkelman/feature- change
74. Thanks https://github.com/dschenkelman/api-secrets-talk @dschenkelman npm i dschenkelman