Web Development with OpenAPI Spec & Fast Testing

Modern web applications are mostly built on API and client side apps. Centralizing the business logic in API can prevent clients duplicating features on their sides.

However, in certain situations, we might face a tight schedule, work with clients who use waterfall workflow, or work with clients who are very sensitive about any unexpected behaviors.

This talk will cover how we utilize OpenAPI Spec for cross-team communication, keep the development speed by using slow but steady integration tests, and how we speed up those tests.

This talk was originally given at WebHack Meetup, a software engineering meetup in Tokyo. https://webhack.connpass.com/event/121030/

I gave this talk as a web engineer at Moneytree KK, which provides financial data platform for general customers, financial institutions and businesses. Check Moneytree's careers page for current open positions. https://www.moneytree.jp/careers

Transcript

1. 2019 © Moneytree KK. Web Development with OpenAPI Spec & Fast Testing Yu-Cheng Chuang Feb 27, 2019 Slides will be available online
2. @yorkxin @chitsaou Yu-Cheng Chuang
 Core Systems Engineer 
 at Moneytree From Taiwan BSD / Linux User since 2005 Web Developer since 2009 #typography #railway #cooking About me
3. Moneytree is a financial data portability platform, comprised of a personal financial management app, and an API (Application Programming Interface) feed of customer data for financial institutions. Moneytree in Brief "
4. Project Background
5. • Backend server built on Ruby on Rails • Server is running on AWS ECS • User management on AWS Cognito • CircleCI / AWS Code Pipeline • Web App built on React • iOS, Android Apps The Stack Project Background
6. # Customer™ $ Backend Developers % $ % Web Developers $ % $ % Mobile Developers $ % $ % & Project Managers '
7. How do we develop individually and keep everything on track? ( ( (
8. Actually I’ve started writing OpenAPI Spec! ' '
9. Yes, our Project Managers can write Open API Spec!
10. Use Machine-friendly API Spec as the Single Source of Truth
11. ) pet-store-api-spec.git pet-store-server.git pet-store-web.git pet-store-mobile.git * + , % % % '%
12. • Allows you to design / define HTTP API in YAML or JSON • Request & Responses can be defined as Schema • Using JSON Schema-like DSL Open API Spec (f.k.a. Swagger) Open API Spec
13. paths: /pets: get: summary: List all pets operationId: listPets tags: - pets parameters: - name: limit in: query description: How many items to return at one time (max 100) required: false schema: type: integer format: int32 responses: '200': description: A paged array of pets headers: x-next: description: A link to the next page of responses schema: type: string content: application/json: schema: $ref: "#/components/schemas/Pets" default: description: unexpected error content: application/json: schema: $ref: "#/components/schemas/Error" components: schemas: Pet: required: - id - name properties: id: type: integer format: int64 name: type: string tag: type: string nullable: true Pets: type: array items: $ref: "#/components/schemas/Pet" Error: required: - code - message properties: code: type: integer format: int32 message: type: string - - - ! ! !
15. • A language to define the data structure of JSON JSON Schema Open API Spec
16. { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/product.schema.json", "title": "Product", "description": "A product from Acme's catalog", "type": "object", "properties": { "productId": { "description": "The unique identifier for a product", "type": "integer" }, "productName": { "description": "Name of the product", "type": "string" }, "price": { "description": "The price of the product", "type": "number", "exclusiveMinimum": 0 }, "tags": { "description": "Tags for the product", "type": "array", "items": { "type": "string" }, "minItems": 1, "uniqueItems": true } }, "required": [ "productId", "productName", "price" ] } { "productId": 1, "productName": "A green door", "price": 12.50, "tags": [ "home", "green" ] } https:/ /json-schema.org/learn/getting-started-step-by-step.html
17. { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "http://example.com/product.schema.json", "title": "Product", "description": "A product from Acme's catalog", "type": "object", "properties": { "productId": { "description": "The unique identifier for a product", "type": "integer" }, "productName": { "description": "Name of the product", "type": "string" }, "price": { "description": "The price of the product", "type": "number", "exclusiveMinimum": 0 }, "tags": { "description": "Tags for the product", "type": "array", "items": { "type": "string" }, "minItems": 1, "uniqueItems": true } }, "required": [ "productId", "productName", "price" ] } schemas: Pet: required: - id - name properties: id: type: integer format: int64 name: type: string tag: type: string nullable: true ᶃ ᶃ ᶄ ᶄ OpenAPI Schema vs. JSON Schema
18. https:/ /swagger.io/docs/specification/data-models/keywords/
19. https:/ /philsturgeon.uk/api/2018/03/30/openapi-and-json-schema-divergence/ (c) 2018 Phil Sturgeon
20. npm install openapi2schema Converting to JSON Schema
21. ⋊> ~/D/pet-store openapi2schema -i pet-store.openapi.yml | jq { "/pets": { "get": { ...}, "post": {…} }, "/pets/{petId}": { "get": { "responses": { "200": { "type": "array", "items": { "required": [“id", "name"], "properties": { "id": { "type": "integer", "format": "int64", "minimum": -9223372036854776000, "maximum": 9223372036854776000 }, "name": { "type": "string" }, "tag": { "type": [ "string", "null" ] } } }, "$schema": "http://json-schema.org/draft-04/schema#" }, "default": {...} } } } } components: schemas: Pet: required: - id - name properties: id: type: integer format: int64 name: type: string tag: type: string nullable: true
22. • YAML / JSON = Machine-friendly • Schema is defined using JSON Schema (*partial compatible) Pros Open API Spec
23. • Most libraries only support version 2 • See https:/ /openapi.tools/ • To validate data structure, you need to
 convert Schema definitions to JSON Schema manually Cons Open API Spec
24. Now I have schema definition for each endpoints…
25. How do we ensure that API server implementation matches the spec? (
26. Test API responses With Schema Definitions
27. gem install json-schema Validating JSON Schema
28. JSON::Validator.fully_validate(schema, hash, strict: true)
29. def load_schema_file(path) # Converts to JSON Schema with a Node CLI tool JSON.parse(`npx openapi2schema -i #{path}`) end ※ Don’t do this in production (only in tests) Since we’re using Ruby…
30. 3) GET /pets/{id} Should be successful matches response schema (success) Failure/Error: expect(response.parsed_body).to match_json_schema(schema) The property '#/pet/name' of type integer did not match the following type: string in schema eb4470d8-8767-56a6-83b7-789839be31e0 Shared Example Group: "Should conform OAS spec" called from ./spec/requests/ shared_examples/response_examples.rb:2
31. • Seed data using test factories or database dumps • Test output with response schema from API Spec • Client developers can start working 
 with fake data and real response structure • Backend developers can start working on 
 JSON serializers Stub Server Extra Benefits
32. Authoring Open API Spec
33. • Swagger Viewer • openapi-lint Tools in VSCode Authoring API Specs
37. Test the API Spec!
38. npm install swagger-cli Test the API spec
39. ⋊> ~/D/pet-store find . -name *.yml -print0 | xargs -n 1 -0 swagger-cli validate Swagger schema validation failed. Additional properties not allowed: responses at #/paths//pets JSON_OBJECT_VALIDATION_FAILED
40. Now it’s time to implement endpoints…
41. ⋊> ~/D/rocketscience on develop ◦ bundle exec rake routes | wc -l 90
42. Can I prevent future regressions? (
43. Request Tests With Detailed Scenarios
44. • Test scenarios are close to business cases • Think from Client Developer’s and User’s perspective • Useful for Regression Tests • Ensuring all responses to match API Spec • So that Client developers will be happy every day Pros Request Tests
45. ⋊> ~/D/rocketscience on develop ◦ bundle exec rake stats +----------------------+--------+--------+---------+---------+-----+-------+ | Name | Lines | LOC | Classes | Methods | M/C | LOC/M | +----------------------+--------+--------+---------+---------+-----+-------+ | Controllers | 1489 | 1151 | 42 | 145 | 3 | 5 | | Helpers | 11 | 10 | 0 | 1 | 0 | 8 | | Jobs | 2 | 2 | 1 | 0 | 0 | 0 | | Models | 2822 | 1781 | 33 | 155 | 4 | 9 | | Mailers | 135 | 95 | 4 | 5 | 1 | 17 | | Channels | 8 | 8 | 2 | 0 | 0 | 0 | | JavaScripts | 1 | 0 | 0 | 0 | 0 | 0 | | Libraries | 155 | 119 | 4 | 8 | 2 | 12 | | Controller tests | 42 | 36 | 1 | 5 | 5 | 5 | | Model tests | 690 | 481 | 4 | 38 | 9 | 10 | | Mailer tests | 0 | 0 | 0 | 0 | 0 | 0 | | Integration tests | 0 | 0 | 0 | 0 | 0 | 0 | | Model specs | 1040 | 834 | 0 | 0 | 0 | 0 | | Presenter specs | 637 | 528 | 4 | 7 | 1 | 73 | | Request specs | 11377 | 9330 | 0 | 78 | 0 | 117 | | Searcher specs | 361 | 298 | 0 | 0 | 0 | 0 | | Lib specs | 1197 | 994 | 0 | 0 | 0 | 0 | | Route specs | 43 | 38 | 0 | 0 | 0 | 0 | +----------------------+--------+--------+---------+---------+-----+-------+ | Total | 20010 | 15705 | 95 | 442 | 4 | 33 | +----------------------+--------+--------+---------+---------+-----+-------+ Code LOC: 3166 Test LOC: 12539 Code to Test Ratio: 1:4.0
46. (Coverage Report from Simplecov)
47. • Hard to debug due to deep stacks • Long Roundtrip → Slow • Very Slow ./0 Cons Request Tests
48. Run tests In Parallel
50. 1 ⚙ ⚙ ⚙ ʊਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓਓʊ
    ʼɹ̴̧̰̲ɹ̸̸̘̳̾̽̓̾̽ɹʻ
    ʉ:?:?:?:?:?:?:?:?:?:?:?:?:?:?:ʉ
51. 1 ⚙ 1 ⚙ 1 ⚙
53. • It assumes each job has same load • You need to write your own script to 
 distribute test files evenly by execution time or size Using GNU Parallel Run Test in Parallel
54. Load Balancing Test Files
55. Time 0 sec 500 sec 1,000 sec 1,500 sec 2,000 sec 2,500 sec ~1600s (70%) ~650s (30%) Bottleneck ~250s
56. • One endpoint per test file • It’s also easier for human to read • Avoid Race Condition — Reset Database for each case • Use as less requests for one scenario as possible • Data preparations (db insertions) are usually fast… • …Unless you’re inserting hundreds of rows Load Balancing Test Files Run Test in Parallel
57. Took 441 seconds (7:21) real 7m21.858s user 26m6.616s sys 0m43.592s Took 278 seconds (4:38) real 4m39.159s user 27m46.976s sys 2m3.372s Took 439 seconds (7:19) real 7m19.914s user 28m23.356s sys 5m35.944s ⚙ ⚙ ⚙ × 4 × 8 × 16 ✅
58. 8 processes might be faster than 16 • /proc/info may != # of CPU Cores • Be aware of startup time overhead (Rails is slow) • Find best number by measuring time • One CI machine might be enough Deciding # of Processes Speed up total test runtime
59. Mistakes…
60. 0 3000 6000 9000 12000 0 ms 120,000 ms 240,000 ms 360,000 ms 480,000 ms 600,000 ms 720,000 ms Past ← Time → Now Total Runtime (ms) Number of Cases Time / Cases Ratio No junit.xml Reduced requests & Split into 2 machines Start using GNU Parallel 4 processes on 1 machine Found some files not tested Switched to Ruby parallel_tests Started 8 Processes Started testing JWT
61. • Find an existing parallel testing solution 
 before you rollout your own one • Store test reports from day 1, if your CI supports it Lessons Parallel Testing
62. AWS Cognito / JWT
63. “Amazon Cognito User Pools provide a secure user directory that scales to hundreds of millions of users. As a fully managed service, User Pools are easy to set up without any worries about server infrastructure. User Pools provide user profiles and authentication tokens for users who sign up directly and for federated users who sign in with social and enterprise identity providers.” AWS Cognito User Pools AWS Cognito https:/ /aws.amazon.com/cognito/details/
64. https:/ /docs.aws.amazon.com/cognito/latest/developerguide/cognito-scenarios.html OAuth 2.0 + Bearer Token (JWT)
65. Use JWT to Remove Dependency on Auth Server
66. • A Stateless, Cryptography-Based Token Standard JSON Web Token (RFC 7519) JWT
68. Authorization: Bearer 11111111.22222222.33333333 Header Signature Payload
69. • HMAC + SHA256 (Shared Secrets) • RSA / ECDSA (Asymmetric Signing) Algorithms JSON Web Token
70. 4 4 JWT Issuer 5 6 API Server Signed by Private Key Verified by Public Key
71. { "sub": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", // Unique user 'sub' in Cognito User Pool "aud": "xxxxxxxxxxxxexample", // OAuth Client ID "email_verified": true, "token_use": "id", "auth_time": 1500009400, "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_example", // Token Issuer "cognito:username": "janedoe", "exp": 1500013000, // Token Valid Until "given_name": "Jane", "iat": 1500009400, "email": "janedoe@example.com" } A typical JWT Claim from AWS Cognito
72. • The Customer™ trusts on AWS’s Security • Real Stateless Authentication • No (hashed) password stored in our database • Client App pinning (checking aud) • Easy to find user in database by inspecting the token Pros JSON Web Token
73. • Token Revocation, obviously • Need to store JWKs (Public Keys) in every server daemon Cons JSON Web Token
74. Just do what Cognito is doing • Generate an RSA Key Pair for Test Environment • Store the JWK of Public Key • Sign the JWT Token with Private Key • Verify the JWT with JWK from Public Key Mocking JWT Verification in Tests JSON Web Token
75. npm install pem-jwk Converting RSA PEM to JWK
76. ⋊> ~/ openssl genrsa 2048 > test.pem Generating RSA private key, 2048 bit long modulus ................................................................+++ ................................................+++ e is 65537 (0x10001) ⋊> ~/ npx pem-jwk test.pem {“kty”:”RSA”,”n”:”m3Knu8N5…”,”e”:”AQAB”,”d”:”X69q6lwo…”,"p":"zrFBKps h...","q":"wIfj...","dp":"f9WHPWe...","qi":"o_O3bEU..."}
77. Black Box Integration Tests
78. • Writing tests in the same framework may have blind spot • Incorrectly assuming Accept: application/json • Sending payload without serializing into JSON • Passing parameters in wrong data type • To make sure your service runs in Docker Why Black Box Integration Tests? Black Box Integration Tests
79. • Docker Build • Docker Run • Database Migrations • PoC of an API Client using OpenAPI Spec • Typical scenarios (sign-up, create data, etc.) • Edge case scenarios that involve multiple actions • Job Workers What to test? Black Box Integration Tests
80. Run your server in Docker and Write HTTP Client in Another Language
81. SQL Database 1 Redis Cache 1 API Server 7 Job Worker ⚙ Fake Clients (Node.js) 89 Docker Compose
82. 4 4 My Test Script 5 6 API Server Sign Token by my Private Key Verify Token by my Public Key
83. npm install openapi-client-axios Blackbox Testing
84. import OpenAPIClientAxios from 'openapi-client-axios'; import * as fs from 'fs'; import * as jwt from 'jsonwebtoken'; import * as jsYaml from 'js-yaml'; export class MyAppClient extends OpenAPIClientAxios { constructor(host = 'localhost:3000') { const url = `http://${host}/`; const definition = jsYaml.safeLoad(fs.readFileSync('/path/to/api/spec.openapi.yml')); definition.servers = [{ url }]; super({ definition }); } async showPetById(id) { return await this.client.query('showPetById', id); } // Sign a JWT Token loginAs(sub, email = null) { const options = { algorithm: 'RS256', issuer: 'test-issuer', audience: 'test-audience' }; const cert = fs.readFileSync('/path/to/jwk.pem'); this.token = jwt.sign({ email, sub }, cert, options); } }
85. async init() { await super.init(); // NOTE: workaround https://github.com/axios/axios/pull/1395 this.client.interceptors.request.use((request) => { request.headers.Authorization = `Token ${this.token}`; return request; }); } Workaround Axios per-instance header issue
86. const client = new MyAppClient(); await client.init(); client.loginAs('john-appleseed', 'john@appleseed.com'); await client.showPetById(42);
87. version: '3' services: app: build: . ports: - "3000:3000" env_file: - __tests__/rails.env - __tests__/shared/jwt/jwks.env depends_on: ['db', 'redis'] db: image: "postgres:9" ports: - "15432:5432" environment: POSTGRES_PASSWORD: "password" redis: image: "redis:5" ports: - "16379:6379" • Expose ports for dockerize heath checking
88. build: machine: true steps: - checkout - run: name: Install Docker Compose command: | curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`una chmod +x ~/docker-compose sudo mv ~/docker-compose /usr/local/bin/docker-compose • Run CircleCI task outside a container so that we can mount local files • Install docker-compose if necessary
89. - run: name: Build image and pull dependencies command: docker-compose up --no-start - run: name: Run Containers background: true command: docker-compose up - run: name: Wait for containers command: | dockerize -wait tcp://localhost:15432 \ -wait tcp://localhost:16379 \ -wait tcp://localhost:3000 • Run Docker build & pull in an individual task • Start containers in background after build & pull were done • Wait until containers ready before running anything else (some RDB takes few seconds to initialize)
90. - run: name: Test - Database preparation command: | docker-compose exec app rake \ db:create \ db:migrate \ db:seed • Prepare database by migration instead of loading schema
91. How do we test features that depend on time? (
92. github.com/wolfcw/libfaketime • A dynamic linking library to modify date() system call • Download a binary from Debian APT Repository • Or build it on a Linux you use for server container • A macOS dylib won’t work libfaketime.so Mocking System Time
93. SQL Database 1 Redis Cache 1 API Server 7 Job Worker ⚙ Fake Clients (Node.js) 89 Docker Compose API Server + 2d 7
94. # An API server that shifts machine time by +2d app-future: build: . ports: - “3001:3000” environment: LD_PRELOAD: /mnt/libfaketime/libfaketime.so.1 FAKETIME: +2d volumes: - ./path/to/libfaketime:/mnt/libfaketime env_file: - __tests__/rails.env - __tests__/shared/jwt/jwks.env depends_on: ['db', 'redis'] 1) Export to another host port 2) Load libfaketime.so for API server 3) Set Shifts 4) Mount faketime binaries into container ! ᶃ " ᶄ : ᶅ : ᶆ
95. How about email content? (
96. MailHog or MailDev
97. SQL Database 1 Redis Cache 1 API Server 7 Job Worker ⚙ Fake Clients (Node.js) 89 Docker Compose API Server + 2d 7 MailHog ; Scrape Emails through its API
98. Conclusion
99. • Write as detailed test cases as possible • Use them as regression tests • Run tests in parallel to reduce total execution time • Test Docker Images to prevent shipping broken image Your Time is < < < Summary
100. 2019 © Moneytree KK. We’re hiring! visit moneytree.jp/careers/