Broken APIs Break Trust: Tips for Creating and Updating APIs

For many of us, APIs and their client libraries are the face of our applications to the world. We need to innovate with new features, but breaking changes are toxic to customer trust.

In this session you will pick-up concrete design patterns that you can use and anti-patterns that you can recognize so that your service API or library can continue to grow without breaking the world. Using real APIs and open source libraries as a case study, we’ll look at actual examples of specific strategies that you can employ to keep your API backwards compatible without painting yourself into a corner.

Transcript

1. BROKEN APIS BREAK TRUST TIPS FOR CREATING AND UPDATING APIS
3. Instagram: @pug_katsu
4. WHO AM I? ▸ AWS SDK for Ruby ▸ Twitter: @alexwwood ▸ Please do tweet your comments and questions about the talk! ▸ Bad Jokes
5. AGENDA ▸ Defining APIs and Backwards Compatibility ▸ Safe API Changes ▸ Constraints and Validation ▸ Exceptions ▸ Undocumented API Contracts ▸ Conclusion: Backwards Compatibility Rules
6. DEFINITIONS
7. "THERE IS A THEORY WHICH STATES THAT IF EVER ANYONE DISCOVERS EXACTLY WHAT THE UNIVERSE IS FOR AND WHY IT IS HERE, IT WILL INSTANTLY DISAPPEAR AND BE REPLACED BY SOMETHING EVEN MORE BIZARRE AND INEXPLICABLE. THERE IS ANOTHER THEORY WHICH STATES THAT THIS HAS ALREADY HAPPENED." Douglas Adams, "Hitchhiker's Guide to the Galaxy"
8. ANATOMY OF A WEB API ▸ Resources: Things that an API manages. ▸ Shape: A structure with members. ▸ Input Shape: A top-level request shape passed to an operation. ▸ Output Shape: A top-level response shape returned from an operation. ▸ Error Shape: A structure that represents an error response from an operation. ▸ Member: Belongs to a shape, represents a property of the shape. ▸ Operation: Something exposed by the service that can be invoked. ▸ NOTE: Can all of this apply to libraries which expose an API but do not make calls to the internet? Absolutely.
9. EVOLUTION OF CLIENTS RELATIVE TO APIS CLIENT API
10. EVOLUTION OF CLIENTS RELATIVE TO APIS CLIENT V2 API V2 CLIENT V1 API V1
11. EVOLUTION OF CLIENTS RELATIVE TO APIS CLIENT V2 API V2 CLIENT V1 API V1
12. EVOLUTION OF CLIENTS RELATIVE TO APIS CLIENT V3 API V3 CLIENT V2 API V2 CLIENT V1 API V1
13. EVOLUTION OF CLIENTS RELATIVE TO APIS CLIENT V3 API V3 CLIENT V2 API V2 CLIENT V1 API V1 CLIENT V4 API V4
14. EVOLUTION OF CLIENTS RELATIVE TO APIS CLIENT V3 API V3 CLIENT V2 API V2 CLIENT V1 API V1 CLIENT V4 API V4 CLIENT ∞ API ∞
15. EVOLUTION OF CLIENTS RELATIVE TO APIS API' CLIENT CLIENT'
16. API Backward Compatibility + Client Forward Compatibility =
17. ANATOMY OF A WEB API Trip travelers [string array] description [string] flight [Flight] ...
18. ANATOMY OF A WEB API Trip travelers [string array] description [string] flight [Flight] ... SHAPE MEMBERS
19. ANATOMY OF A WEB API Trip travelers [string array] description [string] flight [Flight] ... SHAPE MEMBERS SPECIAL MEMBER
20. ANATOMY OF A WEB API Trip travelers [string array] description [string] flight [Flight] ... Flight flight_number [string] airline [string] status [string] ...
21. ANATOMY OF A WEB API ▸ GetTrip ▸ CreateTrip ▸ UpdateTrip ▸ ListTrips ▸ DeleteTrip
22. ANATOMY OF A WEB API ▸ GetTrip ▸ CreateTrip ▸ UpdateTrip ▸ ListTrips ▸ DeleteTrip ▸ GET /trips/:id ▸ POST /trips ▸ PATCH /trips/:id ▸ GET /trips ▸ DELETE /trips/:id
23. ANATOMY OF A WEB API ▸ GetTrip ▸ CreateTrip ▸ UpdateTrip ▸ ListTrips ▸ DeleteTrip ▸ GET /trips/:id ▸ POST /trips ▸ PATCH /trips/:id ▸ GET /trips ▸ DELETE /trips/:id ▸ 'trips#show' ▸ 'trips#create' ▸ 'trips#update' ▸ 'trips#index' ▸ 'trips#destroy'
24. ANATOMY OF A WEB API trip_client.get_trip(id: 123) GET /trips/123 { trip: { travelers: ["Alex"], description: "RailsConf 2018", flight: { flight_number: "DL3378", airline: "Delta", status: "Landed" } } }
25. ANATOMY OF A WEB API trip_client.get_trip(id: 123) GET /trips/123 { trip: { travelers: ["Alex"], description: "RailsConf 2018", flight: { flight_number: "DL3378", airline: "Delta", status: "Landed" } } }
26. ANATOMY OF A WEB API trip_client.get_trip(id: 123) GET /trips/123 { trip: { travelers: ["Alex"], description: "RailsConf 2018", flight: { flight_number: "DL3378", airline: "Delta", status: "Delayed" # Because of course it was. } } }
27. ANATOMY OF A WEB API trip_client.get_trip(id: 321) GET /trips/321 ResourceNotFound
28. ANATOMY OF A WEB API ▸ Resources: Things that an API manages. ▸ Shape: A structure with members. ▸ Input Shape: A top-level request shape passed to an operation. ▸ Output Shape: A top-level response shape returned from an operation. ▸ Error Shape: A structure that represents an error response from an operation. ▸ Member: Belongs to a shape, represents a property of the shape. ▸ Operation: Something exposed by the service that can be invoked. ▸ NOTE: Can all of this apply to libraries which expose an API but do not make calls to the internet? Absolutely.
29. SAFE API CHANGES
30. I MIGHT DROP THE WORLD ...IF I CHANGE HANDS. Lil' Wayne
31. ADDING & REMOVING Trip travelers [string array] description [string] flight [Flight] ...
32. ADDING & REMOVING Trip travelers [string array] description [string] flight [Flight] confirmed [boolean]
33. ADDING & REMOVING Trip travelers [string array] description [string] flight [Flight] confirmed [boolean]
34. ADDING & REMOVING Trip travelers [string array] description [string] flight [Flight] confirmed [boolean]
35. ADDING & REMOVING Trip travelers [string array] description [string] flight [Flight] confirmed [boolean] WRONG
36. ADDING & REMOVING Trip travelers [string array] description [string] flight [Flight] confirmed [booleanstring]
37. ADDING & REMOVING Trip travelers [string array] description [string] flight [Flight] confirmed [booleanstring] WRONG
38. ADDING MEMBERS Trip travelers [string array] description [string] flight [Flight] confirmed [boolean]
39. ADDING MEMBERS - NEW CLIENT trip_client = TripClient.new resp = trip_client.get_trip(id: 321) resp.travelers # => ["Alex"] resp.confirmed # => true
40. ADDING MEMBERS - OLD CLIENT trip_client = TripClient.new resp = trip_client.get_trip(id: 321) resp.travelers # => ["Alex"]
41. INPUT SHAPES - ADDING AND REMOVING ListTripsRequest next_token [string] max_results [integer] ... ...
42. INPUT SHAPES - ADDING AND REMOVING ListTripsRequest next_token [string] max_results [integer] confirmed [boolean] ...
43. INPUT SHAPES - ADDING AND REMOVING ListTripsRequest next_token [string] max_results [integer] confirmed [boolean] ...
44. INPUT SHAPES - ADDING AND REMOVING ListTripsRequest next_token [string] max_results [integer] confirmed [boolean] (Required) ...
45. INPUT SHAPES - ADDING AND REMOVING ListTripsRequest next_token [string] max_results [integer] confirmed [boolean] (Required) ... WRONG
46. INPUT SHAPES - ADDING AND REMOVING trip_client = TripClient.new resp = trip_client.list_trips
47. INPUT SHAPES - ADDING AND REMOVING trip_client = TripClient.new resp = trip_client.list_trips # TripClient::Errors::MissingRequiredParameter
48. INPUT SHAPES - ADDING AND REMOVING trip_client = TripClient.new resp = trip_client.list_trips # TripClient::Errors::MissingRequiredParameter WRONG
49. INPUT SHAPES - ADDING AND REMOVING trip_client = TripClient.new resp = trip_client.list_trips
50. INPUT SHAPES - ADDING AND REMOVING trip_client = TripClient.new resp = trip_client.list_trips # ArgumentError: missing required parameter # :confirmed
51. INPUT SHAPES - ADDING AND REMOVING trip_client = TripClient.new resp = trip_client.list_trips # ArgumentError: missing required parameter # :confirmed WRONG
52. KEY TAKEAWAYS ‣ Having your code break on you is a jarring, painful experience. Make serious efforts to avoid doing this to your users. ‣ Since some changes are harder than others, think ahead when initially launching your API to avoid the likelihood of needing to make "bad changes". ‣ For Web APIs, there is no complete way around this no matter how clever you are with clients. ‣ Community clients will be built. ‣ People will make raw, well-formed HTTP requests outside of your client libraries. ‣ Ruby does NOT get a free lunch.
53. CONSTRAINTS, VALIDATION, AND EXCEPTIONS
54. DID YOU HAVE TO DO THIS? I WAS THINKING THAT YOU COULD BE TRUSTED. Taylor Swift
55. AGAIN: NO ADDING NEW REQUIRED PARAMETERS ▸ Breaks existing code AND old clients. ▸ Removing "required" attribute from parameters is okay.
56. CONSTRAINT VALIDATION ListTripsRequest next_token [string] max_results [integer] (Maximum: 25) confirmed [boolean] ...
57. CONSTRAINT VALIDATION ListTripsRequest next_token [string] max_results [integer] (Maximum: 20) confirmed [boolean] ...
58. CONSTRAINT VALIDATION ListTripsRequest next_token [string] max_results [integer] (Maximum: 20) confirmed [boolean] ... WRONG
59. CONSTRAINT VALIDATION ListTripsRequest next_token [string] max_results [integer] (Maximum: 50) confirmed [boolean] ...
60. TERMINAL STATES Flight flight_number [string] airline [string] status [string] ...
61. TERMINAL STATES ▸ Flight.status ▸ Interim: OnTime, Delayed, Boarding, InFlight ▸ Terminal: Landed, Cancelled
62. TERMINAL STATES ▸ Flight.status ▸ Interim: OnTime, Delayed, Boarding, InFlight ▸ Terminal: Landed, LandedOnTime, LandedLate, Cancelled
63. TERMINAL STATES ▸ Flight.status ▸ Interim: OnTime, Delayed, Boarding, InFlight ▸ Terminal: Landed, LandedLate, Cancelled
64. TERMINAL STATES ▸ Flight.status ▸ Interim: OnTime, Delayed, Boarding, InFlight ▸ Terminal: Landed, LandedLate, Cancelled WRONG
65. TERMINAL STATES ▸ Flight.status ▸ Interim: OnTime, Delayed, Boarding, InFlight ▸ Terminal: Landed, Cancelled
66. TERMINAL STATES ▸ Flight.status ▸ Interim: OnTime, Delayed, Boarding, InFlight, OnApproach ▸ Terminal: Landed, Cancelled
67. EXCEPTIONS trip_client.get_trip(id: 321) GET /trips/321 ResourceNotFound
68. EXCEPTIONS begin resp = trip_client.get_trip(id: id) rescue ResourceNotFound => e logger.info("Trip #{id} does not exist.") nil end
69. EXCEPTIONS trip_client.get_trip(id: 321) GET /trips/321 ResourceNotFound || ResourceDeleted
70. EXCEPTIONS begin resp = trip_client.get_trip(id: id) rescue ResourceNotFound => e logger.info("Trip #{id} does not exist.") nil end # Suddenly this raises ResourceDeleted...
71. EXCEPTIONS begin resp = trip_client.get_trip(id: id) rescue ResourceNotFound => e logger.info("Trip #{id} does not exist.") nil end # Suddenly this raises ResourceDeleted... WRONG
72. EXCEPTIONS begin resp = trip_client.get_trip(id: id) rescue ResourceNotFound => e if e.previously_existed? logger.info("Trip #{id} previously deleted.") else logger.info("Trip #{id} does not exist.") end nil end
73. UNDOCUMENTED API CONTRACTS
74. WHEN GIVEN A CHOICE BETWEEN TWO OPTIONS, CHOOSE ALL THREE. John Carlyle
75. WITH A SUFFICIENT NUMBER OF USERS OF AN API, IT DOES NOT MATTER WHAT YOU PROMISE IN THE CONTRACT, ALL OBSERVABLE BEHAVIORS OF YOUR SYSTEM WILL BE DEPENDED ON BY SOMEBODY. Hyrum's Law
76. CONCLUSION API DESIGN RULES YOU CAN USE!
77. RELATED TALKS ▸ AWS re:Invent 2017: Embracing Change without Breaking the World (DEV319) ▸ AWS re:Invent 2017: Deploying and Managing Ruby Applications on AWS (DEV207) ▸ RailsConf 2018: The GraphQL Way: A new path for JSON APIs. ▸ Note: This talk already happened, but you can view later on YouTube for a different POV on some of these topics.
78. FOUR OUT OF FIVE DENTISTS RECOMMEND SAVING THIS SLIDE FOR REFERENCE DO ▸ APIs ▸ Add members/shapes ▸ Add intermediate workflow states ▸ Add detail to exceptions ▸ Add new opt-in exceptions ▸ Loosening constraints ▸ Clients ▸ Forward compatible (support all of the above) ▸ Focus on discoverability ▸ APIs ▸ Remove/rename members and shapes. ▸ Change member types ▸ Add new terminal workflow states ▸ Add new exceptions without opt-in ▸ Tighten constraints ▸ Clients ▸ Validate API constraints DO NOT