Apidays New York 2024 - The subtle art of API rate limiting by Josh Twist, Zuplo

The subtle art of API rate limiting
Josh Twist, Co-founder & CEO at Zuplo

Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024)

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. The subtle art of API rate limiting Josh Twist CEO @zuplo
3. Me Founded several services in Microsoft Azure: API Management Logic Apps Mobile Services Power Automate Pro Head of Product at Stripe and Facebook Been a vendor and a customer of API gateways / management
4. Agenda Why do we need this? What’s the canonical approach? Private or Public Latency tradeoffs and optimizing the customer experience Observability Demo Content
5. Am I actually going to be attacked?
6. Am I actually going to be attacked? Yes, but not by who you think.
7. Why do we need rate limiting? Protect resource consumption Protect resource starvation Ensure good experience for all customers Discourage abusers Enforce business model
8. A canonical response Status - 429 Status Text - Too Many Requests Header - Retry-After: 3600 Body - Use Problem Details format (IETF RFC 7807)
9. A canonical response Status - 429 Status Text - Too Many Requests Header - Retry-After: 3600 Body - Use Problem Details format (IETF RFC 7807) Do you want to do this?
10. Private or public limits? Disclosing limits allows malicious users to easily maximize their consumption of your resources without hitting the limit Not disclosing limits makes it hard for consumers to avoid hitting your rate limit Informal Partnership 
(e.g. Free Tier) Formal Partnership 
(e.g. Contracted Customer) Hide limits Disclose limits
11. How to apply rate limiting IP based?
12. Rate limiting in distributed systems is hard
13. The latency / accuracy tradeoff Being accurate means completing the rate limit check before allowing the request to proceed == a slower API for everyone You incur the latency on every request, whether an problem is afoot or not Consider a more lenient approach where you run the check asynchronously and cache blocked consumers Here’s how
14. Async rate limiting Have a local cache to store known limited consumers (store the retry time also) Check the rate limit in parallel with performing the primary request If rate limit check comes back blocked, update the cache and, if the primary request not compete, override the response. Otherwise, the next request will be blocked anyway.
15. Observability? RPS (requests per second) report See rate limited responses Break down by bucket (IP, user etc) What are the minimum requirements
16. This content available online now Blog post: zuplo.link/rate-limiting-blog Video: zuplo.link/rate-limiting-video