apidays London 2023 - API Metrics matters in APIOps, Ludovic Pourrat, Lombard Odier Group

apidays London 2023 - APIs for Smarter Platforms and Business Processes
September 13 & 14, 2023

API Metrics matters in APIOps
Ludovic Pourrat, API Architect & API Catalyst, Lombard Odier Group

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. 14/09/2023 Ludovic Pourrat Apidays London 2023 API Metrics Matter in APIOps
2. 1 · Ludovic Pourrat P u b l i c API Architect & API Catalyst | Platform Architect Lombard Odier Group Auxiliary Ski Instructor Ecole Suisse de Ski, Verbier
3. 2 · Ludovic Pourrat P u b l i c Lombard Odier Group Private Bank in Switzerland since 1796 Main businesses • Private Clients • Asset Management • Technology for Banking Technology • Financial Software Solution Developer • BPO activity «Bank as a service»
4. 3 · Ludovic Pourrat P u b l i c Banking Platform 4 functional development streams • Market, Front, Tax & Operations, Finance Modular Service oriented solution • ≈ 800 application components GX - Large Modernization Initiative started in 2020 • Functional (e.g. SMF, Order Management) • Technical • OpenShift, GitOps, APIOps - Kong, Kafka, Vault, Consul, … • Angular micro-frontends • Migration Program - SpringBoot and Quarkus • Service Now, APM & OTEL, OIDC • Cloud - development environments
5. API Metrics & Discovered Landscape
6. 5 · Ludovic Pourrat P u b l i c API Metrics Overview A wide range of available metrics Performance Metrics Latency Time Throughput Cache Hit Rate Resource Utilization Reliability and Availability Metrics Uptime Error Code Breakdown & Error Rate SLA Compliance
7. 6 · Ludovic Pourrat P u b l i c API Metrics Overview A wide range of available metrics Performance Metrics Latency Time Throughput Cache Hit Rate Resource Utilization Reliability and Availability Metrics Uptime Error Code Breakdown & Error Rate SLA Compliance Security Metrics OWASP Top 10 API Conformance Authentication & Authorization Rate Limiting Developer & Consumer Experience Metrics Engagement Rate Adoption Rate Retention & Churn Rate Documentation Quality and Coverage Rate Validation Error Rate
8. 7 · Ludovic Pourrat P u b l i c API Metrics Overview A wide range of available metrics Standard Conformance Metrics OpenAPI / GraphQL / AsyncAPI / gRPC Conformance Custom API Standard Conformance Change Management Metrics API Version Adoption & Retention Rate Deprecation Effectiveness Deprecation Usage API Version Views & Sunset Compliance SDK Version Usage Performance Metrics Latency Time Throughput Cache Hit Rate Resource Utilization Reliability and Availability Metrics Uptime Error Code Breakdown & Error Rate SLA Compliance Security Metrics OWASP Top 10 API Conformance Authentication & Authorization Rate Limiting Developer & Consumer Experience Metrics Engagement Rate Adoption Rate Retention & Churn Rate Documentation Quality and Coverage Rate Validation Error Rate
9. 8 · Ludovic Pourrat P u b l i c API Metrics Overview A wide range of available metrics AI Readiness & Usability Metrics AI & bot consumption readiness AI context prompt awareness Data Governance Metrics Data Classification & Tagging Data Privacy Data Quality Standard Conformance Metrics OpenAPI / GraphQL / AsyncAPI / gRPC Conformance Custom API Standard Conformance Change Management Metrics API Version Adoption & Retention Rate Deprecation Effectiveness Deprecation Usage API Version Views & Sunset Compliance SDK Version Usage Performance Metrics Latency Time Throughput Cache Hit Rate Resource Utilization Reliability and Availability Metrics Uptime Error Code Breakdown & Error Rate SLA Compliance Security Metrics OWASP Top 10 API Conformance Authentication & Authorization Rate Limiting Developer & Consumer Experience Metrics Engagement Rate Adoption Rate Retention & Churn Rate Documentation Quality and Coverage Rate Validation Error Rate
10. 9 · Ludovic Pourrat P u b l i c API Metrics Landscape Identified so far and not exhaustive Reliability and Availability Metrics Standard Conformance Metrics Data Governance Metrics Performance Metrics Change Management Metrics Security Metrics AI Readiness & Usability Metrics Developer Experience Metrics API Gateways API Gateways API Portals OTEL OTEL
11. API Metrics Applied to APIOps
12. 11 · Ludovic Pourrat P u b l i c API Metrics applied to APIOps Design Develop Secure Deploy Publish Monitor Analyze Developer Experience Metrics API documentation via descriptions and attached Markdown files API conformance with the OpenAPI / GraphQL / AsyncAPI standard API conformance with your own API standard API Standard Conformance Metrics API samples explained with a business context Built-in and custom resources constraints definition and documentation API backward compatibility checks & API breaking changes API Security Score OWASP API Security Top 10 2023 Assessment Data Governance Metrics Data classification Data ownership Performance Metrics Resource allocations & limitations Refine Metrics Collections Review metrics accuracy Engage with new metric sensors Focus & Prioritization Review the topmost areas of improvement Data Governance Metrics Data Privacy Nothing Yet ? We discuss that later
13. API Specification Overlay
14. 13 · Ludovic Pourrat P u b l i c Current State Solution Driver But we already got something we invested upon and that is well understood. Problem Statement It looks quite complex to compute all those metrics and to feed them back into your APIOps cycles. Market Readiness There is not yet a single solution or provider to gather all the API metrics you may need.
15. 14 · Ludovic Pourrat P u b l i c Overlaying API Specification with API Metrics API Overlay API Specification API Extensions The API overlay definition provide a common definition of how to apply modifications on the API specification. The API overlay is a method to patch the API specification to apply the API metrics. 3. API Overlay Allows extending the existing API definitions with custom metadata. The extensions are the placeholder for the API metrics. 2. API Extensions We start from what matter the most, so the API specification. 1. API Specification https://github.com/OAI/Overlay-Specification But we selected an easier approach by using the overlayed result instead.
16. 15 · Ludovic Pourrat P u b l i c API Specification Overlayed openapi: 3.1.0 info: […] termsOfService: www.lombardodier.com/terms-and-conditions.html contact: name: Lombard Odier url: 'https://www.lombardodier.com/home.html' email: contact@lombardodier.com x-api-conformance-score: 80 paths: ‘/my-resource’: x-api-operation-latency: 20ms […] asyncapi: 2.6.0 info: […] termsOfService: www.lombardodier.com/terms-and-conditions.html contact: name: Lombard Odier url: 'https://www.lombardodier.com/home.html' email: contact@lombardodier.com x-api-conformance-score: 40 channels: […] graphql: October 2021 info: […] termsOfService: www.lombardodier.com/terms-and-conditions.html contact: name: Lombard Odier url: 'https://www.lombardodier.com/home.html' email: contact@lombardodier.com x-api-conformance-score: 60 spec: | type Query { […] } openapi: 3.1.0 info: x-api-conformance-score: 80 paths: ‘/my-resource’: x-api-operation-latency: 20ms Overlay with Extensions 1. OpenAPI 2. AsyncAPI 3. GraphQL
17. API Overlay Computation
18. 17 · Ludovic Pourrat P u b l i c API Specification & API Mock | A Kubernetes Native Automation The API specification and the API mock definition are handled as Kubernetes custom resource definitions. Those can be also created at deployment for upper environments. API Artefacts as CRDs
19. 18 · Ludovic Pourrat P u b l i c API Specification & API Mock | A Kubernetes Native Automation The API specification is automatically deployed into our API developer portal and an API mock is instantiated within our API sandbox. Kubernetes Controllers
20. 19 · Ludovic Pourrat P u b l i c API Overlay Computation | A Kubernetes Extension The API overlay is ‘yet another’ Kubernetes custom resource definition. API Artefacts as CRDs We added a Kubernetes controller to compute the API metrics from the various sources and to generate the API overlay.. Kubernetes Controllers
21. 20 · Ludovic Pourrat P u b l i c API Overlay Computation | A Kubernetes Extension Then the overlay could extend the API specification definition and the API mock definition. Kubernetes Controllers
22. 21 · Ludovic Pourrat P u b l i c Finally, the API Developer Portal View Once collected the API overlay is published in the API developer portal. The API card can be flipped to display synthetics about the API metrics. API Card
23. 22 · Ludovic Pourrat P u b l i c API Overlay Overview From the source The API artefacts are collected from the source in Git. API Metrics Collection The API metrics are collected from the different sources. Enhance API Developer Portal The API metrics are visible directly from the API specification. API Overlay The API overlay describe the collected API metrics. API Sandbox The API mocks can be updated according to the API metrics. API Artefacts The API specification and the API mock definition are handled as CRDs.
24. 23 · Ludovic Pourrat P u b l i c Recap about the process 1 5 4 3 2 All the organization can leverage and benefits from the API metrics and ease decision making The API overlay can be fed into you APIOps cycles along your iterations The API overlay and the API extensions are a simple vehicle to carry-on API metrics The API metrics are computed from their sources The API specifications are the source of truth and ease any API related analysis API Metrics matter in APIOps As an outcome your APIOps cycles are enriched with valuable API metrics to foster better developer and consumer experiences. API Metrics in the API Portal We think that is the sweet spot to display back the collected API metrics.
25. API Metrics | Maturity Matrix
26. 25 · Ludovic Pourrat P u b l i c Implement fundamental metrics to establish a baseline. Focus on performance metrics Level 1 - Basic Expand your metrics with cover usage, utilization, security and start documenting the APIs Level 2 - Intermediate Add the data governance metrics and gather the reliability and availability metrics Level 4 - Expert Dive deeper by adding conformance, consumer and developer experience metrics Level 3 - Advanced Continuously monitor and optimize the metrics. Explore AI consumption and integration metrics Level 5- Optimized API Metrics | Maturity Matrix The five levels of maturity
27. 26 · Ludovic Pourrat P u b l i c Q&A Wish you some h[API]ness at Apidays ! Better skiers have more fun.