Deep Dive into Lambda Response Streaming

JAWS PANKRATION 2024 でお話しした内容になります。
https://jawspankration2024.jaws-ug.jp/ja/timetable/TT-58/

Transcript

1. Deep Dive into Lambda Response Streaming Yohei Watanabe [AWS Ambassador, AWS Community Builder] [JAWS-UG Tokyo]
2. Happy 10th Anniversary, Lambda! 2 https://aws.amazon.com/jp/blogs/aws/run-code-cloud/
3. Introduction • Introduced in 2022, Response Streaming is a key feature in AWS Lambda. • It's particularly relevant in the current LLM boom, where streaming large volumes of text output is becoming the norm. • While Lambda supports this capability, comprehensive documentation is still limited. 3
4. Me Watanabe Yohei(@watany) • NTT TechnoCross Corporation • JAWS-UG Tokyo Organizer • Title https://jawsug.connpass.com/event/316451/
5. Table of Contents 1. What is Lambda Response Streaming? 2. Managed Runtime 3. Custom Runtime 4. Lambda Web Adapter (LWA) 5
6. １．What is Lambda Response Streaming? 6
7. What is ”Response Streaming” ? Lambda Response Streaming allows immediate sending of available response data back to the caller. • Efficiently returns large data, up to 20MB • Reduces time to first byte (TTFB) to just a few milliseconds, minimizing response latency 7 https://aws.amazon.com/jp/blogs/compute/introducing-aws-lambda-resp onse-streaming/
8. What is ”Response Streaming” ? Key Focus Areas: • LLM Chat UI Responses ◦ Examples: Claude, ChatGPT • Server-Side Rendering (SSR) ◦ Examples: Next.js, Remix 8
9. What is ”Response Streaming” ? Written in SAM/CFn 9
10. What is ”Response Streaming” ? Written in CDK 10
11. Response Streaming Limitations • Types of Streaming • Limits of Data Size • Buffering • Runtime-specific Limitations 11
12. Types of ”Response Streaming” Which types of streaming does Lambda Response Streaming support？ • Transfer-Encoding: chunked • Server-Sent Events (SSE) No • Websocket 12
13. Limits Payload Size Limits • Buffered (Normal): 6MB • Response Streaming: 20MB Bandwidth limits • First 6MB: Uncapped bandwidth for the initial 6MB of your function’s response. • After 6MB: Streaming limited to 2MBps. 13
14. Buffering • Undocumented Issue: Small response data may not be output. • Estimated Flush Threshold: Around 100KB. • Ref: Forcing Lambda Response Streaming to Flush Content ◦ https://betterdev.blog/lambda-response-streaming-flush-c ontent/ 14
15. Runtime-specific Limitations Each runtime has its own limitations; let’s go through them one by one. • Managed Runtime • Custom Runtime • Managed or Container With Lambda Web Adapter 15
16. 2．Managed Runtime 16
17. Managed Runtime • Available in Node.js v14 and above. • Not supported in other languages. • Wrap your function using the decorator implemented in this managed runtime. 17
18. Decorator Wrap with the awslambda.streamifyResponse decorator (right) 18
19. awslambda.streamifyResponse Pipeline is recommended over Write for Streams. 19
20. Write metadata 20 awslambda.HttpResponseStream.from(stream, metadata) Use Case: • custom HTTP response status codes • custom HTTP headers • some cookie data to the client.
21. ３．Custom Runtime 21
22. Custom Runtime Overview • Customizable Environment: Allows you to define your own runtime environment instead of using AWS’s managed runtimes. • bootstrap Script: Central to custom runtimes, handling initialization, request processing, and response. • Runtime API Communication: Interacts with the Lambda Runtime API for event handling and response delivery. 22 https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html
23. Custom Runtime Overview After the Execution Environment starts, the Runtime and Function communicate requests and responses via the Lambda Runtime API. 23 https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html
24. Custom Runtime Overview Request: The Runtime API GET /runtime/invocation/next fetches the event, which the Runtime and Function then process. 24 https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html curl -X GET "http://${AWS_LAMBDA_RUNTIME_API}/r untime/invocation/next"
25. Custom Runtime Overview Response: The Runtime and Function process the result, then POST it to the Runtime API at /runtime/invocation/AwsRequestId/response. 25 https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html curl -X POST "http://${AWS_LAMBDA_RUNTIME_API}/r untime/invocation/${AwsRequestId}/re sponse"
26. How to Enable Response Streaming in a Custom Runtime When POST to Runtime API /runtime/invocation/AwsRequestId/response, do the following • "Add the following to the headers ◦ Lambda-Runtime-Function-Response-Mode: streaming ◦ Transfer-Encoding: chunked • Send the response in chunks, then close the connection. • and other error handling 26
27. How to Enable Response Streaming in a Custom Runtime • The current implementation covers the equivalent of `awslambda.streamifyResponse`. • To implement the equivalent of `awslambda.HttpResponseStream`.from, you need: ◦ Set Content-Type to `application/vnd.awslambda.http-integration-response`. ◦ Send custom headers (status code, headers, cookies) in JSON. ◦ Add 8 NULL characters as separators. ◦ Encode the response using HTTP/1.1 chunked transfer. • Ref：https://aws.amazon.com/jp/blogs/compute/using-response-streaming -with-aws-lambda-web-adapter-to-optimize-performance/ 27
28. Implementation Example Refer to the implemented "Rust Runtime for AWS Lambda" for guidance. 28 https://github.com/awslabs/aws-lambda-rust-runtime/blob/fbf212f4eef8c0fd8bd87f87998239fa17bc2b23/lambda-runtime/src/streaming.rs
29. ４．Lambda Web Adapter (LWA) 29
30. What is Lambda Web Adapter(LWA)? • Run web apps on AWS Lambda using familiar frameworks like Express.js, Flask, and SpringBoot. • Deploy the same Docker image across AWS Lambda, EC2, Fargate, and local environments. 30 https://github.com/awslabs/aws-lambda-web-adapter
31. Why Response Streaming Works with LWA? Lambda Web Adapter includes a "Runtime Interface Client." 31 https://github.com/awslabs/aws-lambda-web-adapter
32. Why Response Streaming Works with LWA? Lambda Web Adapter connects the "Web App" and "Runtime API. 32 https://aws.amazon.com/jp/blogs/compute/using-response-streaming-wit h-aws-lambda-web-adapter-to-optimize-performance/ Triggers the app to start upon receiving an HTTP request. Converts events fetched from the Runtime API into HTTP requests.
33. Why Response Streaming Works with LWA? Lambda Web Adapter connects the "Web App" and "Runtime API. 33 https://aws.amazon.com/jp/blogs/compute/using-response-streaming-wit h-aws-lambda-web-adapter-to-optimize-performance/ Returns the HTTP Response to the Lambda Web Adapter (LWA). Converts the HTTP Response and POSTs it to the Runtime API.
34. Why Response Streaming Works with LWA? Two Common Use Cases for Using LWA 1. Container Runtime 2. Managed Runtime 34
35. 4－1．Lambda Web Adapter With Container Runtime 35
36. Container Runtime + LWA LWA Binary move to `/opt/extensions/` 36
37. Container Runtime + LWA It works with just one additional line. (2nd line) Nothing else is special. 37
38. 4－2．Lambda Web Adapter With Managed Runtime 38
39. Managed Runtime + LWA １．Attach as a Lambda Layer 39
40. Managed Runtime + LWA 2. Lambda Environment Variables 40
41. Managed Runtime + LWA 2. Lambda Environment Variables Must: • AWS_LWA_EXEC_WRAPPER: /opt/bootstrap ◦ Required for LWA + Managed runtimes. • AWS_LWA_INVOKE_MODE: RESPONSE_STREAM ◦ Important for streaming. (Functions URL alone is not enough) 41
42. Managed Runtime + LWA 2. Lambda Environment Variables Optional: • https://github.com/awslabs/ aws-lambda-web-adapter 42
43. Managed Runtime + LWA 3. Run Script 43
44. Managed Runtime + LWA 3. Run Script(FastAPI) 44
45. Managed Runtime + LWA 3. Run Script(Express.js) 45
46. Managed Runtime + LWA Appendix. The LWA GitHub repository contains examples for various frameworks. https://github.com/awslabs/aws-lambda -web-adapter/tree/main/examples/ 46
47. Closing 47
48. Which one should you use? • If you can use Node.js, that's the best choice. ◦ Also, consider using Hono. • If you can trigger your app with an HTTP request, LWA is ideal. • Custom runtimes are challenging, but examples are available. 48
49. See you 👋 49