On-demand image scaling with AWS Lambda and S3

Talk at AWS User Group Singapore Meetup.

Video: https://engineers.sg/v/1707

Transcript

1. On-demand image scaling with AWS Lambda Claudio Mettler
2. Project • Cloudification of a monolithic customized CMS based on TYPO3 CMS (PHP/MySQL) for an online news platform • Image scaling built into CMS/template language (TYPO3/fluid) • Unreliable • Slow
3. Template View Helpers <f:image src="img/test.jpg" width="50m" height="100"/>
 
 <f:image src="{ad.image}" width="50m" height="100"/>
5. Main Issues • Files on disk/EFS (and all in the same folder too!) • Lots of database queries • Unreliable
6. Replacement requirements • Use S3 (and Cloudfront) • Must be on-demand: required sizes can't be anticipated at time of upload • Minimize high-latency and high-CPU operations in page generating process • Minimize shared state/coupling for robustness
7. Step 1: Encode information in filename Original:
 /o/[content hash]-[namespace]-[id]-[w]-[h].jpg • Happens on image upload • Client library takes image file and namespace/id, uploads the file to S3 and returns path
8. Step 2: Derive scaled filename from original Scaled: /s/[content hash]-[namespace]-[id]-[ow]-[oh]/[w]x[h].jpg • Client library takes original path and scaling configuration, returns scaled path • No interaction with S3 or database necessary • This is all that has to be done on page generation
9. Step 3: Get S3 to act when scaled version doesn't exist • S3 can't trigger lambda and hold the request • It can Issue redirects and append the request path
10. S3 Routing Rules <RoutingRules> <RoutingRule> <Condition> <KeyPrefixEquals>s/</KeyPrefixEquals> <HttpErrorCodeReturnedEquals>404</HttpErrorCodeReturnedEquals> </Condition> <Redirect> <Protocol>https</Protocol> <HostName>[api gateway url]</HostName> <ReplaceKeyPrefixWith>prod/s/</ReplaceKeyPrefixWith> <HttpRedirectCode>302</HttpRedirectCode> </Redirect> </RoutingRule> </RoutingRules>
11. Step 4: Lambda scaling task • Derive original path from scaled path • Fetch & scale original image • Save to S3 • Redirect back to S3 • For future requests, image will be directly delivered by S3
12. Step 5: Clean up • Trigger cleanup Lambda task every time an original is deleted • Task deletes all scaled versions of the image
13. Security considerations • If i can create arbitrarily scaled versions of an image, so can everyone on the internet • Easy to drive Lambda and S3 costs up with a few HTTP requests
14. Solution: HMAC • Hash Message Authentication Code • Sign request URL with a shared key • Will not prevent people from invoking Lambda • Will prevent expensive operations on Lambda • Ideally, Cloudfront would handle redirects internally and never expose API Gateway URLs
15. Scaled image URL /s/38fff39b6a847fc2d349bd331d9919adf5648ff8- zna-312468-240-160/317x211m.jpg?dyCQEqMUMa9n0LDzuciu7o5Tvvw%3D
16. Demo • 16 JPEG pictures, 2448x2448, 1.2-1.6 MB in size each • Scaled down to 1000x1000
21. Questions? claudio@ctrlm.sg Terraform module: https://github.com/ponyfleisch/lambdaautoscale
 
 PHP Client lib: https://github.com/ponyfleisch/clownfish-client-php