Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Video streaming acceleration with Varnish

Video streaming acceleration with Varnish

In this presentation, I'm explaining how video streaming via HTTP works. We'll focus on the HLS standard.

The goal of the presentation is to explain the challenges when delivering video via the web and how Varnish can accelerate that delivery.

See https://feryn.eu/speaking/video-streaming-acceleration-varnish/ for more details

Thijs Feryn

May 08, 2019
Tweet

More Decks by Thijs Feryn

Other Decks in Technology

Transcript

  1. #EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:6 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-PLAYLIST-TYPE:VOD #EXTINF:6.000000, stream_00.ts #EXTINF:6.000000, stream_01.ts #EXTINF:6.000000,

    stream_02.ts #EXTINF:6.000000, stream_03.ts #EXTINF:6.000000, stream_04.ts #EXTINF:6.000000, stream_05.ts #EXTINF:6.000000, stream_06.ts #EXTINF:6.000000, stream_07.ts #EXTINF:6.000000, stream_08.ts #EXTINF:6.000000, stream_09.ts #EXTINF:5.280000, stream_010.ts #EXT-X-ENDLIST STREAM.M3U8
  2. <html> <head> <link href="https://vjs.zencdn.net/7.5.4/video-js.css" rel="stylesheet"> </head> <body> <video id='my-video' class='video-js'

    controls preload='auto' height='640' data- setup='{}'> <source src="/live/stream.m3u8" type="application/vnd.apple.mpegurl"> </video> <script src='https://vjs.zencdn.net/7.5.4/video.js'></script> </body> </html>
  3. ffmpeg -i video.mp4 \ -c:v libx264 -crf 21 \ -preset

    veryfast -g 25 -sc_threshold 0 \ -c:a aac -b:a 128k -ac 2 \ -f hls -hls_time 4 -hls_playlist_type vod stream.m3u8
  4. WHY IS VARNISH SO POWERFUL? ✓ EXTREMELY LOW RESOURCE ✓

    EXTREMELY STABLE ✓ 100 GBIT PER SERVER ✓ NO DISK ACCESS AT RUNTIME ✓ REQUEST COALESCING ✓ VARNISH CONFIGURATION LANGUAGE ✓ VMODS ✓ COMPLIES TO HTTP BEST PRACTICES ✓ ENTERPRISE FEATURES*
  5. LIVE ✓ LOW LATENCY ✓ CONSTANT PLAYLIST UPDATES ✓ LOW

    TTL ON PLAYLISTS ✓ ONLY THE LAST X SEGMENTS ARE REQUIRED ✓ SEGMENT SIZE TRADEOFF
  6. VOD ✓ NO PLAYLIST UPDATES ✓ HIGH TTL ON PLAYLISTS

    ✓ ALL SEGMENTS ARE REQUIRED ✓ STORAGE REQUIREMENTS ✓ PRE-FETCHING POSSIBLE
  7. WHERE VARNISH STORES ITS OBJECTS ✓ MEMORY ✓ DISK ✓

    MASSIVE STORAGE ENGINE (ENTERPRISE ONLY)
  8. ffmpeg -i video.mp4 -g 25 -sc_threshold 0 \ -map v:0

    -c:v:0 libx264 -b:v:0 800k \ -map v:0 -c:v:1 libx264-b:v:1 2500k \ -map a:0 -c:a aac -b:a 128k -ac 2 \ -f hls -hls_time 6 hls_playlist_type vod \ -master_pl_name master.m3u8 \ -var_stream_map "v:0,a:0 v:1,a:0" stream_%v.m3u8
  9. ffmpeg -listen 1 -i rtmp://0.0.0.0/live/stream \ -threads 0 -preset veryfast

    -g 25 -sc_threshold 0 \ -map v:0 -c:v:0 libx264 -b:v:0 800k \ -map v:0 -c:v:1 libx264 -b:v:1 2500k \ -map a:0 -c:a aac -b:a 128k -ac 2 \ -f hls -hls_time 6 -hls_flags delete_segments \ -hls_list_size 5 -hls_delete_threshold 1 \ -master_pl_name master.m3u8 \ -var_stream_map "v:0,a:0 v:1,a:1" /storage/hls/live/ stream_%v.m3u8
  10. $ openssl rand 16 > enc.key $ openssl rand -hex

    16 230f26620bfafa3cd420cb68c0647415 ENC.KEYINFO /live/enc.key /storage/hls/live/enc.key 230f26620bfafa3cd420cb68c0647415
  11. ffmpeg -listen 1 -i rtmp://0.0.0.0/live/stream \ -threads 0 -preset veryfast

    -g 25 -sc_threshold 0 \ -map v:0 -c:v:0 libx264 -b:v:0 800k \ -map v:0 -c:v:1 libx264 -b:v:1 2500k \ -map a:0 -c:a aac -b:a 128k -ac 2 \ -f hls -hls_time 6 -hls_flags delete_segments \ -hls_list_size 5 -hls_delete_threshold 1 \ -master_pl_name master.m3u8 \ -hls_key_info_file /storage/hls/live/enc.keyinfo \ -var_stream_map "v:0,a:0 v:1,a:1" /storage/hls/live/ stream_%v.m3u8
  12. vcl 4.1; import http; backend default { .host = "1.2.3.4";

    .port = "80"; } sub vcl_recv { if(req.url ~ "^/live") { if (req.http.Authorization != "Basic YWRtaW46YWRtaW4=") { return (synth(401, "Restricted")); } } unset req.http.Authorization; } sub vcl_backend_response { if(bereq.url ~ "^/live/stream_[0-9]+\.m3u8$" || bereq.url == "/live/master.m3u8") { set beresp.grace = 0s; set beresp.ttl = 1s; } } sub vcl_synth { if (resp.status == 401) { set resp.http.WWW-Authenticate = {"Basic realm="Restricted area""}; } }
  13. VARNISH ENTERPRISE ✓ CLIENT SSL TERMINATION ✓ BACKEND SSL CONNECTIONS

    ✓ PARALLEL ESI ✓ MASSIVE STORAGE ENGINE ✓ ENCRYPTION ✓ THROTTLING ✓ RATE LIMITING ✓ PREFETCHING ✓ GEOLOCATION ✓ AUTHENTICATION ✓ EDGESTASH ✓ CUSTOM STATISTICS ✓ ADMIN MODULE ✓ SUPPORT ✓ HIGH AVAILABILITY
  14. MSE ✓ MIX OF MEMORY AND DISK ✓ SMARTER LRU

    ✓ LESS DISK FRAGMENTATION ✓ INTELLIGENT STORAGE ROUTING ✓ MODULAR
  15. env: { id = "myenv"; memcache_size = "100G"; books =

    ( { id = "book1"; directory = "/var/lib/mse/book1"; database_size = "1G"; stores = ( { id = "store-1-1"; filename = "/var/lib/mse/stores/disk1/store-1-1.dat"; size = "1T"; }, { id = "store-1-2"; filename = "/var/lib/mse/stores/disk2/store-1-2.dat"; size = "1T"; } ); } };
  16. vcl 4.1; import file; backend default { .host = "origin";

    .port = "80"; } sub vcl_init { new root = file.init("/var/www/html/"); } sub vcl_backend_fetch { set bereq.backend = root.backend(); } sub vcl_backend_response { if(bereq.url ~ "^/live/stream_[0-9]+\.m3u8$" || bereq.url == "/live/master.m3u8") { set beresp.grace = 0s; set beresp.ttl = 1s; } }
  17. vcl 4.1; import xbody; include "devicedetect.vcl"; backend default { .host

    = "origin"; .port = "80"; } sub vcl_recv { call devicedetect; set req.http.X-UA-Device-Prefix = regsub(req.http.X-UA-Device,"(\w+)\-.*","\1"); } sub vcl_backend_response { if(bereq.url == "/live/master.m3u8" ){ set beresp.http.vary = "X-UA-Device-Prefix"; if(bereq.http.X-UA-Device-Prefix == "mobile") { xbody.regsub("\#EXT-X-STREAM-INF:BANDWIDTH=2890800.*\nstream_.*", ""); } } }
  18. vcl 4.1; import http; import std; backend default { .host

    = "origin"; .port = "80"; } sub vcl_recv { if (req.url ~ "^/(live|vod)/.+\.ts$") { http.init(0); http.req_set_max_loops(0,1); http.req_copy_headers(0); http.req_set_method(0, "HEAD"); set req.http.x-next-url = http.prefetch_next_url(); std.log("Prefetching " + req.http.x-next-url); http.req_set_url(0, req.http.x-next-url); http.req_send_and_finish(0); } }