Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Video Stream on Android (DevFest Tokyo 2016)

Video Stream on Android (DevFest Tokyo 2016)

Introduction to Video Stream
- Http Live Streaming (HLS)
- Dynamic Adaptive Streaming over HTTP (DASH)
- H.264
- H.265
- Fragmented MP4
- ExoPlayer
- DRM (Widevine, PlayReady)

Daichi Furiya (Wasabeef)

October 09, 2016
Tweet

More Decks by Daichi Furiya (Wasabeef)

Other Decks in Programming

Transcript

  1. Video compression 30fps 1sec 60KB × 30 = 1.8MB 1min

    1.8MB × 60 = 108MB 1hour 108MB × 10 = 1.1GB 60KB 65KB 55KB 50KB
  2. Video compression 60KB 5KB 2KB 7KB 30fps 1sec 60KB +

    (4KB × 30) = 180KB 1min 60KB + (4KB × 1800) = 7.2MB 1hour 7.2MB × 10 = 72MB
  3. .avi .mp4 .mkv .ogg .flv etc.. Container format Video H.264

    H.265 Divx VP9 Audio AAC WMA Voribis PCM Caption SAMI, SMIL Hi-Caption CMML, DXFP Meta Date Author Title
  4. HLS

  5. # ffmpeg -i BigBuckBunny.mp4 \ -vcodec libx264 \ -s 1280x720

    \ -acodec aac -b:a 256k \ -flags +loop-global_header \ -bsf h264_mp4toannexb \ -f segment -segment_format mpegts \ -segment_time 10 \ -segment_list output.m3u8 output_%04d.ts Converting MP4 to HLS With FFmpeg
  6. ts?

  7. #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:0 #EXTINF:10, no desc output0000.ts #EXTINF:10, no desc

    output0001.ts #EXTINF:10, no desc 〜 ... 〜 output180.ts #EXT-X-ENDLIST playlist.m3u8
  8. manifest.mdp <?xml version="1.0" ?> <MPD mediaPresentationDuration="PT9M56.458S" minBufferTime="PT20.00S" profiles="urn:mpeg:dash:profile:isoff-live:2011" type="static" xmlns="urn:mpeg:dash:schema:mpd:2011">

    <-- Created with Bento4 mp4-dash.py, VERSION=1.7.0-613 --> <Period> <-- Video --> <AdaptationSet maxHeight="180" maxWidth="320" mimeType="video/mp4" minHeight="180" minWidth="320" segmentAlignment="true" startWithSAP="1"> <ContentProtection schemeIdUri="urn:mpeg:dash:abmprotection:2016" value="0x55ea5d616e90eaca972bc31a9fb2c7f8,0xaccca4b41de3d9afb029070eb564be40"> </ContentProtection> <SegmentTemplate duration="20000" initialization="$/init.mp4" media="$/seg-$.m4s" startNumber="1" timescale="1000"/> <Representation bandwidth="702842" codecs="avc1.42C00D" frameRate="24" height="180" id="../video_enc/1" scanType="progressive" width="320"/> </AdaptationSet> <-- Audio --> <AdaptationSet mimeType="audio/mp4" segmentAlignment="true" startWithSAP="1"> <SegmentTemplate duration="20000" initialization="$/init.mp4" media="$/seg-$.m4s" startNumber="1" timescale="1000"/> <Representation audioSamplingRate="48000" bandwidth="163859" codecs="mp4a.40.2" id="../audio/und/mp4a"> <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/> </Representation> </AdaptationSet> </Period> </MPD>
  9. <mdp> - Root manifest.mdp <period> - A start time and

    duration. <AdaptationSet> - Audio/Video content <Representation> - Bitrate and Aspect
  10. MediaPlayer String url = “http://wasabeef.jp/sample.mp4”; MediaPlayer mediaPlayer = new MediaPlayer();

    mediaPlayer.setAudioStreamType(STREAM_MUSIC); mediaPlayer.setDataSource(url); mediaPlayer.prepare(); mediaPlayer.setSurface(surface); mediaPlayer.start(); // mediaPlayer.release();
  11. API Level 16 Easy to extend and customize HLS, MPEG-DASH,

    SmoothStreaming DRM (WideVine, PlayReady) ExoPlayer
  12. Devices Use Case Version API Level Audio 4.1 16 Video

    4.1 16 WideVine 4.3 18 PlayReady AndroidTV AndroidTV
  13. Formats (Containers) Feature HLS MPEG-DASH fMP4 ✕ ◯ WebM ✕

    ◯ Matroska ✕ ◯ MPEG-TS ◯ ✕ ADTS(AAC) ◯ ✕ MP3 ◯ ✕
  14. Formats (Closed Caption) Feature HLS MPEG-DASH TTML ✕ ◯ WebVTT

    ◯ ◯ Tx3g ✕ ◯ SubRip ✕ ◯ EIA-608 ◯ ✕
  15. 2.x

  16. A greatly simplified demo app. A VideoView equivalent in SimpleExoPlayerView.

    Improved Audio extension APIs with pre- built extensions for ffmpeg, opus, and flac. Playlist Support. New features
  17. MediaSource // Measures bandwidth during playback. Can be null if

    not required. DefaultBandwidthMeter bandwidth = new DefaultBandwidthMeter(); // Produces DataSource instances through which media data is loaded. DataSource.Factory dataSource = new DefaultDataSourceFactory(this, Util.getUserAgent(this, "wasabeef"), bandwidthMeter); // Produces Extractor instances for parsing the media data. ExtractorsFactory extractors = new DefaultExtractorsFactory(); // This is the MediaSource representing the media to be played. MediaSource videoSource = new ExtractorMediaSource(mp4VideoUri, dataSource, extractors, null, null); // Prepare the player with the source. player.prepare(videoSource);
  18. #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:0 #EXTINF:10, no desc output0000.ts #EXTINF:10, no desc

    output0001.ts #EXTINF:10, no desc 〜 省略 〜 output180.ts #EXT-X-ENDLIST Parser
  19. HlsPlaylistParser private static final String TAG_VERSION = "#EXT-X-VERSION"; private static

    final String TAG_STREAM_INF = "#EXT-X-STREAM-INF"; private static final String TAG_MEDIA = "#EXT-X-MEDIA"; private static final String TAG_DISCONTINUITY = "#EXT-X-DISCONTINUITY"; private static final String TAG_MEDIA_DURATION = “#EXTINF"; // ...
  20. DataSource // Measures bandwidth during playback. Can be null if

    not required. DefaultBandwidthMeter bandwidth = new DefaultBandwidthMeter(); // Produces DataSource instances through which media data is loaded. DataSource.Factory dataSource = new DefaultDataSourceFactory(this, Util.getUserAgent(this, "wasabeef"), bandwidthMeter); // Produces Extractor instances for parsing the media data. ExtractorsFactory extractors = new DefaultExtractorsFactory(); // This is the MediaSource representing the media to be played. MediaSource videoSource = new ExtractorMediaSource(mp4VideoUri, dataSource, extractors, null, null); // Prepare the player with the source. player.prepare(videoSource);
  21. A Hollywood grade DRM is not always needed, sometimes it’s

    enough to just add another layer of security through AES encryption. HLS Encryption
  22. HLS Encryption - DataSource @Override public long open(DataSpec dataSpec) throws

    IOException { Assertions.checkState(dataSource == null); // Choose the correct source for the scheme. String scheme = dataSpec.uri.getScheme(); if (Util.isLocalFileUri(dataSpec.uri)) { if (dataSpec.uri.getPath().startsWith("/android_asset/")) { dataSource = assetDataSource; } else { dataSource = fileDataSource; } } else if (SCHEME_ASSET.equals(scheme)) { dataSource = assetDataSource; } else if (SCHEME_CONTENT.equals(scheme)) { dataSource = contentDataSource; } else { dataSource = baseDataSource; } // Open the source and return. return dataSource.open(dataSpec); }
  23. HLS Encryption - DataSource @Override public long open(DataSpec dataSpec) throws

    IOException { Assertions.checkState(dataSource == null); // Choose the correct source for the scheme. String scheme = dataSpec.uri.getScheme(); if (Util.isLocalFileUri(dataSpec.uri)) { if (dataSpec.uri.getPath().startsWith("/android_asset/")) { dataSource = assetDataSource; } else { dataSource = fileDataSource; } } else if (SCHEME_ASSET.equals(scheme)) { dataSource = assetDataSource; } else if (SCHEME_CONTENT.equals(scheme)) { dataSource = contentDataSource; } else if (SCHEME_WASABEEF.equals(scheme)) { dataSource = wasabeefDataSource; } else { dataSource = baseDataSource; } // Open the source and return. return dataSource.open(dataSpec); }
  24. The timed metadata information can be used by clients as

    cuepoints, for information display, to invoke time- aligned actions, and so on. This information is available to the client on the playback timeline in the form of ID3 tags in HLS. ID3v2 Tags
  25. manifest.mdp <?xml version="1.0" ?> <MPD mediaPresentationDuration="PT9M56.458S" minBufferTime="PT20.00S" profiles="urn:mpeg:dash:profile:isoff-live:2011" type="static" xmlns="urn:mpeg:dash:schema:mpd:2011">

    <-- Created with Bento4 mp4-dash.py, VERSION=1.7.0-613 --> <Period> <-- Video --> <AdaptationSet maxHeight="180" maxWidth="320" mimeType="video/mp4" minHeight="180" minWidth="320" segmentAlignment="true" startWithSAP="1"> <ContentProtection schemeIdUri="urn:mpeg:dash:abmprotection:2016" value="0x55ea5d616e90eaca972bc31a9fb2c7f8,0xaccca4b41de3d9afb029070eb564be40"> </ContentProtection> <SegmentTemplate duration="20000" initialization="$/init.mp4" media="$/seg-$.m4s" startNumber="1" timescale="1000"/> <Representation bandwidth="702842" codecs="avc1.42C00D" frameRate="24" height="180" id="../video_enc/1" scanType="progressive" width="320"/> </AdaptationSet> <-- Audio --> <AdaptationSet mimeType="audio/mp4" segmentAlignment="true" startWithSAP="1"> <SegmentTemplate duration="20000" initialization="$/init.mp4" media="$/seg-$.m4s" startNumber="1" timescale="1000"/> <Representation audioSamplingRate="48000" bandwidth="163859" codecs="mp4a.40.2" id="../audio/und/mp4a"> <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/> </Representation> </AdaptationSet> </Period> </MPD>
  26. Content Protection - DataSource @Override public long open(DataSpec dataSpec) throws

    IOException { Assertions.checkState(dataSource == null); // Choose the correct source for the scheme. String scheme = dataSpec.uri.getScheme(); if (Util.isLocalFileUri(dataSpec.uri)) { if (dataSpec.uri.getPath().startsWith("/android_asset/")) { dataSource = assetDataSource; } else { dataSource = fileDataSource; } } else if (SCHEME_ASSET.equals(scheme)) { dataSource = assetDataSource; } else if (SCHEME_CONTENT.equals(scheme)) { dataSource = contentDataSource; } else if (SCHEME_WASABEEF.equals(scheme)) { dataSource = wasabeefDataSource; } else { dataSource = baseDataSource; } // Open the source and return. return dataSource.open(dataSpec); }
  27. DRM PlayReady Widevine(M) Widevine(C) FairPlay Android (4.3+) ✕ ◯ ◯

    ✕ Android (3+) ✕ ✕ ◯ ✕ iOS (6+) ✕ ✕ ✕ ◯ Windows Phone ◯ ✕ ✕ ✕ $ISPNF  ✕ ◯ ✕ ✕ 'JSFGPY  ✕ ◯ ✕ ✕ *&  ◯ ✕ ✕ ✕ Edge (Win10+) ◯ ✕ ✕ ✕ Safari(8+) ✕ ✕ ✕ ◯ Opera (31+) ✕ ◯ ✕ ✕ Chromecast ◯ ◯ ✕ ✕ Android TV ◯ ◯ ✕ ✕ Apple TV ✕ ✕ ✕ ✕ Amazno Fire TV ◯ ✕ ✕ ✕
  28. DRM HttpMediaDrmCallback callback = new HttpMediaDrmCallback(licenseUrl, dataSource, null); FrameworkMediaDrm mediaDrm

    = FrameworkMediaDrm.newInstance(uuid); DrmSessionManager<FrameworkMediaCrypto> dsm = new StreamingDrmSessionManager<> (uuid, mediaDrm, callback, null, handler, logger); DefaultLoadControl dlc = new DefaultLoadControl() ExoPlayer player = ExoPlayerFactory.newSimpleInstance(this, trackSelector, dlc, dsm, false);