Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Media streaming on Android @droidkaigi 2018

TakuSemba
February 08, 2018

Media streaming on Android @droidkaigi 2018

TakuSemba

February 08, 2018
Tweet

More Decks by TakuSemba

Other Decks in Technology

Transcript

  1. API level 1 API level 14 Simple use cases Advanced

    use cases Black Box Customizable MediaPlayer ExoPlayer
  2. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) Streaming 1 frame
  3. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) Streaming 1 second
  4. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits Streaming 1 second
  5. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits > 60m bps (4G LTE) Streaming 1 second
  6. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits > 60m bps (4G LTE) codec (h264) Streaming 1 second
  7. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits > 60m bps (4G LTE) 14.92m bits < 60m bps (4G LTE) codec (h264) Streaming 1 second
  8. HLS

  9. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 …

    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8
  10. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 …

    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000
  11. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … #EXTINF:10.0, http://media.example.com/segment1.ts
  12. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … http://media.example.com/segment1.ts
  13. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … http://media.example.com/segment2.ts
  14. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … http://media.example.com/segment3.ts
  15. ftyp: file type moof: header data sidx: index of moof

    + mdat Fragmented MP4 mdat: audio or video data
  16. MPD <MPD xmlns="urn:mpeg:dash:schema:mpd:2011" minBufferTime="PT1.500S" type="static" mediaPresentationDuration="PT0H3M10.024S" maxSegmentDuration="PT0H0M10.010S" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011"> <Period duration="PT0H3M10.024S">

    <AdaptationSet segmentAlignment="true" maxWidth="1280" maxHeight="720" maxFrameRate="24000/1001" par="16:9" lang="und" subsegmentAlignment="true" subsegmentStartsWithSAP="1"> <Representation id="1" mimeType="video/mp4" codecs="avc1.64001F" width="1280" height="720" frameRate="24000/1001" sar="1:1" startWithSAP="1" bandwidth="400823"> <BaseURL>400k.mp4</BaseURL> <SegmentBase indexRangeExact="true" indexRange="919-1262"> <Initialization range="0-918"/> </SegmentBase> … </Representation> </AdaptationSet> <AdaptationSet segmentAlignment="true" lang="und" subsegmentAlignment="true" subsegmentStartsWithSAP="1"> <Representation id="3" mimeType="audio/mp4" codecs="mp4a.40.2" startWithSAP="1" bandwidth="197469"> <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/> <BaseURL>output_audio_dashinit.mp4</BaseURL> <SegmentBase indexRangeExact="true" indexRange="861-1132"> <Initialization range="0-860"/> </SegmentBase> </Representation> </AdaptationSet> </Period> </MPD>
  17. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
  18. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() )
  19. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player
  20. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
  21. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true player.prepare(mediaSource) player.playWhenReady = true
  22. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true // clean up player.release()
  23. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
  24. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
  25. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
  26. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val okHttpClient = OkHttpClient.Builder() .connectTimeout(10L, TimeUnit.SECONDS) .writeTimeout(10L, TimeUnit.SECONDS) .readTimeout(30L, TimeUnit.SECONDS) .build() val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
  27. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val okHttpClient = OkHttpClient.Builder() .connectTimeout(10L, TimeUnit.SECONDS) .writeTimeout(10L, TimeUnit.SECONDS) .readTimeout(30L, TimeUnit.SECONDS) .build() val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
  28. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val okHttpClient = OkHttpClient.Builder() .connectTimeout(10L, TimeUnit.SECONDS) .writeTimeout(10L, TimeUnit.SECONDS) .readTimeout(30L, TimeUnit.SECONDS) .build() val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
  29. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val okHttpClient = OkHttpClient(…) val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
  30. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient(…) val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
  31. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient(…) val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl()
  32. Renderer public class MyRenderer implements Renderer { // make your

    own renderer } DefaultRenderersFactory(..., MyRenderer())
  33. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(),

    HIEST_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)
  34. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(),

    DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)
  35. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(),

    DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION) DefaultBandwidthMeter(),
  36. TrackSelector public final class DefaultBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return currentBitrate; } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } }
  37. TrackSelector public final class DefaultBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return currentBitrate; } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } } @Override public long getBitrateEstimate() { return currentBitrate; }
  38. TrackSelector public final class MyBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return Math.min(currentBitrate, limitBitrate); } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } } @Override public long getBitrateEstimate() { return Math.min(currentBitrate, limitBitrate); } MyBandwidthMeter
  39. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( MyBandwidthMeter(),

    DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)
  40. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( MyBandwidthMeter(),

    DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION) MyBandwidthMeter(),
  41. TrackSelector public final class MyBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return Math.max(currentBitrate, limitBitrate); } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } }
  42. TrackSelector public final class MyBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return Math.max(currentBitrate, limitBitrate); } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } } return Math.max(currentBitrate, limitBitrate); MyBandwidthMeter
  43. LoadControl val loadControl = DefaultLoadControl( DefaultAllocator(true, C.DEFAULT_BUFFER_SEGMENT_SIZE), DEFAULT_MIN_BUFFER_MS, DEFAULT_MAX_BUFFER_MS, DEFAULT_BUFFER_FOR_PLAYBACK_MS,

    DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS ) DEFAULT_MIN_BUFFER_MS, DEFAULT_MAX_BUFFER_MS, DEFAULT_BUFFER_FOR_PLAYBACK_MS, DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS
  44. MediaSource val firstSource = HlsMediaSource(firstUri, …) val secondSource = HlsMediaSource(secondUri,

    …) val mediaSource = ConcatenatingMediaSource(firstSource, secondSource) // play firstSource, then secondSource player.prepare(mediaSource) player.playWhenReady = true
  45. Metadata class Logger : MetadataOutput { override fun onMetadata(metadata: Metadata)

    { (1..metadata.length()) .map { metadata[it] } .filterIsInstance<TextInformationFrame>() .forEach { print("metadata: ${it.value}") } } } // metadata: id=1000,place=droidkaigi
  46. Metadata class Logger : MetadataOutput { override fun onMetadata(metadata: Metadata)

    { (1..metadata.length()) .map { metadata[it] } .filterIsInstance<EventMessage>() .forEach { print("metadata: ${it.value}") } } } // metadata: id=1000,place=droidkaigi
  47. RTMP ɾdeveloped by Adobe. not “open” protocol ɾRed5 started rtmp

    in 2005 ɾWowza started rtmp in 2007 ɾrtmpdump released in 2008 ɾrtmp specification released 2012
  48. RTMP PRO CON ɾsmall header (basically 1, 4, 8, 12

    bytes) ɾrealtime ɾdifficult to cache ɾno multiple resolutions
  49. public final class RtmpDataSource implements DataSource { private RtmpClient rtmpClient;

    ... @Override public int read(byte[] buffer, int offset, int readLength) throws IOException { int bytesRead = rtmpClient.read(buffer, offset, readLength); ... return bytesRead; } ... }
  50. public final class RtmpDataSource implements DataSource { private RtmpClient rtmpClient;

    ... @Override public int read(byte[] buffer, int offset, int readLength) throws IOException { int bytesRead = rtmpClient.read(buffer, offset, readLength); ... return bytesRead; } ... } private RtmpClient rtmpClient; rtmpClient.read(buffer, offset, readLength);
  51. public class RtmpClient { … public int read(byte[] data, int

    offset, int size) throws IOException { return nativeRead(data, offset, size, rtmpPointer); } private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer) throws IOException … }
  52. public class RtmpClient { … public int read(byte[] data, int

    offset, int size) throws IOException { return nativeRead(data, offset, size, rtmpPointer); } private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer) throws IOException … } private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer) public class RtmpClient
  53. RTMP ɾcapture data ɾencode data ɾpublish data val audioRecord =

    AudioRecord(...) // camera2 API val manager : CameraManager manager.openCamera(...) // camera API val camera = Camera.(info) Audio Data Video Data
  54. RTMP // MediaCodec.java val encoder = MediaCodec(…) encoder.start() val inputBuffers

    = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) ɾcapture data ɾencode data ɾpublish data
  55. RTMP // MediaCodec.java val encoder = MediaCodec(…) encoder.start() val inputBuffers

    = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) ɾcapture data ɾencode data ɾpublish data // MediaCodec.java val encoder = MediaCodec(…) encoder.start()
  56. RTMP ɾcapture data ɾencode data ɾpublish data // MediaCodec.java val

    encoder = MediaCodec(…) encoder.start() val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface()
  57. RTMP ɾcapture data ɾencode data ɾpublish data // MediaCodec.java val

    encoder = MediaCodec(…) encoder.start() val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) // get encoded data val outputBuffers = encoder.getOutputBuffer(index)
  58. RTMP // rtmpdump public native int writeVideo( byte[] data, int

    offset, int length, int timestamp ) ɾcapture data ɾencode data ɾpublish data
  59. RTMP // rtmpdump public native int writeVideo( byte[] data, int

    offset, int length, int timestamp ) // Socket.java val socket = Socket(“192.xx.xx.xxx", 1935) val outputStream = DataOutputStream(…) outputStream.write(byteArray) ɾcapture data ɾencode data ɾpublish data