Share
Embed
Transcript
108 slides
-
• https://openrec.tv • ήʔϜಛԽͷੜ์ૹαʔϏε • MAU 200ສ • ߴը࣭ɺԆ৴ɺ͍͔͚࠶ੜɺ̐ϲࠃޠ(CJK+E)ରԠ •
Web/iOS/tvOS/Android/AndroidTV -
E CyberAgent group iOS BeerBash։࠵! ݄ Ր ɹ࣌։࢝ʢ࣌ΑΓड։࢝ʣ αΠόʔΤʔδΣϯτʢौ୩ϓϥΠϜϓϥβ'ɹΫϦΤΠςΟϒϥϯδʣ dɹ֓ཁઆ໌ɺסഋ
dɹஊ dɹ৽ઃͷ৽نαʔϏε։ൃ৫ɹ$"54ʢΫϥΠΞϯτٕज़ྖҬΛݗҾ͢Δ৫ʣʹ͍ͭͯ dɹϚονϯάΞϓϦʮλοϓϧੜʯʹ͓͚Δ։ൃͷมԽ dɹஊ ͝Ԡื23ίʔυ͔Β ͪ͜Β͔ΒͷΤϯτϦʔશһ͝ࢀՃ͍͚ͨͩ·͢ connpass͔ΒΤϯτϦʔͷ߹ɺࢀՃநબͱͳΓ·͢ ฐάϧʔϓͷiOSDCొஃऀ͕ࢀՃ͢ΔɹBeerBashͰ͢ʂ দԬࢀՃ͠·͢ʂ ࣌ ॴ ༰ ɹɹɹɹɹɹɹגࣜձࣾαΠόʔΤʔδΣϯτ$MJFOU"EWBODFE5FDIOPMPHZ4UVEJP $"54 ɹҏ౻ɹګฏ ɹɹɹɹɹɹɹגࣜձࣾϚονϯάΤʔδΣϯτɹ∁ڮɹ༏հ -
ΞδΣϯμ •HLSͷ֓ཁ •ReplayKitͷ֓ཁ •ը໘ऩͷϥΠϑαΠΫϧ •SampleBufferͷฤू •ੜ์ૹதͷૢ࡞ •৴ͷҟৗऴྃͱ࠶։ •iOS AppͱBroadcaster Upload
Extensionͷσʔλڞ༗ •ExtensionΛσόοά͢Δ -
৴ΞϓϦ OBS ৴ج൫ ΞϓϦ ಈըใ ࢹௌऀ frame .ts .m3u8 ಈըใ
.ts .m3u8 ৴ऀ Storage/CDN HLSʹΑΔಈը৴ͷશମ૾ URI -
৴ΞϓϦ OBS Storage/CDN ৴ج൫ ΞϓϦ ಈըใ ࢹௌऀ .ts .m3u8 frame
.ts .m3u8 ಈըใ ৴ऀ ࠓճѻ͏෦ ɾReplayKit ɾBroadcast Upload Extension URI -
class SampleHandler: RPBroadcastSampleHandler { override func broadcastStarted(…) { // ৴։࢝
} override func processSampleBuffer(…) { // ը໘ऩϓϩηε͔ΒΩϟϓνϟ͞ΕͨϑϨʔϜͷड͚औΓ } override func finishBroadcastWithError() { // ৴ऴྃ } override func finishBroadcastWithError(…) { // ҟৗऴྃ } } ΤϯτϦʔϙΠϯτ SampleHandler.swift https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler -
֤ϥΠϑαΠΫϧຖͷΞΫγϣϯ Start Recording Finish εϖοΫνΣοΫ/ೝূ ৴ઌͷղܾ ৴ج൫ʹଓ Τϯίʔυͱ৴ ଓΛஅ ৴த
৴։࢝ ৴ऴྃ ঢ়ଶ ঢ়ଶຖͷΞΫγϣϯ ͷঢ়ଶΛࢹ -
Recording Τϯίʔυͱ৴ ɾϝσΟΞຖʹΤϯίʔυ(h264/AAC)͠ɺૹ৴ ɾόοϑΝͷฤू/ࠩ͠ସ͑ ͷঢ়ଶΛࢹ ɾը໘ͷճసใ ɾ৴ઃఆͷมߋ ɾը໘ڞ༗ΦϑϞʔυ ɾ௨ͷON/OFF ɾ௨ͷண৴
ɾͷϩοΫɺిΕ ɾωοτϫʔΫঢ়گͷࢹ ɾΞϓϦ͔Βͷऴྃ ಈըͷݟ͑ํ தஅ/ऴྃ/ϦτϥΠ -
ը໘ऩϓϩηε͔Βड͚औ ΔϝσΟΞσʔλ func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
// process each media data } ϝσΟΞͷੜσʔλɻόΠτใ͔Β ֤ϝσΟΞʹ෮ݩͯ͠ฤूͨ͠ΓΤϯίʔυͨ͠Γ͢Δ video, audioApp, audioMicͷ̏छྨ sampleBufferType sampleBuffer -
NT NT NT NT NT NT SampleBuffer(video) t 154 #VGGFS
αϯϓϦϯάλΠϛϯάը໘ʹߋ৽͕͋ͬͨͱ͖ DurationલճͷαϯϓϦϯά͔ΒࠓճͷαϯϓϦϯά -
࣮ࡍͷ ϑϨʔϜ I Frame P Frame ࠩͷΈΛ࣋ͭ͜ͱͰσʔλྔΛѹॖ HLSʹ͓͍ͯI Frame1ͭͷtsϑΝΠϧຖʹ ࠷ॳͷϑϨʔϜͱؚͯ͠ΜͰ͍ͳ͚ΕͳΒͳ͍
ΩʔϑϨʔϜ(I Frame)ͱ ಈըΛ࠶ݱ͢ΔͨΊʹbitmapใΛؚͯ͢ΜͩϑϨʔϜ -
t NT NT NT NT NT NT videoϑϨʔϜͷϦΧόϦͰରԠ 154 ϑϨʔϜΛૹ৴ͯ͠Ұఆ͕࣌ؒܦͬͯ࣍ͷϑϨʔϜ͕ͳ͍࣌
࠷ޙʹૹ৴ͨ͠ϑϨʔϜΛɺPTSΛ͚ସ͑ͯૹ৴͢Δ #VGGFS -
CMTime ࣌ؒใΛࢄత(ϑϨʔϜ)ʹͰѻ͏ public struct CMTime { public var value: CMTimeValue
public var timescale: CMTimeScale public var flags: CMTimeFlags public var epoch: CMTimeEpoch public init() public init(value: CMTimeValue, timescale: CMTimeScale, flags: CMTimeFlags, epoch: CMTimeEpoch) } extension CMTime { public init(seconds: Double, preferredTimescale: CMTimeScale) public init(value: CMTimeValue, timescale: CMTimeScale) } value: ϑϨʔϜ timescale: 1ඵ͋ͨΓΛԿϑϨʔϜʹׂ͢Δ͔ʁ -
NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈
㽈 t 㽈 αϯϓϦϯάλΠϛϯάఆظతɻ ִؒ1024/44100ඵ(≒23ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS -
NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈
㽈 t 㽈 ϏσΦόοϑΝࠓͷදࣔͷόοϑΝʹର͠ ΦʔσΟΦόοϑΝࠓ·ͰԻ͞ΕͨόοϑΝΛड͚औΔ 㽈 㽈 㽈 㽈 㽈 औಘ λΠϛϯά ᶃ ᶄ ᶅ ᶆ ᶇ ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS -
NT NT NT NT NT CMSampleBuffer(audioApp) 154 㽈 㽈 㽈
㽈 t 㽈 ϚΠΫಉ༷αϯϓϦϯάλΠϛϯά΄΅ఆظతɻ ִؒ22050/44100ඵલޙ(≒500ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS -
ө૾ͷฤू ө૾ͷฤूͱ͍ͬͯCIImageʹΑΔฤू ·ͣCMSampleBuffer͔ΒCIImageܗࣜʹม͢Δ extension CMSampleBuffer { var ciImage: CIImage? {
guard let pixelBuffer = CMSampleBufferGetImageBuffer(self) else { return nil } CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly); defer { CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly) } return CIImage(cvPixelBuffer: pixelBuffer) } } CMSampleBuffer > CVPixelBuffer > CIImage -
CMSampleBufferʹ͢ // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) // ۣܗใ
let dimensions = CMVideoFormatDescriptionGetDimensions(CMSampleBufferGetFormatDescription(sampleBuffer)!) // ϐΫηϧใͷॻ͖ࠐΈઌͷόοϑΝΛ࡞ var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil, &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) // ৽͍͠SampleBufferΛ࡞ var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer) -
var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil,
&outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } CIImageͷॻ͖ࠐΈઌͷόοϑΝΛ࡞͢Δ CMSampleBufferʹ͢ᶃ -
var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo
= self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer) ॻ͖ࠐΜͩBufferͱݩͷCMSampleBufferͷ ࣌ؒใΛͬͯ৽ͨͳCMSampleBufferΛੜ͢Δ CMSampleBufferʹ͢ᶅ -
NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF
όοϑΝͷࠩ͠ସ͑ -
NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF
όοϑΝͷࠩ͠ସ͑ -
όοϑΝͷࠩ͠ସ͑ ֖ֆͷCMSampleBufferΛϗϧμʔʹอ͓͖࣋ͯ͠ ετϦʔϜͷϋϯυϦϯάଆͰόοϑΝΛࠩ͠ସ͑ class MaskHolder { static let shared =
MaskHolder() private let maskBuffer: CMSampleBuffer func mask(with sampleTimingInfo: CMSampleTimingInfo) -> CMSampleBuffer { return maskBuffer.copy(with: sampleTimingInfo) } } class BroadcastService { var isEnabledMask: Bool let rtmpStream: RTMPStream func processSampleBuffer(sampleBuffer: CMSampleBuffer) { let buffer = isEnabledMask ? MaskHolder.shared.mask(with: sampleBuffer.sampleTimingInfo) : sampleBuffer rtmpStream.appendSampleBuffer(sampleBuffer: buffer) } } -
extension CMSampleBuffer { func copy(with timing: CMSampleTimingInfo) -> CMSampleBuffer? {
var sampleTiming = timing var copied: CMSampleBuffer? = nil CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorSystemDefault, self, 1, &sampleTiming, &copied) return copied } var sampleTimingInfo: CMSampleTimingInfo { return CMSampleTimingInfo(duration: self.duration, presentationTimeStamp: self.presentationTimeStamp, decodeTimeStamp: self.decodeTimeStamp) } var duration: CMTime { return CMSampleBufferGetDuration(self) } var presentationTimeStamp: CMTime { return CMSampleBufferGetPresentationTimeStamp(self) } var decodeTimeStamp: CMTime { return CMSampleBufferGetDecodeTimeStamp(self) } } CMSampleBuffer ؔݺͼग़͠ʹΑΔϓϩύςΟऔಘ͕ଟ͍ͷͰ extensionʹcomputed propertyΛੜ͓ͯ͘͠ͱศར όοϑΝͷࠩ͠ସ͑ -
extension CMSampleBuffer { var orientation: CGImagePropertyOrientation { guard let attachment
= CMGetAttachment(self, RPVideoSampleOrientationKey, nil) as? NSNumber else { return .up } if let orientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) { return orientation } return .up } } VQ MFGU SJHIU EPXO https://developer.apple.com/documentation/imageio/cgimagepropertyorientation CGImagePropertyOrientation -
ɾυΩϡϝϯτͰiOS11.0+͕ͩ… ɾiOS11.2.1ҎલͰىಈ࣌ʹϦϯΫΤϥʔͰڧ੍ऴྃ ɾiOS11.2.5Ҏ߱ͳ͠ CMGetAttachment(self, "RPSampleBufferVideoOrientation" as CFString, nil) RPVideoSampleOrientationKey ɾ11.2.5Ҏ߱ͰऔಘͰ͖ΔఆΛϕλଧͪ
ɾ11.2.1ҎલͰnil͕औಘ͞ΕΔ ɾ11.2.1ҎલͰىಈͰ͖Δ͜ͱ͕ॏཁ https://developer.apple.com/documentation/replaykit/rpvideosampleorientationkey -
DispatchGroup override open func broadcastFinished() { let dispathGroup = DispatchGroup()
dispathGroup.enter() broadcastService?.finish { dispathGroup.leave() } dispathGroup.wait() } DispatchGroupͰεϨουΛఀࢭͤ͞Δ -
BroadcastService func finish(completion: (()->Void)) { api.finishStream(movieID: 123) { completion?() }
} αʔϏεͰAPIΛड͚औͬͨΒcompletionΛ࣮ߦ -
ΞϓϦέʔγϣϯ͔Βऴྃ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) }
RPBroadcastSampleHandlerͰ Τϥʔऴྃͤ͞Δඞཁ͕͋Δ https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler/2721526-finishbroadcastwitherror -
৴Λҟৗऴྃͤ͞Δ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) }
దʹΤϥʔΛૹΔ͜ͱͰɺ μΠΞϩάʹऴྃཧ༝͕දࣔͰ͖Δ -
৴Λҟৗऴྃͤ͞Δ let message = “ωοτϫʔΫ͕அ͞Ε·ͨ͠" let error = NSError( domain:
Bundle.main.bundleIdentifier ?? “" , code: 1 , userInfo: [NSLocalizedFailureReasonErrorKey: message]) finishBroadcastWithError(error) userInfoʹϝοηʔδΛࢦఆ͠ͳ͍ͱ ऴྃཧ༝͕(null)ʹͳΓ·͢ɻ -
௨ͷண৴ import CallKit class SampleHandler: RPBroadcastSampleHandler { let callObserver =
CXCallObserver() override init() { super.init() callObserver.setDelegate(self, queue: nil) } } extension SampleHandler: CXCallObserverDelegate { func callObserver(…) { broadcastService.suspendStreaming() // ৴Λதஅ͢Δ } } -
AppGroupΛͬͯσʔλΛ ڞ༗͢Δ • ෳͷΞϓϦؒͰͷσʔλڞ༗ઃఆ • ಉ͡σϕϩούʔͷΞϓϦؒͰ༗ޮ • AppID͝ͱʹ༗ޮԽ͠ɺEntitlementͳͲͷ ઃఆ͕ඞཁ let
userDefaults: UserDefaults = UserDefaults(suiteName: “groups.jp.iosdc2018”) IDΛࢦఆ͢Δ͜ͱͰάϧʔϓؒͷڞ༗ྖҬʹΞΫηεͰ͖Δ -
ͳͷͰLoggerΛ͍·͠ΐ͏ • TCP Socket Logger • Logboard/LogboardConsole • JustLog •
File Logger • XCGLogger • SwiftyBeaver