Behind the Stream - How AbemaTV Engineers Build Video Apps at Scale

AbemaTV, Inc. All Rights Reserved  AbemaTV, Inc. All Rights Reserved 
1 Behind the Stream How AbemaTV Engineers Build Video Apps at Scale 2026 January 15th Tokyo Video Tech #11 Orbit Yusuke Goto （五藤佑典） Developer Expert for Video Technology at CyberAgent, Inc. Director of Device Engineering at AbemaTV, Inc

AbemaTV, Inc. All Rights Reserved  五藤佑典 YUSUKE GOTO https://ygoto3.com/
@ygoto3_ • Majored in Graphic Design at California State University, San Bernardino • CyberAgent Developer Expert at CyberAgent, Inc. • Director of Device Engineering at AbemaTV, Inc. Career History 1. Graphic / Web Designer 2. Marketer 3. Web Engineer 4. Video Engineer 5. Product Manager 6. Director of Engineering

AbemaTV, Inc. All Rights Reserved  ABEMA 4 Hybrid of TV
& video Allowing you to enjoy both 24/7 linear streaming and video on demand that allows you to watch missed or exclusive content at your preferred time without registration. Largest collection of original episodes in Japan Offering over 30,000 episodes available for streaming at any given time with the largest* collection of original episodes among domestic video services. (*as of January 2022, based on in-house research) 100% professional content Delivering high-quality content through a production system that leverages the respective strengths of CyberAgent and TV Asahi. Diverse lineup Broadcasting approximately 20 channels of diverse genres 24/7, including Japan’s only 24-hour news channel, as well as original dramas, romance programs, anime, sports, and more.

AbemaTV, Inc. All Rights Reserved  ABEMA 5 Multi-device compatibility and
diverse channel lineups

AbemaTV, Inc. All Rights Reserved  Evolution of ABEMA's Business Model
Fully free service offering linear broadcasts + advertising • Access to past episodes was partially paid Transition to freemium model (hybrid of FAST + SVOD) Introduction of AVOD offerings Segmented ad delivery implementation Addition of Pay-Per-View models Implementation of programmatic advertising Introduction of split-screen advertising format Launch of partnership programs Release of advertisement-supported premium subscription plan 2016 2017 2024 2018 2023 2021 2020

AbemaTV, Inc. All Rights Reserved  Historical Engineering Structure App Development
Group Server-side development teams Ad Tech Group Client-side development teams Server-side development teams AbemaTV in 2016

AbemaTV, Inc. All Rights Reserved  Initial Stage What we should
do was obvious • There's no VOD functionality => Let’s implement it! • There are no advertising features in VOD => Let’s implement it! • There’s no favorite list functionality => Let’s implement it! • There’s no start over playback functionality => Let’s implement it!

AbemaTV, Inc. All Rights Reserved  Initial Stage What we should
do was obvious • There's no VOD functionality => Let’s implement it! • There are no advertising features in VOD => Let’s implement it! • There’s no favorite list functionality => Let’s implement it! • There’s no start over playback functionality => Let’s implement it! Speed is paramount

AbemaTV, Inc. All Rights Reserved  However, After the Essential Features
were Implemented … • Gradually increased opportunities to question which features are most needed for business growth at this moment • The “speed is paramount” had accumulated technical debt, slowing development velocity

AbemaTV, Inc. All Rights Reserved  After Essential Features were Implemented
… • Gradually increased opportunities to question which features are most needed for business growth at this moment • The “speed is paramount” had accumulated technical debt, slowing development velocity Gradually shifted focus on key performance factors of experience at ABEMA

AbemaTV, Inc. All Rights Reserved  Key Performance Factors of Experience
at ABEMA • Navigation flow efficiency • Natural conversion points for payment initiation • Advertising revenue generation • User viewing duration

at ABEMA • Navigation flow efficiency • Natural conversion points for payment initiation • Advertising revenue generation • User viewing duration How can we focus on them?

at ABEMA • Navigation flow efficiency • Natural conversion points for payment initiation • Advertising revenue generation • User viewing duration How can we focus on them? Transitioning toward a development structure focused on the key performance factors

AbemaTV, Inc. All Rights Reserved  Segmentation to Engage in the
Key Service Factors App Development Group Server-side development teams Client-side development teams 2016 Navigation Development Group Payment Development Group Video Engineering Group 2026

Key Service Factors AbemaTV in 2026 Navigation Development Group Ad Tech Group Server-side Client-side Server-side Payment Development Group Server-side Client-side Video Engineering Group Server-side Client-side

Key Service Factors AbemaTV in 2026 Navigation Development Group Ad Tech Group Payment Development Group Video Engineering Group Establishing domain-speciﬁc commitments leads to gaining deeper understanding of associated challenges and solve them more effectively

AbemaTV, Inc. All Rights Reserved  Difference Between 2016 and 2026
2016 Model Lack of domain expertise accumulation • e.g. Unable to diagnose the cause of DRM not functioning on speciﬁc devices • e.g. End up with a tightly coupled UI and playback functionality implementation 2026 Model Domain experts evolve naturally over time • e.g. Able to resolve issues, understanding device revocations and behavior variations across SDK versions • e.g. Able to design future-proofed content protection system

AbemaTV, Inc. All Rights Reserved  Downside • Create many intersections
of requests between groups • Due to differing primary responsibilities/focus areas, deeper contextual requirements emerged that needed to be negotiated Even if each group could independently come up with optimal solutions, delivering a single feature to users still requires coordination across multiple groups

AbemaTV, Inc. All Rights Reserved  Intersection of Requests Navigation Development
Group Payment Development Group Video Engineering Group Ad Tech Group

Group Payment Development Group Video Engineering Group Ad Tech Group Need player functionality suitable for instant preview features

Group Payment Development Group Video Engineering Group Ad Tech Group Need to modify UI view components to switch to more robust DRM methods

Group Payment Development Group Video Engineering Group Ad Tech Group Want to insert personalize advertisements

Group Payment Development Group Video Engineering Group Ad Tech Group Should optimize encoding parameters of the ads

Group Payment Development Group Video Engineering Group Ad Tech Group Want video download capability to be premium, thus requiring download playback functionality

Group Payment Development Group Video Engineering Group Ad Tech Group Want to track ad viewability, so need to register UI positioning parameters

AbemaTV, Inc. All Rights Reserved  Product Management has Become Harder
Intersections of Requests The probability of discovering issues during ﬁnal integration to increase => Increased delivery time The priorities in the other groups to frequently diverge => Increased development time Old system replacement to go slowly (Scaling quality to enhance market competitiveness requires modiﬁcations to a historic system infrastructure) cause

To manage the quality of product at scale, What should we do?

Group Payment Development Group Video Engineering Group Ad Tech Group To launch partnership collaboration business, need functionality to receive and distribute partner streams

Group Payment Development Group Video Engineering Group Ad Tech Group Want to quickly ﬁnalize payment system changes and conduct early testing of partner’s stream reception and playback

AbemaTV, Inc. All Rights Reserved  Testing Partnership Programs Negative Conditions
• The content of partner streams remains unknown until the systems are fully integrated ◦ Of course, general speciﬁcations are clear, but the actual output can only be determined after actual encoding ◦ Plus, even if we can receive feeds in advance, there was no mechanism to actually test the viewing experience until the integration • ABEMA maintains 22 reference devices as testbeds, with validation based on whether these devices pass at least 2-hour aging tests ◦ 2 hours * 22 devices * 3 viewing modes = 132 hours are required to perform aging tests ◦ At that time, aging testing was performed manually by human observation

• The content of partner streams remains unknown until the systems are fully integrated ◦ Of course, general speciﬁcations are clear, but the actual output can only be determined after actual encoding ◦ Plus, even if we can receive feeds in advance, there is no mechanism to actually test the viewing experience until the integration • ABEMA maintains 22 reference devices as testbeds, with validation based on whether these devices pass at least 2-hour aging tests ◦ 2 hours * 22 devices * 3 viewing modes = 132 hours are required to perform aging tests ◦ At that time, aging testing was performed manually by human observation We managed to complete aging testing and necessary ﬁxes within the limited timeframe to launch with satisfactory quality… However, we still incurred additional costs a lot

• The content of partner streams remains unknown until the systems are fully integrated ◦ Of course, general speciﬁcations are clear, but the actual output can only be determined after actual encoding ◦ Plus, even if we can receive feeds in advance, there is no mechanism to actually test the viewing experience until the integration • ABEMA maintains 22 reference devices as testbeds, with validation based on whether these devices pass at least 2-hour aging tests ◦ 2 hours * 22 devices * 3 viewing modes = 132 hours are required to perform aging tests ◦ At that time, aging testing was performed manually by human observation We managed to complete aging testing and necessary ﬁxes within the limited timeframe to launch with satisfactory quality… However, we still incurred additional costs a lot We can't pay such cost for every additional partner !!

AbemaTV, Inc. All Rights Reserved  Testing Partnership Programs Two problems
we aim to solve 1. We cannot start new feature testing until the system is completely integrated 2. Aging tests for multi-device support takes too long

To manage the quality of product at scale, What should we do?

To manage the quality of product at scale, What should we do? For video engineering, we decided to develop 2 quality assuring products to scale the video viewing experience

AbemaTV, Inc. All Rights Reserved  2 Quality Assuring Products Code
name : Golem Code name : Sphinx

AbemaTV, Inc. All Rights Reserved  We cannot start new feature
testing until the system is completely integrated What can we do to assure the quality before the integration?

AbemaTV, Inc. All Rights Reserved  We cannot start new feature
testing until the system is completely integrated What can we do to assure the quality before the integration? Develop specialized sandbox testing system for viewing test purposes

AbemaTV, Inc. All Rights Reserved  Golem : Making the Product
Requirements from Engineers’ Voice • Video feature development often requires more time setting up reproduction conditions than debugging time itself • With TV devices in particular, setting up reproduction conditions like character input via remote control takes considerable time • I want to enhance quality down to every detail, but there simply isn't enough time Engineer S

AbemaTV, Inc. All Rights Reserved  Golem Multi-platform video functionalities sandbox
testing environment Purpose • If quality assurance can be completed in Golem, all video functions should operate flawlessly • Enable testing of actual viewing experiences even when the systems are not fully integrated yet Main features • Can run any staging videos with arbitrary configurations • Can enable configuration of advertising insertion, including arbitrary VMAP and VAST settings • Can simplify difficult-to-use UI elements (such as keyboard operations via remote control)

AbemaTV, Inc. All Rights Reserved  Golem Google Cloud | GKE
Simpliﬁed Diagram Request the web app (HTML) Command Control message (WebSocket) Control message (WebSocket) Host the web app Controller DUT (Web) DUT (Installed App) Control Device

AbemaTV, Inc. All Rights Reserved  Result Before After Video Engineering
Navigation Development QA Develop Fix Develop Fix Test E2E Re-test Video Engineering Navigation Development QA Develop Develop Fix Test video feats Re-test Fix Test E2E Release Release Faster & Less bugs

AbemaTV, Inc. All Rights Reserved  Result Before After Video Engineering
Navigation Development QA Develop Fix Develop Fix Test E2E Re-test Video Engineering Navigation Development QA Develop Develop Fix Test video feats Re-test Fix Test E2E Release Release Faster & Less bugs Engineer S • Should keep developing Golem to fully automate all repetitive testing related to video viewing, so test frequency becomes inﬁnite, enabling continuous quality improvement However, automation is limited yet

AbemaTV, Inc. All Rights Reserved  Aging tests for multi-device support
takes too long What can we do to to ﬁnish aging tests in just one day Develop automated aging testing systems to work as substitutes for human eyes

AbemaTV, Inc. All Rights Reserved  Golem : Making the Product
Requirements from Engineers’ Voice • We want to adjust encoding parameters to improve coding eﬃciency, but delays in aging tests make this diﬃcult to do casually • Coordinating with QA team planning takes much time Engineer N • We should evaluate viewing experience quality quantitatively • We should automate the evaluation process so anyone can perform it Engineers

AbemaTV, Inc. All Rights Reserved  Sphinx Mechanism ensuring playback stability
through aging testing Purpose • Reduce costs and improve frequency of aging testing execution • In aging testing for video playback experiences, we eliminate human observation and automatically analyze metrics that would otherwise be missed by visual inspection Use cases • When modifying encoding parameters • When changing advertising insertion methods • When adjusting CDN conﬁgurations

AbemaTV, Inc. All Rights Reserved  Sphinx AWS Simpliﬁed Diagram Recorded
video Trigger Analyze the video Record the playback session Analysis result External RAM System RAM data Identiﬁer metadata Record Aggregate the id and analysis

video Trigger Analyze the video Record the playback session Analysis result External RAM System RAM data Identiﬁer metadata Record Aggregate the id and analysis Setup is as easy as this

video Trigger Analyze the video Record the playback session Analysis result External RAM System RAM data Identiﬁer metadata Record Aggregate the id and analysis Setup is as easy as this Able to analyze recordings even for DRM content !

AbemaTV, Inc. All Rights Reserved  Sphinx calculates freeze events by
analyzing the recorded video’s timestamps and frames with RTFDA (Real-Time Video Freezing Detection Algorithm) https://link.springer.com/article/10.1007/s11554-019-00873-y

Intersections of Requests The probability of discovering issues during ﬁnal integration to increase => Increased delivery time The priorities in the other groups to frequently diverge => Increased development time Old system replacement to go slowly (Scaling quality to enhance market competitiveness requires modiﬁcations to a historic system infrastructure) cause

The probability of discovering issues during ﬁnal integration to increase => Increased delivery time The priorities in the other groups to frequently diverge => Increased development time Old system replacement to go slowly (Scaling quality to enhance market competitiveness requires modiﬁcations to a historic system infrastructure) Test all conditions in sandbox environments => Reduced time to identify causes Develop and test future features => Eliminated development time overhead Implementing replacements within the viewing system itself => The system evolves independently

Behind the Stream - How AbemaTV Engineers Build...

Behind the Stream - How AbemaTV Engineers Build Video Apps at Scale

Transcript

More Decks by Yusuke Goto

Other Decks in Technology

Featured