[Journal club] FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging

Transcript

1. FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging ICLR 2026 Oral Ziyang Fan¹ Keyu Chen¹ Ruilong Xing¹ Yulin Li¹ Li Jiang²˒³ Zhuotao Tian¹˒³ ¹Harbin Institute of Technology, Shenzhen ²CUHK-Shenzhen ³Shenzhen Loop Area Institute 慶應義塾大学 杉浦孔明研究室 Hairong Shi 1
2. Abstract ◆ Background ▪ VLLMs process thousands of visual tokens per video ▪ Existing methods compress spatial and temporal redundancy independently ▪ Fixed-location temporal matching breaks under motion, scale, and viewpoint changes ◆ Method: FlashVID ▪ ADTS: selects informative and diverse tokens ▪ TSTM: builds redundancy trees across adjacent frames ▪ Training-free, plug-and-play hybrid compression ◆ Result ▪ 10% tokens retain 99.1% relative accuracy ▪ 6.3× faster prefilling; 2.1× faster TTFT ▪ 10× more frames yield +8.6% relative performance 2
3. BG: Visual Tokens Dominate VLLM Inference Cost ▪ F frames → vision encoder produces F × Nᵥ tokens; text contributes only Nₜ tokens ▪ Visual tokens dominate sequence length n = F × Nᵥ + Nₜ ▪ Longer videos therefore increase compute and memory rapidly ▪ Key Question: Can we compress visual tokens before the LLM without sacrificing video understanding? 3
4. Related Work : Training-free Visual Token Compression Method Where Reduction Happens Core Idea FastV [Chen+, ECCV24 Oral] Inside the LLM Image2Video: Text-to-visual attention pruning VisionZip [Yang+, CVPR25] Before the LLM Image2Video: [CLS] attention + spatial merging FastVID [Shen+, NeurIPS25] Before LLM Video segmentation + clustering  Most methods still decouple spatial and temporal redundancy FastV [Chen+, ECCV24] FastVID [Shen+, NeurIPS25] 4
5. Method: FlashVID → Motivation ◆ Observation: Temporal Redundancy Is Not Location -preserving • • • • • The same object moves, changes scale, rotates, or deforms over time Temporal Token Merging (TTM) matches same patch locations across frames Key Insight: Spatial and temporal redundancy are inherently coupled A useful compressor must follow semantic content rather than fixed coordinates Ron’s hair → Hermione‘s mouth Ron’s coat → Ron‘s hand 5
6. Method: FlashVID → Framework Overview ◆ Framework • • • Partition semantically similar consecutive frames • 1 st : Attention and Diversity-based Token Selection (ADTS) selects informative and diverse tokens • 2 nd : Tree-based Spatiotemporal Token Merging (TSTM) constructs and merges redundancy trees Each token links to its most similar token in the previous frame → merge each tree Advantage: training-free, plug-and-play 6
7. Method: FlashVID → Attention and Diversity-based Token Selection ◆ Attention and Diversity-based Token Selection (ADTS) • Start from frame-wise cosine distance matrix: • • • [CLS] attention calibration: the attention score received from [CLS] token • • ☺ Ensure feature diversity for robust video representations  Diversity alone may overlook the most informative visual tokens ☺ Highlight informative tokens in each frame Event relevance calibration: GAP and compute the event similarity matrices • ☺ Emphasize the tokens relevant to whole video 7
8. Method: FlashVID → Tree-based Spatiotemporal Token Merging ◆ Tree-based Spatiotemporal Token Merging • • 1 st : Construct spatiotemporal redundancy trees • Compute the cosine similarity matrix between visual features in adjacent frames • Each token links to its most similar counterpart in the previous frame 2 nd: Compress spatiotemporal redundancy Compressed representative token Aggregation function (mean pooling) i-th spatiotemporal redundancy tree 8
9. Evaluation: Setup ◆ Models • • • LLaVA-OneVision 7B LLaVA-Video 7B Qwen2.5-VL 7B ◆ Tasks • Video understanding: VideoMME, EgoSchema, LongVideoBench, MVBench, MLVU ◆ Baselines • Video-MME [Fu+, CVPR 25] FastV, VisionZip, PruneVID, FastVID ◆ Protocol • • R ∈ {25%, 20%, 15%, 10%} A800 80GB for evaluation; A100 for latency analysis EgoSchema [Mangalam +, NeurIPS 23] 9
10. Evaluation: Main Result → LLaVA-OneVision • • • FlashVID achieved SOTAs at every tested retention ratio At 10% retention, FlashVID preserved 99.1% relative accuracy At 15–25%, FlashVID exceeded the full-token model → “less is more” effect 10
11. Evaluation: Main Result → Cross-model Generalization • • FlashVID also outperformed other training-free methods on Qwen2.5-VL and LLaVA-Video Showed cross-model generalization 11
12. Evaluation: Results → Longer Context and Inference Efficiency ◆ Longer Video Context • 160 frames × 10% tokens outperformed 16 frames × 100% tokens by +8.6% • The largest absolute gain appeared on MLVU: 40.6 → 47.5 ◆ Efficiency • • token reduction with 99.1% relative accuracy 6.3× prefilling speedup and 2.1× Time-To-First-Token (TTFT) speedup 12
13. Qualitative Analysis: Tree-based Spatiotemporal Token Merging ▪ TSTM tracks semantically related regions despite changes in spatial position, scale, and orientation → captures fine-grained spatiotemporal redundancy beyond rigid spatial correspondence. 13
14. Qualitative Analysis: LLaVA-OneVision with and without FlashVID ▪ “less-is-more” effect: LLaVA OneVision with FlashVID compression could answer the questions correctly, whereas the original model with full visual tokens input gave incorrect answers 14
15. Conclusion ◆ Background ▪ Prior methods compress spatial and temporal redundancy independently ▪ Fixed-location temporal matching breaks under motion, scale, and viewpoint changes ◆ Method: FlashVID ▪ ADTS: selects informative and diverse tokens ▪ TSTM: builds redundancy trees across adjacent frames ▪ Training-free, plug-and-play hybrid compression ◆ Result ▪ Validated on 3 VLLMs and 4 video benchmarks ▪ 10% tokens → 99.1% accuracy; 6.3× prefill; 2.1× TTFT ▪ 10× more frames → +8.6% relative performance 15
16. Links ❖ Code https://github.com/Fanziyang-v/FlashVID ❖ Paper https://arxiv.org/pdf/2602.08024 16