Share
Embed
[輪講] Vision Transformer Adapter for Dense Predictions
Vision Transformerに視覚的帰納バイアスを注入することで密予測タスクへ効率的に適応させる手法である、ViT-Adapterについての輪講資料です。
論文:https://arxiv.org/abs/2205.08534
Naoki Kato
December 09, 2025
Transcript
19 slides
-
<ྠߨ>7JTJPO5SBOTGPSNFS"EBQUFS GPS%FOTF1SFEJDUJPOT ;$IFO :%VBO 88BOH +)F 5-V +%BJ :2JBP *$-3
IUUQTBSYJWPSHBCT ൃදɿ/BPLJ,BUP -
จ֓ཁ 7JTJPO5SBOTGPSNFS 7J5 ʹࢹ֮తؼೲόΠΞεΛೖ͠ ີ༧ଌλεΫޮతʹదԠͤ͞Δ 7J5"EBQUFS ΛఏҊ ଟ͘ͷίϯϖ্Ґղ๏Ͱར༻͞Ε͍ͯΔɿ • TU1MBDF4PMVUJPOGPSUIFUI-4704$IBMMFOHF7JEFP*OTUBODF4FHNFOUBUJPO
• $IBNQJPOTPMVUJPOJO5SBDL %0DDVQBODZ1SFEJDUJPO PGUIF$713 "VUPOPNPVT%SJWJOH$IBMMFOHF • $IBNQJPOTPMVUJPOJOUIF7JEFP4DFOF1BSTJOHJOUIF8JME$IBMMFOHFBU$713 %*/0W <4JNÉPOJ > ͰྖҬׂλεΫʹ࠾༻ʢޙड़ͷ 'FBUVSF*OKFDUPSআ֎ʣ બఆཧ༝ɿ՝ఏى ղܾࡦͱʹཧʹద͓ͬͯΓษڧʹͳΔྖҬׂͰڧ͍ -
7J5ࢹ֮తؼೲόΠΞεͷऑ͍ઃܭͰ͋Γॊೈੑ͕ߴ͍ҰํͰ ྖҬׂମݕग़ͷີ༧ଌλεΫͷੑೳʹྼΔ͕͋Δ 7J5ͷ 1ZSBNJE7JTJPO5SBOTGPSNFS <8BOH *$$7`> 4XJO5SBOTGPSNFS <-JV *$$7`>
ؼೲόΠΞε͕ऑ͍ ؼೲόΠΞε͕ڧ͍ 7J5 <%PTPWJUTLJZ *$-3`> ༧ଌΛՄೳʹ͢ΔͨΊͷԾఆࣄલ͔ࣝΒੜ͡ΔόΠΞε -
7J5"EBQUFS ࣄલֶश͞Εͨ 7J5ʹΞμϓλʔΛՃͯ͠ԼྲྀλεΫͰֶश • ϚϧνϞʔμϧσʔλͰࣄલֶश͞Εͨڧྗͳ 7J5Λ׆༻Ͱ͖Δ • ΞμϓλʔͰࢹ֮తؼೲόΠΞεΛೖ Step1:
Image Modality Pre-training Step2: Fine-tuning (a) Previous Paradigm Vision-Specific Transformer COCO ADE20K Det/Seg Vision-Specific Transformer Image Modality SL/SSL Step1: Multi-Modal Pre-training Step2: Fine-tuning with Adapter (b) Our Paradigm ViT Image, Video, Text, ... SL/SSL COCO ADE20K ViT Adapter Det/Seg -
طଘͷ 7J5׆༻ख๏ͱͷࠩ ೖྗը૾ใΛ༻͍ͨ 7J5ͱͷํͷใୡΛ௨ͯ͠ ಛϐϥϛουΛऔಘ ViT 1/32 up 4x
1/16 up 2x 1/8 1x 1/4 down 2x (a) Previous Method (Li et al., ViTDet) ViT Adapter (b) ViT-Adapter (ours) Image Image 1/32 1/16 1/8 1/4 using task prior using task prior & input 7J5ύονຒΊࠐΈͷ࣌Ͱۭؒใ͕ଛͳΘΕ͍ͯͦ͏ͳͷͰ߹ཧతʹࢥ͑Δ -
ΞʔΩςΫνϟશମ 7J5ʹ 4QBUJBM1SJPS.PEVMF 'FBUVSF*OKFDUPS&YUSBDUPSΛՃ • ࠷ऴ &YUSBDUPSͷಛϐϥϛουΛ༧ଌʹར༻ʢҙͷϔουΛ༻ʣ Patch Embedding
… Spatial Prior Module (a) Vision Transformer (ViT) (b) ViT-Adapter Block N Injector N Block 2 Extractor 2 Injector 2 Extractor 1 Block 1 Injector 1 Det Seg ... Position Embedding Element-wise Addition (c) Spatial Prior Module ℱ!" # Stem ℱ# ℱ$ ℱ% Extractor N (d) Spatial Feature Injector " Key/ Value ℱ!" & Query ℱ'() & Cross-Attention (e) Multi-Scale Feature Extractor " Key/Value Query ℱ!" & ℱ'() &*# ℱ # !" & Norm Cross-Attention FFN ℱ!" &*# ℱ!" & ℱ # '() & -
4QBUJBM1SJPS.PEVMF 41. ύονຒΊࠐΈͱผʹ ೖྗը૾͔ΒہॴతͳۭؒใΛநग़ εςοϓɿ 4UFNɿDPOW NBYQPPM TUSJEFͷ
DPOWͰಛϚοϓͷղ૾Λ ஈ֊తʹॖখ ºDPOWͰ 7J5ͷຒΊࠐΈ࣍ݩͱἧ͑Δ ϚϧνεέʔϧಛΛฒͯτʔΫϯΛߏ -
4QBUJBM'FBUVSF*OKFDUPS ۭؒಛΛ 7J5ͷதؒදݱʹೖ͠ ؼೲόΠΞεΛಋೖ • ΫϩεΞςϯγϣϯΛ༻͍Δ͜ͱͰ λεΫʹ༗༻ͳۭؒใ͕ 7J5ʹΈࠐ·ΕΔ •
ࣄલֶश͞ΕͨಛΛཚ͞ͳ͍Α͏ 𝛾 ΛͰॳظԽ -
.VMUJ4DBMF'FBUVSF&YUSBDUPS 7J5ͷதؒදݱΛ༻͍ͯ ۭؒಛΛߋ৽ • େҬతͳίϯςΩετใ͕औΓࠐ·ΕΔ • %FGPSNBCMF"UUFOUJPO <9JB $713`>
Λར༻ • ''/ɿMJOFBSͱ EFQUIXJTFDPOWͰߏ -
ධՁɿମݕग़ XҟͳΔࣄલֶश PO$0$0 • ϚϧνϞʔμϧࣄલֶश 6OJ1FSDFJWFSW <-J $713`> ͞Εͨ 7J5ͷར༻ʹΑΓେ͖͘ੑೳվળ
• ࠷ྑͰ CPY"1 NBTL"1 ʢ͍ͣΕ࣌ͷ 405"ʣΛୡ -
ධՁɿྖҬׂ ࠷ઌख๏ͱͷൺֱ PO"%&, • 7J5"EBQUFSΛ༻͍ͨ #&J5 <8BOH $713`>͕࣌ͷ 405"Λୡ
.BTL'PSNFS <$IFOH $713`>ࠓͰڧ͍ -
ఆੑ݁Ռ PO"%&, ಛϚοϓͷࡉ෦දݱ͕վળ Segmentation Results Stride-4 Feature Map Stride-8
Feature Map Stride-16 Feature Map Stride-32 Feature Map ViT ViT-Adapter-B ViT-B ViT-Adapter-B ViT ViT-Adapter-B ViT-B ViT-Adapter-B ViT-B -
֤ίϯϙʔωϯτͷ༗ޮੑΛ֬ೝ "CMBUJPO4UVEZ Patch Embedding … Spatial Prior Module (b)
ViT-Adapter Block N Injector N Block 2 Extractor 2 Injector 2 Extractor 1 Block 1 Injector 1 Seg ... sition mbedding ement-wise ddition (c) Spatial Prior Module ℱ!" # Stem ℱ# ℱ$ ℱ% Extractor N (d) Spatial Feature Injector " Key/ Value ℱ!" & Query ℱ'() & Cross-Attention (e) Multi-Scale Feature Extractor " Key/Value Query ℱ!" & ℱ'() &*# ℱ # !" & Norm Cross-Attention FFN ℱ!" &*# ℱ!" & ℱ # '() & -
·ͱΊ • 7J5ࢹ֮తؼೲόΠΞεΛಋೖ͢ΔύϥμΠϜΛఏҊ • 7J5ͷ൚༻ੑͱڧྗͳࣄલֶशදݱΛ׆༻Ͱ͖Δ • 7J5ͷΞʔΩςΫνϟΛมߋ͠ͳ͍λεΫదԠϞδϡʔϧ܈Λઃܭ • 4QBUJBM1SJPS.PEVMFɿہॴۭؒಛͷநग़ •
'FBUVSF*OKFDUPS&YUSBDUPSɿۭؒಛͷೖɾେҬಛͷநग़ • Ճύϥϝʔλ͕গͳ͘ޮతͳసҠֶश͕Մೳ • $0$0ͱ "%&,Ͱ࣌ͷ࠷ઌੑೳΛୡ • ॴײɿ՝ఏىͱղܾͷํੑ͕దͰ͋Ε جຊతͳߏཁૉͷ Έ߹Θ͚ͤͩͰྑ͍Ξτϓοτ͕ग़ͤΔʢͦΕ͕͍͕͠ʣ