[輪講] Vision Transformer Adapter for Dense Predictions

Vision Transformerに視覚的帰納バイアスを注入することで密予測タスクへ効率的に適応させる手法である、ViT-Adapterについての輪講資料です。

論文：https://arxiv.org/abs/2205.08534

Transcript

1. <ྠߨ>
   7JTJPO
   5SBOTGPSNFS
   "EBQUFS GPS
   %FOTF
   1SFEJDUJPOT ;
   $IFO
   :
   %VBO
   8
   8BOH
   +
   )F
   5
   -V
   +
   %BJ
   :
   2JBP *$-3
    IUUQTBSYJWPSHBCT ൃදɿ/BPLJ
   ,BUP
2. ࿦จ֓ཁ 7JTJPO
   5SBOTGPSNFS
   7J5
   ʹࢹ֮తؼೲόΠΞεΛ஫ೖ͠ ີ༧ଌλεΫ΁ޮ཰తʹదԠͤ͞Δ 7J5"EBQUFS ΛఏҊ ଟ͘ͷίϯϖ্Ґղ๏Ͱར༻͞Ε͍ͯΔɿ • TU
   1MBDF
   4PMVUJPO
   GPS
   UIF
   UI
   -4704
   $IBMMFOHF
   7JEFP
   *OTUBODF
   4FHNFOUBUJPO • $IBNQJPO
   TPMVUJPO
   JO
   5SBDL
   
   %
   0DDVQBODZ
   1SFEJDUJPO
   PG
   UIF
   $713
   
   "VUPOPNPVT
   %SJWJOH
   $IBMMFOHF • $IBNQJPO
   TPMVUJPO
   JO
   UIF
   7JEFP
   4DFOF
   1BSTJOH
   JO
   UIF
   8JME
   $IBMMFOHF
   BU
   $713
    %*/0W <4JNÉPOJ
   >
   Ͱ΋ྖҬ෼ׂλεΫʹ࠾༻ʢޙड़ͷ 'FBUVSF
   *OKFDUPS
   ͸আ֎ʣ બఆཧ༝ɿ՝୊ఏى
   ղܾࡦͱ΋ʹཧʹద͓ͬͯΓษڧʹͳΔ
   ྖҬ෼ׂͰڧ͍ 
3. 7J5
   ͸ࢹ֮తؼೲόΠΞεͷऑ͍ઃܭͰ͋Γॊೈੑ͕ߴ͍ҰํͰ ྖҬ෼ׂ΍෺ମݕग़౳ͷີ༧ଌλεΫͷੑೳʹྼΔ໰୊͕͋Δ 7J5
   ͷ໰୊఺  1ZSBNJE
   7JTJPO
   5SBOTGPSNFS <8BOH
   *$$7`> 4XJO
   5SBOTGPSNFS <-JV
   *$$7`> ؼೲόΠΞε͕ऑ͍ ؼೲόΠΞε͕ڧ͍ 7J5 <%PTPWJUTLJZ
   *$-3`>
   ༧ଌΛՄೳʹ͢ΔͨΊͷԾఆ΍ࣄલ஌͔ࣝΒੜ͡ΔόΠΞε
4. 7J5"EBQUFS ࣄલֶश͞Εͨ 7J5
   ʹΞμϓλʔΛ෇Ճͯ͠ԼྲྀλεΫͰֶश • ϚϧνϞʔμϧσʔλͰࣄલֶश͞Εͨڧྗͳ 7J5
   Λ׆༻Ͱ͖Δ • ΞμϓλʔͰࢹ֮తؼೲόΠΞεΛ஫ೖ  Step1: Image Modality Pre-training Step2: Fine-tuning (a) Previous Paradigm Vision-Specific Transformer COCO ADE20K Det/Seg Vision-Specific Transformer Image Modality SL/SSL Step1: Multi-Modal Pre-training Step2: Fine-tuning with Adapter (b) Our Paradigm ViT Image, Video, Text, ... SL/SSL COCO ADE20K ViT Adapter Det/Seg
5. طଘͷ 7J5
   ׆༻ख๏ͱͷࠩ෼ ೖྗը૾৘ใΛ༻͍ͨ 7J5
   ͱͷ૒ํ޲ͷ৘ใ఻ୡΛ௨ͯ͠ ಛ௃ϐϥϛουΛऔಘ  ViT 1/32 up 4x 1/16 up 2x 1/8 1x 1/4 down 2x (a) Previous Method (Li et al., ViTDet) ViT Adapter (b) ViT-Adapter (ours) Image Image 1/32 1/16 1/8 1/4 using task prior using task prior & input 7J5
   ͸ύονຒΊࠐΈͷ࣌఺Ͱۭؒ৘ใ͕ଛͳΘΕ͍ͯͦ͏ͳͷͰ߹ཧతʹࢥ͑Δ
6. ΞʔΩςΫνϟશମ 7J5
   ʹ 4QBUJBM
   1SJPS
   .PEVMF
   'FBUVSF
   *OKFDUPS&YUSBDUPS
   Λ෇Ճ • ࠷ऴ &YUSBDUPS
   ͷϚϧνεέʔϧಛ௃Λ༧ଌʹར༻  Patch Embedding … Spatial Prior Module (a) Vision Transformer (ViT) (b) ViT-Adapter Block N Injector N Block 2 Extractor 2 Injector 2 Extractor 1 Block 1 Injector 1 Det Seg ... Position Embedding Element-wise Addition (c) Spatial Prior Module ℱ!" # Stem ℱ# ℱ$ ℱ% Extractor N (d) Spatial Feature Injector " Key/ Value ℱ!" & Query ℱ'() & Cross-Attention (e) Multi-Scale Feature Extractor " Key/Value Query ℱ!" & ℱ'() &*# ℱ # !" & Norm Cross-Attention FFN ℱ!" &*# ℱ!" & ℱ # '() &
7. 4QBUJBM
   1SJPS
   .PEVMF
   41. ύονຒΊࠐΈͱ͸ผʹ ೖྗը૾͔Βہॴతͳۭؒ৘ใΛநग़ εςοϓɿ  4UFNɿDPOW
   ૚
   NBYQPPM  TUSJEF
   
   ͷ DPOW
   Ͱಛ௃Ϛοϓͷղ૾౓Λ ஈ֊తʹॖখ  º
   DPOW
   Ͱ 7J5
   ͷຒΊࠐΈ࣍ݩͱἧ͑Δ  Ϛϧνεέʔϧͷಛ௃ྔΛऔಘ 
8. 4QBUJBM
   'FBUVSF
   *OKFDUPS ۭؒಛ௃Λ 7J5
   ͷதؒදݱʹ஫ೖ͠ ؼೲόΠΞεΛಋೖ  • ΫϩεΞςϯγϣϯΛ༻͍Δ͜ͱͰ λεΫʹ༗༻ͳۭؒ৘ใ͕ 7J5
   ʹ૊Έࠐ·ΕΔ • ࣄલֶश͞Εͨಛ௃Λཚ͞ͳ͍Α͏ 𝛾 ΛͰॳظԽ
9. .VMUJ4DBMF
   'FBUVSF
   &YUSBDUPS 7J5
   ͷதؒදݱΛ༻͍ͯ ۭؒಛ௃Λߋ৽ • େҬతͳίϯςΩετ৘ใ͕औΓࠐ·ΕΔ  • %FGPSNBCMF
   "UUFOUJPO <9JB
   $713`>
   Λར༻ • ''/ɿMJOFBS
   ͱ EFQUIXJTF
   DPOW
   Ͱߏ੒
10. ΞʔΩςΫνϟઃఆ 7J5
    ʹରͯ͠গͳ͍௥ՃύϥϝʔλͰಋೖՄೳ 
11. ධՁɿ෺ମݕग़ X
    .BTL
    3$//
    PO
    $0$0  طଘͷ 7J5
    ׆༻ख๏ ͷੑೳΛ྇կ
12. ධՁɿ෺ମݕग़ X
    ҟͳΔࣄલֶश PO
    $0$0 • ϚϧνϞʔμϧࣄલֶश 6OJ1FSDFJWFS
    W <-J
    $713`>
    ͞Εͨ 7J5
    ͷར༻ʹΑΓେ͖͘ੑೳվળ • ࠷ྑͰ CPY
    "1
    
    NBTL
    "1
     ʢ͍ͣΕ΋౰࣌ͷ 405"ʣΛୡ੒ 
13. ධՁɿྖҬ෼ׂ PO
    "%&,  طଘͷ 7J5
    ׆༻ख๏ ͷੑೳΛ྇կ ϚϧνϞʔμϧ ࣄલֶशʹΑΓվળ
14. ධՁɿྖҬ෼ׂ ࠷ઌ୺ख๏ͱͷൺֱ PO
    "%&, • 7J5"EBQUFS
    Λ༻͍ͨ #&J5 <8BOH
    $713`>
    ͕౰࣌ͷ 405"
    Λୡ੒ 
15. ධՁɿྖҬ෼ׂ ࠷ઌ୺ख๏ͱͷൺֱ PO
    "%&, • 7J5"EBQUFS
    Λ༻͍ͨ #&J5 <8BOH
    $713`>
    ͕౰࣌ͷ 405"
    Λୡ੒  .BTL'PSNFS <$IFOH
    $713`>
    ΋ࠓͰ΋ڧ͍
16. ఆੑ݁Ռ PO
    $0$0 ಛ௃Ϛοϓͷߴप೾੒෼͕େ͖͘
    ີͳ༧ଌʹد༩ 
17. ఆੑ݁Ռ PO
    "%&, ಛ௃Ϛοϓͷࡉ෦දݱ͕վળ  Segmentation Results Stride-4 Feature Map Stride-8 Feature Map Stride-16 Feature Map Stride-32 Feature Map ViT ViT-Adapter-B ViT-B ViT-Adapter-B ViT ViT-Adapter-B ViT-B ViT-Adapter-B ViT-B
18. ֤ίϯϙʔωϯτͷ༗ޮੑΛ֬ೝ "CMBUJPO
    4UVEZ  Patch Embedding … Spatial Prior Module (b) ViT-Adapter Block N Injector N Block 2 Extractor 2 Injector 2 Extractor 1 Block 1 Injector 1 Seg ... sition mbedding ement-wise ddition (c) Spatial Prior Module ℱ!" # Stem ℱ# ℱ$ ℱ% Extractor N (d) Spatial Feature Injector " Key/ Value ℱ!" & Query ℱ'() & Cross-Attention (e) Multi-Scale Feature Extractor " Key/Value Query ℱ!" & ℱ'() &*# ℱ # !" & Norm Cross-Attention FFN ℱ!" &*# ℱ!" & ℱ # '() &
19. ·ͱΊ • 7J5
    ΁ࢹ֮తؼೲόΠΞεΛಋೖ͢ΔύϥμΠϜΛఏҊ • 7J5
    ͷ൚༻ੑͱڧྗͳࣄલֶशදݱΛ׆༻Ͱ͖Δ • 7J5
    ͷΞʔΩςΫνϟΛมߋ͠ͳ͍λεΫదԠϞδϡʔϧ܈Λઃܭ • 4QBUJBM
    1SJPS
    .PEVMFɿہॴۭؒ৘ใͷऔಘ • 'FBUVSF
    *OKFDUPS&YUSBDUPSɿۭؒ৘ใͷ஫ೖɾେҬಛ௃ͷநग़ • ௥Ճύϥϝʔλ͕গͳ͘ޮ཰తͳసҠֶश͕Մೳ • $0$0
    ͱ "%&,
    Ͱ౰࣌ͷ࠷ઌ୺ੑೳΛୡ੒ • ॴײɿ՝୊ఏىͱղܾͷํ޲ੑ͕ద੾Ͱ͋Ε͹
    جຊతͳߏ੒ཁૉ ͷ૊Έ߹Θ͚ͤͩͰྑ͍Ξ΢τϓοτ͕ग़ͤΔʢͦΕ͕೉͍͕͠ʣ