Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AI application systems realized by HyperCLOVA, ...

AI application systems realized by HyperCLOVA, a large-scale general-purpose language model for Japanese and its challenges

日本語の大規模汎用言語モデル「HyperCLOVA」が実現できるAI応用システムとその課題
佐藤 敏紀(LINE株式会社 AI開発室 NLP開発チーム マネージャー)

CCSE2021での発表資料です
https://ccse.jp/2021/

LINE Developers

December 17, 2021
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. ࠤ౻ හل (@overlast) • γχΞιϑτ΢ΣΞΤϯδχΞ / Ϛωʔδϟʔ • ࣗવݴޠॲཧ •

    ৘ใݕࡧ • LINE CLOVA • ೔ຊޠͷࣗવݴޠཧղ(NLU)γεςϜ • HyperCLOVA • ೔ຊޠ൛ͷ։ൃ੹೚ऀ • ೔ຊޠίʔύε / AIϑΟϧλʔ / Ԡ༻ٕज़ͷ։ൃɾධՁ • OSS: NEologd ϓϩδΣΫτͷϝΠϯ։ൃऀ • mecab-ipadic-NEologd 2
  2. Attention, please ! ࠷ॳʹ͓఻͍͑ͨ͠·͢ɻ - ͜ͷηογϣϯ͸20෼ͱ΍΍ۦ͚଍Ͱ͢ͷͰɺ͢͜͠ूதྗ͕ඞཁͰ͢ ! - Ͳ͏ͯ͠΋ࣗવݴޠॲཧͷઐ໳༻ޠ͕ଟ͘ͳΓ·͕͢ɺ෼͔Βͳ͍༻ޠ͸ 


    Θ͔Γқ͘ղઆͰ͖·͢ͷͰɺTwitterͰ @overlast ʹmention͍ͯͩ͘͠͞ - େن໛൚༻ݴޠϞσϧ͕ ”ະདྷΛͲ͏ม͑Δ͔” ༧૝͠ͳ͕Β͓ฉ͖͍ͩ͘͞
  3. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ -

    ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
  4. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ -

    ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
  5. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ -

    ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
  6. Title 80pt - ༧ଌ͍ͨ͠ςΩετͷτʔΫϯ਺NͱɺίʔύεΛଌఆͯ͠ಘͨط஌ͷτʔΫϯྻछ਺Vͷͱ͖ - ςΩετͷ֬཰෼෍ΛٻΊΔܭࢉ࣌ʹอ࣋͢Δύϥϝλ਺͕VͷN৐ʹͳΔ => ίεύ͕ѱ͍ ֬཰తݴޠϞσϧͷ࣍ݩͷढ͍ χϡʔϥϧݴޠϞσϧ

    - LSTM΍Transformer͕جʹͳ͍ͬͯͯѥछ͕ࢁఔ͋ΔɻBERT΍HyperCLOVA͸Transformerͷ೿ੜ - ಛ௃ϕΫτϧͷ࣍ݩ਺N(e.g. GPT3͸12288) x ޠኮ਺VͷߦྻΛอ࣋͢ΔͷͰVͷnഒ => ίεύ͕ྑ͍ - ֬཰తݴޠϞσϧͷ༷ʹ֬཰෼෍ͷ௿͍ܥྻͷ౷ܭ஋Λཅʹ࣋ͨͳͯ͘ࡁΉ ֬཰తݴޠϞσϧ ݴޠϞσϧ͸֬཰తͱχϡʔϥϧͷ2λΠϓʹ෼͔ΕΔ - ྫ: ୯ޠn-gramϞσϧ = ίʔύεதͷ୯ޠn-gramग़ݱ֬཰ʹج͖ͮɺ೚ҙͷςΩετͷ֬཰෼෍ΛਪఆͰ͖Δ - n-gram = ܥྻΛ୯Ґ௕ͷτʔΫϯྻʹ۠੾Δ͜ͱͰಘΒΕΔɺ͢΂ͯͷ௕͞nͷτʔΫϯྻ - ྫ: ݴޠϞσϧͷจࣈ1-gram: ݴ/ޠ/Ϟ/σ/ϧɺ ݴޠϞσϧͷจࣈ2-gram: ݴޠ/ޠϞ/Ϟσ/σϧ
  7. Title 80pt - ༧ଌ͍ͨ͠ςΩετͷτʔΫϯ਺NͱɺίʔύεΛଌఆͯ͠ಘͨط஌ͷτʔΫϯྻछ਺Vͷͱ͖ - ςΩετͷ֬཰෼෍ΛٻΊΔܭࢉ࣌ʹอ࣋͢Δύϥϝλ਺͕VͷN৐ʹͳΔ => ίεύ͕ѱ͍ ֬཰తݴޠϞσϧͷ࣍ݩͷढ͍ χϡʔϥϧݴޠϞσϧ

    - LSTM΍Transformer͕جʹͳ͍ͬͯͯѥछ͕ࢁఔ͋ΔɻBERT΍HyperCLOVA͸Transformerͷ೿ੜ - ಛ௃ϕΫτϧͷ࣍ݩ਺N(e.g. GPT3͸12288) x ޠኮ਺VͷߦྻΛอ࣋͢ΔͷͰVͷnഒ => ίεύ͕ྑ͍ - ֬཰తݴޠϞσϧͷ༷ʹ֬཰෼෍ͷ௿͍ܥྻͷ౷ܭ஋Λཅʹ࣋ͨͳͯ͘ࡁΉ ֬཰తݴޠϞσϧ HyperCLOVA͸χϡʔϥϧݴޠϞσϧΛ಺แ - ྫ: ୯ޠn-gramϞσϧ = ίʔύεதͷ୯ޠn-gramग़ݱ֬཰ʹج͖ͮɺ೚ҙͷςΩετͷ֬཰෼෍ΛਪఆͰ͖Δ - n-gram = ܥྻΛ୯Ґ௕ͷτʔΫϯྻʹ۠੾Δ͜ͱͰಘΒΕΔɺ͢΂ͯͷ௕͞nͷτʔΫϯྻ - ྫ: ݴޠϞσϧͷจࣈ1-gram: ݴ/ޠ/Ϟ/σ/ϧɺ ݴޠϞσϧͷจࣈ2-gram: ݴޠ/ޠϞ/Ϟσ/σϧ
  8. NLPͷٕज़τϨϯυͷมભ Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2

    
 (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) MT-NLG (2021)
  9. Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2 


    (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴ଎Խ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ
  10. Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2 


    (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴ଎Խ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧ෇σʔλͷେྔੜ੒ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ
  11. Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2 


    (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴ଎Խ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧ෇σʔλͷେྔੜ੒ ʻBreak Through 3ʼ Prompting : ̍ͭͷݴޠϞσϧͰ 
 ༷ʑͳλεΫΛॲཧͰ͖ΔՄೳੑ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ
  12. 3. Prompting ֶश ֶश Ϟσϧ ֶश σʔλ ਪ࿦ σʔλ ਪଌ

    σʔλ ֶश ࣄલֶश Ϟσϧ ֶश σʔλ ਪ࿦ σʔλ ਪଌ σʔλ ֶश ಛԽܕֶश Ϟσϧ λεΫ ಛԽֶश σʔλ ֶश େن໛൚༻ ݴޠϞσϧ ֶश σʔλ ਪ࿦ ਪଌ σʔλ Prompt ≠Program ैདྷ: ֶशϞσϧΛλεΫʹಛԽͤ͞Δ͜ͱͰੑೳΛ֬อ͖ͯͨ͠ ࠓޙ: ࠶ֶश΍ɺϓϩάϥϜͷ࣮૷Λͤͣʹ ໨తλεΫͷਪ࿦Λ࣮ࢪͰ͖Δ
  13. Seq2Seq(2014) • RNN, LSTM based Transformer(2017) GPT-1(2018) BERT(2018) GPT-2 


    (2019) GPT-3 
 (2020) BERTൃలܕ(2019) • RoBERTa(2019) • AlBERT(2019) • DistilBERT(2019) ʻBreak Through 1ʼ Attentionٕज़͕ൃలɺ ֶश͕ߴ଎Խ ʻBreak Through 2ʼ Generative pre-trainingʹΑΔ ϥϕϧ෇σʔλͷେྔੜ੒ ʻBreak Through 3ʼ Prompting : ̍ͭͷݴޠϞσϧͰ 
 ༷ʑͳλεΫΛॲཧͰ͖ΔՄೳੑ ʻBreak Through 4ʼ Scaling Lawͷൃݟ ੑೳ͸ύϥϝʔλ਺ɺσʔληοτ MT-NLG (2021) NLPͷٕज़τϨϯυͷมભ
  14. ڝ߹ଞ͕ࣾެද͍ͯ͠ΔϞσϧͷαΠζ OpenAI Google GPT: Generative Pre-trained Transformer T5: Text-To-Text Transfer

    Transformer MT-NLG: Megatron-Turing Natural Language Generation GPT-3(2020)ɿ1.3B, 2.7B, 6.7B, 13B, 175B GPT-2(2019)ɿ117M, 345M, 762M, 1.5B T5(2019)ɿ60M, 220M, 770M, 2.8B, 11B Microsoft MT-NLG(2021)ɿ530B
  15. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ -

    ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
  16. HyperCLOVA ͷ೔ຊޠϞσϧߏஙͷݱঢ় 1.3B → 6.7B → 13B → 39B 13B

    → 39B 82B 204B ʙ (2022೥த) ଟݴޠϞσϧ େن໛Ϟσϧ 
 ೔ຊޠ / ଟݴޠ ௒େن໛ 
 ೔ຊޠϞσϧ ೔ຊޠϞσϧ ߏங࡞ۀ͕ਐߦத 
  17. HyperCLOVAͷϞσϧߏஙͷख๏ EMNLP 2021 ͷNAVERͷ࿦จ*΋͝ࢀর͍ͩ͘͞ https://arxiv.org/abs/2109.04650 ࠷௿ͷֶश཰Λ Megatron-LMͷඪ४஋ͷ 1/10ʹ͢Δඞཁ͕͋ͬͨ NVIDIA Superpod

    ্Ͱֶश 
 Superpod͸Ԇ΂1024ຕͷA100 Λࢗͨ͠128୆ͷDGXΫϥελ ֶशʹ࢖͏ίʔύεͷτʔΫ ϯ਺͸ɺߏங͢ΔϞσϧͷύ ϥϝλ਺ͷ3ഒҎ্͕๬·͍͠ ByteϨϕϧͷBPE tokenizerʹΑΔίʔύε ࣄྫͷτʔΫϯԽ Transformer Decoder architecture Λ࠾༻ Megatron-LM Λ࢖༻ (Shoeybi et al., 2019)  https://arxiv.org/abs/2109.04650 * What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers, Boseop Kim et.al, EMNLP 2021
  18. HyperCLOVAͷϞσϧߏஙͷख๏ EMNLP 2021 ͷNAVERͷ࿦จ΋͝ࢀর͍ͩ͘͞ ࠷௿ͷֶश཰Λ Megatron-LMͷඪ४஋ͷ 1/10ʹ͢Δඞཁ͕͋ͬͨ NVIDIA Superpod ্Ͱֶश

    
 Superpod͸Ԇ΂1024ຕͷA100 Λࢗͨ͠128୆ͷDGXΫϥελ ֶशʹ࢖͏ίʔύεͷτʔΫ ϯ਺͸ɺߏங͢ΔϞσϧͷύ ϥϝλ਺ͷ3ഒҎ্͕๬·͍͠ ByteϨϕϧͷBPE tokenizerʹΑΔίʔύε ࣄྫͷτʔΫϯԽ Transformer Decoder architecture Λ࠾༻ Megatron-LM Λ࢖༻ (Shoeybi et al., 2019) 
  19. 

  20. LINE LM Corpus(for HyperCLOVA’s LMs) LINE ΍ LINE OpenChat ͷσʔλ͸ຊίʔύεʹ࠾࿥͠·ͤΜ

    - BERTͷϞσϧΛߏங͢ΔͨΊͷίʔύεͱͯ͠2019೥͔Β։ൃΛ։࢝ - LINE ݕࡧػೳͷͨΊʹΫϩʔϧ͞Εͨσʔλ΋ར༻ͨ͠ - "ඇެ։ͳݸਓ৘ใ"ͱͯ͠༰қʹநग़Ͱ͖ͨσʔλ͸আ֎͍ͯ͠Δ - ೔ຊޠදݱΛϞσϧʹؚΊΔ͏͑Ͱେ੾ͦ͏ͳαΠτΛඃ෴ͨ͠ - ͍͔ͭ͘ͷ֎෦ίϯςϯπΛߪೖͯ͠ɺݖར໰୊Λղܾͨ͠͏͑Ͱ࢖༻ ! 
  21. LINE LM Corpus for 39Bͷ಺༁ 82B Ҏ্ͷϞσϧߏஙʹ͍ͭͯ͸ɺߋʹ޿ൣͳൣғ͔ΒίϯςϯπΛऩू͍ͯ͠·͢ ղઆ τʔΫϯ਺ Blog

    ೔ຊޠͷϒϩά 105.8B News ೔ຊޠͷχϡʔεهࣄ 12.4B Q&A ೔ຊޠͷ࣭໰Ԡ౴αΠτ 10.5B WikiJa/En/Ko Wikipedia ͷ dump data 4.7B Novel ೔ຊޠͷখઆαΠτ 1.0B Shopping ೔ຊޠͷγϣοϐϯάαΠτ 20.9B Others ͦͷଞ 0.1B Total 155.4B 
  22. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ -

    ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
  23. 1. ਓ͕ؒͰ͖Δॲཧʹม׵ (ෳ਺ͷ)ೖྗςΩετ (ෳ਺ͷ)ग़ྗςΩετ #### (ෳ਺ͷ)ೖྗςΩετ (ෳ਺ͷ)ग़ྗςΩετ #### ɾɾɾ ͷ༷ʹShotͷ܁Γฦ͠Λॻ͘ɻ

    ྫ֎తͳ৘ใ͸ͳΔ΂্͘ͷํʹຒΊࠐΉ 2. ॲཧͷݟຊΛோΊΔ ʮಓ୺Ͱ୭͔ʹϓϩϯϓτΛݟͤ ͨޙͰɺϓϩϯϓτʹؚ·Ε͍ͯ ͳ͍(ෳ਺ͷ)ೖྗςΩετΛݟͤ ͨΒɺ(ෳ਺ͷ)ग़ྗςΩετΛ࡞ ΕΔՄೳੑ͕͋Δ͔ʯΛߟ͑Δɻ 
 
 Shot͕଍Γͳ͍͜ͱ͕໰୊ͳΒɺ ͞Βʹ଍͢͜ͱͰରॲͰ͖Δ 3. ΫΤϦΛ௥Ճ ϓϩϯϓτͷ຤ඌͷ####ͷ࣍ͷߦʹ ΫΤϦͱͳΔ(ෳ਺ͷ)ೖྗςΩετΛ ೖྗͨ͠Βɺ(ෳ਺ͷ)ग़ྗςΩετͷ ࠷ॳͷه߸෦෼Λ௥هͯ͠ɺ࣮ߦͯ͠ ΈΔ ग़ྗ͕ඍົͳΒύϥϝλͷௐ੔Λ͢Δ ବ໨ͳΒ1΍2ʹ΋ͲΔ ΍ͬͯΈ͍ͨ͜ͱ to ϓϩϯϓτܕͷࢦྩॻ
  24. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ -

    ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
  25. HyperCLOVAΛ༻͍ͨର࿩γεςϜͷओ؍ධՁ with 6.7B/13B/39B JP Model for 4 tasks ͢΂ͯͷλεΫͱϞσϧͷ૊Έ߹ΘͤͰΞϊςʔγϣϯΛ࣮ࢪ ಉ͡5ਓͷΞϊςʔλʔʹΑΔओ؍ධՁ

    ֤ηογϣϯ͸Mԟ෮ͷձ࿩ϖΞ Ϣʔβʔ͸ධՁ༻ͷNݸͷτϐοΫͷϦετΛड͚औΔ ֤ηογϣϯͰ͸ɼϦετ͔Β1ͭͷޠኮΛফඅ͢Δ Play-groundͰ࣮ࢪ 4. ࡶஊΛ͢Δ 3. τϐοΫ΁ͷϢʔβײ৘΁ͷରԠ 2. ҟͳΔτϐοΫ΁ͷભҠʹରԠ 1. جૅతͳޠኮͷཧղ 
  26. ྫ֎: ࡶஊͷλεΫ͸ΰʔϧͷୡ੒౓ΛධՁ͍ͯ͠ͳ͍(ෆཁ͔ͩΒ) ԁ׈ͳԠ౴ Qɿࣗવͳ൓ԠͰ͔ͨ͠ʁ ձ࿩ͷܦҢʹ୼ͼ΍ໃ६͸͋Γ·ͤΜ͔ʁ τϐοΫ΁ͷ௥ै Q: Did it stay

    on topic? τϐοΫΛݟࣦ͍ͬͯͳ͔͔ͬͨʢ͜ͷ৔߹ɺԿʹ͍ͭͯฉ͔Ε͍ͯΔͷ͔Λݟࣦ͍ͬͯͳ͍͔ʁ) ࿩୊ͷ੾Γସ͕͑Ͱ͖͔ͨʢ͜ͷ৔߹ɺલͷ࣭໰ʹҾ͖໭͔ͤͨʁ) τϐοΫͷఏڙ΍ 
 ࣭໰ͷ౤͔͚͛ Qɿ࿩୊Λఏڙ͔ͨ͠ʁ ճ౴தʹൃݴऀͷ࿩ΛҾ͖ग़͢͜ͱ͕Ͱ͖͔ͨʢͰ͖ͳ͍Մೳੑ͕ߴ͍ʣ ໨ඪͷୡ੒ Qɿ໨త͸ୡ੒Ͱ͖·͔ͨ͠ʁ શλεΫʹڞ௨ͳධՁ߲໨ ! 
  27. 1. جૅతͳޠኮͷཧղ ॳ౳ޠኮ த౳ޠኮ খֶߍ ۚͮͪ தֶߍ Ԗච େਓ νϡʔϦοϓ

    ઌੜ ώϚϫϦ ϥΠΦϯ ص ΩϦϯ Ҝࢠ ిं ۺ ं αϯμϧ ηʔλʔ ΓΜ͝ εΧʔτ Έ͔Μ Ωϟϕπ αϯϚ ͖Ύ͏Γ Ϛάϩ εζϝ ϋʔϞχΧ Πϯί ϐΞϊ τϯϘ ΞϦ IUUQTSFQPTJUPSZOJOKBMBDKQ BDUJPOSFQPTJUPSZ@BDUJPO@DPNNPO@EPXOMPBEJUFN@JEJUFN@OPBUUSJCVUF@JE fi MF@OP ToDO: ֤ޠኮʹରͯ͠ɺҙຯ(Ϩϕϧ1)ͱ૝ى͢Δײ৘(Ϩϕϧ2)ʹ͍࣭ͭͯ໰Λ͢Δ 
  28. Topic A Topic B Topic A Topic B ৽ܕίϩφ΢Πϧε Πϯό΢ϯυ

    Πνϩʔ େ୩ᠳฏ ۓٸࣄଶએݴ ৽ܕίϩφϫΫνϯ AR(֦ுݱ࣮) ࣗಈӡసٕज़ YouTuber VTuber ϨΦφϧυɾμɾϰΟϯν ΫϩʔυɾϞω ฏ੒ ྩ࿨ Πϯλʔωοτ 5G σϑϨܦࡁ ௒ߴྸԽࣾձ ւ֎ཱྀߦ ࠃ಺ཱྀߦ ిؾࣗಈं ϦχΞதԝ৽װઢ 2. ҟͳΔෳ਺ͷτϐοΫΛτϥοΩϯά ToDO: τϐοΫAͰձ࿩Λ࢝Ίɺ10ԟ෮͢ΔલʹτϐοΫBʹ੾Γସ͑Δ ฏ੒ 
  29. 3. ͋ΔτϐοΫʹର͢ΔϢʔβʔͷײ৘ʹ൓Ԡ Topic sentiment A sentiment B ৽ܕίϩφ΢Πϧε ؤுΖ͏ ෆ҆ͩ

    Πϯό΢ϯυ ໭ͬͯ͘Δ ໭Βͳ͍ ৽ܕίϩφϫΫνϯ ଴ͱ͏ ͍ͭʹͳΔ YouTuber ΍Γ͍ͨ ΍Γͨ͘ͳ͍ େ୩ᠳฏ ׆༂ͯ͠ཉ͍͠ ࡾৼͯ͠ཉ͍͠ AR(֦ுݱ࣮) ໘ന͍ ๞͖ͨ ௒ߴྸԽࣾձ େৎ෉ ৺഑ ւ֎ཱྀߦ ߦ͖͍ͨ ߦ͖ͨ͘ͳ͍ ిؾࣗಈं ৐Γ͍ͨ ৐Γͨ͘ͳ͍ ϦχΞதԝ৽װઢ ৐Γ͍ͨ ৐Γͨ͘ͳ͍ ToDOɿTopicʹ͍ͭͯ15ԟ෮ͷձ࿩Λ͢Δɻ࠷ॳͷ15ԟ෮͸ײ৘Aɺ࣍ʹײ৘Bͷؾ࣋ͪͰ࿩͢ 
  30. 39B JP Modelͷओ؍ධՁͷ݁Ռͷཁ໿ 1. جૅతͳޠኮͷཧղ 2. ҟͳΔτϐοΫ΁ ͷભҠʹରԠ 3. τϐοΫ΁ͷલ

    ޲͖ͳײ৘΁ͷରԠ 3. τϐοΫ΁ͷޙΖ ޲͖ͳײ৘΁ͷରԠ 4. ࡶஊΛ͢Δ ԁ׈ͳԠ౴ 0.978(356/364) 0.908(1749/1926) 0.908(1198/1320) 0.872(1072/1229) 0.925(86/93) τϐοΫ΁ͷ௥ै 0.984(358/364) 0.952(1834/1926) 0.951(1255/1320) 0.93(1144/1229) 0.935(87/93) τϐοΫͷఏڙ΍ 
 ࣭໰ͷ౤͔͚͛ 0.003(1/364) 0.023(44/1926) 0.033(44/1320) 0.035(43/1229) 0.086(8/93) ໨ඪͷୡ੒ 0.835(304/364) 0.907(1746/1926) 0.899(1187/1320) 0.505(621/1229) - 
  31. 39B JP Modelͷओ؍ධՁͷ݁Ռͷཁ໿ 1. جૅతͳޠኮͷཧղ 2. ҟͳΔτϐοΫ΁ ͷભҠʹରԠ 3. τϐοΫ΁ͷલ

    ޲͖ͳײ৘΁ͷରԠ 3. τϐοΫ΁ͷޙΖ ޲͖ͳײ৘΁ͷରԠ 4. ࡶஊΛ͢Δ ԁ׈ͳԠ౴ 0.978(356/364) 0.908(1749/1926) 0.908(1198/1320) 0.872(1072/1229) 0.925(86/93) τϐοΫ΁ͷ௥ै 0.984(358/364) 0.952(1834/1926) 0.951(1255/1320) 0.93(1144/1229) 0.935(87/93) τϐοΫͷఏڙ΍ 
 ࣭໰ͷ౤͔͚͛ 0.003(1/364) 0.023(44/1926) 0.033(44/1320) 0.035(43/1229) 0.086(8/93) ໨ඪͷୡ੒ 0.835(304/364) 0.907(1746/1926) 0.899(1187/1320) 0.505(621/1229) - 
  32. Application example: HyperCLOVA Friends Talk with any adjustable character using

    HyperCLOVA HyperCLOVA͸͋Δఔ౓ϩʔϧϓϨΠͰ͖Δ
  33. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ -

    ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
  34. RCQA possible only task ͷͨΊͷ 
 39B JPϞσϧʹΑΔࣗಈධՁ݁Ռ model /

    few-shot shot temperature top_p answer match 6.7B / contextual 0 0.5 0.8 - 4 0.1 0.9 66.52 13B / contextual 0 0.5 0.8 - 4 0.4 0.1 70.28 39B / contextual 0 0.4 0.5 80.51 1 0.4 0.5 89.18 2 0.4 0.5 89.31 3 0.4 0.5 89.09 4 0.4 0.5 89.83 39B / non-contextual 0 0.4 0.5 69.50 1 0.4 0.5 76.97 2 0.4 0.5 79.08 3 0.4 0.5 79.38 4 0.4 0.5 80.51 
  35. HyperCLOVA’s LM vs BERT-large TASK: RCQA possible only (௨ৗͷRCQAλεΫ͔Β౴͑ΒΕͳ͍࣭໰Λ࡟আ͠·ͨ͠) -

    ਅʹࠔ೉ͳ໰୊ΛHyperCLOVAͰղܾͰ͖ͳ͍Մೳੑ͕͋Γ·͢ - normal => possible only: train: 43610 => 21091, dev: 6863 => 3524, test: 6178 => 3079 - BERT͸ɺಛఆͷλεΫʹରͯ͠ඍௐ੔͢Δ͜ͱͰɺΑΓߴ͍݁ՌΛಘΒΕΔՄೳੑ͕͋Δ - HyperCLOVA͸PromptingͱϥϑͳύϥϝʔλݕࡧͰBERTͱಉ౳ͷύϑΥʔϚϯεΛ࣮ݱͰ͖Δ test acc test F1 memo HyperCLOVA 85.03 89.95 JP 39B 2-shots, 
 temperature=0.4, top_p= 0.5 BERT-jp-large 86.68 90.49 Using subset of LINE LM corpus 
  36. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ -

    ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ 
  37. ೔ຊޠͷ೉͠͞ शಘͷ೉͠͞ ೔ຊޠͷ࿩ऀ͸ - ͻΒ͕ͳ - ยԾ໊ - ׽ࣈ -

    ϩʔϚࣈ ͳͲͳͲΛ ҰͭͷจॻΛॻͨ͘Ίʹ࢖͏ େྔͷඞਢޠኮ ೔ৗձ࿩ʹඞཁͳޠኮ਺ - 8,000ޠҎ্ ଟ͘ͷ୯ޠΛ஌Δඞཁ͕͋Δ - ಉԻҟٛޠ - ܟޠ - ํݴͳͲ ୯ޠͷলུ ೔ຊޠ࿩ऀ͸ɺจॻதͷҎԼ ͷ୯ޠΛলུ͢Δ͜ͱ͕͋Δ - ओޠ - ໨తޠ লུ͞Εͨ୯ޠ͸Ұҙʹਪ࿦ Ͱ͖ͳ͍Մೳੑ͕͋Δ 
  38. ςΩετੜ੒ͷ೉͠͞ ੜ੒͞ΕͨςΩετͷ 
 જࡏతͳϦεΫ ҎԼͷΑ͏ͳٕज़Λ։ൃ ͢Δඞཁ͕͋Γ·͢ɻ - ίʔύεͷ಺༰ͷภΓ΍ දهํ๏ͷվળ -

    ग़ྗ͞ΕͨςΩετͷਅ ࣮ੑͱ҆શੑͷ֬อ AIྙཧͷ࣮ફ ೖग़ྗςΩετʹ͸༷ʑͳྙཧత ഑ྀ͕ඞཁͰ͢ - ಟੑ - ੑతͳ΋ͷ - ߈ܸతͳ΋ͷ - ๯ಚత - ڴഭత - ΞΠσϯςΟςΟ΁ͷ߈ܸ ຊ࣭తͳධՁͷࣗಈԽ ಈతςΩετੜ੒ͷ݁Ռʹద ༻Ͱ͖ΔϝτϦΫε͕ඞཁ - ࿩୊ੑͷ͋Δίϯςϯπͷਫ਼౓ - ੜ੒͞ΕͨςΩετͷҰ؏ੑ - ໨తͷୡ੒౓ͷ൑ఆ 
  39. LINEͷݴޠϞσϧΛOSSͱͯ͠ຊ೥౓தʹ഑෍։࢝ ΋ͪΖΜHyperCLOVAҎ֎Ͱ͢ ੑೳ໨ඪ: LINEͷݴޠϞσϧ for OSS > other OSS ͳݴޠϞσϧ

    ՄೳͰ͋Ε͹ɺෳ਺೥ؒ͸ఆظతʹߋ৽͍ͨ͠ͱߟ͍͑ͯ·͢ HyperCLOVAʹ࢖͍ͬͯΔίʔύε(LINE LM Corpus)ͷαϒηοτΛ࢖༻͠·͢ !
  40. Agenda - Πϯτϩ - NLPͷٕज़τϨϯυ - HyperCLOVAͷ֓ཁ - ঎඼આ໌จੜ੒γεςϜ΁ͷԠ༻ -

    ର࿩γεςϜ΁ͷԠ༻ - BERTͱͷൺֱ - ࣭໰Ԡ౴λεΫͰ - - ߴ඼࣭Ͱ҆શͳग़ྗΛಘΔͨΊͷ՝୊ - ·ͱΊ