Share
Embed
SNLP2022: Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Reina Akama
September 20, 2022
Transcript
26 slides
-
Documenting Large Webtext Corpora: A Case Study on the Colossal
Clean Crawled Corpus Jesse Dodge et al., EMNLP2021 ಡΈख: ؒ ྯಸ (౦େ) ୈ14ճ࠷ઌNLPษڧձ @͓େ -
େنݴޠϞσϧͱࣄલֶशίʔύε ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 2 • BERT [Devlin+2019] → RoBERTa
[Liu+2019] • BookCorpus [Zhu+2015], English-language Wikipedia • → CommonCrawl News dataset [Nagel 2016], OpenWebtext [Gokaslan+2019], Stories [Trinh+2018] • GPT-3 [Brown+2020] • CommonCrawl (60%), WebText2 (22%) [Kaplan+2020], Books1 (8%) and Books2 (8%) [Brown+2020], English-language Wikipedia (3%) • T5 [Raffel+2019] • Colossal Clean Crawled Corpus (C4; filtered CommonCrawl) [Raffel+2019] • Switch Transformer [Fedus+2021] • Colossal Clean Crawled Corpus (C4; filtered CommonCrawl) [Raffel+2019] -
େنݴޠϞσϧͱࣄલֶशίʔύε ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 3 • BERT [Devlin+2019] → RoBERTa
[Liu+2019] • BookCorpus [Zhu+2015], English-language Wikipedia • → CommonCrawl News dataset [Nagel 2016], OpenWebtext [Gokaslan+2019], Stories [Trinh+2018] • GPT-3 [Brown+2020] • CommonCrawl (60%), WebText2 (22%) [Kaplan+2020], Books1 (8%) and Books2 (8%) [Brown+2020], English-language Wikipedia (3%) • T5 [Raffel+2019] • Colossal Clean Crawled Corpus (C4; filtered CommonCrawl) [Raffel+2019] • Switch Transformer [Fedus+2021] • Colossal Clean Crawled Corpus (C4; filtered CommonCrawl) [Raffel+2019] -
English Colossal Clean Crawled Corpus (C4) ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 4
• Common Crawl (2019-04) ΛϑΟϧλϦϯάɺ Reasonably clean and natural ͳେنӳޠςΩετσʔλ • 5จҎ্͔ΒͳΔจॻɺ3୯ޠҎ্͔ΒͳΔจͷΈอ࣋ • ඌʹ۟ಡʢϐϦΦυɺײ୰ූɺٙූɺҾ༻ූʣ͕͋ΔߦͷΈอ࣋ • langdetect (https://pypi.org/project/langdetect/) Ͱ 0.99 Ҏ্ͷ֬Ͱӳޠʹྨ͞ΕͨจॻͷΈอ࣋ • “List of Dirty, Naughty, Obscene, or Otherwise Bad Words” (https://git.io/vSyEu) ͷ୯ޠΛؚΉจॻΛআڈ • `Javascript` ͱ͍͏୯ޠؚ͕·ΕΔߦɺதׅހ `{` ͕ొ͢ΔϖʔδΛআ • 3จઅҎ্ͷॏෳ1ͭΛͯ͠ଞΛআ [Raffel+ JMLR2020] -
จͷओுɾಘΒΕΔݟ ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 5 • ࣄલֶशίʔύεʹ͍ͭͯΒͳ͍ͷ·͍ͣ • ࣄલֶशίʔύε͕γεςϜʹ༩͑ΔӨڹΛΔ͜ͱ͕Ͱ͖ͳ͍ •
ԼྲྀλεΫʹόΠΞε͕ೖ͞ΕΔՄೳੑ • ͳͥΒͳ͍ʁɿେن͗ͯ͢ମܥతͳੳ͕ࠔ • Ұపఈతʹੳͯ͠ɺυΩϡϝϯτԽ͢Ε͍͍ʢ͜ͷจʣ • Post-hoc ͳੳΛ͍͢͠Α͏ͳڥͷඋॏཁ • ຊจͷߩݙ • ࣄલֶशίʔύε C4 ͷఆྔతੳɾৄࡉͳυΩϡϝϯτͷఏڙ • ఆྔԽͷํ๏ɺੳͷํ๏Λఏࣔ • σʔληοτެ։ʹؔ͢Δఏݴ -
3ͭͷϨϕϧͰͷυΩϡϝϯτԽΛఏএ ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 6 σʔλͷग़ॲɺछྨɺ࣌ظɺҬ… ᶃ ͲͷΑ͏ͳσʔλ͔ʁ ᶄ Կؚ͕·Ε͍ͯΔ͔ʁ
ᶅ Կؚ͕·Ε͍ͯͳ͍͔ʁ ػց͕ੜͨ͠σʔλɺϕϯνϚʔΫσʔλ… τϐοΫɺΞΠσϯςΟςΟ… -
Metadata ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 7 σʔλͷग़ॲɺछྨɺ࣌ظɺҬ… ᶃ ͲͷΑ͏ͳσʔλ͔ʁ ᶄ Կؚ͕·Ε͍ͯΔ͔ʁ
ᶅ Կؚ͕·Ε͍ͯͳ͍͔ʁ ػց͕ੜͨ͠σʔλɺϕϯνϚʔΫσʔλ… τϐοΫɺΞΠσϯςΟςΟ… -
Metadata − Internet domains ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 8 • ௐࠪํ๏
• Internet Archive Λࢀর͢Δ͜ͱʹΑΓ จॻͱϝλσʔλ (ɺҐஔใ) Λඥ͚ • Χϯτͯ͠Λัଊ • ݟ • .com, .org, .net ͕ଟ͍ • (ӳޠͷςΩετΛऩू͍ͯ͠Δͷʹ) ӳޠҎ֎Λ ओݴޠͱ͢ΔࠃɾҬͷυϝΠϯ্Ґ25Ґʹଘࡏ • ্Ґ25Ґʹೖ͍ͬͯͳ͍͕ɺถࠃͷ܉ࣄ༻ υϝΠϯ .mil (33M tokens) ɺӳࠃͷ܉ࣄ༻ɾࠃল υϝΠϯ .mod.uk (1M tokens) ଟ͍ -
Metadata − Websites ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 9 • ௐࠪํ๏ •
Internet Archive Λࢀর͢Δ͜ͱʹΑΓ จॻͱϝλσʔλ (, Ґஔใ) Λඥ͚ • Χϯτͯ͠Λัଊ • ݟ • ಛڐจॻ͕େྔʹؚ·Ε͍ͯΔ • patents.google.com (1Ґ), patents.com (10Ґ) • Wikipedia, χϡʔεɺݴޠϞσϧͷࣄલֶशʹ Α͘༻͍ΒΕΔσʔλଟؚ͘·Ε͍ͯΔ • NYTimes, LATimes, AlJazeera • ॻ੶ྨଟ͍ • Plos, FrontiersIn, Springer -
Metadata − Utterance Date ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 10 • ௐࠪํ๏
• Internet Archive Λࢀর͢Δ͜ͱʹΑΓ จॻͱϝλσʔλ (, Ґஔใ) Λඥ͚ • Χϯτͯ͠Λัଊ • ݟ • C4ͷ92%͕աڈ10ؒ (2011ʙ2019) ʹ ॻ͔Εͨͷͱਪఆ • σʔλϩϯάςʔϧɺσʔλऩूͷ 10ʙ20લʹॻ͔Εͨσʔλগͳ͔Βͣଘࡏ C4͔Βແ࡞ҝʹநग़ͨ͠1,000,000ͷURLʹ͍ͭͯ ࠷ॳʹΠϯσοΫε͞Εͨͷ -
Included data ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 11 σʔλͷग़ॲɺछྨɺ࣌ظɺҬ… ᶃ ͲͷΑ͏ͳσʔλ͔ʁ ᶄ
Կؚ͕·Ε͍ͯΔ͔ʁ ᶅ Կؚ͕·Ε͍ͯͳ͍͔ʁ ػց͕ੜͨ͠σʔλɺϕϯνϚʔΫσʔλ… τϐοΫɺΞΠσϯςΟςΟ… -
Included data − Machine-generated text ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 12 •
ௐࠪํ๏ • ඥ͚ͨϝλσʔλΛར༻ • ಛڐจॻ = ػց͕ੜͨ͠จॻ (ӳޠͷػց༁) ͷ্ݶͱΈͳ͢ - ࠃ͝ͱʹಛڐΛॻͨ͘Ίͷݴޠ͕ఆΊΒΕ͍ͯΔ - patents.google.com ɺ֤ࠃಛڐிͷಛڐΛ ػց༁Ͱӳޠʹ༁͍ͯ͠Δ • ݟ • ಛڐจॻͷେถࠃಛڐிͷͷͰ͋Δ͕ɺ 10%Ҏ্ӳޠҎ֎ͷݴޠͰॻ͘͜ͱΛఆΊ͍ͯΔ ಛڐிͷͷɻ͜ΕΒͷจॻɺػցʹΑͬͯ ੜ͞ΕͨՄೳੑ͕ߴ͍ɻ • Σϒ্ͷػցੜจॻɺ࣌ؒͱͱʹ૿Ճ͢Δ ͜ͱ͕༧͞ΕΔ ಛڐ -
Included data − Benchmark data contamination ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 13
• ௐࠪํ๏ • ࣄલֶशσʔλͱϕϯνϚʔΫྫͷؒͷ શҰகΛଌఆɻେจࣈͱ۟ಡਖ਼نԽɻ • n-gramॏෳ (8<=n<=13) [Brown+2020] • ݟ • 2%ʙ25% ͷҰகΛ؍ • Ұகɺ୯จͷλʔήοτΛؚΉ σʔληοτ (Xsum, TIFU-short, AMR-to-text) ͕ɺ ෳจͷλʔήοτΛؚΉσʔληοτ (TIFU-long, WikiBio) ΑΓߴ͍ • LAMA Λ༻͍ͨ zero-shot ͷੑೳධՁʹ ҙ͕ඞཁʢطʹֶशࡁͷՄೳੑʣ ࣗવݴޠจੜλεΫͷλʔήοττʔΫϯؚ͕·Ε͍ͯΔ͔ʁ -
Included data − Benchmark data contamination ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 14
• ௐࠪํ๏ • ࣄલֶशσʔλͱϕϯνϚʔΫྫͷؒͷ શҰகΛଌఆɻେจࣈͱ۟ಡਖ਼نԽɻ • n-gramॏෳ (8<=n<=13) [Brown+2020] • ݟ • 2%ʙ50% ͷҰகΛ؍ • ͬͱҰகͷߴ͍ QNLI wikipedia ͔Β ߏங͞Εͨσʔλ • GLEU Λ༻͍ͨ few-shot ͷੑೳධՁʹ ҙ͕ඞཁʢطʹֶशࡁͷՄೳੑʣ GLEU [Wang+2019] ͱҰக͢ΔτʔΫϯؚ͕·Ε͍ͯΔ͔ʁ -
ͳͥҰகʁ ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 15 • Ծઆ1: Σϒ্ͷςΩετΛݩʹߏங͔ͨ͠Β (ݩ͕ಉ͔ͩ͡Β) Ұகͨ͠
• QNLI wikipedia σʔλΛݩʹߏங͞Εͨͷ • Ծઆ2: σʔλ࡞ޙʹΣϒʹΞοϓϩʔυͨ͠ͷ͕Ϋϩʔϧ͞Εͨ • ෳͷݚڀऀ͕ GLEU Λ github ϨϙδτϦͰެ։͍ͯ͠Δ -
ͳͥҰகʁ ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 16 • Ծઆ1: Σϒ্ͷςΩετΛݩʹߏங͔ͨ͠Β (ݩ͕ಉ͔ͩ͡Β) Ұகͨ͠
• QNLI wikipedia σʔλΛݩʹߏங͞Εͨͷ • Ծઆ2: σʔλ࡞ޙʹΣϒʹΞοϓϩʔυͨ͠ͷ͕Ϋϩʔϧ͞Εͨ • ෳͷݚڀऀ͕ GLEU Λ github ϨϙδτϦͰެ։͍ͯ͠Δ GLEU Λެ։͍ͯ͠ΔϨϙδτϦʹؚ·ΕΔϑϨʔζ͕ CommonCrawl ʹؚ·ΕΔ͔Λ֬ೝ͠ɺ ͦͷϖʔδ͕σʔλͱؚͯ͠·Ε͍ͯΔ͔ΛਓखͰνΣοΫ ˠ ֘ͳ͠ ✘ -
Excluded data ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 17 σʔλͷग़ॲɺछྨɺ࣌ظɺҬ… ᶃ ͲͷΑ͏ͳσʔλ͔ʁ ᶄ
Կؚ͕·Ε͍ͯΔ͔ʁ ᶅ Կؚ͕·Ε͍ͯͳ͍͔ʁ ػց͕ੜͨ͠σʔλɺϕϯνϚʔΫσʔλ… τϐοΫɺΞΠσϯςΟςΟ… -
ϒϩοΫϦετʹΑΔϑΟϧλॲཧ ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 18 • Common Crawl (2019-04) ΛϑΟϧλϦϯάɺ
Reasonably clean and natural ͳେنӳޠςΩετσʔλ • 5จҎ্͔ΒͳΔจॻɺ3୯ޠҎ্͔ΒͳΔจͷΈอ࣋ • ඌʹ۟ಡʢϐϦΦυɺײ୰ූɺٙූɺҾ༻ූʣ͕͋ΔߦͷΈอ࣋ • langdetect (https://pypi.org/project/langdetect/) Ͱ 0.99 Ҏ্ͷ֬Ͱӳޠʹྨ͞ΕͨจॻͷΈอ࣋ • “List of Dirty, Naughty, Obscene, or Otherwise Bad Words” (https://git.io/vSyEu) ͷ୯ޠΛؚΉจॻΛআڈ • `Javascript` ͱ͍͏୯ޠؚ͕·ΕΔߦɺதׅހ `{` ͕ొ͢ΔϖʔδΛআ • 3จઅҎ্ͷॏෳ1ͭΛͯ͠ଞΛআ -
Excluded data − Topic ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 19 • ௐࠪํ๏
• ϒϩοΫϦετʹΑͬͯ আڈ͞Εͨจॻ͔Βແ࡞ҝʹ 100,000จॻΛநग़ • TF-IDFͰ֫ಘͨ͠ຒΊࠐΈදݱΛ k-means Ͱ 50Ϋϥελʹྨ • PCA ͰՄࢹԽ • ݟ • ϙϧϊ(31%)ϔΠτεϐʔνܥ ͷΫϥελ͕ଟ͍ • Պֶɾҩྍɾ݈߁ɾήʔϜɾ ༮ࣇڭҭͷແͳΫϥελ ؍ ҩྍ -
Excluded data − Topic ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 20 • ௐࠪํ๏
• ϒϩοΫϦετʹΑͬͯ আڈ͞Εͨจॻ͔Βແ࡞ҝʹ 100,000จॻΛநग़ • TF-IDFͰ֫ಘͨ͠ຒΊࠐΈදݱΛ k-means Ͱ 50Ϋϥελʹྨ • PCA ͰՄࢹԽ • ݟ • ϙϧϊ(31%)ϔΠτεϐʔνܥ ͷΫϥελ͕ଟ͍ • Պֶɾҩྍɾ݈߁ɾήʔϜɾ ༮ࣇڭҭͷແͳΫϥελ ؍ ༮ࣇڭҭ ֶߍɾϏδωε -
Excluded data − Identity ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 21 • ௐࠪํ๏
• Identity ʹݴٴ͍ͯ͠ΔจॻΛ ਖ਼نදݱͰநग़ • ͜ΕΒͷจॻʹ͍ͭͯɺ ϒϩοΫϦετʹΑΓআڈ͞Εͨ จॻͱͷPMIΛଌఆ • ݟ • LGBTQ+ ͷ identity Λٞ͢Δจॻ͕ ಛʹภͬͯআڈ͞ΕΔՄೳੑ • ਓछɾຽΑΓॏେͳϦεΫ -
Excluded data − Identity ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 22 • ௐࠪํ๏
• Identity ʹݴٴ͍ͯ͠ΔจॻΛ ਖ਼نදݱͰநग़ • ͜ΕΒͷจॻʹ͍ͭͯɺ ϒϩοΫϦετʹΑΓআڈ͞Εͨ จॻͱͷPMIΛଌఆ • ݟ • LGBTQ+ ͷ identity Λٞ͢Δจॻ͕ ಛʹภͬͯআڈ͞ΕΔՄೳੑ • ਓछɾຽΑΓॏେͳϦεΫ • ແ࡞ҝʹબͨ͠50จॻΛਓखධՁ • ͏ͪɺnon-offensive: 22%, non-sexual: 36% -
·ͱΊ ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 23 • ࣄલֶशίʔύε C4 ͷఆྔతੳɾݟͷఏڙ •
ಛڐɾχϡʔεɾwikipediaυϝΠϯ͕ଟؚ͘·ΕΔ • ͜͜10ͷσʔλ͕΄ͱΜͲɺ10Ҏ্લͷσʔλؚ·ΕΔ • ػց͕ੜͨ͠ςΩετɺϕϯνϚʔΫσʔλͷίϯλϛ • (ࣾձతόΠΞεͷଘࡏ) • ϚΠϊϦςΟͷ͕ෆదʹআ͔Ε͍ͯΔՄೳੑ • σʔληοτެ։ʹؔ͢Δఏݴ • Reporting website metadataʢΣϒαΠτͷϝλσʔλΛใࠂ͢Δ͜ͱʣ • Examining benchmark contaminationʢϕϯνϚʔΫσʔλͷίϯλϛΛௐࠪ͢Δ͜ͱʣ • Social biases and representational harmsʢࣾձతόΠΞεͱฐΛೝࣝ͢Δ͜ͱʣ • Excluded voices and identitiesʢআ֎͞ΕͨͱΞΠσϯςΟςΟΛೝࣝ͢Δ͜ͱʣ Meta Meta Included Included Excluded -
ॻࢽใ ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 24 • ஶऀʹΑΔิʁ → Ͱͳ͍ C4
Λެ։ (JMLR 2020; 2020/06) C4 Λௐࠪ (EMNLP 2021; 2021/11) -
طଘίʔύεʹର͢Δ post-hoc study ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 25 • RealToxicityPrompts: Evaluating
Neural Toxic Degeneration in Language Models [Gehman+EMNLP2020] • OpenWebText [Gokaslan+2019] ʹؚ·ΕΔ༗ੑͷ͋ΔίϯςϯπϑΣΠΫχϡʔεΛௐࠪ • What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus [Luccioni+ACL2021] • Common Crawl ʹؚ·ΕΔ·͘͠ͳ͍ίϯςϯπΛௐࠪɺϔΠτεϐʔνਓ͚ίϯςϯπͷଘࡏΛࢦఠ • Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets [Kreutzer+TACL2022] • ΣϒΫϩʔϧͰ࡞͞Εͨ5ͭͷଟݴޠίʔύε [El-Kishky+2020; Xue+2021; Ortiz Suárez+ 2020; Bañón+2020; Schwenk+2019] ʹ͍ͭͯɺݴޠϥϕϧͷద͞ɾςΩετͷ࣭ΛਓखͰஅ ࣄલֶशίʔύεͷମܥతੳҰൠʹࠔɺΏ͑ʹৄࡉͳυΩϡϝϯτܽམ [Bender+2021; Paullada+2020] ࣄલֶशίʔύε͕γεςϜʹ༩͑ΔӨڹΛΔ͜ͱ͕Ͱ͖ͳ͍ ԼྲྀλεΫʹόΠΞε͕ೖ͞ΕΔՄೳੑ [Li+2020; Gehman+2020; Groenwold+2020] ཱͭ -
ॴײ ୈ14ճ࠷ઌNLPษڧձ | 2022-09-27 26 • ϒϥοΫϘοΫεͷऔΓΈʹߩݙ͢Δॏཁͳ؍ • Ϟσϧͷੳʢe.g., interpretation,
probingʣ • σʔλͷੳ • ੳ؍ͷબཧ༝ʁ • ཏతͳௐࠪݱ࣮తͰͳ͍ɺ·ͣԡ͑͞Δ͖؍ͱͦͷࠜڌ • ஶऀʮ͜ͷαΠζͷσʔληοτͰى͜Γ͏ΔͷҰ෦͔͠ௐ͍ࠪͯ͠ͳ͍͜ͱΛೝ͍ࣝͯ͠Δʯ • ஶऀʮݸਓΛಛఆͰ͖Δใஶ࡞ݖͷ͋ΔςΩετଘࡏ͢ΔͩΖ͏ɺ͜ΕΒͷఆྔԽআࠓޙͷ࡞ۀʹҕͶΔʯ