Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TokyoCL-2016-10-05

penzant
October 05, 2016
200

 TokyoCL-2016-10-05

penzant

October 05, 2016
Tweet

Transcript

  1. Normative Analysis of Reading Comprehension Tasks + etc. Saku Sugawara

    (NII Aizawa-lab, Univ. Tokyo) TokyoCL #6 (NII) October 5, 2016
  2. ࠓ೔ͷ࿩ ϙΤϜ + ࣗ෼ͷݚڀ + म࿦ͷશମ૾ͷߏ૝ ࡶͳٞ࿦ ໨࣍ 1. ʮݴޠཧղʯʹ͍ͭͯ

    2. ʮಡղλεΫʯʹ͍ͭͯ 3. ʮݴޠཧղʹཁٻ͞ΕΔجૅతͳೳྗʯʹ͍ͭͯ 4. ʮఆٛͨ͠ೳྗʹΑΔಡղλεΫͷΞϊςʔγϣϯʯʹͭ ͍ͯ 5. ࠓޙ 2 / 80
  3. ͜ͷ section ͰͲͷΑ͏ͳ࿩Λ͢Δ͔ Reading comprehension ͱ͍͏λεΫܗࣜΛʢNLP ͷશମతͳ ໨తͷதʹʣͲͷΑ͏ʹҐஔ͚ͮΔ͔ 1. େ໨ඪ͸Կ͔

    2. ୹ظత໨ඪͱͯ͠ͲͷΑ͏ͳλεΫܗࣜΛఆΊΔ͔ 3. ͦͷλεΫͷ੍໿͸ԿͰ͋ΓɺͲͷΑ͏ͳൃలɾ֦ு͕ݟࠐ ΊΔ͔ 4 / 80
  4. େ໨ඪ: ҰํͰզʑ (ࢲ) ͸ԿΛ࣮ݱ͍͔ͨ͠ʁ ʮ஻ΕΔϩϘοτʯ ʁ ʮࣗ཯ͨ͠ΤʔδΣϯτʯ ʁ Ͳ͜ͰͲͷΑ͏ʹͲͷΑ͏ͳ໨తͰʁ Subsumption

    architecture? യવͱͨ͠໨తͰ͸ઃܭ͢Δਓ͕ࠔΔ ࠷ۙͷྲྀߦΓͰΑࣖ͘ʹ͢Δ໰୊఺ ͱʹ͔͘ʮίϛϡχέʔγϣϯ͕ͱΕΔ͜ͱʯ ʁ ʢ͜͜ʹϩϘοτͷը૾ʣ 6 / 80
  5. ݴޠߦҝΛݶఆ͢Δ ͍Ζ͍Ζͳݴޠߦҝ͕ఆٛͰ͖Δ ࣭໰ɺԠ౴ɺఏҊɺ໋ྩɺಠݴɺ http://plato.stanford.edu/entries/speech-acts/ ͜͜Ͱ͸Ԡ౴ʹݶఆ͢Δ ʢ༨ஊʣ Stanford Encyclopedia of Philosophy

    (SEP) ͸ݴޠֶɾݴޠ఩ ֶܥͷ༻ޠͷղઆ͕ཉ͍͠ͱ͖ʹ݁ߏศརͰ͢ SEP ͷ Computational Linguistics [url] ͷ߲͸ Lenhart Schubert [url] ͱ͍͏͓͡͞Μ͕ॻ͍͍ͯΔ Semantic Representation (AAAI2015) [pdf] What kinds of knowledge are required for genuine understanding? (IJCAI2015 WS) [pdf] ͍Ζ͍Ζ֓؍ΛಘΔͷʹྑ͍͔΋͠Εͳ͍Ͱ͢ 8 / 80
  6. ਓ͸ʮ͋ΒΏΔ՝୊͸˓˓ͱ͍͏ܗࣜʹؐݩͰ ͖Δʯͱݴ͏͜ͱ͕͋Δ͕ɺͦΕͰ͍͍ͷͩΖ ͏͔ͱ͍͏࿩ ʮRTE/QA/຋༁ ʹؐݩͰ͖Δʯ → ʮೖྗɾ಺෦ঢ়ଶɾग़ྗʯͰ͍͍ͱࢥ͏ → Input /

    State / Action ୯ͳΔؔ਺ͱ͸ݴ͑ͳ͍ɺঢ়ଶͷอଘ͕͋Δ ೖྗ͸ͳ͍͜ͱ͕͋Δɺ಺෦ঢ়ଶ͸࣌ؒతʹભҠ͢Δɺग़ྗ ͸ͳ͍͜ͱ͕͋Δ Context ͸ʮ஌֮Մೳͳڞ༗͞Εͨ৘ใʯ΍ʮ࣌ؒతɾۭؒత ʹͦͷ৔͔Β཭Εͨ఺Ͱ֫ಘ͞Εͨ৘ใʯΛؚΉɺ΋ͬͱݴ ͑͹ʮજࡏతʹ֫ಘͰ͖Δ৘ใʯ΋ؚΉ ͜͜Ͱ͸ਓؒͷݴޠ׆ಈʹ͍ͭͯड़΂͍ͯ·͢ 11 / 80
  7. ࣮༻తͳྫ Input: ࣭໰ʮ౦ژӺ͔ΒਆอொӺ·Ͱͷܦ࿏Λڭ͑ͯʯ Context: ܦ࿏ਤ΍࣌ࠁදͷσʔλɺܭࢉ Action: ฦ౴ʮ౦ژ͔Βؙϊ಺ઢͰେखொ΁ɺେखொ͔Β൒ଂ ໳ઢͰਆอொ΁ʯ Input: ࣭໰ʮ͜ͷը૾ʹࣸͬͯΔೣͷ඼छ͸Կʁʯ

    Context: ը૾ͷೝࣝɾॲཧ Action: ฦ౴ʮࡾໟೣʯ ඇ࣮༻తʹ͸ɺϢʔβʔ͕ҙਤ͠ͳ͍ೖྗΛ͢Δͱಥવ૽͗ ग़͢ϑΝʔϏʔਓܗͳͲ Pepper ͘ΜͬͯಠΓݴΛݴ͏ΜͰ͔͢Ͷ 12 / 80
  8. Կ͕ݴ͍͍͔ͨ ʮಡղͱ͍͏λεΫ͕ɺཧ૝తͳ໨ඪʹରͯ͠ɺͲͷΑ͏ͳ ੍໿Λ͔࣋ͭʯΛҙࣝ͢ΔͱɺΑΓൃలతͳܗ͕ࣜߟ͑΍͢ ͘ͳΔʁ ۠ผ͢ΔͨΊͷཁૉ͸;ͨͭ: ʮೖग़ྗɾ಺෦ঢ়ଶʢจ຺ʣ ʯ ʮ৘ใͷഔମɾͦͷ࣌ؒ/ۭؒత෯ɾ಺༰ʯ ೖग़ྗ: ςΩετͷΈɺ࣌ؒతɾྔతʹมԽ

    จ຺తঢ়ଶ: ςΩετͷΈɺ࣌ؒతɾྔతʹݻఆ ೖग़ྗ͸࣭໰Ԡ౴ʢͱ͍͏ݴޠߦҝʣͷΈ ࢀ༩ऀ͕ར༻͢Δจ຺͸ڞ༗͞Ε໌ࣔత ؾ࣋ͪ Media /modality Time/space Contents/type Input/Output text variable question answering State/Context text invariable shared/explicit 13 / 80
  9. Background: Big Data vs. Realistic ࣗಈతʹ࡞Δͱͨ͘͞Μσʔλ͕Ͱ͖Δ΋ͷͷɺ࣭͕ո͍͠ CNN/Daily Mail (Hermann+ 2015)

    SQuAD (Rajpurkar+ 2016) [url] ਓखͰσʔλΛ࡞Ζ͏ͱ͢ΔͱͲ͏ͯ͠΋খ͘͞ͳΔ MCTest (Richardson+ 2013) [url]: 660*4 questions ProcessBank (Berant+ 2014) [url]: 585 questions ਓखͰൺֱత࣭ͷྑ͍σʔλΛͨ͘͞Μ࡞ͬͨྫʁ LAMBADA (Paperno+ 2016) [url] dev+test Ͱ 10K passage (=question) ͘Β͍ 19 / 80
  10. CNN/Daily Mail Dataset (DeepMind QA Dataset) CNN ΍ Daily Mail

    ͷهࣄλΠτϧ΍ݟग़͕֘͠౰Օॴͷཁ໿ ʹͳ͍ͬͯΔ λΠτϧ΍ݟग़͠ͷ entity ෦෼Λ݀ʹͯͦ͠Ε͕Կ͔Λ౴͑ ͤ͞ΔλεΫ هࣄ͸ͨ͘͞Μ͋ΔͷͰͨ͘͞Μ࡞ΕΔ (context, query, answer) Ͱ 1 ୯Ґ هࣄ಺༰ʹ౴͕͑ग़ͯ͜ͳ͍Α͏ͳ΋ͷ͸࡞Βͳ͍ 20 / 80
  11. SQuAD (2016) Reading comprehension dataset 100,000+, from Wikipedia articles, by

    crowdworkers Human performance: 86.8% Baseline (Logistic Regression): 51.0% using features of dependency trees 23 / 80
  12. Example - SQuAD - dev set (v1.1) - cloze Super

    Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The Amer- ican Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24 10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California. 1. Which NFL team represented the AFC at Super Bowl 50? Denver Broncos (177) 2. Where did Super Bowl 50 take place? Santa Clara, California (403), Levi’s Stadium (355), Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California. (355) 3. Which NFL team won Super Bowl 50? Denver Broncos (177) 25 / 80
  13. MCTest (2013) Reading comprehension dataset Manually written texts by crowdworkers

    Contextual passages and multiple choice questions 660 tasks, and each task has 4 questions Human performance: better than 95% Baseline: 50-60% Bast score: around 70% 26 / 80
  14. Example - MCTest - multiple choice James the Turtle was

    always getting in trouble. One day, James went to the grocery store and pulled all the pudding off the shelves and ate two jars. Then he walked to the fast food restaurant and ordered 15 bags of fries. Q1: What is the name of the trouble making turtle? A1: James Q2: Where did James go after he went to the grocery store? A2: a fast food restaurant 27 / 80
  15. Example - MCTest What we have to understand to answer

    them...? “James the Turtle ...” paraphrase of the subject (James = the Turtle) “the Turtle was always getting in trouble” = “the trouble making turtle” “James” = “the name”: knowledge “The grocery store” and “the fast food restaurant” are places Temporal relation: James went to Place1. After that, he went to Place2. Pick the key information from the context sentences etc... 28 / 80
  16. Example - LAMBADA - cloze He shook his head, took

    a step back and held his hands up as he tried to smile without losing a cigarette. “Yes you can,” Julia said in a reassuring voice. “I’ve already focused on my friend. You just have to click the shutter, on top, here.” Query: He nodded sheepishly, through his cigarette away and took the ʻʼ. camera 30 / 80
  17. Big Data vs. Realistic: ࠶ܝ ࣗಈతʹ࡞Δͱͨ͘͞Μσʔλ͕Ͱ͖Δ΋ͷͷɺ࣭͕ո͍͠ CNN/Daily Mail (Hermann+ 2015)

    SQuAD (Rajpurkar+ 2016) [url] ਓखͰσʔλΛ࡞Ζ͏ͱ͢ΔͱͲ͏ͯ͠΋খ͘͞ͳΔ MCTest (Richardson+ 2013) [url]: 660*4 questions ProcessBank (Berant+ 2014) [url]: 585 questions ਓखͰൺֱత࣭ͷྑ͍σʔλΛͨ͘͞Μ࡞ͬͨྫʁ LAMBADA (Paperno+ 2016) [url] dev+test Ͱ 10K passage (=question) ͘Β͍ 31 / 80
  18. CNN/DailyMail in ACL2016 Chen+, A Thorough Examination of the CNN/Daily

    Mail Reading Comprehension Task CNN/Daily Mail ͸΄ͱΜͲ಄ଧͪ: σʔλʹϊΠζ͕ଟ͍ɺ ࣗಈͰ࡞Εͨͷ͸ྑ͍͚Ͳ࣭΋େࣄʢຊ౰ʹಡղతͳਪ࿦Λ ଌΔͷʹ໾ཱͭͷ͔ʁʣ ͜͏͍͏σʔληοτΛ൱ఆ͢Δඞཁ͸ͳ͘ɺΑΓ realistic ͳσʔληοτͷͨΊͷֶशσʔλͱͯ͠׆͔ͤΔ͸ͣ 32 / 80
  19. ʢ୤ઢʣ Simple questions ͕Α͍ͷ͔ʁ 1. Tom is a student. Q:

    Is Tom a student? A: Yes 2. Descartes was born in 1596. Q: When was Descartes born? A: 1596 (External knowledge is not necessary here) (These questions seem to be solved by simple rules) (They may not be useful to test more intelligent systems) → must be not too simple, but a little easier (and not difficult). 35 / 80
  20. Moderately easy??: bAbI tasks [Weston+ 2015] Task: Three Supporting Facts

    John picked up the apple. John went to the office. John went to the kitchen. John dropped the apple. Where was the apple before the kitchen? A: office ✓ Easy (for human) ✓ Multiple sentences ✓ Automatically generated × Small vocabulary × Simple and unnatural sentences 36 / 80
  21. bAbI tasks [Weston+ 2015] - Task categories Single, two, or

    three supporting facts Two or three argument relations Yes/no questions Counting and lists/sets Simple negation and indefinite knowledge Basic Coreference ,conjunctions and compound coreference Time reasoning Basic decuction and induction Positional and size reasoning Path finding Agent’s motivations 37 / 80
  22. bAbI tasks [Weston+ 2015] - more examples Task: Basic Coreference

    Daniel was in the kitchen. Then he went to the studio. Sandra was in the office. Where is Daniel? A:studio Task: Conpound Coreference Daniel and Sandra journeyed to the office. Then they went to the garden. Sandra and John travelled to the kitchen. After that they moved to the hallway. Where is Daniel? A: garden 38 / 80
  23. bAbI tasks [Weston+ 2015] - more examples Task: Time Reasoning

    In the afternoon Julie went to the park. Yesterday Julie was at school. Julie went to the cinema this evening. Where did Julie go after the park? A:cinema Where was Julie before the park? A:school Task: Basic Induction Lily is a swan. Lily is white. Bernhard is green. Greg is a swan. What color is Greg? A:white 39 / 80
  24. What are problems in bAbI tasks? In order to construct

    more sophisticated tasks, we want to follow the formulation of bAbI. However the problems are: 1. Low expressibility for linguistic expressions the tasks are not constructed in terms of “grammar” its vocabulary can not be extended easily → Re-organize tasks related on linguistic/grammatical elements 2. Skills required in the tasks being not enough/sufficient e.g. analogy, discourse relations, entailments... → Add skills in the reference of other existing tasks 40 / 80
  25. Problem?? We/researchers try to develop a model to solve tasks/datasets

    listed. However: error analysis is difficult and complicated... What is a main cause of error? What does (not) a system understand? Can we distinguish if a system fortunately outputs a correct answer? ͋Δ໰͍ʹۮવਖ਼ղ͍ͯ͠ͳ͍͔ΛݟۃΊ͍ͨ গ͠ม͑ͨઃఆͰ͸ؒҧ͑Δ͜ͱ͕͋Δ͔΋͠Εͳ͍ 41 / 80
  26. ΍Δ͜ͱͷ֓ཁ ໨త: ݴޠཧղʹඞཁͳཁૉ΍ೳྗΛ෼ྨ͢Δ 0. ݴޠཧղͬΆ͍΋ͷΛ෼ղͯ͠ߟ͑Δ ͻͱͭͷࣄ৅ͷදݱ (A) + ࣄ৅૬ޓͷؔ܎ੑ (B)

    1. (A) จ๏ཁૉͱޠኮ಺༰ʹ෼͚ͯɺจ๏ཁૉΛ੔ཧ͢Δ 2. (B) ඞཁͬΆ͍ೳྗΛطଘλεΫ͔Βྻڍ 44 / 80
  27. Overview of natural language understanding We naively assume that readers

    follow the figure below: NL Understanding Facts Chaining Simple Sentence Understanding Logical Reasoning Entailment ... Grammatical Meanings Lexical Meanings → Each task should be intended to test each grammatical meaning and chaining skill?? 45 / 80
  28. 1. List of grammatical elements Table: Grammatical elements; ∗ =

    appearing in the bAbI tasks; †N = appearing in our extra-task N (later). Word class etc. Elements Common nouns number†1, definiteness Personal pronouns case†2, person∗, gender∗, number∗ Determinatives demonstrative, distributive†3, degree†4, etc. Adjectives attributive or predicative, comparison∗ Verbs voice∗, modality†5, tense-aspect†6, type Prepositions temporal∗, spatial∗, etc. Adverbs common adverbs, sentence adverbs Conjunctions coordinates∗, subordinates†7,†8 Sentence mood imperative, subjunctive Interrogative wh-∗, how∗, y/n∗ 46 / 80
  29. Annotation MCTest Λର৅ʹ͢Δ ࢠڙ޲͚ͷจষɺcrowdworker ͕จষͱ໰͍Λ࡞੒ multiple choice 120+200 questions Λ໨ࢹͰ

    200 ͷ΄͏ͷҰக཰͸ 85% multi-labeling ඞཁͱ͠ͳ͍໰͍͸ɺ౴͕͑ͦͷ··ॻ͍ͯ͋Δจ͕ଘࡏ ͢Δ 52 / 80
  30. Example 1 ID: MC160.dev.1 (3) one: C1: Sally had a

    very exciting summer vacation. C2: She went to summer camp for the first time. C3: Sally’s favorite activity was walking in the woods because she enjoyed nature. Q: Why does Sally like walking in the woods? A: She likes nature. Coreference resolution: · she in C3 = Sally in C3 Causal relation: · she enjoyed nature in C3 → Sally’s favorite activity was walking in the woods in C3 Commonsense reasoning: · Sally’s favorite activity was walking ... in C3 ⇒ Sally likes walking in the woods ... in Q · enjoyed nature in C3 ⇒ likes nature in A 55 / 80
  31. Example 2 ID: MC160.dev.29 (1) multiple: C1: The princess climbed

    out the window of the high tower and climbed down the south wall when her mother was sleeping. C2: She wandered out a good ways. C3: Finally she went into the forest where there are no electric poles but where there are some caves. Q: Where did the princess wander to after escaping? A: Forest Coreference resolution: · She in C2 = the princess in C1 · She in C3 = the princess in C1 Temporal relation: · the actions in C1 → wandered out ... in C2 → went into ... in C3 56 / 80
  32. Example 2 ID: MC160.dev.29 (1) multiple: C1: The princess climbed

    out the window of the high tower and climbed down the south wall when her mother was sleeping. C2: She wandered out a good ways. C3: Finally she went into the forest where there are no electric poles but where there are some caves. Q: Where did the princess wander to after escaping? A: Forest Complex sentence: · C1 = the princess climbed out ... and [the princess] climbed down ... Commonsense reasoning: · escaping in Q ⇒ the actions in C1 · wandered out in C2 and went into the forest in C3 ⇒ wander to the forest in Q and A 57 / 80
  33. Example 3 ID: Civil disobedience, paragraph 1, question 1 C1:

    One of its earliest massive implementations was brought about by Egyptians against the British occupation in the 1919 Revolution. C2: Civil disobedience is one of the many ways people have rebelled against what they deem to be unfair laws. Q: What is it called when people in society rebel against laws they think are unfair? A: Civil disobedience Coreference resolution: · they in C2 = people in C2 (different clauses) · they in Q = people in Q (different clauses) Temporal relation: · people have rebelled... in C2 → when people in society rebel... in Q 63 / 80
  34. Example 3 ID: Civil disobedience, paragraph 1, question 1 C1:

    One of its earliest massive implementations was brought about by Egyptians against the British occupation in the 1919 Revolution. C2: Civil disobedience is one of the many ways people have rebelled against what they deem to be unfair laws. Q: What is it called when people in society rebel against laws they think are unfair? A: Civil disobedience Complex sentences: · C2 = one of the many ways people have (relative clause) · C2 = Civil disobedience is... against [the object] and [it is] what they deem to... (relative clause) · Q = What is it called... laws and they think [the laws] unfair? 64 / 80
  35. Example 3 ID: Civil disobedience, paragraph 1, question 1 C1:

    One of its earliest massive implementations was brought about by Egyptians against the British occupation in the 1919 Revolution. C2: Civil disobedience is one of the many ways people have rebelled against what they deem to be unfair laws. Q: What is it called when people in society rebel against laws they think are unfair? A: Civil disobedience Commonsense reasoning: · What is it called in Q ⇒ Civil disobedience is · laws they think... in C2 = what they deem to in Q 65 / 80
  36. ϥϕϧ਺ͷग़ݱස౓ͷൺֱ Table: Numbers of skills required in each question of

    MCTest / SQuAD. # skill(s) 0 1 2 3 4 5 MCTest 10.3% 28.4% 28.4% 23.8% 8.1% 0.9% SQuAD 5.0% 48.8% 37.5% 6.2% 2.5% 0.0% 3 ͭҎ্ཁٻ͢Δͷ͸ MCTest ͷํ͕ଟ͍ ೉қ౓ͱͯ͠ఆٛͰ͖Δʁ 67 / 80
  37. Discussion ͍Ζ͍Ζ 1. Commonsense reasoning 2. Task unification / other

    fomulations 3. Tasks for machine learning 4. Task difficulty 69 / 80
  38. Discussion: Commonsense reasoning ΰϛശঢ়ଶ: ෼ྨෆ໌ͳ΋ͷΛʮ͜ΕৗࣝͬΆ͍ʯͱͯ͠͠·͏ ࢀߟจݙ Levesque 2011, Winograd Schema

    Challenge [url] Davis and Marcus 2015, Commonsense reasoning and commonsense knowledge in artificial intelligence [url] Clark and Etzioni 2016, My computer is an honor student - but how intelligent is it? Standardized Tests as a Measure of AI [pdf] 70 / 80
  39. Davis+ 2015, Commonsense reasoning and ... Commonsense reasoining ͱ͸ʁ ͱ͍͏ղઆΛ͍ͯ͠Δ

    1. Taxonomic reasoning ੈք஌ࣝʹཪ෇͚ΒΕͨਪ࿦ 2. Temporal resoning ࣌ؒతॱংؔ܎ 3. Action and change ҼՌਪ࿦తͳ΋ͷɺ͜ΕΛͨ͠ΒͲ͏ͳΔ͔ 4. Qualitative reasoning ఆੑతͳؔ܎ɺ͋Ε͕มԽͨ͠Β͜Ε΋มԽ͢Δͱ͍͏ରԠ 71 / 80
  40. Davis+ 2015, Commonsense reasoning and ... 1. Taxonomic reasoning Taxonomies

    are something like “semantic networks” Node: entities, abstract concepts, category, ... Edge: subcategory, instance, property, ... 72 / 80
  41. Davis+ 2015, Commonsense reasoning and ... 2. Temporal reasoning Πϕϯτͷॱংؔ܎Λೝࣝ͢Δ

    ࣌ܥྻతͳ஌͔ࣝΒ৽͍͠ਪ࿦Λ͢Δ (Temporal logic [url] తͳ) “ ..., if one knows that Mozart was born earlier and died younger than Beethoven, one can infer that Mozart died earlier than Beethoven.” 73 / 80
  42. Davis+ 2015, Commonsense reasoning and ... 3. Action and change:

    Constraints on events: Events are atomic. Every change in the world is the result of an event. Events are deterministic. ... Domains including: Continuous domains, where change is continuous Simultaneous events Multiple agents ... 74 / 80
  43. Davis+ 2015, Commonsense reasoning and ... 4. Qualitative reasoning If

    the price of an object goes up then (usually, other things being equal) the number sold will go down. If the temperature of gas in a closed container goes up, then the pressure will go up. If an ecosystem contains foxes and rabbits and the number of foxes decreases, then the death rate of the rabbits will decrease (in the short term). 75 / 80
  44. Discussion: Task formulations Output ޠ۟Λ౴͑Δ͚ͩͷλεΫͰྑ͍ͷ͔ʁ ͋Δ͍͸ɺಡղλεΫ͸બ୒ࣜͷ΋ͷͰΑ͍ͷ͔ʁ ޠ۟ͷ QA ͩͱɺ౴͑Λ༻ҙ͢Δͷ͕େม SQuAD(2016)

    Ͱ͸ෳ਺ਓʹղ౴Λ࡞Β͓ͤͯΓɺ౴͕͑ෳ਺௨ Γ͋Δʢผͷදݱɺspan ͷ௕͞ʣ ཧ༝આ໌ͳͲΛͤ͞Δͱ΋ͬͱେม ൑ఆثΛ࡞Δʁ ਓؒͷ࠾఺͢ΒखͰ΍ͬͯΔͷʹʜʜ ͜ͷ͋ͨΓͷٞ࿦͸ Clark and Etzioni 2016, My computer is an honor student - but how intelligent is it? Standardized Tests as a Measure of AI [pdf] ͋ͨΓΛࢀর খֶੜϨϕϧͷࢉ਺ɾཧՊͷ໰୊Λ࡞͍ͬͯΔ͕ɺଟذબ୒͔ࣜ Β୤٫͢Δͷʹ࢛ۤീ͍ۤͯ͠Δ 76 / 80
  45. Discussion: Task formulations Input จ͚ͩ͡Όͳ͘ɺਤ΍දΛ࢖͍͍ͨ ͜ͷ͋ͨΓͷٞ࿦΋ Clark and Etzioni 2016

    Ͱͳ͞Ε͍ͯ·͢ ࢉ਺ͰزԿͷ໰୊͕ग़ͤͳ͍ͷ͸೉͍͠ ཧՊͷ஌ࣝ໰୊Ͱը૾Λ࢖͑ΔͱҰؾʹ෯͕޿͕Δ ֎෦஌ࣝͷѻ͍ΛͲ͏͢Δ͔ʁ ৗࣝɿ஌͍ͬͯͯཉ͍͠ͷͰςετͱͯ͠͸ఏࣔ͠ͳ͍ ಛघͳ஌ࣝɿղ͚ΔΑ͏ʹิॿతͳઆ໌จΛ૿΍͢ʢਓ͕ղ͘ࠃ ޠͷจষ୊Ͱ஫͕෇͘Α͏ʹʣ 77 / 80
  46. Discussion: Tasks for machine learning ͦ΋ͦ΋ reading comprehension ʹ͓͚Δʮֶशʯ͸ԿΛֶशͯ͠ ͍Δͷ͔

    ʢݴޠ֫ಘΛ஍ಓʹ΍Δ΂͖Ͱ͸ʁʣ ʢຊ౰͔ʁʣ ೝ஌ݴޠֶͰ͸ɺ༻๏ج൫ཧ࿦ͱ͍͏ͷ͕͋Γʜʜ Langacker 1987 [amzn] ΛಡΈ͍ͨ ೔ຊޠͩͱ τϚηϩ 2006, ʰ͜ͱ͹Λͭ͘Δʱ[amzn] ͕͓͢ ͢Ί ʮςΩετϕʔεͷݴޠ֫ಘͱ͸ʁʯΛߟ͑Δඞཁ͸͋Δ ֓೦ɾจ๏͸ຊདྷతʹ͸஌֮৘ใͱ݁ͼͭ͘΋ͷͳʢ͸ͣʣͷͰɺ ஌֮৘ใΛ࣋ͨͳֶ͍शث͸֓೦ؒͷؔ܎ੑ͔͠शಘͰ͖ͳ͍ Taxonomy Λ֫ಘ͢Δɺͱ͍͏ײ͡ ʹɺೝ஌ʜʜ 78 / 80