About Missing Values

Transcript

1. ܽଛ஋ॲཧʹ͍ͭͯ
   dࣦΘΕͨ஋ΛٻΊͯd
2. ໨࣍ 1. ܽଛ஋ͱ͸ 2. ܽଛআڈ 3. ܽଛλΠϓ 4. ୯Ұ୅ೖ๏ 5. ଟॏ୅ೖ๏ 6. ࠓճ঺հ͠ͳ͔ͬͨ΋ͷ 7. ࢀߟจݙ
3. ܽଛ஋ͱ͸ Missing Valueʢܽଛ஋ɺܽଌ஋ʣ In statistics, missing data, or missing values, occur when no data value is stored for the variable in an observation. Missing data are a common occurrence and can have a significant effect on the conclusions that can be drawn from the data. ʢhttps://en.wikipedia.org/wiki/Missing_dataʣ
4. ܽଛ͕͋Δͱ Կ͕ࠔΔͷ͔ʁ ܽଛ஋ͱ͸
5. ܭࢉग़དྷͳ͍ Q,10, 20, ܽଛ஋ͷฏۉ͸ʁ ܽଛ஋ͱ͸ > mean(c(10, 20, NA)) [1] NA
6. Ͳ͏͢Δͷ͔ʁ ܽଛ஋ͱ͸
7. 1. ফ͢ 2. ຒΊΔ ܽଛ஋ͱ͸
8. 1. ফ͢ 2. ຒΊΔ 1. ফ͢ ܽଛ஋ͱ͸
9. ܽଛͷ͋ΔϨίʔυΛফ͢ ମॏ ੑผ 50 ঁ 70 உ 55 NA NA உ ܽଛ஋আڈ
10. ମॏ ੑผ 50 ঁ 70 உ 55 NA NA உ ମॏ ੑผ 50 ঁ 70 உ ܽଛͷ͋ΔϨίʔυΛফ͢ ܽଛ஋আڈ
11. ମॏ ੑผ 50 ঁ 70 உ 55 NA NA உ ମॏ ੑผ 50 ঁ 70 உ ϦετϫΠζ๏ ܽଛͷ͋ΔϨίʔυΛফ͢ ܽଛ஋আڈ
12. ফ͍͍ͯ͠ͷʁ 1. ফͯ͠΋ྑ͍৔߹ 2. ফͯ͠͸ବ໨ͳ৔߹ ܽଛ஋আڈ
13. ׬શʹϥϯμϜʹܽଛ ܽଛύλʔϯ͸ੑผ ΍ମॏͱແؔ܎ ܽଛ஋আڈ
14. ׬શʹϥϯμϜʹܽଛ ܽଛύλʔϯ͸ੑผ ΍ମॏͱແؔ܎ ফͯ͠΋໰୊͸ͳ͍ ͨͩ͠ɺޮ཰͸Լ͕Δ ܽଛ஋আڈ
15. ؍ଌม਺ʹґଘͯܽ͠ଛ ঁੑͷํ͕ଟܽ͘ଛ ܽଛ஋আڈ
16. ؍ଌม਺ʹґଘͯܽ͠ଛ ফ͢ͱঁੑͷσʔλͷΈݮগ ภΔ ঁੑͷํ͕ଟܽ͘ଛ ܽଛ஋আڈ
17. ܽଛม਺ࣗମʹґଘͯܽ͠ଛ ମॏͷॏ͍ํ͕ଟ͘ ܽଛ ܽଛ஋আڈ
18. ܽଛม਺ࣗମʹґଘͯܽ͠ଛ ফ͢ͱମॏͷߴ͍σʔλ͕ݮগ ภΔ ମॏͷॏ͍ํ͕ଟ͘ ܽଛ ܽଛ஋আڈ
19. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
20. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
21. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ফͯ͠΋ྑ͍ ܽଛλΠϓ
22. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
23. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ରॲࠔ೉ ܽଛλΠϓ
24. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
25. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
26. 1. ফ͢ 2. ຒΊΔ 1. ফ͢ 2. ຒΊΔ ୯Ұ୅ೖ๏
27. ͲΜͳ஋ͰຒΊΔͷ͔ʁ ୯Ұ୅ೖ๏
28. 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏
29. 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏
30. ฏۉ஋Λܭࢉ͠୅ೖ ୯Ұ୅ೖ๏
31. ฏۉ஋Λܭࢉ͠୅ೖ ฏۉ஋୅ೖ๏ ฏۉ஋ ≒ உੑͷମॏͷฏۉ ୯Ұ୅ೖ๏
32. 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏
33. ܽଌ஋Λ༧ଌ͢ΔϞσϧΛ࡞Δ *ิॿม਺ ܽ ଛ ͷ ͋ Δ ม ਺ *ิॿม਺:ܽଛ஋ͷ༧ଌʹ࢖ΘΕΔม਺ɻඞͣ͠΋໨తม਺ͷ༧ଌʹ࢖ΘΕΔΘ͚Ͱ͸ͳ͍ɻ ୯Ұ୅ೖ๏
34. ܽଌ஋Λ༧ଌ͢ΔϞσϧΛ࡞Δ ิॿม਺ ܽ ଛ ͷ ͋ Δ ม ਺ ୯Ұ୅ೖ๏
35. ܽଌ஋Λ༧ଌ͢ΔϞσϧΛ࡞Δ ܽ ଛ ͷ ͋ Δ ม ਺ ʢ֬ఆతʣ ճؼ୅ೖ๏ ิॿม਺ ୯Ұ୅ೖ๏
36. ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୅ೖ஋͕શͯ௚ઢ্ʹ ୯Ұ୅ೖ๏
37. ܽ ଛ ͷ ͋ Δ ม ਺ ༧ଌ஋ͷޡࠩΛաখධՁ ิॿม਺ ୅ೖ஋͕શͯ௚ઢ্ʹ ୯Ұ୅ೖ๏
38. 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏
39. ୅ೖϞσϧʹޡ߲ࠩΛ௥Ճ ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ޡ߲ࠩ ୯Ұ୅ೖ๏
40. ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୅ೖϞσϧʹޡ߲ࠩΛ௥Ճ ͹Β͖ͭΛ൓ө ୯Ұ୅ೖ๏
41. ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୅ೖϞσϧʹޡ߲ࠩΛ௥Ճ ֬཰తճؼ୅ೖ๏ ୯Ұ୅ೖ๏
42. ୅ೖϞσϧͷޡ߲ࠩ͸൓өग़དྷͨ ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୯Ұ୅ೖ๏
43. ୅ೖϞσϧͦͷ΋ͷͷෆ࣮֬ੑ͸ʁ ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ʁ ʁ ʁ ୯Ұ୅ೖ๏
44. 1ͭͷ஋Λ୅ೖʢ୯Ұ୅ೖ๏ʣ ෳ਺ͷ஋Λ୅ೖ ʢଟॏ୅ೖ๏ʣ ଟॏ୅ೖ๏
45. ୅ೖ ෼ੳ ౷߹ ෳ਺ͷ୅ೖ஋Λਪఆ͠ෳ਺ͷ୅ೖࡁΈσʔλΛੜ੒ ୅ೖࡁΈσʔλΛ࢖༻ͯ͠฼਺ਪఆ ਪఆ஋Λ౷߹͠ɺ࠷ऴ݁Ռͱ͢Δ ଟॏ୅ೖ๏
46. ܽଛσʔλ ୅ೖࡁΈσʔλ1 ୅ೖࡁΈσʔλ2 ୅ೖࡁΈσʔλ3 ෼ੳ݁Ռ1 ෼ੳ݁Ռ2 ෼ੳ݁Ռ3 ࠷ऴ݁Ռ ୅ೖ ෼ੳ ౷߹ ଟॏ୅ೖ๏
47. ܽଛσʔλ ୅ೖࡁΈσʔλ1 ୅ೖࡁΈσʔλ2 ୅ೖࡁΈσʔλ3 ෼ੳ݁Ռ1 ෼ੳ݁Ռ2 ෼ੳ݁Ռ3 ࠷ऴ݁Ռ ୅ೖ ෼ੳ ౷߹ ଟॏ୅ೖ๏
48. ‣ DA๏ : Data Augmentationʢσʔλ֦େ๏ʣ ‣ FCS๏ : Fully Conditional Specificationʢ׬ શ৚݅෇͖ࢦఆʣ ‣ EMB๏ : Expectation-Maximization with Bootstrapping ଟॏ୅ೖ๏
49. ‣ ܽଛআڈɿϖΞϫΠζ๏ɺม਺আڈ ‣ ୯Ұ୅ೖ๏ɿൺ཰୅ೖ๏ɺϗοτσοΫ๏ɺίʔ ϧυσοΫ๏ɺLOCFɺNOCB ‣ ଟॏ୅ೖ๏ɿ୅ೖ਍அɺײ౓෼ੳ ‣ ׬શ৘ใ࠷໬๏ ‣ LightGBMͳͲͷܽଛ஋ࣗಈॲཧ ࠓճ঺հ͠ͳ͔ͬͨ΋ͷ
50. ‣ ߴڮকٓ,౉ลඒஐࢠ, ܽଌσʔλॲཧ: RʹΑΔ୯Ұ୅ೖ๏ͱ ଟॏ୅ೖ๏ (౷ܭֶOne Point), ڞཱग़൛, 2017/12/9, 192ϖʔ δ ‣ ܽଛ஋͕͋Δσʔλͷ෼ੳ, Sunny side up!, http:// norimune.net/1811 ‣ RͰ࣮ફʂܽଛσʔλ෼ੳೖ໳ʲ1ʳ, NHN TECHORUS Tech Blog, https://techblog.nhn-techorus.com/archives/6573 ‣ R ܽଛ஋ͷରԠʢmissing value treatmentʣ, ౷ܭֶͱӸֶͱ ࣌ʑɺॿڭੜ׆, http://jojoshin.hatenablog.com/entry/ 2017/02/03/220118 ࢀߟจݙ
51. ENJOY! A