Share
Embed
PRML Chapter5 Hessian Matrix
PRML復々習レーン #7
http://atnd.org/events/33833
での発表に用いた資料
hassaku
December 16, 2012
Transcript
13 slides
-
ϔοηߦྻ )FTTJBO.BUSJY w ϔοηߦྻͷׂ w ϔοηߦྻΛٻΊΔ͜ͱʹΑΓɺޡࠩؔͷॏΈʹؔ͢ Δ̎֊ඍΛධՁ͢Δ͜ͱ͕Մೳ w ϔοηߦྻͷݻ༗͕શͯਖ਼ʢਖ਼ఆʣͳΒۃখɺݻ ༗͕શͯෛͳΒۃେɺਖ਼ͱෛͷ྆ํͳΒҌΛࣔ͢
z = x2 + y2 z = xe−x2 −y2 λ 1 = 1 2 (3 + 33) λ 2 = 1 2 (3 - 33) λ 1 = 1.7 λ 2 = 0.86 λ 1 = −1.7 λ 2 = −0.86 1212݄16༵ -
χϡʔϥϧίϯϐϡʔςΟϯά Ͱͷׂ w ඇઢܗ࠷దԽΞϧΰϦζϜʹ͓͚Δޡࠩۂ໘ͷධՁ w ֶशσʔλͷมಈ࣌ʹ͓͚Δ࠶ֶशͷߴԽ w ʮמΓࠐΈʯΞϧΰϦζϜʹ͓͚ΔͳॏΈͷಛఆ w ϕΠζωοτϫʔΫͷϥϓϥεۙࣅʹ͓͚Δ༧ଌ
ٴͼύϥϝʔλͷܾఆɺϞσϧΤϏσϯεධՁ ʢઅʣ 1212݄16༵ -
ϔοηߦྻ )FTTJBO.BUSJY ∂2 E ∂w kl ∂w mn → ∂2
E ∂w ij 2 = H ij i, j ∈{1…W} 8ύϥϝʔλͷ૯ //ʹ͓͚Δϔοη ߦྻͷ֤ཁૉ H = ∂2 E ∂w 1 ∂w 1 … ∂2 E ∂w 1 ∂w W ∂2 E ∂w W ∂w 1 ∂2 E ∂w W ∂w W ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ॏΈόΠΞεͱ͍ͬͨ ύϥϝʔλʹؔͯ͠Ұ༷ʹฒ͢ 1212݄16༵ -
jrjͷ֓ཁ w ϔοηߦྻ0 8? ͷܭࢉྔ͕ඞཁͩ͠ɺೋ֊ඍେม w ର֯ۙࣅ w ඇର֯ཁૉ͕ແࢹͰ͖ΔͷͰ͋Ε0 8
ͰܭࢉՄೳ w ֎ੵʹΑΔۙࣅ w ेʹֶशࡁΈͷঢ়ଶͰɺ0 8? ͰܭࢉՄೳ w ٯߦྻͷޮͷྑ͍ஞ࣍ܭࢉ͕Մೳʢˠ४χϡʔτϯ๏ʣ w ༗ݶ෯ͷࠩʹΑΔۙࣅ w ٯൖ๏Λ༻͍ͯܭࢉ͞ΕͨϔοηߦྻͷνΣοΫʹ༗ޮ w Ұ෦ٯൖެࣜΛऔΓೖΕΔ͜ͱʹΑΓ0 8? ͰܭࢉՄೳ w ϔοηߦྻͷݫີͳධՁ w ϔοηߦྻͷੵͷߴͳܭࢉ 1212݄16༵ -
࠷దԽΞϧΰϦζϜ w(τ +1) = w(τ ) −η∇E n w(τ +1)
= w(τ ) −η H n (τ ) ( )−1 ∇E n w(τ +1) = w(τ ) −ηH n −1∇E n ࠷ٸ߱Լ๏ χϡʔτϯ๏ ४χϡʔτϯ๏ ର֯χϡʔτϯ๏ ϔοηߦྻͷஞ࣍ۙࣅʹ͍ͭͯɺ%'1 #'(4 -#'(4 w ڭՊॻͷྲྀΕͰͳ͍Ͱ͕͢ɺΠϝʔδ͔ͭΉͨΊʹɾɾɾ w(τ +1) = w(τ ) −η diagH n ( )−1 ∇E n ΑΓߴͳऩଋ ܭࢉྔݮʢͨͩ͠ෆ҆ఆԽʣ ͏গ͠Ϛγͳۙࣅ H n (τ ) → H n ۙ 1212݄16༵ -
ର֯ۙࣅ w ϔοηߦྻ͕ର֯ߦྻͩͱ͢Εɺٯߦྻݻ༗ΛٻΊΔͷ ༰қ͘ɺඇର߲֯ͷӨڹ͕গͳ͍Ͱ༗ҙٛɻ࣮ࡍܭࢉ ྔ0 8 ͰࡁΉ a j =
w ji z i i ∑ ∂2 E n ∂a j 2 = h'(a j )2 w kj w k' j ∂2 E n ∂a k ∂a k' k' ∑ k ∑ + h''(a j ) w kj ∂E n ∂a k k ∑ ∂2 E n ∂w ij 2 = ∂ ∂w ij ∂E n ∂a j z i ⎛ ⎝ ⎜ ⎞ ⎠ ⎟ = ∂2 E n ∂a j 2 z i 2 ∂2 E n ∂a j 2 = h'(a j )2 w kj ∂2 E n ∂a k 2 k ∑ + h''(a j ) w kj ∂E n ∂a k k ∑ ඇର߲͕֯ແࢹग़དྷΔͱ͢Εର߲֯ͷΈܭࢉ͢Ε͍͍ͷͰɹɹɹͷ ܭࢉྔͰࡁΈɺ֤ͷܭࢉҎԼͷΑ͏ʹ୯७Խ͞ΕΔ ∂E n ∂a j = h'(a j ) w kj ∂E n ∂a k k ∑ ΑΓɺϔοηߦྻͷ֤ ٯൖͷެࣜ ʹର͠ɺੵͷඍΛߦ͑ O(W ) ຊདྷɺ͜Εʹؔͯ͠ϔοηߦ ྻશମΛͷܭࢉͰධՁ O(W 2 ) ͨͩ͠ɺϔοηߦྻΛඞཁͱ͢Δओཁ ͳͰۃʹඇର֯Ͱ͋Δ͜ͱ͕ ଟ͘ɺద༻ʹҙ͕ඞཁɾɾɾ 1212݄16༵ -
֎ੵʹΑΔۙࣅ ʢΨεɾχϡʔτϯ๏ͱ w ޡ͕ࠩؔฏํͱͳΔΑ͏ͳʹ͓͍ͯɺेʹ܇࿅͞Εͨঢ়ଶͰ ɺϔοηߦྻͷҰ෦ͷ߲ແࢹग़དྷΔ͘Β͍খ͘͞ͳΓɺ0 8? ͷ ୯७ͳੵԋࢉͰධՁ͕Մೳ Η =
∇∇E = ∇y n (∇y n )T n=1 N ∑ + (y n − t n )∇∇y n n=1 N ∑ ेʹ܇࿅͞ΕͨωοτϫʔΫͰɺZ͕Uʹेۙͮ͘͜ͱʹΑΓɺୈ߲͕̎θϩʹۙ ͖ͮɺແࢹ͢Δ͜ͱ͕ग़དྷΔͷͰҎԼͷΑ͏ʹۙࣅՄೳ Η ∇y n (∇y n )T n=1 N ∑ ≡ b n b n T n=1 N ∑ -FWFOWFSH.BSRVBSEUۙࣅ ʢ֎ੵʹΑΔۙࣅʣ E = 1 2 (y n − t n )2 n=1 N ∑ ͷΑ͏ͳճؼʹର͠ɺɹɹɹɹɹɹɹɹɹɹΑΓ ∇E = (y n − t n )∇y n n=1 N ∑ 1212݄16༵ -
ϔοηٯߦྻͷஞ࣍ܭࢉ w ֎ੵʹΑΔۙࣅΛϕʔεʹͯ͠ɺ-ݸͷσʔλʹର͠ஞ࣍ ܭࢉ͠ɺ) Ћ*ͷٯߦྻΛঃʑʹݟ͚ͭΔ͜ͱ͕Մೳ Η L+1 = Η L
+ b L+1 b L+1 T 8PPECVSZ߃ࣜ Η N = b n b n T n=1 N ∑ Η L+1 −1 = Η L + b L+1 b L+1 T ( )−1 = Η L −1 + Η L −1b L+1 b L+1 T Η L −1 1+ b L+1 T Η L −1b L+1 Η 0 = b 1 b 1 T = αI (α 1) Η +αI L +1= N ʹͳΔ·Ͱ܁Γฦ͢͜ͱͰ ͷٯߦྻ͕ݟ͔ͭΔ ͱͯ͠ 1212݄16༵ -
༗ݶ෯ͷࠩʹΑΔۙࣅ w ਫ਼্ͷݶք͕͋Δͷͷɺٯൖ๏ͰٻΊΒΕͨϔοηߦ ྻͷݕূʹ༗ޮɻී௨ʹܭࢉ͢Δͱ0 8? ͷͱ͜Ζɺ̍֊ඍ ٯൖͷެࣜΛద༻͢Δ͜ͱͰ0 8? Ͱܭࢉ͢Δ͜ͱՄೳ ∂2
E ∂w ji ∂w lk = 1 4ε2 E(w ji + ε,w lk + ε)− E(w ji + ε,w lk − ε)− E(w ji − ε,w lk + ε)+ E(w ji − ε,w lk − ε) { }+O(ε2 ) ∂2 E ∂w ji ∂w lk = ∂ ∂w lk ∂E ∂w ji = 1 2ε ∂E ∂w ji (w lk + ε)− ∂E ∂w ji (w lk − ε) ⎧ ⎨ ⎪ ⎩ ⎪ ⎫ ⎬ ⎪ ⎭ ⎪ +O(ε2 ) ∂E ∂w ji ٯൖ๏Ͱޮతʹٻ ΊΔ͜ͱ͕ग़དྷΔͷͰ ઁಈЏʹԠͨ͡ཹޡࠩ ϔοηߦྻͷ֤ཁૉ 1212݄16༵ -
ϔοηߦྻͷݫີͳධՁ w ٯൖͷςΫχοΫͷ֦ுΛར༻ͨ͠ɺҙͷϑΟʔ υϑΥϫʔυߏΛͭωοτϫʔΫͷݫີͳϔοη ߦྻΛ0 8? ͰٻΊΔख๏͕ଘࡏ͢Δ Β͍͠ʣ #JTIPQ #JTIPQ
w ̎ωοτϫʔΫͷ߹ɺͦΕͧΕͷॏΈͷΈ߹ ΘͤͰϒϩοΫԽ͢Δ͜ͱͰ༰қʹࣜΛಋग़͢Δ͜ͱ ͕Ͱ͖ΔʢΒ͍͠ʣ 1212݄16༵ -
ϔοηߦྻͷੵͷߴͳܭࢉ w ී௨ʹϔοηߦྻΛධՁͯ͠ɺੵΛٻΊΔͱ 0 8? ɻͨͩ͠ɺϔοηߦྻࣗମͰͳ͘ɺ͋Δϕ Ϋτϧͱͷੵ͕ඞཁͳ͚ͩͰ͋Εɺޡࠩٯൖͷख ๏༻͍Δ͜ͱʹΑΓ0 8 ʹͰ͖Δ
vTH = vT∇ ∇E ( ) Ͱ͋Δ͜ͱ͔Βɺ㲆&ΛٻΊΔͷʹඞཁͳޡࠩٯ ൖʹؔΘΔશͯͷํఔࣜͷܭࢉ෦ʹඍԋ ࢉࢠɹɹɹΛ࡞༻ͤ͞Δɻ vT∇ 1212݄16༵