Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Paper Reading: Noise-Contrastive Estimation of ...

Paper Reading: Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

Avatar for Kento Nozawa

Kento Nozawa

June 12, 2018
Tweet

More Decks by Kento Nozawa

Other Decks in Research

Transcript

  1. Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural

    Image Statistics Student seminar @ Sugiyama-Sato-Honda Lab D1, DC1 @ The University of Tokyo Kento NOZAWA (@nzw0301) Michael U. Gutmann, Aapo Hyvärinen, JMLR, 2012.
  2. Why I choose this paper? • Useful approximation to calculate

    an intractable partition function in probabilistic models • Fast and simple to learn probabilistic models • Sometimes used in recent DNN/ ML papers 2
  3. or . Z(✓) is A sum/integral is intractable over unnormalized

    
 probability distribution for many models. Goal: Estimation parameters in a probabilistic model: Background: Probabilistic Models 3 ✓ <latexit sha1_base64="317VWb7h3beDW2lJpPQYuAuNpOg=">AAACaXichVFNLwNBGH66vqo+2nIRLqIhTs27LkQiGi6O2mpJENldg9Xt7mZ32qQaibOTm+BEIiJ+hgN/wKE/QXokcXHwdttEELyTmXnmmfd555kZ3bVMXxLVQkpbe0dnV7g70tPb1x+NxQfyvlPyDJEzHMvx1nTNF5Zpi5w0pSXWXE9oRd0Sq3phsbG/Whaebzr2iqy4YrOo7drmjmlokqn8htwTUtuKJShJQYz+BGoLJOYfInNHAJad2A02sA0HBkooQsCGZGxBg89tHSoILnObqDLnMTKDfYFDRFhb4izBGRqzBR53ebXeYm1eN2r6gdrgUyzuHitHMU5PdEsv9Eh39Ezvv9aqBjUaXio8602tcLeix0PZt39VRZ4l9j5Vf3qW2MFM4NVk727ANG5hNPXlg9OX7GxmvDpBV1Rn/5dUo3u+gV1+Na7TInOBCH+A+v25f4L8VFKlpJqmRGoBzQhjBGOY5PeeRgpLWEaOz93HCc5wHqorcWVIGW6mKqGWZhBfQkl8APjwje0=</latexit> <latexit sha1_base64="G5x5alePLF85znahbXBopKC7Dq4=">AAACaXichVFNLwNBGH66vtdXy0W4NBri1LzrQiRCuDi2aDUpkd01WLa7m91pk2r8ASc3wQkRET/DgT8g0p8gjiQuDt5umwiCdzIzzzzzPu88M2N4thVIompEaWpuaW1r71A7u7p7eqOxvmzgFn1TZEzXdv2coQfCthyRkZa0Rc7zhV4wbLFi7M7X9ldKwg8s11mWZU+sFfQtx9q0TF0ylV2V20Lq69EEJSmM+E+gNUBi5k6d9i4e1ZQbvcIqNuDCRBEFCDiQjG3oCLjloYHgMbeGCnM+IyvcF9iHytoiZwnO0Jnd5XGLV/kG6/C6VjMI1SafYnP3WRnHCD3QNb3QPd3QE73/WqsS1qh5KfNs1LXCW+89GFh6+1dV4Fli+1P1p2eJTUyGXi327oVM7RZmXV/aO3pZmlocqYzSOT2z/zOq0i3fwCm9mpdpsXgKlT9A+/7cP0F2PKlRUktTYnYO9WjHEIYxxu89gVksIIUMn7uDQxzjJPKsxJQBZbCeqkQamn58CSXxAZ6+j2E=</latexit> <latexit sha1_base64="G5x5alePLF85znahbXBopKC7Dq4=">AAACaXichVFNLwNBGH66vtdXy0W4NBri1LzrQiRCuDi2aDUpkd01WLa7m91pk2r8ASc3wQkRET/DgT8g0p8gjiQuDt5umwiCdzIzzzzzPu88M2N4thVIompEaWpuaW1r71A7u7p7eqOxvmzgFn1TZEzXdv2coQfCthyRkZa0Rc7zhV4wbLFi7M7X9ldKwg8s11mWZU+sFfQtx9q0TF0ylV2V20Lq69EEJSmM+E+gNUBi5k6d9i4e1ZQbvcIqNuDCRBEFCDiQjG3oCLjloYHgMbeGCnM+IyvcF9iHytoiZwnO0Jnd5XGLV/kG6/C6VjMI1SafYnP3WRnHCD3QNb3QPd3QE73/WqsS1qh5KfNs1LXCW+89GFh6+1dV4Fli+1P1p2eJTUyGXi327oVM7RZmXV/aO3pZmlocqYzSOT2z/zOq0i3fwCm9mpdpsXgKlT9A+/7cP0F2PKlRUktTYnYO9WjHEIYxxu89gVksIIUMn7uDQxzjJPKsxJQBZbCeqkQamn58CSXxAZ6+j2E=</latexit> <latexit sha1_base64="B4/KVo8DHTFQ13Zj30fSx7QNPAM=">AAACaXichVG7SgNBFD1Z3/GRRBvRJhgUq3DXRrESbSyNmgckIrvrmKzZ7C67k4AGf8DKTtRKQUT8DBt/wCKfICkVbCy82SyIinqHmTlz5p47Z2Z01zJ9SdSKKD29ff0Dg0PR4ZHRsVg8MZ7znbpniKzhWI5X0DVfWKYtstKUlii4ntBquiXyenWts59vCM83HXtbHrpip6aVbXPfNDTJVK4kK0Jqu/EUpSmI5E+ghiCFMDac+C1K2IMDA3XUIGBDMragwedWhAqCy9wOmsx5jMxgX+AYUdbWOUtwhsZslccyr4oha/O6U9MP1AafYnH3WJnELD3RHb3QI93TM73/WqsZ1Oh4OeRZ72qFuxs7mdx6+1dV41mi8qn607PEPpYCryZ7dwOmcwujq28cnb1sLW/ONufomtrs/4pa9MA3sBuvxk1GbF4iyh+gfn/unyC3kFYprWYotbIafsUgpjGDeX7vRaxgHRvI8rkHOMU5LiJtJaFMKlPdVCUSaibwJZTUB6pdjCA=</latexit> Partition function p(x; ✓) = 1 Z(✓) p0(x; ✓) <latexit sha1_base64="7gY4fYlpOkCxnEDZkQNamIz0Lr0=">AAACmXichVFNSxtBGH5cWz9iq6leBC9LQ0p6Ce+KoFSEqBfxpKbR0CSG3XWii5vdZXcS1CV/wD8g6KkFKaV/Quil7b2H/IOKBw8p9OLBN5tAq0F9h5l55pn3eeeZGcOzrUASNfuU/mfPBwaHhmMjL16OjsVfjW8Gbs03Rc50bdfPG3ogbMsROWlJW+Q9X+hVwxZbxv5ye3+rLvzAcp338tATpaq+61gVy9QlU+X4jJc6mFeLck9I/a26oBYrvm6GWiP8kOqSDTX0Gtuk/pdXjicoTVGovUDrgkQmc/37AsCaG/+MInbgwkQNVQg4kIxt6Ai4FaCB4DFXQsicz8iK9gUaiLG2xlmCM3Rm93nc5VWhyzq8btcMIrXJp9jcfVaqSNIv+kIt+k5f6ZJuHqwVRjXaXg55Njpa4ZXHjiezf59UVXmW2PunetSzRAVzkVeLvXsR076F2dHXj05a2XcbyfANfaIr9v+RmvSNb+DU/5jn62LjDDH+AO3+c/eCzem0RmltnX9iCZ0YwhReI8XvPYsMVrCGHJ97igv8wE9lSllUVpTVTqrS19VM4E4o2VtadJ9I</latexit> <latexit sha1_base64="Oa0xPtXVUZqDwpGLQAeyWNMf0Jk=">AAACmXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuYFGhUWCvElGSkliRqKtgqxKQVJSZXG9ZWR2lABWsVqgtq4wwUkNTFCygb6BmAgQImwxDKUHZweP9ij9gclYB8geUMMQwpDPkMyQylDLkMqQx5DCVAdg5DIkMxEEYzGDIYMBQAxWIZqoFiRUBWJlg+laGWgQuotxSoKhWoIhEomg0k04G8aKhoHpAPMrMYrDsZaEsOEBcBdSowqBpcNVhp8NnghMFqg5cGf3CaVQ02A+SWSiCdBNGbWhDP3yUR/J2grlwgXcKQgdCF180lDGkMFmC3ZgLdXgAWAfkiGaK/rGr652CrINVqNYNFBq+B7l9ocNPgMNAHeWVfkpcGpgbNZuACRoAhenBjMsKM9AwN9AwDgTHhxAABHAzSDEoMGsDwNmdwYPBgCGAIBdo7i+EAw0mGU0zSTI5MHkxeEKVMjFA9wgwogCkYANixoBo=</latexit> <latexit sha1_base64="Oa0xPtXVUZqDwpGLQAeyWNMf0Jk=">AAACmXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuYFGhUWCvElGSkliRqKtgqxKQVJSZXG9ZWR2lABWsVqgtq4wwUkNTFCygb6BmAgQImwxDKUHZweP9ij9gclYB8geUMMQwpDPkMyQylDLkMqQx5DCVAdg5DIkMxEEYzGDIYMBQAxWIZqoFiRUBWJlg+laGWgQuotxSoKhWoIhEomg0k04G8aKhoHpAPMrMYrDsZaEsOEBcBdSowqBpcNVhp8NnghMFqg5cGf3CaVQ02A+SWSiCdBNGbWhDP3yUR/J2grlwgXcKQgdCF180lDGkMFmC3ZgLdXgAWAfkiGaK/rGr652CrINVqNYNFBq+B7l9ocNPgMNAHeWVfkpcGpgbNZuACRoAhenBjMsKM9AwN9AwDgTHhxAABHAzSDEoMGsDwNmdwYPBgCGAIBdo7i+EAw0mGU0zSTI5MHkxeEKVMjFA9wgwogCkYANixoBo=</latexit> <latexit sha1_base64="v6WdAFhAQijKqZkSrsUrDZZNaYY=">AAACmXichVHLSiNBFD22r0x8RWcz4KYxKHETbougKILObGRWaiYqGg3dbSU26XQ33ZWgNvmB+YGBmdUMiIg/IbjR2c/CTxhcKrhx4U2nwRfqLarq1Kl7bp2qMjzbCiTRZZvS3tHZ1Z34kOzp7esfSA0OrQZuzTdF3nRt11839EDYliPy0pK2WPd8oVcNW6wZlS/N/bW68APLdb7JfU9sVfWyY5UsU5dMFVOTXmZvVi3IXSH1cXVOLZR83Qy1RriRicmGGnqNbVIf5RVTacpSFOpLoMUgjTiW3NQRCtiBCxM1VCHgQDK2oSPgtgkNBI+5LYTM+YysaF+ggSRra5wlOENntsJjmVebMevwulkziNQmn2Jz91mpYpT+0TFd0zmd0H+6e7VWGNVoetnn2WhphVcc+P4pd/uuqsqzxO6D6k3PEiVMR14t9u5FTPMWZktfP/hxnZtZGQ3H6A9dsf/fdElnfAOnfmMeLouVX0jyB2jPn/slWJ3IapTVlik9/zn+igSGMYIMv/cU5rGIJeT53J84xQX+KsPKgrKofG2lKm2x5iOehJK7B6+InDE=</latexit> p0(.; ✓) <latexit sha1_base64="uAUJ2Mb9zA7Ly6FPQCyZwgPfrWU=">AAACdnichVFNLwNBGH66vqo+2nKRSKQh9RFJ864L4SJcHBVFojS7a1ob293N7rRC4w/4Aw4iQUIjfoaLP+DgJ4gjCQcHb7dNBME7mZlnnnmfd56Z0V3L9CXRQ0hpam5pbQu3Rzo6u7qjsXjPqu+UPENkDMdyvHVd84Vl2iIjTWmJddcTWlG3xJq+O1/bXysLzzcde0Xuu2KzqBVsM28ammQqF4tX3MMtSoymZhJZuSOkNpaLDVGKgkj8BGoDDM0OZsfPACw6sSqy2IYDAyUUIWBDMragwee2ARUEl7lNVJjzGJnBvsAhIqwtcZbgDI3ZXR4LvNposDavazX9QG3wKRZ3j5UJJOmerumZ7uiGHun911qVoEbNyz7Pel0r3Fz0qG/59V9VkWeJnU/Vn54l8pgKvJrs3Q2Y2i2Mur58cPy8PL2UrAzTBT2x/3N6oFu+gV1+MS7TYukEEf4A9ftz/wSrEymVUmqaf2IO9QijH4MY5feexCwWsIgMn7uHU1yhGnpTBpSkMlJPVUINTS++hEIfpIaROA==</latexit> <latexit sha1_base64="w3lRR1wwNSwmVOx1hCd3eGQJBU8=">AAACdnichVFNLwNBGH66vuujxUUikYbURyTNuy6ES8PFEVUkSrO7hm5sdze700o1/oCzxEEcSBDxM1z8AYf+BHGshIODt9smguCdzMwzz7zPO8/M6K5l+pKoElKamlta29o7wp1d3T2RaG/fmu8UPEOkDcdyvA1d84Vl2iItTWmJDdcTWl63xLq+v1DbXy8Kzzcde1WWXLGV1/Zsc9c0NMlUNtpbdo+2KTaemItlZE5IbSIbHaEEBRH7CdQGGEkOZyZPKsnSkhO9QQY7cGCggDwEbEjGFjT43DahguAyt4Uycx4jM9gXOEKYtQXOEpyhMbvP4x6vNhuszetaTT9QG3yKxd1jZQxxeqRbqtID3dETvf9aqxzUqHkp8azXtcLNRo4HUq//qvI8S+Q+VX96ltjFTODVZO9uwNRuYdT1xcPTamp2JV4epUt6Zv8XVKF7voFdfDGulsXKGcL8Aer35/4J1qYSKiXUZf6JedSjHYMYxji/9zSSWMQS0nzuAc5xjZvQmzKkxJWxeqoSamj68SUU+gBq55K+</latexit> <latexit sha1_base64="w3lRR1wwNSwmVOx1hCd3eGQJBU8=">AAACdnichVFNLwNBGH66vuujxUUikYbURyTNuy6ES8PFEVUkSrO7hm5sdze700o1/oCzxEEcSBDxM1z8AYf+BHGshIODt9smguCdzMwzz7zPO8/M6K5l+pKoElKamlta29o7wp1d3T2RaG/fmu8UPEOkDcdyvA1d84Vl2iItTWmJDdcTWl63xLq+v1DbXy8Kzzcde1WWXLGV1/Zsc9c0NMlUNtpbdo+2KTaemItlZE5IbSIbHaEEBRH7CdQGGEkOZyZPKsnSkhO9QQY7cGCggDwEbEjGFjT43DahguAyt4Uycx4jM9gXOEKYtQXOEpyhMbvP4x6vNhuszetaTT9QG3yKxd1jZQxxeqRbqtID3dETvf9aqxzUqHkp8azXtcLNRo4HUq//qvI8S+Q+VX96ltjFTODVZO9uwNRuYdT1xcPTamp2JV4epUt6Zv8XVKF7voFdfDGulsXKGcL8Aer35/4J1qYSKiXUZf6JedSjHYMYxji/9zSSWMQS0nzuAc5xjZvQmzKkxJWxeqoSamj68SUU+gBq55K+</latexit> <latexit sha1_base64="fNaMbYxB+H4mwjCvQRiZp7000Gc=">AAACdnichVHLSsNAFD2N7/qquhEEKZb62JQbN4puim5camtV8FGSONZgmoRkWtHSH/AHXIgLBS3iZ7jxB1z0E8Slgi5ceJsGREW9w8ycOXPPnTMzumuZviSqR5SW1rb2js6uaHdPb19/bGBwzXdKniFyhmM53oau+cIybZGTprTEhusJrahbYl0/WGzsr5eF55uOvSqPXLFd1Aq2uWcammQqHxuouNUdik+m5uNbcl9IbSofS1CKgoj/BGoIEghj2YnVsIVdODBQQhECNiRjCxp8bptQQXCZ20aFOY+RGewLVBFlbYmzBGdozB7wWODVZsjavG7U9AO1wadY3D1WxpGkB7qhZ7qnW3qk919rVYIaDS9HPOtNrXDz/SfD2dd/VUWeJfY/VX96ltjDbODVZO9uwDRuYTT15ePT5+xcJlkZp0t6Yv8XVKc7voFdfjGuVkTmDFH+APX7c/8Ea9MplVLqCiXSC+FXdGIEY5jk955BGktYRo7PPcQ5rlGLvCmjSlKZaKYqkVAzhC+h0AfTnI+v</latexit> X x p0(x; ✓) <latexit sha1_base64="8k0CcF80RkwWl9z7U75UczfiOXc=">AAACe3ichVFdK0RBGH72+F4fuygpNycbIW3vUSJuRMmlr0VZtnOOwcn56pzZDcsf8AdcuKIk8SOUG3/AhZ8gl5Qbyrtnt4TwTjPzzDPv884zM4ZvW6EkeogpVdU1tXX1DfHGpuaWRLK1bSn08oEpMqZne8GKoYfCtlyRkZa0xYofCN0xbLFs7EyV9pcLIggtz12Ue75Yc/Qt19q0TF0ylUt2ZMO8k9tV/XVS+3bH1azcFlLvzyVTlKYo1J9Aq4DUxPRx+w2AWS95gSw24MFEHg4EXEjGNnSE3FahgeAzt4YicwEjK9oXOESctXnOEpyhM7vD4xavViusy+tSzTBSm3yKzT1gpYoeuqdLeqY7uqJHevu1VjGqUfKyx7NR1go/lzjqXHj9V+XwLLH9qfrTs8QmRiOvFnv3I6Z0C7OsL+wfPy+MzfcUe+mMntj/KT3QLd/ALbyY53Ni/gRx/gDt+3P/BEtDaY3S2hz/xCTKUY8udKOP33sEE5jBLDJ87gHOcIXr2LuSUgaUwXKqEqto2vEllOEPNlmTyw==</latexit> <latexit sha1_base64="H9PCfw1UrI5/SZpZDdMCzd8QFK8=">AAACe3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuIxRSX5sZXKBTEGShoVFgrxJRkpJYkasYLKBvoGYCBAibDEMpQdnCbLrxP6em9gHyB5QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IXXzSUMaQwWYLdmAt1eABYB+SIZor+savrnYKsg1Wo1g0UGr4HuX2hw0+Aw0Ad5ZV+SlwamBs1m4AJGgCF6cGMywoz0DA30DAOBMeHEAAEcDNIMSgwawPA2Z3Bg8GAIYAgF2lvDsIhhNcMaxr9MykxaTDoQpUyMUD3CDCiAyRQApPyVrg==</latexit> <latexit sha1_base64="H9PCfw1UrI5/SZpZDdMCzd8QFK8=">AAACe3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuIxRSX5sZXKBTEGShoVFgrxJRkpJYkasYLKBvoGYCBAibDEMpQdnCbLrxP6em9gHyB5QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IXXzSUMaQwWYLdmAt1eABYB+SIZor+savrnYKsg1Wo1g0UGr4HuX2hw0+Aw0Ad5ZV+SlwamBs1m4AJGgCF6cGMywoz0DA30DAOBMeHEAAEcDNIMSgwawPA2Z3Bg8GAIYAgF2lvDsIhhNcMaxr9MykxaTDoQpUyMUD3CDCiAyRQApPyVrg==</latexit> <latexit sha1_base64="2XjVx2/Q0RR5744YaXto0PhCGiY=">AAACe3ichVHLLgRBFD3T3uM1SERi0zEhiExuS4SwETaWXoPEMOluNXT0K901Eww/4AcsrEhEhL+w8QMWPkEsSWxI3OnpRBDcSlWdOnXPrVNVhm9boSR6SCg1tXX1DY1NyeaW1rb2VEfnSugVA1NkTc/2gjVDD4VtuSIrLWmLNT8QumPYYtXYna3sr5ZEEFqeuyz3fbHh6NuuVbBMXTKVT3XnwqKT31P9TVIH96bUnNwRUh/Kp9KUoSjUn0CLQRpxzHupS+SwBQ8minAg4EIytqEj5LYODQSfuQ2UmQsYWdG+wBGSrC1yluAMndldHrd5tR6zLq8rNcNIbfIpNveAlSr66Z6u6Jnu6Joe6e3XWuWoRsXLPs9GVSv8fPtxz9LrvyqHZ4mdT9WfniUKmIi8Wuzdj5jKLcyqvnRw8rw0udhfHqBzemL/Z/RAt3wDt/RiXiyIxVMk+QO078/9E6yMZjTKaAuUnp6Jv6IRvejDIL/3OKYxh3lk+dxDnOMaN4l3Ja0MKyPVVCURa7rwJZSxD6hKkds=</latexit> Z p0(x; ✓)dx <latexit sha1_base64="WNLv6W7OFBixBnjBW4pOz/mIEdo=">AAAChXichVG7ThtBFD1sQjCGBAcaJJoVCARFrLtRICgUWNBQAsbYEgZrdxnwyPvS7tiysWgp+AEKKpCiCCVt+AEafoDCn4BSEomGguu1JURQkjuamTNn7rlzZsYKHBkpolaP9up175u+RH9yYPDtu6HU++HNyK+GtsjZvuOHBcuMhCM9kVNSOaIQhMJ0LUfkrcpyez9fE2EkfW9DNQKx7Zr7ntyTtqmYKqX0ovSUHuyQPl1f0IuqLJQ5oxddU5VDt7l7WC+lJihNcegvgdEFE5nF9GwGwKqf+oYiduHDRhUuBDwoxg5MRNy2YIAQMLeNJnMhIxnvCxwiydoqZwnOMJmt8LjPq60u6/G6XTOK1Taf4nAPWaljkm7ogu7omr7TLT38tVYzrtH20uDZ6mhFUBo6Hs3e/1fl8qxQflL907PCHuZjr5K9BzHTvoXd0dcOTu6yX9Ynm1N0Tr/Y/xm16Ipv4NV+21/XxPopkvwBxp/P/RJsfkwblDbW+CeW0IkExjCOaX7vz8hgBavI8blH+IGfuNT6tA/aJ22uk6r1dDUjeBba4iNTfZZa</latexit> <latexit sha1_base64="BTMM6AXpgMIWmsAOwd8cGDdlNys=">AAAChXichVFNL2RBFD0eBs2MxkZi80KIWUznPvEVFi1sLH01Em067z1FV7yvvFfdQbO1sLOysCIRkZktf8DGH7DwE0SsSGws3H7diRiZmVupqlOn7rl1qsoKHBkporsarbau/ktDY1OiueXrt9ZkW/ti5BdCW2Rs3/HDZcuMhCM9kVFSOWI5CIXpWo5YsjanyvtLRRFG0vcW1HYgVl1zw5Pr0jYVU7mknpWe0oOfpPdvjetZlRfK/K5nXVPlQ7e0treVS/ZQiuLQPwOjCnom0qmh9O7j4YyfPEcWa/BhowAXAh4UYwcmIm4rMEAImFtFibmQkYz3BfaQYG2BswRnmMxu8rjBq5Uq6/G6XDOK1Taf4nAPWamjl27pgp7ohn7RPb3+tVYprlH2ss2zVdGKINd60Dn/8l+Vy7NC/l31T88K6xiNvUr2HsRM+RZ2RV/cOXqaH5vrLfXRKT2w/xO6o2u+gVd8ts9mxdwxEvwBxp/P/RksDqQMShmz/BOTqEQjutCNfn7vEUxgGjPI8Ln7+I1LXGkN2g9tUBuupGo1VU0HPoSWfgPq1ZhT</latexit> <latexit sha1_base64="BTMM6AXpgMIWmsAOwd8cGDdlNys=">AAAChXichVFNL2RBFD0eBs2MxkZi80KIWUznPvEVFi1sLH01Em067z1FV7yvvFfdQbO1sLOysCIRkZktf8DGH7DwE0SsSGws3H7diRiZmVupqlOn7rl1qsoKHBkporsarbau/ktDY1OiueXrt9ZkW/ti5BdCW2Rs3/HDZcuMhCM9kVFSOWI5CIXpWo5YsjanyvtLRRFG0vcW1HYgVl1zw5Pr0jYVU7mknpWe0oOfpPdvjetZlRfK/K5nXVPlQ7e0treVS/ZQiuLQPwOjCnom0qmh9O7j4YyfPEcWa/BhowAXAh4UYwcmIm4rMEAImFtFibmQkYz3BfaQYG2BswRnmMxu8rjBq5Uq6/G6XDOK1Taf4nAPWamjl27pgp7ohn7RPb3+tVYprlH2ss2zVdGKINd60Dn/8l+Vy7NC/l31T88K6xiNvUr2HsRM+RZ2RV/cOXqaH5vrLfXRKT2w/xO6o2u+gVd8ts9mxdwxEvwBxp/P/RksDqQMShmz/BOTqEQjutCNfn7vEUxgGjPI8Ln7+I1LXGkN2g9tUBuupGo1VU0HPoSWfgPq1ZhT</latexit> <latexit sha1_base64="wYtk8KKaebSYh98G52k+qXbQykU=">AAAChXichVG7ThtBFD1sABuHh5M0kWhWWESmwLqLEERJAQoNJY/YIGFj7S4DHrEv7Y4tG4s2RX4gBRVIKIqghR+g4QdS+BOilERKkyLX65UQIOCOZubMmXvunJmxAkdGiqjTp73oHxhMpYcyL4dHRseyr16XIr8e2qJo+44fblpmJBzpiaKSyhGbQShM13LEhrW/1N3faIgwkr73WbUCUXHNPU/uSttUTFWzell6Sg+2Sc83P+plVRPKnNLLrqlqodveOWxWszkqUBz6Q2AkIIckVvzsd5SxAx826nAh4EExdmAi4rYFA4SAuQrazIWMZLwvcIgMa+ucJTjDZHafxz1ebSWsx+tuzShW23yKwz1kpY5J+kk/6Iau6Yx+0b9Ha7XjGl0vLZ6tnlYE1bGvb9f/PqtyeVao3aqe9Kywi/exV8neg5jp3sLu6RsH327WP6xNtt/RCf1m/8fUoSu+gdf4Y5+uirUjZPgDjPvP/RCUZgoGFYxVyi1+Sr4ijXFMIM/vPY9FLGMFRT73C85xgUstpU1rs9pcL1XrSzRvcCe0hf8+6ZU4</latexit> Unnormalized model: s.t. Z p0(x; ✓)dx 6= 1 <latexit sha1_base64="Xen8d6aBoVjqw8xtsIG75vP5hoM=">AAACinichVG7ThtBFD1sCBDzMqGJRLPCcgSNdRch8WpQSEHJIwYklli7yxiP2Be7Ywti8QPwASmoQEIIUdFCSZMfoOATECVINBS5Xq8UBQTc0cycOXPPnTMzdujKWBHdtGgfWj+2tXd8ynR2dff0Zvs+L8VBNXJE0QncIFqxrVi40hdFJZUrVsJIWJ7timV7c6axv1wTUSwD/4faCcWaZ234siwdSzFVyuZN6Ss9/ElD21OmqghlDeumZ6lK5NXXd7d10xdbulHK5qhASegvgZGCHNKYC7InMLGOAA6q8CDgQzF2YSHmtgoDhJC5NdSZixjJZF9gFxnWVjlLcIbF7CaPG7xaTVmf142acaJ2+BSXe8RKHXm6plO6pz90Rrf09GqtelKj4WWHZ7upFWGpd+/L4uO7Ko9nhco/1ZueFcoYT7xK9h4mTOMWTlNf+/X7fnFyIV//Skd0x/4P6Yau+AZ+7cE5nhcLB8jwBxjPn/slWBopGFQw5kdz09/Sr+jAAAYxxO89hmnMYg5FPncf57jApdaljWgT2lQzVWtJNf34L7TvfwHnhpc/</latexit> <latexit sha1_base64="Xen8d6aBoVjqw8xtsIG75vP5hoM=">AAACinichVG7ThtBFD1sCBDzMqGJRLPCcgSNdRch8WpQSEHJIwYklli7yxiP2Be7Ywti8QPwASmoQEIIUdFCSZMfoOATECVINBS5Xq8UBQTc0cycOXPPnTMzdujKWBHdtGgfWj+2tXd8ynR2dff0Zvs+L8VBNXJE0QncIFqxrVi40hdFJZUrVsJIWJ7timV7c6axv1wTUSwD/4faCcWaZ234siwdSzFVyuZN6Ss9/ElD21OmqghlDeumZ6lK5NXXd7d10xdbulHK5qhASegvgZGCHNKYC7InMLGOAA6q8CDgQzF2YSHmtgoDhJC5NdSZixjJZF9gFxnWVjlLcIbF7CaPG7xaTVmf142acaJ2+BSXe8RKHXm6plO6pz90Rrf09GqtelKj4WWHZ7upFWGpd+/L4uO7Ko9nhco/1ZueFcoYT7xK9h4mTOMWTlNf+/X7fnFyIV//Skd0x/4P6Yau+AZ+7cE5nhcLB8jwBxjPn/slWBopGFQw5kdz09/Sr+jAAAYxxO89hmnMYg5FPncf57jApdaljWgT2lQzVWtJNf34L7TvfwHnhpc/</latexit> <latexit sha1_base64="Xen8d6aBoVjqw8xtsIG75vP5hoM=">AAACinichVG7ThtBFD1sCBDzMqGJRLPCcgSNdRch8WpQSEHJIwYklli7yxiP2Be7Ywti8QPwASmoQEIIUdFCSZMfoOATECVINBS5Xq8UBQTc0cycOXPPnTMzdujKWBHdtGgfWj+2tXd8ynR2dff0Zvs+L8VBNXJE0QncIFqxrVi40hdFJZUrVsJIWJ7timV7c6axv1wTUSwD/4faCcWaZ234siwdSzFVyuZN6Ss9/ElD21OmqghlDeumZ6lK5NXXd7d10xdbulHK5qhASegvgZGCHNKYC7InMLGOAA6q8CDgQzF2YSHmtgoDhJC5NdSZixjJZF9gFxnWVjlLcIbF7CaPG7xaTVmf142acaJ2+BSXe8RKHXm6plO6pz90Rrf09GqtelKj4WWHZ7upFWGpd+/L4uO7Ko9nhco/1ZueFcoYT7xK9h4mTOMWTlNf+/X7fnFyIV//Skd0x/4P6Yau+AZ+7cE5nhcLB8jwBxjPn/slWBopGFQw5kdz09/Sr+jAAAYxxO89hmnMYg5FPncf57jApdaljWgT2lQzVWtJNf34L7TvfwHnhpc/</latexit> <latexit sha1_base64="Xen8d6aBoVjqw8xtsIG75vP5hoM=">AAACinichVG7ThtBFD1sCBDzMqGJRLPCcgSNdRch8WpQSEHJIwYklli7yxiP2Be7Ywti8QPwASmoQEIIUdFCSZMfoOATECVINBS5Xq8UBQTc0cycOXPPnTMzdujKWBHdtGgfWj+2tXd8ynR2dff0Zvs+L8VBNXJE0QncIFqxrVi40hdFJZUrVsJIWJ7timV7c6axv1wTUSwD/4faCcWaZ234siwdSzFVyuZN6Ss9/ElD21OmqghlDeumZ6lK5NXXd7d10xdbulHK5qhASegvgZGCHNKYC7InMLGOAA6q8CDgQzF2YSHmtgoDhJC5NdSZixjJZF9gFxnWVjlLcIbF7CaPG7xaTVmf142acaJ2+BSXe8RKHXm6plO6pz90Rrf09GqtelKj4WWHZ7upFWGpd+/L4uO7Ko9nhco/1ZueFcoYT7xK9h4mTOMWTlNf+/X7fnFyIV//Skd0x/4P6Yau+AZ+7cE5nhcLB8jwBxjPn/slWBopGFQw5kdz09/Sr+jAAAYxxO89hmnMYg5FPncf57jApdaljWgT2lQzVWtJNf34L7TvfwHnhpc/</latexit>
  4. Intuition of Noise Contrastive Estimation (NCE) Solving binary classification problem!

    4 %BUBTBNQMFT /PJTFTBNQMFT data? or noise? Logistic Regression It can be considered as a fixed generator and a trainable discriminator in the GAN game.
  5. Overview of NCE 1. Replace an expensive partition function with

    
 a learnable parameter in a trained model 2. Sample noise data from a noise distribution 3. Classify between observed data and noise data 5
  6. Overview of NCE 1. Replace an expensive partition function with

    
 a learnable parameter in a trained model 2. Sample noise data from a noise distribution 3. Classify between observed data and noise data 6
  7. Replace a partition function with a learnable parameter 7 is

    a learnable parameter instead of the partition function c <latexit sha1_base64="dqnNk2e//D8T2u6qA6gageMEH+k=">AAACZHichVFNS8NAFJzGr1qrrRZBEEQsiqfy4kURxKIXj9ZaLdRSkrhqaJqEJC3UInjWq+LBk4KI+DM86B/w0D8giEcFLx58TQuiRX3L7s7Ovnk7u6vahu56RPWA1NHZ1d0T7A31hfsHItHBoQ3XKjuayGiWYTlZVXGFoZsi4+meIbK2I5SSaohNtbjc2N+sCMfVLXPdq9oiX1J2TX1H1xSPqZRWiMYpQX6MtwO5BeKL96GFQwCrVvQaW9iGBQ1llCBgwmNsQIHLLQcZBJu5PGrMOYx0f1/gACHWljlLcIbCbJHHXV7lWqzJ60ZN11drfIrB3WHlOCbpkW7olR7olp7p49daNb9Gw0uVZ7WpFXYhcjSSfv9XVeLZw96X6k/PHnYw53vV2bvtM41baE19Zf/sNT2/Nlmbokt6Yf8XVKc7voFZedOuUmLtHCH+APnnc7eDjZmETAk5RfHkEpoRxCgmMM3vPYskVrCKDJ8rcIwTnAaepLAUk4abqVKgpYnhW0hjnxmui6w=</latexit> <latexit sha1_base64="LQr8q1wW7eSqPsXuz5Dptsecmko=">AAACZHichVFdKwRRGH52fK2x7CKllGQjV9s7bkjJxo1Li0WhbWacXZP5amZ2i80f4JZcuEKS/AwX/AHJH1BySblx4Z3ZLSG8p3POc57zPu95zjmaaxp+QPQQkxoam5pb4q1yW6K9I5nq7FrynbKni7zumI63oqm+MA1b5AMjMMWK6wnV0kyxrG3NhPvLFeH5hmMvBtuuWLfUkm0UDV0NmMrphVSaMhTFwE+g1EF66kaedM/u5TkndYE1bMCBjjIsCNgIGJtQ4XNbhQKCy9w6qsx5jIxoX2AXMmvLnCU4Q2V2i8cSr1brrM3rsKYfqXU+xeTusXIAQ3RHl/RCt3RFT/T+a61qVCP0ss2zVtMKt5Dc6114+1dl8Rxg81P1p+cARYxHXg327kZMeAu9pq/sHL0sTMwPVYfplJ7Z/wk90DXfwK686uc5MX8MmT9A+f7cP8HSaEahjJKjdHYatYijD4MY4fceQxazmEOezxXYxwEOY49SQuqWemqpUqyu6caXkPo/AL9tjSA=</latexit> <latexit sha1_base64="LQr8q1wW7eSqPsXuz5Dptsecmko=">AAACZHichVFdKwRRGH52fK2x7CKllGQjV9s7bkjJxo1Li0WhbWacXZP5amZ2i80f4JZcuEKS/AwX/AHJH1BySblx4Z3ZLSG8p3POc57zPu95zjmaaxp+QPQQkxoam5pb4q1yW6K9I5nq7FrynbKni7zumI63oqm+MA1b5AMjMMWK6wnV0kyxrG3NhPvLFeH5hmMvBtuuWLfUkm0UDV0NmMrphVSaMhTFwE+g1EF66kaedM/u5TkndYE1bMCBjjIsCNgIGJtQ4XNbhQKCy9w6qsx5jIxoX2AXMmvLnCU4Q2V2i8cSr1brrM3rsKYfqXU+xeTusXIAQ3RHl/RCt3RFT/T+a61qVCP0ss2zVtMKt5Dc6114+1dl8Rxg81P1p+cARYxHXg327kZMeAu9pq/sHL0sTMwPVYfplJ7Z/wk90DXfwK686uc5MX8MmT9A+f7cP8HSaEahjJKjdHYatYijD4MY4fceQxazmEOezxXYxwEOY49SQuqWemqpUqyu6caXkPo/AL9tjSA=</latexit> <latexit sha1_base64="dxmZ8Hqd0XIx3n6Mwtt8nkpiKvQ=">AAACZHichVG7SgNBFD1Z3/GRqAiCIMEQsQp3bRSroI2lMeYBUWR3neiSfbG7CcTgD2irWFgpiIifYeMPWOQHBLGMYGPhzWZBVNQ7zMyZM/fcOTOjOobu+UStiNTT29c/MDgUHR4ZHYvFxycKnl1zNZHXbMN2S6riCUO3RN7XfUOUHFcopmqIolpd6+wX68L1dNva8huO2DGVfUuv6JriM5XVduNJSlMQiZ9ADkESYWzY8RtsYw82NNRgQsCCz9iAAo9bGTIIDnM7aDLnMtKDfYEjRFlb4yzBGQqzVR73eVUOWYvXnZpeoNb4FIO7y8oEUvRIt9SmB7qjZ3r/tVYzqNHx0uBZ7WqFsxs7ns69/asyefZx8Kn607OPCpYDrzp7dwKmcwutq68fnrdzK5up5jxd0Qv7v6QW3fMNrPqrdp0VmxeI8gfI35/7JygspmVKy1lKZlbDrxjEDOawwO+9hAzWsYE8nytwglOcRZ6kEWlSmuqmSpFQM4kvIc1+AMsMid8=</latexit> Ex: 1D Gaussian model Original: NCE: pm(x; µ, ) = 1 p 2⇡ 2 exp (x µ)2 2 2 <latexit sha1_base64="VSlY5Qjqb/hIP4PkB8BlzrqPZx0=">AAAC2nichZFNa9RAGMf/iW91q3bVi+BlsFR2wS6TvSiKsOjFY1/cttCpaxJn16F5GZPJsnXYizf1KnjwpCAifgwvfgA9FD+BeKzgxYNPsitii/qEZJ75z//3zDOZQEcqN5zvOu6hw0eOHps5Xps9cfLUXP30mbU8LbJQdsM0SrONwM9lpBLZNcpEckNn0o+DSK4H2zfL9fWhzHKVJrfNjpZbsT9IVF+FviGpVy90L26M2DUm4uISE7kaxH6TsetW9DM/ZNYbW5E/yAyzbaHV1MDu2PaYggk50kwEatCwi7+IxmiRarFm5SFqH0HmZq1Xn+ctXgU7mHjTZL7T+ajvAlhK628gcA8pQhSIIZHAUB7BR07PJjxwaNK2YEnLKFPVusQYNWILckly+KRu03dAs82pmtC8rJlXdEi7RPRmRDIs8E/8Ld/jH/g7/oX/+GstW9Uoe9mhMZiwUvfmnpxb/f5fKqbR4P5v6p89G/RxpepVUe+6UspThBN++PD53urVlQV7kb/iX6n/l3yXv6cTJMNv4etlufIC5QV4+3/3wWSt3fJ4y1umm7iBSczgPC6gQf/7Mjq4hSV0ad/PjuPUnFlXuI/cx+7TidV1psxZ/BHus5+6BbUk</latexit> <latexit sha1_base64="vc6E7/f7WSYPE/Kug47h1XxApFk=">AAAC2nichZG/axRBFMffrr+Si5rTNAGbwZBwB+Z4e41BEY7YWOaHlwQy8dhd5y5D9sdkd/a4ZLgmXbQVLKwURMQ/w8ZatAj5C4JlhDQp8nbvRExQ37I7b77z/bx5s+OpQKYa8cCyL12+cvXayGhp7PqNm+PlW7dX0jhLfNH04yBO1jw3FYGMRFNLHYg1lQg39AKx6m09ztdXuyJJZRw91TtKbIRuJ5Jt6buapFY5U62w0mMPGQ+ze4ynshO6VcYeGd5OXJ8Zp294up1oZupcyaGBPTP1PgXjoqcY92SnYmZ/EZXeLNVi1cJD1DmCzNVSqzyFNSyCXUycYTLVaHxTfD/8uhCXPwCH5xCDDxmEICACTXkALqT0rIMDCIq0DTCkJZTJYl1AH0rEZuQS5HBJ3aJvh2brQzWieV4zLWifdgnoTYhkMI3f8SMe4xf8hEd4+tdapqiR97JDozdghWqNv5hcPvkvFdKoYfM39c+eNbRhruhVUu+qUPJT+AO+u/v6ePnB0rSZwXf4g/p/iwf4mU4QdX/67xfF0hvIL8A5/7svJiv1moM1Z5FuYh4GMQJ34C5U6H/fhwY8gQVo0r6HlmWVrDGb23v2vv1yYLWtITMBf4T96gzwqbbo</latexit> <latexit sha1_base64="vc6E7/f7WSYPE/Kug47h1XxApFk=">AAAC2nichZG/axRBFMffrr+Si5rTNAGbwZBwB+Z4e41BEY7YWOaHlwQy8dhd5y5D9sdkd/a4ZLgmXbQVLKwURMQ/w8ZatAj5C4JlhDQp8nbvRExQ37I7b77z/bx5s+OpQKYa8cCyL12+cvXayGhp7PqNm+PlW7dX0jhLfNH04yBO1jw3FYGMRFNLHYg1lQg39AKx6m09ztdXuyJJZRw91TtKbIRuJ5Jt6buapFY5U62w0mMPGQ+ze4ynshO6VcYeGd5OXJ8Zp294up1oZupcyaGBPTP1PgXjoqcY92SnYmZ/EZXeLNVi1cJD1DmCzNVSqzyFNSyCXUycYTLVaHxTfD/8uhCXPwCH5xCDDxmEICACTXkALqT0rIMDCIq0DTCkJZTJYl1AH0rEZuQS5HBJ3aJvh2brQzWieV4zLWifdgnoTYhkMI3f8SMe4xf8hEd4+tdapqiR97JDozdghWqNv5hcPvkvFdKoYfM39c+eNbRhruhVUu+qUPJT+AO+u/v6ePnB0rSZwXf4g/p/iwf4mU4QdX/67xfF0hvIL8A5/7svJiv1moM1Z5FuYh4GMQJ34C5U6H/fhwY8gQVo0r6HlmWVrDGb23v2vv1yYLWtITMBf4T96gzwqbbo</latexit> <latexit sha1_base64="nh/59HsTMZFqmDxb3h0CAEJndIA=">AAAC2nichZHPa9RAFMdfYtW6Vbu1F8HL4FLZBbu87KVFKRS9eOwPty106pLE2XVofkyTybLtsJfetNeCB08KIsU/w4v/gIfiXyAeK/TSgy/ZiNiivpDMm+98P2/eZDwVyFQjHlv2pbHLV66OX6tMXL9xc7I6dWstjbPEF20/DuJkw3NTEchItLXUgdhQiXBDLxDr3vbjfH29L5JUxtFTvavEVuj2ItmVvqtJ6lQz1QnrA/aQ8TC7z3gqe6HbYGzB8G7i+sw4Q8PTnUQz0+JKlgb2zLSGFIyLgWLck726mf1F1AezVIs1Cg9R5wgyNyqdag2bWAS7mDhlUoMyluLqB+DwHGLwIYMQBESgKQ/AhZSeTXAAQZG2BYa0hDJZrAsYQoXYjFyCHC6p2/Tt0WyzVCOa5zXTgvZpl4DehEgGM/gFj/AEP+NH/IZnf61lihp5L7s0eiNWqM7kq9urp/+lQho1vPhN/bNnDV2YL3qV1LsqlPwU/ojv770+WX2wMmPu4Tv8Tv2/xWP8RCeI+j/898ti5Q3kF+Cc/90Xk7VW08Gms4y1xUflVYzDHbgLdfrfc7AIT2AJ2rTvV8uyKtaEze19+6V9MLLaVslMwx9hH/4EzCGzAA==</latexit> pm(x; c, µ, ) = 1 c exp (x µ)2 2 2 <latexit sha1_base64="8TaLo4nCxuio5CW5jRO+26vEj8M=">AAACxnichZFLaxRBEMf/GV9xfWTVi5BLY4jsgll69mJQhKCXHPNwk0AmLj1t79pkXsxj3dgMePYLePCkEELwY3jQoxcPOXkW8RQxlxxSM7MhaFB7mO6qf9WvurrbjTydpJzvjVlnzp47f2H8Yu3S5StXJ+rXrq8kYRZL1ZGhF8ZrrkiUpwPVSXXqqbUoVsJ3PbXqbj4q4qsDFSc6DB6nW5Ha8EU/0D0tRUpSty6irt8YsvtM3mGOn9GU6L4vmow9ME4vFpIZOzcyz5mjhhFzXN1vmJnjUGM4QxBrPjHtPDftimWlVxCU3Kx161O8xcvBThv2yJiam/36mQFYCOs7cPAUISQy+FAIkJLtQSChbx02OCLSNmBIi8nSZVwhR43YjLIUZQhSN2nuk7c+UgPyi5pJSUvaxaM/JpJhmn/hu3yff+Lv+Td++NdapqxR9LJFq1uxKupOvLq5fPBfyqc1xbMT6p89p+hhtuxVU+9RqRSnkBU/ePF6f/ne0rS5zd/x79T/W77HP9AJgsFPub2olt6geAD7z+s+bay0WzZv2Yv0Eg9RjXFM4hYadN93MYd5LKBD+37ED/zCgTVvBVZmPa9SrbERcwO/DevlEdPWr7Q=</latexit> <latexit sha1_base64="rt302ArbxwqnOirDpZtry4NzGNg=">AAACxnichZFLaxRBEMf/GV9xfWTVg4qXxhDZBbP07MWgBEK85JiHmwQycelpe9cm82Ie6ybNgGe/QA6eFETEj+FBj148xItXRTxFzMWDNTMrokHtYbqr/lW/6upuN/J0knK+N2YdOXrs+Inxk7VTp8+cnaifO7+ahFksVUeGXhivuyJRng5UJ9Wpp9ajWAnf9dSau3W7iK8NVJzoMLiTbkdq0xf9QPe0FClJ3bqIun5jyG4xeZ05fkZTovu+aDI2a5xeLCQzdm5knjNHDSPmuLrfMNM/Q43hNEGsede089y0K5aVXkFQcrPWrU/yFi8HO2zYI2NybubD24vvP15eDOvP4eAeQkhk8KEQICXbg0BC3wZscESkbcKQFpOly7hCjhqxGWUpyhCkbtHcJ29jpAbkFzWTkpa0i0d/TCTDFH/HX/B9/oa/5J/497/WMmWNopdtWt2KVVF34tGllYP/Uj6tKe7/ov7Zc4oeZspeNfUelUpxClnxg53d/ZWby1PmGn/KP1P/T/gef0UnCAZf5bMltfwYxQPYf173YWO13bJ5y16il5hHNcZxBVfRoPu+gTksYBEd2vc1vuAbDqwFK7Ay60GVao2NmAv4bVgPfwAlBbGG</latexit> <latexit sha1_base64="rt302ArbxwqnOirDpZtry4NzGNg=">AAACxnichZFLaxRBEMf/GV9xfWTVg4qXxhDZBbP07MWgBEK85JiHmwQycelpe9cm82Ie6ybNgGe/QA6eFETEj+FBj148xItXRTxFzMWDNTMrokHtYbqr/lW/6upuN/J0knK+N2YdOXrs+Inxk7VTp8+cnaifO7+ahFksVUeGXhivuyJRng5UJ9Wpp9ajWAnf9dSau3W7iK8NVJzoMLiTbkdq0xf9QPe0FClJ3bqIun5jyG4xeZ05fkZTovu+aDI2a5xeLCQzdm5knjNHDSPmuLrfMNM/Q43hNEGsede089y0K5aVXkFQcrPWrU/yFi8HO2zYI2NybubD24vvP15eDOvP4eAeQkhk8KEQICXbg0BC3wZscESkbcKQFpOly7hCjhqxGWUpyhCkbtHcJ29jpAbkFzWTkpa0i0d/TCTDFH/HX/B9/oa/5J/497/WMmWNopdtWt2KVVF34tGllYP/Uj6tKe7/ov7Zc4oeZspeNfUelUpxClnxg53d/ZWby1PmGn/KP1P/T/gef0UnCAZf5bMltfwYxQPYf173YWO13bJ5y16il5hHNcZxBVfRoPu+gTksYBEd2vc1vuAbDqwFK7Ay60GVao2NmAv4bVgPfwAlBbGG</latexit> <latexit sha1_base64="B0ExEql2Vc5wEZ8QFOvDSkQ05K8=">AAACxnichZFLaxRBEMf/GV9xfWTVi+ClcYnsgllq9qIoQtBLjnm4SSATl5m2d9NkXsxj3dgMePYLePCkICJ+DA/6BTzkI4h4ipiLB2tmR0SD2sN0V/2rftXV3V7s6zQj2p+xjh0/cfLU7OnGmbPnzs81L1xcT6M8kaovIz9KNj03Vb4OVT/Tma8240S5geerDW/3XhnfGKsk1VF4P9uL1XbgjkI91NLNWBo03XgQtCfitpDXhRPkPKV6FLgdIe4YZ5i4Uhi7MLIohKMmsXA8PWqbhZ+h9mSBIdF5YHpFYXpTVlReSXBypzFotqhL1RBHDbs2WqjHctR8DQcPEUEiRwCFEBnbPlyk/G3BBiFmbRuGtYQtXcUVCjSYzTlLcYbL6i7PI/a2ajVkv6yZVrTkXXz+EyYF5ukjvaED+kBv6RN9/2stU9Uoe9nj1ZuyKh7MPb28dvhfKuA1w84v6p89ZxjiZtWr5t7jSilPIaf8+PGzg7Vbq/PmGr2kz9z/C9qnd3yCcPxVvlpRq89RPoD953UfNdZ7XZu69gq1Fu/WTzGLK7iKNt/3DSxiCcvo877v8QXfcGgtWaGVW4+mqdZMzVzCb8N68gOlmK1t</latexit> pm(x; ✓) = 1 Z(✓) p0 m (x; ✓) <latexit sha1_base64="KZAh7y0V0qe3/C6DUlQ3EjB9tIY=">AAAC3XichVFLLwRBEK4Z7/VaXCQuHZsVLpsakRAiES6OXouwTGZaLxPzykzvBps9uoi4OjiRiIif4eIPOLi4iyOJi4Pa2Qlhg5pM99df1Vf9dbfp21YoER8Uta6+obGpuSXR2tbe0Zns6l4OvULARZZ7thesmkYobMsVWWlJW6z6gTAc0xYr5u5MJb9SFEFoee6S3PfFhmNsu1be4oYkSk/u+brDBtkem2A5uSOkwYbYJMvlA4OzEtPKNKxR/jNXZj7bJBLLtbo0+y7kf1XryRRmMApWC7QYpCCOOS95BTnYAg84FMABAS5IwjYYENK3Dhog+MRtQIm4gJAV5QWUIUHaAlUJqjCI3aVxm1brMevSutIzjNScdrHpD0jJII33eI0veIc3+ITvv/YqRT0qXvZpNqta4eudR72Lb/+qHJol7Hyp/vQsIQ9jkVeLvPsRUzkFr+qLB6cvi+ML6dIAXuAz+T/HB7ylE7jFV345LxbOIEEPoP287lqwPJzRMKPNj6SmpuOnaIY+6IdBuu9RmIJZmIMs7fuo1CvtSoeqq4fqsXpSLVWVWNMD30I9/QA8fKyx</latexit> <latexit sha1_base64="KZAh7y0V0qe3/C6DUlQ3EjB9tIY=">AAAC3XichVFLLwRBEK4Z7/VaXCQuHZsVLpsakRAiES6OXouwTGZaLxPzykzvBps9uoi4OjiRiIif4eIPOLi4iyOJi4Pa2Qlhg5pM99df1Vf9dbfp21YoER8Uta6+obGpuSXR2tbe0Zns6l4OvULARZZ7thesmkYobMsVWWlJW6z6gTAc0xYr5u5MJb9SFEFoee6S3PfFhmNsu1be4oYkSk/u+brDBtkem2A5uSOkwYbYJMvlA4OzEtPKNKxR/jNXZj7bJBLLtbo0+y7kf1XryRRmMApWC7QYpCCOOS95BTnYAg84FMABAS5IwjYYENK3Dhog+MRtQIm4gJAV5QWUIUHaAlUJqjCI3aVxm1brMevSutIzjNScdrHpD0jJII33eI0veIc3+ITvv/YqRT0qXvZpNqta4eudR72Lb/+qHJol7Hyp/vQsIQ9jkVeLvPsRUzkFr+qLB6cvi+ML6dIAXuAz+T/HB7ylE7jFV345LxbOIEEPoP287lqwPJzRMKPNj6SmpuOnaIY+6IdBuu9RmIJZmIMs7fuo1CvtSoeqq4fqsXpSLVWVWNMD30I9/QA8fKyx</latexit> <latexit sha1_base64="KZAh7y0V0qe3/C6DUlQ3EjB9tIY=">AAAC3XichVFLLwRBEK4Z7/VaXCQuHZsVLpsakRAiES6OXouwTGZaLxPzykzvBps9uoi4OjiRiIif4eIPOLi4iyOJi4Pa2Qlhg5pM99df1Vf9dbfp21YoER8Uta6+obGpuSXR2tbe0Zns6l4OvULARZZ7thesmkYobMsVWWlJW6z6gTAc0xYr5u5MJb9SFEFoee6S3PfFhmNsu1be4oYkSk/u+brDBtkem2A5uSOkwYbYJMvlA4OzEtPKNKxR/jNXZj7bJBLLtbo0+y7kf1XryRRmMApWC7QYpCCOOS95BTnYAg84FMABAS5IwjYYENK3Dhog+MRtQIm4gJAV5QWUIUHaAlUJqjCI3aVxm1brMevSutIzjNScdrHpD0jJII33eI0veIc3+ITvv/YqRT0qXvZpNqta4eudR72Lb/+qHJol7Hyp/vQsIQ9jkVeLvPsRUzkFr+qLB6cvi+ML6dIAXuAz+T/HB7ylE7jFV345LxbOIEEPoP287lqwPJzRMKPNj6SmpuOnaIY+6IdBuu9RmIJZmIMs7fuo1CvtSoeqq4fqsXpSLVWVWNMD30I9/QA8fKyx</latexit> <latexit sha1_base64="KZAh7y0V0qe3/C6DUlQ3EjB9tIY=">AAAC3XichVFLLwRBEK4Z7/VaXCQuHZsVLpsakRAiES6OXouwTGZaLxPzykzvBps9uoi4OjiRiIif4eIPOLi4iyOJi4Pa2Qlhg5pM99df1Vf9dbfp21YoER8Uta6+obGpuSXR2tbe0Zns6l4OvULARZZ7thesmkYobMsVWWlJW6z6gTAc0xYr5u5MJb9SFEFoee6S3PfFhmNsu1be4oYkSk/u+brDBtkem2A5uSOkwYbYJMvlA4OzEtPKNKxR/jNXZj7bJBLLtbo0+y7kf1XryRRmMApWC7QYpCCOOS95BTnYAg84FMABAS5IwjYYENK3Dhog+MRtQIm4gJAV5QWUIUHaAlUJqjCI3aVxm1brMevSutIzjNScdrHpD0jJII33eI0veIc3+ITvv/YqRT0qXvZpNqta4eudR72Lb/+qHJol7Hyp/vQsIQ9jkVeLvPsRUzkFr+qLB6cvi+ML6dIAXuAz+T/HB7ylE7jFV345LxbOIEEPoP287lqwPJzRMKPNj6SmpuOnaIY+6IdBuu9RmIJZmIMs7fuo1CvtSoeqq4fqsXpSLVWVWNMD30I9/QA8fKyx</latexit> pm(x; ✓) = 1 c p0 m (x; ✓) <latexit sha1_base64="ASnzXrfWOxpQ/pmoxwguySivJ8Y=">AAACpXichVFNSxtBGH6yrV9p1bS9CF4Gg61ewrulYFEKQS+9CH4lCkaX3XGii/vF7iQ0DfkD/oEeerKgpfgzvPTQqwV/gvSo4KWHvtksSBuq7zAzzzzzPu88M+NEnptoosuc8ejxwODQ8Ej+ydPRsfHCs+fVJGzEUlVk6IXxlmMnynMDVdGu9tRWFCvbdzy16Rwudfc3mypO3DDY0K1I7fj2fuDWXWlrpqxCObJ8MSM+iAVR0wdK22JWvBOiVo9tKdrC7PAgRUdEYpcRdfqyrUKRSpSG6AdmBorIYiUsfEUNewgh0YAPhQCasQcbCbdtmCBEzO2gzVzMyE33FTrIs7bBWYozbGYPedzn1XbGBrzu1kxSteRTPO4xKwWm6YK+0TV9pzO6ot//rdVOa3S9tHh2eloVWeNHE+u3D6p8njUO7lT3etao423q1WXvUcp0byF7+ubHT9fr82vT7Zf0hX6x/2O6pHO+QdC8kSerau0z8vwB5r/P3Q+qr0smlczVN8XyYvYVw5jEFGb4vedQxnusoMLnnuIHLvDTeGUsGxtGtZdq5DLNC/wVhvUHufydiw==</latexit> <latexit sha1_base64="ASnzXrfWOxpQ/pmoxwguySivJ8Y=">AAACpXichVFNSxtBGH6yrV9p1bS9CF4Gg61ewrulYFEKQS+9CH4lCkaX3XGii/vF7iQ0DfkD/oEeerKgpfgzvPTQqwV/gvSo4KWHvtksSBuq7zAzzzzzPu88M+NEnptoosuc8ejxwODQ8Ej+ydPRsfHCs+fVJGzEUlVk6IXxlmMnynMDVdGu9tRWFCvbdzy16Rwudfc3mypO3DDY0K1I7fj2fuDWXWlrpqxCObJ8MSM+iAVR0wdK22JWvBOiVo9tKdrC7PAgRUdEYpcRdfqyrUKRSpSG6AdmBorIYiUsfEUNewgh0YAPhQCasQcbCbdtmCBEzO2gzVzMyE33FTrIs7bBWYozbGYPedzn1XbGBrzu1kxSteRTPO4xKwWm6YK+0TV9pzO6ot//rdVOa3S9tHh2eloVWeNHE+u3D6p8njUO7lT3etao423q1WXvUcp0byF7+ubHT9fr82vT7Zf0hX6x/2O6pHO+QdC8kSerau0z8vwB5r/P3Q+qr0smlczVN8XyYvYVw5jEFGb4vedQxnusoMLnnuIHLvDTeGUsGxtGtZdq5DLNC/wVhvUHufydiw==</latexit> <latexit sha1_base64="ASnzXrfWOxpQ/pmoxwguySivJ8Y=">AAACpXichVFNSxtBGH6yrV9p1bS9CF4Gg61ewrulYFEKQS+9CH4lCkaX3XGii/vF7iQ0DfkD/oEeerKgpfgzvPTQqwV/gvSo4KWHvtksSBuq7zAzzzzzPu88M+NEnptoosuc8ejxwODQ8Ej+ydPRsfHCs+fVJGzEUlVk6IXxlmMnynMDVdGu9tRWFCvbdzy16Rwudfc3mypO3DDY0K1I7fj2fuDWXWlrpqxCObJ8MSM+iAVR0wdK22JWvBOiVo9tKdrC7PAgRUdEYpcRdfqyrUKRSpSG6AdmBorIYiUsfEUNewgh0YAPhQCasQcbCbdtmCBEzO2gzVzMyE33FTrIs7bBWYozbGYPedzn1XbGBrzu1kxSteRTPO4xKwWm6YK+0TV9pzO6ot//rdVOa3S9tHh2eloVWeNHE+u3D6p8njUO7lT3etao423q1WXvUcp0byF7+ubHT9fr82vT7Zf0hX6x/2O6pHO+QdC8kSerau0z8vwB5r/P3Q+qr0smlczVN8XyYvYVw5jEFGb4vedQxnusoMLnnuIHLvDTeGUsGxtGtZdq5DLNC/wVhvUHufydiw==</latexit> <latexit sha1_base64="ASnzXrfWOxpQ/pmoxwguySivJ8Y=">AAACpXichVFNSxtBGH6yrV9p1bS9CF4Gg61ewrulYFEKQS+9CH4lCkaX3XGii/vF7iQ0DfkD/oEeerKgpfgzvPTQqwV/gvSo4KWHvtksSBuq7zAzzzzzPu88M+NEnptoosuc8ejxwODQ8Ej+ydPRsfHCs+fVJGzEUlVk6IXxlmMnynMDVdGu9tRWFCvbdzy16Rwudfc3mypO3DDY0K1I7fj2fuDWXWlrpqxCObJ8MSM+iAVR0wdK22JWvBOiVo9tKdrC7PAgRUdEYpcRdfqyrUKRSpSG6AdmBorIYiUsfEUNewgh0YAPhQCasQcbCbdtmCBEzO2gzVzMyE33FTrIs7bBWYozbGYPedzn1XbGBrzu1kxSteRTPO4xKwWm6YK+0TV9pzO6ot//rdVOa3S9tHh2eloVWeNHE+u3D6p8njUO7lT3etao423q1WXvUcp0byF7+ubHT9fr82vT7Zf0hX6x/2O6pHO+QdC8kSerau0z8vwB5r/P3Q+qr0smlczVN8XyYvYVw5jEFGb4vedQxnusoMLnnuIHLvDTeGUsGxtGtZdq5DLNC/wVhvUHufydiw==</latexit> Original: NCE:
  8. Overview of NCE 1. Replace an expensive partition function with

    
 a learnable parameter in a trained model 2. Sample noise data from a noise distribution 3. Classify between observed data and noise data 8
  9. Noise Distribution To classify between observed data and noise data,

    
 NCE’s noise distribution … • has analytical expression for a log PDF/PMF • can be sampled easily • is similar to observed data distribution in some aspect • Ex. Covariance structure for natural images • Ex. Unigram distribution in NLP 9
  10. Overview of NCE 1. Replace an expensive partition function with

    
 a learnable parameter in a trained model 2. Sample noise data from a noise distribution 3. Classify between observed data and noise data 10
  11. Classify between Observed Data and Noise Data Objective function (Bernoulli

    loss) 11 Td X t=1 ln[h(xt; ✓)] + Tn X t=1 ln[1 h(yt; ✓)] <latexit sha1_base64="G/6GZ3Rc5jIp4mNZtxr4wsWZ/zE=">AAAC1XichZHPSxtBFMe/2fqrUWtaL0IvQ0VRpOGtF0tLQRRKj/6KEZK47K6jWdxf7E5C45Kb9NBLjz14sYVSpH9GL/0HWvA/sPSo4EXBl03A1mj7hpl58533efNmxgpdJ1ZExxntXk9vX//A/ezg0PCDkdzDR+txUItsWbADN4g2LDOWruPLgnKUKzfCSJqe5cqitbvY2i/WZRQ7gb+mGqGseOaO72w7tqlYMnJhOa55RqJe6s3NRKwZidgSTW5l1xclURVT4g1ripUXoqyqUpliWlTEjOji/D84/WmKNm5Bjdw45Sk10e3oHWd8/tXJ5SGApSD3BWVsIYCNGjxI+FDsuzARcytBByFkrYKEtYg9J92XaCLLbI2jJEeYrO7yuMOrUkf1ed3KGae0zae43CMmBSboBx3RKX2nr/SLLu7MlaQ5WrU0eLbarAyNkXdjq+f/pTyeFarX1D9rVtjGs7RWh2sPU6V1C7vN1/c+nK4+X5lIJukT/eb6P9IxfeMb+PUz+/OyXDlAlj9Av/nc3c76bF6nvL7MP7GAtg3gMZ5git97DvN4jSUU+NyfuMj0Zvq0otbU9rW37VAt02FG8Zdp768A2SqwkQ==</latexit> <latexit sha1_base64="r/ccuq7XYw1XTNxAkIue4DeX7zQ=">AAAC1XichZHPSxtBFMdf1vor2ibaS8HLoCiW0vDWi6IIYqF4TNQYIYnL7jpJFvcXu5NgusSTePDSowcPoiBF/DO8eBcF/wPFo4IXBV82AbVp9Q0z8+Y77/PmzYzmmoYvEC8jUtuH9o7Oru5oT+/HT7F4X/+S75Q9nad1x3S8ZU31uWnYPC0MYfJl1+OqpZk8o639qO9nKtzzDcdeFFWX5y21aBsFQ1cFSUrczfllSwnEtFxbCdiiErBVVqOWM22WZSU2ytZJE6RMsZwocaGyryzPvrEWzn7Byd9DtPoPVIkPYQJDY62O3HSGZn5ePe6dORtJJ/4HcrAKDuhQBgs42CDIN0EFn1oWZEBwSctDQJpHnhHuc6hBlNgyRXGKUEldo7FIq2xTtWldz+mHtE6nmNQ9IhkM4zke4S2e4jFe48N/cwVhjnotVZq1BstdJbb9ZeH+XcqiWUDpmXqzZgEFmAhrNah2N1Tqt9AbfOXXzu3C5PxwMIIHeEP17+MlntAN7Mqdfpji87sQpQ+Q/37uVmdpLCFjQk7RT8xCw7pgAAZhlN57HGZgDpKQpnMv4CHSHumQMlJN2pS2GqFSpMl8hlcm/X4CB2eyUA==</latexit> <latexit sha1_base64="r/ccuq7XYw1XTNxAkIue4DeX7zQ=">AAAC1XichZHPSxtBFMdf1vor2ibaS8HLoCiW0vDWi6IIYqF4TNQYIYnL7jpJFvcXu5NgusSTePDSowcPoiBF/DO8eBcF/wPFo4IXBV82AbVp9Q0z8+Y77/PmzYzmmoYvEC8jUtuH9o7Oru5oT+/HT7F4X/+S75Q9nad1x3S8ZU31uWnYPC0MYfJl1+OqpZk8o639qO9nKtzzDcdeFFWX5y21aBsFQ1cFSUrczfllSwnEtFxbCdiiErBVVqOWM22WZSU2ytZJE6RMsZwocaGyryzPvrEWzn7Byd9DtPoPVIkPYQJDY62O3HSGZn5ePe6dORtJJ/4HcrAKDuhQBgs42CDIN0EFn1oWZEBwSctDQJpHnhHuc6hBlNgyRXGKUEldo7FIq2xTtWldz+mHtE6nmNQ9IhkM4zke4S2e4jFe48N/cwVhjnotVZq1BstdJbb9ZeH+XcqiWUDpmXqzZgEFmAhrNah2N1Tqt9AbfOXXzu3C5PxwMIIHeEP17+MlntAN7Mqdfpji87sQpQ+Q/37uVmdpLCFjQk7RT8xCw7pgAAZhlN57HGZgDpKQpnMv4CHSHumQMlJN2pS2GqFSpMl8hlcm/X4CB2eyUA==</latexit> <latexit sha1_base64="MUizgpG4hUbY1PLaPd5V/ZORBxQ=">AAAC1XichZE7SwNBEMcn5zu+ojaCzWJQFDHM2SiKINpY+ooRknjcnas5vBd3m2A80omFjaWFlYKI+DFs/AIKfgSxVLCxcHI58BEfs+zu7H/nNzu7q7mm4QvEh5jU0NjU3NLaFm/v6OzqTvT0rvtO0dN5WndMx9vQVJ+bhs3TwhAm33A9rlqayTPa7kJ1P1Pinm849poouzxvqTu2sW3oqiBJSbg5v2gpgZiVK5sBW1MCtsUq1HKmzbKswEbYHmmClBmWEwUuVDbK8myM1XH2J04eD9HyD6iSSGIKQ2P1jhw5SYhsyUlcQg62wAEdimABBxsE+Sao4FPLggwILml5CEjzyDPCfQ4ViBNbpChOESqpuzTu0CobqTatqzn9kNbpFJO6RySDIbzDK3zGW7zGR3z7NVcQ5qjWUqZZq7HcVbqP+ldf/6UsmgUUPqg/axawDVNhrQbV7oZK9RZ6jS/tnzyvTq8MBcN4jk9U/xk+4A3dwC696BfLfOUU4vQB8vfnrnfWJ1IypuRlTM7NR1/RCgMwCCP03pMwB4uwBGk69x7eYk2xZikjVaQD6bAWKsUipg++mHT8DkQkrYY=</latexit> G(x; ✓) = ln pm(x; ✓) ln pn(x) <latexit sha1_base64="nkIHah4WsDUAHBQJ7DokrYE9vWU=">AAACpHichVHLShxBFD22j+j4mNFshGwKRVHE4XY2iiKKLnThQp2MCo4O3W3pNNMvumsGzeAP+AMu3GhARPyMbALZxoWfIFkayMaFd7onBBX1Nl116px7bt2qMgPHjhTRbZPW3NLa9qG9I9XZ1d2TzvT2rUd+JbRk3vIdP9w0jUg6tifzylaO3AxCabimIzfM8kJd36jKMLJ974s6DOS2a+x79p5tGYqpYmZ2UYyIAzEtCqoklSFGxYwoOJ4IijXhiqPn4vg/0Uuk0WJmkLIUh3gJ9AYYnBsojJ0BWPEzlyhgFz4sVOBCwoNi7MBAxN8WdBAC5rZRYy5kZMe6xBFS7K1wluQMg9kyj/u82mqwHq/rNaPYbfEuDv8hOwWG6Iau6J5+0DXd0cOrtWpxjXovhzybiVcGxfRxf+7vuy6XZ4XSf9ebPSvsYTLu1ebeg5ipn8JK/NWvJ/e5qbWh2jB9o9/c/znd0nc+gVf9Y12syrVTpPgB9OfX/RKsf87qlNVX+SXmkUQ7PmEAI3zfE5jDElaQ530v8BO/cKMNa8taTssnqVpTw/MRT0LbeQQxa55+</latexit> <latexit sha1_base64="kzsjsnHuwYrNLdSfmmGNKNs1UBQ=">AAACpHichVHPSxtBFP7cqtXU6tpehF6GiKKI4W0viqU0tAc9eDBJo4HEht111MX9xe4kNA3+Az0LHnqyICL+Gb0UvJpD/oTSo4VeeujLbkqpon3Lznzzfe9782bGCl0nVkTdAe3B4NDww5HRzKOxx+MT+uSTzThoRLYs24EbRBXLjKXr+LKsHOXKShhJ07NcuWUdvOnpW00ZxU7gv1WtUG575p7v7Dq2qZiq669WxZx4L16ImtqXyhTz4qWoub4I623hicOb4uIf0U+l+bo+TTlKQtwGRh9M57O1haNuvrUR6GeoYQcBbDTgQcKHYuzCRMxfFQYIIXPbaDMXMXISXeIQGfY2OEtyhsnsAY97vKr2WZ/XvZpx4rZ5F5f/iJ0CM9Shc7qmr3RB3+jXnbXaSY1eLy2erdQrw/rEx6nSz/+6PJ4V9v+67u1ZYRfLSa8O9x4mTO8Udupvfji+Lq0UZ9qz9Jm+c/8n1KUvfAK/+cM+LcjiJ2T4AYyb130bbD7PGZQzCvwSr5HGCJ4hizm+7yXksYYNlHnfU1ziCh1tVlvXSlo5TdUG+p6n+Ce0d78B972gBA==</latexit> <latexit sha1_base64="kzsjsnHuwYrNLdSfmmGNKNs1UBQ=">AAACpHichVHPSxtBFP7cqtXU6tpehF6GiKKI4W0viqU0tAc9eDBJo4HEht111MX9xe4kNA3+Az0LHnqyICL+Gb0UvJpD/oTSo4VeeujLbkqpon3Lznzzfe9782bGCl0nVkTdAe3B4NDww5HRzKOxx+MT+uSTzThoRLYs24EbRBXLjKXr+LKsHOXKShhJ07NcuWUdvOnpW00ZxU7gv1WtUG575p7v7Dq2qZiq669WxZx4L16ImtqXyhTz4qWoub4I623hicOb4uIf0U+l+bo+TTlKQtwGRh9M57O1haNuvrUR6GeoYQcBbDTgQcKHYuzCRMxfFQYIIXPbaDMXMXISXeIQGfY2OEtyhsnsAY97vKr2WZ/XvZpx4rZ5F5f/iJ0CM9Shc7qmr3RB3+jXnbXaSY1eLy2erdQrw/rEx6nSz/+6PJ4V9v+67u1ZYRfLSa8O9x4mTO8Udupvfji+Lq0UZ9qz9Jm+c/8n1KUvfAK/+cM+LcjiJ2T4AYyb130bbD7PGZQzCvwSr5HGCJ4hizm+7yXksYYNlHnfU1ziCh1tVlvXSlo5TdUG+p6n+Ce0d78B972gBA==</latexit> <latexit sha1_base64="7XKN2W0K+PSt9Bee+eSoyWMMdS8=">AAACpHichVG7ThtBFD0shIdJwECDRDPCAkER624aEAiBoIAiBbYxIGFi7W4GWLEv7Y4twPIP8AMUVCBZKOIzaJDShoJPQJQg0VDker0RApTkrnbmzDn33LkzYwaOHSmi2zatveNDZ1d3T6r346e+/vTA4HrkV0JLFi3f8cNN04ikY3uyqGzlyM0glIZrOnLD3F9q6htVGUa2762pw0Buu8auZ+/YlqGYKqfnl8WEOBCzoqT2pDLEpJgTJccTQbkmXFF/K37+I3otabKczlCW4hDvgZ6ADJJY9dMXKOE7fFiowIWEB8XYgYGIvy3oIATMbaPGXMjIjnWJOlLsrXCW5AyD2X0ed3m1lbAer5s1o9ht8S4O/yE7Bcbohn7QA13TJd3R819r1eIazV4OeTZbXhmU+4+HC0//dbk8K+y9uP7Zs8IOpuNebe49iJnmKayWv3p08lCYyY/Vxumc7rn/M7qlKz6BV320GjmZP0WKH0B/e93vwfqXrE5ZPUeZhcXkKboxglFM8H1PYQErWEWR923gJ37hRhvXvmoFrdhK1doSzxBehfbtN2CBnPU=</latexit> Parametric sigmoid: Log ratio: h(x; ✓) = 1 1 + Tn Td exp( G(x; ✓)) <latexit sha1_base64="Bsnuuk1UBTAi1kaQfN4MvcDfMSw=">AAACp3ichVFNS9xQFD2mfnVqddRNwU1QLDMUhxs3ilIQC9WdnzMOODK8xDdOMJOE5M2ghvwBF2676EYLRaU/oxtX3RX1J5QuLXTTRW8yA/2Q1hvy3rnn3XPfee+ZvmOHiui2S3vU3dPb1/8482Tg6eBQdnikFHrNwJJFy3O8oGyKUDq2K4vKVo4s+4EUDdORW+b+q2R9qyWD0PbcTXXoy52G2HPtmm0JxVQ1u1jPHczrFVWXSuT1l3qlFggrMuLI0F90ks2qG/OwG+sVeeDnpvSl3yT5OFPNTlCB0tDvA6MDJhZez5ycAlj1sueoYBceLDTRgIQLxdiBQMjfNgwQfOZ2EDEXMLLTdYkYGdY2uUpyhWB2n8c9zrY7rMt50jNM1Rbv4vAfsFLHJH2mS7qjK/pAX+jHP3tFaY/EyyHPZlsr/erQ8bON7w+qGjwr1H+p/utZoYbZ1KvN3v2USU5htfWtozd3G3Prk9Fzekdf2f8Z3dJHPoHb+ma9X5Prb5E8gPH3dd8HpemCQQVjjV9iEe3oxxjGkeP7nsEClrGKIu97gU+4xo2W11a0klZul2pdHc0o/ghN/ASCu6LB</latexit> <latexit sha1_base64="5wVZ+BihKzrd504DFBBvARX3mpg=">AAACp3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQs4ZWhUWCvElGSkliRqKtgqxKQVJSZXG9ZWGypoQzkh8Xm1QCKlViEmtaJAQ1fBHUmLZi1XvICygZ4BGChgMgyhDGUHN/O++VmTPwfkCyxniGFIYchnSGYoZchlSGXIYygBsnMYEhmKgTCawZDBgKEAKBbLUA0UKwKyMsHyqQy1DFxAvaVAValAFYlA0WwgmQ7kRUNF84B8kJnFYN3JQFtygLgIqFOBQdXgqsFKg88GJwxWG7w0+IPTrGqwGSC3VALpJIje1IJ4/i6J4O8EdeUC6RKGDIQuvG4uYUhjsAC7NRPo9gKwCMgXyRD9ZVXTPwdbBalWqxksMngNdP9Cg5sGh4E+yCv7krw0MDVoNgMoAgzRgxuTEWakZ2igZxgIjAknBgjgYJBmUGLQAIa3OYMDgwdDAEMo0N4VDOcYrjPcYNJk8mcKY4qAKGVihOoRZkABTIkAB3qksA==</latexit> <latexit sha1_base64="5wVZ+BihKzrd504DFBBvARX3mpg=">AAACp3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQs4ZWhUWCvElGSkliRqKtgqxKQVJSZXG9ZWGypoQzkh8Xm1QCKlViEmtaJAQ1fBHUmLZi1XvICygZ4BGChgMgyhDGUHN/O++VmTPwfkCyxniGFIYchnSGYoZchlSGXIYygBsnMYEhmKgTCawZDBgKEAKBbLUA0UKwKyMsHyqQy1DFxAvaVAValAFYlA0WwgmQ7kRUNF84B8kJnFYN3JQFtygLgIqFOBQdXgqsFKg88GJwxWG7w0+IPTrGqwGSC3VALpJIje1IJ4/i6J4O8EdeUC6RKGDIQuvG4uYUhjsAC7NRPo9gKwCMgXyRD9ZVXTPwdbBalWqxksMngNdP9Cg5sGh4E+yCv7krw0MDVoNgMoAgzRgxuTEWakZ2igZxgIjAknBgjgYJBmUGLQAIa3OYMDgwdDAEMo0N4VDOcYrjPcYNJk8mcKY4qAKGVihOoRZkABTIkAB3qksA==</latexit> <latexit sha1_base64="c/o7XhVG7GfKJA0/DFvy8wegKvo=">AAACp3ichVFNT9tAEH0YWiAtJcAFqReLCJSoajTmAmpVCcGhvZWvhEgERWuzSSwc27I3EdTyH+APcOBEpaqt+jO49NRbVfgJVY9U4sKBsWOpBUQ7lnffvJ03+3bX9B07VETnA9rg0IOHwyOjuUePx56M5ycmq6HXDSxZsTzHC2qmCKVju7KibOXImh9I0TEduWXurSTrWz0ZhLbnbqoDX+50RMu1m7YlFFON/HK7uP9Sr6u2VKKkv9LrzUBYkRFHhv4sSzYbbszDbqzX5b5ffK6//ktSinONfIHKlIZ+FxgZKCCLVS//EXXswoOFLjqQcKEYOxAI+duGAYLP3A4i5gJGdrouESPH2i5XSa4QzO7x2OJsO2NdzpOeYaq2eBeH/4CVOmbpO32mC/pKX+gnXd3bK0p7JF4OeDb7Wuk3xg+nNy7/q+rwrND+o/qnZ4UmFlOvNnv3UyY5hdXX994dXWy8WJ+N5ug9/WL/J3ROp3wCt/fb+rAm14+RPIBx+7rvgup82aCysUaFpeXsKUbwFDMo8n0vYAlvsIoK7/sJ3/ADZ1pJe6tVtVq/VBvINFO4EZq4Bv4VoNY=</latexit> Model’s PDF/PMF Noise PDF/PMF Observed data Noise data
  12. NCE’s Properties NCE has similar properties to MLE • Nonparametric

    estimation • Consistency • Asymptotic normality Check the original paper if you want to know details. 12
  13. Simulations • True data generated by 5-D Gaussian with 0

    mean • Larger number of data is better 13 From Michael U. Gutmann, Aapo Hyvärinen, Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics. JMLR, 2012.
  14. NCE & Neural Probabilistic Language Models [Mnih & Teh, ICML2012]

    • Softmax with large units is a bottleneck in the training time • is set to 1 to avoid storing parameters depend on context words ( ) • Empirically, it did not affect the performance 14 O(|V |#contexts) <latexit sha1_base64="GV40hula22RSeszxaEw3x1HTW14=">AAACgXichVHLLgRBFD3ae7wGGwkLMSFsJrdFQtgIGzvPGRLDpLsVOvqV7poJ2mws/YCFFYkgVvyCjR+w8AliSWJj4U5PJ4LgVqrq1Kl7bp2q0j3LDCTRY5VSXVNbV9/QmGhqbmltS7Z3ZAO34BsiY7iW66/oWiAs0xEZaUpLrHi+0GzdEsv6znR5f7ko/MB0nSW554k1W9tyzE3T0CRT+WTP7OBB9mA9zKVytia3fTs0XEeKXRmUSkP5ZIrSFEXvT6DGIIU45tzkBXLYgAsDBdgQcCAZW9AQcFuFCoLH3BpC5nxGZrQvUEKCtQXOEpyhMbvD4xavVmPW4XW5ZhCpDT7F4u6zshf99EBX9EL3dE1P9P5rrTCqUfayx7Ne0Qov33bUtfj2r8rmWWL7U/WnZ4lNjEVeTfbuRUz5FkZFX9w/flkcX+gPB+iMntn/KT3SHd/AKb4a5/Ni4QQJ/gD1+3P/BNnhtEppdX4kNTkVf0UDutGHQX7vUUxiBnPI8LmHuMQNbpVqZUghZbiSqlTFmk58CWXiAx0DlNg=</latexit> <latexit sha1_base64="GV40hula22RSeszxaEw3x1HTW14=">AAACgXichVHLLgRBFD3ae7wGGwkLMSFsJrdFQtgIGzvPGRLDpLsVOvqV7poJ2mws/YCFFYkgVvyCjR+w8AliSWJj4U5PJ4LgVqrq1Kl7bp2q0j3LDCTRY5VSXVNbV9/QmGhqbmltS7Z3ZAO34BsiY7iW66/oWiAs0xEZaUpLrHi+0GzdEsv6znR5f7ko/MB0nSW554k1W9tyzE3T0CRT+WTP7OBB9mA9zKVytia3fTs0XEeKXRmUSkP5ZIrSFEXvT6DGIIU45tzkBXLYgAsDBdgQcCAZW9AQcFuFCoLH3BpC5nxGZrQvUEKCtQXOEpyhMbvD4xavVmPW4XW5ZhCpDT7F4u6zshf99EBX9EL3dE1P9P5rrTCqUfayx7Ne0Qov33bUtfj2r8rmWWL7U/WnZ4lNjEVeTfbuRUz5FkZFX9w/flkcX+gPB+iMntn/KT3SHd/AKb4a5/Ni4QQJ/gD1+3P/BNnhtEppdX4kNTkVf0UDutGHQX7vUUxiBnPI8LmHuMQNbpVqZUghZbiSqlTFmk58CWXiAx0DlNg=</latexit> <latexit sha1_base64="GV40hula22RSeszxaEw3x1HTW14=">AAACgXichVHLLgRBFD3ae7wGGwkLMSFsJrdFQtgIGzvPGRLDpLsVOvqV7poJ2mws/YCFFYkgVvyCjR+w8AliSWJj4U5PJ4LgVqrq1Kl7bp2q0j3LDCTRY5VSXVNbV9/QmGhqbmltS7Z3ZAO34BsiY7iW66/oWiAs0xEZaUpLrHi+0GzdEsv6znR5f7ko/MB0nSW554k1W9tyzE3T0CRT+WTP7OBB9mA9zKVytia3fTs0XEeKXRmUSkP5ZIrSFEXvT6DGIIU45tzkBXLYgAsDBdgQcCAZW9AQcFuFCoLH3BpC5nxGZrQvUEKCtQXOEpyhMbvD4xavVmPW4XW5ZhCpDT7F4u6zshf99EBX9EL3dE1P9P5rrTCqUfayx7Ne0Qov33bUtfj2r8rmWWL7U/WnZ4lNjEVeTfbuRUz5FkZFX9w/flkcX+gPB+iMntn/KT3SHd/AKb4a5/Ni4QQJ/gD1+3P/BNnhtEppdX4kNTkVf0UDutGHQX7vUUxiBnPI8LmHuMQNbpVqZUghZbiSqlTFmk58CWXiAx0DlNg=</latexit> <latexit sha1_base64="GV40hula22RSeszxaEw3x1HTW14=">AAACgXichVHLLgRBFD3ae7wGGwkLMSFsJrdFQtgIGzvPGRLDpLsVOvqV7poJ2mws/YCFFYkgVvyCjR+w8AliSWJj4U5PJ4LgVqrq1Kl7bp2q0j3LDCTRY5VSXVNbV9/QmGhqbmltS7Z3ZAO34BsiY7iW66/oWiAs0xEZaUpLrHi+0GzdEsv6znR5f7ko/MB0nSW554k1W9tyzE3T0CRT+WTP7OBB9mA9zKVytia3fTs0XEeKXRmUSkP5ZIrSFEXvT6DGIIU45tzkBXLYgAsDBdgQcCAZW9AQcFuFCoLH3BpC5nxGZrQvUEKCtQXOEpyhMbvD4xavVmPW4XW5ZhCpDT7F4u6zshf99EBX9EL3dE1P9P5rrTCqUfayx7Ne0Qov33bUtfj2r8rmWWL7U/WnZ4lNjEVeTfbuRUz5FkZFX9w/flkcX+gPB+iMntn/KT3SHd/AKb4a5/Ni4QQJ/gD1+3P/BNnhtEppdX4kNTkVf0UDutGHQX7vUUxiBnPI8LmHuMQNbpVqZUghZbiSqlTFmk58CWXiAx0DlNg=</latexit> From Andriy Mnih, Yee Whye Teh, A Fast and Simple Algorithm for Training Neural Probabilistic Language Models. ICML, 2012. c <latexit sha1_base64="lu/Cf4S7fI2pX3rx5Qu1Ikpy69Q=">AAACZHichVHLSsNAFD2Nr1ofrYogCFIsFVflRgTFVdGNS9taW1CRJE5raJqEJC3U4g/oVnHhSkFE/Aw3/oALf0AQlxXcuPA2DYgW9Q4zc+bMPXfOzKi2obse0VNI6unt6x8ID0aGhkdGo7Gx8S3XqjmayGuWYTlFVXGFoZsi7+meIYq2I5SqaoiCWllr7xfqwnF1y9z0GrbYrSplUy/pmuIxldH2YglKkR/xbiAHIIEgNqzYDXawDwsaaqhCwITH2IACl9s2ZBBs5nbRZM5hpPv7AkeIsLbGWYIzFGYrPJZ5tR2wJq/bNV1frfEpBneHlXEk6ZFuqUUPdEcv9PFrraZfo+2lwbPa0Qp7L3o8lXv/V1Xl2cPBl+pPzx5KWPa96uzd9pn2LbSOvn543sqtZJPNObqiV/Z/SU90zzcw62/adUZkLxDhD5B/Pnc32FpIyZSSM4uJ9GrwFWFMYxbz/N5LSGMdG8jzuQInOMVZ6FkaliakyU6qFAo0E/gW0swnzEyJ4w==</latexit> <latexit sha1_base64="lu/Cf4S7fI2pX3rx5Qu1Ikpy69Q=">AAACZHichVHLSsNAFD2Nr1ofrYogCFIsFVflRgTFVdGNS9taW1CRJE5raJqEJC3U4g/oVnHhSkFE/Aw3/oALf0AQlxXcuPA2DYgW9Q4zc+bMPXfOzKi2obse0VNI6unt6x8ID0aGhkdGo7Gx8S3XqjmayGuWYTlFVXGFoZsi7+meIYq2I5SqaoiCWllr7xfqwnF1y9z0GrbYrSplUy/pmuIxldH2YglKkR/xbiAHIIEgNqzYDXawDwsaaqhCwITH2IACl9s2ZBBs5nbRZM5hpPv7AkeIsLbGWYIzFGYrPJZ5tR2wJq/bNV1frfEpBneHlXEk6ZFuqUUPdEcv9PFrraZfo+2lwbPa0Qp7L3o8lXv/V1Xl2cPBl+pPzx5KWPa96uzd9pn2LbSOvn543sqtZJPNObqiV/Z/SU90zzcw62/adUZkLxDhD5B/Pnc32FpIyZSSM4uJ9GrwFWFMYxbz/N5LSGMdG8jzuQInOMVZ6FkaliakyU6qFAo0E/gW0swnzEyJ4w==</latexit> <latexit sha1_base64="lu/Cf4S7fI2pX3rx5Qu1Ikpy69Q=">AAACZHichVHLSsNAFD2Nr1ofrYogCFIsFVflRgTFVdGNS9taW1CRJE5raJqEJC3U4g/oVnHhSkFE/Aw3/oALf0AQlxXcuPA2DYgW9Q4zc+bMPXfOzKi2obse0VNI6unt6x8ID0aGhkdGo7Gx8S3XqjmayGuWYTlFVXGFoZsi7+meIYq2I5SqaoiCWllr7xfqwnF1y9z0GrbYrSplUy/pmuIxldH2YglKkR/xbiAHIIEgNqzYDXawDwsaaqhCwITH2IACl9s2ZBBs5nbRZM5hpPv7AkeIsLbGWYIzFGYrPJZ5tR2wJq/bNV1frfEpBneHlXEk6ZFuqUUPdEcv9PFrraZfo+2lwbPa0Qp7L3o8lXv/V1Xl2cPBl+pPzx5KWPa96uzd9pn2LbSOvn543sqtZJPNObqiV/Z/SU90zzcw62/adUZkLxDhD5B/Pnc32FpIyZSSM4uJ9GrwFWFMYxbz/N5LSGMdG8jzuQInOMVZ6FkaliakyU6qFAo0E/gW0swnzEyJ4w==</latexit> <latexit sha1_base64="lu/Cf4S7fI2pX3rx5Qu1Ikpy69Q=">AAACZHichVHLSsNAFD2Nr1ofrYogCFIsFVflRgTFVdGNS9taW1CRJE5raJqEJC3U4g/oVnHhSkFE/Aw3/oALf0AQlxXcuPA2DYgW9Q4zc+bMPXfOzKi2obse0VNI6unt6x8ID0aGhkdGo7Gx8S3XqjmayGuWYTlFVXGFoZsi7+meIYq2I5SqaoiCWllr7xfqwnF1y9z0GrbYrSplUy/pmuIxldH2YglKkR/xbiAHIIEgNqzYDXawDwsaaqhCwITH2IACl9s2ZBBs5nbRZM5hpPv7AkeIsLbGWYIzFGYrPJZ5tR2wJq/bNV1frfEpBneHlXEk6ZFuqUUPdEcv9PFrraZfo+2lwbPa0Qp7L3o8lXv/V1Xl2cPBl+pPzx5KWPa96uzd9pn2LbSOvn543sqtZJPNObqiV/Z/SU90zzcw62/adUZkLxDhD5B/Pnc32FpIyZSSM4uJ9GrwFWFMYxbz/N5LSGMdG8jzuQInOMVZ6FkaliakyU6qFAo0E/gW0swnzEyJ4w==</latexit>
  15. Summary • Proposed new estimation method for a model with

    ineffective partition function • Replace partition function then estimate parameters by classifying between observed data and noise data • Useful for many applications 15
  16. Further Reading • Bregman divergence perspective • An extension paper

    • Review paper by authors • Recent paper: Ciwan Ceylan, Michael Gutmann, Conditional Noise-Contrastive Estimation of Unnormalised Models. ICML, 2018. 16
  17. Implementations • Author’s Matlab code • Auto Differential Frameworks •

    Official TensorFlow function • PyTorch issue 17