How should we manage experiments?

)PXTIPVMEXFNBOBHFFYQFSJNFOUT .-LJUDIFO $IJF)BZBTIJEB

$IJF)BZBTIJEB !DIJF •  .BDIJOF-FBSOJOHBOE*OGSBTUSVDUVSF&OHJOFFSBU $PPLQBE*OD •  $POUSJCVUPSPG"QBDIF4QBSL )BEPPQ BOETPPO
•  (SBEVBUFE0TBLB6OJWBOE/"*45 –  &OHBHFEJO7PJDFDPOWFSTJPOSFTFBSDI *DPOUSJCVUFEBSUJDMFTBCPVUTDJFOUJGJD DPNQVUJOHXJUI1ZUIPO *UbTPOTBMF"QSJM

5IJOHT*XBOUUPEP •  *OBEEJUJPOUPXPSLPONBDIJOFMFBSOJOH *`EMJLFUP JNQSPWFUIFNBDIJOFMFBSOJOHFYQFSJNFOU EFQMPZNFOU BOENPOJUPSJOHQSPDFTTGSPNUIFWJFXQPJOUPGTPGUXBSF FOHJOFFSJOH

1BTUXPSLFBTZ@OPUJGJFS •  4JNQMFEFDPSBUPSGVODUJPOUPOPUJGZPOTMBDL NBJM PS .BDEFTLUPQXIFOZPVSTDSJQUGJOJTIFE –  IUUQTHJUIVCDPNDIJFFBTZ@OPUJGJFS But Hyperdash
is very convenient

5IFMJGFDZDMFPGUIFNBDIJOFMFBSOJOHFYQFSJNFOU •  8IFOXFXPSLPONBDIJOFMFBSOJOH XFMPPQGPMMPXJOH TUFQT –  %FDJEFFWBMVBUJPOJOEFY –  &YFDVUFFYQFSJNFOUT • 
'FBUVSF&OHJOFFSJOH •  5SBJONPEFM •  &WBMVBUFNPEFM •  $IBOHFDPOEJUJPOT –  4VNNBSJ[FBOEWJTVBMJ[FUIFSFTVMUPGFYQFSJFODFTGPSSFQPSUTPS QBQFST Loop many times

"WBSJFUZPGQBUUFSOT •  $SFBUFGFBUVSFEBUBTFUGSPNIVHFEBUB –  )PXNVDIEBUBEPXFVTF –  8IBULJOEPGEBUBEPXFVTF BDUJPOMPH
VTFSBOEJUFNDPOUFYUT JNBHFT FUD –  )PXUPJNQMFNFOUGFBUVSFFOHJOFFSJOH /PSNBMJ[BUJPO .JTTJOH WBMVFQSPDFTTJOH FUD •  5SZBWBSJFUZPGBMHPSJTNT •  1BSBNFUFS5VOJOH –  &HCBUDITJ[F MFBSOJOHSBUF FUD

%JGGJDVMUZJONBOBHJOHFYQFSJNFOUT •  8FIBWFUPNBOBHFTPNBOZJOGPSNBUJPO –  %P[FOTPGFYQFSJNFOUT –  .BOZQBSBNFUFST –  TPVSDFDPEFWFSTJPO – 
&OWJSPONFOUBM*OGPSNBUJPO •  3FTPVSDFTJ[F •  6TF(16 •  'SBNFXPSLWFSTJPOT •  &YQFSJNFOUQJQFMJOFPGUFODPOUBJOTTFWFSBMOVNCFSPG TUFQT

*NQPSUBOUUIJOHTGPSFYQFSJNFOUNBOBHFNFOU •  4BWFBMMOFDFTTBSZJOGPSNBUJPOFGGJDJFOUMZ •  #FBCMFUPTFBSDIBOEBDDFTTQBTUJOGPSNBUJPOFBTJMZ •  #FBCMFUPTVNNBSJ[FTFWFSBMFYQFSJNFOUTFBTJMZ

&YJTUJOHXPSLT •  4BDSFE IUUQTHJUIVCDPN*%4*"TBDSFE •  '(-BC IUUQTHJUIVCDPN,BJYIJO'(-BC • 
5IFZBSFUPPMTUPNBOBHFFYQFSJNFOUMPHT •  5IFZVTF.POHP%#GPSEBUBTUPDLFSBOEQSPWJEFTUIF SJDI6* •  *U`TWFSZDPOWFOJFOUGPSTDBMBCMFFYQFSJNFOUT –  4IBSFFYQFSJNFOUTXJUI.VMUJQMFFOWJSPONFOUT NVMUJQMF SFTFBSDIFS FUD •  #VUUIFZBSFPWFSTQFDGPSNFʜ –  /POFDFTTBSZPGSJDIQMBUGPSN –  *EPO`UXBOUUPNBJOUBJOB.POHP%# –  *EPO`UMJLFUPDIBOHFNZTDSJQUTGPSMPHNBOBHFNFOU

&YJTUJOHXPSLT •  $PNFUNM IUUQTXXXDPNFUNM –  $MPVE4FSWJDFGPSMPHUSBDLJOH –  4IBSFMPHT NFNPSBOEVNGVODUJPO
FBTZUPVTF –  -PPLTHPPE #VUʜ –  8FDBO`UVQMPBEBEEJUJPOBMGJMFTTVDIBTUSBJOFENPEFMTPSEBUB –  /PDPMMBCPSBUJPOXJUIHJU –  8FIBWFUPUBLFDBSFOPUUPVQMPBEDPOGJEFOUJBMJOGPSNBUJPO

FYQTUPDL •  *DSFBUFEBUPPMXIJDITPMWFTUIFDPNQMFYJUZPGUIF FYQFSJNFOUNBOBHFNFOU –  IUUQTHJUIVCDPNDIJFFYQTUPDL

$PODFQUPGFYQTUPDL •  *UIBTWFSZTJNQMFBSDIJUFDUVSF –  /PEFQFOEFODZ –  /PNBJOUFOBODF •  8FEPO`UOFFEUPDIBOHFPVSTDSJQUT – 
0OMZBEEJOHUXPPSUISFFDPNNBOET •  *UIBTHPPEMPH"DDFTTJCJMJUZXJUI7JTVBMJ[BUJPO –  8FDBODPNQBSFQBTUFYQFSJNFOUT –  8FDBOBDDFTTFBDIMPHGSPNBCSPXTFS

6TBHFPOGVODUJPOBMJ[FETDSJQUT *GZPVBMSFBEZIBWFGVODUJPOBMJ[FEUBSHFUQSPDFTT ZPVDBO VTFUIFGPMMPXJOHEFDPSBUPS from expstock import expstock a=1 b=2
e = expstock.ExpStock(exp_name=‘test_experiment’) @expstock.expstock(e) def run(a, b): return a + b e.append_param(a=a, b=b) run(a, b)

6TBHFXJUI+VQZUFS/PUFCPPL 8JUIJOUFSBDUJWFTDSJQUTFHFYQFSJNFOUTPO+VQZUFS /PUFCPPL ZPVDBOVTFUIFGPMMPXJOHDPNNBOE from expstock import expstock a=1 b=2
e = expstock.ExpStock(exp_name=‘test_experiment’) e.append_param(a=a, b=b) e.pre_stock() a + b e.post_stock()

%JSFDUPSZ4USVDUVSF -PHTBSFTUPDLFEBUFBDITVCEJSFDUPSZ experiments ᵓᴷᴷ <yyyymmdd_hhmmss>_<experiment_name> ᴹ ᵓᴷᴷ exec_time.txt ᴹ ᵓᴷᴷ
git_diff.txt ᴹ ᵓᴷᴷ git_head.txt ᴹ ᵓᴷᴷ machine_info.txt ᴹ ᵓᴷᴷ memo.txt ᴹ ᵓᴷᴷ params.txt ᴹ ᵓᴷᴷ report.txt ᴹ ᵓᴷᴷ result.txt ᴹ ᵓᴷᴷ stderr.txt ᴹ ᵋᴷᴷ stdout.txt

4BWFBEEJUJPOBMGJMFT •  :PVDBOTBWFBEEJUJPOBMGJMFTBUTBNFEJSFDUPSZ # save keras model from keras.models import
load_model model_file = os.path.join(e.log_dirname, ‘model.h5’) model.save(model_file) # save matplotlib firure plt_png = os.path.join(e.log_dirname, ‘plt.png’) plt.savefig(plt_png)

FYQTUPDLTFSWFS •  &YQTUPDLIBTBTJNQMF6*JNQMFNFOUBUJPOXJUICPUUMF BOE42-JUF

'VUVSFXPSL •  *ODSFBTFMPHNFUSJDT –  .BDIJOF3FTPVSDFVUJMJ[BUJPO –  'SBNFXPSLWFSTJPO •  'VODUJPOUPFEJUNFUSJDTPO6* • 
FUD

How should we manage experiments?

How should we manage experiments?

chie8842

More Decks by chie8842

Other Decks in Technology

Featured

Transcript

)PXTIPVMEXFNBOBHFFYQFSJNFOUT .-LJUDIFO $IJF)BZBTIJEB

$IJF)BZBTIJEB !DIJF •  .BDIJOF-FBSOJOHBOEOGSBTUSVDUVSF&OHJOFFSBU $PPLQBEOD •  $POUSJCVUPSPG"QBDIF4QBSL )BEPPQ BOETPPO

5IJOHTXBOUUPEP •  OBEEJUJPOUPXPSLPONBDIJOFMFBSOJOH *`EMJLFUP JNQSPWFUIFNBDIJOFMFBSOJOHFYQFSJNFOU EFQMPZNFOU BOENPOJUPSJOHQSPDFTTGSPNUIFWJFXQPJOUPGTPGUXBSF FOHJOFFSJOH

1BTUXPSLFBTZ@OPUJGJFS •  4JNQMFEFDPSBUPSGVODUJPOUPOPUJGZPOTMBDL NBJM PS .BDEFTLUPQXIFOZPVSTDSJQUGJOJTIFE –  IUUQTHJUIVCDPNDIJFFBTZ@OPUJGJFS But Hyperdash

5IFMJGFDZDMFPGUIFNBDIJOFMFBSOJOHFYQFSJNFOU •  8IFOXFXPSLPONBDIJOFMFBSOJOH XFMPPQGPMMPXJOH TUFQT –  %FDJEFFWBMVBUJPOJOEFY –  &YFDVUFFYQFSJNFOUT •

"WBSJFUZPGQBUUFSOT •  $SFBUFGFBUVSFEBUBTFUGSPNIVHFEBUB –  )PXNVDIEBUBEPXFVTF –  8IBULJOEPGEBUBEPXFVTF BDUJPOMPH

%JGGJDVMUZJONBOBHJOHFYQFSJNFOUT •  8FIBWFUPNBOBHFTPNBOZJOGPSNBUJPO –  %P[FOTPGFYQFSJNFOUT –  .BOZQBSBNFUFST –  TPVSDFDPEFWFSTJPO –

*NQPSUBOUUIJOHTGPSFYQFSJNFOUNBOBHFNFOU •  4BWFBMMOFDFTTBSZJOGPSNBUJPOFGGJDJFOUMZ •  #FBCMFUPTFBSDIBOEBDDFTTQBTUJOGPSNBUJPOFBTJMZ •  #FBCMFUPTVNNBSJ[FTFWFSBMFYQFSJNFOUTFBTJMZ

&YJTUJOHXPSLT •  4BDSFE IUUQTHJUIVCDPN%4"TBDSFE •  '(-BC IUUQTHJUIVCDPN,BJYIJO'(-BC •

&YJTUJOHXPSLT •  $PNFUNM IUUQTXXXDPNFUNM –  $MPVE4FSWJDFGPSMPHUSBDLJOH –  4IBSFMPHT NFNPSBOEVNGVODUJPO

FYQTUPDL •  *DSFBUFEBUPPMXIJDITPMWFTUIFDPNQMFYJUZPGUIF FYQFSJNFOUNBOBHFNFOU –  IUUQTHJUIVCDPNDIJFFYQTUPDL

$PODFQUPGFYQTUPDL •  *UIBTWFSZTJNQMFBSDIJUFDUVSF –  /PEFQFOEFODZ –  /PNBJOUFOBODF •  8FEPO`UOFFEUPDIBOHFPVSTDSJQUT –

6TBHFPOGVODUJPOBMJ[FETDSJQUT *GZPVBMSFBEZIBWFGVODUJPOBMJ[FEUBSHFUQSPDFTT ZPVDBO VTFUIFGPMMPXJOHEFDPSBUPS from expstock import expstock a=1 b=2

6TBHFXJUI+VQZUFS/PUFCPPL 8JUIJOUFSBDUJWFTDSJQUTFHFYQFSJNFOUTPO+VQZUFS /PUFCPPL ZPVDBOVTFUIFGPMMPXJOHDPNNBOE from expstock import expstock a=1 b=2

%JSFDUPSZ4USVDUVSF -PHTBSFTUPDLFEBUFBDITVCEJSFDUPSZ experiments ᵓᴷᴷ <yyyymmdd_hhmmss>_<experiment_name> ᴹ ᵓᴷᴷ exec_time.txt ᴹ ᵓᴷᴷ

4BWFBEEJUJPOBMGJMFT •  :PVDBOTBWFBEEJUJPOBMGJMFTBUTBNFEJSFDUPSZ # save keras model from keras.models import

FYQTUPDLTFSWFS •  &YQTUPDLIBTBTJNQMF6*JNQMFNFOUBUJPOXJUICPUUMF BOE42-JUF

'VUVSFXPSL •  ODSFBTFMPHNFUSJDT –  .BDIJOF3FTPVSDFVUJMJ[BUJPO –  'SBNFXPSLWFSTJPO •  'VODUJPOUPFEJUNFUSJDTPO6 •