Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Indaba2019.pdf

 Indaba2019.pdf

sambaiga

April 12, 2019
Tweet

More Decks by sambaiga

Other Decks in Technology

Transcript

  1. @sambaiga Spices for Successful ML Project IndabaXTanzania 2019 Anthony Faustine

    PhD machine learning researcher IDLab research group-Ghent University Belgium. 12th April 2019 1
  2. @sambaiga Prole PhD machine learning researcher, IDlab, imec A machine

    learning researcher passionate on using cutting-edge technology to create intelligence system that can reason and understand Figure 1: Research: NILM co-founder: pythontz, indabatz, parrotai strive for excellence money will follow.. 3
  3. @sambaiga Introduction To have successfully and publishable ML project: •

    Identify open problem • Design experiment • Get dataset • Dene evaluation metric • Write code • Run an experiment • Analyse results 4
  4. @sambaiga Identify open problem Dont do the obvious 1 Do

    literature review • Learn about common, methods, dataset and libraries. • Identify open questions that need answers 2 Establish hypothesis about the problem 5
  5. @sambaiga Design experiment 1 Dene performance metric 2 Establish baseline

    performance • Any publishable performance with simplest approach. • Dene your baseline. • Use best published performance. 3 Establish upper bound. 4 Establish project management. • Folder structure. • Version control (gitlab, github etc). 6
  6. @sambaiga Dataset You may need more than one data-set to

    benchmark your solution. 1 At least one dataset that appeared in related prior work. 2 Source of dataset • Build them. • Scrape them. • Find them (contact authors). • Generate them (Articial data). • Folder structure. • Version control (gitlab, github etc). 3 Prepare them for your experiment. 7
  7. @sambaiga Write code quickly: Get a good starting point First

    get a baseline running ⇒this is good research practise. 9
  8. @sambaiga Write code quickly: Use good code style Write code

    for people, not machines • Add comments and include expression in your module. • Use meaningful names. • Add comments about tensors shape • Add comments describing non-obvious logic 10
  9. @sambaiga Write code quickly: Include minimum testing • Test some

    parts of your code. • Make sure data processing works consistently. • Test if tensor operations runs as expected • Test weather gradients are non-zero. 11
  10. @sambaiga Write code quickly: Reduce hard-coding Reduce hard coding as

    much as you can. • Use congurations les (JSON, YAML, or text les) and or argparse module. • Allow you to start simple and later expand without rewriting your code. • Make controlled experiments easier. 12
  11. @sambaiga Write code: important take-away • Build and test code

    to load and process your data. • Build and test code for simple baseline. • Build and test code to evaluate results. • Write re-usable codes. 13
  12. @sambaiga Run experiment Keep track of what you ran •

    keep track of what happen, when and with what code. • Save model checkpoint les for all reasonably eective/interesting experiments • Not recommended: modifying code to run dierent variants → hard to keep track of what you ran. • Analyse model behaviour during training →Use Tensor board, Logging etc. • Take notes of what each experiment was meant to test. 14
  13. @sambaiga Quantitative evaluation • Follow prior work precisely in how

    to choose and implement main evaluation metric. • Show metric as many variants of your model as you can • Test for statistical signicance (for highly variable models or small dierence performance). • If your results are not signicant. say so and explain what you found. 15
  14. @sambaiga Qualitative-evaluation This is the analysis section • convince reader

    for your hypothesis. • Look to prior work to get started • Show examples of system output. • Present error analysis. • Visualize your hidden states. • Plot how your model performance varies with the amount of data. • Include an on-line demo. • If your results are not signicant. say so and explain what you found. 16
  15. @sambaiga Formative vs summative evaluation When the cook tastes the

    soup that is formative; when the customer testes that is summative. Formative evaluation • They guide further investigations • Compare design option A to B, tune hyper-parameters etc Summative evaluation • compare your approach to previous approaches, • compare dierent major variants of your approach. • only use test set. Note: Don't save all your qualitative evaluation for the summative evaluation. 17
  16. @sambaiga Strategies to improve ML performance The challenges → so

    many things to try or change (hyper-parameters etc). • Be specic on what to tune in order to try achieving one eect. • For Supervised ML system focus to achieve: 1 Best performance in training set. 2 Best performance in validation/dev set. 3 Best performance in test set. 4 Perform well in real world. Use dierent knobs (parameters) to improve performance of each part. 18
  17. @sambaiga Strategies to improve ML performance 1 To improve performance

    in training set • use bigger neural network or switch to a better optimization algorithms (adam etc) 2 To improve performance in validation/dev set • Apply regularization or use bigger training set. 3 To improve performance in test set • Increase size of dev set. 4 Poor performance in real world. • Change development set, modify your objective function/hypothesis. 19
  18. @sambaiga Bias Variance Analysis: Avoidable bias • If avoidable bias

    > variance focus on reducing bias. • If avoidable bias < variance focus on reducing variance. 20
  19. @sambaiga Error-Analysis If the performance of your ML algorithm is

    still poor compared to human level performance →perform error analysis • Manually examine mistakes that your ML algorithm is making → gain insight of what to do next. 21
  20. @sambaiga References 1 My personal experiences. 2 Writing Code for

    NLP Research, Joel Grus. 3 Foundations: How to design experiments in NLU, Sam Bowman. 4 Machine Learning Yearning, Andrew Ng. 25