Amandine Lee - Passing Exceptions 101: Paradigms in Error Handling

Exception handling in Python can sometimes feel like a Wild West. If you have a `send_email` function, and the caller inputs an invalid email address, should it:
A) Return `None` or some other special return value,
B) Let the underlying exception it might cause bubble up,
C) Check via a regex and type checking and raise a `ValueError` immediately, or
D) Make a custom `EmailException` subclass and raise that?

What if there is a network error while the email was sending? Or what if the function calls a helper `_format_email` that returns an integer (clearly wrong!), or raises an `TypeError` itself? Should it crash the program or prompt a retry?

This talk will introduce the concept of an exception, explain the built-in Python exception hierarchy and the utility of custom subclasses, demonstrate try/except/finally/else syntax, and then explore different design patterns for exception control flow and their tradeoffs using examples. It will also make comparisons to error handling philosophy in other languages, like Eiffel and Go.

https://us.pycon.org/2017/schedule/presentation/572/

Transcript

1. PARADIGMS IN ERROR HANDLING Passing Exceptions 101 Amandine Lee, PyCon 2017 Portland, OR Actually write speaker notes or notecards if I won’t able to see them live.
2. ABOUT THIS TALK CONTEXT AS INTRODUCTION
3. write bullet points or scripts for this
4. ABOUT THIS TALK QUESTIONS I WANTED TO ANSWER ‣How do make my code correct and reliable? ‣But also readable and maintainable?
5. ABOUT THIS TALK TOPICS NOT COVERED ‣Underlying implementation of exceptions ‣Performance
6. WHAT IS AN EXCEPTION? Tools of the trade
7. WHAT IS AN EXCEPTION? EXCEPTION USAGE Function B Function A Function C EXCEPTION TRY Function D [ETC] EXCEPT 1 try: 2 do_thing() 3 except OSError: 4 retry() 5 except Exception as exc: 6 crash_dump(exc) 7 raise exc 8 9 class MyError(Exception): 10 pass 11 12 raise MyError()
8. WHAT IS AN EXCEPTION? BUILT IN EXCEPTIONS (PYTHON 3.6) BaseException Exception GeneratorExit Your own exceptions SystemExit LookupError KeyboardInterrupt KeyError ValueError IndexError OSError Warning FileNotFoundError ConnectionError … … …
9. AN EXAMPLE Let’s start digging
10. AN EXAMPLE 1 CONFIG_FILE = 'config.json' 2 3 def get_id(conf_fname=CONFIG_FILE): 4 with open(conf_fname, 'r') as fobj: 5 data = json.load(fobj) 6 try: 7 return data['id'] 8 except KeyError: 9 return None FIRST DRAFT
11. AN EXAMPLE REASONS FOR EXCEPTIONS 1) Something exceptional happened
12. AN EXAMPLE 1 def get_id(conf_fname=CONFIG_FILE): 2 try: 3 with open(conf_fname, 'r') as fobj: 4 data = json.load(fobj) 5 except FileNotFoundError, JSONDecodeError: 6 raise MalformedDataError() 7 8 try: 9 return data['id'] 10 except TypeError: 11 raise MalformedDataError() 12 except KeyError: 13 return None ENUMERATING POSSIBLE ERRORS
13. AN EXAMPLE REASONS FOR EXCEPTIONS 1) Something exceptional happened 1) Control flow 2) Unexpected error
14. AN EXAMPLE 1 def get_id(conf_fname=CONFIG_FILE): 2 try: 3 with open(CONFIG_FILE, 'r') as fobj: 4 data = json.loads(fobj) 5 return data.get('id') 6 except Exception: 7 raise MalformedDataError() CATCH ‘EM ALL
15. AN EXAMPLE REASONS FOR EXCEPTIONS 1) Control flow 2) Unexpected *external* error 3) Bug in my code
16. CAN WE BE MORE SYSTEMATIC?
17. EXCEPTION USE CASES Architecture patterns
18. Function B EXCEPTION USE CASES 1. CONTROL FLOW Function A Function D EXCEPTION ???? IF/ELSE HELPER CALL ‣ Do you expect this exception? ‣ Do you *need* to jump the stack? ‣ Can you avoid provoking control flow exceptions?
19. EXCEPTION USE CASES 3. BUG IN MY CODE >>> deg get_id(): File "<stdin>", line 1 deg get_id(): ^ SyntaxError: invalid syntax Raise errors early ‣ Mypy! Static analysis! Raise ‘em explicitly where they came from ‣ Don’t hide with general try/catch ‣ Communicate the context
20. EXCEPTION USE CASES 2. UNEXPECTED EXTERNAL ERROR Problems with callees Problems with callers
21. DESIGN BY CONTRACT Assigning responsibility
22. DESIGN BY CONTRACT HOW CONTRACTS WORK Function Contract Client: - Supply pre-conditions Supplier: - Once pre-conditions are met, fulfill post- conditions Everyone: - Maintain invariants Replace with a diagram?
 delegation and agreements?
23. DESIGN BY CONTRACT CONTRACT ELEMENTS Input types/values Outputs types/value Error and exceptions raised Side effects Preconditions Postconditions Invariants
24. EIFFEL class interface ACCOUNT feature -- Element change deposit (sum: INTEGER) -- Add `sum' to account. require non_negative: sum >= 0 ensure one_more_deposit: deposit_count = old deposit_count + 1 updated: balance = old balance + sum invariant consistent_balance: balance = all_deposits.total end -- class interface ACCOUNT https://www.eiffel.org/doc/eiffel/ET%3A%20Design%20by%20Contract%20%28tm%29%2C%20Assertions%20and%20Exceptions What exactly do these keywords DO?
25. DESIGN BY CONTRACT WHO ENFORCES THE CONTRACT? 1 def get_id(conf_fname=CONFIG_FILE): 2 """Return the user id from CONFIG_FILE, or None if it hasn't been set yet""" 3 if not isinstance(conf_fname, str): 4 raise TypeError('conf_fname must be a string') 5 if not (os.path.exists(conf_fname) and os.path.isfile(conf_fname): 6 raise ValueError('Invalid value for config filename: %r' % conf_fname) 7 try: 8 with open(conf_fname, 'r') as fobj: 9 data = json.loads(fobj) 10 except JSONDecodeError, ValueError: 11 raise MalformedDataError('Issue JSON decoding config file') 12 13 if not isinstance(data, dict): 14 raise MalformedDataError('Must be a JSON-encoded dictionary') 15 16 if 'id' not in data: 17 return None 18 19 uid = data.get('id') 20 if isinstance(uid, str): 21 uid = int(uid) 22 23 assert isinstance(uid, int), 'The user id must be an integer' 24 25 return uid
26. DESIGN BY CONTRACT 1 def get_id(conf_fname=CONFIG_FILE: str) -> Optional[int]: 2 """Return the user id from conf_fname, or None 3 if it hasn't been set yet 4 5 Precondition: 6 Valid JSON-encoded dictionary at conf_name 7 """ 8 9 with open(conf_fname, 'r') as fobj: 10 data = json.loads(fobj) 11 12 uid = data.get('id') 13 14 assert uid is None or isinstance(uid, int)), \ 15 'The user id must be an integer if it exists' 16 17 return uid MY FINAL VERSION
27. RECOVERING FROM ERROR Self-renovation
28. RECOVERING FROM ERROR WAYS TO DEAL WITH ERRORS ‣ Error codes (e.g. Go) ‣ Checked exceptions (e.g. Java) ‣ Unchecked exceptions (Python!) ‣ Abandonment (e.g. Erlang, Midori) RECOVERING FROM ERROR
29. GOLANG 1 type UserInfo struct { 2 id int 3 } 4 5 func getId(configfile) { 6 file, err := ioutil. ReadFile(configfile) 7 if err != nil { 8 return 0, err 9 } 10 11 var userinfo UserInfo 12 13 err := json.Unmarshal( file, &userinfo) 14 if err != nil { 15 return 0, err 16 } 18 19 return userinfo.id, nil 20 }
30. JAVA 1 public int get_id(String conf_fname) throws MalformedDataError { 2 try { 3 JSONParser jsonParser = new JSONParser(); 4 File file = new File(conf_fname); 5 JSONObject config = jsonParser.parse( 6 new FileReader(file)); 7 parseJson(jsonObject); 8 return object.id; 9 } catch (FileNotFoundException | 10 JsonDecodeException ex) { 11 throw new MalformedDataError(ex); 12 } 13 }
31. ‣ Network flakiness ‣ Database out of connections ‣ Disk unavailable ‣ Re-reading corrupt file Well-isolated systems With clear, enforceable interfaces/ contracts RECOVERING FROM ERROR WHAT IS RECOVERABLE?
32. RECOVERING FROM ERROR HOW DO I RECOVER? Abandonment isolates at the process level ‣ What level of isolation makes sense from you? ‣ How can you make sure all bad state is cleared away to retry?
33. LESSONS LEARNED There are many kind of Exceptions in Python ‣ Control flow, bugs, contract failures Make a contract ‣ Document the contract ‣ Use exceptions to attribute breaches in contract Self-heal cleanly ‣ Think about what is recoverable ‣ Clear out bad state, get back to happy ‣ Isolated components are helpful
34. THANKS
35. ADDITIONAL READING http://joeduffyblog.com/2016/02/07/the-error-model/ http://dafoster.net/articles/2016/05/17/abandonment-vs- unchecked-exceptions-for-error-handling/ http://cecs.wright.edu/~pmateti/Courses/7140/ Lectures/OOD/meyer-design-by-contract-1992.pdf