Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Brett Slatkin - Refactoring Python: Why and how...

Brett Slatkin - Refactoring Python: Why and how to restructure your code

As programs gain complexity, it becomes harder to add features and fix bugs. Reorganizing code is an effective way to make programs more manageable. This talk will show you Pythonic ways to do the most imporant "refactorings": Extract variables with __nonzero__; Change signatures with *args and **kwargs; Extract fields and classes with @property; Create stateful closures with __call__; and more!

https://us.pycon.org/2016/schedule/presentation/2073/

PyCon 2016

May 29, 2016
Tweet

More Decks by PyCon 2016

Other Decks in Programming

Transcript

  1. Refactoring Python: Why and how to restructure your code Brett

    Slatkin @haxor onebigfluke.com 2016-05-30T11:30-07:00
  2. • What, When, Why, How • Strategies ◦ Extract Variable

    & Function ◦ Extract Class & Move Fields ◦ Move Field gotchas • Follow-up • Bonus ◦ Extract Closure Agenda
  3. Repeatedly reorganizing and rewriting code until it's obvious* to a

    new reader. What is refactoring? * See Clean Code by Robert Martin
  4. • In advance • For testing • "Don't repeat yourself"

    • Brittleness • Complexity When do you refactor?
  5. What's the difference between good and great programmers? (anecdotally) Me

    usually Great Time spent 0 100% Good Writing & testing Refactoring Style & docs
  6. 1. Identify bad code 2. Improve it 3. Run tests

    4. Fix and improve tests 5. Repeat How do you refactor?
  7. How do you refactor Python? Image © Hans Hillewaert Creative

    Commons Attribution-Share Alike 4.0 International
  8. • Thorough tests • Quick tests • Source control •

    Willing to make mistakes Prerequisites
  9. MONTHS = ('January', 'February', ...) def what_to_eat(month): if (month.lower().endswith('r') or

    month.lower().endswith('ary')): print('%s: oysters' % month) elif 8 > MONTHS.index(month) > 4: print('%s: tomatoes' % month) else: print('%s: asparagus' % month) When should you eat certain foods?
  10. Before if (month.lower().endswith('r') or month.lower().endswith('ary')): print('%s: oysters' % month) elif

    8 > MONTHS.index(month) > 4: print('%s: tomatoes' % month) else: print('%s: asparagus' % month)
  11. lowered = month.lower() ends_in_r = lowered.endswith('r') ends_in_ary = lowered.endswith('ary') index

    = MONTHS.index(month) summer = 8 > index > 4 if ends_in_r or ends_in_ary: print('%s: oysters' % month) elif summer: print('%s: tomatoes' % month) else: print('%s: asparagus' % month) After: Extract variables
  12. def oysters_good(month): lowered = month.lower() return ( lowered.endswith('r') or lowered.endswith('ary'))

    def tomatoes_good(month): index = MONTHS.index(month) return 8 > index > 4 Extract variables into functions
  13. Before if (month.lower().endswith('r') or month.lower().endswith('ary')): print('%s: oysters' % month) elif

    8 > MONTHS.index(month) > 4: print('%s: tomatoes' % month) else: print('%s: asparagus' % month)
  14. After: Using functions if oysters_good(month): print('%s: oysters' % month) elif

    tomatoes_good(month): print('%s: tomatoes' % month) else: print('%s: asparagus' % month)
  15. After: Using functions with variables time_for_oysters = oysters_good(month) time_for_tomatoes =

    tomatoes_good(month) if time_for_oysters: print('%s: oysters' % month) elif time_for_tomatoes: print('%s: tomatoes' % month) else: print('%s: asparagus' % month)
  16. def oysters_good(month): lowered = month.lower() return ( lowered.endswith('r') or lowered.endswith('ary'))

    def tomatoes_good(month): index = MONTHS.index(month) return 8 > index > 4 These functions will get complicated
  17. class OystersGood: def __init__(self, month): lowered = month.lower() self.r =

    lowered.endswith('r') self.ary = lowered.endswith('ary') self._result = self.r or self.ary def __bool__(self): # aka __nonzero__ return self._result Extract variables into classes
  18. class TomatoesGood: def __init__(self, month): self.index = MONTHS.index(month) self._result =

    8 > index > 4 def __bool__(self): # aka __nonzero__ return self._result Extract variables into classes
  19. time_for_oysters = oysters_good(month) time_for_tomatoes = tomatoes_good(month) if time_for_oysters: print('%s: oysters'

    % month) elif time_for_tomatoes: print('%s: tomatoes' % month) else: print('%s: asparagus' % month) Before: Using functions
  20. After: Using classes time_for_oysters = OystersGood(month) time_for_tomatoes = TomatoesGood(month) if

    time_for_oysters: # Calls __bool__ print('%s: oysters' % month) elif time_for_tomatoes: # Calls __bool__ print('%s: tomatoes' % month) else: print('%s: asparagus' % month)
  21. test = OystersGood('November') assert test assert test.r assert not test.ary

    test = OystersGood('July') assert not test assert not test.r assert not test.ary Extracting classes facilitates testing
  22. Things to remember • Extract variables and functions to improve

    readability • Extract variables into classes to improve testability • Use __bool__ to indicate a class is a paper trail
  23. Keeping track of your pet's age class Pet: def __init__(self,

    name, age): self.name = name self.age = age
  24. >>> pet = Pet('Gregory the Gila', 3) print('%s is %d

    years old' % (pet.name, pet.age)) Keeping track of your pet's age Gregory the Gila is 3 years old
  25. class Pet: def __init__(self, name, age): self.name = name self.age

    = age self.treats_eaten = 0 def give_treats(self, count): self.treats_eaten += count Keeping track of your pet's treats
  26. >>> pet = Pet('Gregory the Gila', 3) pet.give_treats(2) print('%s ate

    %d treats' % (pet.name, pet.treats_eaten)) Keeping track of your pet's treats Gregory the Gila ate 2 treats
  27. class Pet: def __init__(self, name, age, *, has_scales=False, lays_eggs=False, drinks_milk=False):

    self.name = name self.age = age self.treats_eaten = 0 self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk Keeping track of your pet's needs
  28. class Pet: def __init__(self, ...): ... def give_treats(self, count): ..

    @property def needs_heat_lamp(self): return ( self.has_scales and self.lays_eggs and not self.drinks_milk) Keeping track of your pet's needs
  29. >>> pet = Pet('Gregory the Gila', 3, has_scales=True, lays_eggs=True) print('%s

    needs a heat lamp? %s' % (pet.name, pet.needs_heat_lamp)) Keeping track of your pet's needs Gregory the Gila needs a heat lamp? True
  30. class Pet: def __init__(self, name, age, *, has_scales=False, lays_eggs=False, drinks_milk=False):

    self.name = name self.age = age self.treats_eaten = 0 self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk It's getting complicated
  31. 1. Add an improved interface ◦ Maintain backwards compatibility ◦

    Issue warnings for old usage 2. Migrate old usage to new usage ◦ Run tests to verify correctness ◦ Fix and improve broken tests 3. Remove code for old interface How do you redraw boundaries?
  32. import warnings warnings.warn('Helpful message') • Default: Print messages to stderr

    • Force warnings to become exceptions: python -W error your_code.py What are warnings?
  33. Before class Pet: def __init__(self, name, age, *, has_scales=False, lays_eggs=False,

    drinks_milk=False): self.name = name self.age = age self.treats_eaten = 0 self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk
  34. After: Extract Animal from Pet class Animal: def __init__(self, *,

    has_scales=False, lays_eggs=False, drinks_milk=False): self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk
  35. class Pet: def __init__(self, name, age, animal=None, **kwargs): if kwargs

    and animal is not None: raise TypeError('Mixed usage') if animal is None: warnings.warn('Should use Animal') animal = Animal(**kwargs) self.animal = animal self.name = name self.age = age self.treats_eaten = 0 After: Backwards compatible
  36. >>> Mixed usage raises exception animal = Animal(has_scales=True, lays_eggs=True) pet

    = Pet('Gregory the Gila', 3, animal, has_scales=False) Traceback ... TypeError: Mixed usage
  37. >>> pet = Pet('Gregory the Gila', 3, has_scales=True, lays_eggs=True) Old

    constructor works, but warns UserWarning: Should use Animal
  38. >>> My pet is Gregory the Gila New constructor usage

    doesn't warn animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal) print('My pet is %s' % pet.name)
  39. class Pet: def __init__(self, name, age, *, has_scales=False, lays_eggs=False, drinks_milk=False):

    ... self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk Before: Fields on self
  40. class Pet: ... @property def has_scales(self): warnings.warn('Use animal attribute') return

    self.animal.has_scales @property def lays_eggs(self): ... @property def drinks_milk(self): ... After: Move fields to inner object
  41. >>> Old attributes issue a warning UserWarning: Use animal attribute

    Gregory the Gila has scales? True animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal) print('%s has scales? %s' % (pet.name, pet.has_scales))
  42. >>> New attributes don't warn Gregory the Gila has scales?

    True animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal) print('%s has scales? %s' % (pet.name, pet.animal.has_scales))
  43. class Pet: def __init__(self, ...): ... def give_treats(self, count): ..

    @property def needs_heat_lamp(self): return ( self.has_scales and self.lays_eggs and not self.drinks_milk) Before: Helpers access self
  44. class Pet: def __init__(self, ...): ... def give_treats(self, count): ..

    @property def needs_heat_lamp(self): return ( self.animal.has_scales and self.animal.lays_eggs and not self.animal.drinks_milk) After: Helpers access inner object
  45. >>> Existing helper usage doesn't warn Gregory the Gila needs

    a heat lamp? True animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', 3, animal) print('%s needs a heat lamp? %s' % (pet.name, pet.needs_heat_lamp))
  46. • Split classes using optional arguments to __init__ • Use

    @property to move methods and fields between classes • Issue warnings in old code paths to find their occurrences Things to remember
  47. class Animal: def __init__(self, *, has_scales=False, lays_eggs=False, drinks_milk=False): ... class

    Pet: def __init__(self, name, age, animal): ... Before: Is this obvious?
  48. class Animal: def __init__(self, age=None, *, has_scales=False, lays_eggs=False, drinks_milk=False): ...

    class Pet: def __init__(self, name, animal): ... After: Move age to Animal
  49. class Animal: def __init__(self, age=None, *, has_scales=False, lays_eggs=False, drinks_milk=False): if

    age is None: warnings.warn('age not specified') self.age = age self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk After: Constructor with optional age
  50. class Pet: def __init__(self, name, maybe_age, maybe_animal=None): if maybe_animal is

    not None: warnings.warn('Put age on animal') self.animal = maybe_animal self.animal.age = maybe_age else: self.animal = maybe_age ... After: Pet constructor with optional age
  51. class Pet: def __init__(self, name, maybe_age, maybe_animal=None): ... def give_treats(self,

    count): ... @property def age(self): warnings.warn('Use animal.age') return self.animal.age After: Compatibility property age
  52. >>> animal = Animal(has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila',

    3, animal) print('%s is %d years old' % (pet.name, pet.age)) After: Old usage has a lot of warnings UserWarning: age not specified UserWarning: Put age on animal UserWarning: Use animal.age Gregory the Gila is 3 years old
  53. >>> After: New usage has no warnings Gregory the Gila

    is 3 years old animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) print('%s is %d years old' % (pet.name, pet.animal.age))
  54. animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila',

    animal) pet.age = 5 Gregory is older than I thought
  55. >>> Assigning to age breaks! AttributeError: can't set attribute animal

    = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 # Error
  56. class Pet: ... @property def age(self): warnings.warn('Use animal.age') return self.animal.age

    @age.setter def age(self, new_age): warnings.warn('Assign animal.age') self.animal.age = new_age Need a compatibility property setter
  57. >>> animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the

    Gila', animal) pet.age = 5 Old assignment now issues a warning UserWarning: Assign animal.age
  58. >>> New assignment doesn't warn Gregory the Gila is 5

    years old animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.animal.age = 5 print('%s is %d years old' % (pet.name, pet.animal.age))
  59. class Animal: def __init__(self, age, *, has_scales=False, lays_eggs=False, drinks_milk=False): self.age

    = age self.has_scales = has_scales self.lays_eggs = lays_eggs self.drinks_milk = drinks_milk ... Finally: age is part of Animal
  60. class Pet: def __init__(self, name, animal): self.animal = animal self.name

    = name self.treats_eaten = 0 ... Finally: Pet has no concept of age
  61. animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila',

    animal) pet.age = 5 print('%s is %d years old' % (pet.name, pet.animal.age)) Again: Gregory is older than I thought
  62. >>> Surprise! Old usage is doubly broken Gregory the Gila

    is 3 years old animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 print('%s is %d years old' % (pet.name, pet.animal.age))
  63. >>> Surprise! Old usage is doubly broken Gregory the Gila

    is 3 years old animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 # No error! print('%s is %d years old' % (pet.name, pet.animal.age))
  64. class Pet: ... @property def age(self): raise AttributeError('Use animal') @age.setter

    def age(self, new_age): raise AttributeError('Use animal') Need defensive property tombstones
  65. >>> Now accidental old usage will break Traceback ... AttributeError:

    Use animal animal = Animal(3, has_scales=True, lays_eggs=True) pet = Pet('Gregory the Gila', animal) pet.age = 5 # Error
  66. Things to remember • Use @property.setter to move fields that

    can be assigned • Defend against muscle memory with tombstone @propertys
  67. Links • PMOTW: Warnings - Doug Hellmann • Stop Writing

    Classes - Jack Diederich • Beyond PEP 8 - Raymond Hettinger
  68. • This talk's code & slides: ◦ github.com/bslatkin/pycon2016 • My

    book: EffectivePython.com ◦ Discount today: informit.com/deals • Me: @haxor and onebigfluke.com Links
  69. class Grade: def __init__(self, student, score): self.student = student self.score

    = score grades = [ Grade('Jim', 92), Grade('Jen', 89), Grade('Ali', 73), Grade('Bob', 96), ] Calculating stats for students
  70. def print_stats(grades): total, count, lo, hi = 0, 0, 100,

    0 for grade in grades: total += grade.score count += 1 if grade.score < lo: lo = grade.score elif grade.score > hi: hi = grade.score print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi)) Calculating stats for students
  71. Before def print_stats(grades): total, count, lo, hi = 0, 0,

    100, 0 for grade in grades: total += grade.score count += 1 if grade.score < lo: lo = grade.score elif grade.score > hi: hi = grade.score print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi))
  72. After: Extract a stateful closure def print_stats(grades): total, count, lo,

    hi = 0, 0, 100, 0 def adjust_stats(grade): # Closure ... for grade in grades: adjust_stats(grade) print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi))
  73. Stateful closure functions are messy def print_stats(grades): total, count, lo,

    hi = 0, 0, 100, 0 def adjust_stats(grade): nonlocal total, count, lo, hi total += grade.score count += 1 if grade.score < lo: lo = grade.score elif grade.score > hi: hi = grade.score ...
  74. class CalculateStats: def __init__(self): self.total = 0 self.count = 0

    self.lo = 100 self.hi = 0 def __call__(self, grade): ... @property def avg(self): ... Instead: Stateful closure class
  75. class CalculateStats: def __init__(self): ... def __call__(self, grade): self.total +=

    grade.score self.count += 1 if grade.score < self.lo: self.lo = grade.score elif grade.score > self.hi: self.hi = grade.score Instead: Stateful closure class
  76. class CalculateStats: def __init__(self): ... def __call__(self, grade): ... @property

    def avg(self): return self.total / self.count Instead: Stateful closure class
  77. def print_stats(grades): total, count, lo, hi = 0, 0, 100,

    0 for grade in grades: total += grade.score count += 1 if grade.score < lo: lo = grade.score elif grade.score > hi: hi = grade.score print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi)) Before
  78. Before: Closure function def print_stats(grades): total, count, lo, hi =

    0, 0, 100, 0 def adjust_stats(grade): # Closure ... for grade in grades: adjust_stats(grade) print('Avg: %f, Lo: %f Hi: %f' % (total / count, lo, hi))
  79. def print_stats(grades): stats = CalculateStats() for grade in grades: stats(grade)

    print('Avg: %f, Lo: %f Hi: %f' % (stats.avg, stats.lo, stats.hi)) After: Using stateful closure class
  80. • Extracting a closure function can make code less clear

    • Use __call__ to indicate that a class is just a stateful closure • Closure classes can be tested independently Things to remember