$30 off During Our Annual Pro Sale. View Details »

Move Fast and Break Nothing

Zach Holman
October 01, 2014

Move Fast and Break Nothing

Move fast and break things. Scale later. Ship it. They're the rallying cries of our industry. While they've helped propel us forward, they're still crude solutions. We're moving fast, but what specifically is okay for us to break, and how much can we break it? How do we quickly ship really difficult changes? The next step is being smarter about the answers to these questions. Turns out, there are some really cool approaches we can take these days to keep us fast while your company grows.

You can also read the full text accompaniment to this talk at: http://zachholman.com/talk/move-fast-break-nothing/

Zach Holman

October 01, 2014
Tweet

More Decks by Zach Holman

Other Decks in Programming

Transcript

  1. break
    nothing
    move fast
    &

    View Slide

  2. View Slide

  3. Move fast and break things
    — Mark Zuckerberg, Facebook
    “ ”

    View Slide

  4. Move fast and fix things
    — Mark Zuckerberg, Facebook
    “ ”

    View Slide

  5. Move fast with stable infra
    — Mark Boringberg, Facebook
    “ ”

    View Slide

  6. Move fast and be bold

    — Miguel Velazquez, Facebook

    View Slide

  7. Drop the ‘the’ — it’s cleaner.

    — JUSTIN TIMBERLAKE SWOOOOOON

    View Slide

  8. good philosophy
    moving quickly and safely is

    View Slide

  9. what does it mean
    in practice?

    View Slide

  10. @ holman

    View Slide

  11. @ holman
    !

    View Slide

  12. View Slide

  13. View Slide

  14. got me thinking about
    building environments
    where
    success
    is a byproduct

    View Slide

  15. code
    3 areas
    , process& talk

    View Slide

  16. code

    View Slide

  17. move fast

    View Slide

  18. people like fast
    better product

    View Slide

  19. people like fast
    see what works

    View Slide

  20. people like fast
    momentum

    View Slide

  21. break things
    and

    View Slide

  22. break things
    fix later and… stuff
    and

    View Slide

  23. some things
    you can’t
    break

    View Slide

  24. billing· permissions
    · upgrades· migrations

    View Slide

  25. how do you
    move fast
    in dicey code?

    View Slide

  26. we redid github’s
    permissions code

    View Slide

  27. hella scary.

    View Slide

  28. .01% errors
    unacceptable
    =

    View Slide

  29. 0 errors
    mandatory
    =

    View Slide

  30. how do we ship this
    fast + safe?

    View Slide

  31. tests are great,
    of course

    View Slide

  32. production can differ
    from what your tests expect

    View Slide

  33. production can differ
    from what you even expect

    View Slide

  34. ensure that your code
    doesn’t change production behavior

    View Slide

  35. parallel
    codepaths

    View Slide

  36. request

    View Slide

  37. user.old_code
    always gets run
    request

    View Slide

  38. user.new_code
    also gets run
    request
    user.old_code
    always gets run

    View Slide

  39. user.new_code
    also gets run
    request
    user.old_code
    always gets run
    response

    View Slide

  40. run both,
    compare

    View Slide

  41. /github/dat-science
    p

    View Slide

  42. science “new-auth” do | e |
    e.control { user.slow_auth }
    e.candidate { user.fast_auth }
    end

    View Slide

  43. science “new-auth” do | e |
    e.control { user.slow_auth }
    e.candidate { user.fast_auth }
    end
    always runs and
    returns your result

    View Slide

  44. can be run as
    a percentage
    science “new-auth” do | e |
    e.control { user.slow_auth }
    e.candidate { user.fast_auth }
    end

    View Slide

  45. collects the
    two results
    science “new-auth” do | e |
    e.control { user.slow_auth }
    e.candidate { user.fast_auth }
    end

    View Slide

  46. attempts vs. mismatches

    View Slide

  47. 75th and 99th percentile performance

    View Slide

  48. quickly iterate
    until you can
    prove safety

    View Slide

  49. existing process
    build into your

    View Slide

  50. what do you have today
    that you can already use?

    View Slide

  51. each layer of
    process added
    is expensive

    View Slide

  52. instead grow
    process laterally

    View Slide

  53. tests rule everything
    around me

    View Slide

  54. ci as code
    maintenance

    View Slide

  55. failing test:
    removing html class
    without cleaning css

    View Slide

  56. failing test:
    removing css without
    removing html class

    View Slide

  57. failing test:
    adding that’s
    not on our cdn

    View Slide

  58. failing test:
    invalid css (scss linting)

    View Slide

  59. automates gruntwork
    out of code review

    View Slide

  60. we test blog posts, too

    View Slide

  61. failing test:
    images off our cdn

    View Slide

  62. failing test:
    image non-retina
    or too big

    View Slide

  63. failing test:
    using an oft-used
    phrase

    View Slide

  64. pro-
    cess

    View Slide

  65. what’s
    important?

    View Slide

  66. feature launches need:
    · designers
    developers
    · product managers
    · qa
    · lawyers
    · marketing· ops

    View Slide

  67. text files
    are rad

    View Slide


  68. View Slide


  69. anpp
    (apple new product process)

    View Slide


  70. (apple new product process)
    giant-ass
    checklist
    anpp

    View Slide


  71. (apple new product process)
    lists at tim-level
    and small team-level
    anpp

    View Slide


  72. (apple new product process)
    happens as
    very first step
    anpp

    View Slide

  73. d

    View Slide

  74. d
    ship lists

    View Slide

  75. d
    ship lists
    happens often
    as last step

    View Slide

  76. d
    ship lists
    all the todos for
    teams for their +1

    View Slide

  77. View Slide

  78. simple todo lists
    make you think less

    View Slide

  79. ownership

    View Slide

  80. don’t throw code
    over the wall

    View Slide

  81. tie code to writers
    d

    View Slide

  82. tie code to writers
    d
    notifies you
    if you screw up

    View Slide

  83. View Slide

  84. authors know
    their own code

    View Slide

  85. authors should
    fix their bugs

    View Slide

  86. how can
    your process
    help out?

    View Slide

  87. tie code to writers

    View Slide

  88. tie code to writers

    View Slide

  89. tie code to writers
    chrome has
    owner files

    View Slide

  90. tie code to writers
    dictates
    responsibility

    View Slide

  91. tie code to writers
    makes
    mentors visible

    View Slide

  92. tie code to writers
    d

    View Slide

  93. tie code to writers
    d
    per-file
    ownership

    View Slide

  94. tie code to writers
    d
    class BranchesController
    areas_of_responsibility :git
    end

    View Slide

  95. View Slide

  96. View Slide

  97. tie code to writers
    d
    issues get assigned
    to the appropriate team

    View Slide

  98. talk

    View Slide

  99. more
    communication!
    yayyyyyyyycrap

    View Slide

  100. email
    meetings
    chat
    issues
    im
    on-call
    video
    al

    View Slide

  101. more is good
    better is better

    View Slide

  102. most things
    aren’t emergencies

    View Slide

  103. most things
    aren’t emergencies
    be mindful of time

    View Slide

  104. measure impact
    of your actions
    be mindful of time

    View Slide

  105. imagine:

    View Slide

  106. 1. page a coworker for help
    2. they get woken up
    3. their phone takes a selfie of them
    4. selfie gets posted into chat

    View Slide

  107. View Slide

  108. would this make you respect
    your coworker’s time more?

    View Slide

  109. add empathy to
    your process

    View Slide

  110. empathy
    stems from being
    exposedto real pain

    View Slide

  111. everyone should
    do some support

    View Slide

  112. there’s a difference
    between
    reading about it
    helping with it
    &

    View Slide

  113. institutional
    teaching

    View Slide

  114. [we] have a responsibility to
    be teachers—that this should
    be a central part of [our] jobs

    — Ed Catmull, Creativity, Inc.

    View Slide

  115. …it’s just logic that someday
    we won’t be here.
    — Ed Catmull, Creativity, Inc.

    View Slide

  116. more productive
    if workers are smart

    View Slide

  117. how do you teach
    without making it lame?

    View Slide

  118. chatops:
    learning by osmosis

    View Slide

  119. app crashes. here’s a normal flow:

    View Slide

  120. 1. employee gets paged
    2. they ssh into… something
    3. they fix it… somehow
    app crashes. here’s a normal flow:

    View Slide

  121. no visibility means no teaching

    View Slide

  122. no visibility means no improvement

    View Slide

  123. d
    HUBOT

    View Slide

  124. 1. employee gets paged
    2. they manage it in a chat room
    3. they fix it and people can watch
    here’s a better flow:
    app crashes.

    View Slide

  125. visibility means teaching

    View Slide

  126. visibility means improvement

    View Slide

  127. real-life debugging

    View Slide

  128. github’s wifi is down
    floor two, san francisco
    (right now!)

    View Slide

  129. View Slide

  130. View Slide

  131. View Slide

  132. View Slide

  133. debug in the open

    View Slide

  134. feedback

    View Slide

  135. blue angels
    the
    feedback
    on

    View Slide

  136. View Slide

  137. two-way
    street

    View Slide

  138. two-way
    street
    how well do
    you receive
    feedback?

    View Slide

  139. two-way
    street
    how well do
    you give
    feedback?

    View Slide

  140. improving
    feedbackacross an org

    View Slide

  141. View Slide

  142. caution
    with a degree of
    moving fast

    View Slide

  143. need to be fast

    View Slide

  144. need to be safe

    View Slide

  145. takes effort
    to achieve both

    View Slide

  146. @ holman

    View Slide