Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Changing the Laws of Engineering with GitHub Pu...

Changing the Laws of Engineering with GitHub Pull Requests (Velocity Santa Clara 2015)

Every engineering team has processes they follow to get things done in the way that works best for them. Whether they’re in someone’s head or formally documented, they have to be communicated and shared because they are an implementation of the team’s culture at that point in time. But for a company and its engineering team to sustain its culture as it grows, even more important than what those processes are or how they’re recorded is how they undergo change.

At New Relic, we have a culture of openness and inclusivity that we intend to keep, so we’ve been using the same process to change our culture that we use to change our code: GitHub Pull Requests. We’ve learned some advantages to this approach over what we had used before: meetings, email, and wikis. These advantages include:

- A workflow that is a natural fit for engineers
- Less discussion happening in exclusive settings like meetings
- The “Why” for every change recorded in one place
- Freedom of thought that arises when you can clone the entire “book of laws” and reimagine it as you think it should be

Our entire engineering team, and others in the company we collaborate with, are now contributors to shaping our process in real time. We are closing the gap between our culture and our processes every day as engineers and managers alike propose and debate changes. We’ll share what’s worked well and what still needs help.

After this talk, you’ll be thinking about your culture and process, how to keep them in step, and most of all why its important to do so.

Ralph Bodenner

May 29, 2015
Tweet

More Decks by Ralph Bodenner

Other Decks in Technology

Transcript

  1. Changing the Laws of Engineering With GitHub Pull Requests Ralph

    Bodenner Director of Engineering, New Relic @ralphbod Slides https://bit.ly/pull-request-your-culture Wednesday, May 27, 15 30 minutes Culture. Relation to process, documentation -When you leave, you'll be thinking about your engineering processes. -Asking yourself why and how you change them. -You will have more powerful tools for sustaining your engineering culture.
  2. Wednesday, May 27, 15 Tell our story of how New

    Relic Engineerings makes our processes serve our culture. Our story starts with a story about another company.
  3. Copyright 2015 Valve Corporation Wednesday, May 27, 15 Inspiration is

    this. (Read it?) Back in 2012, this document was leaked. Valve, the famous game developer, had a flat org structure.
  4. Copyright 2015 Valve Corporation Wednesday, May 27, 15 Their handbook

    paints a picture a lovely culture of autonomy and shipping awesome software. Described how to do it. Methods/processes.
  5. LET’S DO THIS Wednesday, May 27, 15 In our way

    at the time, with managers in a room... 3 years later
  6. Wednesday, May 27, 15 This is ours! That is not

    an image of an insignia from a space-traveling TV show owned by a multinational media conglomerate. But the artifact is not as important as how we create it.
  7. Wednesday, May 27, 15 And our way of writing it:

    “Process PRs” ... After much debate and learning. Needed the document, but the process of writing is the most important
  8. Culture Why you do things. Metaprocess How you change. Process

    What you do, today. Entropy Holding you down. Wednesday, May 27, 15 Terms Aspirational
  9. 1,054 64 187 25 comments commenters (~30% of Engineering) PRs

    merged (+30 closed) engineers committing (+25 others) Wednesday, May 27, 15 Results since July 2014 (10 months) More engineers than managers are commenting (38 vs. 26)
  10. Processes embody culture Processes are repeatable (efficient, fair) Transparent Inclusive

    Discoverable Single source of truth, written down Update/delete easily Bias for action Autonomy Culture > Process What we want Wednesday, May 27, 15 Learned what we want today after 5 years of experimentation
  11. The Great Title Debate 1 / 5 Wednesday, May 27,

    15 Right around the time of Valve handbook, engineering management discussed, in meetings over the course of many weeks, what our titles should be.
  12. Titles Compensation Promotions Reviews Wednesday, May 27, 15 We want

    these to be consistent and understood by everyone. That would embody our culture, of transparency and equity. Anyone should be able to read what titles are defined by.
  13. Wednesday, May 27, 15 Engineer who coined “software engineering” Margaret

    Hamilton, lead flight software engineer on Apollo mission What did we end up with?
  14. Software Engineer Senior Software Engineer Lead Software Engineer Principal Software

    Engineer Distinguished Software Engineer Engineering Manager Senior Engineering Manager Director of Engineering Wednesday, May 27, 15 We didn’t put on the cowboy hat. Lots of reasons for what we ended up with. Not ideal (as in idealistic). How you promote and recognize people embodies your culture.
  15. Process == What you do because it works at the

    time. Wednesday, May 27, 15 This talk: Not about which ones you’re using. It works for the people, it works for the business. It works now. You want to keep doing what works. It embodies your culture.
  16. Ye Olde Metaprocess Wednesday, May 27, 15 Meeting + Email

    + Wiki People get left out who can’t (remote) or shouldn’t (engineers) be in the meeting. Conversation happening for those privileged by role or location or start date.
  17. 150 Wednesday, May 27, 15 Dunbar’s Number, which taught us

    that we must write things down. We’re too big to convey all the nuances of how things have changed since you wrote it down, since the changes are only in your heads. Multiple offices in multiple timezones. Oral culture --> written culture. Slow-to-change written word is the medium.
  18. Processes embody culture Processes are repeatable (efficient, fair) Transparent Inclusive

    Discoverable Single source of truth, written down What we want Wednesday, May 27, 15 We’re well beyond “oral tradition” scale. How will we write it down?
  19. HOW ABOUT A WIKI? Wednesday, May 27, 15 This is

    Ward. - Backpack - Wikispaces - Confluence
  20. The Append-only Wiki 2 / 5 Wednesday, May 27, 15

    People just added on, they didn’t change or remove. There were comments. Big pile of crap hiding the gems. We had to do annual purges. Became untrustworthy. But we have Ward!
  21. Copypasta feeds inertia Wednesday, May 27, 15 Not just the

    time spent rewriting that slows you down. People read stuff and believe it. You continue in wrong directions.
  22. Processes embody culture Processes are repeatable (efficient, fair) Transparent Inclusive

    Discoverable Single source of truth, written down Update/delete easily What we want Wednesday, May 27, 15 “Easily” not just in UI, but organizationally Reinforced by our next story...
  23. The Zombie Manual Tests 3 / 5 Wednesday, May 27,

    15 We used to do manual tests after every production deploy. We had a wiki page on it, a list of accounts to test with. New engineers would read about the tests and take them really literally. We never found problems, but spend hundreds of person-hours doing it. Can’t afford that. We deleted the wiki pages.
  24. Wednesday, May 27, 15 There was a good reason for

    the crap that doesn’t work anymore. Well-meaning! But confusing. We care about customer experience. We operate a SaaS, where production is the one true reality. (I know the appendix is actually useful... but this is funny.)
  25. Wednesday, May 27, 15 Why delete them? We didn’t want

    our process to become untrustworthy. Misleading is worse than missing. Read well-meaning but wrong stuff too many times and it loses any credibility it might have had.
  26. Culture > Process Wednesday, May 27, 15 If had left

    process the same, people would’ve continued to waste time on things that didn’t have value. We value spending time... valuably. With this process, that aspect of culture was being eroded. Culture should outlast Process.
  27. Culture is the reason for Process. Wednesday, May 27, 15

    This is the core problem: We care about our culture. We want it to live and thrive. That’s why we have processes.
  28. Entropy Wednesday, May 27, 15 As soon as you write

    it, it’s out of date. If processes don’t change to close the gap with your culture, you lose to entropy.
  29. You need a way to close the Culture/Process gap. Wednesday,

    May 27, 15 Metaprocess! If that dev had updated the manual test doc, we’d have wasted less time.
  30. Processes embody culture Processes are repeatable (efficient, fair) Transparent Inclusive

    Discoverable Single source of truth, written down Update/delete easily Autonomy Bias for action What we want Wednesday, May 27, 15 “Easily” not just in UI, but organizationally
  31. This document and the information herein (including any information that

    may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided. Copyright 2008-2015 New Relic, Inc. All rights reserved. Our business is not a democracy. 4 / 5 Wednesday, May 27, 15 Laws, boards of directors get to say no.
  32. The Boss Wednesday, May 27, 15 Can’t have just anyone

    change processes, in particular those that affect money or private information.
  33. Open Source is not a democracy. Wednesday, May 27, 15

    We have a few committers, everyone’s a contributor. Everyone has a voice, but not a vote.
  34. Processes embody culture Processes are repeatable (efficiency) Transparent Inclusive Discoverable

    Single source of truth, written down Update/delete easily Autonomy Bias for action Editorial control What we want Wednesday, May 27, 15
  35. The idea 5 / 5 Wednesday, May 27, 15 Growing.

    20 five years ago, to 500-something employees by the time we filed our S-1 last year. A company we admire, Simple, doing engineering in Portland. They wrote architecture docs in GitHub. Commit the docs first, then the software becomes real after review. Moved our operational docs into READMEs, some repos just about describing our systems.
  36. Culture as Code Wednesday, May 27, 15 Culture as code.

    Same workflow as for code. That works for engineers. Lower mental barrier to contribution. Wiki was “a manager tool”.
  37. Processes embody culture Processes are repeatable (efficient, fair) Inclusive Transparent

    Discoverable Single source of truth, written down Update/delete easily Bias for action Autonomy Editorial control What we got Wednesday, May 27, 15 Not working perfectly for non-technical people UI could be easier What did we learn?
  38. Do write together Wednesday, May 27, 15 Pair programming. Focused

    conversation of a few people generates a proposal that is worth sharing. Draft outside the PR.
  39. Actively discourage email discussion Wednesday, May 27, 15 Structure your

    repos/orgs so people can use notifications. Close email threads with BCC and paste them into PRs.
  40. Keep comments open after merge ~50% of discussion after merge

    Wednesday, May 27, 15 I typed into a comment field while someone was talking to me.
  41. “Lowers the contribution barrier...” Wednesday, May 27, 15 Everyone has

    to understand git/GitHub to contribute. No Edit button. Wiki or Quip is better at this. Do anything you can to lower barrier. Train your non-technical staff. Train your technical staff.
  42. git clone culture 6 / 5 Wednesday, May 27, 15

    You can edit the whole world and propose something new. No need to get it right before you edit, or need to get rights. Freedom to think of the world as it should be.
  43. Why did we think this was a good idea? Wednesday,

    May 27, 15 What did you know at the time? What assumptions went unchallenged? What was important at the time?
  44. Why > What Wednesday, May 27, 15 It’s hard to

    read the why in a document meant to efficiently describe a process.
  45. Junior Software Engineer Software Engineer Senior Software Engineer Principal Software

    Engineer Distinguished Software Engineer Engineering Manager Senior Engineering Manager Director of Engineering Wednesday, May 27, 15
  46. Software Engineer Senior Software Engineer Lead Software Engineer Principal Software

    Engineer Distinguished Software Engineer Engineering Manager Senior Engineering Manager Director of Engineering Wednesday, May 27, 15
  47. Ask why. Wednesday, May 27, 15 Everyone should be able

    to ask why. Our culture (autonomy, boldness, urgency, customer focus) demands it.
  48. Acknowledgements This talk was the work of at least 18

    people. @belindarunkle @danarelic @feministy @foliosus @kfrugia @kwugirl Matt Otis @mewzherder @mflaming @nerdygirl @nicbenders @qkate @relistan @spkane @SXavier_NwRelic @WardCunningham @xensesthegreat Thanks for your help in shaping this talk and in making New Relic awesome! Wednesday, May 27, 15
  49. References • Valve Handbook for New Employees http://www.valvesoftware.com/company/ Valve_Handbook_LowRes.pdf •

    GitHub: Quick Pull Requests https://github.com/blog/1945-quick-pull-requests Wednesday, May 27, 15
  50. Image References • Valve Working without a Boss http://www.blogcdn.com/www.joystiq.com/media/2012/04/screen-shot-2012-04-23-at-11.24.33-am.jpg •

    Valve cover http://i.kinja-img.com/gawker-media/image/upload/s--ANyPdiwb--/17k9nffv4tn36jpg.jpg • Constitutional Convention, from http://en.wikipedia.org/wiki/Convention_to_propose_amendments_to_the_United_States_Constitution (Public Domain) • Margaret Hamilton with her code http://upload.wikimedia.org/wikipedia/commons/2/2e/Margaret_Hamilton.gif (Public Domain) • Ward Cunningham http://upload.wikimedia.org/wikipedia/commons/3/30/Ward_Cunningham_1.jpg (Used with permission) • CA carcinogen warning sign http://images.mysafetysign.com/img/lg/S/chemical-cause-cancer-warning-sign-s-0289.png • Scumbag appendix http://imgur.com/gallery/254ANHP • Pair keypunching http://en.wikipedia.org/wiki/File:Keypunching_at_Texas_A%26M2.jpg Wednesday, May 27, 15