Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa 2014

Have you ever wrote code that worked when you wrote it, but failed when it went to QA? Or staging? Or production? Learning how to build a high quality development environment is a serious undertaking and will involve a lot of trial and error — unless you come to this talk. Having spent the better part of the last year prepping, implementing, rolling out, maintaining, stabilizing, and growing a large scale development environment platform, I will show you the ropes. The real secret to building a system everyone can love: it’s all about making eggs, not bacon.

Find out more at: http://jmather.com/talks/2014-02-26/cookbook/

Transcript

1. Much more Eggs than Bacon 
 A Development Environment
 Jacob Mather Software Engineer, Mashery Cookbook

2. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Who am I? What I do: • Software Engineer at Mashery • Co-organizer of the
 San Francisco PHP User Group • Community Evangelist Where I can be found: • Blog: http://jmather.com • Twitter: @thejmather
3. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Why I am giving this talk
4. Let’s talk about
 why we need environments
5. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 What environments could we have? • Production • Where people paying money interact • Staging • Test your production deployment plan • Quality Assurance • Cheaper version of staging to do comprehensive testing • Development • Designed for easy debugging
6. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Production Server Server Server Load Balancer Service 1 Server Server Server Load Balancer Service 2 Server Server Server Load Balancer Web Portal DB Cache
7. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Staging Server Server Load Balancer Service 1 Server Server Load Balancer Service 2 Server Server Load Balancer Web Portal DB Cache
8. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 QA Server Service 1 Server Service 2 Server Web Portal DB Cache
9. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Development Server Service 1 Service 2 DB Cache Web Portal
10. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Why are these differences important?
11. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development
12. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA
13. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA Staging
14. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA Staging Production
15. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 What happens when they are too different?
16. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development
17. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA
18. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA
19. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA
20. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA Staging
21. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA Staging
22. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA Staging
23. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA Staging Production
24. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA Staging Production
25. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Lifecycle of Code Development QA Staging Production
26. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Why where you develop matters • Environment based bugs are costly to track down • If you can't reproduce in dev, you are shooting in the dark • Devs need to share code without slowing down • Makes infrastructure more transparent
27. Let’s talk about
 getting ready for change
28. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Devise your minimum viable stack
29. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Server configuration automation • If you have no current automation • Chef • Puppet • Ansible • If you do have automation • Ops will need to find and remove all production secrets • Ops will need to add a way for you to customize
30. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Code cleanup • Force all configuration to be explicit • Not based on path or host name • Remove magic strings
31. Let’s talk about
 your choices when implementing
32. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Choose your platform • Traditional VM • VirtualBox (version 4.2.16 — trust me.) • VMWare • Parallels • Cloud • AWS • Digital Ocean • Rackspace • Container VM (Docker)
33. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Verification • Smoke tests • Manual inspection • Functional tests
34. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Points to consider • CLI interface • Learn from my mistake • Use un-routed domains for virtual hosts • Test with all internet disabled
35. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 At this point you should be able to build a fully functional environment that you are capable of developing software in.
36. Let’s talk about
 getting in to the hands of users
37. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Make or break time • Prepare as much documentation as you can stomach • Prepare for lots of confusion and frustration
38. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Choose your timing carefully You only get one chance at a first impression. ! The first time a user spins up their new environment, it must work flawlessly. ! The entire rest of their experience will be influenced by this very first interaction.
39. Let’s talk about
 what happens after deployment
40. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 One thing we all have in common... Everyone loves to customize their computer! ! Someone will use CDPATH which makes cd output text. ! A few will use rvm and not appreciate your automated (and incorrect) help installing gems. ! Nobody likes you fiddling with their computer.
41. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Start building a troubleshooting document The less time you have to spend debugging other people’s environments, the more time you have to rapidly iterate. ! No only do you need to have them, they need to be accessible, and make sense, and have enough context.
42. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Document how to tinker with the system The sooner you can get others involved with adding features, the faster you will be able to keep up with the ever changing needs of your new development environment. ! It also lets others try things experimentally, without involving you at all!
43. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Tips for making improvements • Slow and steady wins the race. • Do not change your environment’s interface. • Except when there’s great value in it. • Even then it’s really hard to justify.
44. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 48 Minutes
45. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 30 Minutes
46. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 22 Minutes
47. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Replicating production bugs in development
48. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Make reporting problems easy . easy
49. Let’s talk about
 chasing the myth of stability
50. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 The march of progress will continue • Ops updates will break your dev environments. • You will find new third parties your builds depend on. • 100 scratch environment builds a week will make 1 in 100 events happen once a week. • You’ll start to have others contribute infrastructure changes to meet their project’s needs.
51. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Finding your environment’s dependencies • Your environment depends on: • Your own bootstrapping code • Ops automation code • Random third party software Ops is installing • … anything else?
52. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Managing your environment’s dependencies • Time to build a CI process • You’ll need a desktop to test accurately • Watch for changes from all relevant repos • Remember it is pairs of commits that are tested
53. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Making a quick recovery Check to see if your platform of choice supports snapshots. ! When environments go sideways, they can be hard to restore. ! Snapshots eliminate troubleshooting, and take ~20 seconds to run a restore.
54. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 “I just updated trunk and EVERYTHING IS BROKEN!”
55. Let’s talk about
 handing off to a team
56. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Eventually you will need to scale, but… • There’s a lot of domain specific knowledge in your head. • Do you know how to run a team? • Do you know how to lead a team? • Do you know how to be a product owner?
57. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Plus, letting go is hard.
58. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 So…
59. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 this is cool and all but…
60. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 what’s that stuff about eggs and bacon?
61. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 The chicken and the pig.
62. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Cool story bro, but what are you really trying to say?
63. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 ! We are hiring at
64. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014 Thank you! More information about this talk can be found at: ! http://bit.ly/clp14-cookbook ! RATE MY TALK AT: http://spkr8.com/t/29431 ! PLEASE LEAVE FEEDBACK It’s the only way I can improve