Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP 2014

Have you ever wrote code that worked when you wrote it, but failed when it went to QA? Or staging? Or production? Learning how to build a high quality development environment is a serious undertaking and will involve a lot of trial and error — unless you come to this talk. Having spent the better part of the last year prepping, implementing, rolling out, maintaining, stabilizing, and growing a large scale development environment platform, I will show you the ropes. The real secret to building a system everyone can love: it’s all about making eggs, not bacon.

Find out more at http://jmather.com/talks/midwest-php-2014/cookbook

Transcript

1. Much more Eggs than Bacon 
 A Development Environment
 Jacob Mather Software Engineer, Mashery Cookbook

2. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Who am I? What I do: • Software Engineer at Mashery • Co-organizer of the
 San Francisco PHP User Group • Community Evangelist Where I can be found: • Blog: http://jmather.com • Twitter: @thejmather
3. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Why I am giving this talk
4. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 People who are really serious about software should make their own hardware. - Alan Kay
5. Let’s talk about
 what environments we need
6. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 What environments should we have?
7. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 What environments should we have… and what is each environment’s job?
8. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Production
9. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Staging
10. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Quality Assurance
11. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Development
12. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Cache Cache Cache Server Production Server Server Server Load Balancer Service 1 Server Server Server Server Load Balancer Service 2 Server Server Server Server Load Balancer Web Portal DB Cache
13. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Cache Cache Staging Server Server Server Load Balancer Service 1 Server Server Server Load Balancer Service 2 Server Server Server Load Balancer Web Portal DB Cache
14. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Cache Quality Assurance Server Server Load Balancer Service 1 Server Server Load Balancer Service 2 Server Server Load Balancer Web Portal DB Cache
15. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Development Server Service 1 Service 2 DB Cache Web Portal
16. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Why are these differences important?
17. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development
18. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development QA
19. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development QA Staging
20. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development QA Staging Production
21. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 What happens when they are too different?
22. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development
23. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development QA
24. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development QA
25. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development QA Staging
26. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development QA Staging
27. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development QA Staging Production
28. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Lifecycle of Code Development QA Staging Production
29. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Why where you develop matters • Environment based bugs are costly to track down • If you cannot reproduce a bug in dev, you are shooting in the dark • Devs need to share code without slowing down • Makes infrastructure more transparent
30. I’m on board,
 so how do I get started?
31. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Devise your minimum viable stack
32. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Server configuration automation • If you have no current automation • Chef • Puppet • Ansible • If you do have automation • Ops will need to find and remove all production secrets • Ops will need to add a way for you to customize
33. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Code cleanup • Force all configuration to be explicit • Not based on path or host name • Remove magic strings
34. Whew, that’s done, but how do I actually do it?
35. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Choose your platform • Traditional VM • Cloud • Container VM
36. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Verification • Smoke tests • Manual inspection • Functional tests
37. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Points to consider • Use un-routed domains for virtual hosts • Test with all internet disabled
38. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 At this point you should be able to build a fully functional environment that you are capable of developing software in.
39. Ok, now I am
 ready to give this to other people
40. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Make or break time • Prepare as much documentation as you can stomach • Prepare for lots of confusion and frustration
41. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Choose your timing carefully You only get one chance at a first impression. ! The first time a user spins up their new environment, it must work flawlessly. ! The entire rest of their experience will be influenced by this very first interaction.
42. WOW! Look!
 Everyone is using it now!
43. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 One thing we all have in common... Everyone loves to customize their computer! ! Someone will use CDPATH which makes cd output text. ! A few will use rvm and not appreciate your automated (and incorrect) help installing gems. ! Nobody likes you fiddling with their computer.
44. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Start building a troubleshooting document The less time you have to spend debugging other people’s environments, the more time you have to rapidly iterate. ! No only do you need to have them, they need to be accessible, and make sense, and have enough context.
45. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Document how to tinker with the system The sooner you can get others involved with adding features, the faster you will be able to keep up with the ever changing needs of your new development environment. ! It also lets others try things experimentally, without involving you at all!
46. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Tips for making improvements • Slow and steady wins the race. • Do not change your environment’s interface. • Except when there’s great value in it. • Even then it’s really hard to justify.
47. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 48 Minutes
48. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 30 Minutes
49. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 22 Minutes
50. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Replicating production bugs in development
51. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Make reporting problems easy . easy
52. Let’s talk about
 chasing the myth of stability
53. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 The march of progress will continue • Ops updates will break your dev environments. • You will find new third parties your builds depend on. • 100 scratch environment builds a week will make 1 in 100 events happen once a week. • You’ll start to have others contribute infrastructure changes to meet their project’s needs.
54. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Finding your environment’s dependencies • Your environment depends on: • Your own bootstrapping code • Ops automation code • Random third party software Ops is installing • … anything else?
55. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Managing your environment’s dependencies • Time to build a CI process • You’ll need a desktop to test accurately • Watch for changes from all relevant repos • Remember it is pairs of commits that are tested
56. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Making a quick recovery Check to see if your platform of choice supports snapshots. ! When environments go sideways, they can be hard to restore. ! Snapshots eliminate troubleshooting, and take ~20 seconds to run a restore.
57. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 “I just updated trunk and EVERYTHING IS BROKEN!”
58. Eventually, it will be time to
 work on something else
59. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Eventually you will need build a team • There’s a lot of domain specific knowledge in your head. • Do you know how to run a team? • Do you know how to lead a team? • Do you know how to be a product owner?
60. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Plus, letting go is hard.
61. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Let’s get an overview of the system…
62. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Further out…
63. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Regrets… • Start with a CLI abstraction of vagrant • Never change the access URLs • Don’t use a git repo for distribution unless it works 100% like a git repo
64. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 So…
65. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 What’s that stuff about eggs and bacon?
66. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 The chicken and the pig.
67. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Every time you change your code to work in an environment other than production, you intentionally break your code.
68. Much more Eggs than Bacon: A Development Environment Cookbook - MidwestPHP, March 15th, 2014 Thank you! More information about this talk can be found at: ! http://bit.ly/midwestphp-cookbook ! RATE MY TALK AT: https://joind.in/10550 ! PLEASE LEAVE FEEDBACK It’s the only way I can improve