Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Dark Side of DevOps

Chris Short
September 08, 2017

The Dark Side of DevOps

The DevOps journey is a combination of people, processes, and tooling. The culmination of these is a team of allies pushing the envelope and never resting on their laurels. There is no such thing as a completed DevOps transition. Resting on your laurels is not a DevOps mindset. DevOps and its allies should be iterating and improving upon what they have learned daily.

Chris Short

September 08, 2017
Tweet

More Decks by Chris Short

Other Decks in Programming

Transcript

  1. THE DARK SIDE OF DEVOPS
    DEVOPSDAYS RALEIGH
    @ChrisShort devopsish.com

    View full-size slide

  2. INTRODUCTION
    chrisshort.net
    @ChrisShort devopsish.com
    • Hello! HAPPY, LOUD, GOOD MORNING

    • A quick intro about me

    • I've worked in IT for over twenty years now

    • I've worked in DevOps capacities since 2011

    • I'm currently at working at bankrate.com in Detroit, MI

    • There's a new project I'm participating in with opensource.com called DevOps Stories

    • It's about the intersection of open source and DevOps

    • If you're interested in helping out head over to opensource.com/devops-team

    View full-size slide

  3. • True Story: Last year I lived in Raleigh, attended DevOpsDays here. But, I spoke at DevOpsDays Detroit.

    • This year, I live in Detroit, will attend DevOpsDays there and am speaking here in Raleigh.

    • The moral of that story is: be careful where you speak at your first DevOpsDays, you might end up living there.

    • But, Don't Knock Detroit

    • Blight is so 2000s

    • These are just a few pictures I've taken of Detroit since moving there

    • The city is going through a massive rebirth and it's amazing

    • If you are ever in town let me know, I'll give you a tour

    • On to the DevOps part of this talk

    View full-size slide

  4. • One thing I love about DevOps is how failure is embraced

    • I love failure because it teaches us things we never would know about ourselves

    • Many people present DevOps as this great thing filled with

    • Roses

    • Rainbows

    • Unicorns

    • Maybe even a random Leprechaun

    • Sometimes it's nice to remind ourselves that DevOps has a dark side to it

    View full-size slide

  5. • The Dark Side of DevOps is Complete Abject Failure

    • Failed DevOps Transformations Are a Real Thing

    • Organizations try to move their organization towards The Three Ways and fail

    • It happens; failure is a part of life

    • Let's Explore a Couple of Examples from My Own DevOps Journey

    View full-size slide

  6. DISCLAIMER
    @ChrisShort devopsish.com
    CSHORT.CO/DARKOPS
    • Let me drop this quick disclaimer

    • This talk was adapted from an article I wrote in March: http://cshort.co/darkops

    • These are real world experiences I have had within organizations that I have been employed by

    • Identifying information is intentionally anonymous

    • If you think you are in one of these organizations please consider keeping that to yourself for the sake of others

    • Or don't if you're the rebellious type

    • Ready? Let’s walk into the cave on Dagobah together.

    View full-size slide

  7. THE OPS MUTINY
    @ChrisShort devopsish.com
    When I say "abject failure", this is what I'm referring to

    View full-size slide

  8. THE OPS MUTINY: BACKSTORY
    ▸ Dev teams adopted DevOps practices
    ▸ Traditional IT team was not adapting
    ▸ IT team was actively resisting DevOps
    ▸ Mutiny!!! Majority of IT team left company en masse
    ▸ Tribal knowledge left with remaining two team members
    @ChrisShort devopsish.com
    • Dev teams adopted DevOps practices for flagship product

    • The traditional IT team was not adapting

    • IT team was actively resisting the transition towards a DevOps workflow

    • There was a mutiny and the majority of the IT team left the company en masse

    • Tribal knowledge rested with two remaining team members

    View full-size slide

  9. @ChrisShort devopsish.com
    Qatar, 2002
    • This is where I came into this sordid affair (in the twenty-teens)

    • I joined on as an operations lead to push the team into the future

    • A younger, less experienced, and slightly mislead Chris Short

    • LOOK AT SCREEN
    • "Well maybe not that young"

    • I should've used young Ewan McGregor but... Ya know... Episode 1

    View full-size slide

  10. @ChrisShort devopsish.com
    • Week One was a sign of how bad this was going to be

    • ZERO formal processes and on boarding was terrible

    • Changed desks halfway through the second day

    • I had a loaner laptop and never had a work phone number while I worked there

    • Spent my first week working on compliance training I did not need to do

    • Did not have access to what I needed to complete the goals laid out for me

    View full-size slide

  11. • Then the architecture review happened...

    • Company had been acquired for a HUGE sum of cash

    • The architecture review was not for me, the new lead, it was for the company that had bought them

    • I learned the office server room with faulty HVAC held some production functions

    • Learned that Windows NT 4.0 was still present in their infrastructure in 2015

    • Changes could not happen due to business concerns

    • Legal coverage was good enough for the corporate lawyers

    • No data stored on archaic OS versions; this was pretty much a lie

    View full-size slide

  12. @ChrisShort devopsish.com
    • Then I started poking around the network

    • Was immediately told to stop

    • IT was concerned about the load I was putting on the network (from my laptop)

    • The "concern" was bogus and it was actually to keep the old guard in a position of power

    • The mutiny was on-going as the lingering staff continued to consolidate power

    View full-size slide

  13. • Ambitious plan underway to move the flagship project over to a cloud provider and use containers

    • Majority of that work was being handled by a third party integrator

    • Plan was for the vendor to integrate and stand up the new infrastructure

    • Team would then take over operations

    • Company was relying more and more on the vendor when the opposite was the desired outcome

    • It was becoming clear the team was going to continue their resistance and I had no allies

    View full-size slide

  14. @ChrisShort devopsish.com
    • Shortly before a major US holiday, I determined enough was enough

    • After three weeks, I was putting in my two weeks notice

    • The anti-DevOps mutiny was successful

    • The leadership tried very hard to get me to stay on

    • Leadership offered to terminate the remaining holdouts days before a major holiday

    • Not okay with morally bankrupting myself in the name of DevOps

    View full-size slide

  15. THE DEV REBELLION
    @ChrisShort devopsish.com
    • I had high hopes for my next opportunity

    • Fresh out of one DevOps failure I unintentionally jumped into another one

    View full-size slide

  16. @ChrisShort devopsish.com
    • A full blown DevOps rebellion against the monolithic IT department was planned

    • DevOps was brought to bear within a small team working on an open source project

    • I had high hopes for this opportunity

    View full-size slide

  17. @ChrisShort devopsish.com
    Team was already using cloud native tools because existing tooling was inadequate:

    • Docker

    • Papertrail

    • Postgres

    • Slack for communication

    View full-size slide

  18. @ChrisShort devopsish.com
    • CTO said cloud was prohibited; everything had to stay in house

    • Weeks or months to get a VM; DNS change requests took days or weeks

    • Team was going to utilize Heroku despite the IT organization stating “cloud” was prohibited

    • Shortly before my arrival they had done all the appropriate paperwork to deploy this new project

    • Assumption from the IT staff and paper pushers was that this new project was deploying in-house

    • All the documentation clearly stated Heroku was the workload’s destination

    • CRAFTY and I loved it!

    • It was so gangster

    View full-size slide

  19. • IT department approval authorities signed off on utilizing cloud resources despite saying they never would

    • They just signed off on a full blown shadow IT project and it was right there in their own documents

    • This proved their ITIL and ITSM death grip process was not meeting their desired objectives

    • The cloud cat was out of the bag

    View full-size slide

  20. @ChrisShort devopsish.com
    • IT attempted to change and satisfy the team's needs in-house

    • The IT process was a strict, by the book adherence to ITIL

    • It would never be able to shoehorn itself into a speedier process until the need for hand jamming tickets was eliminated

    • OpenStack instance as a developer playground was going to take a year or more

    • Tooling to enable API calls for IT resources was outsourced to a vendor with little feedback from our team

    • IT did not get it but the team was working around the bottlenecks

    • It was working for the team but my job of trying to get the IT org looking towards the future was next to impossible

    View full-size slide

  21. • Resistance and educating the IT organization were critical components of helping them on their path towards transformation

    • The alternative was to join in the IT organization's thinking which as stifling innovation

    • Join them and you can ruin the galaxy together!

    View full-size slide

  22. • The rebellion was in full force

    • Microservices were being built and deployed

    • IT organization had no way of keeping up with the multiple releases a day

    • Change occurred when IT organization realized it was lagging way behind

    • IT organization realized they couldn't compete on costs

    • Do to a host of other issues I decided to leave this position for another opportunity

    • Eventually compliant Azure resources were made available to the teams

    • Last I heard it was still taking months to get a VM in their datacenter

    View full-size slide

  23. DEVOPS STRIKES BACK
    (LESSONS LEARNED)
    @ChrisShort devopsish.com
    What did I learn from these two journeys?

    Aside from changing jobs is easy in Raleigh/Durham

    View full-size slide

  24. "OUR GREATEST GLORY IS NOT IN NEVER
    FAILING, BUT IN RISING UP EVERY TIME
    WE FAIL."
    Ralph Waldo Emerson
    EMBRACE FAILURE, LEARN FROM IT

    View full-size slide

  25. @ChrisShort devopsish.com
    • A mutiny and a rebellion might look the same from opposing perspectives

    • The rebellion was to change for the better while the mutiny was to maintain the status quo

    • This is not to say Ops is bad and Devs are great (far from it)

    • These jobs along my journey proves that motivated teams can act towards their common goal be it good or bad

    View full-size slide

  26. @ChrisShort devopsish.com
    • I have learned that DevOps needs allies at all levels

    • I wish I would have learned this earlier in my career

    • Not just leadership

    • Not just individual contributors

    • People have to want to better their environments

    View full-size slide

  27. @ChrisShort devopsish.com
    • The Dev Rebellion had people at all levels pointing at successes of the team

    • The Ops Mutiny had a core group of people trying to cast off change regardless of outcomes

    • But both were personal failures since I wasn't able to rally enough support in enough time to keep me from going mad

    View full-size slide






  28. @ChrisShort devopsish.com
    • In Conclusion...

    • The DevOps journey is a combination of people, processes, and tooling

    • The culmination of these is a team of allies pushing the envelope

    • Never resting on their laurels

    • There is no such thing as a completed DevOps transition

    • Resting on your laurels is not a DevOps mindset

    • DevOps and its allies should be iterating and improving upon what they have learned daily

    View full-size slide

  29. @ChrisShort devopsish.com
    • Thank you for your time today

    • Feel free to reach out to me on Twitter if you ever need another opinion or a nudge in a direction

    • Take care of each other out there

    View full-size slide