Over the last year and a half my team and I replaced a large subsystem in our application without (hardly) anyone noticing. I talked about that story, the tools and techniques we used, and what we learned along the way.
Scientist is open-source: https://github.com/github/scientist
Talk given at denver.rb and boulder.rb, December 2014.