Velocity 2010: Change ManagementPosted: July 6, 2010
Change Management was another huge theme at the conference. I actually heard more than one speaker say something to the effect of “If you don’t have change management in place in some form, you should leave the conference right now, put it in place, and then come back.” Developers (almost) always have some manner of change management in place – source control, peer review, approval processes, release schedules, etc. – in order to…errm…manage changes to the codebase on which they are working. …but what about the systems – the machines in which that code is running? …and why is that important? Why isn’t it okay to have a sysadmin fire up emacs, slam in a config change, and send out an email saying “all’s well”?
Well…let me give a non-tech example here. Let’s say you took your car to the mechanic. He takes a look-see at it, sort of twiddles about a bit, and hits you with a bill for a few hundred bucks. (I realize that this is almost exactly how most folks’ visits to the mechanic go, but bear with me here). Now suppose he can’t tell you exactly what he did or document it in any way, but it’s running so “we’re good, right?” Oh…and suppose he tells you that your car may or may not start the next time you turn the key; “Just bring ‘er on back and we’ll have another look!” How comfortable would you be with the arrangement?
Okay, so the car analogy doesn’t carry over so well (and is rather unfair to sysadmins, I might add)…but I can tell you that I’ve seen this sort of thing happen in the datacenter many a time. As a Linux admin, I was terrified of rebooting machines, largely due to the inverse relationship between the uptime of a system and the likelihood of it actually coming back up correctly after a reboot. Having a change management tool like Chef (very cool; demoed at the conference), cfengine, puppet – pick your poison – backed by version control (of course) is a means of raising the level of confidence about changes being made to systems.
Oh, and how about auditing? Suppose your method for determining what’s going on with your systems is to walk around to all the sysadmins who might have touched Machine 7 of 3,956 and ask politely “Have you changed anything on this sum’bitch in the past decade?” Repeatability? “Can you build Machine 3,957 and make it look just like Machine 7?” CYA during a postmortem witch hunt? “Prove to me that you didn’t have a hand in bringing down our production database this afternoon.” A good change management system goes a step beyond the typical “get all of management to sign off on the change before putting it in” approach. Just a few of the reasons for implementing change management.