Tuesday, August 15, 2006

Test Driven Systems Administration

I wrote this in 2005. It was lost with my server. Thanks to Ade for the cached copy!

I have been thinking about how to apply Agile to the field of Systems Administration for a while now. I haven't been thinking too hard, mind you; but I have been thinking.

I googled, and found this and this, but nothing that really provided any great insight.

So I decided that a good starting place would be testing. Not monitoring, but testing. I had intended to play with Ruby or Python and extend their test frameworks so I could make assertions about the operating system, e.g. asserting that a process was bound to a socket, or that a server responded correctly. This week, I was asked to work on a project doing some migrations of servers, so it was a nice opportunity to test the theory out.

All the servers had Perl installed, so I chose Test::More as a test framework - again it was installed everywhere, and I didn't need much. So then it was simply:

ok(connect_host($ip,$port)==0,""can_connect_to_foo01"");

and it was good. connect_host is a subroutine to connect to a given host and TCP port using IO::Socket. So now I had tests to make sure that the services were available on the correct IP address. Will definitely try and expand on this.

3 comments:

Mike Vincent said...

Looking forward to reading your future writings on the subject. I remember reading your original posting and thought it was a pretty interesting idea.

Back in 2004, Harry Fuecks had written about using the SimpleTest php testing framework to monitor servers in this article http://ezcommunity.net/article/view/68/1/39.

jason huggins said...

I've been thinking about how system administration fits into the agile world, too, but from a different angle. For day-to-day testing of our production apps these days we just use Nagios. :-)

But the thought that I had is though Continuous Integration is highly valued and a known quantity, Continuous Deployment (to production) is still very much an emerging concept for many people. By continuous deployement, I mean having every step automated for moving from dev/test to production and doing it all the time. With tools like Capistrano this idea becomes easier to implement.

But since the thought of *actually* deploying to the production server on every source commit is scary, then you could deploy to a virtual machine configured just like the production box. (Consider it a mock object for the actual production box).

And to tie together your thoughts here on test driven sys admin, you would use your sys admin tests as a post deployment functional test of the vm to make sure the deployment worked as planned. And again, when you run the "real" deployment, you could run the same tests, for a sanity check against production.

Anyway, I've been thinking that building in "CD" practices (like setting up mock production virtual machines, installing capistrano and writing sysadmin tests will become part of my "Iteration Zero" tasks in the future.

Mike Vincent said...

I think more than system monitoring and deployment strategies. My own confidence would/will be increased a thousand fold when my server's configurations are version controlled same as my source is on development projects. It's the next item on my list of things to do to improve sys admin work.

The virtual server idea is interesting, I started thinking about how I would replicate one of our servers in a virtual space. Maybe through making a custom package that installs everything necessary and sets it up (semi-interactively) the way we need it all setup. And it is -that- which we maintain, make changes to, test on the virtual server and if/when things work to their expectations (tests! :) it can be rolled to the production machine(s).

Think I may've just re-iterated what you said. :) Awesome idea.