Friday, August 03, 2007

The cost of change for build and deploy

Deploying software in Agile projects is often considered a black art or a thankless task.

Flattening the cost of change ...

We've pretty much always known that in waterfall projects, later changes cost a lot more than early changes. Agile methodologies attempt to flatten the cost of change by the use of practises such as testing, continuous integration, iterative developmet, collective code ownership and early deployment. This is successful in a majority of Agile development projects; and we continue to write tools and frameworks to allow us to write software faster and cheaper. This is all good and I wouldn't want to work any other way.

... for everything?

The software project that I am working on is delivering well software when we plan to deliver it: we have a large team turning out Java applications that are delivering value to the business.

So why is deployment such a poor cousin to the code that it deploys? Almost every project I have worked on has ended up with a deployment script that is a polyglot of Unix shell script commands. I don't attribute the problem to the choice of scripting language; such scripts make computer systems around the world run, and will continue to do so for years to come. What I want to explore is how we deploy software quickly without creating a deployment script that nobody wants to touch.

The safety net

Refactoring poor code can be challenging, but it's certainly possible to do so without breaking the build. Chances are that you'll be able to ensure that at least your new code has test coverage to give you some confidence. You can always fall back on the debugger in your IDE. When you do break the build, you'll most likely be able to use the feedback to resolve the issue.

Compare this activity to changing build and release scripts. You don't often have test coverage for what you are doing in this world. Your scripts may not be executed until days later, perhaps more.

Finding out that every released version of code since last week has been broken by your change could be considered a gumption trap.

The 1990's called. They want their deployment system back.

The cost of change, which we have done a great job of flattening in software projects comes back with a vengeance here. Once we leave the IDE , we seem to give up our agile practises. We cheerfully deploy fully tested code with an untested deployment system.

We seem to confuse several activities on software projects: build, deploy and systems administration.

Because deployment is the area where code artifacts meet the systems run by the operations people, it makes sense in a narrow way to have those people own the deployment scripts and process. But to do that overlooks the fact that developers can bring a lot to the deployment process. I'd much rather see collaboration between the developers, who bring coding expertise and the systems people, who know a thing or two about how to deploy applications. In my experience, throwing deployment entirely over the wall leads to a less than optimal process and developers who know nothing about how the code that they write gets run.

All very depressing. what can we do?



Consider your options for deployment systems

At the very least, consider writing your deployment system in a language that is easily testable. Python and Ruby come to mind. You may also be able to adopt a framework like SmartFrog or Capistrano, rather than rolling your own.

Exercise the production deployment process in Continuous Integration


We rehearse plays, weddings and deployments. But are we rehearsing the same thing that we perform on the night? Not on some projects: we have seen farce being rehearsed during development, and then drama being rehearsed in the deployments to test environments. Many projects suffer from a scarcity of test environments. Use your time on them testing software rather than debugging deployments.


Think up front about your requirements
Spend some time at the whiteboard, thinking about your requirements for deployment, and what might come down the road. If you have time to implement them and you can, do it. This goes against the grain of Agile software projects, but when you don't have the benefit of refactoring, or other nifty practises, you'll constantly be playing catch-up.

No comments: