Adventures in Geekland

Scalability of development process

My current project has an agile mind-set, but a slow development process. The overhead of picking up a CR, checking out, testing, checking-in, integrating, and deploying build is currently getting up towards the 3 hour mark. This is a fixed overhead associated with any development work, not including the actual engineering effort of implementing a change request.

This overhead can be divided into parts - There is firstly that overhead that is fixed for each developer - e.g. as the number of developers increases, the individual overhead remains relatively constant. The other sort of overhead is the sort that increases with the number of developers.

For instance, if the build is broken, no-one can check-in. This is block on all development. If you have 100 developers, each will be blocked for at least 1 hour plus the length of any fix to the broken build. 100 developer hours lost, just like that.

However, if the local build process breaks, only the developer who broke it will be affected. Alternatively, if each local build process takes 1 hour (i.e. 1 hour of lost developer time waiting for the local build to complete) then while each and every developer suffers that overhead, they are free to pair with other developers.

If the build is denegenerate, then effort must be placed first on foremost on reducing the impact of a broken shared build. Only if it is not cost-effective to improve the build server should work commence on reducing the overhead of the local build process.

To that end, we have been looking at making our build server extremely scalable and cost effective. We have been looking at clustering, and trying to use free software to do it.

posted by Nick Drew @ 9:57 pm