A better automated build system

I’m a big fan of automating builds. In non-trivial software, it becomes time consuming and error prone to build releases by hand. It’s a job for a machine. A machine can notice when you change things, and rebuild everything affected automatically, in a reliable way. It’s also a natural place to run all affected unit tests.  Plus, in statically typed languages, just recompiling everything that depends on some piece of modified code will often be enough to point out problems (ie: the build breaks), but with a complex set of interelated projects, you’ll forget to compile all dependant projects every time.

Professionally, I’m a c# coder, and I use CruiseControl.net for automating builds. I use svn for my source control. I’ve extended CruiseControl slightly, to include a straightforward way of deploying builds. My extensions take an msi to be deployed, rename it to include the version/build number, zip it up into a similarly named zip file, and ftp it to a desired location. This goes well with a stupidly simple little web app I wrote called SimpleDeployPortal, which provides a web app interface to the folder to which you deploy, and maintains an xml version file that can be consumed by automated processes to figure out what versions of what apps are available through a particular portal instance. In turn, I have some c# libraries which help you implement auto-update for your apps, based on this portal with xml versions file approach. This is described further in The Wandering Developer Build Tools, and the source is on sourceforge. I have windows installers for all of this stuff, built and deployed using these tools, but unfortunately the bit on my website that displays the builds is broken at the moment. For what it’s worth, it’s here http://emlynoregan.com/Software.aspx, I’ll fix it eventually. But I digress.

CruiseControl.net has been annoying me. It conflates some problems, causing it to be cumbersome and inflexible. The one I’m most concerned with is that it makes you define a project by defining where it is in source control (a folder), defining where it is on your local drive (another folder), and basic autobuilding on seeing a change in source control, whereapon it gets the modified files and rebuilds.

Now, automating a build is a bit of work. The resulting configuration is large, complex, and requires maintaining. So you want to get maximum value out of it.

Unfortunately, cruisecontrol ties the build automation and syncing from source together, so you end up having something which only works based on the branches you have coded it to look at in source control, and also assumes it is building in a folder where you are not also developing.

How I think it should work:

The central thing is the local copy of source code. If I have a local copy of the source defined (ie: a folder where it can go), then I should be able to automate my build based on that alone. I should be able to define projects in there, based in sub folders. I should be able to define dependencies between projects (ie: project B depends on project A, so changes to A are also changes to B). The build tool should be able to see when relevant files change in the local copy, figuring out what projects are affected, coming up with a dependency sorted list of projects that require rebuilding. The build tool should be able to actually perform that rebuild if required.

Separately, I should be able to define the relationship between the local copy and a source repository if I so desire. I should be able to break this into repository relevant projects, likely different (less finely grained) to the build automation projects. Here, there are no dependencies. For each project I should be able to define the local folder, and what branches are available (trunk, branches 1 to N, etc). There is then a separate location in source control for each branch for the project. Also, I need to be able to define various repository get profiles, which are a list of all desired projects, along with the branch to get for each project. Also, I need a service which can maintain synchronisation for a given build profile between the repository and the local folders, by getting changes from the repository when changes appear.

Now these two pieces, (source control -> local copy) and (local copy -> builds) are conceptually separate, and could be used in these separate use cases:

1: Production Build Machine: Simply pair the two pieces above. You define the repository get profiles to sync with the localcopy folder(s), and separately define the build projects (a directed graph of dependencies) to build. You rely on the detecting filesystem changes when new source comes from the repository, to make the two pieces interoperate.

2: Other Build Machines (eg: daily): Just like the production build machine, but you use a different set of compatible repository get profiles (eg: pointing at unstable dev branches of the repos). The build projects config is identical, can be used unchanged.

3: Development machine: You can use just the build projects stuff, with no repository sync. This gives you the same ability to build consistently, but based on your local changes. Also, it shouldn’t autobuild if it sees changes; it should require the developer to fire it off, but give prompting about what needs rebuilding.

4: Give me Build X: A system with a build machine and a web app. It allows users to specify a project that they want to build, and to choose the versions of the various projects in the repos to use. This machine then extracts all the code as required to somewhere appropriate, does the build (including all dependencies), and puts the result somewhere available to the web server, which can then provide it to the user.

And there are many other ways you could use this, I think.

A note: It might be a good idea to do the actual building via NAnt, that’s what it’s for after all.

That’ll do for now. But this beasty, it wants building.

About these ads