Continuous Integration

NxtGenUG Oxford - CI Factory talk

Liam Westley — Thu, 07 Aug 2008 08:02:50 GMT

Thanks to all who attended the NxtGenUG Oxford meeting last night, always good to see a different user group in operation. Apologies for not achieving the promised 20 minutes to create a CI server from a completely clean Win2003 server installation - but 26 minutes isn't too shoddy.

As promised you can download the slidedeck for my CI Factory talk from here. Make sure you view it with the speaker notes as they give all the background information. On the 'Demonstration' slide the notes include the exact steps and installation list for creating that CI server in 26 minutes.

And finally, thanks to Barry Dorrans for ferrying me to and from Didcot Parkway station (and providing a Hot Wax chilli seedling) as well as Research Machines for providing the factilities for the evening.

Liam

P.S. In the demonstration I used a full .NET 3.5 redistributable package to install the .NET SDK without requiring an internet connection. Thanks to the articles on Aaron Stebner's WebLog and Bret Grinslade blog I idenitified the correct package to download. The full package also arrived this week on MSDN CD 2426.29 (grey).

London .NET User Group - CI Factory talk

Liam Westley — Fri, 25 Jul 2008 08:29:24 GMT

Thanks to all who attended the London .NET User Group meeting last night, a great turnout and lovely weather for slurping beers in the streets of Clerkenwell afterwards.

As promised you can download the slidedesk for my CI Factory grok talk from here, and if yoiu want an extended version I'm presenting at NxtGenUG in Oxford on Wednesday 6th August.

In the one hour or so I'll hopefully have time to actually create a SubVersion repository, edit the CI Factory setup file to create a new build server and demonstrate it automatically building an app checked in from Visual Studio 2008.

And finallyl, thanks to Ian Cooper and Skillsmatter.com for providing the factilities for yesterday evening.

DDD6 - 24th November - You have until 19th October to get your vote in

Liam Westley — Sun, 14 Oct 2007 22:04:14 GMT

Voting now open on DDD6 here, http://www.developerday.co.uk/ddd/votesessions.asp, so get your 10 votes in to make sure it's the conference you want it to be.

Those who are eagle eyed will note that my bribes to the organisers were obviously well received as my yap about CIFactory has appeared at the top of the voting table .....

Grok talk at Fest07 - 23rd May 2007

Liam Westley — Sun, 20 May 2007 21:50:06 GMT

I will be presenting a grok talk at Fest07 at Microsoft campus in Reading; 'Continuous Integration for newbies made easy - A quick guide to CI Factory'.

Those who have read my posts on CI Factory installation will be familiar with the subject, but feel free to come along and say hello if your attending. I'll have my build server with me - an external notebook drive with VirtualPC image and I'll be giving a brief overview of how CI Factory got my build server up and running in a couple of days, with a comprehensive set of code analysis tools.

The Fest07 conference is organised by the NxtGenUG (Next Generation User Group).

UPDATE: Presentation now available here (PowerPoint 2003 and Acrobat PDF versions).

Technorati tags: ci factory, Continuous Integration

CI Factory - my final setup

Liam Westley — Mon, 23 Apr 2007 21:08:39 GMT

Here's the culmination of my research into CI Factory and how I'm not using it to develop my open source RapidDB libraries. This is a fairly personal setup, as it only has a single developer (me), two development machines (currently an office desktop PC and a roaming laptop) and a build server which is not publicly accessible via the Internet. An MSDN and Microsoft Partner Action Pack subscription are busy covering my software license requirements.

Summary:

Development PC 1 : Desktop, XP Pro SP2, Program Files on D: drive, C:\Projects for CI Factory tree.
Development PC 2 : Laptop, XP Pro SP2, Program Files on C: drive, C:\Projects for CI Factory tree.
Build Server : Virtual PC 2004 image, XP Pro SP2, C:\Projects for CI Factory tree.
Source code repository : www.hosted-projects.com.

By using Microsoft Virtual PC 2004 (or VPC), which is a free download now, I have the VPC disk image (6.3Gb currently) which is on a portable laptop hard drive, running over USB 2.0. When in the office this connects to my desktop, when out of the office this connects to my laptop. That way, I always have the build server available, where ever I am working. And when my build server finally settles down and I require external access, I can easily install it under Virtual Server on my main Windows Server 2003 box and setup suitable port forwarding on my broadband connection.

Both development PCs and the build server have Microsoft SQL Server 2000 Developer and Microsoft SQL Server 2005 Developer installed. To aid with unit testing, I have created aliases to the servers on all machines as NUnit-Sql2000 and NUnit-Sql2005. This ensures that whatever machine name and instance name I might have used when installing SQL Server, unit tests can use the same alias names across all machines (and please don't flame me that unit tests should not really include database access, I have my reasons).

Build server o/s and tools installation

Windows XP Professional SP2
.NET framework 1.1, .NET framework 2.0, .NET framework 3.0
Microsoft SQL Server 2000 Developer Edition, Microsoft SQL Server 2005 Developer Edition
NCover v1.5.5 (beta)
SubVersion 1.4.3, TortoiseSVN 1.4.3
.NET Framework 1.1 SDK, .NET Framework 2.0 SDK [blog update, 6 June 2007]

CI Factory installation on VPC build server image

.\Install Scripts\Arguments.xml;

Change ProjectName; testproject.
Change the portname number (really important for multiple installations!)
Email details; VPC-Build, vpc.build@yourdomain.co.uk
Developer lists; your.name@yourdomain.co.uk
SVN URI Root; https://svn1.hosted-projects.com/youraccountname
SVN URI ProjectName, Shared Repo; ProjectName" value="${SVN.URI.Root}/${ProjectName}"
SVN.WebRepoURL; https://svn1.hosted-projects.com/youraccountname/${ProjectName}
SVN.Credentials.SafeStorage; true
SVN.Username / SVN.Password; <set to your username/password>

\Packages;

Install NCover 1.5.5 Beta to prog files; C:\Program Files\NCover
Install SubVersion 1.4.3 to prog files; C:\Program Files\Subversion
(optional) Install TortoiseSVN 1.4.3 to prog files; C:\Program Files\TortoiseSVN

CI Factory Installation

Create repository/project for testproject in SubVersion, edit ACL list to include user permissions RW.
Run C:\Tools\CI Factory\run.bat to install.
After install Visual Studio will not start as it doesn't exist on the build server; instead use
   Notepad to edit c:\Projects\testproject\Current\Build\Main.build.xml
   --> look for warnings about first call/last call and move (as in Jay Flower screencast)
   --> (SourceModificationReport.PublishOldSource to first item)
   --> (SourceModificationReport.PublishNewSource to last item)
Commit, Main.build.xml to SubVersion.
Run C:\Projects\testproject\Current\Build\Packages\CSDiff\bin\CSDiff.exe once to store path to CSDiff in registry (used in later scripts).

Post installation configuration; c:\Projects\testproject\Current\Build\ccnetproject.xml

Locate <!ENTITY email line (at top of file)
Edit the from attribute to a suitable 'from' address
Edit the mailhost attribute to by your SMTP server
For authenticated logins add the mailhostUsername and mailhostPassword attributes and enter the login and password used to send e-mails.

Afterwards, it should look something like;
<!ENTITY email '<email from="testproject.build@yourdomain.co.uk" mailhost="mail.yourdomain.co.uk" mailhostUsername="testproject.build@yourdomain.co.uk" mailhostPassword="testprojectbuild" includeDetails="true">

Developer PC 2 - C:\Program Files (99% of users)

You can just check out the tree from SubVersion to C:\Projects\testproject and run C:\Projects\testproject\Current\Build\OpenSolution.bat to load the project into Visual Studio 2005. If you want to run unit tests you will need to have installed the appropriate tools (NUnit for myself, or feel free to use MbUnit which comes with CI Factory).

Developer PC 1 - D:\Program Files

Check out the tree from SubVersion to C:\Projects\testproject.

Copy C:\Projects\testproject\Current\Build\OpenSolution.bat to C:\Projects\testproject\Current\Build\OpenSolutionVSOnD.bat
Edit OpenSolutionVSOnD.bat in Notepad, and edit the path C:\Program Files\Microsoft Visual Studio 8\VC\vcvarsall.bat to be on the D: drive.
Repeat with C:\Projects\testproject\Current\Product\OpenSolution.bat.

Adding your projects

Now you continue with the normal CI Factory process of adding your existing projects into C:\projects\testproject\Current\Product\Production as detailed in the Jay Flowers/CI Factory screen cast.

Conclusion

For those thinking that all this sounds like a whole heap of work to 'reproduce' your existing setup, this is really is not the case. I cannot tell you the relief to have CruiseControl.NET setup for me automatically and have it branching versions seamlessly in the background. It also encourages you down the unit testing path which is not a bad thing at all (although I may be breaking it with database reliant testing).

I have also been impressed by the range of default packages installed by CI Factory. This provides an immediate wake up call to how you might improve the quality of your software by using automated tools, instead of just hoping you have written reliable code that you have tested properly. I have yet to use all the default packages properly, but then, even programmers I know who have in depth experience of Test Driven Development (TDD) and CI have found new tools that CI Factory includes, such as Simian.

Technorati tags: ci factory, Continuous Integration

Update log
6 June 2007 : Updated to include installation of .NET Framework SDK to prevent Unable to load DLL 'svn_client-1' error in build log (as in CI Factory FAQ, here).

CI Factory with NUnit (rather than MbUnit)

Liam Westley — Sun, 15 Apr 2007 23:24:23 GMT

I decided to add some unit tests into my first CI Factory project. First I used a reference to MbUnit.Framework.dll in my test project, this is the unit test framework which ships with CI Factory, and all worked fine.

No disrespect to MbUnit, but I decided I really wanted to use NUnit so I can utilise the larger amount of documentation and community knowledge that exists for NUnit (even though it might not be as flexible as MbUnit). So I added NUnit 2.2.8 (.NET 2.0 version) underneath my CI Factory Third Party folder and flipped the test project over to use NUnit with a reference to NUnit.Framework.dll instead of MbUnit.Framework.dll.

Cruise Control blew up with an error in the NAnt Output log;

External Program Failed: C:\Program Files\NCover\NCover.Console.exe (return code was 1)

Oops!! I know the issue is within the file Build/Packages/NCover/NCover.Target.xml, probably leading to Build/Packages/DotNetUnitTest/UnitTest.Target.xml but I couldn't find anything obvious to edit.

In the end the Solution was horribly simple and I let out a loud DOH!. I just left the reference to MbUnit.Framework.dll in my test project, in addition to the NUnit.Framework.dll reference which actually performs the unit tests.

I can now run the NUnit GUI to run my tests on my development machine. If they pass, I check in, Cruise Control builds the project and happily runs my unit tests in automated mode.

Happy, happy, joy, joy

P.S. I did muse that it could be the lack of an NUnit.Tasks.dll for NAnt to hook into which might be the issue, but I'm no expert so this is a bit of a wild stab in the dark.

Technorati tags: ci factory, Continuous Integration

SubVersion - make sure SVN.EXE and Tortoise are kept in sync when using automated tools

Liam Westley — Tue, 10 Apr 2007 13:29:00 GMT

A minor glitch while getting CI Factory up and running.

I discovered how important it is to make sure that you keep your version of SVN.EXE (command line SubVersion) in step with that of TortoiseSVN. As TortoiseSVN informs you of upgrades, I'd happily updated it to v1.4.x automatically, leaving my SVN.EXE at 1.3.x.

The gotcha was that the installation of CI Factory was fine - it used the older v1.3.x client to create the project tree. After I had used TortoiseSVN to add some projects, it was only when I forced a rebuild by CruiseControl.NET that I saw errors in the CC.NET interface detailing the SubVersion mismatch.

The issue was that you cannot use older SubVersion clients to update a local tree created by a different SubVersion client. Quite a reasonable requirement and a terrible 'DOH !' moment on my part. A quick update of my command line SubVersion tools and all was well in CC.NET.

CI Factory - getting it working with your development tools in D:\Program Files

Liam Westley — Tue, 03 Apr 2007 16:06:00 GMT

Before we dive in make sure you read my introduction post here, and that you have followed the screen cast on how to setup CI Factory. My notes will basically follow that procedure and highlight the differences when you have development tools on your D: drive.

This is an early list, but hopefully it should give people pointers on how to get going quickly;

I'm assuming that your development PC already has Visual Studio 2005, and SubVersion 1.4.3. In addition I had to install NCover 1.5.5 beta and download the latest CIFactory beta (I used 0.8.0.121) which I unzipped to C:\Tools\CI Factory. I have hosted SubVersion at hosted-projects.com for $15 a month (note: my host area has been changed to xxxxxx on this server for obvious reasons).

Once unzipped, look at C:\Tools\CI Factory;

.\Install Scripts\Arguments.xml (standard edits)

1. Change ProjectName; testproject.
2. Change the port number if you want multiple installations.
3. Edit the email details and developer list e-mails.

4. SVN.URI.Root; https://svn1.hosted-projects.com/xxxxxx
5. SVN.URI.ProjectName, Shared Repo; ProjectName" value="${SVN.URI.Root}/${ProjectName}"

6. SVN.WebRepoURL; https://svn1.hosted-projects.com/xxxxxx/${ProjectName}
7. SVN.Credentials.SafeStorage; true
8. SVN.Username / SVN.Password; <set to known login for hosted-projects.com>

.\Packages (D: drive specific)

1. Install NCover 1.5.5 Beta to program files; D:\Program Files\NCover
2. Edit .\NCover\NCover.Properties.xml; Ncover.ExePath to D:\Program Files\NCover\NCover.Console.exe
3. Edit .\NCover\Install\Properties.xml; Ncover.ExePath to D:\Program Files\NCover\NCover.Console.exe
4. Edit .\MSBuild\Install; %Compile.DevEnv.vsvars32bat% to D:\Program Files\Microsoft Visual Studio 8\VC\vcvarsall.bat
5. Edit .\VS.NETCompile\Install; %Compile.DevEnv.vsvars32bat% to D:\Program Files\Microsoft Visual Studio 8\VC\vcvarsall.bat

.\Power Tools (D: drive specific)

1. OpenSolution.bat; edit C:\Program Files\Microsoft Visual Studio 8\VC\vcvarsall.bat to be on D: drive.
2. Still in OpenSolution.bat; edit C:\Program Files\Subversion\bin to D:\Program Files\Subversion\bin

.\Default\Build (D: drive specific)

1. CCNETServer.bat; edit C:\Program Files\Subversion\bin to D:\Program Files\Subversion\bin
2. OpenSolution.bat; edit C:\Program Files\Microsoft Visual Studio 8\VC\vcvarsall.bat to C:\Program Files\Microsoft Visual Studio 8\VC\vcvarsall.bat.

To prevent NDepends build errors
Copy D:\WINDOWS\FONTS to C:\WINDOWS\FONTS to prevent error in CCNet Build Report erroring on missing Arial font.

>> INSTALLATION

1. Create repository/project in SubVersion, edit the ACL/permissions list to include user permissions RW for the login you configured.
2. Run C:\Tools\CI Factory\run.bat to install CIFactory components (I left them with the default modules).
3. After install Visual Studio fires up; edit c:\Projects\testproject\Current\Build\Main.build.xml
   --> look for warnings about first call/last call and move (as in the CIFactory screen cast)
   --> (SourceModificationReport.PublishOldSource to first item)
   --> (SourceModificationReport.PublishNewSource to last item)
4. Commit, Main.build.xml to SVN.
5. Run CSDIff once to store path to CSDiff in registry (this is used in later scripts).

>> ADD INITIAL PROJECTS (to C:\Projects\testproject\Current\Product\Production)

1. Run, C:\Projects\testproject\Current\Product\OpenSolution.bat; this should load Visual Studio 2005.
2. Copy your existing projects to C:\Projects\testproject\Current\Product\Production (use TortoiseSVN->Export to get a clean tree with no .svn folders)
3. Add your projects in Visual Studio, check it builds.
4. Add your projects in C:\Projects\testproject\Current\Product\Production, to SVN (remove/exclude the bin/obj directories).
5. Commit your additions, and the changes to the main solution (new projects) to SVN.
6. CruiseControl.NET should now pick up changes and compile your new solution.

Miscellaneous notes

The resource usage for an 'empty' CIFactory install with default modules but none of your own projects came in at 14Mb with SVN on hosted-projects.com (peaked at 17Mb during commit). Important to note if you have usage limits.

The other wrinkle I had was a SubVersion certificate issue; hosted-projects.com are still using a certificate based on the URL http://www.hosted-projects.com/ which was created prior to their configuring multiple servers. My repositories are now on svn1.hosted-projects.com, however they still use the old certificate for encrypting content. This doesn't bother me but it can bother svn.exe, the command line tool, especially when used in an automated mode.

To fix this run SVN.EXE at the command line to your repository;
     svn list https://svn1.hosted-projects.com/xxxxxx/cifactory --username xxxx --password xxxx
You will be asked to accept the certificate, if you choose permanently then the CIFactory and CCNET scripts will work without a problem.

CI Factory - my first day of real use - phew!

Liam Westley — Tue, 03 Apr 2007 15:13:00 GMT

A friend from the London .NET users group, Peter Ibbotson, on hearing that I had started looking into Cruise Control .NET for continuous integration said that before doing anything I should look at Zi Makki's blog about CIFactory, a master setup tool for creating a source code tree complete with various tools such as SubVersion, NCover, NDepends all hooked into CruiseControl.NET with NAnt as the build script automating the processing.

Well, I've just spent a challenging but very satisfying day and a half getting to grips with CI Factory and I'm very impressed. I know the principles of TDD and CI, but being a one man micro-ISV and not having the luck to work for a client with a TDD/CI software methodology I've never got round to sorting my own development out.

CI Factory provides an ideal kickoff point as you get a large number of standard modules, configured on the fly for your 'new' project in your repository. It doesn't have to be a really new project, as long as you don't have a mega project with 100's of projects you can probably move your code into the CI Factory Production tree quite quickly.

For those starting out new I cannot overstate how important the screen cast training videos were to my getting the system up and running so quickly. The longer video based on 0.8.0.27 beta gives you a great installation guide to setting up the default options correctly. It helped that I run SubVersion as my source control system but it was so good I downloaded the whole lot to my local drive (not trivial with various .swf, .flv and .xml files being required) so I can browse the video under a local IIS when not connected to the Internet.

The only real issues I had were configuring the system to work on a development machine where only source code and data is on my C: partition, with my programs all in C:\Program Files. There'll be an accompanying blog on how to solve this issue. However, few people are probably in this position so they'll have a much easier time.

Many thanks to Jay Flowers, the developer behind CIFactory for replying very quickly with my D: drive queries. A patient man, as I'm not someone who has dealt with NAnt scripts before and had to be helped out a bit more than should have been necessary.