Perforce

Sample CruiseControl build using Perforce

EltonStoneman — Thu, 29 May 2008 19:34:17 GMT

Tim asked for some more detail on using Perforce as your SCM for CruiseControl from this post, so here are some sample config files. This post may be too detailed for a general audience…

Perforce Setup – Build Server Workspace

You'll need a dedicated workspace for the build server to use – this will be used by CC.Net to check for changes and to resync. You don't need a separate user account, so it can be created with the build manager as the owner. You'll define the location to monitor in the CC.Net config, so the workspace only needs a view over the root project. So you may create a workspace called build.server with a view of //depot/… //build.server/… (while you're setting this up you can use it from your dev machine with a different root from your normal workspace, then change the host to be the build server when your scripts are finished).

CruiseControl setup - CCNet.config

This sample config uses the build server's workspace to monitor //depot/ProjectX/MAIN/source (assumes the workspace depot is c:\depot):

<cruisecontrol>
<project name="ProjectX">
    <sourcecontrol type="p4">
      <view>//depot/ProjectX/MAIN/source...</view>
      <client>build.server</client>
      <applyLabel>true</applyLabel>
      <autoGetSource>true</autoGetSource>
    </sourcecontrol>
    <workingDirectory>C:\depot\ProjectX\MAIN\source</workingDirectory>
    <webURL>http://localhost/ccnet </webURL>
    <artifactDirectory>C:\buildOutput\ProjectX\artifacts</artifactDirectory>
    <modificationDelaySeconds>5</modificationDelaySeconds>
    <triggers>
      <intervalTrigger name="continuous" seconds="10"/>
    </triggers>
    <state type="state" directory="C:\buildOutput\ProjectX\artifacts" />
    <labeller type="iterationlabeller">
      <prefix>1.1</prefix>
      <duration>4</duration>
      <releaseStartDate>2008/5/29</releaseStartDate>
      <separator>.</separator>
    </labeller>
    <tasks>
      <msbuild>
        <executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable>
        <projectFile>build\ProjectX.build</projectFile>
        <targets>FullBuild</targets>
        <buildArgs>/noconsolelogger /p:Configuration=Debug /verbosity:minimal</buildArgs>
        <logger>C:\Program Files\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
      </msbuild>
    </tasks>
</project>
</cruiseControl>

This tells CC to connect to Perforce in the build.server workspace, and monitor all files under //depot/ProjectX/MAIN/source. When it finds a change, it will resynchronise (bringing the build server's copy of the source up to date), determine a label for the build, and start the FullBuild target in the MSBuild file.

If the build is successful, the label is applied to all files in the view, so Perforce is stamped with a label for every successful build (this means at any time we can create a branch for a specific build which we know will work). There's no publishing here, but you typically configure CCNet to issue build failure mails to people/lists named in the project.

For multiple projects in the same CCNet config, you can use the same workspace but specify a different view.

MSBuild setup – projectX.build

The build file specifies targets as entry points. Targets can contain other targets, so typically the build process will be split into component parts which are brought together by composite targets. A sample build file split like this is:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="FullBuild">

<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>

<PropertyGroup>
    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
    <OutputDir>bin\$(Configuration)</OutputDir>
    <ObjOutputDir>obj\$(Configuration)</ObjOutputDir>
    <ProjectDir>..\ProjectX</ProjectDir >
    <ProjectTestDir>..\ProjectX.Test</ProjectTestDir >
    <ProjectFile>$(ProjectDir)\ProjectX.csproj</ProjectFile >
    <TestProjectFile>$(ProjectTestDir)\ProjectX.Test.csproj</TestProjectFile >
    <ExcludeTestCategories>InProgress,LongRunning</ExcludeTestCategories>
    <Version Condition=" '$(CCNetLabel)' == '' ">1.0.0.0</Version>
    <NUnitOutputXmlFile>C:\buildOutput\ProjectX\artifacts\testResults_$(Version).xml</NUnitOutputXmlFile>
</PropertyGroup>

<Target Name="FullBuild">
    <CallTarget Targets="BuildNoTests"/>
    <CallTarget Targets="UnitTestContinuous"/>
    <CallTarget Targets="Package" />
</Target>
<Target Name="BuildNoTests">
    <CallTarget Targets="Build"/>
    <CallTarget Targets="PostBuild"/>
</Target>
<ItemGroup>
    <OutputDirectory Include="d1">
      <DirectoryName>
        $(ProjectDir)\$(OutputDir)
      </DirectoryName>
    </OutputDirectory>
    <OutputDirectory Include="d2">
      <DirectoryName>
        $(ProjectDir)\$(ObjOutputDir)</DirectoryName>
    </OutputDirectory>
    <OutputDirectory Include="d3">
      <DirectoryName>$(ProjectTestDir)\$(OutputDir)</DirectoryName>
    </OutputDirectory>
    <OutputDirectory Include="d4">
      <DirectoryName>$(ProjectTestDir)\$(ObjOutputDir)</DirectoryName>
    </OutputDirectory>
</ItemGroup>
<ItemGroup>
    <ProjectDirectory Include="p1">
      <DirectoryName>$(ProjectDir)</DirectoryName>
    </ProjectDirectory>
    <ProjectDirectory Include="p2">
      <DirectoryName>$(ProjectTestDir)</DirectoryName>
</ProjectDirectory>
</ItemGroup>
<Target Name="CleanDirectories">
    <Attrib Directories="%(OutputDirectory.DirectoryName)" ReadOnly="false" ContinueOnError="true"/>
    <RemoveDir Directories="%(OutputDirectory.DirectoryName)" ContinueOnError="true"/>
</Target>
<Target Name="SetAssemblyVersions">
    <Attrib Files="%(ProjectDirectory.DirectoryName)\Properties\AssemblyInfo.cs" ReadOnly="false"/>
    <AssemblyInfo CodeLanguage="CS" OutputFile="%(ProjectDirectory.DirectoryName)\Properties\AssemblyInfo.cs" AssemblyVersion="$(Version)" AssemblyFileVersion="$(Versiom)" />

</Target>

<Target Name="PreBuild">
    <Message Importance="high" Text="In target PreBuild"/>
    
    <CallTarget Targets="CleanDirectories"/>
    
    <CallTarget Targets="SetAssemblyVersions"/>
</Target>

<Target Name="Build" DependsOnTargets="PreBuild">
    <Message Importance="high" Text="In target Build"/>
    <MSBuild Projects="$(ProjectFile)">
    </MSBuild>
    <MSBuild Projects="$(TestProjectFile)">
    </MSBuild>
</Target>
<ItemGroup>
    <ProjectOutput Include="$(ProjectDir)\$(OutputDir)\*.*"/>
</ItemGroup>
<ItemGroup>
    <ProjectTestOutput Include="$(ProjectTestDir)\$(OutputDir)\*.*"/>
</ItemGroup>

<Target Name="PostBuild" DependsOnTargets="Build">
    <Message Importance="high" Text="In target PostBuild"/>
    <Copy SourceFiles="@(ProjectOutput)" DestinationFolder="C:\buildOutput\ProjectX\$(Version)"/>
    <Copy SourceFiles="@(ProjectTestOutput)" DestinationFolder="C:\buildOutput\ProjectX\Test\$(Version)"/>
</Target>

<Target Name="UnitTestContinuous">
    <Message Importance="high" Text="In target UnitTestContinuous"/>
    <CreateItem Include="SmartRename.Test.Dummy.dll">
      <Output TaskParameter="Include" ItemName="TestAssembly" />
    </CreateItem>
    <Message Importance="high" Text="Assemblies: @(TestAssembly)"/>
    <NUnit ToolPath="C:\Program Files\NUnit 2.4.6\bin" WorkingDirectory="$(ProjectTestDir)\$(OutputDir)" Assemblies="@(TestAssembly)" ExcludeCategory="$(ExcludeTestCategories)" OutputXmlFile="$(NUnitOutputXmlFile)"/>

</Target>

<Target Name="Package" DependsOnTargets="Build">
<Message Importance="high" Text="In target Package"/>
</Target>
</Project>

(note some of the targets are placeholders).

This is more involved and MSBuild is well documented, but the key things to note are:

Properties defined in the initial PropertyGroup can be overridden by values passed to the MSBuild exe (with /p:name=value;name2=value2); they are referenced in the script with $(name);
ItemGroup allows you to define an array of items that can be operated on (e.g. in CleanDirectories target); the loop is started and entries accessed with %(GroupName.ElementName);
Individual items in an ItemGroup can be accessed with @(ItemName);
DependsOnTargets flag will ensure the necessary targets are run before the executing target, but will not invoke the target if it has already been called;
The SetAssemblyVersions target uses a MSBuild Community Task (http://msbuildtasks.tigris.org/) to update the build number in the projects' AssemblyInfo files - this ensures the built assemblies have the same build number as the Perforce source label (available in $(CCNetLabel) property.

Scripting Branch Creation in Perforce

EltonStoneman — Fri, 04 Apr 2008 16:17:40 GMT

To simplify the branching process, allow automation and remove the errors likely from manually branching, I'd like the whole process to be scripted. The P4 command allows us to do this, but there are complications as it requires user input via the specification forms for most of its commands. You can direct it to read the input from a file, but that's not much help when you want to use variables to specify branch names etc.

Generating Specification Forms on the Fly

My solution was to create a simple console app to generate spefication forms based on input parameters. The variables in the script go into the generator which stores them as files, which P4 can use as input. It's not terribly elegant, but it's efficient and I haven't found any other solutions out there. The code is on CodePlex here: Generate Perforce Specification Form

Using the console app, it's easy to take the manual steps from Branching with Perforce and script them for automation. In this case, I'm expecting branching to be an occassional job so the purpose of scripting it is to reduce the possibility of error and simplify the branching process for anyone who needs to do it and doesn't want to trawl through these blogs.

With this in mind, the script is a Windows batch file (the full script is BranchWorkspace.bat in the Samples directory of the CodePlex source) rather than a build task. It should be straightforward to read, but I'll dissect it here to show how the commands fit in with the manual process.

Walkthrough

Firstly the batch file sets up parameters for all the variables we'll use - branch name, branch master workspace name, views etc. Then it follows the same order as the manual process, generating the specification forms where needed, then invoking the P4 command and telling it to read from the spec form.

1. Create the branch view

::generate spec form for the branch:
set branchSpecPath=C:\%branchName%.p4
GenerateP4SpecForm "WRITE" "%branchSpecPath%" "|" "Branch: %branchName%|Owner: elton.stoneman|Description: Created by elton.stoneman.|Options: unlocked|View: %mainView% %branchView%"

The console app takes parameters telling it where to create the spec form, how the form options are separated (I use a pipe as there are colons in the options), and then the pipe-separated option list.

What goes into the list depends on the type of spec form P4 wants; there are samples in the CodePlex source, or you can copy and paste from a real spec form, or many of the commands take a -o parameter so you can output an existing spec form and use that as the basis.

::create the branch view:
p4 branch -i <%branchSpecPath%
- the branch view is created and we pass the spec form path to P4 so there's no user intervention.

2. Create the master workspace

As before, generate the relevant spec form and pass it into P4:

::generate spec form for the branch master workspace:
set branchClientSpecPath=C:\%branchClient%.p4
GenerateP4SpecForm "WRITE" "%branchClientSpecPath%" "|" "Client: %branchClient%|Owner: elton.stoneman|Host: myHost|Description: Created by elton.stoneman.|Root: %branchClientRoot%|Options: noallwrite noclobber nocompress unlocked nomodtime normdir|SubmitOptions: submitunchanged|LineEnd: local|View: %branchView% %branchClientView%"

::create branch workspace:
p4 client -i <%branchClientSpecPath%

3. Flag files to be branched

No spec form needed here, but the batch file allows you to branch from the current revisions or from a specified label:

::integrate the branch back to main - branch to label if provided:
IF "%label%"==" " p4 -c%branchClient% integrate -b %branchName%
IF NOT "%label%"==" " p4 -c%branchClient% integrate -b %branchName% @%label%

4. Submit the change list

This is a bit different as the submit command launches a spec form listing the files to be submitted, and requires you to add a description. We can't generate the spec form here, as we'll lose the file list, so instead we output the file list first:

::get a list of the changes (required for Submit):
set changePath=c:\%branchClient%.chg
p4 -c%branchClient% change -o >%changePath%

- and then use the UPDATE mode of the console app to set the description (P4 always uses the same text for the default description):

IF "%label%"==" " GenerateP4SpecForm "UPDATE" "%changePath%" "<enter description here>" "MASTER created from branch: %branchName%"
IF NOT "%label%"==" " GenerateP4SpecForm "UPDATE" "%changePath%" "<enter description here>" "MASTER created from branch: %branchName%, label: %label%"

This way we get the full change list from Perforce and add in our description (in this case, specifying the branch view and label used to get the files). Now we submit using the updated change list:

::submit to create branch in depot:
p4 -c%branchClient% submit -i <%changePath%

And the branch is complete and populated.

Note, the sample batch file leaves the generated spec forms in place, in case of any issues. It's a simple change to make the script delete them at the end.

Branching with Perforce

EltonStoneman — Fri, 04 Apr 2008 14:27:59 GMT

My preferred structure is for each project to have a root in the depot, then each branch to have a root within the project. So as the codebase matures you end up with:

\\depot
    \ProjectX
       \MAIN
          \source
       \R1
          \source
        \R2
          \source
    \ProjectY
       \MAIN
          \source
etc.

Nothing revolutionary there, but it gives some context for the examples.

Branching the Codeline

There are four steps to creating and populating a branch from the main codeline:

Create a branch view
Create a master workspace (client view) for the branch
Flag files to be branched (integrated in Perforce terminology)
Submit the change list from 3) to the depot

(there are other approaches but this is straightforward and recommended. See Practical Perforce - Google Book Search
and Perforce Branching for more details).

Walkthrough

I'll follow this through with the example that ProjectX has just been released to version 3.0. The build applied a label of 3.0.4.86 to the code which has gone into release, and work has been checked into the main branch since, so we want to create a new R3 branch and populate it from the MAIN branch at label 3.0.4.86.

The example will use the command line tool p4.exe, so when we look at automating this in a script the commands will be familiar. Note that in Perforce most command lines take a small number of arguments, then launch a text editor with a pre-populated specification form. After editing and saving the form, the command will complete.

1. Create the branch view

We start by creating a branch with a view that links it back to the main codeline. Following best practice guidelines, the branch name reflects the project and the views it links: ProjectX_MAIN-R3. Creating it is as simple as issuing:

    p4 branch ProjectX_MAIN-R3

- and setting the view in the specification form to map the existing MAIN branch to the new R3 branch:

//depot/ProjectX/MAIN/... //depot/ProjectX/R3/...

This creates the branch view which you will see listed in the Branches tab of P4V, but the new branch is not yet in the depot.

2. Create the master workspace

We'll create a workspace for the branch which will be the master - by integrating the R3 master workspace with the main branch, we'll populate the depot with the desired source. Using the p4 client command, we'll create the workspace ProjectX_R3-MASTER:

    p4 client ProjectX_R3-MASTER

- and in the specifcation form, set the view to map the R3 master to the R3 branch:

//depot/ProjectX/R3/... //ProjectX_R3-MASTER/...

You should now see the new master in the P4V Workspaces view, but the depot won't have an R3 branch populated yet.

3. Flag files to be branched

Using p4 integrate, we create a changelist which will populate the R3 master from MAIN, via the R3 branch:

p4 -cProjectX_R3-MASTER integrate -b ProjectX_MAIN-R3 @3.0.4.86

There is no specification form for this command. Note that we tell P4 to use the R3 master workspace with the -c global option before issuing integrate. This will flag all the files from the R3 branch with the 3.0.4.86 label in the R3 master's default changelist as new additions. To populate the branch in the depot we just need to submit the changelist.

4. Submit the change list

p4 -cProjectX_R3-MASTER submit

The specification form here shows you all the files that will be added and requires you to give the changelist a description - you must change it from the default <enter description here>.

Assuming there were no errors, we'll now have the populated R3 branch under ProjectX, without the changes made in MAIN after the 3.0.4.86 build. You may well have errors as the views you enter in the spec forms are not validated against what's in the depot. This is a brittle operation which is likely to be needed repeatedly, so in the next post we look at scripting the whole branch creation.

Developing Against Branches

You can create your dev workspaces with a view over the whole project (e.g. \\depot\\ProjectX\... \\a.developer\ProjectX\...), so when the new branch is added it will be automatically brought into the developers' workspaces when they Get Latest.

I'd prefer to set workspaces up to a specific branch (\\depot\\ProjectX\MAIN\... \\a.developer\ProjectX\MAIN\...). This means you only have people working in the branch they need. When a new branch is made the people who need to work on it can edit their workspace to add the new view (\\depot\\ProjectX\R3\... \\a.developer\ProjectX\R3\...). The main dev team continue as they are and don't end up having to synchronize to branches they never use.

Alternatively, you can set up a workspace for each branch you work on, so developers would have a main workspace and a Release 1 workspace. The new dev workspace can be created using the branch master as a template. I'm disinclined to use this as there's a greater admin/maintenance/licensing overheead, with no benefits above the single-workspace-multiple-views approach.

Using Perforce with CruiseControl.Net

EltonStoneman — Fri, 04 Apr 2008 13:31:44 GMT

On my latest project we're using Perforce for source control and my first tasks have been to set up CruiseControl.Net for continuous integration, and define the branching strategy to use. I'm new to P4 and wanted to document the learning curve - the next few posts will cover what I've put in place.

Perforce Concepts

Being an SCM with all the usual features, Perforce naturally has a different set of terminology for all the usual concepts. The repository on the server is the depot and the local dev repository is a workspace. The workspace is also known as a client view, which has a mapping to the depot that looks like:

//depot/ProjectX/... //a.developer/ProjectX/...

- meaning all files within the ProjectX node of the depot are mapped to the ProjectX node of a.developer's workspace.

Views are also used in branching and labelling, which I'll cover more in the branch strategy post.

CruiseControl Config

Perforce with CC is a well trodden path, but for reference the sourceControl project node is:

    <sourcecontrol type="p4">
    <view>//depot/ProjectX/MAIN/source/...</view>
        <client>BuildServer</client>
        <applyLabel>true</applyLabel>
        <autoGetSource>true</autoGetSource>
    </sourcecontrol>

The client node specifies the workspace to connect to, and the view specifies the exact mapping for the project. That view is used to monitor the folder for changes, and if you specify autoGetSource and applyLabel, they'll run over all the files in the view. This allows you to set up one workspace for the build server which has a view over the entire depot, and then specify different views for each build project.

There are various warnings saying that Perforce doesn't work with purely numeric labels, but in version 2007.3 that seems fine. With the standard iteration labeller for every successful build CC will create a label in Perforce and label every file in the specified view.

Perforce seems like a nice SCM, with excellent cross-platform support. The licensing model allows you to download and use it for free with up to 2 users and 5 workspaces: http://www.perforce.com/perforce/loadprog.html