Geeks With Blogs
Alex Hildyard

The past few weeks I've been playing with Octopus Deploy, evaluating it as a possible replacement for our existing WiX/Powershell deployment process. I wanted to share with you a problem that many users of Octopus Deploy have encountered, along with a somewhat rough and ready solution to that problem that I've come up with in the short term.

The Problem

I don't just need to use Octopus to deploy to DEV and QA; I need a deployment solution that will allow me to deploy to Production environments in the same way I deploy to pre-Production environments. I have multiple Production environments, some of which are in Data centres, and some of which will not be willing to open up ports to support Octopus, nor prepared to permit the installation of any third party software (in this case, an Octopus tentacle).

The problem is informally addressed here, and Octopus Deploy look at the scenario specifically in the section "Tentacle can't be installed (offline deployments).” This in fact is something Octopus Deploy had previously considered; see also here.

First steps


The penny dropped when I ran octo.exe, and exported all my projects from Octopus. Since the output format was JSON, I was able to write:

$project = Get-Content "project.json" -Raw | ConvertFrom-Json

In one line, I now had my project, workflow, Varable sets, environments and scopes all exposed as an object graph within Powershell. Surely all I needed to do was write a lightweight "workflow runner", and I would have achieved my objective -- a pure Powershell-based deployment script that took an OD project for input?

Actually, things are not quite so straightforward. To their credit, Octopus make this clear.

Look at the section "Project Export File Format", and specifically the comment that LibraryVariableSets are exported by reference. Note also -- though it doesn't say so here -- that ActionTemplates are also exported by reference. What this means is that the JSON you've just imported hasn't pulled in any variables from external LibraryVariableSets you referenced in your project. Neither has it pulled in the content of any ActionTemplates upon which your project might depend.

There is a good reason for this, and I need to make clear that what I'm proposing in this post runs somewhat counter to the set of design decisions made by the OD team. If you pull down the Octopus Tools source code from GitHub, you can see that there is plenty of code there to cover what is effectively a synchronisation activity during project import -- resource keys are tallied with existing keys and updated as required. And, if you take a step back, this makes perfect sense: If you have a Production Octopus server, and you want to import a project exported from a Development Octopus server, you don't want to be overwriting the presumably bonafide Production VariableSet(s). What about script and Step templates? That'a slightly greyer area. The bottom line is this: the idea of code promotion, by means of which you move a project without its variables from one environment to another -- simply doesn't sit well with the idea of a Tentacle-free deployment in which your "project" and your "release" are essentially conflated into a single object which holds all the configuration information it needs to be deployed. That's why I feel I'm trying to get Octopus to do something it was never intended to do. But that said, here's how to do it ...

Hacking Octopus Tools


Rather than fiddle with the existing import/export format, and thereby prevent you from using octo.exe with existing projects, we're just going to export a little bit more data than it currently exports.

First, open ProjectExport.cs. You'll need to add a couple of new properties:

        public List<VariableSetResource> LibraryVariableSetsResources { get; set; }
        public List<ActionTemplateResource> ActionTemplatesResources { get; set; }

We'll use these to export the Library variables and Action templates by value rather than by reference.

Next, make the following changes to ProjectExporter.cs:

          Log.Debug("Finding action templates for project");
            var actionTemplates = new List<ReferenceDataItem>();
            var actionTemplatesResources = new List<ActionTemplateResource>();
            foreach (var step in deploymentProcess.Steps)
            {
                foreach (var action in step.Actions)
                {
                    string templateId;
                    if (action.Properties.TryGetValue("Octopus.Action.Template.Id", out templateId))
                    {
                        Log.Debug("Finding action template for step " + step.Name);
                        var template = actionTemplateRepository.Get(templateId);
                        if (template == null)
                            throw new CommandException("Could not find action template for step " + step.Name);
                        if (actionTemplates.All(t => t.Id != templateId))
                        {
                            actionTemplates.Add(new ReferenceDataItem(template.Id, template.Name));
                            actionTemplatesResources.Add(template);
                        }
                    }
                }
            }

       var libraryVariableSets = new List<ReferenceDataItem>();
            var libraryVariableSetsResources = new List<VariableSetResource>();
            foreach (var libraryVariableSetId in project.IncludedLibraryVariableSetIds)
            {
                var libraryVariableSet = Repository.LibraryVariableSets.Get(libraryVariableSetId);
                var libraryVariableSetResource = Repository.VariableSets.Get(String.Format("VariableSet-{0}", libraryVariableSetId));
               
                if (libraryVariableSet == null)
                {
                    throw new CommandException("Could not find Library Variable Set with Library Variable Set Id " + libraryVariableSetId);
                }

                libraryVariableSets.Add(new ReferenceDataItem(libraryVariableSet.Id, libraryVariableSet.Name));
                libraryVariableSetsResources.Add(libraryVariableSetResource);
            }

  
var export = new ProjectExport
            {
                Project = project,
                ProjectGroup = new ReferenceDataItem(projectGroup.Id, projectGroup.Name),
                VariableSet = variables,
                DeploymentProcess = deploymentProcess,
                NuGetFeeds = nugetFeeds,
                ActionTemplates = actionTemplates,
                ActionTemplatesResources = actionTemplatesResources,
                LibraryVariableSets = libraryVariableSets,
                LibraryVariableSetsResources = libraryVariableSetsResources
            };


Finally, recompile octo.exe. You can now invoke it with a command line like:

octo.exe export --server=http://my-octopus-svr --apiKey=API-MYOCTOPUSAPIKEY --type=project --name="someProject" --filePath="C:\temp\someProject.json"


Introducing Calamari


Why Calamari? Because that's what you get when you chop an Octopus' tentacle into little pieces. Calamari is a pure Powershell "Octopus Workflow Runner" generator, that pulls in the JSON exported from our modified octo.exe, and exports a standalone Powershell file that will run that workflow in the context of a specified environment and server role. It's an MVP, with all the usual caveats. No logging, no error handling; it only handles rudimentary scope resolution (environment and role), and exposes only the most basic environment variables; it makes no attempt to embed offline resources, etc., but I was able to use it successfully to create offline installers for the various Websites and Windows services I needed to deploy. I've tried to comment the code as much as possible to make clear what it's doing, and why.

You can invoke it like this:

Calamarify -Project "project.json" -Environment "DEV1" -Role "app" > offline_app_install.ps1

The code is here:


function Calamarify
{
     param(
        [parameter(mandatory=$True, HelpMessage='The file path to the Octopus project to load')]
        [string]$Project,
 
        [parameter(mandatory=$True, HelpMessage='The Environment for which you want to generate a deployment script')]
        [string]$Environment,
      
        [parameter(mandatory=$True, HelpMessage='The Role for which you want to generate a deployment script')]
        [string]$Role
        )


    # Fail on errors
    $ErrorActionPreference = "Stop"

    # Set some locals
    $role = $Role
    $environment = $Environment
    $projectName = $Project

    Write-Output "# Deployment script for project: $Project, environment: $environment, role: $role"

    # Import the project
    $obj = ConvertFrom-Json(Get-Content $projectName -Raw)

    # Get the Environment ID
    $environmentId = ($obj.VariableSet.ScopeValues.Environments | Where { $_.Name -match $environment }).Id

    # Collect together our immediate (project-local) variables; cast to an ArrayList so that we can append further items in a moment
    $vars = [System.Collections.ArrayList]$obj.VariableSet.Variables
   
    # Now add in any other Library VariableSets that were included; this piece of code depends on changes I've made to Octo.exe. If you don't have these,
    # LibraryVariableSetsResources won't exist, and this code loop will be skipped
    $obj.LibraryVariableSetsResources |% `
    {
        $libId = [string]($_.Id)
        Write-Output "# Including Library Variables for $libId"
        $vars.AddRange($_.Variables)
    }

    # Perform an initial resolution of the variables, based on our scope; ignore duplicates
    $vars = $vars | Where `
    {
        # Filter: No scope and no environment; no scope and same enviroment; same scope and same environment; same scope and no environment; note that this
        # is a simplification of Octopus' actual scope resolution, since where a variable matches multiple filters, the most inclusive filter should provide
        # the ultimate value
        $newVar = $_
        ([string]$newVar.Scope.Role -match $role -or [string]$newVar.Scope.Role -eq "") -and ([string]$newVar.Scope.Environment -match $environmentId -or [string]$newVar.Scope.Environment -eq "")
    }

    # Now cull any duplicates; there's probably a neater way to do it, but I'm piping the ArrayList into a Hashtable, and then piping it back again to remove duplicates
    $hash = @{}
    $vars |% { $hash[$_.Name] = $_ }
    $vars = $hash.Values

    # Now expand the variables found in the VariableSet and LibraryVariableSet objects (ie. the ones of the form #{TOKEN})
    $expanded = $True

    while ($expanded -eq $True)
    {
        $expanded = $False
       
        # For every variable in the collection of variables,
        $vars |% `
        {
            # considered as a key/value pair of the form #{KEY} = VALUE,
            $thisVar = $_
            $thisVarValue = [string]$thisVar.Value
            $thisToken = "#{" + [string]$thisVar.Name + "}"
           
            # locate all the variables whose VALUE includes a reference to this KEY
            $vars |? `
            {
                $search = $_.Value
                $search -match $thisToken
            } |% `
            {
                # and replace the reference with its immediate expansion
                $subst = $_
                $subst.Value = $subst.Value.Replace($thisToken, $thisVarValue)
                $expanded = $True
            }
        }
    }

    # Create an OctopusParameters hashtable (this is required for OD variables with restricted characters in their names, for example "-"
    Write-Output "`$OctopusParameters = @{}"

    # Add a few constants
    Write-Output "`$OctopusParameters['Octopus.Machine.Name'] = '$env:COMPUTERNAME'"

    # Add the contents of any script modules
    $scripts = $vars |? `
    {
        $_.Name -match "Octopus.Script.Module"
    }
   
    $scripts |% `
    {
        # Write out the script
        $script = $_       
        Write-Output $script.Value
       
        # And remove it from the collection; since the collection is fixed size, we remove it by replacing the collection with itself,
        # having first filtered out the item we no longer require
        $vars = $vars |? { $_.Name -ne $script.Name }   
    }  
    # And write out the variables; put everything into the OctopusParameters collection, and also create PS variables for those permitted
    # within PS (those which don't contain a hyphen)
    $vars |% `
    {
        Write-Output([System.String]::Format('$OctopusParameters[''{0}''] = ''{1}''', $_.Name, $_.Value))
       
        if ($_.Name -notmatch "-")
        {
            Write-Output([System.String]::Format('${0} = ''{1}''', $_.Name, $_.Value))
        }
    }   
  
    # Now walk $obj.DeploymentProcess.Steps, writing out the step bodies in order where Role and Environment correctly scope
    $steps = $obj.DeploymentProcess.Steps | Where { $_.Properties.'Octopus.Action.TargetRoles' -match $role }

    Write-Output "# Workflow follows"

    $steps |% `
    {
        $step = $_
        Write-Output([System.String]::Format("# Step: {0}", $step.Name))

        ($_.Actions | Where { $action = $_; [string]$action.Environments -eq "" -or [string]$action.Environments -match $environmentId }) |% `
        {
            $action = $_
       
            Write-Output([System.String]::Format("# Action: {0}", $action.Name))
           
            # Write out the PSCustomObject properties; omit the script body, because we need to initialise
            # the variables first
            $action.Properties | Get-Member -MemberType *Property |? { $_.name -inotmatch "Octopus.Action.Script.ScriptBody" -and $_.name -ne "Value" } | Select-Object |% `
            {
                $prop = $_
               
                $propIndex = $prop.Definition.IndexOf('=')
                #$keyVal = "'" + $prop.Definition.Substring($propIndex+1, $prop.Definition.Length - $propIndex-1) + "'"
                $keyVal = $prop.Definition.Substring($propIndex+1, $prop.Definition.Length - $propIndex-1)
               
                # Resolve references to OD #{TOKEN}; we use a match that permits hyphens and underscores as well as alphabetic letters
                [regex]::Matches($keyVal, '#{[A-Z|a-z|-]+}') |% `
                {
                    $_.groups |% `
                    {
                        $key = $_.Value
                        $key_untokenised = $key.Substring(2, $key.Length-3)
                        $substitutions = $vars | Where { $sub = $_; $sub.Name -eq $key_untokenised }
                        $val = $vars | Where { $_.Name -eq $key_untokenised }

                        $substitutions |% { $keyVal = $keyVal.Replace($key, $val.Value) }
                    }
                }
   
                # Escape any single quotes in the resultant string
                $keyVal = "'" + $keyVal.Replace("'", "''") + "'"
               
                # Write out the resolved variable as both a PS and an OD variable
                $psVar = '$' + $prop.name + " = $keyVal"
                $octVar = '$OctopusParameters[''' + $prop.Name + "'] = $keyVal"
               
                # If it begins with "Octopus" and contains dots, then it's just an "Octopus" parameter, so don't generate a PS-style parameter in this instance
                if (!$psVar.StartsWith('$Octopus') -and !$psVar -match ".")
                {
                    Write-Output $psVar
                }

                Write-Output $octVar
            }

            # Now write out the script body, leaving all the variables "as is"
            Write-Output $action.Properties.'Octopus.Action.Script.ScriptBody'
           
            Write-Output([System.String]::Format("# End of Action: {0}", $action.Name))
        }

        Write-Output([System.String]::Format("# End of Step: {0}", $step.Name))
    }

    Write-Output "# End of workflow"
}

Posted on Thursday, January 1, 2015 2:00 PM | Back to top


Comments on this post: Calamari and Octopus Deploy: Deploying Applications without a Tentacle

# re: Calamari and Octopus Deploy: Deploying Applications without a Tentacle
Requesting Gravatar...
I am trying to run this PS script and is prompting for some parameters. can't get this script run properly. Is it something i am missing?

cmdlet ForEach-Object at command pipeline position 1
Supply values for the following parameters:
Process[0]:
Left by Suresh on Mar 22, 2015 4:07 AM

# re: Calamari and Octopus Deploy: Deploying Applications without a Tentacle
Requesting Gravatar...
Hi Suresh,

You may be missing a backtick, or perhaps you aren't running under the same version of Powershell (I used Powershell 4.0).

In the first instance, I'd recommend stepping through the script line by line in the ISE, and see if you can identify the problem that way.

Regards,

Alex
Left by Alex Hildyard on Mar 25, 2015 12:20 AM

Your comment:
 (will show your gravatar)


Copyright © Alex Hildyard | Powered by: GeeksWithBlogs.net