ILearnable .Net

May 8, 2010

Build/Deploysetup for a fairly large SOA project, part 2: deployment

Filed under: Uncategorized — andreakn @ 20:17

so, in part one I explained some context for the project and the build setup ending up with how ccnet gets nant to deploy compiled code artifacts etc. to a local folder on the build machine (a subfolder under d:\deploy) so that a completely fine runnable version of the SOA stack runs on the build server. in this part I’ll explain how we take the artifacts from this location and install them on our other environments.

Specifically, I’ll be talking about:
1) Databases
2) code
3) configuration
4) backup / restore

Let’s start
1) Databases
DBs are usually the hardest to work with deployment-wise, no exception here. Let me just start off by saying that I’m convinced that using a migration scheme for DBs for instance using for instance migrator.net would be SO much easier than the route we’ve taken (hindsight is always 20-20, oh well) but we do have a solid getup ourselves even though it reeks of MacGyverisms.
There are approx 15 databases (most solutions have their own database) and they are all built by the default.build file as explained in part one just before the unittests are run. The mechanism for building the DB is basically as follows: a dev does modifications to the DB in SQL Server Manager, then runs a proprietary .exe file scripting out the DB schema into sql script files which are then checked in to svn. When the build process later runs a different proprietary .exe file those same sql files ar run in the correct order on each build. effectively re-creating the DB from scratch on the buildserver.

Naturally we cannot use the same process in our other environments, so what we do is run a product called Redgate Sql Compare to sync the changes into the “next” environment (the environments are build => devtest => test => production.) This is a somewhat tedious and error-prone manual process (especially when deploying to production at 00:15 am) and the tool is not perfect (especially when it comes to the norwegian letters ø å æ )

2) code
We use powershell to deploy our code. In our scripts we divide the codebase into the website, the internal services, the external services and the scheduled tasks as this distinction reflects what we usually deploy on every go. I’ll omit the external services from the examples as they are really fragmented into a lot of subsets.

Powershell is a really powerful scripting environment, especially if you’re already familiar with Microsoft.Net, but it has two major drawbacks (at least as of version 1 which is the one we’re using):

1) it’s hard to split scripts into multiple files. Effectively what you are left with is doing the following:

$scriptToBeIncluded = Resolve-Path 'D:\temp\some\absolute\script\path\ScriptToBeIncluded.ps1'
. $scriptToBeIncluded

with no baked in concept of “has this script been loaded before?”, so the start of the script files to follow is all about managing the inclusion of other scripts.

2) If you do a foreach($item in $someItems) and $someItems is null (or $null rather in powershell) you will actually get one “hit” into the foreach body where $item is set to the value $null. Exactly which use case the powershell team had in mind when they coded that monstrosity I will not even begin to speculate, but it does lead to a helluvalot null checks throughout the scripts.

When developing the scripts I used Powershell GUI which is an opensource free editor with inbuilt debugger. Works like a charm!.

So, on to the code:

We have 4 powershell script files:
a) DeployMachines.ps1 (contains the definitions of all the machines in all the environments, where stuff is placed. These definitions are placed inside anonymous objects called $server throughout the code)
b) DeploymentCommonFunctions.ps1 (contains powershell functions which operate on the before mentioned server objects)
c) DEPLOY_SCRIPT.ps1 (contains a wizard guiding the user through the deployment process)
d) Backup_Script.psq (contains wizard for more fine-grained control of backups, I’ll ignore it for this writeup)

# Deploy_machines.ps1
#
# This file contains the definition of the servers to which deploys are done.
# This means that no servers should be hard-coded in other scripts, but they should include this script
# to get the server definitions. This script is included in the script DeploymentCommonFunctions.ps1
# so any script including that script will also get this script
# 
#

#
#  Common paths / filters to be used for most or all deploy scripts
#

$script_loaded_machines = $true
"loaded script Deploy_Machines.ps1"

$development_mode = $false #when in development mode, no actual changes will happen on disk, but script logs all actions it would have done

$deploy_staging_area = "d:\serverdeploy\"

$config_exclude = "*.config","thumbs.db","*.txt","*.pdb","*.properties"
$config_include = "*.config","robots.txt","*.properties"

$servicedirexclude = @("LoggingService","Deploy.*")

$wait_seconds_after_task_shutdown = 60 # 60 seconds default
$wait_seconds_after_services_shutdown = 30 # 30 seconds default

$deploy_log_file = "d:\deployment\Logs\deployment_log.txt"

$all_servers = @()
$servertype_customizations = @()


#
# Servertype definitions
#
$servertypes = New-Object Object `
| Add-Member -MemberType NoteProperty -Name InternalServices -Value  "InternalServices" -PassThru `
| Add-Member -MemberType NoteProperty -Name Web -Value  "Web" -PassThru `
| Add-Member -MemberType NoteProperty -Name Tasks -Value  "Tasks" -PassThru 

#available object properties, since we have created this object the key values can be accessed through $key.
$key = New-Object Object `
| Add-Member -MemberType NoteProperty -Name ServerType -Value  "ServerType" -PassThru `
| Add-Member -MemberType NoteProperty -Name ServerName -Value  "ServerName" -PassThru `
| Add-Member -MemberType NoteProperty -Name Drive -Value "Drive" -PassThru `
| Add-Member -MemberType NoteProperty -Name Environment -Value "Environment" -PassThru `
| Add-Member -MemberType NoteProperty -Name Path -Value "Path" -PassThru `
| Add-Member -MemberType NoteProperty -Name Number -Value "Number" -PassThru `
| Add-Member -MemberType NoteProperty -Name BuildPath -Value "BuildPath" -PassThru `
| Add-Member -MemberType NoteProperty -Name BackupPath -Value "BackupPath" -PassThru `
| Add-Member -MemberType NoteProperty -Name OnlyUpgradeExistingDirs -Value "OnlyUpgradeExistingDirs" -PassThru

#create a server object from the input data
function Create-Machine([string] $Id,[string] $Environment, [string] $ServerType, [string] $ServerName,`
                        [string] $Drive,[string] $Path,[string] $BackupPath,[bool] $OnlyUpgradeExistingDirs, `
						[string] $DeployFromTestServer, [string] $BuildPath){
	$valid_servertype = Get-Member -InputObject $servertypes -Name $ServerType
	if($valid_servertype -eq $null){
		Throw [System.ArgumentException]
	}
	foreach($server in $all_servers){
		if($Id -eq $server.Id){
			$msg = [System.String]::Format("Server with Id = {0} already defined in server list",$Id)
			Throw ($msg)
		}
	}
	
	$new_server = New-Object Object `
	| Add-Member -MemberType NoteProperty -Name Id -Value $Id -PassThru `
	| Add-Member -MemberType NoteProperty -Name Environment -Value $Environment -PassThru `
	| Add-Member -MemberType NoteProperty -Name ServerType -Value  $ServerType -PassThru `
	| Add-Member -MemberType NoteProperty -Name ServerName -Value  $ServerName -PassThru `
	| Add-Member -MemberType NoteProperty -Name Drive -Value $Drive -PassThru `
	| Add-Member -MemberType NoteProperty -Name Path -Value $Path -PassThru `
	| Add-Member -MemberType NoteProperty -Name BackupPath -Value $BackupPath -PassThru `
	| Add-Member -MemberType NoteProperty -Name OnlyUpgradeExistingDirs -Value $OnlyUpgradeExistingDirs -PassThru `
	| Add-Member -MemberType NoteProperty -Name DeployFromTestServer -Value $DeployFromTestServer -PassThru `
	| Add-Member -MemberType NoteProperty -Name BuildPath -Value $BuildPath -PassThru 

	return $new_server
}


#
# machine definitions
#
# The machine definitions are created by making dynamic objects containing the following properties
# - Id  -> the Id of the deployment target on the form <environment>_<servertype>_<number>
# - Environment -> devtest, test (also used when hotfixing), prod
# - ServerType -> must match entries in $servertypes object
# - ServerName -> must match an entry in c:\windows\system32\drivers\etc\hosts on the build machine 
# - Drive -> the disk drive on which stuff is installed / backed up
# - Path -> the root path (on the defined drive) to deploy to (and backup from)
# - BackupPath -> the path (on the defined drive) to backup to
# - OnlyUpgradeExistingDirs (SPECIAL) -> Will only deploy a directory if the directory already exists in deploy target
# - DeployFromTestServer ->  (when preparing for deploy to test or prod: which deployment target Id to get files (dlls etc) from)
# - BuildPath -> (when preparing for deploy to devtest or hotfixing to test: which path on the build server to get files (dlls etc) from)
# 
# a separate array is defined for each type of server. If a server has more than one type 
# (web servers also host a service for instance) then it needs to be defined in all the relevant arrays
# 
#Explanation of parameters:    *Id*                   *env*     *type*  *machinename*  X:\ *depl.path*    *backuppath*  *onlyUpgr.*  *GetFromId*       *BuildPath*     
$all_servers += Create-Machine "devtest_tasks_1"      "devtest" "tasks" "10.67.2.15" "e" "Tasks"        "Backups"       $true      $null             "d:\DEPLOY\Tasks\"  
$all_servers += Create-Machine "test_tasks_1" 		  "test"    "tasks" "10.50.1.133" "c" "Code\Tasks" "Backups" 		$true      "devtest_tasks_1" "\\HotfixBuildServer\d$\DEPLOY\Tasks\"  
$all_servers += Create-Machine "prod_tasks_1" 		  "prod"    "tasks" "192.168.8.16" "d" "Code\Tasks" "Backups" 		$true      "test_tasks_1"    $null 

#and on and on the list goes.. we have 58 entries in total, only 3 showed here

$servertype_customizations += New-Object Object `
| Add-Member -MemberType NoteProperty -Name ServerType -Value  $servertypes.Web -PassThru `
| Add-Member -MemberType NoteProperty -Name ServiceNames -Value  @() -PassThru `
| Add-Member -MemberType NoteProperty -Name StopTaskSqlAgent -Value $false -PassThru  `
| Add-Member -MemberType NoteProperty -Name RunsIIS -Value $true -PassThru  `
| Add-Member -MemberType NoteProperty -Name ExcludeSubDirs -Value @() -PassThru 


$servertype_customizations += New-Object Object `
| Add-Member -MemberType NoteProperty -Name ServerType -Value  $servertypes.InternalServices -PassThru `
| Add-Member -MemberType NoteProperty -Name ServiceNames -Value  @("BEProtocolService","SystemBetProcessingService") -PassThru `
| Add-Member -MemberType NoteProperty -Name StopTaskSqlAgent -Value $true -PassThru  `
| Add-Member -MemberType NoteProperty -Name RunsIIS -Value $true -PassThru  `
| Add-Member -MemberType NoteProperty -Name ExcludeSubDirs -Value @( "ExternalInfoService") -PassThru 


$servertype_customizations += New-Object Object `
| Add-Member -MemberType NoteProperty -Name ServerType -Value  $servertypes.Tasks -PassThru `
| Add-Member -MemberType NoteProperty -Name ServiceNames -Value  @("SQLSERVERAGENT") -PassThru `
| Add-Member -MemberType NoteProperty -Name StopTaskSqlAgent -Value $false -PassThru  `
| Add-Member -MemberType NoteProperty -Name RunsIIS -Value $false -PassThru  `
| Add-Member -MemberType NoteProperty -Name ExcludeSubDirs -Value @() -PassThru 






As the trained eye will spot, the machines defined in the devtest environment fetches their compiled code from d:\deploy from the buildserver(the last parameter to Create-Machine) but the test environment fetches its code from the devtest environment (second last parameter), and the prod environment get its code from the test environment, there is also the possibility of getting code into test from the hotfix-buildserver, effectively bypassing the devtest environment

b) Deployment_Common_Functions.ps1
this is the real “meat” of the deployment script. it contains functionality for
1) preparing the deployment to a directory on the build server
– fetching the code to be deployed from “previous environment” (for devtest that means the code just built)
– fetching the configuration files from the very environment we are preparing a deploy for. The assumption being that whatever configuration is currently in that environment is the correct one, and any changes that needs to be done because of this deploy needs to be done manually on those files after they’ve been prepared.
2) deploying the prepared files to a target in an environment
3) backing up from a target in an environment
4) restoring a backup
5) cleaning up list of backups from environments


if($script_loaded_machines -eq $false){
	$deploy_machines_init = Resolve-Path 'D:\deployment\Deploy_Machines.ps1' 
	. $deploy_machines_init
}
"loaded script DeploymentCommenFunctions.ps1"
$script_loaded_common = $true
#
# DeploymentCommonFunction.ps1
#  
# This script contains useful functions to be called from other more specific scripts
# To include this script in another script the following should be typed in the top of the other script:
#
# $deploy_functions_init = Resolve-Path 'D:\deployment\DeploymentCommonFunctions.ps1' 
# . $deploy_functions_init
#
# 


################## ENVIRONMENT STUFF #############

function log([string] $message){
	Write-Host $message
}

function trace([string] $message){
	if($development_mode){
		Write-Host $message
	}
}

function Log-Deploy([string] $message){
	$log_time = [DateTime]::Now.ToString("yyyy-MM-dd HH-mm")
	$log_message = $log_time +" - "+$message
	Add-Content $deploy_log_file $log_message
}

function Exit-Deploy(){
	Log-Deploy "-----------------EXITed from deploy-script"
	exit
}

########## PARSING #############

function Prompt-YesNo($text){
	Write-Host $text
	$input = Read-Host
	if($input -eq "y" -or $input -eq "Y"){
		Write-Host "You selected YES"
		return $true		
	}elseif($input -eq "n" -or $input -eq "N"){
		Write-Host "You selected NO"
		return $false		
	}
	Write-Host "could not parse your answer, exiting"
	Exit-Deploy	
}

########### PROMPTS ###################

function Prompt-WhichServers($servers){
	log ""
	log "On which servers do you want to operate"
	log "input their ids comma separated"
	log "(default = all )"
	foreach($server in $servers){
		if($server -eq $null){
			continue
		}
		$n = $server.Id
		$sn = $server.ServerName
		Write-Host " $n - $sn"
	}
	$input = Read-Host
	$deploy_servers = @()
	
	switch($input){
		""{$deploy_servers = $servers}
		default{
			foreach($n in $input.Split(',')){
				$n = $n.Trim()
				$deploy_servers += $servers | Where{$_.Id -eq $n}
			}
		}
	}
	return $deploy_servers
}



function Prompt-BeginDeploy($servers){
	log ""
	log "We are ready to deploy to the following targets:"
	$a = @{Expression={$_.Id};Label="ID";width=4}, `
	@{Expression={$_.ServerName};Label="Server";width=25}, `
	@{Expression={$_.Drive};Label="Disk";width=4}, `
	@{Expression={$_.Path};Label="Deploy path";width=25}, `
	@{Expression={$_.ServerType};Label="Type";width=25}
	foreach($server in $deploy_servers){
		$sn = $server.ServerName
		$sd = $server.Drive
		$sp = $server.Path
		$st = $server.ServerType
		$si = $server.Id
		log "($si): \\$sn\$sd`$\$sp  ($st)"	
	}
	
	log "Press enter to continue"
	$input = Read-Host
}


function Prompt-ChildrenFilter($servers){
	$childrenFilter = @()
	
	foreach($server in $servers){
		$server_root = Get-DeployUNCPath $server
		
		foreach($temp in Get-ChildItem $server_root){
			$addIt = $true
			foreach($alreadyInList in $childrenFilter){
				if($alreadyInList -eq $temp.Name){
					$addIt = $false
					break
				}
			}
			if($addIt){
				$childrenFilter += $temp.Name
			}
		}
	}
	
	Write-Host "Which Children do you want?"
	Write-Host "(comma separated)"
	for($i = 1; $i -le $childrenFilter.Length; $i++){
		$child = $childrenFilter[$i-1]
		Write-Host " $i - $child"
	}
	$input = Read-Host
	
	$newChildrenFilter = @()
	
	switch($input){
		""{$newChildrenFilter = $childrenFilter}
		default{
			foreach($n in $input.Split(',')){
				$n = $n.Trim()
				$i = [int]::Parse($n)
				$newChildrenFilter += $childrenFilter[$i-1]
			}
		}
	}
	
	Write-Host "So, you want:"
	for($i = 1; $i -le $newChildrenFilter.Length; $i++){
		$child = $newChildrenFilter[$i-1]
		Write-Host " $child"
	}	
	Write-Host "Press Esc now if this is wrong, enter to continue"
	$input = Read-Host
	return $newChildrenFilter
}


function Prompt-WantsChildrenFilter($servertype){
	return Prompt-YesNo "Do you want to deploy single instances of $servertype (y/n)"
}


############# GETTERS ON "SERVER" OBJECTS 

#
# Gets the path on which stuff will be deployed for a given server object
#
function Get-DeployUNCPath($server){
	return "\\"+$server.ServerName+"\"+$server.Drive+"$\"+$server.Path+"\"
}

#
# Gets the path from which stuff will be deployed to test for a given server object
#
function Get-PrepareSourcePath($server){
	if($server.IsTest){	
		return $server.BuildPath
	}
	else{
		$source_server = $all_servers | where{$_.ServerType -eq $server.ServerType -and $_.IsTest -eq $true -and $_.Id -eq $server.DeployFromTestServer}
		return Get-DeploySourcePath $source_server
	}
}

#
# Gets the path on which stuff will be prepared
#
function Get-PreparePath($server){
	
	return $deploy_staging_area+$server.Environment+"\"+$server.ServerType+"\"+$server.ServerName+"\"+$server.Path+"\"
}

#
# Gets the path on which stuff will be backed up to
#
function Get-BackupUNCPath($server){
	return "\\"+$server.ServerName+"\"+$server.Drive+"$\"+$server.BackupPath+"\"
}


############## START / STOP REMOTE SERVICES #####################

#
# starts a service on a specified machine
#
function Start-RemoteService([string] $machine_name, [string] $service_name){
	log "starting service $service_name on $machine_name"

	if($development_mode){return}
	
	$servicecontroller = [System.ServiceProcess.ServiceController]::GetServices($machine_name) | where{ (($_.name -eq $service_name) -or ($_.displayname -eq $service_name))}
	if ($servicecontroller -eq $null)
	{
		continue;
	}
	if($servicecontroller.Status -ne 'Running'){
		$servicecontroller.Start() 
		$servicecontroller.WaitForStatus('Running',(new-timespan -seconds 60))
	}
	log "service $service_name started on $machine_name"
	
}

#
# Stops a service on a specifiec machine
#
function Stop-RemoteService([string] $machine_name, [string] $service_name){
	log "stopping service $service_name on $machine_name"
	if($development_mode){return}

	$servicecontroller = [System.ServiceProcess.ServiceController]::GetServices($machine_name) | where{ (($_.name -eq $service_name) -or ($_.displayname -eq $service_name))}
	if ($servicecontroller -eq $null)
	{
		return;
	}
	
	if($servicecontroller.Status -ne 'Stopped'){
		$servicecontroller.Stop() 
		$servicecontroller.WaitForStatus('Stopped',(new-timespan -seconds 60))
	}
	log "service $service_name stopped on $machine_name"
}



#################  BASE OPERATIONAL FUNCTIONS (works on paths, no prompting) ###################

#
# Backs up from a given location to a different given location on the same machine and same drive
#
function Backup-DeployedStuff([string]$machine_name, [string] $drive_letter, [string] $deployment_location, [string] $backup_location="", [string] $backup_name){
	if($backup_name -eq $null -or $backup_name -eq ""){
		$backup_name = [DateTime]::Now.ToString("yyyy-MM-dd HH-mm")
	}
	$full_deploy_path ="\\"+$machine_name+"\"+$drive_letter+"$\"+$deployment_location
	$backups_path ="\\"+$machine_name+"\"+$drive_letter+"$\"+$backup_location+"\"+$backup_name
	log "backing up from $full_deploy_path to $backups_path"
	
	if($development_mode){return}
	
	if(! (Test-Path	 ($backups_path) -PathType Container) ){
		md ($backups_path)
	}
	
	Get-ChildItem $full_deploy_path -Recurse -Force | Copy-Item -Destination {Join-Path $backups_path $_.FullName.Replace($full_deploy_path,"")}
}

function Restore-DeployedStuff([string]$machine_name, [string] $drive_letter, [string] $deployment_location, [string] $backup_location="", [string] $backup_name){
	if($backup_name -eq $null -or $backup_name -eq ""){
		throw "No backup-name specified"
	}
	
	
	$backups_path ="\\"+$machine_name+"\"+$drive_letter+"$\"+$backup_location+"\"+$backup_name
	$full_deploy_path ="\\"+$machine_name+"\"+$drive_letter+"$\"+$deployment_location
	log "restoring up from $backups_path to $full_deploy_path"
	
	if($development_mode){return}
	
	if(! (Test-Path	 ($backups_path) -PathType Container) ){
		throw "no backup found at $backups_path"
	}

	Remove-Item $full_deploy_path -Recurse -Force
	md $full_deploy_path

	Get-ChildItem $backups_path -Recurse -Force | Copy-Item -Destination {Join-Path $full_deploy_path $_.FullName.Replace($backups_path,"")}
	
}

#
# Deploys Services from one root path to another. 
# It actually deploys the child directories of the root path, and not the root path itself
#
function Deploy-Subdirectories([string] $source_root_path, [string] $destination_root_path, $childrenFilter){	
	log "starting to deploy children inder under $source_root_path to $destination_root_path"
	$filterChildren = $true
	if($childrenFilter -eq $null -or $childrenFilter.Length -eq 0){
		$filterChildren = $false
	}
	foreach($child_path in Get-ChildItem $source_root_path){
		$childname = $child_path.Name
		
		if($filterChildren){
			if(!($childrenFilter -contains $childname)){
				continue;
			}
		}
			
		$destination_path = $destination_root_path + $childname
		$source_path = $source_root_path + $childname
		Deploy-Directory $source_path $destination_path 
	}
}



#
# Deploys from one directory to another, 
# It first deletes every file (but leaves directories alone) under the destination directory
# then copies everything from the source to the destination
# (this is done to keep directory security settings)
#
function Deploy-Directory([string] $source_path, [string] $destination_path){
	log "deploying everything under $source_path to $destination_path"
	if($development_mode){return}

	$removequeue = Get-ChildItem $destination_path -Recurse -Force | Where{$_.psIsContainer -eq $false} 
	foreach($item_to_remove in $removequeue){
		if($item_to_remove -ne $null -and $item_to_remove.FullName -ne $null){
			Remove-Item $item_to_remove.FullName -Force 
		}
	}
	Copy-Directory $source_path $destination_path
}

#
# Copies the content recursively from one directory to another directory
# however it only copies things that does not already exist in the destination 
# it never replaces files or folders (this is done to keep directory security settings)
#
function Copy-Directory([string] $source_path, [string] $destination_path){
	if($development_mode){return}

	$copyqueue = New-Object System.Collections.ArrayList
	Get-ChildItem $source_path -Recurse -Force | foreach {$null = $copyqueue.Add($_)}
	foreach($copyqueue_item in $copyqueue){
		$destination = $destination_path + $copyqueue_item.FullName.Substring($source_path.Length)
		if(!(Test-Path $destination)){
			Copy-Item $copyqueue_item.FullName $destination -Force 
		}
	}
}


function Prepare-Directory([string]$code_source_path, [string] $config_source_path, [string] $prepare_area_path){
	log "Copying compiled code from $code_source_path to $prepare_area_path"
	log "Copying configs from $config_source_path to $prepare_area_path"
	
	if($development_mode){return}

	$source_files = Get-ChildItem -Recurse -Exclude $config_exclude $code_source_path  
	if($source_files -eq $null){
		Log-Deploy "WARNING: Could not find any files to deploy from source $code_source_path Should the target be removed from the environment? "
		return;
	}

	if (!(Test-Path $prepare_area_path))
	{
		md $prepare_area_path
	}
	
	foreach($source_file in $source_files){
		$destination_file = Join-Path $prepare_area_path $source_file.FullName.Substring($code_source_path.length)
		$directory_exists = Test-Path $destination_file -PathType Container
		if($source_file.PSIsContainer -and $directory_exists){
			continue
		}
		else{
			Copy-Item $source_file $destination_file -Force
		}
	}
#	Copy-Item $source_files -Destination {Join-Path $prepare_area_path $_.FullName.Substring($code_source_path.length)} -ErrorAction SilentlyContinue
	$configs = Get-ChildItem -Recurse -Include $config_include $config_source_path 
	if($configs -eq $null){
		continue
	}
	foreach($config in $configs){
		if($config_source_path.EndsWith("\")){
			$config_source_path = $config_source_path.Substring(0,$config_source_path.Length - 1)
		}
		if($prepare_area_path.EndsWith("\")){
			$prepare_area_path = $prepare_area_path.Substring(0,$prepare_area_path.Length - 1)
		}
		$config_target_dir = $prepare_area_path + $config.Directory.FullName.Substring($config_source_path.Length)
				
		if(Test-Path -Path $config_target_dir -PathType Container){
			Copy-Item $config.FullName -Destination  $config_target_dir 
		}
	}
}


function Do-Remove-AllExceptSvn($path){
	$fullPath = $path.FullName
	trace "deleting area $fullPath"

	if(! $development_mode){
		if (!(Test-Path $path))
		{
			md $path
		}
		
		$escExcDir = "\.svn"
		$items = Get-ChildItem $path -Recurse -Force  | Where {$_.FullName -notMatch $escExcDir} |sort FullName -Descending
		
		foreach($item in $items){
			if(!$item.PSIsContainer -and $item -ne $null  ) {
				Remove-Item $item.FullName -Force
			}
		}
		
		$items = Get-ChildItem $path -Recurse -Force  | Where {$_.FullName -notMatch $escExcDir -and $_.PSIsContainer -eq $true} |sort FullName -Descending
		foreach($item in $items){
			if($item -eq $null){
				continue	
			}
			$subItems = Get-ChildItem $item.FullName -Recurse -Force | Where { $_.PSIsContainer -eq $false}
			if($subItems -eq $null -or $subItems.Length -eq 0 ){
				Remove-Item $item.FullName -Force
			}
		}
	}
}

function Remove-AllExceptSvn($path, $childrenFilter){
	if (!(Test-Path $path))
	{
		md $path
	}
	
	if($childrenFilter -eq $null -or $childrenFilter.Length -eq 0 ){
		Do-Remove-AllExceptSvn $path
	}else{
		$childPaths = Get-ChildItem $path
		foreach($childPath in $childPaths){
			if($childrenFilter -contains $childPath.Name){
				Do-Remove-AllExceptSvn $childPath.FullName
			}
		}
	}
}



############### FUNCTIONS WORKING ON OTHER FUNCTIONS (takes server objects as input) ############

#
# Backs up a given server object
#
function Backup-Server($server){
	Backup-DeployedStuff $server.ServerName, $server.Drive, $server.Path, $server.BackupPath
}

function Prepare-Deploy([object] $server, [bool] $fetch_from_buildserver,  $childrenToPrepare){
	$are_children_tentative = $server.OnlyUpgradeExistingDirs
	
	$environment = $server.Environment
	$server_prepare_path = Get-PreparePath $server
	$customizations = $servertype_customizations | where {$_.ServerType -eq $server.ServerType}
	Remove-AllExceptSvn $server_prepare_path $childrenToPrepare
	
	$deploy_source = $server.BuildPath
	
	if(!$fetch_from_buildserver ){
		$deploy_source = $null
		$s = $all_servers | where {$_.Id -eq $server.DeployFromTestServer -and $_.ServerType -eq $server.ServerType }
		$deploy_source = Get-DeployUNCPath $s
	}
	if($deploy_source -eq $null){
		$Id = $server.Id
		$msg = [System.String]::Format("deployment source not initialized for server {0}.Fetch from build = {1}",$Id, $fetch_from_buildserver)
		Throw 
	}
	
	$filterChildren = $true
	if($childrenToPrepare -eq $null -or $childrenToPrepare.Length -eq 0){
		$filterChildren = $false
	}
	
	$prepare_target_base = $server_prepare_path
	$serverUNCPath = Get-DeployUNCPath $server 
	$thetype = $are_children_tentative.GetType()
	"are children tentative? $are_children_tentative ( $thetype )"	
	if($are_children_tentative){		
		foreach($sub_dir in Get-ChildItem $deploy_source){
			$child_name = $sub_dir.Name
			if($filterChildren){
				$skip = $true;
				foreach($child_filter_name in $childrenToPrepare){
					if($child_name -eq $child_filter_name){
						$skip = $false;
						break;
					}
				}
				if($skip){
					continue;
				}
			}
				
			
			$deploy_path =  $serverUNCPath + $sub_dir.Name
			$prepare_target = $prepare_target_base +$sub_dir+"\"
			$child_deploy_source = $deploy_source + $sub_dir.Name+"\"
			if(!(Test-Path $deploy_path -PathType Container)){
				#this child dir does not exist on server, so we won't update it automagically
				continue
			}
			$should_deploy = $true
			foreach($exclude in $customizations.ExcludeSubDirs){
				if($exclude -eq $child_name){
					$should_deploy = $false
				}
			}
			if(! $should_deploy){
				continue
			}
			
			trace "Creating prepare area $prepare_target"
			if(! $development_mode -and !(Test-Path $prepare_target -PathType Container)){
				md $prepare_target
			}		
			Prepare-Directory $child_deploy_source $deploy_path $prepare_target
		}
	}else{
		$prepare_target = $prepare_target_base 
		trace "Creating prepare area $prepare_target"
		if(! $development_mode -and !(Test-Path $prepare_target)){
			md $prepare_target
		}
		$exists_source = Test-Path $deploy_source
		
		Prepare-Directory $deploy_source $serverUNCPath $prepare_target		 
	}
}

function Deploy-PerServer($deploy_servers, $childrenFilter){
	foreach($deploy_server in $deploy_servers){
		if($deploy_server -eq $null){
			continue
		}
		$deploy_path = Get-DeployUNCPath $deploy_server
		$source_path = Get-PreparePath $deploy_server
		
		Log-Deploy " - $source_path => $deploy_path"
	
		$stopstart = $servertype_customizations | where {$_.ServerType -eq $deploy_server.ServerType}
		
	
		if($stopstart.StopTaskSqlAgent){
			foreach($task_server in $all_servers | Where{$_.Environment -eq $deploy_server.Environment -and $_.ServerType -eq $servertypes.Tasks} ){
				log "stopping SQLSERVERAGENT on $task_server"
				Stop-RemoteService $task_server.ServerName "SQLSERVERAGENT"
			}
			log "waiting for $wait_seconds_after_task_shutdown secs to be sure tasks are done before we begin"
			if(!$development_mode){
				Start-Sleep -Seconds $wait_seconds_after_task_shutdown  #sleep so tasks can finish before we pull down applications
			}
			log "wakey wakey, get on with it"
		}
		
		foreach($service_name in $stopstart.ServiceNames){
			Stop-RemoteService $deploy_server.ServerName $service_name
		}
		if($stopstart.ServiceNames.length -gt 0){
			log "waiting for $wait_seconds_after_services_shutdown secs to be sure tasks are done"
			if(!$development_mode){
				Start-Sleep -Seconds $wait_seconds_after_services_shutdown  #sleep so tasks can finish before we pull down applications
			}
			log "wakey wakey, get on with it"
		}
		if($deploy_server.OnlyUpgradeExistingDirs){
			Deploy-Subdirectories $source_path $deploy_path $childrenFilter	
		}
		else{
			Deploy-Directory $source_path $deploy_path
		}
	
	
		foreach($service_name in $stopstart.ServiceNames){
			Start-RemoteService  $deploy_server.ServerName $service_name
		}
	
		if($stopstart.StopTaskSqlAgent){
			foreach($task_server in $all_servers | Where{$_.Environment -eq $deploy_server.Environment -and $_.ServerType -eq $servertypes.Tasks} ){
				log "starting SQLSERVERAGENT on $task_server"
				Start-RemoteService $task_server.ServerName "SQLSERVERAGENT"
			}
		}
	
	}
}

function Deploy-ServerType($servertype, $environment, $shouldPromptForSubDirectories){
	
	$start_time = [DateTime]::Now
	
	
	Log-Deploy "Starting $servertype deployment to $environment"
	
	
	$servers = $all_servers | Where {$_.Environment -eq $environment -and $_.ServerType -eq $servertype}
	$deploy_servers = Prompt-WhichServers $servers
	$childrenFilter = $null
	if($shouldPromptForSubDirectories){
		$shouldPromptForSubDirectories = Prompt-WantsChildrenFilter $servertype
	}
	
	if($shouldPromptForSubDirectories){
		$childrenFilter = Prompt-ChildrenFilter $deploy_servers
	}
	Prompt-BeginDeploy $deploy_servers
	
	Deploy-PerServer $deploy_servers $childrenFilter
	
	$end_time = [DateTime]::Now
	$duration = $end_time - $start_time
	Write-Host "FINISHED!!! and it only took $duration"
	
	Log-Deploy "Finished $servertype deployment to $deploy_environment_name ($duration)"
}


function Deploy-All($environment){
	$start_time = [DateTime]::Now
	
	Log-Deploy "Are you sure you want to do a full deployment? (have you made a backup?)"
	Log-Deploy "Press Escape to exit, Enter to continue"
	$input = Read-Host
	
	Log-Deploy "Starting deployment of everything to $environment"
	
	$servers = $all_servers | Where {$_.Environment -eq $environment}
	
	#stop tasks
	foreach($task_server in $all_servers | Where{$_.Environment -eq $deploy_server.Environment -and $_.ServerType -eq $servertypes.Tasks} ){
		if($deploy_server -eq $null){
			continue
		}
		log "stopping SQLSERVERAGENT on $task_server"
		Stop-RemoteService $task_server.ServerName "SQLSERVERAGENT"
	}
	
	#stop running services on all machines
	foreach($deploy_server in $servers){
		if($deploy_server -eq $null){
			continue
		}
		$stopstart = $servertype_customizations | where {$_.ServerType -eq $deploy_server.ServerType}
		foreach($service_name in $stopstart.ServiceNames){
			Stop-RemoteService $deploy_server.ServerName $service_name
		}
	}
	
	#wait until stuff has quieted down
	log "waiting for $wait_seconds_after_task_shutdown secs to be sure tasks are done before we begin"
	if(!$development_mode){
		Start-Sleep -Seconds $wait_seconds_after_task_shutdown  #sleep so tasks can finish before we pull down applications
	}
	log "wakey wakey, get on with it"
		
	# Perform file deploy
	foreach($deploy_server in $servers){
		if($deploy_server -eq $null){
			continue
		}
		$deploy_path = Get-DeployUNCPath $deploy_server
		$source_path = Get-PreparePath $deploy_server
		
		Log-Deploy " - $source_path => $deploy_path"
			
		if($deploy_server.OnlyUpgradeExistingDirs){
			Deploy-Subdirectories $source_path $deploy_path $null
		}
		else{
			Deploy-Directory $source_path $deploy_path
		}
	
	
	}
	
	#stop running services on all machines
	foreach($deploy_server in $servers){
		if($deploy_server -eq $null){
			continue
		}
		$stopstart = $servertype_customizations | where {$_.ServerType -eq $deploy_server.ServerType}
			foreach($service_name in $stopstart.ServiceNames){
			Start-RemoteService  $deploy_server.ServerName $service_name
		}
	}


	
	foreach($task_server in $all_servers | Where{$_.Environment -eq $deploy_server.Environment -and $_.ServerType -eq $servertypes.Tasks} ){
		log "starting SQLSERVERAGENT on $task_server"
		Start-RemoteService $task_server.ServerName "SQLSERVERAGENT"
	}
	
	
	
	$end_time = [DateTime]::Now
	$duration = $end_time - $start_time
	Write-Host "FINISHED!!! and it only took $duration"
	
	Log-Deploy "Finished $servertype deployment to $deploy_environment_name ($duration)"

}




function Prepare-ServerType($servertype, $environment, [bool]$shouldPromptForSubDirectories = $false){
	Log-Deploy "Starting $servertype Deploy Preparations to $environment"
	$start_time = [DateTime]::Now

	$servers =  $all_servers | where{$_.ServerType -eq $servertype -and $_.Environment -eq $environment}
	
	if($shouldPromptForSubDirectories){
		$shouldPromptForSubDirectories = Prompt-WantsChildrenFilter $servertype
	}
	

	if($shouldPromptForSubDirectories){
		$childrenFilter = Prompt-ChildrenFilter $servers
	}
	

	foreach($server in $servers){	
		if($server -ne $null){
			Prepare-Deploy $server $prepare_from_buildserver $childrenFilter
		}
	}	
	
	$end_time = [DateTime]::Now
	$duration = $end_time - $start_time
	Write-Host "FINISHED!!! and it only took $duration"
	Log-Deploy "Finished $servertype Preparations ($deploy_environment_name) - ($duration)"
}


3)
The last piece of the puzzle is the wizard guiding the user through the process. really simple .bat file stuff:


$script_loaded_machines = $false
$script_loaded_common = $false
$script_loaded_backup = $false
$script_loaded_externaldeploy = $false
$script_loaded_externalprepare = $false
$script_loaded_failsafe = $false


if($script_loaded_backup -eq $false){
	$include = Resolve-Path 'D:\deployment\Make_Backups.ps1'  #contains the wizard for doing backups, nothing much to see here
	. $include
}


$ui = (Get-Host).UI.RawUI
$ui.ForegroundColor = "black"

function Do-Backup(){
	$ui.ForegroundColor = "darkblue"
	Create-Backups -environment $environment
}

function Do-Restore(){
	$ui.ForegroundColor = "darkcyan"
	Restore-Backups -environment $environment
}

function Do-CleanBackups(){
	$ui.ForegroundColor = "darkcyan"
	Delete-Backups -environment $environment
}


function Prepare-Everything($environment){
	""
	"About to prepare for a full deploy"
	"Press escape to abort, Enter to continue"
	""
	$input = Read-Host
	$all_start_time = [DateTime]::Now

	foreach($typeholder in $servertype_customizations){
		$servertype = $typeholder.ServerType
		Log "Preparing servertype $servertype for $environment"
		Prepare-ServerType -environment $environment -servertype $servertype $false
	}
	$all_end_time = [DateTime]::Now
	$duration = $all_end_time - $all_start_time
	Write-Host "FINISHED!!! and it only took $duration"
}

function Do-Prepare(){
	$prepare_from_buildserver = $false
	if($environment -eq "devtest"){
		$prepare_from_buildserver = $true
	}
	if($environment -eq "test"){
		Log ""
		Log "Is this a hotfix? (y / n) (y==fetch from hotfix server)"
		$input = Read-Host
		if($input -eq "n" -or $input -eq "N"){
			Log "Not hotfix: Fetch files from previous env"
		}
		elseif($input -eq "y" -or $input -eq "Y")
		{
			Log "Hotfix: Fetches from hotfix build server"
			$prepare_from_buildserver = $true
		}
		else{
			Exit-Deploy
		}
		
	}

	$ui.ForegroundColor = "darkgreen"
	""
	""
	"What do you want to prepare?"
	"1 - Services"
	"2 - Web"
	"3 - Tasks"
	"100 - Everyting in $environment"
	
	"(default = quit the script)"
	$input = Read-Host
	""
	"(you chose $input)"
	switch($input){
		"1"{ Prepare-ServerType $servertypes.InternalServices $environment $true}
		"2"{ Prepare-ServerType $servertypes.Web $environment $false}
		"3"{ Prepare-ServerType $servertypes.Tasks $environment $true}
		"100"{Prepare-Everything $environment}
		default{Exit-Deploy}
	}

}



function Do-Deploy(){
	$ui.ForegroundColor = "DarkRed"

	""
	"What do you want to deploy?"
	"1 - Web Services"
	"2 - Web Site "
	"3 - Database Tasks"
	"100 - Everyting"
	"(default = quit the script)"
	""
	$input = Read-Host
	switch($input){
		"1" { Deploy-ServerType $servertypes.InternalServices $environment $true}
		"2" { Deploy-ServerType $servertypes.Web $environment $false}
		"3" { Deploy-ServerType $servertypes.Tasks $environment $true}
		"100" { Deploy-All -environment $environment }
		default{Exit-Deploy }
	}

}


Log-Deploy "----------------STARTed deploy script"
$ui.BackgroundColor = "White"
$ui.ForegroundColor = "Black"

"Welcome to the deployment script"
""
""
"Before you run this script to deploy to devtest you need to have:"
" - run c:\ProjectWork\ThisProject\buildsolutions.bat locally"
" - checked in inn c:\ProjectWork\ThisProject\services\SharedAssemblies to subversion"
" - built all the projects in CruiseControl.net ( http://buildsvc ) - (zzz_FullBuild_ClickToBuildEverything) "
""
"[Dev mode=$development_mode] (when in dev mode, no changes are made to anything)"
""
$do_quit = $false

""
""
"Which environment do you want to operate on?"
""
"0 - devtest"
"1 - test/hotfix"
"99 - prod"
""
$environment = "devtest"
$input = Read-Host
switch($input){
	"0"{
		$environment = "devtest" 		
		}
	"1"{	
		$environment = "test"
		}
	"99"{ 
		$environment = "prod"
		$ui.BackgroundColor = "Red"
		$ui.ForegroundColor = "White"
	""
	"ARE YOU REALLY SURE YOU WANT TO OPERATE ON THE PRODUCTION ENVIRONMENT? (press escape to abort)"
	""
		$ui.ForegroundColor = "Black"
		$ui.BackgroundColor = "White"
		$input = Read-Host
	}
	default{Exit-Deploy}
}

while(! $do_quit){
	$ui.ForegroundColor = "black"
	"What do you want to do?"
	"1 - Back up the code and config currently in $environment "
	"2 - Prepare a deploy to staging area on buildserver ( $environment )"
	"3 - deploy to $environment "
	"8 - Delete an old backup from $environment"
	"9 - Restore an old backup into $environment"
	"exit - avslutte"
	"(default = avslutte)"
	$input = Read-Host
	switch($input){
		"1"{ Do-Backup }
		"2"{ Do-Prepare}
		"3"{ Do-Deploy}
		"8"{ Do-CleanBackups}
		"9"{ Do-Restore}
		default{Exit-Deploy}
	}
}

I’ve covered a lot of ground here, it took me bloody ages to write it up (sorry to the commenter who needed it some weeks ago).

This strategy for build and deployment has a few weaknesses i know of (and probably a lot I don’t know), most notably:
– we use commonly compiled dlls which are checked into svn. ideally we should be referencing them directly from the buildserver so we didn’t need a two-phased build
– Database changes are completely decoupled from the code, so as the code is moved from environment to environment, the database has to be kept manually (though through tools) updated.
– configuration changes are bloody hard to get from environment to environment in .Net, you either have to check in web.config in N versions (one for each machine in each environment) or you have to keep them all manually up to date.

All in all this strategy has worked out very well for us these last 6 months, and errors stemming from deployment issues are now almost nonexistent. (we still get the odd DB-sync glitch or configuration sync glitch) but now the script just does its thing for 15 minutes and then you’re set.

I hope this has been of use to someone.
good night

Blog at WordPress.com.