My never ending quest for finding a build script solution that satisfies my requirements for facilitating builds within .NET might just have come to an end. I’ve gone from Albacore and Rake, to Phantom with Boo syntax, to psake and now finally GulpJS. I’ve glanced at Nant and others and none of the technologies has really “failed“, but there has been drawbacks. Now, I put my faith in GulpJS, which works great for e.g. compiling, minifying and bundling Less resources, but as it turns out, it also works great for managing builds within .NET.

Disclaimer & notes

No guru warning, hence feel free to comment and help improve on the solution described. Also. Please note, line feeds have been added for readability.

Requirements

My requirements are quite simple:

1. Should be able to be triggered locally as well as in a CI server (TeamCity)
2. Should be able to do: simple clean, copy, MS-Builds, unit-testing (XUnit) and NuGet interaction

Given this, it should be quite clear that pretty much any building framework/technology should be able to satisfy my demands.

Why don’t you just use build steps in the CI-server?

First, lets talk about the need for executing my builds locally. Basically, I want the same behavior on my local dev machine as on the CI-server. Why? Easier to manage as it’s easier to debug and author. And I really, really, really don’t like setting up 395 build steps with a plethora of configuration options in a build server like TeamCity. For me, configuring one simple step that checks out the source code (and the build script) and then one step that executes my build script, is enough. This gives me flexibility and portability; potentially allowing me to switch to other environments, which for a .NET developer becomes more important for every day, since the upcoming release of .NET will support other environments than Windows (Running ASP.NET 5 applications in Linux Containers with Docker, Running ASP.NET 5 on Linux).

Keep dependencies down

Preferably, it should have no (or few) extra dependencies in a Windows environment. This requirement would scratch most, but not psake, which bases it self upon PowerShell. Being written in PowerShell means that you can pretty much do anything with it. How-ever, I’m really not a PowerShell guy, and I have very little interest in learning it. Sure I get around with Google and some years of programming, but PowerShell is nothing I want to master. Also, with .NET moving onto other platforms, it would be nice if you could get it to run on those platforms as well.

Given that I don’t really like PowerShell, and multi environment wish, I think Node, and GulpJS is a good candidate. It also makes me learn more about Node and JavaScript, which feels more relevant that learning PowerShell, which feels ops focused.

Solution

Node, JavaScript and GulpJS. Gulp has a nice task and stream based flow, where you can define task that works on streams that you can chain | pipe to get many “steps” within one task, all working on the same resource. Hence you could start out with providing a glob expression to target specific resources and then pipe the matching resources into one or more steps.

You can of course also define dependencies between tasks, stating "task2" can only be executed if "task1"has executed.

The GitHub repo and the layout

The repo for this sample is my simple guard clause validation project “Ensure.That” which is distributed via NuGet. Hence the end result of this build script, will be two NuGet packages. One for the compiled distribution and one for the source distribution.

The layout of the repo is:

deploy
  |-tools
      |-packages.config
  |-Ensure.That.Source.nuspec
  |-Ensure.That.nuspec
  |-gulpfile.json
  |-package.json
src
  |-projects
      |-EnsureThat
  |-tests
      |-EnsureThat.UnitTests
  |-Ensure.That.sln
  |-SharedAssemblyInfo.cs

What happens upon execution?

When the build script in the file gulpfile.json is executed for CI (continuous-integration) purposes, a clean build folder is created in the deployfolder. The solution is compiled with a Release profile and the resulting dll and xml (for documentation) is (by the script) copied to the build folder. Then the unit-tests are executed, and finally, two NuGet packages are created in the build folder.

I will not checkin my node modules but have decided to keep a package.json (definition) file in my deploy folder, keeping track of all my dependencies (used modules) so that that I easily can restore them before executing my gulpfile.json.

Getting started with Gulp

Gulp has many, many, many plugins, distributed via npm (as Gulp itself is). There are packages for compiling less, minifying- and bundling css, but also packages for .NET developers, e.g. for invoking MSBuild, NuGet, NUnit or e.g. for patching AssemblyInfo.cs files. You an of course also execute shell commands, which I find simple enough. We will have a look at some of these packages below, but first, lets look at how we can get going. Please, note, I will not go into any details about Node, Node package manager etc. For more detailed info on how to get started, you could consolidate the Gulp documentation.

Execute a Gulp task

Once Gulp has been installed globally: npm i gulp -g; as well as locally: npm i gulp --save; You create the file containing your Gulp tasks: gulpfile.json. Inside of it, you define at least one task. The default task is named default and will be executed if no task name is provided at the command line.

var gulp = require('gulp');

gulp.task('default', function (cb) {
  ...
});

Task-to-task dependencies

When using the stream and pipe construct or when returning a promise from your task you do not need to indicate “task completion”. Otherwise you have to manually indicate that a task is complete by invoking the callback function cb. Lets define a few task using callback. One flow-control task, "default", and then two simple task simulating work do be done.

gulp.task('default', ['task1', 'task2']);

gulp.task('task1', function(cb){
  console.log('task1 doing work');
  cb();
});

gulp.task('task2', function(cb){
  console.log('task2 doing work');
  cb();
});

Remember, node is by nature async, hence, running the "default" task could potentially get "task2" to complete before "task1":

//task1 and task2 is executed asynchronously
gulp.task('default', ['task1', 'task2']);

Sequenced task depdendecies

You could state that "task2" should be dependent on "task1" and then "task1" would be executed before "task2" is.

gulp.task('task2', ['task1'], function(cb){
  ...
});

But what if, you would like to execute the work in "task2" without triggering any dependencies? For me, with the current version of GulpJS, I went with a dependency on the module run-sequence

var gulp = require('gulp'),
    sequence = require('run-sequence');
      
gulp.task('a', function(cb){
    //Add timeout to get it to execute longer than B
    var t = setTimeout(function () {
      clearTimeout(t);
      console.log('a');
      cb();
    }, 1000);
});

gulp.task('b', function(cb){
    console.log('b');
    cb();
});

gulp.task('c', function(cb){
    sequence('a', 'b', cb);
});

Executing "task c" would lead to the following output.

cmd> gulp c
=== output ===>
[11:06:34] Starting 'c'...
[11:06:34] Starting 'a'...
a
[11:06:35] Finished 'a' after 995 ms
[11:06:35] Starting 'b'...
b
[11:06:35] Finished 'b' after 223 μs
[11:06:35] Finished 'c' after 1 s

Using task dependencies it would look like this:

gulp.task('a', function(cb){
    ...
});

gulp.task('b', ['a'], function(cb){
    ...
});

gulp.task('c', ['b']);

Why “run-sequence”?

I have two “flow controlling tasks“: default and ci.
Other than that I have “simple worker tasks“: init-tools, nuget-restore, clean, assemblyinfo, build, copy, unit-test and nuget-pack. (View complete version in the GitHub repo)

// *** flow controlling tasks ***
//for local execution on developer machine
gulp.task('default', ['clean', 'assemblyinfo', 'build', 'copy', 'unit-test']);

//for execution on build server
gulp.task('ci', ['init-tools', 'nuget-restore', 'default', 'nuget-pack']);

// *** worker tasks ***
gulp.task('init-tools', function () { ... });
gulp.task('nuget-restore', function () { ... });
gulp.task('clean', function() { ... };
gulp.task('assemblyinfo', function() { ... });
gulp.task('build', ['clean', 'nuget-restore', 'assemblyinfo'], function() { ... });
gulp.task('copy', ['build'], function() { ... });
gulp.task('unit-test', ['init-tools', 'build'], function () { ... });
gulp.task('nuget-pack', ['copy', 'unit-test'], function () { ... });

One requirement I have, is to be able to execute each of the simple task isolated, without triggering any dependencies, which would not be feasible using the “task dependencies construct” above. But using "run-sequence" I could remove all task dependencies in my worker tasks and instead define my flow controlling tasks like this instead:

gulp.task('default', function (callback) {
  sequence(
    //run in parallel
    ['clean', 'assemblyinfo'],
    
    //run sequenced
    'build', 'copy', 'unit-test',
    callback
  );
});

gulp.task('ci', function (cb) {
  sequence(
    //run in parallel
    ['init-tools', 'clean', 'assemblyinfo', 'nuget-restore'],
    
    //run sequenced
    'build', 'copy', 'unit-test', 'nuget-pack',
    callback
  );
});

// *** worker tasks ***
gulp.task('init-tools', function () { ... });
gulp.task('nuget-restore', function () { ... });
gulp.task('clean', function() { ... };
gulp.task('assemblyinfo', function() { ... });
gulp.task('build', function() { ... });
gulp.task('copy', function() { ... });
gulp.task('unit-test', function () { ... });
gulp.task('nuget-pack', function () { ... });

The relevant tasks & modules for my build

Lets look at them one by one, the final script can be viewed on GitHub. Remember that the execution of the script is in: the[repo]deploy folder. So, e.g. a path like ./tools would refer to [repo]deploytools.

Configuration of the build script

I allow injection of two command line parameters: buildrevision and buildprofile. To parse the command line arguments, I use the package yargs. Locally, I will not pass any values, but on the build server I will pass the sequentially incremented build counter (managed by TeamCity): buildcounter => buildrevision.

var config = {
  slnname: 'Ensure.That',
  src: './../src/',
  build: {
    outdir: './build/',
    version: '2.0.0',
    revision: argv.buildrevision || '*',
    profile: argv.buildprofile || 'Release'
  }
};

Why not inject the whole build version from TeamCity?

Because I want as little dependencies as possible on my build server. I want the version bump to show in my commits in my GitHub repository. So when doing a release branch that release branch will have a commit bumping the value in the build script.

task: init-tools

I don’t want to have xUnit registered globally via an environment path, but instead be a tool part of the project. This since I use different versions of xUnit in my various projects and I want it to be easy to see what external resources my projects and builds depend upon. For this, I have a tools folder. In which I have a NuGet packages.config file, with the xUnit.Runners package in. All I need to do, is to invoke a nuget restore on that packages.config file.

There’s a gulp-nuget plugin, but I went with a simple shell invoke instead. For shell execution, I’ve decided to use gulp-shell.

var gulp = require('gulp'),
    shell = require('gulp-shell');

gulp.task('init-tools', shell.task([
  'nuget restore ./tools/packages.config -o ./tools/']
));

I’ve decided to have NuGet installed once on my machine, and not part of the build. This is easy to do using the NuGet CLI Chocolatey package.

task: nuget-restore

Before I compile my solution using MSBuild, I want to ensure that all NuGet packages for the solution is restored. Again, as with the init-tools task, a simple shell invoke of my nuget.exe is enough.

var gulp = require('gulp'),
    shell = require('gulp-shell');

gulp.task('nuget-restore', function () {
  return gulp.src(config.src + '*.sln', { read: false })
    .pipe(shell('nuget restore '));
});

task: clean

You will see some code using depracated gulp plugins like: gulp-clean, gulp-rimraf ; but by recommendation, you should instead use the gulp friendly del package. This will assist me in removing the deploybuild folder.

var gulp = require('gulp'),
    del = require('del');

gulp.task('clean', function(cb) {
  del([config.build.outdir + '**'], cb);
});

task: assemblyinfo

I have one shared assembly info file: SharedAssemblyInfo.cs that is included as link in my projects. In that I store all shared project values. But for each build I need to patch it and set a new version. For this I use the gulp-dotnet-assembly-info package.

var gulp = require('gulp'),
    assemblyInfo = require('gulp-dotnet-assembly-info');

gulp.task('assemblyinfo', function() {
  return gulp
    .src(config.src + 'SharedAssemblyInfo.cs')
    .pipe(assemblyInfo({
      version: config.build.version + '.' + config.build.revision,
      fileVersion: config.build.version,
    }))
    .pipe(gulp.dest(config.src));
});

task: build

Time for the meat of the build script, the actual build itself. I just want a simple invoke of MSBuild, against my solution file. For this I could have used a shell invoke, but decided to go with the gulp-msbuild package.

var gulp = require('gulp'),
    msbuild = require('gulp-msbuild');

gulp.task('build', function() {
  return gulp.src(config.src + '*.sln')
    .pipe(msbuild({
      toolsVersion: 12.0,
      configuration: config.build.profile,
      targets: ['Clean', 'Build'],
      errorOnFail: true,
      stdout: true,
      verbosity: 'minimal'
    }));
});

If I wanted a “silent” build, I could have left out stdout and verbosity, but I want to see what is build in the build server log, hence why I include it. I also want an error to fail the build process, hence errorOnFail:true.

task: copy

I could have left out the copy part and instead reference e.g. the dlls in the bin folder instead. But I think it’s easier to work with the resources like this. Especially when “debugging/understanding” my builds. , To get the files, instead of getting the full nested structure, to be copied into the root of my deploybuild folder, I’ve chosen to use the gulp-flatten package.

var gulp = require('gulp'),
    flatten = require('gulp-flatten');
    
gulp.task('copy', function() {
  return gulp.src(config.src
                + 'projects/**/bin/'
                + config.build.profile + '/*.{dll,XML}')
    .pipe(flatten())
    .pipe(gulp.dest(config.build.outdir));
});

task:unit-test

Simple shell invoke of my xUnit runner that was restored (downloaded from NuGet) in my init-tools task.

var gulp = require('gulp'),
    shell = require('gulp-shell');

gulp.task('unit-test', function () {
  return gulp.src(config.src + 'tests/**/bin/'
                + config.build.profile + '/*.UnitTests.dll')
    .pipe(shell(
           'xunit.console.clr4.exe <%= file.path %>  /silent /noshadow',
           { cwd: './tools/xunit.runners.1.9.2/tools/' })
    );
});

task:nuget-pack

Final step. Shell invoke nuget.exe to pack my tow NuGet artifacts. One for the dll distribution and one for the source distribution.

gulp.task('nuget-pack', shell.task([
  'nuget pack ./' + config.slnname + '.nuspec
     -version ' + config.build.version + '
     -basepath ' + config.build.outdir + '
     -o ' + config.build.outdir,

  'nuget pack ./' + config.slnname + '.Source.nuspec
     -version ' + config.build.version + '
     -basepath ' + config.src + '
     -o ' + config.build.outdir,]
));

Execute it

Easy. Configure your build server to have one build step restoring the NPM packages:

npm install

Then one build step that executes gulp, and the ci task. Also pass a buildrevision if you want.

gulp ci --buildrevision=%build.counter%

Summary

That was all the steps necessary for managing this project. Works for me. And makes me take yet another step closer to the Node community and gets me out of the PowerShell swamp. Feel free to help me improve the solution. The full script can be found here.

//Daniel