Getting the tooling to do TDD with JavaScript code has been something that I’ve been struggling with for the last year. There have been lots of tools that can handle one aspect or another of JavaScript testing, but nothing was a complete solution. I thought our needs would be fairly common since we’re using pretty standard tool sets.

I wanted:

  • Ability to run JS tests automatically or with a simple key command within Visual Studio (ala Resharper)
  • The ability to use wildcards for our source and test files. Listing each file out is too painful on a large project.
  • TeamCity integration just like we have for our C# unit tests
  • Code coverage generation, preferably that could also hook into TeamCity

Some Nice To Haves:

  • Allow devs to generate code coverage locally
  • A configuration that could be checked into our source control

I’m finally happy with the current setup we’re using now. We’ve setup Karma which fits our needs and hits just about every point we wanted.

Our Setup

Here’s a bit more detail on what we’re using and testing against.

Our JS code is mostly using Knockout.js. We try to keep jQuery use to a minimum, and keep it out of our ViewModels completely, with the exception of $.ajax. Knockout makes it very easy to test our client side logic because there is no reliance on the DOM.

On the testing side we use QUnit mainly because it is very close to NUnit which is our testing framework on the C# side of things. We’ve recently introduced Sinon.js for our mocking/spies/stubbing framework. We had been using one I wrote, but Sinon is just so much better.

A Brief History of Testing

When we started with JavaScript testing we just used a web page setup from the QUnit tutorials. That was fine for local testing, but didn’t work with TeamCity. It didn’t take long to get PhantomJS setup and having our tests run in TeamCity that way.

To get code coverage working, we found the YUI-Coverage tool. It’s a Java app that instruments your code then parses the output created when the tests run. It worked but was a pain to maintain. Since the files were modified when they were instrumented, we had to make sure we saved off a copy of the originals otherwise we’d see coverage percentages like 56000%. It has no issue instrumenting an already instrumented file for bonus coverage fun.

We were able to get this setup working, but it wasn’t quite where we wanted it to be.

Enter Angular.js & Karma

I had seen the limits of Knockout when it came to very complicated Single Page Apps (SPA) that we had worked on. Knockout worked, but the code was not as clean and clear as I would have liked it to be. I started reading about Angular.js and it’s approach as a client-side framework. I came across the test framework that the Angular team had created. At the time it had a rather unfortunate name (which has been corrected), but it appeared to be everything we were looking for.

Karma is a command line tool that runs in Node.js. It uses more modern approaches to testing by being just a modular test runner. It supports all the major testing libraries, including QUnit. It also has a code coverage module which runs Istanbul, also by the YUI team. Istanbul uses another library calls Esprima which allows for the instrumentation to be done in memory saving us the step of saving off the originals.

How it works is actually really cool. You configure Karma with your source and test files and tell it how you want the results reported back to you. There are a variety of reporters; we just use the progress one. You also tell Karma which browsers you would like your tests run in. It defaults to Chrome, but supports the major browsers and PhantomJS. You can configure as many as you like and have your tests run on each concurrently.

Karma hosts the web server itself and uses websockets to establish a connection to the testing browser. When you update your files, it re-sends them to the browsers and re-runs your tests. This provides instant and automatic feedback. Exactly what we want for doing TDD.

As of 0.10, Karma is now plugin based. The team did a good job of breaking out the existing functionality into Node modules, and the community has filled the gaps. The TeamCity reporter works great, so we’re still covered there.

Karma on Windows

For the most part, getting Karma to work on Windows was painless. We’re using Node 0.10.15 and all of the Node modules that are used compile just fine. We did run into an issue with how the location of the Chrome and Firefox executables is determined, but I have already submitted pull requests to correct that (Chrome Reporter, Firefox Reporter).

We have two Karma config files setup. Once for local development that runs after files are saved, and another for TeamCity and code coverage enabled. This allows us to see the coverage without having to check in, which is actually pretty nice.

My Contribution to the Karma Community

As I was learning how to get Karma going I didn’t like how I had to keep the console window visible to know if my tests failed. I wanted to hear that my tests failed.

Introducing Karma-Beep-Reporter. It’s a simple reporter that outputs the ASCII character 0x07 (Bell) when you have a failed test or your tests fail to run altogether. It’s meant to run along side one of the other reporters since it only beeps. I’ve only tested it on Windows so far, but it works great. I welcome comments and feedback!