Showing posts with label unit test. Show all posts

Tuesday, July 2, 2013

baseapp: Continuous Integration

baseapp provides all the boilerplate to get your JavaScript web application started off right, this Part 7.

(or binge read them all on the baseapp wiki!)

baseapp achieves Continuous Integration in 3 (yes 3) ways:

They are all complimentary and do similar things so take your pick!

grunt watch

The grunt watch plugin monitors our files and runs grunt tasks as those files are modified - to the configuration!

    watch: {
        // If the main app changes (don't have any specific tests for this file :( YET! )
        mainApp: {
            files: [ 'app.js' ]
            , tasks: ['jshint', 'webdriver' ]
        } 
        // If any server-side JS changes
        , serverSide: {
            files: [ 'routes/**/*.js' ]
            , tasks: ['jshint', 'jasmine_node_coverage', 'webdriver' ]
        }
        // If any server-side JS TEST changes
        , serverSideTests: {
            files: [ 'spec/server/**/*.js' ]
            , tasks: ['jshint', 'jasmine_node_coverage' ]
        }
        // If any client-side JS changes
        , clientSide: {
            files: [ 'public/javascripts/**/*.js' ]
            , tasks: ['jshint', 'jasmine', 'webdriver' ]
        }
        // If any client-side JS TEST changes
        , clientSideTests: {
            files: [ 'spec/client/**/*.js' ]
            , tasks: ['jshint', 'jasmine' ]
        }
        // If any integration/webdriver JS TEST changes
        , webDriverTests: {
            files: [ 'spec/webdriver/**/*.js' ]
            , tasks: ['jshint', 'webdriver' ]
        }
    }

The deal here is each file can only be in ONE stanza, if a file is represented in more than one grunt-watch stanza only the last one will 'win'. SO we slice and dice our files to ensure they only show up in one place. So the game is determine which grunt tasks should be run if a given file changes.

Starting with 'app.js' - if it changes changes run the 'jshint' and 'webdriver' tasks because those are the only two tasks that app.js could effect.

Similarly for editing a server-side JS file, if one of those changes run 'jshint', 'jasmine_node_coverage', and 'webdriver' because all of those target are potentially effected by a change to one of those files.

If any server-side unit test file changes we only run 'jshint' and 'jasmine_node_coverage' - we do NOT need to run any webdriver tests because change a server-side unit test file does not potentially effect those.

The same logic applies to changed client-side JavaScript files vs. client-side test files.

Finally if any webdriver tests change then we run the 'jshint' and 'webdriver' tasks because those are the only tasks potentially effected by a change to any of those files.

So in a terminal kick it all off by executing:

% grunt watch

Now in another terminal edit away and the 'grunt watch' terminal will run jshint and tests after you save changed files.

Karma

Karma functions similarly to 'grunt watch' - we tell it which files to watch, and if there is a change it should kick off some tests. Karma however ONLY runs out client-side jasmine unit tests. What is snazzy about it is support for multiple simultaneous browsers and built-in coverage generation.

All of its configuration is in karma.conf.js (yes this is a JavaScript file). Since it is going to run jasmine it needs the same information as our 'grunt jasmine' task: namely where all of our client-side JavaScript files are and what extra JavaScript to load into the browser to run our tests. That configuration is here:

// list of files / patterns to load in the browser
files = [
  JASMINE,
  JASMINE_ADAPTER,
  'public/vendor/jquery-2.0.2.min.js',
  'public/vendor/jasmine-jquery.js',
  'public/vendor/dust-core-1.2.3.min.js',
  'public/javascripts/*.js',
  'spec/client/*.js'
];

'JASMINE' and 'JASMINE_ADAPTER' are karma built-ins for running jasmine tests - convenient! This section adds in coverage support:

preprocessors = {
    'public/javascripts/*.js': 'coverage'
};

This tells Karma to generate coverage information for all files in 'public/javascripts/*js' - where all of our client-side JavaScript resides.

coverageReporter = {
    type : 'lcov',
    dir : 'public/coverage/client'
};

Here is where we tell Karma where to put the coverage output. When the tests are all done running Karma will generate a new code coverage report.

browsers = ['ChromeCanary', 'PhantomJS', 'Safari', 'Firefox'];

Here is our array of browsers we want Karma to run - every time one of our watched files change Karma will execute all of the unit tests in each of those browsers. When Karma starts up it spawns off each of those and they sit around and wait to run tests.

Finally:

singleRun = false;

Tells Karma to keep running in the background, you can run it in single shot mode if being run in a QA environment.

So to start the whole thing up:

% node_modules/.bin/karma start

Karma will spawn off all the browsers you requested, you can now minimize all of those windows, Karma will print all relevant output to the terminal window. As you edit client-side JavaScript or the tests Karma will automatically re-run all the tests.

travis-ci

Travis-CI executes the 'script' property from our .travis.yml file each time we push to github, if that is not present (which it is not in our config), it runs 'npm test' by default for NodeJS jobs (which for us just turns around and runs 'grunt test'). Our travis-ci configuration is in the .travis.yml file. Here it is:

language: node_js
node_js:
  - "0.8"
  - "0.10"
services: redis-server
before_install:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"
before_script:
  - "java -jar ./node_modules/webdriverjs/bin/selenium-server-standalone-2.31.0.jar &"
  - "sleep 10"

This just says this application is a NodeJS application and we want to test it against version .8 and .10 of node. You need to link your github account with travis-ci so it can watch your github commits and act on them accordingly.

Set up information for Travis CI is here. Basically you just need to create a Travis CI account and activate the GitHub service hook. Then visit your Travis CI profile and enable Travis CI for your repository. With our .travis.yml file in place, the next commit to our github repo will trigger Travis CI.

Travis CI conveniently has both redis and phantomjs (and firefox!) preinstalled for webdriver tests. Redis however is NOT automatically started so we need to tell Travis to start redis-server for us. We also have to start the selenium server and then sleep for 10 seconds to ensure it is up and ready to go. In case we want to use 'firefox' for our Selenium tests we fire up the virtual X framebuffer and set the DISPLAY environment variable accordingly.

You may notice that our travis tests will generate coverage information and total it all up and run plato - none of which we actually use as the result of the travis tests. Oh well.

If all goes well Travis will blow through all of our tests successfully. On a Travis result state change, like from success to failure or vice versa you will get an email with a link to the log so you can debug if one of your tests has failed.

You can see baseapp's Travis CI dashboard here. Yes it took several tries to get it right!! But lucky for you all of the 'hard' work has been done for you.

Finally we add the 'travis build status' badge to the top of our README.md file to show all comers we use travis-ci to test our project and show build status:

[![build status](https://secure.travis-ci.org/zzo/baseapp.png)](http://travis-ci.org/zzo/baseapp)

Don't freak out at the markdown syntax, we are just enclosing an image (baseapp.png) with a link to the baseapp travis-ci page.

Wednesday, June 26, 2013

baseapp: Server-Side Unit Tests

baseapp provides all the boilerplate to get your JavaScript web application started off right, this Part 3.

(or binge read them all on the baseapp wiki!)

baseapp handles server-side unit tests using jasmine-node and istanbul for code coverage. The grunt task to run the tests is 'grunt jasmine_node_coverage'. The tests always run with code coverage enabled.

Unlike the client-side unit tests I do not use any third party grunt plugin to execute this grunt task. Let's go to the configuration!

    jasmine_node_coverage: {
        options: {
            coverDir: 'public/coverage/server'
            , specDir: 'spec/server'
            , junitDir: './build/reports/jasmine_node/'
        }
    }

Not a lot here - 'options.coverDir' is the directory where coverage information will be dropped, 'options.specDir' is where all of the server-side unit tests live, and finally 'options.junitDir' is where the JUnit XML test output (one file per suite) is placed.

Jasmine-node works almost identically to client-side jasmine. The biggest (only?) difference is how asynchronous tests are handled. While both jasmines match 'runs()' and 'waitsFor()' functions to handle asynchronous tests, jasmine-node also provides 'jasmine.asyncSpecDone();' matched with 'jasmine.asyncSpecWait();' to handle async tests easier.

Let's take a look at the userSpec.js suite. This file unit tests the authentication code in routes/user.js. Unlike the client-side tests these tests are not run in a browser and objects under test are pulled in via the normal 'require' method.

In this case I expect the redis database to be up, you can use jasmine spies to mock it all out, but instead I select redis database '15' to not interfere with production data (redis has 16 databases named 0-15, by default you get database 0 unless you change it as I do in the 'select' call).

Also I have an 'afterEach' method that completely blows out the contents of the redis database after each test so I'm guaranteed each test gets a fresh database (redis.flushdb()).

The async tests themsevles are very straightforward, just call methods and verify resopnses, easy peasy.

Application Architecture

A quick note about ease of testing, you will note in user.js, I was clear about separating HTTP from the functionality of the module to make testing easier. I was very careful to not mix dealing with HTTP-protocol based values with authentication routines. I specifically did not want to mix up HTTP with authentication to make testing easier. This way I do not have to mock/deal with HTTP when testing the authentication routines. Be sure to keep separation of concerns in mind while writing your code. Writing tests first will help keep your code clean.

Code Coverage

Istanbul has great integration with jasmine-node, looking further down in the Gruntfile you can see the implementation of the jasmine_node_coverage task. Simply running istanbul with jasmine-node will automatically generate code coverage and the coverage output is written to 'public/coverage/server/lcov-report/index.html' in fancy HTML.

Tuesday, June 25, 2013

baseapp: Client-Side Unit Tests

baseapp provides all the boilerplate to get your JavaScript web application started off right, this Part 2.

(or binge read them all on the baseapp wiki!)

baseapp handles clide-side unit tests using jasmine and istanbul for code coverage. The grunt task to run the tests is 'grunt jasmine'. They always run with code coverage enabled.

Test Configuration

Let's look at the configuration in the Gruntfile:

    jasmine : {
        test: {
            src : 'public/javascripts/**/*.js',
            options : {
                specs : 'spec/client/**/*.js'
                , keepRunner: true  // great for debugging tests
                , vendor: [ 
                      'public/vendor/jquery-2.0.2.min.js'
                    , 'public/vendor/jasmine-jquery.js' 
                    , 'public/vendor/dust-core-1.2.3.min.js' 
                    , 'vendor/bootstrap/js/bootstrap.min.js'
                ]
                , junit: {
                    path: "./build/reports/jasmine/"
                    , consolidate: true
                }
                , template: require('grunt-template-jasmine-istanbul')
                , templateOptions: {
                    coverage: 'public/coverage/client/coverage.json'
                    , report:   'public/coverage/client'
                }
            }
        }
    },

Here is what is happening - the grunt-contrib-jasmine plugin collects all of your client side JavaScript (the 'src' property), all of your test files ('opitons.specs') and any other JavaScript files we need to execute our tests ('options.vendor') and creates a single HTML file 'SpecRunner.html' which is loaded into a browser (phantomjs). The grunt jasmine plugin automatically adds the jasmine client-side libraries to actually run the tests too. SO when SpecRunner.html is loaded into a browser (phantomjs) all of your tests are run.

The output of these tests (in JUnit XML format) is dumped into the options.junit.path directory - one XML file per test suite.

Finally the grunt-template-jasmine-istanbul package is leveraged to generate code coverage information, the HTML output of which is dumped into the 'options.templateOptions.report' directory. We also persist the 'coverage.json' file so it can be aggragated later with other tests (like server-side unit tests and webdriver tests).

Ok that's the setup - for now just know that any '*Spec.js' file placed into the 'spec/client' directory will get executed and any client-side JavaScript file you write in public/javascripts will get loaded to be tested.

Executing:

% grunt jasmine

Will run all of this stuff.

Test Files

How about actually writing the tests? First get basically familiar with jasmine if not already. Now Let's take a look at logoutSpec.js first - a suite called 'logout' is created with two tests.

The most interesting bits are the fixture setup and the jQuery AJAX spy.

Fixtures

Most client-side JavaScript manipulates the DOM, but we don't want to load in all of our application's HTML to test our code, so we use 'fixtures' instead. The 'setFixtures' call is provided by the jasmine-jquery JavaScript library we loaded in via the 'vendors' array in our Gruntfile.js. This lets us set some HTML for the following test that is automatically cleaned up after the test ends. So note I have to 'setFixtures' for each test. If your HTML is the same for each test then you should use a 'beforeEach' function so you DRY (Don't Repeat Yourself). You'll see the HTML is slightly different for each test so I wasn't able to do that here.

Jasmine-jquery and also load fixtures from a webserver, which is especially nice if you are using templates (which you are!), so you can test using your application's actual HTML. As these fixtures are very simple I did not do that here. Also beware of how the fixture loading interacts with jQuery's AJAX object which I will discuss with more detail next.

Spies

Jasmine is espcially nice for providing spies for interacting with your code's dependencies. The logout component relies on an AJAX call to actually log the user out, but while unit testing we do not have a web server running so we need to mock out the AJAX call. We do that using the "spyOn($, 'ajax')" function call.

All subsequent calls that use jQuery's AJAX mechanism will instead get routed to this spy, the underlying AJAX call will NOT get executed (note jasmine does provide a way to also call through to the actualy implementation but we do not want to do that here).

The spy allows us to verify the ajax method was called with the expect arguments. Spies let us do more than just verify arguments and call through to the real underlying implementation, take a look at loginSpec.js to see more!

Search for the 'andCallFake' method - using that method you can have the spied on method execute and return any code you like! Let's take a close look at the "should handle failure default response" test case.

First I set up the HTML fixture for this test, then I create the login component I will test along with some other canned values I will be using more than once so I DRM (Don't Repeat Myself). Now I create a spy for the $.ajax method and inform jasmine whenever that method is called to actually execute the given function instead. Note my function receives the argument list, and in this case I simply turn around and call the provided 'error' callback with some canned data to test that code path. I then set some form elements and 'submit' the form. As this is all synchronous the error callback will get called via 'andCallFake' and finally I expect the error message is being shown properly. Finally note the 'toHaveText' matcher came from jasmine-jquery, which comes chock full of lots of great jQuery-specific matches for us to leverage. Be sure to check them out.

Test Files

So all of our client-side unit test files must reside in the spec/client directory and be named with the word 'Spec' in them - follow that pattern for your own sanity! As you create more component keep adding tests, they can be at any depth in the spec/client tree, jasmine will find them!

_SpecRunner.html and Debugging Tests

Very rarely things do not work the very first time. You'll note there is a grunt configuration property 'keepRunner'. I told you about the SpecRunner.html file grunt-contrib-jasmine generates each time you run 'grunt jasmine' - by default that file is deleted after all the tests complete. But if there a problem running the tests it is very hard to tell what is going on, especially as the tests are all run in phantomjs. By setting 'keepRunner' to 'true' grunt-contrib-jasmine will not delete SpecRunner.html so you can load it up into Chrome (or any other browser not named Chrome) and manually execute unit tests yourself. They will run quickly! Most importantly you can open up a JavaScript console/debugger window to really tell what is going on when your tests are not running correctly.

Test Output

Test results are output in the JUnit XML format with pass/fail information and dumped to the build/reports/jasmine/ directory with one XML file per-suite. The most interesting thing about this format is its wide support.

Code Coverage

All JavaScript files under test are automatically instrumented for code coverage information. Regardless of tests passing/failing you will get code coverage information dumped to public/coverage/client/index.html - simply point your browser and that file and bask in the coverage'ness. You can drill down to each individual file and see line-by-line output of coverage as provided by istanbul. Let this information help guide your next set of tests!

Monday, June 24, 2013

Intro To baseapp: Making JavaScript Best Practices Easy

baseapp provides all the boilerplate to get your JavaScript web application started off right, this Part 1.

(or binge read them all on the baseapp wiki!)

Getting started is always the hardest part. Whether a web app, testing, or a blog post staring at a blank sheet of metaphorical paper is tough. You have a great idea of what it should do, what you want, but how to start? You just want to write code that brings your idea to life, but how to get to that part? You need a web server, you need a database, you need authentication, you want it to look nice, you want to use best practices, you want to use the latest technology, and you want it to be fun. That is why you are doing all of this at some level, it has got to be fun! That means no tedium, no slogging through boilerplate, how can you skip all of that and just jump straight to the fun part? Writing code that will change the world, or at least a corner of it, how can you get to that part as quickly as possible? There is so much cruft to fight through to get there, wouldn't it be nice if all of that stuff could just 'blow away...'?

My friends I have been there, I have had lots of great ideas, start hacking, and then got quickly crushed under the weight of boilerplate and 'best practices'. I want to 'test first'. I want to have a solid foundation for my web app. I want to do things the 'right way'. But there always seems to be a horrible choice at the beginning of a new project: either start trying to do things the 'right way' by setting up testing, automation, and foundation before any of the 'exciting' and 'interesting' work (because you cannot bake that stuff into your code later), OR dive right in and start coding the 'good' stuff and be left with a mess soon thereafter. What a crappy set of choices! I think we all agree we'd LIKE to start our new project off 'right' with all the test and automation infrastructure built in from the beginning, but doing that saps all of the joy out of coding, out of turning our great idea into an awesome product that the world loves. And what's the fun in that?

So I give to you, stymied developer, the boilerplate. I tend to use the same technology in most of my web apps: Express and Redis on the server-side, Bootstrap and LinkedIn-DustJS on the client side. I use Jasmine and WebDriver for testing and Istanbul for code coverage. Grunt and Travis-CI for automation and continuous integration. I run everything through JSHint. I like Plato's static code analysis. And most of my apps need user registration and login/logout. And I am sick of having to rewrite all of those pieces from scratch for each web app I dream up! So, as of today, no more. I have created 'baseapp', a bare application with all of that all thrown in and ready to go. The idea is you (and me!) fork this repo for every new web app you write so you can jump straight into the good parts and all the boilerplate test/automation/code coverage crap is already taken care of for us. I'm not just talking the packages are downloaded for you. I'm talking test code has already been written for all the base stuff. Most people, me included, just really want to see how something is done and then we can go off and do it ourselves, 'see one, do one, teach one' kinda thing. So I have pre-populated baseapp with client-side unit tests, server-side unit tests, and webdriver integration tests. No need to try to re-figure out how to piece all of these things together yourself. Just look at what I did and make more of them just like that.

Here are the technologies baseapp leverages to the hilt:

* github
* grunt
* jshint
* dustjs-linkedin
* express
* jasmine
* jasmine-node
* webdriver
* redis
* authentication
* istanbul
* plato
* karma
* phantomjs
* bootstrap
* travis-ci

This is all setup and baked into the baseapp, the packages aren't just installed, they are all actually used and setup and ready to go. If this stack is up your alley starting your web app with 'best practices' and 'test first' cannot be easier as all of the setup/boilerplate is already done.

Here is how to start your next web app:

1. Fork the baseapp repo
- so you have your own copy of it - you will be hacking away adding your special sauce!
2. Clone into your local dev environment
3. Update 'package.json' with the name of your web app
- This is your biggest decision - what to name your app!
4. % npm install
- This will suck down all packages necessary for your awesome new dev environment
5. Install & start redis from redis.io
- But you've already got it installed & running right?
6. To run WebDriver/Selenium tests you need to start the Selenium jar
- % java -jar ./node_modules/webdriverjs/bin/selenium-server-standalone-2.31.0.jar
7. % grunt test

You will see tests pass hopefully! If not you will get an informative error message to as why.

To see the app running yourself simply:

% npm start

And then in your browser go to:

http://127.0.0.1:3000

And you will see your beautiful application so far - buttons to login and register - that actually work - and that's it.

This article is the beginning in a series explaining exactly what it going on in 'baseapp' - I will explain everything that has already been set up for you and how to leverage it as your develop the world's next great webapp - stay tuned and have fun!