Double automation as performance test

My system under test is so clunky, that I can observe performance differences between builds. That gives me an idea to double automation as performance benchmark test. 

The implementation is very simple: for important actions, automation cases will log their start and end time and save them to a database; a web application will analyze and show reports of performance comparison.

I am fairly new to JavaScript world, if it was in Java, logging time of important actions is a cross-cutting concern and would be implemented through AOP. But in my test framework, I just created a utility class and throw it into places where it is needed.

var ActionRecoder = function () {}

var actionStack=[];


ActionRecoder.prototype.renew=function(){
    this.actionList=[];
    this.caseHardWait=0;
}

ActionRecoder.prototype.bastart=function(action, actioner){

    logger.info();
    logger.info();
    actioner=actioner || glue.operator;
    logger.info("*********************"+actioner+" started "+action+"*********************");

    this.dostart_(action, actioner, true);
}


ActionRecoder.prototype.bastop=function(){

    var lastAction=actionStack.pop();
    logger.info("*********************"+lastAction.actioner+" exited "+lastAction.action+"*********************");

    this.dostop_(lastAction);
}


ActionRecoder.prototype.start=function(action, actioner){

    actioner=actioner || glue.operator;

    logger.info(actioner + " start:" + action+"~~~~~~~~");

    this.dostart_(action, actioner);
}


ActionRecoder.prototype.stop=function(){
    var lastAction=actionStack.pop();
    logger.info(lastAction.actioner+" stop:"+lastAction.action+"~~~~~~~~");

    this.dostop_(lastAction);
}

ActionRecoder.prototype.dostart_=function(action, actioner, isBusinessAction){

    actioner=actioner || glue.operator;
    actionStack.push({actioner:actioner, action:action, start:new moment(), businessAction:isBusinessAction});
}

ActionRecoder.prototype.dostop_=function(lastAction){
    lastAction.end=new moment();
    lastAction.duration=lastAction.end.diff(lastAction.start,"milliseconds");
    lastAction.start=lastAction.start
    lastAction.end=lastAction.end

    this.actionList.push(lastAction);

    return true;
}

There are two kinds of actions. My automation test framework is built around domain objects so that overtime, it becomes easier and easier to write automation test cases:

A Login.login is a business action, it involves opening up the login page, entering user name and password and clicking on the login button. A login.login.clickBtnLogin is a finer action, it records the time the login button is clicked and the result page is shown.  Recording finer actions helps tracking down performance issues. 

One thing to look out for is that because in Protractor almost all actions on a UI is a promise, ActionRecorder.stop has to be invoked in then . My automation test framework uses generator, so I can just invoke it at the last step of the function. 

3 levels of data are recorded, one is for action:

One is for test case:

The final one is for test suite:

The web application is much more fun, I used React, D3 and Nodejs to build it. This diagram shows the comparison of action “Login.login.clickBtnLogin” between builds 490 and 678.

This kind of comparison serves to give an overview, but it doesn’t make much sense: login of different users will have different performance, even the same user, login in different context will have different performance, so comparison has to be done between apple and apple. 

This configure page allows to configure apple and apple comparison:

And here is the result:

The column underneath the zero is where build 678 is quicker than 490,  above is where build 679 is slower than 490. 

I am thinking now if automation can triple as stress test. Because my automation uses Protractor which uses Selenium, first I need to figure out the stress level of Selenium. Stay tuned.