Double automation as performance test

My system under test is so clunky, that I can observe performance differences between builds. That gives me an idea to double automation as performance benchmark test. 

The implementation is very simple: for important actions, automation cases will log their start and end time and save them to a database; a web application will analyze and show reports of performance comparison.

I am fairly new to JavaScript world, if it was in Java, logging time of important actions is a cross-cutting concern and would be implemented through AOP. But in my test framework, I just created a utility class and throw it into places where it is needed.

var ActionRecoder = function () {}

var actionStack=[];


ActionRecoder.prototype.renew=function(){
    this.actionList=[];
    this.caseHardWait=0;
}

ActionRecoder.prototype.bastart=function(action, actioner){

    logger.info();
    logger.info();
    actioner=actioner || glue.operator;
    logger.info("*********************"+actioner+" started "+action+"*********************");

    this.dostart_(action, actioner, true);
}


ActionRecoder.prototype.bastop=function(){

    var lastAction=actionStack.pop();
    logger.info("*********************"+lastAction.actioner+" exited "+lastAction.action+"*********************");

    this.dostop_(lastAction);
}


ActionRecoder.prototype.start=function(action, actioner){

    actioner=actioner || glue.operator;

    logger.info(actioner + " start:" + action+"~~~~~~~~");

    this.dostart_(action, actioner);
}


ActionRecoder.prototype.stop=function(){
    var lastAction=actionStack.pop();
    logger.info(lastAction.actioner+" stop:"+lastAction.action+"~~~~~~~~");

    this.dostop_(lastAction);
}

ActionRecoder.prototype.dostart_=function(action, actioner, isBusinessAction){

    actioner=actioner || glue.operator;
    actionStack.push({actioner:actioner, action:action, start:new moment(), businessAction:isBusinessAction});
}

ActionRecoder.prototype.dostop_=function(lastAction){
    lastAction.end=new moment();
    lastAction.duration=lastAction.end.diff(lastAction.start,"milliseconds");
    lastAction.start=lastAction.start
    lastAction.end=lastAction.end

    this.actionList.push(lastAction);

    return true;
}

There are two kinds of actions. My automation test framework is built around domain objects so that overtime, it becomes easier and easier to write automation test cases:

A Login.login is a business action, it involves opening up the login page, entering user name and password and clicking on the login button. A login.login.clickBtnLogin is a finer action, it records the time the login button is clicked and the result page is shown.  Recording finer actions helps tracking down performance issues. 

One thing to look out for is that because in Protractor almost all actions on a UI is a promise, ActionRecorder.stop has to be invoked in then . My automation test framework uses generator, so I can just invoke it at the last step of the function. 

3 levels of data are recorded, one is for action:

One is for test case:

The final one is for test suite:

The web application is much more fun, I used React, D3 and Nodejs to build it. This diagram shows the comparison of action “Login.login.clickBtnLogin” between builds 490 and 678.

This kind of comparison serves to give an overview, but it doesn’t make much sense: login of different users will have different performance, even the same user, login in different context will have different performance, so comparison has to be done between apple and apple. 

This configure page allows to configure apple and apple comparison:

And here is the result:

The column underneath the zero is where build 678 is quicker than 490,  above is where build 679 is slower than 490. 

I am thinking now if automation can triple as stress test. Because my automation uses Protractor which uses Selenium, first I need to figure out the stress level of Selenium. Stay tuned.

Yield out of Promise hell

In Protractor, every action on an UI element is a promise. A promise is something that happens in the future. It is easy to mix present with the future, for example:

it('test_promise', function () {
    browser.get('http://localhost:8080/examples/test.html')
        .then(function() {
            var btnEle=element(by.id("b1"));
            for(var i=0; i<5; i++){
               console.log("clicking at "+i+" times");
               btnEle.click().then(function(){
                   console.log("clicked at "+i+" times");
               })
           }
        });
})

The output from the above code is:

clicking at 0 times

clicking at 1 times

clicking at 2 times

clicking at 3 times

clicking at 4 times

clicked at 5 times

clicked at 5 times

clicked at 5 times

clicked at 5 times

clicked at 5 times

What happens is the click() action is put into the promise queue and scheduled to happen in the future, while the for loop finishes in the present. If we want click() to record the clicking times correctly, we have to use another variable to record the times: 

it('test_promise2', function () {
    browser.get('http://localhost:8080/examples/test.html')
        .then(function() {
            var btnEle=element(by.id("b1"));
            var j=0;
            for(var i=0; i<5; i++){
                btnEle.click().then(function(){
                    j++;
                    console.log("clicked at "+j+" times");
                })
            }

        });
})

 

This trick can be used to get values from an array of promises:

it('test_getAllElements', function () {

    browser.get('http://localhost:8080/examples/test.html')
        .then(function() {
            var messages=[];
            var j=0;
            var defer = protractor.promise.defer();
            element.all(by.xpath("//body")).then(function(eles) {
                for (var i = 0; i < eles.length; i++) {
                    util.getNonInputText(eles[i]).then(function (text) {
                        if (util.isDefined(text) && text.length > 0) {
                            console.log("message is:" + text);
                            messages.push(text);
                        }



                        j++;
                        if (j === eles.length) {
                            console.log("fulfilled");
                            defer.fulfill(messages);
                        }
                    });
                }

            });


            defer.promise.then(function(result){
                console.log(result);
            });
        });
})

Even something as innocent as getting the ID of an element is a promise, to get the ID, you have to get it in then():

 Util.prototype.getId = function(element){
    return element.getAttribute("id").then(function(id){
       return id;
    });
};

In my tests, on my occasions, I have to calculate IDs based on known elements, so I have to write the logic like this:

var ele=element(by.name("abc"));
util.getId(ele).then(function(id){
    var anotherEle=element(by.id(id+"_btn"));
    anotherEle.click().then(function(){
        ...
    })
})

This is certainly less intuitive than getting the id directly:

var ele=element(by.name("abc"));

var id=util.getId(ele);
var anotherEle=element(by.id(id+"_btn"));

I wanted to find a way to make writing async code as intuitive as writing sync code, this led me to Generator.Here is some code that shows the basics of Generator:

function* f() {
    console.log('before a');
    yield 'a';
    console.log('before b');
    yield 'b';
    console.log('before c');
    yield 'c';
    console.log('before d');
    return 'd';
}

var g = f();
console.log("1");
console.log(g.next());
console.log("2");
console.log(g.next());
console.log("3");
console.log(g.next());
console.log("4");
console.log(g.next());

The output is:

1

before a

{ value: 'a', done: false }

2

before b

{ value: 'b', done: false }

3

before c

{ value: 'c', done: false }

4

before d

{ value: 'd', done: true }

function* f(), * makes function f a generator function. The code inside the generator function won’t get executed until next() is called; and execution inside the generator function will stop at yield until next() is called again. 

This feature is useful in making async code into sync. Say, one line of code creates a promise, the next line of code won’t get executed until the promise is executed. Library co (https://github.com/tj/co ) does exactly that. 

With co,   I can write code in a more intuitive way. For example, I can now get the id of an element:

co(function*() {
    var ele = element(by.name("abc"));
    var id=yield util.getId(ele);
    var anotherEle=element(by.id(id+"_btn"));
});

The resulting code is much more structured and readable:

 

And by the way, in Webstorm, you need to configure javascript language in order for Webstorm to support yield syntax: