I had an interesting conversation with our Test Automation team lead Stefan – who Andi interviewed for our “Eating our own Dog Food ” article – on his experiences with the willingness of developers to write performance tests.

I asked a provocative question: do developers really want to write them in the first place? First he smiled but then he said that they do. I honestly was a bit surprised because in my own experience as a developer was that I rather wanted to write code instead of tests.

He agreed that it was not that easy in the beginning; however, after the first developers started it became a more and more common practice. I then asked him what he thinks are the main reasons why developers want to write performance tests. The following sums up our experience at dynaTrace into what causes developers themselves write performance tests.

Reason 1: Everyone wants to be a hero
Developers like to write well-performing code. Even more, they like to write faster code. Just like Captain Kirk saying “I need the ship ready in 2 hours” – and Scottie replying “I will do it in 20 minutes”.

Every developer wants to show how well his code performs. The performance testing hype at dynaTrace really started when a developer sent out an email about how his performance tests showed him a regression immediately after checking in. The figure below shows exactly the image the developer sent out.

CI Performance Graph Showing Several Performance Regressions

The developer wanted to share his experience and also show others the benefits of the tests he wrote. A couple of weeks after this email another developer sent out an email about the improvements he achieved for a component between two releases.

Another team of developers managed to optimize the throughput of our server by up to four times. This was a really great achievement and the result of hard work – earning them the dynaTrace culture award. So their contribution was acknowledged as one of the most significant in the last quarter.

So if you want people to deliver certain results then award them for those results and give them the fame they deserve.

Reason 2: The “It will come back anyway” Experience
This one is nice as nobody believes it first. You can always tell people that if they do not get things right during development, a potential problem will come back to them later. So while performance testing might mean some additional work right now it is better than having even more work later on when you have to fix performance bugs.

As I said, nobody believes this until they experience it for the first time themselves. Then people start to write their tests to avoid running into this situation again. There is nothing more frustrating than fixing bugs – possibly in older versions – when all your colleagues are working on the cool new features.

Reason 3: He is working on my code too
At dynaTrace we have always taken performance seriously. Developers have always been profiling their code to optimize it or add additional output for performance measurement. This works fine if you are the only one working on your code and do not have any dependencies on other components. If you not earning money by writing “Hello World” applications this is not the case ; -). Other developers are working on the code base as well.

So beside the fact that your profiling results are only available to you locally, they might already be outdated after the next check-in. As you have no automated means to find those changes, you will not be aware of it. Only a standardized test suite that is automatically executed as part of your Continuous Integration environment can help here

Some people also use specific logging code which they think will help them to track down problems. While this works fine sometimes, our experience has shown that others might change the code causing part of the logging statement not to be executed anymore. The reason might be that your code just got deprecated or somebody added an additional conditional statement, so logging code now only gets executed in certain cases.

Reason 4: Only let developers write test code
This is a key lesson we learned in making performance testing successful. The more complex it is to write test code, the fewer tests are written. Especially performance tests can easily become quite complex – sometimes even more complex than the actual code you want to test. In our case, we want to start our own software, start a couple of application server instances and then trigger a load test. All this has to be managed across multiple nodes.

If we forced developers to write that code themselves they would not want to write any tests at all. So we took a massive investment in writing our own testing framework that takes away that burden from the developer. The example below shows how to start a remote WebSphere server; all done by just adding some simple annotations.

@DtdRemoteSud(

host = "lab2",

name = "WAS7.0",

startupPriority = 1,

postStartClosure = WaitForWebSphereSudIsUp.class

)

private SudInterface webSphereSud;

Additionally, dynaTrace itself provides extensions which developers can use to control our own software components. Developers now simply focus on writing their test cases. All this together help us to create a good test suite quickly and make it attractive to developers to write tests.

Reason 5: Provide Feedback
If you invest your time, you want to get something back. So the key to make developers write more tests is to make the test results easily accessible to them. The immediate value for the developer is to get feedback about the performance impact of his code changes very quickly. We run our performance test suite two times a day. So ideally you can fix a performance bug the same day you created it :-

Developers get this feedback via email just as build or functional test results. So they invest time once and get feedback anytime they – or somebody else – changes the code.

Reason 6: The old code threat
Not only we but also our code gets older. The older code is, the less you want to touch it. However very often you have no choice. Martin Fowler already wrote in his Refactoring book that you also start refactoring by having the proper tests in place. While Martin was talking about functional tests here, the same is true for performance tests.

The really evil thing is that if you have any modifying components which heavily interact with each other you can very easily introduce performance problems. So instead of just ruining your own code, you will also ruin the code of other developers.

We have seen especially bug fixes introducing such problems. Just as you can see in the picture above, a functional bug fix introduced a serious performance problem. Additionally you want to be fast in fixing the old code and have more time to develop the really cool new features – don’t you?

Reason 7: I might be the newb but not the noob
Stefan realized that the time when the most test cases are written is when a new developer takes over a feature. This is for two main reasons: First, you do not want to break features of other developers; Second, you are new and want to learn how the code works. The best way to do this is – except reading source code and asking former developers – by testing. So developers who take over a feature have the highest acceptance of writing tests.

Related reading:

1. Performance Management in Continuous Integration I recently gave  presentations on Performance Management as part of...
2. dynaTrace Application Performance Almanac 2010 Inspired by the work of Stoyan on his performance advent...
3. Do more with Functional Testing – Take the Next Evolutionary Step Functional Testing has always been an activity done by Test...
4. Randomizing Input Data for Visual Studio Load Tests While preparing for my presentation Load and Performance Testing: How...
5. Eating Our Own Dog Food: How dynaTrace does Continuous APM Internally in Development with dynaTrace TA Lead Stefan Frandl I sat together with Stefan Frandl, Test Automation Lead in...