Testing through the Credit Crunch - Part 5 - Visualization

Visualization of testing is one of the new and exciting areas of testing. It is being born out of the need to see results and coverage of testing quickly. Remember the old days of writing test strategy documents at the beginning of a project, then rewriting them a week later, and then a week later until you get to the end of the project and the document looks nothing like it did at the beginning of the project. Oh and did I mention that the document never gets completed?!

Now you have stakeholders just assuming you did the right thing, documents that will just pass an ISO audit and a slight resentment to your project manager for allowing scope creep! Then came along strategy documents in a state diagram, like Graphical Test Strategy I discussed before, that allows you to get your document done in a day and sent out to stakeholders. All of a sudden you a hero to your "customers" as they become more involved in the quality of the end product.

You have drawn a couple pictures to show people what you are going to do but now lets have a look at drawing pictures to actually do the testing. If you carry on breaking down the state diagram you are given a beautiful test case diagram. Translating that into a automated test can be done with a tool called CubicTest. CubicTest is an Eclipse plugin that allows you create abstractions for test cases and then gives you Selenium or Watir Tests using a state diagram. Its a good tool if you are constantly using Eclipse but I never use Eclipse

Visualization also offers a lot of value to code metrics. Unit tests are only of value if they test different scenarios and if you are testing different scenarios you will inheritently get good code coverage. Code Coverage is a good way to see that you are testing properly. Well, that you are testing in the right areas. My tool of choice is NCover but thats because I work in a .NET shop.

All of this can be placed into a CI server and all it can then be reported on easily. I use CruiseControl.NET to do my CI and I love that it shows all the passes,ignores and fails of tests from the web front end. I can check up on the developers without having to get their code out of the source repository. The manual testing guys make their scripts update a database. Thanks to simple charting APIs( Google Charts ) I can see how projects are going from one page. So without too much cost and fuss we can see how testing is going and the stakeholders can feel attached to QA of the product from Development to Delivery! As always tools to do what I have talked have open source alternatives.

Unfortunately none of these tools make the lives of testers easier. It works is showing that we are doing well, or not if thats the case, but how do we know that we have a decent amount of code coverage on core libraries? The Microsoft XBoX team analyse their logs and apply Heat Maps so that they can see where their testers have been doing. But what if you took your normal code coverage report and put that through a heat map but put a weighting on the function. The weighting relates to how bad a bug would be if found in this area and if we are to work it how well its tested by the code coverage and the number of tests we can actually get a feel for what we are developing and testing.

Finally the thing that is desperately needed is a Testers Heads Up Display (T.H.U.D). James Whittaker discussed this in his series on the Future of Testing and in his Keynote at GTAC 2008. It takes the idea from games where the H.U.D. allows gamers to be really good at the game. The T.H.U.D will give you little notes when you are working saying that there are potentially bugs in this area and telling you about previous bugs in that area. My idea of the T.H.U.D would give you the information and then record where you have been so that you get a heat map over your application when you want to check weather you are finished in this area.

The faster the iterations get the more testers will need something like this. The average ratio of developers to testers seem to sit around 3:1 and it takes just one developer not to unit test their code to slow a tester down since they have to look for smaller bugs as well as those hard to find that we testers look forward to finding.

This leads me onto my final post in the series: Innovation