Testing through the Credit Crunch - Part 6 - Innovation

This is the last in the series and I thought that I would finish it off discussing the innovative culture that is needed in today's climate. If you work in an Agile environment trying to create the next big thing you know not all the components you need have been created. And if they haven't been created you have very little to zero chance that you will find a testing tool to automate your application.

You can automated the units of the code, obviously, but there are parts of the application that you can't test with a unit/integration testing framework or your continuous integration server is taking too long to build and test. I know we have all seen the last issue.

So what do we do then? We need think outside of the box to get a solution. A lot of the testing frameworks have grown from the need for something to fit a certain issue.

The image on the left shows the increase in testing projects on sourceforge.net over 9 years. As you can see there has been a nice growth of testing frameworks. A lot of these have been grown from the need to test something and there are no tools out there. One example of this is Selenium.

Selenium started out as an in-house testing framework for ThoughtWorks. Now its one the worlds best functional testing tool. Next tool to talk about is a Selenium derivative. Selenium Grid was born out of the need to make the tests run quicker. The build was taken forever so Philippe Hanrigou and a few other ThoughtWorkers extended Selenium RC to get it to runs tests in parallel without having to worry the infrastructure out there.

I am currently building a system to pull in usability stats from our web application with a developer using 3 free tools that anyone out there can find and making a nice hybrid test tool. I will document this in a future blog post with my colleague. But this has been born out of the need to do something that no other tool seems to do at the moment.

The skill that is needed by testers all across the board is the ability to tackle projects by thinking laterally. Tools are not always going to be available for everything and record-and-replay will not solve all of the worlds issues. There are a number of harder issues that need to be solved like the Testers-Heads-Up-Display that I discussed in my last post.

All of these items can easily be solved with a little innovation by all the testers out there. I guess the question is, what can you offer to the testing world.

Testing through the Credit Crunch - Part 5 - Visualization

Visualization of testing is one of the new and exciting areas of testing. It is being born out of the need to see results and coverage of testing quickly. Remember the old days of writing test strategy documents at the beginning of a project, then rewriting them a week later, and then a week later until you get to the end of the project and the document looks nothing like it did at the beginning of the project. Oh and did I mention that the document never gets completed?!

Now you have stakeholders just assuming you did the right thing, documents that will just pass an ISO audit and a slight resentment to your project manager for allowing scope creep! Then came along strategy documents in a state diagram, like Graphical Test Strategy I discussed before, that allows you to get your document done in a day and sent out to stakeholders. All of a sudden you a hero to your "customers" as they become more involved in the quality of the end product.

You have drawn a couple pictures to show people what you are going to do but now lets have a look at drawing pictures to actually do the testing. If you carry on breaking down the state diagram you are given a beautiful test case diagram. Translating that into a automated test can be done with a tool called CubicTest. CubicTest is an Eclipse plugin that allows you create abstractions for test cases and then gives you Selenium or Watir Tests using a state diagram. Its a good tool if you are constantly using Eclipse but I never use Eclipse

Visualization also offers a lot of value to code metrics. Unit tests are only of value if they test different scenarios and if you are testing different scenarios you will inheritently get good code coverage. Code Coverage is a good way to see that you are testing properly. Well, that you are testing in the right areas. My tool of choice is NCover but thats because I work in a .NET shop.

All of this can be placed into a CI server and all it can then be reported on easily. I use CruiseControl.NET to do my CI and I love that it shows all the passes,ignores and fails of tests from the web front end. I can check up on the developers without having to get their code out of the source repository. The manual testing guys make their scripts update a database. Thanks to simple charting APIs( Google Charts ) I can see how projects are going from one page. So without too much cost and fuss we can see how testing is going and the stakeholders can feel attached to QA of the product from Development to Delivery! As always tools to do what I have talked have open source alternatives.

Unfortunately none of these tools make the lives of testers easier. It works is showing that we are doing well, or not if thats the case, but how do we know that we have a decent amount of code coverage on core libraries? The Microsoft XBoX team analyse their logs and apply Heat Maps so that they can see where their testers have been doing. But what if you took your normal code coverage report and put that through a heat map but put a weighting on the function. The weighting relates to how bad a bug would be if found in this area and if we are to work it how well its tested by the code coverage and the number of tests we can actually get a feel for what we are developing and testing.

Finally the thing that is desperately needed is a Testers Heads Up Display (T.H.U.D). James Whittaker discussed this in his series on the Future of Testing and in his Keynote at GTAC 2008. It takes the idea from games where the H.U.D. allows gamers to be really good at the game. The T.H.U.D will give you little notes when you are working saying that there are potentially bugs in this area and telling you about previous bugs in that area. My idea of the T.H.U.D would give you the information and then record where you have been so that you get a heat map over your application when you want to check weather you are finished in this area.

The faster the iterations get the more testers will need something like this. The average ratio of developers to testers seem to sit around 3:1 and it takes just one developer not to unit test their code to slow a tester down since they have to look for smaller bugs as well as those hard to find that we testers look forward to finding.

This leads me onto my final post in the series: Innovation

Testing through the Credit Crunch - Part 3 - Virtualization

So far in the series we have learnt how to save money by becoming more Agile and by implementing Test Driven Development. The move to these can take a little while to implement because of the learning curve involved and/or the inability of some people to move out of their comfort zone.

This post is going to discuss virtualizing our test environments. Virtualisation is fast becoming the norm in data centers because it allow infrastructure managers to build up disaster recovery machines fairly quickly. They can also utilize the hardware of the host more efficiently. Good thing when everyone is trying to become 'greener".

So if the infrastructure people think its a good idea then it must be a good idea for testers. Getting real machines for testing is expensive, not only in terms of the hardware and software, but also the human resources to maintain them. Think about having 10 computers with 10 Windows licences that then need to be maintained. Having a quick look at Dell for a basic computer it would cost at least £2990 for the 10 computers and windows licences and just to make it a round number lets say that it will take £100, £10 per machine, to maintain them for a test cycle.

So we now have a cost of at least £3000 to get all the computers and maintain them for one cycle. That doesn't include the electricity to run all of this hardware. Since its the credit crunch we don't that amount of money to waste. The best thing to do is to virtualise the whole lot.

Spend the same amount of money on a low range server . So now we only have one machine that needs to be maintains. Maintenance cost goes from £100 down to £20. I say £20 because maintenance on servers can be a little more because the hardware is slightly more expensive. http://www.vmware.com/go/calculator will give you a better calculation. I did it and for 500000 machines, defaults in the calculator, I saved £350000+ over a time frame.

So whats the best Virtualisation software available? This is a very hard question to answer. I am big fan of free software so use Microsoft Virtual Server 2005 and VMWare ESXi. I have heard good things about XEN if you have a Linux. The main thing is to chose an application that you feel that you can work with quite well. Both MS Virtual Server and VMWare ESXi have APIs that allow you to manipulate the machines that are running on it. This means that you can clone machines and create a new machine by just running a script. As with Test Automation you won't see the saving straight away but it will save time, hence money, in the long run.

In the next post I will discuss visualisation of testing

Testing through the Credit Crunch - Part 2 - Test Driven Development

In the last post of this series I discussed briefly about Technical Debt and how Test Driven Development can help in dropping the debt and dropping the amount of interest gained over the life of the product.

Test Driven Development is the concept that you write your tests before you write your code. Once your code is finished you refactor with the safety that your newly created test will prevent any stupid bugs that you could have introduced. It's what the TDD community call Red-Green-Refactor. Now to most people writing tests first sounds stupid, "How do you know what your code is going to do until you have finished coding it?"
My retort tends to be "You know what you are expecting back from the call. Your test needs to be really small and since an Assert is a test, Assert what you are expecting your method to return! Image that your code exists and test against it." Below is an example of a small test and when writing tests we should try keep them as small as we physically can.

[Test]
public void creditcrunch_Test(){
int FutureTechnicalDebt = 100;
Assert.LessThan(FutureTechnicalDebt,TechnicalDebt.Current);
}

So we know that we need tests first to give us a good safety net for writing our code.In Technical Debt terms we have spent a little less to create/run and report on the test and we prevent anyone else breaking our code. Cost-- and Interest-- which is what we want. Running the test above will fail because there is no code to run it against. Write some code to make it pass if you have the time.

The other reason why developers are starting to like TDD, other than the removal of the ear ache that testers would give them about throwaway builds, is that challenges them to think about the interface into their new object. We have all sat down with the keyboard in front of us, tea/coffee next to us and found that someone has changed the interface your were developing against! Queue the expletives! $*@%*$£ !@)@)£)&! Luckily for us we have good Test Driven Developers who have have discussed the interface and we by proxy prevented interest being added to our debt. Interest--! Communication is key in Test Driven Developments, especially when it comes to interfaces because there may be some integration bugs that will creep in without you realising!

Knowing what the Interface is going to look like is has another major benefit. Mocks and stubs! If we know what the interface to an assembly is and how its going to react we can write tests to test our code and when it needs to speak to the other assembly, which may not have been created yet or makes our tests run slowly like accessing a database or filesystem, we can return the results back that we want our code to handle using one of them. This could be exceptions,dodgy data or good data. Testing those code paths means we can increase our code coverage therefore increasing the size of our safety net! It will also mean that our tests run a lot quicker because we don't have to interact with slow drives or have anyone elses tests interfere with our data. Cost-- and Interest--.

So by creating this safety net we have prevented stupid mistakes creeping into our code. Its these stupid mistakes that could cost our team/company a lot of money. Since companies don't have much money it could mean your job!

The graph below shows how by doing the, what I call, simple things we have brought down the technical debt that we could potentially owe. NOTE: The values used are just arbitary values I found on the internet with a little searching.

In the next post in the series I will discuss Virtualization and what the it can mean to you as  a tester!

Is documenting test processes a bad idea in an Agile Development Environment?

I work in a smallish development team in Southampton as some of you know. There are other development teams in the group in United Kingdom and in France that all fall under the same group of companies of different sizes.

The unfortunate problem that we have is that there isn't a properly documented process for getting everything in place and ready for each testing cycle. Once we have everything ready we have another problem that I will hopefully discuss in a future posting.

This is where I had the thought "Is documenting test processes a bad idea in an Agile Development Environment?". If you have worked in an office with ISO9001 accreditation you will know that everything needs to be documented or if you have worked in a Six Sigma office you will have noticed that you have to put everything you do in a database so the "Black Belts" or "Master Black Belts" can see areas that need improvement.

So back to my question, "Is documenting test processes a bad idea in an Agile Development Environment?". My answer is definitely! And I am going to admit this is a fairly biased answer having started my career in large financial institutions where I was taught that everyone is expendable but their process knowledge isn't. This means that everything needs to be documented. However my thoughts on this is that things need to documented and followed to a point as long as that point does not impact on the creativity of the software developers and the testers.

Its this creativity that allows people to come up with new and exciting ways of developing and testing. Tools like the xUnit and Selenium and many more have come from this creativity. This then needs to be balanced with some form of Total Quality Management (TQM) so that everything can be reproduced if and when needed.

Documented processes can't be completely wrong, if they were then large companies like IBM, Citrix and Microsoft would have never got to where they are now!

Graphical Test Planning

In December I went to conference in London and one of the lectures that I went to listen to was about Graphical Test Planning by Hardeep Sharma of Citrix Systems.

Graphical Test Planning is a way of creating a test plan without having to write entire documents. I am sure that most of the readers
that will be reading this will be practise full agile development and are probably
thinking that I am mad.

Maybe I am mad but I really think that this is a great way to work out what needs to be testing and
is a good way to track progress of testing.

The entry below is more my thoughts having tried this technique on web projects
and have used a web examples. I hope you enjoy reading it!

Graphical Test Planning is done by creating a structural relationship diagrams.
These
allow product managers to know what is going to be tested from the day that they decide there is going to be a new version of the product.

What a Graphical Test Plan is or isn't

What a Graphical Test Plan is:

• A structured relationship diagram


• A list of behavioural areas that need to be tested


• A method of getting greating feedback about what needs to be tested


• A method of tracking progress of design and execution of testing

What a Graphical Test Plan isn't:

• A flow diagram


• A tester managers thoughts or mind map


• A feature list


• A hand-holding exercise for a test engineer on how to use the application

How To Create A Structured Relationship Diagram

A structured relationship diagram will show what behavioural areas need to be
tested on different platforms and different compilers. Below is an example of
what a SRD looks like for Selenium.

Its not 100% correct for selenium but I wanted to show something
with no coverage.

From this you can see what areas need to be tested and what doesn't. If we carry this theme on we can then
start drilling the behavioural areas down into more detailed SRDs and if we were to
carry on drilling down we will start to create another form of these diagrams
called a Test Case Diagram.

A Test case diagram looks a lot like a SRD except at the end of the digram instead of showing the behavioural areas it will show the expected results.
The great thing about this is that you can create the SRD in about 10 minutes and then do the test case diagrams in a few hours and this means that you
can have all this test information done before any code is done.

So if we use this metric to create an entire test strategy for a project using graphical test planning compared to the waterfall method of doing things

you would have saved yourself somewhere in the region of 1 man month and you are likely to have fixed a number
of flaws in the design before any code is near completion.

I hope that you found this very interesting and I would like to the thank

Hardeep Sharma for showing off this technique at the SIGIST Conference in
December.