Designing and Testing with Behaviours rather than requirements

A few months ago I decided that I wanted to learn something new. I decided that I would have a go on the Google App Engine, learning Python and jQuery. I decided that I would start working on something that would be useful, at least for me, and go from there. I also thought that I would give Behaviour Driven Development a try

I decided that I would work on a project management application. So the first thing I needed was a way to store my backlog and visually see where items were on the stack.

Since I have a stack on bits of paper and want to do this by describing my behaviors for each of my requirements I need a way to describe them that I can easily translate into a test. I went looking through all the different BDD Frameworks (NBehave, Behaviour, JBehave, RBehave,StoryQ, and many many more!) to look for the common phrases and it came back with the phrase that I think is brilliant. It flows off the tongue so easily that getting product owners involved to write them can make an agile project work like clockwork.

The phrase is below. NOTE: The bold items make the main phrase up and the italics make it more readable.

As a product owner
I want functionality
so that something is done
with Scenarios

Scenario 1:
Given start point
When do something
Then expections will be met
.
.
.
Scenario n:
.
.
.

This can easily be coded into different languages and in some of the languages the natural language above is all that is needed but just a little tweaking to get the functions calls sorted. If you are a Selenium nut like me you could even drive with something like FitNesse. Gojko Adzic has explained it really well on his blog so suggest you have a look there.

The other plus point for BDD in Agile environments is its ability to change the BDD test very quickly if the product owner changes their mind. And since product owners change their minds every time they have a cup of tea, and in the UK it can be very often, you need to be able to keep your tests up to date.

Testing through the Credit Crunch - Part 6 - Innovation

This is the last in the series and I thought that I would finish it off discussing the innovative culture that is needed in today's climate. If you work in an Agile environment trying to create the next big thing you know not all the components you need have been created. And if they haven't been created you have very little to zero chance that you will find a testing tool to automate your application.

You can automated the units of the code, obviously, but there are parts of the application that you can't test with a unit/integration testing framework or your continuous integration server is taking too long to build and test. I know we have all seen the last issue.

So what do we do then? We need think outside of the box to get a solution. A lot of the testing frameworks have grown from the need for something to fit a certain issue.

The image on the left shows the increase in testing projects on sourceforge.net over 9 years. As you can see there has been a nice growth of testing frameworks. A lot of these have been grown from the need to test something and there are no tools out there. One example of this is Selenium.

Selenium started out as an in-house testing framework for ThoughtWorks. Now its one the worlds best functional testing tool. Next tool to talk about is a Selenium derivative. Selenium Grid was born out of the need to make the tests run quicker. The build was taken forever so Philippe Hanrigou and a few other ThoughtWorkers extended Selenium RC to get it to runs tests in parallel without having to worry the infrastructure out there.

I am currently building a system to pull in usability stats from our web application with a developer using 3 free tools that anyone out there can find and making a nice hybrid test tool. I will document this in a future blog post with my colleague. But this has been born out of the need to do something that no other tool seems to do at the moment.

The skill that is needed by testers all across the board is the ability to tackle projects by thinking laterally. Tools are not always going to be available for everything and record-and-replay will not solve all of the worlds issues. There are a number of harder issues that need to be solved like the Testers-Heads-Up-Display that I discussed in my last post.

All of these items can easily be solved with a little innovation by all the testers out there. I guess the question is, what can you offer to the testing world.

Testing through the Credit Crunch - Part 5 - Visualization

Visualization of testing is one of the new and exciting areas of testing. It is being born out of the need to see results and coverage of testing quickly. Remember the old days of writing test strategy documents at the beginning of a project, then rewriting them a week later, and then a week later until you get to the end of the project and the document looks nothing like it did at the beginning of the project. Oh and did I mention that the document never gets completed?!

Now you have stakeholders just assuming you did the right thing, documents that will just pass an ISO audit and a slight resentment to your project manager for allowing scope creep! Then came along strategy documents in a state diagram, like Graphical Test Strategy I discussed before, that allows you to get your document done in a day and sent out to stakeholders. All of a sudden you a hero to your "customers" as they become more involved in the quality of the end product.

You have drawn a couple pictures to show people what you are going to do but now lets have a look at drawing pictures to actually do the testing. If you carry on breaking down the state diagram you are given a beautiful test case diagram. Translating that into a automated test can be done with a tool called CubicTest. CubicTest is an Eclipse plugin that allows you create abstractions for test cases and then gives you Selenium or Watir Tests using a state diagram. Its a good tool if you are constantly using Eclipse but I never use Eclipse

Visualization also offers a lot of value to code metrics. Unit tests are only of value if they test different scenarios and if you are testing different scenarios you will inheritently get good code coverage. Code Coverage is a good way to see that you are testing properly. Well, that you are testing in the right areas. My tool of choice is NCover but thats because I work in a .NET shop.

All of this can be placed into a CI server and all it can then be reported on easily. I use CruiseControl.NET to do my CI and I love that it shows all the passes,ignores and fails of tests from the web front end. I can check up on the developers without having to get their code out of the source repository. The manual testing guys make their scripts update a database. Thanks to simple charting APIs( Google Charts ) I can see how projects are going from one page. So without too much cost and fuss we can see how testing is going and the stakeholders can feel attached to QA of the product from Development to Delivery! As always tools to do what I have talked have open source alternatives.

Unfortunately none of these tools make the lives of testers easier. It works is showing that we are doing well, or not if thats the case, but how do we know that we have a decent amount of code coverage on core libraries? The Microsoft XBoX team analyse their logs and apply Heat Maps so that they can see where their testers have been doing. But what if you took your normal code coverage report and put that through a heat map but put a weighting on the function. The weighting relates to how bad a bug would be if found in this area and if we are to work it how well its tested by the code coverage and the number of tests we can actually get a feel for what we are developing and testing.

Finally the thing that is desperately needed is a Testers Heads Up Display (T.H.U.D). James Whittaker discussed this in his series on the Future of Testing and in his Keynote at GTAC 2008. It takes the idea from games where the H.U.D. allows gamers to be really good at the game. The T.H.U.D will give you little notes when you are working saying that there are potentially bugs in this area and telling you about previous bugs in that area. My idea of the T.H.U.D would give you the information and then record where you have been so that you get a heat map over your application when you want to check weather you are finished in this area.

The faster the iterations get the more testers will need something like this. The average ratio of developers to testers seem to sit around 3:1 and it takes just one developer not to unit test their code to slow a tester down since they have to look for smaller bugs as well as those hard to find that we testers look forward to finding.

This leads me onto my final post in the series: Innovation

Sorry about the spam

I would like to apologize for the RSS Spam that everyone has been getting lately. I have moved my Feedburner account to Google. It appears that Google did not test the migration process. I spammed everyone when sorting my CNAME so I can use http://feeds.theautomatedtester.co.uk/TheAutomatedTester as my Feed URL again instead of it returning 404 messages that Google had decided would be better after the migration. I spammed you guys then and then on the weekend Google decided to spam you with a fix that they added.

Sorry for the annoyance that everyone has suffered. Hopefully everything is now sorted and it won't try resend everything again!!!!

Sorry again for the spam!!!!

Testing through the Credit Crunch - Part 4 - Value Open Source

By now you will have hopefully realised that running a Test Department doesn't have to be expensive. We have seen that by making the entire company responsible for the quality of the product we take the strain off us doing the developers jobs and lets us find those show stopper bugs.

In this post I am going to discuss why you should value open source tools to automate finding those showstoppers.

Lets start at the small thing to be tools we need at the beginning:

Continuous Integration

How many time in the past have you released something and then suddenly have to roll back the release because not everything works together. Continuous Integration Servers get code out of the source repository. I am a big fan of Subversion as a source reprository, which just happens to be open source!

The top CI server application out there are CruiseControl, Cruise Control.NET, Hudson and TeamCity. 3 of those 4 are Open Source. The one that is not open source, TeamCity, requires a you pay for a licence if you want all the bells and whistles. You should be able to manage on the free version of TeamCity if that the one you decide on.

One of the most important features of all these tools is the ability to automatically run the unit tests that have been created by the developers.

Unit Testing

The de facto testing tools for Unit Testing are NUnit, JUnit and TestNG for .NET and Java. There are loads more out there, like unittest for python, that are used all the time. These tools are great for integrating into Continuous Integration Servers.

In the 2nd post of the series I mentioned that having unit tests allows us to have a safety net to work against. The CI servers will run every developers unit test. This allows developers to do a check-in/commit, once their tests pass, and then get the rest of the tests run. This means that if you make a change you will find out if you have by accident broke something. You will learn to appreciate this safety net but make sure that you don't become lazy and use that as your only testing.

Functional Testing

Web applications have the largest amount of Automation tools for functional testing. Since I also work in this industry I will be very biased towards the following applications.

• Selenium
  Selenium is slowly becoming the test tool of choice for Rich Internet Applications. Selenium allows you to write functional tests against AJAX applications. Selenium is used by the top Internet application companies and Agile Consultants.
  
  Its developed using JavaScript and can work in all Browsers on any OS. As you would have noticed I am a big fan!
  
• Watir
  Developed in Ruby its has a very good following. Test Scripts need to be developed in Ruby and you just import Gems for the Browser that you want to test.
  
  I found that it needed a lot of work to test multiple browsers compared to Selenium. I know some people would say its easier to use that Selenium. I guess its down to the user.
  

Acceptance Testing

More and more companies are starting to use acceptance testing tools to capture their requirements. Acceptance test frameworks can all integrated into Continuous Integration Servers. They take a plain text requirement and turn it into a Test Fixture.

• Fit
• FitNesse
• Concordion

The main tool for acceptance in my opinion is a manual tool commonly known as the customer. Remember that customers are both internal and external to the company! End users are the best acceptance testers as they are the subject matter experts and spot bugs quite quickly.


In the next post of the series I will discuss Visualization of Testing

Testing through the Credit Crunch - Part 3 - Virtualization

So far in the series we have learnt how to save money by becoming more Agile and by implementing Test Driven Development. The move to these can take a little while to implement because of the learning curve involved and/or the inability of some people to move out of their comfort zone.

This post is going to discuss virtualizing our test environments. Virtualisation is fast becoming the norm in data centers because it allow infrastructure managers to build up disaster recovery machines fairly quickly. They can also utilize the hardware of the host more efficiently. Good thing when everyone is trying to become 'greener".

So if the infrastructure people think its a good idea then it must be a good idea for testers. Getting real machines for testing is expensive, not only in terms of the hardware and software, but also the human resources to maintain them. Think about having 10 computers with 10 Windows licences that then need to be maintained. Having a quick look at Dell for a basic computer it would cost at least £2990 for the 10 computers and windows licences and just to make it a round number lets say that it will take £100, £10 per machine, to maintain them for a test cycle.

So we now have a cost of at least £3000 to get all the computers and maintain them for one cycle. That doesn't include the electricity to run all of this hardware. Since its the credit crunch we don't that amount of money to waste. The best thing to do is to virtualise the whole lot.

Spend the same amount of money on a low range server . So now we only have one machine that needs to be maintains. Maintenance cost goes from £100 down to £20. I say £20 because maintenance on servers can be a little more because the hardware is slightly more expensive. http://www.vmware.com/go/calculator will give you a better calculation. I did it and for 500000 machines, defaults in the calculator, I saved £350000+ over a time frame.

So whats the best Virtualisation software available? This is a very hard question to answer. I am big fan of free software so use Microsoft Virtual Server 2005 and VMWare ESXi. I have heard good things about XEN if you have a Linux. The main thing is to chose an application that you feel that you can work with quite well. Both MS Virtual Server and VMWare ESXi have APIs that allow you to manipulate the machines that are running on it. This means that you can clone machines and create a new machine by just running a script. As with Test Automation you won't see the saving straight away but it will save time, hence money, in the long run.

In the next post I will discuss visualisation of testing

Testing through the Credit Crunch - Part 2 - Test Driven Development

In the last post of this series I discussed briefly about Technical Debt and how Test Driven Development can help in dropping the debt and dropping the amount of interest gained over the life of the product.

Test Driven Development is the concept that you write your tests before you write your code. Once your code is finished you refactor with the safety that your newly created test will prevent any stupid bugs that you could have introduced. It's what the TDD community call Red-Green-Refactor. Now to most people writing tests first sounds stupid, "How do you know what your code is going to do until you have finished coding it?"
My retort tends to be "You know what you are expecting back from the call. Your test needs to be really small and since an Assert is a test, Assert what you are expecting your method to return! Image that your code exists and test against it." Below is an example of a small test and when writing tests we should try keep them as small as we physically can.

[Test]
public void creditcrunch_Test(){
int FutureTechnicalDebt = 100;
Assert.LessThan(FutureTechnicalDebt,TechnicalDebt.Current);
}

So we know that we need tests first to give us a good safety net for writing our code.In Technical Debt terms we have spent a little less to create/run and report on the test and we prevent anyone else breaking our code. Cost-- and Interest-- which is what we want. Running the test above will fail because there is no code to run it against. Write some code to make it pass if you have the time.

The other reason why developers are starting to like TDD, other than the removal of the ear ache that testers would give them about throwaway builds, is that challenges them to think about the interface into their new object. We have all sat down with the keyboard in front of us, tea/coffee next to us and found that someone has changed the interface your were developing against! Queue the expletives! $*@%*$£ !@)@)£)&! Luckily for us we have good Test Driven Developers who have have discussed the interface and we by proxy prevented interest being added to our debt. Interest--! Communication is key in Test Driven Developments, especially when it comes to interfaces because there may be some integration bugs that will creep in without you realising!

Knowing what the Interface is going to look like is has another major benefit. Mocks and stubs! If we know what the interface to an assembly is and how its going to react we can write tests to test our code and when it needs to speak to the other assembly, which may not have been created yet or makes our tests run slowly like accessing a database or filesystem, we can return the results back that we want our code to handle using one of them. This could be exceptions,dodgy data or good data. Testing those code paths means we can increase our code coverage therefore increasing the size of our safety net! It will also mean that our tests run a lot quicker because we don't have to interact with slow drives or have anyone elses tests interfere with our data. Cost-- and Interest--.

So by creating this safety net we have prevented stupid mistakes creeping into our code. Its these stupid mistakes that could cost our team/company a lot of money. Since companies don't have much money it could mean your job!

The graph below shows how by doing the, what I call, simple things we have brought down the technical debt that we could potentially owe. NOTE: The values used are just arbitary values I found on the internet with a little searching.

In the next post in the series I will discuss Virtualization and what the it can mean to you as  a tester!