Testing through the Credit Crunch - Part 3 - Virtualization

So far in the series we have learnt how to save money by becoming more Agile and by implementing Test Driven Development. The move to these can take a little while to implement because of the learning curve involved and/or the inability of some people to move out of their comfort zone.

This post is going to discuss virtualizing our test environments. Virtualisation is fast becoming the norm in data centers because it allow infrastructure managers to build up disaster recovery machines fairly quickly. They can also utilize the hardware of the host more efficiently. Good thing when everyone is trying to become 'greener".

So if the infrastructure people think its a good idea then it must be a good idea for testers. Getting real machines for testing is expensive, not only in terms of the hardware and software, but also the human resources to maintain them. Think about having 10 computers with 10 Windows licences that then need to be maintained. Having a quick look at Dell for a basic computer it would cost at least £2990 for the 10 computers and windows licences and just to make it a round number lets say that it will take £100, £10 per machine, to maintain them for a test cycle.

So we now have a cost of at least £3000 to get all the computers and maintain them for one cycle. That doesn't include the electricity to run all of this hardware. Since its the credit crunch we don't that amount of money to waste. The best thing to do is to virtualise the whole lot.

Spend the same amount of money on a low range server . So now we only have one machine that needs to be maintains. Maintenance cost goes from £100 down to £20. I say £20 because maintenance on servers can be a little more because the hardware is slightly more expensive. http://www.vmware.com/go/calculator will give you a better calculation. I did it and for 500000 machines, defaults in the calculator, I saved £350000+ over a time frame.

So whats the best Virtualisation software available? This is a very hard question to answer. I am big fan of free software so use Microsoft Virtual Server 2005 and VMWare ESXi. I have heard good things about XEN if you have a Linux. The main thing is to chose an application that you feel that you can work with quite well. Both MS Virtual Server and VMWare ESXi have APIs that allow you to manipulate the machines that are running on it. This means that you can clone machines and create a new machine by just running a script. As with Test Automation you won't see the saving straight away but it will save time, hence money, in the long run.

In the next post I will discuss visualisation of testing

Testing through the Credit Crunch - Part 2 - Test Driven Development

In the last post of this series I discussed briefly about Technical Debt and how Test Driven Development can help in dropping the debt and dropping the amount of interest gained over the life of the product.

Test Driven Development is the concept that you write your tests before you write your code. Once your code is finished you refactor with the safety that your newly created test will prevent any stupid bugs that you could have introduced. It's what the TDD community call Red-Green-Refactor. Now to most people writing tests first sounds stupid, "How do you know what your code is going to do until you have finished coding it?"
My retort tends to be "You know what you are expecting back from the call. Your test needs to be really small and since an Assert is a test, Assert what you are expecting your method to return! Image that your code exists and test against it." Below is an example of a small test and when writing tests we should try keep them as small as we physically can.

[Test]
public void creditcrunch_Test(){
int FutureTechnicalDebt = 100;
Assert.LessThan(FutureTechnicalDebt,TechnicalDebt.Current);
}

So we know that we need tests first to give us a good safety net for writing our code.In Technical Debt terms we have spent a little less to create/run and report on the test and we prevent anyone else breaking our code. Cost-- and Interest-- which is what we want. Running the test above will fail because there is no code to run it against. Write some code to make it pass if you have the time.

The other reason why developers are starting to like TDD, other than the removal of the ear ache that testers would give them about throwaway builds, is that challenges them to think about the interface into their new object. We have all sat down with the keyboard in front of us, tea/coffee next to us and found that someone has changed the interface your were developing against! Queue the expletives! $*@%*$£ !@)@)£)&! Luckily for us we have good Test Driven Developers who have have discussed the interface and we by proxy prevented interest being added to our debt. Interest--! Communication is key in Test Driven Developments, especially when it comes to interfaces because there may be some integration bugs that will creep in without you realising!

Knowing what the Interface is going to look like is has another major benefit. Mocks and stubs! If we know what the interface to an assembly is and how its going to react we can write tests to test our code and when it needs to speak to the other assembly, which may not have been created yet or makes our tests run slowly like accessing a database or filesystem, we can return the results back that we want our code to handle using one of them. This could be exceptions,dodgy data or good data. Testing those code paths means we can increase our code coverage therefore increasing the size of our safety net! It will also mean that our tests run a lot quicker because we don't have to interact with slow drives or have anyone elses tests interfere with our data. Cost-- and Interest--.

So by creating this safety net we have prevented stupid mistakes creeping into our code. Its these stupid mistakes that could cost our team/company a lot of money. Since companies don't have much money it could mean your job!

The graph below shows how by doing the, what I call, simple things we have brought down the technical debt that we could potentially owe. NOTE: The values used are just arbitary values I found on the internet with a little searching.

In the next post in the series I will discuss Virtualization and what the it can mean to you as  a tester!

Testing through the Credit Crunch - Part 1 - Development Environments

The credit crunch has gripped the world and not so long ago Gartner suggested that the investment in technology was going to drop. They have dropped the increase in spending from 5.8% down to 2.3%. This may not seem a lot but when companies spend millions on IT each year it adds up very quickly. Unfortunately past experience has shown that testing departments are the ones who suffer first.

More employers are predicting a rise in redundancies in the UK and the other week a politician predicted a run on British Pound.

This doesn't mean all doom and gloom for IT professionals. Over the next few weeks I am going be writing a series of posts with some best practices that I have found. Lets start this series by discussing the Environments we build and test software.

Environments

This is a very ambiguous term so lets split it into the 2 main things I mean. Development Methodology and Testing Environments.

Development Methodology

When creating products it is starting to become common practise to follow Total Quality Management (TQM). The idea is to make sure that quality is in every process of the organisation. By moving to your company to this principle you will be adding confidence to your product to all stakeholders. More confidence in your stakeholders will mean that your department will get a little more funding and since its the credit crunch it must be good!

A lot of companies actually practise this without realising it. The main thing to take away from TQM is that everyone in the company has a part to play in the quality of the product that is delivered!

Agile or Not!

My first few jobs were at financial institutions and thought it was normal to do testing at the end. I worked as a WinRunner Automated Tester. I then started working on some Open Source projects in Java and was asked "Can we see your unit tests? Did you use JUnit?".
I answered quite sheepishly "I tested it by checking that it did what it needed. I don't have any JUnit tests."

Obviously my code got rejected because it wasn't neat and didn't have proper unit tests. There was a few other issues that they raised and to be honest if I was doing the code review I would have rejected it too!

Now I work for an ASP/ SaaS and we do a little Agile. I have been able to see the value of being agile while developing software. The main thing that I have learnt is that Technical/Design debt is something that is inherently low in an Agile Evironment.

Technical debt is the idea that when you release software you go into debt. If you have cut corners it will affect the level of debt you have. If you leave these items in your code it will start gaining interest on your debt. So if you have any TODO's in there that is interest on your debt. If you have poorly designed interfaces then you have added interest on your debt. Putting testing at the end of a project will definitely add interest to your debt! Doing tests at the beginning of the development process will bring down the "interest" accrued against your debt in the development process.

So to help us get through this credit crunch we need to bring down the debt in our software. The best way to do this in my opinion is to achieve this is to move your development team to a Test Driven Development way of thinking. As I said above if you place your tests earlier in the process you will lessen your technical debt. The simple rule when it comes to testing is the earlier you test the cheaper it is to solve bugs. So putting in automated tests at the beginning of the development process will go far in bring down your actual development costs for that project but also for future developments.


In the next post I will discuss further the value of Test Driven Development.

GTAC 2008 - Day 2

Day 2 of GTAC 2008 has been quite interesting. The talks that have interested me were the talk on Context Driven Testing.

This talk, by Pete Schneider of F5, dealt with what context that tests were being created. It dealt with a common issue that testers and deverlopers have and most of them ignore. It was really interesting in seeing that they were 11 different applications that testers had created and were maintaining but there was a lot of overlap. They found that creating tests in the right context and with the right owners of tests has dramatically made a difference in getting their application tested.

The main thing that I took away from it is; When creating a testing framework ask yourself the following questions

• Who is going to write the framework
• Who is going to build and maintain it
• How are you going to use it
• How long will the tests live

The next talk that was really interesting was on Automated Model-based testing by Atif Memon and Oluwaseun Akinmade of University of Maryland. It was a really good talk on creating models for test cases, similar to my Graphical Test Strategy post from the beginning of the year. The difference between my post and this talk was the model is created on the fly by the test code that then builds a WebDriver test case. There is one draw back that I noticed and I know I wasn't the only one. This modelling works well on websites that don't have AJAX. Its a big draw back and they did mention that they are working to get it sorted.

Its value for Desktop applications is really good and I would definitely recommend it. It is able to recreate states for the test and make sure that works well and has good coverage.


The next talk was on the value of unit tests by Christopher Semturs. He was advocating the use of Mocks in your unit tests so that you can get better granularity of your tests. This is something that really interests me because I like making my tests fast because if a test is fast a developer is more likely to run the tests. He was advocating the idea of small,medium and large tests. Its an idea from some Google guys where you do unit tests in isolation and then do pair-wise integration tests. This means that you drop your large end to end tests number dramatically.


GTAC 2008 has been really cool with a lot of clever people talking about different aspects of testing and describing what is needed from testing in the future. Its definitely been worth the jetlag!

GTAC 2008 - Day 1

Day one of GTAC has been really cool. There have been a number of really good talks. I will put links to the YouTube videos when I know them.

James Whittaker's opening talk was a really good start to the day. One of the main things he talked about was the visualization of testing for the future. He discussed the way that people can visualise when there are new code differences. This can be very useful in making sure that testers can see what is new and what should be concentrated on. I am always keen on using visual ways to test and make sure that the quality is high.

The next talk that caught my attention was "Advances in Automated Software Technologies". This talk discussed the idea of Autonomous Computing in software testing and the idea that you can auto generate test cases against APIs. It is an interesting idea but the way that it was put across was that you can only use this on items where the requirements are rock solid. This is not really something that can be applied to Agile developments.

The talk on Groovy was really good. I have started playing with Groovy when trying to automate SOA testing using SoapUI. Its a nice language that can be strong typed or weak typed at the same time and really useful for scripting. Since I haven't played with it a lot I did learn a lot like you can hook into all languages that use the JVM.

The next talk that really interested me was the last talk of the day. It was on the testability of code. It was looking to see how easy is it to test and making tests simpler. This is important because it means that your tests become manageable and supportable. Vishal Chowdhary was advocating SOCK:

• Simplicity of your code
• Observe how things work and interact
  
• Control of your tests
• Knowledge of what it should be done
  

He was also saying that we need to make sure that we don't overdesign software because this can add extra complexity when there doesn't need to be!

The talks have been very good and have stimulated some ideas that I want to take back to work! I am now off to the Google Seattle office for a tour the lightning talks.

.Net Gem - How to Unit Test Internal Methods

I have recently started redoing unit tests for bit of code at work and came across a internal methods and classes that needed to be tested.

This posed a question of: how do I access the internal methods and classes to test them properly from an external assembly?

I found this little gem that a lot of the developers where I work had not heard of, this doesn't mean that its not common knowledge but thought I would share it a little more.

I have put a scenario below of how it might work.

Being a good TDD developer you decide that you want to write your unit tests for an internal method. You create your .Net Class Library structure so you know what your tests need to call to do your asserts. It may looks something like the image below.

So now you want to start writing your tests and notice that your intellisense is not bringing up your internal class. So what do you do next?

Open the AssemblyInfo.cs file that is in your Properties folder. Add the line [assembly:InternalsVisibleTo("Unit.Tests")] somewhere near the top.

Now your Internal Classes and methods are only visible to Unit.Tests assembly.

Your start writing your unit tests and intellisense should be playing Mr. Nice Guy and we should see something like below.

I hope that it has been useful!

Are regression packs still worthwhile?

Recently I was on forum and noticed the thread "Why do we do Regression testing?". In it the person was complaining about their company's regression pack and asking if it was worth the effort of fixing it?

My answer was a little quick off the mark. I replied saying "Definitely! Its the backbone for each and every build!". But then this got me thinking, How many testers out there DO NOT practice Agile development? How many testers DO NOT work in an environment where automation of tests is 2nd nature?

I do not ask this question in a malicious way, I genuinely would like to know. When I go to SIGiST there are a number of questions in the presentations/workshops by people who have not worked in an Agile environment and, to be honest, seem scared by it. These are the same people when you mention something like "Continuous Integration" and they get that deer-in-headlights look on their face.

This leads me to my next question; Is there a "Rich get richer, Poor get poorer" divide happening in testing? People in conferences mention that the lines between development and testing is blurring. It's called grey-box testing. But your average manual tester in a large non-software institution (Banks, Insurance Companies, etc) have got their "break" in testing by being seconded to the test team for a project and never wanted to leave their secondment. They may not have been given opportunities to learn how to program or may not even have heard about concepts like Agile, TDD and all the other new ways testers do their jobs.

To these people; the answer to my original question is still the same: "Yes! Definitely! Otherwise things that used to work may not work anymore. I would also suggest trying to automate the regression pack as much as possible so that you can concentrate on making testing more efficient. Try automate as much as possible so that you can concentrate on doing testing on those hard to test areas. Quality of software is still the main objective of testing."