ISEB/ISTQB - Is it really worth it?

I have been reading James Bach's blog for a while and in the latest post he has commented on Alan Richardson's commentary on ISEB/ISTQB.

Finally I have someone that agrees with me that ISEB/ISTQB while being good in theory is pretty useless in real life. I know that most people reading this will be saying "That is a very bold statement!". I know that it is and below I will explain why I agree with Alan.

Background
Testers have been fighting for years to remove the stigma that testing software is easy and that any person can do it. This, I feel, is because software testers used to be glorified data inputters with a little technical background. No wonder developers saw a secondment to testing as a demotion. Then Kent Beck came along and wrote Test-Driven Development: By Example. A developer who came along and made testing fun for developers! Something that testers could never get right! So lets have a look at the pros and cons of having a ISEB/ISTQB certificate.

Pros

• Lets start why I think ISEB/ISTQB is good. I just said that testers have been fighting for years to remove a stigma that testing software is easy. ISEB came along and people who didn't work in the industry suddenly thought that testing was a little harder because you needed a certificate.
• It also sets out standard names for things so that if one tester speaks to another they are speaking in the same tongue. Its also the latest buzzword so HR people will more than likely to give you an interview when you apply for a job.

Cons

• The ISEB/ISTQB foundation course teaches the exact same things that I was taught during my first year of university. So not only do you have to pay your way through university but then have to pay from £140 to £700 to get the certificate.
• I met a ISTQB examiner in my last job and he did not seem to understand testing in the real world. So are the testers that are getting these certificates also getting a working understanding of testing? I honestly don't think so!
• The other thing, and this is a big thing for me, is that the foundation course book mentions Agile Development 3 times. Those three entries are no more than a quick mention. Is it worth getting a certificate that doesn't give a chapter what is becoming the most common development methodology.

Results
ISEB/ISTQB is unfortunately going the way of the the MSCE certificate of the late 90s. Its becoming no more than a buzz word and I feel is lowering the standard of testing. I know that I am not the only person who thinks this. I read a blog entry not so long ago where testers in India are seen as the poor cousins. Something that we've been working against!

This has to stop and we as testers are the ones who can stop this! Come on testers, lets start a revolt and get certificates to mean something!

Selenium Grid - Why is this the way of the future?

Update: If you are looking for a Selenium Grid Tutorial I have put it here

I have been working with Selenium Grid since Philippe Hanrigou and others at ThoughtWorks released it at the end of last year. I knew then that it was going to become the standard way for automated GUI web application testing.

As an application it has great potential since it is has all the best bits of Selenium but it does multiple browsers in parallel. Suddenly your testing that used to take n units of time to run are taking 1/n to run. As a practitioner you can get things done quicker, as a manager you can get things done for cheaper! Everyone is a winner.

The other thing that makes the grid work well, and this is a thing that makes Selenium Remote Control work brilliantly, is the ability to use any language to control the test. People use Java, Ruby, .Net, Python and many more to create their test scripts and run it with the appropriate client. If you have the appropriate client application like TestNG for Java or DeepTest for Ruby or PNUnit for .Net you can start running tests with one script in parallel. The one limitation toof using these test tools is that you only really have one browser application, like Firefox, being tested at a time. This is because you only have one instance of the Selenium object in your code.

The way that I have got around this is to have many instances of the Selenium object in my code that each control a certain type. This type could be Internet Explorer 7 on Vista or Firefox 2 on XP.

An example of this would be (Note: I use c# to create my tests and then run them with NUnit)

[Setup]
public void setup(){
//Vista IE7 Instance
VistaIE7selenium = new DefaultSelenium("hub", 4444, "IE7 on Vista", "http://testserver");

//XP Firefox2 Instance
XPFirefox2selenium = new DefaultSelenium("hub",4444 , "Firefox2 on XP", "http://testserver");
XPFirefox2selenium.Start();
}


[Test]
public void GridTest(){
VistaIE7selenium.Open("/index.html");
XPFirefox2selenium.Open("/index.html");
}

As you can see it has the ability to create multiple instances of Selenium and then run them. This then leaves the handling of the Selenium Remote Control up to the Grid. As Philippe says, "It allows you to gridify your tests".

The negatives of Selenium Grid is that its still a work in progress. I know, thats hardly a reason to call it a negative. There are also a number of bugs but its growing section of OpenQA.org so give it a chance! I have 7 Selenium Remote Control instances running against my Grid (all virtualised with MS Virtual Server and WMWare) and I haven't had many issues. I have hooked Selenium Grid into our CruiseControl.NET server to test all the different browsers. I still have a lot of work to do with this but I know that it will be worth it in the end!

If you want to automate web testing and you want to do it efficiently use Selenium Grid! It will make your life a lot easier!

What makes a good automated test?

I have been asked to try automate the testing of a wider range of for the group. This is going to be quite challenging since I have not automated desktop applications for about 15 months and the software we make is quite unique. I will also hopefully be teaching the other testers how to maintain the automated tests and eventually create their own.

This challenge has made me think back on my automated testing experience and ask "What makes a good automated test?". A good automated test has all the hallmarks of a good manual test except that its comparatively quick and cheap to run. This means that an automated test is successful when it finds bugs and repeatable where ever the software is installed.

In terms of web testing it would be a test script that can be created and then stored in a central place and be run from the central place. Ideally this would be also run on the continuous integration server when doing builds.

The thing that makes a good automated test great is scalability. I recently watched a YouTube video of the Selenium Users Meet-up. There were Google staff there saying that their testing farm handles 51000 tests and most Friday's all of these tests run. They then went on to complain that Selenium struggled to scale to that size. To be honest at that size I am not surprised!




P.S. To those who are waiting for the next Selenium Tutorial I am hoping to have it complete in the next week for everyone to use.

What would people like for the next Selenium Tutorial?

Thanks to everyone who has been using my site and all those who are subscribed to my RSS Feed. I haven't released another tutorial for couple of weeks and have started thinking of what would people like next for a Selenium tutorial.

I am struggling to think what people would really like. I would appreciate it if people would take the time to let me know, via the comments section of this blog, what tutorial they would like next. Please don't think that your questions are too novice. I would like to help as many people out as possible!

I look forward to your replies!

Is documenting test processes a bad idea in an Agile Development Environment?

I work in a smallish development team in Southampton as some of you know. There are other development teams in the group in United Kingdom and in France that all fall under the same group of companies of different sizes.

The unfortunate problem that we have is that there isn't a properly documented process for getting everything in place and ready for each testing cycle. Once we have everything ready we have another problem that I will hopefully discuss in a future posting.

This is where I had the thought "Is documenting test processes a bad idea in an Agile Development Environment?". If you have worked in an office with ISO9001 accreditation you will know that everything needs to be documented or if you have worked in a Six Sigma office you will have noticed that you have to put everything you do in a database so the "Black Belts" or "Master Black Belts" can see areas that need improvement.

So back to my question, "Is documenting test processes a bad idea in an Agile Development Environment?". My answer is definitely! And I am going to admit this is a fairly biased answer having started my career in large financial institutions where I was taught that everyone is expendable but their process knowledge isn't. This means that everything needs to be documented. However my thoughts on this is that things need to documented and followed to a point as long as that point does not impact on the creativity of the software developers and the testers.

Its this creativity that allows people to come up with new and exciting ways of developing and testing. Tools like the xUnit and Selenium and many more have come from this creativity. This then needs to be balanced with some form of Total Quality Management (TQM) so that everything can be reproduced if and when needed.

Documented processes can't be completely wrong, if they were then large companies like IBM, Citrix and Microsoft would have never got to where they are now!

What has changed in the last year at work

Not that long ago it was my year anniversary with my employer. As with most people it just seemed to blow over but I have had a chance to reflect over what
changes there have been in my time there. The one thing that I can definitely say is that I has been a good year.

When I started there was no testing team which means that there were no definitive test processes and no definitive automation testing.

There was however a team of developers eager to make sure that they delivered a
good quality product.

In the last year I have managed to automate around 40% of the site with Selenium. This has then been installed on the development machines so that the developers can have a

way of developers to unit test thier front end code.

I have managed to get the development team to start using NUnit to create their unit tests instead of developing their own applications to unit test their work. This has meant the number of bugs that get into the builds has dropped dramatically.

These two changes have created a mindshift for all of the developers to a more
agile development cycle. This shift has not been that difficult and the reason
for this is the management team that I report to. They are always open to ideas
especially ones that save money. And agile development on web projects
definitely has this this appeal.

So next thing in my plan is to create an integration server so that all the
different development teams all over the UK and France can check in their work
and be confident that their changes will work on another environment.

Graphical Test Planning

In December I went to conference in London and one of the lectures that I went to listen to was about Graphical Test Planning by Hardeep Sharma of Citrix Systems.

Graphical Test Planning is a way of creating a test plan without having to write entire documents. I am sure that most of the readers
that will be reading this will be practise full agile development and are probably
thinking that I am mad.

Maybe I am mad but I really think that this is a great way to work out what needs to be testing and
is a good way to track progress of testing.

The entry below is more my thoughts having tried this technique on web projects
and have used a web examples. I hope you enjoy reading it!

Graphical Test Planning is done by creating a structural relationship diagrams.
These
allow product managers to know what is going to be tested from the day that they decide there is going to be a new version of the product.

What a Graphical Test Plan is or isn't

What a Graphical Test Plan is:

• A structured relationship diagram


• A list of behavioural areas that need to be tested


• A method of getting greating feedback about what needs to be tested


• A method of tracking progress of design and execution of testing

What a Graphical Test Plan isn't:

• A flow diagram


• A tester managers thoughts or mind map


• A feature list


• A hand-holding exercise for a test engineer on how to use the application

How To Create A Structured Relationship Diagram

A structured relationship diagram will show what behavioural areas need to be
tested on different platforms and different compilers. Below is an example of
what a SRD looks like for Selenium.

Its not 100% correct for selenium but I wanted to show something
with no coverage.

From this you can see what areas need to be tested and what doesn't. If we carry this theme on we can then
start drilling the behavioural areas down into more detailed SRDs and if we were to
carry on drilling down we will start to create another form of these diagrams
called a Test Case Diagram.

A Test case diagram looks a lot like a SRD except at the end of the digram instead of showing the behavioural areas it will show the expected results.
The great thing about this is that you can create the SRD in about 10 minutes and then do the test case diagrams in a few hours and this means that you
can have all this test information done before any code is done.

So if we use this metric to create an entire test strategy for a project using graphical test planning compared to the waterfall method of doing things

you would have saved yourself somewhere in the region of 1 man month and you are likely to have fixed a number
of flaws in the design before any code is near completion.

I hope that you found this very interesting and I would like to the thank

Hardeep Sharma for showing off this technique at the SIGIST Conference in
December.