GTAC Day 2

Day 2 of GTAC started out with something that always fascinates me in the computing world. Something more than test automation or the user experience that users are given. The day started by a talk given by Dr Alberto Di Meglio about grid computing.

Opening talk for Day 2 - Dr Alberto Di Meglio

Alberto works for CERN working on the ETICS project. This is part of a project to get Grid computing working so that scientists can use it to complex calculations. This grid is also going to be use by many other scientists around the world use the E-sciencE grid. Alberto started explaining that with new technologies, like the LHC, we need something to process a large amount of data and we need to do it fairly quickly. The LHC can produce in the region of 15 Petabytes worth of data.

All of this work needs to be done and needs to tie in with the emerging standards that are appearing for Grid computing. The next challenge that they have seen is that they need to find a way of doing Quality Assurance on the system. ETICS is a continuous build, test and QA verification system. This system is designed so that developers can work in many different languages but they all access a standard API to get builds and tests done. ETICS is developed against a QA model called A-QCM, which stands for Automated Quality Control Model. They also implement a number of ISO codes and have submitted how they do their work to potentially become its own ISO code.

Tests are done in a multinode environment. This environment is built dynamically for what is needed and when it is needed. This is done by creating VM images on the fly because maintaining a set of VM Images can be quite the prohibitive in maintaining all of this. He also mentioned that there has been a major shift in culture with the ETICS project because the scientists that develop now do so thinking about the quality of their code.

A very enjoyable talk about the issues of grids.

Testing Applications on mobile devices - Doron Reuveni

Doron, the CEO of uTest, started by explaining what his company does and how goes about doing these things. He talked about the story from England where an entire village tried to guess the weight of a bull. Not one person got the correct value but the average of all the entries worked out the weight of the bull to within 2 Grams. This is called the Wisdom of the Crowd.

He then started explaining about the differences between crowdsourcing and in house testers or outsourced testers. He said that crowdsourcing fitted in somewhere in the middle between in-house and outsourced. He then also started describing the differences between scripted manual testing and exploratory testing. He said that a lot of the uTest testers fell in the exploratory testing realm. These testers are really good at finding the edge cases that no scripted tester would find and that they are a lot more creative.

Doron said that all of this is really good when testing mobile apps because the smartphone that will win the market will be the one that has the best apps and has the best entry to market for these apps. He said that the crowd is really good with mobile apps because they can use a lot more providers, phone types and data types than an in house team could.

It was a good insight to crowdsourcing

JSTestDriver - Jeremie Lenfant-Engelman

Jeremies talk was discussing the new JavaScript Testing framework that he and other Googlers have created. The new framework allows people to develop JavaScript in a Test Driven Development environment. The new framework was designed with the need to make tests run really quickly in the same way that JUnit tests do and also to be run in the same way. It needed to remove the need to move from an IDE to a browser and press refresh.

Developers like to see that their code is doing a lot of work and that it is doing this work quickly. JSTestDriver does this by having a server that captures browsers and the browsers then poll the server very regularly to see if there are bits of information out there. When there is it runs it and reports the results. This means that we can either run this really quickly or we add this to a continuous integration server.

It also has a very good code coverage model that is being developed so that we can see what code has been executed. Overall it is a very good framework and since it came out I have been playing with it and have even submitted a number of patches. This is definitely something that we need to watch in the future.

Selenium: to 2.0 and beyond - Simon Stewart and Jason Huggins

This is probably the most anticipated talk of the entire conference. It was quite funny to see the entire conference room get all filled up just before the start of the talk. Jason and Simon started talking about the benefits of Selenium and the benefits of web driver. It is great for doing the work that you want when testing the browsers. They then discussed the different issues with doing this as well.

So the solution? Well to merge the two frame works and make one "Uber" framework. Simon and Jason have been doing a lot of work to get the two development branches merged resulting in the code now being moved over to http://selenium.googlecode.com and the next step was to start merging the bug tracking for this. They are going to be creating a hybrid system that allows developers to write WebDriver tests and run them in Selenium and vice versa.

Looking at Selenium 2.x there is going to be a lot more work on browser support for Webdriver natively and work to get it doing a number of this so that we can move away from the "Uber-jar" that is going to be released shortly. There was also a talk by a guy from Opera, who's name I can't remember, who said that they use WebDriver to help them do all of there rendering tests for new versions of Opera, including mobile versions.

This was a very entertaining talk.

Score one for Quality - Joshua Williams and Ross Smith

This was one of the talks that I was looking forward to for the entire time in GTAC. Joshua and Ross, who both work on Windows 7, talked about how they introduced a number of games to do quality assurance. They started out by showing that some people are Actively Disengaged with there job. This means that they don't always enjoy what they are doing and how they are doing it. They then started introducing games into the work people did but they needed to work out a few things like working out what would excite people. They went through a number of games that testers and developers would play until they found the right mix.

During their talk they even got the group involved by doing a competition for who had the "Best Bug Story" and sharing that with everyone. The showed with this that people are competitive and want to compete all the time and its a natural thing. They also mentioned that new developers and testers are coming from the Gaming Generation so love playing games all the time.

The said that we need to do a lot more to get people engadged with their work and they need to start feeling like they are controbuting to this.

A very good talk all in all from the Microsoft guys!

Automated Performance Test Data Collection and Reporting - David Burns and David Henderson

This, unfortunately, you will have to wait for a seperate blog post for this.

GTAC Day 1

Today was the first day of Google Test Automation Conference. There have been some really good talks and below I am going to give my thoughts on the talks.

Keynote by Prof. Nicklaus Wirth

Prof. Nicklaus Wirth opened the first day of GTAC. Prof. Wirth is a winner of the Turing award and currently working at ETH Zurich. Prof. Wirth worked through the years of Computer Science showing how things have improved over the years. he explained that through those years you couldn't just pull up a debugger and find the issues.

Prof. Wirth also used the famous Dijkstra quote that "Testing shows the presense of bugs, not the absence of them" so we can't just assume that we are making high quality code. He also had a complaint that universities are not teaching programming to the students. He likened computer programming to playing the piano, its easy to learn to play with 2 fingers but to play with all 10 fingers is extremely difficult so its easy to create programs but hard to make really good applications.

Precondition Satisfaction by Smart Object Selection in Random Testing - Yi Wei and Serge Gebhardt

Yi and Serge did a really good talk about creating random objects to go through testing objects. They were talking about how they go about generating random objects and how they are used within testing.

They talked about how they generate lots of objects and generate lots of object very quickly so they needed to work out what they needed exactly so started working out what objects meet the pre- and post-conditions.

They then started talking about how the objects start working in tests and were left to run for many hours. They found over 500 bugs found in 2 open source libraries by applying this random object testing. They found minor bugs to bugs within a lexer which can be quite scary.

Fighting Layout Bugs - Michael Tamm

Michael Tamm gave a very good presentation on how to fight layout bugs. Michael started by talking about all the basic things that we can do and should we be applying them to the our continuous integration servers. This can be done by collecting the page structure,parsing it and passing it to W3C. This can be done for the both HTML and CSS. Unfortunately CSS validator can be too strict and doesn't like browser specific styles.

Michael then moved on to a project that he has been working on about an Open Source project that he has created. This can be found at http://code.google.com/p/fighting-layout-bugs. This project works by firing a WebDriver instance and then takes a number of screen shots and does a bitwise operation to work out where things are overlapping. It was a very a good presentation and will be watching this project very carefully.

Even Better than the real thing - Lessons learned from GWT application - Nicolas Wettstein

Nicolas gave a very good talk about Google Web Toolkit. He talked through all the different issues that can be found when developing software. He showed what can be really bad code examples and how this can be cleaned up.

Nicolas also talked about that there should be a very good design structure such as MVC/MVP. That way people can start creating very good unit tests because a developer can then fake out the right items. Favourite quote from this talk is "Developers are responsible for the quality of the code"

Automatic Workarounds for web applications - Alessandra Goria and Mauro Pezze

This talk was a very interesting talk about having a system that sat between the browser and the backend server. The have developed a system that when a user encounters a bug you can tell the proxy and it will go off and find a solution from a repository.

They were using bugs in the Google Maps API bug tracker and analyzed the issues and if there were any potential workarounds apply them to the page. Unfortunately this is very academic work and does not meet any of the potential psychological needs of users. It also doesn't differentiate between bugs that could be happening.

The talk was very good but academic so don't think it will ever become a real world application.

Achieving Web test automation with a mixed skills team - Mark Micallef

Mark gave a very good presentation on how to get Test Analysts and Test Engineers working very well together. Mark was showing how you start with a team with lots of tasks but they are all manual and how, with the right tools, you can get them writing basic tools.

Mark said that he got the teams using Ruby and Cucumber to start doing basic ATDD and BDD with the test analysts and then getting the test engineers to fill in the technical aspects. Mark did this by seeing what motivated the team and trying to work with their strings.

Mark said that this is the start of the work but the BBC still has a lot of work to do! It was a very good presentation and makes me wonder what motivates testers out there.

And finally...

The lightning talks were really good. Everyone was very entertained by the talks. When they are out on YouTube I recommend everyone having a watch of them.