Saturday 20 February 2010

Process.Start() issues with Mono on Mac OS X: My Story

Over the last month or so I have been spending my time writing .NET code for the new Selenium 2 Bindings. I have spent most of my time working on the Google Chrome Driver. But below is the story of 1 Selenium bug and 1 Mono bug and my exploration of how processes work in .NET on a Mac.


I was quite confident about the .NET bindings since it was extremely easy to get them working on my laptop. I admit I got nervous about how it would work on Non-Windows platforms since it was my first time writing .NET code on a platform that isn't windows. I shouldn't have been nervous. It was fairly easy to install Mono and MonoDevelop on my Ubuntu VM and ran the NUnit tests that we have. Google Chrome started and we had tests passing. Brillant because there was little code changes, I just updated the code where to find the Google Chrome application.

I was confident until I started working Mac OS X. I have very little experience on a Mac. Other than using it to run Selenium Grid or using it with a Browser I know very little about how the OS works.

On the Mac I installed Mono and MonoDevelop and fired them up. I went in and updated the code so that if the .NET code was running on a Mac it knew where to find the Google Chrome application and would then start up the process using the standard Process.Start() method from System.Diagnostics library. Feeling extremely confident I fired up NUnit and clicked Run. They all failed. Why? Well the browser wasn't loading. Selenium without a browser, excluding when used with HTMLUnit, isn't going to work.

I started debugging my code to see why the browser wasn't loading. The first issue that I found was the Selenium 2 .NET code wasn't recognising the Mac OS platform. We have logic that checks which platform you are on and we hadn't put a conditional for Mac OS. Updated my code and ran the tests again and they went red again. Why? No browser again.

Debugging again I came across a rather large and very annoying bug in Mono. Mono when running on Mac OS platform reports that it is running on Unix/Linux. This isn't too ridiculous since Mac OS is based on BSD however it means now that when ever the code needs to launch the browser it needs to up to 3 hits to the disk to see if it is where I think its. It also means the fix I added in Selenium earlier doesn't really matter. Anyway the code now has the ability to launch the browser...or does it? It doesn't as my tests are still failing.

I started debugging again. The error I was getting was that I was trying to open the application and open wouldn't accept the arguments I was passing in because they are not valid for Open. Since I have never developed on a Mac I had to do some research and found out that the defaults for Process.Start() want to Open the file rather than execute it. This is perfect if you want to open text files but not applications. So I tried to see about creating a bash script to start up Google Chrome. One obvious downside if it did start up was that I wouldn't have an easy way to kill it. I was starting to think that I was going to be doing Yak Shaving to get this to work.

I next had a look at using Monobjc. It is a library that allows you to write Cocoa code in .NET and there it uses a bridge to get your code working. It is something that one of the Watin guys had done to get Watin working on a Mac. After a lot of pulling my hair out for a while I gave up and asked a question on StackOverflow.

The 5th answer I was given, and the one I accepted, was that I had forgot to set a property called UseShellExecute. My code for starting up went from

chromeProcess = Process.Start(GetChromeFile(), String.Concat(Arguments, serverUrl));

to

chromeProcess = Process.Start(new ProcessStartInfo(GetChromeFile(), String.Concat(Arguments, serverUrl)) { UseShellExecute = false });

The subtle change of explicitly creating a new instance of ProcessStartInfo and updating the property { UseShellExecute = false } meant that my code went from opening an application to executing a application. The difference tells Process not to open a shell according to MSDN but on a Mac means it does what I want. I hope this is helpful for anyone else in the future.

kick it on DotNetKicks.com

Thursday 4 February 2010

Why should I move to Selenium 2?

As most of you know I am a really big fan of Selenium for testing of Web UI and as of Tuesday I joined the development team working on the next release of the Selenium. I am helping Jim Evans on the .NET bindings.

Selenium 2 is the merging of Selenium and WebDriver to make the whats going to be the best Web UI test framework out there.

Selenium 2 will help both WebDriver and Selenium get over their respective weaknesses. So, for those who have a decent coverage and your tests have been running for a while why should you think about moving your tests to Selenium 2?

Well I will explain why I have started moving my tests. Firstly, I love to dogfood my own code so writing my tests was a natural progression. Secondly, the ability to write really verbose tests without having to try really hard.

I have started moving my Internet Explorer Selenium Tests to use the WebDriver API to test the Editor component of our site. This is one of the hardest areas to automate with Selenium 1 because its using contentEditable on a div. I managed to test this using the components JavaScript API to inject HTML and then manipulate it and then use the API to get the HTML out again and check that it is correct. Running 1 test in Internet Explorer for this would take up to 90 seconds to complete. I would like to add a disclaimer that the test had been optimized as much as it could with no XPath, etc so it was running as fast as possible

Now to start porting the tests over to the new API. After working out the little issues in how to get to certain elements I was able to port over a number of tests. I was able to port over 4 tests in a little bit of work and then thought to run the tests. Any guesses to how long it took to run all 4 tests?

360 seconds?
270 seconds?
120 seconds?
60 seconds?
30 seconds?

The answer is 60 seconds. Well it was 57 seconds but I needed to pick from the list I gave. 4 tests in 2/3 of the time to run the original test to check for the exact same thing. The new test have very little JavaScript pokng in the test because I use the native keystrokes to type into the div where I could not with Selenium 1. This means that the tests do not have to handle the JavaScript scope chain to do basic things and makes the tests lightning fast.

The next test will be to port the WebDriverBackedSelenium code from Java to .NET and then run the first tests again to see how much faster those tests run. So if you want really fast tests then start thinking about moving to Selenium 2.

p.s. If you haven't filled in my questionaire on Selenium 2 presentations and demo's please do so here