Monday, December 04, 2006

The Temptation of Free and Easy to Use Test Tools

&otAfter recommending those nice open source test tools such as Watir and Selenium to my colleagues @work (and to my boss of course) for quite some time, I begin to see certain dangers which come along with the availabilty of free high quality test tools. This fact combined with all those success stories all over the net, which state how easy to use those tools are or how fast a newbie could learn Ruby (which would be required in order to use Watir) seem to lead people to the wrong conclusion test automation was a simple task.

In the last view weeks I have seen several posts on the Watir mailing list from people asking how to read/write data in/from external files, how to get all the links on page (and after getting the answer they ask the same for all buttons...), asking what happens if you do $some_ruby_statement_here instead of trying it in irb or reading the (very fine) documentation. The attack of the newbies has begun ;-)

Don't get me wrong, I like the idea of test automation becoming more popular and I'd like to see more testers trying to improve their work with some automated testing. The point is, test automation has always been a serious engineering task. we all have seen test automation projects fail because people were not aware of that fact. Where is the difference in the vendors telling you record&playback is easy and works perfectly for your needs and "us" telling open source test tools and languages like Ruby would be well suited for beginners?

Many organisations didn't use test automation because of the high costs of a commercial test tool and lack of skilled personel to use them. Now with one of those restrictions gone some people seem to think of open source tools as the new and improved record&replay. Test automation with a free tool is still the same challenging task as it was with a commercial one.

There are some points which people who are new to test automation and start using Watir for example should keep in mind:

  • Ruby is not a good language for beginners! (im curious how Brian Marrick's forthcoming book Everyday Scripting with Ruby will prove me wrong on this one) It may be easier to learn than C or Perl but there are some very advanced concepts involved in Ruby and its syntax is huge compared to that of Python or VB. On the other hand if you are an experienced programmer Ruby is a very nice toy to play with - That's for sure.

  • You have to learn a few things about software engineering in order to be a good test automation engineer. Not to mention reading the documentation, asking your friend Google and finally use your own brain ;-)
The examples from the Watir mailing list I described before are very similar to experiences we have made with offshore test automation engineers in India. Many of them (not all, but many of them) dont have a deeper understanding of how test automation should be done and somehow they prefer to ask their team lead everything they could have googled or found in the documentation of the test tool on their own.

I believe the point im trying to make is: no matter how nice, sophisticated or "easy" those tools are and how much we like to encourage others to try them, we should never forget: a fool with a tool is still a fool...

People like Cem Kaner, James Bach, Bret Pettichord, Elisabeth Hendrickson and others have done a terrific job in educating testers and test managers about "good" test automation - Testers who take the new free test tools as the new record&playback will fail and we test automation engineers can start education our test managers and customers again.