When you deal with test automation of web pages these days, there seems to be just one player to choose: Selenium. And for all which I have seen, it is the best you can get! But still it is a pain and feels quite wrong when you need to work with it on a professional level.

Let me first make one point very clear: Selenium is a great achievement, and I totally respect the immense work and results the people around Selenium have achieved! But since test automation of web applications became such a crucial part of current software development, we really need to move on to a more sound and solid approach. Whatever that will be…

The Bliss

Let me first be more precise, I refer here to the Selenium WebDriver API, as being realized in the Selenium Server. It allows you to easily setup a distributed automated testing infrastructure, which is absolutely great.She Flashes
Together with the browser drivers, which are available for basically all common browsers, you can run your tests cross-browser, even on mobile devices. You just pick the browser and/or device, and you can run the same test wherever you want. At least in theory. The API gives you most of the stuff you need, even though you would like to do without explicit waits, scrolling up and down at the page, and such things, but I am coming to that, next.

The Pain

Waiting in Vain

Once you have written a test case using the WebDriver API, you can run the test on different browsers and devices. You commonly start with one browser, say Chrome, and then you realize that your test fails or is unstable (it sometimes passes and fails). You find out, that you need to wait for some elements to appear, to load, or whatever. So you use the several waiting operations from the API. The last thing you want is to do an explicit wait for a given number of seconds, since test execution time is relative to the executing system. But sometimes you cannot do without, or you just don’t know how. But usually you somehow manage to get your test stable by waiting enough here and there. And then you are happy. Until you run it on a different browser. Since there, it can very likely happen, that the test case fails again, or becomes unstable. Why? Because that other browser wants you to wait for different elements, events or whatever.

So you start to extend your test case to make it run stable on the second browser (and keeping it stable on the first one). And this game continues for each browser you add. And with each browser it gets a bigger mess.

Incompatible API Interpretations

Another painful observation is, that different browsers behave differently for the same API call. For instance, Firefox prefers to open the pages in a new window, whereas the default for Chrome is a new tab. You can then try to fix this with a changed browser profile, and that may work out, or it may not. Or Firefox shows you a tenacious modal dialog (which you cannot access with the WebDriver) when you leave a secure page, but Chrome is just fine with it.
DSC01266_v1So you need to fiddle around all these peculiarities, and even if it works, your test case gets more and more complex.

Unexpected Firefox Changes

In my current project we are now in the not so funny situation that we cannot run our test automation on a Firefox having a version > 47. Why? Because Selenium 2 cannot deal with it. And Selenium 3 needs the geckodriver, which is so immature, that very many test cases fail due to that. And the current version 0.13.0 of the geckodriver does not sound as if there will be a feature-complete version, soon. With each new version of Firefox the situation gets worse, and we may soon need to invest a huge effort to somehow define a new, adapted, restricted, whatever test suite for the newer browser versions. And then analyze both test suites separately, etc.
Airport Sunday Afternoon
It is hard to understand how Firefox/Selenium can make such a move, without offering any workaround, for such popular technologies so many people rely on.

The Relief?

So what can you do? First of all, it is easy to complain without making a better suggestion. And I don’t have one. If you do, please let me know! You could consider using a headless browser like PhantomJS, but trying that showed that it is equally difficult (or worse) to get stable tests. And since tests via browser automation are also usually slow, I am considering more and more to step back from those as much as possible. At least unless there is a stable, reliable, and hassle-free approach available.

One should consider if it is really necessary to run that test case via a full browser automation. Unless you really need all the JavaScript magic, etc., you may discover that it is totally fine to run your tests via plain CURL, or via the functional testing interface of your PHP framework. Luckily, tools like Codeception offer these options in addition to Selenium. And you may even use the same test case here as for the WebDriver case. So doing makes your tests stable and fast. Reducing the test cases you run via Selenium to a minimum definitely pays off a lot.

Summarized, web test automation is a hassle. And a real relief is only available for a subset of the technologies and testing goals you may need. It would be great to have an API available which is not defined on the abstraction level of WebDriver, so it can do without most waits, and is interpreted the same way by all implementations.
Berlin in Canonet 2
If that is a realistic wish, let’s see.

1 Trackback

Trackback specific URI for this entry

  • *
    Lars Frantzen R&D 
    Why Tests are Flaky - External Dependencies
    When dealing with test automation – especially when doing web testing with a full-blown browser interface – one of the biggest issue is what is called flaky tests. You may also call it unstable tests, and often this notion is overloaded with ...

No comments

Add Comment

You can use [geshi lang=lang_name [,ln={y|n}]][/geshi] tags to embed source code snippets.
Gravatar, Twitter author images supported.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.