The Selenium Bliss and Pain
When you deal with test automation of web pages these days, there seems to be just one player to choose: Selenium. And for all which I have seen, it is the best you can get! But still it is a pain and feels quite wrong when you need to work with it on a professional level.
Let me first make one point very clear: Selenium is a great achievement, and I totally respect the immense work and results the people around Selenium have achieved! But since test automation of web applications became such a crucial part of current software development, we really need to move on to a more sound and solid approach. Whatever that will be…
Let me first be more precise, I refer here to the Selenium WebDriver API, as being realized in the Selenium Server. It allows you to easily setup a distributed automated testing infrastructure, which is absolutely great.
Together with the browser drivers, which are available for basically all common browsers, you can run your tests cross-browser, even on mobile devices. You just pick the browser and/or device, and you can run the same test wherever you want. At least in theory. The API gives you most of the stuff you need, even though you would like to do without explicit waits, scrolling up and down at the page, and such things, but I am coming to that, next.
Waiting in Vain
Once you have written a test case using the WebDriver API, you can run the test on different browsers and devices. You commonly start with one browser, say Chrome, and then you realize that your test fails or is unstable (it sometimes passes and fails). You find out, that you need to wait for some elements to appear, to load, or whatever. So you use the several waiting operations from the API. The last thing you want is to do an explicit wait for a given number of seconds, since test execution time is relative to the executing system. But sometimes you cannot do without, or you just don’t know how. But usually you somehow manage to get your test stable by waiting enough here and there. And then you are happy. Until you run it on a different browser. Since there, it can very likely happen, that the test case fails again, or becomes unstable. Why? Because that other browser wants you to wait for different elements, events or whatever.
So you start to extend your test case to make it run stable on the second browser (and keeping it stable on the first one). And this game continues for each browser you add. And with each browser it gets a bigger mess.
Incompatible API Interpretations
Another painful observation is, that different browsers behave differently for the same API call. For instance, Firefox prefers to open the pages in a new window, whereas the default for Chrome is a new tab. You can then try to fix this with a changed browser profile, and that may work out, or it may not. Or Firefox shows you a tenacious modal dialog (which you cannot access with the WebDriver) when you leave a secure page, but Chrome is just fine with it.
So you need to fiddle around all these peculiarities, and even if it works, your test case gets more and more complex.
Unexpected Firefox Changes
In my current project we are now in the not so funny situation that we cannot run our test automation on a Firefox having a version > 47. Why? Because Selenium 2 cannot deal with it. And Selenium 3 needs the geckodriver, which is so immature, that very many test cases fail due to that. And the current version 0.13.0 of the geckodriver does not sound as if there will be a feature-complete version, soon. With each new version of Firefox the situation gets worse, and we may soon need to invest a huge effort to somehow define a new, adapted, restricted, whatever test suite for the newer browser versions. And then analyze both test suites separately, etc.
It is hard to understand how Firefox/Selenium can make such a move, without offering any workaround, for such popular technologies so many people rely on.
So what can you do? First of all, it is easy to complain without making a better suggestion. And I don’t have one. If you do, please let me know! You could consider using a headless browser like PhantomJS, but trying that showed that it is equally difficult (or worse) to get stable tests. And since tests via browser automation are also usually slow, I am considering more and more to step back from those as much as possible. At least unless there is a stable, reliable, and hassle-free approach available.
Summarized, web test automation is a hassle. And a real relief is only available for a subset of the technologies and testing goals you may need. It would be great to have an API available which is not defined on the abstraction level of WebDriver, so it can do without most waits, and is interpreted the same way by all implementations.
If that is a realistic wish, let’s see.