Modern Manual and Automated Cross Browser Testing

In this article we will take a look at the current state of modern cross browser testing and how it is possible to leverage proven techniques to gain the most out of the limited testing time available. It is intended for web developers and testers working on small static websites and large scale modern web applications.

My goal is to provide you, the reader, with a detailed guide that answers all questions about cross browser testing from start to finish. To get the most out of this guide you should read it once until the end and then take action in all sections that apply to you, your team and the project at hand. No prior knowledge of browser testing is required.

What Is Cross Browser Testing?

The goal of cross browser testing is to make sure that a website or web application tool looks and works correctly on all desktop and mobile browsers it targets. With the correct thought out testing strategy and process you can achieve this goal without the need to test the system out manually on hundreds of system combinations.

Is Browser Testing Necessary?

One might reasonably assume that because HTML, CSS and JavaScript have a detailed specification and all modern browsers are developed with this specification in mind, then the standards-compliant code should just work in all major browsers without any browser specific adjustment. Sadly this is not really the case!

Why is that you might ask?

Web Standards Are Not Bulletproof

Did you ever attempt to create a non-trivial application using nothing more than a written specification? Even if the specification is exceptionally well written, thought out questions will still surface during the implementation process because the spec cannot cover all edge cases in advance. This means that the actual implementations in the browser will differ slightly to the specification and other implementations.

Browsers Are Moving Targets

Most major browser vendors are locked in a race to implement new features in their rendering and JavaScript engines as fast as possible. Google and Mozilla currently of their browsers roughly every 6 weeks which makes it impossible to know off the top of your head what features are supported and what bugs might be present at any given time.

Recently, the vendors have started to take features away in newer versions (in the name of security), which in turn makes ongoing testing even more important and time sensitive.

Testing Strategy

Before we get into the nitty gritty details of actual testing, it is a good idea to take a step back and define a well thought out testing strategy specific to the current website or web application under test. Ideally this will be done in advance to any significant implementation work.

What Browsers Should You Test?

This is one of the most important decisions you will need to make to minimize the testing effort necessary. Nobody wants to spend hours testing older IE versions only to find out that the targeted users don’t actually use it.

To determine the list of targeted browsers you have a few different options:

• Take a look at existing statistics of the current / old webpage or web application to figure out which browsers are actually used.
• If no such statistics exist you can take a look at overall browser usage statistics.
• For desktop browsers it is always advisable to test for the latest version of the big 4 (Safari, Microsoft Edge, Chrome and Firefox) as this will cover a lot of bases.
• Internet Explorer is a special case because it is no longer actively developed but many Windows 7 users are still using it. If you don’t have a compelling reason not to you should add at least Internet Explorer 11 to the targeted browser list.
• In almost all cases you can ignore other Blink based browsers like Opera or Brave because they will work just as good as Google Chrome.
• For mobile browsers there are really only 2 targets: Safari on iOS and Chrome on Android. In most cases no other browsers have the necessary market share to make testing worthwhile – possibly with the exception of Firefox for Android.

Manual or Automated Testing?

Cross browser testing can be performed either manually or by using some kind of automation. While automating this process sounds like a good idea, in general you need to keep in mind that there is significant upfront and ongoing work involved in creating all the automated test cases.

Generally, it depends on the specific project you need to test. If you are creating a static website then the chances are that you won’t need to run a big number of regression tests. In this case, relying only on manual testing makes complete sense. For bigger web applications on the other hand, automated test cases could be worth the upfront time investment as you can run them again and again for each new feature or bug fix. This is especially true for applications with a long lifespan where development (and therefore testing) is ongoing.

Even if you decide to go with automated testing you will need the ability to carry out manual browser testing, both during development and for smoke testing a single browser. Automation will never be able to eliminate all manual testing time (nor should it).

Manual Browser Testing

By far, the easiest cross browser testing method is manual testing. For this, you need to gain access to the specific browsers you want to target and test manually.

Optimal Testing Process

As manual testing can be a time consuming and boring task, it is wise to optimise the testing process and reduce the time spent testing. A nice clean testing process can look like this:

• During development of a single feature or subpage, you should rely on a single browser to ensure everything works. Here you should use a modern and standards compliant browser like Chrome or Firefox. As an added benefit these browsers come equipped with a wide range of built-in developer tools which can make debugging much easier.
• Once the feature or subpage is complete and works correctly in your preferred development browser, you can move on and test it in all targeted browsers.
• If you encounter any bug or layout issues fix these and test your fix in all browsers so you don’t introduce new bugs in the browsers already tested.

For fixing bugs you can rely on compatibility data from http://caniuse.com. If at all possible don’t use browser detection and fall back to feature detection. This is so that your page keeps working out if a browser adds new features later on.

Desktop Browser Testing

The easiest desktop browsers to test are Chrome and Firefox as these can be installed for testing directly on any operating system. Keep in mind that these browsers auto-update so if you need to test for a specific version take steps to disable this feature (Chrome, Firefox).

The Microsoft browsers Internet Explorer and Edge only run on Windows, so if you are using macOS or Linux you cannot run them directly. Luckily Microsoft provides us with virtual machines for testing free of charge at http://modern.ie.

For testing Apple’s Safari browser you will need access to a machine running macOS as Safari does not run on any other operating system.

Of course, there a tools and services available to make these browsers available for testing either on your local machine or in the cloud. You can find a list of popular tools here in our blog.

Mobile Device Testing

Testing any page for mobile browsers is a tad more complicated than the desktop browsers. There are really 2 different test scenarios to be aware of here.

Test for Responsiveness

A responsive website automatically adapts to the current screen size of the device under test. This can be tested easily by simply resizing a desktop browser window or by using the mobile device simulation in Google Chrome. This way you can make sure that the page is visually correct at a lot of different screen sizes.

Test Mobile Browsers

To start. let us define what mobile browsers are. On iOS (iPhone, iPad …) there is only one rendering engine available: mobile Safari. All other browsers available for this platform use the mobile Safari rendering engine under the hood even if it is called Chrome for iOS.

On Android apps, in theory, the apps can use different rendering engines. In practice, most Android browsers rely on the Chrome rendering engine with the only notable exception being Firefox for Android.

This means that testing can be reduced to mobile Safari and mobile Chrome in almost all scenarios. The easiest solution for testing is using a physical device each. There is no need for testing all available form factors as the responsive page layout is already covered by our responsive testing (see above).

If you don’t have any real devices available you can use the iOS and Android emulator or an online testing solution. The Android emulator can run on any operating system but the iOS emulator is only available through XCode on mac

Automated Browser Testing

Many people think that automating ongoing browser testing is the Holy Grail to create a powerful web application without having to spend countless hours on manual testing labour. The problem often overlooked is, that creating and maintaining a fully automated test suite can be a huge task which will take up a significant amount of developer time. Still it is a viable strategy if you keep a few key points in mind:

• Your automated tests need to run as often as possible, ideally after each commit use some sort of Continuous Integration solution.
• If at all possible tests should run using the real targeted browsers.
• Tests need to be reproducible, for example through screenshots during test execution so you can see what went wrong after a failed test.

Luckily, there is a really powerful browser automation framework available that supports all major browsers called Selenium. Let us take a look at how the above points can be achieved using this framework.

Selenium Framework

Selenium is an open source browser automation framework backed by all major browser vendors including Microsoft, Apple, Google and Mozilla. It gives you the opportunity to create automated test cases that can run against all major browsers ­– and of course you can run the same test code against all supported browsers, reducing the browser specific overhead greatly.

Selenium has bindings for many popular programming languages like Java, C#, Ruby, Python and JavaScript. This means that you can write your tests in the same programming language used for the web project itself. There are lots of great Selenium tutorials out there including our own upcoming web automation course over at Selenium.Academy.

Working directly with Selenium might not always be the most suitable option for you. There are a lot of different frameworks out there that try to hide the complexity of Selenium to make the test creation steps easier for you. Of course you can also skip programming altogether and record your tests directly in the browser with a test recorder like the one included in BrowseEmAll.

Test Execution Setup

Once you have created your first test cases, it is essential to create some kind of environment where these tests can be run. Ideally this would be separated from the actual development environment so the tests can be run as much as possible.

For this, Selenium can create a so called Selenium Grid which contains one Selenium Hub and one or more Selenium Nodes. The Selenium Hub orchestrates the test execution on the different nodes. Each node is defined by the browsers it can run. Of course you can have many different nodes on different machines to be able to test with different browser and operating system combinations.

Unfortunately the setup and maintenance for this kind of environment can be a daunting task. Of course there are on-Premise and cloud solutions available if you don’t want to do all this manually.

Resources

• W3C Specifications:
  https://www.w3.org/TR/
• Browsers Rapid Release Cycle:
  https://dzone.com/articles/case-rapid-release-cycles
• Chrome HTTPS feature removal:
  https://searchenginewatch.com/2017/03/31/google-chrome-ssl-certificate-proposal-could-affect-millions-of-websites/
• Browser Usage Statistics:
  http://gs.statcounter.com/
• Browser Compatibility Data:
  http://caniuse.com
• Google Chrome Auto Update Disable:
  https://stackoverflow.com/questions/18483087/how-to-disable-google-chrome-auto-update
• Firefox Disable Auto Update:
  https://www.technipages.com/enable-disable-automatic-updates-in-firefox
• Edge and IE Virtual Machines:
  http://modern.ie
• Cross browser testing tools:
  https://www.browseemall.com/Blog/index.php/2015/04/21/cross-browser-testing-solutions-the-ultimate-guide/
• Responsive Websites Explained:
  http://hello.rindle.com/explaining-responsive-web-design-to-your-old-school-clients/
• Selenium Automation Framework:
  http://seleniumhq.org
• Lean automated browser testing with Selenium:
  http://selenium.academy
• Selenium Testing Frameworks:
  http://www.softwaretestinghelp.com/test-automation-frameworks-selenium-tutorial-20/
• Selenium Grid Explained:
  https://www.browseemall.com/Blog/index.php/2016/11/03/get-started-with-selenium-3-and-selenium-grid/

Key terms [Glossary]

• Blink – The rendering engine behind Google Chrome. Many other smaller browsers like Opera and Brave also use this engine.
• Cross Browser Testing – Means to make sure that a website or web application works in all major targeted browsers.
• Compatibility Data – A detailed guide on which web features are supported in a specific browser version.
• Desktop Browser – Is a browser running on a desktop machine or notebook, e.g. Google Chrome, Microsoft Internet Explorer …
• Feature Detection – Web programming approach where the browser support for a web feature is detected without using the browsers user agent string or other browser specific identifiers.
• Mobile Browser – A browser running on a mobile device like a phone or tablet.
• Responsive Website – The layout of a responsive website automatically adapts to the current available screen size.
• Static Website – A website without (much) dynamic functionality like a marketing page or a blog.
• Test Execution Setup – Means to execute automated test cases. This can be a internal server, software on the developers machine or a third party testing grid.
• Web Application – A website with lots of dynamic functionality like Facebook or Twitter.