Let’s say, you develop a website. All functionalities are tested and work well. But it has a bunch of misspelled words and broken layouts here and there. Can it be considered a high-quality user experience? Certainly not. And we believe many users share the same thought.
Visual aspects matter.
When developing an application, most of the resources are usually put into testing the functionalities. This is very understandable. We want the thing to behave and function as it should be. Now with the option of automation, we increase the efficiency of functional testing multiple times.
But what about the look and feel? When we talk about a high-standard user experience, the interface and visual aspects must be in the conversation and cannot be overlooked.
Most teams are aware of this, still spending resources on visual testing. The problem is the majority of teams perform visual testing manually. Considering a large number of pages or elements to verify on an application’s interface, naked eyes are easily prone to errors and missing details. Not to mention this task is very boring and depressing to most people, which can decrease the testing efficiency.
Katalon had supported visual testing, but it was not sufficient enough.
Back in 2020, Katalon supported visual testing with pixel-by-pixel image comparison (and we still do). For the newbies, this means it will compare two images (the base image and the captured-as-checkpoint image) by every pixel.
Frankly speaking, the results of this method is not very reliable. Just a few unmatched pixels can cause the tests to fail, but these mismatches are totally acceptable to our human eyes.
Let’s have a quick example. Below is a baseline image:
We adjust the space between intro-header div and container-fluid for the checkpoint image (which will be compared to the baseline):
The result of using the pixel comparison:
While the changes are acceptable for us humans, the test results turned out to be failed, ringing a false alarm.
So, we decided to make enhancements for visual testing.
Fast forward two years, after intensive research and implementation, in 2022, we’ve introduced two new comparison methods that leverage the technology of AI. Those are the content-based and layout-based comparisons, which are expected to solve the problem of false positives caused by the common pixel method. These methods only focus on major changes that really matter to humans, giving users much more reliable results.
The layout-based comparison method
It automatically identifies and maps zones that are similar in the two images. If the zones are just slightly different (due to different spacing for example), it will look past the minorities and still mark the test result as pass.
Here’s the outcome of the pixel method comparing a base image and a checkpoint (red means unmatched zones):
The content-based comparison method
As its name suggests, it identifies and compares the texts between two images. It also tells users whether the compared texts are in different positions.
Below is an example of when comparing the base (first image) and the checkpoint (second image). Similar texts are in the green boxes. Texts that are the same but in different positions are highlighted in blue. No mismatch (red) in the case.
Other enhancements
We also add the ability to specify ignored zones while comparing two images, which can be a solid solution for dynamic content like advertisement sections displayed on many websites.
We also enhance the baseline collections to give users more options to manage their baseline images.
For a quick demonstration, you can watch this video: https://www.youtube.com/watch?v=J5y48iX0MO0
Enhanced visual testing means more value to Katalon users.
Katalon allows users to not just stop at functional testing. They can now extend to visual testing with more reliable results. Creating new or adding visual testing as part of functional tests can be done easily and quickly by adding predefined keywords in Katalon Studio in the tests.
Users can then leverage TestCloud to take screenshots of the elements under test in multiple cloud environments with different browsers, OSs, and devices. Katalon takes care of the need for both functional and visual regression testing.
All of these happen natively within the Katalon platform. No complicated setup or third-party configurations are needed. Even testers with minimal experience can perform visual testing with Katalon. The general steps include:
- Integrate Katalon Studio and TestOps
- Enable screenshot/image capture in Katalon Studio by using predefined keywords
- Execute test suites in Katalon Studio
- Go to TestOps > Visual Testing > Visual Test Runs to view the results for visual tests
For more information, read this documentation.
What’s next?
After going through the trial period, we officially released the enhanced visual testing functionalities in July 2022. The next thing to do is listen to our users’ feedback through various channels and look into feature enhancements and new development.
If you’re a business or enterprise TestOps user, you can purchase or request a trial to use the full capabilities of visual testing (and provide your valuable feedback to us). More information here.
Note: The visual testing is still available for free users but it’s limited to only the pixel-comparison method.
If you want to read more articles like this, visit our Product Insights corner.
Have a nice day!