Visual Testing in Katalon Studio

I have created a project at

Please refer to this Github repository for running codes and description in detail.

----

Problem to solve

What do I mean by the word Visutal Testing? Any toolset that provides a solution to my requirements as follows, I would regard it a Visual Testing solution.

  • Often I have to look at the pages of a Web site. Maybe I am a developer, a designer, a tester or just a fan of the site.
  • I want to look at as many pages of the site as possible. Wide coverage matters.
  • I want to compare the view of 2 environments; e.g. Production and Development.
  • I want to take full-page screen shots of all pages as evidence
  • After taking screenshots, I want to check them to find out if there are any differences in view between the two.

If the target Web site has 100 pages, it would take me more than 60 minutes to run through. Itā€™s too tiring, too boring. I donā€™t like doing such a bullshit job! Therefore I would add the final term:

  • I want to automate it.

I just want to be notified if 2 environments have some significant visual differences. I would not require the tool to be equipped with full fledged debugging functionalities. Preferably it should be free of charge.

Solution

As for the file path problem, I have developed another project titled:

This project demonstrates how to take multiple sets of web page screen shots and store them into a well-structured file system tree. This demo project uses another GitHub repository:

This project provides a jar file which implements MaterialRepository = well-defined file tree and provides access methods.

As for the full-page screenshot problem, I found that the library aShot solves like a charm. I wrote a report how I utilized aShot in Katalon Studio:

I was impressed that aShot provides ImageDiff. This utility class enables me to check very easily how much differences there are amongst 2 images. I wanted to use ImageDiff in Katalon Studio. So I have developed another GibHub repository:

In there I have developed Katalon Custume Keyword com.kazurayam.ksbackyard.ScreenshotDriver. ScreenshotDriver is just a small wrapper for aShot. This makes it a bit easier to use aShot functionality in Katalon Test Cases.

I have integrated those external resources to build a toolset of ā€˜Visual Testingā€™ in Katalon Studio.

------------------

Input

A Test Suite Collection Test Suites/Execute_twins executes a Test Suite called Test Suites/twins_capture twice. Each execution takes screenshots of the following URLs:

  1. http://demoaut.katalon.com/ --- called Production environment
  2. http://demoaut-mimic.kazurayam.com/ --- called Development environment

Once screenshots are taken and stored in PNG files, Test Suites/Execute_twins executes a Test Suite called Test Suites/twins_exam. This test suite compares each pairs of screenshots to find if any significant visual differences are there. It generates a HTML file to show screenshots with the assertion result.

Output

You should open index.html with your browser by double-clicking the local file.

index.html

The test suite collection generates ./Materials/index.html. This HTML shows a list of source images plus the images as comparison result.

File path: <projectDir>/Materials/index.html index

ImageDiff

If you click the line with purple background color, you will see a ImageDiff with a lot of red-portion. The red portion shows the differences between the two source images.

ImageDiff

Production page

Production|

Development page

Development|

Twins-mode and Chronos-mode

Initially VisualTestingInKatalonStudio supported Twins mode which compares a pair of URLs of Application-Under-Test at a given time. This aimed to verify if a development environment is identical to a production environment of my AUT.

Later in the version 1.10.0, VisualTestingInKatalonStudio supported Chronos mode which compares the current set of screenshots against the previous set of screenshots of a Web site. The Chronos mode enables me (and you) to check status of the AUT at different timing: before/after any system changes ā€” application software upgrades, changes to configs, OS patches, network reconfiguration, database migration, etc.

How to enable your own Katalon project to perform ā€œVisual Testingā€

See the following documentation:

35 Likes

wow. cool! +1

1 Like

Awesome work Kaz! Upvoted.

KATALON Ɖ ZICA

Congratz for the work guys!

3 Likes

Thank you kazurayam, for this once again very practical solution proposal for my current, not yet mentioned challenge at exactly the right moment! :slight_smile:

However, Iā€™ve just gotten a little away from the idea of actually tackling my problem with the ashot library, i.e. purely graphically based. Since I stumbled across the Galen framework, my claim is actually more in the direction you mentioned, that I would like to get some more debugging information about which concrete web elements deviate from my expected values in position and/or dimension. I think Iā€™ll first try to see if I can move forward with this approach. But Iā€™m very sure that your solution will be a great addition, so many thanks again for your great work!

1 Like

Congrats Kazurayam. May I know if this works on mobile testing? With different size of screen

MaKatalon,

On PC using desktop browser, I have tested 3 view port sizes: 1600x900, 1024x768, 375x677 (=iPhone6 size). You can change the size by WebUI.setViewPortSize(width, height)

----

You asked me ā€œif this works on mobile testing?ā€ ā€” well I am not sure how I should answer to it. The only part I am concerned about is the takeEntirePage() method of ScreenshotDriver class

This method requires org.openqa.selenium.WebDriver instance as 1st argument, depends on aShot. According to GitHub - pazone/ashot: WebDriver Screenshot utility. Take screenshots, crop, prettify, compare;

aShot supports taking a screenshot of a WebElement on different platforms (i.e. desktop browsers, iOS Simulator Mobile Safari, Android Emulator Browser) as documented at

So, VisualTestingInKatalonStudio should be able to test WebUI on iOS Simulator Mobile Safari, Android Emulator Browser.

----

Do you want to use this with ā€œAppiumā€ ? ā€” well. I do not know much about Mobile App testing.

(edit at 11/26/2018 AShot does not support android and ios with Appium. See Appium Support Ā· Issue #112 Ā· pazone/ashot Ā· GitHub. Hence my Visual Testing with Katalon Studio does not support Appium. If someone can tell me how to take entire-page screenshot for Appium, then I would be able to extend my ā€œVisual Testing in Katalon Studioā€ to cover Android and IOS)

1 Like

thanks for your work
i tried earlier to use ashot in my project but i failed ( didnā€™t know the reason that time )
so, what exactly step by step should i do to merge your project to mine ?

Writing a step-by-step instruction how to build your own ā€œVisual Testing in Katalon Studioā€ ā€” this is a very tough task for me; much more difficult than writing codes. I havenā€™t written it yet.

Download the zip of the ā€œVisualTestingInKatalonStudioā€ project and see how it is built. Thatā€™s all I can say now.

Katalon Experts would be able to cook my project into parts, digest it, and pickup what they need to transplant into their own projects. I hope they do so.

I think that the Custom Keyword com.kazurayam.ksbackyard.ScreenshotDriver is immature, poorly written. I would refactor it soon.

1 Like

Hesham,

If the AShot is your current issue, then the following demo project would be useful for you:

This demo project is focused how to take full-page screenshot with AShot.

My ā€œVisual Testing in Katalon Studioā€ is far more complicated than this. It would not be easy to digest.

2 Likes

I wrote 2 more posts where I used AShot library to verify slideshow and YouTube video.

Just for your interest.

1 Like

I wrote another tips:

This post introduces the Materials library which is used in my VisualTestingInKatalonStudio project.

1 Like

One more use case of the Materials library

Here I used it for testing RESTful API

1 Like

(edit at May 2019)
The version 1.10.0 of VisualTestingInKatalonStudio has equipped with a easier method of enabling your own Katalon Studio project to do ā€œVisual Testingā€. See https://github.com/kazurayam/VisualTestingInKatalonStudio#how-to-make-your-katalon-studio-project-capable-of-visual-testing section.

The ā€œStep by step instructionā€ document got obsolete in v1.10.0, therefore was removed.


I have written and published a new document

Step by step instruction - how to create a visual testing project from scratch

https://github.com/kazurayam/VisualTestingInKatalonStudio/blob/master/docs/Step_by_step_instruction.md

In this document I have tried to describe in detail how to get started with ā€œVisual Testing in Katalon Studioā€ ā€” comparing 2 almost identical sites by taking screenshots and making image differences.

I believe that automated visual regression testing is an effective method to detect unplanned/accidental changes made somehow in your Web app.

Please try.

7 Likes

+10!

2 Likes

Katalon Studio ā€œThe Missing Manualā€ -> this thread! B)

1 Like

For everyone reading this, @kazurayam has added a follow-up article:

1 Like

Don Welsh raised an interesting question, so I replied to it. Let me share the discussion here.

1 Like