Visual Testing in Katalon Studio


#1

I have created a project at

Please refer to this Github repository for running codes and description in detail.

----

Problem to solve

What do I mean by the word Visutal Testing? Any toolset that provides a solution to my requirements as follows, I would regard it a Visual Testing solution.

  • Often I have to look at the pages of a Web site. Maybe I am a developer, a designer, a tester or just a fan of the site.
  • I want to look at as many pages of the site as possible. Wide coverage matters.
  • I want to compare the view of 2 environments; e.g. Production and Development.
  • I want to take full-page screen shots of all pages as evidence
  • After taking screenshots, I want to check them to find out if there are any differences in view between the two.

If the target Web site has 100 pages, it would take me more than 60 minutes to run through. It’s too tiring, too boring. I don’t like doing the job everyday! Therefore I would add the final term:

  • I want to automate it.

I just want to be notified if 2 environments have any significant enough visual differences now. I would not require the tool to be equipped with full fledged debugging functionalities. Preferably it should be free of charge.

Solution

As for the file path problem, I have developed another project titled:

This project demonstrates how to take multiple sets of web page screen shots and store them into a well-structured file system tree. This demo project uses another GitHub repository:

This project provides a jar file which implements MaterialRepository = well-defined file tree and provides access methods.

As for the full-page screenshot problem, I found that the library aShot solves like a charm. I wrote a report how I utilized aShot in Katalon Studio:

I was impressed that aShot provides ImageDiff. This utility class enables me to check very easily how much differences there are amongst 2 images. I wanted to use ImageDiff in Katalon Studio. So I have developed another GibHub repository:

In there I have developed Katalon Custume Keyword com.kazurayam.ksbackyard.ScreenshotDriver. ScreenshotDriver is just a small wrapper for aShot. This makes it a bit easier to use aShot functionality in Katalon Test Cases.

I have integrated those external resources to build a toolset of ‘Visual Testing’ in Katalon Studio.

------------------

Input

The Test Suite Collection Executes calls Test Suite Main/TS1 twice; each time targeting the following URLs:

  1. http://demoaut.katalon.com/ --- called Production environment
  2. http://demoaut-mimic.kazurayam.com/ --- called Development environment

Output

The tool creates screen shots of the Input site and compare them, for example:

Production page

Production|

Development page

Development|

index.html

The test suite collection generates ./Materials/index.html. This HTML shows a list of source images plus the images as comparison result.

File path: <projectDir>/Materials/index.html index

ImageDiff

If you click the line with purple background color, you will see a ImageDiff with a lot of red-portion. The red portion shows the differences between the two source images.

ImageDiff


Visual Based Testing
#2

wow. cool! +1


#3

Awesome work Kaz! Upvoted.


#4

KATALON É ZICA

Congratz for the work guys!


#5

Thank you kazurayam, for this once again very practical solution proposal for my current, not yet mentioned challenge at exactly the right moment! :slight_smile:

However, I’ve just gotten a little away from the idea of actually tackling my problem with the ashot library, i.e. purely graphically based. Since I stumbled across the Galen framework, my claim is actually more in the direction you mentioned, that I would like to get some more debugging information about which concrete web elements deviate from my expected values in position and/or dimension. I think I’ll first try to see if I can move forward with this approach. But I’m very sure that your solution will be a great addition, so many thanks again for your great work!


#6

Congrats Kazurayam. May I know if this works on mobile testing? With different size of screen


#7

MaKatalon,

On PC using desktop browser, I have tested 3 view port sizes: 1600x900, 1024x768, 375x677 (=iPhone6 size). You can change the size by WebUI.setViewPortSize(width, height)

----

You asked me “if this works on mobile testing?” — well I am not sure how I should answer to it. The only part I am concerned about is the takeEntirePage() method of ScreenshotDriver class

This method requires org.openqa.selenium.WebDriver instance as 1st argument, depends on aShot. According to https://github.com/yandex-qatools/ashot;

aShot supports taking a screenshot of a WebElement on different platforms (i.e. desktop browsers, iOS Simulator Mobile Safari, Android Emulator Browser) as documented at

So, VisualTestingInKatalonStudio should be able to test WebUI on iOS Simulator Mobile Safari, Android Emulator Browser.

----

Do you want to use this with “Appium” ? — well. I do not know much about Mobile App testing.

(edit at 11/26/2018 AShot does not support android and ios with Appium. See https://github.com/pazone/ashot/issues/112. Hence my Visual Testing with Katalon Studio does not support Appium. If someone can tell me how to take entire-page screenshot for Appium, then I would be able to extend my “Visual Testing in Katalon Studio” to cover Android and IOS)


#8

thanks for your work
i tried earlier to use ashot in my project but i failed ( didn’t know the reason that time )
so, what exactly step by step should i do to merge your project to mine ?


#9

Writing a step-by-step instruction how to build your own “Visual Testing in Katalon Studio” — this is a very tough task for me; much more difficult than writing codes. I haven’t written it yet.

Download the zip of the “VisualTestingInKatalonStudio” project and see how it is built. That’s all I can say now.

Katalon Experts would be able to cook my project into parts, digest it, and pickup what they need to transplant into their own projects. I hope they do so.

I think that the Custom Keyword com.kazurayam.ksbackyard.ScreenshotDriver is immature, poorly written. I would refactor it soon.


#10

Hesham,

If the AShot is your current issue, then the following demo project would be useful for you:

This demo project is focused how to take full-page screenshot with AShot.

My “Visual Testing in Katalon Studio” is far more complicated than this. It would not be easy to digest.


#11

I wrote 2 more posts where I used AShot library to verify slideshow and YouTube video.

Just for your interest.


#12

I wrote another tips:

This post introduces the Materials library which is used in my VisualTestingInKatalonStudio project.


#13

One more use case of the Materials library

Here I used it for testing RESTful API


#14

I have written and published a new document

Step by step instruction - how to create a visual testing project from scratch

In this document I have tried to describe in detail how to get started with “Visual Testing in Katalon Studio” — comparing 2 almost identical sites by taking screenshots and making image differences.

I believe that automated visual testing is an effective method to detect** unplanned/accidental changes **made somehow in your Web app.

Please try.


#15

+10!


#16

Katalon Studio “The Missing Manual” -> this thread! B)


#17