Visual Inspection in Katalon Studio - Reborn

kazurayam · August 15, 2021, 5:06am

I have created a GitHub repository:

@date Aug 2021
@update 15 Oct 2022, ver 0.10.3
@author kazurayam

My previous work

Sep 2018, I published a project named “Visual Testing in Katalon Studio” at

GitHub - kazurayam/VisualTestingInKatalonStudio: This is a Katalon Studio project which performs 2 ways of screenshot-comparison testing for WebUI in full automation. Twins mode compares 2 host names at a time. Chronos mode compares the current view of a AUT to the previous one. You can make your own Katalon Studio project capable of the same screenshot-comparison. approach. Instruction provided.

Please have a look at the “Motivation” section in its README document to know why I made it. I tried to implement what I called “Visual Testing” in Katalon Studio. That project worked for me.

But I wasn’t satisfied with it. Why? I would enumerate 3 problems about that project.

The code set of Visual Testing In katalon Studio project is too complicated. The project contains 28 Test Cases, 7 Test Suites, 4 Test Suite Collections, 1 Test Listener, 12 Keywords. After 3 years, I forgot them and unable to maintain them any longer.
The project enabled me to compare a pair of 2 screenshots of Web pages in PNG image format. I found that it is not enough. Often I wanted to compare 2 texts scraped from web. For example, I wanted to compare 2 HTML files. Also I wanted to compare 2 JSON files downloaded from a Web Service.
The project wasn’t distributable to others easily because it wasn’t packaged as “jar”. It was too hard for people to reuse the code set of the project to achieve their own “Visual Testing” for their custom targets.

My new development

In 2021 July-August, I have developed a new project named materialstore. It is a domain-specific file system written in Groovy, accompanied with a set of utility class that implements “Visual Testing”. In this project I have re-designed and re-implemented my idea from scratch. In the materialstore library, I have achieved the followings.

(1) Simpler code set

The materialstore provides a self-contained API in Java8, which encapsulates all of the useful functionalities of the previous Visual Testing In katalon Studio project. A single Test Case in Katalon Studio empowered by the materialstore library can achieve whole job equivalent to the previous one. No need to struggle with that many components (28 Test Cases + 7 Test Suites + 4 Test Suite Collections + 1 Test Listener + 12 Keywords).

(2) Domain-specific file system indexed by metadata

By the word material I mean any type of files downloaded from Web applications during automated tests. Screenshot images (PNG), Web page source (HTML), JSON and XML responded by Web Services — I call all of these as materials.

The materialstore provides capability of materializing (storing) files downloaded from web sites in a pre-designed directory structure (I call it the store).

An application writes files into the “store” associating “metadata”. The “materials” in the “store” are indexed by the associated metadata. An application retrieves files from the store by querying for metadata. An application does not look up files by physical name. The application is no longer responsible for deciding and remembering the path of physical files.

A metadata of a material in the store is merely an instance of java.util.Map<String, String> with arbitrary key=value pairs. You can programme any kind of metadata and associate it to each materials so that the materials are well described.

For example, you can associate the URL from which the web resource was retrieved; or you can associate the name of browser (“Chrome”, “FireFox”, “Safari”, etc) which you used to take the screenshots; or you can associate the name of “Execution Profile” in Katalon Studio which you applied when you executed your Test Case.

And you are responsible for creating metadata that clearly and uniquely identifies a material amongst a set of materials created a single run of test.

Metadata composition is entirely up to the user application. Composing metadata is difficult part of the materialstore library. It looks similar to designing database table in SQL-backed application.

(3) Packaged in a jar

The artifact of materialstore is distributed as a single jar file. The jar file is available at the Maven Central Repository. Any Java/Groovy-based application can automate downloading the materialstore-x.x.x.jar to resolve the dependency using the build tools (Gradle, Maven, Ant).

(4) Usable outside Katalon Studio

The materialstore library is a set of Plain Old Java Objects, which has no immediate dependency on the Katalon Studio API. It has no dependency on the Selenium WebDriver API either.

So the materialstore library can be used in any Java/Groovy project outside Katalon Studio. I can use it in a plain Selenium WebDriver-based automated test project for Web UI on Gradle. I can use it in a plain Appium Java Client-based automated test project for Mobile UI on Maven. I can use it in a plain Apache HttpClient-based automated test project for Web Services on Ant.

This “VisualTestingInKatalonStudio_Reborn” project is a mere demonstration how to utilize the materialstore library in Katalon Studio.

Examples in Katalon Studio

Creating a project, resolving external dependencies

You can create a new Katalon Studio project, import the required external dependencies, and write your Test Cases for “Visual Inspection”. Let me describe the procedure how to create your “Visual Inspection Reborn” project.

Install “Gradle” build tool into your PC. Please follow this guide to install Gradle on your machine.

You can use Gradle ver 7.x as well as ver 6.x.

Open Katalon Studio GUI. Create a new project as usual in whichever directory you like.

I will write a symbol $projectDir to express this project directory.

create $projectDir/build.gradle. You should copy and paste the source of:

build.gradle

In the commandline (outside Katalon Studio GUI) you want to execute the following command:

$ cd $projectDir
$ gradle drivers

The gradle driver will display some lines of messages in 10 seconds, and will finish successfully.

BUILD SUCCESSFUL in 1s
1 actionable task: 1 executed

Once the command finished, in the $projectDir/Drivers directory, you will find some jar files are automatically imported.

$ tree Drivers

Drivers
├── AUTOIMPORTED_ExecutionProfilesLoader-1.2.1.jar
├── AUTOIMPORTED_ashot-1.5.4.jar
├── AUTOIMPORTED_freemarker-2.3.31.jar
├── AUTOIMPORTED_java-diff-utils-4.11.jar
├── AUTOIMPORTED_jsoup-1.14.3.jar
├── AUTOIMPORTED_materialstore-0.10.3-SNAPSHOT.jar

The exact version number of the jars may change in future. Please regard this as an example.

All of these jar files are downloaded from the Maven Central Repository.

If you are going to push your new project into Git repository, you should write the .gitignore file so that it ignores the Drivers/ directory.

Stop Katalon Studio, restart it and re-open the project.

You have resolved external dependencies. Now you can start writing a Test Case.

Demo outputs

Visual Inspection in Katalon Studio | VisualInspectionInKatalonStudio_Reborn

Sample1: simply visit a web site to take screenshots

First example. We will write a Test Case in Katalon Studio that visits the Google Search page. We will take screenshots and HTML page sources of the Web page. We will store PNG files and HTML files into the store directory using the materialstore library. We will finally generate a HTML file in which we can view the stored PNG and HTML files.

(1) Test Case

You want to newly create a Test Case Test Cases/main/GoogleSearch/scrapeGoogleSearch in your project. Copy and paste the following sample source:

Test Cases/main/GoogleSearch/searchGoogleSearch

You want to run it by clicking the green button in Katalon Studio GUI.

(2) The report generated

Once the Test Case finished, a HTML file will be created at store/scrapeGoogleSearch.html. Please open it in any web browser. It renders a view of the stored 6 files. You can see an working example here: pls. click here.

(3) The “store” directory structure

When the Test Case finished, you will find a new directory $projectDir/store is created. In there you will find a tree of directories and files, like this:

$ tree store
store
...
├── GoogleSearch_scrape
│   └── 20221016_084617
│       ├── index
│       └── objects
│           ├── 09be6ef6ef37f04bf34977fcce96088195a0cd76.html
│           ├── 1ae20a4dc1fce5fdec3ee457b1af1106e632880b.png
│           ├── 63cc4d50d6140aa3db07ed1da6477f528ede9fa6.png
│           ├── 7203b58eba10f9b7d9220532b72990f0732ce673.png
│           └── d331434cb6bfed9772c54727bdaaed4ac3fdeefa.html
├── GoogleSearch_scrape-list.html
...
17 directories, 39 files

(4) The objects/ directory

Under the store/GoogleSearch_scrape/yyyyMMdd_hhmmss/objects/ directory, there are 5 files. Among them you will find 3 files with postfix png. These are the screenshot of web pages in PNG image format. Also you will find 2 files with postfix html. These are HTML sources of web pages.
The file name comprises with 40 hex-decimal characters appended with extension (.png, .html). The hex-decimal string (I call this “ID”) is derived from the file content without compression by SHA1 Secure Hash algorithm.

(5) The index file

The store/GoogleSearch_scrape/yyyyMMdd_hhmmss/index file would be interesting. An example of the index file is like this:

09be6ef6ef37f04bf34977fcce96088195a0cd76	html	{"q":"katalon", "step":"03", "URL.host":"www.google.com", "URL.path":"/search", "URL.port":"80", "URL.protocol":"https", "URL.query":"q=katalon"}
7203b58eba10f9b7d9220532b72990f0732ce673	png	{"q":"katalon", "step":"03", "URL.host":"www.google.com", "URL.path":"/search", "URL.port":"80", "URL.protocol":"https", "URL.query":"q=katalon"}
d331434cb6bfed9772c54727bdaaed4ac3fdeefa	html	{"step":"01", "URL.host":"www.google.com", "URL.path":"/", "URL.port":"80", "URL.protocol":"https"}
63cc4d50d6140aa3db07ed1da6477f528ede9fa6	png	{"step":"01", "URL.host":"www.google.com", "URL.path":"/", "URL.port":"80", "URL.protocol":"https"}
09be6ef6ef37f04bf34977fcce96088195a0cd76	html	{"step":"02", "URL.host":"www.google.com", "URL.path":"/search", "URL.port":"80", "URL.protocol":"https", "URL.query":"q=katalon"}
1ae20a4dc1fce5fdec3ee457b1af1106e632880b	png	{"step":"02", "URL.host":"www.google.com", "URL.path":"/search", "URL.port":"80", "URL.protocol":"https", "URL.query":"q=katalon"}

The index file is a plain text file. Each lines corresponds to each files stored in the objects directory.

A line of the index file has 3 parts delimited by TAB characters.

<SHA1 Hash value of each file>\t<file type>\t<metadata>

(6) The metadata

In the test Case script, the code created metadata for each objects. Typically a metadata will include information derived from the URL of the source Web Page. For example, an URL

http://www.google.com/

will be digested to form a metadata

{"URL.host":"www.google.com", "URL.path":"/", "URL.port":"80", "URL.protocol":"https"}

You can add any key-value pair into the metadata as you like. For example, you would want to add some sequential numbering like "step":"01", "step":"02" and "step":"03". This will enable to sort the entities in the file report sorted in the order you like.

In the index file, lines are sorted by the ascending order of metadata text.

The materialstore API restricts that metadata texts in a index file MUST be unique. Your application can not create multiple objects (= multiple lines in the index file) with the same metadata value.

(7) How to sort the entries in the report

In the report, the entries are sorted by the ascending order of the “Identification” string. The “Identification” is the portion highlighted in the following image:

If you want to sort multiple screenshots in the order as they were taken, then you would want to do two things:

while your script takes screenshots, you want to add an Metadata attribute that represents the sequence. For example, add "step":"01", "step":"02" and "step":"03". See the Line#46 of Test Cases/main/GoogleSearch/materialize

Metadata metadata1 = Metadata.builder(searchPageURL).put("step", "01").build()

when your script generates a report, specify attribute names as SortKeys. The attribute specified as SortKeys will come left-most of each Identification string, so that will have higher priority of sorting order. See the Line#25 of Test Cases/main/GoogleSearch/report

Inspector inspector = Inspector.newInstance(store)
SortKeys sortKeys = new SortKeys("step","URL.host", "URL.path")
inspector.setSortKeys(sortKeys)
Path report = inspector.report(materialList, fileName)

Sample2: Visual Inspection in Chronos mode

Second example. We will write a Test Case in Katalon Studio that visits the http://demoaut-mimic.kazurayam.com/. The top page displays a current timestamp in the unit of seconds. So everytime you retrieve this page, the page changes slightly at the timestamp portion.

How is the current page of a web system different from what it was previously since 5 minutes ago, 3 hours ago, or 2 days ago? I want to see the differences in HTML code, not only visually as screenshots comparison. — the second example will show you how to implement a test that helps.

(1) Test Case

In your project, you want to copy&paste the following Test Case code.

Test Cases/main/CURA/VisualInspectionChronos

You will execute the Test Case two times. You run it once; wait for some period (seconds, minutes, hours, or days, … up to you); then run it again. The Test Case will preserve the output of previous runs.

(2) Execution Profile

This Test Case expects an Execution Profile is provided like this:

Profiles/CURA_DevelopmentEnv

Please create it in your project.

(3) The “store” directory

You will get 2 directories named in the format of yyyyMMdd_hhmmss. The directories will look like this:

$ tree store
store
├── CURA_Main_Chronos
│   ├── 20221016_084722
│   │   ├── index
│   │   └── objects
│   │       ├── 3e7cfb9128556d0f5bca16af1c7be00396f15d8b.png
│   │       ├── 46bd12e0dbfeb39bbbbc67c1996395b0ea666c20.png
│   │       ├── 61ed7eeb5d1bc3388808e4f747537a599768acc6.html
│   │       ├── 919580be9a8fb2e195ffe40bfe687caf4718d312.html
│   │       ├── d85de77a8e6df464d89a3d8501e243f53ac527c3.png
│   │       └── f3de4cfcf448606fa7b15250d88f792dacb62e6c.html
│   ├── 20221016_084759
│   │   ├── index
│   │   └── objects
│   │       ├── 5ce9b37a8632a97a7355fb3a09014a7105854927.png
│   │       ├── 9835d49d33407bfe0cd7f9ef41e69c151a4ee3d3.html
│   │       ├── a3e78a6b3ed0005dcef5f8419da3ce7f0d151f13.png
│   │       ├── d85de77a8e6df464d89a3d8501e243f53ac527c3.png
│   │       ├── d9e7abf1d048e4f0e512a0f499db98dd01e4620a.html
│   │       └── e62bda5328f450387273668749cc7ca525988da5.html
│   └── 20221016_084824
│       ├── index
│       └── objects
│           ├── 06bf1cdb8d3778092dcc488403a909463bd8c987.html
│           ├── 15a6cdaad19f4c2f22b4722f33717d625f33a550.html
│           ├── 1e865231bdc6a6cc6570d05a82f17e429995e049.png
│           ├── 2af8aef025af08139ab96dbc71378279c775af6d.png
│           ├── b716d42df3912f11c350edcad74221b90bd41e83.html
│           └── d85de77a8e6df464d89a3d8501e243f53ac527c3.png
├── CURA_Main_Chronos-index.html
...

(4) making Chronological diff

The Test Case compares takes materials at chronologically different timings, and then compare the 2 sets.

Test Cases/CURA/VisualInspectionChronos does the following processing:

make a List of materials in the current timestamp directory
make another List of materials in the previous timestamp directory
make differences between the two lists, store the diff info into the store as well
generate a HTML report, where you can view the detail with diff information.

(5) The report generated

Once the Test Case finished, a HTML file will be generated at store/CURA_VisualInspectionChronos-index.html. Please open it in any web browser. It renders a view of the stored 6 files. You can see an working example here: pls. click here.

You can see examples of screenshot comparison result: Previous screenshot - diff image - current screenshot.

Also you can see examples of HTML page sources comparison result.

(6) Interactively memorizing ignorable diffs

You can interact with the report generated by the Visual Inspection. See the top-left corner of the report. You will find the count of “diff” entities are displayed.

2w 0i 6t

This means:

There are 2 diff entities as Warnings that have a diff-ratio greater than the criteria given.
There are 0 diff entities as Ignored that have a diff-ratio greater than the criteria given, and is “checked to be ignorable”
There are 6 diff entities in Total.

In this sample case, I created 6 materials (PNG screenshot and HTML source saved). I set 1.0% as the criteria of evaluating the diffRatio; if an entity has a diffRatio greater than 1.0, it will be marked warning, hence the figure 46.66% is highlighted with the background color of yellow.

A large diffRatio indicates to me that there must be some reason that caused significant visual differences; therefore I should look into the system. Definitely I will do for every entities warned. I would find some reason why the diffRatio is as large as 46.66%. No matter what the reason is, I would decide whether the entity is ignorable or not.

In fact in most cases, the diffs detected by automated tests are ignorable for human. For example, when I compare a development environment against a production environment of a Web app, it is quite likely that there are some significant visual differences caused by the recent changes in the application software in the development environment. Any visual differences caused by known factors are ignorable for human, and I want to differentiate them from the diffs caused by unexpected factors.

Only human can decide if a visual difference is ignorable or not; my Visual Inspection tool does not decide it. But the tool offers you a feature of persisting your decision into the report and memorize it across sessions. Let me assume I have found out that the entity with 46.33% in the following screenshot is ignorable. Then I would interact with the report displayed in a browser. I will click the checkbox at the left end of the entity.

When the checkbox is checked, the diffRatio 43.66% changes its background color less highlighted. And the “Count” display on the top-left corner changes:

2w 0i 6t => 1w 1i 6t

This display indicates that I have 1 warning, 1 ignorable, among 6 entities in total.

I will continue debugging. I will look at the entity with the diffRatio 1.87%. Let me assume I found it ignorable again. Then I will check it also. Eventually the report would give the following display:

Now I have got

0w 2i 6t

I have zero warnings! Congratulations! I have finished inspecting this web application!

The status checkbox toggled ON/OFF by human is persisted into localStorage of browser. Therefore your decisions (which entities to be ignored) will be retained across browser sessions. You can reload the report, close and reopen the report. The checkboxes in the report will keep the status you put previously.

2w 0i 6t => 1w 1i 6t => 0w 2i 6t — I would name this display as WIT — makes my inspection task spirally upward. It helps me gradually improve the target system through executing Visual Inspection repeatedly.

Sample3: Visual Inspection in Twins mode

The 3rd example visits 2 URLs at almost the same timing.

The former URL is meant to represent a Production environment of a web system, the latter URL is meant to represent a Development environment. The pages look similar at a glance, but in fact these are different in detail.

How the pages of 2 environments differ now? — the 3rd example will show you.

(1) Test Case

In your project, you want to copy the following Test Case code.

Test Cases/main/MyAdmin/VisualInspectionTwins

You will execute the Test Case only once. The Test Case will visit the 2 URLs as one batch. And it creates a HTML report.

(2) Execution Profile

The Test Case script expects 2 Execution Profiles are provided like this:

Profiles/MyAdmin_DevelopmentEnv
Profiles/MyAdmin_ProductionEnv

Please create them in your project.

(3) The “store” directory

Once finished, the Test Case creates a directory named in the format of yyyyMMdd_hhmmss. The directory looks like this:

$ tree store
store
...
├── MyAdmin_Main_Twins
│   ├── 20221016_084837
│   │   ├── index
│   │   └── objects
│   │       ├── 1f2f0176d802db36521219ce281432dc0e67d920.png
│   │       └── 4419c44062177d8cb12a839169bc63b53cba003b.html
│   ├── 20221016_084846
│   │   ├── index
│   │   └── objects
│   │       ├── b59bdac875b80dc47fe3345b4dbe8968db48188d.png
│   │       └── b95fe452b63a6a9b5d5c4329ffaab6945e414943.html
│   └── 20221016_084853
│       ├── index
│       └── objects
│           ├── 5c75382c591698c749db0e03850135ddc72310fb.html
│           └── 843303409b05d859e8af8617c347fe19f88290f3.png
└── MyAdmin_Main_Twins-index.html

(4) making diffs of Twins

The Test Case looks up 2 sets of materials and compare them.

Test Cases/main/MyAdmin/VisualInspectionTwins does the following processing:

Amongst the stored materials, select 2 (a screenshot and a HTML) of the Development environment to make a List.
Also select 2 of the Production environment to make a List.
compare the two Lists and find differences between them.
will insert files into the “store”, which contains the diff information.
generate a HTML report, where you can view the detail with diff information.

See Selection algorithm for more detail

(5) The report generated

Finally the Test Case will generate a HTML report of which path is store/MyAdmin_VisualInspectionTwins-index.html. Please open it in any web browser. It renders a view of the stored 1 page, both in PNG screenshot and HTML page source. You can see an working example here: pls.click here

You can see a screenshot comparison result: Production Env - Diff - Development Env.

Also you can see the diff of HTML page source.

Notes on Extensibility

The materialstore library is usable for various cases. Once the materials are saved into the store, same diff + reporting features work.

for Mobile

The materialstore library can be used by a Appium-based Mobile testing in Java. All you need to do is to take screenshots of Mobile app using Katalon Studio’s built-in keyword (e.g, Mobile taksScreenshot) and save the image into the store.

for Web Service

The materialstore library can be used by a Apache HttpClient-based Web Service testing in Java. Using Katalon Studio’s sendRequest keyword, you can get a response. Then your script extract the response body (JSON, XML) and save them into the store.

Conclusion

The Visual Testing In Katalon Studio project is now superseded by this Visual Inspection In Katalon Studio - Reborn project backed by the materialstore library.

kazurayam · January 22, 2022, 11:20am

Aprin 2022, The Sample4 has now its own GitHub project at

Sample4: Navigating multiple URLs in Twins mode

The sample3 showed how to compare just a single pair of URLs. The next sample4 shows how to compare multiple pairs of URLs. The sample4 shows how to navigate though a site for 7 pages while taking screenshots and HTML sources.

(1) What the test case does?

The test case Flaskr/VisualInspectionTwins does the following:

It compares 2 web sites visually: http://127.0.0.1 and http://127.0.0.1:3090 .

(You can setup these URLs on your PC locally. I will explain how to later in this document.)

A web application named Flakr run in the URLs. Let me call them "Flaskr Prod env " and "Flaskr Dev env ". These 2 URLs have just the same functionality. But the pages look slightly different. The production environment has no site logo, but the development environment has a small site logo. .

the Prod env
the Dev env

The test case starts visiting the index page, then navigates through pages by clicking menu anchors, typing texts, clicking buttons, etc.

Step	description	Prod env URL	Dev env URL
step1	index page	http://127.0.0.1/	http://127.0.0.1:3090/
step2	register credential	http://127.0.0.1/auth/register	http://127.0.0.1:3090/auth/register
step3	log in	http://127.0.0.1/auth/login	http://127.0.0.1:3090/auth/login
step4	posting - blank	http://127.0.0.1/create	http://127.0.0.1:3090/create
step5	posting - text typed	http://127.0.0.1/create	http://127.0.0.1:3090/create
step6	list of blogs	http://127.0.0.1/	http://127.0.0.1:3090/
step7	log out	http://127.0.0.1/	http://127.0.0.1:3090/

The test case compares each pairs of URLs and generates diff images. The following is an example of a diff image. Please find the section on the top-left side painted red, which is the diff.
The test case generates a HTML report which shows a list all of the materials (screenshot images and HTML page sources) attached with detail diff information. You can find an example here.

(2) How to run the test

Just open the “Test Cases/Flaskr/VisualInspectionTwins” and run it. As default Chrome Headless browser git will be used, but you can choose any browser. You can choose any Execution Profile. The test won’t be affected by the profile you chose.

The test case will take approximately 30 seconds to finish.

The test case will write the report in the <projectDir>/store/Flaskr_VisualInspectionTwins-index.html file.

(3) What is Flaskr?

Flaskr is coded in Python language on top of the “Flask” web application framework. I learned the Flaskr web app at the Flask Tutorial authored and published by the Pallets project.

This tutorial will walk you through creating a basic blog application called Flaskr. Users will be able to register, log in, create posts, and edit or delete their own posts.

I used the source code 99% as is. I amended it slightly to diplay the site logo in the Dev env only.

(4) How the test is coded

You can read the sources

and a lot of related Groovy classes

Why do I have these Groovy classes? — It is because I employed the “Page Object Model”. The Page Object Model helped me in writing compact and readable codes.

The sample4 is highly extensible while avoiding code publication. It would be a foundation of large scale Visual Inspection projects.

kazurayam · January 22, 2022, 11:21am

April 2022, The sample4 which requires Docker. now has its own GitHub project,

https://github.com/kazurayam/VisualInspection_with_PageObjectModel_in_KatalonStudio

Setup Docker & Flaskr on your PC

In the sample4, we need 2 URLs available on our own PC.

You can make it work on your PC. You need Docker installed into your PC.

Installing Docker Desktop

On my MacBook Air, I installed Docker Desktop. Docker Desktop for Windows is also available.

Command Line Interface

Here I assume you work on a Command Line Interface. I use the Terminal.app of macOS. For Windows user, install Git for Windows which bundles “Git Bash”.

Starting up Flaskr

Open a window of Command Line Interface, then execute:

$ cd $VisualInspectionInKatalonStudio_Reborn
$ ./startup-flaskr-prod.sh

This shell script will emit 2 lines of messages and will block:

you can visit http://127.0.0.1/
Serving on http://0.0.0.0:8080

Next, you want to open one more window of Command Line Interface, then execute:

$ cd $VisualInspectionInKatalonStudio_Reborn
$ ./startup-flaskr-dev.sh

This will emit 2 lines of messages and will block:

$ ./startup-flaskr-dev.sh
you can visit http://127.0.0.1:3090/
Serving on http://0.0.0.0:8080

By lsof command, you can make sure that 2 processes are listening to the IP port #80 and #3090 on your localhost.

$  lsof -i -P | grep LISTEN | grep com.docke
com.docke   709 kazuakiurayama  107u  IPv6 0x84f53716e8d7cb33      0t0  TCP *:3090 (LISTEN)
com.docke   709 kazuakiurayama  111u  IPv6 0x84f53716e8d771d3      0t0  TCP *:80 (LISTEN)

Shutting down Flaskr gracefully

You can stop the docker process gracefully by typing CTRL + C.

You should NEVER close the window of Command Line Interface without stopping the docker process by CTLR + C.

If you forced to close the window, then IP Port #80 and #3090 might be left USED status. In that case you would fail to start a new process of Flaskr again. When it occurred, you need to stop & restart your PC/OS to release the ports.

Initializing Database in the web app

The Flaskr has a backend database where credentials and blog posts are stored. After you repeated running tests several times you will find many blog posts are stored, and you would feel like to clean the database out.

Just type CTRL + C to stop the docker process and restart it. The start-up script will automatically initialize the internal database and make it empty.

How I used Docker

Hava a look at the code of startup-flaskr-prod.sh:

CWD=$(pwd)
cd $(mktemp -d)

echo you can visit http://127.0.0.1/

docker run -it -p 80:8080 kazurayam/flaskr-kazurayam:1.1.0

cd $CWD

As you see, it runs docker run command with a docker image kazurayam/flaskr-kazurayam:1.1.0 . I created this docker image and published at the Docker Hub.

kazurayam · January 27, 2022, 5:26am

I continued development to inspect CSS and JavaScript referred by web pages.

Dan_Bown · February 23, 2022, 5:27pm

Hi @kazurayam
I’m interested in functionality to support comparison of multiple URLs in Twins mode, since sometime a SAAS application will make changes that our company has no control over and it would be useful to supplement functional automated tests. This work looks great, although I will need some time to get my head around it. Can I ask whether there is a need for Docker or Python in the project or is this just for illustration? Working in a restricted environment without Gradle, so I’m unable to run the example and also installing dependencies by hand. I assume that if I wanted to make my own test cases I would need to replace the POM versions you have (eg Include\scripts\groovy\com\kazurayam\uitestjava\flaskr\pom\actions)? Just a note that the groovy script URLs in Sample 4, Section 4 are not valid (give a 404, I found them in the project though). Thanks Dan

kazurayam · February 24, 2022, 1:24am

Only the “Sample4 Flaskr” requires Docker. The “visual inspection” framework does not require Docker. The Sample1, 2, 3 do not require Docker either.

See https://github.com/kazurayam/VisualInspectionInKatalonStudio_Reborn#target-web-application-under-sample-tests for more info.

The source code in the Include/scripts/groovy/com/kazurayam/uitestjava/flaskr/ folder is a part of the Sample4. It is not a part of the “visual inspection” framework. If you are going to develop your own test suite, your code will not use Include/scripts/groovy/com/kazurayam/uitestjava/flaskr/ at all.

You misunderstood.

The URL “http://127.0.0.1:3080” and “http://127.0.0.1:3090” are the right ones. That’s what I prepared for the Sample 4. These URLs will be provided by the Docker container that runs on your localhost(127.0.0.1). The Test Listeners/TLFlaskr silently&automatically starts/stops the Docker container on your PC for the Sample4 provided that you have installed “Docker for Desktop” into your PC.

If you want to see the Sample4 running, you need to install Docker into your PC. But if you do not want the Sample4, you won’t need Docker.

kazurayam · February 24, 2022, 1:29am

It’s OK not to use Gradle as far as you can manually download the required versions of external jars into the Drivers folder.

However please note that I frequently update the materialstore and other jars. Quite sometimes I would change the “visual inspection” framework’s API drastically. These are still in unstable status. The consistency of the jar versions really matters. Therefore you had better use Gradle to manage the external dependencies consistently.

By the way, yesterday I published the latest materialstore-0.3.5.jar but the Maven Central has not yet accepted, has not published it yet on their download site. It would take a few more days before the v0.3.5 of materialstore available publicly available.

kazurayam · February 24, 2022, 1:48am

@duyluong
@Russ_Thomas
@Dan_Bown

I wrote

My Groovy/Java code (including inside Katalon Studio) can start & stop any Docker Container on the localhost(127.0.0.1). Don’t you find it interesting?

If you have a web app of yours (in any language: Python, Node, Ruby, .NET, Java/Groovy) in a Docker image, you can start it as a Docker process on your PC at http://127.0.0.1:xxxx, and run your test suites with Selenium-java against that URL.

See the following doc for more detail:

https://github.com/kazurayam/SeleniumTestInJavaBackedByDockerContainer

This was difficult, required several months of my development efforts.

Dan_Bown · February 24, 2022, 7:52pm

Thanks for clarifications, and thanks for your hard work

kazurayam · April 10, 2022, 4:08am

I have released the v0.7.0 of “Visual Inspection in Katalon Studio”

https://github.com/kazurayam/VisualInspectionInKatalonStudio_Reborn/releases/tag/0.7.0

I added a small feature: Interactively memorizing ignorable diffs

Interactively memorizing ignorable diffs

You can interact with the report generated by the Visual Inspection. See the top-left corner of the report. You will find the count of “diff” entities are displayed.

Please click THIS to see a demo in action.

2w 0i 6t

This means:

There are 2 diff entities that have a diff-ratio greater than the criteria given.
There are 0 diff entities that have a diff-ratio greater than the criteria given, and is “checked to be ignorable”
There are 6 diff entities in total.

In other words, I created 6 materials (PNG screenshot and HTML source saved). I set 1.0% as the criteria of evaluating the diffRatio; if an entity has a diffRatio greather than 1.0, it will be marked warning , hence the figure 46.66% is highlighted with the background color of yellow.

A large diffRatio indicates to me that there must be some reason that caused significant differences; therefore I should look at the system to find out the reason of the difference. Definitely I will look into every entities warned. Then I would find out some reasons why the diffRatio is as large as 46.66% . No matter what the reason is, I would decide whether the entity is ignorable or not.

In fact the most of diffs detected by automated tests are ignorable for human. For example, when I compare a development environment against a production environment of a Web app, it is quite likely that there are some significant visual differences caused by the recent changes in the application software in the development environment.

Any visual difference caused by known factors are ignorable for human, and I want to differentiate them from the diffs caused by unexpected factors. Only human can decide if a visual difference is ignorable or not.

So my Visual Inspection tool does not decide it. But the tool offers you a feature of persisting your decision into the report and memorize it across sessions. Let me assume I have found out that the entity with 46.33% in the following screenshot is ignorable. Then I would interact with the report displayed in a browser. I will click the checkbox at the left end of the entity.

When the checkbox is checked, the diffRatio 43.66% changes its background color less highlighted. And the “Count” display on the top-left corner changes:

2w 0i 6t => 1w 1i 6t

This display indicates that I have 1 warning, 1 ignorable, among 6 entities in total.

I will continue debugging. I will look at the entity with the diffRatio 1.87% . Let me assume I found it ignorable again. Then I will check it also. Eventually the report would give the following display:

Now I have got

0w 2i 6t

I have zero warnings! Congrat! I have finished inspecting this web application!

The status of checkbox toggled by human is persisted into localStorage of browser. Therefore your decisions (which entities to be ignored) will be retained across browser sessions. You can reload the page, close and reopen the page. The checkboxes will keep the status you put previously.

2w 0i 6t => 1w 1i 6t => 0w 2i 6t — I would name this display as WIT — makes my inspection task spirally upward. It helps me gradually improve the target system through executing Visual Inspection repeatedly.

kazurayam · October 16, 2022, 2:21am

I have updated the “VisualTestingInKatalonStudio_Reborn” project to ver 0.10.3. In this version, you can specify the sorting order of entries in the report. See Visual Inspection in Katalon Studio - Reborn for more.

Other discussion motivated to develop the ver 0.10.3.

kazurayam · January 1, 2023, 12:03pm

I have made a new version of Visual Inspection.

Please see the following post:

The new version is empowered by a custom Gradle plugin com.kazurayam.inspectus4katalon which enables you to make your own Katalon projects capable of “Visual Inspection” quickly in an automated fashion.

I would no longer maintain the “Visual Inspection in Katalon Studio - Reborn” project.