Visual Inspection in Katalon Studio - Reborn

I have created a GitHub repository:

My previous work

Sep 2018, I published a project named “Visual Testing in Katalon Studio” at

Please have a look at the “Motivation” section in its README document to know why I made it. I tried to implement what I called “Visual Testing” in Katalon Studio. The project worked for me.

But I wasn’t satisfied with it. Why? I would enumerate 3 problems about that project.

  1. The codeset of Visual Testing In katalon Studio project is too complicated. The project contains 28 Test Cases, 7 Test Suites, 4 Test Suite Collections, 1 Test Listener, 12 Keywords. After 3 years, I forgot them and unable to maintain them any longer.
  2. The project enables me to compare a pair of 2 screenshots of Web pages in PNG image format. That’s all. Not enough. Often I wanted to compare 2 texts scraped from web. E.g, 2 HTML files as Web Page source; 2 JSON files downloaded from a Web Service.
  3. The project isn’t distributable to others easily because it isn’t packaged. It’s too hard for people to reuse the code set of the project to achieve their own “Visual Testing” for their custom targets.

My new development

In 2021 July-August, I have developed a new project named materialstore . It is a domain-specifc file system written in Groovy. In this project I have re-designed and re-implemented my idea from scratch. I have achieved the followings.

(1) Simpler codeset

The materialstore provides a self-contained API in Groovy, which encapsulates all of the useful functionalities of the previous Visual Testing In katalon Studio project. A single Test Case in Katalon Studio empowered by the materialstore library can achieve whole job equivalent to the previous one. No need to struggle with that many components (28 Test Cases + 7 Test Suites + 4 Test Suite Collections + 1 Test Listener + 12 Keywords).

(2) Domain-specific file system indexed by metadata

By the word material I mean any type of files downloaded from Web applications during automated tests. Screenshot images (PNG), Web page source (HTML), JSON and XML responded by Web Services — I call all of these as materials .

The materialstore provides capability of materializing (storing) files downloaded from web sites in a pre-designed directory structure (I call it the store ).

An application writes files into the “store” associating “metadata” . The “materials” in the “store” are indexed by the associcated metadata . An application retrieves files from the store by metadata as key. An application does not look up files by name (Path). The application is no longer responsible for deciding and remembering the path of physical files.

A metadata of a material in the store is merely an instance of java.util.Map<String, String> with arbitrary key=value pairs. You can programme any kind of metadata and associate it to each materials so that the materials are well described and clearly identified. For example, you can associate the URL from which the web resource was retrieved; or you can associate the name of browser (“Chrome”, “FireFox”, “Safari”, etc) which you used to take the screenshots; or you can associate the name of “Execution Profile” in Katalon Studio which you applied when you executed your Test Case.

Metadata composition is entirely up to the user application. Composing metadata is a bit difficult part of the materialstore library. It looks similar to designing database table in SQL-backed application.

(3) Packaged in a jar

The artifact of materialstore is distributed as a single jar file. The jar file is available at the Maven Central Repository. Any Java/Groovy-based application can automate downloading the materialstore-x.x.x.jar to resolve dependency using the build tools (Gradle, Maven, Ant).

(4) Usable outside Katalon Studio

The materialstore 's jar has no immediate dependency on the Katalon Studio API. It is not dependent even on the Selenium Webdriver API. So the materialstore library can be used in any Java/Groovy project, not only in Katalon Studio. I can use it in a plain Selenium WebDriver-based automated test project for Web UI on Gradle. I can use it in a plain Appium Java Client-based automated test project for Mobile UI on Maven. I can use it in a plain Apache HttpClient-based automated test project for Web Services on Ant.

Inspection, not Test

I named my previous project as “Visual Testing”. I think this naming was not very good. To me, a word testing means to check if a system behaves as I expect. On the other hand, inspection means to me looking around a system for anything unexpected observed. This project does not really test it , but it does help me to inspect it .

Now I want to name my new project as Visual Inspection . I think it is a better naming.

The documentation of “materialstore”

The following repository contains sample codes and documentation of the materialstore API. The sample codes are written in Java, run on Gradle + Selenium WebDriver + Apache HttpClient.

Examples in Katalon Studio

Here I will present 3 examples which show how to use the materialstore library in Kataloon Studio. I intend these will supercede my previous “Visual Testing in Katalon Studio” project.

Creating a project, resolving external dependencies

You can create a new Katalon Studio project, import the required external dependencies, and write your Test Cases for Visual Inspection. Let me describe the procedure how to create your “Visual Inspection Reborn” project.

  1. Install “Gradle” build tool into your PC. Please follow this guide to install Gradle on your machine.

You can use Gradle ver 7.x as well as ver 6.x.

  1. Open Katalon Studio GUI. Create a new project as usual in whichever directory you like.

I will write a symbol $projectDir to express this project directory.

  1. create $projectDir/build.gradle . You should copy and paste the source of:
  1. In the commandline (outside Katalon Studio GUI) you want to execute the following command:
$ cd $projectDir
$ gradle drivers
  1. The gradle driver will display some lines of messages in 10 seconds, and will finish successfully.
1 actionable task: 1 executed
  1. Once the command finished, in the $projectDir/Drivers directory, you will find some jar files are automatically imported.
$ tree Drivers
├── AUTOIMPORTED_ExecutionProfilesLoader-1.2.1.jar
├── AUTOIMPORTED_ashot-1.5.4.jar
├── AUTOIMPORTED_java-diff-utils-4.9.jar
├── AUTOIMPORTED_jsoup-1.13.1.jar
├── AUTOIMPORTED_materialstore-0.1.0.jar

The exact version number of the jars may change in future. Please regard this as an example.

All of these jar files are downloaded from the Maven Central Repository. These are required to run the “Visual Inspection” code in your new project in Katalon Studio locally.

If you are going to push your new project into Git repository, you should write .gitignore file so that it ignores the Drivers/ directory.

  1. Stop Katalon Studio, restart it and re-open the project.

You have resolved external dependencies. Now you can start writing a Test Case.

Sample1: simply visit a URL and scrape

First example. We will write a Test Case in Katalon Studio that visits the Google Search page. We will take screenshots and HTML page sources of the Web page. We will store PNG files and HTML files into the store directory using the materialstore library. We will finally generate a HTML file in which we can view the stored files.

(1) Test Case

You want to newly create a Test Case Test Cases/main/GoogleSearch/scrapeGoogleSearch in your project. Copy and paste the following sample source:

Once you have created the Test Case, you want to run it as usual by clicking the green button run button in Katalon Studio GUI.

(2) The report generated

Once the Test Case finished, a HTML file will be created at store/scrapeGoogleSearch.html . Please open it in any web browser. It renders a view of the stored 6 files. You can see an working example here: pls. click here.


(3) The “store” directory structure

When the Test Case finished, you will find a new directory $projectDir/store is created. In there you will find a tree of directories and files, like this:

$ tree store
├── scrapeGoogleSearch
│   └── 20210813_221052
│       ├── index
│       └── objects
│           ├── 01014deef318115a75ac1c3ab0f9844832c81c86.html
│           ├── 02625f7607199d99ca58b803d6fe51b7c94835e7.html
│           ├── 2563a225cb7bcd438ae12a6126b2091eb8e09e7d.png
│           ├── 5c002fbe44438341d3d92832d1e004198153976b.png
│           ├── 8370ecd0081e1fb9ce8aaecb1618ee0fc16b6924.html
│           └── efaed8443417a62faf35ee9d9b858592cd67bbae.png
└── scrapeGoogleSearch.html

(4) The objects/ directory

  • Under the store/scrapeGoogleSearch/yyyyMMdd_hhmmss/objects/ directory, there are 6 files. Among them you will find 3 files with postfix png . These are the screenshot of web pages in PNG image format. Also you will find 3 files with postfix html . These are HTML sources of web pages.
  • The file name comprises with 40 hex-decimal characters appended with extension ( .png , .html ). The hex-decimal string (I call this “ID”) is derived from the file content without compression by SHA1 Secure Hash algorithm.

(5) The index file

  • The store/scrapeGoogleSearch/yyyyMMdd_hhmmss/index file would be interesting. An example of the index file is like this:
8370ecd0081e1fb9ce8aaecb1618ee0fc16b6924	html	{"":"", "URL.path":"/", "URL.protocol":"https"}
2563a225cb7bcd438ae12a6126b2091eb8e09e7d	png	{"":"", "URL.path":"/", "URL.protocol":"https"}

The index file is a plain text file. Each lines corresponds to each files stored in the objects directory.

A line of the index file has 3 parts delimited by TAB characters.

<SHA1 Hash value of each file>\t<file type>\t<metadata>

(6) The metadata

In the test Case script, the code created metadata for each objects. Typically a metadata will include information derived from the URL of the source Web Page. For example, an URL


will be digested to form a metadata

  • {"":"", "URL.path":"/", "URL.protocol":"https"}

Plus, you can add any key-value pair into the metadata as you like.

In the index file, lines are sorted by the ascending order of metadata text.

The materialstore API restricts that metadata texts in a index file MUST be unique. Your application can not create multiple objects (= multiple lines in the index file) with the same metadata value.

Sample2: Visual Inspection in Chronos mode

Second example. We will write a Test Case in Katalon Studio that visits the The top page displays a current timestamp in the unit of seconds. So everytime you retrieve this page, the page changes slightly at the timestamp portion.

How is the current page of a web system different from what it was previously since 5 minutes ago, 3 hours ago, or 2 days ago? I want to see the differences in HTML code, not only visually as screenshots comparison. — the second example will show you how to implement a test that helps.

(1) Test Case

In your project, you want to copy&paste the following Test Case code.

You will execute the Test Case two times. You run it once; wait for some period (seconds, minuits, hours, or days, … up to you); then run it again. The Test Case will preserve the output of previous runs.

(2) Execution Profile

This Test Case expects an Execution Profile is provided like this:

  • Profiles/CURA_DevelopmentEnv


Please create it in your project.

(3) The “store” directory

You will get 2 directories named in the format of yyyyMMdd_hhmmss . The directories will look like this:

$ tree store
├── CURA_VisualInspectionChronos
│   ├── 20210814_143440
│   │   ├── index
│   │   └── objects
│   │       ├── 081a2a3e3174f13f60a9707f9464d1d73339d0e3.html
│   │       ├── 4d671f2cd14839164840a520cb185c2d1bb68586.png
│   │       ├── 504259db4e6562f62f8f70a6991fbce42aa55407.html
│   │       ├── 668b2a28455d9524fc1da35317f44d3797ea5344.html
│   │       ├── 6b82ac13ae98ca3c055d28469b75c8f377c1d8b1.png
│   │       ├── 6b8a699d921520c10b9e7e61cf62e528bf263fa8.html
│   │       ├── 8dd629a5d1b542b80bebc571c1ee35d27157ea1e.html
│   │       ├── a11357d7fbc475f7fefabd50f468b9b4fd4e8b35.png
│   │       ├── ed401b9bff2a687e41cedd9a63b3dd15880964eb.png
│   │       └── f8664dfa1c3a3af60914c4c90ddfaf0286910133.html
│   └── 20210814_143722
│       ├── index
│       └── objects
│           ├── 01746787bffa40e2ae0997da12e66c3f6ed50a87.png
│           ├── 25e752bc500be3a94fad9a4e6fc3ebc25431b558.png
│           ├── 3d56d836f5c778d0ec31f72fd74fcc5d3cb348fd.html
│           ├── 4d671f2cd14839164840a520cb185c2d1bb68586.png
│           ├── 582791931452081590447c71d8769ae185ff7cb9.html
│           ├── 668b2a28455d9524fc1da35317f44d3797ea5344.html
│           ├── 6b82ac13ae98ca3c055d28469b75c8f377c1d8b1.png
│           ├── e18882a39ba9632e9b0018e642d90a289fd1bcb7.html
│           ├── f6b51712f25ba02a5b39eedb30a57b31d455eafc.html
│           └── f8664dfa1c3a3af60914c4c90ddfaf0286910133.html
└── CURA_VisualInspectionChronos-index.html

(4) making Chronological diff

The Test Case compares takes materials at chronologically different timings, and then compare the 2 sets.

Test Cases/CURA/VisualInspectionChronos does the followng processing:

  1. make a List of materials in the current timestmp directory
  2. make another List of materials in the previous timestamp directory
  3. make differences between the two lists, store the diff info into the store as well
  4. generate a HTML report, where you can view the detail with diff information.

(5) The report generated

Once the Test Case finished, a HTML file will be generated at store/CURA_VisualInspectionChronos-index.html . Please open it in any web browser. It renders a view of the stored 6 files. You can see an working example here: pls. click here.

You can see examples of screenshot comparison result: Previous screenshot - diff image - current screenshot.

Chronos Left

Chronos Diff

Chronos Right

Also you can see examples of HTML page sources comparison result.

Chronos HTML

Sample3: Visual Inspection in Twins mode

The 3rd example visits 2 URLs. These are useless pages solely for this example.

The former URL is meant to represent a Production environment of a web system, the latter URL is meant to represent a Development environment . The pages look similar at a glance, but are different in detail.

How the pages of 2 environments differ now? — the 3rd example will show you.

(1) Test Case

In your project, you want to copy the followiing Test Case code.

You will execute the Test Case only once. The Test Case will visit the 2 URLs as one batch. And it creates a HTML report.

(2) Execution Profile

The Test Case script expects 2 Execution Profiles are provided like this:

  • Profiles/MyAdmin_DevelopmentEnv profile_MyAdmin_Dev
  • Profiles/MyAdmin_ProductionEnv profile_MyAdmin_Prod

Please create them in your project.

(3) The “store” directory

Once finished, the Test Case creates a directory named in the format of yyyyMMdd_hhmmss . The directory looks like this:

$ tree store
├── MyAdmin_VisualInspectionTwins
│   └── 20210814_224127
│       ├── index
│       └── objects
│           ├── 2cf3afe9ff4104e4055f0dc2bff53b9166e80a0c.html
│           ├── 4cbd1ac3812a5251325202f86a8dfe76bc82dbdc.png
│           ├── 5e3ff331ef4ff3feb3222f0d2951aed284e47ac2.png
│           ├── adb4c03ac0f0928a4dff3328845c4db460fc72b5.html
│           ├── b0456c0ae964825a1508b28ac1042340b29c9357.html
│           └── e911ce587553fc42f24bf18279ee3f5214eb75ba.png
└── MyAdmin_VisualInspectionTwins-index.html

(4) makiing diffs of Twins

The Test Case looks up 2 set of materials and compare them.

Test Cases/main/MyAdmin/VisualInspectionTwins does the following processing:

  1. Amongst the stored materials, select 2 (a screenshot and a HTML) of the Development environment to make a List.
  2. Also select 2 of the Projection environment to make a Liist.
  3. do take differences between the two lists.
  4. will insert files into the “store”, which contains the diff information.
  5. generate a HTML report, where you can view the detail with diff information.

How can I select materials amongst many appropriately and make pairs to compare?

Well, it is difficult to explain. I wonder if I could describe it in English, but let me try.

In the objects directory, there are 6 files. These are a mixture of materials downloaded from 2 URLs. How can I tell which file is from which URL? — The index file keeps enough information. The content of index file is as follows:

In there you can find the following 2 lines. These lines point to the PNG screenshots of the 2 environments. I call the 3 column delimited by \t ( {"": ... :"body"} ) as metadata .

5e3ff331ef4ff3feb3222f0d2951aed284e47ac2	png	{"":"", "URL.path":"/", "URL.protocol":"http", "profile":"MyAdmin_DevelopmentEnv", "selector":"body"}
4cbd1ac3812a5251325202f86a8dfe76bc82dbdc	png	{"":"", "URL.path":"/", "URL.protocol":"http", "profile":"MyAdmin_ProductionEnv", "selector":"body"}

These 2 lines have a common portion in the metadata :

{"URL.path":"/", "selector":"body"}

Also they have a different portion:

{{"":"", "profile":"MyAdmin_DevelopmentEnv"}
{{"":"", "profile":"MyAdmin_ProductionEnv"}

The Test Case script can select 2 lines as a pair amongst others by looking at their metadata . I would ignore the known different portions in the metadata resulting a common portion. Taking the common portion as a key of lookup, I could find a pair or materials to compare.

Difficult to understand? Well, I agree. But this is the best flexible and expressive method I could develop.

(5) The report generated

Finally the Test Case will generate a HTML report of which path is store/MyAdmin_VisualInspectionTwins-index.html . Please open it in any web browser. It renders a view of the stored 1 page, both in PNG screenshot and HTML page source. You can see an working example here: here

You can see a screenshot comparison result: Production Env - Diff - Development Env.




Also you can see the diff of HTML page source.


Notes on Extensibility

The materialstore library is usable for various cases. Once the materials are saved into the store, same diff + reporting features work.

for Mobile

The materialstore library can be used by a Appium-based Mobile testing in Java. All you need to do is to take screenshots of Mobile app using Katalon Studio’s built-in keyword (e.g, Mobile taksScreenshot) and save the image into the store .

for Web Service

The materialstore library can be used by a Apache HttpClient-based Web Service testing in Java. Using Katalon Studio’s sendRequest keyword, you can get a response. Then your script extract the response body (JSON, XML) and save them into the store .


The Visual Testing In Katalon Studio project is now superseded by this Visual Inspection In Katalon Studio - Reborn project backed by the materialstore library.


Aprin 2022, The Sample4 has now its own GitHub project at

Sample4: Navigating multiple URLs in Twins mode

The sample3 showed how to compare just a single pair of URLs. The next sample4 shows how to compare multiple pairs of URLs. The sample4 shows how to navigate though a site for 7 pages while taking screenshots and HTML sources.

(1) What the test case does?

The test case Flaskr/VisualInspectionTwins does the following:

  1. It compares 2 web sites visually: and .
  • (You can setup these URLs on your PC locally. I will explain how to later in this document.)
  1. A web application named Flakr run in the URLs. Let me call them "Flaskr Prod env " and "Flaskr Dev env ". These 2 URLs have just the same functionality. But the pages look slightly different. The production environment has no site logo, but the development environment has a small site logo. logo.
  • the Prod env Prod
  • the Dev env Dev
  1. The test case starts visiting the index page, then navigates through pages by clicking menu anchors, typing texts, clicking buttons, etc.
  1. The test case compares each pairs of URLs and generates diff images. The following is an example of a diff image. Please find the section on the top-left side painted red, which is the diff. diff
  2. The test case generates a HTML report which shows a list all of the materials (screenshot images and HTML page sources) attached with detail diff information. You can find an example here.


(2) How to run the test

Just open the “Test Cases/Flaskr/VisualInspectionTwins” and run it. As default Chrome Headless browser git will be used, but you can choose any browser. You can choose any Execution Profile. The test won’t be affected by the profile you chose.

The test case will take approximately 30 seconds to finish.

The test case will write the report in the <projectDir>/store/Flaskr_VisualInspectionTwins-index.html file.

(3) What is Flaskr?

Flaskr is coded in Python language on top of the “Flask” web application framework. I learned the Flaskr web app at the Flask Tutorial authored and published by the Pallets project.

This tutorial will walk you through creating a basic blog application called Flaskr. Users will be able to register, log in, create posts, and edit or delete their own posts.

I used the source code 99% as is. I amended it slightly to diplay the site logo in the Dev env only.

(4) How the test is coded

You can read the sources

and a lot of related Groovy classes

Why do I have these Groovy classes? — It is because I employed the “Page Object Model”. The Page Object Model helped me in writing compact and readable codes.

The sample4 is highly extensible while avoiding code publication. It would be a foundation of large scale Visual Inspection projects.

April 2022, The sample4 which requires Docker. now has its own GitHub project,

Setup Docker & Flaskr on your PC

In the sample4, we need 2 URLs available on our own PC.

You can make it work on your PC. You need Docker installed into your PC.

Installing Docker Desktop

On my MacBook Air, I installed Docker Desktop. Docker Desktop for Windows is also available.

Command Line Interface

Here I assume you work on a Command Line Interface. I use the of macOS. For Windows user, install Git for Windows which bundles “Git Bash”.

Starting up Flaskr

Open a window of Command Line Interface, then execute:

$ cd $VisualInspectionInKatalonStudio_Reborn
$ ./

This shell script will emit 2 lines of messages and will block:

you can visit
Serving on

Next, you want to open one more window of Command Line Interface, then execute:

$ cd $VisualInspectionInKatalonStudio_Reborn
$ ./

This will emit 2 lines of messages and will block:

$ ./
you can visit
Serving on

By lsof command, you can make sure that 2 processes are listening to the IP port #80 and #3090 on your localhost.

$  lsof -i -P | grep LISTEN | grep com.docke
com.docke   709 kazuakiurayama  107u  IPv6 0x84f53716e8d7cb33      0t0  TCP *:3090 (LISTEN)
com.docke   709 kazuakiurayama  111u  IPv6 0x84f53716e8d771d3      0t0  TCP *:80 (LISTEN)

Shutting down Flaskr gracefully

You can stop the docker process gracefully by typing CTRL + C.

You should NEVER close the window of Command Line Interface without stopping the docker process by CTLR + C.

If you forced to close the window, then IP Port #80 and #3090 might be left USED status. In that case you would fail to start a new process of Flaskr again. When it occurred, you need to stop & restart your PC/OS to release the ports.

Initializing Database in the web app

The Flaskr has a backend database where credentials and blog posts are stored. After you repeated running tests several times you will find many blog posts are stored, and you would feel like to clean the database out.

Just type CTRL + C to stop the docker process and restart it. The start-up script will automatically initialize the internal database and make it empty.

How I used Docker

Hava a look at the code of

cd $(mktemp -d)

echo you can visit

docker run -it -p 80:8080 kazurayam/flaskr-kazurayam:1.1.0

cd $CWD

As you see, it runs docker run command with a docker image kazurayam/flaskr-kazurayam:1.1.0 . I created this docker image and published at the Docker Hub.

I continued development to inspect CSS and JavaScript referred by web pages.

Hi @kazurayam
I’m interested in functionality to support comparison of multiple URLs in Twins mode, since sometime a SAAS application will make changes that our company has no control over and it would be useful to supplement functional automated tests. This work looks great, although I will need some time to get my head around it. Can I ask whether there is a need for Docker or Python in the project or is this just for illustration? Working in a restricted environment without Gradle, so I’m unable to run the example and also installing dependencies by hand. I assume that if I wanted to make my own test cases I would need to replace the POM versions you have (eg Include\scripts\groovy\com\kazurayam\uitestjava\flaskr\pom\actions)? Just a note that the groovy script URLs in Sample 4, Section 4 are not valid (give a 404, I found them in the project though). Thanks Dan

Only the “Sample4 Flaskr” requires Docker. The “visual inspection” framework does not require Docker. The Sample1, 2, 3 do not require Docker either.

See for more info.

The source code in the Include/scripts/groovy/com/kazurayam/uitestjava/flaskr/ folder is a part of the Sample4. It is not a part of the “visual inspection” framework. If you are going to develop your own test suite, your code will not use Include/scripts/groovy/com/kazurayam/uitestjava/flaskr/ at all.

You misunderstood.

The URL “” and “” are the right ones. That’s what I prepared for the Sample 4. These URLs will be provided by the Docker container that runs on your localhost( The Test Listeners/TLFlaskr silently&automatically starts/stops the Docker container on your PC for the Sample4 provided that you have installed “Docker for Desktop” into your PC.

If you want to see the Sample4 running, you need to install Docker into your PC. But if you do not want the Sample4, you won’t need Docker.

It’s OK not to use Gradle as far as you can manually download the required versions of external jars into the Drivers folder.

However please note that I frequently update the materialstore and other jars. Quite sometimes I would change the “visual inspection” framework’s API drastically. These are still in unstable status. The consistency of the jar versions really matters. Therefore you had better use Gradle to manage the external dependencies consistently.

By the way, yesterday I published the latest materialstore-0.3.5.jar but the Maven Central has not yet accepted, has not published it yet on their download site. It would take a few more days before the v0.3.5 of materialstore available publicly available.


I wrote

My Groovy/Java code (including inside Katalon Studio) can start & stop any Docker Container on the localhost( Don’t you find it interesting?

If you have a web app of yours (in any language: Python, Node, Ruby, .NET, Java/Groovy) in a Docker image, you can start it as a Docker process on your PC at, and run your test suites with Selenium-java against that URL.

See the following doc for more detail:

This was difficult, required several months of my development efforts.


Thanks for clarifications, and thanks for your hard work :grinning:

I have released the v0.7.0 of “Visual Inspection in Katalon Studio”

I added a small feature: Interactively memorizing ignorable diffs

Interactively memorizing ignorable diffs

You can interact with the report generated by the Visual Inspection. See the top-left corner of the report. You will find the count of “diff” entities are displayed.

Please click THIS to see a demo in action.

Count initial

  • 2w 0i 6t

This means:

  • There are 2 diff entities that have a diff-ratio greater than the criteria given.
  • There are 0 diff entities that have a diff-ratio greater than the criteria given, and is “checked to be ignorable”
  • There are 6 diff entities in total.

In other words, I created 6 materials (PNG screenshot and HTML source saved). I set 1.0% as the criteria of evaluating the diffRatio; if an entity has a diffRatio greather than 1.0, it will be marked warning , hence the figure 46.66% is highlighted with the background color of yellow.

A large diffRatio indicates to me that there must be some reason that caused significant differences; therefore I should look at the system to find out the reason of the difference. Definitely I will look into every entities warned. Then I would find out some reasons why the diffRatio is as large as 46.66% . No matter what the reason is, I would decide whether the entity is ignorable or not.

In fact the most of diffs detected by automated tests are ignorable for human. For example, when I compare a development environment against a production environment of a Web app, it is quite likely that there are some significant visual differences caused by the recent changes in the application software in the development environment.

Any visual difference caused by known factors are ignorable for human, and I want to differentiate them from the diffs caused by unexpected factors. Only human can decide if a visual difference is ignorable or not.

So my Visual Inspection tool does not decide it. But the tool offers you a feature of persisting your decision into the report and memorize it across sessions. Let me assume I have found out that the entity with 46.33% in the following screenshot is ignorable. Then I would interact with the report displayed in a browser. I will click the checkbox at the left end of the entity.

Count checked ignorable

When the checkbox is checked, the diffRatio 43.66% changes its background color less highlighted. And the “Count” display on the top-left corner changes:

  • 2w 0i 6t => 1w 1i 6t

This display indicates that I have 1 warning, 1 ignorable, among 6 entities in total.

I will continue debugging. I will look at the entity with the diffRatio 1.87% . Let me assume I found it ignorable again. Then I will check it also. Eventually the report would give the following display:

Count 3 done all

Now I have got

  • 0w 2i 6t

I have zero warnings! Congrat! I have finished inspecting this web application!

The status of checkbox toggled by human is persisted into localStorage of browser. Therefore your decisions (which entities to be ignored) will be retained across browser sessions. You can reload the page, close and reopen the page. The checkboxes will keep the status you put previously.

2w 0i 6t => 1w 1i 6t => 0w 2i 6t — I would name this display as WIT — makes my inspection task spirally upward. It helps me gradually improve the target system through executing Visual Inspection repeatedly.

1 Like