Visual Testing in Katalon Studio

I am currently working on enhancement of “Visual Testing in Katalon Studio”. The typical usecase in my mind is to compare screenshots Before and After any work on a production Web system.

I would take a set of screenshots (may be some hundreds of pages) before any work (application upgrade, applying security patches, DB data migration, network resources reconfiguration, etc) around my Web App. After a few hours once the work is done, I would take another set of screenshots of the same Web App. I will compare the 2 sets of images. If the 2 sets are completely identical, ok, I can go home with relief. But when the comparison revealed any significant image differences, I must have made silly mistakes and caused unplanned/unexpected changes in the system. I will go back to the system and will fix problems BEFORE my customers notice it.

I think I can release the new version in a month by the end of April 2019. This enhancement required quite significant changes of the Materies library, but the job has been done and near completion.

My current idea is to comapre the latest status of a web system against a previous status specifying a timestamp. Your requirement (@eb1, comparing 2019-01-01 and 2019-02-30) is a bit different, but principally similar type of problem. I can imagine a future enhancement to support comparing 2 time slices specifying 2 timestamps.

The coming new version will be capable of marking tests as failed if “one or more significant enough differences found”.

2 Likes

@kazurayam

I am currently working on a enhancement. The typical usecase in my mind is Compare screenshots Before and After any work on a production system .

The coming new version will be capable of marking tests as failed if “one or more significant enough differences found”.

Awesome! Good to know. I will look forward to the next release!

Is there an easy way to compare screenshots that aren’t taken at the same time and aren’t the most recent screenshots?

After a bit of thought, I realized that it is possible to fulfill the first part of this requirement by making a few changes to the test suite.

Workaround: how compare images across two different test executions:

Changes:

  1. Delete the call to “Test Suites/cleanMaterials” in the test suite. This prevents our old screenshots from being deleted.
  2. Delete one of the two calls to “Test Suites/Main/TS1”. Leave the one that uses the profile of the AUT you want to test. This way only one set of screenshots will be taken on each test execution

Execution workflow:

  1. Run the tests on your pre-deploy “Before” app.
    • “Before” images will be saved to the materials folder with timestamp
    • “index” file will be generated but no image diff will be performed
  2. Deploy the changes to your app.
  3. Run the tests on your post-deploy “After” app.
    • “After” images will be saved to the materials folder with the most recent timestamp
    • The most two recent sets of screen shots will be diffed, and the diff results will be written to the “index” file.
      • note: If you have run the tests between steps #1 and #3, you will need to move or delete any sets of screenshots that were saved between your “Before” and “After” screenshots

Thank you again for putting this all together, and I’ll look forward to the upcomming release!

-eb

1 Like

This is great - nice job @kazurayam

1 Like

Last weekend, the new version of “Visual Testing in Katalon Studio” has come up and running on my Mac. Still I need several weekends to write sufficient documentations.

3 Likes

Would it be premature to share it under “develop” or a feature branch on github?

Thanks,
eb

1 Like

@manpreet.mukkar

Please have a look at the following posts:

3 Likes

Let me announce the release of new version 1.10.0 of VisualTestingInKatalonStudio project.

Changes include:

  1. Chronological approach of screenshot-comparison testing of a single URL, namely “Chronos”, was newly supported in addition to the existing approach of comparing 2 URLs (dev & production) of a web application, namely “Twins”.
  2. Materials/index.html now presents 3 screenshot images (Back > diff > Forth) in Carousel format.
  3. Gradle Plugin com.github.kazurayam.visualtestinginks has been newly developed. This Gradle Plugin makes resources (Test Cases, Test Suites, Test Objects, Test Listeners, Keywords) of VisualTestingInKatalonStudio portable to other projects.

Please start with the revised README doc

9 Likes

Great stuff @kazurayam thank you for sharing!

1 Like

Thanks. Awesome work @kazurayam. Much Appreciated!

1 Like

Katalon Studio v6.1.5 was released. This version includes:

  • Added the capability to display all user folders/files in Test Explorer.

Now you can view the <projectDir>/Materials/index.html file generated by VisualTestingInKatalonStudio with browser inside the Katalon Studio’s GUI. You do not have to open a browser window outside KS.

I would recommend to you to upgrade KS to v6.1.5 if you are going to evaluate VisualTestingInKatalonStudio.

You can also edit and save .gitignore, README.md file. Any other type of text files including *.js, *.css, *.bat, *.sh, *.properties, *.xml and *.json. It seems that plain (no syntax-highlighting) editor is only available.

4 Likes

Here is an example output from VisualTestingInKatalonStudio in action

1 Like

Hello Kazurayam,

The link which is shared for step by step instruction for ignoring web elements - > Says page not found
https://github.com/kazurayam/VisualTestingInKatalonStudio/blob/master/docs/Step_by_step_instruction.md

Can you please share the new link if it is moved from current path.

1 Like

You want to visit the following url:

1 Like

@rajani.gangadhara

Which version of the “VisualTestingInKatalonStudio” project are you looking at? I presume you are looking at v1.9.2 or earlier. The most up-to-date version is v.1.10.9.

At the version 1.10.0, I made a significant change to this project. I introduced a Gradle plugin visualtestinginks, which makes it very easy to “enable your own Katalon Studio project to perform Visual Testing — twins mode and chronos mode”. I changed completely the method to make your own project to be capable of Visual Testing. Therefore I DELETED the document “Step by step instruction” as its content is no longer valid as of v1.10.x.

1 Like

Now @rajani.gangadhara showed his interest to this project. I got much pleased, thank you, but a bit frightened. I must confess, the version 1.10.x has some compatibility problem. The users of 1.9.x may encounter problem upgrading to 1.10.0.

To be honest, I thought nobody (except myself) would be looking at this project, so I did not take care about compatibility at all.

2 Likes

Thanks a lot for the Quick reponse and your help :slight_smile:

1 Like

@kazurayam Hi can I use this for testing images within larger image? Say I have a full page screenshot, then I take a smaller section of the page and say find image1 inside image 2. I tried to do this a few years ago with but there were too many false positives.

Thanks.

1 Like

Hi,

For this use case you can try to search for Sikuli library which provides you with a set of APIs to find an image within another image.

Cheers !

1 Like

Hey Thanh, Sikuli was actually what I used back then and it wouldn’t work with finding say an element button in a full screen image.

1 Like

Hi.

What was the difficulties ? I tried Sikuli before and I think it works rather nicely.

1 Like