Visual Testing in Katalon Studio

One more use case of the Materials library

Here I used it for testing RESTful API

1 Like

(edit at May 2019)
The version 1.10.0 of VisualTestingInKatalonStudio has equipped with a easier method of enabling your own Katalon Studio project to do “Visual Testing”. See https://github.com/kazurayam/VisualTestingInKatalonStudio#how-to-make-your-katalon-studio-project-capable-of-visual-testing section.

The “Step by step instruction” document got obsolete in v1.10.0, therefore was removed.


I have written and published a new document

Step by step instruction - how to create a visual testing project from scratch

https://github.com/kazurayam/VisualTestingInKatalonStudio/blob/master/docs/Step_by_step_instruction.md

In this document I have tried to describe in detail how to get started with “Visual Testing in Katalon Studio” — comparing 2 almost identical sites by taking screenshots and making image differences.

I believe that automated visual regression testing is an effective method to detect unplanned/accidental changes made somehow in your Web app.

Please try.

7 Likes

+10!

2 Likes

Katalon Studio “The Missing Manual” -> this thread! B)

1 Like

For everyone reading this, @kazurayam has added a follow-up article:

1 Like

Don Welsh raised an interesting question, so I replied to it. Let me share the discussion here.

1 Like

I did not took the time to read this thread before. And I feel bad now cause it is very good and add an important value to Katalon. And to my projects.
You did a pretty fine job, @kazurayam

1 Like

I have updated VisualTestingInKatlaonStudio to version 1.8.0.

This version supported a new feature “ignoring web elements”. See Step by step instruction, section #10 for detail.

You can specify specific HTML elements to be painted with grey rectangles in the screenshot as follows.

The elements painted grey are effectively ignored when a pair of screen shots are compared.

2 Likes

Hi @kazurayam

Wonderful feature. This is irrelevant to that, but I suggest you create some sort of index (or table of content) since your README file is long. I believe it will prove useful to people who are looking for particular features to identify if your project offers it immediately.

Regards !

1 Like

I have added Table Of Contents to the Step by step instruction document. TOC is generated by DocToc

3 Likes

wow, a single question get this huge effort from you @kazurayam
thanks for this huge effort :heart_eyes::heart_eyes:

1 Like

good work.

1 Like

Hi @kazurayam,

Thank you for the great work.

I’m looking at using this in a project, but I have two questions/problems. I was wondering if you or anyone else has any ready-made solutions for these issues?

(1.) Is there an easy way to compare screenshots that aren’t taken at the same time and aren’t the most recent screenshots?

In the sample project you provided, screenshots are taken from two different environments and then compared. Instead of taking screen shots from two different environments at the same time, I would like to compare screenshots from different releases of the same environment. For example:

  • 2019-01-01: Take screenshot of website after v1.0 release.
  • 2019-02-30: Take screenshot of website after v1.1 release and diff with old screenshot.

Basically, I want to remove the “compare two most recent materials” restriction. Is there an easy way to do this currently without making major changes to the Materials library?

(2.) Is there a easy way to mark tests as failed if images are different?

In the sample project, diffing the images is done after the actual tests are run, so it’s not possible to mark the tests as “failed” if the images differ. Is there an easy way to diff the images during execution and mark tests as failed if the images are different?

Regards,
eb

1 Like

I am currently working on enhancement of “Visual Testing in Katalon Studio”. The typical usecase in my mind is to compare screenshots Before and After any work on a production Web system.

I would take a set of screenshots (may be some hundreds of pages) before any work (application upgrade, applying security patches, DB data migration, network resources reconfiguration, etc) around my Web App. After a few hours once the work is done, I would take another set of screenshots of the same Web App. I will compare the 2 sets of images. If the 2 sets are completely identical, ok, I can go home with relief. But when the comparison revealed any significant image differences, I must have made silly mistakes and caused unplanned/unexpected changes in the system. I will go back to the system and will fix problems BEFORE my customers notice it.

I think I can release the new version in a month by the end of April 2019. This enhancement required quite significant changes of the Materies library, but the job has been done and near completion.

My current idea is to comapre the latest status of a web system against a previous status specifying a timestamp. Your requirement (@eb1, comparing 2019-01-01 and 2019-02-30) is a bit different, but principally similar type of problem. I can imagine a future enhancement to support comparing 2 time slices specifying 2 timestamps.

The coming new version will be capable of marking tests as failed if “one or more significant enough differences found”.

2 Likes

@kazurayam

I am currently working on a enhancement. The typical usecase in my mind is Compare screenshots Before and After any work on a production system .

The coming new version will be capable of marking tests as failed if “one or more significant enough differences found”.

Awesome! Good to know. I will look forward to the next release!

Is there an easy way to compare screenshots that aren’t taken at the same time and aren’t the most recent screenshots?

After a bit of thought, I realized that it is possible to fulfill the first part of this requirement by making a few changes to the test suite.

Workaround: how compare images across two different test executions:

Changes:

  1. Delete the call to “Test Suites/cleanMaterials” in the test suite. This prevents our old screenshots from being deleted.
  2. Delete one of the two calls to “Test Suites/Main/TS1”. Leave the one that uses the profile of the AUT you want to test. This way only one set of screenshots will be taken on each test execution

Execution workflow:

  1. Run the tests on your pre-deploy “Before” app.
    • “Before” images will be saved to the materials folder with timestamp
    • “index” file will be generated but no image diff will be performed
  2. Deploy the changes to your app.
  3. Run the tests on your post-deploy “After” app.
    • “After” images will be saved to the materials folder with the most recent timestamp
    • The most two recent sets of screen shots will be diffed, and the diff results will be written to the “index” file.
      • note: If you have run the tests between steps #1 and #3, you will need to move or delete any sets of screenshots that were saved between your “Before” and “After” screenshots

Thank you again for putting this all together, and I’ll look forward to the upcomming release!

-eb

1 Like

This is great - nice job @kazurayam

1 Like

Last weekend, the new version of “Visual Testing in Katalon Studio” has come up and running on my Mac. Still I need several weekends to write sufficient documentations.

3 Likes

Would it be premature to share it under “develop” or a feature branch on github?

Thanks,
eb

1 Like