Visual Testing in Katalon Studio

Product Forums Katalon Studio
,
21

I did not took the time to read this thread before. And I feel bad now cause it is very good and add an important value to Katalon. And to my projects.
You did a pretty fine job, @kazurayam

1 Like
22

I have updated VisualTestingInKatlaonStudio to version 1.8.0.

This version supported a new feature “ignoring web elements”. See Step by step instruction, section #10 for detail.

You can specify specific HTML elements to be painted with grey rectangles in the screenshot as follows.

step10_stats_painted_grey
step10_stats_painted_grey1265×218 15.3 KB

The elements painted grey are effectively ignored when a pair of screen shots are compared.

2 Likes
23

Hi @kazurayam

Wonderful feature. This is irrelevant to that, but I suggest you create some sort of index (or table of content) since your README file is long. I believe it will prove useful to people who are looking for particular features to identify if your project offers it immediately.

Regards !

1 Like
24

I have added Table Of Contents to the Step by step instruction document. TOC is generated by DocToc

3 Likes
25

wow, a single question get this huge effort from you @kazurayam
thanks for this huge effort :heart_eyes::heart_eyes:

1 Like
26

good work.

1 Like
27

Hi @kazurayam,

Thank you for the great work.

I’m looking at using this in a project, but I have two questions/problems. I was wondering if you or anyone else has any ready-made solutions for these issues?

(1.) Is there an easy way to compare screenshots that aren’t taken at the same time and aren’t the most recent screenshots?

In the sample project you provided, screenshots are taken from two different environments and then compared. Instead of taking screen shots from two different environments at the same time, I would like to compare screenshots from different releases of the same environment. For example:

  • 2019-01-01: Take screenshot of website after v1.0 release.
  • 2019-02-30: Take screenshot of website after v1.1 release and diff with old screenshot.

Basically, I want to remove the “compare two most recent materials” restriction. Is there an easy way to do this currently without making major changes to the Materials library?

(2.) Is there a easy way to mark tests as failed if images are different?

In the sample project, diffing the images is done after the actual tests are run, so it’s not possible to mark the tests as “failed” if the images differ. Is there an easy way to diff the images during execution and mark tests as failed if the images are different?

Regards,
eb

1 Like
28

I am currently working on enhancement of “Visual Testing in Katalon Studio”. The typical usecase in my mind is to compare screenshots Before and After any work on a production Web system.

I would take a set of screenshots (may be some hundreds of pages) before any work (application upgrade, applying security patches, DB data migration, network resources reconfiguration, etc) around my Web App. After a few hours once the work is done, I would take another set of screenshots of the same Web App. I will compare the 2 sets of images. If the 2 sets are completely identical, ok, I can go home with relief. But when the comparison revealed any significant image differences, I must have made silly mistakes and caused unplanned/unexpected changes in the system. I will go back to the system and will fix problems BEFORE my customers notice it.

I think I can release the new version in a month by the end of April 2019. This enhancement required quite significant changes of the Materies library, but the job has been done and near completion.

My current idea is to comapre the latest status of a web system against a previous status specifying a timestamp. Your requirement (@eb1, comparing 2019-01-01 and 2019-02-30) is a bit different, but principally similar type of problem. I can imagine a future enhancement to support comparing 2 time slices specifying 2 timestamps.

The coming new version will be capable of marking tests as failed if “one or more significant enough differences found”.

2 Likes
29

@kazurayam

I am currently working on a enhancement. The typical usecase in my mind is Compare screenshots Before and After any work on a production system .

The coming new version will be capable of marking tests as failed if “one or more significant enough differences found”.

Awesome! Good to know. I will look forward to the next release!

Is there an easy way to compare screenshots that aren’t taken at the same time and aren’t the most recent screenshots?

After a bit of thought, I realized that it is possible to fulfill the first part of this requirement by making a few changes to the test suite.

Workaround: how compare images across two different test executions:

Changes:

  1. Delete the call to “Test Suites/cleanMaterials” in the test suite. This prevents our old screenshots from being deleted.
  2. Delete one of the two calls to “Test Suites/Main/TS1”. Leave the one that uses the profile of the AUT you want to test. This way only one set of screenshots will be taken on each test execution

edited%20test%20suite
edited%20test%20suite1804×664 95 KB

Execution workflow:

  1. Run the tests on your pre-deploy “Before” app.
    • “Before” images will be saved to the materials folder with timestamp
    • “index” file will be generated but no image diff will be performed
  2. Deploy the changes to your app.
  3. Run the tests on your post-deploy “After” app.
    • “After” images will be saved to the materials folder with the most recent timestamp
    • The most two recent sets of screen shots will be diffed, and the diff results will be written to the “index” file.
      • note: If you have run the tests between steps #1 and #3, you will need to move or delete any sets of screenshots that were saved between your “Before” and “After” screenshots

Thank you again for putting this all together, and I’ll look forward to the upcomming release!

-eb

1 Like
30

This is great - nice job @kazurayam

1 Like
31

Last weekend, the new version of “Visual Testing in Katalon Studio” has come up and running on my Mac. Still I need several weekends to write sufficient documentations.

3 Likes
32

Would it be premature to share it under “develop” or a feature branch on github?

Thanks,
eb

1 Like
34

@manpreet.mukkar

Please have a look at the following posts:

3 Likes
35

Let me announce the release of new version 1.10.0 of VisualTestingInKatalonStudio project.

Changes include:

  1. Chronological approach of screenshot-comparison testing of a single URL, namely “Chronos”, was newly supported in addition to the existing approach of comparing 2 URLs (dev & production) of a web application, namely “Twins”.
  2. Materials/index.html now presents 3 screenshot images (Back > diff > Forth) in Carousel format.
  3. Gradle Plugin com.github.kazurayam.visualtestinginks has been newly developed. This Gradle Plugin makes resources (Test Cases, Test Suites, Test Objects, Test Listeners, Keywords) of VisualTestingInKatalonStudio portable to other projects.

Please start with the revised README doc

9 Likes
36

Great stuff @kazurayam thank you for sharing!

1 Like
37

Thanks. Awesome work @kazurayam. Much Appreciated!

1 Like
38

Katalon Studio v6.1.5 was released. This version includes:

  • Added the capability to display all user folders/files in Test Explorer.

5_supports_rendering_any_folder_any_text
5_supports_rendering_any_folder_any_text3840×2080 1.41 MB

Now you can view the <projectDir>/Materials/index.html file generated by VisualTestingInKatalonStudio with browser inside the Katalon Studio’s GUI. You do not have to open a browser window outside KS.

I would recommend to you to upgrade KS to v6.1.5 if you are going to evaluate VisualTestingInKatalonStudio.

You can also edit and save .gitignore, README.md file. Any other type of text files including *.js, *.css, *.bat, *.sh, *.properties, *.xml and *.json. It seems that plain (no syntax-highlighting) editor is only available.

4 Likes
39

Here is an example output from VisualTestingInKatalonStudio in action

1 Like
40

Hello Kazurayam,

The link which is shared for step by step instruction for ignoring web elements - > Says page not found
https://github.com/kazurayam/VisualTestingInKatalonStudio/blob/master/docs/Step_by_step_instruction.md

Can you please share the new link if it is moved from current path.

1 Like
41

You want to visit the following url:

1 Like