Visual Testing (image comparison) with Katalon Studio 7.8.0 and Katalon TestOps

Hello Katalon Devs,

I have started trying out the visual testing and Im impressed so far, thank you for your efforts. I still need more clarification please. When I run a different TestSuite with takeFullPageScreenshotAsCheckpoint(), this TestSuite is being compared to previous Baselines which don’t exist here.

Is there a way to separate each TestSuite’s Baselines so Katalon knows which screenshots to expect in each TestSuite?

Thank you

1 Like

Hi @Tarek,

I’ve updated the tutorial for Katalon Studio 7.7. I hope it would be helpful for you. If not, could you please give us more details (a piece of sample code would help a lot)?

2 Likes

Hi @devalex88,

Thank you for your reply. The link for Katalon TestOps Visual is down in : https://docs.katalon.com/katalon-studio/new/version-70.html#official-release---version-770---771

I have multiple test cases for multiple screens which requires different visual testing of course. I called:
WebUI.takeFullPageScreenshot(RunConfiguration.getReportFolder() + ‘/keyes-search.png’)
in the Search screen and saved the initial screenshot as my baseline.

However, if I execute the other test case independently and take a screenshot at the Search Result screen:
WebUI.takeFullPageScreenshot(RunConfiguration.getReportFolder() + ‘/keyes-searchResult.png’)
It shows that a screenshot is missing “keyes-search.png”, but I am of no need of the first screenshot.

Is there a way to separate the screenshots’. For example, a folder for the set of screenshots of Search screen and separate screenshots for the Search Result screen instead of having them in the same baseline?

1 Like

Hi @Tarek,
So you need a way to organizing your screenshots to a set of baseline collections and define which collection is used when run execution. Instead of comparing all baseline screenshots.

Hi @Thong,

Yes, that is exactly what I’m trying to explain, sorry if my wording wasn’t correct. Is it available right now with the current release? Or would it be a future feature perhaps?

Thank you

Hi @Tarek,

My team will work on it, it’s not available now.

1 Like

Hi @Thong,

Thank you for the consideration! I’m looking forward for this project. Can I please add one idea too: allow the user to delete/rename Baseline screenshots.

Thank you

2 Likes

A post was split to a new topic: Katalon Studio 7.7.1 bug

Received a report today via email of past week executions(assume from testops). Just an fyi… love this.

3 Likes

How do you delete baseline images? I see no option to delete. Also Katalon seems to add duplicate images to the baseline when there is a mismatch rather than “know” that some elements on the same page have failed. It other words, it creates a new baseline when it should not.

Hi. We will add a button for deleting baselines next week.

I’m sorry but could you please clarify the situation so that I can troubleshoot. Thank you!

1 Like

Katalon seems to think that mismatches on a page are new page. So rather than compare against the existing baseline, it creates a new baseline as if it were a new page added.

Hi @devalex88 , do you know if the delete option was already impelemented?

See my report at

1 Like

You can “re-baseline” image if you tick the green checkmark as correct, refresh the page and button save to baseline will be enabled, this will “re-baseline” your image :slight_smile: It’s a workaround for the delete of baseline images.

If you see one that says “missing” and you tick the green checkmark and click save to baseline it will “delete” it

1 Like

Hi @marryko Implementation has been done but it will take about two weeks to deploy.

@kevin.mcandrew1 The Keywords for Mobile are available now with Katalon Studio 7.8.0, and they can ignore regions when capturing screenshots. I added them to the first post.

2 Likes

@devalex88 That’s great news - thank you - I’ll check this out very soon :slight_smile:

@devalex88

So far looking good. The takeScreenshotAsCheckpoint command works well - no false positives so far (not sure if you have built-in tolerances or whether our app is pixel perfect each time?).

I didn’t try takeAreaScreenshotAsCheckpoint (not sure if I understand the rectangle aspect), and takeElementScreenshotAsCheckpoint didn’t work for me - the image it captured did not resemble the element I’d identified (I tried three separate objects).

I also couldn’t see how to exclude regions? Unless you mean by being able to capture an element or area that they are in effect excluding other areas?

Please can I make a couple of feature requests and report a possible small bug…

Possible bug: in the TestOps Visual Testing dashboard, within an execution, the ‘Save to Baseline’ button is not immediately enabled after accepting a new or changed image…as a workaround, refreshing the page enables this button.

New Feature Request #1: Would it be possible to take ‘full page’ screenshots (i.e. the full height, where a vertically-scrollable page is displayed) please?

New Feature Request #2: For excluding regions, would it be possible to take a screenshot of the whole app (either full-height, or as it is now), and be able to exclude specified element(s)? For example, in the following image (just a random example), the pink area might be some sort of container element for dynamic content that could change with every login? (obviously you may have covered such a scenario already?)

Thanks for your support :slight_smile:

Hi there,
Love this feature, think it has lots of potential and so far, for my minor use case (verifying a blob pdf view) it works great (Version 7.8.1).

However, 2 questions:

In my test I want to verify twice against the same baseline. So I have in my test check 1:

KeywordUtil.logInfo("1) Verify recap page (via elipsis action button) by visual testing = " + WebUI.takeScreenshotAsCheckpoint('blobVisualRecapPage', FailureHandling.CONTINUE_ON_FAILURE))

and somewhere later on check 2:

KeywordUtil.logInfo("2) Verify recap page (via recap page pdf button) by visual testing = " + WebUI.takeScreenshotAsCheckpoint('blobVisualRecapPage', FailureHandling.CONTINUE_ON_FAILURE))

However, in the executions view of the Visual Testing I only get to see 1 checkpoint!
Question one: I would expect to see 2 checkpoints, no?

Question two: If the screenshot does not match the baseline the test case itself will still pass. I presume because the actual verification happens only AFTER the test is finished and uploaded to the TestOps…
So we will have no failure in the test management executions, but we will have to monitor this in the visual testing executions. Isn’t there a way to make the test case fail in the test management executions, or make it at least visual over there?

Thanks in advance for any reply!