Visual Testing (image comparison) with Katalon Studio 7.8.0 and Katalon TestOps

Hi @Tarek,

My team will work on it, it’s not available now.

1 Like

Hi @Thong,

Thank you for the consideration! I’m looking forward for this project. Can I please add one idea too: allow the user to delete/rename Baseline screenshots.

Thank you

2 Likes

A post was split to a new topic: Katalon Studio 7.7.1 bug

Received a report today via email of past week executions(assume from testops). Just an fyi… love this.

3 Likes

How do you delete baseline images? I see no option to delete. Also Katalon seems to add duplicate images to the baseline when there is a mismatch rather than “know” that some elements on the same page have failed. It other words, it creates a new baseline when it should not.

Hi. We will add a button for deleting baselines next week.

I’m sorry but could you please clarify the situation so that I can troubleshoot. Thank you!

1 Like

Katalon seems to think that mismatches on a page are new page. So rather than compare against the existing baseline, it creates a new baseline as if it were a new page added.

Hi @devalex88 , do you know if the delete option was already impelemented?

See my report at

1 Like

You can “re-baseline” image if you tick the green checkmark as correct, refresh the page and button save to baseline will be enabled, this will “re-baseline” your image :slight_smile: It’s a workaround for the delete of baseline images.

If you see one that says “missing” and you tick the green checkmark and click save to baseline it will “delete” it

1 Like

Hi @marryko Implementation has been done but it will take about two weeks to deploy.

@kevin.mcandrew1 The Keywords for Mobile are available now with Katalon Studio 7.8.0, and they can ignore regions when capturing screenshots. I added them to the first post.

2 Likes

@devalex88 That’s great news - thank you - I’ll check this out very soon :slight_smile:

@devalex88

So far looking good. The takeScreenshotAsCheckpoint command works well - no false positives so far (not sure if you have built-in tolerances or whether our app is pixel perfect each time?).

I didn’t try takeAreaScreenshotAsCheckpoint (not sure if I understand the rectangle aspect), and takeElementScreenshotAsCheckpoint didn’t work for me - the image it captured did not resemble the element I’d identified (I tried three separate objects).

I also couldn’t see how to exclude regions? Unless you mean by being able to capture an element or area that they are in effect excluding other areas?

Please can I make a couple of feature requests and report a possible small bug…

Possible bug: in the TestOps Visual Testing dashboard, within an execution, the ‘Save to Baseline’ button is not immediately enabled after accepting a new or changed image…as a workaround, refreshing the page enables this button.

New Feature Request #1: Would it be possible to take ‘full page’ screenshots (i.e. the full height, where a vertically-scrollable page is displayed) please?

New Feature Request #2: For excluding regions, would it be possible to take a screenshot of the whole app (either full-height, or as it is now), and be able to exclude specified element(s)? For example, in the following image (just a random example), the pink area might be some sort of container element for dynamic content that could change with every login? (obviously you may have covered such a scenario already?)

Thanks for your support :slight_smile:

Hi there,
Love this feature, think it has lots of potential and so far, for my minor use case (verifying a blob pdf view) it works great (Version 7.8.1).

However, 2 questions:

In my test I want to verify twice against the same baseline. So I have in my test check 1:

KeywordUtil.logInfo("1) Verify recap page (via elipsis action button) by visual testing = " + WebUI.takeScreenshotAsCheckpoint('blobVisualRecapPage', FailureHandling.CONTINUE_ON_FAILURE))

and somewhere later on check 2:

KeywordUtil.logInfo("2) Verify recap page (via recap page pdf button) by visual testing = " + WebUI.takeScreenshotAsCheckpoint('blobVisualRecapPage', FailureHandling.CONTINUE_ON_FAILURE))

However, in the executions view of the Visual Testing I only get to see 1 checkpoint!
Question one: I would expect to see 2 checkpoints, no?

Question two: If the screenshot does not match the baseline the test case itself will still pass. I presume because the actual verification happens only AFTER the test is finished and uploaded to the TestOps…
So we will have no failure in the test management executions, but we will have to monitor this in the visual testing executions. Isn’t there a way to make the test case fail in the test management executions, or make it at least visual over there?

Thanks in advance for any reply!

@Kevin_McAndrew,

Thank you for the feature requests, I think they are very helpful. We will add them to our backlog.

Regarding you question, you can pass a list of redacted objects, e.g. Mobile.takeScreenshotAsCheckpoint('checkpoint_name', [findTestObject('Application/android.widget.TextView - App')]). The feature currently does not work as expected on some Android models.

In the future, we will allow specifying redacted regions on Katalon TestOps UI (maybe by draw a rectangular shape on baseline images).

@joost.degeyndt,

Thank you so much for the feedback. Currently, the checkpoint IDs are unique, e.g.

WebUI.takeScreenshotAsCheckpoint(‘blobVisualRecapPage-1’, FailureHandling.CONTINUE_ON_FAILURE))
WebUI.takeScreenshotAsCheckpoint(‘blobVisualRecapPage-2’, FailureHandling.CONTINUE_ON_FAILURE))

Please let us know if you have a reason for using the same checkpoint ID multiple times - we will find a way to solve your use cases.

For the second question, we are working on integrating the visual validation results into the executions, so that you won’t have to monitor them.

@marryko,

Thank you for your feedback. Now you can delete a baseline by navigating to its details page.

Hi @devalex88,
Thanks for your reply!

The reason why I use the same checkpointId twice is because there are 2 pathways of user actions that leads to this screen. I want to make sure that the user sees the pdf view (blob recap page) through path A and the same pdf view should open when going through path B. I could make a blobVisualRecapPage-1 & blobVisualRecapPage-2, but both would be exactly the same. For maintenance purposes I’d prefer to be able to use the same checkpoint twice, otherwise when the screen changes I’ll have to update 2 baselines instead of 1.
If you feel my use case is low on your prio backlog, I’ll go with creating two checkpoint Id’s (luckily I don’t have 47 pathways :wink: )

Great news on the intention to integrate visual validations results into the executions!

1 Like

Thank you for the clarification. I think it’s a valid (and helpful) use case - we will support this soon.

1 Like