Visual Testing (image comparison) with Katalon Studio 7.8.0 and Katalon TestOps

[Tarek]

Hello Katalon Devs,

I have started trying out the visual testing and Im impressed so far, thank you for your efforts. I still need more clarification please. When I run a different TestSuite with takeFullPageScreenshotAsCheckpoint(), this TestSuite is being compared to previous Baselines which don’t exist here.

Is there a way to separate each TestSuite’s Baselines so Katalon knows which screenshots to expect in each TestSuite?

Thank you

[devalex88]

Hi @Tarek,

I’ve updated the tutorial for Katalon Studio 7.7. I hope it would be helpful for you. If not, could you please give us more details (a piece of sample code would help a lot)?

[Tarek]

Hi @devalex88,

Thank you for your reply. The link for Katalon TestOps Visual is down in : https://docs.katalon.com/katalon-studio/new/version-70.html#official-release---version-770---771

I have multiple test cases for multiple screens which requires different visual testing of course. I called:
WebUI.takeFullPageScreenshot(RunConfiguration.getReportFolder() + ‘/keyes-search.png’)
in the Search screen and saved the initial screenshot as my baseline.

However, if I execute the other test case independently and take a screenshot at the Search Result screen:
WebUI.takeFullPageScreenshot(RunConfiguration.getReportFolder() + ‘/keyes-searchResult.png’)
It shows that a screenshot is missing “keyes-search.png”, but I am of no need of the first screenshot.

Is there a way to separate the screenshots’. For example, a folder for the set of screenshots of Search screen and separate screenshots for the Search Result screen instead of having them in the same baseline?

[Thong]

Hi @Tarek,
So you need a way to organizing your screenshots to a set of baseline collections and define which collection is used when run execution. Instead of comparing all baseline screenshots.

[Tarek]

Hi @Thong,

Yes, that is exactly what I’m trying to explain, sorry if my wording wasn’t correct. Is it available right now with the current release? Or would it be a future feature perhaps?

Thank you

[Thong]

Hi @Tarek,

My team will work on it, it’s not available now.

[Tarek]

Hi @Thong,

Thank you for the consideration! I’m looking forward for this project. Can I please add one idea too: allow the user to delete/rename Baseline screenshots.

Thank you

[rtetrault]

Received a report today via email of past week executions(assume from testops). Just an fyi… love this.

[michaelmakar1]

How do you delete baseline images? I see no option to delete. Also Katalon seems to add duplicate images to the baseline when there is a mismatch rather than “know” that some elements on the same page have failed. It other words, it creates a new baseline when it should not.

[devalex88]

Hi. We will add a button for deleting baselines next week.

@michaelmakar1

Also Katalon seems to add duplicate images to the baseline when there is a mismatch rather than “know” that some elements on the same page have failed. It other words, it creates a new baseline when it should not.

I’m sorry but could you please clarify the situation so that I can troubleshoot. Thank you!

[michaelmakar1]

Katalon seems to think that mismatches on a page are new page. So rather than compare against the existing baseline, it creates a new baseline as if it were a new page added.

[marryko]

Hi @devalex88 , do you know if the delete option was already impelemented?

[kazurayam]

@kazurayam

They do not take screenshots! Very different from the design of other “Visual Testing” tools

See my report at

Saving Web Page as MHTML in Katalon Studio

I have made a GitHub project to share my findings. I made it publicly accessible. Problem to solve I want to automate the following tasks by scripts in Katalon Studio: to take snapshot of current DOM of Web Page in browser, save it into a local file in the MHTML format. to take full screenshots of a web page in multiple viewport sizes (with*height) just as I can do in Chrome DevTools using Device Mode like Galaxy S5, Pixel 2, iPhone 6/7/8, iPad. I want screenshots in the size of PC display…

[marryko]

@kevin.mcandrew1

Also, I can’t seem to re-baseline executions…in one case, I have only one screenshot taken, but I see two in my execution report), and the Save to Baseline button is unavailable…

You can “re-baseline” image if you tick the green checkmark as correct, refresh the page and button save to baseline will be enabled, this will “re-baseline” your image It’s a workaround for the delete of baseline images.

If you see one that says “missing” and you tick the green checkmark and click save to baseline it will “delete” it

[devalex88]

Hi @marryko Implementation has been done but it will take about two weeks to deploy.

[devalex88]

@kevin.mcandrew1 The Keywords for Mobile are available now with Katalon Studio 7.8.0, and they can ignore regions when capturing screenshots. I added them to the first post.

[Kevin_McAndrew]

@devalex88 That’s great news - thank you - I’ll check this out very soon

[Kevin_McAndrew]

@devalex88

So far looking good. The takeScreenshotAsCheckpoint command works well - no false positives so far (not sure if you have built-in tolerances or whether our app is pixel perfect each time?).

I didn’t try takeAreaScreenshotAsCheckpoint (not sure if I understand the rectangle aspect), and takeElementScreenshotAsCheckpoint didn’t work for me - the image it captured did not resemble the element I’d identified (I tried three separate objects).

I also couldn’t see how to exclude regions? Unless you mean by being able to capture an element or area that they are in effect excluding other areas?

Please can I make a couple of feature requests and report a possible small bug…

Possible bug: in the TestOps Visual Testing dashboard, within an execution, the ‘Save to Baseline’ button is not immediately enabled after accepting a new or changed image…as a workaround, refreshing the page enables this button.

New Feature Request #1: Would it be possible to take ‘full page’ screenshots (i.e. the full height, where a vertically-scrollable page is displayed) please?

New Feature Request #2: For excluding regions, would it be possible to take a screenshot of the whole app (either full-height, or as it is now), and be able to exclude specified element(s)? For example, in the following image (just a random example), the pink area might be some sort of container element for dynamic content that could change with every login? (obviously you may have covered such a scenario already?)

Thanks for your support

[joost.degeyndt]

Hi there,
Love this feature, think it has lots of potential and so far, for my minor use case (verifying a blob pdf view) it works great (Version 7.8.1).

However, 2 questions:

In my test I want to verify twice against the same baseline. So I have in my test check 1:

KeywordUtil.logInfo("1) Verify recap page (via elipsis action button) by visual testing = " + WebUI.takeScreenshotAsCheckpoint('blobVisualRecapPage', FailureHandling.CONTINUE_ON_FAILURE))

and somewhere later on check 2:

KeywordUtil.logInfo("2) Verify recap page (via recap page pdf button) by visual testing = " + WebUI.takeScreenshotAsCheckpoint('blobVisualRecapPage', FailureHandling.CONTINUE_ON_FAILURE))

However, in the executions view of the Visual Testing I only get to see 1 checkpoint!
Question one: I would expect to see 2 checkpoints, no?

Question two: If the screenshot does not match the baseline the test case itself will still pass. I presume because the actual verification happens only AFTER the test is finished and uploaded to the TestOps…
So we will have no failure in the test management executions, but we will have to monitor this in the visual testing executions. Isn’t there a way to make the test case fail in the test management executions, or make it at least visual over there?

Thanks in advance for any reply!