Visual Testing (image comparison) with Katalon Studio 7.8.0 and Katalon TestOps

Does Katalon TestOps Vision support taking screenshots and visual testing for Mobile apps?

1 Like

@kazurayam Yes, we are working on the Keywords to capture screenshot for Web, Mobile, and Windows App.

1 Like

Where can the early release be downloaded?

Currently, we are working the Katalon Studio keywords, which will be available for a couple weeks. But you can follow the first topic above to try it.

Hi, do you have any future plans for the following…

  • Some kind of dashboard (or other mechanism) to easily review visual differences from a test-run, and either mark any diffs as a fail, or select an option to ‘re-baseline’ - similar to Applitools and percy.io?

  • Some way of dealing with subtle rendering differences (or will it just be pixel-by-pixel comparison, with no tolerance?)

  • The ability to compare visual differences on a per device / browser level? (e.g. I might want to run the same test script against 10 devices, but I want each comparison to be per-device)?

Thanks,
Kevin

I had a quick look at Percy.

In the FAQ

So, I upload screenshots?

Nope! You might be surprised to learn that under the hood, Percy is not designed to accept screenshots, but instead captures DOM snapshots and page assets (CSS, images, etc.).

They do not take screenshots! Very different from the design of other “Visual Testing” tools like Applitools. This design would enable tests on top of Percy to cover multiple devices / browsers easily. This design would be well fitting as a component of the BrowserStack service who recently merged Percy.

@kazurayam yes I was aware of this from a demo I’ve seen, and it looks a very interesting product.

Bearing in mind that with a free account, you can get 5,000 screenshots per month (maybe enough for my company, at least to start with), if it could integrate with Katalon easily like Applitools does (eyes.open…), then this could be a very cost-effective and easy-to-implement solution.

By the way, I perhaps didn’t make it totally clear, but my third question which was about multiple devices and browsers was more about the upcoming Katalon solution!

I understand that you created a visual testing solution for Katalon that I’ve seen in another post - do you have any thoughts or suggestions for a good visual testing solution, that is preferably codeless (or at least simple enough for a non-coder to implement), primarily for mobile applications?

Have you tried the Katalon solution detailed here yet?

You should ask this question to @devalex88

My VisualTestingInKatalonStudio does not support mobile applications at all. So it is not worth of your attention.

5000 screenshots per month — is it really enough for you? During evaluation stage, may be enough. But once you start using it in a real project, I suppose, you would consume the capacity very soon. In my case I will create (and discard) 5000 screenshots in a single day to carry out a full regression testing of my business app. You need to make up your mind how much you can afford to pay for the tool.

@kazurayam

Thanks for clarifying the situation regarding your solution.

We’re probably working on different size projects - my ininitial usage would be for a mobile app with a limited number of pages. But like you say, it would be good for the evaluation stage at least, before spending the money after gaining confidence that the solution works on a smaller scale.

@devalex88 - please could you have a look at my questions about?

Thanks :slight_smile:

@devalex88 also, please may I ask…

  • Will the visual testing utility become available for mobile applications, and windows desktop?

I have tried the web side of things based on your example above, and I have to say it looks very promising, and the web-based management of visual diffs looks very promising for an early version (thought for me, the diffs weren’t highlighted).

Thanks,
Kevin

Hi @kevin.mcandrew1,
Can you share with us some screenshots? The diffs can be toggled so make sure you have turned it on.

.

We are developing the following keywords which support both web and mobile testing.

WebUI.takeScreenshotAsCheckpoint(String checkpointName)
WebUI.takeAreaScreenshotAsCheckpoint(String checkpointName, Rectangle rect, FailureHandling flowControl)
WebUI.takeElementScreenshotAsCheckpoint(String checkpointName, TestObject to, FailureHandling flowControl)
WebUI.takeFullPageScreenshotAsCheckpoint(String checkpointName, List<TestObject> ignoredElements, FailureHandling flowControl)
Mobile.takeScreenshotAsCheckpoint(String checkpointName)
Mobile.takeElementScreenshotAsCheckpoint(String checkpointName, TestObject to)
Mobile.takeAreaScreenshot(String checkpointName, Rectangle rect)
1 Like

Hi @anhqle,

Thanks for getting back to me. Here’s a screenshot of what I see…this is exactly what I see whether I have ‘Show Diff’ or ‘Hide Diff’ as the button text…

However, I think I might know what’s going on. From your screenshot, it looks like you’re highlighting diffs in red. In my example, I have several validation messages that are in the baseline, but not the checkpoint. The validation messages are displayed in red (as is often the case). Therefore, I believe that the red validation messages are being highlighted - in red - giving the impression that they’re not being highlighted.

This is an educated guess, but if I’m correct, can I please suggest that you use another colour to highlight diffs, and/or perhaps (if you’re able to), some additional highlighting method, such as highlighting the area around the diff?

Also, I can’t seem to re-baseline executions…in one case, I have only one screenshot taken, but I see two in my execution report), and the Save to Baseline button is unavailable…

Note: this could just be user error on my part! Although I now can’t seem to create / execute new test cases (in suites) without a similar issue.

Are there any plans to implement this for Windows Desktop testing?

Finally…I’d be happy to help beta test this - it’s an area I’m particularly keen to learn more about.

Thanks :slight_smile:

[EDIT] - I removed the error message details from this post as it was just me not including the libraries - apologies!

1 Like

Thank you for your suggestion about diffs highlighting. We will investigate further on this issue.

About re-baseline execution, you can click Save to baseline after making some changes to the checkpoints (mark it as passed or failed). Checkpoints that are New, Mismatch, or Missing will have status Unresolved by default. It’s up to you to approve (mark it as passed) or reject the changes (mark it as failed).


Checkpoint with Passed status will be saved to baseline

@anhqle

Thanks - looks like you’ve improved the highlighting already?

I think I understand the base-lining workflow now - thank you.

Do you have any news on when this solution will work for mobile applications? I tried the ‘Take Screenshot’ mobile keyword, but just got errors (sorry can’t recall the specifics, but I presumed it’s because it’s still under development)?

Let me know if you want me to provide more error details if this isn’t something you’re expecting.

Thanks :slight_smile:

1 Like

@anhqle, also, please may I ask how you might be able to handle dynamic items, such as dates and times?

I’m guessing that the time in the top left hand corner of this example wouldn’t be captured, because it’s the OS, not the app, but will there be provision to exclude certain items or regions, such as the date in each item (so as to avoid constant false-positives)?

Thanks!

The OS status bar will not be captured when using keywords so don’t worry.

Thank you for your suggestion about region exclusion. We will keep you informed if there is any update.

Thank you @anhqle, I think the region exclusion thing is really important; without such a feature, any page with any kind of dynamic data can’t be visually tested (as I understand it). Please could you keep us updated in this thread?

Many thanks :slight_smile:

Hello Katalon Devs,

I have started trying out the visual testing and Im impressed so far, thank you for your efforts. I still need more clarification please. When I run a different TestSuite with takeFullPageScreenshotAsCheckpoint(), this TestSuite is being compared to previous Baselines which don’t exist here.

Is there a way to separate each TestSuite’s Baselines so Katalon knows which screenshots to expect in each TestSuite?

Thank you

1 Like