[Update with Katalon Studio 7.7] Early release of Katalon TestOps Visual Testing (image comparison)

Hi everyone,

An early release of Katalon TestOps Visual Testing feature (https://analytics.katalon.com) is available now. This is a handy feature to help you compare screenshots between executions when assertion steps do not provide enough coverage or the webpage layout is important.

Katalon Studio version 7.7 added new screenshot-taking keywords for Web UI testing. Similar keywords for Mobile is coming soon.

In this tutorial, I will show you how to leverage this feature to deliver more effective test cases in less time.

You can download the sample code here https://github.com/katalon-studio-samples/visual-testing-sample.

  1. Make use of the above keywords to take screenshots of the whole viewport or parts of it.

Here is a sample test case:

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.checkpoint.CheckpointFactory as CheckpointFactory
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as MobileBuiltInKeywords
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testcase.TestCaseFactory as TestCaseFactory
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testdata.TestDataFactory as TestDataFactory
import com.kms.katalon.core.testobject.ObjectRepository as ObjectRepository
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WSBuiltInKeywords
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUiBuiltInKeywords
import internal.GlobalVariable as GlobalVariable
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS

WebUI.comment('Story: Book an appointment')

WebUI.comment('Given that the user has logged into their account')

WebUI.openBrowser(GlobalVariable.G_SiteURL)

WebUI.takeScreenshotAsCheckpoint('login', null)

WebUI.callTestCase(findTestCase('Common Test Cases/Login'), [('Username') : 'John Doe', ('Password') : 'ThisIsNotAPassword'], FailureHandling.STOP_ON_FAILURE)

WebUI.comment('And Appointment page is displayed')

WebUI.takeScreenshotAsCheckpoint('appointment', null)

WebUI.selectOptionByLabel(findTestObject('Page_CuraAppointment/lst_Facility'), 'Hongkong CURA Healthcare Center', false)

WebUI.check(findTestObject('Page_CuraAppointment/chk_Medicaid'))

WebUI.check(findTestObject('Page_CuraAppointment/chk_Readmission'))

WebUI.setText(findTestObject('Page_CuraAppointment/txt_VisitDate'), '27/12/2016')

WebUI.setText(findTestObject('Page_CuraAppointment/txt_Comment'), 'Please make appointment as soon as possible.')

WebUI.comment('When he fills in valid information in Appointment page')

WebUI.click(findTestObject('Page_CuraAppointment/btn_BookAppointment'))

WebUI.takeScreenshotAsCheckpoint('confirmation', null)

WebUI.verifyTextPresent('Appointment Confirmation', false)

WebUI.comment('Then he should be able to book a new appointment')

WebUI.verifyMatch('Hongkong CURA Healthcare Center', WebUI.getText(findTestObject('Page_AppointmentConfirmation/lbl_Facility')), false)

WebUI.verifyMatch('Yes', WebUI.getText(findTestObject('Page_AppointmentConfirmation/lbl_HospitalReadmission')), false)

WebUI.verifyMatch('Medicaid', WebUI.getText(findTestObject('Page_AppointmentConfirmation/lbl_Program')), false)

WebUI.verifyMatch('27/12/2016', WebUI.getText(findTestObject('Page_AppointmentConfirmation/lbl_VisitDate')), false)

WebUI.verifyMatch('Please make appointment as soon as possible.', WebUI.getText(findTestObject('Page_AppointmentConfirmation/lbl_Comment')), false)

WebUI.takeScreenshot()
WebUI.closeBrowser()
  1. Integrate the current Katalon Studio project with Katalon TestOps.

  1. Execute your test suite. Test results will be submitted automatically to Katalon TestOps. Visit the Katalon TestOps project where the execution has been submitted to, and navigate to Visual Testing.

  1. In KatalonTestOps Vision, the first execution’s screenshots will serve as baseline images. In later steps, I’ll show you how to update baselines to reflect new system-under-test UI if necessary.

  1. Images from subsequent executions (checkpoints) will be compared with the baselines automatically by Katalon TestOps.

This is a match.

This is a mismatch.

  1. Resolve mismatches.

The “Mismatch” label indicates that the checkpoints are different from their baselines. The yellow color indicates that the corresponding checkpoint has not been reverified manually.

You can show/hide the difference layer to closely inspect the mismatch. If the checkpoint is marked as “passed”, it will be used as the baseline for the subsequent executions.

Don’t forget to click “Save as Baseline” after finish the review.

  1. You can also review the current baselines and the corresponding checkpoints.

In the future, you can also exclude region or tuning comparison parameters to better reflect the application UI.

I hope this feature will be helpful for your test projects.

7 Likes

Does Katalon TestOps Vision support taking screenshots and visual testing for Mobile apps?

1 Like

@kazurayam Yes, we are working on the Keywords to capture screenshot for Web, Mobile, and Windows App.

1 Like

Where can the early release be downloaded?

Currently, we are working the Katalon Studio keywords, which will be available for a couple weeks. But you can follow the first topic above to try it.

Hi, do you have any future plans for the following…

  • Some kind of dashboard (or other mechanism) to easily review visual differences from a test-run, and either mark any diffs as a fail, or select an option to ‘re-baseline’ - similar to Applitools and percy.io?

  • Some way of dealing with subtle rendering differences (or will it just be pixel-by-pixel comparison, with no tolerance?)

  • The ability to compare visual differences on a per device / browser level? (e.g. I might want to run the same test script against 10 devices, but I want each comparison to be per-device)?

Thanks,
Kevin

I had a quick look at Percy.

In the FAQ

So, I upload screenshots?

Nope! You might be surprised to learn that under the hood, Percy is not designed to accept screenshots, but instead captures DOM snapshots and page assets (CSS, images, etc.).

They do not take screenshots! Very different from the design of other “Visual Testing” tools like Applitools. This design would enable tests on top of Percy to cover multiple devices / browsers easily. This design would be well fitting as a component of the BrowserStack service who recently merged Percy.

@kazurayam yes I was aware of this from a demo I’ve seen, and it looks a very interesting product.

Bearing in mind that with a free account, you can get 5,000 screenshots per month (maybe enough for my company, at least to start with), if it could integrate with Katalon easily like Applitools does (eyes.open…), then this could be a very cost-effective and easy-to-implement solution.

By the way, I perhaps didn’t make it totally clear, but my third question which was about multiple devices and browsers was more about the upcoming Katalon solution!

I understand that you created a visual testing solution for Katalon that I’ve seen in another post - do you have any thoughts or suggestions for a good visual testing solution, that is preferably codeless (or at least simple enough for a non-coder to implement), primarily for mobile applications?

Have you tried the Katalon solution detailed here yet?

You should ask this question to @devalex88

My VisualTestingInKatalonStudio does not support mobile applications at all. So it is not worth of your attention.

5000 screenshots per month — is it really enough for you? During evaluation stage, may be enough. But once you start using it in a real project, I suppose, you would consume the capacity very soon. In my case I will create (and discard) 5000 screenshots in a single day to carry out a full regression testing of my business app. You need to make up your mind how much you can afford to pay for the tool.

@kazurayam

Thanks for clarifying the situation regarding your solution.

We’re probably working on different size projects - my ininitial usage would be for a mobile app with a limited number of pages. But like you say, it would be good for the evaluation stage at least, before spending the money after gaining confidence that the solution works on a smaller scale.

@devalex88 - please could you have a look at my questions about?

Thanks :slight_smile:

@devalex88 also, please may I ask…

  • Will the visual testing utility become available for mobile applications, and windows desktop?

I have tried the web side of things based on your example above, and I have to say it looks very promising, and the web-based management of visual diffs looks very promising for an early version (thought for me, the diffs weren’t highlighted).

Thanks,
Kevin

Hi @kevin.mcandrew1,
Can you share with us some screenshots? The diffs can be toggled so make sure you have turned it on.

.

We are developing the following keywords which support both web and mobile testing.

WebUI.takeScreenshotAsCheckpoint(String checkpointName)
WebUI.takeAreaScreenshotAsCheckpoint(String checkpointName, Rectangle rect, FailureHandling flowControl)
WebUI.takeElementScreenshotAsCheckpoint(String checkpointName, TestObject to, FailureHandling flowControl)
WebUI.takeFullPageScreenshotAsCheckpoint(String checkpointName, List<TestObject> ignoredElements, FailureHandling flowControl)
Mobile.takeScreenshotAsCheckpoint(String checkpointName)
Mobile.takeElementScreenshotAsCheckpoint(String checkpointName, TestObject to)
Mobile.takeAreaScreenshot(String checkpointName, Rectangle rect)

Hi @anhqle,

Thanks for getting back to me. Here’s a screenshot of what I see…this is exactly what I see whether I have ‘Show Diff’ or ‘Hide Diff’ as the button text…

However, I think I might know what’s going on. From your screenshot, it looks like you’re highlighting diffs in red. In my example, I have several validation messages that are in the baseline, but not the checkpoint. The validation messages are displayed in red (as is often the case). Therefore, I believe that the red validation messages are being highlighted - in red - giving the impression that they’re not being highlighted.

This is an educated guess, but if I’m correct, can I please suggest that you use another colour to highlight diffs, and/or perhaps (if you’re able to), some additional highlighting method, such as highlighting the area around the diff?

Also, I can’t seem to re-baseline executions…in one case, I have only one screenshot taken, but I see two in my execution report), and the Save to Baseline button is unavailable…

Note: this could just be user error on my part! Although I now can’t seem to create / execute new test cases (in suites) without a similar issue.

Are there any plans to implement this for Windows Desktop testing?

Finally…I’d be happy to help beta test this - it’s an area I’m particularly keen to learn more about.

Thanks :slight_smile:

[EDIT] - I removed the error message details from this post as it was just me not including the libraries - apologies!

1 Like

Thank you for your suggestion about diffs highlighting. We will investigate further on this issue.

About re-baseline execution, you can click Save to baseline after making some changes to the checkpoints (mark it as passed or failed). Checkpoints that are New, Mismatch, or Missing will have status Unresolved by default. It’s up to you to approve (mark it as passed) or reject the changes (mark it as failed).


Checkpoint with Passed status will be saved to baseline

@anhqle

Thanks - looks like you’ve improved the highlighting already?

I think I understand the base-lining workflow now - thank you.

Do you have any news on when this solution will work for mobile applications? I tried the ‘Take Screenshot’ mobile keyword, but just got errors (sorry can’t recall the specifics, but I presumed it’s because it’s still under development)?

Let me know if you want me to provide more error details if this isn’t something you’re expecting.

Thanks :slight_smile:

@anhqle, also, please may I ask how you might be able to handle dynamic items, such as dates and times?

I’m guessing that the time in the top left hand corner of this example wouldn’t be captured, because it’s the OS, not the app, but will there be provision to exclude certain items or regions, such as the date in each item (so as to avoid constant false-positives)?

Thanks!

The OS status bar will not be captured when using keywords so don’t worry.

Thank you for your suggestion about region exclusion. We will keep you informed if there is any update.

Thank you @anhqle, I think the region exclusion thing is really important; without such a feature, any page with any kind of dynamic data can’t be visually tested (as I understand it). Please could you keep us updated in this thread?

Many thanks :slight_smile: