Comparing Screenshots of а WebElement

I have made a demo project at

This project proposes a set of Custom Keywords which enable you to test arbitrary HTML elements visually by taking and comparing screenshots in various ways.


Problem to solve

What is built-in

Katalon Studio provides a limited number of built-in keywords for testing web pages visually.

  1. WebUI.verifyImagePresent(TestObject expected)
  2. WebUI.takeScreenshot(File)

Shortages

I would enumerate shortages of the built-in keywords above.

  1. WebUI.takeScreenshot(File) enables you to take screenshot of viewport of the target web page, but it does not allow you to take screenshot of a particular HTML element of your interest.
  2. WebUI.verifyImagePresent(TestObject expected) requires you to prepare a image file as the expected image before running test. But Katalon Studio does not provide any method for you to prepare the expected image out of the target web page. Katalon just assumes you can find the expected image file somewhere.
  3. WebUI.verifyImagePresent(TestObject expected) tries to find the expected image anywhere one or more times in the target web page. You can not specify which HTML element to compare against the expected image.

Solution proposed

aShot, WebDriver Screenshot utility, enables you to take screenshot of particular HTML element. Also it provides images comparison. My Custom Keyword com.kazurayam.ksbackyard.ScreenshotDriver wraps aShot so that you can utilize aShot features in Katalon Studio easily.


Custom Keywords

ScreenshotDriver#saveElementImage

Description

Takes screenshot of a HTML element in the target web page and store into file

Parameters

Param Param Type Mandatory Description
element TestObject Required target HTML element to take screenshot of
file File Required File into which PNG image is saved into

Example

// save screenshot of Google Logo area
TestObject logoArea = findTestObject('Page_Google/div_logoArea')
File file1 = workdir.resolve('logoArea.png').toFile()
CustomKeywords.'com.kazurayam.ksbackyard.ScreenshotDriver.saveElementImage'(logoArea, file1)

ScreenshotDriver#verifyImagesAreSimilar(File, TestObject, …)

Description

Compare actual screenshot of a HTML element in the target web page against the expected PNG file. Difference of 2 images are calculated. If the difference is less than or equal to the given criteria, then verifies images are regarded similar.

Parameters

Param Param Type Mandatory Description
expected File Required PNG File to compare HTML element against
actual TestObject Required target HTML element in the target web page
criteriaPercent Double Required e.g. 5.0 percent
snapshotDir File Optional default to /tmp directory. When FAILED, 3 PNG files will be saved into this directory. These are the snapshots of the expected image, the actual image, and the difference image. The snapshots will NOT be saved when PASSED as default. But you can toggle it ON by calling ScreenshotDriver.setForceSnapshots(true)
flowControl FailureHandling Optional default to CONTINUE_ON_FAILURE

Example

TestObject logoArea = findTestObject('Object Repository/Page_Google/div_logoArea')
File file1 = datadir.resolve('google_logo.png').toFile()
CustomKeywords.'com.kazurayam.ksbackyard.ScreenshotDriver.verifyImagesAreSimilar'(
    file1,
    logoArea,
    3.0,
    workdir.resolve('a').toFile(),
    FailureHandling.CONTINUE_ON_FAILURE)

Or you can simplify the above code as follows:

...
CustomKeywords.'com.kazurayam.ksbackyard.ScreenshotDriver.verifyImagesAreSimilar'(
    file1,
    logoArea,
    3.0)

There are a few more keywords supported. see the GitHub project for detail



How to run the demo

  1. download the zip file of this project from Realses page
  2. unzip it
  3. open the project with your Katalon studio

and then

4. TC1_saveElementImage

Open Test Cases/TC1_saveElementImage and run it. When finished, you will find the screenshot of

element, which contains the Google Logo graphics, is saved in the /tmp/TC1 directory.

logoArea

5. TC2_verifyImagesAreSimilar_or_Different(File,TestObject)

Open Test Cases/TC2_verifyImagesAreSimilar_or_different(File,TestObject) and run it. This test case will fail. The log was as follows:

11-01-2018 04:42:58 PM - [FAILED] - Test Cases/TC2_verifyImagesAreSimilar_or_Different(File,TestObject)    FAILED because (of) (Stack trace: com.kms.katalon.core.exception.StepFailedException:     images are expected to be similar but are different, difference=99.35%,    snapshots were saved in C:\Users\qcq0264\katalon-workspace\ComparingScreenshotsOfWebElement\tmp\TC2\b	at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:15)	at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)	at com.kazurayam.ksbackyard.Assert.stepFailed(Assert.groovy:68)	at com.kazurayam.ksbackyard.Assert.assertTrue(Assert.groovy:27)	at com.kazurayam.ksbackyard.Assert$assertTrue.call(Unknown Source)	at com.kazurayam.ksbackyard.ScreenshotDriver.verifyImagesAreSimilar(ScreenshotDriver.groovy:316)	...

You can find the reason why it failed by looking at the snapshots. See the following snapshots.

The expected image is tmp/TC2/b/verifyImagesAreSimilar(File,TestObject).expected.png expected

The actual image is tmp/TC2/b/verifyImagesAreSimilar(File,TestObject).actual.png actual

The difference image is tmp/TC2/b/verifyImagesAreSimilar(File,TestObject).diff(99.35).png diff

The expected image and actual image is NOT similar, therefore the test case TC2 fails.

7 Likes

+1

How to make those Custom Keywords operational in your Katalon projectDir

  1. import the jar file of aShot into your Katalon Studio Project. You can download the jar at Maven Repository. Or you can reuse the jar bundled in this project. See Katalon Docs External Library for operation.
  2. Using Katalon Studio GUI, create 2 Groovy class files in the Keywords folder, and copy&paste the source code from this project:
  3. Now you can create your test case. See the following sample codes.
4 Likes

Hi kazurayam,

Thank you for the very complete example. I was wondering, is there a method here that allows the images to be attached to the Katalon reports?

Thank you,
Paul

No.

The report is compiled by Katalon Studio, and it provides no extension point.

1 Like

Hi @kazurayam,

Will this approach work for mobile keyword? Also, I tried to use the default ‘Verify Image Present’ mobile keyword but it seems to ‘pass’ even when the reference image does not match the actual one(say 50% match). Do you have any idea on that?

Regards,
Vergil

No. This post of mine used the AShot library to take screenshot of Web HTML elements. It does not support Mobile.

Katalon Studio supports a keywords to take screenshot of a Mobile element:

https://docs.katalon.com/katalon-studio/docs/mobile-take-element-screenshot-as-checkpoint.html

Possibly you want to try it.

Once you have got a pair of PNG files, then you will find the next question:

how to compare these 2 PNG files? how much they are similar or different?

My GitHub repository Comparing Screenshots of Web Element provides a class com.kazurayam.ksbackyard.ScreenshotDriver. This class implements the following method:

    static ImageDifference compareImages(
			BufferedImage expectedImage,
			BufferedImage actualImage,
			Double criteriaPercent)
	{
		ImageDifference imgDifference =
				new ImageDifference(expectedImage, actualImage)
		imgDifference.setCriteria(criteriaPercent)
		return imgDifference
	}

This method would be similar to what you would want. This method wraps AShot to compare 2 BufferedImage instances.

Thank you very much for sharing! I will check.

@kazurayam Thank you so much for your guidance! The approach that you shared works perfectly! The accuracy of image comparison is good too.