[KShare] How to Click on Desktop Elements by using Images

Hi Community members, :wave:

Our team noticed that there had been several threads on the forum regarding Desktop Testing with Katalon as of late. As a result, we will be dedicating June to producing Desktop Testing-related articles for our KShare series!

Let’s get started!


So, you have a list of elements from an object’s locator → you want to click on one of the elements in the list. → Do you wish to use images to determine the desired element?

If you see yourself in this kind of situation, then stick with us till the end of this article for a tutorial on how to do so.

Scenario

image

There are 2 combo boxes , which can be found with the locator below:

//Window/ComboBox

Now, we expect to click on the combo box including the default value named Telex. What should we do? Read more below to find out :point_down:

Instructions

Step 1: Capture an image containing the text Telex

We recommend you to use the key-chord: Win + Shift + S instead of using any tools like Snipping Tool since it supports you capturing precisely the text.

image

Step 2: Add the custom keywords below into your Custom Keyword file

@Keyword
	def clickElementContainsImage(WindowsTestObject testObject, String image, float minScore = 0.5) {
		def element = findElementContainsImage(testObject, image, minScore);
		if (element == null) {
			KeywordUtil.markFailedAndStop("Cannot find any elements that contain images");
			return;
		}
		element.click();
	}
	
	
@Keyword
	def RemoteWebElement findElementContainsImage(WindowsTestObject testObject, String image, float minScore = 0.5) {
		List<RemoteWebElement> elements = Windows.findElements(testObject);

		def driver = Windows.getDriver();
		def window = driver.manage().window();

		def winWidth = window.getSize().getWidth();
		def winHeight = window.getSize().getHeight();
		def winX = window.getPosition().getX();
		def winY = window.getPosition().getY();
		//		println "> Window(${winX}, ${winY}, ${winWidth}, ${winHeight})";

		def desktop = WindowsDriverFactory.getWindowsSession().getDesktopDriver();
		def device = GraphicsEnvironment.getLocalGraphicsEnvironment().getDefaultScreenDevice();
		def screenWidth = device.getDisplayMode().getWidth();
		def screenHeight = device.getDisplayMode().getHeight();
		//		println "> Screen(${screenWidth}, ${screenHeight})";

		def bounds = new DesktopScreenRegion().getBounds();
		def scale = bounds.getWidth() / screenWidth;

		def sikuliImage = resizeImage(image, scale);
		Target target = new ImageTarget(sikuliImage);
		target.setMinScore(minScore);

		for (def elementI : elements) {
			def info = elementI.getText();
			def x = winX + elementI.getLocation().getX();
			def y = winY + elementI.getLocation().getY();
			def width = elementI.getSize().getWidth();
			def height = elementI.getSize().getHeight();

			def region = new DesktopScreenRegion((int)(x * scale), (int)(y * scale), (int)(width * scale), (int)(height * scale));
			def result = region.find(target);

			if (result) {
				//				println ">>> (${x}, ${y}, ${width}, ${height}) ${info}";
				return elementI;
			} else {
				//				println "> Region(${x}, ${y}, ${width}, ${height}) ${info}";
			}
		}

		return null;
	}

Usage

In the custom keyword named findElementContainsImage and clickElementContainsImageyou will notice a parameter named float minScore = 0.5

0.5 is the default value of the exact index when you use those custom keywords. However, you can update that number to make sure the keyword can recognize your image exactly.

For example, in this scenario, we will need to update the minScore from 0.5 to 0.9

CustomKeywords. 'Windows ImprovedKeywords.clickElementContainsImage' (DropownList, Path, 0.9)

Result


If you find this article helpful, then don’t forget to show your support by leaving a like :+1: or a heart :heart: and share it with your colleagues or teammates!

2 Likes

Thank you the Product Support team (@support.squad) and Linh Nguyen (@linh.nguyen) for yet another helpful article!

Linh Nguyen Thong Tran
Linh Nguyen (@linh.nguyen) - Product Support Manager at Katalon Thong Tran (@thong.tran) - Senior Software Engineer at Katalon
Linh is the Product Support team Manager at Katalon. She spent many years working as an Automation Testing QA before joining Katalon Product Support as a technical support expert. Based on her experiences, she usually provides consumers with highly applicable solutions. She now manages her team with a user-centric approach to guarantee customers greater success with Katalon Products. A passionate Katalon developer with a wealth of programming and testing expertise. Thong has been dedicated to providing exceptional enterprise support for the past five years, helping Katalon’s customers achieve their testing goals with ease.
2 Likes