# \[KShare\] How to Click on Desktop Elements by using Images

**URL:** <https://forum.katalon.com/t/kshare-how-to-click-on-desktop-elements-by-using-images/132574>\
**Category:** Kshare\
**Tags:** linh-nguyen, thong-tran, desktop, gatedknowledge, support, katalon-studio\
**Created:** [June 4, 2024, 8:00am UTC](https://forum.katalon.com/t/kshare-how-to-click-on-desktop-elements-by-using-images/132574 "2024-06-04T08:00:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![support.squad](https://dub1.discourse-cdn.com/katalon/user_avatar/forum.katalon.com/support.squad/32/23139_2.png) [@support.squad](https://forum.katalon.com/u/support.squad)\
**Post date:** [June 4, 2024, 8:00am UTC](https://forum.katalon.com/t/kshare-how-to-click-on-desktop-elements-by-using-images/132574/1 "2024-06-04T08:00:17Z")

</div>

![Forum thumbnail](https://europe1.discourse-cdn.com/katalon/original/3X/f/f/ffdbbc76e3e07eabe8084d6fe302d1092c0cf3ed.jpeg)

Hi Community members, 👋

Our team noticed that there had been several threads on the forum regarding Desktop Testing with Katalon as of late. As a result, we will be dedicating June to producing Desktop Testing-related articles for our [KShare](http://forum.katalon.com/c/product-insights/73) series!

Let’s get started!

* * *

So, you have a **list of elements** from an object’s locator → you want to **click on one** of the elements in the list. → **Do you wish to use images to determine the desired element?**

If you see yourself in this kind of situation, then stick with us till the end of this article for a tutorial on how to do so.

### Scenario

 ![image](https://europe1.discourse-cdn.com/katalon/original/3X/c/a/ca8e547ac8fc2c95f65ae814f250a244d8bbf488.png)

There are **2 combo boxes** , which can be found with the locator below:

```auto
//Window/ComboBox

```

Now, we expect to click on the combo box including the default value named `Telex`. What should we do? Read more below to find out 👇

### Instructions

#### Step 1: Capture an image containing the text `Telex`

We recommend you to use the key-chord: **Win + Shift + S** instead of using any tools like Snipping Tool since it supports you capturing precisely the text.

![image](https://europe1.discourse-cdn.com/katalon/original/3X/6/0/60e548268027c7a0fd8df197395b8c8f91d50546.png)

#### Step 2: Add the custom keywords below into your Custom Keyword file

```auto
@Keyword
	def clickElementContainsImage(WindowsTestObject testObject, String image, float minScore = 0.5) {
		def element = findElementContainsImage(testObject, image, minScore);
		if (element == null) {
			KeywordUtil.markFailedAndStop("Cannot find any elements that contain images");
			return;
		}
		element.click();
	}
	
	
@Keyword
	def RemoteWebElement findElementContainsImage(WindowsTestObject testObject, String image, float minScore = 0.5) {
		List<RemoteWebElement> elements = Windows.findElements(testObject);

		def driver = Windows.getDriver();
		def window = driver.manage().window();

		def winWidth = window.getSize().getWidth();
		def winHeight = window.getSize().getHeight();
		def winX = window.getPosition().getX();
		def winY = window.getPosition().getY();
		// println "> Window(${winX}, ${winY}, ${winWidth}, ${winHeight})";

		def desktop = WindowsDriverFactory.getWindowsSession().getDesktopDriver();
		def device = GraphicsEnvironment.getLocalGraphicsEnvironment().getDefaultScreenDevice();
		def screenWidth = device.getDisplayMode().getWidth();
		def screenHeight = device.getDisplayMode().getHeight();
		// println "> Screen(${screenWidth}, ${screenHeight})";

		def bounds = new DesktopScreenRegion().getBounds();
		def scale = bounds.getWidth() / screenWidth;

		def sikuliImage = resizeImage(image, scale);
		Target target = new ImageTarget(sikuliImage);
		target.setMinScore(minScore);

		for (def elementI : elements) {
			def info = elementI.getText();
			def x = winX + elementI.getLocation().getX();
			def y = winY + elementI.getLocation().getY();
			def width = elementI.getSize().getWidth();
			def height = elementI.getSize().getHeight();

			def region = new DesktopScreenRegion((int)(x * scale), (int)(y * scale), (int)(width * scale), (int)(height * scale));
			def result = region.find(target);

			if (result) {
				// println ">>> (${x}, ${y}, ${width}, ${height}) ${info}";
				return elementI;
			} else {
				// println "> Region(${x}, ${y}, ${width}, ${height}) ${info}";
			}
		}

		return null;
	}

```

### Usage

In the custom keyword named `findElementContainsImage` and `clickElementContainsImage`you will notice a parameter named `float minScore = 0.5`

0.5 is the default value of the exact index when you use those custom keywords. However, you can update that number to make sure the keyword can recognize your image exactly.

For example, in this scenario, we will need to update the minScore from **0.5 to 0.9**

```auto
CustomKeywords. 'Windows ImprovedKeywords.clickElementContainsImage' (DropownList, Path, 0.9)

```

### Result

* * *

If you find this article helpful, then don’t forget to show your support by leaving a like 👍 or a heart ❤ and share it with your colleagues or teammates!

---

<div class="post-metadata">

**Author:** ![albert.vu](https://dub1.discourse-cdn.com/katalon/user_avatar/forum.katalon.com/albert.vu/32/20595_2.png) [@albert.vu](https://forum.katalon.com/u/albert.vu)\
**Post date:** [June 4, 2024, 8:00am UTC](https://forum.katalon.com/t/kshare-how-to-click-on-desktop-elements-by-using-images/132574/2 "2024-06-04T08:00:17Z")

</div>

Thank you the Product Support team (@support.squad) and Linh Nguyen (@linh.nguyen) for yet another helpful article!

| ![Linh Nguyen](https://europe1.discourse-cdn.com/katalon/original/3X/8/4/84e17374fb62b33f2d3ea89a16c9bf09f01aaf0b.png) | ![Thong Tran](https://europe1.discourse-cdn.com/katalon/original/3X/2/4/2473c611894c7134b42923bcd8045dfc1afcef73.png) |
| --- | --- |
| **Linh Nguyen** (@linh.nguyen) - Product Support Manager at Katalon | **Thong Tran** (@thong.tran) - Senior Software Engineer at Katalon |
| Linh is the Product Support team Manager at Katalon. She spent many years working as an Automation Testing QA before joining Katalon Product Support as a technical support expert. Based on her experiences, she usually provides consumers with highly applicable solutions. She now manages her team with a user-centric approach to guarantee customers greater success with Katalon Products. | A passionate Katalon developer with a wealth of programming and testing expertise. Thong has been dedicated to providing exceptional enterprise support for the past five years, helping Katalon’s customers achieve their testing goals with ease. |

---

<div class="post-metadata">

**Author:** ![system](https://europe1.discourse-cdn.com/katalon/original/3X/d/5/d55596ab02785a255798e87ede769734747b30c4.svg) [@system](https://forum.katalon.com/u/system)\
**Post date:** [June 4, 2025, 8:00am UTC](https://forum.katalon.com/t/kshare-how-to-click-on-desktop-elements-by-using-images/132574/3 "2025-06-04T08:00:19Z")

</div>

Closed on June 4, 2025, 8:00am UTC

This topic was automatically closed after 365 days. New replies are no longer allowed.