verifyOptionPresentByLabel is very slow for select elements with many options

sbrod · October 11, 2022, 8:36pm

Operating System

Windows 10

Katalon Studio version

8.4.1

Log Folder:

.log (65.5 KB)

Environment (for Web Testing)

Chrome 105.0.5195.102

Steps to reproduce

Run WebUI.verifyOptionPresentByLabel() on a select element with several thousand or more options. See that it takes several minutes to run.

Expected Behavior
Method adheres to specified maximum timeout. If method is unable to perform within the allotted time, it should throw an appropriate error.

Actual Behavior
Method runs in excess of timeout, including and up to excessive runtimes for select elements with many options.

More Information
The select element is being selected via a TestObject with a CSS selector (‘select#${id}’ with the appropriate ID passed to it). The select element has about 8500 options, so I’m not surprised it takes some time to verify a label is present, but taking several minutes to check the label against those options is absurd. Regex enabled/disabled makes no noticeable difference.

One of our test cases has multiple consecutive calls to this method, and some interesting behavior ensues. The test case appears to freeze, and it does so for long enough for our website to log out due to inactivity. As soon as it does so, the test suddenly jumps forward several steps, and each of the calls to verifyOptionPresentByLabel is printed to the log viewer with various runtimes between 50 seconds and 5 minutes. If it was hanging on the first call, I’d expect only the first one to have been called and then see the test hit the next calls after the logout (where they would immediately fail because the element doesn’t exist on the login page), but that’s not what’s happening.

It’s also weird how varied the runtimes are. If the method was succeeding, I wouldn’t be surprised, since it could be explained by the method finding the label and terminating early. But these all failed to find the option in the dropdown.

Screenshots / Videos (please attach screenshots or videos if necessary to reproduce the issue)

IsAttributeVisible has a couple WaitFors (5 seconds max) and then a call to verifyOptionPresentByLabel. The WaitFors aren’t contributing anything substantial, as I was able to reproduce this issue with an isolated call to verifyOptionPresentByLabel on the same select element.

kazurayam · October 11, 2022, 10:00pm

Please share the source code of your AdditionalAttributeAssociationPageModel.IsAttributeVisible() class/method.

kazurayam · October 11, 2022, 10:10pm

Does the method repeats calling WebUI.verifyOptionPresentByLabel() for 8500 times?

Then it would innevitably slow. Each single call of WebUI.verifyOptionPresentByLable() triggers a conversation between your code and browser over TCP. It would take lo…oong to execute 8500 times of conversations over TCP.

I would suggest you an alternative design. You can fetch the 8500 <option> elements from the page in browser into the runtime memory space of your test case script. Use WebUI.findWebElements keyword. It will return a List<WebElement>. Your test case script will be able to iterate over it locally. This processing will be far faster as it makes only single TCP conversation. ---- maybe in 10 seconds?

List<WebElement> options = WebUI.findWebElements(TestObject that select <option> elements)

options.eachWithIndex { opt, index ->
    if (opt.getText() == "foo") {
        println "foo is found at ${index}!"
    }
}

sbrod · October 11, 2022, 10:27pm

As I said, IsAttributeVisible has a couple WaitFors and a call to verifyOptionPresentByLabel. Here’s the code

public static boolean IsAttributeVisible(String attributeName, AssociationUsage usage=AssociationUsage.USER_TYPE, FailureHandling failureMode=FailureHandling.OPTIONAL) {
		TestObject attributeSelect = GetAttributeSelectByUsage(usage)
		WebUI.waitForElementPresent(attributeSelect, GlobalVariable.TIME_OUT)
		WebUI.waitForElementVisible(attributeSelect, GlobalVariable.TIME_OUT)
		return WebUI.verifyOptionPresentByLabel(attributeSelect, attributeName, false, GlobalVariable.TIME_OUT, failureMode)
	}

private static TestObject GetAttributeSelectByUsage(AssociationUsage usage) {
		return (usage == AssociationUsage.USER_TYPE) ? userTypeAttributeSelect : clientTypeAttributeSelect
	}

public static TestObject userTypeAttributeSelect = findTestObject('Common/selectById', ['id' : USER_TYPE_ATTRIBUTE_SELECT_ID])
public static TestObject clientTypeAttributeSelect = findTestObject('Common/selectById', ['id' : CLIENT_TYPE_ATTRIBUTE_SELECT_ID])

(Yeah I know the waits are redundant, they’ve already been fixed on another branch)

As you can see, there are no loops; I’m calling the verify method only once on the object, and there are no other sources that could reasonably contribute to the observed runtimes. Both TestObjects are simple CSS selectors of the format previously specified (select#${id}) and the value of GlobalVariable.TIME_OUT is 5.

I appreciate the suggestion, however I’m not looking for a solution at the moment. I’m only posting because I believe there are optimization problems with this method that the Katalon team should resolve. Our solution to our specific problem was to periodically and automatically clear our testing environment’s database so that there will not be anything approaching 8500 options.

kazurayam · October 11, 2022, 10:53pm

I looked at the source code of verifyOptionPresentByLable keyword at

github.com

katalon-studio/katalon-studio-testing-framework/blob/master/Include/scripts/groovy/com/kms/katalon/core/webui/keyword/builtin/VerifyOptionPresentByLabelKeyword.groovy

package com.kms.katalon.core.webui.keyword.builtin

import groovy.transform.CompileStatic

import java.text.MessageFormat
import java.util.concurrent.TimeUnit

import org.apache.commons.io.FileUtils
import org.openqa.selenium.Alert
import org.openqa.selenium.By
import org.openqa.selenium.Dimension
import org.openqa.selenium.JavascriptExecutor
import org.openqa.selenium.NoSuchElementException
import org.openqa.selenium.NoSuchWindowException
import org.openqa.selenium.TimeoutException
import org.openqa.selenium.WebDriver
import org.openqa.selenium.WebDriverException
import org.openqa.selenium.WebElement
import org.openqa.selenium.interactions.Actions
import org.openqa.selenium.support.ui.ExpectedCondition

This file has been truncated. show original

I found that it triggers a single call to

WebElement webElement = WebUIAbstractKeyword.findWebElement(to, timeOut)

It seems that this took over 3minutes to iterate over 8500 elements.

I think that this speed is genuine to Selenium; not to Katalon Studio.

If you are not satisfied with this speed, then I think you should try other WebUI testing tools which is NOT runnging on top of Selenium. Possiblly Playwright over ChromeDevToolsProtocol is worth trying.

sbrod · October 11, 2022, 11:02pm

findWebElement() is still a Katalon method, right? I’m not sure we can conclude that it’s Selenium’s fault yet.

Also, this still leaves exposed the timeout issue. Timeout is passed to verifyOptionPresentByLabel and to findWebElement and yet runtime still exceeds the timeout provided.

kazurayam · October 12, 2022, 12:07am

Yes.

I can read the source codes of Katalon Studio. I could trace the chain of method calls from WebUI.findElements() to the WebUICommonHelper. I found at the Line#1014 of

github.com

katalon-studio/katalon-studio-testing-framework/blob/master/Include/scripts/groovy/com/kms/katalon/core/webui/common/WebUiCommonHelper.java

package com.kms.katalon.core.webui.common;

import java.awt.Graphics2D;
import java.awt.Image;
import java.awt.Rectangle;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.text.MessageFormat;
import java.util.AbstractMap;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import java.util.function.Function;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

This file has been truncated. show original

      foundElements = webDriver.findElements(defaultLocator);

This line calls the Selenium’s API.

I am sure we can conclude that it’s Selenium’s performance issue.

kazurayam · October 12, 2022, 12:08am

I have no idea about this. There might be some bug in Katalon Studio.

kazurayam · October 12, 2022, 11:12am

I tried to examine how WebUI.verifyOptionPresentByLabel() keyword performs. I created a Katalon project on my mac.

I created a html as test target. It’s content is as simple as this:

<html>
<head>
<title>ks_performance_VerifyOptionPresentByLabelKeyword</title>
</head>
<body>
<div id="main">
<select id="combo_facility" name="facility">
    <option value="1">Label1</option>
    <option value="2">Label2</option>
    <option value="3">Label3</option>
    <option value="4">Label4</option>
    <option value="5">Label5</option>
    <option value="6">Label6</option>
    <option value="7">Label7</option>
    <option value="8">Label8</option>
....
    <option value="9">Label8500</option>
</select>
</div>
</option>
</html>

I wrote a Test Case script TC1:

import java.nio.file.Path
import java.nio.file.Paths

import com.kms.katalon.core.configuration.RunConfiguration
import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

import internal.GlobalVariable

Path html = Paths.get(RunConfiguration.getProjectDir()).resolve("fixture.html")
WebUI.openBrowser('')
WebUI.setViewPortSize(800,600)
WebUI.navigateToUrl(html.toFile().toURI().toURL().toExternalForm())
TestObject tObj = makeTestObject("//*[@id='combo_facility']")
WebUI.verifyElementPresent(tObj, 10)

WebUI.comment("start verifyOptionPresentByLabel")
WebUI.verifyOptionPresentByLabel(tObj, "Label" + GlobalVariable.RANGE_END, false, 5000)
WebUI.comment("end verifyOptionPresentByLable")

WebUI.closeBrowser()

TestObject makeTestObject(String xpath) {
	TestObject tObj = new TestObject(xpath)
	tObj.addProperty("xpath", ConditionType.EQUALS, xpath)
	return tObj
}

I set GlobalVariable.RANGE_END to be 8500.

When I executed TC1 I got the following log output:

2022-10-12 20:05:36.822 INFO  c.k.k.c.keyword.builtin.CommentKeyword   - start verifyOptionPresentByLabel
2022-10-12 20:08:10.322 INFO  c.k.k.c.keyword.builtin.CommentKeyword   - end verifyOptionPresentByLable

TC1 took 2min 24 seconds to verify if <option>Label8500</option> is found.

While TC1 is running, I looked at system performance monitor to see CPU usage.

I found that the process of Chrome browser renderer was running busy over 2min.

It would fair to say that Katalon Studio is not busy while peforming “WebUI.verifyOpitonPresentByLabel” keyword.

“WebUI.verifyOptionPresentByLabel” against a <select> element which has 8500 child <option> elements are slow because Chrome browser is too busy.

If I could do similar test with Playwright with Chrome DevTools Protocol, the time taken would not be the same as this experiment driven by Selenium. Slower or faster, I don’t know.

kazurayam · October 12, 2022, 11:35am

I would be grateful if somebody who is experienced with Playwright kindly reexamine this and report how quick it is.

@peter.hevesi

Will you?

peter.hevesi · October 12, 2022, 12:50pm

Hi @kazurayam,
No sorry, I don’t have time for it, I am very busy, I am developing 2 games in Unity beside my main job But Playwright is always much faster then Selenium and Katalon Studio…
I would always just recommend Playwright for every tester out there… It works more reliably and definitely much faster, but to this question, yeah, it might be, that the slowness comes from Chrome itself… But I would bet it will be faster in Playwright

sbrod · October 12, 2022, 9:11pm

This line calls the Selenium’s API.
I am sure we can conclude that it’s Selenium’s performance issue.

~~Dang, that’s unfortunate. I’m surprised (and disappointed) that Selenium has such limitations.~~

Nvm just read the comment about how chrome is the limitation. Could you run the same test on other browsers and see if there’s a substantial difference?

kazurayam · October 12, 2022, 9:21pm

Here is my project repository on GitHub. You can download a zip of the project and try whatever you want.

Once unzipped, execute the test suite TS1.

kazurayam · October 12, 2022, 10:00pm

An experiment recorded the following result.

Browser	Duration
Chrome	208
Firefox	30
Safari	225
Edge Chromium	196

Duration is in seconds.

I used the version 0.1.1 of the test project.

Different types of browser performed differently. Firefox was best performant, others were equally slow. The performance of selecting an <option> by label seems to be quite dependent on how the internal of browsers are implemented.

I suppose that the internal implementation of Chrome browser for WebDriver API and for Chrome Devtools Protocol would be quite different. The CDP is newer than WebDriver. Therefore I guess Playwright on CDP would run far faster than KS because of the difference in Chrome’s internal implementation. Just a guess.

At least, I think that Katalon Studio is not guilty for the slowness reported in the original post. As for this issue, KS has very little room for improvement.

sbrod · October 12, 2022, 10:42pm

At least, I think that Katalon Studio is not guilty for the slowness reported in the original post. As for this issue, KS has very little room for improvement.

Agreed, that much is clear. Thank you for all the time and effort to investigate this.

Topic		Replies	Views
Unable to verify options present Web Testing katalon-studio	11	1116	April 11, 2019
Unable to Select an option from the dropdown : Error : Element should have been "select" but was "div" Web Testing katalon-studio	3	4867	April 6, 2021
[WebUI] Verify Option Selected By Label Miscellaneous	0	480	January 4, 2019
Stale Element all of a sudden Web Testing katalon-studio	37	3535	March 30, 2022
[WebUI] Verify Option Not Selected By Label Miscellaneous	0	336	January 5, 2019