How does one get Katalon to handle asynchronous executeJavaScript calls?

kevin.jackey · January 31, 2023, 7:04pm

I am leveraging WebUI.executeJavaScript to retrieve web timing events that I currently have as JavaScript snippets in my browser console, the JavaScript in the jsCommandString below works correctly when executed in a browser console, however in Katalon it returns a zero which indicates that it’s not waiting for the promise to complete.

How can I get the promise to complete within Katalon before moving on to the next step?

//Calculate First Byte [needs work]
jsCommandString = "new PerformanceObserver((entryList) => {const [pageNav] = entryList.getEntriesByType('navigation');var TTFB = (pageNav.responseStart/1000).toFixed(2); return TTFB;}).observe({type: 'navigation',	buffered: true});"
def ttfb  = WebUI.executeJavaScript(jsCommandString, null)
log.logInfo ('Returned amount (TTFB)= ' + ttfb)

Russ_Thomas · January 31, 2023, 8:25pm

Short answer: no, you can’t, not using the regular WebUI API (or even Se as far as I am aware).

Longer answer: it’ll take quite some work, but it can be done, using a sentinel approach.

In Groovy you’ll need a loop (e.g while etc) which calls a JavaScript sentinel¹. When the sentinel changes state (becomes true perhaps?) it signals that the async job is done. Back in Groovy, you break out of the loop and read the results (again via JavaScript).

Why not do the whole thing in JS? While your code “returns” to the Groovy loop, the JS gets chance to run in the browser (non-blocking akin to node/deno).

Let me know if that’s enough to get you going or have more Qs.

1 A sentinel is the arbiter or coordinator over a signal or “change of state”.

kevin.jackey · January 31, 2023, 9:05pm

Hmm… will need to look up more details around the sentinel approach.

As for the second part (“why not do the whole thing in JS?”) I’m not quite sure what you mean. I need to ensure that all these promises have finished in my Katalon script before I can go on to the next phase though.

mwarren04011990 · January 31, 2023, 10:11pm

kevin.jackey:

//Calculate First Byte [needs work]
jsCommandString = "new PerformanceObserver((entryList) => {const [pageNav] = entryList.getEntriesByType('navigation');var TTFB = (pageNav.responseStart/1000).toFixed(2); return TTFB;}).observe({type: 'navigation',	buffered: true});"
def ttfb  = WebUI.executeJavaScript(jsCommandString, null)
log.logInfo ('Returned amount (TTFB)= ' + ttfb)

Your code, refactored…

Let’s clean up your code here:

final double ttfb = WebUI.executeJavaScript("""
new PerformanceObserver((entryList) => {
  const [pageNav] = entryList.getEntriesByType('navigation');
  var TTFB = (pageNav.responseStart/1000).toFixed(2); 
  return TTFB;
}).observe({
  type: 'navigation',	
  buffered: true,
});
""",
null);

log.logInfo("Returned amount (TTFB) = ${ttfb}")

Breakdown of your approach, and an initial suggestion…

If we look at your function, we see that you are writing a variable to the global JavaScript scope, called TTFB, with use of that var keyword.

Also, PerformanceObserver.observe() does not return a value, so this approach, as it is written, is incorrect.

Why not pull the var declaration out of the function, and separate concerns?

Implementing Russ’s suggestion

Russ is spot on about how to handle this. Let’s jump-start our implementation with some code at this blog post.

We’re going to create a general web UI util keyword, let’s call it WaitForCondition:

public final class GeneralWebUIUtils {
    public static boolean WaitForCondition(Closure<Boolean> onCheckCondition, int timeOut, Closure<String> onErrorMessage, FailureHandling failureHandling = FailureHandling.STOP_ON_FAILURE) {
        final long startTime = System.currentTimeMillis()
        boolean isConditionSatisfied = false;
        while ((System.currentTimeMillis() < startTime + timeOut * 1000) && (!isConditionSatisfied)) {
            isConditionSatisfied = onCheckCondition()
        }
        if ((!isConditionSatisfied) && (failureHandling.equals(FailureHandling.STOP_ON_FAILURE))) {
            KeywordUtil.markFailedAndStop("${onErrorMessage()} after ${(System.currentTimeMillis() - startTime) / 1000} seconds");
        }
        return isConditionSatisfied;
    }
}

We now have our util method, now let’s use it:

WebUI.executeJavaScript("TTFB = 0;", null); // set up the sentry variable

WebUI.executeJavaScript("""
new PerformanceObserver((entryList) => {
  const [pageNav] = entryList.getEntriesByType('navigation');
  TTFB = (pageNav.responseStart/1000).toFixed(2); 
}).observe({
  type: 'navigation',	
  buffered: true,
});
""",
null);

// TODO: do some action to trigger that obsever here, I guess...

// you might want to make this another general WebUI util...
GeneralWebUIUtils.WaitForCondition({ return (boolean)WebUI.executeJavaScript("return (TTFB > 0)", null)},
    10, // or however long you want to wait before failing the test
    { return "Time To First Byte (TTFB) exceeded the wait time."},
)

final double ttfb = (double)WebUI.executeJavaScript("return TTFB;", null);

WebUI.verifyNotEqual(ttfb, 0);

DISCLAIMER: I have NOT run this.

kazurayam · February 1, 2023, 1:33am

Let me show you my experiment.

import java.time.Duration
import java.time.LocalDateTime

import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

TestObject makeTestObject(String xpath) {
	TestObject tObj = new TestObject(xpath)
	tObj.addProperty("xpath", ConditionType.EQUALS, xpath)
	return tObj
}

WebUI.openBrowser("")
WebUI.navigateToUrl("https://katalon-demo-cura.herokuapp.com/profile.php#login")

String js = """
const sleep = m => new Promise(r => setTimeout(r, m));
(async () => {
    await sleep(3000);

    var newElement = "<div id='marker'>Marker</div>";
    var bodyElement = document.body;
    bodyElement.innerHTML = newElement + bodyElement.innerHTML;
})();
"""

LocalDateTime start = LocalDateTime.now()

WebUI.executeJavaScript(js, null)
WebUI.verifyElementPresent(makeTestObject("//div[@id='marker']"), 10)

LocalDateTime end = LocalDateTime.now()
WebUI.comment("${Duration.between(start,end).toMillis()} milliseconds passed")

WebUI.closeBrowser()

I think, this code is good as it looks simple. It uses usual Katalon WebUI keywords only. The js in this code is an old school javascript. There is nothing puzzling.

This code is bad as it modifies the DOM of the target HTML page. Modifying the target DOM could be unsafe. It is difficult to predict what will happen. You may encounter something strange.

This experiment is based on my idea as follows:

A Katalon script can not be notified of ant events in JavaScript at all. No way.
however a Katalon script can monitor JavaScript’s behavior indirectly via the DOM of the pages.

If your JavaScipt is designed to cause a change in the target DOM, then your Katalon test case script should just wait for the change to be present in the DOM using good old “WebUI.waitFor*” or “WebUI.verify*” keywords. That is enough to make your test case script to be in sync with asynchronous JavaScript. What sort of changes in DOM? that is case-by-case. This approach is not a magic spell that applies to all cases.

If your original JavaScript does not make any change in the DOM, … well your test script can intentionally make a change: insert a marker element as I did above. You would be able to make the marker element harmless somehow; e.g., by styling it with CSS property display:none.

kevin.jackey · February 1, 2023, 1:25pm

The idea is not to modify the DOM of the page, so if I’m doing that then I am definitely doing it wrong

The goal here is to be able to capture TTFB as outlined in the web.dev TTFB article [Time to First Byte (TTFB)] .

There is also a much simpler implementation of TTFB capture using the web-vitals library, but I’m not sure how to get that library into my Katalon project:

import {onTTFB} from 'web-vitals';

// Measure and log TTFB as soon as it's available.
onTTFB(console.log);

I will eventually need to capture CLS [Cumulative Layout Shift (CLS)] though, which does require waiting for a Promise to fulfill, so figured I might as well learn the techniques sooner rather than later

let clsValue = 0;
let clsEntries = [];

let sessionValue = 0;
let sessionEntries = [];

new PerformanceObserver((entryList) => {
  for (const entry of entryList.getEntries()) {
    // Only count layout shifts without recent user input.
    if (!entry.hadRecentInput) {
      const firstSessionEntry = sessionEntries[0];
      const lastSessionEntry = sessionEntries[sessionEntries.length - 1];

      // If the entry occurred less than 1 second after the previous entry and
      // less than 5 seconds after the first entry in the session, include the
      // entry in the current session. Otherwise, start a new session.
      if (sessionValue &&
          entry.startTime - lastSessionEntry.startTime < 1000 &&
          entry.startTime - firstSessionEntry.startTime < 5000) {
        sessionValue += entry.value;
        sessionEntries.push(entry);
      } else {
        sessionValue = entry.value;
        sessionEntries = [entry];
      }

      // If the current session value is larger than the current CLS value,
      // update CLS and the entries contributing to it.
      if (sessionValue > clsValue) {
        clsValue = sessionValue;
        clsEntries = sessionEntries;

        // Log the updated value (and its entries) to the console.
        console.log('CLS:', clsValue, clsEntries)
      }
    }
  }
}).observe({type: 'layout-shift', buffered: true});

kazurayam · February 1, 2023, 1:48pm

I can not understand your code, your intention. Sorry, I willl quit this topic.

kevin.jackey · February 1, 2023, 2:42pm

This worked exceedingly well, now i’ll try to apply it to the remainder of the “promise-based” web timings (CLS, FCP, FID, LCP)

kevin.jackey · February 1, 2023, 4:03pm

Have been able to implement this successfully for all of the standard measurement timings except one - Cumulative Layout Shift (CLS). The reference calculation can be found at the following link: Cumulative Layout Shift (CLS)

So I have the following code implementation of this logic:

//Calculate Cumulative Layout Shift (CLS)
	///Setup Sentry Variable
	WebUI.executeJavaScript("""
			let clsValue = 0;
			let clsEntries = [];
			let sessionValue = 0;
			let sessionEntries = [];
			let CLS = 0;
		""", null);
	
	////Start Promise Action
	WebUI.executeJavaScript("""
		new PerformanceObserver((entryList) => {
		  for (const entry of entryList.getEntries()) {
		    // Only count layout shifts without recent user input.
		    if (!entry.hadRecentInput) {
		      const firstSessionEntry = sessionEntries[0];
		      const lastSessionEntry = sessionEntries[sessionEntries.length - 1];
		
		      // If the entry occurred less than 1 second after the previous entry and
		      // less than 5 seconds after the first entry in the session, include the
		      // entry in the current session. Otherwise, start a new session.
		      if (sessionValue &&
		          entry.startTime - lastSessionEntry.startTime < 1000 &&
		          entry.startTime - firstSessionEntry.startTime < 5000) {
		        sessionValue += entry.value;
		        sessionEntries.push(entry);
		      } else {
		        sessionValue = entry.value;
		        sessionEntries = [entry];
		      }
		
		      // If the current session value is larger than the current CLS value,
		      // update CLS and the entries contributing to it.
		      if (sessionValue > clsValue) {
		        clsValue = sessionValue;
		        clsEntries = sessionEntries;
		      }
		    }
		  }
		  CLS = clsValue
		
		}).observe({type: 'layout-shift', buffered: true});
		""",null);
	
	///Call wait_for_condition function
	Common_Functions.wait_for_condition({ return (boolean)WebUI.executeJavaScript("return (CLS > 0)", null)},
		10, // or however long you want to wait before failing the test
		{ return "Cumulative Layout Shift (CLS) exceeded the wait time."},
	)
		
	///Return final CLS Value and display
	def cls = WebUI.executeJavaScript("return CLS;", null);
	log.logInfo ('Returned amount (CLS)= ' + cls)

yet I’m getting this error, which I’m not getting with the other similar functions:

Unable to execute JavaScript. (Root cause: com.kms.katalon.core.exception.StepFailedException: Unable to execute JavaScript.
	at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.stepFailed(WebUIKeywordMain.groovy:64)
	at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.runKeyword(WebUIKeywordMain.groovy:26)
	at com.kms.katalon.core.webui.keyword.builtin.ExecuteJavaScriptKeyword.executeJavascript(ExecuteJavascriptKeyword.groovy:42)
	at com.kms.katalon.core.webui.keyword.builtin.ExecuteJavaScriptKeyword.execute(ExecuteJavascriptKeyword.groovy:37)
	at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
	at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords.executeJavaScript(WebUiBuiltInKeywords.groovy:4873)
	at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords$executeJavaScript$1.call(Unknown Source)
	at Script1675184790214$_run_closure9.doCall(Script1675184790214.groovy:226)
	at Script1675184790214$_run_closure9.doCall(Script1675184790214.groovy)
	at btt_portal.Common_Functions.wait_for_condition(Common_Functions.groovy:120)
	at btt_portal.Common_Functions.wait_for_condition(Common_Functions.groovy)
	at btt_portal.Common_Functions$wait_for_condition.call(Unknown Source)
	at Browser Console Retrieval Experimentation.run(Browser Console Retrieval Experimentation:226)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:448)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:439)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:418)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:410)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:285)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
	at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
	at TempTestCase1675267145594.run(TempTestCase1675267145594.groovy:25)
Caused by: org.openqa.selenium.JavascriptException: javascript error: CLS is not defined
  (Session info: chrome=109.0.5414.76)
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'
System info: host: 'BTT0247', ip: '192.168.0.100', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_282'
Driver info: com.kms.katalon.selenium.driver.CChromeDriver
Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 109.0.5414.76, chrome: {chromedriverVersion: 109.0.5414.74 (e7c5703604da..., userDataDir: C:\Users\KEVINJ~1\AppData\L...}, goog:chromeOptions: {debuggerAddress: localhost:64343}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: WINDOWS, platformName: WINDOWS, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: b15fd01a21fd3c9ddb72f90349194f85
	at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
	at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
	at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
	at com.kms.katalon.selenium.driver.CChromeDriver.execute(CChromeDriver.java:19)
	at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:489)
	at org.openqa.selenium.support.events.EventFiringWebDriver.lambda$new$1(EventFiringWebDriver.java:105)
	at com.sun.proxy.$Proxy13.executeScript(Unknown Source)
	at org.openqa.selenium.support.events.EventFiringWebDriver.executeScript(EventFiringWebDriver.java:229)
	at com.kms.katalon.core.webui.keyword.builtin.ExecuteJavaScriptKeyword$_executeJavascript_closure1.doCall(ExecuteJavascriptKeyword.groovy:48)
	at com.kms.katalon.core.webui.keyword.builtin.ExecuteJavaScriptKeyword$_executeJavascript_closure1.call(ExecuteJavascriptKeyword.groovy)
	at com.kms.katalon.core.webui.keyword.internal.WebUIKeywordMain.runKeyword(WebUIKeywordMain.groovy:20)
	at com.kms.katalon.core.webui.keyword.builtin.ExecuteJavaScriptKeyword.executeJavascript(ExecuteJavascriptKeyword.groovy:42)
	at com.kms.katalon.core.webui.keyword.builtin.ExecuteJavaScriptKeyword.execute(ExecuteJavascriptKeyword.groovy:37)
	at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
	at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords.executeJavaScript(WebUiBuiltInKeywords.groovy:4873)
	at com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords$executeJavaScript$1.call(Unknown Source)
	at Script1675184790214$_run_closure9.doCall(Script1675184790214.groovy:226)
	at Script1675184790214$_run_closure9.doCall(Script1675184790214.groovy)
	at btt_portal.Common_Functions.wait_for_condition(Common_Functions.groovy:120)
	at btt_portal.Common_Functions.wait_for_condition(Common_Functions.groovy)
	at btt_portal.Common_Functions$wait_for_condition.call(Unknown Source)
	at Script1675184790214.run(Script1675184790214.groovy:226)
	... 11 more
)

You two have gotten me this far, I’m almost there but can’t for the life of me figure out why this last item is failing when the other metrics ran perfectly. Any thoughts?

mwarren04011990 · February 1, 2023, 4:18pm

It looks like it’s failing around the CLS variable check…

It seems like you declared that with let instead of var…

kevin.jackey · February 1, 2023, 4:36pm

Grrr… I’m at a loss with JS sometimes, the official JS guidelines are that var is deprecated, yet here in this intance it may be causing a failure. sigh.

Will check this when I get back to my desk, and thanks for the reply!

mwarren04011990 · February 1, 2023, 4:52pm

Yea, it’s all about scoping:

when you declare with let or const, it is declared in the local scope. Only things in the local scope, or any inner scope (e.g. in functions) can access it.

This is crucial because the way WebUI.executeJavaScript() (and Selenium equivalent) works, is that it runs your JavaScript inside an immediately invoked function expression. Hence, when you declare something with let inside that statement, it won’t be available to another piece of code inside another WebUI.executeJavaScript() call .

when you declare a variable with var, or simply define it, it is declared in global scope. ANYTHING can access it, even code outside the function it was defined in!

Try it and it should work

UPDATE: Try declaring your sentry variables without any prefix keywords:

no let
no var

kevin.jackey · February 1, 2023, 7:36pm

Glad I double checked and looked at your last comment, as I had changed it to var and it still failed. Was getting ready to pull my hair out until I saw your suggestion to not prefix the declaration at all…and that worked!! Thank you!!!

kazurayam · February 2, 2023, 12:42am

My idea is to allow modifying the DOM of the page. I changed my code

specify <div style="display:none" to the inserted element
insert the marker element at the very bottom of the page so that it should not cause any “Cumulative Layout Shift”.
give an “id” attribute to the marker element: <div id="xxxxx" where xxxxx is a unique string based on the current timestamp of miiliseconds.

import java.time.Duration
import java.time.LocalDateTime

import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

TestObject makeTestObject(String xpath) {
	TestObject tObj = new TestObject(xpath)
	tObj.addProperty("xpath", ConditionType.EQUALS, xpath)
	return tObj
}

WebUI.openBrowser("")
WebUI.navigateToUrl("https://katalon-demo-cura.herokuapp.com/profile.php#login")

String MARKER_ID = "marker${new Date().getTime()}"

String js = """
const sleep = m => new Promise(r => setTimeout(r, m));
(async () => {
    await sleep(3000);

	// insert a invisible marker at the bottom of the page
    var newElement = "<div id='${MARKER_ID}' style='display:none'>Marker</div>";
    var bodyElement = document.body;
    bodyElement.innerHTML = bodyElement.innerHTML + newElement;
})();
"""

LocalDateTime start = LocalDateTime.now()

WebUI.executeJavaScript(js, null)
WebUI.verifyElementPresent(makeTestObject("//div[@id='${MARKER_ID}']"), 10)

LocalDateTime end = LocalDateTime.now()
WebUI.comment("${Duration.between(start,end).toMillis()} milliseconds passed")

WebUI.closeBrowser()