TestClosure --- executing Groovy Closures in multi-Threads in a Test Case simultaneously

I have made a GitHub project:


This project proposes a new concept TestClosure for Katalon Studio. This project provides a preliminary implementation and demonstrations.

Problem to solve

Many of Katalon Studio users want to execute multiple test processings parallelly . Me too. They have their own requirements. In my case, I want to take a lot of web page screenshots as quickly as possible.

As I wrote in the README of ExecutionProfilesLoader project, last year I wanted to take screenshots of 6000 URLs. The job took me 6000 * 10 seconds = over 17 hours. Obviously it’s too long. I wanted to make the job run faster.

It is a simple job to take a screenshot of a web page.

Open browser, navigate to URL, take screenshot, save image to file, close browser .

Every time we navigate to a new URL, we are forced to wait for a few seconds until the page is fully loaded. This wait makes our sequential processing very slow.

If have a machine with 8 core-CPU, I can process these 6000 pages with 8 threads, then the job will be done in 17 hours / 8 threads = 2.2 hours.

But how can I do multi-threading in a Test Case of Katalon Studio?

I know that Katalon Studio offers a feature of executing Test Suites in parallel mode. It isn’t satisfactory for me. It is too coarse-grained. I want a way to execute a piece of Groovy scripts in fine-grained / light-weighted Java Threads.

Solution

1. You can call Groovy Closure in Katalon Test Case

It is quite easy to create a Groovy Closure that contains WebUI.* statements in a Test Case. Executing it is a breeze. See the following example.

When you execute this, you will see a browser window opens, shows a web page, and closes.

2. You can excute Groovy Closures using ExecutorService utility

java.util.concurrent.ExecutorService makes concurrent programming in Java/Groovy much easier. The ExecutorService accepts objects that implements the java.util.concurrent.Callable interface and executes them. A Groovy Closure implements the java.util.concurrent.Callable interface. So, you can execute Closures of WebUI.* statements using ExecutorService utility in a Katalon Test Case.

The following sample code shows 2 Closures are in 2 threads sequantially in Katalon Test Case.

When you execute this code, you will see 2 browser windows opened/closed sequentially.

3. You can parametrize Closures

I want to parametrize a Closure. I want to execute 2 instances of a Groovy Closure with different values of parameters. The following code shows how to do it. I made a Groovy Class URLVisitor which implements the java.util.concurrent.Callable interface.

When you execute this code, you will see 2 browser windows opened/closed sequentially.

4. You can only see 1 browser window

Now I want to execute 2 Closures simultaneously, in parallel. The following code does that:

The code difference between this and the previous one is only 1 character. The size of Thread Pool is changed: 1 → 4.

ExecutorService exService = Executors.newFixedThreadPool(4)

Of course I would expect to see 2 Browser windows to open.

When I executed this example, I got puzzled. I executed 2 threads in parallel, and I saw only 1 window opened. why?

What you see is not what you get, sometimes. The fact is, acutally 2 windows were opened by the script, but the windows were displayed overlayed at the same position (x, y coordinate) with just the same dimension (width, height). So I could see only one. This behavior is very confusing.

5. Introduce TestClosure to manage layout of browser windows

I want to manage the layout of browser windows. I want them displayed at different positions (x, y) so that I can see them identifiable. But how can I do it?

WebDriver API provides tools for us to solve this problem. You can explicitly set Window Position and Size. For example,

driver.manage().window().setPosition(new Point(50,200));

and

driver.manage().window().setSize(new Dimension(300,500));

I can should be able to set position and size to the windows opened by WebUI.openBrowser('') call inside the Closures in Test Cases. The following code shows how I managed it.

I introduced a custom helper BrowserWindow.layout(metrics, position) that moves the browser window appropriately.

Problems are resolved.

Miscellaneous demo videos

You can view the videos by clicking the following links:

demo video Test Suite elasped (seconds) what it does
demo1A 27 visits 3 URLs sequentially, windows are not managed
demo1B 30 visits 3 URLs sequetioally, windows are managed in Tile layout
demo1C 55 visits 5 URLs sequentially, windows are managed in Stack layout, takes screenshots
demo2A 20 visits 3 URLs simultaneously, windows are overlayed at the same (x,y) position with the same width/height
demo2B 21 visits 3 URLs simultaneously, windows are managed in Tile layout
demo2C 48 visits 5 URLs simultaneously, windows are managed in Stacklayout, takes screenshots
demo3D 156 takes Full Page screenshots of 8 URLs using 2 browsers simultaneously. A browser is reused to process 4 URLs each. Unfortunately WebUI.takeFullPageScreenshot() takes long time (approx. 30 seconds each). Unable to post video for demo.

Using the following environment:

  • Mac Book Air, Processor: 1.6GHz Dual Core Intel Core i5, Memory 16GB
  • macOS Big Sur
  • Katalon Studio v7.9.1

Caution: videos are compressed

When I executed a Test Suite in Katalon Studio to take the demo2A, the Test Suite acutally took 20 seconds to finish processing. But you will find the “movie” plays in 7 seconds. The movie plays far quicker than the acutal scene. I suppose that, while I uploaded the files to Google Drive, the movies were compressed to reduce the size by chomping the still frames off.

Design Description

This project includes a set of working codes. Please read the source to find the detail. The code will tell you everything I could.

Here I write quick pointers:

Entry point

Have a look at a Test Case that opens 3 browser windows simultaneously:

The code is very short. Is it simple? — not really. This short code triggers a lot.

What is TestClosure? How to write it?

Essentially a TestClosure object is a pair of a Groovy Closure and a list of parameters for the closure. Plus a TestClosure carries information where the browser window should be located and how it should be sized. The source of TestClosure is this:

The following Test Case shows an example of creating a list of TestClosure objects.

How to execute TestClosures in parallel?

The Test Case Test Cases/demo/demo2B_simultaneous_tiling calles TestClosureCollectionExcecutor . This object executes the collection of TestClosures simultaneoutly.

The source code is here:

TestClosuresCollectionExecutor creates a thread pool of 4 as default. You can change the maxThread by parameter to the Buiilder. The value 1 - 16 is accepted.

2 strategies of window layout

This project provides 2 strategies of browser window layout:

  • Tiling : see demo2B for example
  • Stacking : see demo2C for example

TestClosuresCollectionExecutor uses the Tiling strategy as default. You can explicitly specify the strategy as a parameter to the Builder.

Multiple threads — does it run faster?

Yes, it does on a machine with 2 or more core-CPU. If my test case executes mutilple TestClosures with multiple threads, it rus faster than with a single thread.

Of course, the faster processing demands more powerful machine resources. If I can afford Mac Book Air M1 with 8-core CPU, my test case with Thread Pool of 8 will run far faster. I want to see it!

It is pointless to give the maximum number of threads for TestClosureCollectionExecutor with a value larger (8, 16, 32) than the number of cores you have.

Conclusion

I could implement a Test Case that executes multiple Groovy Closures with multiple Threads. The code ran faster on my dual-core machine than with single Thread.

The codes of this project is yet preliminary. It has a lot more to develop. Especially, it does not consider failure handling seriously.

5 Likes

A plug-in jar named kazurayam-ks-testobject-x.x.x.jar is available at the Releases page. This jar contains com.kazurayam.ks.testclosure.TestClosure and related classes. Once you install this jar into your project’s Plugins directory, you can quickly start multi-threading in Katalon Studio.

Is this still a “preliminary” release, Kaz?

I have considered failure handling in the recent version.
So, it is not preliminary.
I find no more to add in this project.

1 Like

I have published a new versin 0.21.0 of TestClosure.

In v0.20.0, a typical TestClosure was constructed as this:

import com.kazurayam.ks.testclosure.TestClosure
import org.openqa.selenium.Dimension
import org.openqa.selenium.Point
import com.kazurayam.ks.windowlayout.BrowserWindowLayoutKeyword as BrowserWindow

new TestClosure({ Point position, Dimension dimension ->
	String url = 'http://forum.katalon.com/'
	WebUI.openBrowser('')
	//
	BrowserWindow.layout(position, dimension)
	//
	WebUI.navigateToUrl(url)
	WebUI.comment("processing ${url}")
	TestObject tObj = newTestObjectXPath("//a[contains(text(),'How To Help Us Help You')]")
	WebUI.verifyElementPresent(tObj, 5)
	WebUI.scrollToElement(tObj, 5)
	WebUI.delay(1)
	WebUI.closeBrowser()
	}, [])

As you can see, this TestClosure is responsible for opening/closing browser and for positioning the browser window. The code is lengthy, therefore is error prone.

Now in v0.21.0, a typical TestClosure is constructed as this:

import com.kazurayam.ks.testclosure.TestClosure
import org.openqa.selenium.WebDriver
import com.kms.katalon.core.webui.driver.DriverFactory

TestClosure tc = new TestClosure({ WebDriver driver ->
    DriverFactory.changeWebDriver(driver)
	String url = 'http://forum.katalon.com/'
	WebUI.navigateToUrl(url)
	WebUI.comment("processing ${url}")
	TestObject tObj = newTestObjectXPath("//a[contains(text(),'How To Help Us Help You')]")
	WebUI.verifyElementPresent(tObj, 5)
	WebUI.scrollToElement(tObj, 5)
	}, [])

Now the framework com.kazurayam.ks.testclosure.TestClosureCollectionExecutor is entirely responsible for opening/closing browser, positioning browser window. An instance of WebDriver is passed to a TestClosure as argument. A TestClosure just uses the given WebDriver instance. So the code of a TestClosure in v0.21.0 is much simpler than v0.20.0.

1 Like

Recently I bought a new Mac Book Air with M1 Processor (8 core). I tried to execute 8 Threads in parallel on it. The result was interesting — far slower than 2 threads.

The reason was obvious — my home Wifi network was too slow. 8 simultaneous HTTP sessions jammed it. I learned that a TestClosure project requires a fast-enough network connection. I would need an EC2 on AWS to prove the potential of TestClosure.

I haven’t studied enough why. I can see 8 Chrome browser windows are opened. But it seems that only 1 Chrome browser is active at a time.

1 Like

The following post may be relevant:

Solution
We have to use different user-data directory for each profile. We no need to create profile manually in chrome browser and also no need to provide --profile-directory argument in chrome options. But you can maintain the sessions and history by mentioning different user-data-dir path for each chrome driver instance

In the TestClosure project I have a Test Case which opens 4 Chrome browser windows and visits several URLs starting with http://www.katalon.com/. The Test Case drivers 4 threads in parallel. I expected that test runs faster than 1 thread.

However, to my surpise, the test run very slowly. See the following measurement.

It took 60 seconds for this test case script to open https://www.katalon.com/. When I manually open the same URL, it opens in a few seconds. It does not take 60 seconds. Why does it take so long for my test case script to open https://www.katalon.com/?

I have studied this issue in the last 10 months. Now I have found out the reason.

My program opened a Chrome browser like this:

			// open a browser window for this TestClosure
			WebUI.openBrowser('')

This is the usual way in Katalon Studio scripting. This call opens a Chrome browser with completely empty cache.

As you know, this browser has no HTML files cached, no CSS cached, no JavaScript cached, no GIF/JPEG/PNG images cached. When my script navigate to the URL https://www.katalon.com/, yes, the page will eventually open. But it takes long seconds to download all these resources.

Now I have realised that WebU.openBrowser('') is not enough for me to make my test scripts with TestClosure framework run faster. I need to invent a way to launch Chrome browser specifying custom profiles Katalon1, Katalon2, Katalon3 and Katalon4 which keep the web resources are cached and reusable.

Provided with a new way of launchig Chrome with custom profile, I will prepare these 4 profiles to cache web resources of the target URL https://www.katalon.com prior to testing. Then my test script should launch 4 Chrome browser with these well-prepared custom profiles which supply cached web resources. Then the test script will run much faster.

But 60 seconds??? It shouldn’t take that long, regardless.

Good eyes you have.

60 second is the timeout limit I set to Katalon Studio to wait for the page load. It seemed the page did not finish loading within that duration.

I noticed that “http://www.katalon.com/” has some links to external host’s URL, and somehow HTTP session to that external URL does keep on going longer than the timeout seconds; or maybe HTTP session is kept unclosed forever, so that Katalon Studio does not recognise the end of page loading. But I find this long page loading only when I opened browser by automation script using “WebUI.openBrowser(‘’)”; I do not see it when I open the page manually; it is puzzling. I think that this puzzle has nothing to do with multi-threading. This puzzle would have something to do with the way how I launch Chrome browser ---- with profile or without it, how desired-capabilities are set, etc.

Anyway, I haven’t look at the nature of this external URL careful enough. I will look into the external URLs that seem to be never finishing to load. If I find the session is really never finishing and not really important, then I would change the timeout setting more quicker: 60 → 20 seconds for example.

1 Like

I have found out a reason of 60 seconds.

I used a secret method of activating Chrome browser using existing User Profile, not a default profile. I developed a class to do this. The class took very long time to copy files under an existing profile’s directory to a temporary directory. The files include browsing history, cached pages+images, etc. When I specified my “Default” Chrome profile which contained so many files (nearly 30,000), it took 30 seconds to copy files to a temporary directory. I examined a different case. I altered my code to use another Chrome profile, which contains almost zero history and cache files, to launch Chrome. Then it took just a few seconds to open browser. Well, there is a lot of pitfalls before launching a browser with XxxxDriver.

1 Like

Now I do not recommend for others to follow my TestClosure approach any longer.

See the following post for the reason why I don’t.

Primarily because

However I do not recommend you to follow this way. Because any problems in this system are difficult to analyse and fix. I haven’t developed any easy-to-use debugging facility for “multi-threaded Test Cases” (I am not capable of developing such magical thing). I only have the Stack Trace messages emitted by Exceptions. I am OK with it, but you would be not. I do not think I can support you remotely.

2 Likes

worth to mention also, quoting:
I learned a simple & fundamental lesson: I need more machine + network resources to process more tasks in a limited time frame.

A year and 10 months have passed since I wrote:

Today, I happened to find the cause why my tests ran so slow. The Smart Wait was enabled in this project. That was the reason why it took so long for my tests to open https://www.katalon.com/.

When I disabled the Smart Wait, my tests ran jumping fast!

I guess that http://www.katalon.com/ pages employ some AJAX technique. The HTML DOM of the pages continue moving; AJAX event stream never stops. Therefore the Smart Wait continues waiting long until the timer expires after 30 seconds.

My mistake! I should have been more cautious about the Smart Wait and the nature of the target web application. Now I am pleased that I could resolve this long-outstanding issue for me.

I understand that multi-threading was not the cause of slowness. Multi-Threaded execution of Test Cases by “TestClosure” might be a useful trick though I previously thought it isn’t.

I have updated the project

and published a new releases 0.25.1. With it I did an experiment how much multi-threading can improve the performance of a test case. I used my MacBook Air, M1, 8GB memory. Katalon Studio Arm64 9.0 with Internet connection 6Mbps

By executing the Test Suites/demo/demo3D_1_2_4Threads, a test case processes 18 URLs. The test case runs multiple Java Threads. Each thread executes a Groovy closure. Each closure visits a URL and take a screenshoot and save it into disk. I executed the test case with 1 thread, 2 threads and 4 threads. The durations measured were as follows:

ID duration graph
1 Thread 05:06 #########1#########2#########3#
2 Threads 03:54 #########1#########2####
4 Threads 02:51 #########1########

A single # means 10 seconds.

It took approximately 5 minutes to process 18 URLs with a single thread.
With 2 threads, it took approximately 4 minutes.
With 4 threads, it took approximately 3 minutes.

@kazurayam interresting …
anyway, I saw this sentence in your git project:

It is pointless to give the maximum number of threads for TestClosureCollectionExceutor with a value larger (8, 16, 32) than the number of cores you have.

Is this based on experiments, or an assumption?
Why do I ask? Well, there are two known methods for parallelization in Java (and not only in Java).
Processes and threads, a short description about can be found here:
Process vs Thread: What's the Difference? - javatpoint.

Not sure exactly how ExecutorService actually works, but it seems to be an interface to help create and execute threads (and async top of that, which is yet another clever mechanism for this case as there is no obvious reason for inter-thread communication)

In the case of processes, it make sense to limit the number of them to numproc (or numproc -1 sometime) because those are handled by the kernel scheduler. So having more processes than the number of cores, some will be queued until resources are released.

However, with threads, the limitations usually came from other factors (mem available / heapsize, iowaits etc)

So, if i am guessing right and the ExecutorService is using just threading, you can try to experiment a bit by going with the pool size with larger values, until you find an optimum (it is not a guarantee it will perform better, since, as I mentioned, there are lot of other factors involved, but worth to try)

You can also keep an eye on the taskmanager and see how many java processes are running when you exectute the tests.

If you see just a single process (but with plenty threads, ofcourse), then everything is just threaded, if you see more than one, it may be that under-the-hood it is forking processes, so the limitation to numproc start to make sense.

Just a guess, no backing experiments.

I would erase this sometime in future.

I am going to do one more variation of experiments. In the current experiments, the controller class repeats opening/closing a browser window for each invokation of Groovy Closure. I took this design because it is easy to implement. But, as you know, launching a browser window is a heavy processing. It would be a good idea to launch a limited number of browser windows (as many as the number of threads), and to reuse a single browser window for multiple times of closure invocation. I expect that the test will run a bit faster.

worth to try but you may face another issue.
not sure if next thread in queue will know what browser instance to use.
so, you may have to pass also the driver instance from one thread to another and in this case async thread may not help.
you will have to use interthread comunication but the botleneck may be the browser instance.
some funny stuff may happen if you start the browser from the main thread and execute two or more ‘navigate to’ at the same time. so each thread will have to wait until the previous one releases the browser, which may not help at all.

will dig a bit, i think for such forking processes may be a better aproach, so each process to use a dedicated browser, and further execute the tasks either sequentially or threaded.
with processes, you should actually benefit by the cpu cores available.
i think i saw once some doc about a process executor similar to the one you used (or i may confuse it with python…)
anyway, it is not a guarantee this aproach will be better, since multiprocessing it usually used to solve cpu bound / compute issues, threads are better when the problem to solve is IO bound

@kazurayam I thing the equivalent of ExecutorService for multiprocessing may be

worth to explore, imho