My code uses java.util.concurrent.ExecutorService
Line#115:
// create Thread pool
ExecutorService executorService = Executors.newFixedThreadPool(capacity)
List<Future<String>> futures = executorService.invokeAll(loadedTestClosures)
...
I wonāt do that. It makes the code too messy.
Unfortunately I am not able to find a pretty explanatory guide about Java and when to use thread or process.
And how. And when
So, despite is for python, a good introductory article can be found here, from where I learned a lot about such:
ā¦ and from there, following the links, are more example on how to choose the right solution, how to tune, and so on.
Since there is not a clear guidance for java (or at least I am not able to find such), on this matter ā¦I just throw that link and this is it, at this moment.
I played with the python examples and make sense what is written there at least for me, however I donāt have the needed bandwith right now to achieve the same with java, once the basic concepts are understood.
So I will just let that link here for somebody willing to actually explore.
Well, I modified the code and did a measurement. The resullt was as follows.
| ID | duration | graph |
|---|---|---|
| Renew browser windows, 1 Thread | 05:06 | #########1#########2#########3# |
| Renew browser windows, 2 Threads | 03:54 | #########1#########2#### |
| Renew browser windows, 4 Threads | 02:51 | #########1######## |
| Reuse a browser window, 1 Thread | 04:05 | #########1#########2##### |
| Reuse 2 browser windows, 2 Threads | 02:57 | #########1######## |
| Reuse 4 browser windows, 4 Threads | 02:00 | #########1### |
Previously, my code renewed browser window for every page browsing, 1 Thread processed 18 URLs in approximately 5 minuites.
This time, my new code resused browser windows for multiple page browsing, 1 Thread processed the same job in approximately 4 minuites; 4 Threads processed it in 2 minutes. I think the code change was effective enough for faster run.
Why the test which reuses a browser window for multiple page visits can run faster than the test which renews browser windows for every single page visit?
The reason is simple: Browserās content cache worked.
Browser can cache the web resources (images, css, js, etc) and reuse them to respond faster. In fact my test visits a single site http://www.katalon.com/ and its sub-pages which share a lot of web resources. If my test reuses a browser window to visit multiple sub-pages, the cached resources contributes for faster response.
On the other hand, WebDriver always opens a browser with empty cache. Therefore if I repeat open/close browser window for every single visit to the site, the cache would NOT contribute for faster response.
By the way, the Parallel mode of Test Suite Collection uses the multi-processes approach, not multi-threads. See the following screeshot:
I have a Test Suite Collection named TCS, which comprises with 4 Test Suites TS1, TS2, TS3 and TS4. Each Test Suite calls the same Test Case named blocker which just continues
WebUI.delay(1)
println "Hi"
for 100 times. When I execute it, I saw 4 Processes were created. I could see it in the monitor app of OS.
So, if you want to try a parallel execution of of Test Cases with multi-processes, the easiest way is to use Test Suite Collection in parallel mode.
Why am I not satisfied with the parallelizm by Test Suite Collection? Why did I challenge the TestClosure project? What was my motivation?
I am not concerned about the choice of Multi-Process / Multi-Thread at all.
I wanted to implement the following āinit - Map - Reduceā architecture, where the āMapā stages are executed parallelly, and the āReduceā stage must be executed after all of the āMapā stages finished.
+------+ +-------+
| init | ----+---> | Map 1 | ----+
+------+ | +-------+ | +--------+
... +----> | Reduce |
| +-------+ | +--------+
+---> | Map N | ----+
+-------+
I was not satisfied with the parallel mode of Test Suite Collection because it does not let me implement the āReduceā step.
So, I designed the āTestClosureā project let me do this MapReduce-like architecture cheaply. See the following source code if you are interested in
This is just a single Test Case script (which uses my custom Groovy classes); this implements the āinit - Map - Reduceā architecture that I wanted. TestClosure is cheap. This does not require any change in Katalon products; does not require any priced Cloud-based infrastructures.
Good to know, thx!
On a certain topic, TestNG has been mentioned as a better solution for parallel mode.
Well, I suppose the author was reffering to this:
Quoting:
Suppose, for an application you need to execute a sanity automation suite of 50 test cases in Chrome and Firefox browser. If you go with the traditional sequential flow, you need to execute the suite for Chrome browser first which would take 1 hr and then you need to execute for Firefox browser which takes another 1 hr. So, in total you would need 2 hrs to test in both the browsers. By using a parallel mechanism, you can run simultaneously for both the browsers in just 1 hr thereby reducing the execution time by 50%.
I donāt remember right now if TestNG is using multithreading or multiprocessing, I was playing with such long time ago. I think is multiprocessing (but I may be wrong)
However, is to be noted that, this mode is recomended when running a certain suite on two or more browsers at the same time.
Multiprocessing is better in this case, due to process isolation.
And, in this regard, Katalon does somehow exactly the same. Just duplicate the needed suite and set it with the needed profile.
To address different scenarios (e.g as requested in that topic, to run all tests from a suite in parallel ) a bit more caution is needed, as proved by @kazurayam experiments.
LE: to be clear, such feature has to be threaded. If not obvious, I can explain more why. And the side effects of this approach.
with this in mind, kazu approach was, intuitively, the right one ā¦ despite the webui test. the tests executed are quick enough to not overhelm the machine
First, it is to be determined what is the problem to be solved, before to choose the right solution.
After that, the harware resources has to be handled properly, otherwise if this mode is not used appropiate can very quick lead to resource exhaust.
Multithreading is more appropiate to solve pure IO bound issues (e.g. just api testing, or db testing, data post-processing etc).
Web testing involve also a CPU bound component.
Not pure CPU bound, such are mostly computational issues, but still, each thread will fire up his own browser instance, which is a process.
With this in mind, a threaded approach for webtesting does not solve a CPU bound issue, but it can create one if is not implemented properly.
Things may work fine on a small scale, but with a large amount of testcases ā¦ mhm.
So, I can understand why Katalon is reluctant to implement this, since everything can go wrong very easy.
Therefore, the user should implement his own custom solution for now. Which is possible, as demonstrated.
To be noted, with threaded parallelizations, a custom reporting solution may be needed, but this is another discussion.
Yes. Very true.
+1
Iām sorry to bother you over and over.
I found a room of performance improvement in my code that does the following steps:
My test script had a section that waits for an element to appear with timeout of 3 second, and I realized that this section was unnecessary. So I removed this section. Consequently my test script run faster for 3 seconds for each URL. I measured how long my test takes with 1 Thread, 2 Threads and 4 Threads as I did previously.
The resullt was as follows.
| ID | duration | graph |
|---|---|---|
| Renew browser windows, 1 Thread | 05:06 | #########1#########2#########3# |
| Renew browser windows, 2 Threads | 03:54 | #########1#########2#### |
| Renew browser windows, 4 Threads | 02:51 | #########1######## |
| Reuse a browser window, 1 Thread | 04:05 | #########1#########2##### |
| Reuse 2 browser windows, 2 Threads | 02:57 | #########1######## |
| Reuse 4 browser windows, 4 Threads | 02:00 | #########1### |
| Eliminate unnecessary wait, 1Thread | 02:16 | #########1#### |
| Eliminate unnecessary wait, 2Threads | 01:42 | #########1# |
| Eliminate unnecessary wait, 4Threads | 01:33 | #########1 |
In the begining, my test took 5 minutes to achieve what I wanted. In the end, my revised test took 1minutes 33secons to achieve the same.
In this series of experiments, I have identified several factors of performance improvementsļ¼
Now I know that the parallelism is just one of the factors for faster processing.