Parallel execution is inconsistent &/or not reliable

Hi Katalon Team,
We are in process to setup &/or evaluate automation using Katalon studio v 7.7.2 for our application under testing. The AUT has WEB, API and thin tiers. To start with I created 3 test cases for REST API & 3 test-cases for web-ui and tried to ran these test cases in parallel execution mode. Katalon studio always throwing different errors for above 3 web-ui test cases though I’m using same platform(I.e chrome) for all runs.

I analyzed that Katalon studio throwing different errors for every run.

Per Katalon documentation &/or forum responses I tried below options:

  1. Reduced thread pool size to 2 threads still Katalon studio throwing errors for 3 web-ui tests.

  2. I tried to increase global project page load time - 30 seconds & actions time - 30 seconds underneath project ->settings->execution - web-ui setting but no luck.

  3. I tried to increase heap size and tweak garbage collection algorithm but still web-ui test cases steps failing for different reasons.

  4. Then I added is web-ui element visible method to detect web component visibility and based on visibility traversing subsequent logic. However those web-ui test cases failing for different reasons.

Per one Katalon studio forum thread - This is just my guess based on the symptoms, but it’s probably a consequence of multi-threading. When you use “Parallel Execution” mode in a Test Suite Collection, underneath Katalon will create many execution threads, and every now and then it switches to a different thread and continue that execution.

A side-effect is that during these switches, the browsers are in and out of focus, which may cause the web page (specifically the elements) to react unexpectedly. For example, an element may not be interactable if it’s not focused, and this may be the case when the entire browser is not focused.

Does this means parallel execution modes is not working at all ?

Please advice is there is any problem in my approach for running parallel mode & suggest how to rectify errors occurred while executing web-ui tests in parallel execution mode.

Thanks,
Ganesh

@ThanhTo,

@ganeshpunde naively assumes here that the parallel execution of Test Suite Collection is implemented by multiple Java Threads. Is it really the case?

Yes, it is the case. I spoke in some other posts about this. Test Suite Collection’s parallel execution is akin to a more thread-switching rather than true parallel executions. To overcome the limitations, one should use multiple Runtime Engine instances for parallel executions.

Q. Does Katalon studio create thread per web component action/ test case / test suite ?

Merely want to understand if I do logical grouping of test cases / test suites based on test case/ test suites / tiers I.e. if services test cases in one group and execute in parallel mode and write output in persistent storage may be file or db and then use that to run web-ui test cases/ test suites in serial execution mode.

@ganeshpunde

A thread is created per Test Suite in a Test Suite Collection, you would want to be careful when writing from one thread and reading from another thread.

thank you so much, it will help.

@ThanhTo

No. This is not correct. I believe that an OS-level Process (=Java VM instance) is forked per Test Suite in a Test Suite Collection. I have got an evidence.

I have made a project published on GitHub:

In this project, you will find a Test Suite Collection named “TSC_usingTimestampedLogger” which binds 4 Test Suites and execute them in parallel mode. I executed the TSC and monitored Activity Monitor on Mac to see how OS-level processes behave.

When I executed the TSC, I could see 4 “java” processes are newly forked and finished just in sync with 4 Test Suites in the TSC started and finished. The following screenshot shows it.

If Test Suites in parallel mode run in Java Threads, I should have never observed OS-level Processes being forked.