Parallel execution give more error

#1

Hi

in order for us to reduce our total execution time, running test suits in parallel after creating Suit collection

We used to see more failures of the test cases/steps when run in parallel after putting them in suit collection. The failures reasons appears in reports are mainly - script not able to click on specific elements/object and element not found even there are sufficient delay provided. I wanted to know why this happens and what action can be taken to resolve this issue

There are very less failures when we run the suit sequentially

Please help me with root cause of more failures during Parallel run the action needed to fix this problem

Thanks in advance
Deelip

#2

hi,

first of all check your pc mem & cpu usage during test running

#3

Hi

Thanks for your response

The VM capacity is 8 core and 64 GB memory so i do not think VM resource is issue but will capture the utilization of VM during parallel run

the scripts needs to perform operations in different browser during parallel run so may be katalon framework may be the issue. i used to run 3 suits in parallel and have observed this failures. if i increase more suits to run in parallel then more failures happens

Thanks

#4

hi,

are you using non GUI or with GUI model?

#5

Hi Timo

you mean Command line (Console mode) or running using Katalon Studio?

i run in Katalon Studio only and not command line

just wanted to give more info on failure

suppose i have total 9 suits and running 3 in parallel. in some suits specific test case/steps pass and in some suits the same test case/steps fails

Thanks
Deelip

#6

hi,

yes cmd or Katalon Studio
are you able to debug it?

#7

Hi Timo

1st of all, i appreciate all your time on this

what should i debug there as i am not seeing issue with scripts as the same scripts is passing for other suits and failing for some other suits

For example - If i have total 9 suits then may 6 suits will pass and 3 suits will fail. some time 7 pass and 2 fails. sometime 6 fails and 3 pass

I am not expert in Katalon so please guide by providing more details as what are the different things i should check and take appropriate action to fix the issue

Thanks with Regards
Deelip

#8

CPU utilization on this VM - below 50%
Memory utilization - below 30%

the above utilization is when 3 suits are running parallel

#9

Hi @Deelip_Kumar_Saxena

This is just my guess based on the symptoms, but it’s probably a consequence of multi-threading. When you use “Parallel Execution” mode in a Test Suite Collection, underneath Katalon will create many execution threads, and every now and then it switches to a different thread and continue that execution.

A side-effect is that during these switches, the browsers are in and out of focus, which may cause the web page (specifically the elements) to react unexpectedly. For example, an element may not be interactable if it’s not focused, and this may be the case when the entire browser is not focused.

I think for true (and safe) parallel execution, you would need to execute separate Test Suite Collections / Test Suites.

#10

Hi Thanh

thanks for your response

what do you mean by the below - I think for true (and safe) parallel execution, you would need to execute separate Test Suite Collections / Test Suites.

could you please provide more details on this

currently i have 9 suits and i put those all in one suit collection and run from suit collection (3 Parallel Run)

Thanks
Deelip

#11

Hi @Deelip_Kumar_Saxena

By that I simply mean you’d need to start separate execution with separate Runtime Engine / Katalon Studio.

#12

Hi Thanh

you mean, run another suit collection on separate machine/VM, right??

or same machine opening another katalon Studio and run it, per my understating if we do on same machine then we may face same issue, right??

Thanks
Deelip

1 Like
#13

Hi @Deelip_Kumar_Saxena

Unfortunately It’s hard to say, you’d have to try out each solution, since executing multiple Test Suites in parallel and using Parallel Execution mode in a TSC to execute them is not the same.

#14

@ThanhTo We need some guidance or some kind of direction to improve the performance of parallel test. Let me highlight some detail here: Currently, we are using VM’s for our regression test. There are 4 virtual machine with all have the same specification. The VM capacity is 8 core and 64 GB memory(E8_vs3).
We have 3 regression suite with each suite has 260 test scripts. When we run test parallel through HUB & NODE configuration, we saw failures in the regression test because of script can’t call the test case or object is not traceable. After rerunning those failure again, don’t see failures. Is there any way I can improve the performance of Regression Test ?
We have 2 KSE & 2 KRE licenses. I tried to run the test separately on each machine with each separate license.
Regression Suite -1 on 1st VM from KSE
Regression Suite -2 on 2nd VM from KSE
Regression Suite -3 on 3rd VM from KRE(cmd)

Is there any other suggestions anyone would recommend? I have also tried Katalon katilium hub & node configuration.

#15

Hi @sshah

If you’re talking about the error where element is not located, and that it’s resolved after re-running, then it sounds like a timing problem. I would suggest you identify the flaky Test Objects and put waiting statements (waitForElementPresent, etc) where appropriate. Additionally, you can try to enable Smart Wait in Project > Settings > Execution. It increases the execution time but perhaps it can make up by increasing stability.

I am not sure what “script can’t call the test case” means. Please provide the console log and relevant information so we can make a suggestion.

#16

Hello!

We are facing issues when there is a parallel execution of test suites on node locked licensed version of KRE.

We have a web application which is making a call to KRE. So when multiple users hit the application concurrently then we are facing issues.

We were able to see successful result for parallel execution on trial version.
But that’s not the case once we had licensed version.
We are receiving random errors or, sometimes it works perfectly fine.
Please let us know if we can do any changes on our end to handle this.

An explanation of why trial version works fine for parallel execution and licensed version doesn’t would help us to understand more.

Couple of random errors we received for parallel execution are-

  • Error: Offline activation failed. org.eclipse.core.internal.resources.ResourceException: Could not set the project description for ‘C%%Users%sa-eztools%Documents%KatalonWebDevTesting%WebDev Testing.prj’ because the project description file (.project) is out of sync with the file system.

  • Error: org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.NoClassDefFoundError: commonlib/Common$_googleAnalyticsLink_closure4