in order for us to reduce our total execution time, running test suits in parallel after creating Suit collection
We used to see more failures of the test cases/steps when run in parallel after putting them in suit collection. The failures reasons appears in reports are mainly - script not able to click on specific elements/object and element not found even there are sufficient delay provided. I wanted to know why this happens and what action can be taken to resolve this issue
There are very less failures when we run the suit sequentially
Please help me with root cause of more failures during Parallel run the action needed to fix this problem
The VM capacity is 8 core and 64 GB memory so i do not think VM resource is issue but will capture the utilization of VM during parallel run
the scripts needs to perform operations in different browser during parallel run so may be katalon framework may be the issue. i used to run 3 suits in parallel and have observed this failures. if i increase more suits to run in parallel then more failures happens
what should i debug there as i am not seeing issue with scripts as the same scripts is passing for other suits and failing for some other suits
For example - If i have total 9 suits then may 6 suits will pass and 3 suits will fail. some time 7 pass and 2 fails. sometime 6 fails and 3 pass
I am not expert in Katalon so please guide by providing more details as what are the different things i should check and take appropriate action to fix the issue
This is just my guess based on the symptoms, but itâs probably a consequence of multi-threading. When you use âParallel Executionâ mode in a Test Suite Collection, underneath Katalon will create many execution threads, and every now and then it switches to a different thread and continue that execution.
A side-effect is that during these switches, the browsers are in and out of focus, which may cause the web page (specifically the elements) to react unexpectedly. For example, an element may not be interactable if itâs not focused, and this may be the case when the entire browser is not focused.
I think for true (and safe) parallel execution, you would need to execute separate Test Suite Collections / Test Suites.
what do you mean by the below - I think for true (and safe) parallel execution, you would need to execute separate Test Suite Collections / Test Suites.
could you please provide more details on this
currently i have 9 suits and i put those all in one suit collection and run from suit collection (3 Parallel Run)
Unfortunately Itâs hard to say, youâd have to try out each solution, since executing multiple Test Suites in parallel and using Parallel Execution mode in a TSC to execute them is not the same.
@ThanhTo We need some guidance or some kind of direction to improve the performance of parallel test. Let me highlight some detail here: Currently, we are using VMâs for our regression test. There are 4 virtual machine with all have the same specification. The VM capacity is 8 core and 64 GB memory(E8_vs3).
We have 3 regression suite with each suite has 260 test scripts. When we run test parallel through HUB & NODE configuration, we saw failures in the regression test because of script canât call the test case or object is not traceable. After rerunning those failure again, donât see failures. Is there any way I can improve the performance of Regression Test ?
We have 2 KSE & 2 KRE licenses. I tried to run the test separately on each machine with each separate license.
Regression Suite -1 on 1st VM from KSE
Regression Suite -2 on 2nd VM from KSE
Regression Suite -3 on 3rd VM from KRE(cmd)
Is there any other suggestions anyone would recommend? I have also tried Katalon katilium hub & node configuration.
If youâre talking about the error where element is not located, and that itâs resolved after re-running, then it sounds like a timing problem. I would suggest you identify the flaky Test Objects and put waiting statements (waitForElementPresent, etc) where appropriate. Additionally, you can try to enable Smart Wait in Project > Settings > Execution. It increases the execution time but perhaps it can make up by increasing stability.
I am not sure what âscript canât call the test caseâ means. Please provide the console log and relevant information so we can make a suggestion.
We are facing issues when there is a parallel execution of test suites on node locked licensed version of KRE.
We have a web application which is making a call to KRE. So when multiple users hit the application concurrently then we are facing issues.
We were able to see successful result for parallel execution on trial version.
But thatâs not the case once we had licensed version.
We are receiving random errors or, sometimes it works perfectly fine.
Please let us know if we can do any changes on our end to handle this.
An explanation of why trial version works fine for parallel execution and licensed version doesnât would help us to understand more.
Couple of random errors we received for parallel execution are-
Error: Offline activation failed. org.eclipse.core.internal.resources.ResourceException: Could not set the project description for âC%%Users%sa-eztools%Documents%KatalonWebDevTesting%WebDev Testing.prjâ because the project description file (.project) is out of sync with the file system.