Scripts are working fine individually but not in batches in katalon
1 Like
When you say “in batches”, are you talking about having multiple Test Cases in a Test Suite, or having multiple Test Suites in a Test Collection?
If you are talking Test Cases in a Test Suite, then I would say not to go beyond having 5 Test Cases within a Test Suite. What I do is try to find a position that I can Save my work before I need 6 TC and then restart another Test Suite and carry on where I was last at.
Edit: What would happen is somewhere within my 6th Test Case the writing to log would just stop. I assumed it was something to do with the JVM, and I increased the numbers for it, but I use the Log to review my TCs and find all faults that show, so it is just easier to not write past a safety, which I (arbitrarily) put at 5 TCs
Edit: Maybe the below feature which can activate the Java Garbage Collector, to clean up the Heap, can help.
What !?!? 
Man, that’s crazy. And rather than exporting your poor experience running suites to other users and assume they have the same unresolved issues, I think it would be better to get @himanisaxena19 to explain better what they mean by “batches” and give us the failure/error messages, so we can figure out what’s wrong.
For reference: I run KS in Window10 VMs with 4GB ram and 4 virtual processors. Suite size: 117 test cases. Time to execute: ~2.5 hours.
I am trying to execute multiple test suites in collection.
Execution mode: Parallel (Max concurrent instances 5)
Delay between instances: 15 seconds
Approx 40 test suites in 1 collection
Running this collection in Windows 10 with 16GB installed RAM
Don’t use Parallel mode of Test Suite Collection of Katalon Studio. You should execute it in Sequential mode only. Parallel mode is a gamble which you are highly likely to loose.
Parallel mode exposes the fragility of your test scripts. The fragilities of your test scripts tend to stay halmless in the sequential mode. But once executed in the parallel mode, the fragilities get chance to harm you.
If you want run your test suites successfully in the Parallel mode, you need to make your test scripts much more robust (less fragile) against any resource contentions.
What is the resource contention I mean? For example, 5 of your Test Suites may try to open 5 windows of web browsers simultaneously. Then the browsers will surely reponds very slowly. The browser software is NOT designed to process 5 requests in a real parallelism. You should know that it is a heavy CPU-intensive processing to open a browser window and let it navigate to a URL. The first browser will respond in 10 seconds, the second window will respond in 20 seconds, the third window in 30 seconds, the fourth and the fifth window will be slower than the MAX WAIT of 30 seconds that Katalon Studio is configured. So, in the Parallel mode, the 4th and 5th Test Suite may fail to mange the slow response of browsers. On the other hand, in the Sequential mode, the browser will open in 10 seconds everytime so that all your Test Suites will have less troubles.
It would require a lot of work for you to develop your test scripts more robust. If you are not willing to work for the robustness, then you should step back and run your test in Sequential mode only.
Don’t ask us, guys in this forum, why you got failures in the Parallel mode. It is only YOU who can debug your fragile tests. You should read the error logs very carefully, check how web pages in browsers perform while tests are going on, review your test scripts etc. Nobody other than you can do it.
Why do you want to run your Test Suites in parallel? I guess, you are expecting that your Collection of 5 Tests Suites in Parallel mode will run 5 times quicker than in Sequential mode. — Well, that is not the case. You should examine the speed gain of parallelism. I would tell you, the Parallel mode on a single machine is no faster than the Sequential mode, because you have only limited CPU/memory resources on a box. The parallelism will benefit you nothing on a single box.
If you want your tests to run faster, then you need to distribute your Katalon tests into multiple machines, say 5 servers. The only distribution feature that Katalon offers is the following one:
You will be asked to purchace sufficient number of Katalon Runtime Engine licenses (5?) to utilize this feature.
40? 
Red flag. 
- Parallel testing should work since Katalon says it should work. If not, post a bug report.
- I don’t do parallel suite execution. I never had much luck with it and left it behind a long time ago.
- I sometimes execute suites/test cases in parallel but I start them by hand in KS. That works for me.