I know that we can run test suites concurrently, but is there a way to run the tests within a test suite simultaneously?
Not directly with a Katalon API.
However, I think @kazurayam has done some work in this area. Sit tight, he’ll be around in a couple of hours or so.
Here is what I have done.
I tried to run Groovy Closures in multiple Java Threads (2 threds, 4 threads, 8 threads, …) in a Test Case script. It worked for me.
However I do not recommend you to follow this way. Because any problems in this system are difficult to analyse and fix. I haven’t developed any easy-to-use debugging facility for “multi-threaded Test Cases” (I am not capable of developing such magical thing). I only have the Stack Trace messages emitted by Exceptions. I am OK with it, but you would be not. I do not think I can support you remotely.
I hoped that executing multiple Groovy Closures in parallel will make my Test Case run far faster than running them sequentially. 2 times faster, 4 times faster, 8 times faster.
But our life is not as easy as that.
My research revealed that multi-threaded Test Case did not run faster than the single-threaded process. Even slower sometimes. There were several reasons; ChromeDriver somehow takes longer time to launch browser; running multiple browsers in parallel requires faster network connection to the Internet but my WiFi is too slow; the CPU of my Mac Book Air got too busy; tests failed due to delays caused by various occasional reasons. Still I do not think that multi-threading caused slowness. I learned a simple & fundamental lesson: I need more machine + network resources to process more tasks in a limited time frame.
Plus I need to write my tests very robust. Otherwise, a multi-threaded test becomes a chaos.
in order to make your test run faster, it would be easier and safer for you to allocate more machine resources . You should distribute the jobs to multiple machines. This is where Cloud computing environment (AWS, GCP) shines. If you want your test run 10 times faster, then you should distribute the job onto 10 instances of AWS lambda with 10 licenses of KRE, perhaps.
Do you have a very large test project which takes hours and days to run? Do you want it run faster?
Then split your test into multiple test suites, and run the test suite in parallel on multiple distributed machines on a cloud service using multiple Katalon Runtime Engine. That would be most practical approach, I think.
Are you experiencing you tests run very slow? Do you want to make your tests run faster? Then have a look at the following topic of mine:
Thank you for the detailed explanation. Could you please let me know a little bit more about this option. I have already divided the test cases into multiple suites, how can I run them on multiple machines using Katalon runtime engine? I have never done this before, how can this be done?
I have never done it in fact.
I mentioned this as a theory.
If you are a payed user, you should be able to request technical support by Katalon. They would welcome you as you would need to purchase more KRE licenses
Ah ok, understood
For example, have a look at this
The Jenkins controller is the orignal node in the Jenkins installation. The Jenkins controller administers the Jenkins agents and orchestrates their work, including scheduling jobs on agents and monitoring agents. Agents may be connected to the Jenkins controller using either local or cloud computers.
You want to configure a set of Jenkins jobs which execute Katalon Studio Runtime with your project specifying which Test Suites to run.
Once you figure out with Katalon licensing (each process require a katalon license) … the only thing you have to do is to learn about your orchestrator.
e.g. for jenkins, instead of using freestyle project, you can use a matrix approach: