What is the maximum number of test suites, I can add to a Test suite collection in Cmd line execution and in docker execution

What is the maximum number of test suites, I can add to a Test suite collection in Cmd line execution and in docker execution

Katalon version in use : 8.6.0

Things I tried,

  1. put 175 Test suites in one Testsuite collection. (in Customized docker run)
    Always(or Atleast 5/175) suffered errors like
    Some Testsuites (pass) failed to generate JUnit report .
    Some Testsuites (fail) failed to go to automatic re-run.
    Almost all testsuites took longer time to generate reports (PDF & JUnit) despite plenty resources available.

  2. put 87 Test suites in one Testsuite collection and rest 88 Test suites in another Testsuite collection (in Customized docker run)
    Always(or Atleast 2-4/175) suffered errors like
    Some Testsuites (pass) failed to generate JUnit report .
    Some Testsuites (fail) failed to go to automatic re-run.
    Almost all testsuites took longer time to generate reports (PDF & JUnit) despite plenty resources available.

  3. made 4 Testsuites collections with (45+45+45+40) (in Customized docker run)
    Rarely (or 2/175) suffered errors like
    Some Testsuites (pass) failed to generate JUnit report .
    some testsuites took longer time to generate reports (PDF & JUnit) despite plenty resources available.

Its a mind boggling thing to make assumptions on what should we use as we do not have any standard document of what running inside katalon (like maximum threads katalon can generate/handle or maximum number of testsuites katalon can handle internally in a test suite collection.

Request your assistance on this.

Regards
A katalon learner

These issues concern about the internal of Katalon Studio. I think, only the development team of Katalon can look into.

This is a user forum where nobody knows the internal of Katalon Studio.

Therefore, if you are a paying licencee, you should raise an official techinal support request to Katalon.

Personally I have no experience with mass-scale Test Suite Collection like you “put 175 Test suites in one Testsuite collection”. I have never tried it because I know such massive Test Suite Collection does not work stable enough. haven’t researched why. I just gave up.

You are a learner yet. If so, let me ask you, why don’t you create 175 Test Cases in one Test Suite Collection, rather than 175 Test Suites in one Test Suite?

I believe that 175 Test Cases in one Test Suite would run far stable than 175 Test Suites in one Test Suite Colleciton.

I think, simply put, 175 Test Suites in one Test Suite Collection — it is a strategic mistake. You should rewrite your project to be 175 Test Cases in one Test Suite.

Running a Test Suite in Katalon Studio always forks an OS process per each. If you run 8 Test Suites, then 8 OS processes will be forked. Obviously multiple OS processes consumes more resourcess, and error-prone due to various resource conflicts at OS-level. Katalon Studio will not be able to control them. It is very difficult to find the reason why any error occur.

Running 8 Test Cases bundled in a Test Suite will fork only one OS process where 8 test cases will be executed sequentially one after another.

Running Test scenarios at Test case level doesn’t give me a JUnit_Report.xml and PDF of individual scenario (It just runs the tests and produce one JUnit report of test suite - only … no pdfs for individual runs or collective pdf of all runs) which I need for test run record keeping.

Also running testcases in one test suite is not giving me advantage/option of concurrencies… all testcases will run sequential, which will sky-rocket my execution time. (An impossible option for project)

As you put it right, putting 175 Testsuites in one collection is a strategic mistake but was just trying to use the advantage of concurrencies in here. Alas, it isn’t doing what we normally would believe katalon to do with stability.

You are right.

You are right.

In this forum some wrote that their test suites run very long: for 10 hours, 20 hours or even days. :wink:

Concurrencies in Katalon Studio cost very high. It certainly involves a lot of techinical difficulties. I do not think Katalon can provide the beautiful concurrency out of box.

You are right.

Life is not as easy as you (we) expect.