Pipeline Execution Does Not Complete After 3–4 Test Suites

Web Testing
1

We are currently facing an issue and would appreciate your assistance in identifying the cause of a failure in our automation schedule within the CI/CD pipeline.

We are executing a test suite collection, where one suite contains approximately 100–150 tests. During pipeline execution, after running 3–4 suites, the process appears to hang and continues running indefinitely without completing. In the pipeline reports, this is reflected as very slow or stalled progress.

Could you please help us understand:

  • Whether this is a known issue
  • If there is any limitation related to the number of tests within a suite
  • Or if there are recommended solutions or configurations to prevent this behavior

We appreciate your guidance on resolving this issue.

2

Dear Sophia,

Practical recommendation now:

  1. If possible, split the collection into smaller suites/batches as a mitigation.
  2. Check whether the affected run uses dynamic suite, True Parallel, or Standalone mode
  3. Review the stuck execution logs for signs like git clone failure, test suite not found, or queued/provisioning never transitions.
  4. Make sure you are on the latest TestCloud/TestOps release train available to your environment.

If you could kindly share the execution ID / schedule ID / environment mode (True Parallel vs Standalone) or one failed pipeline link. Our community can better assist you.

Bella

3

Have you ever successfully run the test suite collection on any other environment but on the CI/CD pipeline? Are you sure, on other environment, the test suite collection runs fine? Any evidence?

150 test cases in a test suite — this looks very big. I think it is too big to debug when any little trouble occurs.

A run of Test Suite will try to compile an execution report (HTML,PDF,CSV etc) that contains all the test cases’ results combined. The report will become big as the number of test cases grows. The report compilation process is pretty memory hungry, requires larger memory accordingly. I guess, the process got an OutOfMemory error, which will cause the JVM process to stop suddenly and uncontrollably.

I would recommend:

  1. Make sure that the Test Suite Collection ran in the sequential mode, not in the parallel mode.
  2. try setting to emit no execution report. Then, the test suite collection may run fine and faster.
  3. extend the JVM Heap memory for Katalon runtime.
  4. split the large test suites into smaller pieces; prefer smaller number of test cases per test suite.

How can you extend the JVM Heap memory? Where is the explanation how to in the official Katalon documentation? ------- I was surprised to find there is none! => @nghi.phan

See OutOfMemoryError in console mode for my old and poor description. You want to change -Xmx2048m in katalon.ini file to, for example, -Xmx4096m or -Xmx8g

4

I asked for internal help :grinning_face: and this is what I have been shown:

Katalon officially documents JVM launch arguments in Execution Settings, including -Xmx to raise the maximum heap size.

Execution settings in Katalon Studio | Katalon Docs

Execution settings help define the desired behaviors that apply to the project during and after test execution. From Katalon Studio version 10.4.0 onwards, execution settings are now scoped per platform (WebUI, Mobile, Windows), allowing you to configure timeouts and delays specific to each testing environment.

View test suite and test suite collection reports in Katalon Studio | Katalon Docs

→ is this what you are looking for?

5

I would like to wait @sophia.n for trying the info and report her result back.

6

kindly compare/share the error logs trace from local and pipeline execution. kindly add additional details to rule out issues one by one