When running a test suite via KRE command line from a jenkins pipeline, and checking console output, we get detailed run logs which is perfect, example:
Will print execution status every 15 second(s)
---------------------------------------------------------------------------------
Test Suites/My test suite - Chrome - 20260210_151655..
.........................................................................0/3(0%)
---------------------------------------------------------------------------------
2026-02-10 15:17:04.162 INFO c.k.k.c.l.logback.LogbackConfigurator - Logback default configuration initialized from: /opt/katalonstudio/configuration/org.eclipse.osgi/102/0/.cp/resources/logback/logback-execution.xml
2026-02-10 15:17:04.164 INFO c.k.k.c.l.logback.LogbackConfigurator - Logback custom configuration initialized from: /tmp/project/Include/config/log.properties
2026-02-10 15:17:04.521 INFO c.k.katalon.core.main.TestSuiteExecutor - START Test Suites/My test suite
etc....
2026-02-10 15:17:09.703 DEBUG testcase.My test case - 1: enhancedClick(findTestObject("App/input - Email"))
Whereas when running a test suite collection, we don’t have detailed logs any more, and only execution status, example:
Will print execution status every 15 second(s)
---------------------------------------------------------------------------------
My test suite collection.....................................0/6(0%)
---------------------------------------------------------------------------------
My test suite A - Chrome - 20260210_154617..
.........................................................................0/3(0%)
My test suite B - Chrome - 20260210_154618..
.........................................................................0/3(0%)
---------------------------------------------------------------------------------
Test suite My test suite A - Chrome - 20260210_154618 is ready to start at 15:46:30.421173572
Running: My test suite A - Chrome - 20260210_154618
---------------------------------------------------------------------------------
My test suite collection....................................3/6(50%)
---------------------------------------------------------------------------------
My test suite A - Chrome - 20260210_154617..
.......................................................................3/3(100%)
My test suite B - Chrome - 20260210_154618..
.........................................................................0/3(0%)
---------------------------------------------------------------------------------
I tried to add the following arugment to the docker run katalon command line with no effect: -runMode=console -consoleLog -logLevel=DEBUG
→ Is there a way to keep detailed execution logs in addition to execution status for test suite collection?
KRE doesn’t currently stream the per-test-suite execution log with Test Suite Collection. You’ll mainly see the progress/status lines like you pasted. Your -runMode=console -logLevel=DEBUG won’t change that behavior for TSC runs.
Test suite collections don’t output detailed execution logs to the console in CI/CD environments like Jenkins, even though individual test suites do. The arguments you tried (-consoleLog,-logLevel=DEBUG) don’t work because:
-consoleLog
is Windows-only: According to the official KRE documentation, this argument explicitly states: “Display log in the console. Only use this option when running Katalon Studio in Windows Command Prompt. Do not use this option in other OSes or CI tools, for example Jenkins.”
-logLevel=DEBUG
is not a valid KRE argument: This parameter doesn’t exist in the KRE command-line syntax. The documentation lists all supported arguments, and log level control isn’t available as a command-line parameter for KRE.
Design limitation: Test suite collections have different logging behavior by design—they’re optimized for batch execution and report generation rather than console output streaming.
I would recommend those who has the KRE licenses, including @yann.sautreuil, to stop using Test Scuite Collection.
In my humble opinion, “Test Suite Collection” is a feature designed solely for Katalon Studio GUI, not for KRE. Imagine a GUI user has mutilple Test Suites and he wants to run all of them by a single trigger, but he doesn’t like to repeat clicking the green arrow button to run the Test Suites one by one. Then, the only option for him is to bind the Test Suites to form a TSC.
Remember, the Katalon Studio GUI user will rarely look at the Console; they prefer the “Log Viewer”. Therefore TSC is designed to make very little output into the Console.
Provided with KRE license, you can create a TSC-equivalent process by various methods; a simple shell script, Jenkins pipeline, or a GitHub Workflow.
Moreover, those external methods (shell, Jenkins pipeline) offers better controllability than TSC. You can see all messages logged in the console. Jenkins Pipeline enables you to introduce the Conditional Constructs with which you get the flexibility to change the flow of execution based on dynamic criteria, such as environment variables, results from previous steps, etc. If you allocate enough server resources for the pipeline, you would be able to execute multiple Test Suites really in parallel to let them run far faster than a sequential execution on a single box. Katalon’s TSC offers no such power. You should make the most use of the pipelines.
Thank you all for your responses. I was afraid it would be negative and by design not possible.
Agree with you @kazurayam in general I orchestrate myself via pipelines and do not rely on test suite collection, but here I need to come back to it because I have so many individual test suites that it makes pipelines script not maintainable / huge and also it takes 1-2min per test suite to bootup the docker / katalon environment even if it’s for a few scenario per test suite, extending the overall execution time. By using test suite collection we suffer the bootup once and can chain as many test suite as we like, in this use case a lot, without additional delay.
I will continue with test suite collection approach here, and will have to wait for result upload to testops for easy access to details. Current testops is very slow but the new one is supposed to be fast, let’s put it to the test.
I understand that you are suffering from the long processing durations, the burden of code maintenance. I think that you would need to refactor your codebase to decrease the number of test suites.
To achieve that, I think you should refactor a Test Suite into a single “controller” Test Case that calls other “worker” Test Cases using WebUI.callTestCase() keyword. That will run far faster than a Test Suite that comprises with worker Test Cases.
Too much works … yes, probably. Then all you could do would be to stay with your system as is.
No annoucement, discussion with support, where I discovered that there is a new version of testops and that it could solve some issues we have with current one including some perf issues.
