- Using Katalon Runtime Engine 8.6.8
- In Azure DevOps release pipeline, locally hosted.
- Have a valid KRE license.
- Recently stopped renewal of TestCloud license.
The test cases are successfully executed: there’s no mentioning of any test failure in the (attached) logs. The only mention of a ‘fail’ is the inability to license TestCloud (obviously):
Activated successfully with Katalon Runtime Engine license.
Failed to activate with TestCloud license.
What I have tried:
- Checked ‘Build CMD’ options, added ‘-orgID=123…’
- Created a new apiKey, deleted all unused.
- Updated/triggered updating of ‘analytics.authentication.token’ in ‘com.kms.katalon.integration.analytics.properties’ by opening the project settings in Katalon Studio under Katalon admin account.
- Explicitly added ‘analytics.testcloud.integration.enable’ and ‘analytics.testreport.autoupload.enable’ settings to ‘com.kms.katalon.integration.analytics.properties’.
- Noteworthy, even though disabled, the properties file set above settings to ‘true’ when triggering the update…
- Verified tests run ok when executed from Katalon Studio against same environment.
Where does that ‘Exit code: 2’ come from? How to fix this? Might it be the failing TestCloud license check?
com.kms.katalon.integration.analytics.properties (760 Bytes)
error.log (1.1 KB)
kre_fail_2.txt (10.1 KB)
1 Like
Hi @coen.schalkwijk,
Where does that ‘Exit code: 2’ come from? How to fix this?
According to this document: https://docs.katalon.com/docs/execute/katalon-runtime-engine/get-started-with-katalon-runtime-engine#kre-exit-codes. Error #2 means your tests contain at least one error. You should check your detailed test report for further troubleshooting.
Might it be the failing TestCloud license check?
No, Test Cloud license checking doesn’t affect this error. Test Cloud license only affects your pipeline if you wants to run with Test Cloud devices. Since you run with -browserType=Web Service, the errors must come from others
2 Likes
@duyluong Thank you for your reply.
I’m familiar with the exit codes. I’ve checked the logs/test results. I cannot find any failing tests. There’s also no errors reported. From the CLI I read:
2023-11-14T16:10:19.5224487Z --------------------------------------------------------------------------------
2023-11-14T16:10:19.5224901Z Test Suites/Test Suite Collection - 20231114_164834…68/68(100%)
2023-11-14T16:10:19.5225309Z --------------------------------------------------------------------------------
I will check again, just in case.
I’ve just tried to run my test cases with KRE 9.0.0 and then I get this exception. Not sure if related to the issue, but at least different compared to the 8.6.8 results:
Search for valid offline licenses in folder: C:\Users\pro.katalon\license
The number of valid offline licenses: 0
Start activating online…
Tue Nov 14 16:48:22 CET 2023
java.lang.NumberFormatException: Cannot parse null string
at java.base/java.lang.Long.parseLong(Long.java:674)
at java.base/java.lang.Long.valueOf(Long.java:1163)
at com.kms.katalon.integration.analytics.providers.TestOpsFeatureActivatorImpl.getRenewInterval(TestOpsFeatureActivatorImpl.java:39)
at com.kms.katalon.integration.analytics.providers.TestOpsFeatureActivatorImpl.activate(TestOpsFeatureActivatorImpl.java:58)
at com.kms.katalon.application.utils.ActivationInfoCollector.getLicenseFromTestOps(ActivationInfoCollector.java:625)
at com.kms.katalon.application.utils.ActivationInfoCollector.activate(ActivationInfoCollector.java:441)
at com.kms.katalon.application.utils.ActivationInfoCollector.checkAndMarkActivatedForConsoleMode(ActivationInfoCollector.java:261)
at com.kms.katalon.console.handlers.ActivationHandler.activateWithConsoleArgOptions(ActivationHandler.java:176)
at com.kms.katalon.console.handlers.ActivationHandler.activationWithTestOps(ActivationHandler.java:226)
at com.kms.katalon.console.handlers.ActivationHandler.activate(ActivationHandler.java:275)
at com.kms.katalon.console.application.ConsoleMain.launch(ConsoleMain.java:117)
at com.kms.katalon.console.application.ConsoleApplicationStarter.start(ConsoleApplicationStarter.java:36)
at com.kms.katalon.application.Application.runConsole(Application.java:129)
at com.kms.katalon.application.Application.start(Application.java:93)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:402)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596)
at org.eclipse.equinox.launcher.Main.run(Main.java:1467)
Activated successfully with Katalon Runtime Engine license.
Failed to activate with TestCloud license.
There’s more exceptions in the KRE log with v9.0.0. See attached file.
kre_fail_3.txt (19.5 KB)
Hi @coen.schalkwijk,
Can you share with me some test suite HTML reports, such as: D:\agent[…]Reports\20231114_164834\API Test Suites\Setup\20231114_164835
2 Likes
@duyluong I was looking at the same. Turns out that path is incorrect and that in the correct path there is a test report containing a test that results in an error. Huzzah, that’s nice, now I’ve got something to solve 
<testsuites name=“Test Suite Collection” time=“939.957” tests=“68” failures=“0” errors=“10”>…
The nature of this failing test accounts for a successful running local test and a failing one in the pipeline.
So the question now becomes why this test is listed as success in the CLI output (concerns ‘WS Test Suites’) or am I misinterpreting the output?
2023-11-14T16:10:19.5224487Z --------------------------------------------------------------------------------
2023-11-14T16:10:19.5224901Z Test Suites/Test Suite Collection - 20231114_164834…68/68(100%)
2023-11-14T16:10:19.5225309Z --------------------------------------------------------------------------------
2023-11-14T16:10:19.5226772Z Test Suites/API Test Suites/Settings - - 20231114_164853…3/3(100%)
2023-11-14T16:10:19.5227177Z Test Suites/API Test Suites/EyeTrackingLiveview - - 20231114_164855…3/3(100%)
2023-11-14T16:10:19.5227562Z Test Suites/API Test Suites/StageFile - - 20231114_164857…1/1(100%)
2023-11-14T16:10:19.5227963Z Test Suites/WS Test Suites/StagePlayer - - 20231114_164859…10/10(100%)
2023-11-14T16:10:19.5228344Z Test Suites/API Test Suites/Teardown - - 20231114_164901…2/2(100%)
2023-11-14T16:10:19.5228740Z --------------------------------------------------------------------------------
1 Like
Post Mortem: hints I missed:
- I have 68 test cases, correctly counted in the CLI.
Test reporting mentioned 58 test cases successfully executed. So I was missing results from 10, which were failing.
- Could have checked reports on disk, instead of solely relying on CICD reporting.