I integrated to execute my tests using gitlab pipeline but found that in some cases, the exit codes retruned by KRE is not as per the actual problem (based on the info in user guide docs). Recently, I ran the scripts and it throws an error due to unavailability of runtime license as you can see here
Start activating online...
Failed to activate with Katalon Runtime Engine license. The number of the licenses cannot exceed the license quota. {"timestamp":"2023-09-12T10:14:48.536+0000","status":400,"error":"Bad Request","message":"The number of the licenses cannot exceed the license quota.","path":"/api/v1/license-keys/activate"}
Failed to activate with TestCloud license. null
Cannot execute this test suite. There is no Katalon Runtime Engine license available.
All launchers terminated
+ ret_code=3
+ exit 3
As you can see the exit code is 3 in this case but docs mentioning something different
KRE exit codes
Below is the list of exit codes of KRE:
0: The execution passed with no failed or error test case.
1: The execution has failed test cases.
2: The execution has error test cases.
3: The execution has failed test cases and error test cases.
4: The execution cannot start because of invalid arguments.
5: Invalid project.
6: Out of quota.
7: Unable to check quota.
For license unavailability, it should return exit code 6 rather than 3 which is for failed/error test cases. Now this is creating problem for me because based on these exit codes, I added some conditions in pipeline to show the job status or describe the exact preoblem why the tests are failing/not able to execute.
If this is the case, you should accept this behavior as given. You shouldn’t waste your time asking why. Nobody knows. Nobody in this forum can change KRE as you want it to be.
You should just try to find a solution to your problem.
You wrote that the exit code is not enough for you. OK, then, you need to aquire more information out of the stdout and stderr of a katalonc execution.
If I were you, I would write a bash shell script so that it redirect stdout and stderr from KRE into a temporary file, and parse the text in the file. The shell script would be able to find lines
Failed to activate with Katalon Runtime Engine license. The number of the licenses cannot exceed the license quota. {"timestamp":"2023-09-12T10:14:48.536+0000","status":400,"error":"Bad Request","message":"The number of the licenses cannot exceed the license quota.","path":"/api/v1/license-keys/activate"}
Failed to activate with TestCloud license. null
Cannot execute this test suite. There is no Katalon Runtime Engine license available.
In addition to the exit code from katalonc, the shell script can evaluate these information to plan what to do.
How to redirect stdout and stderr into file? See
If you want to write the stdout and the stderror into files as well was into the console at the same time, you want to use the tee command.
Nope!
KRE should properly return proper exit-codes, otherwise why the doc?
Just exit with 1 in case of any error and let the user figure-it out by parsing the console output? Is this your proposal?
See also this thread:
KRE exit codes are totally messed and this should be adressed by the development team.
So, with your mode powers, kindly move this into the bugs section.
The complain is valid for my eyes.
attn: @vu.tran@Elly_Tran
I am inclined to guess that Katalon Team will be reluctant to invest their human resources to improve the exit code of KRE, which is fundamental but not appealing for marketing. I guess they are too busy for developing great new features.
That shouldn’t be reason to not correct old/ignored/less used features. KRE integrated with some CI tool usually used by those who paid for this so each claimed feature of KRE/KSE should work properly.
Aside of that point, Katalon are delivering a product to customers with certain features and if one is not working properly, we, as end users, expecting to get resolved this one issue as well because we want to use it and also we paid for it.
I believe you are a paying customer of KRE. You are qualified to raise an official support ticket. You should raise it. I am afraid, a small discussion in the user forum may not be significant enough for the Katalon developers.
If Katalon development is reluctant to fix valid and obvious bugs, fine for me.
Will only encourage more users to stop using it for production, as i did few years ago.
To develop more and more workarounds was a waste of my time so i opted to simply devellop my own ‘framework’ based on more sane resources.
I am looking into this issues only as a hobby.
Thank you so much for great discussion. If you are paid user, please submit ticket here Katalon Help Center to gain better support. In the meanwhile, I will raise the ticket for this issue and back to you if any updates. Can you please provide your device and Katalon version? Thank you!
I don’t see any reason for this particular request, but we do understand you are not a techicall person.
Anyway, KRE is broken in handling exit codes, no matter on what platform is used.
Period!
Kindly provide this additional info to the development team exactly as it is.
But yeah, KRE is available only for licensed users… so why the comunity should bother anymore?
Sit down, @bionel!
Cheers!