I have running the Katalon Docker Image in Gitlab-CI.
But there is a Problem that I could not solve.
If everything is going well and my tests are starting then it works like expected:
If a testsuite finishes with success the container exits with code 0
If a testsuite failed the container exits with code 1
But there are some special cases, like the browser could not start because of an incompatible webdriver:
2023-08-10 16:22:43.012 ERROR c.k.k.core.keyword.internal.KeywordMain - ❌ Unable to open browser with url: 'http://my-url/' (Root cause: com.kms.katalon.core.exception.StepFailedException: Unable to open browser with url: 'http://my-url/'
[...]
Caused by: org.openqa.selenium.SessionNotCreatedException: session not created: This version of ChromeDriver only supports Chrome version 110
Current browser version is 115.0.5790.170 with binary path /usr/bin/google-chrome
This is an example and I experienced other cases where the tests also couldn’t start because of an error.
Katalon always delivers exit code 0 in these cases.
And the gitlab-pipeline is marked as successful because of this.
This is a big problem and right now I can’t see if I can do anything to solve it. I think this comes out of the Katalon Engine and I am not a Katalon developer.
Am I right or is there a solution for this?
Would be just awesome!
It is not about Chrome and Webdrivers. I fixed this specific problem with the incompatible webdriver already. It was just an example to show you that there can be error cases where Katalon gives an exit code 0 (that stands for success)
And that was totally not a successful test run.
When we run hundreds of tests later on, we cannot check each individual test log to make sure that such a problem is not occuring again spontaneously. In such a case, Katalon must return an ‘exit code 1’, so that we can immediately see: There are issues.
But for this case, looks like there is an uncaught situation.
No test was started, so KRE wrongly reports 0 because no failure was detected (or other known situation)
Which is wrong.
Should report at least 4:
4: The execution cannot start because …
Kindly pass this info to the development team to investigate
@simon.klotz since you are using KRE, most likely you have a licence.
So you can rise also a support ticket using the HelpDesk
@bionel thanks for your further information.
This documentation is helpful, and yes this should be probably exit code 4.
I meanwhile remembered by myself that we’re actually paying for this and I reached out to our account manager and described the issue. It is up to them now to forward the issue to a developer team or whatever.
