How to pass exitCode from Katalon to Jenkins in order to inform of Test Case failures

I have made a demo project and pushed in GitHub:

-----

Problem to solve

I made a simple Katalon Studio project, which has a test suite named TS0. It invokes 2 test cases named TC1 and TC2. The TC1 always gets success while the TC2 always fails. Consequently the TS0 always finishes with 1 failure.

I wanted to execute the test suite in console. I read the documentation of Console Mode Execution. I made a Windows batch file named run_console_mode.bat. Its initial version was like this:

@echo off
echo KATALONSTUDIO_HOME="%KATALONSTUDIO_HOME%"
set DIR=%~dp0set SCRIPT_DIR=%DIR:~0,-1%
echo SCRIPT_DIR=%SCRIPT_DIR%
cd "%KATALONSTUDIO_HOME%"
.\katalon.exe -noSplash -runMode=console -summaryReport -projectPath="%SCRIPT_DIR%" -testSuitePath="Test Suites\TS0" -browserType="Firefox" -headless --config -proxyOption=MANUAL_CONFIG -proxy.server.type=HTTP -proxy.server.address="172.24.2.10" -proxy.server.port="8080"cd /d "%SCRIPT_DIR%"

(Please note that here is no code to deal with the exitCode from .\katalon.exe)

When I ran this bat file in the Windows command prompt TS0 was executed. As expected TC1 passed and TC2 failed.

Then I put the project into GitHub and configured my Continuous Integration server (Jenkins) so that it continuously build the project. When I tried for the first time, in the Jenkins dashboard I found a blue ball icon for the project. blue_ball The blue ball icon means that Jenkins sees the project ran passed.

Why PASSED? — I expected to see a red ball red_ball instead, because TC2 always fails.

Obviously Jenkins was not informed of the TC2 failure. I realized that I need to rewrite the run_console_mode.bat script.

Solution

Here is my revised run_console_mode.bat:

The point is that it receives the exitCode from .\katalon.exe and pass it to the caller process.

...
.\katalon.exe -noSplash -runMode=console ...
set exitCode=%ERRORLEVEL%
...
exit /B %exitCode%

The line: exit /B %exitCode% solved my problem. Jenkins is now informed of the TC2 failure and can deal with the job correctly.

6 Likes

nice +1

1 Like

In Jenkins, in the settings of the job, on Mac, I wrote a shell script as follows. This script tells Jenkins how to invoke my run_console_mode.sh. I needed to export Environment Variable KATALONSTUDIO_HOME with value of the path where Katalon Studio is installed.

If on Windows, I will run my run_console_mode.bat (windows bat file). And the way to set Environment Variable will be as follows:

set KATALONSTUDIO_HOME="C:\Katalon Studio-5.7.0"
.\run_console_mode.bat

jenkins_settings_shell_script.png

1 Like

kazurayam said:

In Jenkins, in the settings of the job, on Mac, I wrote a shell script as follows. This script tells Jenkins how to invoke my run_console_mode.sh. I needed to export Environment Variable KATALONSTUDIO_HOME with value of the path where Katalon Studio is installed.

You don’t have to use export, if the variable is needed only in the given shell step, just use KATALON_HOME=“blah/blah”

But that may not be available inside the script.

A better approach is to put in your script something like:

KATALON_HOME=$1

and call the script with parameter, e.g:

./run_console_mode.sh “path/to/app”

If you need a variable to be available for all steps, you can use the EnvironmentInjectotPlugin:

Ibus,

You are right. I take your point.

1 Like

@kazurayam do jenkins wont generate reports if the build is failed with exit code 1? Is there any way to generate the reports in jenkins even if build is failed.

Please advise. I am referring to HTML or pdf reports

What do you mean by saying “reports” here?

It is basically run results like we get after running test suite in katalon which tells which all the test cases are passed or failed.

You mean the report prodcued by the Basic Report plugin of Katalon Studio?
https://store.katalon.com/product/59/Basic-Report

Jenkins knows nothing about it. Jenkins has nothing to do with the Katalon’s Basic Report.

So I would reword your question as such:

Well, possibly there is something wrong in your test project. You need to fix it before getting the Basic Report.

What’s wrong? — I do not know, as you have provided no information. If you want somebody to help you, please have a look at the following post:

Your question would be “off topic” from the point of view of this old thread.
If you want to ask question, please make a new post and provide information (log, HTML, page screenshot, test case script etc)

I will close this topic.