How to pass exitCode from Katalon to Jenkins in order to inform of Test Case failures

I have made a demo project and pushed in GitHub:

-----

Problem to solve

I made a simple Katalon Studio project, which has a test suite named TS0. It invokes 2 test cases named TC1 and TC2. The TC1 always gets success while the TC2 always fails. Consequently the TS0 always finishes with 1 failure.

I wanted to execute the test suite in console. I read the documentation of Console Mode Execution. I made a Windows batch file named run_console_mode.bat. Its initial version was like this:

@echo off
echo KATALONSTUDIO_HOME="%KATALONSTUDIO_HOME%"
set DIR=%~dp0set SCRIPT_DIR=%DIR:~0,-1%
echo SCRIPT_DIR=%SCRIPT_DIR%
cd "%KATALONSTUDIO_HOME%"
.\katalon.exe -noSplash -runMode=console -summaryReport -projectPath="%SCRIPT_DIR%" -testSuitePath="Test Suites\TS0" -browserType="Firefox" -headless --config -proxyOption=MANUAL_CONFIG -proxy.server.type=HTTP -proxy.server.address="172.24.2.10" -proxy.server.port="8080"cd /d "%SCRIPT_DIR%"

(Please note that here is no code to deal with the exitCode from .\katalon.exe)

When I ran this bat file in the Windows command prompt TS0 was executed. As expected TC1 passed and TC2 failed.

Then I put the project into GitHub and configured my Continuous Integration server (Jenkins) so that it continuously build the project. When I tried for the first time, in the Jenkins dashboard I found a blue ball icon for the project. blue_ball The blue ball icon means that Jenkins sees the project ran passed.

Why PASSED? — I expected to see a red ball red_ball instead, because TC2 always fails.

Obviously Jenkins was not informed of the TC2 failure. I realized that I need to rewrite the run_console_mode.bat script.

Solution

Here is my revised run_console_mode.bat:

The point is that it receives the exitCode from .\katalon.exe and pass it to the caller process.

...
.\katalon.exe -noSplash -runMode=console ...
set exitCode=%ERRORLEVEL%
...
exit /B %exitCode%

The line: exit /B %exitCode% solved my problem. Jenkins is now informed of the TC2 failure and can deal with the job correctly.

6 Likes

nice +1

1 Like

In Jenkins, in the settings of the job, on Mac, I wrote a shell script as follows. This script tells Jenkins how to invoke my run_console_mode.sh. I needed to export Environment Variable KATALONSTUDIO_HOME with value of the path where Katalon Studio is installed.

If on Windows, I will run my run_console_mode.bat (windows bat file). And the way to set Environment Variable will be as follows:

set KATALONSTUDIO_HOME="C:\Katalon Studio-5.7.0"
.\run_console_mode.bat

jenkins_settings_shell_script.png

1 Like

kazurayam said:

In Jenkins, in the settings of the job, on Mac, I wrote a shell script as follows. This script tells Jenkins how to invoke my run_console_mode.sh. I needed to export Environment Variable KATALONSTUDIO_HOME with value of the path where Katalon Studio is installed.

You don’t have to use export, if the variable is needed only in the given shell step, just use KATALON_HOME=“blah/blah”

But that may not be available inside the script.

A better approach is to put in your script something like:

KATALON_HOME=$1

and call the script with parameter, e.g:

./run_console_mode.sh “path/to/app”

If you need a variable to be available for all steps, you can use the EnvironmentInjectotPlugin:

Ibus,

You are right. I take your point.

1 Like