I have made a demo project and pushed in GitHub:
-----
Problem to solve
I made a simple Katalon Studio project, which has a test suite named TS0
. It invokes 2 test cases named TC1
and TC2
. The TC1 always gets success while the TC2 always fails. Consequently the TS0 always finishes with 1 failure.
I wanted to execute the test suite in console. I read the documentation of Console Mode Execution. I made a Windows batch file named run_console_mode.bat
. Its initial version was like this:
@echo off
echo KATALONSTUDIO_HOME="%KATALONSTUDIO_HOME%"
set DIR=%~dp0set SCRIPT_DIR=%DIR:~0,-1%
echo SCRIPT_DIR=%SCRIPT_DIR%
cd "%KATALONSTUDIO_HOME%"
.\katalon.exe -noSplash -runMode=console -summaryReport -projectPath="%SCRIPT_DIR%" -testSuitePath="Test Suites\TS0" -browserType="Firefox" -headless --config -proxyOption=MANUAL_CONFIG -proxy.server.type=HTTP -proxy.server.address="172.24.2.10" -proxy.server.port="8080"cd /d "%SCRIPT_DIR%"
(Please note that here is no code to deal with the exitCode from .\katalon.exe
)
When I ran this bat file in the Windows command prompt TS0
was executed. As expected TC1
passed and TC2
failed.
Then I put the project into GitHub and configured my Continuous Integration server (Jenkins) so that it continuously build the project. When I tried for the first time, in the Jenkins dashboard I found a blue ball icon for the project. The blue ball icon means that Jenkins sees the project ran passed.
Why PASSED? — I expected to see a red ball instead, because TC2
always fails.
Obviously Jenkins was not informed of the TC2
failure. I realized that I need to rewrite the run_console_mode.bat
script.
Solution
Here is my revised run_console_mode.bat
:
The point is that it receives the exitCode from .\katalon.exe
and pass it to the caller process.
...
.\katalon.exe -noSplash -runMode=console ...
set exitCode=%ERRORLEVEL%
...
exit /B %exitCode%
The line: exit /B %exitCode%
solved my problem. Jenkins is now informed of the TC2
failure and can deal with the job correctly.