Katalon console Failing TestRun when tests passes on retry

Bug Report
,
1

i created a batch file to run the Katalon in console mode for jenkins using the below

katalon -runMode=console -projectPath=%projectpath% -testSuiteCollectionPath=%path% -retry=1 -retryFailedTestCases=true -browserType=%browser% -email=%email% -password=%password%

I ran the testsuite collections using console mode with -retry=1 option. 2 Tests failed on the first run and passed on the second run. Even if the tests passed on the secondrun, console mode is failing the whole test run in Jenkins.

2

Can you help to post me your Jenkins log?

3

this issue is only when we run testSuitecollection

i attchig the part of log,

--------------------------------------------------------------------------------

Test Suites/12.1/Suite Collections/TSL_retrys - 20180313_113017…0/2(0%)

--------------------------------------------------------------------------------

Test Suites/12.1/Suite Collections/TestLoginSuite - Chrome - 20180313_113018…

…0/2(0%)

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

Test Suites/12.1/Suite Collections/TSL_retrys - 20180313_113017…1/2(50%)

--------------------------------------------------------------------------------

Test Suites/12.1/Suite Collections/TestLoginSuite - Chrome - 20180313_113018…

…1/2(50%)

--------------------------------------------------------------------------------

Uploading report to qTest…

Report has been sent to qTest

Re-run test suite: Test Suites/12.1/Suite Collections/TestLoginSuite # 1.

--------------------------------------------------------------------------------

Test Suites/12.1/Suite Collections/TSL_retrys - 20180313_113017…2/3(67%)

--------------------------------------------------------------------------------

Test Suites/12.1/Suite Collections/TestLoginSuite - Chrome - 20180313_113018…

…2/2(100%)

Test Suites/12.1/Suite Collections/TestLoginSuite - Chrome - 20180313_113039 - R

e-run 1…0/1(0%)

--------------------------------------------------------------------------------

Uploading report to qTest…

Report has been sent to qTest

--------------------------------------------------------------------------------

Test Suites/12.1/Suite Collections/TSL_retrys - 20180313_113017…3/3(100%)

--------------------------------------------------------------------------------

Test Suites/12.1/Suite Collections/TestLoginSuite - Chrome - 20180313_113018…

…2/2(100%)

Test Suites/12.1/Suite Collections/TestLoginSuite - Chrome - 20180313_113039 - R

e-run 1…1/1(100%)

--------------------------------------------------------------------------------

Build step ‘Execute Windows batch command’ marked build as failure

Finished: FAILURE

4

when i tried to attach log it says too long, then i shorten it and now it says file not allowed

5

Hi Srujan,

Thanks for your information. I will take a look at this and reply to you soon.

Regards

6

same issue herer, and I have report the same issue in your git hub: https://github.com/katalon-studio/docker-images/issues/14

7

@Vinh Nguyen We were wondering that any update for this issue?

8

@Srujan Varma Patchamatla We were wondering that any kinds of walk-around for this issue?

We tried this, if the docker run success, then chrome_success=true. If the docker run fails, the chrome_success is empty.
But, if retry triggered, no matter the the docker run is success or not, the chrome_success is still empty.

set +xe  
  
docker run --rm --name ${BUILD\_TAG} -v "$project\_dir":/katalon/katalon/source:ro -v "$reports\_dir/chrome":/katalon/katalon/project/Reports/xxx -e KATALON\_OPTS="$katalon_opts" VM20:5000/katalon:latest **&& chrome_success=true  
**  

if \[\[ $chrome_success != true \]\]; then 

    exit -1

fi
9

any update, guys?

10

This is now related to Jenkins itself, exit code from Jenkins is not the same when you do it normally using console mode execution , so I can think of adjusting exit code for Jenkins execution in this case

11

@Vinh Nguyen

Thank you for your suggestions.

However, we found that, if retry happens, no matter the final test is success or fail, the exit code is 1. That is the reason why the jenkins job failed.

Here is our suggestion to fix the bug.

1. If retry happens, and final test is success. Set the exit code to 0.
2. If retry happens, and final test is fail. Set the exit code to 1.

For more info about exit code, you could refer here: http://docs.katalon.com/pages/viewpage.action?pageId=786527#CI/CDIntegrations-ExitCodes

Thank you,
Ken

12

We were wondering that any script could update the exit code to walk-around the bug?

Here is my idea:
1. New a Test Listener with “Generate sample After Test Case method” -> 2. if test case success, use groovy script for force the exit code to 0

And this maybe work.

@Vinh Nguyen: I need you. Any suggestions from your side are high valued. Thank you in advance.

Thank you,
Ken

13

Here is the walk-around: https://github.com/katalon-studio/docker-images/issues/14#issuecomment-399404943

@Vinh Nguyen Please take time to fix the CRITICAL bug

14

any update about the bug?

15

any update about the bug?

16

any update about the bug?

17

Hi ken,

We will consider fixing this issue in later releases. Thanks for your information.

18

Hi,
is there any update on this issue?
IMO this makes the retry-Option pretty much useless on Jenkins at the moment.

Thanks,
Dennis

19

Hi,

Is there any update on this issue?

Thanks

20

I am facing this issue as well. Any plan to fix soon?