How to continue Test Suites after fail in Jenkins?

Hello,
Let’s say that I have a test (Test1). And in this test I want to verify if something isn’t present.
If it’s not visible, throw new exception and mark test as failed.
If it’s visible - do nothing.

'Check if object isn\'t present.'
if (WebUI.verifyElementNotPresent(findTestObject('Table/NameOfObject', [('name') : 'AutoTestLB']), 0)) {
    'Throw an error if object isn\'t visible.'
    throw new com.kms.katalon.core.exception.StepErrorException('Object isn\'t visible')
}

So, in this case, if something won’t be visible, test suite will be marked as FAILED.

Now, in Jenkins, if I have more Test Suites, other tests after that above aren’t anymore executed.

cd /home/dev/Documents/Katalon/
./katalon -runMode=console -projectPath="Project.prj" -reportFolder="Reports" -reportFileName="report1" -retry=0 -testSuitePath="Test Suites/Test1" -browserType="Chrome"
./katalon -runMode=console -projectPath="Project.prj" -reportFolder="Reports" -reportFileName="report2" -retry=0 -testSuitePath="Test Suites/Test2" -browserType="Chrome"
./katalon -runMode=console -projectPath="Project.prj" -reportFolder="Reports" -reportFileName="report3" -retry=0 -testSuitePath="Test Suites/Test3" -browserType="Chrome"

What I want to do is to “Start test, if TestSuite is failed, mark that and continue tests in another TestSuite”.

What should be changed?

This topic is moved to New Feature/Suggestion box for further discussion.

Mariusz said:

Hello,
Let’s say that I have a test (Test1). And in this test I want to verify if something isn’t present.
If it’s not visible, throw new exception and mark test as failed.
If it’s visible - do nothing.

'Check if object isn\'t present.'

if (WebUI.verifyElementNotPresent(findTestObject(‘Table/NameOfObject’, [(‘name’) : ‘AutoTestLB’]), 0)) {
‘Throw an error if object isn't visible.’
throw new com.kms.katalon.core.exception.StepErrorException(‘Object isn't visible’)
}


So, in this case, if something won't be visible, test suite will be marked as FAILED.  
  
Now, in Jenkins, if I have more Test Suites, other tests after that above aren't anymore executed.

cd /home/dev/Documents/Katalon/
./katalon -runMode=console -projectPath=“Project.prj” -reportFolder=“Reports” -reportFileName=“report1” -retry=0 -testSuitePath=“Test Suites/Test1” -browserType=“Chrome”
./katalon -runMode=console -projectPath=“Project.prj” -reportFolder=“Reports” -reportFileName=“report2” -retry=0 -testSuitePath=“Test Suites/Test2” -browserType=“Chrome”
./katalon -runMode=console -projectPath=“Project.prj” -reportFolder=“Reports” -reportFileName=“report3” -retry=0 -testSuitePath=“Test Suites/Test3” -browserType=“Chrome”


  
What I want to do is to "Start test, if TestSuite is failed, mark that and continue tests in another TestSuite".  
  
What should be changed?  

  

A Test Suite should be run completely (all test cases within it). I believe this is a normal behavior and it should not be changed accordingly if one of the executed test cases are failed.

I am not entirely convinced of this.
So let us assume one more thing. Three successful tests should give us 100% efficacy (~33% for each test). If one test is not performed, we should get a score of ~66% (assuming that the other tests are successful), and in this case we really have 0%, because Jenkins (or Katalon?) did not continue further testing.
It seems that some extra option would be useful to continue all the TCs, independently, until the end. However, I am not sure on which side this should be implemented (Katalon or some Jenkins modification).
Or maybe instead of doing it this way, I should switch to Test Suite Collection?

Finally, I found a solution.

$KATALON -reportFileName="API" -retry=0 -testSuiteCollectionPath="Test Suites/API/API" -browserType="Chrome" || error=true
$KATALON -reportFileName="Dashboard" -retry=0 -testSuitePath="Test Suites/Dashboard" -browserType="Chrome" || error=true

Fail the build if there was an error
if [ $error ]
then 
    exit -1
fi

In this case, if one test fail, it will be marked, but all other test will be continued.
In last step, we will check if there were any errors. If yes, Jenkins will mark whole job as FAILED (or not).

Hi, what about this? I have a Test Suite with 5 test cases. Let’s say Test Case 1 fails, how can I have it proceed to Test Case 2 and so on?

This is what I am getting when a test case fails in a Test Suite, it was like it keeps on retrying the failed step on that test case even the retry is equal to 0:

Could you show the structure of your TS and TCs?

Hi @Mariusz,

Here is my sample Test Suite:

Here are the test cases:

Also, I set the default failure handling of the project to CONTINUE_ON_FAILURE.

Here’s the job:

Add missing conditions from my Jenkins code above and check.

Hi @Mariusz,

Sorry but is this the condition that you want me to add to my build?

$KATALON -reportFileName=“API” -retry=0 -testSuiteCollectionPath=“Test Suites/API/API” -browserType=“Chrome” || error=true
$KATALON -reportFileName=“Dashboard” -retry=0 -testSuitePath=“Test Suites/Dashboard” -browserType=“Chrome” || error=true

Fail the build if there was an error
if [ $error ]
then
exit -1
fi

May I ask for the exact condition that I should copy? Please bear with me as I’m just new to using Katalon and Jenkins :disappointed:

At the end of the command you need to set || error=true
And then

if [ $error ]
then
exit -1
fi

I added it:

-projectPath=“C:\Users\Admin\git\pimmdesktoptestscripts\PIMM Desktop CCM.prj” -retry=0 -testSuitePath=“Test Suites/Test_TS” -executionProfile=“default” -browserType=“Chrome (headless)” -apiKey=5beb57ba-6f7a-4897-bae6-970f4313cc14 --config -webui.autoUpdateDrivers=true || error=true

if [ $error ]
then
exit -1
fi

But still getting the same issue: