Stopping Test Suite execution if Test Case fails


#1

Is there a way to stop Test Suite execution if one of its Test Cases fails?
I.e. in case of creating certain elements some crucial part is missing, and then there is no point of reproducing that issue again and again in every Test Case.


How to skip all remaining tests if initial login test fails?
Stop execution of all test cases in a test suite collection on any failure?
#2

Hi Alexander

You can use com.kms.katalon.core.util.Keywordutil for this purpose.
Please use either markErrorAndStop or markFailedAndStop methods.

You can find samples in below link:
http://www.sanspantalones.com/2018/02/16/marking-tests-as-passed-failed-or-in-error-using-markfailed-markpassed-markerror-in-katalon-studio/


#3

I think it stops the failed Test Case only, but the Test Suite keeps executing the rest of Test Cases.


#4

If your test cases are dependent on the successful execution of another one, IMHO the right approach is to add them into that test case after the assertion, see https://docs.katalon.com/display/KD/Call+Test+Case

With such design, if the assert fail and the handling is set to ‘Stop on Failure’, the execution will just stop at that point without executing further test-cases.


#5

Ibus said:

If your test cases are dependent on the successful execution of another one, IMHO the right approach is to add them into that test case after the assertion, see https://docs.katalon.com/display/KD/Call+Test+Case

With such design, if the assert fail and the handling is set to ‘Stop on Failure’, the execution will just stop at that point without executing further test-cases.

It might work, but it complicates the scripts structure, it makes that Test Case to work instead of Test Suite


#6

well … agreed, makes the structure more complicated.

other option may be … set a global variable based on the result of the first test and execute the others conditionally (sort of 'if $passed then do_test else println(‘test skipped’)) but this is ugly too, will still call all other testcases but just with a different wokflow


#7

Ibus said:

If your test cases are dependent on the successful execution of another one, IMHO the right approach is to add them into that test case after the assertion, see https://docs.katalon.com/display/KD/Call+Test+Case

With such design, if the assert fail and the handling is set to ‘Stop on Failure’, the execution will just stop at that point without executing further test-cases.

Well I really don’t agree with this approach. It’s important to keep every test cases as simple and clean as possible. There should be a way to stop a whole Test Suite if a Test Case fails. My actual scenario : I need to login to my portal to start my test cases. If the login doesn’t succeed, the rest of the cases will fail for sure. And I need to have that login process in a case by itself (and not in a Setup/Teardown) because I would like to retry my login 3x times before declaring a FAIL (and if I use a Setup/Teardown, it seems like the retry count doesn’t apply).


#8

Ibus said:

well … agreed, makes the structure more complicated.

other option may be … set a global variable based on the result of the first test and execute the others conditionally (sort of 'if $passed then do_test else println(‘test skipped’)) but this is ugly too, will still call all other testcases but just with a different wokflow

That could help me for a while… do you have an example of code that catches the result of a test case ?


#9

that depends on the assert you use in the main test case.

But since that assert, when is failing, it will stop that test case, you can put immediately after something like ‘GlobalVariable.excuteOthers = true’.

If the failure handling is set to STOP_ON_FAILURE, when the assert in that main test case will fail, that line won’t be executed and the ‘excuteOthers’ global variable will remain untouched (therefore you have to set it with a default ‘false’)


#10

any update on this concern?


#11

Hi.

There is a important difference between CallTestCase and “TestSuite with several TestCases”. For example, I have to start after every TestCase @AfterTestCase test listener script. This script inserts TestCase results data into DB through WebService. When I used CallTestCase method, only parent TestCase call @AfterTestCase, but TestCases called by CallTestCase do not call this script. So, I must use TestSuit. But in this case I need to skip other TestCases after one before has fail or error status.

Now, I understand I need to use Global Variable and manual scripting (or @BeforeTestCase???) to stop remaining TestCases execution. Unfortunaltely it’s not elegant solution. Therefore, that is not bad idea to enchance the KATALON with new functionality: stop whole TestSuit after some TestCase will finish with fail or error status.

Regards,

Alex


#12

Hi.

There is a important difference between CallTestCase and “TestSuite with several TestCases”. For example, I have to start after every TestCase @AfterTestCase test listener script. This script inserts TestCase results data into DB through WebService. When I used CallTestCase method, only parent TestCase call @AfterTestCase, but TestCases called by CallTestCase do not call this script. So, I must use TestSuit. But in this case I need to skip other TestCases after one before has fail or error status.

Now, I understand I need to use Global Variable and manual scripting (or @BeforeTestCase???) to stop remaining TestCases execution. Unfortunaltely it’s not elegant solution. Therefore, that is not bad idea to enchance the KATALON with new functionality: stop whole TestSuit after some TestCase will finish with fail or error status.

Regards,

Alex


#13

Even I was not able to find anything to fulfill this requirement. Alternatively You can use the call Test cases in such scenario and can club the depended test cases in one test case. Suppose you have to login and then perform some Test case. you can first create a test case where login will happen and then call the other depended test cases in that same Test case. Make sure you mark the login failure scenario as markfailureandstop method so that it will stop the rest of the execution.


#14

Hi All,

I came across all the discussions here. However, I have an example scenario wherein TestCase#1 creates a user and Testcase#2 logins to application with that user account. And TestSuite#1 contains Testcase#1 and TestCase#2.

Now when I execute TestSuite#1 and suppose Testcase#1 fails, is there a way to stop executing TestSuite#1 completely. Since it is meaningless to start executing TestCase#2 without user account created(Testcase#1 fails)

Any help is appreciated. Thanks in advance.

Regards,
Sridhar


#15

I tried with this listener inside my suit:

import com.kms.katalon.core.context.TestCaseContext
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testcase.TestCaseFactory as TestCaseFactory

    @TearDownTestCase(skipped = true) // Please change skipped to be false to activate this method.
    def tearDownTestCase(TestCaseContext testCaseContext) {
    	// Put your code here.
    	if(testCaseContext.getTestCaseStatus()=="FAILED"){
    		FailureHandling.STOP_ON_FAILURE
    	}
    }

But throws me the following error:
Cannot invoke method getTestCaseStatus() on null object

Anyone could solve this problem in another way?


#16

Hi @rodrigocalabretta

Methods annotated with @TearDownTestCase don’t accept (by design) a testCaseContext object, which is why it is null, and any attempt to call methods on a null object would result in the error you see. Object testCaseContext is available only in Test Listeners.


#17

I tried to run that from the @AfterTestCase listener and handle that failure, but the problem is that it didn’t stop the Suit run. Where should I set that failure handling?