Stopping Test Suite execution if Test Case fails

Is there a way to stop Test Suite execution if one of its Test Cases fails?
I.e. in case of creating certain elements some crucial part is missing, and then there is no point of reproducing that issue again and again in every Test Case.

2 Likes

Hi Alexander

You can use com.kms.katalon.core.util.Keywordutil for this purpose.
Please use either markErrorAndStop or markFailedAndStop methods.

You can find samples in below link:
http://www.sanspantalones.com/2018/02/16/marking-tests-as-passed-failed-or-in-error-using-markfailed-markpassed-markerror-in-katalon-studio/

I think it stops the failed Test Case only, but the Test Suite keeps executing the rest of Test Cases.

2 Likes

If your test cases are dependent on the successful execution of another one, IMHO the right approach is to add them into that test case after the assertion, see https://docs.katalon.com/display/KD/Call+Test+Case

With such design, if the assert fail and the handling is set to ā€˜Stop on Failureā€™, the execution will just stop at that point without executing further test-cases.

1 Like

Ibus said:

If your test cases are dependent on the successful execution of another one, IMHO the right approach is to add them into that test case after the assertion, see https://docs.katalon.com/display/KD/Call+Test+Case

With such design, if the assert fail and the handling is set to ā€˜Stop on Failureā€™, the execution will just stop at that point without executing further test-cases.

It might work, but it complicates the scripts structure, it makes that Test Case to work instead of Test Suite

1 Like

well ā€¦ agreed, makes the structure more complicated.

other option may be ā€¦ set a global variable based on the result of the first test and execute the others conditionally (sort of 'if $passed then do_test else println(ā€˜test skippedā€™)) but this is ugly too, will still call all other testcases but just with a different wokflow

1 Like

Ibus said:

If your test cases are dependent on the successful execution of another one, IMHO the right approach is to add them into that test case after the assertion, see https://docs.katalon.com/display/KD/Call+Test+Case

With such design, if the assert fail and the handling is set to ā€˜Stop on Failureā€™, the execution will just stop at that point without executing further test-cases.

Well I really donā€™t agree with this approach. Itā€™s important to keep every test cases as simple and clean as possible. There should be a way to stop a whole Test Suite if a Test Case fails. My actual scenario : I need to login to my portal to start my test cases. If the login doesnā€™t succeed, the rest of the cases will fail for sure. And I need to have that login process in a case by itself (and not in a Setup/Teardown) because I would like to retry my login 3x times before declaring a FAIL (and if I use a Setup/Teardown, it seems like the retry count doesnā€™t apply).

1 Like

Ibus said:

well ā€¦ agreed, makes the structure more complicated.

other option may be ā€¦ set a global variable based on the result of the first test and execute the others conditionally (sort of 'if $passed then do_test else println(ā€˜test skippedā€™)) but this is ugly too, will still call all other testcases but just with a different wokflow

That could help me for a whileā€¦ do you have an example of code that catches the result of a test case ?

that depends on the assert you use in the main test case.

But since that assert, when is failing, it will stop that test case, you can put immediately after something like ā€˜GlobalVariable.excuteOthers = trueā€™.

If the failure handling is set to STOP_ON_FAILURE, when the assert in that main test case will fail, that line wonā€™t be executed and the ā€˜excuteOthersā€™ global variable will remain untouched (therefore you have to set it with a default ā€˜falseā€™)

any update on this concern?

Hi.

There is a important difference between CallTestCase and ā€œTestSuite with several TestCasesā€. For example, I have to start after every TestCase @AfterTestCase test listener script. This script inserts TestCase results data into DB through WebService. When I used CallTestCase method, only parent TestCase call @AfterTestCase, but TestCases called by CallTestCase do not call this script. So, I must use TestSuit. But in this case I need to skip other TestCases after one before has fail or error status.

Now, I understand I need to use Global Variable and manual scripting (or @BeforeTestCase???) to stop remaining TestCases execution. Unfortunaltely itā€™s not elegant solution. Therefore, that is not bad idea to enchance the KATALON with new functionality: stop whole TestSuit after some TestCase will finish with fail or error status.

Regards,

Alex

1 Like

Hi.

There is a important difference between CallTestCase and ā€œTestSuite with several TestCasesā€. For example, I have to start after every TestCase @AfterTestCase test listener script. This script inserts TestCase results data into DB through WebService. When I used CallTestCase method, only parent TestCase call @AfterTestCase, but TestCases called by CallTestCase do not call this script. So, I must use TestSuit. But in this case I need to skip other TestCases after one before has fail or error status.

Now, I understand I need to use Global Variable and manual scripting (or @BeforeTestCase???) to stop remaining TestCases execution. Unfortunaltely itā€™s not elegant solution. Therefore, that is not bad idea to enchance the KATALON with new functionality: stop whole TestSuit after some TestCase will finish with fail or error status.

Regards,

Alex

Even I was not able to find anything to fulfill this requirement. Alternatively You can use the call Test cases in such scenario and can club the depended test cases in one test case. Suppose you have to login and then perform some Test case. you can first create a test case where login will happen and then call the other depended test cases in that same Test case. Make sure you mark the login failure scenario as markfailureandstop method so that it will stop the rest of the execution.

Hi All,

I came across all the discussions here. However, I have an example scenario wherein TestCase#1 creates a user and Testcase#2 logins to application with that user account. And TestSuite#1 contains Testcase#1 and TestCase#2.

Now when I execute TestSuite#1 and suppose Testcase#1 fails, is there a way to stop executing TestSuite#1 completely. Since it is meaningless to start executing TestCase#2 without user account created(Testcase#1 fails)

Any help is appreciated. Thanks in advance.

Regards,
Sridhar

I tried with this listener inside my suit:

import com.kms.katalon.core.context.TestCaseContext
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testcase.TestCaseFactory as TestCaseFactory

    @TearDownTestCase(skipped = true) // Please change skipped to be false to activate this method.
    def tearDownTestCase(TestCaseContext testCaseContext) {
    	// Put your code here.
    	if(testCaseContext.getTestCaseStatus()=="FAILED"){
    		FailureHandling.STOP_ON_FAILURE
    	}
    }

But throws me the following error:
Cannot invoke method getTestCaseStatus() on null object

Anyone could solve this problem in another way?

1 Like

Hi @rodrigocalabretta

Methods annotated with @TearDownTestCase donā€™t accept (by design) a testCaseContext object, which is why it is null, and any attempt to call methods on a null object would result in the error you see. Object testCaseContext is available only in Test Listeners.

I tried to run that from the @AfterTestCase listener and handle that failure, but the problem is that it didnā€™t stop the Suit run. Where should I set that failure handling?

Hi guys, I think I have a solution to this.

Itā€™s not my favourite because I have to call this on every SetupTesCase() when I should prefer to stop all the suit immediate but least the test didnā€™t run

First of all, save the status at the finish test case run creating a string global variable

Listerner
	@AfterTestCase
	def AfterTestCase(TestCaseContext testCaseContext) {
		if(testCaseContext.getTestCaseStatus()!="PASSED"){
		GlobalVariable.TestCaseStatus= testCaseContext.getTestCaseStatus();
		}
}
Test Suit Script

@SetupTestCase(skipped = false)
def setupTestCase() {
// Put your code here.
if(GlobalVariable.TestCaseStatus==ā€œFAILEDā€ || GlobalVariable.TestCaseStatus==ā€œERRORā€){
KeywordUtil.markFailedAndStop(ā€˜ERROR: The previous test failā€™)

	}
}

I have to use in the listener that condition because when I mark as Failed or as Error in the previous test that test Fails RIGHT! but his status is still PASSED when I get testCaseContext.getTestCaseStatus() so thatā€™s the reason (I think maybe this is some error by katalon) :confused:

@ThanhTo let me know If you have some improvement in your mind, thatā€™s should be great

My solution for this years-old problem: