Is there a way to stop Test Suite execution if one of its Test Cases fails?
I.e. in case of creating certain elements some crucial part is missing, and then there is no point of reproducing that issue again and again in every Test Case.
Hi Alexander
You can use com.kms.katalon.core.util.Keywordutil for this purpose.
Please use either markErrorAndStop or markFailedAndStop methods.
You can find samples in below link:
http://www.sanspantalones.com/2018/02/16/marking-tests-as-passed-failed-or-in-error-using-markfailed-markpassed-markerror-in-katalon-studio/
I think it stops the failed Test Case only, but the Test Suite keeps executing the rest of Test Cases.
If your test cases are dependent on the successful execution of another one, IMHO the right approach is to add them into that test case after the assertion, see https://docs.katalon.com/display/KD/Call+Test+Case
With such design, if the assert fail and the handling is set to āStop on Failureā, the execution will just stop at that point without executing further test-cases.
Ibus said:
If your test cases are dependent on the successful execution of another one, IMHO the right approach is to add them into that test case after the assertion, see https://docs.katalon.com/display/KD/Call+Test+Case
With such design, if the assert fail and the handling is set to āStop on Failureā, the execution will just stop at that point without executing further test-cases.
It might work, but it complicates the scripts structure, it makes that Test Case to work instead of Test Suite
well ā¦ agreed, makes the structure more complicated.
other option may be ā¦ set a global variable based on the result of the first test and execute the others conditionally (sort of 'if $passed then do_test else println(ātest skippedā)) but this is ugly too, will still call all other testcases but just with a different wokflow
Ibus said:
If your test cases are dependent on the successful execution of another one, IMHO the right approach is to add them into that test case after the assertion, see https://docs.katalon.com/display/KD/Call+Test+Case
With such design, if the assert fail and the handling is set to āStop on Failureā, the execution will just stop at that point without executing further test-cases.
Well I really donāt agree with this approach. Itās important to keep every test cases as simple and clean as possible. There should be a way to stop a whole Test Suite if a Test Case fails. My actual scenario : I need to login to my portal to start my test cases. If the login doesnāt succeed, the rest of the cases will fail for sure. And I need to have that login process in a case by itself (and not in a Setup/Teardown) because I would like to retry my login 3x times before declaring a FAIL (and if I use a Setup/Teardown, it seems like the retry count doesnāt apply).
Ibus said:
well ā¦ agreed, makes the structure more complicated.
other option may be ā¦ set a global variable based on the result of the first test and execute the others conditionally (sort of 'if $passed then do_test else println(ātest skippedā)) but this is ugly too, will still call all other testcases but just with a different wokflow
That could help me for a whileā¦ do you have an example of code that catches the result of a test case ?
that depends on the assert you use in the main test case.
But since that assert, when is failing, it will stop that test case, you can put immediately after something like āGlobalVariable.excuteOthers = trueā.
If the failure handling is set to STOP_ON_FAILURE, when the assert in that main test case will fail, that line wonāt be executed and the āexcuteOthersā global variable will remain untouched (therefore you have to set it with a default āfalseā)
any update on this concern?
Hi.
There is a important difference between CallTestCase and āTestSuite with several TestCasesā. For example, I have to start after every TestCase @AfterTestCase test listener script. This script inserts TestCase results data into DB through WebService. When I used CallTestCase method, only parent TestCase call @AfterTestCase, but TestCases called by CallTestCase do not call this script. So, I must use TestSuit. But in this case I need to skip other TestCases after one before has fail or error status.
Now, I understand I need to use Global Variable and manual scripting (or @BeforeTestCase???) to stop remaining TestCases execution. Unfortunaltely itās not elegant solution. Therefore, that is not bad idea to enchance the KATALON with new functionality: stop whole TestSuit after some TestCase will finish with fail or error status.
Regards,
Alex
Hi.
There is a important difference between CallTestCase and āTestSuite with several TestCasesā. For example, I have to start after every TestCase @AfterTestCase test listener script. This script inserts TestCase results data into DB through WebService. When I used CallTestCase method, only parent TestCase call @AfterTestCase, but TestCases called by CallTestCase do not call this script. So, I must use TestSuit. But in this case I need to skip other TestCases after one before has fail or error status.
Now, I understand I need to use Global Variable and manual scripting (or @BeforeTestCase???) to stop remaining TestCases execution. Unfortunaltely itās not elegant solution. Therefore, that is not bad idea to enchance the KATALON with new functionality: stop whole TestSuit after some TestCase will finish with fail or error status.
Regards,
Alex
Even I was not able to find anything to fulfill this requirement. Alternatively You can use the call Test cases in such scenario and can club the depended test cases in one test case. Suppose you have to login and then perform some Test case. you can first create a test case where login will happen and then call the other depended test cases in that same Test case. Make sure you mark the login failure scenario as markfailureandstop method so that it will stop the rest of the execution.
Hi All,
I came across all the discussions here. However, I have an example scenario wherein TestCase#1 creates a user and Testcase#2 logins to application with that user account. And TestSuite#1 contains Testcase#1 and TestCase#2.
Now when I execute TestSuite#1 and suppose Testcase#1 fails, is there a way to stop executing TestSuite#1 completely. Since it is meaningless to start executing TestCase#2 without user account created(Testcase#1 fails)
Any help is appreciated. Thanks in advance.
Regards,
Sridhar
I tried with this listener inside my suit:
import com.kms.katalon.core.context.TestCaseContext import com.kms.katalon.core.model.FailureHandling as FailureHandling import com.kms.katalon.core.testcase.TestCase as TestCase import com.kms.katalon.core.testcase.TestCaseFactory as TestCaseFactory @TearDownTestCase(skipped = true) // Please change skipped to be false to activate this method. def tearDownTestCase(TestCaseContext testCaseContext) { // Put your code here. if(testCaseContext.getTestCaseStatus()=="FAILED"){ FailureHandling.STOP_ON_FAILURE } }
But throws me the following error:
Cannot invoke method getTestCaseStatus() on null object
Anyone could solve this problem in another way?
Methods annotated with @TearDownTestCase
donāt accept (by design) a testCaseContext
object, which is why it is null, and any attempt to call methods on a null object would result in the error you see. Object testCaseContext
is available only in Test Listeners.
I tried to run that from the @AfterTestCase listener and handle that failure, but the problem is that it didnāt stop the Suit run. Where should I set that failure handling?
Hi guys, I think I have a solution to this.
Itās not my favourite because I have to call this on every SetupTesCase() when I should prefer to stop all the suit immediate but least the test didnāt run
First of all, save the status at the finish test case run creating a string global variable
Listerner
@AfterTestCase def AfterTestCase(TestCaseContext testCaseContext) { if(testCaseContext.getTestCaseStatus()!="PASSED"){ GlobalVariable.TestCaseStatus= testCaseContext.getTestCaseStatus(); } }
Test Suit Script
@SetupTestCase(skipped = false)
def setupTestCase() {
// Put your code here.
if(GlobalVariable.TestCaseStatus==āFAILEDā || GlobalVariable.TestCaseStatus==āERRORā){
KeywordUtil.markFailedAndStop(āERROR: The previous test failā)} }
I have to use in the listener that condition because when I mark as Failed or as Error in the previous test that test Fails RIGHT! but his status is still PASSED when I get testCaseContext.getTestCaseStatus() so thatās the reason (I think maybe this is some error by katalon)
@ThanhTo let me know If you have some improvement in your mind, thatās should be great
My solution for this years-old problem: