How to let TestCases in a TestSuite to quit once a TestCase failed

I have published a demo project on GitHub:


Problem to solve

A topic in the Katalon User Forum wrote:

Hi, there is a way to stop a test suite if one of its cases fails? I need some method or form to stop the suite, does anyone have a solution?

Fair enough requirement, I think. Let me assume I have a Test Suite TS1 is comprised with 3 Test Cases: TC1 , TC2 , TC3 . The TC3 runs very long (e.g, 20 minutes). The TC2 normally passes but occasionally fails. When I run the TS1 and unfortunately the TC2 failed, I want the TS1 stops as soon as the TC2 failed. I do not want to wait for the TC3 to finish after 20 minutes.

However Katalon Studio does not support the feature to stop a Test Suite when a comprising Test Case failed.

What else can I do practically to let a Test Suite to finish as soon as a Test Case failed?

Solution

I will not ask Katalon Studio to control if it should invoke each Test Cases (TC1, TC2, TC3) or not. I will let it invoke all Test Cases in a Test Suite as defined.

Rather, I would write each Test Cases to check if any of preceding Test Cases in the Test Suite has failed . If there are any failed Test Cases, then a Test Case should quit immediately. A Test Case should check it before executing the body of test processes to prevent consuming time. I will introduce a few custom Groovy classes. A jar file that includes the module will be provided. Provided that Test Cases are informed of the results of preceding Test Cases, following Test Cases can be self-deterministic.

Demonstration

You want to run Test Suites/TSa . The TSa is comprised with 3 Test Cases: TS1_passes , TS2_passes , TS3_passes . When you run TSa , all of compont Test Cases will pass.

TSa

Next, please run the Test Suites/TSb . The TSb is comprised with 3 Test Cases: TS1_passes , TS2_fails , TS3_passes . When you run it, TS2_fails will fail intentionally, and TS3_passes quits soon before doing any meaningful actions .

TSb

As you see, the TSb can shorten the duration required for the TC3 after the failed TC2 .

Yes, Katalon Studio still executes all of 3 Test Cases defined in the Test Case TSb . But the Test Cases are coded so that they quit soon if one or more preceding Test Cases failed. Therefore TSb can eliminate redundant duration.

How you should write your code

Test Cases

I wrote 4 Test Cases.

TC1_passes

import com.kazurayam.ks.testsuite.Advisor
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

if (Advisor.shouldQuit()) return;

WebUI.comment("TC1 ran")
for (int i in 1..3) {
    WebUI.comment("TC1 is doing a heavy task: ${i}")
}

TC2_fails

import com.kazurayam.ks.testsuite.Advisor
import com.kms.katalon.core.util.KeywordUtil
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

if (Advisor.shouldQuit()) return;

WebUI.comment("TC2 ran")
KeywordUtil.markFailed("TC2 failed")

TC2_passes

import com.kazurayam.ks.testsuite.Advisor
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

if (Advisor.shouldQuit()) return;

WebUI.comment("TC2 ran")
for (int i in 1..3) {
    WebUI.comment("TC2 is doing a heavy task: ${i}")
}

TC3_passes

import com.kazurayam.ks.testsuite.Advisor
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

if (Advisor.shouldQuit()) return;

WebUI.comment("TC3 ran")
for (int i in 1..3) {
    WebUI.comment("TC3 is doing a heavy task: ${i}")
}

Please note that all these Test Cases has a common section at the very beginning:

import com.kazurayam.ks.testsuite.Advisor

if (Advisor.shouldQuit()) return;

Design

com.kazurayam.ks.testsuite.Advisor is a Custom Groovy class that I developed. This is included in the TestSuiteAdvisor-x.x.x.jar file. A call to Advisor.shouldQuit() would return a Boolean value. If one or more preceding Test Cases in a Test Suite have failed, then shouldQuit() will return true. Then the test case should decide; if it wants to, it can quit immediately by calling the statement return;

Katalon Studio will ignorantly trigger your Test Cases, and your Test Cases are supposed to choose for themselves. This is a sort of Inversion of control.

Test Listener

You will wonder how Advisor is informed of the status preceding Test Cases (passed or failed)? The trick is performed by a Test Listener.

Test Listeners/TL1

import com.kazurayam.ks.testsuite.ProgressListener
import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.annotation.BeforeTestSuite
import com.kms.katalon.core.context.TestCaseContext
import com.kms.katalon.core.context.TestSuiteContext

class TL1 {
    ProgressListener listener

    TL1() {
        this.listener = new ProgressListener()
    }

    @BeforeTestSuite
    def beforeTestSuite(TestSuiteContext testSuiteContext) {
        listener.beforeTestSuite(testSuiteContext)
    }

    @AfterTestCase
    def afterTestCase(TestCaseContext testCaseContext) {
        listener.afterTestCase(testCaseContext)
    }
}

The TL1 delegates another custom class com.kazurayam.ks.testsuite.ProgressListener to inform the Advisor of the status ( PASSED or FAILED ) of all Test Cases.

The Test Listener is not bundled in the jar file. You need to write a Test Listener like this manually. You can just copy and paste the above code.

Other source codes

The source code of the other classes are disclosed on another GigHub repository. Please have a look if you are interested in the internal.

Dependencies

At the Releases page, you can download the jar of TestSuiteAdvisor-x.x.x.jar . You want to copy that jar into your project’s Drivers folder.

TestSuiteAdvisor internally depends on the ExecutionProfilesLoader project’s jar. You want to copy the jar into your project’s `Drivers’ folder as well.

See the following screenshot how the Drivers folder would look like:

Drivers

I have just found an article “Stop Test Suite Execution with ‘X’ Failed Tests | Fail-fast Principle”. According to this, Katalon Runtime Engine supports a command line param -maxFailedTests=T since v8.1. OK. I understand it. My solution is different from this. My solution works in a Katalon Studio Free version as well as in KSE+KRE. My solution works the same as specifying -maxFailedTest=1 to KRE. I personally do not need to be able to specify -maxFailedTests=2 or larger.

Although I can imagine a code like this:

if (Advisor.exceedingMaxFailedTests(2)) return;

or

if (Advisor.checkCondition({List<ProgressEntry> entries -> 
    entries.size() >= 2 })) return;

these are possible to implement; but I don’t think I need these.

I would rather be interested in “Test Cases Dependency Tree” like Gradle’s tasks in build.gradle.

Say, let me suppose I can somehow express:

TC5 dependsOn TC1
TC4 dependsOn TC2
TC3 dependsOn TC2
TC2 dependsOn TC1

when TC1 passes then TC2 fails, TC3 and TC4 will not run, but TC5 will run.

A difficulty is that a Test Suite can be comprised with the repetition of a single Test Case with variables bound to data. For example

row1 TC2 with {"ID":"foo"}, depends on TC1
row2 TC2 with {"ID":"bar"}, depends on TC1
row3 TC2 with {"ID":"baz"}, depends on TC1
row4 TC2 with {"ID":"foo"}, depends on TC1    // "foo" again!

Interesting, but I suppose, I do not practically need this complexity.

For such cases would be nice to have a keyword like KeywordUtil.markSkipped("Reason for skipping").
Will be more relevant, imho.
I think this has been discussed few times but ignored by the development team.

As a workaround, markWarning can be used instead:
https://docs.katalon.com/javadoc/com/kms/katalon/core/util/KeywordUtil.html#markWarning(java.lang.String)

Advisor .shouldQuit() actually uses KeywordUtil.markWarning()

See Line26 of https://github.com/kazurayam/TestSuiteAdvisor/blob/master/Keywords/com/kazurayam/ks/testsuite/Advisor.groovy

1 Like

Aha, cool.
Sorry, I didn’t check into the code, because you know me, I am lazy :))