I have published a GitHub project:
Problem to solve
In the following repository, you can find a set of codes for explanation.
In this repository, you will find a Katalon Studio project with a Test Suite named TSa
. The TSa
consists of 4 Test Cases: TCa1
, TCa2
, TCa3
and TCa4
. The TSa
will invoke 4 Test Cases just sequentially. The TCa1
, TCa2
and TCa4
— these 3 will finish quickly in a few seconds; but the TCa3
could run long (20 seconds, actually).
Regardless accidentally or intentionally, the TCa2
could fail. Even if the TCa2
failed, the Test Suite TSa
will continue invoking the following Test cases TCa3
and TCa4
, as the following screenshot shows:
Please note that:
-
The Test Case
TCa2
failed intentionally. -
The Test Case
TCa3
was invoked and took 20 seconds to finish. -
The Test Suite
TSa
took 23 seconds to finish.
Now I want to introduce a condition:
When the
TCa2
failed, I do not like to wait for theTCa3
to finish, because theTCa3
is no longer worth executing (due to some reasons) when its predecessorTCa2
failed. Rather I want to stop the Test Suite as soon as possible so that I can start debugging theTCa2
.
In the Katalon Studio GUI, I would keep watching it. I would notice any failures during a Test Suite runs, and I would able to stop the Test Suite by some manual intervention (clicking a “stop” button).
But the Katalon Runtime Engine provides very little chance to intervene the progress of a Test Suite run. When the TCa2
failed, still the TCa3
will be invoked. We have to wait for the TCa3
to finish (for 20 seconds, 2 minutes, 20 minutes or possibly 2 hours) before we can start trouble-shooting the TCa2
’s failure. No other option is provided by Katalon.
Proposed Solution
This project provides a jar file which contains a Groovy class named
which is supposed to be used in a Katalon Studio project. Here is a sample Test Case script that utilizes it:
// Test Cases/conditional-withFailure/TCb3
import com.kazurayam.ks.TestCaseResults
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.model.FailureHandling as FailureHandling
TestCaseResults.assertTestCasePASSED("conditional-withFailure/TCb2")
WebUI.comment("conditional-withFailure/TCb3 is running")
WebUI.delay(5)
WebUI.comment("conditional-withFailure/TCb3 is running still")
WebUI.delay(5)
WebUI.comment("conditional-withFailure/TCb3 is running yet")
WebUI.delay(5)
WebUI.comment("conditional-withFailure/TCb3 is running even more")
WebUI.delay(5)
This is a Test Case named TCb3
. The TCb3
is bundled in a Test Suite named TSb
which consists of 4 Test Cases: TCb1
, TCb2
, TCb3
and TCb4
. These 4 Test Cases will be executed just sequentially.
Let’s read the source code; the TCb3
is calling
TestCaseResults.assertTestCasePASSED("conditional-withFailure/TCb2")`
With this statement, the TCb3
makes an assertion that a Test Case "conditional-withFailure/TCb2"
ran and finished PASSED
before the TCb3
. If the TCb2
is found PASSED
, the call to assertTestCasePASSED()
will return silent. The TCb3
will continue its processing. If the TCb2
is found Not PASSED
, the call to assertTestCasePASSED()
will raise a StepFailedException
. The TCb3
will quit soon, it will skip its processing body. A call to TestCaseResults.assertTestCasePASSED(String testCaseId)
enables you to conditionally execute your Test Case depending on the result of preceding Test Case in a Test Suite.
Solution Explained
Installing the jar file
A jar file that contains com.kazurayam.ks.TestCaseResult
class is downloadable at the Releases page. You want to create your own Katalon Studio project. Download the jar into the Drivers
folder in the project.
Create a Test Listener
You have to create a Test Listener. The name of Test Listener can be any. For example TL1
. The code should be like this:
import com.kazurayam.ks.TestCaseResults
import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.annotation.AfterTestSuite
import com.kms.katalon.core.context.TestCaseContext
import com.kms.katalon.core.context.TestSuiteContext
class TL1 {
@AfterTestCase
def afterTestCase(TestCaseContext testCaseContext) {
TestCaseResults.put(testCaseContext)
}
@AfterTestSuite
def afterTestSuite(TestSuiteContext testSuiteContext) {
TestCaseResults.println()
}
}
You would not need to customize it. Just copy & paste this sample.
This code transfers the execution results of Test Cases in a Test Suite to the instance of com.kazurayam.ks.TestCaseResults
class. This code is mandatory to inform the TestCaseResults
instance of the Test Cases’ result so that it can serve useful assertTestCasePASSED(String testCaseId)
method.
Sample codes explained
Test Suite TSb
consists of 4 Test Cases: TCb1
, TCb2
, TCb3
and TCb4
.
TCb1
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
WebUI.comment("conditional-withFailure/TCb1 is running")
TCb2
// Test Cases/conditional-withFailure/TCb2
import com.kazurayam.ks.TestCaseResults
import com.kms.katalon.core.util.KeywordUtil
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.model.FailureHandling as FailureHandling
TestCaseResults.assertTestCasePASSED("conditional-withFailure/TCb1")
// or
//Boolean result = TestCaseResults.assertTestCasePASSED("conditional-withFailure/TCb1", FailureHandling.OPTIONAL)
WebUI.comment("conditional-withFailure/TCb2 is running")
KeywordUtil.markFailedAndStop("Test Cases/conditional-withFailure/TCb2 failed intentionally")
Please note that TCb2
will fail intentionally for demonstration purpose.
TCb3
// Test Cases/conditional-withFailure/TCb3
import com.kazurayam.ks.TestCaseResults
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.model.FailureHandling as FailureHandling
TestCaseResults.assertTestCasePASSED("conditional-withFailure/TCb2")
WebUI.comment("conditional-withFailure/TCb3 is running")
WebUI.delay(5)
WebUI.comment("conditional-withFailure/TCb3 is running still")
WebUI.delay(5)
WebUI.comment("conditional-withFailure/TCb3 is running yet")
WebUI.delay(5)
WebUI.comment("conditional-withFailure/TCb3 is running even more")
WebUI.delay(5)
Please note that TCb3
asserts that the preceding TCb2
passed. If the TCb2
is found NOT passed, then the TCb3
will quit soon without doing its long-running processing.
TCb4
import com.kazurayam.ks.TestCaseResults
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
TestCaseResults.assertTestCasePASSED("conditional-withFailure/TCb1")
WebUI.comment("conditional-withFailure/TCb4 is running")
How the TSb
ran
When I ran the TSb
, I saw the following result:
Please note the following points:
-
The Test Case
TCb2
failed intentionally -
The Test Case
TCb3
started but failed quickly without performing its long-running processing. TheTCb3
failed because theTCb2
failed. -
The Test Suite
TSb
finished quickly in just 2 seconds.
Principally, The Test Suite TSa
and TSb
are quite similar. Both are intended to do the same test processing. However the TSb
behaves quite differently when any of Test Case failed. The TSb
is enabled to execute Test Cases conditionally using the com.kazurayam.ks.TestCaseResults.assertTestCasePASSED(String testCaseId)
method call. So a Test Suite empowered by the assertTestCasePASSED
can quit as soon as its member Test Case failed.
No longer you need to wait for a long-running Test Suite to finish after a failure even in the Katalon Runtime Engine.