I made a demo project on GitHub:
Problems to solve
Reproducing a problem on try-catch block which encloses callTestCase()
duyluong, a Katalon Developer, made an issue
“CallTestCase using in try-catch block throws StepFailedException”.
Expected result:
If users use callTestCase in try-catch block, KS should NOT throw StepFailedException. Main Test Case should pass if nothing throws in the catch or finally block.
Actual result:
If users use callTestCase in try-catch block, KS throws StepFailedException and marks the main test case failed.
I was not aware of the reported problem. I wanted to reproduce the problem on my PC. I made a main test case like the following psuedo code:
try {
WebUI.callTestCase(findTestCase('sub'), [:])
} catch (Exception ex) {}
And I made a test case named sub
, which is called by the main, like the following psuedo code:
import com.kms.katalon.core.util.KeywordUtil
KeywordUtil.markFailureAndStop("so long")
Expected behavior:
The main test case is expected to pass. Even if the sub test case failed and threw StepFailedException, the try
block in the main is expected to catch it.
Behavior to be reproduced:
The issue reports that KS 5.10.1 will actually behave different. The main test case will actually fail in KS 5.10.1. When the sub test case threw StepFailedException, the try
block in the main is unable to catch it.
I wanted to witness the behavior.
A custom keyword that always fail
In order to investigate the reported issue, I wanted to create a comprehensive suite of test cases, which should include:
- a caller that calls a sub test case without
try-catch
block; the sub test case does not log failure, does not throwStepFailedException
. - a caller that calls a sub test case without
try-catch
block; the sub test case does log failure, but does not throwStepFailedException
. - a caller that calls a sub test case without
try-catch
block; the sub test case does log failure, and does throwStepFailedException
. - a caller that calls a sub test case with
try-catch
block; the sub test case does not log failure, does not throwStepFailedException
. - a caller that calls a sub test case with
try-catch
block; the sub test case does log failure, but does not throwStepFailedException
. - a caller that calls a sub test case with
try-catch
block; the sub test case does log failure, and does throwStepFailedException
.
A long list of test cases. I wanted to make my test cases concise.
Solution
I should develop a new custom keyword, namely fail()
, which encapsulate the detail behavior.
- to log as info or as failure
- to throw exception or not
Description
a primitive keyword: fail
I made a custom keyword com.kazurayam.ksbackyard.PrimitiveKeywords#fail
:
package com.kazurayam.ksbackyard
import com.kms.katalon.core.annotation.Keyword
import com.kms.katalon.core.exception.StepFailedException
import com.kms.katalon.core.model.FailureHandling
import com.kms.katalon.core.util.KeywordUtil
public class PrimitiveKeywords {
/**
* This keyword emits the message given as 1st argument into Katalon log.
* The FailureHandling object given as 2nd argument is respected.
*
* @param message
* @param flowControl
* @throws StepFailedException
*/
@Keyword
static void fail(String message, FailureHandling flowControl) throws StepFailedException {
switch (flowControl) {
case FailureHandling.OPTIONAL:
KeywordUtil.logInfo(message)
break
case FailureHandling.CONTINUE_ON_FAILURE:
KeywordUtil.markFailed(message)
break
case FailureHandling.STOP_ON_FAILURE:
KeywordUtil.markFailedAndStop(message)
break
}
}
}
How it works?
fail
does no more than emiting the given message into the Katalon Log- if
FailureHandling.OPTIONAL
is specified as the 2nd arg, it emits the given message as Info. That’s all. It works just likeWebUI.comment
. - if
FailureHandling.CONTINUE_ON_FAILURE
is specified as the 2nd arg, it emits the message as failure plus the stack trace, but does NOT throw any StepFailedException. - if
FailureHandling.STOP_ON_FAILURE
is specified as the 2nd arg, it emits the message as failure plus the stack trace, and throw new StepFailedExcption which is raised up to the caller
Let me see …
When I run Test Cases/caller - OPTIONAL
, I got the following output:
>>> Intentional failure. Caller can safely ignore this
...
>>> callTestCase() failed but we ignored it as FailureHandling.OPTIONAL specfied
OK, this was just what I expected.
When I run Test Cases/caller - CONTINUE_ON_FAILURE
, I got the following output:
2019-01-28 15:04:17.961 ERROR c.k.katalon.core.main.TestCaseExecutor - ❌ com.kazurayam.ksbackyard.PrimitiveKeywords.fail(message, CONTINUE_ON_FAILURE) FAILED.
Reason: com.kms.katalon.core.exception.StepFailedException:
>>> Intentional failure. Caller can continue
at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:18)
at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
at com.kazurayam.ksbackyard.PrimitiveKeywords.fail(PrimitiveKeywords.groovy:25)
at com.kazurayam.ksbackyard.PrimitiveKeywords.invokeMethod(PrimitiveKeywords.groovy)
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:49)
at sub - CONTINUE_ON_FAILURE.run(sub - CONTINUE_ON_FAILURE:3)
...
Hmm, that was not what I expected. There I expected to see a message of
>>> callTestCase() failed but Caller continued as FailureHandling.CONTINUE_ON_FAILURE specified
But actually I could not see it. Test Cases/caller - CONTINUE_ON_FAILURE
should not throw a StepFailedException but it actually does. I think it is a bug in Katalon Stdudio 5.10.1.
Finally, Test Cases/caller - STOP_ON_FAILURE
will emit messages is similar to Test Cases/caller - CONTINUE_ON_FAILURE
Reproducing the problem of try-catch
I made a test case Test Cases/caller with try - STOP_ON_FAILURE
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
try {
WebUI.callTestCase(findTestCase('sub - STOP_ON_FAILURE'),
["message":">>> Intentional failure. Caller should stop immediately"])
WebUI.comment(">>> callTestCase() failed")
WebUI.comment(">>> but you will not seed this message")
} catch (Exception ex) {
WebUI.comment(">>> caught an Exception: " + ex.getMessage())
}
Expected behavior
>>> Internal failure. Caller should stop immediately
...
>>> caught an Exception: ...
Reproduced behavior
2019-01-28 15:27:15.420 ERROR k.k.c.m.CustomKeywordDelegatingMetaClass - ❌ >>> Intentional failure. Caller should stop immediately
2019-01-28 15:27:15.439 ERROR c.k.katalon.core.main.TestCaseExecutor - ❌ com.kazurayam.ksbackyard.PrimitiveKeywords.fail(message, STOP_ON_FAILURE) FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: >>> Intentional failure. Caller should stop immediately
at com.kms.katalon.core.util.KeywordUtil.markFailedAndStop(KeywordUtil.java:27)
at com.kms.katalon.core.util.KeywordUtil$markFailedAndStop.call(Unknown Source)
at com.kazurayam.ksbackyard.PrimitiveKeywords.fail(PrimitiveKeywords.groovy:28)
at com.kazurayam.ksbackyard.PrimitiveKeywords.invokeMethod(PrimitiveKeywords.groovy)
at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:49)
at sub - STOP_ON_FAILURE.run(sub - STOP_ON_FAILURE:5)
Conclusion
In the above message of “Reproduced behavior”, I could see:
- the
try-catch
block in the caller is expected to catch the StepFailedException raised by the callee test case, but it does not. - rather the test case entirely failed
- the full stack trace was printed in the log
I could reproduce the problem reported by “CallTestCase using in try-catch block throws StepFailedException”.
My custom keyword fail()
greatly simplified the test cases for investigation.
Memo
I was inspired by org.junit.Assert#fail
.