Fail() - a primitive keyword


#1

I made a demo project on GitHub:


Problems to solve

Reproducing a problem on try-catch block in a test using callTestCase()

duyluong, a Katalon Developer, made an issue
“CallTestCase using in try-catch block throws StepFailedException”.

Expected result:
If users use callTestCase in try-catch block, KS should NOT throw StepFailedException. Main Test Case should pass if nothing throws in the catch or finally block.
Actual result:
If users use callTestCase in try-catch block, KS throws StepFailedException and marks the main test case failed.

I was not aware of the reported problem. I wanted to reproduce the problem on my PC. I made a main test case like the following psuedo code:

try {
    WebUI.callTestCase(findTestCase('sub'), [:])
} catch (Exception ex) {}

And I made a test case named sub, which is called by the main, like the following psuedo code:

import com.kms.katalon.core.util.KeywordUtil
KeywordUtil.markFailureAndStop("so long")

Expected behavior:

The main test case is expected to pass. Even if the sub test case failed and threw StepFailedException, the try block in the main is expected to catch it.

Behavior to be reproduced:

The issue reports that KS 5.10.1 will actually behave different. The main test case will actually fail in KS 5.10.1. When the sub test case threw StepFailedException, the try block in the main is unable to catch it.

I wanted to witness the behavior.

A custom keyword that always fail

In order to investigate the reported issue, I wanted to create a comprehensive suite of test cases, which should include:

  1. a caller that calls a sub test case without try-catch block; the sub test case does not log failure, does not throw StepFailedException.
  2. a caller that calls a sub test case without try-catch block; the sub test case does log failure, but does not throw StepFailedException.
  3. a caller that calls a sub test case without try-catch block; the sub test case does log failure, and does throw StepFailedException.
  4. a caller that calls a sub test case with try-catch block; the sub test case does not log failure, does not throw StepFailedException.
  5. a caller that calls a sub test case with try-catch block; the sub test case does log failure, but does not throw StepFailedException.
  6. a caller that calls a sub test case with try-catch block; the sub test case does log failure, and does throw StepFailedException.

A long list of test cases. I wanted to make my test cases concise.

Solution

I should develop a new custom keyword, namely fail(), which encapsulate the detail behavior.

  • to log as info or as failure
  • to throw exception or not

Description

a primitive keyword: fail

I made a custom keyword com.kazurayam.ksbackyard.PrimitiveKeywords#fail:

package com.kazurayam.ksbackyard

import com.kms.katalon.core.annotation.Keyword
import com.kms.katalon.core.exception.StepFailedException
import com.kms.katalon.core.model.FailureHandling
import com.kms.katalon.core.util.KeywordUtil

public class PrimitiveKeywords {

	/**
	 * This keyword emits the message given as 1st argument into Katalon log.
	 * The FailureHandling object given as 2nd argument is respected.
	 *
	 * @param message
	 * @param flowControl
	 * @throws StepFailedException
	 */
	@Keyword
	static void fail(String message, FailureHandling flowControl) throws StepFailedException {
		switch (flowControl) {
			case FailureHandling.OPTIONAL:
				KeywordUtil.logInfo(message)
				break
			case FailureHandling.CONTINUE_ON_FAILURE:
				KeywordUtil.markFailed(message)
				break
			case FailureHandling.STOP_ON_FAILURE:
				KeywordUtil.markFailedAndStop(message)
				break
		}
	}
}

How it works?

  1. fail does no more than emiting the given message into the Katalon Log
  2. if FailureHandling.OPTIONAL is specified as the 2nd arg, it emits the given message as Info. That’s all. It works just like WebUI.comment.
  3. if FailureHandling.CONTINUE_ON_FAILURE is specified as the 2nd arg, it emits the message as failure plus the stack trace, but does NOT throw any StepFailedException.
  4. if FailureHandling.STOP_ON_FAILURE is specified as the 2nd arg, it emits the message as failure plus the stack trace, and throw new StepFailedExcption which is raised up to the caller

Let me see …

When I run Test Cases/caller - OPTIONAL, I got the following output:

>>> Intentional failure. Caller can safely ignore this
...
>>> callTestCase() failed but we ignored it as FailureHandling.OPTIONAL specfied

OK, this was just what I expected.

When I run Test Cases/caller - CONTINUE_ON_FAILURE, I got the following output:

2019-01-28 15:04:17.961 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ com.kazurayam.ksbackyard.PrimitiveKeywords.fail(message, CONTINUE_ON_FAILURE) FAILED.
Reason: com.kms.katalon.core.exception.StepFailedException:
>>> Intentional failure. Caller can continue
	at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:18)
	at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
	at com.kazurayam.ksbackyard.PrimitiveKeywords.fail(PrimitiveKeywords.groovy:25)
	at com.kazurayam.ksbackyard.PrimitiveKeywords.invokeMethod(PrimitiveKeywords.groovy)
	at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:49)
	at sub - CONTINUE_ON_FAILURE.run(sub - CONTINUE_ON_FAILURE:3)
...

Hmm, that was not what I expected. There I expected to see a message of

>>> callTestCase() failed but Caller continued as FailureHandling.CONTINUE_ON_FAILURE specified

But actually I could not see it. Test Cases/caller - CONTINUE_ON_FAILURE should not throw a StepFailedException but it actually does. I think it is a bug in Katalon Stdudio 5.10.1.

Finally, Test Cases/caller - STOP_ON_FAILURE will emit messages is similar to Test Cases/caller - CONTINUE_ON_FAILURE

Reproducing the problem of try-catch

I made a test case Test Cases/caller with try - STOP_ON_FAILURE

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

try {
	WebUI.callTestCase(findTestCase('sub - STOP_ON_FAILURE'),
		["message":">>> Intentional failure. Caller should stop immediately"])

	WebUI.comment(">>> callTestCase() failed")
	WebUI.comment(">>> but you will not seed this message")
} catch (Exception ex) {
	WebUI.comment(">>> caught an Exception: " + ex.getMessage())
}

Expected behavior

>>> Internal failure. Caller should stop immediately
...
>>> caught an Exception: ...

Reproduced behavior

2019-01-28 15:27:15.420 ERROR k.k.c.m.CustomKeywordDelegatingMetaClass - ❌ >>> Intentional failure. Caller should stop immediately
2019-01-28 15:27:15.439 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ com.kazurayam.ksbackyard.PrimitiveKeywords.fail(message, STOP_ON_FAILURE) FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: >>> Intentional failure. Caller should stop immediately
	at com.kms.katalon.core.util.KeywordUtil.markFailedAndStop(KeywordUtil.java:27)
	at com.kms.katalon.core.util.KeywordUtil$markFailedAndStop.call(Unknown Source)
	at com.kazurayam.ksbackyard.PrimitiveKeywords.fail(PrimitiveKeywords.groovy:28)
	at com.kazurayam.ksbackyard.PrimitiveKeywords.invokeMethod(PrimitiveKeywords.groovy)
	at com.kms.katalon.core.main.CustomKeywordDelegatingMetaClass.invokeStaticMethod(CustomKeywordDelegatingMetaClass.java:49)
	at sub - STOP_ON_FAILURE.run(sub - STOP_ON_FAILURE:5)

Conclusion

In the above message of “Reproduced behavior”, I could see:

  • the try-catch block in the caller is expected to catch the StepFailedException raised by the callee test case, but it does not.
  • rather the test case entirely failed
  • the full stack trace was printed in the log

I could reproduce the problem reported by “CallTestCase using in try-catch block throws StepFailedException”.

My custom keyword fail() greatly simplified the test cases for investigation.

Memo

I was inspired by org.junit.Assert#fail.


How to stop test case in catch block
#2

Pure science, Kaz. Exceptional.

concise -> rigorous