What's the use of FailureHandling.OPTIONAL?

[vadim.nechunaev]

As i see, FailureHandling.CONTINUE_ON_FAILURE marks the case as failed, but continues running it. But OPTIONAL doesn’t seem to affect the test run results in Log Viewer. It does mark the failed steps with a yellow mark, but you need to specifically look at all the steps (that can be numerous and nested). It doesn’t show up in the Passed, Failed, Errors list on top of the log viewer.

I would like to get a mandatory value from a test case in a bigger test case (so it’s stop on failure), but also do checks inside the case that aren’t blocking the overall execution.

So basically, is there an easy way to see the failed optional checks without extra button presses every time?

[kazurayam]

@vadim.nechunaev

I would like to get a mandatory value from a test case in a bigger test case (so it’s stop on failure), but also do checks inside the case that aren’t blocking the overall execution.

I don’t really understand what you want to achieve.

Please show your code. Please describe how your are not satisfied about the code. Then others may be able to suggest to you something how to modify your code.

[vadim.nechunaev]

Basically i’m trying to deal with this problem:

callTestCase allways returns null if it fails, even If exists a return sentence

Hello. I´ve some shared test cases that return data. Some days ago I´ve realized that, if the test fails, it is returning null instead of the value I give back. For example: --log string-- KeywordUtil.logInfo("Received: ${text}") def size = text.length() TestNGKW.verifyGreaterThan(size, 1, FailureHandling.CONTINUE_ON_FAILURE) KeywordUtil.logInfo("size: ${size}") return "${size}" --Main test-- def returned = WebUI.callTestCase(findTestCase('Work In Progress/log string'), [('text') : tex…

For the sake of example, let’s say i have a test case:

WS.verifyEqual(1, 2, FailureHandling.CONTINUE_ON_FAILURE)
return "response"

So i’m calling this test case from within another:

def returned = WebUI.callTestCase(findTestCase('Test case name'), [('text') : text], FailureHandling.CONTINUE_ON_FAILURE)
KeywordUtil.logInfo("Returned: ${returned}")

So the called test case marked as failure and even though it reached its end, the returned value is null.

The suggested solution seems to be writing the returned value into a global variable and calling that, though that would be bothersome.

But i’m confused what FailureHandling.OPTIONAL is even supposed to be used for. So i change the called test case to have optional handling of the assertion:

WS.verifyEqual(1, 2, FailureHandling.OPTIONAL)
return "response"

As the result the intended value is returned by the test case (“response”), but not only is the failed test case marked successful, the failed optional assertion isn’t displayed anywhere in the log viewer, unless i specifically check every step in the log tree.

And here it is buried inside nested calls:

Naturally, manually checking every step for optional failures isn’t viable with bigger size and number of test cases.

[kazurayam]

@vadim.nechunaev

So the called test case marked as failure and even though it reached its end, the returned value is null.

You can find the reasone why the returned value is null. Look at the source code of CallTestCase keyword at https://github.com/katalon-studio/katalon-studio-testing-framework/blob/master/Include/scripts/groovy/com/kms/katalon/core/keyword/builtin/CallTestCaseKeyword.groovy

Line#58-74

                TestResult result = TestCaseMain.runTestCase(calledTestCase.getTestCaseId(), new TestCaseBinding(
                        calledTestCase.getTestCaseId(), binding), flowControl, false, false)
                switch (result.getTestStatus().getStatusValue()) {
                    case TestStatus.TestStatusValue.FAILED:
                        throw new StepFailedException(MessageFormat.format(StringConstants.KW_MSG_CALL_TC_FAILED, calledTestCase.getTestCaseId()),
                            result.getCause())
                        break
                    case TestStatus.TestStatusValue.ERROR:
                        throw new StepErrorException(MessageFormat.format(StringConstants.KW_MSG_CALL_TC_X_FAILED_BECAUSE_OF_ERROR, calledTestCase.getTestCaseId()), result.getCause())
                        break
                    case TestStatus.TestStatusValue.PASSED:
                        logger.logPassed(MessageFormat.format(StringConstants.KW_LOG_PASSED_CALL_TC_X_SUCCESSFULLY, calledTestCase.getTestCaseId()))
                        break
                    default:
                        break
                }
                return result.getScriptResult()   // Line#74

The line#74 is reached only when you specify FailureHandling.OPTIONAL in the called test case.

When you specify FailureHandling.CONTINUE_ON_FAILURE in the called test case, the execution flow would not reach to the line#74 of the CallTestCase class.

Therefore the caller Test Case will find as if the callee returned null. But in fact the callee does NOT reach the the line#74 = the callee does not return anything <= from the caller point of view, you may see the callee returned null, but the fact is not.

[kazurayam]

I guess you want to do 2 things

1. you want your callee test case always returns some meaningful value, not null
2. you want to mark red with “FAILED” message when something unexpected happened in your callee test case : for example WS.verifyEquals(1, 2, FailureHandling.OPTIONAL)

How can you invent your own solution?

Perhaps you would want to use KeywordUtil.markFailed() in your caller Test Case
https://api-docs.katalon.com/com/kms/katalon/core/util/KeywordUtil.html#markFailed(java.lang.String)

Your callee test may be something like:

boolean var1 = WS.verifyEquals(a, b, FailureHandling.OPTIONAL)
boolean var2 = WS.verifyEquals(a, c, FailureHandling.OPTIONAL)
if (var1 && var2) {
    return "good"
} else if (var1 && !var2 || !var1 && var2) {
    return "so so"
} else {
    return "bad"
}

Your caller can check the returned value and switch:

def result = WS.callTestCase("callee", ["a": valueA, "b": valueB, "c":valueC], FailureHandling.OPTIONAL)

switch (result) {
    case: "good"
        KeywordUtil.markPassed("happy")
        break;
    case: "so so"
        KeywordUtils.markFailed("something went wrong")
        break
    default:
        KeywordUtils.markFailedAnsStop("I am sad")
        break
}

When you give “a”=1, “b”=2, “c”=1 then the callee would return “so so”, the caller will mark the test failed, would show the message in read in the Log Viewer, as you possibly want it to.

Now you have a full control over the failure handling. A wheel has been re-invented!

[vadim.nechunaev]

I’m thinking of using markWarning or something like that, but i’ll come back to it later. Right now i’m putting test automation on hold for a few days.