Test Case level is set to passed even if one of the steps are mark failed (using KeywordUtil.markFailed)

Murphy_Memije · February 17, 2021, 3:02pm

Hi,

I created a custom keyword to verify if two values are matched utilizing the built in keyword for WS.

Snippet:

@Keyword
def verifyValuesMatch(String actual, String expected) {
	if (WS.verifyMatch(actual, expected, false)) {
		log.resultLog("PASSED", expected, actual);
	} else {
		log.resultLog("FAILED", expected, actual);
	}
}

I then call another function to mark the step failed which uses the built-in KeywordUtil.markFailed

All is working fine and the steps are successfully mark failed however I just noticed that even though one of the steps are mark failed, the Test Case level is still set to PASSED which gives me a wrong impression that the actual test has no issues. Is it intended that way? Here’s a sample report in TestOps.

I wanted to set the test case to failed if it has any steps that are marked as failed. I know we can use other keywords such as Assertions.assertThat but I want my test to continue on failure since I’m testing a web service which involves verification of multiple attributes.

Is there any workaround that I can do? Thanks in advance for the inputs and suggestions!

kazurayam · February 18, 2021, 3:12am

~~I think, yes, it is intended way.~~

I can find the javadoc:

https://docs.katalon.com/javadoc/com/kms/katalon/core/util/KeywordUtil.html

~~Please find 2 methods available~~

KeywordUtil.markFailed()
KeywordUtil.makrFailedAndStop()

public static void markFailed (String message)
Mark a keyword to be failed and continue execution

~~The markFailed method is intended to mark A KEYWORD to be failed, no Exception will be thrown, no affect to the TEST CASE level.~~

~~You should look at another method:~~

public static void markFailedAndStop (String message)
Mark a keyword to be failed and stop execution

~~This will affect to the test case level. By this method, the test case will be mark FAILED and an Exception will be thrown.~~

kazurayam edited: May be I am wrong.

Murphy_Memije · February 18, 2021, 5:58am

Hi @kazurayam,

It’s an instance of com.kms.katalon.util.KeywordUtil

Murphy_Memije · February 18, 2021, 5:58am

Thanks @kazurayam, I’ll have a check on this and get back if this will satisfy my use case! Thanks!

Murphy_Memije · February 18, 2021, 5:58am

Hi @kazurayam ,

Thanks for the reply and suggestion!

That’s right, I wanted to continue and proceed on the proceeding validations after previous steps were marked failed. Using the markFailedAndStop will force stop the execution on the first instance of failure.

To give some examples with my project also, I’m using a BDD approach and I have multiple test case in a single feature file.

e.g.

GIVEN I do this…
WHEN I do this…
THEN I verify this… <— (1st validation)
AND I verify this… <----(2nd validation)

I wanted my test to proceed on the 2nd validation without force exiting the execution on the 1st validation.

kazurayam · February 18, 2021, 6:38am

I tried to reproduce this on my side.

Here is my skeltal example:

Keyword: my.ResultLogger

package my

import com.kms.katalon.core.util.KeywordUtil

public class ResultLogger {
	int numberOfFailures = 0

	def getNumberOfFailres() {
		return numberOfFailures
	}

	def resultlog(type, expected, actual) {
		switch(type) {
			case "R02":
				KeywordUtil.markFailed("failed")
				numberOfFailures += 1
				break
		}
	}
}

WebUI Test Case:

import com.kms.katalon.core.util.KeywordUtil

import my.ResultLogger

ResultLogger logger = new ResultLogger()

logger.resultlog("R02", "expected", "actual")
logger.resultlog("R02", "expected", "actual")
logger.resultlog("R02", "expected", "actual")

/*
if (logger.getNumberOfFailures() > 0) {
	KeywordUtil.markFailedAndStop("one or more failures occured")
}
*/

When I executed the Test Case, I got output in the console:

2021-02-18 15:42:08.849 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/TC3
2021-02-18 15:42:10.144 DEBUG testcase.TC3                             - 1: logger = new my.ResultLogger()
2021-02-18 15:42:10.163 DEBUG testcase.TC3                             - 2: logger.resultlog("R02", "expected", "actual")
2021-02-18 15:42:10.203 ERROR com.kms.katalon.core.util.KeywordUtil    - ❌ failed
2021-02-18 15:42:10.260 DEBUG testcase.TC3                             - 3: logger.resultlog("R02", "expected", "actual")
2021-02-18 15:42:10.269 ERROR com.kms.katalon.core.util.KeywordUtil    - ❌ failed
2021-02-18 15:42:10.284 DEBUG testcase.TC3                             - 4: logger.resultlog("R02", "expected", "actual")
2021-02-18 15:42:10.291 ERROR com.kms.katalon.core.util.KeywordUtil    - ❌ failed
2021-02-18 15:42:10.319 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/TC3 FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: failed
	at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
	at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
	at my.ResultLogger.resultlog(ResultLogger.groovy:15)
	at my.ResultLogger$resultlog.call(Unknown Source)
	at TC3.run(TC3:7)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:398)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:389)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:368)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:360)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:255)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:105)
	at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
	at TempTestCase1613630524859.run(TempTestCase1613630524859.groovy:25)

2021-02-18 15:42:10.371 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/TC3

As you see, the Test Case was marked FAILED when it called KeywordUtil.markFailed(). On the other hand, you wrote that your Test Case is marked PASSED.

I do not see the reason why we see this discrepancy.

Murphy_Memije · February 18, 2021, 7:34am

Thanks for this detailed example @kazurayam really helps. After further checking with my logs, it is indeed failing my test case. Sample log below:

2021-02-18 15:57:19.780 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2021-02-18 15:57:19.782 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/xcore/orders/order-creation
2021-02-18 15:57:21.727 DEBUG testcase.order-creation                  - 1: runFeatureFile("Include/features/xcore/orders/test.feature")
2021-02-18 15:57:21.782 INFO  c.k.k.c.c.k.CucumberBuiltinKeywords      - Starting run keyword runFeatureFile: 'Include/features/xcore/orders/test.feature' and extract report to folder: 'C:\Users\Murphy\AppData\Local\Temp\Katalon\Test Cases\xcore\orders\order-creation\20210218_155717/cucumber_report/1613635041772'...
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
2021-02-18 15:57:24.048 INFO  c.k.k.core.webservice.common.HarLogger   - HAR: C:\Users\Murphy\AppData\Local\Temp\Katalon\Test Cases\xcore\orders\order-creation\20210218_155717\requests\main\0.har
2021-02-18 15:57:24.254 DEBUG com.kms.katalon.core.util.KeywordUtil    - ✓ [PASSED]: Expected = 200 Actual = 200
2021-02-18 15:57:24.279 INFO  com.kms.katalon.core.util.KeywordUtil    - [INFO]: Shopee order created with tracking number 0642-4178-CBRQ
2021-02-18 15:57:24.904 INFO  c.k.k.core.webservice.common.HarLogger   - HAR: C:\Users\Murphy\AppData\Local\Temp\Katalon\Test Cases\xcore\orders\order-creation\20210218_155717\requests\main\1.har
2021-02-18 15:57:24.909 DEBUG com.kms.katalon.core.util.KeywordUtil    - ✓ [PASSED]: Expected = 200 Actual = 200
2021-02-18 15:57:24.935 DEBUG com.kms.katalon.core.util.KeywordUtil    - ✓ [PASSED]: Expected = Shopee Actual = Shopee
2021-02-18 15:57:25.380 INFO  c.k.k.core.webservice.common.HarLogger   - HAR: C:\Users\Murphy\AppData\Local\Temp\Katalon\Test Cases\xcore\orders\order-creation\20210218_155717\requests\main\2.har
2021-02-18 15:57:25.383 DEBUG com.kms.katalon.core.util.KeywordUtil    - ✓ [PASSED]: Expected = 200 Actual = 200
2021-02-18 15:57:25.392 ERROR c.k.k.core.keyword.internal.KeywordMain  - ❌ Unable to verify match between actual text 'MMB' and expected text 'LUZ' (Root cause: com.kms.katalon.core.exception.StepFailedException: Actual text 'MMB' and expected text 'LUZ' are not matched
	at com.kms.katalon.core.keyword.builtin.VerifyMatchKeyword$_verifyMatch_closure1.doCall(VerifyMatchKeyword.groovy:57)
	at com.kms.katalon.core.keyword.builtin.VerifyMatchKeyword$_verifyMatch_closure1.call(VerifyMatchKeyword.groovy)
	at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)
	at com.kms.katalon.core.keyword.builtin.VerifyMatchKeyword.verifyMatch(VerifyMatchKeyword.groovy:60)
	at com.kms.katalon.core.keyword.builtin.VerifyMatchKeyword.execute(VerifyMatchKeyword.groovy:45)
	at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:73)
	at com.kms.katalon.core.keyword.BuiltinKeywords.verifyMatch(BuiltinKeywords.groovy:73)
	at utils.Validator.verifyValuesMatch(Validator.groovy:62)
	at utils.Validator$verifyValuesMatch$0.call(Unknown Source)
	at step_definitions.TestFlowOrders.the_order_attribute_should_be_in_the_GET_order_response(TestFlowOrders.groovy:152)
	at cucumber.runtime.Utils$1.call(Utils.java:26)
	at cucumber.runtime.Timeout.timeout(Timeout.java:16)
	at cucumber.runtime.Utils.invoke(Utils.java:20)
	at cucumber.runtime.java.JavaStepDefinition.execute(JavaStepDefinition.java:48)
	at cucumber.runtime.PickleStepDefinitionMatch.runStep(PickleStepDefinitionMatch.java:50)
	at cucumber.runner.TestStep.executeStep(TestStep.java:55)
	at cucumber.runner.TestStep.run(TestStep.java:42)
	at cucumber.runner.PickleStepTestStep.run(PickleStepTestStep.java:53)
	at cucumber.runner.TestCase.run(TestCase.java:47)
	at cucumber.runner.Runner.runPickle(Runner.java:44)
	at cucumber.runtime.Runtime.runFeature(Runtime.java:120)
	at cucumber.runtime.Runtime.run(Runtime.java:106)
	at cucumber.api.cli.Main.run(Main.java:35)
	at cucumber.api.cli.Main$run.call(Unknown Source)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy:106)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy)
	at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)
	at com.kms.katalon.core.keyword.internal.KeywordMain$runKeyword.call(Unknown Source)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:73)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile$0.callStatic(Unknown Source)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:246)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile.call(Unknown Source)
	at order-creation.run(order-creation:19)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:398)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:389)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:368)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:360)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:255)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:105)
	at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
	at TempTestCase1613635037152.run(TempTestCase1613635037152.groovy:25)
)
2021-02-18 15:57:25.396 ERROR com.kms.katalon.core.util.KeywordUtil    - ❌ [FAILED]: Expected = LUZ Actual = MMB
2021-02-18 15:57:25.783 INFO  c.k.k.core.webservice.common.HarLogger   - HAR: C:\Users\Murphy\AppData\Local\Temp\Katalon\Test Cases\xcore\orders\order-creation\20210218_155717\requests\main\3.har
2021-02-18 15:57:25.786 DEBUG com.kms.katalon.core.util.KeywordUtil    - ✓ [PASSED]: Expected = 200 Actual = 200
2021-02-18 15:57:25.790 DEBUG com.kms.katalon.core.util.KeywordUtil    - ✓ [PASSED]: Expected = cod Actual = cod
2021-02-18 15:57:26.235 INFO  c.k.k.core.webservice.common.HarLogger   - HAR: C:\Users\Murphy\AppData\Local\Temp\Katalon\Test Cases\xcore\orders\order-creation\20210218_155717\requests\main\4.har
2021-02-18 15:57:26.237 DEBUG com.kms.katalon.core.util.KeywordUtil    - ✓ [PASSED]: Expected = 200 Actual = 200
2021-02-18 15:57:26.240 ERROR c.k.k.core.keyword.internal.KeywordMain  - ❌ Unable to verify match between actual text 'next_day' and expected text 'same_day' (Root cause: com.kms.katalon.core.exception.StepFailedException: Actual text 'next_day' and expected text 'same_day' are not matched
	at com.kms.katalon.core.keyword.builtin.VerifyMatchKeyword$_verifyMatch_closure1.doCall(VerifyMatchKeyword.groovy:57)
	at com.kms.katalon.core.keyword.builtin.VerifyMatchKeyword$_verifyMatch_closure1.call(VerifyMatchKeyword.groovy)
	at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)
	at com.kms.katalon.core.keyword.builtin.VerifyMatchKeyword.verifyMatch(VerifyMatchKeyword.groovy:60)
	at com.kms.katalon.core.keyword.builtin.VerifyMatchKeyword.execute(VerifyMatchKeyword.groovy:45)
	at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:73)
	at com.kms.katalon.core.keyword.BuiltinKeywords.verifyMatch(BuiltinKeywords.groovy:73)
	at utils.Validator.verifyValuesMatch(Validator.groovy:62)
	at utils.Validator$verifyValuesMatch$0.call(Unknown Source)
	at step_definitions.TestFlowOrders.the_order_attribute_should_be_in_the_GET_order_response(TestFlowOrders.groovy:152)
	at cucumber.runtime.Utils$1.call(Utils.java:26)
	at cucumber.runtime.Timeout.timeout(Timeout.java:16)
	at cucumber.runtime.Utils.invoke(Utils.java:20)
	at cucumber.runtime.java.JavaStepDefinition.execute(JavaStepDefinition.java:48)
	at cucumber.runtime.PickleStepDefinitionMatch.runStep(PickleStepDefinitionMatch.java:50)
	at cucumber.runner.TestStep.executeStep(TestStep.java:55)
	at cucumber.runner.TestStep.run(TestStep.java:42)
	at cucumber.runner.PickleStepTestStep.run(PickleStepTestStep.java:53)
	at cucumber.runner.TestCase.run(TestCase.java:47)
	at cucumber.runner.Runner.runPickle(Runner.java:44)
	at cucumber.runtime.Runtime.runFeature(Runtime.java:120)
	at cucumber.runtime.Runtime.run(Runtime.java:106)
	at cucumber.api.cli.Main.run(Main.java:35)
	at cucumber.api.cli.Main$run.call(Unknown Source)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy:106)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$_runFeatureFile_closure1.doCall(CucumberBuiltinKeywords.groovy)
	at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:68)
	at com.kms.katalon.core.keyword.internal.KeywordMain$runKeyword.call(Unknown Source)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:73)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile$0.callStatic(Unknown Source)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords.runFeatureFile(CucumberBuiltinKeywords.groovy:246)
	at com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords$runFeatureFile.call(Unknown Source)
	at order-creation.run(order-creation:19)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:398)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:389)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:368)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:360)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:255)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:105)
	at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
	at TempTestCase1613635037152.run(TempTestCase1613635037152.groovy:25)
)
2021-02-18 15:57:26.241 ERROR com.kms.katalon.core.util.KeywordUtil    - ❌ [FAILED]: Expected = same_day Actual = next_day

1 Scenarios (e[32m1 passede[0m)
8 Steps (e[32m8 passede[0m)
0m3.698s

2021-02-18 15:57:26.293 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/xcore/orders/order-creation FAILED.

However I’m not sure how this is being interpreted by TestOps because clearly it’ showing that the test case/ test suite passed where there are actually failed assertions (as for with my original post.)

kazurayam · February 18, 2021, 8:47am

I found that com.kms.katalon.core.main.TestCaseExecutor and com.kms.katalon.core.main.WSVerificationExecutor — these 2 classes execute test case scripts differently.

TestCaseExecutor throws Exception and report that the Test Case has FAILED when the test case script calls com.kms.katalon.util.KeywordUtil.markFailed() once or more times.
WSVerificationExcutor does NOT throw any Exception, does NOT report Test Case failed when the test case script calls com.kms.katalon.util.KeywordUtil.markFailed() once or more times

I could read the source codes and got sure of this.

Murphy_Memije · February 18, 2021, 9:14am

Good to know how these two execute scripts. So TestCaseExecutor is the way for me. But then I’m not sure why TestOps reports behave differently (previous screenshots - Test Case/Suite passed but with failed assertions). Should I raise in the TestOps category?

Appreciate your time taking a deeper look into this @kazurayam very helpful thanks!

kazurayam · February 18, 2021, 9:15am

I know nothing about TestOps.

kazurayam · February 19, 2021, 3:09am

I guess, TestOps is not interpreting the result of test case/test suites. TestOps just displays what was reported by the class which executed your test.

With which executor class you executed your test — TestCaseExecutor or WSVerificationExecutor or anything else?

Murphy_Memije · February 19, 2021, 4:57am

I have tried both, here’s an example of the html report generated if this will help.
Edit: Actually in order to publish the report to TestOps you must run it via Test Suite level which always uses TestCaseExecutor class I believe.

Using TestExecutorClass:
20210219_112254.html (222.2 KB)

In TestOps:

kazurayam · February 19, 2021, 5:21am

Thank you for sharing the screenshot of TestOps.

I have never used TestOps. I know nothing about the report.

I would like to pass this issue Katalon teaｍ

@ThanhTo
@duyluong

Murphy_Memije · February 19, 2021, 5:26am

Thanks @kazurayam ! Appreciate it.

Looking forward to hear from the Katalon Team. Thanks!

kate.mallinson · December 8, 2022, 8:27pm

@here I’d be really interested to know if there is a recommended way forward, we also have Studio and TestOps and are seeing what looks like different results in both?