Execution log: all test cases marked passed despite KeywordUtil.markFailed()

Miscellaneous
1

Hi,

I’m facing an issue when using the mail reports.

My test suite calls a test case that uses an Excel data file to iterate (about 120 lines).

The structure of the test case is the following:

try
{
	[...]
	
	if (myCondition)
	{
		throw new Exception("Test failed")
	}
}
catch (Exception ex)
{
	KeywordUtil.markFailed(ex.getMessage())
}

Despite some iterations fail during execution, the mail report mark all the iterations as “passed”: ${totalPassed} = 120 / ${totalFailed} = 0 / ${totalError} = 0
Same for execution log.

I can understand why… Thanks in advance for your help.

1 Like
2

In fact, this does not concern mail reports, as the execution log marks all test cases as passed too.

3

I presume that the myCondition variable has value of false in all iterations. Because the myCondition is false, thefore your code will not raise any Exception("Test failed). Therefore number of totalFailed will be 0 and totalError will be 0.

Why that variable is always false? ---- I don’t know. You did not disclose any detail. Only you can investingate your code.

Only one thing I woud suggest is to inseart a line of debug print, like

    println "myCondtion=${myCondtion}"
    if (myCondition) {
        ...
4

Thank you for your answer.
I can confirm that the exception is risen for some iterations so the condition is properly working. I’ve no explaination on why the related iterations are not marked as failed.

5

How about calling markFailedAndStop() rather than markFailed() ? just to see what happens.

6

I already tried with this keyword but no luck :frowning:

7

I wrote a Test Case as follows:

import com.kms.katalon.core.util.KeywordUtil

try
{
    boolean myCondition = true
    if (myCondition)
    {
        throw new Exception("Test failed")
    }
}
catch (Exception ex)
{
    KeywordUtil.markFailed(ex.getMessage())
}

I ran it and got the following output:

2023-10-12 00:03:41.606 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2023-10-12 00:03:41.609 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/ejacquemoud
Test Cases/ejacquemoud
[:]
2023-10-12 00:03:42.382 ERROR com.kms.katalon.core.util.KeywordUtil    - ❌ Test failed
2023-10-12 00:03:42.423 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/ejacquemoud FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: Test failed
	at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
	at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
	at ejacquemoud.run(ejacquemoud:13)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:448)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:439)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:418)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:410)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:285)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:144)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:135)
	at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
	at TempTestCase1697036617778.run(TempTestCase1697036617778.groovy:25)

Test Cases/ejacquemoud
FAILED
2023-10-12 00:03:42.481 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/ejacquemoud

This result is just as I expected. Nothing myterious.
I failed to reproduce the ejacquemoud’s case.
I have no more idea.
I will quit.

8

I have the same behavior as you in the console: “StepFailedException”, “ERROR” and “:x: Test Cases/MyTestCase FAILED”.
But when I switch to the log viewer, the same Test Case appears as passed (“Passes: 1”), which is non-sense… Unfortunatly I think the mail report is based on the log viewer.

I should precise the way my code works.
In my Test Case, I have a handlefailure() method that calls another method located in a keyword. I guess this is part of the problem…

9

I can not reproduce this on my machine.
I have no more idea.

10

Thanks for your help.
I finally implemented my custom mail reports as I can not find any solution.