Please let us know what you are using Katalon Studio for?
- I have currently applied Katalon Studio in my project
How would your work be affected if this issue has not been resolved?
- I cannot continue my job and have to work on something else while awaiting your response
Operating System
Windows 10
Katalon Studio Version
7.8.1
Screenshots / Videos
When using a negative assertion, e.g. assert !true, katalon studio shows the test step passed, and in the test suite the test step shows as passed.
sample script that will produce this error:
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.configuration.RunConfiguration as RunConfiguration
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.mobile.keyword.internal.MobileDriverFactory as MobileDriverFactory
import com.kms.katalon.core.model.FailureHandling
import com.kms.katalon.core.util.internal.PathUtil as PathUtil
import internal.GlobalVariable as GlobalVariable
import io.appium.java_client.HasSettings
import io.appium.java_client.Setting
'Should not work'
assert !true
execution of test suite with test script
individual execution of test script
Please try the foollowing test case script.
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
'Should not work'
//assert !true
WebUI.verifyEqual(!true, true)
This will show as failed.
Marking a test case failed — it is a part of functionality of the built-in WebUI.* keywords.
Groovy language’s assert statement throws an instance of java.lang.AssertionError but it does not concern about the Katalon Studio’s internal status.
1 Like
Thanks for the reply. Yah it is true that we can use built in keywords for verification statement, but if the default assert class in groovy throws an Assertion exception Katalon should pick up that Exception thrown and flag the test step as failed.
Following test case would illustrate what you desire (but KS is not implemented as such, of course)
import com.kms.katalon.core.util.KeywordUtil
try {
assert !true
} catch (AssertionError error) {
KeywordUtil.markFailed('Should not work')
throw error
}
This test case runs as follows:
2020-12-11 09:56:23.625 ERROR c.k.katalon.core.main.TestCaseExecutor - ❌ Test Cases/TC2 FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: Should not work
at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
at TC2.run(TC2:6)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
...
Thus you can mark your test case as “failed” when “assert expression” statement failed.
Not correct. Groovy’s assert throws an instance of java.lang.AssertionError, not an Exception.
By a general convention of Java programming, you should not catch any Throwable and Errors in your java/groovy codes. Here I quote from https://www.baeldung.com/java-error-catch
Please note that the Java Virtual Machine throws errors to indicate severe problems from which it can’t recover , such as lack of memory and stack overflows, among others.
Thus , we must have a very, very good reason to catch an error!
I think, Katalon Studio follows this convention and refrains from catching AssertionError. This is the reason why Katalon test step shows as passed when asserting a negative statement.
Therefore, I think, you should no use Groovy’s assert in test case scripts in Katalon Studio, as you will loose control of Test Case status (passed, failed). Yes, you can still use assert statement, but should be aware of the consequence. In fact I do use the assert statement of Groovy lang often (as assert XXX is short and easy to write) while casually playing on KS. But in a serious cases I would rewrite assert xxx to WebUI.verifyXXX or KeywordUtils.markFailed().
It is not accurate to report a test step as passing if that test step failed the test. I know for most this may not be a huge issue because test still fails, but for my industry we need the log evidence to be accurate.
Cucumber will report steps failing if an assertion fails, so I would hope it would follow a similar paradigm seeing that you can execute Cucumber tests in Katalon. If Katalon wants to continue to not reporting failed assertions as step failures, then they should at a minimum put a warning or error in the editor when someone uses an assert. Because we can’t really use any static analysis tools in Katalon IDE to help steer people clear of using asserts in their tests.
I confirmed in v8.2.0, Katalon Studio is able to report FAILED when Goovy’s assert directive raised an error. This is good.
