I wonder why Katalon test case turns green although there is a warning “Test object with id ‘Object Repository/’ does not exist”. I expect a test case to stop once a warning event occurs. Windows 8.1/Katalon 7.0.8 I use STOP_ON_FAILURE setting by default.

This is how Failure Handling works: [image] Failure Handling Failure handling settings allow users to decide whether Katalon Studio will continue running or not in case of errors occurs during execution. Currently, Katalon Studio supports the following failure handling...

I know. But how to make Katalon stop a test case once a warning event occurs?

Typically, “warnings” are not treated as aborts or assertion failures. You can try this: [image] The other possibility is to trap the boolean from one of the verifyElement* APIs and use KeywordUtil. https://docs.katalon.com/javadoc/com/kms/katalon/core/util/KeywordUtil.html https://docs.katalon.com/katalon-studio/docs/mark-tests-status-for-a-custom-keyword.html

I’ll ask this question another way. What must happen in order to stop a test case containing the following code? WebUI.verifyElementPresent(findTestObject(

@devalex88 @ThanhTo I think this may be a bug. As you’re aware, I don’t run tests like this, so treat this report with due distrust :wink: Tested in Katalon 7.0.8 @anbanbanb Okay, here’s my Project Setting: [image] Here is my script (wam is just my way of writing comments - safe to ignore). wam 'Nav to Katalon "About us" page' WebUI.navigateToUrl("https://www.katalon.com/about-us/") WebUI.waitForPageLoad(10) wam 'Check some links are accessible' WebUI.verifyLinksAccessible(["https://www.katalon.com/resources-center/blog/", "https://www.katalon.com/partners-and-contributors"]) wam 'Create a failing test object using makeTO' TestObject to = makeTO("input.russ") wam 'Try to verify the fake test object - THIS SHOULD FAIL!' WebUI.verifyElementPresent(to, 5) wam 'Test should NOT reach this line!' WebUI.click(to) The test reaches the last line but should not. This also does not stop the test: wam 'Try to verify the fake test object - THIS SHOULD FAIL!' WebUI.verifyElementPresent(to, 5, FailureHandling.STOP_ON_FAILURE) This works but should not be necessary: wam 'Try to verify the fake test object - THIS SHOULD FAIL!' if(!WebUI.verifyElementPresent(to, 5, FailureHandling.STOP_ON_FAILURE)) { KeywordUtil.markFailedAndStop("Element not present") }

[image] Russ_Thomas: if(!WebUI.verifyElementPresent(to, 5, FailureHandling.STOP_ON_FAILURE)) { KeywordUtil.markFailedAndStop(“Element not present”) } This works. But I have many lines with checking objects, and it is inconvenient to use three lines of code for verification instead of one. Are you sure that Katalon developers initially considered WebUI.verifyElementPresent(findTestObject(... as warning and not as failure?

[image] anbanbanb: This works. But I have many lines with checking objects, and it is inconvenient to use three lines of code for verification instead of one. That’s the point of the bug report. Sit back and wait for the devs to check it out.

Hi @anbanbanb [image] [WebUI] Verify Element Present Description Verify if the given web element does present on DOM. Parameters Param Param Type Mandatory Description to TestObject Required Represent a web element. timeout int Required System will wait... So as documented, the behavior of verifyElementPresent should return true or false, indicating wether an element is present or not. It throwing an exception and failing the test was an explicit behavior, but making it harder for people who simply want to proceed with their logic, at least without try catch (which is bulky when used multiple times). Therefore we adjusted the behavior to match the expectations against the documentation. If you want to fail your test, then there are two ways: Implement a custom keyword that checks for false value returned from verifyElementPresent and throw the exception. Use: assert present == true // this will fail the test if verifyElementPresent returns false This adjustment is meant to balance the frequently observed use cases.

[image] ThanhTo: Therefore we adjusted the behavior to match the expectations against the documentation. @ThanhTo @devalex88 @duyluong Guys, keep in mind when you read this, I don’t use these APIs, so it doesn’t really matter to me, but, as someone trying to help others, now I have a problem: Thanh - the expectations are going to vary based on two use-cases: The boolean aspect The FailureHandling aspect What we have is one “thing” (verifyElementPresent) documented as offering two things[1] but failing to do one of them: Return true/false based on presence/absence. (PASS) Support FailureHandling as per the docs. (FAIL at least when STOP_ON_FAILURE is used) FailureHandling, as I understand it, is meant as an interface on to Katalon’s TestCase state management (they end up as throws in many cases, right?) But… If verifyElementPresent now ignores STOP_ON_FAILURE then it seems fair to say the FailureHandling argument has no purpose, or worse, only partial purpose. If so, then it doesn’t “match the expectations against the documentation”. You have a TestCase step that cannot fail - have I got that right? I took a look at VerifyElementPresentKeyword.groovy and WebUIKeywordMain.groovy but my mental closure parser threw a StackOverflowException :nerd_face::dizzy_face: To totally grok it I’d need to step it through properly. Therefore this is still buggy if only because the documentation is now wrong. If I’ve said STOP_ON FAILURE as both… a parameter to the method AND as my default handling setting and NEITHER can stop a test case, that’s a 100%, bona fide, de facto bug. Repeat. I don’t use this stuff so it doesn’t really matter to me, personally. But now I can’t support/help users understand what STOP_ON_FAILURE even means. My advice, for what it’s worth: deprecate WebUI, begin work on WebUI2 and fix all these irksome pain points. :sunglasses: Sorry if this was a ramble (been here 19 hours!) [1] It’s rarely a good idea to have one thing do two jobs: Single-responsibility principle - Wikipedia. Yeah, I know, we can all end up with tangles like this but that’s what the route cause is here – you have tension between a boolean and a throwable. It’s an arms race no one can win.

The purpose of Failure Handling is to handle all types of failures, not just the “considered-as-failure” of verifying element presents or exists. There are many exceptions which the implementation of verifyElementPresent may throw that may have nothing to do with the intention of the keyword (i.e NullPointerException). Specifying Failure Handling strategy, then, is for obtaining more/less control over these scenarios. To quote the documentation: Failure handling settings allow users to decide whether Katalon Studio will continue running or not in case of errors occurs during execution. https://docs.katalon.com/katalon-studio/docs/failure-handling.html#default-failure-handlingbehavior [image] Russ_Thomas: then it seems fair to say the FailureHandling argument has no purpose, or worse, only partial purpose While I agree that the expectation regarding the Failure Handling, in the context of this topic, would usually be about handling the case where verifyElementPresent determines that element is not present, it is not a “failure” per se, but more of a valid piece of information to base your test execution further (deciding what to do if element is not present). To decide to fail the test given this piece of information is a decision that would need to be made explicitly by using assert, or implicitly (if you have reasons to do so) by implementing a wrapper custom keyword. Nevertheless, this decision is entirely up to the users. Of course, the behavior switch from the old version to the new version may cause frictions because with more power to make decisions come more responsibility (to add alert/custom keywords), but I think returning a boolean indicating the state of presence/existence for that element is a good balance between the use cases.

FailureHandling Change breaks legacy code (was STOP_ON_FAILURE does not stop a test when verifyElementPresent returns false)

Bug Report

Russ_Thomas November 10, 2019, 4:45pm 2

This is how Failure Handling works:

Tests are now false-passing with a broken step

Topic		Replies	Views
Unable to verify IF element present or not Web Testing katalon-studio	18	8265	March 12, 2024
How to check element status in conditional statement? Miscellaneous	6	1748	August 12, 2021
Katalon 7.08 doesn't detect errors Web Testing katalon-studio	10	1525	November 12, 2019
[WebUI] Verify Element Present Miscellaneous new-doc-request	32	9937	April 7, 2022
Execute if/else when Element exists Web Testing katalon-studio	58	40201	January 17, 2023

FailureHandling Change breaks legacy code (was STOP_ON_FAILURE does not stop a test when verifyElementPresent returns false)

Related topics