my ‘jump’ was a comparison between some coding paradigms (some people may read it as a way to move forward in the sense of how to approach the code, withouth being blocked by specific keywords/language implementations)
your ‘jump’ was just a give up.
just sayin …
I’ve just noticed my statements, like this one…
make it sound like I was missing one of your points above. I didn’t miss it. I’m saying that the documentation does not make it clear. I’m saying the FH argument does not handle all errors produced during the execution of the step.
I offer the following as evidence - if I’ve misunderstood, please show me where I’m wrong
Firstly, the documentation:
My code:
wam 'Nav to Katalon "About us" page'
WebUI.navigateToUrl("https://www.katalon.com/about-us/")
WebUI.waitForPageLoad(10)
wam 'Check some links are accessible'
WebUI.verifyLinksAccessible(["https://www.katalon.com/resources-center/blog/", "https://www.katalon.com/partners-and-contributors"])
wam 'Create a null test object'
TestObject to = null
wam 'Try to verify a null test object (1)'
if(!WebUI.verifyElementPresent(to, 5, FailureHandling.OPTIONAL)) {
wam "null object, OPTIONAL"
}
wam 'Try to verify a null test object (2)'
if(!WebUI.verifyElementPresent(null, 5, FailureHandling.CONTINUE_ON_FAILURE)) {
wam "null object, CONTINUE"
}
wam 'Try to verify a null test object - THIS SHOULD STOP THE TEST! (3)'
if(!WebUI.verifyElementPresent(null, 5, FailureHandling.STOP_ON_FAILURE)) {
wam "null object, STOP (shouldn't see this)"
}
wam 'Test should NOT reach this line!'
Result -> I see ALL lines executed. I see warnings but I don’t see errors or failures.
(Now you see how my comments are using KeywordUtil warnings)
Somebody needs to document clearly the definition of “failure” and “error”. I tried forcing errors as you can see. I went further, passing in strings for the timeout argument. They all caused exceptions but not errors proving the FH arg is pretty much meaningless.
Am I wrong?
If it’s the way I conduct (and manage) testing and reporting, please let me know. And if so, I’d be interested how the essence of the code above works for others.
1 Like
I jumped back to version 6.3.4 waiting of results of this discussion.
Problem of version 7.0 is wider and not only in failure handling STOP_ON_FAILURE. In my code I verify existence of web elements on pages using WebUI.verifyElementPresent and CONTINUE_ON_FAILURE failure handling. In v 6.3.4 program reported if element was absent and continue to run. Now (in version 7.0) program returns boolean value but doesn’t report failure. Now I need to use artificially created CustomKeywords to catch failure like:
@Keyword
def verifyElementPresent (String objectName, Integer n){
def present = false
present = WebUI.verifyElementPresent(findTestObject(objectName), n)
if (present == false){
KeywordUtil.markFailed(“Web element with id '” + objectName+"’ not present")
}
}
So now I need to fix thousand lines of code to use version 7.0. Is it good approach?
Sit tight. I’m hoping the devs will recategorize this as a regression. Not enough consideration was given to legacy behavior. I can understand the reasoning, v7 was a step-change and v6 has a death-date. But that doesn’t mean it was a good idea.
We need this code to be reverted and work to begin on WebUI2.
1 Like
In the meantime, I just add assert in front of every WebUI.verifyElementPresent(findTestObject(...
It works fine.
It is about a couple of dozens lines to add to. So it is acceptable for me.
it works. but is an abomination. like assert an assertion.
e.g:
assert 1 == 1
assert works as STOP_ON_FAILURE and doesn’t work if you need CONTINUE_ON_FAILURE
verifyElementPresent checks for null parameter and return false, so that’s why your test seems to ignore the FailureHandling flag, it is debatable wether or not we should throw exception when the parameter is null or just return false.
However, as a way to demonstrate my point regarding the kind of failures that FailureHandling would assist you with, considering the test case with just one line, no opening web driver or anything:
WebUI.verifyElementPresent(new TestObject(), 10, FailureHandling.OPTIONAL);
If you execute this, then the log would be as:
Considering the exact same script minus the failure handling:
WebUI.verifyElementPresent(new TestObject(), 10, FailureHandling.STOP_ON_FAILURE);
The log viewer:
If you have not opened a browser, then this should necessarily be an error, because the pre-condition to run the verification is not satisfied.
But, I would definitely discuss with the team about this and the documentations as well. It seems there’s a mismatch between what users think failures and errors are and what we think they are.
1 Like
Thanks Thanh - that’s very helpful.
Agreed. Also, there is the unfortunate choice of language around verifyElementPresent and FailureHandling which strongly implies that failure to verify the presence (the false return) is handled by FailureHandling.
This same point was made here:
As you’re no doubt aware by now, I don’t think you can fix this in this API. You need another API and let this one wither on the vine of history. (WebUI2 etc).
To help users with reams of legacy code derived from v6 code bases, you should revert the behavior. Then perhaps introduce isElementPresent as per @Ibus’ suggestion.
Thanks again!
2 Likes
Hi guys,
I googled “Verify vs Assert” and got lots of results, but there’s this particular result that attempts to explain in terms of Selenium.
Specifically:
- When the “assert” command fails, the test execution will be canceled. So when the Assertion fails, then all test steps are skipped / omitted after that line of code.
and
When the “Verify” command fails, then execution will continue and logging the failure.
Which I think may be helpful to our problem at hand. It is interesting to note that another source agrees with this conception. What do you guys think ?
2 Likes
Good idea. Thanks @ThanhTo
I think you guys should publish that in the docs anywhere the advice is relevant. Something like…
Where applicable and appropriate, Katalon Studio’s API set conforms to the published philosophies and definitions of the Selenium API set. In particular, the published definitions for
failure,errorandexception.
Of course, that also means, where you differ, you should either make that clear, or fix the problem.
1 Like
When the “Verify” command fails, then execution will continue and logging the failure.
That makes sense. My issue arises when I can run a test case that’s verifying an entire web page and its elements, with each step using FailureHandling of “Continue”, because I want it to check the entire page and check what elements may not be present. The current behavior doesn’t mark the test case as failed but instead as passing, and the step that failed only produces a warning.
I understand that with “verify”, execution continues and the failure is logged. But when I look at Test Suite results, any failing test cases won’t be marked as Failed. So now, I’m stepping through each step in the Test Case trying to see if there’s a warning that shows that the Test Case has actually failed.
The previous behavior from v6 seemed to work for me and made sense to me, though I do understand the point you were making earlier. I just want test cases or test steps to be marked as failures. I’d hate to have to go back and make changes to most of my tests to include a workaround. I don’t know what the plan is on reverting or not, but I’d like to get some suggestions on how to proceed forward. If I have to make changes or workarounds to follow this new behavior, then so be it. I would just like to know what the plan is.
4 Likes
I do have the same problem. In all my test cases ,I verify multiple elements on page using CONTINUE_ON FAILURE.If it only produces warning, then have to rework on all test cases. 
I just spent ages on this issue as my test cases are not being marked as failed even though I have set failure handling to stop on failure if a element is not present after verification.
After reading some discussions here it seems that this is the intended behavior even though this is not how it used to work. Which also means I need to go back through all my test cases and replace my verifys with something else.
If verifyelementpresent does not cause my tests to fail even when I have included failure handling then what do I use in its place? Its not as if there is a keyword assertelementpresent.
If this is to be the intended behavior going forward then why is this not the case for all other verify’s? VerifyMatch for example, you could use the same argument for this as was done for verifyelementpresent that its a bit of useful information that you would want to do something else based on if it is a match or not.
This is why we have verifypresenet and verifynotpresent as if either passes we can do what we want, but now that it does not fail when what we expect to happen doesnt happen, what are we supposed to do instead?
Hey Katalon-Team,
- verifyElementPresent does not fail anymore if Element is NOT found. It does only show a warning but testcase does “Pass”
- since Katalon 7.0.x (unsure which hotfix-version)
Steps to reproduce:
- Set up any object
- Use “VerifyElementPresent (TestObject to, int timeOut)”
- Check by letting the test run against any website (e.g. google)
- already verified that only this verify-function does not work as intended anymore
Workaround: use of assert-functionality
We used it to quick check user instances after deployment. All testcases passed…actually half of them had a problem and auto-testcases didn’t show them. I am pretty sure that’s not an unlikely testcase for this function.
Cheers
Alex
1 Like
I’m in the same boat as you, as are other users. Here’s a similar thread/discussion going on, with some input from a developer and community moderator: STOP_ON_FAILURE does not stop a test when verifyElementPresent returns false
It seems as though they changed the behavior of the FailureHandling parameter (anyone correct me if I’m wrong). Basically, it seems like the exception that was normally thrown and marking the test case as failed, isn’t behaving this way anymore. The exception may be getting handled differently and we only get a warning as a result, which makes it hard to see what test cases failed from a top level view (e.g., looking at Test Suite results).
1 Like
you are … somehow wrong.
the change was in the returned result of the keyword.
won’t throw an exception anylonger (which can be catched) but just an insensitive boolean.
(that made failure handling obsoleted for this particular case)
therefore breaking few ${unknown number} happy implementations to make happy few (not common) implementations 
LE. i know, i am evil but cannot hold it:
this is the Microsoft release model 
2 Likes
that seems to be a nice article to read, i will digest it … however.
the main issue here is … some users requested new approach (arguing they have to change few hundred lines of code because they did not understood the behaviour of the keyword from the begining)
but nobody cares about the users who already understood and use it accordingly.
how about backward compatibility strategy?
LE: i just finished reading the article. quote:
if(isElementPresent(By.linkText("Submit_Button")))
------------
assertTrue(isElementPresent(By.cssSelector("div.stqatools")));
------------
public class Verify_Example {
public static void main(String args[]) {
// Creating softAssert object
SoftAssert softAssert = new SoftAssert();
// Assertion failing
softAssert.fail("Failing first assertion");
System.out.println("Failing 1");
// AssertAll use to make test case fail in report all scripts will execute until assertAll() statement
softAssert.assertAll();
}
}
Told you!
@Ibus et al
I’m closing this thread and moving it to Bug Reports. While it’s not a bug per se, at the very least those with legacy code deserve due consideration. I have it on @ThanhTo’s word that a conversation has been started backstage.
There are plenty of ways this can be addressed - some mentioned above and no doubt other ways the devs themselves may prefer.
That for now is good enough, I think.
By all means, do keep referring people to this thread - because what we don’t need is 25 threads all saying the same thing.
Thanks everyone.
3 Likes
Hi everyone,
After reviewing the implications of new behavior of verifyElementPresent, we have decided to revert it back to the old one. Just want you guys to know, there will be an official announcement later.
2 Likes