[Katalon Studio 7.8.2] Make use of assertions for better test case quality and failure investigation

Hi everyone!

Since Katalon Studio 7.8.2, Katalon TestOps will give more details about assertions in execution reports. Assertions ensure that the application is working correctly by checking whether a condition is true, i.e., whether labels, data, API responses, etc., are rendered correctly.

This new feature will give you a better grasp of the severity of a failure. For example, a test result with 1% of its assertions have failed usually indicates a better product quality than that with 99% of the assertions have failed.

Moreover, the number of assertions of a test case also tells something about this test case’s quality. A test case without any assertions is usually less useful than a test case with a dozen assertions.

As usual, this new feature required the integration between Katalon Studio projects and Katalon TestOps.

Whenever an execution is submitted to Katalon TestOps, you can see the number of assertions for each test result.

In a specific test result, you can see a list of all failed assertions without having to go through the log file.

Katalon TestOps also shows the number of assertions a test case normally has so that you can identify test cases that need more assertions.

Built-in assertion keywords have names prefixed by verify, e.g. landingPage = WebUI.verifyElementPresent(findTestObject('Page_CuraAppointment/div_Appointment'), GlobalVariable.G_Timeout).

Custom assertion keywords are Groovy/Java methods that make sure of KeywordUtil.markPassed(message), KeywordUtil.markFailed(message), KeywordUtil.markFailedAndStop(message).

You can also exclude a custom keyword from being identified as an assertion in the Project Setting page of Katalon TestOps.

3 Likes

Hi there. (Best wishes 2021!)
I’m very much interested to see this feature implemented in my projects. I do wonder, it does not seem to pick up any assertions that I coded like this:
assert actualStatusAfterRequestForDelta != myRequestsStatusProcessed : " We expect the actualStatusAfterRequestForDelta ($actualStatusAfterRequestForDelta) to be updated and thus no longer in status $myRequestsStatusProcessed - See screenShot:" + WebUI.takeScreenshot()

What happens now is that I can see my test failed :
Reason:
java.lang.AssertionError: We expect the actualStatusAfterRequestForDelta (Verwerkt) to be updated and thus no longer in status Verwerkt - See screenShot:C:\X\Reports\20210104_081102\Y\Z\20210104_081113\1609744403404.png. Expression: (actualStatusAfterRequestForDelta != myRequestsStatusProcessed). Values: actualStatusAfterRequestForDelta = Verwerkt, myRequestsStatusProcessed = Verwerkt

Yet in the Test Case’s Log I can not see any failed test steps. I can be mistaking, but I really thought in previous versions this show at least as a failed test step?
image

In TestOps it says 0/8 failed assertions, so it isn’t picking up on the “assert”…
Am I missing something?

Thanks

1 Like

Hi @joost.degeyndt

Currently, Grovvy assert keyword is not supported. For fully supported with continue-on-failure, try our verifyEqual built-in keyword. verifyEqual keyword is available for WebUI, Mobile, and Windows.

About your code of assertion actualStatusAfterRequestForDelta, I think you should use Objects.equals for object comparison if actualStatusAfterRequestForDelta is not a primitive or enum type.

Thanks

If I use the verifyEqual built-in keyword, it seems I lose the possibility to provide a customized assertion-failure-message. In my above example the assertion failure is quite straight forward, but in others that message can very much help the debugging effort. I do agree that continue-on-failure is something I lack with the current Groovy assert keyword.

Update: I did find this topic JUnit-like assertion method that can emit custom message for a custom keyword that provides best of both worlds!

1 Like

That’s a great set of Custom Keywords. Thank you @joost.degeyndt for the mention. Thank you @kazurayam for sharing the Keywords.

@devalex88

On this old post, Want to pass as argument Message to WebUI.verifyEqual(message, actual, expected)
there was a discussion that dev might implement this custom code. Is there any chance that the Katalon asserts could be expanded to include a third parameter for custom message using what @kazurayam created?

Any place we can upvote this?

1 Like

@duyluong Please help.

Hi @devalex88, I guess my request is a low priority. Thanks for trying.

Hi @jim.sears,

On this old post, Want to pass as argument Message to WebUI.verifyEqual(message, actual, expected)
there was a discussion that dev might implement this custom code. Is there any chance that the Katalon asserts could be expanded to include a third parameter for custom message using what @kazurayam created?
Any place we can upvote this?

We will put this keyword request/enhancement in our backlog for further evaluation. The final solution may come with adding another parameter to log messages or introduce a new keyword.

Thanks

2 Likes