Is it possible to make soft assert in Katalon?

yuliya.h · August 7, 2019, 6:08am

Hello dear community (so warm when I think how active you are guys and how this place is incredible)!

I’m just wondering if there any way to make asserts work in a soft way. Here I was actually posting the question, but it looks like that documentation is checked not so often:

https://docs.katalon.com/katalon-studio/docs/assert-statements.html#assert-statement-in-scripting-view

I configured my project that it goes on test execution on failures, but for asserts it doesn’t work. I’ve seen that there is an interface for configuring failure handling:

But it doesn’t change behavior if you choose “Continue On Failure”, when assert fails, your test case stops running. Is there any way to make soft asserts in Katalon?

Would be very grateful to get any advise!

Marek_Melocik · August 7, 2019, 6:12am

Hello there,

what do you mean by soft assert? What should be expected output?

yuliya.h · August 7, 2019, 6:14am

HI Marek,

By soft assert I mean when my test fails at assert, it goes on working (such behaviour I want to achieve), I mean executing the next steps in my test case after this failed assert till the end. And then show in log failed step. Right now the test just stops executing when assert fails. Let me know, if you need more clarifications.

Marek_Melocik · August 7, 2019, 6:25am

You can try to set CONTINUE_ON_FAILURE globally in Project - Settings menu. But I think this is related only to Katalon’s keywords. When you call assert method, it throws other exception which must be handled separately.

But you can still use condition instead of assert, like this:

if(similar) {
    // do whatever for true result
} else {
    // false action
}

Here you can call KeywordUtil.markFailed("custom message") and thanks to global CONTINUE_ON_FAILURE settings, it’d fail at the end of the test.

ThanhTo · August 7, 2019, 6:30am

Hi @yuliya.h

Maybe the following article would be of interest for you.

You can use SoftAssertions in Katalon Studio’ script mode. I haven’t tried it yet, but it looks promising, so please do try it out.

Cheers !

yuliya.h · August 7, 2019, 6:34am

Thank you Marek,

My project is already configured so that it continues on failures:

But yeah, this assumption I think correct that this applies only to Katalon keywords, since I’m using my own keywords with assert.

I will think about conditions, but actually I don’t like this approach. Anyway, thank you!

yuliya.h · August 7, 2019, 6:38am

Hey Thanh,

Yeah, I found the similar way to use JUnit for soft assert, but decided to ask if there any out-of-the-box method before.

Thank you, looks like I must try this way

discover.selenium · August 7, 2019, 8:25am

import org.testng.asserts.SoftAssert;

SoftAssert softAssertion=new SoftAssert();

// Katalon Code

System.out.println(“Test Step 1”);
softAssertion.assertTrue(true);

System.out.println(“Test Step 2”);
softAssertion.assertTrue(false);

System.out.println(“Test Step 3”);
System.out.println(“Test Step 4”);
softAssertion.assertTrue(false);

System.out.println(“Test Step 5”);
softAssertion.assertEquals(“Google”, “Bing”)

System.out.println(“Test Step 6”);

softAssertion.assertAll();

yuliya.h · August 7, 2019, 8:33am

Thank you discover.selenium! Will try this out!

shanme · November 29, 2023, 9:43pm

is there way to make assertionError log more specific
instead of expected [true] but found [false]
some thing like , expected true on this statement but found false ??

grylion54 · November 30, 2023, 12:06am

How about using WebUI.verifyMatch(first item, second item, false) if it is to do with String comparisons? The “false” is to do with using Wild Card matching, which you likely won’t be using, so set if as false. The verifyMatch just gives you a green colour if it matches and a yellow colour if it doesn’t, assuming you have your project set with the default FailureHandling set at OPTIONAL. Or you can add the FailureHandling.OPTIONAL as the last parameter.

I use verifyEqual for number comparison.
WebUI.verifyEqual(first number, second number)

Edit: And there are also WebUI.verifyNotMatch and WebUI.verifyNotEqual as well.
And you could always try:
softAssertion.assertNotTrue(false);

kazurayam · November 30, 2023, 9:00am

I would use com.kms.katalon.core.util.KeywordUtil class.

Sample1:

import com.kms.katalon.core.util.KeywordUtil
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

KeywordUtil.markFailed("I am down")

for (int i = 0; i < 10; i++) {
	WebUI.delay(1)
	println ("hello" + i)
	
}

This will print in the console:

2023-11-30 17:55:53.372 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2023-11-30 17:55:53.373 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/markFailed
2023-11-30 17:55:53.478 ERROR com.kms.katalon.core.util.KeywordUtil    - ❌ I am down
hello0
hello1
hello2
hello3
hello4
hello5
hello6
hello7
hello8
hello9
2023-11-30 17:56:03.650 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/markFailed FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: I am down
	at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
	at markFailed.run(markFailed:4)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:448)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:439)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:418)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:410)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:285)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:144)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:135)
	at TempTestCase1701334551847.run(TempTestCase1701334551847.groovy:25)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2023-11-30 17:56:03.679 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/markFailed

Please find the helloX messages are printed after the KeywordUtil.markFailed().

Sample2

import com.kms.katalon.core.util.KeywordUtil
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

KeywordUtil.markFailedAndStop("I am down")

for (int i = 0; i < 10; i++) {
	WebUI.delay(1)
	println ("hello" + i)
	
}

When I ran this, I saw

2023-11-30 17:59:18.374 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2023-11-30 17:59:18.375 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/markFailed
2023-11-30 17:59:18.492 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/markFailed FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: I am down
	at com.kms.katalon.core.util.KeywordUtil.markFailedAndStop(KeywordUtil.java:30)
	at markFailed.run(markFailed:4)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:448)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:439)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:418)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:410)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:285)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:144)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:135)
	at TempTestCase1701334756843.run(TempTestCase1701334756843.groovy:25)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2023-11-30 17:59:18.532 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/markFailed

The call to KeywordUtil.markFailedAndStop stopped the test case immediately.

So my soft assert will be something like:

boolean result = myFunction1(args)
if (!result) {
    KeywordUtils.markFailed("so so")
}

result = myFunction2(args)
if (!result) {
    KeywordUtils.markFailed("ha ha")
}

...

shanme · November 30, 2023, 3:14pm

Actually we were tyring to do some thing like this to check data type however user is object but this is failing when we did something like this , could you advise way to check data type on response and have descriptive or more specific message in logs too

def data = response.responseBodyContent

def obj = slurper.parseText(data)
softAssertion.assertTrue obj[‘id’] instanceof Integer

softAssertion.assertTrue obj[‘user’] instanceof Object

kazurayam · November 30, 2023, 11:44pm

I would rewrite this as follows:

import org.assertj.core.api.SoftAssertions

import groovy.json.JsonSlurper

/* https://forum.katalon.com/t/is-it-possible-to-make-soft-assert-in-katalon/31059/13
 */

def assertData(obj) {
	println "start processing data with id:${obj['id']}"
	SoftAssertions softly = new SoftAssertions();

	softly.assertThat(obj['id'])
		.as("id must be a sequence of digits 0,1,2,...9")
		.matches("\\d+")

	softly.assertThat(obj['user'])
		.as("user must be a sequence of word characters")
		.matches("\\w+(\\s+\\w+)*")

	softly.assertAll();
}

def gooddata = '''{
    "id": "123",
    "user": "John Doe"
}'''

def baddata = '''{
    "id": "abc123",
    "user": "James Bond 007"
}'''

def slurper = new JsonSlurper()

assertData(slurper.parseText(gooddata))
assertData(slurper.parseText(baddata))

Output:

2023-12-01 08:43:00.476 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2023-12-01 08:43:00.477 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/MakeSoftAssert
start processing data with id:123
start processing data with id:abc123
2023-12-01 08:43:00.904 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/MakeSoftAssert FAILED.
Reason:
org.assertj.core.api.SoftAssertionError: 
The following 2 assertions failed:
1) [id must be a sequence of digits 0,1,2,...9] 
Expecting:
 "abc123"
to match pattern:
 "\d+"
2) [user must be a sequence of word characters] 
Expecting:
 "James Bond 007"
to match pattern:
 "\w+(\s\w+)?"

	at org.assertj.core.api.SoftAssertions.assertAll(SoftAssertions.java:155)
	at MakeSoftAssert.assertData(MakeSoftAssert:26)
	at MakeSoftAssert.run(MakeSoftAssert:42)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:448)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:439)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:418)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:410)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:285)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:144)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:135)
	at TempTestCase1701387778948.run(TempTestCase1701387778948.groovy:25)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2023-12-01 08:43:00.910 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/MakeSoftAssert FAILED.
Reason:
org.assertj.core.api.SoftAssertionError: 
The following 2 assertions failed:
1) [id must be a sequence of digits 0,1,2,...9] 
Expecting:
 "abc123"
to match pattern:
 "\d+"
2) [user must be a sequence of word characters] 
Expecting:
 "John Doe 007"
to match pattern:
 "\w+(\s+\w+)*"

	at org.assertj.core.api.SoftAssertions.assertAll(SoftAssertions.java:155)
	at MakeSoftAssert.assertData(MakeSoftAssert:26)
	at MakeSoftAssert.run(MakeSoftAssert:42)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:448)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:439)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:418)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:410)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:285)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:144)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:135)
	at TempTestCase1701387778948.run(TempTestCase1701387778948.groovy:25)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2023-12-01 08:43:00.931 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/MakeSoftAssert

For AssertJ matches(), see

Topic		Replies	Views
How to use Soft Assertion Katalon Studio	4	262	October 13, 2023
How to fail a test if the condition is false? Katalon Studio katalon-studio , web-testing	13	6411	June 30, 2020
Katalon Recorder - Continue on Failure Archive katalon-recorder	4	3216	February 16, 2019
How to handle Soft Asserts in katalon Katalon Studio katalon-studio , web-testing	2	236	December 3, 2023
Assertion inside the custom keyword Katalon Studio integrations , katalon-studio , api-testing	3	559	July 12, 2023

Is it possible to make soft assert in Katalon?

Related Topics