Because test case files can’t be more than 800 or 900 lines (I can’t remember the exact number), I’ve divided my test cases to more than 1 file.
For Example, I’ve created a folder and added related testcases under it. TestCase_User (Folder)
1_CreateUser
2_UpdateUser
3_DeleteUser
When running testcases from test suite, if a testcase 2_UpdateUser fails, I want it to Re-run failed test case 1_CreateUser, 2_UpdateUser and 3_DeleteUser (not just 2_UpdateUser). How can I accomplish this?
callTestCase option doesn’t work. For example I’ve tried the following:
Under 4_RunAll, I’ve the following:
WS.callTestCase(findTestCase(‘1_CreateUser’), [:], FailureHandling.CONTINUE_ON_FAILURE)
WS.callTestCase(findTestCase(‘2_UpdateUser’), [:], FailureHandling.CONTINUE_ON_FAILURE)
WS.callTestCase(findTestCase(‘3_DeleteUser’), [:], FailureHandling.CONTINUE_ON_FAILURE)
If I run the testcase 4_RunAll from test suite, it creates, updates and deletes the user like it should; However, after each testcase(1_CreateUser, 2_UpdateUser and 3_DeleteUser) @AfterTestCase listener is not called, I need that listener to be called after create, update and delete user. Listener is only called after 4_RunAll.
For this reason I cannot use callTestCase for this purpose; so how can I Re-run failed testcases like I’ve mentioned above?
With current Katalon capabilities, using callTestCase() is the only way to achieve what you’re trying to accomplish.
4_RunAll needs to be the supervisor. It also needs to perform the code required by the @AfterTextCase for each TC, 1, 2 and 3. So move the @AfterTextCase code into methods in the supervisor (or a helper class).
I think I would wrap each callTestCase() in a try-catch and when a failure occurs, you’ll know then what you need to do to recover and rerun the tests.
Given that I can’t see your AUT, that’s the best I can offer.
import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.context.TestCaseContext
class TL {
@AfterTestCase
def afterTestCase(TestCaseContext testCaseContext) {
onIncident("@AfterTestCase for " + testCaseContext.getTestCaseId())
}
static def void onIncident(String message) {
// you can do anything you like here
println message
}
}
When I ran 4_RunAll, I saw the following messages in the console:
022-03-09 12:58:35.311 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-09 12:58:35.316 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/4_RunAll
2022-03-09 12:58:36.019 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-09 12:58:36.019 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/1_CreateUser
1_CreateUser was invoked
2022-03-09 12:58:36.063 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/1_CreateUser
2022-03-09 12:58:36.063 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-09 12:58:36.167 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-09 12:58:36.168 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/2_UpdateUser
2_UpdateUser was invoked
2022-03-09 12:58:36.240 ERROR com.kms.katalon.core.util.KeywordUtil - ❌ fatal error occured
2022-03-09 12:58:36.260 ERROR c.k.katalon.core.main.TestCaseExecutor - ❌ Test Cases/2_UpdateUser FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: fatal error occured
at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
at 2_UpdateUser.run(2_UpdateUser:5)
... (trimmed the stack trace)
2022-03-09 12:58:36.283 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/2_UpdateUser
2022-03-09 12:58:36.286 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-09 12:58:36.292 ERROR c.k.k.core.keyword.internal.KeywordMain - ❌ Unable to call Test Case 'Test Cases/2_UpdateUser' (Root cause: com.kms.katalon.core.exception.StepFailedException: Call Test Case 'Test Cases/2_UpdateUser' failed
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:63)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
...
Failure in 2_UpdateUser
2022-03-09 12:58:36.390 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-09 12:58:36.390 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/3_DeleteUser
3_DeleteUser was invoked
2022-03-09 12:58:36.416 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/3_DeleteUser
2022-03-09 12:58:36.416 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
@AfterTestCase for Test Cases/4_RunAll
2022-03-09 12:58:36.452 INFO c.k.katalon.core.main.TestCaseExecutor - END Test Cases/4_RunAll
Hi Kazurayam,
Thank you for a detailed response. This does not solve my problem. I need listener to be called after 1_CreateUser, 2_UpdateUser, 3_DeleteUser. In your example I’ve added the following print line in TL.
“I am here” should be printed 4 times but it’s only getting printed once. Listener has to be called regardless of testcase passes or fails.
Thanks
@AfterTestCase
def afterTestCase(TestCaseContext testCaseContext)
{
System.out.println(“I am here”)
onIncident("@AfterTestCase for " + testCaseContext.getTestCaseId())
}
Hi,
Thanks for a quick response. I can’t use Helper class or move the code to Supervisor method because the listener has some functions that I do not see outside of that class, for example TestCaseContext(this is just an example of one method that I need Listener for, there are multiple other methods that I use which are only available in Listener)
I don’t understand why Listener isn’t called after each test case, all of these are test cases and the listener is @AfterTestCase.
import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.annotation.BeforeTestCase
import com.kms.katalon.core.context.TestCaseContext
class TL {
private static Stack<TestCaseContext> tccStack = new Stack<>()
@BeforeTestCase
def beforeTestCase(TestCaseContext testCaseContext) {
tccStack.push(testCaseContext)
}
@AfterTestCase
def afterTestCase(TestCaseContext testCaseContext) {
if (!tccStack.empty()) {
TestCaseContext tcc = tccStack.pop()
onIncident("@AfterTestCase for " + tcc.getTestCaseId())
}
}
static def void onIncident(String message) {
// you can do anything you like here
if (!tccStack.empty()) {
TestCaseContext tcc = tccStack.peek()
println "[" + tcc.getTestCaseId() + "] " + message
}
}
}
When I ran the 4_RunAll, I got the following output in the console.
2022-03-10 09:50:13.592 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-10 09:50:13.596 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/4_RunAll
2022-03-10 09:50:14.331 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-10 09:50:14.332 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/1_CreateUser
1_CreateUser was invoked
2022-03-10 09:50:14.376 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/1_CreateUser
2022-03-10 09:50:14.376 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
[Test Cases/4_RunAll] has finished
2022-03-10 09:50:14.538 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-10 09:50:14.538 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/2_UpdateUser
2_UpdateUser was invoked
2022-03-10 09:50:14.606 ERROR com.kms.katalon.core.util.KeywordUtil - ❌ fatal error occured
2022-03-10 09:50:14.636 ERROR c.k.katalon.core.main.TestCaseExecutor - ❌ Test Cases/2_UpdateUser FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: fatal error occured
at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
at 2_UpdateUser.run(2_UpdateUser:5)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:442)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:433)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:412)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:404)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:281)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:350)
at 4_RunAll.run(4_RunAll:15)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:442)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:433)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:412)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:404)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:281)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1646873408911.run(TempTestCase1646873408911.groovy:25)
2022-03-10 09:50:14.656 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/2_UpdateUser
2022-03-10 09:50:14.659 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-10 09:50:14.664 ERROR c.k.k.core.keyword.internal.KeywordMain - ❌ Unable to call Test Case 'Test Cases/2_UpdateUser' (Root cause: com.kms.katalon.core.exception.StepFailedException: Call Test Case 'Test Cases/2_UpdateUser' failed
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:63)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.call(CallTestCaseKeyword.groovy)
at com.kms.katalon.core.keyword.internal.KeywordMain.runKeyword(KeywordMain.groovy:74)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.callTestCase(CallTestCaseKeyword.groovy:81)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword.execute(CallTestCaseKeyword.groovy:44)
at com.kms.katalon.core.keyword.internal.KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.groovy:74)
at com.kms.katalon.core.keyword.BuiltinKeywords.callTestCase(BuiltinKeywords.groovy:350)
at 4_RunAll.run(4_RunAll:15)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:442)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:433)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:412)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:404)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:281)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at TempTestCase1646873408911.run(TempTestCase1646873408911.groovy:25)
Caused by: com.kms.katalon.core.exception.StepFailedException: fatal error occured
at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
at 2_UpdateUser.run(2_UpdateUser:5)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:442)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:433)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:412)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:404)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:281)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)
at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:59)
... 18 more
)
[Test Cases/4_RunAll] has finished
2022-03-10 09:50:14.770 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-03-10 09:50:14.771 INFO c.k.katalon.core.main.TestCaseExecutor - CALL Test Cases/3_DeleteUser
3_DeleteUser was invoked
2022-03-10 09:50:14.800 INFO c.k.katalon.core.main.TestCaseExecutor - END CALL Test Cases/3_DeleteUser
2022-03-10 09:50:14.800 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
[Test Cases/4_RunAll] has finished
2022-03-10 09:50:14.818 INFO c.k.katalon.core.main.TestCaseExecutor - END Test Cases/4_RunAll
Let me review your comments.
OK. My demo does it.
OK. You need a Stack. The TestListener of my demo does it. You can add more Stacks for other “functions” private to the Test Listener as many as you would like.
I can explain it. You need to know how “Groovy Script” runs behind Katalon Studio. Please have a look at the official Groovy Language documentation to understand what’s going on behind the scene.
When you run the Test Case 4_RunAll, the following processing is fired.
the script sources of 4_RunAlll + 1_CreateUser + 2_UpdateUser + 3_DeleteUser are dynamically merged to form a Groovy Class. The class would be concepturally something like this:
import org.codehaus.groovy.runtime.InvokerHelper
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import com.kms.katalon.core.exception.StepFailedException
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
class Main extends Script {
def run() {
try {
// the code of "1_CreateUser" comes here
} catch (StepFailedException e) {
// ignore
} finally {
TL.onIncident("has finished")
}
try {
// the code of "2_UpdateUser" comes here
} catch (StepFailedException e) {
// ignore
} finally {
TL.onIncident("has finished")
}
try {
// the code of "3_DeleteUser" comes here
} catch (StepFailedException e) {
// ignore
} finally {
TL.onIncident("has finished")
}
}
static void main(String[] args) {
InvokerHelper.runScript(Main, args)
}
}
Let me call this Groovy class as Main class derived from Test Cases sources.
Katalon Studio execute the Goovy runtime specifying the Main class. The Main#main() is invoked, which invokes the Main#run(). The run() is written, as you see above, to evaluate the combined lines written in 1_CreateUser + 2_UpdateUser + 3_DeleteUser files.
Katalon Studio calls the @AfterTestCase-annotated method of the Test Listener when the Main#main() finished.
The @AfterTestCase-annotated method has no chance to be called after 1_CreateUser + 2_UpdateUser + 3_DeleteUser source codes.
Quod Erat Demonstrandum
By the way, I would agree that the name @AfterTestCase is not describing what it actually does. I suppose, it should rather be named @AfterMain_run. But obviously people would not understand what @AfterMain_run means.
You thought that you can workaround the “Method code too large” problem by splitting a single large Test Cases into smaller pieces and re-assembling them by callTestCase().
But in fact, no, you can’t. The “Method code too large” error occurs when the run() method of the Main class derived from 4_RunAll will become larger than the JVM limitation (64K bytes) in size of the compiled JVM byte code. AFAIK, the run() becomes as large as the sum of component sources (1_CreateUser + 2_UpdateUser + 3_DeleteUser).
Q. How can you definitely workaround the “Method code too large” problem?
A1. You can make a Test Suite which combines the 3 Test Cases. If it is favourable for you and you can avoid using callTestCases(), it is the easiest.
A2. Or You want to rewrite the source codes of 1_CreateUser, 2_UpdateUser, 3_DeleteUser to be more compact. How to? — I don’t see, as it depends how your codes are written.
I have used an approach very similar to what you are doing. How I handled recovery was to use a csv file that contained when a call to a testcase from the main testcase was made with something like:
testcase1,started
Then I would update that entry with:
testcase1,completed
Only if no error occurred.
That way my main testcase could recover from any failed called testcase by the main testcase reading the csv and skipping those that are ‘completed’
I have had to split our testcases for the same reason being to big in one testcase.
This is a simplistic example based on what we do, but it is powerful in that we can run it hierarchically without issue and full recoverability.
Thank you so much for your help, unfortunately this does not solve my problem. I need @AfterTestCase to be called after each TC_1, TC_2 and TC_3(regardless of test case is passing or failing) but you mentioned above it will only be called after mainclass.
Hello again,
I’ve followed what has been proposed in V0.2.1. I’ve copied my code under onIncident() method.
Here’s the short version of onIncident() method.
static def void onIncident()
{
// you can do anything you like here
if (!tccStack.empty())
{
TestCaseContext testCaseContext = tccStack.peek()
HashMap<ResponseObject, ResponseObject> responsedata = new HashMap<>();
responsedata = testCaseContext.getTestCaseVariables()
}
}
My question is, if I have the testCaseContext info here, then why is responsedata always empty? Test case variables have been defined, that’s not the issue. Please help.
This code suggests to me that you expect (hope? require? believe?) testCaseContext.getTestCaseVariable() will return an object of type Map<ResponseObject, ResponseObject>.
No. Katalon Studio will never set an instance of ResponseObject into the TestCaseContext object automatically.
And your TestCase script has no chance to set any value into the TestCaseContext object explicitly.
If you want to pass any type of data from Test Case A to Test Case B, you need to use either of
a GlobalVariable
a public static property of appropriate type declared a TestListener
local file on disk
In the original post you wrote:
Your original code was too large to be a single test case so that you splitted the original code into 3 test cases, then you have got another aspect of difficulties. I would suggest to you going back to the original code and make it fit so that you can do 3 processings in a single Test Case.
If I were you, I will try to find a code fragment that repeats many times in a Test Case. I will extract the fragment and make it a method of external Groovy class. The Groovy class file can be located under <projectDir>/Keywords dir or <projectDir>/Include/scripts/groovy dir. I will change the TestCase script to call the external classes (methods). Hopefully by such refactoring, the number of lines of Test Case decreases. — I do not see how much feasible it is in your case.
Large codes as Test Case script tend to make your project difficult to manage. (Unfortunately “Recorder” tools tends to generate gigantic test case scripts). It would worth trying for you to review+rewrite your test cases to make them as slim & fit as possible. You should refactor test cases well modularised with minimum repetition of codes.
Hi Kazurayam,
Thank you for assisting me. The TestCaseContext that we are pushing on stack in @BeforeTestCase, is TestCaseContext for RunAll testcase or for CreateUser/UpdateUser/DeleteUser?
Katalon Studio invokes @BeforeTestCase-annoteded method in a TestListener for ALL of Test Cases. And my sample code does not chose. It pushes all of the TestCaseContext instance of 4 Test Cases: “RunAll” + “CreateUser” + “UpdateUser” + “DeleteUser”. i was wrong. when i wrote this sentese, i was blank. i did not remember the previous posts at all.
But you can change the behaviour as you want. If you want to push the TestCaseContext of CreateUser only, then you can achieve it by a if statement inside the @BeforeTestCase-annotated method; for example.
Thanks Kazurayam.
What you have suggested here is exactly what I need but it does not work. CreateUser is never pushed on stack, only RunAll is pushed on stack.
