Get number of runs when retrying failed cases

How can I get the number of a TC’s run when running test suites via command line with ‘retryFailedTestCases=true’? By code?

Nothing? Let me elaborate a bit.

Command line output has the following line:

Re-run test suite: Test Suites/TS name reports # 3.

So, this information obviously is known to Katalon.

Does anyone know how to extract it? Where could it be written?


I am going to tell you what I have done.

Yesterday I updated one of my demo project published on GitHub:

This project has a Test Suite named ‘Test Suites/TS1_screenshot_taken_when_failed’. I changed the config of the Test Suite so that it retries failed test cases.

Then I executed the Test Suite. It invoked the Test Case ‘TC1_screenshot_taken_when_failed’. This test case is coded to fail always. I could see the Test Suite certainly re-ran the Test Case for 2 times.

Then I checked the log in the Reports folder looking for some messages which describes how the Test Suite re-ran the Test Case. But I could not find any messages like “Re-run test suite: Test Suites/TS name reports # 3.”

Rather the log messages impressed me that Test Suite is not designed to log anything about the re-run. The log messages looked just the same as the run without test case re-run at all.

So, I have a doubt about your previous comment. You said you saw a message in the Command Line output:

Re-run test suite: Test Suites/TS name reports # 3.

I wonder if this message was emitted by Katalon Studio or by your custom Test Case.


I played on. In the

I did some research.

I added 2 GlobalVariables.

- NUMBER_OF_TESTCASE_RUNS of Map<String, Number> type

I added a TestListener as follows:

import java.nio.file.Filesimport java.nio.file.Pathimport java.nio.file.Pathsimport com.kms.katalon.core.annotation.BeforeTestCaseimport com.kms.katalon.core.annotation.BeforeTestSuiteimport com.kms.katalon.core.configuration.RunConfigurationimport com.kms.katalon.core.context.TestCaseContextimport com.kms.katalon.core.context.TestSuiteContextimport com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUIimport internal.GlobalVariable as GlobalVariableclass MyTestListener {		/**	 * Executes before every test case starts.	 * @param testCaseContext related information of the executed test case.	 */	@BeforeTestCase	def beforeTestCase(TestCaseContext testCaseContext) {		String testCaseId = testCaseContext.getTestCaseId()		WebUI.comment(">>> testCaseId is ${testCaseId}")		GlobalVariable.CURRENT_TESTCASE_ID = testCaseId		Map runsRecord = (Map)GlobalVariable.NUMBER_OF_TESTCASE_RUNS		if (runsRecord.containsKey(testCaseId)) {			WebUI.comment(">>> runRecord contains key ${testCaseId}")			Number count = runsRecord.get(testCaseId)			runsRecord.put(testCaseId, count + 1)		} else {			WebUI.comment(">>> runRecrod does not contain key ${testCaseId}")			runsRecord.put(testCaseId, 1)		}		GlobalVariable.NUMBER_OF_TESTCASE_RUNS = runsRecord	}}

And I changed the Test Case ''TC1_screenshot_taken_when_failed"

import static com.kms.katalon.core.testobject.ObjectRepository.findTestObjectimport java.nio.file.Filesimport java.nio.file.Pathimport java.nio.file.Pathsimport com.kms.katalon.core.configuration.RunConfigurationimport com.kms.katalon.core.model.FailureHandling as FailureHandlingimport com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUIimport internal.GlobalVariable as GlobalVariableString nth(Number v) {	if (v == 1) {		return '1st'	} else if (v == 2) {		return '2nd'	} else if (v == 3) {		return '3rd'	} else {		return "${v}th"	}}String testCaseId = (String)GlobalVariable.CURRENT_TESTCASE_IDMap runRecord = (Map)GlobalVariable.NUMBER_OF_TESTCASE_RUNSNumber runs = runRecord.get(testCaseId)String message = ">>> ${testCaseId} ran for the ${nth(runs)} time"WebUI.comment(message)Path messagePath = Paths.get(RunConfiguration.getProjectDir()).resolve("./message.txt")WebUI.comment(">>> ${messagePath.toString()}")File messageFile = messagePath.toFile()messageFile.append("${message}\n")...

Finally I changed the Test Suite ‘TS1_screenshot_taken_when_failed’ to re-run the test case 2 times.

# What I expected

I expected the message.txt file wil be as this:

>>> Test Cases/TC1_screenshot_taken_when_failed ran for the 1st time>>> Test Cases/TC1_screenshot_taken_when_failed ran for the 2nd time>>> Test Cases/TC1_screenshot_taken_when_failed ran for the 3rd time

# What I actually got**

I executed the Test Suite. It resulted “./message.txt” as follows:

>>> Test Cases/TC1_screenshot_taken_when_failed ran for the 1st time

# What I found**

I actually saw the Test Suite executed the Test Case which failed, and the Test Case re-run 2 times.

Therefore I thought the handler method annotated as BeforeTestCase in the MyTestListener class would have been called 3 times. If the handler method is called multiple times including re-run, then I would be able to count the number of runs myself. I thought this would be informative for you, Mate.

However, to my surprise, the handler method annotated as BeforeTestCase seems to be called by Katalon only once. I am not yet confident enough that I understand this behavior.

Now I know at least that I can not count the number of re-runs myself.

Thanks, @4280-kazurayam!

This is very informative. I am writing some kind of a test reporting code, but I thought it would be much simpler if this info already exists.

And about the #runs message - it is not emitted by my test case. It is shown when a TS is re-run inside of a Test Suite Collection.

I tried to reproduce the ‘#Re-run test suite’ message in my project.

I added a Test Suite Collection and a bat file to execute Katalon in console mode:

When I execute the bat, yes, I saw the message:

I learned a lot, thank you.


I guess there seem to be no API which allows you to get access to ‘#1’ and ‘#2’. No reasoning I have. Just I feel.


I was able to count the number of test suite runs with this un-elegant piece of code:

def reporterAfterTestSuite(TestSuiteContext testSuiteContext) {
WebUI.comment('>>>>>>>>>>>>>>>>>>> EmailReporter <<<<<<<<<<<<<<<<<<<<<')
if (GlobalVariable.testSuiteStatus=='FAILED'){
String counterFile = RunConfiguration.getProjectDir() + '/Data Files/' + 'tsFailCounter.txt'
File tsFailCounter = new File(counterFile)
int counter = tsFailCounter.text.toInteger()
Date zavrsetak = new Date()
String endTime = RunConfiguration.getProjectDir() + '/Data Files/' + 'tsLastEndTime.txt'
File tsLastEndTime = new File(endTime)
String lastSuite = RunConfiguration.getProjectDir() + '/Data Files/' + 'tsLastExecutedTS.txt'
File tsLastExecutedTS = new File(lastSuite)
def string = RunConfiguration.getLogFolderPath()
println string
def list1 = string.tokenize('\\')
println list1
println list1[list1.size()-2]

The problem is, now I cannot run other test suites in parallel, otherwise my counter is reset.