How can I get the number of a TC’s run when running test suites via command line with ‘retryFailedTestCases=true’? By code?
Nothing? Let me elaborate a bit.
Command line output has the following line:
Re-run test suite: Test Suites/TS name reports # 3.
So, this information obviously is known to Katalon.
Does anyone know how to extract it? Where could it be written?
Mate,
I am going to tell you what I have done.
Yesterday I updated one of my demo project published on GitHub:
This project has a Test Suite named ‘Test Suites/TS1_screenshot_taken_when_failed’. I changed the config of the Test Suite so that it retries failed test cases.
Then I executed the Test Suite. It invoked the Test Case ‘TC1_screenshot_taken_when_failed’. This test case is coded to fail always. I could see the Test Suite certainly re-ran the Test Case for 2 times.
Then I checked the log in the Reports folder looking for some messages which describes how the Test Suite re-ran the Test Case. But I could not find any messages like “Re-run test suite: Test Suites/TS name reports # 3.”
Rather the log messages impressed me that Test Suite is not designed to log anything about the re-run. The log messages looked just the same as the run without test case re-run at all.
So, I have a doubt about your previous comment. You said you saw a message in the Command Line output:
Re-run test suite: Test Suites/TS name reports # 3.
I wonder if this message was emitted by Katalon Studio or by your custom Test Case.
I played on. In the
I did some research.
I added 2 GlobalVariables.
- CURRENT_TESTCASE_NAME of String type
- NUMBER_OF_TESTCASE_RUNS of Map<String, Number> type
I added a TestListener as follows:
import java.nio.file.Filesimport java.nio.file.Pathimport java.nio.file.Pathsimport com.kms.katalon.core.annotation.BeforeTestCaseimport com.kms.katalon.core.annotation.BeforeTestSuiteimport com.kms.katalon.core.configuration.RunConfigurationimport com.kms.katalon.core.context.TestCaseContextimport com.kms.katalon.core.context.TestSuiteContextimport com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUIimport internal.GlobalVariable as GlobalVariableclass MyTestListener { /** * Executes before every test case starts. * @param testCaseContext related information of the executed test case. */ @BeforeTestCase def beforeTestCase(TestCaseContext testCaseContext) { String testCaseId = testCaseContext.getTestCaseId() WebUI.comment(">>> testCaseId is ${testCaseId}") GlobalVariable.CURRENT_TESTCASE_ID = testCaseId Map runsRecord = (Map)GlobalVariable.NUMBER_OF_TESTCASE_RUNS if (runsRecord.containsKey(testCaseId)) { WebUI.comment(">>> runRecord contains key ${testCaseId}") Number count = runsRecord.get(testCaseId) runsRecord.put(testCaseId, count + 1) } else { WebUI.comment(">>> runRecrod does not contain key ${testCaseId}") runsRecord.put(testCaseId, 1) } GlobalVariable.NUMBER_OF_TESTCASE_RUNS = runsRecord }}
And I changed the Test Case ''TC1_screenshot_taken_when_failed"
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObjectimport java.nio.file.Filesimport java.nio.file.Pathimport java.nio.file.Pathsimport com.kms.katalon.core.configuration.RunConfigurationimport com.kms.katalon.core.model.FailureHandling as FailureHandlingimport com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUIimport internal.GlobalVariable as GlobalVariableString nth(Number v) { if (v == 1) { return '1st' } else if (v == 2) { return '2nd' } else if (v == 3) { return '3rd' } else { return "${v}th" }}String testCaseId = (String)GlobalVariable.CURRENT_TESTCASE_IDMap runRecord = (Map)GlobalVariable.NUMBER_OF_TESTCASE_RUNSNumber runs = runRecord.get(testCaseId)String message = ">>> ${testCaseId} ran for the ${nth(runs)} time"WebUI.comment(message)Path messagePath = Paths.get(RunConfiguration.getProjectDir()).resolve("./message.txt")WebUI.comment(">>> ${messagePath.toString()}")File messageFile = messagePath.toFile()messageFile.append("${message}\n")...
Finally I changed the Test Suite ‘TS1_screenshot_taken_when_failed’ to re-run the test case 2 times.
# What I expected
I expected the message.txt file wil be as this:
>>> Test Cases/TC1_screenshot_taken_when_failed ran for the 1st time>>> Test Cases/TC1_screenshot_taken_when_failed ran for the 2nd time>>> Test Cases/TC1_screenshot_taken_when_failed ran for the 3rd time
**
# What I actually got**
I executed the Test Suite. It resulted “./message.txt” as follows:
>>> Test Cases/TC1_screenshot_taken_when_failed ran for the 1st time
**
# What I found**
I actually saw the Test Suite executed the Test Case which failed, and the Test Case re-run 2 times.
Therefore I thought the handler method annotated as BeforeTestCase in the MyTestListener class would have been called 3 times. If the handler method is called multiple times including re-run, then I would be able to count the number of runs myself. I thought this would be informative for you, Mate.
However, to my surprise, the handler method annotated as BeforeTestCase seems to be called by Katalon only once. I am not yet confident enough that I understand this behavior.
Now I know at least that I can not count the number of re-runs myself.
Thanks, @4280-kazurayam!
This is very informative. I am writing some kind of a test reporting code, but I thought it would be much simpler if this info already exists.
And about the #runs message - it is not emitted by my test case. It is shown when a TS is re-run inside of a Test Suite Collection.
I tried to reproduce the ‘#Re-run test suite’ message in my project.
I added a Test Suite Collection and a bat file to execute Katalon in console mode:
When I execute the bat, yes, I saw the message: 
I learned a lot, thank you.
-------
I guess there seem to be no API which allows you to get access to ‘#1’ and ‘#2’. No reasoning I have. Just I feel.
I was able to count the number of test suite runs with this un-elegant piece of code:
@AfterTestSuite
def reporterAfterTestSuite(TestSuiteContext testSuiteContext) {
WebUI.comment('>>>>>>>>>>>>>>>>>>> EmailReporter <<<<<<<<<<<<<<<<<<<<<')
if (GlobalVariable.testSuiteStatus=='FAILED'){
String counterFile = RunConfiguration.getProjectDir() + '/Data Files/' + 'tsFailCounter.txt'
File tsFailCounter = new File(counterFile)
int counter = tsFailCounter.text.toInteger()
tsFailCounter.write((counter+1).toString())
Date zavrsetak = new Date()
String endTime = RunConfiguration.getProjectDir() + '/Data Files/' + 'tsLastEndTime.txt'
File tsLastEndTime = new File(endTime)
tsLastEndTime.write(zavrsetak.toString())
String lastSuite = RunConfiguration.getProjectDir() + '/Data Files/' + 'tsLastExecutedTS.txt'
File tsLastExecutedTS = new File(lastSuite)
def string = RunConfiguration.getLogFolderPath()
println string
def list1 = string.tokenize('\\')
println list1
println list1[list1.size()-2]
tsLastExecutedTS.write(list1[list1.size()-2])
}
}
The problem is, now I cannot run other test suites in parallel, otherwise my counter is reset.