ReportUtilDriver --- how to generate test reports by user code in Katalon Studio

Problem to solve

When Katalon Studio generates a bunch of test reports (HTML, XML, CSV) ? How KS does it?

Let me elaborate this question with a demo project.

import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

WebUI.openBrowser('')
WebUI.navigateToUrl('https://katalon-demo-cura.herokuapp.com/profile.php#login')
WebUI.setText(findTestObject('Login/Page_CURA Healthcare Service_Login/input_Username_username'), 'John Doe')
WebUI.setEncryptedText(findTestObject('Login/Page_CURA Healthcare Service_Login/input_Password_password'), 'g3/DOGG74jC3Flrr3yH+3D/yKbOqqUNM')
WebUI.click(findTestObject('Login/Page_CURA Healthcare Service_Login/button_Login'))
WebUI.closeBrowser()
  • I created a Test Suite [ main/TS1 ](Test Suites/main/TS1.ts), which just executes the TC1 .
  • I created a Test Listener [ TL_listBunchDir ](Test Listeners/TL_listBunchDir.groovy):
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths

import com.kms.katalon.core.annotation.AfterTestSuite
import com.kms.katalon.core.configuration.RunConfiguration
import com.kms.katalon.core.context.TestSuiteContext

class TL_listBunchDir {

	@AfterTestSuite
	def sampleAfterTestSuite(TestSuiteContext testSuiteContext) {
		Path projectDir = Paths.get(RunConfiguration.getProjectDir())
		Path bunchDir = Paths.get(RunConfiguration.getReportFolder())
		Files.list(bunchDir).sorted().each ({ Path p ->
			println projectDir.relativize(p).toString()
		})
	}
}

The TL_listBunchDir prints a list of files in a directory where HTML/XML reports should be written into.

When I execute the TS1 , I got the following output in the Console.

2021-11-21 14:03:26.017 DEBUG testcase.TC1                             - 6: closeBrowser()
2021-11-21 14:03:26.226 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/main/TC1
Reports/20211121_140311/main/TS1/20211121_140311/execution.properties
Reports/20211121_140311/main/TS1/20211121_140311/execution0.log
Reports/20211121_140311/main/TS1/20211121_140311/execution0.log.lck
Reports/20211121_140311/main/TS1/20211121_140311/testCaseBinding
Reports/20211121_140311/main/TS1/20211121_140311/tsc_id.txt
2021-11-21 14:03:26.957 INFO  com.kms.katalon.core.util.KeywordUtil    - Start generating HTML report folder at: /Users/kazuakiurayama/katalon-workspace/ReportUtilDriver/Reports/20211121_140311/main/TS1/20211121_140311...
2021-11-21 14:03:26.995 INFO  com.kms.katalon.core.util.KeywordUtil    - HTML report generated
2021-11-21 14:03:26.997 INFO  com.kms.katalon.core.util.KeywordUtil    - Start generating CSV report folder at: /Users/kazuakiurayama/katalon-workspace/ReportUtilDriver/Reports/20211121_140311/main/TS1/20211121_140311...
2021-11-21 14:03:27.002 INFO  com.kms.katalon.core.util.KeywordUtil    - CSV report generated
2021-11-21 14:03:27.154 INFO  c.k.katalon.core.main.TestSuiteExecutor  - --------------------
2021-11-21 14:03:27.154 INFO  c.k.katalon.core.main.TestSuiteExecutor  - END Test Suites/main/TS1
2021-11-21 14:03:27.154 INFO  c.k.katalon.core.main.TestSuiteExecutor  - ====================

Once everything done, I looked at the reports directory and found this:

list_bunch_after_gen

Please remark the 2 points of interest.

  1. When @AfterTestSuite -annotated method of TL_listBunchDir ran, there were not the report files (HTML,XML,CSV). At that time, you can only find files execution0.log etc.
  2. As you can see in the Console log, Katalon Studio emitted message that told KS generated reports in HTML and CSV.

Now I got the answer to my 1st question: when Katalon Studio generates a bunch of test reports (HTML, XML, CSV)?

When the @AfterTestSuite -annotated method of TestListener was invoked, the HTML report was not there yet. Katalon Studio generates it sometime after my Test Suite execution has completely finished.

But my 2nd question is still open: How Katalon Studio generetes a HTML report?

I have investigated the source code of Katalon Studio, and have got an answer to it.

Answer

Katalon Studio API contains a class com.kms.katalon.core.reporting.ReportUtil . It implements entry-point methods that transform the execution0.log file into the various format of reports.

If you write a custom Groovy script that drives the ReportUtil class appropriately, then your custom script can generate the test result reports in HTML/XML and write them into any location. Your Test Listener can generate the reports. It can write them outside the project, or can zip them and send it to anywhere you want.

Please read the source for the answer.

Test Listener

Please read Test Listeners/TL_generateReports

The @AfterTestSuite-annotated method will be invoked when a Test Suite finished running. It will call my keyword ReportUtilDriver .

Keyword

My keyword class ReportUtilDriver will call com.kms.katalon.core.reporting.ReportUtil to generate reports.

ReporrttUtilDriver#generateBunches() method scans the Reports directory of the project, and saves the new HTML files into /Users/[myName]/tmp/katalon-reports directory. If the Reports dicretory contains multiple results of Test Suite execution, say 3 sets, then generateBunches() finds all 3 sets and re-generate reports and saves them into the output directory.

Please read the source code at com.kazurayam.ks.reporting.ReportUtilDriver for detail.

Demo

You want to execute Test Suites/main/TS1 . When it finished, visit /Users/[yourName]/tmp/katalon-reports directory using Windows Explorer or Mac Finder.

output

There you will find HTML/XML/CSV reports have been re-generated there. These files were generated by com.kms.katalon.core.reporting.ReportUtil which was driven my custom keyword and a TestListener.

Conclusion

com.kazurayam.ks.reporting.ReportUtilDriver shows you how a custom Groovy script can re-generate the Katalon built-in format reports.

… and customize them to your heart’s content.

@kazurayam I think you know I do something quite similar except I don’t mess around with Katalon’s output files at all.

I have a wrapper method around KeywordUtil.markWarning which saves my output to two files: An individual TC report and an accumulated report file for a suite (only present during suite runs). One is plain text, the other JSON format. The benefit is, I am in complete control of the text in the messages and complete control of the report:

wam 'My fantastic message'  // becomes a warning message

wam = writeAUTMessage().

Because they are warnings, I can control their display in the log view like we discussed elsewhere.

@Russ_Thomas

I think you know, many people want to mess around with Katalon’s output files. They often ask us how to customise it. Personally I do not need it at all, but am interested how to customise it using Groovy’s Meta-programming technique just for fun.

Before playing on customisation, I needed to find out how to generate the built-in reports by my custom script. So I have solve the prerequisite.

I didn’t mean to imply that was a bad thing – far from it, I think what you achieved was very good!

I know. :slight_smile: