Hello all,
I have specific needs for my project (make summary reports with only the errors details in the additional pages) and I wanted to customize the reports generation to reach my goals.
When I tried to compile the report plugin on my machine, Gradle failed while downloading the “com.katalon.gradle-plugin” version “0.1.1” Gradle plugin.
Does someone know where to get this plugin or have an alternative solution ?
Thanks in advance and kind regards
Hi there,
Thank you very much for your topic. Please note that it may take a little while before a member of our community or from Katalon team responds to you.
Thanks!
I tried reproducing what you tried. I also experienced a lot of problems.
I did “git clone GitHub - katalon-studio/katalon-studio-basic-report-plugin” on my machine.
I cd into the project, I did “gradle katalonCopyDependencies”. Then I got a series of errors.
In short, I realized that
I initially used Gradle 8.5, but the project does not work with it. So I had to switch to Gradle 7.x.
I initially used JDK21 to run Gradle, but Gradle 7.6 does not work on JDK21. So I had to switch to JDK17.
then I could run “gradle katalonCopyDependencies” successfully. I opened the project with Katalon Studio 9.0.
Then I found that a Groovy source code of the “Basic Report Plugin” had a compilation error (syntax error). Wow! It requires a lot of works to build the plugin.
I think that it is a poor idea to try to customize the “Basic Report” which was NOT designed for customization.
When you run a Test Suite in Katalon Studio, the test runner in KS serializes all the logs into a file named execution0.log in the report directory of the test suite.
When it finished running the Test Suite, KS will call com.kms.katalon.core.reporting.ReportUtil#writeLogRecordToFiles(String logFolder). This method will read a large XML document out of execution0.log file, transform it into files in various formats: HTML, CSV, PDF, etc.
The ReportUtil class is a bunch of Groovy code that transforms a XML document into a HTML document using JasperReports framework. The katalonCopyDependency Gradle task is required to resolve the extenal dependencies required by JasperReports.
I have ever looked into this XML → HTML transformation processing in the ReportUtil class. If you are going to customize this transformation processing, you have to start with looking at the data structure of the XML document in execution0.log file. I am sure, Katalon has published no document about execution0.log. You have to find out all the secrets about execution0.log for yourself without any help. It will be too damn hard.
The “Basic Report Plugin” contains a copy of com.kms.katalon.core.reporting.ReportUtil and related classes. I do not understand why Katalon made this module and published it as a Plugin. I don’t see what they want users to do with the plugin. It seems to me that the plugin just increased code duplication, which all developers should try to eliminate.
It is fairely easy to write a TestListener class in Katalon project in Groovy language. A TestListener can compile a summary report in your custom format.
For example, have a look at my previous project: Custom Test Report compiled by TestListener (kazurayam-ks-plainreport)
The TLPlainReport will give you an idea what you should write.
import com.kazurayam.ks.plainreport.ReportGenerator
import com.kms.katalon.core.annotation.AfterTestCase
import com.kms.katalon.core.annotation.AfterTestSuite
import com.kms.katalon.core.annotation.BeforeTestCase
import com.kms.katalon.core.annotation.BeforeTestSuite
import com.kms.katalon.core.context.TestCaseContext
import com.kms.katalon.core.context.TestSuiteContext
import com.kms.katalon.core.configuration.RunConfiguration
import java.nio.file.Path
import java.nio.file.Paths
class TLPlainReport {
private ReportGenerator reportgen
TLPlainReport() {
reportgen = new ReportGenerator()
reportgen.setOutputDir("./PlainReport")
}
@BeforeTestSuite
def beforeTestSuite(TestSuiteContext testSuiteContext) {
reportgen.beforeTestSuite(testSuiteContext)
}
@BeforeTestCase
def beforeTestCase(TestCaseContext testCaseContext) {
reportgen.beforeTestCase(testCaseContext)
}
@AfterTestCase
def afterTestCase(TestCaseContext testCaseContext) {
reportgen.afterTestCase(testCaseContext)
}
@AfterTestSuite
def afterTestSuite(TestSuiteContext testSuiteContext) {
reportgen.afterTestSuite(testSuiteContext)
}
}
This TestListener redirects the event stream originated by Katalon’s Test Runner into my reporting class com.kazurayam.ks.plainreport.ReportGenerator. Once my reporting object is notified of @AfterTestXXXX events, then my reporting object can generate a file with error details in any format I’d like.
TestListener is the gearbox of custom reporting mechanism. Easy, isn’t it?
You should create a new TestListener and a reporting class that meets your specific needs.
Here is a recent development by mohit.kumar: ExtentReports + Katalon
I have tried it. It just worked. It generated a report like this:
This looks nice, doesn’t it? mohit.kumar developed his own TestListener in KS and a reporting class on top of ExtentReports. That’s the right way.
You would be able to make a twist of ExtentReports + Katalon combination for your need.
Hi [kazurayam]
Thank you very much for your investigations and your ideas.
Actually, I thought as you that using the TestSuiteContext would be sufficient.
However, my main goal was to highlight the failures and their cause.
I will take a sight on your work, that’s very kind of you to share it.
And your suggestions might already help me to reach my goals, again thanks for them.
Kind regards
See TestCaseContext (Katalon Studio API Specification)
The TestCaseContext class implements several methods that inform you of error detail.
Your reporting module can print these information in your report.
Do you want to print more information that high light the failure and their cause? Do you want to include all of messages emitted by WebUI.comment(String) keyword? Well, it is impossible in Katalon Studio. Katalon Studio serializes all logs into execution0.log file. KS won’t share the result of WebUI.comment(String msg) to your TestListeners.
If you want the detail logs, you have to parse the XML and look up what you want. That task is difficult enough. Additinally, there is a timing problem, which is even more difficult to overcome. The XML document in execution0.log file will be incomplete while the TestListener is running; the document will be finalized and ready to read for your TestListener only @AfterTestSuite.
If you can switch to TestNG to develop your automated Web UI test suites, TestNG enables you to do more. TestNG provide its own TestListener interface which transports versatile information from a test class to a test listener. The following article describes what is possible in the combination of WebDriver + TestNG + ExtentReports.
https://www.swtestacademy.com/extent-reports-in-selenium-with-testng/
So, I mean, in Katalon Studio, you are limited. All your TestListerner in KS can know is the information carried by TestSuiteContext and TestCaseContext. Ah, I forgot to mention, your TestListener can refer to the GlobalVariables and DriverFactory.getWebDriver() statically. KS does not allow you to know more. However, I suppose that the information carried by TestCaseContext object is good enough for @sebastien.treyvaud to get started with custom reporting.
Again thank you very much for the useful details and recommendations.
I will be able to make my way using them.
At the moment, I was able to reuse interesting pieces from the GitHub repository mentionned in the title.
The users of the tests are already quite happy with the results we reached.
Of course, they want now additional features and I will continue to dig into the information you provided to check if I am able to reach these new demands.
Best regards
I am glad to hear you have found out your way.
