A fake KeywordLogger could integrate Extent Reports into Katalon project nicely

[kazurayam]

I made a GitHub repository:

GitHub - kazurayam/KS_Fake_KeywordLogger_Integrates_ExtentReports

---

Problem to solve

I want to create a test execution report of a Katalon Studio project using the Extent Reports. Let me give you an sample problem.

I made a Test Suite TS1:

Also I made 2 Test Cases. The TC1 is as follows:

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

// TC1

WebUI.comment("雨ニモマケズ")
WebUI.comment("風ニモマケズ")

And TC2 is as follows:

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

// TC2

WebUI.comment("Psalm 201 – En vänlig grönskas rika dräkt")
WebUI.comment("")
WebUI.comment("En vänlig grönskas rika dräkt har smyckat dal och ängar.")
WebUI.comment("Nu smeker vindens ljumma fläkt de fagra örtes-ängar;")
WebUI.comment("Och solens ljus och lundens sus och vågens sorl bland viden")
WebUI.comment("förkunna sommartiden.")
WebUI.comment("")
WebUI.comment("Sin lycka och sin sommar-ro de yra fåglar prisa;")
WebUI.comment("Ur skogens snår, ur stilla bo framklingar deras visa.")
WebUI.comment("En hymn går opp med fröjd och hopp från deras glada kväden")
WebUI.comment("från blommorna och träden")
WebUI.comment("")
WebUI.comment("Men Du, o Gud, som gör vår jord så skön i sommarns stunder,")
WebUI.comment("Giv, att jag aktar främst ditt ord och dina nådesunder,")
WebUI.comment("Allt kött är hö, och blomstren dö och tiden allt fördriver")
WebUI.comment("blott Herrens ord förbliver.")
WebUI.comment("")
WebUI.comment("Musik: Waldemar Åhlén")
WebUI.comment("Text: Carl David af Wirsén")
WebUI.comment("quoted from https://1.se/text-psalm-201-en-vanlig-gronskas-rika-drakt-sommarpsalm/")

When I exected the TS1, Katalon Studio generated an HTML report like this:

Every Katalon users will find there is nothing special in TS1, TC1, TC2 and the HTML report. It’s a boring stuff.

Now I want to add another format of test execution report generated by Extent Reports. The report looks something like this:

Now I would set a constraint to myself in achieving the Extent Reports integration into Katalon project.

The Test Case TC1 should not be changed. It should remain the same as before. Test Cases shouldn’t make any call to the Extent Reports API. The ordinary WebUI.comment(String message) should print the message into a new report generated by Extent Reports as well.

How can I achieve it?

Solution

We can read the source code of com.kms.katalon.core.** packages contained in the Katalon Studio distributables. For example, on my Mac, I could find the jar files that contain the sources :

$ pwd
/Applications/Katalon Studio.app/Contents/Eclipse/configuration/resources/source
$ tree -P *.jar
.
├── com.kms.katalon.core
│   └── com.kms.katalon.core-sources.jar
├── com.kms.katalon.core.cucumber
│   └── com.kms.katalon.core.cucumber-sources.jar
├── com.kms.katalon.core.mobile
│   └── com.kms.katalon.core.mobile-sources.jar
├── com.kms.katalon.core.testng
│   └── com.kms.katalon.core.testng-sources.jar
├── com.kms.katalon.core.webservice
│   └── com.kms.katalon.core.webservice-sources.jar
├── com.kms.katalon.core.webui
│   └── com.kms.katalon.core.webui-sources.jar
└── com.kms.katalon.core.windows
    └── com.kms.katalon.core.windows-sources.jar

8 directories, 7 files

I started reading the source codes to find out how a call WebUI.comment("雨ニモマケズ") propagates through the call chains and how the message is written into the Console tab and the HTML report file located at Reports/yyyyMMdd_hhmmss/TS1/yyyyMMdd_hhmmss/execution0.log file. Eventurally I found it. Let me trace the path that I went through.

• A call to WebUI.comment("雨ニモマケズ") in a Test Case script calls the comment() method of the com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords class but you can not find the method in the source. The comment() method is actually implemented in the com.kms.katalon.core.keyword.BuiltInKeywords which is the parent class.

• The comment() method of com.kms.katalon.core.keyword.BuiltInKeyword calls the executeKeywordForPlatform() method of the com.kms.katalon.core.keyword.internal.KeywordExecutor.

• A call to KeywordExecutor.executeKeywordForPlatform(KeywordExecutor.PLATFORM_BUILT_IN, "comment", message) calls comment(String message) of the com.kms.katalon.core.keyword.builtin.CommentKeyword. The method is implemented as follows:

    public void comment(String message) {
        // Just a comment line, do nothing
        logger.logInfo(message)
    }

• The variable named logger is declared in the com.kms.katalon.core.keyword.internal.AbstractKeyword, which is the parent class of the CommentKeyword class, as follows:

...
import com.kms.katalon.core.logging.KeywordLogger
...
public abstract class AbstractKeyword implements IKeyword {

    protected final KeywordLogger logger = KeywordLogger.getInstance(this.getClass());

    ...

• The logInfo(String) method of com.kms.katalon.core.logging.KeywordLogger is implemented as follows:

    public void logInfo(String message, Map<String, String> attributes) {
        logger.info(message);         // emit message into the Console in GUI via org.slf4j.Logger object
        xmlKeywordLogger.logInfo(this, message, attributes); // emit message into the execution0.log file
    }

Finally, I got to the heart of the matter! The logInfo(String) method of the com.kms.katalon.core.logging.KeywordLogger object actually prints messages into

1. the LogViewer in the Katalon Studio GUI, and
2. the execution0.log file under the <projectDir>/Reports directory. Katalon Studio will later transform the file into the builtin test execution reports in HTML/CSV/PDF.

So, I want to change the logInfo method of the KeywordLogger so that the message is also transferred into a report generated by Extent Reports. In short I want to change it as:

   public void logInfo(String message, Map<String, String> attributes) {
        logger.info(message);                                // write into the LogViewer
        xmlKeywordLogger.logInfo(this, message, attributes); // write into the execution0.log file
        /*
         * kazurayam inserted the following
         */
        for (Map.Entry<String, ReportBuilder> pair: reportBuilders.entrySet()) {
            String className = pair.getKey()
            ReportBuilder rb = pair.getValue()
            rb.getInstance().logInfo(message)
            // com.kazurayam.ks.reporting.ReportBuilderSkeletonImpl.getInstance().logInfo(message) will write the message into the console
            // com.kazurayam.ks.reporting.ReportBuilderExtentImpl.getInstance().logInfo(message) will write the message into the html generated by Extent Reports
        }
    }

Simple, isn’t it?

Difficulty

I want to change the logInfo method of com.kms.katalon.core.logging.KeywordLogger object. Can I do it?

No, I can’t. Katalon Studio is not an open-source software. It is a proprietary software product of Katalon who exclusively owns the source code; though a set of copy is published.

But I am really interested in the idea. It will be a fun. I would try.

Bad Hack

After a few weeks of studies, I @kazurayam have found out a hack. Let me tell you about it here.

Every Katalon Studio project has a file named .classpath where all libraries available to the project are listed. It starts with the following lines:

The line#8 declares the /Applications/Katalon Studio.app/Contents/Eclipse/plugins/com.kms.katalon.core_1.0.0.202501201829.jar. This jar contains the binary of the com.kms.katalon.core.logging.KeywordLogger. And a line above the <classpathentry kind="src" output="bin/groovy" path="Include/scripts/groovy"/> is declared. As you know, Katalon Studio allows you to create any custom Groovy class in the <projectDir>/Include/scripts/groovy folder. The classes created in the Include/scripts/groovy folder is declared first. The precedence depends on the line order. Therefore the classes in the Include/scripts/groovy folder will have the higher precedence to the classes in the com.kms.katalon.core_1.0.0.202501201829.jar.

Now, I can create a fake com.kms.katalon.core.logging.KeywordLogger in the Include/scripts/groovy. Katalon Studio will allow me to do it.

Then what will happen? — My fake KeywordLogger will have higher precedence to the real KeywordLogger provided by Katalon. Effectively I can change the source code of the KeywordLogger as I like.

Description

I created this project and tried my idea: “A fake KeywordLogger integrates Extent Reports into Katalo project”. It worked!

How to resolve external dependencies

I need to import several external dependencies such as Extent Reports, etc into my project. I used the Katalon Studio’s Gradle Plugin.

I created a build.gradle file.

In the command line, I ran:

$ pwd
~/katalon-workspace/KS_Fake_KeywordLogger_Integrates_ExtentReports
$ gradle katalonCopyDependencies
...

Then a few jar files will be downloaded from the Maven Central repository in to the Drivers folder, as follows:

Codes created

• Include/scripts/groovy/com/kms/katalon/core/logging/KeywordLogger.groovy

• Include/scripts/groovy/com/kazurayam/ks/reporting/ReportAdapter.groovy

• Include/scripts/groovy/com/kazurayam/ks/reporting/ReportAdapterExtentImpl.groovy

• Include/scripts/groovy/com/kazurayam/ks/reporting/ReportAdaptersLoader.groovy

• reporting.config.json

• Test%20Listeners/CustomReportsListener.groovy

• Test Cases/main/TC1

• Test Cases/main/TC2

• Test%20Suites/TS1.ts

I learned a lot out of the GitHub repository extent-report-sample by @coty.

How to run the demo

Just run the Test Suites/TS1

Final result

The <projectDir>/Extent directory will be newly created where the reports will be generated by Extent Reports that look like

Conclusion

I think that it is the best approach to modify the com.kms.katalon.core.logging.KeywordLogger class to transfer the log messages into Extent Reports. My fake KeywordLogger implementation proved my idea is possibly good. I am contented with this result.

However, I am aware that my work is just the start of long development efforts to accomplish integrating Extent Reports into Katalon to a satisfactory level. I just worked on a single keyword WebUI.comment. There are dozens of more keywords to work on: WebUI.click, WebUI.setText, WebUI.openBrowser, WebUI.verifyElementPresent, and so on. We would need to amend the KeywordLogger class more significantly.

Who can achieve this task? — Only Katalon can do it, as the KeywordLogger is their own property. Nobody else can.

[vu.tran]

Thank you @kazurayam for sharing this. This is well-informed to our product managers.

[kazurayam]

Finally, I would point out: the com.kms.katalon.core.logging.KeywordLogger is NOT extensible at all. I believe, this is a design shortage. If we want various types of test reports to be really pluggable, the KeywordLogger should be entirely refactored to be extensible.

[kazurayam]

I couldn’t stop working on a research. I wanted to change my fake KeywordLogger extensible so that I can inject one or more classes that compile test execution report.

I got an idea:

Include/scripts/groovy/com/kms/katalon/core/logging/KeywordLogger.groovy

See the logInfo(String) method implementation.

	public void logInfo(String message, Map<String, String> attributes) {
		logger.info(message);                                // write into the LogViewer
		xmlKeywordLogger.logInfo(this, message, attributes); // write into the execution0.log file
		/*
		 * kazurayam inserted the following
		 */
		for (Map.Entry<String, ReportBuilder> pair: reportBuilders.entrySet()) {
			String className = pair.getKey()
			ReportBuilder rb = pair.getValue()
			rb.getInstance().logInfo(message)
			// com.kazurayam.ks.reporting.ReportBuilderSkeletonImpl.getInstance().logInfo(message) will write the message into the console
			// com.kazurayam.ks.reporting.ReportBuilderExtentImpl.getInstance().logInfo(message) will write the message into the html generated by Extent Reports
		}
	}

The reportBuilders contains a map that contains anonymous class that implements the ReportBuilder interface, and implements getInstance() method that returns a Singleton object.

My fake KeywordLogger’s logInfo() iterates over the injected ReportBuilders and calls the longInfo method of the injected ReportBuilders.

In a JSON config file, the Fully Qualified Class Name of custom classes to inject are declared:

{
	"ReportBuilder_classes" : [
		"com.kazurayam.ks.reporting.ReportBuilderSkeletonImpl",
		"com.kazurayam.ks.reporting.ReportBuilderExtentImpl"
	]
}

If you create your own class that implements the ReportBuilder interface and implements getInstance() method, you can add it in the JSON file. Your class will be recognized by my fake KeywordLogger. Effectively your own class will be fed with the logging stream out of Katalon Studio. Your class will be enabled to compile a report with equivalent content as the Katalon-builtin report in whatever custom format you like.

[kazurayam]

I have updated my demo project. It generates a html report compiled by Extent Reports, which includes

• screenshots taken by Katalon Keywords on failure
• Java stacktrace taken on keyword failure

With this change, the report compiled by Extent Reports covers the equivalent range of test execution logs as the Katalon built-in report covers.

The test case has nothing unusual; it looks just as an ordinary Katalon Test case, like this:

import com.kms.katalon.core.model.FailureHandling
import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

TestObject makeTestObject(String id, String xpath) {
	TestObject tObj = new TestObject(id)
	tObj.addProperty("xpath", ConditionType.EQUALS, xpath)
	return tObj
}
TestObject invalidBtn = makeTestObject("invalidBtn", "//a[@id='btn-make-disappointment']")

WebUI.openBrowser("http://demoaut.katalon.com/")
WebUI.verifyElementPresent(invalidBtn, 10, FailureHandling.CONTINUE_ON_FAILURE)
WebUI.closeBrowser()

Please note that you also have the source code in your hands so that you have a good chance to customize the report. For example, you can add “Git Branch Name: master” in the report as I did here.

However, as I mentioned in the original post, my project requires the com.kms.katalon.core.logging.KeywordLogger class to be modified. Only Katalon can do it in production; I can not change their proprietary product. So what I have done (the fake KeywordLogger class) will stay just a proof of concept.

Now I have done everything I wanted. It was a fun. I won’t do anymore on this issue.