Custom Test Report compiled by TestListener (kazurayam-ks-plainreport)

[kazurayam]

I have made a demo project on GitHub and published it:

GitHub - kazurayam/kazurayam-ks-plainreport

last edited at 30 Apr, 2021

---

Problem to solve

In the Katalon User Forum, there was a question that asked how to automatically copy the Test Reports in HTML/PDF format to another location once a test suite finished. In response to the question, I replied with a post where I described my previous solution. To be honest, I am not very much happy with my previous solution. Russ Thomas replied a post where he mentioned that, instead of bothering around the built-in Reports, he developed his own reporting functionality.

So I was motivated to seek for what custom report I can develop in Katalon Studio.

Solution

Katalon Studio provides a feature called Test Listeners. If you make full use of the TestListener feature, you can compile your own reports of test execution with full control over contents/location/timing. You can compile report in any format you like. You can save the file wherever you want.

I have made a skeletal implementation of my custom Test Report. The report is in JSON text format. It is plain; I mean the report is not ornamented at all.

I have produced a jar file named kazurayam-ks-plainreport-x.x.x.jar for general use. You can download and use it in your own Katalon project.

I used the Gson library to jsonize anonymous Groovy objects. Gson is bundled in Katalon Studio.

Description

How to install the plugin

• create your own Katalon Studio project
• visit the Releases page and download the latest version of kazurayam-ks-plainreport-x.x.x.jar`
• save the downloaded jar file into your project’s Plugins folder
• stop/restart Katalon Studio

Add a Test Listener

• In the Test Listener directory, create a new Test Listenner with any name.
• Copy the sample code Test Listeners/TLPlainReport and paste it into you new Test Listener, save it to disk.
• The TLPlainReport is common to every case. No code editing is required.

This new Test Listener is required.

Create your Test Cases and Test Suite.

You can create your Test Cases and Test Suite as you like.

For demo purpose, you can reuse the following test cases as example.

• Test Cases/TC1
• Test Cases/TC2

Here I assume you will create a Test Suite named “TS1” which binds the TC1 and TC2.

Running the demo

Select the Test Suite “TS1” and run it.

Demo output

Output location

Once the Test Suites/TS1 finished, a new folder <projectDir>/PlainReport will be created. Inside it you will find a 3 files.

• execution0.log
• messages.<TestSuiteID>.<yyyyMMdd_hhmmss>.txt
• report.<TestSuiteID>.<yyyyMMdd_hhmmss>.json

The execution0.log file is copied from the Reports folder just to for easier reference. In the log file you can find all messages emitted by your tests with a lot of additives including timestamp.

The report.TestSuiteID.yyyyMMdd_hhmmss.json file contains information from:

• the TestSuiteContext object
• the TestCaseContext objects
• the execution.properties file in the Report folder

The messages.TestSuiteID.yyyyMMdd_hhmmss.txt file contains the messages emitted by failed Test Cases in Java printStackTrace format.

Console log

If you look at the console log, you can find output like this:

@AfterTestSuite
Reports/20210101_133234/TS1/20210101_133234/execution0.log 30098bytes
Reports/20210101_133234/TS1/20210101_133234/execution0.log.lck 0bytes
Reports/20210101_133234/TS1/20210101_133234/execution.properties 2660bytes
Reports/20210101_133234/TS1/20210101_133234/testCaseBinding 128bytes

This message proves that 2 files in the Report folder ( execution.properites and execution0.log ) are available at the event of @AfterTestSuite . In these 2 files you can find almost all information out of Katalon Studio how the test suite was configured and how it ran.

How the demo designed

Please read the source of the project to find how the demo designed.

1. Test Listeners/TLPlainReport.groovy
2. Keywords/com/kazurayam/ks/plainreport/ReportGenerator
3. Keywords/com/kazurayam/ks/plainreport/PlainReport

[Russ_Thomas]

@kazurayam

But he did not describe how his code looks like.

He/I did, many times. For one of those times, @devalex88 mentioned he may use it or a derivation of it in a future release (not sure if that’s still the case, though).

There is, in my view, an ill-defined cut-off point when it comes to sharing code. My reporting system is well beyond the “it’s just too big” to share, pretty much only because it would take me way too long to document it clearly for a 3rd party to follow. And then there’s the support. I frankly don’t have the time.

@kazurayam

So, I am afraid, the readers may feel lost where to go for developing custom reports as Russ did.

Really? What a strange thing to say, especially under Problem to solve and while summing up the problem at that. I could go so far as to say that’s more than a little disingenuous - there is, as you well know, a search facility on the forum…

@kazurayam

However, I would remind you that it would involve a lot of effort compiling your custom report.

Exactly, like I said. And now I’m adding what I said above. You have to support it. That is quite significant and the author of such a system should not consider it a trivial task.

[anon46315158]

@Russ_Thomas @kazurayam i think sometime it is enough to just prove ‘it is possible’ and both of you did it.
In the end it is a community forum, nobody can be enforced here to provide ‘out-of-the-box’ solutions. (or to share/document/maintain his own code)
And IMHO, providing just guidance it is by far more constructive, let the end user explore and develop his own version.
Lazy people should just open a feature request and wait for it to be considered.
Now they know what to ask (or not)

[gengland]

Thanks for taking the time to put this together, I’ll definitely have a play around with your idea and I think that it (along with your initial suggestion of having a test collection to copy the built-in results) are great workarounds for a feature that probably should be already available within Katalon out of the box

[kazurayam]

I updated the demo project with tag 0.3

kazurayam/CompilingCustomReportInKatalonStudio

I changed my Custom Report in JSON to make it easier to read; the keys of JSON Objects are sorted in alpha-numerically ascending order while ignoring cases.

Sample output:

{
  "executedBrowser": "Chrome",
  "executionProfile": "default",
  "executionProperties": {
    "allowCustomizeRequestResponseSizeLimit": true,
    "allowCustomizeRequestTimeout": true,
    "allowUsingSelfHealing": true,
    "allowUsingTimeCapsule": true,
    "description": "",
    "execution": {
      "drivers": {
        "preferences": {
          "WebUI": {}
        },
        "system": {
          "WebUI": {
            "browserType": "CHROME_DRIVER",
            "chromeDriverPath": "/Applications/Katalon Studio.app/Contents/Eclipse/configuration/resources/drivers/chromedriver_mac/chromedriver"
          }
        }
      },
      "general": {
        "actionDelay": 0,
        "autoApplyNeighborXpaths": false,
        "defaultFailureHandling": "STOP_ON_FAILURE",
        "defaultPageLoadTimeout": 30,
        "enablePageLoadTimeout": false,
        "excludeKeywords": [
          "verifyElementPresent",
          "verifyElementNotPresent"
        ],
        "executionProfile": "default",
        "ignorePageLoadTimeoutException": false,
        "methodsPriorityOrder": [
          {
            "left": "XPATH",
            "right": true
          },
          {
            "left": "BASIC",
            "right": true
          },
          {
            "left": "CSS",
            "right": true
          },
          {
            "left": "IMAGE",
            "right": true
          }
        ],
        "proxy": "{\"proxyOption\":\"NO_PROXY\",\"proxyServerType\":\"HTTP\",\"username\":\"\",\"password\":\"\",\"proxyServerAddress\":\"\",\"proxyServerPort\":0,\"exceptionList\":\"\",\"applyToDesiredCapabilities\":true}",
        "report": {
          "reportFolder": "/Users/kazuakiurayama/katalon-workspace/CompilingCustomReportInKatalonStudio/Reports/20210318_125423/TS1/20210318_125423",
          "screenCaptureOption": true,
          "videoRecorderOption": {
            "allowedRecordIfFailed": true,
            "allowedRecordIfPassed": false,
            "enable": false,
            "recordAllTestCases": false,
            "useBrowserRecorder": true,
            "videoFormat": "AVI",
            "videoQuality": "LOW"
          }
        },
        "selfHealingEnabled": true,
        "terminateDriverAfterTestCase": false,
        "terminateDriverAfterTestSuite": true,
        "testDataInfo": {},
        "timeCapsuleEnabled": false,
        "timeout": 30,
        "useActionDelayInSecond": "SECONDS",
        "xpathsPriority": [
          {
            "left": "xpath:attributes",
            "right": true
          },
          {
            "left": "xpath:idRelative",
            "right": true
          },
          {
            "left": "dom:name",
            "right": true
          },
          {
            "left": "xpath:link",
            "right": true
          },
          {
            "left": "xpath:neighbor",
            "right": true
          },
          {
            "left": "xpath:href",
            "right": true
          },
          {
            "left": "xpath:img",
            "right": true
          },
          {
            "left": "xpath:position",
            "right": true
          }
        ]
      },
      "globalSmartWaitEnabled": true,
      "logTestSteps": false
    },
    "host": {
      "hostAddress": "192.168.43.135",
      "hostName": "kazuakiurayama - 192.168.43.135",
      "hostPort": 53056,
      "os": "Mac OS X 64bit"
    },
    "id": "Test Suites/TS1",
    "logbackConfigFileLocation": "/Applications/Katalon Studio.app/Contents/Eclipse/configuration/org.eclipse.osgi/73/0/.cp/resources/logback/logback-console.xml",
    "Name": "Chrome",
    "name": "TS1",
    "pluginTestListeners": [
      "KatalonReportListener"
    ],
    "projectDir": "/Users/kazuakiurayama/katalon-workspace/CompilingCustomReportInKatalonStudio",
    "runningMode": "GUI",
    "sessionServer.host": "localhost",
    "sessionServer.port": 52680,
    "source": "/Users/kazuakiurayama/katalon-workspace/CompilingCustomReportInKatalonStudio/Test Suites/TS1.ts",
    "testops": {}
  },
  "testCases": [
    {
      "isMainTestCase": true,
      "isSkipped": false,
      "message": "",
      "testCaseId": "Test Cases/TC1",
      "testCaseIndex": 0,
      "testCaseStatus": "PASSED",
      "testCaseVariables": {
        "var1": "foo",
        "var2": 99
      }
    },
    {
      "isMainTestCase": true,
      "isSkipped": false,
      "message": "Test Cases/TC2 FAILED.\nReason:\ncom.kms.katalon.core.exception.StepFailedException: Failed intentionally\n\tat com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)\n\tat com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)\n\tat TC2.run(TC2:6)\n\tat com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)\n\tat com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)\n\tat com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:398)\n\tat com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:389)\n\tat com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:368)\n\tat com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:360)\n\tat com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:255)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:203)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:152)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:95)\n\tat com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)\n\tat com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)\n\tat TempTestSuite1616039663642.run(TempTestSuite1616039663642.groovy:39)\n",
      "testCaseId": "Test Cases/TC2",
      "testCaseIndex": 1,
      "testCaseStatus": "FAILED",
      "testCaseVariables": {}
    }
  ],
  "testSuite": {
    "status": "COMPLETE",
    "testSuiteId": "Test Suites/TS1"
  }
}

[kazurayam]

I have made a further improvement in https://github.com/kazurayam/CompilingCustomReportInKatalonStudio tagged as 0.4

In the JSON file, newlines and tab characters in a string must be escaped as \n and \t like this:

"message": "Test Cases/TC2 FAILED.\nReason:\ncom.kms.katalon.core.exception.StepFailedException: Failed intentionally\n\tat com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)\n\tat com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)\n\tat TC2.run(TC2:6)\n\tat com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)\n\tat com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)\n\tat com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:398)\n\tat com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:389)\n\tat com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:368)\n\tat com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:360)\n\tat com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:255)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:203)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:152)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:95)\n\tat com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)\n\tat com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)\n\tat TempTestSuite1616039663642.run(TempTestSuite1616039663642.groovy:39)\n",

A message which contains newlines and tabs escaped in this way is very difficult to read.

So, I extended my reporting script to create another file messages_TS1.txt. Its content is as this (as everyone would expect):

Test Cases/TC2 FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: Failed intentionally
  at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
  at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
  at TC2.run(TC2:6)
  at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
  at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
  at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:398)
  at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:389)
  at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:368)
  at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:360)
  at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:255)
  at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:203)
  at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:152)
  at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:95)
  at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
  at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
  at TempTestSuite1616051889386.run(TempTestSuite1616051889386.groovy:39)

[kazurayam]

I have made one more improvement in
https://github.com/kazurayam/CompilingCustomReportInKatalonStudio/ tagged as 0.5

The output plainreport.XXX.yyyyMMdd_hhmmss.json file now includes GlobalVariables:

{
  "executedBrowser": "Chrome",
  "executionProfile": "default",
  "executionProfile_GlobalVariables": {
    "DEBUG_MODE": false,
    "foo": "bar",
    "Password": "",
    "Products": [
      88,
      99
    ],
    "URL": "http://demoaut.katlaon.com/",
    "Username": "John Doe"
  },
  ...

[kazurayam]

The version 0.6.0 of “kazurayam-ks-plainreport” publishes a plugable jar file. See the doc README How to install the plugin.