Custom Test Report compiled by TestListener

I have made a demo project on GitHub and published it:

last edited at 1 Jan, 2021


Problem to solve

In the Katalon User Forum, there was a question that asked how to automatically copy the Test Reports in HTML/PDF format to another location once a test suite finished.

In response to the question, I replied with a post where I described my previous solution. To be honest, I am not very much happy with my previous solution.

Russ Thomas replied a post where he mentioned that, instead of bothering around the built-in Reports, he developed his own reporting functionality. In that post, Russ did not described how his code looks like. So, I was concerned that the readers may feel lost where to go for developing custom reports as Russ did.

Solution

Katalon Studio provides TestListener which is annotated with the TestListener interface.

If you make full use of the TestListener feature, you can compile your own reports of test execution with full control over contents/location/timing. You can compile report in any format you like. You can save the file wherever you want.

Description

How to run the demo

open the Test Suite Test Suites/TS1 , and just run it.

Demo output

Console log

If you look at the console log, you can find output like this:

@AfterTestSuite
Reports/20210101_133234/TS1/20210101_133234/execution0.log 30098bytes
Reports/20210101_133234/TS1/20210101_133234/execution0.log.lck 0bytes
Reports/20210101_133234/TS1/20210101_133234/execution.properties 2660bytes
Reports/20210101_133234/TS1/20210101_133234/testCaseBinding 128bytes

This message proves that 2 files in the Report folder ( execution.properites and execution0.log ) are available at the event of @AfterTestSuite . In these 2 files you can find almost all information out of Katalon Studio how the test suite was configured and how it ran.

CustomReport dir

Once the Test Suites/TS1 finished, a new folder <projectDir>/CustomReport will be created. Inside it you will find a 2 files.

  • memo_TS1.json
  • execution0.log

The CustomReport/memo_TS1.json file contains information from:

  • TestSuiteContext object
  • TestCaseContext objects
  • execution.properties file

The execution0.log file is copied from the Reports folder just to for reference. In the log file you can find all messages emited by your tests with a lot of additives including timestamp.

How the demo designed

Please read the source of the project to find how the demo designed.

The core part is Test Listeners/CustomReportCompiler.groovy . The CustomReportCompiler and a custom Groovy class my.Memo do everything needed to produce the JSON report.

Desired Reporting frameworks …

In the Katalon Forum, many people have expressed their wishes that they want to view the test reports using their favorites reporting frameworks. For example;

Yes, you can develop your code so that it satisfies your requirements by extracting necessary information out of what Katalon Studio provides. Your code should feed the information to your favourites reporting frameworks. That is no different from what I have done here. However, I would remind you that it would involve a lot of your efforts compiling nicely formatted custom report.

2 Likes

He/I did, many times. For one of those times, @devalex88 mentioned he may use it or a derivation of it in a future release (not sure if that’s still the case, though).

There is, in my view, an ill-defined cut-off point when it comes to sharing code. My reporting system is well beyond the “it’s just too big” to share, pretty much only because it would take me way too long to document it clearly for a 3rd party to follow. And then there’s the support. I frankly don’t have the time.

Really? What a strange thing to say, especially under Problem to solve and while summing up the problem at that. I could go so far as to say that’s more than a little disingenuous - there is, as you well know, a search facility on the forum…

Exactly, like I said. And now I’m adding what I said above. You have to support it. That is quite significant and the author of such a system should not consider it a trivial task.

@Russ_Thomas @kazurayam i think sometime it is enough to just prove ‘it is possible’ and both of you did it.
In the end it is a community forum, nobody can be enforced here to provide ‘out-of-the-box’ solutions. (or to share/document/maintain his own code)
And IMHO, providing just guidance it is by far more constructive, let the end user explore and develop his own version.
Lazy people should just open a feature request and wait for it to be considered.
Now they know what to ask :stuck_out_tongue: (or not)

2 Likes

Thanks for taking the time to put this together, I’ll definitely have a play around with your idea and I think that it (along with your initial suggestion of having a test collection to copy the built-in results) are great workarounds for a feature that probably should be already available within Katalon out of the box :slight_smile:

I updated the demo project with tag 0.3

I changed my Custom Report in JSON to make it easier to read; the keys of JSON Objects are sorted in alpha-numerically ascending order while ignoring cases.

Sample output:

{
  "executedBrowser": "Chrome",
  "executionProfile": "default",
  "executionProperties": {
    "allowCustomizeRequestResponseSizeLimit": true,
    "allowCustomizeRequestTimeout": true,
    "allowUsingSelfHealing": true,
    "allowUsingTimeCapsule": true,
    "description": "",
    "execution": {
      "drivers": {
        "preferences": {
          "WebUI": {}
        },
        "system": {
          "WebUI": {
            "browserType": "CHROME_DRIVER",
            "chromeDriverPath": "/Applications/Katalon Studio.app/Contents/Eclipse/configuration/resources/drivers/chromedriver_mac/chromedriver"
          }
        }
      },
      "general": {
        "actionDelay": 0,
        "autoApplyNeighborXpaths": false,
        "defaultFailureHandling": "STOP_ON_FAILURE",
        "defaultPageLoadTimeout": 30,
        "enablePageLoadTimeout": false,
        "excludeKeywords": [
          "verifyElementPresent",
          "verifyElementNotPresent"
        ],
        "executionProfile": "default",
        "ignorePageLoadTimeoutException": false,
        "methodsPriorityOrder": [
          {
            "left": "XPATH",
            "right": true
          },
          {
            "left": "BASIC",
            "right": true
          },
          {
            "left": "CSS",
            "right": true
          },
          {
            "left": "IMAGE",
            "right": true
          }
        ],
        "proxy": "{\"proxyOption\":\"NO_PROXY\",\"proxyServerType\":\"HTTP\",\"username\":\"\",\"password\":\"\",\"proxyServerAddress\":\"\",\"proxyServerPort\":0,\"exceptionList\":\"\",\"applyToDesiredCapabilities\":true}",
        "report": {
          "reportFolder": "/Users/kazuakiurayama/katalon-workspace/CompilingCustomReportInKatalonStudio/Reports/20210318_125423/TS1/20210318_125423",
          "screenCaptureOption": true,
          "videoRecorderOption": {
            "allowedRecordIfFailed": true,
            "allowedRecordIfPassed": false,
            "enable": false,
            "recordAllTestCases": false,
            "useBrowserRecorder": true,
            "videoFormat": "AVI",
            "videoQuality": "LOW"
          }
        },
        "selfHealingEnabled": true,
        "terminateDriverAfterTestCase": false,
        "terminateDriverAfterTestSuite": true,
        "testDataInfo": {},
        "timeCapsuleEnabled": false,
        "timeout": 30,
        "useActionDelayInSecond": "SECONDS",
        "xpathsPriority": [
          {
            "left": "xpath:attributes",
            "right": true
          },
          {
            "left": "xpath:idRelative",
            "right": true
          },
          {
            "left": "dom:name",
            "right": true
          },
          {
            "left": "xpath:link",
            "right": true
          },
          {
            "left": "xpath:neighbor",
            "right": true
          },
          {
            "left": "xpath:href",
            "right": true
          },
          {
            "left": "xpath:img",
            "right": true
          },
          {
            "left": "xpath:position",
            "right": true
          }
        ]
      },
      "globalSmartWaitEnabled": true,
      "logTestSteps": false
    },
    "host": {
      "hostAddress": "192.168.43.135",
      "hostName": "kazuakiurayama - 192.168.43.135",
      "hostPort": 53056,
      "os": "Mac OS X 64bit"
    },
    "id": "Test Suites/TS1",
    "logbackConfigFileLocation": "/Applications/Katalon Studio.app/Contents/Eclipse/configuration/org.eclipse.osgi/73/0/.cp/resources/logback/logback-console.xml",
    "Name": "Chrome",
    "name": "TS1",
    "pluginTestListeners": [
      "KatalonReportListener"
    ],
    "projectDir": "/Users/kazuakiurayama/katalon-workspace/CompilingCustomReportInKatalonStudio",
    "runningMode": "GUI",
    "sessionServer.host": "localhost",
    "sessionServer.port": 52680,
    "source": "/Users/kazuakiurayama/katalon-workspace/CompilingCustomReportInKatalonStudio/Test Suites/TS1.ts",
    "testops": {}
  },
  "testCases": [
    {
      "isMainTestCase": true,
      "isSkipped": false,
      "message": "",
      "testCaseId": "Test Cases/TC1",
      "testCaseIndex": 0,
      "testCaseStatus": "PASSED",
      "testCaseVariables": {
        "var1": "foo",
        "var2": 99
      }
    },
    {
      "isMainTestCase": true,
      "isSkipped": false,
      "message": "Test Cases/TC2 FAILED.\nReason:\ncom.kms.katalon.core.exception.StepFailedException: Failed intentionally\n\tat com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)\n\tat com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)\n\tat TC2.run(TC2:6)\n\tat com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)\n\tat com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)\n\tat com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:398)\n\tat com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:389)\n\tat com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:368)\n\tat com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:360)\n\tat com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:255)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:203)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:152)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:95)\n\tat com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)\n\tat com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)\n\tat TempTestSuite1616039663642.run(TempTestSuite1616039663642.groovy:39)\n",
      "testCaseId": "Test Cases/TC2",
      "testCaseIndex": 1,
      "testCaseStatus": "FAILED",
      "testCaseVariables": {}
    }
  ],
  "testSuite": {
    "status": "COMPLETE",
    "testSuiteId": "Test Suites/TS1"
  }
}

I have made a further improvement in https://github.com/kazurayam/CompilingCustomReportInKatalonStudio tagged as 0.4

In the JSON file, newlines and tab characters in a string must be escaped as \n and \t like this:

"message": "Test Cases/TC2 FAILED.\nReason:\ncom.kms.katalon.core.exception.StepFailedException: Failed intentionally\n\tat com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)\n\tat com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)\n\tat TC2.run(TC2:6)\n\tat com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)\n\tat com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)\n\tat com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:398)\n\tat com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:389)\n\tat com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:368)\n\tat com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:360)\n\tat com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:255)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:203)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:152)\n\tat com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:95)\n\tat com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)\n\tat com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)\n\tat TempTestSuite1616039663642.run(TempTestSuite1616039663642.groovy:39)\n",

A message which contains newlines and tabs escaped in this way is very difficult to read.

So, I extended my reporting script to create another file messages_TS1.txt. Its content is as this (as everyone would expect):

Test Cases/TC2 FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: Failed intentionally
  at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
  at com.kms.katalon.core.util.KeywordUtil$markFailed.call(Unknown Source)
  at TC2.run(TC2:6)
  at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
  at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
  at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:398)
  at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:389)
  at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:368)
  at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:360)
  at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:255)
  at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:203)
  at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:152)
  at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:95)
  at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
  at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
  at TempTestSuite1616051889386.run(TempTestSuite1616051889386.groovy:39)

I have made one more improvement in
https://github.com/kazurayam/CompilingCustomReportInKatalonStudio/ tagged as 0.5

The output memo_TS1.json file now includes GlobalVariables:

{
  "executedBrowser": "Chrome",
  "executionProfile": "default",
  "executionProfile_GlobalVariables": {
    "DEBUG_MODE": false,
    "foo": "bar",
    "Password": "",
    "Products": [
      88,
      99
    ],
    "URL": "http://demoaut.katlaon.com/",
    "Username": "John Doe"
  },
  ...