Post Test Suite Processing --- how to zip Report Folder

I have published a demo project in GitHub


Caution

I used Mac. This project assumes that OS command /bin/sh is available, which is usually true on Mac and Linux. You should be able to rewrite this for Windows + PowerShell environment easily.

Demonstration

This project demonstrates how to automate making a zip file of Reports generated by a Test Suite execution.

Please download this project from https://github.com/kazurayam/Post_TestSuite_processing/releases , open it with your local Katalon Studio, run a Test Suite Collection named TSC . Just run it to see what TSC does.

  1. Open a Test Suite Collection named Test Suites/TCS . It comprises with 2 Test Suites.
  • Test Suites/TS1
  • Test Suites/TS_post_TS1_processing
  1. Click the Execute button to run it. It will run for a few seconds. 01_TSC
  2. The TS1 will run for quckly and pass; TS_post_TS1_proocessing will fail. It fails intentionally for demostration purpose. Don’t mind it.
  3. Once TSC finished, a folder named out will be created under the project directory. 02_out

However, Katalon Studio has a bug; the out folder, which is newly created by script, will not become visible in the Test Explorer pane. You need to close the project and reopen it. By doing so Katalon Studio will find the new out folder.

  1. In the out folder, you will find two files created by the TSC .
  • out/consume_TS_report.sh
  • out/Reports_yyyyMMdd_hhmmss.zip

Here yyyyMMdd_hhmmss represent a varying time stamp.

  1. Please check the content of the Reports_yyyyMMdd_hhmmss.zip file using your favorite archiver software. In the zip file, you will find the test report in HTML/CSV/XML generated by the TS1 . For example, like this: 03_zip
  2. TSC calls Test Suite/TS1 , which calls a Test Case TC1 . TC1 is a skeltal Test Case example, like this:
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

WebUI.comment("Here you can do whatever you like")

Problem to solve

Why I developed this demo project? Let me tell it you.

Katalon Studio users often want to make a zip file of the Reports of a Test Suite execution. If you don’t mind doing it manually after a Test Suite finished, you can:

  1. In Windows Explorer GUI, open the project folder
  2. Dig the folder tree to reach <projectDir>/Reports/yyyyMMdd_hhmmss/testSuiteName/yyyyMMdd_hhmmss2 folder
  3. Right-click the folder and choose “archive it” menu.
  4. You are done. You will get a zip file which contains HTML/CSV/XML reports.

But I don’t like to do it manually. I want to automate zipping the Reports . Not only zipping, I want more. For example, I want to transfer the file to some remote storage for further processing.

How can I make it possible in a Katalon Studio project?

Technical difficulties

I found 3 technical issues that I had to overcome.

Timing issue

Katalon users naively expects that HTML/XML reports of a Test Suite TS1 to be present and accessible in the Report folder as soon as finished. They expects the reports should be accessible in the @AfterTestSuite -annotated method in a TestListener.

Unfortunately, it is not the case. When a @AfterTestSuite -annotated method invoked, the HTML/XML reports are not present there yet.

I can show you an evidence.

When you run the Test Suite Collection TSC , you should be able to find the following messages in the Console.

TS1ReportDirLister output ----------------------------------------------
executing: ls -la ./Reports/20211023_172734/TS1/20211023_172736
return code = 0
total 40
drwxr-xr-x  6 kazurayam  staff    192 Oct 23 17:27 .
drwxr-xr-x  3 kazurayam  staff     96 Oct 23 17:27 ..
-rw-r--r--  1 kazurayam  staff   2649 Oct 23 17:27 execution.properties
-rw-r--r--  1 kazurayam  staff  11108 Oct 23 17:27 execution0.log
-rw-r--r--  1 kazurayam  staff      0 Oct 23 17:27 execution0.log.lck
-rw-r--r--  1 kazurayam  staff     64 Oct 23 17:27 testCaseBinding

This message is emitted by Test Listeners/TS1ReportDirLister. It executes a Command ls -la ./Reports/20211023_172734/TS1/20211023_172736 in the @AfterTestSuite -annotated method. As you can see, there is a bulky execution0.log file but no HTML/XML reports yet.

The fact is, Katalon Studio will compile HTML/XML report after @AfterTestSuite -annotated method finished. Therefore you can not make a zip of the report in the @AfterTestSuite -annotated method after TS1 .

Variable path

In order to write a script to zip a folder, the script must know the path of target folder. On the other hand, Katalon Studio assigns the path of a report folder for TS1 like:

  • Reports/20211023_125201/TS1/20211023_125203

As you can see, this path has variable parts (time stamp). How can my script find the concrete value of the variable folder path?

How to run a Command from Groovy

In order to create a zip file of a folder contents, I want to use the good old bash shell commands: zip , mv , cp , rm , mkdir , echo etc. Also I want to use curl command to transfer files over network.

I will create, by Groovy script (TestListener), a shell script file named consume_TS_report.sh . Then, how can I execute that shell script with /bin/sh (bash shell interpreter) from my Groovy script (TestCase)?

Proposed solution

Resolving timing issue

You can not make a zip of the Test Suite TS1 by TS1 itself. Possible solution is to create another Test Suite TS_post_TS1_processing . You want to create a Test Suite Collection TSC and let it call TS1 and TS_post_TS1_processing sequentially in this order. At the timing when TS_post_TS1_processing is activated, the reports of TS1 has been already compiled. The following Test Suites should be able to read the reports.

Resolving variable report folder path issue

com.kms.katalon.core.configuration.RunConfiguration class implements String getReportFolder() method. With this method call, we can get to know the report folder path of the current Test Suite.

Resolving how to run command from Groovy

The java.lang.ProcessBuilder class enables running arbitrary commands in a new process forked from Groovy script. The following article covers how to use it.

However I found that ProcessBuilder is difficult to use for simple use cases. So I have developed a small wrapper for ProcessBuilder named subprocessj .

The jar is already bundled in the Drivers folder of this Git project.

You can download the jar of subprocessj from the Maven Central repository.

The tutorial of subprocessj is here:

The source code of subprocessj is hosted here:

Description

First half

When you run the Test Suite Collection TSC and when the Test Suite TS1 has finished, the Test Listeners/PostTS1Processor will be invoked. It has @AfterTestSuite -annotated method. Please read the source.

What does the method do ?

  1. if the Test Suite TS1 has been finished,
  2. identify the path of report folder of TS1
  3. in the out folder, create a file named consume_TS_report.sh
  4. write a few lines of bash commands. The commands will do the following:
  • make a zip file of the report folder
  • move the zip file into the out folder
  • try to transfer the zip file by HTTP POST request to a remote URL
  1. The out/consume_TS_Report.sh is something like the following:
zip Reports_20211023_172736 -r "Reports/20211023_172734/TS1/20211023_172736"
mv Reports_20211023_172736.zip "out/"

# I know this would fail
curl -X POST https://localhost:80 -F 'file=@out/Reports_20211023_172736.zip'
  1. do " chmod +x consume_TS_report.sh " to change the file permission to make it executable in the OS command line.

In the out/consume_TS_Report.sh script file, I wrote a line of curl command to transfer the zip file to a URL (https://localhost:80), which will inevitably fail as the URL is not alive. This is just an example. You can do anything you like using shell commands here.

Second half

The Test Suite Collection TSC will call the Test Suite TS_post_TS1_processing after TS1 .

TS_post_TS1_processing calls a Test Case TC_consume_TS1_report .

What this test case does? For the detail, read the source code.

  1. The test case script locates the out/consume_TS_report.sh
  2. It forks a new OS process to calls /bin/sh (bash shell interpreter) while specifying the script file. In short, it execute the script file. It waits for the forked process to finish.
  3. When the forked process finished, the test case checks the return code. If the return code != 0, the test case fails.
  4. It prints the error messages from the forked process if any.
1 Like

@kazurayam looks nice
however, you can use the java.util.zip to create the archive, see:
https://www.baeldung.com/java-compress-and-uncompress
and whatever method you like to make a post request (predefined request object or build the request object dynamically)
with a bit or care, using those will make your project OS agnostic (a shell script is no longer needed)
you only have to pass the path of the Report folder from one suite to the post process one, here a simple temporary text file is good enough

Also, you can improve it a bit to take a list of report folders as input (let’s suppose you run more than one test suites and you zip all reports)

another approach (for just zipping the reports) may be to use a gradle task.
but here i cannot help much, i never played with custom gradle tasks in katalon
see: https://docs.gradle.org/current/userguide/working_with_files.html#sec:creating_archives_example

for the upload part, provided the gradle approach is working, can be something like here:

The POC here may be a start:

provided one may find a solution to register a gradle task which executes after the report is generated,
you are in full control

@Jass @ThanhTo @duyluong any hints if this is achievable?

@bionel

Thank you for your comment.

Here I aimed to present an idea of “Post TestSuite processing” to those who visited the following thread

I wanted to show a working example of “Post Test Suite processing” for their reference.

java.util.zip, Gradle, Ant — Yes, of course. You can use any approach to implement post-processing. Here I chose shell commands just because I love those good old one-liners. Zip, mv and curl still shine to me.

OS agnostic

You are right. But it doesn’t matter to me very much. I know it is easy to rewrite this for Windows + PowerShell environment when I need it.

v0.1.2 enabled you to make a zip using PowerShell

https://github.com/kazurayam/Post_TestSuite_processing/releases/tag/0.1.2

1 Like