I have published a demo project in GitHub
Caution
I used Mac. This project assumes that OS command /bin/sh
is available, which is usually true on Mac and Linux. You should be able to rewrite this for Windows + PowerShell environment easily.
Demonstration
This project demonstrates how to automate making a zip file of Reports generated by a Test Suite execution.
Please download this project from Releases · kazurayam/Post_TestSuite_processing · GitHub , open it with your local Katalon Studio, run a Test Suite Collection named TSC
. Just run it to see what TSC
does.
- Open a Test Suite Collection named
Test Suites/TCS
. It comprises with 2 Test Suites.
Test Suites/TS1
Test Suites/TS_post_TS1_processing
- Click the Execute button to run it. It will run for a few seconds.
- The
TS1
will run for quckly and pass;TS_post_TS1_proocessing
will fail. It fails intentionally for demostration purpose. Don’t mind it. - Once
TSC
finished, a folder namedout
will be created under the project directory.
However, Katalon Studio has a bug; the
out
folder, which is newly created by script, will not become visible in the Test Explorer pane. You need to close the project and reopen it. By doing so Katalon Studio will find the newout
folder.
- In the
out
folder, you will find two files created by theTSC
.
out/consume_TS_report.sh
out/Reports_yyyyMMdd_hhmmss.zip
Here
yyyyMMdd_hhmmss
represent a varying time stamp.
- Please check the content of the
Reports_yyyyMMdd_hhmmss.zip
file using your favorite archiver software. In the zip file, you will find the test report in HTML/CSV/XML generated by theTS1
. For example, like this: TSC
callsTest Suite/TS1
, which calls a Test CaseTC1
.TC1
is a skeltal Test Case example, like this:
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
WebUI.comment("Here you can do whatever you like")
Problem to solve
Why I developed this demo project? Let me tell it you.
Katalon Studio users often want to make a zip file of the Reports of a Test Suite execution. If you don’t mind doing it manually after a Test Suite finished, you can:
- In Windows Explorer GUI, open the project folder
- Dig the folder tree to reach
<projectDir>/Reports/yyyyMMdd_hhmmss/testSuiteName/yyyyMMdd_hhmmss2
folder - Right-click the folder and choose “archive it” menu.
- You are done. You will get a zip file which contains HTML/CSV/XML reports.
But I don’t like to do it manually. I want to automate zipping the Reports . Not only zipping, I want more. For example, I want to transfer the file to some remote storage for further processing.
How can I make it possible in a Katalon Studio project?
Technical difficulties
I found 3 technical issues that I had to overcome.
Timing issue
Katalon users naively expects that HTML/XML reports of a Test Suite TS1
to be present and accessible in the Report folder as soon as finished. They expects the reports should be accessible in the @AfterTestSuite
-annotated method in a TestListener.
Unfortunately, it is not the case. When a @AfterTestSuite
-annotated method invoked, the HTML/XML reports are not present there yet.
I can show you an evidence.
When you run the Test Suite Collection TSC
, you should be able to find the following messages in the Console.
TS1ReportDirLister output ----------------------------------------------
executing: ls -la ./Reports/20211023_172734/TS1/20211023_172736
return code = 0
total 40
drwxr-xr-x 6 kazurayam staff 192 Oct 23 17:27 .
drwxr-xr-x 3 kazurayam staff 96 Oct 23 17:27 ..
-rw-r--r-- 1 kazurayam staff 2649 Oct 23 17:27 execution.properties
-rw-r--r-- 1 kazurayam staff 11108 Oct 23 17:27 execution0.log
-rw-r--r-- 1 kazurayam staff 0 Oct 23 17:27 execution0.log.lck
-rw-r--r-- 1 kazurayam staff 64 Oct 23 17:27 testCaseBinding
This message is emitted by Test Listeners/TS1ReportDirLister. It executes a Command ls -la ./Reports/20211023_172734/TS1/20211023_172736
in the @AfterTestSuite
-annotated method. As you can see, there is a bulky execution0.log
file but no HTML/XML reports yet.
The fact is, Katalon Studio will compile HTML/XML report after @AfterTestSuite
-annotated method finished. Therefore you can not make a zip of the report in the @AfterTestSuite
-annotated method after TS1
.
Variable path
In order to write a script to zip a folder, the script must know the path of target folder. On the other hand, Katalon Studio assigns the path of a report folder for TS1
like:
Reports/20211023_125201/TS1/20211023_125203
As you can see, this path has variable parts (time stamp). How can my script find the concrete value of the variable folder path?
How to run a Command from Groovy
In order to create a zip file of a folder contents, I want to use the good old bash shell commands: zip
, mv
, cp
, rm
, mkdir
, echo
etc. Also I want to use curl
command to transfer files over network.
I will create, by Groovy script (TestListener), a shell script file named consume_TS_report.sh
. Then, how can I execute that shell script with /bin/sh
(bash shell interpreter) from my Groovy script (TestCase)?
Proposed solution
Resolving timing issue
You can not make a zip of the Test Suite TS1
by TS1 itself. Possible solution is to create another Test Suite TS_post_TS1_processing
. You want to create a Test Suite Collection TSC
and let it call TS1
and TS_post_TS1_processing
sequentially in this order. At the timing when TS_post_TS1_processing
is activated, the reports of TS1
has been already compiled. The following Test Suites should be able to read the reports.
Resolving variable report folder path issue
com.kms.katalon.core.configuration.RunConfiguration
class implements String getReportFolder() method. With this method call, we can get to know the report folder path of the current Test Suite.
Resolving how to run command from Groovy
The java.lang.ProcessBuilder
class enables running arbitrary commands in a new process forked from Groovy script. The following article covers how to use it.
However I found that ProcessBuilder is difficult to use for simple use cases. So I have developed a small wrapper for ProcessBuilder named subprocessj
.
The jar is already bundled in the Drivers
folder of this Git project.
You can download the jar of subprocessj
from the Maven Central repository.
The tutorial of subprocessj
is here:
The source code of subprocessj
is hosted here:
Description
First half
When you run the Test Suite Collection TSC
and when the Test Suite TS1
has finished, the Test Listeners/PostTS1Processor will be invoked. It has @AfterTestSuite
-annotated method. Please read the source.
What does the method do ?
- if the Test Suite
TS1
has been finished, - identify the path of report folder of
TS1
- in the
out
folder, create a file namedconsume_TS_report.sh
- write a few lines of bash commands. The commands will do the following:
- make a zip file of the report folder
- move the zip file into the
out
folder - try to transfer the zip file by HTTP POST request to a remote URL
- The
out/consume_TS_Report.sh
is something like the following:
zip Reports_20211023_172736 -r "Reports/20211023_172734/TS1/20211023_172736"
mv Reports_20211023_172736.zip "out/"
# I know this would fail
curl -X POST https://localhost:80 -F 'file=@out/Reports_20211023_172736.zip'
- do "
chmod +x consume_TS_report.sh
" to change the file permission to make it executable in the OS command line.
In the
out/consume_TS_Report.sh
script file, I wrote a line ofcurl
command to transfer the zip file to a URL (https://localhost:80), which will inevitably fail as the URL is not alive. This is just an example. You can do anything you like using shell commands here.
Second half
The Test Suite Collection TSC
will call the Test Suite TS_post_TS1_processing
after TS1
.
TS_post_TS1_processing
calls a Test Case TC_consume_TS1_report
.
What this test case does? For the detail, read the source code.
- The test case script locates the
out/consume_TS_report.sh
- It forks a new OS process to calls
/bin/sh
(bash shell interpreter) while specifying the script file. In short, it execute the script file. It waits for the forked process to finish. - When the forked process finished, the test case checks the return code. If the return code != 0, the test case fails.
- It prints the error messages from the forked process if any.