Hi. I want to get a report ONLY on errors, not all the steps, how can I do this?
Is this what you’re asking about?
And yes and no, but I meant about exporting the report, of course I can use the option as in the screenshot. But it would be great if, immediately after the first run of the test suite, the unloading was only steps with an error.
I feared you might be asking for that. As far as I know, there is no way to accomplish what you need, directly in Katalon out of the box.
All I care about is failures/errors. For that reason, I decided to write my own reporting system. All I see now is exactly what I need to see – failures.
+1
The built-in report does not support any filtering at all.
A proposal from me:
I made a Test Suite “TS1”, which comprises 2 Test Cases: “FailingTC” and “PassingTC”:
When I executed the TS1, I got some files in the <projectDir>/Reports folder, as this:
$ tree -h Reports
[ 128] Reports
├── [ 96] 20230222_080435
│ └── [ 96] TS1
│ └── [ 320] 20230222_080435
│ ├── [ 486] 20230222_080435.csv
│ ├── [201K] 20230222_080435.html
│ ├── [8.9K] JUnit_Report.xml
│ ├── [2.7K] execution.properties
│ ├── [ 36] execution.uuid
│ ├── [ 17K] execution0.log
│ ├── [ 206] testCaseBinding
│ └── [ 0] tsc_id.txt
└── [ 96] Self-healing
└── [ 31] broken-test-objects.json
5 directories, 9 files
The execution0.log file will be newly created with size=0 as soon as the TS1 started. TS1 will append logs to the execution0.log file as it goes. When TS1 reached to the end of processing, the execution0.log file will have fully filled. The file will be available to the @AfterTestSuite-annotated method of a Test Listener.
However, all the other files (.csv, .html, .xml) will NOT be there in the Reports folder when the @AfterTestSuite-annoted method of a Test Listener is invoked. These files will be created by Katalon Studio after the TS1 finished and ceased away. The files will only be available to the following processes after the TS1’s run.
There is a CSV file, which containes lines like:
$ cat 20230222_080435.csv
Suite/Test/Step Name,Browser,Description,Tag,Start time,End time,Duration,Status
TS1,,,,2023-02-22 08:04:38,2023-02-22 08:04:39,0.827s,FAILED
,,,,,,,
Test Cases/FailingTC,,,,2023-02-22 08:04:39,2023-02-22 08:04:39,0.204s,FAILED
"markFailed(""I am bad"")",,,,2023-02-22 08:04:39,2023-02-22 08:04:39,0.039s,FAILED
,,,,,,,
Test Cases/PassingTC,,,,2023-02-22 08:04:39,2023-02-22 08:04:39,0.205s,PASSED
"comment(""I am OK"")",,,,2023-02-22 08:04:39,2023-02-22 08:04:39,0.114s,PASSED
It would be easy for you to develop some program that processes this CSV file. You can make the result of “failure only”.
Is this CSV not enough for you? Then you can look at the execution0.log file in which every outcome from the TS1 execution is recored. You can write a reporting program that consumes this file in the way whatever you like.
How do you find this idea (wrote some post-processing programs)? Not satisfactory? ---- Then tell us what you find unsatisfactory. That will tell Katalon team what you want more clearly.
Possibly you have an instance of HTML report, which could be very large, which you are not satified about somehow. Please show it to us and tell how you want to change it. With a concrete requirement, Katalon team might be interested in it.
1 Like
Yes. In my case, TS turns out to be huge. Sometimes you have to do such tests and manually handle each unsuccessful case is very tedious. And coming up with crutches is precious time. It would be very cool if there was some kind of filter on reports before launching.
Остановки 1 часть.html (14.4 MB)
Thank you for sharing your sample HTML.
I found it is as large as 14.4MB. Terribly large! I guess, this gigantic size of the file would cause various difficulties for you. It would be great if you can shrink the size of the report HTML.
Are you a paying customer? If you have a paid license, you can turn the “Log executed test steps” option to “Disabled”.
If this option is set Disabled, the execution log would become far smaller, the Report will be far light-weighted. You would find it preferable. Please try.
Unfortunately, neither I nor the organization can afford a subscription. And the functionality is limited. I have sent the full text of the suite, at the moment it is divided into 4 parts. In any case, in total time, the run takes more than 8 hours. I can put up with such a long run, but the result does not correspond to what I want so far and I have to use the “Retry failed executions only” option, which increases the time by another 1 hour + - depending on the number of errors found.
Terrible.
Have you ever seen my post below. You may find a way to run your test much faster: 8hours → less than 4 hours, possibly.
Thanks. I will try to implement this in my case and unsubscribe based on the result. 
??? what do you mean?
I wanted to say that I will compare the results obtained with previous tests. I already have reports in my mind to the management because I spent a lot of time on this situation. 
I noticed that your test case calls WebUI.delay(2) twice. This will make your test run slower for 4 seconds than necessary.
You should not use WebUI.delay() for fixed duration of wait. Instead you should use WebUI.waitForXXXX(TestObject, timeout) keywords.
- [WebUI] Wait For Element Clickable | Katalon Docs
- [WebUI] Wait For Element Present | Katalon Docs
- [WebUI] Wait For Element Visible | Katalon Docs
- [WebUI] Wait For Page Load | Katalon Docs
The WebUI.waitForXXXX keywords will return immediately in milli-seconds when the condition test returned true; therefore it runs far faster than WebUI.delay(fixed 2 seconds).
The above sample test case took 6.922 second, which includes 4 seconds of fixed duration of wait. Well, it is waiting too long, isn’t it ?
I noticed your test suite contained 1804 times of Test Case execution. If each executions waited 4 seconds, that makes 1804 * 4 = 2 hours of explicit wait. Your test suite execution was slowed down for 2 hours due to unnecessary calls to WebUI.delay(2).
You wrote that test suite took 8 hours. Repetitive call to WebUI(2) occupied 2 hours. WebUI.click() and WebUI.setText() calls (plus interactions with your web app) took possibly 3 hours. 2+3 makes only 5 hours. Then 8hours minus 5hours => 3 hours: for what reason this 3 hours passed? ----> I presume that the LogViewer delayed your test suite execution for 3 hours. So, I think your 8 hours could be shortened to 4, possibly.
I’ll try to use it. Delay had to be used because the tested site is so crookedly assembled that even xpath sometimes does not pick up when recording and you have to manually register. Therefore, I came to the conclusion that delay will be easier, albeit at the expense of time. + api jambs and the answer has to wait. Thanks
I have made a prototype:
page.html (49.1 KB)
Please download this HTML to your local PC, open it with your browser.
Initially it will show both of the PASSED and FAILED, and will show a button labelled “Hide the PASSED”.
When you click the button, the PASSED will be turned hidden; only the FAILED will remain shown. The button label will change to “Show the PASSED”.
When you click the button again, the PASSED will be displayed again.
The prototype HTML is small. It has only 1 <div> of PASSED and 1 <div> of FAILED. But the number of divs does not matter much. Even with 2000 <div>s, the button would work in a few seconds.
A small twist in the UI design as I showed here would help you. How do you like this idea?
However I can not change Katalon Studio to generate the HTML report like this. Only Katalon team can work on this issue.
I changed the title of your topic
“How to get a bug report ONLY” → “Report only FAILED tests; no PASSED tests please”
to express the original question better.
I know, the following URL shows us the source code of the program modules that compiles the Report HTML.
public static void writeLogRecordToFiles(TestSuiteLogRecord suiteLogEntity, File logFolder) throws Exception {
// compile report in HTML
writeHtmlReport(suiteLogEntity, logFolder);
// compile report in CSV
writeCSVReport(suiteLogEntity, logFolder);
writeSimpleHTMLReport(suiteLogEntity, logFolder);
// compile report in JUnit XML format
writeJUnitReport(suiteLogEntity, logFolder);
}
I believe, Katalon Studio calls this method of this class to compile the reports when a Test Suite finished.
Do you have your Katalon Studio project integrated with Test Ops?
You should review if you need it or not.
Your execution log is huge; therefore it would take very long time to transfer logs from your local PC to the Test Ops. In order to shorten the processing time, possibly you had better not integrate the project with TestOps.
I have made a public GitHub project
This is a small Katalon Studio project for demonstration purpose.
You can download the zip of this project from the Releases page, unzip it locally to open in your Katalon Studio.
Second best proposal
I believe, Katalon Studio Standalone Edition calls com.kms.katalon.core.reporting.ReportUtil#writeLogRecordToFiles() to compile the reports when a Test Suite finished.
I can not change the Katalon Studio itself, but I would be able to reuse the open-sourced codes to produce a second best solution.
I created a GitHub project “ReportFailedTestOnly”:
There I developed a set of Groovy classes, and created a jar, which is published at
Using this jar, any Katalon Studio users can let their projects to produce a HTML Test Suite report which can display Failed tests only.
How to use the solution
You want to create a Katalon Studio project with a Test Suite.
You want to download the ReportFailedTestsOnly-0.1.1.jar and locate it in the Drivers folder. You want to close the project and reopen it to let Katalon Studio recognize the downloaded jar.
You want to create a Test Case like this:
Here I named it as AltTestSuiteReport, but you can name it any.
The Test Case contains the name of Test Suite in the code, like
Path reportDir = rdu.getLatestTestSuiteReportDirOf("TS1")
The example has a Test Suite named as "TS1". You MUST change this string constant to be your own Test Suite name.
Now you are ready.
You want to run your Test Suite as you have done many times.
Once your Test Suite finished, you want to run the new Test Case AltTestSuiteReport. It wil run fast enough. It will take 10 seconds or so, but will not take any hours.
The test case will create a new file at
<projectDir>/AltTestSuiteReport/AltTestSuiteReport.html
You want to open this HTML with any web browser.
You will find the page looks similar to the built-in report, but slightly different. It has a button on the top.
It starts with a page where both of PASSED tests and FAILED tests are displayed:
By clicking the button “Hide the PASSED”, you can change the page to show Failed tests only.
The label of the button will change to “Show the PASSED”. By clicking the button again, the PASSED test cases will come up.
Movie
Here is a video that shows how I operated this project:
Why I made this
Katalon Studio offers Data driven testing. I suppose this featured is liked by users a lot. Once accustomed, users tend to create large data sets to drive their tests. For example, the OP created approximately 2000 lines of data. I suppose he/she is not exceptional. Rather he/she is a typical user of “Data driven testing”. Provided with this volumy data, Katalon Studio Standalone Edition will create a HTML Report that contains 2000 test cases, which is too large. The report is so large that it becomes useless. Anyone feels like to see only FAILED tests, just a few, amongst 2000. They do not like to wander in the mass. A huge report without filtering capability makes us frustrated.
However, I guess that Katalon team would not address this issue because I know in the last few years they have been busy for developing their TestOps product. They may say “please stop using the free version, please evalutate the Platform Editon on the Cloud”. I don’t know if the Platform Editon makes them happy.
I was worried that users of local Katalon Standalone Edition, including @jmad and me, will be left unsatisfied for more years to come. Therefore I have worked out for this 2nd-best solution “ReportFailedTestsOnly”. Please try it.
3 Likes
@kazurayam as always, another example of, if issues are taken seriously, solutions may be found.
Unfortunately, as already mentioned, looks like the efforts from Katalon staff are focused on the platform edition promote mostly.
I am still unsure if that is a good thing or a bad thing.
I do understand the advantages of cloud and SaaS and I am a big fan on such.
But details matters, if the needs of the users are ignored and the focus is just to ‘embrace the new and amazing platform’… in the detriment of functionalities… mhm.
I raised my concerns already on this matter since the early begining of Platform Edition.
The Katalon staff aparently have a vision.
If it is good vision or not, time will tell.