KS 8.1.0 stucked in running (after more than 5 days of running long haul test) - need help to generate partial report

[anon46315158]

@kazurayam

You may try enlarging it, for example:

-Xmx20g

raise also the Xms to at least half of Xmx to became faster available for JVM

@kazurayam

Memory leak, if any, is caused by your own test scripts; you should not doubt Katalon Studio. You should check your code more carefully.

well, let’s don’t forget that as long as the test is still running, is collecting data for the final report.
(which, apparently, it happens in memory)
we cannot say that this is a pure memory leak but more like unhappy design (if that is the case).
to confirm this, some voodoo debugging may be needed.
perhaps the dev team may want to review that part and find better solutions for such cases (.e.g flush the data to disk)
the main issue is caused due to the large dataset
but yeah, unhandled network resources can create also strange issues

[Russ_Thomas]

@anon46315158

perhaps the dev team may want to review that part and find better solutions for such cases (.e.g flush the data to disk)

Exactly.  

@Russ_Thomas

@duyluong @ThanhTo

Feature Request

Devs – can I suggest that this is taken up as a feature request? A means to access on-going results as a test/suite is running? If KS is waiting until the test is complete to flush a report to disk, that’s really not good in long haul tests like this (@duyluong recall I have a test that runs for 13 hours overnight).

[kazurayam]

@david.casiano

It is possible that there is a memory leak somewhere

If you want to see more about the runtime memory usage, you should try

• Visual VM

which is free, a proven profiling tool for any JVM applications.

A sample how Visual VM looks:

This screenshot shows a case in my project, not yours of course. Have a look at how the figure of “Heap size” and “Used heap” are moving. Please note the “Heap size” goes up and stay to 7GB (as I specified -Xmx7g), while “Used heap” is going around 2GB; there is 5GB of Heap space once used and left un-freed. In this situation, Garbage Collection becomes too busy trying to free the heaps space, then application processes do less and less, eventually stops. I mean this phenomenon a “Memory-leak stuff”. This memory-leak stuff was caused by a fault in my test script; it left a lot of i/o buffers unclosed.

@Russ_Thomas

Devs – can I suggest that this is taken up as a feature request? A means to access on-going results as a test/suite is running?

I don’t think this request is practical. When a test script is faulty and causes this sort of “Memory-leak stuff” in the Java VM layer, I believe, all of application processing on a Java VM process will stop working. The code of Katalon Studio in the process will also stop functioning.

[anon46315158]

@kazurayam

This screenshot shows a case in my project, not yours of course.

Exactly!
So, up to now you just have assumptions on what is the actual root cause (as we do also).

Therefore I propose:

@david.casiano should take an in depth look into the code and debug it, with the hope it may be able to identify the actual “leak”.
the hints from @kazurayam will be of a great help, provided the tools are used properly.

@duyluong the dev team should consider the proposal from @Russ_Thomas an make the reporting part better.

Until then, we are just running into circles.
Good enough?

[Russ_Thomas]

@anon46315158

@duyluong the dev team should consider the proposal from @Russ_Thomas an make the reporting part better.

The nice and friendly thing to do would be to make it readable by a browser:

file:///<katalon-project>/reports/<date-time>/<suite-name>/report.temp.html

Then just wait for a (configurable) auto-refresh or hit F5.

I don’t see any engagement though, and that’s a shame. Probably not sexy enough – and doesn’t involve TestOps, either, which is where most of the development effort seems to be going these days. But really, devs, it just needs to work.

---

The cognoscenti will recall how my reporting system works, documented in bits around this forum. I could implement this in my stuff in an afternoon (or two).

[david.casiano]

Updating this thread. It seems that the approach to use “Test collection” seems to work better. Our observation seems to reducing the entries in the “Report” folder decreases the memory. As I move out the previous reports(created by Katalon after test finishes) to another directory seems to reduce the memory(still big at over 1.2 GB of memory at startup with the project - no tests have been run).
Also, with the “Test Collection” approach to distribute the data across multiple test suites increased performance as I can process quicker thru the data files. However, this time, it got stucked at the end (‘frozen’ UI after 3 days). But the good news was I got the reports for the test suites that finished.

Thanks for all the tips!

[kazurayam]

@Russ_Thomas

@Russ_Thomas

The nice and friendly thing to do would be to make it readable by a browser:

file:////reports///report.temp.html

Then just wait for a (configurable) auto-refresh or hit F5.

I have a solution for me.

1. save this Groovy script httpserver.groovy into your Katalon projects root folder.
2. here I assume you have Groovy interpreter is installed and operational in the commandline. If not, install it as Official doc tells.
3. In the command line, you want to :

$ cd <your_katalon_project_folder>
$ groovy httpserver.groovy

4. You will see in the command line, a HTTP server starts and displays such message:

$ groovy httpserver.groovy

port=80
base-dir=/Users/kazurayam/katalon-workspace/VisualInspectionInKatalonStudio_Reborn/.

5. In any browser, open the URL http://localhost
6. You will find the project directory list is rendered in the browser, like this:
   
   
7. you can view any files in the Katalon project. Not only the built-in reports, you can view any custom-made reports in browser through this local HTTP server.
8. You do not have to transfer report files to remote servers (like TestOps), which takes long time. As soon as the custom reports are generated locally, you can view it at URL http://localhost. Quite handy.
9. to stop the local HTTP server, type CTL+c in the commandline
10. you can change the listening port from #80 to any you want by command line parameter. Also you can change the base-directory by parameter.

This HTTP server is developed by opengl-8080 and published at GitHub - opengl-8080/groovy-http-server: Simple HTTP Server by Groovy. I modified the original slightly.

[kazurayam]

I find this local HTTP server is useful for quick researches. For example, I used this server to witness a bug in Katalon Studio’s RequestObject class.

• Katalon POSt Request with multipart/form-data and formdata fields conrtains utf-8 text not submittd the text in the same format

In this Katalon User Form, many people report problems about WebService tests. However, quite often it is difficult to reproduce problems without a RESTful/SOAP server counterpart up and running. This local HTTP server makes it easy to mimic the problem with test fixtures (JSON, XML files) manually prepared in the project directory locally.

[kazurayam]

@david.casiano

Our observation seems to reducing the entries in the “Report” folder decreases the memory.

Then, you would want to decrease the size of files in the “Report” folder. That may result less memory occupied.

If you have a payed license of Katalon Studio Enterprise, then you can disable “log executed steps”, which will significantly reduce the size of report files. The “step execution logs” occupies approximately 90% of bytes of report files; but that information is not useful (I suppose you do not need the step execution logs). See the following post also:

TestSuite with 127 tests freezes

@artsoft, The problem may come from Log Viewer. Please share the report folder and the execution log with us, we will investigate later. For the workaround, you can try to disable Log executed test step in Project Settings/Execution [Screen Shot 2021-11-16 at 11.21.05 AM]

[anon46315158]

@kazurayam

I have a solution for me.

1. save this Groovy script httpserver.groovy into your Katalon projects root folder.

well, of-course, one may use python too, provided is already installed.
just do:

$python -m SimpleHTTPServer <port_number>

in whatever folder you like.
no need to grab any third-party script, this module is already provided with a standard installation.

but the idea of the current topic was that the reporting part has to be improved, in regards with memory efficiency.
note that the final html report it is not generated until the test suite is ended gracefully.

[Russ_Thomas]

@anon46315158

note that the final html report it is not generated until the test suite is ended gracefully.

Precisely. (He gets it !) We want it as it is being constructed, not wait until it’s flushed to disk - that’s the point. This requires new code in Katalon to flush the steps to disk, step by step (not TC by TC and certainly not per suite, which is what we have right now). I’d even settle for simple “outcomes” - pass/fail/error.

Great post though, @kazurayam. I’m sure it will help somebody. But like @anon46315158 said, browsing local folders is easy. I use file:/// stuff all the time – I have seven tiddlyservers (node) open right now.

[kazurayam]

@Russ_Thomas

A means to access on-going results as a test/suite is running?

Katalon Studio emits runtime logs into execution0.log file.

$ pwd
/Users/kazurayam/katalon-workspace/ReportWriterDriver
:~/katalon-workspace/ReportWriterDriver (master #)
$ tree Reports -s
Reports
├── [         96]  20211117_145442
│   └── [         96]  TS1
│       └── [        320]  20211117_145442
│           ├── [     197916]  20211117_145442.html
│           ├── [       1803]  JUnit_Report.xml
│           ├── [       2663]  execution.properties
│           ├── [         36]  execution.uuid
│           ├── [     127247]  execution0.log   <====== THIS!
│           ├── [        144]  testCaseBinding
│           ├── [          0]  tsc_id.txt
│           └── [        128]  videos
│               ├── [     861216]  screen_recording_1_0.avi
│               └── [      10079]  screen_recording_1_0.srt
└── [         96]  Self-healing
    └── [         31]  broken-test-objects.json

5 directories, 10 files

The content of the execution0.log file is something like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE log SYSTEM "logger.dtd">
<log>
<record>
  <date>2021-05-31T11:54:33</date>
  <millis>1622429673295</millis>
  <sequence>0</sequence>
  <level>START</level>
  <class>com.kms.katalon.core.logging.XmlKeywordLogger</class>
  <method>startSuite</method>
  <thread>1</thread>
  <message>Start Test Suite : Test Suites/test/All</message>
  <nestedLevel>0</nestedLevel>
  <escapedJava>false</escapedJava>
  <property name="name">All</property>
  <property name="description"></property>
  <property name="id">Test Suites/test/All</property>
  <property name="stopImmediately">false</property>
</record>
...

The execution0.log file is generated by the Java Standard Logging framework: java.util.logging.XMLFormatter etc.

The content of execution0.log file is exactly the same as what we see in the Log Viewer in the Katalon Studio GUI. This means, the execution0.log file contains the on-going results as a test suite is running.

Katalon Studio transforms the execution0.log file to what we call “Basic Report as HTML” and “JUnitReport as XML”.

Therefore, if you want a means to access on-going results as a test/suite is running, then you want 2 things:

1. You want to trigger Katalon Studio to do execution0.log → HTML report transformation intentionally even when a Test Suite has crashed for some reason.

2. When you forcefully stoped Katalon Studio because the Test Suite got frozen, then the execution0.log file might become a mal-formed XML document. For example,

   • the file may miss the closing </log> tag at the end of the file
   • a <record> tag may be broken at the end of the file.

Possibly, the execution0.log file is generated when a Test Suite has finished (regardless with failure or not). I am not sure how the log file will be generated when the OS process was stopped irregularly.

Do you want Katalon studio to be capable of transforming a mal-formed execution0.log to a well-formed Basic HTML report? I do not think it is possible to implement.

A super-skillful Java programmer may be able to write a purifier tool that accept a mal-formed execution0.log file to make it a well-formed XML. Then the magic may become possible. — sorry, I am not capable.

[anon46315158]

In addition to what has been said above, perhaps I have to be a bit more precise.
When I am saying ‘reporting’ I refer also to the log viewer.
And that i know for sure it is memory hungry, since it is populating the GUI ‘on the fly’ but seems like it is not releasing the memory until the test finishes (or sometime not at all).
So the log viewer definitely need some love.

[kazurayam]

@anon46315158

the log viewer.
And that i know for sure it is memory hungry

Yes, it is.

as @duyluong wrote yesterday as follows, the Log Viewer seems to have problem.

TestSuite with 127 tests freezes

@artsoft, The problem may come from Log Viewer. Please share the report folder and the execution log with us, we will investigate later. For the workaround, you can try to disable Log executed test step in Project Settings/Execution [Screen Shot 2021-11-16 at 11.21.05 AM]

[kazurayam]

A Katalon Studio Enterprise license holder can disable the “step execution logs”, but this option is not available to the Katalon Studio Free license holder. I am uncomfortable about this design decision. In Dec 2020, I wrote that Katalon should change this option to be free.

Performance of logging execution steps

I agree with you. @ThanhTo @devalex88 @duyluong In this couple of weeks, I found number of posts in this forum raised performance issues, like this. The verbose Step Execution Log affects them badly. Let me remind you that, as your products are developed, so are the users as well. They will try larger scale projects, and they are likely to find “Katalon Studio is too slow”. This reputation would harm your business. I would suggest to you Katalon team to change your mind. The “Enable/Disabl…

But the team has not taken this seriously.

I am afraid, I should advise those who want to drive a large scale test (like you @david.casiano) should not use Katalon Studio Free version. They need to pay for the Enterprise license. Otherwise, they should choose other UI-testing software.

@Jass

Let me tell this you. I think this was a bad design decision.

[anon46315158]

@kazurayam

A Katalon Studio Enterprise license holder can disable the “step execution logs”, but this option is not avaiable to the Katalon Studio Free license holder.

And this is were we are starting to converge.
This option it is available only for enterprise users, as you correctly pointed.

And, even if this will be made available for free version, the user may still like to have an option to watch the reports ‘on the fly’, particularly for such cases with large datasets / long running suites.

So, no matter if i am an enterprise user or free user, I should have the option to turn off the log viewer, but refreshing the browser some time to time (I won’t mind doing this manually)

This is why both requests should be considered.

Of-course, running the tests with KRE may be a solution (since the console runner shouldn’t bother with the log viewer) but KRE … guess what, it is yet another licensed only feature …

[kazurayam]

to the product owner of Katalon Studio, @Jass

Let me tell you, I see Katalon Studio is not good for large scale testing.

[anon46315158]

@kazurayam

I am afraid, I should advise those who want to drive a large scale test (like you @david.casiano) should not use Katalon Studio Free version. You need to pay for the Enterprise license. Otherwise, you should choose other UI-testing software.

I am already on the ‘dark-side’ (pyhton + robot framework).
And actually … funny, the main reason for this was not the development inconsistencies and licensing schema … but something more serious for me … the Linux support.
Yet again, starting from 7.9.0, the linux version is broken, at least on Fedora, due to the eclipse upgrade most probably.
But I won’t bother to submit another bug report, looks like the Linux usage it is simply ignored … some people are thinking that Linux is running only in the cloud …

[kazurayam]

@anon46315158

But I won’t bother to submit another bug report,

Have you ever reported a bug report of Katalon GUI on Fedora? Which URL? Let me bookmark it!

[kazurayam]

@anon46315158

at least on Fedora, due to the eclipse upgrade most probably.

Is the Eclipse project on a decline nowadays?