I have published a demo project at GitHub.
Problem to solve
Katalon Studio’s “Log Viewer” slows down your test execution significantly. Are you aware of this fact?
Maybe not. So I will report my analysis here. I can explain how to make it better.
Measurement result
Let me go straight to the point. The following table shows the result I measured how long a Test Suite took to finish running. I used a single Test Suite while I applied several variation of the “Log Viewer” setups.
As the following table shows, in the case 1, my test suite took 5 minutes 37 seconds to finish. But the same code finished in 25 seconds in the case 9 . This difference proves that the “Log Viewer” slows down your tests. How you set up Log Viewer — it matters significantly to the speed of your tests.
Case | log executed test steps | Widget is | Mode | log level | scroll | duration | duration graph |
---|---|---|---|---|---|---|---|
1 | Enabled | Attached | Tree | - | enabled | 5 minutes 37 seconds | ################################## |
2 | Enabled | Attached | Tree | - | locked | 3 m 25 s | ##################### |
3 | Enabled | Attached | Table | All | enabled | 3 m 4 s | ################## |
4 | Enabled | Attached | Table | All | locked | 1 m 36 s | ######### |
5 | Enabled | Attached | Table | F+E+W | enabled | 56 s | ###### |
6 | Enabled | Attached | Table | F+E+W | locked | 1 m 10 s | ####### |
7 | Enabled | Detached | Tree | - | enabled | 5 m 35 s | ################################## |
8 | Enabled | Detached | Tree | - | locked | 4 m 15 s | ########################## |
9 | Enabled | Closed | - | - | - | 25 s | ### |
X1 | Disabled | Attached | Tree | - | enabled | 3 m 53 s | ####################### |
X2 | Disabled | Attached | Tree | - | locked | 3 m 2 s | ################## |
X3 | Disabled | Attached | Table | All | enabled | 2 m 33 s | ############### |
X4 | Disabled | Attached | Table | All | locked | 1 m 41 s | ########## |
X5 | Disabled | Attached | Table | F+E+W | enabled | 2 m 9 s | ############# |
X6 | Disabled | Attached | Table | F+E+W | locked | 1 m 59 s | ############ |
X7 | Disabled | Detached | Tree | - | enabled | 5 m 3 s | ############################## |
X8 | Disabled | Detached | Tree | - | locked | 4 m 3 s | ######################## |
X9 | Disabled | Closed | - | - | - | 25 s | ### |
Code to run
I made a Test Suite, a Test Case, and a CSV file as a test fixture.
Test Suite TS1
I made a Test Suite TS1
, which applies “A. Execution from test suites” as described in the article “Data-driven testing approach with Katalon Studio”.
TS1
calls the Test Case printID
for all rows in the data.csv
file, which contains 1000 lines. TS1
repeats calling the printID
script 1000 times.
Test Case printID
I made a Test Case Test Case/printID
, which is minimal:
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
WebUI.comment("ID=${ID}")
This printID
declares a variable ID
as:
The ID
variable will be populated by TS1
with data picked up from data.csv
file.
data.csv
file
I made a CSV file: data.csv.
ID
#0000
#0001
#0002
#0003
#0004
...
#0998
#0999
This file contains 1000 lines. TS1
reads this, iterate all lines, find the value of ID
column, which is passed to the printID
test case.
How I measured the duration
When I say “the TS1 took 5 minutes 37 seconds”, how did I record when it started and when it ended?
As soon as I clicked the run button to start TS1
, a “Job Progress” modal window will open.
In the “Job Progress” window, I found a figure, like 37/1000
which goes on incrementing. This means, the Test Suite TS1
is repeating to call the Test Case printID
for 1000 times as total, and it has finished 37 times.
I used a Timer app on my Android mobile phone to measure the duration. I started it as soon as I clicked the run button; wait for a while. When the “Job Progress” showed 1000/1000
, I stopped the timer. This is the way how I measured the duration of TS1
.
Log Viewer setup options
Here I will enumerate the options of Log Viewer setups.
Log Viewer widget can be Attached/Detached/Closed
Usually a Log Viewer widget is attached in the Katalon Studio’s window.
By a right-clicking the name tab, you can detach the Log Viewer widget from the Katalon Studio’s window. The following image shows how it looks like.
Even if detached, the Log Viewer widget is still alive and in action.
You can even close the window of the detached Log Viewer widget.
Once Log Viewer widget has disappeared, Log Viewer is no longer there.
If you stop and restart Katalon Studio GUI, the Log Viewer widget will revive.
Mode of Log Viewer
Log Viewer has 2 formats. Namely, “Log view” and “Tree view”. You can choose by toggling the button.
Log view
Tree view
Log type options
In the Log view, you can select which type of logs to be displayed: All, Info, Passed, Failed, Error, Warning, Not Run
Step Execution Log
If you select “All” in the Log view and run a test, you will see quite a lot of “START” and “END” logs are printed.
If you deselect “All”, then no START and END logs will be visible.
Log executed test steps - Enabled/Disabled
The START and END logs are also called “step execution logs” .
In the “Project Settings > Execution” dialog, you will find an option: “Log executed test steps”:
If you have a Katalon Studio Enterprise license, you can disable logging START and END.
If you do not have the Enterprise license, you will be advised to purchase it.
Scroll Lock
Log Viewer widget has a toggle button with “Lock”-like icon, which is labeled “Scroll Lock”.
If you toggle it ON, the Log Viewer stops automatic scrolling. Even when a test emits thousands of logs, the “Log view” will show only 10 or 20 lines at the top only and stay quiet. But the “tree view” will continue trembling even if it is “Scroll Locked” while a test is running.
How is “Log Viewer” setup initially
When you newly installed Katalon Studio or you have upgrade it to a newer version, the Log Viewer will be automatically re-configured as follows:
- Log Viewer widget is attached into the Katalon Studio window.
- Log Viewer shows Tree view, rather than Log view, initially
- In the Log view, “All” level is selected initially
- The “Scroll Lock” is off initially
I believe that most of the Katalon Studio users are using it with this Log Viewer setups unchanged. This means, you are running your tests as the slowest “case1”.
Conclusion
If your test is running quick enough and you are happy with it, forget me! You don’t need my advise.
In order to make your tests run faster, I would advise you to follow this:
- You should not use the Tree view of the Log Viewer; you should prefer the Log view.
- In the Log view, you should never select the “All” level to print, as it emits bulky “step execution logs”.
- In the Log view, You should select only the levels you need: e.g, “Failure” + “Error” + “Warning”. This will reduce the volume of logs to be printed. You can add “Info” if you like.
- If you have an Enterprise license, you should set the “Log executed test steps” disabled.
- You could detach the Log Viewer widget and close it. Then your tests will run at the highest speed without being slowed down by the Log Viewer.