Refresh issue in test suite collection

[p.kayam]

Operating System

Windows 11

Katalon Studio version

e.g. version 8.5.5

Log Folder:

• Windows logs folder: Katalon Studio folder>\config\.metadata\.log
• MacOS logs folder: file:///Applications/Katalon%20Studio.app/Contents/MacOS/config/.metadata/.plugins/org.eclipse.ui.workbench/log

Environment (for Web Testing)

• Chrome & Version 108.0.5359.125 (Official Build) (64-bit)

Steps to reproduce

When ever there is new test case is added in the test suite then in the suite collection need to remove that test suite and add again before execution else those test cases are not included

[waqas]

If you want to include newly added test cases in a test suite before execution, you can do that by updating the test suite in the suite collection. Here’s an example of how you could remove a test suite from a suite collection and add it again to include newly added test cases:

import com.kms.katalon.core.testcase.TestCaseFactory

// Get the test suite you want to update
TestSuite testSuite = TestSuiteFactory.getTestSuiteByName("My Test Suite")

// Remove the test suite from the suite collection
TestSuiteCollection.removeTestSuite(testSuite)

// Add the test suite to the suite collection again
TestSuiteCollection.addTestSuite(testSuite)

[p.kayam]

@waqas

newly added test cases in a test suite before execution, you can do that by updating the test suite in the suite collection. Here’s an example of how you could remove a test suite from a suite collection and add it again to include newly added test case

I will try to implement this, Thanks for reply.

[kazurayam]

@p.kayam

I tried to reproduce your case on my machine. I could. I think you, @p.kayam, made a small mistake. And Katalon Studio has a defect that confused you. Let me tell you what I did.

Scene A

Explation of scene A

1. I made a Test Case “TCx”, which just displays “xxxxxxx TCx is running xxxxxxxxxx”
2. I also made a Test Case “TCy”, which just displays “yyyyyyyyy TCy is running yyyyyyyy”
3. I made a Test Suite named “TS1”. I included only TCx.
4. I made a Test Suite Collection named “TCS”. I included the TS1.
5. When I execute the TCS, in the console a line of message “xxxxxxxx TCx is running xxxxxx” appeared. This was just expected. TCS runs fine.

Scene B

Explanation of scene B

1. I changed the TS1. I included the TCy in the TS1.
2. I exectuted the TSC. It showed a message “xxxxxxxx TCx is running xxxxxxxx” only. It did not emitted a message “yyyyyy TCy is running yyyyyy”. This is an insident!
3. I realised that I forgot to save the TS1. I did not click the green arrow “save” button. I left the TS1 unsaved, marked with an * mark.

Scene C

Explation of scene C

9. I realized my mistake that I did not saved the changed TS1. So I clicked the green button to save the TS1. The * mark disappered.
10. I executed the TSC. It emitted 2 lines of messages: “xxxxxx TCx is running xxxxxxx” and “yyyyyyy TCy is running yyyyyyyy”. This was as expected. TCS ran fine.

---

@p.kayam

I guess, you forgot saving the changed Test Suite, didn’t you?

[bionel]

@kazurayam

This behavior of Katalon Studio doesn’t look professinal.

May not look professional, but I made a habit of using Ctrl-Shift-S vs Ctrl-S when using various apps.
Usually this saves me from headeache (but sometime creates additional troubles)
‘Save All’ vs ‘Save’ sometime make the difference, and those keybindings are most common used.

[kazurayam]

I think there is a small defect in Katalon Studio. When I tried to execute the TSC while the TS1 was unsaved, I think Katalon Studio should not have started executing the TSC with unsaved TS1.

Katalon Studio should automatically save the TS1 before continuing; or should stop and warn “please save the TS1 befre running TCS”.

Unfortunately Katalon Studio ran TSC with unsaved TS1; this confused @p.kayam. This behavior of Katalon Studio doesn’t look professinal to me.

[kazurayam]

@bionel

May not look professional

IntelliJ IDEA, Visual Studio Code — these professional editors make sure that everything saved before user actions.

@bionel

but I made a habit of using Ctrl-Shift-S vs Ctrl-S when using various apps.

Me too.

[bionel]

@kazurayam

Katalon Studio should automatically save the TS1 before continuing; or should stop and warn “please save the TS1 befre running TCS”.

For me it did.

I used your sample project for browsermobproxy.
I opened two testcases and the testsuite.

I did few changes in the testcases (added some comments)

Without saving, I ran the suite.

Both testcases were automatically saved prior to run (the * disapeared and my changes are present after the suite run)

Well … i tested on linux, if it matters (and I am lazy to create a collection also) …

[kazurayam]

@bionel

Both testcases were automatically saved prior to run (the * disapeared and my changes are present after the suite run)

Perhaps Katalon developers implemented autosaving of Test Cases but did not implemented autosaving of Test Suite triggered by TCS execution.

[bionel]

@kazurayam

they did not implemented autosaving of Test Suite.

Ah … well, lazy to test that also, but probably you are right.
Will autosave testcases but this is the only test I did.
Indeed, looks like a bug to be considered.

[kazurayam]

→ @vu.tran

[p.kayam]

@kazurayam

r case on my machine. I co

My scenario is different, I pulled changes of my colleague in that there were some new testcases added in a test suite and that test suite is added in my test suite collection.
When I try to execute the test suite collection those test cases were not included if I remove the test suite and add it again those test cases was included.
I hope this is clear!

[kazurayam]

@p.kayam

I pulled changes of my colleague

After pulling the changes, did you close & reopen the project?

After pulling the changes, did you ever opened the new testcases in the Test Case editor in your Katalon Studio GUI?

My Guess

Unless you manually open a test case script in GUI, Katalon Studio may not recognize that it has got a new script in the pulled project.

In my experience, unless given a chance to refresh, Katalon Studio often goes out of sync with files on disk. See the following ancient post:

Nothing Refreshes Project Root folder

I’ve noticed something similar. If I ever copy a project across to a bunch of VMs, some sessions of Katalon notice the changes and ask if I want to update them. Other’s don’t - they either update blindly with no report, or, rarely, don’t update at all. Whatever they’re using in the Java core for file system watching is flaky at best.

Katalon developer stated they have fixed this issue. But I believe there remain some edge cases where Katalon Studio gets out of sync with files on disk. I’ve got bored with this issue. I am not going to debate this bug any longer.

What I do

I would just repeat closing and reopening projects when necessary. Especially I always do close&reopen a project after pulling it from a Git remote repository. It’s a rule of thumb.

[bionel]

@p.kayam

I pulled changes of my colleague in that there were some new testcases added in a test suite and that test suite is added in my test suite collection.

This is a different storry.
Did your colleague saved everything needed before comitting/pushing the changes?

To confirm this ‘bug’ additional test scenarios are needed …

@p.kayam

I hope this is clear!

Actually, this brings more confusion on what you and your colleague did …

[p.kayam]

@bionel

Did your colleague saved everything needed before comitting/pushing the changes?

Without saving you can’t commit/push the file

[bionel]

@p.kayam

Without saving you can’t commit/push the file

are you sure about this statement?
do you know how git actually works?

[kazurayam]

@p.kayam

Without saving you can’t commit/push the file

I don’t think so.

Git looks at the files on disk only. Git does not communicate with Katalon Studio at all. Git never asks Katalon Studio to flush the changes you made.

[Elly_Tran]

Hi,

I have tried this solution macos - Eclipse Auto Save before build? - Stack Overflow and I can autosave it. Hope this can help!
This is my reproduce - auto save.mov - Google Drive

[kazurayam]

@Elly_Tran

Thank you for the information: we can configure Eclipse to save dirty files before “build”.

The terminology “build” is a term used in Eclipse. It is not a term used in Katalon Studio.

Please clarify what “build” means in Katalon Studio.

Is it a sort of “build” to run a Test Case in Katalon Studio?
Is it a sort of “build” to run a Test Suite in Katalon Studio?
Is it a sort of “build” to run a Test Suite Collection in Katalon Studio?
Is it a soft of “build” to pull the project from a remote Git repository in Katalon Studio using “Git integration”?

Will a dirty Test Suite be autosaved when a Test Suite Collection is executed?

Do you mean that in all these cases, provided that the autosave is configured ON, the dirty files will be saved by Katalon Studio before those actions?

[p.kayam]

Thank you for the information