Refresh issue in test suite collection

Operating System

Windows 11

Katalon Studio version

e.g. version 8.5.5

Log Folder:

  • Windows logs folder: Katalon Studio folder>\config\.metadata\.log
  • MacOS logs folder: file:///Applications/Katalon%20Studio.app/Contents/MacOS/config/.metadata/.plugins/org.eclipse.ui.workbench/log

Environment (for Web Testing)

  • Chrome & Version 108.0.5359.125 (Official Build) (64-bit)

Steps to reproduce

When ever there is new test case is added in the test suite then in the suite collection need to remove that test suite and add again before execution else those test cases are not included

If you want to include newly added test cases in a test suite before execution, you can do that by updating the test suite in the suite collection. Here’s an example of how you could remove a test suite from a suite collection and add it again to include newly added test cases:

import com.kms.katalon.core.testcase.TestCaseFactory

// Get the test suite you want to update
TestSuite testSuite = TestSuiteFactory.getTestSuiteByName("My Test Suite")

// Remove the test suite from the suite collection
TestSuiteCollection.removeTestSuite(testSuite)

// Add the test suite to the suite collection again
TestSuiteCollection.addTestSuite(testSuite)

I will try to implement this, Thanks for reply.

@p.kayam

I tried to reproduce your case on my machine. I could. I think you, @p.kayam, made a small mistake. And Katalon Studio has a defect that confused you. Let me tell you what I did.

Scene A

Explation of scene A

  1. I made a Test Case “TCx”, which just displays “xxxxxxx TCx is running xxxxxxxxxx”
  2. I also made a Test Case “TCy”, which just displays “yyyyyyyyy TCy is running yyyyyyyy”
  3. I made a Test Suite named “TS1”. I included only TCx.
  4. I made a Test Suite Collection named “TCS”. I included the TS1.
  5. When I execute the TCS, in the console a line of message “xxxxxxxx TCx is running xxxxxx” appeared. This was just expected. TCS runs fine.

Scene B

Explanation of scene B

  1. I changed the TS1. I included the TCy in the TS1.
  2. I exectuted the TSC. It showed a message “xxxxxxxx TCx is running xxxxxxxx” only. It did not emitted a message “yyyyyy TCy is running yyyyyy”. This is an insident!
  3. I realised that I forgot to save the TS1. I did not click the green arrow “save” button. I left the TS1 unsaved, marked with an * mark.

Scene C

Explation of scene C

  1. I realized my mistake that I did not saved the changed TS1. So I clicked the green button to save the TS1. The * mark disappered.
  2. I executed the TSC. It emitted 2 lines of messages: “xxxxxx TCx is running xxxxxxx” and “yyyyyyy TCy is running yyyyyyyy”. This was as expected. TCS ran fine.

@p.kayam

I guess, you forgot saving the changed Test Suite, didn’t you?

May not look professional, but I made a habit of using Ctrl-Shift-S vs Ctrl-S when using various apps.
Usually this saves me from headeache (but sometime creates additional troubles)
‘Save All’ vs ‘Save’ sometime make the difference, and those keybindings are most common used.

I think there is a small defect in Katalon Studio. When I tried to execute the TSC while the TS1 was unsaved, I think Katalon Studio should not have started executing the TSC with unsaved TS1.

Katalon Studio should automatically save the TS1 before continuing; or should stop and warn “please save the TS1 befre running TCS”.

Unfortunately Katalon Studio ran TSC with unsaved TS1; this confused @p.kayam. This behavior of Katalon Studio doesn’t look professinal to me.

IntelliJ IDEA, Visual Studio Code — these professional editors make sure that everything saved before user actions.

Me too.

For me it did.

I used your sample project for browsermobproxy.
I opened two testcases and the testsuite.

I did few changes in the testcases (added some comments)

Without saving, I ran the suite.

Both testcases were automatically saved prior to run (the * disapeared and my changes are present after the suite run)

Well … i tested on linux, if it matters (and I am lazy to create a collection also) …

Perhaps Katalon developers implemented autosaving of Test Cases but did not implemented autosaving of Test Suite triggered by TCS execution.

Ah … well, lazy to test that also, but probably you are right.
Will autosave testcases but this is the only test I did.
Indeed, looks like a bug to be considered.

@vu.tran

My scenario is different, I pulled changes of my colleague in that there were some new testcases added in a test suite and that test suite is added in my test suite collection.
When I try to execute the test suite collection those test cases were not included if I remove the test suite and add it again those test cases was included.
I hope this is clear!

After pulling the changes, did you close & reopen the project?

After pulling the changes, did you ever opened the new testcases in the Test Case editor in your Katalon Studio GUI?

My Guess

Unless you manually open a test case script in GUI, Katalon Studio may not recognize that it has got a new script in the pulled project.

In my experience, unless given a chance to refresh, Katalon Studio often goes out of sync with files on disk. See the following ancient post:

Katalon developer stated they have fixed this issue. But I believe there remain some edge cases where Katalon Studio gets out of sync with files on disk. I’ve got bored with this issue. I am not going to debate this bug any longer.

What I do

I would just repeat closing and reopening projects when necessary. Especially I always do close&reopen a project after pulling it from a Git remote repository. It’s a rule of thumb.

This is a different storry.
Did your colleague saved everything needed before comitting/pushing the changes?

To confirm this ‘bug’ additional test scenarios are needed …

Actually, this brings more confusion on what you and your colleague did …

Without saving you can’t commit/push the file

are you sure about this statement?
do you know how git actually works?

I don’t think so.

Git looks at the files on disk only. Git does not communicate with Katalon Studio at all. Git never asks Katalon Studio to flush the changes you made.

Hi,

I have tried this solution macos - Eclipse Auto Save before build? - Stack Overflow and I can autosave it. Hope this can help!
This is my reproduce - auto save.mov - Google Drive

@Elly_Tran

Thank you for the information: we can configure Eclipse to save dirty files before “build”.

The terminology “build” is a term used in Eclipse. It is not a term used in Katalon Studio.

Please clarify what “build” means in Katalon Studio.

Is it a sort of “build” to run a Test Case in Katalon Studio?
Is it a sort of “build” to run a Test Suite in Katalon Studio?
Is it a sort of “build” to run a Test Suite Collection in Katalon Studio?
Is it a soft of “build” to pull the project from a remote Git repository in Katalon Studio using “Git integration”?

Will a dirty Test Suite be autosaved when a Test Suite Collection is executed?

Do you mean that in all these cases, provided that the autosave is configured ON, the dirty files will be saved by Katalon Studio before those actions?

Thank you for the information