Katalon Studio erased a Test Case code when it was moved, and then renamed

This post follows my previous post:

I believe that I have found out a procedure how to reproduce the incident (Test Case code is erased by Katalon Studio). So I would explain it here.

I used 2 machine environments. The issue was reproduced on both.

  • MacOS12.6 + Katalon Studio 8.3.0
  • Windows 10 + Katalon Studio Standard Editon 8.4.1

I did:

  1. Create a new Katalon Project

  2. Create a new Test Case at Test Cases/sub/subsub/TC1. The code is just skeltal as follows:

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

WebUI.comment("Hello, world!")
  1. The file tree looked as follows.
.
β”œβ”€β”€ Checkpoints
β”œβ”€β”€ Data Files
β”œβ”€β”€ Drivers
β”œβ”€β”€ Include
β”‚   β”œβ”€β”€ config
β”‚   β”‚   └── log.properties
β”‚   β”œβ”€β”€ features
β”‚   └── scripts
β”‚       └── groovy
β”œβ”€β”€ Keywords
β”œβ”€β”€ Libs
β”‚   β”œβ”€β”€ CustomKeywords.groovy
β”‚   └── internal
β”‚       └── GlobalVariable.groovy
β”œβ”€β”€ Object Repository
β”œβ”€β”€ Plugins
β”œβ”€β”€ Profiles
β”‚   └── default.glbl
β”œβ”€β”€ Reports
β”‚   └── Self-healing
β”‚       └── broken-test-objects.json
β”œβ”€β”€ Scripts
β”‚   └── sub
β”‚       └── subsub
β”‚           └── TC1
β”‚               └── Script1675984340226.groovy
β”œβ”€β”€ Test Cases
β”‚   └── sub
β”‚       └── subsub
β”‚           └── TC1.tc
β”œβ”€β”€ Test Listeners
β”‚   └── TL1.groovy
β”œβ”€β”€ Test Suites
β”‚   β”œβ”€β”€ TS1.groovy
β”‚   └── TS1.ts
β”œβ”€β”€ TestScritVanishesWhenRenamed.prj
β”œβ”€β”€ bin
β”‚   β”œβ”€β”€ groovy
β”‚   β”œβ”€β”€ keyword
β”‚   β”œβ”€β”€ lib
β”‚   β”‚   β”œβ”€β”€ CustomKeywords.class
β”‚   β”‚   └── internal
β”‚   β”‚       └── GlobalVariable.class
β”‚   β”œβ”€β”€ listener
β”‚   β”‚   └── TL1.class
β”‚   └── testcase
β”œβ”€β”€ build.gradle
β”œβ”€β”€ console.properties
└── settings
    β”œβ”€β”€ external
    β”‚   β”œβ”€β”€ com.kms.katalon.composer.execution.settings.properties
    β”‚   └── com.kms.katalon.core.db.DatabaseSettings.properties
    └── internal
        β”œβ”€β”€ com.kms.katalon.composer.testcase.properties
        β”œβ”€β”€ com.kms.katalon.composer.testcase.settings.properties
        β”œβ”€β”€ com.kms.katalon.execution.properties
        β”œβ”€β”€ com.kms.katalon.execution.webui.properties
        β”œβ”€β”€ com.kms.katalon.integration.analytics.properties
        └── com.kms.katalon.integration.qtest.properties

36 directories, 24 files


This looked fine.

  1. I moved the Test Case from Test Cases/sub/subsub/TC1 to Test Cases/sub/TC1. I did it in the GUI by mouse click + drag + drop operation. The file tree looked as follows:
.
β”œβ”€β”€ Checkpoints
...
β”œβ”€β”€ Scripts
β”‚   └── sub
β”‚       β”œβ”€β”€ TC1
β”‚       β”‚   └── Script1675984340226.groovy
β”‚       └── subsub
β”œβ”€β”€ Test Cases
β”‚   └── sub
β”‚       β”œβ”€β”€ TC1.tc
β”‚       └── subsub
...

So far, so good.

  1. In Katalon Studio GUI, I tried to rename Test Cases/sub/TC1 to Test Cases/sub/Main. I got an Error dialog.

After the error, the file tree looked as follows:

.
β”œβ”€β”€ Checkpoints
...
β”œβ”€β”€ Scripts
β”‚   └── sub
β”‚       β”œβ”€β”€ TC1
β”‚       β”‚   └── Script1675984340226.groovy
β”‚       └── subsub
β”œβ”€β”€ Test Cases
β”‚   └── sub
β”‚       β”œβ”€β”€ Main.tc
β”‚       └── subsub
...

It is broken. No longer consistent.

I looked into the .log file where I found an StackTrace was recorded


!ENTRY org.eclipse.e4.ui.workbench 4 0 2023-02-10 08:21:38.401
!MESSAGE 
!STACK 1
org.eclipse.core.internal.resources.ResourceException(/%Users%kazurayam%tmp%TestScritVanishesWhenRenamed%TestScritVanishesWhenRenamed.prj/Scripts/sub/subsub/TC1/Script1675984340226.groovy)[368]: java.lang.Exception: File not found: /Users/kazurayam/tmp/TestScritVanishesWhenRenamed/Scripts/sub/subsub/TC1/Script1675984340226.groovy.
	at org.eclipse.core.internal.resources.ResourceException.provideStackTrace(ResourceException.java:42)
	at org.eclipse.core.internal.resources.ResourceException.<init>(ResourceException.java:38)
	at org.eclipse.core.internal.localstore.FileSystemResourceManager.read(FileSystemResourceManager.java:835)
	at org.eclipse.core.internal.resources.File.getContents(File.java:275)
	at org.eclipse.core.internal.resources.File.getContents(File.java:266)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.getScriptContent(TestArtifactScriptRefactor.java:206)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.replace(TestArtifactScriptRefactor.java:158)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.updateReferences(TestArtifactScriptRefactor.java:289)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.updateReferenceForTestCaseFolder(TestArtifactScriptRefactor.java:260)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.updateReferenceForProject(TestArtifactScriptRefactor.java:245)
	at com.kms.katalon.dal.fileservice.manager.TestCaseFileServiceManager.updateTestCase(TestCaseFileServiceManager.java:131)
	at com.kms.katalon.dal.fileservice.dataprovider.TestCaseFileServiceDataProvider.updateTestCase(TestCaseFileServiceDataProvider.java:27)
	at com.kms.katalon.controller.TestCaseController.updateTestCase(TestCaseController.java:132)
	at com.kms.katalon.composer.testcase.handlers.RenameTestCaseHandler$2.run(RenameTestCaseHandler.java:83)
	at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)
!SUBENTRY 1 org.eclipse.core.resources 4 368 2023-02-10 08:21:38.402
!MESSAGE File not found: /Users/kazurayam/tmp/TestScritVanishesWhenRenamed/Scripts/sub/subsub/TC1/Script1675984340226.groovy.
!STACK 0
java.lang.Exception: File not found: /Users/kazurayam/tmp/TestScritVanishesWhenRenamed/Scripts/sub/subsub/TC1/Script1675984340226.groovy.
	at org.eclipse.core.internal.resources.ResourceException.provideStackTrace(ResourceException.java:42)
	at org.eclipse.core.internal.resources.ResourceException.<init>(ResourceException.java:38)
	at org.eclipse.core.internal.localstore.FileSystemResourceManager.read(FileSystemResourceManager.java:835)
	at org.eclipse.core.internal.resources.File.getContents(File.java:275)
	at org.eclipse.core.internal.resources.File.getContents(File.java:266)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.getScriptContent(TestArtifactScriptRefactor.java:206)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.replace(TestArtifactScriptRefactor.java:158)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.updateReferences(TestArtifactScriptRefactor.java:289)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.updateReferenceForTestCaseFolder(TestArtifactScriptRefactor.java:260)
	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.updateReferenceForProject(TestArtifactScriptRefactor.java:245)
	at com.kms.katalon.dal.fileservice.manager.TestCaseFileServiceManager.updateTestCase(TestCaseFileServiceManager.java:131)
	at com.kms.katalon.dal.fileservice.dataprovider.TestCaseFileServiceDataProvider.updateTestCase(TestCaseFileServiceDataProvider.java:27)
	at com.kms.katalon.controller.TestCaseController.updateTestCase(TestCaseController.java:132)
	at com.kms.katalon.composer.testcase.handlers.RenameTestCaseHandler$2.run(RenameTestCaseHandler.java:83)
	at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)

The problem was reproduced.

1 Like

Please note, at the step 4 above, I moved the TC1 from the folder Test Cases/sub/subsub to Test Cases/sub.

It seems that moving the test case across folders is a key factor to reproduce the incident .

If I do NOT move it accross the folders, just renaming a Test Case does not cause any problem.

!MESSAGE File not found: /Users/kazurayam/tmp/TestScritVanishesWhenRenamed/Scripts/sub/subsub/TC1/Script1675984340226.groovy.
!STACK 0

This message is interesting.

This message proves that Katalon Studio looked for the TC1 folder under the Scripts/sub/subsub folder, but the TC1 folder was no longer there because I moved the TC1 from Scripts/sub/subsub to other folder Scripts/sub before renaming it.

Katalon Studio tried to find TC1 in a wrong place. Why?

This must be the core issue of the problem.

1 Like

Why Katalon Studio tried to find TC1 in the Scripts/sub/subsub folder?

The StackTrace suggests something about this question.

	at com.kms.katalon.groovy.reference.TestArtifactScriptRefactor.getScriptContent(TestArtifactScriptRefactor.java:206)

This line tells us that the TestArtifactScriptRefactor class tried to open the Scripts/sub/subsub/TC1/Script1675984340226.groovy. file, which was already not there.

The class name TestArtifactScriptRefactor reminds me of a fact that Katalon Studio tries to update all references to the Test Case which you are to rename, as the following screenshot tells:

I would presume that the Exception was raised while Katalon Studio was updating the references to this Test Case.

I guess, the path information Scripts/sub/subsub/TC1 was recorded somewhere hidden inside Katalon Studio. That could be some sort of cache that keeps a snapshot of the file tree structure at some timing. And the TestArtifactScriptRefactor class happened to refer to the old/wrong path information recorded in the cache. Therefore it failed.

Why did the invalid path remain in the cache? β€” Please remember, before renaming the Test Case, I moved the TC1 from the folder Scripts/sub/subsub into the new Scripts/sub. I suppose that all of the old path should have been updated so that they point the correct path. However, I believe, Katalon Studio failed to do it. Unfortunately a wrong path remained in the cache, which later caused an Exception when I tried to rename the Test Case.

1 Like

I just experienced this… and i’m almost certain before as well… lost 5 of my test cases.

How is this issue not taken seriously? Maybe I should use the help desk and raise a ticket… so this gets taken seriously.
Things like this should not be in prod… :exploding_head:

since you appear to be a licensed user, why didn’t you already did it?
here, on the comunity forum, the only thing we can do is to provide workarounds, provided we have time to look into it.

we are not here to fix anything

I guess, the following issue is caused by the similar cause

I mean, Katalon team should look at the β€œData Files” as well, not only β€œTest Cases”.

I guess, Eclipse GUI component (β€œTest Explorer”) might be the problematic cache. Katalon Studio may have a bug in the way how to use Eclipse components.

I noticed that v8.6.0 included the fix for this issue.

I have confirmed that this issue was fixed by v8.6.0. Thank you for your efforts.

1 Like

Unfortunately, I noticed an edge case where my Test Case code was erased by Katalon Studio v8.6.0.

The fix was not as complete as it should be.

I am not sure if I could identify the procedure to reproduce the problem.

@Shin
@vu.tran

The problem still remains in the v8.6.0. I could reproduce it.

I used Katalon Studio 8.6.0, macOS 12.6

  1. I created a Test Case β€œTest Cases/hello”

  2. The β€œhello” was:

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

WebUI.comment("Hello, world!")
  1. The hello worked fine.

  2. I created a folder β€œTest Cases/subfolder”

  3. I moved the hello into the folder. I got β€œTest Cases/subfolder/hello”.

  4. I renamed β€œTest Cases/subfolder/hello” to β€œTest Cases/subfolder/greeting”.

  5. I found the β€œTest Cases/subfolder/greeting” got empty: zero lines contained.

It’s so easy to reproduce this problem.


I would warn all the Katalon users again: you should never rename your Test Cases; Katalon Studio may possibly erase it to blank!

If you need to rename it, you should do β€œgit add” and β€œgit commit” to save the file before you try to rename it just in case the code is erased.

1 Like

When I renamed β€œTest Cases/subfolder/hello” to β€œTest Cases/subfolder/greeting”, the script code was turned to be empty. I checked the /Applications/Katalon Studio.app/Contents/MacOS/config/.metadata/.log file. I found the following StackTrace message was recorded.

!ENTRY org.eclipse.jdt.groovy.core 4 0 2023-04-05 17:56:56.720
!MESSAGE Groovy-Eclipse Type Inferencing: Error visiting method main in class Script1680684894106
!STACK 0
java.lang.NullPointerException
	at org.eclipse.jdt.groovy.search.SimpleTypeLookup.findType(SimpleTypeLookup.java:349)
	at org.eclipse.jdt.groovy.search.SimpleTypeLookup.lookupType(SimpleTypeLookup.java:98)
	at org.eclipse.jdt.groovy.search.TypeInferencingVisitorWithRequestor.lookupExpressionType(TypeInferencingVisitorWithRequestor.java:2702)
	at org.eclipse.jdt.groovy.search.TypeInferencingVisitorWithRequestor.handleSimpleExpression(TypeInferencingVisitorWithRequestor.java:2004)
	at org.eclipse.jdt.groovy.search.TypeInferencingVisitorWithRequestor.handleSimpleExpression(TypeInferencingVisitorWithRequestor.java:2009)
	at org.eclipse.jdt.groovy.search.TypeInferencingVisitorWithRequestor.visitStaticMethodCallExpression(TypeInferencingVisitorWithRequestor.java:1671)
	at org.codehaus.groovy.ast.expr.StaticMethodCallExpression.visit(StaticMethodCallExpression.java:43)
	at org.codehaus.groovy.ast.CodeVisitorSupport.visitExpressionStatement(CodeVisitorSupport.java:120)
	at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitExpressionStatement(ClassCodeVisitorSupport.java:269)
	at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42)
	at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:148)
	at org.eclipse.jdt.groovy.search.TypeInferencingVisitorWithRequestor.visitConstructorOrMethod(TypeInferencingVisitorWithRequestor.java:1112)
	at org.eclipse.jdt.groovy.search.TypeInferencingVisitorWithRequestor.visitMethodInternal(TypeInferencingVisitorWithRequestor.java:659)
	at org.eclipse.jdt.groovy.search.TypeInferencingVisitorWithRequestor.visitJDT(TypeInferencingVisitorWithRequestor.java:467)
	at org.eclipse.jdt.groovy.search.TypeInferencingVisitorWithRequestor.visitJDT(TypeInferencingVisitorWithRequestor.java:376)
	at org.eclipse.jdt.groovy.search.TypeInferencingVisitorWithRequestor.visitCompilationUnit(TypeInferencingVisitorWithRequestor.java:256)
	at org.codehaus.groovy.eclipse.search.GroovyOccurrencesFinder.internalFindOccurences(GroovyOccurrencesFinder.java:138)
	at org.codehaus.groovy.eclipse.search.GroovyOccurrencesFinder.getOccurrences(GroovyOccurrencesFinder.java:86)
	at org.codehaus.groovy.eclipse.search.GroovyOccurrencesFinder.findOccurrences(GroovyOccurrencesFinder.java:194)
	at org.codehaus.groovy.eclipse.editor.GroovyEditor.updateOccurrenceAnnotations(GroovyEditor.java:1314)
	at org.eclipse.jdt.internal.ui.javaeditor.JavaEditor$ActivationListener.windowActivated(JavaEditor.java:1224)
	at org.eclipse.ui.internal.Workbench$8.run(Workbench.java:1039)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
	at org.eclipse.ui.internal.Workbench.fireWindowActivated(Workbench.java:1036)
	at org.eclipse.ui.internal.WorkbenchWindow$8.shellActivated(WorkbenchWindow.java:2489)
	at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:92)
	at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:89)
	at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4385)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1512)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1535)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1516)
	at org.eclipse.swt.widgets.Shell.windowDidBecomeKey(Shell.java:2228)
	at org.eclipse.swt.widgets.Display.windowProc(Display.java:6104)
	at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
	at org.eclipse.swt.widgets.Widget.callSuper(Widget.java:228)
	at org.eclipse.swt.widgets.Widget.becomeKeyWindow(Widget.java:389)
	at org.eclipse.swt.widgets.Shell.becomeKeyWindow(Shell.java:533)
	at org.eclipse.swt.widgets.Display.windowProc(Display.java:5823)
	at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
	at org.eclipse.swt.widgets.Display.applicationSendEvent(Display.java:5386)
	at org.eclipse.swt.widgets.Display.applicationProc(Display.java:5522)
	at org.eclipse.swt.internal.cocoa.OS.objc_msgSend(Native Method)
	at org.eclipse.swt.internal.cocoa.NSApplication.sendEvent(NSApplication.java:117)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3786)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$5.run(PartRenderingEngine.java:1158)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1047)
	at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:155)
	at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:658)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338)
	at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:557)
	at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:154)
	at com.kms.katalon.core.application.WorkbenchApplicationStarter.start(WorkbenchApplicationStarter.java:23)
	at com.kms.katalon.application.Application.runGUI(Application.java:191)
	at com.kms.katalon.application.Application.start(Application.java:102)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1447)

I have no idea what happend.