Thanks @Dimas_Tekad_Santosa its good to know im not the only one with this issue. Are you using a Mac by any chance? I dont recall seeing this issue on Ubuntu, and dont currently see it when running tests on Windows.
I have bumped the start and maximum memory allocation in the katalon.ini on Mac (where im seeing the issues), and think 4GB should be fine, see my katalon.ini below:
-startup
../Eclipse/plugins/org.eclipse.equinox.launcher_1.3.201.v20161025-1711.jar
--launcher.library
../Eclipse/plugins/org.eclipse.equinox.launcher.cocoa.macosx.x86_64_1.1.401.v20161122-1740
-data
@noDefault
-vm
../../Contents/Eclipse/jre/Contents/Home/jre/lib/jli/libjli.dylib
-vmargs
-Xms4096m
-Xmx4096m
-XstartOnFirstThread
-Dorg.eclipse.swt.internal.carbon.smallFonts
For reference, this post helped - Katalon Studio, memory issue adding variables for data driven testing