You do not need KSE license to increase the size of memory allocated for a KRE runtime.
On my windows 10 desktop, I installed Katalon Studio (not Enterprize) in %USER_PROFILE%\Katalon_Studio_Windows_64-7.2.1.
In that directory I find a plain text file named katalon.ini.
In the katalon.ini file, I can find a line like
-Xmx2048m
This means my test runs with maximum memory for JVM of 2048 Mega bytes.
With a plain text editor like VIM or Emacs, I can edit this line to
-Xmx4096m
or
-Xmx8192m
The appropriate memory size would depend on how much your tests want and how much physical memory size the machine has.
You run your test on Azure DevOps. Hopefully you know how to edit the katalon.ini file and configure the machine image on which your tests run.