I wish to run some automated tests on my Windows software using Katalon. However, I face the following issue:
The software I am testing has OS-wide presence and influence. Thus, the way it is developed, every time the user changes some setting, he must also reboot his machine for the changes to take effect. Similarly I as the tester also must reboot my machine to test functionality in such cases. However, restarting my computer would obviously also terminate the Katalon process. This is quite a big problem.
I have explored alternatives, in particular using Virtual Machines (VMs). The idea is, run Katalon on my local machine and then do the tests on the VM. This way, rebooting the VM does not end Katalon’s process. But it doesn’t work for me because I cannot create any Objects out of the elements I see on the VM. Even if they have identifying characteristics, they are not selectable or even “visible” to Katalon which is running on a different, ‘outer’ machine. The entire Windows11 VM app is just shown as one big “Pane”. So I’m unable to control anything from Katalon from outside the VM.
(I am using VMWare Workstation)
I have looked online and there seem to be this few ideas:
Use a VM [tried, doesn’t seem to work as explained above]
CI Server [unsure what this entails, or if it’s even relevant]
Use “retry” or some similar method to rerun or resume running of test upon rebooting. There might be some way to configure Katalon to continue where it left off after it opens again.
My questions are:
Is there a different VM that allows me to work in this way?
Is #3 possible (ie run the Katalon test for a bit, until it gets to the reboot; pause here; upon restarting, Katalon can continue where it left off)?
In general, any different ideas from the ones I’ve listed? Honestly, I’m not confident about any of these, and would love to try something different.
I don’t believe there is a way to easily accomplish this with OOTB Katalon tools. That said, and even though your question isn’t worded as such, I think there is an implied Feature Request lurking in there.
All I can think of is a mess of hacky bits and pieces using KRE commandline, batch files managing state on disk and sprinkled over the VM and the outer bare metal machine – too embarrassing to mention (and horrible to maintain).
Let’s get these fine people to have a look and see what they think:
this is out of the scope of (automated) testing but more for having basic devops knowledge.
i never meat a certain application able to reboot the OS is running it and be able to do a full recovery of the initial state.
therefore, such approach is defective by design.
just saying.
whatever else you guys read, is a lie.
just my two cents
That’s insightful. But in my case, since changes have been made to the system settings via the pre-reboot part of the test, there’s no need to really save/restore initial state after rebooting — the “context” lies in the updated settings which are effective after the reboot.
Before I give up on automating this, let me explore this other idea:
Let’s say I have a test that can be broken into part A, reboot, and part B, in that order. (Indeed, most of my test cases would be structured like this. Part A makes the edits to settings, part B checks if the edits worked as expected.) I just use Katalon for part A up to when I need to reboot. Add a few lines in the script that creates a scheduled task under Task Scheduler to run part B upon next start up. Call the reboot. Then Part B begins automatically.
Any lethal difficulty you envision with this idea? Or is this basically some implementation of Russ’ mess of hacky bits, and not recommended nor sustainable? @Russ_Thomas@bionel
It may work somehow … well, you will have to take care also about cleanup …
But first, I will explore a bit on the application side.
Why the system has to be rebooted after any change?
Perhaps you can achieve the same without reboot, e.g by restarting a certain service, or updating some registry …
So I will first speak with the development team to see what can be done on this matter.
The main issue with your approach is, during phase A, after you create the needed startup scripts for phase B, you will trigger somehow machine reboot from Katalon, which is running at this moment.
Whitout a gracefully kill of Katalon, during next run your project may be in an inconsistent state.
And I have no ideea how you can do a gracefull kill of Katalon from Katalon himself
perhaps, during teardown of phase A, the last commands may be a delayed shutdown followed by a taskkill on katalon
this is still meh. you have no guarantee for a gracefull termination, at least with KSE
with KRE it may work (taskkill is not needed)
If reboot cannot be avoided, I will take the following approach, using a certain CI tool (jenkins, bamboo, ADO whatever) and a VM with the AUT installed as executor.
I assume, by Katalon here we speak about KRE, otherwise everything became more complicated.
Will setup a pipeline, job whatever with 3 stages:
Stage 1: CI starts KRE with phase A, where:
KRE starts the AUT
do the necesary changes
KRE gracefully stops the AUT
KRE end his scope so gracefully terminate
Stage2: CI triggers VM reboot and wait for it to became available (this stage may have to run on another executor / node so it can ping the needed VM until it’s up)
Stage3: CI starts KRE with phase B, which does the remaining.