For the last few weeks I’ve been using Katalon to test the performance of our new Sharepoint Online system. We’re in the process of moving over from an on-premises Sharepoint, but we’ve had complaints from clients about performance. This has been going well for the last few weeks: the results are consistent and are giving us a good baseline for further tweaks.
Since Thursday last week, I’ve noticed that the runtime of test suites has quadrupled from 3 minutes to 12-15 minutes. This seems to be caused by certain test cases, but it happens intermittently and while some pages are never affected, none are always affected. When I look at the screen while a problematic test is running, it seems to just sit there: the page is loaded and ready to go, Katalon doesn’t click until a long time has passed and it continues without visible explanation. Katalon does not throw up any errors. I did find some warnings in the event log. They are included.
I’ve added screenshots from one of the problematic test case (2 minutes), including a screenshot of the script. Mind that while this concerns an if\then case, not all affected test cases contain if\then statements. The if\then was added due to the xpath slightly changing if somebody is editing the page.
I’ve checked the site manually and I cannot find any delays clicking myself, nor do my colleagues find any anomalies when they test latency.
Stranger still, I just saw a run (112) that failed after waiting 15 minutes on a job. After a bathroom break, the run completed successfully and the test case that lasted 15 minutes, is now registered at 1 minute and a few seconds. I checked Analytics and saw the failed run. At least I’m not crazy. Tried to reimport the run to see if something changed and now run 112 is completely empty and a new run (113) appeared with the data from the client. I never ran 113.
Does anybody have any idea what’s going on here, because I am stumped.
Additional info:
I’m using Chrome that’s pushed via app-v. I insert Katalon into the App-v via a Powershell script. This hasn’t given any problems before.
Running Windows 7, until Katalon becomes available on the win10 vm’s.
Using Katalaon 6.3.2. Updating is a hassle due to company policies.