I have a Test Suite Collection with 2 Test Suites inside of it. I run them periodically throughout the day locally on my Windows computer. I loved the idea of Test Suite Collections to organize tests better so I was very happy to start using it once the feature became available in version 6.0.0?
Whenever I do run these tests it seems that during a very random part of running one of the Test Suites it will just stop executing, but it will act like it’s still executing. Katalon will not send any errors or failures, and will still show the Stop button at the very top. It sometimes happens in the first Test Suite or the second, it truly does seem random.
This seems to only have started since the version 6.1.0 release, but I’m still not quite sure. Has anyone else experienced this problem? The Test Suites run fine on their own, it just seems to be in the collection.
Me too. I can generally run two in parallel, but the moment I go to three, the third goes batshit (run seq’ly they’re mostly fine).
I get a weird problem sporadically where one or more suites decides it can’t run test due to “Browser not open” which is just nuts. Once in a while, I know Firefox (in my case) can decide it’s going to update something and that kills a test or two. But when I see 15-20 tests do that… has to be something else.
Yep. Me too. No such issues on my Win7 box where I develop my tests.
I thought that might have been my problem, but I see that you have quote a number of test cases as well. I do have a lot of tests with 50+ steps so that could be a factor?
Yeah, mine seem to go all crazy when I run in parallel and honestly they don’t run any faster, the whole execution time is the same as in sequential so I don’t see a need for it at the moment.
Are you using the Test Suite to open your browser? That’s what I have done with mine and I haven’t encountered that issue. I’ve had 10+ tests fail for no apparent reason at all and I’ve noticed if you close Katalon and open it back up that you can run them flawlessly after that. Very strange behavior.
I appreciate the link!
I do have a screenshot that can explain my dilemma a little further.
The test was started at 7:59 AM as shown in the picture, but my screenshot was taken an hour later at 8:59 AM. If you add up the seconds for the setup and 2 test cases to run that equates to a little bit over 2 minutes… yet it still acts like it’s running and even in Katalon Analytics it shows the test cases haven’t finished. Hopefully this helps a little bit more.