Katalon Analytics seems to cause Suite Collections to stop executing

[Brett.Holmes]

(EDIT: For the bug report as it stands, please see this post: Katalon Analytics seems to cause Suite Collections to stop executing - #11 by Brett.Holmes ~Russ)

I have a Test Suite Collection with 2 Test Suites inside of it. I run them periodically throughout the day locally on my Windows computer. I loved the idea of Test Suite Collections to organize tests better so I was very happy to start using it once the feature became available in version 6.0.0?

Whenever I do run these tests it seems that during a very random part of running one of the Test Suites it will just stop executing, but it will act like it’s still executing. Katalon will not send any errors or failures, and will still show the Stop button at the very top. It sometimes happens in the first Test Suite or the second, it truly does seem random.

This seems to only have started since the version 6.1.0 release, but I’m still not quite sure. Has anyone else experienced this problem? The Test Suites run fine on their own, it just seems to be in the collection.

[Russ_Thomas]

Hi Brett

This is not quite a bug report (yet). I’ll move this to Web Testing.

I have a few issues with collections too, but not the one you describe here. Could you describe the Suites better?

1. Number of tests per suite.
2. Are the suites running in parallel?
3. Do they ever execute correctly?
4. What browser(s)?
5. Which OS?

[Brett.Holmes]

Thanks for the response Russ!

1. 13 in the first Test Suite and 24 in the second Test Suite.
2. They are running Sequential since I’ve had some minor problems with Parallel.
3. The Test Suites always execute correctly, but it seems that 3 out of 5 times when the Test Collection is run that it just stops.
4. I’m only running the tests in Firefox at the moment.
5. Windows 10 64 bit.

[Russ_Thomas]

@Brett.Holmes

Thanks for the response Russ!

1. 13 in the first Test Suite and 24 in the second Test Suite.

Mine are in their 20s. One is 35.

2. They are running Sequential since I’ve had some minor problems with Parallel.

Me too. I can generally run two in parallel, but the moment I go to three, the third goes batshit (run seq’ly they’re mostly fine).

3. The Test Suites always execute correctly, but it seems that 3 out of 5 times when the Test Collection is run that it just stops.

I get a weird problem sporadically where one or more suites decides it can’t run test due to “Browser not open” which is just nuts. Once in a while, I know Firefox (in my case) can decide it’s going to update something and that kills a test or two. But when I see 15-20 tests do that… has to be something else.

4. I’m only running the tests in Firefox at the moment.

Funny that.

5. Windows 10 64 bit.

Yep. Me too. No such issues on my Win7 box where I develop my tests.

Just in case it helps you…

[Tip] How to Speed up Katalon Studio on Windows 10

For a while I have noticed that the load time for Katalon on Windows 10 was substantially longer than the load time for Windows 7. I knew also that Win10’s protection system is much more aggressive and pervasive than Win7. So, today, I decided to fix the problem. First, what do I mean by “load time”? The time it takes to start Katalon.exe, open a project (in my case, a quite large project) and wait for Katalon to finish preparing the IDE so it’s ready to be used. <<< Load Time. The exclusi…

[Brett.Holmes]

@Russ_Thomas

Mine are in their 20s. One is 35.

I thought that might have been my problem, but I see that you have quote a number of test cases as well. I do have a lot of tests with 50+ steps so that could be a factor?

@Russ_Thomas

Me too. I can generally run two in parallel, but the moment I go to three, the third goes batshit (run seq’ly they’re mostly fine).

Yeah, mine seem to go all crazy when I run in parallel and honestly they don’t run any faster, the whole execution time is the same as in sequential so I don’t see a need for it at the moment.

@Russ_Thomas

I get a weird problem sporadically where one or more suites decides it can’t run test due to “Browser not open” which is just nuts. Once in a while, I know Firefox (in my case) can decide it’s going to update something and that kills a test or two. But when I see 15-20 tests do that… has to be something else.

Are you using the Test Suite to open your browser? That’s what I have done with mine and I haven’t encountered that issue. I’ve had 10+ tests fail for no apparent reason at all and I’ve noticed if you close Katalon and open it back up that you can run them flawlessly after that. Very strange behavior.

@Russ_Thomas

Just in case it helps you…

I appreciate the link!

I do have a screenshot that can explain my dilemma a little further.

The test was started at 7:59 AM as shown in the picture, but my screenshot was taken an hour later at 8:59 AM. If you add up the seconds for the setup and 2 test cases to run that equates to a little bit over 2 minutes… yet it still acts like it’s running and even in Katalon Analytics it shows the test cases haven’t finished. Hopefully this helps a little bit more.

I appreciate your input @Russ_Thomas

[Russ_Thomas]

@Brett.Holmes

Are you using the Test Suite to open your browser? That’s what I have done with mine and I haven’t encountered that issue.

No. Each test (well, 99% of them) opens a browser, logs in, performs the test, closes browser.

I’ve had 10+ tests fail for no apparent reason at all and I’ve noticed if you close Katalon and open it back up that you can run them flawlessly after that. Very strange behavior.

Only reason I close Kat is when it’s out of memory and crashed - but that’s dev work, not suite executions. Start here if you want the gory details…

Katalon Studio freezes and non-responsive

Me too! So, you keep KS open “all the time”, too? BTW, your working set is about half mine, it seems. But yeah, I’ve done the same and once I see KS “start” to crash, there’s not much you can do to stop it. First indication I get is weird UI update issues… Here’s my current process: [image]

Brett, I could easily return this to Bug Reports, but I know what will happen, there’s nothing in this that can point a developer to the cause.

[Russ_Thomas]

Idea:

When you see this happening, bring up task manager and see how much memory geckodriver is using.

I have mine set to be killed when suites are running but stay open when executing tests. Sometimes I forget to kill them. I use this in a batch file to do that:

taskkill /f /im geckodriver.exe
taskkill /f /im chromedriver.exe
taskkill /f /im IEDriverServer.exe
taskkill /f /im MicrosoftWebDriver.exe

[Brett.Holmes]

@Russ_Thomas

Idea:

When you see this happening, bring up task manager and see how much memory geckodriver is using.

I have mine set to be killed when suites are running but stay open when executing tests. Sometimes I forget to kill them. I use this in a batch file to do that:

taskkill /f /im geckodriver.exe
taskkill /f /im chromedriver.exe
taskkill /f /im IEDriverServer.exe
taskkill /f /im MicrosoftWebDriver.exe

It did happen again today, I brought up task manager and geckodriver is using 4mb of memory, which seems like average. Hmmm interesting, but these stop mid test suite.

[Russ_Thomas]

@Brett.Holmes

geckodriver is using 4mb of memory

Seems reasonable.

[Brett.Holmes]

This keeps happening, and it’s making it very hard for me to be confident in Katalon Test Collection Executions.

[Brett.Holmes]

I think I’ve found the solution! After much testing, I’m almost 99% certain that this problem is caused by Katalon Analytics.

I’ve ran each test collection > 5 times, when I run it with Katalon Analytics the tests stopped executing 2-3/5 times.

When I run it without Katalon Analytics the test collection finishes 5/5 times.

@Russ_Thomas, is there anyway this could be looked into? I wish I had some other way to show this is a consistent problem with the Katalon Analytics integration.

[Russ_Thomas]

@Brett.Holmes

I think I’ve found the solution! After much testing, I’m almost 99% certain that this problem is caused by Katalon Analytics.

I don’t use KA. Not ever.

I’ve ran each test collection > 5 times, when I run it with Katalon Analytics the tests stopped executing 2-3/5 times.

There might be log somewhere that caught that problem. Try: katalon-install\config\.metadata\.log

@Russ_Thomas, is there anyway this could be looked into?

Sure. But I’m not able to make that happen. I’ll move this to bug reports and mark it critical.

Callouts: @devalex88 @ThanhTo @duyluong

[Brett.Holmes]

There might be log somewhere that caught that problem. Try: katalon-install\config\.metadata\.log

I couldn’t find anything specifically in there as it refers me to the xml log file that doesn’t seem to have anything, but the console log did have some information that might be useful:

1561045898886	Marionette	INFO	Stopped listening on port 53337
[Parent 24616, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1559936101/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 14688, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1559936101/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, l[Child 15056, ine 341
Chrome_C[Child 14688, Chrome_ChildThread] WAhildThread] WARNING: pipe error: 109: file z:/task_1559936101/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 15056, Chrome_Child[Parent 24616, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1559936101/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 22616, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1559936101/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 22616, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1559936101/build/src/ipc/chromium/
###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost

[GPU 19284, Chrome_ChildThread] WARNING: pipe error: 109: 
###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv

[Russ_Thomas]

Marionette/Gecko, that’s all pertaining to Firefox’s webdriver.

Doesn’t really tell us much… might be the cause, might be just a symptom.