I think I’ve found the solution! After much testing, I’m almost 99% certain that this problem is caused by Katalon Analytics.
I’ve ran each test collection > 5 times, when I run it with Katalon Analytics the tests stopped executing 2-3/5 times.
When I run it without Katalon Analytics the test collection finishes 5/5 times.
@Russ_Thomas, is there anyway this could be looked into? I wish I had some other way to show this is a consistent problem with the Katalon Analytics integration.