Automation Suite Results Differ between Co-workers


I am reaching out to see if anyone has any ideas as to what may be causing an issue I have noticed with my QA team while running our API automation suites.

While running our suite, which has over 1000 API test cases, each employee receives drastically differing results.

For example, when one employee runs the entire suite sequentially they only receive on average 2-5 errors/failures. When this occurs these test cases all re-run perfectly when ran by themselves outside the suite. This is a daily occurrence for this employee and usually remains the same.

Another employee will run the suite, sequentially as well, the next week and will on average receive 30 - 50 total errors/failures. Again, when these test cases are all re-ran separately from the suite they function perfectly.

Does anyone have any ideas as to what may be causing these drastic differences in results when running the automation suite?

Thank you!


Do you have any idea why these 2-5 errors/failures occured? Why not you fix all these errors one-by-one?

You can raise questions here for each individual errors attaching the sufficient information (scripts, logs, etc) disclosed. The guys in this forum would help you to solve each issues.

Simply put, your codes are not robust enough, possibly. You need to improve them for yourself.

Hi Jacob, :wave:

Welcome to Katalon Community.

May I ask if you are one of our paid customers?

If yes, then you can reach out to our Product Support team regarding your team’s issue below:

Alternatively, since we are also running the Ask Katalon Anything session, you could also ask our Product Managers as well :point_down:

I am using the free version of katalon stuido. Our company hasn’t needed any of the paid features as of yet.

The test cases fail due to timeouts from what I can tell. They are robust and re-run perfectly when ran by themselves. Usually the authentication endpoint times out or one of the verifications randomly doesn’t pass but when re-ran the verifications pass and the authentication works.

Usually when a specific employee runs it on his laptop everything passes with the occasional one or two that don’t.

Not certain why another employee runs it on theirs and many more timeout.

I have no idea. I think it’s only you who have chances to see how your system (tests) worked. Nobody on this forum can see it. So you should investigate the logs etc for yourself.

Without enough enformation discloses, others would not be able to help you.

1 Like

Are there any common project settings or katalon settings that may be causing this? In the past we had issues with the smart wait feature for example. This feature would cause a lot of random failures but when turned off they were no longer present.

No idea. I do not see what is this.

Only you can find what this is.

Understood, I will dig into the logs and see what I can find. I will try to post as many details as I can (if disclosable).

I appreciate the answers and help that has been provided thus far.