I am reaching out to see if anyone has any ideas as to what may be causing an issue I have noticed with my QA team while running our API automation suites.
While running our suite, which has over 1000 API test cases, each employee receives drastically differing results.
For example, when one employee runs the entire suite sequentially they only receive on average 2-5 errors/failures. When this occurs these test cases all re-run perfectly when ran by themselves outside the suite. This is a daily occurrence for this employee and usually remains the same.
Another employee will run the suite, sequentially as well, the next week and will on average receive 30 - 50 total errors/failures. Again, when these test cases are all re-ran separately from the suite they function perfectly.
Does anyone have any ideas as to what may be causing these drastic differences in results when running the automation suite?
Do you have any idea why these 2-5 errors/failures occured? Why not you fix all these errors one-by-one?
You can raise questions here for each individual errors attaching the sufficient information (scripts, logs, etc) disclosed. The guys in this forum would help you to solve each issues.
Simply put, your codes are not robust enough, possibly. You need to improve them for yourself.
The test cases fail due to timeouts from what I can tell. They are robust and re-run perfectly when ran by themselves. Usually the authentication endpoint times out or one of the verifications randomly doesn’t pass but when re-ran the verifications pass and the authentication works.
Usually when a specific employee runs it on his laptop everything passes with the occasional one or two that don’t.
Not certain why another employee runs it on theirs and many more timeout.
I have no idea. I think it’s only you who have chances to see how your system (tests) worked. Nobody on this forum can see it. So you should investigate the logs etc for yourself.
Without enough enformation discloses, others would not be able to help you.
Are there any common project settings or katalon settings that may be causing this? In the past we had issues with the smart wait feature for example. This feature would cause a lot of random failures but when turned off they were no longer present.
