Working tests fail when part of a large suite

Archive
,
1

We’ve written about 50 stable tests that all run perfectly when run independently. When we put them together in a suite of 2 - 4 tests, they still seem to run pretty well. But when we try to run them all together in one big suite, we get lots of random failures and the whole thing seems unstable. Does anybody have any ideas or any experience with this? Each test starts and finishes in the same place, so it’s not a matter of navigation within our application.

2

I have 72 tests in one suite. The whole suite takes ~2hrs to complete. Average time for a test is ~2mins. I sometimes see 100% green but frequently get 2 or 3 fails.

It’s rarely the same 2 or 3, though.

3

Did you do anything to try to make the suite more robust? Anything in project settings? Any set up or tear down code in the suite? Just looking for ideas.

4

Nothing specifc, no.

I just offered you my experience for comparison (not that it helps much).

I don’t do setup/teardown but I do use suite and test listeners. They don’t do anything I would consider helpful to you (mostly about preparing my own reports and logging into the app).

Are you doing anything you think might be an issue when running in a suite?

5

I had the same issue when first starting out running large suites - usually it just came down to performance of the site and the server i was testing it on. If theres any recommendation i can give you is make sure to add more than enough delays for areas which are flaky and not consistent.

Now they will never be random, maybe inconsistent, but never random. When you look through the report just have a read through which tests failed and pay attention to the step at which it failed, more often than not it will be down to a element not being fully loaded and the next step attempting to run.

Again, like Russ, im only speaking from personal experiences

6

Hi,

if you are executed those testsuites with GUI it eats a lot of cpu process, maybe your cpu or mem are used. Should check how much your tests will eat them.
Best is run tests with cmd and use headless mode of browser if not needed to see how tests will proceed.
7

I do face similar issue when I run my test cases collectively and they form large suite but that was not because of Katalon but was from Google.

  • The test cases try to perform payments on our website using different type of credit cards ( country wise) so it has become a very large test suite.
  • There is one step in every test case which trigger a google address suggestion every time to fill up the address.

So when I am continuously hitting the same API inside my test case It starts responding slowly randomly breaking the test cases. To overcome this I added delay after entering values to trigger the suggestion and before selecting the suggestion.