Rerunning Failed Tests Immediately Is Skipping Subsequent Tests

  • Katalon Studio Version: 7.6.1
  • OS Version: Windows 10
  • Browser Version: Chrome 83.0.4103.97

Steps to Reproduce:

1.Create a Test Suite with the Rerun Failed Tests Immediately checked with a retry of 1
2.Ensure the Test Suite has a test case that will fail at some point during the middle of the suite.
3.)Once it reaches the failed test case, the suite will close and reopen and launch from that failed point.
4.) The test will run again and fail and all other tests that follow will be skipped.

Expected behavior: I was under the impression that the failed test case would rerun immediately and if it failed again it would continue the suite just like normal and fail that singular test for the suite. The way that this functions currently seems incorrect and not useful for our purposes. Is this intended functionality?

Screenshots:

Hi @Chase_Emerick

That is the intended behavior. The use case of Retry Immediately is for a Test Suite in which following Test Cases depend on previous Test Cases. It follows that executing subsequent Test Cases when a Test Case fails will not make sense. Hope that helps !

Understood, will there be implementation in the future for the functionality I previously mentioned? It seems to me to be more efficient and would allow for more accurate data to populate Katalon Analytics with. Thanks for the help and God Bless!

@Chase_Emerick

Thank you for the feeback ! Can you elaborate on your use case, and your reasoning as to which behavior would best suite your need ?

Essentially when a test case fails inside of a suite, the suite will rerun the fails that occurred. This does not occur immediately but towards the end of the suite collection. Thus, what happens is the results are spread out. I was suggesting they run immediately after they fail so that the reports at the end of the suite are concise and easier to read and would also make Katalon analytics more accurate.

Current Functionality Example:
Test Suite1 3/15 Fail
Test Suite2 0/10 Fail
Test Suite1 0/3 Fail
Info is submitted to Analytics with information on Suite 1 twice.

Proposed Example:
Test Suite1 3/15 Fail (All fails were run again and the total fails are collected into one result)
Test Suite2 0/10 Fail
Info is submitted to Analytics

With the above example if there was also an option to see more details or less depending on the need. So if you needed to find out why it failed the first run but then passed the second run you could just expand the results or collapse if the information is not needed.

Make sense? If not I can elaborate more.