1.Create a Test Suite with the Rerun Failed Tests Immediately checked with a retry of 1
2.Ensure the Test Suite has a test case that will fail at some point during the middle of the suite.
3.)Once it reaches the failed test case, the suite will close and reopen and launch from that failed point.
4.) The test will run again and fail and all other tests that follow will be skipped.
Expected behavior: I was under the impression that the failed test case would rerun immediately and if it failed again it would continue the suite just like normal and fail that singular test for the suite. The way that this functions currently seems incorrect and not useful for our purposes. Is this intended functionality?
That is the intended behavior. The use case of Retry Immediately is for a Test Suite in which following Test Cases depend on previous Test Cases. It follows that executing subsequent Test Cases when a Test Case fails will not make sense. Hope that helps !
Understood, will there be implementation in the future for the functionality I previously mentioned? It seems to me to be more efficient and would allow for more accurate data to populate Katalon Analytics with. Thanks for the help and God Bless!
Essentially when a test case fails inside of a suite, the suite will rerun the fails that occurred. This does not occur immediately but towards the end of the suite collection. Thus, what happens is the results are spread out. I was suggesting they run immediately after they fail so that the reports at the end of the suite are concise and easier to read and would also make Katalon analytics more accurate.
Current Functionality Example:
Test Suite1 3/15 Fail
Test Suite2 0/10 Fail
Test Suite1 0/3 Fail
Info is submitted to Analytics with information on Suite 1 twice.
Proposed Example:
Test Suite1 3/15 Fail (All fails were run again and the total fails are collected into one result)
Test Suite2 0/10 Fail
Info is submitted to Analytics
With the above example if there was also an option to see more details or less depending on the need. So if you needed to find out why it failed the first run but then passed the second run you could just expand the results or collapse if the information is not needed.
Would just like to bump this and say this is what I would want as well. I expected “Retry failed executions immediately”, to not immediately execute a new test suite with the failed test(s) only in it. I expected instead for Katalon to just rerun the current test that failed N number of times. This is what many testing frameworks do from what I’ve observed. For example Spock Testing Framework does this. For our use case mobile tests can be flakey, so sometimes we just want to rerun flakey tests after they fail.
Implementing this would be great thanks!
@ThanhTo have there been any thoughts on this as of recent?
Can you elaborate a bit more ? Currently, the behavior of “Retry failed executions immediately” does execute a failed test case again and again until it either passes or the specified threshold is reached. Can you tell me what do you expect instead, and your use case for doing so ?
Sure, EXAMPLE test set = test1, test2 test suite = test suite Retry failed executions immediately = 1 Expected behavior:
(1) Run test suite
(2) Run test1
(3) test1 fails
(4) Run test1 again
(4) test1 fails
(5) Run test2
(6) test2 passes
(7) Generate test report with test1, test2 for test suite
Actual behavior
(1) Run test suite
(2) Run test1
(3) test1 fails
(4) Generate test report with test1 for test suite
(6) Run test suite again
(5) Run test1 again
(6) test1 fails
(7) Generate test report with test1 for test suite
So I guess my issues are:
1.If test always fails none of the other tests execute - would expect all tests to be executed at least once, this makes us unable to use this retry option in Jenkins…
2. Test suite starts from beginning again - Would hope just the test itself would be reran not the entire test suite if the failure is in the test case itself.
3. Multiple test reports are generated - Would hope for an option to have only one test report to be generated, with results for test1 (last retry) and test2. Right now it looks confusing with multiple test suite runs for just one test suite!
I would say 1-3 seem to be issues that are not present in Spock or Junit testing frameworks from my knowledge, which do the following:
Execute all tests - regardless of failed tests
Test suite is not re-executed from beginning for failing test - Failed tests are retried when they fail, and test suite is not re-executed from scratch.
Generate only 1 test report (Report last test result of failed test that was retried) - one report being generated is clearer
Agree with Nathaniel, I am disabling this feature for now as I hoped it would help solve random glitchy tests but due to current behavior it is instead messing with rest of my tests.
It can happen that running a test is fail and running exact same again just after is ok. I agree that in such cases script should be made more robust to avoid that, and instead make sure tests always ok first run if they should, but application behavior / complexity / more and more virtual environment and test environment performance issues make it very complicated sometimes.
I was more expecting to have the ability to say in case of failure in the middle of test suite “ok let us note a warning there (optional), and let us consider this run never existed, let us rerun again but just X times, if really fail after x times then we mark fail, else we mark pass, in both cases we continue to next test in test suite like if we made only 1 run (the last one)”.
Thank you everyone for your suggestions and feedbacks ! We are working on some enhancement on this retry feature. We’ll notice you when there’s something new.
Katalon Studio has improved the Retry Failed Executions Immediately feature and introduced the consolidated execution reports to address test flakiness in version 8.1.0. Please see the Release note for more details.
