This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/new/what-is-new-810.html
This is a companion discussion topic for the original entry at https://docs.katalon.com/katalon-studio/new/what-is-new-810.html
Hi,
While I really appreciate your effort to “Tackle test flakiness”, I also think, that one of the very best approach to tackle test flakiness would be to use Playwright instead of Selenium. Because Playwright has it’s internal much better coded, and is therefore much more reliable, and even faster as a bonus.
Playwright is also in Java, so it probably wouldn’t be that hard to bring it into Katalon Studio 
It also has Playwright Recorder now, so it’s easy to just record test.
I’ve already created forum post about starting using Playwright here. So every fan of Playwright, please upvote it, thanks
Pleaeease developers, consider to bring it into Katalon Studio 
Thanks so much
2 Likes
Hi,
The Katalon documentation has not been updated with the 8.1.0 release. The terminate logic isn’t documented that I could find.
I would also like to request an enhancement. The terminate logic is based on test count or it appears to be. My request is to change from a fixed test count to a percentage of all tests in the Test Suite. I.E. - if 10% of the tests are failing then terminate execution. Fixed test count will not be the same for all test suites. For example, Test Suite with 30 tests should stop if 10 are failing where Test Suite of 100 should fail when 30 are failing are two different values. Our CI/CD process executes a pipeline script with the CLI parameters and the same arguments are used for all KS Test Suite, like KS version for example.
Percentage is easier to apply to any test suite and will be consistent across all test suites when executing with Jenkins Pipeline script where the total test count will vary. Using fixed test count is inconsistent and can lead to confusion.
Hi,
I was excited to use the retry logic but it may have a bug when executing from Jenkins. I’m referring to quote from What’s New v8.1.0
“The changed function now consolidates retried test case execution logs into one report so that you can identify flaky tests at a glance.”
The Junit_Report.xml that is generated at run-time, still logs multiple test runs for the same test. This shows as two tests in Jenkins. This execution image below was of one test, set to retry 5 times. The test executed twice and shows in Jenkins as two tests.
I’m not able to use retry logic if the reports will double test count in Jenkins.
As a side note, the retry stopped after the second execution and did not try 3,4 or 5th execution. That might be a different bug but I did not investigate it.
Hi @jim.sears,
Did you try Retry Immediately mechanism or Terminating Execution Conditionally or it’s better if you can show us the command parameters you were using? These are 2 different features.
Hi @peter.hevesi,
Thanks for your suggestion. This idea has been sent to our R&D team, they are experimenting Playwight Java with Katalon Studio.
It is too soon to talk about Playwright can be a part of our engine or not but I think this is also a promising feature.
2 Likes
Thank you that you noticed my post and even started experimenting with Playwright 
I really hope, that you will bring it to Katalon Studio, as it will increase user experience significantly
Hi Duyluong,
When I tried this yesterday, I was trying to do so with both options but I did something wrong. Today’s execution worked and captured only one test run (as expected). So this is no longer issue.
- -retry=1 -retryStrategy=immediately
However, there is something odd about the re-execution. The sql query is invalid on the second execution but it works on the first execution. The sql uses the test name and details from the listener class. I’m investigating and searching for something that changed (or did not change) from the first execution to the second execution. My guess is that everything from the first test run is still “in-memory” because the second execution doesn’t reload the test. What are your thoughts?
Hi @duyluong,
I need to follow up on the retry logic. The strategy I implemented calls KS tests, treating them like reusable modules. The testcase context has worked within this reusable strategy until retry.
When the retry logic is applied and the second execution begins, the testcase context still contains the testCaseContext.message from the previous execution. Should the testCaseContext be reset or cleared between executions by KS or myself?
First execution; from TestListener beforeTest
JIM TListener MSG is empty. (nothing between the brackets, as expected.)
18:16:44 [2021-08-20T18:16:44.229] 2021-08-20 18:16:44.228 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
18:16:44 [2021-08-20T18:16:44.229] 2021-08-20 18:16:44.229 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/SCENARIO/3-ADMINISTRATION/3-6-API_TOKENS/TA_GuiTokenAutoDelete
18:16:44 [2021-08-20T18:16:44.288] 2021-08-20 18:16:44.287 INFO com.kms.katalon.core.util.KeywordUtil - JIM TListener MSG
Start of second execution, I have printed the contents of the testCaseContext again from testListener and it is not empty.
18:17:25 [2021-08-20T18:17:25.607] 2021-08-20 18:17:25.606 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/SCENARIO/3-ADMINISTRATION/3-6-API_TOKENS/TA_GuiTokenAutoDelete
18:17:25 [2021-08-20T18:17:25.613] 2021-08-20 18:17:25.613 INFO com.kms.katalon.core.util.KeywordUtil - JIM TListener MSG [Test Cases/SCENARIO/3-ADMINISTRATION/3-6-API_TOKENS/TA_GuiTokenAutoDelete FAILED.
Should KS clear this context or should I because I have a custom solution? If I need to clear the context Status and Message, then how should I do that correctly?
Thanks,
Jim
It would be great if you could use Playwright as part of your engine.
Closing this thread for now. In case anyone needs assistance locating documentation here is a link to the different Release Notes: Version 8.x | Katalon Docs
Thank you for your participation, Sara