Hi All,
I’m looking for some guidance
What is the best approach to distinguish between failures in our Katalon Studio test scripts (coding failures) and failures in the Application Under Test, both are reported the same in TestOps as Failures.
The scenario is that we have built lots of tests. When I run them some fail and some pass. For the ones which fail, some are expected to fail as the AUT has defects, but without going through all the log files and re-running debugging, how would I know?
The idea scenario (?), would be to have some way for marking / logging / distinguishing between an expected failure due to defects in the AUT which is pending a fix, and those which the scripts fails because something unexpected has happened, either because the AUT has changed in some way or just a plain old bug in the test script.
I’m sure others must have faced this scenario, your thoughts pls,
A Philosophical question. I think, you can not forecast the cause of failures, so you can never distinguish these two categories before a failure occurs.
When I encounter a failure in UI tests and got to know that it is due to defects in the AUT, I would make a JIRA Ticket for it for fix, and tentatively change my test code to accept the current (defective) behaviour of AUT being reproduced. It is a good idea to write the ticket ID in the assertion messages. This tentative change is a good marker to drive my work.
The changed test should pass for a while until the AUT is fixed. Later when the changed test fails again, it will remind me of the JIRA Ticket where I can review the details of the known defect. I will look at the new failure carefully and, if the fix in the AUT is acceptable, I would amend my test back to the right one.
I think, this way is a sort of Test-driven development approach. Do you think it is cumbersome? — Yes, it is; but it works.
This way of test maintenance is possible only when the target AUT has been well accomplished and stable enough. If the AUT is in the early stage of development and has lots of defects, then this way would be hard to follow.
