The AI enabled feature – Analyze doesn’t give proper result. Everytime or most of the time it shows as automation issue only even when it is related with Test data or some Test script issue.
So even when I know it is test data issue then also it shows Automation issue, and 95% times it shows automation issue only
just curious do you have any suggestion for your problem @arvind.choudhary ?
is it something like this: Improving AI efficiency in test analysis - #2 by depapp
Suggestion would be–
the AI analyse button choose the closer enough option
If for example, the issue is related with test data and instead of that if it shows automation issue , and although we need to see the log file indeed then what is the use of the Analyze button
ah i see. noted @arvind.choudhary
Ticket has been created: Jira
Thanks Bella, it will help
Just thinking out loud –
How would the AI know that it is test data issue?
How would the AI know what is valid data or invalid data for your application?
Unless it has access to the application’s business rules, expected datasets, test case intent, or historical context, it is mostly analyzing logs and error messages. In many situations, the same failure pattern could be caused by automation, test data, environment issues, or an actual product defect.
Let me explain
Out of many let us discuss one example, provide me your view.
If I retrieve a text or numeric value from the UI and use it as the actual value, I then compare it with the expected value provided through the test data. According to the requirement, the actual value fetched from the UI should exactly match the expected value from the test data. If both values match, the test is considered passed; however, if there is any mismatch between the actual and expected values, the test is considered failed.
Now Analyze button shows me Automation failure.
So whether it is automation failure or the test data issue?
Most AI analyzers classify failures based on the failed test step rather than true root cause analysis and categorize it as an automation issue because the failure originates from an automated verification step. However, from a QA perspective, this is more accurately a Test/Data Validation Failure or Application/Test Data Issue, assuming the locator and comparison logic are verified to be correct.
Simple rule I expect from the AI enabled Analyse button that,
If the script is unable to perform the intended action → Automation Failure.
If the script performs the action successfully and detects an incorrect result → Application/Data Failure.
And eventually if I have to debugg into deep to get the actual root cause of the issue then why will I use the AI enabled Analyze button
In this scenario, the automation is functioning as expected because it successfully fetches the value from the UI and performs the comparison. Since the test fails due to a mismatch between the actual and expected values, the root cause is more likely a test data or application issue rather than an automation failure.
So this one example which I gave, however I can provide more based on the result I get in the automation in my project
