Hallucinations in Katalon Studio

AI & Agentic Testing
1

Katalon Studio is suggesting me to use waitForElementText().

But wait this Keyword does not exist.

Somehow it confidently suggests to use this keyword for waiting for element text.

This is definitely not expected at this level, given Katalon AI should have all the Katalon keywords handy.

image
image847×392 29.2 KB

2

Hehe, that’s exactly why every AI tool comes with a disclaimer: “AI-generated content may be incorrect. Please verify the information before relying on it.

3

exactly explains why human judgement critical while using AI

4

Yeah correct..

But what about beginners, they paid to Katalon Studio to get specific answers and if they implement in scripts, they might end up spending hours in finding out why the script is failing.

5

I completely agree with your point. In fact, the Katalon AI team should take this seriously, as it can significantly impact the credibility and trustworthiness of the AI tool.

6

Hi @akshay_sartabe, thanks for flagging this and for the screenshot. You’re right, waitForElementText() isn’t a real Katalon keyword, so the AI shouldn’t be suggesting it.

We’ll review this with the team and work on improving it. Thanks for helping us make it better! Also can you share the context, at which question or prompt did it get hallucinated.