The Landscape of AI-Enabled Test Automation Tools

Hi community members,

Let’s check out our recent article about AI-related topic. AI-Enabled End-to-End Test Automation Tools: The Landscape

We would like to grab this chance to know more about you and your AI stories. Share with us more in this dedicated area!

Best,
Katalon Community

1 Like

Hi @vu.tran

I’m not sure what to make of this. The article seems to rest on the publication of an “academic paper” (PDF) written by Phuoc Pham et al. Having read both, I’m not sure either lends itself well to feedback in a public forum. The former is a marketing piece filled with (as you would expect) marketing claims, the latter is just barely good enough for submission to a recognized academic institution, though not without some diligent rewriting and proofreading – indeed, some parts are too weak to be taken seriously.

IMO, testing is a vast topic area drawing on the disciplines of engineering and science – not marketing and tenuously drawn conclusions from so-so “academic papers”. Sorry.

As to your prompt…

I don’t see AI (or for that matter, ML) anywhere in any software in my tool-chain at the current time. How then might I share my “AI stories” in a relevant (i.e. relevant to testing) manner?

1 Like

Thank you so much @Russ_Thomas for sharing the thought. Since AI becomes a trendy topic with the ChatGPT emergence, I would like to know more what other benefits the new technology could help testing more efficient.

One of our team members have tried and found an interesting helpful area

ChatGPT can also be used to generate Test Data and SQL queries, which is very helpful for Testers.
Query: Create a sample CSV data with 5 having headers name,last-name,age with random values, comma separated


I would like to see how you would approach all of the new trends and as an experienced tester and Katalon veteran, we have a lot to learn from your perspective regardless you’re using any AI-augmented practices or not. This place is more to share and learn among others.

We also tried with other scenarios and found a reasonable answer from ChatGPT. Meaning, a newbie or a manual tester could learn from this source and verify with more experienced practitioners, maybe that’s the case?

Then we tried to challenge the bot a bit more :sweat_smile:




(Please note that we have not verified those codes, just a testing question)

I’ve wrangled and wrestled with ChatGPT quite a lot, though I have never tried directing it to write test cases.

Interesting aside – the ChatGPT steps output looks remarkably like MY test script report output: :slight_smile:

Here is mine:

But yes, you should take care with code generated by ChatGPT. It seems to convey confidence in the code it produces, even when it is completely wrong. I tried to get it to produce macro code for TiddlyWiki. The code looked like TiddlyWiki code but was in fact completely wrong. I couldn’t find a way to teach it how to write the code correctly.

Here’s an example. This is my conversation with ChatGPT:

The following is ChatGPT’s (bad) code rendered in TiddlyWiki followed by my corrected code:

The construct ChatGPT is trying to use to access the incoming parameter, {{{ [[text]] }}} is called a “filtered transclusion”. This boils down to producing the WORD text, not the value of the incoming parameter text.

ChatGPT has two choices, use $text$ or <<__text__>> but I could not get it to produce the correct result.

1 Like

Okay, so I decided to persevere, and try to teach ChatGPT how to write the macro correctly (I almost gave up again!)

Notice too, I made a typo. In my last sentence I meant to say “with-out wrapping the parameter”. As it turned out, ChatGPT managed to get to the correct version despite my mistake… eventually.

Here’s the thread:

Notice in the above, ChatGPT is still trying to access the text param as a variable (using <<...>>)

Yes! ChatGPT has finally started to see the light – now we have $text$. But there are still problems to deal with.

I found this part of the conversation very interesting. And all it took was me prompting ChatGPT to RTFM :wink:

All of which sounds well informed and justifiable… except for one thing: it’s complete nonsense!

But that’s just more bad code! <html> tags? There is indeed a <wikify> widget but it has no place here!

I was starting to think I was getting nowhere and was close to giving up again. As a last attempt, I got a little short with my student… I really didn’t expect the response I got:

Just as a reminder, here’s mine from earlier:

image

:tada:

You’d think that was the end of the story. But just before I signed off I thought I’d end the conversation as though I were truly talking to a human student:

There is a minor syntactical error in there, but I chose to let it ride. :wink:

So, it is possible to teach ChatGTP to code. But I’m not sure I have enough years left to teach it Groovy. :confused:

do you really think AI is … that smart?
or it has been trained on existing resources?

worth to read:

if needed, i can provide more articles.

my question, on this matter, up to no, is:

@vu.tran are you sure ChatGPT is creating new content, or it is just an ordinary (but ‘smart’) plagiarist?
(subject to various copyright and licensing infrigements)

Of course. Like any actor, it’s only as good as its last response and should only be trusted based on its worst response.

for now, me grabbing popcorn :smiley:

And, all conversations about AI (more strictly speaking, LLMs) are better conducted after you (we, all of us) have digested and taken on board “Talking About Large Language Models”, (Murray Shanahan, 2022). To do otherwise risks unfortunate (if not dangerous) misunderstanding.

2 Likes

proof of cconcept.
for fun i installed this and asked something simple:

at the moment of writing, the amazing bot is still lagging to find an answer for the last question.

ofcourse, you will say, i dont have an open Ai account.
why should i?
attempting to create one with a ‘decent’ password for my mind, with combined characters, it fails because must be ‘6 to 12 alphanumeric characters’ like the suggested ‘BigMandy’.
therefore, as an inpatient human, just uninstalled this crap.
will never try again.

for my eyes, is just a big fuss (will spend some time one day to chat with the ‘real’ one) heavily abused by various smart guys masking themself as developpers to gain quick money, without any guarantee the answer is received from a verified source.

attempting to speak with ‘the real one’ i got

which, ofcourse, ‘increased’ my confidence level a lot

:rofl:

   

Hi @Russ_Thomas,

Just out of curiousity to learn more about AI use case in testing. Could we consider autonomous testing as AI-augmented technology? One of the highlighted use cases is test generation which I found super helpful to testers. However, AI backed claim is a bit overrated?

Also, my colleague also shared something interesting about use cases for QA Augmenting QA processes with OpenAI | by Gary Parker | Medium.

This is a tough one to unravel, but I’ll take a shot. And note, I reserve the right to disagree with myself (perhaps as soon as tomorrow!) since this is a rapidly changing environment with loosely defined jargon that makes it difficult to nail one’s colors to the mast with any degree of certainty.

So, while answering you, I’m using the definitions as published here.

Based on my understanding, it would be fully AI (not “AI-augmented”).

  • “Autonomous testing is the process of creating, maintaining, and analyzing a series of automated tests (i.e. a test suite) without human assistance.” (My emphasis) (Link)

Just for fun, ChatGPT’s answer:

image
(Edited for length)

If your “testers” are humans, I don’t see how they’re involved.

I think that remains to be seen. In 2-5 years time, Testing will likely look quite different. How big a part AI will be playing is not easy to know. I suspect it will be playing a big part, but equally, we may be looking back and wondering, “What the hell were we thinking?” :wink:

Indeed. BDD and AI would seem to be a good fit. I imagine it might be easier to train an LM to understand a subset of “English” suitable for use as a BDD script – possibly easier than training it to handle the more exacting demands of typical coding languages. Though, of course, neither is impossible.

My answer on this will be long but short.
One of my favorite novel some time ago was a fiction about a tester.
Will be useless to name it since it is written in Romanian language only.
The idea was the tester was using an automated system for testing some more automated stuff like human imitation (androids?)

But nobody was concerned on ‘who is testing the tester’…