Hello! This is a simple outreach question for forum users.
As we know, artificial intelligence is increasingly present in everyday products.
I would like to know how you approach the testing stage of products and if it has changed the way you work.
How do you test that a bot, for example, generates an acceptable or excellent response to the user?
How do you include prompts testing in your work?
Are you experiencing many changes in your day-to-day work as QA’s performing these tasks? Do you know any interesting tools about prompt testing/IA?
I would just like to see your points of view and if you know interesting information, I think there are some good professionals here.
Regards!
3 Likes
This is a really interesting topic. Hope this gets lots of discussion from others. Meanwhile, what is your point of view?
1 Like
This seems like an area where manual testing (or another LLM) would be of most interest but if one were to automate the tests for some reason, then tester should be very careful to aim for deterministic tests. Eg if a question demanding more than a one-word answer is asked in ChatGPT, we know by now that repeatedly asking the same question will rarely (never?) result in the exact same response being given.
This is superficially useful for cheating students for example - feeding the same prompts across a class of students results in slightly different papers for each student, but these can be picked up by anti-cheating software that looks for specific patterns, and of course hopefully the teacher knows their class well enough to spot papers written by a third party! As testers, perhaps we should be more knowledgeable about these patterns that LLMs apparently demonstrate?