Great question, and it’s one we think about a lot on the Studio team.
My honest take: the faster AI gets at producing tests, the more a tester’s value moves from writing tests to deciding what’s worth testing — and then standing behind that decision. AI can generate a hundred test cases in seconds. It can’t tell you which one, if it failed silently in production, would actually hurt a patient. That judgment about risk and consequence stays human.
A few skills I’d bet on continuing to matter:
Questioning the requirement, not just testing against it. AI tests what the spec says. The valuable work is catching what the spec missed — the validation nobody wrote down, the ambiguous acceptance criterion. AI optimizes within the frame; people question the frame.
Critically reviewing AI’s output. AI is confidently wrong sometimes. Knowing when a generated assertion is hallucinated, or when an edge case got quietly skipped, is becoming a core skill — closer to auditing than authoring.
Owning the sign-off. In regulated domains someone is accountable. AI can assist that person, but it can’t be that person.
The way we try to design for this in Studio is to keep the human in the driver’s seat — AI proposes, you decide. The goal isn’t to replace the tester’s judgment, it’s to give them more time to apply it where it counts.
Curious how others here are drawing that line in their own teams.
Hi @Monty_Bagati — really appreciate you pushing on this, and you’re asking exactly the right question. Let me give you my honest take as straightforwardly as I can.
You’re right about Playwright. It’s genuinely excellent — fast, free, brilliantly engineered, and the AI tooling around it keeps getting better. I’m not going to argue otherwise, because I don’t think there’s a clean right-or-wrong answer here. There’s only what a given team is optimizing for, and for plenty of teams Playwright is the right call.
I won’t repeat the great points already made above — @depapp and @arvind.choudhary covered scope and buy-vs-build better than I could. Depa’s line stuck with me:
That’s the part I’d just gently expand on. “Free” is true at the framework level — but the moment you pair Playwright with Claude, Copilot, Cursor, a CI grid, reporting, and the engineering time to wire it all together and keep it healthy, you’re back to a real number. Sometimes a very good number, sometimes not — it just depends on the team. The cost rarely disappears; it moves around.
And honestly, the framing I’d most gently push back on is “ditch.” The strongest teams I see don’t pick a winner — they let each tool own what it’s best at. @arvind.choudhary already said it well:
So you may not love this answer, but I think it’s the truthful one: it’s less “which tool wins” and more “what is your team set up to get the most out of.” Both can be right.
AI keeps suggesting test patterns that work in theory but break in our specific environment (e.g., suggesting setText() for Firefox when we know it needs sendKeys() + blur()). What’s your framework for deciding when to trust AI vs. when to push back? Do you have a checklist of ‘red flags’ that signal AI is generating fragile tests? How do you teach junior testers to critically evaluate AI output instead of blindly accepting it?"
The Buy Vs Build argument was valid and great and fantastic before AI , MCP , Claude , AI in every aspect.
I was one of the contributors of the katalons buy Vs Build white paper.
Katalon was great option until 2025 and best for test engineers not good in coding etc. but now somehow I feel , organisations would prefer playwright with MCP and other AI utilities etc as code is not an issue now.
And katalon is licensed per user per machine , while playwright is free.
If AI can generate a hundred test cases in seconds, what should testers actually spend their time on two years from now? Which skills are you betting still matter - and which ones would you tell your own team to stop investing in?
Writing tests was never the hard part; maintenance and flakiness were. Does AI actually reduce maintenance, or does it just generate more tests for me to maintain? What’s the concrete answer on self-healing and upkeep?
If I build up my test history and intelligence inside a platform, how do I avoid lock-in when the next framework comes along? Can I bring my own framework - say, run Playwright - and still get the platform layer on top?
Katalon is licensed per user per machine; Playwright is free. As AI commoditizes the framework itself, how do you justify the licensing model to a cost-conscious team that feels code is no longer the barrier?
One thing I’ve learned while using AI for test automation is that it becomes much more effective when you provide it with the context of your framework and application. For example, in your Firefox case (setText() vs. sendKeys() + blur()), AI will probably suggest the generic approach the first time because that’s what works in many environments. Once you explain that your framework requires sendKeys() + blur() for Firefox, AI can use that context throughout the conversation (or through project-specific instructions). So human review remains essential.
For junior testers, I don’t think AI should be used only to generate test scripts. The real value comes from using it as a learning partner. After AI generates a script, they should keep asking questions like:
Why was this approach chosen?
How does this keyword work?
What are the alternatives?
What could fail in this implementation?
Is this solution browser-specific or generic?
How can this script be made more robust and maintainable?
This habit helps them understand the reasoning behind the automation instead of simply copying and pasting code. Over time, they’ll develop the judgment to recognize when AI is giving a solid recommendation and when it’s applying a generic pattern that doesn’t fit their project’s specific environment.
Where do you think AI in testing is being overused today? In which parts of the testing lifecycle do you think teams should deliberately avoid applying AI for now because the trade-offs are not worth it yet?
As teams start relying on AI to generate tests, debug failures, and even suggest fixes, who is ultimately accountable when AI makes a wrong decision that leads to a production issue?
In practice, how should QA teams design governance and approval workflows so they gain speed from AI without introducing hidden risk?
Hi @arvind.choudhary , how does AI change the traditional software testing lifecycle (STLC), and how can teams design a feedback loop to continuously improve AI‑generated tests based on production defects and test failures?
Hi @viet.nguyen What practical skills should testers focus on today to effectively work with AI in daily testing tasks, beyond just theoretical understanding?
test generation is the obvious overuse. teams ship 50 AI cases and nobody asks which 5 actually matter. more tests just means more to maintain
i’d also hold back in regulated stuff. healthcare, finance, anything where a missed validation hurts someone. AI doesn’t know what a wrong test costs
and tbh i’d avoid letting AI do the final sign off. it can draft, it can triage, but when production breaks someone has to own it and that someone isn’t the model
the thing i keep seeing though is teams grab AI for the easy loud stuff like generation and scripting, then skip the boring parts. root cause, flakiness, coverage gaps. that’s where people actually spend their time and where AI helps the most imo
How does Katalon’s AI-driven testing platform help Quality Engineering teams reduce test maintenance, improve defect detection, and increase release confidence without replacing human expertise?
As organizations increasingly integrate APIs, cloud services, Big Data, and analytics platforms, how do Katalon’s AI-powered solutions help QA teams automate intelligent testing across complex enterprise ecosystems, and how should testers evolve their skills to support this digital transformation and give a clear solution?
maintenance, defect detection, release confidence, all downstream of the same thing. AI patches broken locators so people aren’t fixing the same thing every sprint, and it spots patterns across runs faster than a human would bother to
but it doesn’t decide what’s worth testing and it doesn’t sign off on a release. that’s still QE
imho, so really it’s not AI replacing anyone, it’s AI clearing the noise so QE spends time where it actually matters
As we already know in the classic STLC we have these following steps (Requirement Analysis → Test Planning → Test Case Design → Environment Setup → Execution → Closure, where a human had to manually think through every step.
So AI is neither replacing any step nor bypassing any step, but reducing the time of each step drastically.
Requirement Analysis → Test Design: with NLP/LLMs now the tester can directly read a Jira ticket or acceptance criteria and directly draft test cases, edge cases, and even Gherkin scenarios directly. so Now with AI as well, we still have bottleneck but shifed from writing the test to "review whether the AI understood the requirement correctly or not.
Test Data task: You can directly get Synthetic data via ML models instead of hand-crafted fixtures.
Dealing the lengthy regression pack: AI looks at previous code changes and bugs to predict where new issues are most likely to occur. Based on this, one can decide which tests should be run first and thus very useful when the project has too many test cases to execute in every build. And if you create Agent it will take care of tis own.
Report dealing: log analysis, flaky-test detection, auto-clustering of failures into root-cause buckets, and defect triage suggestions gets speed up.
