Katalon’s AI-powered solutions help QA teams by automating intelligent test generation, execution, and analysis across APIs, cloud services, Big Data, and analytics platforms, ensuring faster validation and reduced risk in complex enterprise ecosystems. Testers should evolve by strengthening skills in automation frameworks, API/cloud testing, and data-driven validation, while embracing AI-assisted tools to shift from manual checks to strategic quality engineering that supports digital transformation.
this is a different ball game , cant be done so easily with just some scripts or utilities
Yes, hmm that I understand
Really?
Do we really provide all code base to AI?
Good question, Akshay — and no, you don’t need to feed your entire codebase into an AI tool for this to work. That’s a common misconception, so glad you raised it.
What’s actually happening in “AI predicts risk areas” is usually one of two things, and neither requires exposing your full soource:
1. Metadata-based risk models (most common approach)
These models don’t read your code line-by-line — they train on metadata about your codebase history:
- File/module change frequency (churn) from git logs
- Which files were touched in commits linked to bug-fix tickets (via Jira/commit message correlation)
- Code complexity metrics (cyclomatic complexity, file size, number of dependencies)
- Historical defect density per module
This is basically statistics on your commit and defect history, not the code itself. Tools like Launchable and some CI risk-scoring plugins work this way — they ingest commit metadata + test results + Jira links, not your source tree.
2. Static analysis + local/on-prem models
For deeper code-level risk analysis (e.g. flagging risky diffs), teams either:
- Run static analysis locally and only send derived signals (complexity scores, diff size, affected function names) to the AI layer, or
- Use a private/on-prem LLM deployment (or an enterprise-tier tool with a data-processing agreement) if they genuinely want code-level review — which is a very different trust boundary than pasting code into a public chatbot.
So Practically : if your org is doing this with a SaaS AI tool, check what data leaves your network — for most risk-prediction use cases it’s commit metadata and defect history, not raw code.
The metadata-based approach makes sense for identifying potentially risky modules.
But how does the AI determine (just from the metadata)
- Which specific functionality is impacted by this new change?
- Whether the change affects existing behavior, introduces new behavior, or is simply a refactoring?
- Which specific set of tests to re run?
- Which specific set of regression tests to run?
I’ll explain this based on my understanding of how these systems typically work.
To make it easier I have taken a generic example. So here it is not purely AI driven but much of it relies on deterministic engineering principles.
1. How does the system decide which tests to rerun?
This is generally not an AI enabled. Instead, it is handled through Test Impact Analysis (TIA).
Each time a test suite executes, coverage tools like Jacoco and others record which tests interact with which parts of the codebase. It tracks exactly which lines, methods, and branches (such as if/else statements) are executed during a test run. ANd gradually this creates a traceability map, for example: Test_X covers Functions A, B, and C.
When a code change is introduced, the system analyzes the modified functions and consults this mapping to identify the tests associated with those areas. It then reruns only the relevant tests rather than executing the entire suite.
Most implementations also consider dependency chains. For example, if Function A depends on Function B, and Function B is modified, tests covering Function A may also be included, even if they do not directly interact with Function B.
So that’s why I say that this process is largely deterministic: identify changed code, map it to impacted tests, and execute those tests. In the tools like Launchable it is using the same. And now the AI component is typically used to prioritize or rank tests based on risk, rather than determine the impacted tests themselves.
- How are additional regression tests selected?
Here, it is based on the historic data like whenever Module X experiences failures, Module Y is also affected approximately 30% of the time. Based on such historical trends, the system may recommend or include additional regression tests from related areas, even when there is no explicit code-level dependency between them. So here unlike above Test Impact Analysis, the deterministic thing, it is based on statistical inference and historical behavior patterns.
So AI can only assist with impact estimation and risk assessment, but human judgment is still critical when interpreting the true intent and significance of code changes.
Hi @jahnavi.b, I’d say the most practical skill is learning how to work with AI throughout an actual testing workflow, not just learning about AI or collecting prompting techniques.
A few skills I would focus on:
Giving AI the right context: requirements, business rules, application behavior, test data, and expected outcomes.
Reviewing AI-generated tests critically: checking coverage, assertions, edge cases, and whether the tests address real business risks.
Testing and refining AI output: run what AI creates, identify what does not work, and use that evidence to improve the result.
Using AI beyond test generation: for requirement analysis, test data creation, automation, failure investigation, and reporting.
Maintaining testing and technical fundamentals: testers still need to understand APIs, locators, assertions, logs, and basic scripting to know whether AI’s suggestions make sense.
Deciding what is worth testing and automating: AI can generate many test cases, but testers still need to identify which ones actually matter.
The best way to build these skills is to start with one real task. For example, use AI to generate tests from a requirement, review the output, run the tests, and note what context helped improve the result.
I second this
Wrapping up Ask Katalon Anything #11 — thank you all!
That’s a wrap on AKA Episode 11! A huge thank you to everyone who jumped in with thoughtful, challenging, and practical questions on AI in testing and what it means to be a future-ready tester.
A special shout-out to the community members who drove the conversation with their questions: @Monty_Bagati, @masroor.sadiq, @sneha.sajeev, @dineshh, @ss1512, @nimisha.r, @akshay_sartabe, @jahnavi.b, and @stevio052002. Your questions from the Playwright-vs-Katalon debate to AI governance, ROI, migration, and the skills that actually matter, made this one of our liveliest rounds yet.
Thank you also to our panel: @xuan.tran, @arvind.choudhary, @depapp, and @viet.nguyen for the honest, in-depth answers.
Next up: we’ll be sharing the top 3 most-upvoted questions by our panels for the gift card rewards, so keep an eye out for the winners announcement. And stay tuned a community poll is coming shortly!
See you in the next episode.
AKA Episode 11 — Top Questions
A huge thank you to everyone who joined Ask Katalon Anything Episode 11: AI in Testing & the Future-Ready Tester! We received so many thoughtful questions. Here are the standout questions from this round — cast your vote in the poll below for the one you’d most want the panel to dig into!
@akshay_sartabe — As teams rely on AI to generate tests, debug failures, and suggest fixes, who is ultimately accountable when AI makes a wrong decision that leads to a production issue? How should QA teams design governance and approval workflows to gain speed from AI without introducing hidden risk?
@sneha.sajeev — Finding a bug is often easier than understanding its root cause. How do you see AI helping testers with bug investigation and root cause analysis, rather than just generating test cases and automation scripts?
@jahnavi.b — How does AI change the traditional software testing lifecycle (STLC), and how can teams design a feedback loop to continuously improve AI-generated tests based on production defects and test failures?
@dineshh — With AI agents now able to run tests autonomously, what skills should QA engineers prioritise investing in, and what would a “future-ready” cloud QA engineer actually spend 60% of their time doing in 2027?
@Slowey — If I build up my test history and intelligence inside a platform, how do I avoid lock-in when the next tool comes along?
@akshay_sartabe — Where do you think AI in testing is being overused today? In which parts of the testing lifecycle does it add little or negative value?
@Monty_Bagati — Why do you think Playwright, with all the AI embedded in its ecosystem and frequent updates from Microsoft, will not outshine the Katalon platform? Why would organizations choose Katalon over Playwright?
Which question resonates with you the most? Cast your vote below!
- @akshay_sartabe — AI accountability & governance/approval workflows
- @sneha.sajeev — AI for bug investigation & root cause analysis
- @jahnavi.b — AI reshaping the STLC & feedback loops
- @dineshh — Skills for a future-ready QA engineer in the age of AI agents
- @Slowey — Avoiding platform/vendor lock-in for test history & intelligence
- @akshay_sartabe — Where AI in testing is being overused today
- @Monty_Bagati — Playwright vs. Katalon: why choose Katalon?
0
voters
Everywhere Monty wins-- he should change his surname from Bagati to Bugatti 