Problem statement: I want to the AI agent to check how many test cases have been using a Custom Keyword which I have created so that I can decide on how to efficiently update the function.
Limitation: The AI agent is not able to perform this task of even checking how many test cases is using this keyword. As a next step i wanted to update the scripts using this keyword.
Summary: So I have around 400+ test cases in my project now. Now there is a major change in functionality and i want to update test cases impacting the functionality, for which the count is around 150. So I was curious if the agent can help me in updating these scripts. But seems like the AI Agent is not able to this task here.
It keeps giving me below error -
There was an issue processing your request. This is typically due to the question size or current rate limitations. To help get your answer, please try:
⢠Clearing the conversation
⢠Turning off the project context
⢠Breaking your question into smaller pieces
Are there any custom configurations that needs to be done to make the Agent more capable of doing these kind of stuffs? OR Currently this is just the limitation with what we can do with AI agents in Katalon.
Want to understand more before deep diving.
Maybe some experts can guide me here.
2 Likes
Hi there, and thanks for posting in the Katalon community! 
To help you faster, please review our guide on Custom Keyword here: Introduction to custom keywords in Katalon Studio | Katalon Docs. Double-checking the steps and configurations might resolve the issue.
If the doc doesnât help, feel free to provide more details, and a community member will assist you soon. Thanks for being a part of our community!
Best,
Elly Tran
Bumping for more help, may I bring @xuan.tran to help with this case 
@akshay_sartabe For the use case you share, KS MCP can definitely do it. However, it needs to read through every test case in your project to do so, in which 400+ tools call per question is impossible. My suggestion here is that:
-
Start with a small batch first, maybe 5 or 10 test cases, then just ask StudioAssist to continue with the rest
-
Increase the max tool call to avoid disruption (by default, we set 5 tool calls per question)
-
You can also bring your own key to StudioAssist instead of using Katalon AI Service, A stronger model can also help
Hope that this can resolve your concern 
2 Likes
If I were you, I would develop a classical bash shell script which greps through the Groovy scripts under the Scripts folder. It would be enough to identify which Test Case scripts refer to your custom keyword at which line number in the .groovy files. The goodness of this low-tech non-AI approach is that it would run fine with large number of target files. 400+ would be easily processed.
2 Likes
Itâs too easy.
On my Mac, using Visual Studio Code, I made a file grepTCs.sh in the root directory of my Katalon Studio project. I wrote the following bash script:
grep -r -n "my.sample.Greeter" "./Scripts"
I changed the mode of the file to be executable:
$ chmod +x ./grepTCs.sh
I ran this using the Terminal window in Visual Studio Code. I got the following output.
$ ./grepTCs.sh
./Scripts/TC1/Script1763708242058.groovy:1:CustomKeywords.'my.sample.Greeter.sayHelloTo'('World')
./Scripts/TC3/Script1763708465245.groovy:20:CustomKeywords.'my.sample.Greeter.sayHelloTo'('Moon')
This result implies that âTC1â and âTC3â refer to the custom keyword my.sample.Greeter. The line number is displayed also.
I believe that the shell script will scan 400+ files in a few seconds.
By the way, I asked Copilot to give me a sample code saying âshow me a bash script that greps text files under a folderâ. Copilot instantly replied the code that I copy&pasted above. How AI is useful!
2 Likes
@kazurayam
I totally agree with your solution, I was able to do this even with a simple Python script, which took hardly few seconds.
I can even use the Find and Replace functionality to update all the required test cases.
However Iâm asking here from AI agent point of view. If Iâm to use the agent for some complicated tasks I definitely want it to be smart enough to determine how to smartly identify such scenarios.
This is just a basic scenario that i came across and wanted to see how can I truly leverage AI capabilities.
Also if I were to use Copilot or any other any AI tool (which are available for free) why should I use Katalon Studio Assist or Katalonâs AI Agentâs capabilities?
2 Likes
@akshay_sartabe
I donât think that the AI can invent anything new from scratch. AI needs some text sources that define a problem to solve and propose a reliable solution. The source must be published on the Internet.
Would you post your solution in Python here so that AI can read it?
AI will learn it. It wouldnât take long before they find the published solutions. Based on that, they would get ready to provide nice suggestions based on your knowledge to those who ask about the same issue in future.
1 Like
I see where @akshay_sartabe is coming from, and I think that the term agent is deceptive to the user using Studio Assist. Asking it about my Katalon billing is something we wish we could do from Katalon Studio, but the agent is specific to Test Generation. The fact is that not all agents are a one fit solution. Thatâs why in a perfect world, there would be a âtest summary expertâ - an agent that could interpret test results and retrieve info quickly. There would be a âusage/billing summary expertâ - an agent that could summarize the organization users license usage.
Iâve been creating my own solutions to this by creating a Claude Code SKILL. You can call that skill with a Custom Keyword. Claude Code packages a skill-creator skill so you can easily create a skill that works best for your tests.
I asked to ChatGPT the following question
In Katalon Studio, I want to grep text files under the Scripts folder with the fully qualified name of Custom Keyword as pattern. How can I do it?
It replied to me:
In Katalon Studio, the Scripts folder contains Groovy files that represent your test steps. If you want to grep all script files for usages of a specific Custom Keyword (using its fully-qualified name), there are a few reliable methods depending on whether you prefer:
- Katalonâs internal search
- External command-line tools
- A small Groovy script run inside Katalon
Below are the recommended approaches.
1. Use the Grep-like Search Built Into Katalon Studio
Katalon Studio is built on Eclipse, so you can use the Eclipse âFile Searchâ.
Steps
- Press Ctrl+H to open the Search dialog.
- Select the File Search tab.
- In Containing text, enter the fully qualified keyword name, e.g.
I tried the first recommendation: In Katalon Studio GUI, Press Ctrl+H to open the Search dialog.
Well, this is good enough, isnât it?
1 Like
