Katalon creates different results in Test Suite

Hi Team,

I am using katalon since last 6 months, and i m facing this one issue continuously.
Whenever i run katalon test suit it gives different result set. It gives different result when system is being changed or updated. Can anyone please help me with this?

When you say the “system” is being changed or updated, do you mean the browser code or the hardware? If you mean the browser code, then it is possible that your XPaths to the elements may be different.

System means Application URL with different versions, in which classes remains the same. Still sometime XPaths gets changed to relative XPaths by its own. Our team facing this issue again and again.

Are you entering the XPaths manually, or are you using Web Spy or some other tool? Web Spy does not always use the most “reliable” XPath in a changing environment. You may need to modify the path to improve it. Do you use dynamic id or static id? If you use static id, then you should try to shorten the XPath by using the closest static id, such as:

id("DataModel.Status)/…/preceding-sibling::div[@class=“label 2b class”]/label

Notice that the above XPath does not use double slashes to start. Getting to know the ancestries/helpers of XPaths will assist in better locating the elements. (Note: it should be only two dots before the preceding-sibling phrase but I cannot get the format to show proper.)

We are not using Web Spy we are creating test objects and entering XPaths manually. We are using dynamic XPaths, and our code structure is based on classes so we are using classes instead of id’s. I just want to confirm that is it known issue that in some extent katalon test cases needs maintenance.

Can we run our sanity and regression suits in katalon. Because right now we have just completed POC for smoke suit which has nearly 200-300 cases. can we go ahead with sanity and regression suits which will be holding 700-800 test cases in test suits based on different modules. if you guys can help me with this scenario it will be really great.

Well, you probably want a community moderator, like @Russ_Thomas, to answer some of your concerns, however, my general belief is that code does not change on its own. You should talk to other developers/testers to see if anyone is changing the XPath of objects to suit a “new” build, breaking the path for previous versions. (I may not understand the whole picture though.)

Just curious, in 6 months of using KS, how many Test Cases have you created? Maybe I am doing something wrong in how I am using the app. especially if you want hundreds.

There is nothing wrong with doing that. I never use the spy tool.

By the sound of it, you are using a POM (page object model). Good.

Not sure I’d describe it as an issue as such, but yes, unfortunately, “things change” because a website “evolves” over time. Tests need to reflect the current state of the website.

However, take this advice on board: Make your selectors as SHORT and generic as possible. This is an art and a science. Too generic and they may become flaky; too short and they may become brittle/volatile; all down to changes in the document hierarchy and structure. Long selectors are pretty much guaranteed to fail sooner.

A conversation with the page developers is vital and helpful. It’s to their benefit to know which CSS ids and classes you are relying on to create and operate test cases. Since you are using a POM, make sure that a given page object has all the child elements’ xpaths stored as member variables:


public class Mypage extends Basepage {

  String xpathBtnHome = "xpath to find the home button"
  String xpathFirstName = "xpath to find the first-name field"
  String xpathLastName = "xpath to find the last-name field"

  // etc
}

Following is a link to a post by @Brandon_Hein that shows this model in much more detail:

Without seeing your tests and test suites it’s hard to pass comment on your use case. Very much depends on your intentions for coverage. In my world, I aim to cover “user stories”. For each Test Case, I try to capture an entire user story from one end (login) to the point where they have completed the task. If, for you, that results in 800 tests, that’s fine. If you have issues, post a question!

1 Like