Clicking on 'a' tags

I want to find broken links of a site. I use the

'links = WebUI.getAllLinksOnCurrentPage(true, [])
for (def url : links) {
println(url)
}

method to gather all links. But the issue is this method gathers unwanted links as well which when clicked leads to script page of .js .

What i found is that the links which are visible on front end and i am concerned with have ‘a’ tags with href . If i locate the links with ‘a’ tag i would not be dealing with unwanted links like earlier. But i dont know how to gather the ‘a’ tag hrefs.
Please help!!!

Check the image the unwanted links i am getting.

We will need to see some of the HTML showing the links you want to get and the links you don’t want to get.

These are the ones i dont want

and these ones i am concerned with

Is there some reason you can’t use the excludedLinks argument to getAllLinksOnCurrentPage ?

@suraj.thakur

You are interested in only <a href="..."> tags and not in other tags like <link>, <script>, <img>. Then you should not bother yourself with WebUI.getAllLinksOnCurrentPage. You can find all <a> elements by WebUI.findWebElements.

You disclosed the URL of your target web app : https://malasuat.express.shoptimize.in/
So I could make a sample code. Make a Test Case script, copy&paste the following code and run it.

import org.openqa.selenium.WebElement

import com.kms.katalon.core.util.KeywordUtil
import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.RequestObject
import com.kms.katalon.core.testobject.ResponseObject
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

String pageUrl = "https://malasuat.express.shoptimize.in/"

WebUI.openBrowser("")
WebUI.navigateToUrl(pageUrl)

TestObject tObjAnchors = 
	new TestObject("all anchors").addProperty("xpath", ConditionType.EQUALS, 
			"//a")
	
// find all <a> elements
List<WebElement> anchors = WebUI.findWebElements(tObjAnchors, 10)

boolean hasBrokenLinks = false
StringBuilder sb = new StringBuilder()
for (aTag in anchors) {
	String url = aTag.getAttribute("href")
	String statusCode = getResponseStatus(url)
	sb.append("${statusCode} ${url}${System.lineSeparator}")
	// check if this link is broken
	if (statusCode != '200' && statusCode != '302') {
		hasBrokenLinks = true
	}
}
println("*** All <a> elements in ${pageUrl} ***")
print(sb.toString())
println("size of anchors: ${anchors.size()}")
WebUI.closeBrowser()

if (hasBrokenLinks) {
	KeywordUtil.markFailed("${pageUrl} has one or more broken links")
}


/**
 * try to make HTTP GET for this url and return the HTTP Status Code of the response
 * @param url
 * @return
 */
String getResponseStatus(String url) {
	String status = '???'
	if (url != null && url.startsWith('http')) {
		RequestObject req = new RequestObject('check if this url is reachable')
		req.setServiceType('REST')
		req.setRestUrl(url)
		req.setRestRequestMethod('GET')
		// Send the request and get the response
		ResponseObject res = WS.sendRequest(req)
		status = res.getStatusCode()  // '200' OK, '302' redirected, '403' etc	
	}
	return status
}

This script emitted the following messages to the console.

2020-11-21 11:26:54.477 DEBUG testcase.TC10                            - 9: println(*** All <a> elements in $pageUrl ***)
*** All <a> elements in https://malasuat.express.shoptimize.in/ ***
2020-11-21 11:26:54.480 DEBUG testcase.TC10                            - 10: print(sb.toString())
200 https://malasuat.express.shoptimize.in/#contentarea
200 https://www.shiprocket.in/shipment-tracking/
302 https://malasuat.express.shoptimize.in/wishlist
200 https://malasuat.express.shoptimize.in/
200 https://malasuat.express.shoptimize.in/checkout/cart/
200 https://malasuat.express.shoptimize.in/#social-login-popup
200 https://malasuat.express.shoptimize.in/#social-login-popup
??? null
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates.html
200 https://malasuat.express.shoptimize.in/crushes.html
200 https://malasuat.express.shoptimize.in/crushes.html
200 https://malasuat.express.shoptimize.in/jams-and-spreads.html
200 https://malasuat.express.shoptimize.in/jams-and-spreads.html
200 https://malasuat.express.shoptimize.in/bakery.html
200 https://malasuat.express.shoptimize.in/bakery.html
200 https://malasuat.express.shoptimize.in/confectionery.html
200 https://malasuat.express.shoptimize.in/confectionery.html
200 https://malasuat.express.shoptimize.in/food-service.html
200 https://malasuat.express.shoptimize.in/food-service.html
200 https://malasuat.express.shoptimize.in/recipe
200 https://malasuat.express.shoptimize.in/about-us
200 https://malasuat.express.shoptimize.in/contact-us
200 https://malasuat.express.shoptimize.in/#store.links
??? null
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/#
??? null
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/bakery.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates/cordials.html
200 https://malasuat.express.shoptimize.in/confectionery.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates/mocktails.html
200 https://malasuat.express.shoptimize.in/jams-and-spreads/high-fruit-jam.html
200 https://malasuat.express.shoptimize.in/bakery.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates/cordials.html
200 https://malasuat.express.shoptimize.in/confectionery.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates/mocktails.html
200 https://malasuat.express.shoptimize.in/jams-and-spreads/high-fruit-jam.html
200 https://malasuat.express.shoptimize.in/bakery.html
200 https://malasuat.express.shoptimize.in/bakery.html
200 https://malasuat.express.shoptimize.in/confectionery.html
200 https://malasuat.express.shoptimize.in/confectionery.html
200 https://malasuat.express.shoptimize.in/food-service.html
200 https://malasuat.express.shoptimize.in/food-service.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates.html
200 https://malasuat.express.shoptimize.in/crushes.html
200 https://malasuat.express.shoptimize.in/crushes.html
200 https://malasuat.express.shoptimize.in/jams-and-spreads.html
200 https://malasuat.express.shoptimize.in/jams-and-spreads.html
200 https://malasuat.express.shoptimize.in/bakery.html
200 https://malasuat.express.shoptimize.in/bakery.html
200 https://malasuat.express.shoptimize.in/confectionery.html
200 https://malasuat.express.shoptimize.in/confectionery.html
200 https://malasuat.express.shoptimize.in/food-service.html
200 https://malasuat.express.shoptimize.in/food-service.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates.html
200 https://malasuat.express.shoptimize.in/fruit-syrups-and-concentrates.html
200 https://malasuat.express.shoptimize.in/crushes.html
200 https://malasuat.express.shoptimize.in/crushes.html
200 https://malasuat.express.shoptimize.in/jams-and-spreads.html
200 https://malasuat.express.shoptimize.in/jams-and-spreads.html
200 https://malasuat.express.shoptimize.in/#first
200 https://malasuat.express.shoptimize.in/#second
200 https://malasuat.express.shoptimize.in/mango-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/mango-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/black-currant-crush-1000ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/black-currant-crush-1000ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/pineapple-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/pineapple-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/litchi-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/litchi-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/lemon-barley-water-cordial-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/lemon-barley-water-cordial-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/mix-fruit-jam-100g-cup.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/mix-fruit-jam-100g-cup.html
200 https://malasuat.express.shoptimize.in/pineapple-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/pineapple-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/mango-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/mango-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/black-currant-crush-1000ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/black-currant-crush-1000ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/pineapple-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/pineapple-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/litchi-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/litchi-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/lemon-barley-water-cordial-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/lemon-barley-water-cordial-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/mix-fruit-jam-100g-cup.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/mix-fruit-jam-100g-cup.html
200 https://malasuat.express.shoptimize.in/pineapple-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/pineapple-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/mango-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/mango-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/strawberry-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/strawberry-crush-5-ltr-pet-bottle.html
200 https://malasuat.express.shoptimize.in/pineapple-whole-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/pineapple-whole-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/litchi-squash-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/litchi-squash-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/mango-jam-500g-refill.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/mango-jam-500g-refill.html
200 https://malasuat.express.shoptimize.in/mixberry-high-fruit-jam-350gm-glass-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/mixberry-high-fruit-jam-350gm-glass-bottle.html
200 https://malasuat.express.shoptimize.in/kiwi-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/kiwi-crush-750ml-pet-bottle.html
200 https://malasuat.express.shoptimize.in/about-us
200 https://malasuat.express.shoptimize.in/about-us
200 https://malasuat.express.shoptimize.in/blog/post/april-lady-milkshake-recipe.html
200 https://malasuat.express.shoptimize.in/blog/post/citrus-punch-recipe.html
200 https://malasuat.express.shoptimize.in/blog/post/choco-biscuit-recipe.html
200 https://malasuat.express.shoptimize.in/blog/post/blue-curacao-recipe.html
200 https://malasuat.express.shoptimize.in/blog/post/april-lady-milkshake-recipe.html
200 https://malasuat.express.shoptimize.in/blog/post/citrus-punch-recipe.html
200 https://malasuat.express.shoptimize.in/blog/post/choco-biscuit-recipe.html
200 https://malasuat.express.shoptimize.in/blog/post/blue-curacao-recipe.html
200 https://malasuat.express.shoptimize.in/blog/post/april-lady-milkshake-recipe.html
200 https://malasuat.express.shoptimize.in/blog/post/citrus-punch-recipe.html
200 https://malasuat.express.shoptimize.in/recipe
200 https://www.facebook.com/malasfruitproducts/
200 https://twitter.com/malas_fruit
999 https://www.linkedin.com/company/malas-fruit-products
302 https://www.instagram.com/malasfruitproducts/
302 https://www.instagram.com/malasfruitproducts/
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/#
200 https://malasuat.express.shoptimize.in/products.html
200 https://malasuat.express.shoptimize.in/recipe
200 https://malasuat.express.shoptimize.in/bulk-enquiry
200 https://malasuat.express.shoptimize.in/about-us
200 https://malasuat.express.shoptimize.in/contact-us
200 https://malasuat.express.shoptimize.in/terms-and-conditions
200 https://www.shiprocket.in/shipment-tracking/
200 https://malasuat.express.shoptimize.in/cancellation-and-refund-policy
200 https://malasuat.express.shoptimize.in/privacy-policy
??? tel:+91-7767921551
??? mailto:%20%20info@malasfruit.com
200 https://www.facebook.com/malasfruitproducts/
200 https://twitter.com/malas_fruit
999 https://www.linkedin.com/company/malas-fruit-products
302 https://www.instagram.com/malasfruitproducts/
200 https://www.shoptimize.ai/
200 https://malasuat.express.shoptimize.in/customer/account/create/
??? null
200 https://malasuat.express.shoptimize.in/customer/account/forgotpassword/
2020-11-21 11:26:54.483 DEBUG testcase.TC10                            - 11: println(size of anchors: $anchors.size())
size of anchors: 171

And finally this script failed because it found some links in doubt:

2020-11-21 11:26:54.805 ERROR com.kms.katalon.core.util.KeywordUtil    - ❌ https://malasuat.express.shoptimize.in/ has one or more broken links
2020-11-21 11:26:54.811 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/TC10 FAILED.
Reason:
com.kms.katalon.core.exception.StepFailedException: https://malasuat.express.shoptimize.in/ has one or more broken links
	at com.kms.katalon.core.util.KeywordUtil.markFailed(KeywordUtil.java:19)
	at TC10.run(TC10:40)
	at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
	at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
	at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:339)
	at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:330)
	at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:309)
	at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:301)
	at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:235)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:114)
	at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:105)
	at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
	at TempTestCase1605925398853.run(TempTestCase1605925398853.groovy:25)

2020-11-21 11:26:54.820 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/TC10

The sample code above has 2 shortages.

  1. It runs too slowly. It runs slowly because Katalon Studio (free version) logs every single steps in Test Cases, and it emits very verbose output to the LogViewer.
  2. The code is not modularised, not reusable at all. It can not process multiple web pages.

So have worked out to refactor the sample to solve these problems. I have made another post:

You can manipulate the logging level through the logging.something file in your project, even in free version.
Default for testcase execution is Debug, set it at Info level and see if it helps.
The logger you have to set it is shown into the console log :slight_smile:

@bionel

Thank you for your advice.

I am aware that we can use Include/config/log.properties to configure the execution log level.

But I think it is not enough in some cases. Imagine that Mr.X learned the above sample code and created his own “Finding broken links” project where he imitated my code. Let me suppose Mr.X is NOT aware of “log.properties”. Then he will find his project runs slowly.

I thought it would be better to refactor the code. I moved the code fragments, which are repeatedly executed, out of Test Cases script into Keywords class. Katalon Studio does not emit any execution logs for Keywords, so the Keyword codes will be executed hundreds of times without emitting verbose logs regardless how log.properties file is configured.

Thanks a lot guys @Russ_Thomas @kazurayam . You guys really helped . Really appreciate the efforts. Great community !!