Opening article's href link while iterating a List<WebElement>

Hi everyone, kinda new to Katalon Studio, had several months using Katalon Recorder tho.

I’m trying to open found articles from https://scitechdaily.com/ and I have tried different ways of looping, WebUI commands, extracting the href link as string and then open it, etc. But I’m not getting it right. Help would be much appreciated.


import org.openqa.selenium.By
import org.openqa.selenium.WebDriver
import org.openqa.selenium.WebElement

import com.kms.katalon.core.webui.driver.DriverFactory
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI


WebUI.openBrowser('')

WebUI.navigateToUrl('https://scitechdaily.com/')

//detect number of found archive articles 
WebDriver driver = DriverFactory.getWebDriver()
List<WebElement> archiveArticles = driver.findElements(By.cssSelector(
	// selector option 1 
       //"#main-content > div.archive-list.mh-section.mh-group > article > div.content-thumb.content-list-thumb > a"
       // selector option 2
	 "#main-content > div.archive-list.mh-section.mh-group > article > header > h3 > a"))
println archiveArticles.size()

// First try open each article from the found list

for(WebElement el :archiveArticles) {
	//get article's title
	println el.getText()
        //get article's URL to click
	el.getAttribute("href")
        el.click()
}

/* Second failed try
Iterator i = archiveArticles.iterator();

while(i.hasNext()) {

	WebElement element = i.next();

		//link = element.getAttribute("href").toURI().
                element.click()
		break;

}

Third try

for  (WebElement article : archiveArticles) {
      link = article.getAttribute("href")
      println link
      WebUI.openBrowser(link)
      WebUI.waitForPageLoad(3)}*/

For the first and second try error log is:

org.openqa.selenium.ElementClickInterceptedException: element click intercepted: Element <a href="https://scitechdaily.com/patient-case-strongly-suggests-link-between-pfizer-covid-19-vaccine-and-bells-palsy/" title="Patient Case Strongly Suggests Link Between Pfizer COVID-19 Vaccine and Bell’s Palsy" rel="bookmark">...</a> is not clickable at point (486, 656). Other element would receive the click: <iframe id="aswift_8" name="" sandbox="allow-forms allow-popups allow-popups-to-escape-sandbox allow-same-origin allow-scripts allow-top-navigation-by-user-activation" width="1005" height="124" frameborder="0" src="https://googleads.g.doubleclick.net/pagead/html/r20210712/r20190131/zrt_lookup.html?fsb=1#RS-1-&amp;adk=1812271801&amp;client=ca-pub-1492496027964988&amp;fa=1&amp;ifi=9&amp;uci=a!9&amp;btvi=3" marginwidth="0" marginheight="0" vspace="0" hspace="0" allowtransparency="true" scrolling="no" allowfullscreen="true" style="display: block; margin: 0px auto;" data-google-container-id="a!9" data-google-query-id="CMPEwuis8vECFaDuKAUdr5kCQg" data-load-complete="true"></iframe>

For third try error log is this:

org.openqa.selenium.NoSuchSessionException: Session ID is null. Using WebDriver after calling quit()?(...) Driver info: driver.version: CChromeDriver(...)

Hi @a.oliva! Welcome to Katalon Studio.

First and foremost, even assuming that you resolve the errors you are getting (which I will show you how to do below), this will not work as you intend. The reason is that there is an exception in the Selenium API (which Katalon is based upon) called a StaleElementReferenceException that you will inevitably encounter in this scenario. It happens when you try and use an element that is no longer valid in the DOM. There are many reasons why an element/list of elements may become invalid, but in your case, here’s the issue:

1.) You get a list of links. Fine.
2.) You iterate that list of links, clicking each one in the list.
3.) The problem is, after the first link you click, your page will reload (if even partially), and therefore your list has a bunch of links that no longer exist. The second iteration will fail, and you will get a StaleElementReferenceException.

I have a thorough discussion about the dangers of StaleElementReferenceExceptions in another topic, and I would recommend you read through it:

Ok, with that out of the way, lets go through your errors:

org.openqa.selenium.ElementClickInterceptedException: element click intercepted: Element <a href="https://scitechdaily.com/patient-case-strongly-suggests-link-between-pfizer-covid-19-vaccine-and-bells-palsy/" title="Patient Case Strongly Suggests Link Between Pfizer COVID-19 Vaccine and Bell’s Palsy" rel="bookmark">...</a> is not clickable at point (486, 656). Other element would receive the click: <iframe id="aswift_8" name="" sandbox="allow-forms allow-popups allow-popups-to-escape-sandbox allow-same-origin allow-scripts allow-top-navigation-by-user-activation" width="1005" height="124" frameborder="0" src="https://googleads.g.doubleclick.net/pagead/html/r20210712/r20190131/zrt_lookup.html?fsb=1#RS-1-&amp;adk=1812271801&amp;client=ca-pub-1492496027964988&amp;fa=1&amp;ifi=9&amp;uci=a!9&amp;btvi=3" marginwidth="0" marginheight="0" vspace="0" hspace="0" allowtransparency="true" scrolling="no" allowfullscreen="true" style="display: block; margin: 0px auto;" data-google-container-id="a!9" data-google-query-id="CMPEwuis8vECFaDuKAUdr5kCQg" data-load-complete="true"></iframe>

This error says that WebDriver tried to click on a link, but something was on top of your link that “intercepted” the click. In this case, it was an <iframe> element. My best guess at what happened here, without seeing the app you are working with, is that you clicked a link, something popped up (a modal window maybe?), and when you tried to click the second link, that something was overlapping where the link would be on the page. If this is the case, there are a couple solutions:

1.) Make sure that you close whatever popup or modal is opened before clicking the next link.
2.) Click it using JavaScript instead:

JavascriptExecutor jse = (JavascriptExecutor) driver;
jse.executeScript("arguments[0].click()", element);

For the next error:

org.openqa.selenium.NoSuchSessionException: Session ID is null. Using WebDriver after calling quit()?(...) Driver info: driver.version: CChromeDriver(...)

That happened because you called WebUI.openBrowser() when you already had a browser open:

for  (WebElement article : archiveArticles) {
      link = article.getAttribute("href")
      println link
      WebUI.openBrowser(link) <-- this will cause an error
      WebUI.waitForPageLoad(3)}*/

Instead of WebUI.openBrowser(link), you would use WebUI.navigateToUrl(link)

However, as I mentioned at the beginning, I’m fairly certain that your List<WebElement> archiveArticles will become invalid the second you click the first link.

Also, since the link is actually the title found on the element list, should I split and replace spaces with “-” and then click or open that link?
i.e

Science Article 1. find article-nth:child(1) 2. getText() //which would be "Science Article" 3. Manipulate that text = "science-article" 4. then concatenate "https://url.com/"+text?

Does that makes any sense since the staleElement behavior?

Hi Brandon, tysm for all your advice and helpful resources. Actually yes, I already encountered the StaleElementReferenceExeption .

I refactored the code but still have doubts about how to click with JavascriptExecutor and how to fix this stale error, that could be open the article element in a new window? or maybe once I’m there wait for page to load and then redirect to main Url?

Your help is really appreciated. BR

the snippet now looks like this:

`import org.openqa.selenium.By
import org.openqa.selenium.WebDriver
import org.openqa.selenium.chrome.ChromeDriver

System.setProperty(“webdriver.chrome.driver”, “C:\Users\Ana\Desktop\configuration\resources\drivers\chromedriver_win32\chromedriver.exe”);
WebDriver driver = new ChromeDriver();

driver.get(‘https://scitechdaily.com/’)

//detect number of archive articles
articlesSize = driver.findElements(By.cssSelector("#main-content > div.archive-list.mh-section.mh-group > article > div.content-thumb.content-list-thumb > a")).size()

int j;
for(i=1; i <= articlesSize; i++) {

```
if (i == 1) {
	 j = i 
} 
else if ( i == 2 ) {
	 j = j + i 
} 
else {
	 j = j + 2 
} 

String articleSelector = "#main-content > div.archive-list.mh-section.mh-group > article:nth-child(" + j + ")"
println articleSelector
/*******************************************/
driver.findElement(By.cssSelector(articleSelector)).click()
/******************************************/
//JavascriptExecutor jse = (JavascriptExecutor) driver;
//jse.executeScript("arguments[0].click()", element);
/*****************************************/
//methods to quit, re-navigate to main page
```

}`

Here’s roughly how I would approach it:

// Count the number of links using an xpath:
WebDriver driver = DriverFactory.getWebDriver()
int count = driver.findElements(By.xpath("//article//div[contains(@class, 'content-list-thumb')]//a")).size();

// Locate and click each link individually using the same xpath with an index:
for(int i = 1; i <= count; i++) {

    WebElement link = driver.findElement(By.xpath("(//article//div[contains(@class, 'content-list-thumb')]//a)[" + i + "]"));
    link.click();

    // If clicking the link causes a redirect/a modal to appear, you'll need to navigate back/close the modal...
}

When we locate each link immediately before we want to click it, we ensure that we avoid a StaleElementReferenceException. You should never re-use an element reference (or list of references) if you want to avoid that exception.

Thank you! solved now :slight_smile:
JavaScriptExecutor helped A LOT

1 Like

You’re quite welcome :slight_smile: