Extract url from website and save to CSV or Database

Community Discussion Katalon Studio
,
#1

Hello,
i am new with Katalon, and i am trieing arround for somedays,
but i dont get any key how to extract a list of urls from a website and save them into a csv file.

With the samples i also didn´t get any idear.
Maybe someone here could help me?!

thanks

#2

Hi,

this is quite complex question, but not so difficult to implement. You need to:

  1. Identify all elements (in your case links) and store them in a list
  2. Create CSV file/establish DB connection
  3. Loop through list elements to flush them into file/DB

I can assist you with it, but I need more info from you - HTML code, which storage you prefer etc.

#3

Thanks - thats nice!

On the website there the urls listed in these way:

20 per site on nearly 50 sites.

i used imacro before. and there i could def. via position and href and extract to.
but with katalon i am realy new and “green”

i am looking for a way to store the 20 links per site in a CSV file and than get the next site and do the same

#4

the url link looks like:
https:// domain. com/profile/ and the profil ID

in the imacro i could add an * after the last /

#5

This is the code:

import org.openqa.selenium.By
import org.openqa.selenium.WebElement

import com.kms.katalon.core.webui.driver.DriverFactory
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

WebUI.openBrowser('http://yoursite.com')

String commonXpath = "//a[contains(@href, 'https://domain.com/profile/')]"
List<WebElement> urlElements = DriverFactory.getWebDriver().findElements(By.xpath(commonXpath))
List<String> urls = new ArrayList()

for(WebElement elem in urlElements) {
	urls.add(elem.getAttribute("href"))
}

File csvFile = new File("C:\\test\\csvTest.csv")
FileWriter fw = new FileWriter(csvFile, true)

for(String url in urls) {
	fw.write(url + ",")
}

fw.close()

The output is CSV file containing all hrefs which contain https://domain.com/profile/ substring

#6

Thanks - i will trie it.