Why doesn't this FOR set work?

Product Forums Katalon Studio
,
1

I’ve been in this problem for two days.
The brazilian electoral authority makes this website available for consultation on the results of each ballot box in Brazil.
I want to set up a parallel counting structure to check if there is a discrepancy with photographs taken by voters on election day.
It is a work of public utility.

I did the program below, but the nested structure of FOR isn’t working as expected from a compiler.

If I delete all FORs and leave only the last one, the program automatically moves from one value to the next and writes the html file correctly.

When I place any other FOR, the loop is lost.

Code:

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import static com.kms.katalon.core.testobject.ObjectRepository.findWindowsObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.windows.keyword.WindowsBuiltinKeywords as Windows
import com.kms.katalon.core.webui.driver.DriverFactory as DriverFactory
import org.openqa.selenium.WebDriver as WebDriver
import internal.GlobalVariable as GlobalVariable
import org.openqa.selenium.By
import org.openqa.selenium.support.ui.Select
import com.kms.katalon.core.webui.driver.DriverFactory
WebUI.openBrowser(‘’)
WebUI.navigateToUrl(‘http://inter04.tse.jus.br/ords/eletse/f?p=111:1::PESQUISAR:NO:::’)

ComboTurno = ‘Page_Boletim de Urna na WEB/select_12’
ComboUF = ‘Object Repository/Page_Boletim de Urna na WEB/select_UFACALAMAPBACEDFESGOMAMGMSMTPAPBPEPI_a973a7’
ComboMunicipio = ‘Object Repository/Page_Boletim de Urna na WEB/select_Selecione um municpio localidadeACR_454ffd’
ComboZona = ‘Object Repository/Page_Boletim de Urna na WEB/select_–0008’
ComboSeção = ‘Object Repository/Page_Boletim de Urna na WEB/select_–0008000900640072007700830084008500_2e62b7’
BotãoPesquisar = ‘Object Repository/Page_Boletim de Urna na WEB/span_Pesquisar’

//Primeiros valores
WebUI.selectOptionByIndex(findTestObject(ComboTurno), 0)
WebUI.selectOptionByIndex(findTestObject(ComboUF), 1)
WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), 1)
WebUI.selectOptionByIndex(findTestObject(ComboZona), 1)
WebUI.selectOptionByIndex(findTestObject(ComboSeção), 1)

//Primeira pesquisa
WebUI.click(findTestObject(BotãoPesquisar))
WebUI.waitForPageLoad(0)

//alimenta quantidades em cada combo
int QtdTurno = WebUI.getNumberOfTotalOption(findTestObject(ComboTurno))
println(QtdTurno)
int QtdUF = WebUI.getNumberOfTotalOption(findTestObject(ComboUF))
println(QtdUF)
int QtdMunicipio = WebUI.getNumberOfTotalOption(findTestObject(ComboMunicipio))
println(QtdMunicipio)
int QtdZona = WebUI.getNumberOfTotalOption(findTestObject(ComboZona))
println(QtdZona)
int QtdSeção = WebUI.getNumberOfTotalOption(findTestObject(ComboSeção))
println(QtdSeção)

//Ninho de for para percorrer todas as opções
for (int T = 0; T < (QtdTurno-1); T++){

WebUI.selectOptionByIndex(findTestObject(ComboTurno), T);	
 	for (int U = 1; U < (QtdUF-1); U++){
 		WebUI.selectOptionByIndex(findTestObject(ComboUF), U);		
 		for (int M = 1; M < (QtdMunicipio-1); M++) {
 			WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), M);			
 			for (int Z = 1; Z < (QtdZona-1); Z++) {
 				  WebUI.selectOptionByIndex(findTestObject(ComboZona), Z);				  
 				for (int S = 1; S < (QtdSeção-1); S++) {				 	
 					WebUI.selectOptionByIndex(findTestObject(ComboSeção), S)
 					WebUI.click(findTestObject(BotãoPesquisar));
 					WebUI.waitForPageLoad(0);

    				//Gravando o arquivo a partir do valor selecionado nas combos
 					Select selectTurno = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_TURNO']")))
 					Turno = selectTurno.getFirstSelectedOption().getText()
 					Select selectUF = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_UF']")))
 					UF = selectUF.getFirstSelectedOption().getText()
 					Select selectMUN = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_MUN']")))
 					Municipio = selectMUN.getFirstSelectedOption().getText()
 					Select selectZONA = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_ZONA']")))
 					Zona = selectZONA.getFirstSelectedOption().getText()
 					Select selectSECAO = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_SECAO']")))
 					Seção = selectSECAO.getFirstSelectedOption().getText()
 	
 					String NomeDoArquivo = ('E:/TSE/2018/TSE2018_'+
 						'Turno-'+Turno+
 						'_UF-'+UF+
 						'_Municipio-'+Municipio+
 						'_Zona-'+Zona+
 						'_Seção-'+Seção+
 						'.html')
 					WebDriver driver = DriverFactory.getWebDriver()
 					def src = driver.getPageSource()
 					def Arquivo = new File(NomeDoArquivo)
 					Arquivo.write(src)
 				 } 				
 			 }   
 		}
 	}
 }			 
 //WebUI.closeBrowser()

Console

2020-11-25 14:00:49.495 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2020-11-25 14:00:49.511 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/TSE 2018
2020-11-25 14:00:54.171 DEBUG testcase.TSE 2018 - 1: openBrowser(“”)
2020-11-25 14:00:55.391 INFO c.k.k.core.webui.driver.DriverFactory - Starting ‘Firefox’ driver
nov 25, 2020 2:00:55 PM org.openqa.selenium.remote.DesiredCapabilities firefox
INFORMAÇÕES: Using new FirefoxOptions() is preferred to DesiredCapabilities.firefox()
2020-11-25 14:00:55.798 INFO c.k.k.core.webui.driver.DriverFactory - Action delay is set to 0 milliseconds
1606320060493 mozrunner::runner INFO Running command: “C:\Program Files\Mozilla Firefox\firefox.exe” “-marionette” “-foreground” “-no-remote” “-profile” “C:\Users\Laudelino\AppData\Local\Temp\rust_mozprofile.aKcwnv9g6VjI”
Can’t find symbol ‘eglSwapBuffersWithDamageEXT’.
Can’t find symbol ‘eglSetDamageRegionKHR’.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can’t find profile directory.
1606320067536 Marionette INFO Listening on port 57985
1606320067808 Marionette WARN TLS certificate errors will be ignored for this session
1606320067809 Marionette INFO Proxy settings initialised: {“proxyType”:“direct”}
nov 25, 2020 2:01:07 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFORMAÇÕES: Detected dialect: W3C
2020-11-25 14:01:08.179 INFO c.k.k.core.webui.driver.DriverFactory - sessionId = a319f07b-c32c-4c57-bc6d-7698ee55b689
2020-11-25 14:01:08.217 INFO c.k.k.core.webui.driver.DriverFactory - browser = Firefox 83.0
2020-11-25 14:01:08.217 INFO c.k.k.core.webui.driver.DriverFactory - platform = Windows 10
2020-11-25 14:01:08.218 INFO c.k.k.core.webui.driver.DriverFactory - seleniumVersion = 3.141.59
2020-11-25 14:01:08.221 INFO c.k.k.core.webui.driver.DriverFactory - proxyInformation = ProxyInformation { proxyOption=NO_PROXY, proxyServerType=HTTP, username=, password=********, proxyServerAddress=, proxyServerPort=0, executionList=“”, isApplyToDesiredCapabilities=true }
2020-11-25 14:01:08.260 DEBUG testcase.TSE 2018 - 2: navigateToUrl(“http://inter04.tse.jus.br/ords/eletse/f?p=111:1::PESQUISAR:NO:::”)
2020-11-25 14:01:09.207 DEBUG testcase.TSE 2018 - 3: ComboTurno = “Page_Boletim de Urna na WEB/select_12”
2020-11-25 14:01:09.209 DEBUG testcase.TSE 2018 - 4: ComboUF = “Object Repository/Page_Boletim de Urna na WEB/select_UFACALAMAPBACEDFESGOMAMGMSMTPAPBPEPI_a973a7”
2020-11-25 14:01:09.210 DEBUG testcase.TSE 2018 - 5: ComboMunicipio = “Object Repository/Page_Boletim de Urna na WEB/select_Selecione um municpio localidadeACR_454ffd”
2020-11-25 14:01:09.212 DEBUG testcase.TSE 2018 - 6: ComboZona = “Object Repository/Page_Boletim de Urna na WEB/select_–0008”
2020-11-25 14:01:09.213 DEBUG testcase.TSE 2018 - 7: ComboSeção = “Object Repository/Page_Boletim de Urna na WEB/select_–0008000900640072007700830084008500_2e62b7”
2020-11-25 14:01:09.214 DEBUG testcase.TSE 2018 - 8: BotãoPesquisar = “Object Repository/Page_Boletim de Urna na WEB/span_Pesquisar”
2020-11-25 14:01:09.215 DEBUG testcase.TSE 2018 - 9: selectOptionByIndex(findTestObject(ComboTurno), 0)
2020-11-25 14:01:10.838 DEBUG testcase.TSE 2018 - 10: selectOptionByIndex(findTestObject(ComboUF), 1)
2020-11-25 14:01:12.355 DEBUG testcase.TSE 2018 - 11: selectOptionByIndex(findTestObject(ComboMunicipio), 1)
2020-11-25 14:01:13.814 DEBUG testcase.TSE 2018 - 12: selectOptionByIndex(findTestObject(ComboZona), 1)
2020-11-25 14:01:15.302 DEBUG testcase.TSE 2018 - 13: selectOptionByIndex(findTestObject(ComboSeção), 1)
2020-11-25 14:01:17.178 DEBUG testcase.TSE 2018 - 14: click(findTestObject(BotãoPesquisar))
2020-11-25 14:01:18.271 DEBUG testcase.TSE 2018 - 15: waitForPageLoad(0)
2020-11-25 14:01:18.497 WARN c.kms.katalon.core.helper.KeywordHelper - Timeout ‘0’ is invalid. Using default page load timeout: ‘30’
2020-11-25 14:01:18.764 DEBUG testcase.TSE 2018 - 16: QtdTurno = getNumberOfTotalOption(findTestObject(ComboTurno))
2020-11-25 14:01:19.666 DEBUG testcase.TSE 2018 - 17: println(QtdTurno)
> 2
2020-11-25 14:01:19.674 DEBUG testcase.TSE 2018 - 18: QtdUF = getNumberOfTotalOption(findTestObject(ComboUF))
2020-11-25 14:01:20.569 DEBUG testcase.TSE 2018 - 19: println(QtdUF)
> 29
2020-11-25 14:01:20.570 DEBUG testcase.TSE 2018 - 20: QtdMunicipio = getNumberOfTotalOption(findTestObject(ComboMunicipio))
2020-11-25 14:01:21.456 DEBUG testcase.TSE 2018 - 21: println(QtdMunicipio)
> 23
2020-11-25 14:01:21.457 DEBUG testcase.TSE 2018 - 22: QtdZona = getNumberOfTotalOption(findTestObject(ComboZona))
2020-11-25 14:01:22.332 DEBUG testcase.TSE 2018 - 23: println(QtdZona)
> 2
2020-11-25 14:01:22.333 DEBUG testcase.TSE 2018 - 24: QtdSeção = getNumberOfTotalOption(findTestObject(ComboSeção))
2020-11-25 14:01:23.220 DEBUG testcase.TSE 2018 - 25: println(QtdSeção)
> 42
2020-11-25 14:01:23.221 DEBUG testcase.TSE 2018 - 26: for ([T = 0, T < QtdTurno - 1, (T++)])
2020-11-25 14:01:23.222 DEBUG testcase.TSE 2018 - 1: selectOptionByIndex(findTestObject(ComboTurno), T)
2020-11-25 14:01:24.404 DEBUG testcase.TSE 2018 - 2: for ([U = 1, U < QtdUF - 1, (U++)])
2020-11-25 14:01:24.405 DEBUG testcase.TSE 2018 - 1: selectOptionByIndex(findTestObject(ComboUF), U)
2020-11-25 14:01:25.555 DEBUG testcase.TSE 2018 - 2: for ([M = 1, M < QtdMunicipio - 1, (M++)])
2020-11-25 14:01:25.556 DEBUG testcase.TSE 2018 - 1: selectOptionByIndex(findTestObject(ComboMunicipio), M)
2020-11-25 14:01:26.761 DEBUG testcase.TSE 2018 - 2: for ([Z = 1, Z < QtdZona - 1, (Z++)])
2020-11-25 14:01:26.762 DEBUG testcase.TSE 2018 - 1: selectOptionByIndex(findTestObject(ComboMunicipio), M)
2020-11-25 14:01:28.187 DEBUG testcase.TSE 2018 - 2: for ([Z = 1, Z < QtdZona - 1, (Z++)])
2020-11-25 14:01:28.188 DEBUG testcase.TSE 2018 - 1: selectOptionByIndex(findTestObject(ComboMunicipio), M)
2020-11-25 14:01:29.604 DEBUG testcase.TSE 2018 - 2: for ([Z = 1, Z < QtdZona - 1, (Z++)])
2020-11-25 14:01:29.605 DEBUG testcase.TSE 2018 - 1: selectOptionByIndex(findTestObject(ComboMunicipio), M)
2020-11-25 14:01:31.478 DEBUG testcase.TSE 2018 - 2: for ([Z = 1, Z < QtdZona - 1, (Z++)])
2020-11-25 14:01:31.479 DEBUG testcase.TSE 2018 - 1: selectOptionByIndex(findTestObject(ComboMunicipio), M)

2

Is it possible for you to make a complete ZIP file and share it here so that we just unzip, open and run it on our side?

You shared

  1. the URL of your Application Under Test http://inter04.tse.jus.br/ords/eletse/f?p=111:1::PESQUISAR:NO:::
  2. your test case script

but these are not enough. You have created a few Test Objects

  • Page_Boletim de Urna na WEB/select_12
  • …

and so on. In order to reproduce your problem on our side, we need these Test Objects to be shared as well.

1 Like
3

Thank you

TSE 2018.zip (42.4 KB)

4

Having played around with the page, I notice a few things…

  1. Selecting various options causes the page to repopulate selects - but I don’t see any evidence you are waiting for the repopulation to take effect.

  2. Some of the selects have no data (or almost none).

First I would deal with #1. That’s probably a big part of the problem.

I notice also that the log shows issues between the M and Z loops - it gets stuck and never reaches the S loop. More evidence that the repopulation is likely screwing up the code.

1 Like
5

Just I played on your code, changed a bit

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import static com.kms.katalon.core.testobject.ObjectRepository.findWindowsObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.windows.keyword.WindowsBuiltinKeywords as Windows
import com.kms.katalon.core.webui.driver.DriverFactory as DriverFactory
import org.openqa.selenium.WebDriver as WebDriver
import internal.GlobalVariable as GlobalVariable
import org.openqa.selenium.By
import org.openqa.selenium.support.ui.Select
import com.kms.katalon.core.webui.driver.DriverFactory
WebUI.openBrowser('')
WebUI.setViewPortSize(1024,400)
WebUI.navigateToUrl('http://inter04.tse.jus.br/ords/eletse/f?p=111:1::PESQUISAR:NO:::')
ComboTurno = 'Page_Boletim de Urna na WEB/select_12'
ComboUF = 'Object Repository/Page_Boletim de Urna na WEB/select_UFACALAMAPBACEDFESGOMAMGMSMTPAPBPEPI_a973a7'
ComboMunicipio = 'Object Repository/Page_Boletim de Urna na WEB/select_Selecione um municpio  localidadeACR_454ffd'
ComboZona = 'Object Repository/Page_Boletim de Urna na WEB/select_--0008'
ComboSeção = 'Object Repository/Page_Boletim de Urna na WEB/select_--0008000900640072007700830084008500_2e62b7'
BotãoPesquisar = 'Object Repository/Page_Boletim de Urna na WEB/span_Pesquisar'
//Primeiros valores
WebUI.selectOptionByIndex(findTestObject(ComboTurno), 0)
WebUI.selectOptionByIndex(findTestObject(ComboUF), 1)
WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), 1)
WebUI.selectOptionByIndex(findTestObject(ComboZona), 1)
WebUI.selectOptionByIndex(findTestObject(ComboSeção), 1)
//Primeira pesquisa
WebUI.click(findTestObject(BotãoPesquisar))
WebUI.waitForPageLoad(10)
//alimenta quantidades em cada combo

//Ninho de for para percorrer todas as opções
int QtdTurno =     WebUI.getNumberOfTotalOption(findTestObject(ComboTurno))
println(QtdTurno)
for (int T = 0; T <= QtdTurno; T++){
	WebUI.selectOptionByIndex(findTestObject(ComboTurno), T);	
	WebUI.verifyElementPresent(findTestObject(ComboUF), 3)
	int QtdUF = WebUI.getNumberOfTotalOption(findTestObject(ComboUF))
	WebUI.comment("QtdUF=${QtdUF}")
	for (int U = 1; U <= QtdUF; U++){
		WebUI.selectOptionByIndex(findTestObject(ComboUF), U);	
		WebUI.verifyElementPresent(findTestObject(ComboMunicipio), 3)
		int QtdMunicipio = WebUI.getNumberOfTotalOption(findTestObject(ComboMunicipio))
		WebUI.comment("QtdMunicipio=${QtdMunicipio}")
		for (int M = 1; M <= QtdMunicipio; M++) {
			WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), M);
			WebUI.verifyElementPresent(findTestObject(ComboZona), 3)
			int QtdZona = WebUI.getNumberOfTotalOption(findTestObject(ComboZona))
			WebUI.comment("QtdZona=${QtdZona}")
			for (int Z = 1; Z <= QtdZona; Z++) {
				WebUI.selectOptionByIndex(findTestObject(ComboZona), Z);	
				WebUI.verifyElementPresent(findTestObject(ComboSeção), 3)
				int QtdSeção = WebUI.getNumberOfTotalOption(findTestObject(ComboSeção))
				WebUI.comment("QtdSeção=${QtdSeção}")
				for (int S = 1; S <= QtdSeção; S++) {				 	
					WebUI.selectOptionByIndex(findTestObject(ComboSeção), S)
					WebUI.verifyElementPresent(findTestObject(BotãoPesquisar), 3)
					WebUI.click(findTestObject(BotãoPesquisar));
					WebUI.waitForPageLoad(0);
   					//Gravando o arquivo a partir do valor selecionado nas combos
					Select selectTurno = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_TURNO']")))
					Turno = selectTurno.getFirstSelectedOption().getText()
					Select selectUF = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_UF']")))
					UF = selectUF.getFirstSelectedOption().getText()
					Select selectMUN = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_MUN']")))
					Municipio = selectMUN.getFirstSelectedOption().getText()
					Select selectZONA = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_ZONA']")))
					Zona = selectZONA.getFirstSelectedOption().getText()
					Select selectSECAO = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_SECAO']")))
					Seção = selectSECAO.getFirstSelectedOption().getText()
	
					String NomeDoArquivo = ('/Users/username/tmp/TSE2018_'+
						'Turno-'+Turno+
						'_UF-'+UF+
						'_Municipio-'+Municipio+
						'_Zona-'+Zona+
						'_Seção-'+Seção+
						'.html')
					WebDriver driver = DriverFactory.getWebDriver()
					def src = driver.getPageSource()
					def Arquivo = new File(NomeDoArquivo)
					Arquivo.write(src)
				 } 				
			 }   
		}
	}
}			 
//WebUI.closeBrowser()

The for loops look doing something meaningful.

@xupacabra

Please compare the codes and find what I changed. Possibly most interesting change is:

Your original: for (int Z = 1; Z < (QtdZona-1); Z++) {
… when QtdZona is 2, then this loop will exit without doing anything

My change: for (int Z = 1; Z <= QtdZona; Z++) {
… when QtdZona is 2, then this loop will do something twice

1 Like
6

Thanks.
3:00am now here. I will test soon, in the morning.
Thanks

7

Excelente point.
the count in combos, change all time…
I will test.

8

Final solution

for (int T = 0; T <= (QtdTurno); T++){
WebUI.selectOptionByIndex(findTestObject(ComboTurno), T);

for (int U = 1; U <= (QtdUF); U++){
	WebUI.selectOptionByIndex(findTestObject(ComboUF), U);		
	QtdMunicipio = WebUI.getNumberOfTotalOption(findTestObject(ComboMunicipio))-1

	for (int M = 1; M <= (QtdMunicipio); M++) {
		WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), M);						
		QtdZona =      WebUI.getNumberOfTotalOption(findTestObject(ComboZona))-1

		for (int Z = 1; Z <= (QtdZona); Z++) {
			  WebUI.selectOptionByIndex(findTestObject(ComboZona), Z);				  			  
			  QtdSeção =     WebUI.getNumberOfTotalOption(findTestObject(ComboSeção))-1

			 for (int S = 1; S <= (QtdSeção); S++) {				 	
				WebUI.selectOptionByIndex(findTestObject(ComboSeção), S)
				WebUI.click(findTestObject(BotãoPesquisar));
				WebUI.waitForPageLoad(0);
9

Glad you’ve solved the problem.

Is it OK to close this topic?

10

Yes, solve. Thx.
I’m work now on another problem…

At the end of the work, according to the combinations, I have 454 thousand files and that is correct. We have 454,000 ballot boxes in Brazil.
Katalon is generating 7 files per minute and that means that I will need 46 days of processing.

My final code (Working) :upside_down_face: :

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import static com.kms.katalon.core.testobject.ObjectRepository.findWindowsObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.windows.keyword.WindowsBuiltinKeywords as Windows
import com.kms.katalon.core.webui.driver.DriverFactory as DriverFactory
import org.openqa.selenium.WebDriver as WebDriver
import internal.GlobalVariable as GlobalVariable
import org.openqa.selenium.By
import org.openqa.selenium.support.ui.Select
import com.kms.katalon.core.webui.driver.DriverFactory
WebUI.openBrowser('')
WebUI.setViewPortSize(437,1144)
WebUI.navigateToUrl('http://inter04.tse.jus.br/ords/eletse/f?p=111:1::PESQUISAR:NO:::')
 
ComboTurno =     'Object Repository/Page_Boletim de Urna na WEB/select_12'
ComboUF =        'Object Repository/Page_Boletim de Urna na WEB/select_UFACALAMAPBACEDFESGOMAMGMSMTPAPBPEPI_a973a7'
ComboMunicipio = 'Object Repository/Page_Boletim de Urna na WEB/select_Selecione um municpio  localidadeACR_454ffd'
ComboZona =      'Object Repository/Page_Boletim de Urna na WEB/select_--0008'
ComboSeção =     'Object Repository/Page_Boletim de Urna na WEB/select_--0008000900640072007700830084008500_2e62b7'
BotãoPesquisar = 'Object Repository/Page_Boletim de Urna na WEB/span_Pesquisar'

//Primeiros valores
WebUI.selectOptionByIndex(findTestObject(ComboTurno), 0)
WebUI.selectOptionByIndex(findTestObject(ComboUF), 1)
WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), 1)
WebUI.selectOptionByIndex(findTestObject(ComboZona), 1)
WebUI.selectOptionByIndex(findTestObject(ComboSeção), 1)

//Primeira pesquisa
WebUI.click(findTestObject(BotãoPesquisar))
WebUI.waitForPageLoad(10)

//alimenta quantidades em cada combo
int QtdTurno =     WebUI.getNumberOfTotalOption(findTestObject(ComboTurno))-1
println(QtdTurno)
int QtdUF =        WebUI.getNumberOfTotalOption(findTestObject(ComboUF))-1
println(QtdUF-1)
int QtdMunicipio = WebUI.getNumberOfTotalOption(findTestObject(ComboMunicipio))-1
println(QtdMunicipio-1)
int QtdZona =      WebUI.getNumberOfTotalOption(findTestObject(ComboZona))-1
println(QtdZona-1)
int QtdSeção =     WebUI.getNumberOfTotalOption(findTestObject(ComboSeção))-1
println(QtdSeção-1)
String Diretorio = ('E:/TSE/2018/TSE2018')
//Ninho de for para percorrer todas as opções
for (int T = 0; T <= (QtdTurno); T++){
	WebUI.selectOptionByIndex(findTestObject(ComboTurno), T);	
	QtdUF = WebUI.getNumberOfTotalOption(findTestObject(ComboUF))-1
	for (int U = 1; U <= (QtdUF); U++){
		WebUI.selectOptionByIndex(findTestObject(ComboUF), U);		
		QtdMunicipio = WebUI.getNumberOfTotalOption(findTestObject(ComboMunicipio))-1
		for (int M = 1; M <= (QtdMunicipio); M++) {
			WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), M);						
			QtdZona =      WebUI.getNumberOfTotalOption(findTestObject(ComboZona))-1
			for (int Z = 1; Z <= (QtdZona); Z++) {
				  WebUI.selectOptionByIndex(findTestObject(ComboZona), Z);				  			  
				  QtdSeção =     WebUI.getNumberOfTotalOption(findTestObject(ComboSeção))-1
				 for (int S = 1; S <= (QtdSeção); S++) {				 	
					WebUI.selectOptionByIndex(findTestObject(ComboSeção), S)
					//WebUI.click(findTestObject(BotãoPesquisar));
					//WebUI.waitForPageLoad(5);
   					//Gravando o arquivo a partir do valor selecionado nas combos
					Select selectTurno = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_TURNO']")))
					Turno = selectTurno.getFirstSelectedOption().getText()
					Select selectUF = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_UF']")))
					UF = selectUF.getFirstSelectedOption().getText()
					Select selectMUN = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_MUN']")))
					Municipio = selectMUN.getFirstSelectedOption().getText()
					Select selectZONA = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_ZONA']")))
					Zona = selectZONA.getFirstSelectedOption().getText()
					Select selectSECAO = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_SECAO']")))
					Seção = selectSECAO.getFirstSelectedOption().getText()
						
					String NomeDoArquivo = ('_'+
						'Turno-'+Turno+
						'_UF-'+UF+
						'_Municipio-'+Municipio+
						'_Zona-'+Zona+
						'_Seção-'+Seção+
						'.html')
					
					File file = new File(Diretorio+NomeDoArquivo);
					exists = file.exists();
					if (!file.exists())					
					{
						WebUI.click(findTestObject(BotãoPesquisar));
						WebUI.waitForPageLoad(5);
						
						WebDriver driver = DriverFactory.getWebDriver()
						def src = driver.getPageSource()
						def Arquivo = new File(Diretorio+NomeDoArquivo)
						Arquivo.write(src)
					}
				 } 				
			 }   
		}
	}
}			 
//WebUI.closeBrowser()
11

I assume that you want to find out how to make your test run faster.

I found a few points to make your test run a bit faster so that it can generate 9 or 10 files per minutes. I am afraid, the improvement is not drastic enough. So I would not mention it here.

I checked your AUT and your code. I found that it is difficult to make your test generates drastically larger number of files per minute, because your AUT

  • http://inter04.tse.jus.br/ords/eletse/f?p=111:1::PESQUISAR:NO:::

takes approximately 6 seconds to respond to each click on the Pesquisar button.

Unless you have a super power to force the AUT to respond quicker, your test code can not scrape the target pages faster than 6 seconds per view. If you are to process 454000 items by a single process sequentially, it would take more than 454000*6 seconds.

12

If you introduce parallelism, you would be able to make your work done within a day.

If I were you, I will do the following changes:

  1. In the page you have 5 parameters: ‘ComboTurno’, ‘ComboUF’, ‘ComboMunicipio’, ‘ComboZona’, ‘ComboSeção’. I will write a script in Katalon Studio that retrieves the values of these parameters and write all value combinations into a file, namely “AllComboValues”.
  2. I will write another script which reads the “AllComboValues” file and split it into smaller chunks, say 50 files.
  3. I will configure a cloud resource on AWS Device Farm where I create 50 EC2 instances. I will configure Selenium WebDriver for Java. I will execute page-scraping jobs on the cloud environment providing 50 parameter files. The code will visit pages, take page sources, save them into S3 storage.
  4. If you have 50 processes to scrape 454000 pages, and a process can emit 7 files a minute, then the work will be done in 22 hours.
  5. The work might be done even quicker, as the cloud services use very fast backbone networks to the Internet.
  6. As soon as the work done, I will drop the EC2 instances to save money.
  7. Katalon Runtime Engine would not be suitable; too costly to run them on 50 machines for just a few days.

If you have 454000 items to process, a single desktop PC is not powerful enough. I am sure you need some scalable computing resources such as AWS, GCP, Azure. Of course you need enough budget. :grinning:

You can try a smaller scale of parallelism by Test Suite Collection in parallel. With this feature you can try running up to 5 Test Suites in parallel on a PC. Provided that you do the same design change as I mention above, create 2 parameter files to run 2 Test Suites in parallel on your desktop PC. This would reduce the processing hours to 1/2 hopefully.

Be careful, your AUT might go down if your test makes spike traffic.

13

If you get immediate access to the Database behind the AUT rather than scraping the Web pages, it would be much better; far quicker, easier and much more robust.

14

I like this idea about one file with search options.
Great idea!

15

I think that Katalon is not the ideal tool for me to do this … after two weeks of running the script, the performance proved to be very low.
It starts fast and then reduces until it starts to give an error due to timeout.
Of the 545 thousand files, I can’t reach more than 1800. When everything stops.

My katalon.ini

-startup
plugins/org.eclipse.equinox.launcher_1.3.201.v20161025-1711.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.401.v20161122-1740
-data
@noDefault
-vmargs
-XX:+UseG1GC
-XX:+UseStringDeduplication
-Xms256m
-Dfile.encoding=utf-8
-Xmx2048m

It was a good experience.
I wanted to know the tool.
I thank everyone who helped me.

16

@duyluong Sounds like something you might want to know.

17

Seriously? What did you expected? Magic?
Processing large amount of data require strong knowledge about programming (threading, batch processing and so on)
I doubt any tool in this world targeted on functional testing will suit your needs wrt propossed exercise.
You may have to start learning real programming.

18

@xupacabra

I have ever experienced a similar situation in my VisualTestingInKatalonStudio project. I saw, my project in Katalon Studio ran too slow, consumed too large memory, eventually stopped while processing large number of URLs and writing hundreds of files.

I changed my codes and, yes, I could solve it. My code now runs far faster than before, requires fair size of memory, never stops. The tool should not be blamed. God dwells in the details of my code. :wink:

You disclosed your code above. There I find 3 essential points to be improved. I would tell you what.

… still authoring

2 Likes
19

let’s do a simple math (simplified):
you have 454000 items to grab. supposed you can grab a single item in 1 sec (in fact it takes longer with your AUT but we ignore this for the moment) you will need 454000 / 3600 >> 126 hours only to fetch the data
If you want to reduce this time to 1 hr you need a machine able to run 126 threads.
Now … have some fun, run your project to fetch, let’s say … 10-20 items a take a note about RAM consumption before and during the project run. Make the diff and multiply it with 126 < this is the amount of memory you will need to be available at a minimum.
And here i just ignore the CPU cycles needed for it …

And for the moment i ignore also the storage size needed. Supposed each file is just 1K in size, you need ~443M available. Not huge but you can do the math with the real size.

And the above is just for fetching the data … If you need also to post-process the data things looks even funnier.

So you have to totally reconsider your approach, IMHO, by using pure programming with parallel threading, async calls to fetch the data and store it in a certain DB so it can be easy analyzed further.

And definitely you need a better method to fetch the data (direct connection to DB or at least, straight http requests)

Here Katalon himself indeed, cannot help you much. It is designed mostly for sequential code-run, not for magic stuff.
Although it does have support for parallel’ execution, that feature is note pure parallel threading but some sort of switching fast between the threads.

Choose any language you like here, you can do it in Java, Python, C++ … but you have to read about the techniques I mentioned a bit.

If you really want you can use Katalon as IDE, nothing wrong with it as long as you write your code properly.

20

It was a learning experience.
I have been programming since 1986 and finished the job with 1 day of work and processing in Ruby. I liked the tool and its concept.

1 Like