Why doesn't this FOR set work?

Thanks.
3:00am now here. I will test soon, in the morning.
Thanks

Excelente point.
the count in combos, change all time…
I will test.

Final solution

for (int T = 0; T <= (QtdTurno); T++){
WebUI.selectOptionByIndex(findTestObject(ComboTurno), T);

for (int U = 1; U <= (QtdUF); U++){
	WebUI.selectOptionByIndex(findTestObject(ComboUF), U);		
	QtdMunicipio = WebUI.getNumberOfTotalOption(findTestObject(ComboMunicipio))-1

	for (int M = 1; M <= (QtdMunicipio); M++) {
		WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), M);						
		QtdZona =      WebUI.getNumberOfTotalOption(findTestObject(ComboZona))-1

		for (int Z = 1; Z <= (QtdZona); Z++) {
			  WebUI.selectOptionByIndex(findTestObject(ComboZona), Z);				  			  
			  QtdSeção =     WebUI.getNumberOfTotalOption(findTestObject(ComboSeção))-1

			 for (int S = 1; S <= (QtdSeção); S++) {				 	
				WebUI.selectOptionByIndex(findTestObject(ComboSeção), S)
				WebUI.click(findTestObject(BotãoPesquisar));
				WebUI.waitForPageLoad(0);

Glad you’ve solved the problem.

Is it OK to close this topic?

Yes, solve. Thx.
I’m work now on another problem…

At the end of the work, according to the combinations, I have 454 thousand files and that is correct. We have 454,000 ballot boxes in Brazil.
Katalon is generating 7 files per minute and that means that I will need 46 days of processing.

My final code (Working) :upside_down_face: :

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import static com.kms.katalon.core.testobject.ObjectRepository.findWindowsObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import com.kms.katalon.core.windows.keyword.WindowsBuiltinKeywords as Windows
import com.kms.katalon.core.webui.driver.DriverFactory as DriverFactory
import org.openqa.selenium.WebDriver as WebDriver
import internal.GlobalVariable as GlobalVariable
import org.openqa.selenium.By
import org.openqa.selenium.support.ui.Select
import com.kms.katalon.core.webui.driver.DriverFactory
WebUI.openBrowser('')
WebUI.setViewPortSize(437,1144)
WebUI.navigateToUrl('http://inter04.tse.jus.br/ords/eletse/f?p=111:1::PESQUISAR:NO:::')
 
ComboTurno =     'Object Repository/Page_Boletim de Urna na WEB/select_12'
ComboUF =        'Object Repository/Page_Boletim de Urna na WEB/select_UFACALAMAPBACEDFESGOMAMGMSMTPAPBPEPI_a973a7'
ComboMunicipio = 'Object Repository/Page_Boletim de Urna na WEB/select_Selecione um municpio  localidadeACR_454ffd'
ComboZona =      'Object Repository/Page_Boletim de Urna na WEB/select_--0008'
ComboSeção =     'Object Repository/Page_Boletim de Urna na WEB/select_--0008000900640072007700830084008500_2e62b7'
BotãoPesquisar = 'Object Repository/Page_Boletim de Urna na WEB/span_Pesquisar'

//Primeiros valores
WebUI.selectOptionByIndex(findTestObject(ComboTurno), 0)
WebUI.selectOptionByIndex(findTestObject(ComboUF), 1)
WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), 1)
WebUI.selectOptionByIndex(findTestObject(ComboZona), 1)
WebUI.selectOptionByIndex(findTestObject(ComboSeção), 1)

//Primeira pesquisa
WebUI.click(findTestObject(BotãoPesquisar))
WebUI.waitForPageLoad(10)

//alimenta quantidades em cada combo
int QtdTurno =     WebUI.getNumberOfTotalOption(findTestObject(ComboTurno))-1
println(QtdTurno)
int QtdUF =        WebUI.getNumberOfTotalOption(findTestObject(ComboUF))-1
println(QtdUF-1)
int QtdMunicipio = WebUI.getNumberOfTotalOption(findTestObject(ComboMunicipio))-1
println(QtdMunicipio-1)
int QtdZona =      WebUI.getNumberOfTotalOption(findTestObject(ComboZona))-1
println(QtdZona-1)
int QtdSeção =     WebUI.getNumberOfTotalOption(findTestObject(ComboSeção))-1
println(QtdSeção-1)
String Diretorio = ('E:/TSE/2018/TSE2018')
//Ninho de for para percorrer todas as opções
for (int T = 0; T <= (QtdTurno); T++){
	WebUI.selectOptionByIndex(findTestObject(ComboTurno), T);	
	QtdUF = WebUI.getNumberOfTotalOption(findTestObject(ComboUF))-1
	for (int U = 1; U <= (QtdUF); U++){
		WebUI.selectOptionByIndex(findTestObject(ComboUF), U);		
		QtdMunicipio = WebUI.getNumberOfTotalOption(findTestObject(ComboMunicipio))-1
		for (int M = 1; M <= (QtdMunicipio); M++) {
			WebUI.selectOptionByIndex(findTestObject(ComboMunicipio), M);						
			QtdZona =      WebUI.getNumberOfTotalOption(findTestObject(ComboZona))-1
			for (int Z = 1; Z <= (QtdZona); Z++) {
				  WebUI.selectOptionByIndex(findTestObject(ComboZona), Z);				  			  
				  QtdSeção =     WebUI.getNumberOfTotalOption(findTestObject(ComboSeção))-1
				 for (int S = 1; S <= (QtdSeção); S++) {				 	
					WebUI.selectOptionByIndex(findTestObject(ComboSeção), S)
					//WebUI.click(findTestObject(BotãoPesquisar));
					//WebUI.waitForPageLoad(5);
   					//Gravando o arquivo a partir do valor selecionado nas combos
					Select selectTurno = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_TURNO']")))
					Turno = selectTurno.getFirstSelectedOption().getText()
					Select selectUF = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_UF']")))
					UF = selectUF.getFirstSelectedOption().getText()
					Select selectMUN = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_MUN']")))
					Municipio = selectMUN.getFirstSelectedOption().getText()
					Select selectZONA = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_ZONA']")))
					Zona = selectZONA.getFirstSelectedOption().getText()
					Select selectSECAO = new Select(DriverFactory.getWebDriver().findElement(By.xpath("//select[@id='P1_SECAO']")))
					Seção = selectSECAO.getFirstSelectedOption().getText()
						
					String NomeDoArquivo = ('_'+
						'Turno-'+Turno+
						'_UF-'+UF+
						'_Municipio-'+Municipio+
						'_Zona-'+Zona+
						'_Seção-'+Seção+
						'.html')
					
					File file = new File(Diretorio+NomeDoArquivo);
					exists = file.exists();
					if (!file.exists())					
					{
						WebUI.click(findTestObject(BotãoPesquisar));
						WebUI.waitForPageLoad(5);
						
						WebDriver driver = DriverFactory.getWebDriver()
						def src = driver.getPageSource()
						def Arquivo = new File(Diretorio+NomeDoArquivo)
						Arquivo.write(src)
					}
				 } 				
			 }   
		}
	}
}			 
//WebUI.closeBrowser()

I assume that you want to find out how to make your test run faster.

I found a few points to make your test run a bit faster so that it can generate 9 or 10 files per minutes. I am afraid, the improvement is not drastic enough. So I would not mention it here.


I checked your AUT and your code. I found that it is difficult to make your test generates drastically larger number of files per minute, because your AUT

  • http://inter04.tse.jus.br/ords/eletse/f?p=111:1::PESQUISAR:NO:::

takes approximately 6 seconds to respond to each click on the Pesquisar button.

Unless you have a super power to force the AUT to respond quicker, your test code can not scrape the target pages faster than 6 seconds per view. If you are to process 454000 items by a single process sequentially, it would take more than 454000*6 seconds.

If you introduce parallelism, you would be able to make your work done within a day.

If I were you, I will do the following changes:

  1. In the page you have 5 parameters: ‘ComboTurno’, ‘ComboUF’, ‘ComboMunicipio’, ‘ComboZona’, ‘ComboSeção’. I will write a script in Katalon Studio that retrieves the values of these parameters and write all value combinations into a file, namely “AllComboValues”.
  2. I will write another script which reads the “AllComboValues” file and split it into smaller chunks, say 50 files.
  3. I will configure a cloud resource on AWS Device Farm where I create 50 EC2 instances. I will configure Selenium WebDriver for Java. I will execute page-scraping jobs on the cloud environment providing 50 parameter files. The code will visit pages, take page sources, save them into S3 storage.
  4. If you have 50 processes to scrape 454000 pages, and a process can emit 7 files a minute, then the work will be done in 22 hours.
  5. The work might be done even quicker, as the cloud services use very fast backbone networks to the Internet.
  6. As soon as the work done, I will drop the EC2 instances to save money.
  7. Katalon Runtime Engine would not be suitable; too costly to run them on 50 machines for just a few days.

If you have 454000 items to process, a single desktop PC is not powerful enough. I am sure you need some scalable computing resources such as AWS, GCP, Azure. Of course you need enough budget. :grinning:


You can try a smaller scale of parallelism by Test Suite Collection in parallel. With this feature you can try running up to 5 Test Suites in parallel on a PC. Provided that you do the same design change as I mention above, create 2 parameter files to run 2 Test Suites in parallel on your desktop PC. This would reduce the processing hours to 1/2 hopefully.


Be careful, your AUT might go down if your test makes spike traffic.

If you get immediate access to the Database behind the AUT rather than scraping the Web pages, it would be much better; far quicker, easier and much more robust.

I like this idea about one file with search options.
Great idea!

I think that Katalon is not the ideal tool for me to do this … after two weeks of running the script, the performance proved to be very low.
It starts fast and then reduces until it starts to give an error due to timeout.
Of the 545 thousand files, I can’t reach more than 1800. When everything stops.

My katalon.ini

-startup
plugins/org.eclipse.equinox.launcher_1.3.201.v20161025-1711.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.401.v20161122-1740
-data
@noDefault
-vmargs
-XX:+UseG1GC
-XX:+UseStringDeduplication
-Xms256m
-Dfile.encoding=utf-8
-Xmx2048m

It was a good experience.
I wanted to know the tool.
I thank everyone who helped me.

@duyluong Sounds like something you might want to know.

Seriously? What did you expected? Magic?
Processing large amount of data require strong knowledge about programming (threading, batch processing and so on)
I doubt any tool in this world targeted on functional testing will suit your needs wrt propossed exercise.
You may have to start learning real programming.

@xupacabra

I have ever experienced a similar situation in my VisualTestingInKatalonStudio project. I saw, my project in Katalon Studio ran too slow, consumed too large memory, eventually stopped while processing large number of URLs and writing hundreds of files.

I changed my codes and, yes, I could solve it. My code now runs far faster than before, requires fair size of memory, never stops. The tool should not be blamed. God dwells in the details of my code. :wink:

You disclosed your code above. There I find 3 essential points to be improved. I would tell you what.

… still authoring

2 Likes

let’s do a simple math (simplified):
you have 454000 items to grab. supposed you can grab a single item in 1 sec (in fact it takes longer with your AUT but we ignore this for the moment) you will need 454000 / 3600 >> 126 hours only to fetch the data
If you want to reduce this time to 1 hr you need a machine able to run 126 threads.
Now … have some fun, run your project to fetch, let’s say … 10-20 items a take a note about RAM consumption before and during the project run. Make the diff and multiply it with 126 < this is the amount of memory you will need to be available at a minimum.
And here i just ignore the CPU cycles needed for it …

And for the moment i ignore also the storage size needed. Supposed each file is just 1K in size, you need ~443M available. Not huge but you can do the math with the real size.

And the above is just for fetching the data … If you need also to post-process the data things looks even funnier.

So you have to totally reconsider your approach, IMHO, by using pure programming with parallel threading, async calls to fetch the data and store it in a certain DB so it can be easy analyzed further.

And definitely you need a better method to fetch the data (direct connection to DB or at least, straight http requests)

Here Katalon himself indeed, cannot help you much. It is designed mostly for sequential code-run, not for magic stuff.
Although it does have support for parallel’ execution, that feature is note pure parallel threading but some sort of switching fast between the threads.

Choose any language you like here, you can do it in Java, Python, C++ … but you have to read about the techniques I mentioned a bit.

If you really want you can use Katalon as IDE, nothing wrong with it as long as you write your code properly.

It was a learning experience.
I have been programming since 1986 and finished the job with 1 day of work and processing in Ruby. I liked the tool and its concept.

1 Like

Thank you for your help.
Yes, I liked your suggestions.

@xupacabra

Have you already abandoned your project in Katalon Studio, or still continuing/learning?

finished the job with 1 day of work

Once you mentioned that you expect the job takes 46 days to run. But now you wrote you’ve done the same in 1 day, do you?

What’s the difference? Do you still do Web Page scraping? Or you have changed the design and you uses immediate DB connection?

Great!
Now, reimplement your solution in java. For fun!
:smiley:

To me, it seems that he required a new way of executing test processings in parallel on a machine with Multiple-core-CPU (e.g. AWS EC2 M5: 96 vCPU). I have invented it for Katalon Studio. See the following topic:

… sorry, I’m joking. I haven’t tested any project empowered by TestClosure on EC2 M5. Just I hoped it.