Comparing Prod vs Dev website using multiple urls per website

[mgagnequebec]

Hello all, I’ve been trying out Katalon for a while and stumbled on @kazurayam scripts which seemed like a great starting point for my needs. (GitHub - kazurayam/VisualInspectionInKatalonStudio_Reborn: A Katalon Studio project. This project performs automated Web UI testing with screenshot comparison and HTML source comparison.)

I had made a question on github but i’ll paste it here for visibility. Seems it’s more appropriate to ask here.

the scripts i’m trying are part of the " VisualInspectionTwins" sets. The instructions are for one link (one page) per environment. But my test/website has multiple urls for both Prod and Dev environments (around 50 urls for one website on Prod, and the same for the website version on Dev)

I’ve modified the code to use data files and tried to make it work for this “multiple urls per website” scenario. But I have a strange behaviour now. After my test finishes, it passes but shouldn’t. As a trial, I purposely made the scripts compare 2 sets of completely different websites so that it fails when checking the visual comparison. But it ends in a “test pass”!

I had to split the “visitMyAdminTopPage” testcase into 2 scripts tho. Actually, i’ll paste all scripts that I modified below:

“VisualInspectionTwins”

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase

import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths

import com.kazurayam.ks.globalvariable.ExecutionProfilesLoader
import com.kazurayam.materialstore.DiffArtifacts
import com.kazurayam.materialstore.JobName
import com.kazurayam.materialstore.JobTimestamp
import com.kazurayam.materialstore.MaterialList
import com.kazurayam.materialstore.IgnoringMetadataKeys
import com.kazurayam.materialstore.MetadataPattern
import com.kazurayam.materialstore.Store
import com.kazurayam.materialstore.Stores
import com.kms.katalon.core.configuration.RunConfiguration
import com.kms.katalon.core.util.KeywordUtil
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

import internal.GlobalVariable

Path projectDir = Paths.get(RunConfiguration.getProjectDir())
Path root = projectDir.resolve("store")
Store store = Stores.newInstance(root)
JobName jobName = new JobName("MyAdmin_VisualInspectionTwins")
ExecutionProfilesLoader profilesLoader = new ExecutionProfilesLoader()

// --------------------------------------------------------------------

// visit the Production environment
String profile1 = "ProdEnv"
profilesLoader.loadProfile(profile1)
WebUI.comment("Execution Profile ${profile1} was loaded")
JobTimestamp timestampP = JobTimestamp.now()
WebUI.callTestCase(
	findTestCase("utils/goto_ProdSite"),
	["profile": profile1, "store": store, "jobName": jobName, "jobTimestamp": timestampP]
)

	
// visit the Development environment
String profile2 = "DevEnv"
profilesLoader.loadProfile(profile2)
WebUI.comment("Execution Profile ${profile2} was loaded")

JobTimestamp timestampD = JobTimestamp.now()
WebUI.callTestCase(
	findTestCase("utils/goto_DevSite"),
	["profile": profile2, "store": store, "jobName": jobName, "jobTimestamp": timestampD]
)
	
// --------------------------------------------------------------------

// compare the materials obtained from the 2 sites, compile a diff report

// pickup the materials that belongs to the 2 "profiles"
MaterialList left = store.select(jobName, timestampP,
			MetadataPattern.builderWithMap([ "profile": profile1 ]).build()
			)

MaterialList right = store.select(jobName, timestampD,
			MetadataPattern.builderWithMap([ "profile": profile2 ]).build()
			)

// difference greater than the criteria should be warned
double criteria = 0.0d

// make DiffArtifacts
DiffArtifacts stuffedDiffArtifacts = store.makeDiff(left, right, IgnoringMetadataKeys.of("profile", "URL.host"))
int warnings = stuffedDiffArtifacts.countWarnings(criteria)

// compile HTML report
Path reportFile = store.reportDiffs(jobName, stuffedDiffArtifacts, criteria, jobName.toString() + "-index.html")
assert Files.exists(reportFile)
WebUI.comment("The report can be found ${reportFile.toString()}")

if (warnings > 0) {
	KeywordUtil.markFailed("found ${warnings} differences.")
}

“visitMyAdminTopPage1” renamed to “goto_ProdSite”

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kazurayam.materialstore.Metadata
import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable

// def linksDataDev = findTestData("dev")
def linksDataProd = findTestData("prod")
def dbDevProd_data = [('urlprod') : '',('username') : '', ('password') : '']
def rows = findTestData("prod").getRowNumbers()

// check params which should be passed as the arguments of WebUI.callTestCases() call
Objects.requireNonNull(profile)
Objects.requireNonNull(store)
Objects.requireNonNull(jobName)
Objects.requireNonNull(jobTimestamp)

// check the GlobalVariables
// assert GlobalVariable.URL != null, "GlobalVariable.URL is not defined"

WebUI.openBrowser(null)
WebUI.setViewPortSize(1024, 800)

for (def rowNumIndex = 1; rowNumIndex <= rows; rowNumIndex++) {
	
	dbDevProd_data.urlprod = linksDataProd.getValue(1, rowNumIndex)
	WebUI.navigateToUrl(dbDevProd_data.urlprod)
	WebUI.waitForPageLoad(10)
	
	URL url = new URL(WebUI.getUrl())
	
	WebUI.callTestCase(findTestCase("utils/takeScreenshotAndPageSource"),
		[
			"store": store,
			"jobName": jobName,
			"jobTimestamp": jobTimestamp,
			"metadata": Metadata.builderWithUrl(url)
										.put("profile", profile)
										.put("selector", "body")
										.build()
		]
	)
	
	
}


WebUI.closeBrowser()

“visitMyAdminTopPage2” renamed to “goto_DevSite”

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kazurayam.materialstore.Metadata
import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable

def linksDataDev = findTestData("dev")
// def linksDataProd = findTestData("prod")
def dbDevProd_data = [('urldev') : '',('username') : '', ('password') : '']
def rows = findTestData("dev").getRowNumbers()

// check params which should be passed as the arguments of WebUI.callTestCases() call
Objects.requireNonNull(profile)
Objects.requireNonNull(store)
Objects.requireNonNull(jobName)
Objects.requireNonNull(jobTimestamp)

// check the GlobalVariables
// assert GlobalVariable.URL != null, "GlobalVariable.URL is not defined"

WebUI.openBrowser(null)
WebUI.setViewPortSize(1024, 800)

for (def rowNumIndex = 1; rowNumIndex <= rows; rowNumIndex++) {
	
	dbDevProd_data.urldev = linksDataDev.getValue(1, rowNumIndex)
	WebUI.navigateToUrl(dbDevProd_data.urldev)
	WebUI.waitForPageLoad(10)
	
	URL url = new URL(WebUI.getUrl())
	
	WebUI.callTestCase(findTestCase("utils/takeScreenshotAndPageSource"),
		[
			"store": store,
			"jobName": jobName,
			"jobTimestamp": jobTimestamp,
			"metadata": Metadata.builderWithUrl(url)
										.put("profile", profile)
										.put("selector", "body")
										.build()
		]
	)
	
	
}


WebUI.closeBrowser()

And the “takeScreenshotAndPageSource” did not change so i guess pasting it here is not needed. But I will just in case:

import java.nio.file.Files
import java.nio.file.Path

import org.jsoup.Jsoup
import org.jsoup.nodes.Document
import org.openqa.selenium.WebDriver

import com.kazurayam.materialstore.FileType
import com.kazurayam.materialstore.Material
import com.kazurayam.materialstore.Metadata
import com.kms.katalon.core.webui.driver.DriverFactory
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI


Objects.requireNonNull(store)
Objects.requireNonNull(jobName)
Objects.requireNonNull(jobTimestamp)
Objects.requireNonNull(metadata)
assert metadata instanceof Metadata

// take a screenshot and save the image into a temporary file using Katalon's built-in keyword
Path tempFile = Files.createTempFile(null, null);
WebUI.takeFullPageScreenshot(tempFile.toAbsolutePath().toFile().toString(), [])

// copy the image file into the materialstore
Material image = store.write(jobName, jobTimestamp, FileType.PNG, metadata, tempFile)
assert image != null

// save the page source HTML into the materialstore
WebDriver driver = DriverFactory.getWebDriver()
Document doc = Jsoup.parse(driver.getPageSource())
Material html = store.write(jobName, jobTimestamp, FileType.HTML, metadata, doc.toString())
assert html != null

Not sure why this passes, I can see the test going to the correct webpages, using the proper data files, like so:

set1
WebUI.navigateToUrl(dbDevProd_data.urlprod) goes to (bogus websites just for explanation’s sake)
1- www.testprod01.com
2- www.testprod02.com
3- www.testprod03.com
4- www.testprod04.com

set2
WebUI.navigateToUrl(dbDevProd_data.urldev) goes to:
1- www.testdev01.com
2- www.testdev02.com
3- www.testdev03.com
4- www.testdev04.com

of course i’m using other links in the data files but set1 and set2 are completely different websites and could not possibly result in a “pass”. Is there a kind soul that could help me out to see what I’m doing wrong ? I’m just starting to learn and I’m stuck on this problem.

[kazurayam]

The Test Case “VisualInspectionTwins” is designed with an assumption that it would be executed once per a single pair of Prod and Dev host. Usually a pair of hosts have many URLs with various sub-paths. You can process these bunch of URLs in a single execution of “VisualInspectionTwins”.

In case where you have 2 or more pairs of Prod host and Dev host, you should execute the “VisualInspectionTwins” 2 or more times.

See the following example.

Test Case “VisualInspectionTwins” execution #1

prod URL	dev URL	recognised as a pair by the “path”
https://www.testprod01.com/	https://www.testdev01.com/	/
https://www.testprod01.com/a	https://www.testdev01.com/a	/a
https://www.testprod01.com/b/c	https://www.testdev01.com/b/c	/b/c
https://www.testprod01.com/d/e/f	https://www.testdev01.com/d/e/f	/d/e/f
https://www.testprod01.com/g/1	https://www.testdev01.com/g/1	/g/1
https://www.testprod01.com/g/2	https://www.testdev01.com/g/2	/g/2
https://www.testprod01.com/g/3	https://www.testdev01.com/g/3	/g/3
https://www.testprod01.com/g/4	https://www.testdev01.com/g/4	/g/4
…	…	…

Test Case “VisualInspectionTwins” execution #2

prod URL	dev URL	recognised as a pair by the “path”
https://www.testprod02.com/	https://www.testdev02.com/	/
https://www.testprod02.com/h	https://www.testdev02.com/h	/h
https://www.testprod02.com/i/j	https://www.testdev02.com/i/j	/i/j
https://www.testprod02.com/k/l/m	https://www.testdev02.com/k/l/m	/k/l/m
https://www.testprod02.com/o/5	https://www.testdev02.com/o/5	/o/5
https://www.testprod02.com/o/6	https://www.testdev02.com/o/6	/o/6
https://www.testprod02.com/o/7	https://www.testdev02.com/o/7	/o/7
https://www.testprod02.com/o/8	https://www.testdev02.com/o/8	/o/8
…	…	…

Test Case “VisualInspectionTwins” execution #3

prod URL	dev URL	recognised as a pair by the “path”
https://www.testprod03.com/	https://www.testdev03.com/	/
https://www.testprod03.com/a	https://www.testdev03.com/a	/a
https://www.testprod03.com/b/c	https://www.testdev03.com/b/c	/b/c
https://www.testprod03.com/d/e/f	https://www.testdev03.com/d/e/f	/d/e/f
https://www.testprod03.com/g/1	https://www.testdev03.com/g/1	/g/1
https://www.testprod03.com/g/2	https://www.testdev03.com/g/2	/g/2
https://www.testprod03.com/g/3	https://www.testdev03.com/g/3	/g/3
https://www.testprod03.com/g/4	https://www.testdev03.com/g/4	/g/4
…	…	…

Test Case “VisualInspectionTwins” execution #4

prod URL	dev URL	recognised as a pair by the “path”
https://www.testprod04.com/	https://www.testdev04.com/	/
https://www.testprod04.com/a	https://www.testdev04.com/a	/a
https://www.testprod04.com/b/c	https://www.testdev04.com/b/c	/b/c
https://www.testprod04.com/d/e/f	https://www.testdev04.com/d/e/f	/d/e/f
https://www.testprod04.com/g/1	https://www.testdev04.com/g/1	/g/1
https://www.testprod04.com/g/2	https://www.testdev04.com/g/2	/g/2
https://www.testprod04.com/g/3	https://www.testdev04.com/g/3	/g/3
https://www.testprod04.com/g/4	https://www.testdev04.com/g/4	/g/4
…	…	…

A Test Suite in Katalon Studio can not bind a single Test Case (“VisualInspectionTwins”) multiple times. Also you would want to parameterize the name of “Data” to consume.

Therefore you would make 4 individual Test Cases TC1, TC2, TC3, TC4 which indirectly call “VisualInspectionTwins” with parameter. As a parameter you would pass the name of “Data” to consume.

TC1 would look something like this:

// TC1

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

WebUI.callTestCase(findTestCase("VisualInspectionTwins"), ["name":"Data1"])

TC2 would look something like this:

// TC2

import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

WebUI.callTestCase(findTestCase("VisualInspectionTwins"), ["name":"Data2"])

TC3, TC4 and so on like this.

And finally you want to create a Test Suite “TS_all” which calls TC1, TC2, TC3 and TC4.

[mgagnequebec]

Thank you for helping out! Looks like I overdid it tho, I apologize because I am confused a little. I’ll make my scenario a bit simpler to help me understand, hopefully this is not too annoying.

Starting with a vanilla “VisualInspectionInKatalonStudio_Reborn” project, without using data files.

In that scenario, trying to compare only one Prod + Dev website pair, if I want to test multiple urls per environment like this:

Prod

• http://prod_site/page1.foo
• http://prod_site/page2.foo
• http://prod_site/page3.foo

Dev

• http://dev_site/page1.foo
• http://dev_site/page2.foo
• http://dev_site/page3.foo

do I just fill out the profile with the urls I need to test for both environments ? If so, how?

The template profile “MyAdmin_ProductionEnv” has 1 entry pre-filled:

• Name: URL
• Value Type: String
• Value: http://myadmin.kazurayam.com/

Should I just add more entries so that it goes through more urls? Like this:

Name: URL2
Value Type: String
Value: http://myadmin.kazurayam.com/page1

Name: URL3
Value Type: String
Value: http://myadmin.kazurayam.com/page2

etc… ?

But then it means I need to change “visitMyAdminTopPage” to handle those new entries in the profile, correct?

I am asking because this line

WebUI.navigateToUrl("${GlobalVariable.URL}

seems to handle the first “URL” only. So I get confused when I read that the script can process a bunch of URLs in a single execution. Any chance I could get some clarifications about that ?

[kazurayam]

@mgagnequebec

Should I just add more entries so that it goes through more urls? Like this:

That would be an idea, but I do not like it.

I would rather make an Excel file or CSV file that contains a list of URLs to process; and in the profile I will write the name of the data file.

@mgagnequebec

But then it means I need to change “visitMyAdminTopPage” to handle those new entries in the profile, correct?

Yes, correct.

@mgagnequebec

I am asking because this line

WebUI.navigateToUrl("${GlobalVariable.URL}

seems to handle the first “URL” only. So I get confused when I read that the script can process a bunch of URLs in a single execution.

Yes, you are right.

---

The Test Case “VisualInspectionTwins” in my sample project is just an example that can visit only a single URL. I should have named it as

• sample_VisualInspectionTwins_for_a_single_pair_of_URLs

If you want to visit multiple URLs of your target Web App, the “VisualInspectionTwins” is not to be used. You should newly develop a Test Case in your own project for yourself, namely for example

• mgagnequebec_VisualInspectionTwins_for_multiple_pairs_of_URLs

You can name it whatever. You can code your test case as you like, referring to the sample test case I provided.

[kazurayam]

The way how to visit a Web site depends on the nature of each Web App.

• In a simple case, you may be able to list all the target URLs. The way how to present the URLs does not matter much; in Excel accessed via “Data”, in text files in JSON, or URL1+URL2+UR3 in an Execution Profile.

• In another case, you may have to interactively trace a tree of navigation menus to find out the URLs to visit.

• In one more case, your code have to type a “symbol” of some entity (equity ticker, search key, etc) into a <input type="text"> and click the SEND button.

There would be unlimited number of patterns of page navigation. I (@kazurayam) can not provide any “one-for-all” solution for visiting sites. You should develop your own Test Case that visits your target Web App.

Once your test case script finished taking the materials (screenshot, HTML, JSON etc) and saving them in the “store” in the project, then my materialstore library comes up to provide a “one-for-all” solution of processing the files in the “store” directory. It will compare the screenshots and generate a report automagically. You do not have to develop any complicated codes for file comparison and reporting. The practical value of my solution resides in the latter phase of processing.

[mgagnequebec]

Thank you these were great responses. I understand now I think.

Once your test case script finished taking the materials (screenshot, HTML, JSON etc) and saving them in the “store” in the project, then my materialstore library comes up to provide a “one-for-all” solution of processing the files in the “store” directory.

I will try to go for a simpler method since I was under the impression that the mod I made in my first post was doing what this “store” feature required. Gather all screenshots that is. But still ended in a pass so I guess it won’t do. I did learn a couple of nice things tho

[mgagnequebec]

Alright, I figured it out using the “VisualInspectionChronos” template. At least it works for what I first imagined. I just fill out my data file with urls and it goes through all of those to take screenshots, then do the comparing / reporting as usual. (after running twice as intended)

I just changed “visitCURA” to this, in case someone whoul be interested

Objects.requireNonNull(GlobalVariable.URL)
Objects.requireNonNull(GlobalVariable.Username)
Objects.requireNonNull(GlobalVariable.Password)

Objects.requireNonNull(store)
Objects.requireNonNull(jobName)
Objects.requireNonNull(jobTimestamp)

WebUI.openBrowser('')

def dataProd = findTestData("Data Files/prod")

def pENprod_data = [('urlprod') : '']
def rows = findTestData("Data Files/prod").getRowNumbers()


//JobTimestamp jobTimestampP = JobTimestamp.now()
for (def rowNumIndex = 1; rowNumIndex <= rows; rowNumIndex++) {
	pENprod_data.urlprod = dataProd.getValue(1, rowNumIndex)
	_url = pENprod_data.urlprod
	
	WebUI.navigateToUrl(_url)
	WebUI.waitForPageLoad(10)
	URL url = new URL(WebUI.getUrl())
	
	WebUI.callTestCase(findTestCase("main/common/takeScreenshotAndPageSource"), [
		"store": store,
		"jobName": jobName,
		"jobTimestamp": jobTimestamp,
		"metadata": Metadata.builderWithUrl(url).build()
		])
	
}

// we are done
WebUI.closeBrowser()

[kazurayam]

I added a new sample code. See

• Visual Inspection in Katalon Studio - Reborn

The sample4 in the “Visual Inspection in Katalon Studio” project demonstrates how to write a set of test codes that visits a target web app for multiple pages.

I tried to write the code of the sample 4 as concise, readable and extensible as possible. I employed the Page Object Model there. I think that the sample4 can be a foundation for developing a large scale Visual Inspection project. This requires seasoned skill of Java/Groovy programming.

[kazurayam]

Let me add a comment to the “Sample 4” in the “Visual Inspection in Katalon Studio - Reborn” project. See the following comment.

(POM) Page Object Model Structure?

I have ever developed a Katalon Studio project for WebUI Automation (not REST API Automation) applying Page Object Model. Here is a Test case that uses my “Page Objects”: One of my “page object” is here for example This “Page Object” has a line static final By APP_HEADER = By.xpath('//h1[contains(text(),"Flaskr")]') Here I use org.openqa.selenium.By class to make a query for a tex “Flaskr” in the target HTML. You can imagine, you want to use JMESPath instead of By. Then your “R…