The only thing you can do right now is, re-do your project to use only some 2-3 images, for a proof of concept.
200 is too much, as proven …
So, once you have it working like this, you can somehow present it (with caveats) and further keep in touch with Katalon team to solve the remaining issues for better scalability
In the Katalon’s doc “Use TestOps Visual Testing”, I noticed a description.
we highly recommend using[WebUI] Take Screenshot As Checkpoint
@anna.kotliarevsckiy mentioned she needs a full page. +1 from me to her. I have no doubt that a full page screenshot is far more useful for users’ pre-release smoke tests than a partial screenshot. Why does Katalon team not recommend [WebUI] Take Full Page Screenshot As Checkpoint ?
Now I guess, they are already aware that takeFullPageScreenshotAsCheckpoint() tends to create a huge size of PNG image which causes the die-hard OutOfMemoryError. But they do not like to talk about it aloud. They are trying to lead users not to use the “Full Page” keyword in order to avoid any mess.
To me, TestOps’ Visual Testing seems to be a Proof Of Concept yet.
@anna.kotliarevskiy - I believe you are executing test scripts in Katalon Studio and the results (captured images) are then uploaded to TestOps. After that, Katalon Visual Testing will be triggered to analyze the captured images to find out the differences.
So if you are going to have a presentation, I wonder if you can try to execute test scripts within TestOps. And the steps are:
With this approach, you can have a Baseline Collection with less images than the default collection (called System-generated Common Baseline Collection). And in this baseline collection, it only contain images with the new resolution captured from your new machine. Hope this can help you to prepare for your presentation.
Thanks @kazurayam , I do need a full page screenshot and I hope Katalon team will make it work.
Percy does it, so I think it is important for Katalon to stay competitive…
Katalon uses the shutterbug to take screeenshots.
In the Issues of the shutterbug, I found a similar post as @anna.kotliarevskiy found:
This issue was unresolved but was closed.
So I presume that what @anna.kotliarevskiy saw was caused by the shutterbug.
Katalon uses the shutterbug to take screeenshots and compare them.
In the Issues of the shutterbug, I found an interesting post concerning the width and height of images.
I learned by this thread that
shootPage(driver, CaptureFULL) method occasionally have different height, like image1 - 1295x7740; image2 - 1295x7726
I suppose that @anna.kotliarevskiy was annoyed by this unstability of the height of the images created by shutterbug.
One of the post into this discussion tells a work-around:
And this seems to have fixed it: use a delay such as Thread.sleep(5000) after visiting the page and before the screenshot.
And I am finding that if I use Firefox, it seems to take a better more consistent full-page image with the setting Capture.FULL.
You should explicitly wait somehow for the full-page is loaded completely.
The primary issue for @anna.kotliarevskiy was that the screenshot images of a single URL taken by WebUI.takeFullPageScreenshot at 2 timings had different height; the height of the image changed time to time. Why does she get different image heights?
I presume that the Test Case script of anna does NOT wait for the URL to load into the browsers’ window completely (so that the HTML DOM gets static; not moving anymore) before taking a screenshot.
If a screenshot was taken after the page was loaded, then the image will be the same as the previous complete shot. However, if a screenshot was taken while the page was still incomplete (the elements are yet being loaded), then the screenshot will be different from the previous one. There could be various accidental factors; e.g, the HTTP server may be busy so it responds slowly, the network may be jammed, some other CPU-instensive processes may be running in parallel on the tester’s machine.
In order to refrain from such uncertainty, the tester should make sure that the Test Case script waits long enough (but not too long) for the target page to load completely before taking any screenshots.
But, how to make it? The following code shows what I would recommend : “explicitly wait for a HTML element as a handle to be present in the page” :
import java.awt.image.BufferedImage
import javax.imageio.ImageIO
import com.kms.katalon.core.testobject.TestObject
import com.kms.katalon.core.testobject.ConditionType
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
TestObject makeTestObject(String id, String xpath) {
TestObject tObj = new TestObject(id)
tObj.addProperty("xpath", ConditionType.EQUALS, xpath)
return tObj
}
String url = 'https://www.adm.com/en-us/insights-and-innovation/formulation-challenges/sugar-reduction/'
String fileName = "sugarReduction.png"
File png = new File("./" + fileName)
WebUI.openBrowser('')
WebUI.setViewPortSize(1200, 800)
WebUI.navigateToUrl(url)
// now we should explicitly wait for the page to load completely before taking its screenshot
TestObject footerLogo = makeTestObject('footerLogo', '//footer//img[@alt="ADM Logo Footer"]')
WebUI.verifyElementPresent(footerLogo, 10)
WebUI.delay(1)
// now the page would have been loaded complete
// let's take a full page screenshot using the built-in keyword
WebUI.takeFullPageScreenshotAsCheckpoint(png.toString())
// done; close the browser
WebUI.closeBrowser()
// show information
BufferedImage bi = ImageIO.read(png)
WebUI.comment("png image width=" + bi.getWidth())
WebUI.comment("png image height=" + bi.getHeight())
WebUI.comment("png file path=" + png.toString())
WebUI.comment("png file length=" + png.length())
The WebUI.verifyElementPresent() call will wait for the <img alt="ADM Logo Footer"> element (the handle element) to be present in the DOM, at the longest 10 seconds, before the test case takes the full page screenshot.
My idea is that it would be appropriate to assume that the page will be complete when the handle element is present, as it is located near the tail of the whole HTML DOM. So let’s wait for the handle to appear in the DOM, then we will be ready to go forward to take screenshots.
Katalon Studio provides “Smart Wait Function”:
The doc says this would help you for waiting the pages to be loaded completely. With it in theory, @anna.kotliarevskiy wouldn’t need to change her Test Case scripts.
Though I personally do not use this function, simply because I do not understand it. The document explains nothing what it does and does not internally.
The “Object Name” is the:
In your script repo, please add your test cases to a Test Suite folder, then refresh the Script Repo in TestOps to see the Test Suite. Finally, you can select the corresponding path in the “Object Name” drop-down list as the following example:
I observed anna’s target web site http://www.adm.com/ and found that the pages are very difficult to take clean full page screenshot. See the following movie, especially look at the timeframe 10-15secs
You will find some graphics are moving along you scroll down the page. This moving graphics makes it very difficult to take “stable screenshots”.
Let me step back and explain how a full page screenshot is composed. See the following figure:
The screenshooter library (Shutterbug and AShot) can create a full page screenshot which is generated out of a set of partial screenshots of the viewport. The library start taking a viewport screenshot at the top, scroll down to the next view port and take a viewport screenshot, … scroll and shoot, scroll and shoot, … until it reaches to the bottom of the page.
Now let me assume that the target page has any slowly moving graphics that moves as you scrool the view port downward. The library has to wait long enough until the moving graphics settles. So, how long of milliseconds Shutterbug waits as default? ---- just 100 milliseconds as the source tells
The moving graphics at https://www.adm.com is moving so gracefully (slowly) that it is quite likely that when the shutterbug scrolls down the viewport into the portion where the graphics is located, it would find the graphics is still moving for some pixels per 100 milliseconds. The resulting screenshot image will be unstable. Therefore the library is likely to produce a “diff” like this:
Here the different pixels are painted in red. But I am only interested in the diff caused by the unexpected problems. I am not interested in the diff caused by the known factors (moving graphics). How to avoid the boring diffs?
There is a trick. The Shutterbug allows users to specify how long milliseconds it should wait after each scroll operations before taking the partial viewport screenshots. Quote from selenium-shutterbug/src/main/java/com/assertthat/selenium_shutterbug/core/Shutterbug.java at master · assertthat/selenium-shutterbug · GitHub
/**
* To be used when screenshotting the page
* and need to scroll while making screenshots, either vertically or
* horizontally or both directions (Chrome).
*
* @param driver WebDriver instance
* @param capture Capture type
* @param betweenScrollTimeout Timeout to wait between each scrolling operation
* @param useDevicePixelRatio whether to account for device pixel ratio
* @return PageSnapshot instance
*/
public static PageSnapshot shootPage(WebDriver driver, Capture capture,
int betweenScrollTimeout, boolean useDevicePixelRatio) {
...
If user specify a longer value to betweenScrollTimeout, for example 2000, the gracefully moving graphis will get settled, and the screenshot image will become stable. You will get less magnitude of pixel differences.
Now, I would tell you a unfortunate news. Katalon’s WebUI.takeFullPageScreenshotAsCheckout() does not allow you to specify a longer value of the betweenScrollTimeout. See the source
Therefore, if you use this keyword, you will always find the screenshot of https://www.adm.com/ page is reported “different” due to the moving graphics.
A quick work-around: you can call the Shutterbug API directly from your Test Case script. you do not have to use WebUI.* keywords.
the moving graphics are only present on Home page.
My test cases are in the Test Cases folder, Test suites folder contains test suites.
Do you mean I need to move test cases from Test Cases folder to the Test Suites Folder?
But this is not the way it is structured in the sample project.
@kazurayam this moving effect is only applicable on the first page scroll. I have as step in the test cases to scroll down to the page (to let all the elements load first) and then I take screenshot.
I have no problem with taking screenshot for with Home page (and it was always analyzed with no problem before)
You once disclosed your code:
Here I noticed a mysterious line:
WebUI.scrollToPosition(99999, 99999)
I did not see what this line is meant for. But now I understand it. The moving graphics in the https://www.adm.com/ move only when they are loaded, and they would not move once they have been loaded into the DOM. The above scrollToPosition(99999,99999) is meant for triggering the onload events for all graphics.
Ah, good tips. Thank you.
You have to create a test suite, add to the test suite only the relevant test-cases, and further use that suite in TestOps (after you commit your changes and refresh the repository, as indicated)
It has all been done before and I have test cases, test suites.
I can commit and push changes to the repo from the Katalon Studio.
But I am not able to see the repo content when adding it to the Test Ops.
When I try to use the demo repo you have I can see the test suite, but when I am using my repo I cannot see the test suites and test cases within TestOps.
I am not sure what the difference is…
You have to refresh the repository in TestOps to pick up latest commits, it is not doing this automatically.
If still not working, delete it from TestOps and add it again.
I did all of this and still no luck
