Saving Web Page as MHTML in Katalon Studio

Product Forums Katalon Studio
,
1

I have made a GitHub project to share my findings. I made it publicly accessible.

Problem to solve

I want to automate the following tasks by scripts in Katalon Studio:

  1. to take snapshot of current DOM of Web Page in browser, save it into a local file in the MHTML format.
  2. to take full screenshots of a web page in multiple viewport sizes (width*height) just as I can do in Chrome DevTools using Device Mode like Galaxy S5, Pixel 2, iPhone 6/7/8, iPad. I want screenshots in the size of PC display of XGA (1024px,768px) and SXGA (1280px,1024px) as well. I want to save the screenshots as *.png files.

Demo

Prerequisite

  1. Into your Katalon Studio, you need to install the Katalon Studio Plugin: Chrome DevTools Protocol Integration
  2. Download the zip file of this project from the zip file from the Releases page, unzip and open it with your local Katalon Studio.
  3. I tested this project with KS ver 7.2.2. But the older versions should do.
  4. You should choose Google Chrome as browser to run the demo scripts

Description of the demos

Save web page as MHTML

  1. Run Test Cases/Save web page as MHTML
  2. It will open a web page ‘http://demoaut.katalon.com/’ and save it as a single MHTML file
  3. It will write a file tmp/snapshot.mht

Capture full page in multiple viewport sizes

  1. Run TestCases/Capture full page
  2. It will open a web page ‘http://demoaut-mimic.kazurayam.com/’
  3. It will take a full page screenshot of the page
  4. It will write a file tmp/screenshot.png

Background

Motivations

In October 2020 in a post in the Katalon Forum, Katalon 7.8 beta was announced where included a new Feater called Time Capsule .

Time Capsule captures the state of your application when the test fails. No not a picture of your application when the test fails but the actual HTML, CSS of the application. Yes, that also includes the visual aspect of your application as well.

ThanTo, Katalon Developer, added a description about this feature.

Time Capsule captures the MHTML of the page when a test fails.

MHTML? What is it? Quoting from Wikipedia:

MHTML, an initialism of MIME encapsulation of aggregate HTML documents, is a web page archive format used to combine, in a single computer file, the HTML code and its companion resources (such as images, Flash animations, Java applets, and audio and video files) that are represented by external hyperlinks in the web page’s HTML code. The content of an MHTML file is encoded using the same techniques that were first developed for HTML email messages, using the MIME content type multipart/related.[1] MHTML files use a .mhtml or .mht filename extension.

OK. Now I understand what MHTML is.

Inspired by the Time Capsule, I got a new requirement to myself. I want a Custom Keyword which enables me to save snapshot of web page opened in browser into a single file in MHTML format . I want to use the keyword as often as I like during WebUI testing in Katalon Studio. For what? — For my VisualTestingInKatalonStudio project. The current version of this project takes web page screenshots in raster images of PNG format. I thought that MHTML might be better than PNG for recording the web page. Especially when I want to do Visual Testing for multiple viewPortSizes, the MHTML would be useful.

I had another motivation. In another disucusion in the Katalon Forum I was informed of a commercial product Percy, Visual Testing as a service. Percy does visual page comparison, but they wrote “it is not designed to accept screenshots, but instead captures DOM snapshots and page assets (CSS, images, etc.)”. — How they do that? In October 2020 I had no idea. But now in November I have got a guess — Percy possibly utilizes MHTML.

So I have studied how to save pages in MHTML within Katalon Studio. I have learned some points that might be interesting for Katalon users. I have developed a demo project, and here I share it.

My researches

Chrome DevTools Protocol supports taking snapshot of current DOM

I happend to find the following thread in the Puppeteer issues.

This suggests that Chrome DevTools Protocol (also known as CDP ) supports Page.captureSnapshot method, which turns the current DOM of web page into a String as MHTML.

This looked exactly what I want to do! I realised that my question was how to make use of Page.captureSnapshot method of CDP in Katalon Studio.

Chrome DevTools Protocol (CDP)

But what is “Chrome DevTools Protocol” at all? Quoting from :Chrome DevTools Protocol

The Chrome DevTools Protocol allows for tools to instrument, inspect, debug and profile Chromium, Chrome and other Blink-based browsers. Many existing projects currently use the protocol. The Chrome DevTools uses this protocol and the team maintains its API.

Instrumentation is divided into a number of domains (DOM, Debugger, Network etc.). Each domain defines a number of commands it supports and events it generates. Both commands and events are serialized JSON objects of a fixed structure.

Katalon Plugin: Chrome DevTools Protocol Integration

How can I make use of CDP in Katalon Studio? In Nov 2019, Katalon LLC released a Katalon Plugin Chrome DevTools Protocol Integration. In its README, it writes:

Integrate Chrome Devtools Protocol with Katalon Studio using GitHub - kklisura/chrome-devtools-java-client: Chrome DevTools java client..

This plugin bundles the jar of Chrome DevTools Java Client and make it available to Katalon projects.

And this plugin provides a class com.katalon.cdp.CdpUtils which implements a factory method that creates an instance of com.github.kklisure.cdt.services.ChromeDevToolsSerivce from com.kms.katalon.core.webui.dirver.DriverFactory . Provided with this utility, we can use CDP Java Client API together with Katalon’s WebUI API.

Chrome DevTools Java Client by Kenan Clisura

What is Chrome DevTools Java Client? Quoting from its README:

Chrome DevTools Java Client is a DevTools client - in Java.

Of course this library should work in the Groovy scripts in Katalon Studio. The list of examples would best describe what we can do using Chrome DevTools Protocol in Katalon Studio.

Conclusion

Katalon Studio has been solely dependent on Selenium WebDriver for Web UI testing. What we could do in Katalon Studio stays within the scope of WebDriver. Now I see, Chrome DevTools Protocol enables Web UI testers to do something exceeding the capability of WebDriver. For example, we can save web page into MHTML easily. CDP is interesting. I would look at CDP more deeply looking forward to its better use in Katalon Studio.

2 Likes
2

Truly exceptional, @kazurayam

:clap:

3

This is great work, I’ll look into it to see if there’s something we can adapt to Katalon.

4

10 months have passed since I published this custom keyword. How much this feature (take MHTML of web pages) is helpful for resolving problems? Let me look back.

The primary question is: who makes a MHTML, and who reads the MHTML.

I wanted people to take MHTML of their webpages and attached it to the post for questions in this forum. I thought the MHTML would make it possible for me to reproduce their problem on my side easily. The following is an example where I requested the questioner to post a MHTML :

But in fact, the exchanged MHTML file did not help. I could not reproduce his/her problem on my side.

I guess, the javascript in the page is AJAX-driven, requires a live continuous connection to the backend server, so that the page frozen in a MHTML does not act properly.

Once MHTML looked hopeful for me. But now I find it useless.

By the way, I personally do not need MHTML to resolve problems that occur in my local project. I have all the resources (HTML/CSS/JS, Test scripts, logs) on my PC. Why do I need an archive of them = MHTML? — No, I don’t.

I know, Katalon Studio v8.0.5 changed its “Time Capsule” implementation so that it takes MHTML instead of screenshot images. To me, this change looks nonsense. “Time Capsule” creates MHTML files for the tester assuming that he/she would find them useful. — Well, I would disagree with this assumption.

1 Like