Sample usage of BrowserMob Proxy in Katalon Studio

I have published a GitHub project

Background

In the Katalon User Forum, there is a Frequently Asked Question.

I want to do a Web UI test using browsers and at the same time I want to save the payload of HTTP requests and responses into a file. How can I do it?

Katalon Studio v8.5.x does not support recording the payload of HTTP messages exchanged between a web browser and HTTP servers. You need to employ an additional technology. The following post by matteo.lauria at Feb '21 suggested BrowserMob Proxy for this purpose.

BrowserMob Proxy will enable us to make a HAR file which contains all HTTP requests and responses recorded during tests in Katalon Studio. The HTTP messages will be formated in JSON.

The post suggested the way how to make use of BrowserMob Proxy in Katalon Studio. Unfortunately the presented sample codes were not complete (e.g, the import statements were trimmed off), so it was a bit difficult to reuse on your machine.

What’s this

Here I would show you a runnable Katalon Studio project empowered by BrowserMob Proxy which makes a HAR file that contain HTTP messages exchanged by browser and HTTP servers.

sequence

You can download the zip of the project from the Releases page. You want to download the zip; unzip it; open the project with your local Katalon Studio; run the “Test Suite/TS1”; when it finished you will find sample.har file is created in the project folder.

How to view

The sample.har file will be a very large JSON text file. It is difficult to see it and grasp overall in a text editor. You would need a tailored viewer for HAR file. I use Visual Studio Code with HAR Viewer plugin installed. The following image shows an example:

VSCode HAR Viewer

Hope this helps.

3 Likes

@kazurayam I like it.
Note that, you have some typo (altough the code works, is still a typo)

	@Keyword
	def initProxy() {
        BrowserMobProxyServer initPorxy = new BrowserMobProxyServer()
-----
		return [
			initPorxy, 
----
	}

The startProxy and stopProxy functionality can be moved in a Test Listener, for the rest looks nice.

To view HAR files, there are also some online tools available, where you can analyze them like in DevTools networking tab, e.g:

https://toolbox.googleapps.com/apps/har_analyzer/

http://www.softwareishard.com/har/viewer/

Or you can import the HAR file straight in your browser:

@kazurayam
Side note, browsermob-core-2.15 is already provided in Katalon
(I think it is used to intercept API calls with webservice projects)
so the .jar in your Drivers folder is redundant.
See the classpath file:

Katalon_Studio_Linux_64-8.5.0/configuration/resources/lib/browsermob-core-2.1.5.jar
1 Like

How to post-process the HAR file as JSON using Jayway JsonPath

Additionally, I wanted to automate verifying the content of HAR file. I expect that the sample.har file contains a record of a HTTP request to a url that ends with jquery.min.js.

{
  "log": {
    ...
    "entries": [
      ...
      {
        "request": {
          "method": "GET",
          "url": "https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js",
          ...
        }
      }
      ...
    ]
  }
}

How can I do it?

There could be many ways to meet this requirement.

I tried to solve this problem using Jayway JsonPath library.

You can find my solution in the code of Test Cases/process_har_with_jsonpath.

The following screenshot shows how the demo TS1 runs with verifying HAR file.

verify HAR

The Test Case process_har_with_jsonpath verifies if the sample.har file contains a URL that ends with jquery.min.js. Of course you can modify the condition as you want. Please study the code and the Jayway JsonPath documentation.

How to import required jar files.

Katalon Studio does not bundle the jar files required to use the Jayway JsonPath. You need to import external jar files from Maven Central repository into the <projectDir>/Drivers folder. The following document explains how to do it:

To do this, you need to install Java and Gradle into your machine. The following post has an instruction how to:

Once you installed Java and Gradle, you want to write <projectDir>/build.gradle as follows:

plugins {
  id 'java'
  id "com.katalon.gradle-plugin" version "0.1.1"
}

repositories {
  mavenCentral()
}

dependencies {
  implementation group: 'com.jayway.jsonpath', name: 'json-path', version: '2.7.0'
  implementation group: 'org.slf4j', name: 'slf4j-api', version: '2.0.6'
  implementation group: 'org.slf4j', name: 'slf4j-simple', version: '2.0.6'
}

And you want to execute in the command line:

First, you want to change the current directory to the home directory of your Katalon project. For example:

$ cd ~/BrowserMobProxyInKatalonStudio

then you want to do:

$ gradle katalonCopyDependencies

then you will see some jar files are imported into the Drivers dir:

$ ls -ls Drivers
total 1240
drwxr-xr-x   8 kazuakiurayama  staff     256  2 11 11:55 .
drwxr-xr-x  31 kazuakiurayama  staff     992  2 11 11:21 ..
-rw-r--r--   1 kazuakiurayama  staff   29489  2 11 11:21 katalon_generated_accessors-smart-2.4.7.jar
-rw-r--r--   1 kazuakiurayama  staff  121790  2 11 11:21 katalon_generated_asm-9.1.jar
-rw-r--r--   1 kazuakiurayama  staff  271159  2 11 11:21 katalon_generated_json-path-2.7.0.jar
-rw-r--r--   1 kazuakiurayama  staff  119227  2 11 11:21 katalon_generated_json-smart-2.4.7.jar
-rw-r--r--   1 kazuakiurayama  staff   62531  2 11 11:21 katalon_generated_slf4j-api-2.0.6.jar
-rw-r--r--   1 kazuakiurayama  staff   15239  2 11 11:21 katalon_generated_slf4j-simple-2.0.6.jar

Now you are ready to run the Test Suites/TS1 which indirectly execute the Test Case ‘process_har_with_jsonpath’.

process HAR using JMESPath

An alternative way to process JSON could be JMESPath.

JMESPath Python implementation is used by AWS Commandline Interface which I am used to.

JMESPath Java implementation is available at

I tried to implement an example to post-process HAR using JMESPath.

I added a few lines in the build.gradle

  implementation group: 'com.amazonaws', name: 'jmespath-java', version: '1.12.405'
  implementation group: 'io.burt', name: 'jmespath', version: '0.5.1', ext: 'pom'
  implementation group: 'io.burt', name: 'jmespath-jackson', version: '0.5.1'

and did

$ gradle katalonCopyDependencies

then I got a few jar files added in the Drivers folder.

I wrote the Test Cases/process_har_with_JMESPath which worked just the same as Test Cases/process_har_with_jsonpath.

I personally would prefer JMESPath as the JMESPath spec is expressive and proved to be powerful.

Hi @kazurayam thanks a lot for this amazing project. However, after implemented it, I have few questions regarding that,

  1. I am not able to execute the test in the Headless mode. Even if I choose Chrome (headless), it still opens a normal browser. Please let me know what I can add to make it work in a Headless mode.

  2. When I change the name of the file from (Sample.har) to a different name like “report .har”, then I do not see the report downloaded in file explorer.

Thanks a lot.

I suppose you chose Chrome (headless) in the Katalon GUI when you ran the test case,

This GUI operation is effective only when your test case script calls WebUI.openBrowser() keyword to start browsers.

On the other hand, the sample script “startProxy_openBrowser” does not use WebUI.openBrowser(). See the source code of https://github.com/kazurayam/BrowserMobProxyInKatalonStudio/blob/main/Scripts/startProxy_openBrowser/Script1673684243630.groovy

Line#24. It calls:

WebDriver driver = new ChromeDriver(...)

Therefore GUI operation is not effecetive in this case.

The Test Case starts the ChromeDriver by code. If you want to make the Chrome browser to be headless, you need to specify so in the test case code, not by GUI.

1 Like

I have changed the startProxy_openBrowser script to launch Chrome Headless browser at v 1.3.0.

1 Like

It must be due to your mistake. I don’t see what.

1 Like

thanks a lot for all the suggestions and update <3