Request interception with chrome console

I was wondering if there is a better way to do things.
We are waiting for a few rest request to finish and collect the data to it.

With the Chrome console plugin getting the requests is handy, but waiting for the response and getting the data is quiet “ugly” as getting the response and getting the body data to it is split into two functions and the call needs to be finished up front

So currently we do something like:
Init…

ChromeDevToolsService cdts = CdpUtils.getService()
Page page = cdts.getPage()
def map = [:]
Network network = cdts.getNetwork()
Log log = cdts.getLog()

Register Request and Response events and store the data in a map…

network.onRequestWillBeSent({ def event ->
        if (event.type.toString() == 'XHR') {
            if (!(map.containsKey(event.requestId))) {
                def requestEnty = [('RequestID') : event.requestId, ('RespondeID') : '', ('ResponseStatusCode') : '', ('RequestBody') : ''
                    , ('RequestUrl') : event.getRequest().getUrl(), ('RequestMethod') : event.getRequest().method]
                map.put(event.requestId, requestEnty)
            }
        }
    })

network.onResponseReceived({ def event ->
        if (event.type.toString() == 'XHR') {
            WebUI.comment((('onResponseReceived Request ID: ' + event.requestId) + ' - ') + event.response.status.toString())
            if (map.containsKey(event.requestId)) {
                valueList = map.get(event.requestId)
                valueList.RespondeID = event.requestId
                valueList.ResponseStatusCode = event.response.status.toString()
                map.put(event.requestId, valueList)
            }
        }
    })

Enable the listener

page.enable()
network.enable()

Now katalon runs sequential and the calls do come in one by one as some may take a few seconds or longer…so waiting for all to finish seams the only way…

boolean hasNotAllRequest = true
while (hasNotAllRequest) {
    def map2 = bar = map.getClass().newInstance(map)
    def sizebefore = map.size()
    for (def e : map2) {
        if (e != null) {
            def valueList = e.value
            if (e.key == valueList.RespondeID) {
                hasNotAllRequest = false
            } else {
                hasNotAllRequest = true
            }
        }
    }
    if (map.size() != sizebefore) {
        hasNotAllRequest = true
    }
    if (hasNotAllRequest) {
        WebUI.delay(1)
    }
}

and than getting the body data to it once all calls are finished…

def map2 = bar = map.getClass().newInstance(map)

for (def entry : map2) {
    if (entry != null) {
        try {
            String body = network.getResponseBody(entry.key.toString()).body
            entry.value.RequestBody = body
        }
        catch (Exception e) {
        } 
    }
}

Am i overlooking something that is a bit more smart or elegant?

We tried and call the getResponseBody in the onResponseReceived event and that results in a error.

Any ideas are welcome :slight_smile:

2 Likes

Hello! I was needing to read the body of each request that was being generated. So far I did something similar to yours but the getbody doesn’t work for me. Could you fix it?

Wow. Thanks for posting this @Jens.gr (and sorry I missed it first time around).

For the Groovy programmer, I think you have produced a pretty good interface to the CDTS API set. Had I developed something like this, I would have gone the JavaScript route (i.e. I wouldn’t need CDTS, I don’t think). But that’s not a criticism. In Groovy, I would still need a while loop to force the Test Case to wait until the entire JS/XHR set was complete.

Aside: There is some talk about making JavaScript a first-class citizen in Katalon Studio along with the ability to target the chromium V8 engine directly. Perhaps then, native browser tasks like this might become a little cleaner and easier to implement. (@duyluong)

@Matias

Keywords (10.0 KB)
CustomKeywords.‘rd.ChromeConsole.RegisterListender’(true, true)

#Add any code that produces the data

CustomKeywords.‘rd.ChromeConsole.WaitForRequestData’()
CustomKeywords.‘rd.ChromeConsole.GetRequestString’(true)
CustomKeywords.‘rd.ChromeConsole.ResetDataCollection’()

The GetResponse() holds than all calls that have been finished.

that is how we keep using it as of now and it works okayisch. nur sure it meats your use cases.

@Jens.gr Brilliant! Thank you. They only use these Keywords that you sent me or the code above the post also have to be loaded?

@Matias It is using as mentioned the plugin: https://github.com/katalon-studio/katalon-studio-chrome-devtools-protocol-plugin and builds up on it. (used in the keywords)
And with the code above we get all send requests in between the listen / wait.

if your not getting it to work just let me know may have a few spare minutes to make a test project for it.

@Jens.gr I sent you a private message. Thank you!

As for usage out of the conversation we had. i just add this here

```
import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable as GlobalVariable


WebUI.openBrowser('YOUR URL THAT SENDS a XHR Request')

CustomKeywords.'rd.ChromeConsole.RegisterListender'(true, true)

WebUI.navigateToUrl('YOUR URL THAT SENDS a XHR Request')
CustomKeywords.'rd.ChromeConsole.WaitForRequestData'()

//CustomKeywords.'rd.ChromeConsole.GetRequestString'(true)

 Map map =  CustomKeywords.'rd.ChromeConsole.GetResponse'()
for (def entry : map) {
    if (entry != null) {
		println("Entry to look at...")
		println(entry.value)
    }
}
println(map.size())
```

Thank you, it worked locally but not on BrowserStack.
I got this message on the first instruction ChromeDevToolsService cdts = CdpUtils.getService() :

Failed sending HTTP request.

More details:

com.github.kklisura.cdt.services.exceptions.ChromeServiceException: Failed sending HTTP request.

at com.github.kklisura.cdt.services.impl.ChromeServiceImpl.request(ChromeServiceImpl.java:292)

at com.github.kklisura.cdt.services.impl.ChromeServiceImpl.getTabs(ChromeServiceImpl.java:127)

at com.github.kklisura.cdt.services.ChromeService$getTabs.call(Unknown Source)

at com.katalon.cdp.CdpUtils.getService(CdpUtils.groovy:42)

at com.katalon.cdp.CdpUtils$getService.call(Unknown Source)

at mock.run(mock:9)

at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)

at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)

at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:445)

at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:436)

at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:415)

at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:407)

at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:284)

at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:142)

at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:133)

at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)

at TempTestCase1654765836164.run(TempTestCase1654765836164.groovy:25)

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at com.github.kklisura.cdt.services.impl.ChromeServiceImpl.request(ChromeServiceImpl.java:273)

at com.github.kklisura.cdt.services.impl.ChromeServiceImpl.getTabs(ChromeServiceImpl.java:127)

at com.github.kklisura.cdt.services.ChromeService$getTabs.call(Unknown Source)

at com.katalon.cdp.CdpUtils.getService(CdpUtils.groovy:42)

at com.katalon.cdp.CdpUtils$getService.call(Unknown Source)

at Script1654699810967.run(Script1654699810967.groovy:9)

Years have passed. Selenium4 has been released, which provides Chrome DevTools Protocol support. Selenium4 is what @Jens.gr wanted.

5. Support for Chrome Debugging Protocol

Selenium 4 comes with native support for Chrome DevTools Protocol. This means QAs can now use Chrome development properties like Fetch, Network, Profiler, Performance, Application cache, and more. QAs can also leverage the APIs offered by Chrome DevTools to simulate poor network conditions and perform geolocation testing.

Using this API will also help developers or QAs to test and resolve critical bugs for specific web-pages faster and on the fly.

Please note that this article is published by Browserstack.

Unfortunately Katalon Studio is NOT yet ready for Selenium 4. I am afraid KS is getting behind.

> Unfortunately Katalon Studio is NOT yet ready for Selenium 4. I am afraid KS is getting behind.

Indeed. More than half a year has passed. But thank you for linking your post, did not catch it before :slight_smile: nice to read!