Selenium 4 supported Chrome DevTools Protocol. Chrome DevTools Protocol is also called CDP for short. See this article.
I wanted to use CDP Network commands. For example, I wanted to fetch HTTP Request’s Headers (including Cookies) sent from Chrome to Web server. So I tried to write a Test Case that calls CDP through Selenium.
In Katalon Studio v8.2.1-alpha, I wrote the following Test Case. This script is supposed to retrieve console logs in the DevTools. I copied this from this article
import java.time.Duration;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v96.log.Log;
import com.kms.katalon.core.webui.driver.DriverFactory
// https://www.qed42.com/insights/coe/quality-assurance/selenium-4-api-chrome-devtools
System.setProperty("webdriver.chrome.driver", DriverFactory.getChromeDriverPath())
ChromeDriver driver = new ChromeDriver();
driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(10));
// try to view DevTools Console Logs
DevTools devTools = driver.getDevTools();
devTools.createSession()
devTools.send(Log.enable())
devTools.addListener(Log.entryAdded(), { logEntry ->
println("-------------------------------------------")
println("Request ID = " + logEntry.getNetworkRequestId())
println("URL = " + logEntry.getUrl())
println("Source = " + logEntry.getSource())
println("Level = " + logEntry.getLevel())
println("Text = " + logEntry.getText())
println("Timestamp = " + logEntry.getTimestamp())
println("-------------------------------------------");
})
driver.get("https://www.qed42.com/404");
driver.quit()
It worked. It ran and printed the following output
2022-01-03 17:48:29.385 INFO c.k.katalon.core.main.TestCaseExecutor - --------------------
2022-01-03 17:48:29.389 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/test/Selenium4CDP/CDPViewConsoleLogs
2022-01-03 17:48:30.087 DEBUG testcase.CDPViewConsoleLogs - 1: System.setProperty("webdriver.chrome.driver", getChromeDriverPath())
2022-01-03 17:48:30.140 DEBUG testcase.CDPViewConsoleLogs - 2: driver = new org.openqa.selenium.chrome.ChromeDriver()
Starting ChromeDriver 96.0.4664.45 (76e4c1bb2ab4671b8beba3444e61c0f17584b2fc-refs/branch-heads/4664@{#947}) on port 55856
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
1 03, 2022 5:48:33 午後 org.openqa.selenium.remote.ProtocolHandshake createSession
情報: Detected dialect: W3C
1 03, 2022 5:48:33 午後 org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
情報: Found exact CDP implementation for version 96
2022-01-03 17:48:33.970 DEBUG testcase.CDPViewConsoleLogs - 3: timeouts().implicitlyWait(Duration.ofSeconds(10))
2022-01-03 17:48:34.068 DEBUG testcase.CDPViewConsoleLogs - 4: devTools = driver.getDevTools()
2022-01-03 17:48:34.073 DEBUG testcase.CDPViewConsoleLogs - 5: devTools.createSession()
2022-01-03 17:48:34.185 DEBUG testcase.CDPViewConsoleLogs - 6: devTools.send(Log.enable())
2022-01-03 17:48:34.221 DEBUG testcase.CDPViewConsoleLogs - 7: devTools.addListener(Log.entryAdded(), { java.lang.Object logEntry -> ... })
2022-01-03 17:48:34.229 DEBUG testcase.CDPViewConsoleLogs - 8: driver.get("https://www.qed42.com/404")
-------------------------------------------
-------------------------------------------
Request ID = Optional[7D829D787FD04230D68FFB87F39497DF]
Request ID = Optional.empty
URL = Optional[https://www.qed42.com/404]
URL = Optional.empty
Source = network
Source = security
Level = error
Level = warning
Text = Failed to load resource: the server responded with a status of 404 ()
Text = Error with Permissions-Policy header: Unrecognized feature: 'interest-cohort'.
Timestamp = 1.641199714972959E12
-------------------------------------------
Timestamp = 1.641199714979007E12
-------------------------------------------
2022-01-03 17:48:37.296 DEBUG testcase.CDPViewConsoleLogs - 9: driver.quit()
But I found a problem. My Test Case needed the following import statement.
import org.openqa.selenium.devtools.v96.log.Log;
But Katalon Studio v8.2.1 does not bundle the required jar file to resolve this import statement.
So I had to find out which jar file is required, find out from which repository on the Internet I can get the jar. Once found, I download it and save it into the Drivers folder of my Katalon project manually.
In fact, in order to find it, I created a Gradle project using IntelliJ IDEA where I copy & pasted sample code of some tutorial, and let Gradle to resolve the required dependencies. I found that the jar of Selenium 4 depends on a lot of external jar files. Gradle told me the following dependency tree
+--- org.seleniumhq.selenium:selenium-java:4.1.1
| +--- org.seleniumhq.selenium:selenium-api:4.1.1
| +--- org.seleniumhq.selenium:selenium-chrome-driver:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1
| | | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | | +--- com.google.auto:auto-common:1.2
| | | | \--- com.google.guava:guava:31.0.1-jre
| | | | +--- com.google.guava:failureaccess:1.0.1
| | | | +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
| | | | +--- com.google.code.findbugs:jsr305:3.0.2
| | | | +--- org.checkerframework:checker-qual:3.12.0
| | | | +--- com.google.errorprone:error_prone_annotations:2.7.1
| | | | \--- com.google.j2objc:j2objc-annotations:1.3
| | | \--- com.google.guava:guava:31.0.1-jre (*)
| | +--- com.google.guava:guava:31.0.1-jre (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | +--- org.seleniumhq.selenium:selenium-chromium-driver:4.1.1
| | | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | | +--- com.google.guava:guava:31.0.1-jre (*)
| | | +--- org.seleniumhq.selenium:selenium-json:4.1.1
| | | | \--- org.seleniumhq.selenium:selenium-api:4.1.1
| | | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1
| | | +--- com.beust:jcommander:1.81
| | | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | | +--- com.google.guava:guava:31.0.1-jre (*)
| | | +--- io.netty:netty-buffer:4.1.70.Final
| | | | \--- io.netty:netty-common:4.1.70.Final
| | | +--- io.netty:netty-codec-http:4.1.70.Final
| | | | +--- io.netty:netty-common:4.1.70.Final
| | | | +--- io.netty:netty-buffer:4.1.70.Final (*)
| | | | +--- io.netty:netty-transport:4.1.70.Final
| | | | | +--- io.netty:netty-common:4.1.70.Final
| | | | | +--- io.netty:netty-buffer:4.1.70.Final (*)
| | | | | \--- io.netty:netty-resolver:4.1.70.Final
| | | | | \--- io.netty:netty-common:4.1.70.Final
| | | | +--- io.netty:netty-codec:4.1.70.Final
| | | | | +--- io.netty:netty-common:4.1.70.Final
| | | | | +--- io.netty:netty-buffer:4.1.70.Final (*)
| | | | | \--- io.netty:netty-transport:4.1.70.Final (*)
| | | | \--- io.netty:netty-handler:4.1.70.Final
| | | | +--- io.netty:netty-common:4.1.70.Final
| | | | +--- io.netty:netty-resolver:4.1.70.Final (*)
| | | | +--- io.netty:netty-buffer:4.1.70.Final (*)
| | | | +--- io.netty:netty-transport:4.1.70.Final (*)
| | | | \--- io.netty:netty-codec:4.1.70.Final (*)
| | | +--- io.netty:netty-common:4.1.70.Final
| | | +--- io.netty:netty-transport-classes-epoll:4.1.70.Final
| | | | +--- io.netty:netty-common:4.1.70.Final
| | | | +--- io.netty:netty-buffer:4.1.70.Final (*)
| | | | +--- io.netty:netty-transport:4.1.70.Final (*)
| | | | \--- io.netty:netty-transport-native-unix-common:4.1.70.Final
| | | | +--- io.netty:netty-common:4.1.70.Final
| | | | +--- io.netty:netty-buffer:4.1.70.Final (*)
| | | | \--- io.netty:netty-transport:4.1.70.Final (*)
| | | +--- io.netty:netty-transport-classes-kqueue:4.1.70.Final
| | | | +--- io.netty:netty-common:4.1.70.Final
| | | | +--- io.netty:netty-buffer:4.1.70.Final (*)
| | | | +--- io.netty:netty-transport:4.1.70.Final (*)
| | | | \--- io.netty:netty-transport-native-unix-common:4.1.70.Final (*)
| | | +--- io.netty:netty-transport-native-unix-common:4.1.70.Final (*)
| | | +--- io.netty:netty-transport:4.1.70.Final (*)
| | | +--- io.opentelemetry:opentelemetry-api:1.9.1
| | | | \--- io.opentelemetry:opentelemetry-context:1.9.1
| | | +--- io.opentelemetry:opentelemetry-context:1.9.1
| | | +--- io.opentelemetry:opentelemetry-exporter-logging:1.9.1
| | | | +--- io.opentelemetry:opentelemetry-sdk:1.9.1
| | | | | +--- io.opentelemetry:opentelemetry-api:1.9.1 (*)
| | | | | +--- io.opentelemetry:opentelemetry-sdk-common:1.9.1
| | | | | | +--- io.opentelemetry:opentelemetry-semconv:1.9.1-alpha
| | | | | | | \--- io.opentelemetry:opentelemetry-api:1.9.1 (*)
| | | | | | \--- io.opentelemetry:opentelemetry-api:1.9.1 (*)
| | | | | \--- io.opentelemetry:opentelemetry-sdk-trace:1.9.1
| | | | | +--- io.opentelemetry:opentelemetry-api-metrics:1.9.1-alpha
| | | | | | +--- io.opentelemetry:opentelemetry-api:1.9.1 (*)
| | | | | | \--- io.opentelemetry:opentelemetry-context:1.9.1
| | | | | +--- io.opentelemetry:opentelemetry-semconv:1.9.1-alpha (*)
| | | | | +--- io.opentelemetry:opentelemetry-api:1.9.1 (*)
| | | | | \--- io.opentelemetry:opentelemetry-sdk-common:1.9.1 (*)
| | | | +--- io.opentelemetry:opentelemetry-sdk-metrics:1.9.1-alpha
| | | | | +--- io.opentelemetry:opentelemetry-api-metrics:1.9.1-alpha (*)
| | | | | \--- io.opentelemetry:opentelemetry-sdk-common:1.9.1 (*)
| | | | \--- io.opentelemetry:opentelemetry-sdk-logs:1.9.1-alpha
| | | | +--- io.opentelemetry:opentelemetry-api-metrics:1.9.1-alpha (*)
| | | | +--- com.fasterxml.jackson.core:jackson-databind:2.13.0
| | | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.13.0
| | | | | | \--- com.fasterxml.jackson:jackson-bom:2.13.0
| | | | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.13.0 (c)
| | | | | | +--- com.fasterxml.jackson.core:jackson-core:2.13.0 (c)
| | | | | | \--- com.fasterxml.jackson.core:jackson-databind:2.13.0 (c)
| | | | | +--- com.fasterxml.jackson.core:jackson-core:2.13.0
| | | | | | \--- com.fasterxml.jackson:jackson-bom:2.13.0 (*)
| | | | | \--- com.fasterxml.jackson:jackson-bom:2.13.0 (*)
| | | | \--- io.opentelemetry:opentelemetry-sdk:1.9.1 (*)
| | | +--- io.opentelemetry:opentelemetry-sdk-common:1.9.1 (*)
| | | +--- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.9.1
| | | | \--- io.opentelemetry:opentelemetry-sdk:1.9.1 (*)
| | | +--- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.9.1-alpha
| | | | +--- io.opentelemetry:opentelemetry-semconv:1.9.1-alpha (*)
| | | | +--- io.opentelemetry:opentelemetry-sdk:1.9.1 (*)
| | | | +--- io.opentelemetry:opentelemetry-sdk-metrics:1.9.1-alpha (*)
| | | | \--- io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.9.1 (*)
| | | +--- io.opentelemetry:opentelemetry-sdk-trace:1.9.1 (*)
| | | +--- io.opentelemetry:opentelemetry-sdk:1.9.1 (*)
| | | +--- io.opentelemetry:opentelemetry-semconv:1.9.1-alpha (*)
| | | +--- io.ous:jtoml:2.0.0
| | | +--- net.bytebuddy:byte-buddy:1.12.2
| | | +--- org.apache.commons:commons-exec:1.3
| | | +--- org.asynchttpclient:async-http-client:2.12.3
| | | | +--- org.asynchttpclient:async-http-client-netty-utils:2.12.3
| | | | | +--- io.netty:netty-buffer:4.1.60.Final -> 4.1.70.Final (*)
| | | | | +--- org.slf4j:slf4j-api:1.7.30 -> 1.7.32
| | | | | \--- com.sun.activation:jakarta.activation:1.2.2
| | | | +--- io.netty:netty-codec-http:4.1.60.Final -> 4.1.70.Final (*)
| | | | +--- io.netty:netty-handler:4.1.60.Final -> 4.1.70.Final (*)
| | | | +--- io.netty:netty-codec-socks:4.1.60.Final
| | | | | +--- io.netty:netty-common:4.1.60.Final -> 4.1.70.Final
| | | | | +--- io.netty:netty-buffer:4.1.60.Final -> 4.1.70.Final (*)
| | | | | +--- io.netty:netty-transport:4.1.60.Final -> 4.1.70.Final (*)
| | | | | \--- io.netty:netty-codec:4.1.60.Final -> 4.1.70.Final (*)
| | | | +--- io.netty:netty-handler-proxy:4.1.60.Final
| | | | | +--- io.netty:netty-common:4.1.60.Final -> 4.1.70.Final
| | | | | +--- io.netty:netty-buffer:4.1.60.Final -> 4.1.70.Final (*)
| | | | | +--- io.netty:netty-transport:4.1.60.Final -> 4.1.70.Final (*)
| | | | | +--- io.netty:netty-codec:4.1.60.Final -> 4.1.70.Final (*)
| | | | | +--- io.netty:netty-codec-socks:4.1.60.Final (*)
| | | | | \--- io.netty:netty-codec-http:4.1.60.Final -> 4.1.70.Final (*)
| | | | +--- io.netty:netty-transport-native-epoll:4.1.60.Final
| | | | | +--- io.netty:netty-common:4.1.60.Final -> 4.1.70.Final
| | | | | +--- io.netty:netty-buffer:4.1.60.Final -> 4.1.70.Final (*)
| | | | | +--- io.netty:netty-transport:4.1.60.Final -> 4.1.70.Final (*)
| | | | | \--- io.netty:netty-transport-native-unix-common:4.1.60.Final -> 4.1.70.Final (*)
| | | | +--- io.netty:netty-transport-native-kqueue:4.1.60.Final
| | | | | +--- io.netty:netty-common:4.1.60.Final -> 4.1.70.Final
| | | | | +--- io.netty:netty-buffer:4.1.60.Final -> 4.1.70.Final (*)
| | | | | +--- io.netty:netty-transport:4.1.60.Final -> 4.1.70.Final (*)
| | | | | \--- io.netty:netty-transport-native-unix-common:4.1.60.Final -> 4.1.70.Final (*)
| | | | +--- org.reactivestreams:reactive-streams:1.0.3
| | | | +--- com.typesafe.netty:netty-reactive-streams:2.0.4
| | | | | +--- io.netty:netty-handler:4.1.43.Final -> 4.1.70.Final (*)
| | | | | \--- org.reactivestreams:reactive-streams:1.0.3
| | | | +--- org.slf4j:slf4j-api:1.7.30 -> 1.7.32
| | | | \--- com.sun.activation:jakarta.activation:1.2.2
| | | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | | +--- org.seleniumhq.selenium:selenium-http:4.1.1
| | | | +--- com.google.guava:guava:31.0.1-jre (*)
| | | | +--- net.jodah:failsafe:2.4.4
| | | | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | | | \--- org.seleniumhq.selenium:selenium-json:4.1.1 (*)
| | | \--- org.seleniumhq.selenium:selenium-json:4.1.1 (*)
| | +--- org.seleniumhq.selenium:selenium-json:4.1.1 (*)
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-devtools-v85:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- com.google.guava:guava:31.0.1-jre (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | +--- org.seleniumhq.selenium:selenium-json:4.1.1 (*)
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-devtools-v94:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- com.google.guava:guava:31.0.1-jre (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | +--- org.seleniumhq.selenium:selenium-json:4.1.1 (*)
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-devtools-v95:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- com.google.guava:guava:31.0.1-jre (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | +--- org.seleniumhq.selenium:selenium-json:4.1.1 (*)
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-devtools-v96:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- com.google.guava:guava:31.0.1-jre (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | +--- org.seleniumhq.selenium:selenium-json:4.1.1 (*)
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-edge-driver:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- com.google.guava:guava:31.0.1-jre (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | +--- org.seleniumhq.selenium:selenium-chromium-driver:4.1.1 (*)
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-firefox-driver:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- com.google.guava:guava:31.0.1-jre (*)
| | +--- org.seleniumhq.selenium:selenium-devtools-v85:4.1.1 (*)
| | +--- org.seleniumhq.selenium:selenium-json:4.1.1 (*)
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-firefox-xpi-driver:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- com.google.guava:guava:31.0.1-jre (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | +--- org.seleniumhq.selenium:selenium-firefox-driver:4.1.1 (*)
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-ie-driver:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-opera-driver:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| +--- org.seleniumhq.selenium:selenium-safari-driver:4.1.1
| | +--- com.google.auto.service:auto-service-annotations:1.0.1
| | +--- com.google.auto.service:auto-service:1.0.1 (*)
| | +--- com.google.guava:guava:31.0.1-jre (*)
| | +--- org.seleniumhq.selenium:selenium-api:4.1.1
| | \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
| \--- org.seleniumhq.selenium:selenium-support:4.1.1
| +--- com.google.auto.service:auto-service-annotations:1.0.1
| +--- com.google.auto.service:auto-service:1.0.1 (*)
| +--- com.google.guava:guava:31.0.1-jre (*)
| +--- org.seleniumhq.selenium:selenium-api:4.1.1
| +--- org.seleniumhq.selenium:selenium-json:4.1.1 (*)
| \--- org.seleniumhq.selenium:selenium-remote-driver:4.1.1 (*)
This means that a user who want to utilise the goodness of Selenium 4 (for example, CDP support) have to resolve external dependencies them selves as I did, because Katalon Studio does not help us resolving external dependencies as Maven/Gradle does.
Especially you should note the name of jar file:
org.seleniumhq.selenium:selenium-devtools-v96:4.1.1
Here you find v96. This represents the version of Chrome browser. When Chrome browser is upgraded to a new version sometime near future, a new jar org.seleniumhq.selenium:selenium-devtools-v97 will be published in the Maven Central repository. Then users who want to utilise CDP feature through Seleium 4 in Katalon Studio >8.2.1 have to resolve the dependency themselves manually.
This reminds me that Katalon Studio provides Tools > Upgrade WebDriver > Chrome menu in the tools bar. I suppose we need Tools > Upgrade Selenium 4 DevTools jar as well.
By the way, I found that it is harmless to switch the jar file of Selenium bundled in Katalon Studio from v3 to v4. Selenium v4 is well backward compatible. My old projects safely ran in KS v8.2.1-alpha. Note that DesiredCapability class is deprecated in Selenium 4; no longer to be used by Katalon users.