Katalon Studio and BrowserMob Proxy


#1

Greetings,

I am currently working on a automation project that requires the ability to manipulate request headers to mock authentication (our platform allows access to content based on IP ranges, referer’s, etc…).

I understand that Selenium by design does not allow the modification of headers, but have seen various implementations that use a proxy such as BrowserMob Proxy to intercept the requests for this to be done.

Has anyone successfully integrated BrowserMob Proxy with Katalon to achieve this?

Below is some example code that I’ve tried, both with Firefox and Chrome. I’m not an expert in Java or Groovy, but it seems like for Firefox at least, its not respecting the port I’m trying to start on.

For Firefox, it starts the browser and navigates to the web page, but it looks like it’s listening on a different port than the proxy is on.

Port started:34531172.254.253.150172.254.253.150:345311541618735347	geckodriver	INFO	Listening on 127.0.0.1:33137

For Chrome, I’ve read that it won’t respect the DesiredCapabilties.Proxy setting and you have to pass it in as a option. When I try that it fails with the below error:

Port started:33537172.254.253.150172.254.253.150:33537Starting ChromeDriver 2.29.461591 (62ebf098771772160f391d75e589dc567915b233) on port 22277Only local connections are allowed.14:12:28.370 [LittleProxy-0-ClientToProxyWorker-1] ERROR org.littleshoot.proxy.impl.ClientToProxyConnection - (AWAITING_INITIAL) [id: 0xfee38cc1, L:/172.254.253.150:33537 - R:/172.254.253.150:33553]: Caught an exception on ClientToProxyConnectionjava.lang.NoSuchMethodError: com.google.common.net.HostAndPort.getHostText()Ljava/lang/String;

And my Code:

package bmpimport org.openqa.selenium.Proxyimport org.openqa.selenium.WebDriverimport org.openqa.selenium.firefox.FirefoxDriverimport org.openqa.selenium.remote.CapabilityTypeimport org.openqa.selenium.remote.DesiredCapabilitiesimport com.kms.katalon.core.annotation.Keywordimport io.netty.handler.codec.http.HttpRequestimport io.netty.handler.codec.http.HttpResponseimport net.lightbody.bmp.BrowserMobProxyServerimport net.lightbody.bmp.client.ClientUtilimport net.lightbody.bmp.filters.RequestFilter as RequestFilterimport net.lightbody.bmp.filters.ResponseFilter as ResponseFilterimport net.lightbody.bmp.util.HttpMessageContentsimport net.lightbody.bmp.util.HttpMessageInfopublic  class bmp {	public static WebDriver driver	public static BrowserMobProxyServer proxy	@Keyword	public  proxySetup() {		proxy = new BrowserMobProxyServer()		proxy.setTrustAllServers(true)		proxy.start()		int port = proxy.getPort()		System.out.println("Port started:" + port)		proxy.addRequestFilter(new RequestFilter() {					@Override					public HttpResponse filterRequest(HttpRequest request, HttpMessageContents contents, HttpMessageInfo messageInfo) {						//proxy.addHeader("X-Forwarded_for", "66.249.69.245")						request.headers().add("X-Forwarded_for", "66.249.69.245")						String q = URLDecoder.decode(messageInfo.getOriginalUrl(), "UTF-8");						System.out.println("Request: "+ q );						return null;					}				});		proxy.addResponseFilter(new ResponseFilter() {					@Override					public void filterResponse(HttpResponse response, HttpMessageContents contents, HttpMessageInfo messageInfo) {						headers = proxy.getAllHeaders()						System.out.println("Headers: " + headers)					}				});		Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);		/** * Chrome Spike with options *  **//*		ChromeOptions options = new ChromeOptions();		// Add the WebDriver proxy capability.		String hostIp = Inet4Address.getLocalHost().getHostAddress();			println(hostIp)		seleniumProxy.setHttpProxy(hostIp+":"+ proxy.getPort());			println(hostIp+":"+ proxy.getPort())		seleniumProxy.setSslProxy(hostIp+":"+proxy.getPort());;		options.setCapability("proxy", seleniumProxy);		ChromeDriver driver = new ChromeDriver(options);*/			/**		 * Chrome Spike with desired capabilites		 *		 **//*				 ArrayList<String> switches = new ArrayList<String>();		 String hostIp = Inet4Address.getLocalHost().getHostAddress();		 switches.add("--proxy-server=" + hostIp+":"+proxy.getPort());		 seleniumCapabilities.setCapability("chrome.switches", switches);		 driver  = new ChromeDriver(seleniumCapabilities) */				try {			String hostIp = Inet4Address.getLocalHost().getHostAddress();			println(hostIp)			seleniumProxy.setHttpProxy(hostIp+":"+proxy.getPort());			println(hostIp+":"+proxy.getPort())			seleniumProxy.setSslProxy(hostIp+":"+proxy.getPort());		} catch (UnknownHostException e) {			e.printStackTrace();			println("invalid Host Address")		}				DesiredCapabilities seleniumCapabilities = new DesiredCapabilities()		seleniumCapabilities.setCapability(CapabilityType.PROXY, seleniumProxy)		seleniumCapabilities.setCapability(CapabilityType.ACCEPT_INSECURE_CERTS, true)		driver = new FirefoxDriver(seleniumCapabilities)		driver.get("xxx_Sample_URL_XXX")	}	@Keyword	public static void proxyShutdown() {		//driver.quit()		proxy.stop()	}}

Any help would be greatly appreciated, This is the last hurdle to get wide scale adoption of Katalon at my workplace, as this is a critical aspect of our testing.

Other alternatives appreciated, but I have to be able to set the headers in Katalon. So no external proxies.

Thanks in advance.


#2

Chrome error was fixed by upgrading the the latest BrowserMobProxy jar (2.1.5)

Any thoughts on the driver not respecting the proxy settings?