Cloudflare is now blocking Katalon on certain websites. I confirmed this with 3 browsers (Firefox, Chrome, and Edge). I can write a simple test in Katalon that will open a URL on a site that is doing this. The Katalon controlled browser will instantly get stuck in an endless “Checking your browser before allowing you to access the site” loop from cloudfare. Katalon will never get past this part and you are denied access to the website. On the same computer, I can open the browser on its own and type in the same site URL, and it will open without a problem. Everything is 100% the same, same computer, same ip, same browser, same time, the only difference is one browser is opened through Katalon, and the other manually. Katalon will never access the website, I’ve tried for 4 hours.
That means some sites are now able to detect you’re using Katalon, and deny you access to the site as a result.
And there was talk about some of these settings on this thread here:
However nothing on either thread has worked on whatever they are doing now. If I use Katalon, doesn’t matter what my browser is, or my ip, or what computer I’m on, you cannot access the site. If I open the same browser, on the same pc, on the same IP, I can access the site without issue.
So based on the above threads, this code sets what they say would fix the problem:
System.setProperty(“webdriver.chrome.driver”, DriverFactory.getChromeDriverPath())
ChromeOptions options = new ChromeOptions()
options.setExperimentalOption(‘excludeSwitches’, [‘enable-automation’])
options.setExperimentalOption(‘useAutomationExtension’, false)
options.addArguments("–disable-blink-features=AutomationControlled")
WebDriver driver = new ChromeDriver(options)
DriverFactory.changeWebDriver(driver)
This doesn’t work, there is a new way that cloudflare is detecting Katalon usage, and when someone is using this feature, you can’t access the website.
I read this, and understood that Cloudflare CDN (Contents Delivery Network) can block any Selenium-based automation tools including Katalon.
With many websites implementing anti-bot mechanisms, Undetected Chromedriver in Java is a useful addition to any web scraping arsenal. However, it isn’t foolproof and doesn’t work against advanced anti-bot protection.
I found this artcile very interesting. So, I created a Gradle project with Selenium 4.0 + Undetected Chromedriver. I copy&pasted the sample Selenium test. It worked. The test tried to open a web site “NowSecure” which is guarded by Cloudflare. The test could reach to the web site behind the Cloudflare CDN.
How does it work?
The Cloudflare CDN can distinguish two groups of HTTP clients — a HTTP client driven by automation software like Katalon Studio, and a HTTP client driven by human.
How does Cloudflare detect the type runtime? Cloudflare examines the HTTP headers of requests from browsers. Among the HTTP headers, some types headers carry information that tells the type of HTTP client — if the client is automation software or human. When Cloudflare could identify a request raised by automation software, then Cloudflare will reject it.
How can the “Undetected Chromedriver” cheat Cloudflare? — See the following project:
The Undetected Chromedriver let a Chrome browser to execute the JavaScript stealth.min.js which affects the internal of javascript rutime environment. Chrome is told to upload a set of specially tailored HTTP headers to a HTTP server. The tailored HTTP headers will look very similar to the one when a human manually operated the browser.
