Headless Chrome fails with NextJS website but headless Firefox (and normal chrome/firefox) work fine

Web Testing
1

This has been a thorn in my side for a while.

Issue: headless chrome seems to take an endless amount of time to load a login page, and so it always times out and fails to find elements.

This does not happen in headless Firefox, or Chrome/Firefox in normal mode.

What I’ve tried so far:

Added a WebUI.delay(500) for example to see if it actually gets the page loaded (this still fails). The waitForElement type methods also fail, along with the verifyElement ones too.

Screenshots taken at the failed test, show a half loaded page mostly.

So it seems like there’s some sort of problem with my headless chrome loading these dynamic pages generated by NextJS.

I’ve tried adding all sorts of Desired Capabilities and none of them made a difference.

image
image697×310 49 KB

I know this has to be something to do with headless chrome, but I’m not entirely sure what. So I’m hoping that someone here has had some experience with this and knows the answer to my problem. As I’ve literally been fighting this for months. It worked fine at one point. Then stopped after a Chrome update (and yes, i do keep my web drivers up to date with the browser updates. Every week I make sure to update both browser and katalon drivers for firefox and chrome.

Anyone got any ideas that are useful?

Thanks in advance.

2 Likes
2
  1. Emulate a real user environment by adding these ChromeOptions flags:
import org.openqa.selenium.chrome.ChromeOptions
import com.kms.katalon.core.webui.driver.DriverFactory
import org.openqa.selenium.chrome.ChromeDriver

ChromeOptions options = new ChromeOptions()
options.addArguments("headless")
options.addArguments("window-size=1920,1080")
options.addArguments("disable-gpu")
options.addArguments("no-sandbox")
options.addArguments("disable-dev-shm-usage")
options.addArguments("enable-features=NetworkService,NetworkServiceInProcess")
options.addArguments("disable-extensions")
options.addArguments("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36")

DriverFactory.changeWebDriver(new ChromeDriver(options))
  • The custom user-agent can avoid bot detection.
  • Flags adjust sandbox and GPU for headless stability.
  1. Add explicit waits for critical page elements beyond WebUI.delay():
WebUI.waitForElementVisible(findTestObject('LoginPage/LoginButton'), 30)

Wait for elements tied to hydration completion.

  1. Try experimental flags:
  • --force-webrtc-ip-handling-policy=default_public_interface_only
  • --ignore-certificate-errors-spki-list
    (These can help with network and security related hangs.)
  1. Test with Chromium vs. Chrome drivers:
1 Like
3

In order to understand your concern, we would need to reproduce this issue on our side and investigate it hands-on. Please provide us enough information how to reproduce the issue on our side.

  1. share the Next.js-based webapp project zipped. You can assume that we have node and npm installed on our side and we can do npm install to recreate the project, npm run dev to launche the React app.
  2. share your Katalon Studio project zipped
1 Like
4

hi @Sixstring

i’m just curious how the execution on the katalon testops using that Chrome-headless issue profile. can you try to run your test on the katalon testops?

1 Like
5

Unfortunately I cannot run executions on testops. I’m only on the free version currently.

6

Thanks, dineshh. I’ll give some of these flags a try and see if anything helps here. If it does, I’ll make sure to update this thread with the info so others can benefit from it too.

1 Like
7

It appears that something with using the ChromeOptions rather than desired capabilities seems to be working. So now i’ll troubleshoot what was causing my issue there.

These are the ones I used:

ChromeOptions options = new ChromeOptions()
options.addArguments("headless")
options.addArguments("window-size=1920,1080")
options.addArguments("disable-gpu")
options.addArguments("no-sandbox")
options.addArguments("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36")

WebDriver driver = new ChromeDriver(options)

DriverFactory.changeWebDriver(driver)

If i can’t get these working in Desired Capabilities properly, then I’ll just set something up in my listener setup to only set these options when the browser is set to Chrome Headless.

Hopefully that is the end of my woes with headless chrome :smiley:

Thanks for those that gave some input. I appreciate it.