Could you please share your test URL.
The below is sample URL with Windows Authentication popup. Please check if it is working in Chrome browser for you. I have no success with Firefox.
http://admin:admin@the-internet.herokuapp.com/basic_auth
// Method -1 (Works in Chrome only)
WebUI.openBrowser('http://admin:admin@the-internet.herokuapp.com/basic_auth')
// Method -2
WebUI.openBrowser('')
WebUI.authenticate('http://the-internet.herokuapp.com/basic_auth', 'admin', 'admin', 10, FailureHandling.OPTIONAL)