Problem with jnlp files, I can't remove the pop up

If this is truly a browser-level notification, you will need to set a desired capability to handle it. See this:

https://docs.katalon.com/katalon-studio/docs/introduction-to-desired-capabilities.html#web-testing

Also see my topic on how to figure out which capabilities you need to set: