Download file in chrome headless custom folder

[joost.degeyndt]

Hi there,
(First topic. I am using KS free for a year now and recently purchased a licenced KSE version. First topic creation for me here )

I have a script to download a file, store it in a custom folder and the check it’s indeed downloaded. It works fine on Chrome, but on Chrome headless, with the same desired capabilities it doesn’t work. The file does not seem to be downloaded. I use enough sleep time to make sure it’s get downloaded, but still nothing. I assume it doesn’t get downloaded as A) It doesn’t get saved in the wanted custom folder & B) cannot find the file anywhere else

I have searched the community and found this topic: Downloading the files in headless chrome but I lack a bit the information how to apply this to the desired capabilities.

My setup: ChromeDriver 81.0.4044.138 - browser = Chrome 81.0.4044.138 (headless)
What I tried:
From above topic I took:
chromePreferences.put(“profile.default_content_settings.popups”, 0);
chromePreferences.put(“download.prompt_for_download”, “false”);
chromePreferences.put(“download.default_directory”, downloadFilepath);
…and translated this to desired capabilities settings:

To me it’s not always clear how to translate these preferences into the desired capabilities (prefs dictionay? args? Anyway, I also tried some other setting as well (like path C:\\KatalonDownloads, (escaping additional \ ) but no luck.

Topic: Download files in Chrome headless also raised this question but no answer was provided.

Secondly, I’d like to have that custom download folder dynamicly. As I understood, it’s currently not possible to make a reference to (for instance) a global variable in the desired Capabilities settings, correct? Is there some other workaround possible?

Thanks a million in advance!

[Timo_Kuisma1]

hi,
i guess that i have an solution for that,
wait few hours when i am back to home, now i am in office

[Timo_Kuisma1]

hello,
this is example how to download file to wanted folder with chome headless

import org.apache.hc.client5.http.classic.HttpClient
import org.apache.hc.client5.http.classic.methods.HttpPost
import org.apache.hc.client5.http.impl.classic.HttpClientBuilder
import org.apache.hc.core5.http.io.entity.StringEntity
import org.openqa.selenium.chrome.ChromeDriver
import org.openqa.selenium.chrome.ChromeDriverService
import org.openqa.selenium.chrome.ChromeOptions
import org.openqa.selenium.remote.CapabilityType
import org.openqa.selenium.remote.DesiredCapabilities

import com.fasterxml.jackson.databind.ObjectMapper
import com.kms.katalon.core.webui.driver.DriverFactory
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

String downloadFilepath = "C:\\Users\\fitim\\Desktop\\data\\pdf\\"
HashMap<String, Object> chromePreferences = new HashMap<String, Object>();
chromePreferences.put("profile.default_content_settings.popups", 0);
chromePreferences.put("download.prompt_for_download", "false");
chromePreferences.put("download.default_directory", downloadFilepath);
ChromeOptions chromeOptions = new ChromeOptions();
System.setProperty("webdriver.chrome.driver", DriverFactory.getChromeDriverPath())

chromeOptions.addArguments("start-maximized");
chromeOptions.addArguments("disable-infobars");

//HEADLESS CHROME
chromeOptions.addArguments("headless");

chromeOptions.setExperimentalOption("prefs", chromePreferences);
DesiredCapabilities cap = DesiredCapabilities.chrome();
cap.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
cap.setCapability(ChromeOptions.CAPABILITY, chromeOptions);

ChromeDriverService driverService = ChromeDriverService.createDefaultService();
ChromeDriver driver = new ChromeDriver(driverService, chromeOptions);

Map<String, Object> commandParams = new HashMap<>();
commandParams.put("cmd", "Page.setDownloadBehavior");
Map<String, String> params = new HashMap<>();
params.put("behavior", "allow");
params.put("downloadPath", downloadFilepath);
commandParams.put("params", params);
ObjectMapper objectMapper = new ObjectMapper();
HttpClient httpClient = HttpClientBuilder.create().build();
String command = objectMapper.writeValueAsString(commandParams);
String u = driverService.getUrl().toString() + "/session/" + driver.getSessionId() + "/chromium/send_command";
HttpPost request = new HttpPost(u);
request.addHeader("content-type", "application/json");
request.setEntity(new StringEntity(command));
try {
	httpClient.execute(request);
	driver.get("https://docs.oracle.com/javaee/7/JEETT.pdf");
	WebUI.delay(30)
	System.out.println("Task complete, please go to save folder to see it.");
	driver.close()
} catch (IOException e2) {
	e2.printStackTrace();
}

[Russ_Thomas]

Desired Capabilities - The Missing Manual

One of the biggest hurdles with using Katalon/Webdriver is setting the appropriate Desired Capabilities that configure various browser settings programmatically. There are a couple of reasons for this: 1.) Browsers do not expose these back-end properties in the settings menu. Instead, they offer a user-friendly GUI for configuring settings that are based on these properties. This of course makes sense for 99.9999% of browser users, but isn’t helpful for us. 2.) The appropriate Desired Capabili…

[joost.degeyndt]

Thank you @Timo_Kuisma1. But like I mentioned in my post: I did already try to apply your solution via the Katalon project settings > Desired Capabilities for Chrome (headless). See my supplied screenshot above.
I can see below settings in my settings\internal com.kms.katalon.core.webui.chrome (headless)

{"HEADLESS_DRIVER":{"prefs":{"download.prompt_for_download":false,"download.directory_upgrade":true,"safebrowsing.enabled":false,"safebrowsing.disable_download_protection":true,"download_dir":"C:\\KatalonDownloads","profile.default_content_settings.popups":"0","download.default_directory":"C:\\KatalonDownloads","download":{"default_directory":"C:\\KatalonDownloads"}},"excludeSwitches":["enable-automation"],"args":["--disable-infobars"]}}

I tried also “profile.default_content_settings.popups” as a number which is then set to 0.0 by Katalon. Doesn’t seem to download it. The click action was successful, I left enough time to allow to finish the download,… Works on Chrome, not on headless Chrome.

@Russ_Thomas : Thanks for your reply. I did however already read your handy topic, but again, it works fine on normal Chrome (with the below settings), but it’s the Chrome Headless where it doesn’t.

{"CHROME_DRIVER":{"excludeSwitches":["enable-automation"],"useAutomationExtension":false,"prefs":{"download.default_directory":"C:\\KatalonDownloads"}}}

As a workaround, I tried to get it working for Firefox. But there I get annoyed by the save/open dialog which I can’t seem to get through. I supplied following desired capabilities (like suggested in topic Automatic download on firefox and here https://stackoverflow.com/questions/36309314/set-firefox-profile-to-download-files-automatically-using-selenium-and-java ), but my profile does NOT seem to be applied when running in Katalon. Can’t figure out why… These are my firefox desired capabilites settings:

{"FIREFOX_DRIVER":{"firefox_binary":"C:\\Users\\ABC\\AppData\\Local\\Mozilla Firefox\\firefox.exe","firefox_profile":{"browser.download.folderList":2.0,"browser.helperApps.alwaysAsk.force":false,"browser.download.manager.showWhenStarting":false,"browser.download.dir":"C:\\KatalonDownloads","browser.download.downloadDir":"C:\\KatalonDownloads","browser.download.defaultFolder":"C:\\KatalonDownloads","browser.helperApps.neverAsk.saveToDisk":"application/pdf;charset\u003dUTF-8","security.insecure_field_warning.contextual.enabled":false,"browser.helperApps.neverAsk.openFile":"application/pdf;charset\u003dUTF-8","marionette":true,"pdfjs.disabled":true}}}

When I debug and click manually on the download button in the firefox popup I notice it doesn’t even get downloaded in my expected download.dir C:\KatalonDownloads. So I suspect my profile to not be applied for Firefox. Also here, no idea why. This is the log message in the Console:

2020-05-20 14:40:48.931 INFO c.k.katalon.core.main.TestCaseExecutor - START Test Cases/Projects/XYZ/Test/Functional/downloadFullRequest
2020-05-20 14:40:49.803 INFO c.k.katalon.core.main.TestCaseExecutor - (Default) downloadRequestType = fullRequest
2020-05-20 14:40:49.952 INFO com.kms.katalon.core.util.KeywordUtil - testCaseContext.getTextCaseId()=downloadFullRequest
2020-05-20 14:40:51.253 INFO c.k.k.core.webui.driver.DriverFactory - Starting ‘Firefox’ driver
mei 20, 2020 2:40:51 PM org.openqa.selenium.remote.DesiredCapabilities firefox
INFO: Using new FirefoxOptions() is preferred to DesiredCapabilities.firefox()
2020-05-20 14:40:51.591 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set preference: [‘firefox_binary’, ‘C:\Users\ABC\AppData\Local\Mozilla Firefox\firefox.exe’]
2020-05-20 14:40:51.592 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘browser.download.folderList’, ‘2’]
2020-05-20 14:40:51.594 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘browser.helperApps.alwaysAsk.force’, ‘false’]
2020-05-20 14:40:51.595 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘browser.download.manager.showWhenStarting’, ‘false’]
2020-05-20 14:40:51.595 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘browser.download.dir’, ‘C:\KatalonDownloads’]
2020-05-20 14:40:51.618 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘browser.download.downloadDir’, ‘C:\KatalonDownloads’]
2020-05-20 14:40:51.621 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘browser.download.defaultFolder’, ‘C:\KatalonDownloads’]
2020-05-20 14:40:51.623 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘browser.helperApps.neverAsk.saveToDisk’, ‘application/pdf;charset=UTF-8’]
2020-05-20 14:40:51.625 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘security.insecure_field_warning.contextual.enabled’, ‘false’]
2020-05-20 14:40:51.626 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘browser.helperApps.neverAsk.openFile’, ‘application/pdf;charset=UTF-8’]
2020-05-20 14:40:51.627 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘marionette’, ‘true’]
2020-05-20 14:40:51.628 INFO c.k.k.c.w.util.WebDriverPropertyUtil - User set Firefox profile preference: [‘pdfjs.disabled’, ‘true’]
2020-05-20 14:40:51.630 INFO c.k.k.core.webui.driver.DriverFactory - Action delay is set to 0 milliseconds
1589978462305 mozrunner::runner INFO Running command: “C:/Users/ABC/AppData/Local/Mozilla Firefox/firefox.exe” “-marionette” “-foreground” “-no-remote” “-profile” “C:\Users\ABC~1\AppData\Local\Temp\rust_mozprofile.KwcXF9b8qprj”
1589978463412 addons.webextension. WARN Loading extension ‘null’: Reading manifest: Warning processing permissions: Error processing permissions.12: Value “debugger” must either: must either [must either [be one of [“clipboardRead”, “clipboardWrite”, “geolocation”, “idle”, “notifications”], be one of [“bookmarks”], be one of [“find”], be one of [“history”], be one of [“menus.overrideContext”], be one of [“search”], be one of [“topSites”], be one of [“activeTab”, “tabs”, “tabHide”], be one of [“browserSettings”], be one of [“cookies”], be one of [“downloads”, “downloads.open”], be one of [“privacy”], be one of [“webNavigation”], or be one of [“webRequest”, “webRequestBlocking”]], be one of [“alarms”, “mozillaAddons”, “storage”, “unlimitedStorage”], be one of [“browsingData”], be one of [“captivePortal”], be one of [“devtools”], be one of [“identity”], be one of [“menus”, “contextMenus”], be one of [“normandyAddonStudy”], be one of [“pkcs11”], be one of [“sessions”], be one of [“urlbar”], be one of [“geckoProfiler”], be one of [“contextualIdentities”], be one of [“dns”], be one of [“activityLog”], be one of [“management”], be one of [“networkStatus”], be one of [“proxy”], be one of [“nativeMessaging”], be one of [“telemetry”], be one of [“theme”], or match the pattern /^experiments(.\w+)+$/], or must either [be one of ["<all_urls>"], must either [match the pattern /^(https?|wss?|file|ftp|*)://(*|*.[^/]+|[^/]+)/.$/, or match the pattern /^file:///.$/], or match the pattern /^resource://(*|*.[^/]+|[^/]+)/.$|^about:/]
1589978463413 addons.webextension. WARN Loading extension ‘null’: Reading manifest: Warning processing offline_enabled: An unexpected property was found in the WebExtension manifest.
1589978463534 addons.webextension.doh-rollout@mozilla.org WARN Loading extension ‘doh-rollout@mozilla.org’: Reading manifest: Invalid extension permission: networkStatus
1589978463582 addons.webextension.screenshots@mozilla.org WARN Loading extension ‘screenshots@mozilla.org’: Reading manifest: Invalid extension permission: mozillaAddons
1589978463582 addons.webextension.screenshots@mozilla.org WARN Loading extension ‘screenshots@mozilla.org’: Reading manifest: Invalid extension permission: telemetry
1589978463583 addons.webextension.screenshots@mozilla.org WARN Loading extension ‘screenshots@mozilla.org’: Reading manifest: Invalid extension permission: resource://pdf.js/
1589978463583 addons.webextension.screenshots@mozilla.org WARN Loading extension ‘screenshots@mozilla.org’: Reading manifest: Invalid extension permission: about:reader
1589978465021 addons.webextension.{54dcd9e9-903f-43e9-95f0-22bd0e0f3c0a} WARN Loading extension ‘{54dcd9e9-903f-43e9-95f0-22bd0e0f3c0a}’: Reading manifest: Warning processing permissions: Error processing permissions.12: Value “debugger” must either: must either [must either [be one of [“clipboardRead”, “clipboardWrite”, “geolocation”, “idle”, “notifications”], be one of [“bookmarks”], be one of [“find”], be one of [“history”], be one of [“menus.overrideContext”], be one of [“search”], be one of [“topSites”], be one of [“activeTab”, “tabs”, “tabHide”], be one of [“browserSettings”], be one of [“cookies”], be one of [“downloads”, “downloads.open”], be one of [“privacy”], be one of [“webNavigation”], or be one of [“webRequest”, “webRequestBlocking”]], be one of [“alarms”, “mozillaAddons”, “storage”, “unlimitedStorage”], be one of [“browsingData”], be one of [“captivePortal”], be one of [“devtools”], be one of [“identity”], be one of [“menus”, “contextMenus”], be one of [“normandyAddonStudy”], be one of [“pkcs11”], be one of [“sessions”], be one of [“urlbar”], be one of [“geckoProfiler”], be one of [“contextualIdentities”], be one of [“dns”], be one of [“activityLog”], be one of [“management”], be one of [“networkStatus”], be one of [“proxy”], be one of [“nativeMessaging”], be one of [“telemetry”], be one of [“theme”], or match the pattern /^experiments(.\w+)+$/], or must either [be one of ["<all_urls>"], must either [match the pattern /^(https?|wss?|file|ftp|*)://(*|*.[^/]+|[^/]+)/.$/, or match the pattern /^file:///.$/], or match the pattern /^resource://(*|*.[^/]+|[^/]+)/.$|^about:/]
1589978465021 addons.webextension.{54dcd9e9-903f-43e9-95f0-22bd0e0f3c0a} WARN Loading extension ‘{54dcd9e9-903f-43e9-95f0-22bd0e0f3c0a}’: Reading manifest: Warning processing offline_enabled: An unexpected property was found in the WebExtension manifest.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can’t find profile directory.
1589978469736 Marionette INFO Listening on port 56800
1589978470263 Marionette WARN TLS certificate errors will be ignored for this session
1589978470265 Marionette INFO Proxy settings initialised: {“proxyType”:“direct”}
mei 20, 2020 2:41:10 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
2020-05-20 14:41:10.658 INFO c.k.k.core.webui.driver.DriverFactory - sessionId = e8a00148-0a6c-4e7a-b22e-f327c6710143
2020-05-20 14:41:10.695 INFO c.k.k.core.webui.driver.DriverFactory - browser = Firefox 76.0
2020-05-20 14:41:10.696 INFO c.k.k.core.webui.driver.DriverFactory - platform = Windows 10
2020-05-20 14:41:10.697 INFO c.k.k.core.webui.driver.DriverFactory - seleniumVersion = 3.141.59
2020-05-20 14:41:10.698 INFO c.k.k.core.webui.driver.DriverFactory - proxyInformation = ProxyInformation{proxyOption=NO_PROXY, proxyServerType=HTTP, username=, password=*****, proxyServerAddress=, proxyServerPort=0, exceptionList=}
JavaScript error: resource://gre/modules/NetworkGeolocationProvider.jsm, line 555: TypeError: xhr.response.location is undefined

It does show a bunch of warnings and at the end a JavaScript error, but don’t think those are related? After that my tests loads fine up until where I expect my download to be automatically downloaded. Yet I’m blocked by that firefox popup

I tried to go and check that temp profile mentioned in the console log
C:\Users\ABC~1\AppData\Local\Temp\rust_mozprofile.KwcXF9b8qprj and indeed, I do not retrieve the prefs settings there in the prefs.js file. Might that be a clue my custom profile is not applied?

Sorry for the long post, but I tried to be as detailed as possible. Any help here would be greatly appreciated!!!

[Timo_Kuisma1]

hi,
did you tried my example code in KS script view window?
only copy and paste and run with chrome
Not tried to do this with FF never, needed new code for that

[joost.degeyndt]

I appreciate your help @Timo_Kuisma1, but I am looking for a solution that integrates with the build in functionalities of Katalon. When using your approach I think I’ll lose the ability to run my test on different browsers.
Also, I did try your script in a standalone script test case (run with Chrome Headless, no?, you said Chrome). Anyhow, I don’t get it working as there seem to be something wrong with the imports. I get the error: unable to resolve class org.apache.hc.client5.http.classic.methods.HttpPost
Indeed, I can see the IDE indicating it doesn’t know the .create action for the HttpClientBuilder

Auto import didn’t provide any fix.

[Timo_Kuisma1]

hi,
ouh, this .jar need to be in Drivers folder
https://jar-download.com/artifacts/org.apache.httpcomponents.client5/httpclient5/5.0-beta4/source-code

[joost.degeyndt]

Again, thanks for your time @Timo_Kuisma1, but I expect Katalon to have an “out-of-the-box” possibility to download files in an headless browser through the custom capabilities. I don’t want to go through the path of full customization as in the end we might just as well use Selenium.

So for now, I am blocked in downloading a pdf in a headless browser for Chrome and Firefox.