Open Browser with Custom Profile

What if a custom profile created need to run via Chrome headless Browser.

I tried Running the same script its working fine for normal chrome, when i used the same with Profile its opening the chrome.

1 Like

Hey ThanhTo, thanks much for this code, although I’m able to successfully launch the Chrome browser with a new custom profile, Katalon is displaying the following failure:
org.openqa.selenium.WebDriverException: Timed out waiting for driver server to start.

The only change I made to your code was the pathToChromeDriver, I’m using my chrome executalbe at C: Program Files …, but that shouldn’t be causing this error, right??

1 Like

I keep getting this from the above code - please be gentle with me I am very very new to this and in the process of teaching myself.

Reason:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
file:/C:/Users/deric/Katalon%20Studio/TN/TrueNorth/Scripts/TrueNorth/TNGTCLogin/Script1636153465987.groovy: 26: unable to resolve class ChromeOptions
@ line 26, column 15.
ChromeOptions chromeProfile = new ChromeOptions();
^

file:/C:/Users/deric/Katalon%20Studio/TN/TrueNorth/Scripts/TrueNorth/TNGTCLogin/Script1636153465987.groovy: 26: unable to resolve class ChromeOptions
@ line 26, column 31.
ChromeOptions chromeProfile = new ChromeOptions();
^

file:/C:/Users/deric/Katalon%20Studio/TN/TrueNorth/Scripts/TrueNorth/TNGTCLogin/Script1636153465987.groovy: 31: unable to resolve class WebDriver
@ line 31, column 11.
WebDriver driver = new ChromeDriver(chromeProfile);
^

file:/C:/Users/deric/Katalon%20Studio/TN/TrueNorth/Scripts/TrueNorth/TNGTCLogin/Script1636153465987.groovy: 31: unable to resolve class ChromeDriver
@ line 31, column 20.
WebDriver driver = new ChromeDriver(chromeProfile);

Please can you help - all I want is to open chrome with the default profile.
there is an option to launch Chrome (with profile) but it is only available in the web recorder for some reason. and it does not work - it does not open chrome with my normal profile and settings and extensions , still just opens a blank chrome in dark mode with no profile or extensions

1 Like

https://www.selenium.dev/selenium/docs/api/java/org/openqa/selenium/chrome/ChromeOptions.html

https://www.selenium.dev/selenium/docs/api/java/org/openqa/selenium/WebDriver.html

you need to add two import statements:

import org.openqa.selenium.chrome.ChromeOptions
import org.openqa.selenium.WebDriver
1 Like

Thanks @ThanhTo It worked. Although am able to open the browser with a custom profile, its showing below error
Reason:
com.kms.katalon.core.webui.exception.BrowserNotOpenedException: Browser is not opened

Looks like am getting error in the below code :
WebDriver webDriver = DriverFactory.changeWebDriver(driver)

1 Like

Its underlined

How to use several browsers with different profiles simultaneously?

1 Like

Please read the doc of my “chromedriverfactory” project.

In this document, please find a sample code titled " Launch Chrome browser with UserProfile specified". Please look at a @Test method named void test_launch_browser_with_profile_TO_GO() . This method opens a single Chrome browser using a profile named “Picasso”. Just in the same way using any profile names, you can open 2 or more chrome browsers simultatiously.

In the code, the following fragment is important.

                ChromeDriverFactory.UserDataAccess.TO_GO)

please read the doc what this fragment means.

1 Like

You can download the latest version of the jar of “chromedriverfactory” here

1 Like

See also this:

This shows a Katalon Studio project that opens 2 chrome browsers while specifiying 2 difference profiles and perform test simultaneously.

WARNING This sample project used the artifact of my “WebDriverFactory4ks” project. But I have already abondoned the “WebDriverFactory4ks” project because there were bugs due to poor design failures. I would not maintain “WebDriverFactory4ks” project anylonger.

I have created another project “chromedriverfactory” project where I fixed problems as far as I could. If you are willing to try, please use the “chromedriverfactory” in favour of “WebDriverFactory4ks”

1 Like

Hello kazurayam,

We’ve been using code copied from an old contribution of yours, as a Chrome Profile Factory, for almost two years, since Autumn 21. This has been very helpful to us. Merci encore !!

It does work very well, but recently, I’ve hit a limitation, which we had seen in Winter 22, which I haven’t had the time to investigate further.

We use Microsoft Windows.

We are in the situation of getting Katalon working on a copied profile (what you now call the “TO_GO” option).

The original profile is:
AppData\Local\Google\Chrome\User Data\Profile 21
It does contain the cookies (SQLite database file) as:
AppData\Local\Google\Chrome\User Data\Profile 21\Network\Cookies

The profile factory creates a copy of the entire profile directory, like:
AppData\Local\Temp\User Data8672441348483056915
When the initial copy is done, if you get Katalon to stop, we can control that the copy was successful. The cookies are in:
AppData\Local\Temp\User Data8672441348483056915\Network\Cookies

But, after the browser has been instantiated, when Katalon can move to the next step after WebDriver myBrowser = WebDriverFactory.newWebDriver(DriverFactory.getExecutedBrowser(), “my profile name)” … we observe that the file has been modified. And the precious cookies are gone. Re-opening the Cookies file with our favorite SQLite tool shows it’s just empty!

Why would Chrome delete the cookies upon starting a profile in this situation (operating on a copy, started by ChromeDriver, etc)? For some sort of “security” reason?

Whatever the reason, my issue is to keep my cookies!

I’ve spent a few hours looking for a clue, typically musing about a possible ChromeDriver command-line switch that would allow to skip this behavior (List of Chromium Command Line Switches « Peter Beverloo), but I’ve not found. I’m still looking for a way to get the Chrome profile copy started with the cookies of the original profile.

If you have any idea, or suggestion or guess or question or whatever, I would greatly appreciate it.

Thanks in advance,
– Michel

1 Like

I have 2 GitHub repositories that concern opening browser with custom profile.

The #1 is an old project, which I have already abandoned maintenance. You seem to be using the #1 project.

The #2 is the successor of the #1 which I am willing to maintain for issues.

@michel.g

I would not maintain the #1 project any longer. Please consider switching to the #2 project.

I am reluctant to look into your question about the #1 project.

1 Like

I renamed the project “WebDriverFactory4ks” to “chromedriverfactory”. Why I abandoned the #1? I had 2 reasons.

  1. I wanted my library to have no dependency on the com.kms.katalon.** packages. It should be independent from Katalon Studio so that I could use it in a pure Gradle projects (non Katalon projects). See issue13;

  2. I gave up working on multiple types of browsers. I worked only on Chrome. Firefox and Edge are not supported. Why? I found that Firefox has a completely different architecture of custom profile from the one of Chome. I could not support Firefox and Chrome by a single codeset. Therefore I changed (shrinked) the project name: “WebDriverFactory” → “chromedriverfactory”. What about Edge Chromium? — Well, I don’t use Edge, so I am not interested in working on it. I learned that it is a difficult and tiring task to develop a single library that claims support for multiple types of browsers.

1 Like

Hello kazurayam,

I understand the history of your contributions, yet my issue is possibly not bound to using an older version. I’ve read the latest version: the logic, for the part which I’m investigating, is the same.

My question is more general.

a) I have a Chrome profile “MyProfName”, whose data is in the normal “User Data” file structure:
AppData\Local\Google\Chrome\User Data\Profile 21

b) I get this “Profile 21” directory copied into a temporary file structure:
AppData\Local\Temp\MyUserData12345etc\MyProfName

The original Cookies file (SQLite database) has been copied as part of the process:

From:   AppData\Local\Google\Chrome\User Data\Profile 21\Network\Cookies
To:     AppData\Local\Temp\MyUserData12345etc\MyProfName\Network\Cookies

c) The copy has been successful: If I had stopped execution after the copy (breakpoint), I can verify its contents using my favorite SQLite tool.

d) Then the ChromeDriver is instantiated, with various options including:
[window-size=1024,768, --no-sandbox, disable-infobars, disable-gpu, disable-dev-shm-usage, --disable-features=ChromeWhatsNewUI, --ignore-certificate-errors, user-data-dir=C:\Users\MICHEL~1\AppData\LocalTemp\MyUserData12345etc, profile-directory=MyProfName]

e) Later on, after the WebDriver is created, I observe that the Cookies file has been altered. The SQLite database has been emptied. :astonished:

My question is: How to tell the WebDriver instantiation process not to empty the Cookies?

I’ve looked for switches/options/preferences that would drive this behavior, but I haven’t found.

Any idea?

Cheers,
–Michel

1 Like

Which version of Chrome are you using? Prior to v115, or newer?
Have you updgraded it recently?

1 Like

I believe that ChromeDriver does nothing to the Cookies. Only Chrome browser manipulates the Cookie, not ChromeDriver.


Each cookie has its own “Time to Live” (TTL). A TTL can be as short as 1 seconds, 1minutes, 1hour, 1day, 1year,… If the TTL of your cookie is too short, then Chrome will erase the expired Cookie out of the database when activated. Have you checked the TTL of your cookie?


I tried the junit test set for the chromedriverfactory project with Chrome v116. It passed. I could not reproduce what you observed.


If your observation is right, I guess that Chrome browser of newer version might have changed its internal behavior — it has become “more secure”.

I know, the development team of Google Chrome are very strict and defensive about the risk of malicious programmers to drive Chrome browser.

I guess that Chrome browser do not like to be controlled by WebUI testers. Chrome browser tries to detect malicious invasion by WebUI testers. It is likely that it empties the any Cookies prepared in the user-data-dir when it recognized (I don’t know how) that WebUI testers are trying to trick Chrome.

This is just what I guess. I have no evidence.

In case my guess is right, I am negative if I could trick Chrome any longer.

1 Like

Hello kazurayam,

Thanks for your many interesting suggestions.

I’ve had first observed “my” issue in Dec 21, so I don’t think it’s linked to some Chrome version/update. I run 116.0.5845.111 this week.

The persistent cookie I would like to operate has a 1 week TTL. I’ve checked again in the last profile I’ve used: that cookie expires at 13337974245878000, which in Chrome time standard means Aug 31st.

I’ve read carefully your unit test code (CarryingCookieOverSessionsViaProfileTest.groovy). Very interesting.

What is the host_key for your “timestamp” test cookie? Is it “127.0.0.1”? Mine comes from another server in my company’s private cloud, with a host_key like “blabla.server.int.mycompany.cloud”. Maybe this is why Chrome deletes mine, while it does not delete yours?

Overall, I agree with your general conclusion: We are probably facing some “security decision” of the “Google Fortress”, and we can probably do very little against it.

Have a nice week-end.
–Michel

1 Like

I executed the com.kazurayam.webdriverfactory.chrome.CarryingCookiesOverSessionsViaProfileTest using Chrome v116 + the latest chromedriver in IntelliJ IDEA.

The test class printed the following output into the console:


> Task :compileJava UP-TO-DATE
> Task :compileGroovy NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :compileTestJava NO-SOURCE
> Task :compileTestGroovy UP-TO-DATE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :test
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/Users/kazuakiurayama/github/dotfiles/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy-all/2.4.21/8e4f4c30dbb9123fbf703f256cd721bbac5c902a/groovy-all-2.4.21.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
SLF4J: Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
SLF4J: Ignoring binding found at [jar:file:/Users/kazurayam/github/dotfiles/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-simple/1.7.25/8dacf9514f0c707cbbcdd6fd699e8940d42fb54e/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
Starting ChromeDriver 116.0.5845.96 (1a391816688002153ef791ffe60d9e899a71a037-refs/branch-heads/5845@{#1382}) on port 18769
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
8月 26, 2023 5:37:12 午前 org.openqa.selenium.remote.ProtocolHandshake createSession
情報: Detected dialect: W3C
options => Capabilities {browserName: chrome}
userProfile => "{\"userProfile\":\"Picasso\",\"userDataDir\":\"/Users/kazurayam/Library/Application Support/Google/Chrome\",\"profileDirectoryName\":\"Profile 14\"}"
userDataAccess => FOR_HERE


<<<< Request received
{
"method": "GET",
"uri": "/",
"headers": [
    {"Accept-encoding": ["gzip, deflate, br"]}
    {"Sec-ch-ua": ["\"Not)A;Brand\";v=\"24\", \"Chromium\";v=\"116\""]},
    {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"]},
    {"Sec-fetch-dest": ["document"]},
    {"Sec-fetch-user": ["?1"]},
    {"Connection": ["keep-alive"]},
    {"Host": ["127.0.0.1"]},
    {"Sec-fetch-site": ["none"]},
    {"Sec-ch-ua-platform": ["\"macOS\""]},
    {"Sec-fetch-mode": ["navigate"]},
    {"User-agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"]},
    {"Accept-language": ["ja,en-US;q=0.9,en;q=0.8"]},
    {"Upgrade-insecure-requests": ["1"]},
    {"Sec-ch-ua-mobile": ["?0"]},
],
"body": """"""
}
==== Cookies cooked
in the request:  no cookies found
in the response: cookies=[timestamp=Fri, 25 Aug 2023 20:37:13 GMT; Max-Age=60;]

>>>> Response sent
/ is a directory.
Starting ChromeDriver 116.0.5845.96 (1a391816688002153ef791ffe60d9e899a71a037-refs/branch-heads/5845@{#1382}) on port 44035
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
8月 26, 2023 5:37:18 午前 org.openqa.selenium.remote.ProtocolHandshake createSession
情報: Detected dialect: W3C
options => Capabilities {browserName: chrome}
userProfile => "{\"userProfile\":\"Picasso\",\"userDataDir\":\"/var/folders/7m/lm7d6nx51kj0kbtnsskz6r3m0000gn/T/__user-data-dir__12139686102370365831\",\"profileDirectoryName\":\"Profile 14\"}"
userDataAccess => TO_GO


<<<< Request received
{
"method": "GET",
"uri": "/",
"headers": [
    {"Accept-encoding": ["gzip, deflate, br"]}
    {"Cookie": ["timestamp=Fri, 25 Aug 2023 20:37:13 GMT"]},
    {"Sec-ch-ua": ["\"Not)A;Brand\";v=\"24\", \"Chromium\";v=\"116\""]},
    {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"]},
    {"Sec-fetch-dest": ["document"]},
    {"Sec-fetch-user": ["?1"]},
    {"Connection": ["keep-alive"]},
    {"Host": ["127.0.0.1"]},
    {"Sec-fetch-site": ["none"]},
    {"Sec-ch-ua-platform": ["\"macOS\""]},
    {"Sec-fetch-mode": ["navigate"]},
    {"User-agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"]},
    {"Accept-language": ["ja,en-US;q=0.9,en;q=0.8"]},
    {"Upgrade-insecure-requests": ["1"]},
    {"Sec-ch-ua-mobile": ["?0"]},
],
"body": """"""
}
==== Cookies cooked
in the request:  cookies=[timestamp=Fri, 25 Aug 2023 20:37:13 GMT]
in the response: cookies=[timestamp=Fri, 25 Aug 2023 20:37:13 GMT; max-age=60]

>>>> Response sent
/ is a directory.
timestamp1 => timestamp=Fri, 25 Aug 2023 20:37:13 GMT; expires=Fri, 25 Aug 2023 20:38:13 GMT; path=/; domain=127.0.0.1
timestamp2 => timestamp=Fri, 25 Aug 2023 20:37:13 GMT; expires=Fri, 25 Aug 2023 20:38:18 GMT; path=/; domain=127.0.0.1
> Task :jacocoTestReport
BUILD SUCCESSFUL in 17s
4 actionable tasks: 2 executed, 2 up-to-date
5:37:21: Execution finished ':test --tests "com.kazurayam.webdriverfactory.chrome.CarryingCookieOverSessionsViaChromeProfileTest.test_carrying_cookie_over_sessions_via_profile"'.

Please find the following 2 lines especially

>>>> Response sent
/ is a directory.
timestamp1 => timestamp=Fri, 25 Aug 2023 20:37:13 GMT; expires=Fri, 25 Aug 2023 20:38:13 GMT; path=/; domain=127.0.0.1
timestamp2 => timestamp=Fri, 25 Aug 2023 20:37:13 GMT; expires=Fri, 25 Aug 2023 20:38:18 GMT; path=/; domain=127.0.0.1

This “Response sent” message was emitted by com.kazurayam.webdriverfactory.CookieServer. The message was NOT printed by the test.

Please note that these lines have domain=127.0.0.1 portion.

@michel.g

You asked me:

I’m not sure what host_key you mean, but possibly the domain=127.0.0.1 is my answer to your question.

1 Like

Here I would quote a javadoc comment of the @Test void test_carrying_cookie_over_sessions_via_profile() of the test.

/**
     * This code will open Chrome browser and navigate to the URL "http://127.0.0.1" twice.
     * The http server will send a cookie named "timestamp" with value of
     * 1. if the HTTP Request has no "timestamp" cookie, will create a new cookie with current timestamp
     * 2. if the HTTP Request sent a "timestamp" cookie, will echo it
     * The cookie has a expiry that lasts only 60 seconds.
     *
     * At the 1st time, Chrome is opened with UserDataAccess option of "FOR_HERE".
     * Then the "timestamp" cookie will be persisted in the profile storage.
     *
     * At the 2nd time, Chrome is opened with UserDataAccess option of "TO_GO".
     * TO_GO means that the files in the profile directory will be copied from the genuine location
     * to the temporary location. Therefore I expect the cookies are carried over to
     * the second session. In the second session,
     * I expect the "timestamp" cookie should be sent from Chrome to the server again.
     *
     * This code makes assertion if the values of "timestamp" cookie of the 1st session
     * and the 2nd session are equal.
     * If these are not equal, it means that cookie was not carried over.
     */
    @Test
    void test_carrying_cookie_over_sessions_via_profile() {
    

I reviewed this comment. I think this comment describes what the test method does precisely enough.

1 Like

I don’t think so.

Both 127.0.0.1 and blabla.server.int.mycompany.cloud should be accepted as long as the host names were accessible for Chrome.

So, I do not see the reason of your problem yet.

1 Like