Katalon does not support Data URLs (rfc2397)

[Russ_Thomas]

Katalon Should Support All URL Schemes and Protocols

I thought hard about whether to post this as a bug or as a suggestion. The problem with “suggestion” is it implies it can be declined or even ignored. And since this is a failing in terms of Katalon’s ability to automate a browser, it seems valid to call it what it is: a bug.

Data URLs are a thing. Katalon does not support them. Katalon should support them.

Example:

WebUI.navigateToUrl('data:text/html;charset=utf-8,<div>Katalon</div>')

Which should result in this being produced in the browser:

But Katalon stops with an error and note that the failure occurs in Java/Groovy code, not the webdriver:

Unable to navigate to 'data:text/html;charset=utf-8,<div>Katalon</div>' 
(Root cause: com.kms.katalon.core.exception.StepFailedException: 
Unable to navigate to 'data:text/html;charset=utf-8,<div>Katalon</div>'
--snip--
at TempTestCase1556024744947.run(TempTestCase1556024744947.groovy:21)
Caused by: java.nio.file.InvalidPathException: 
Illegal char <:> at index 4: data:text/html;charset=utf-8,<div>Katalon</div>

Supporting Information:

Data URLs - HTTP | MDN

RFC 2397: The "data" URL scheme

[Russ_Thomas]

@ThanhTo @duyluong Any traction on this?

[ThanhTo]

@Russ_Thomas

Sorry I wasn’t aware of this. Could you elaborate more on the use case of Data URLs ? Highly appreciate if you can pour the details in a ticket:

katalon-studio/katalon-studio

[Russ_Thomas]

That’s hard to answer properly - they can be used for many things. Perhaps most commonly images? But as my little demo shows, they’re fully supported directly in the address bar, too.

There’s more discussion below (not quite sure why we ended up with two reports )

URLs supplied to Katalon APIs MUST support data URIs

Example: the following should work: WebUI.navigateToUrl('data:text/html;charset=utf-8,<div>Katalon</div>') But KS (java.nio) issues the following error: Unable to navigate to ‘data:text/html;charset=utf-8, Katalon’ (Root cause: java.nio.file.InvalidPathException: Illegal char <:> at index 4: data:text/html;charset=utf-8,Katalon) which is silly – a colon is certainly NOT an illegal character when part of a Scheme, otherwise http:, https: etc should fail also… goodbye the interwebz. https:…

[Russ_Thomas]

I now recall my original intent.

Side Note: I would fully understand if the devs decide this is too niche a requirement. But the fact remains, Data URIs are legal and Katalon fails to support them in any form.

Using callTestCase(), I developed something called a Super Test Case or Suite-Test-Case (STCs). They are a means to run something like a suite without some of the baggage of suites (mostly avoiding conventional reporting) plus provide a way to control the context around all the member Test Cases (you can specify things like before-tc to ensure a preparatory TC is executed before a given TC.

STCs are of course meant for Test development purposes and perhaps less useful for published suite-type purposes.

Here’s a screenshot of some of my notes on the topic.

To complete the story, I wanted to output the STC report directly in the browser. A Groovy string would be used to store the HTML for the report and passed to the browser as a data URI.

Because Data URIs are not supported, I had to write the string to a file and then navigate to it using a file: schema. That works but it’s a messier way to do such a simple thing. (Plus the files need to be cleaned up at some point.)

So, again, you might consider that to be a niche requirement but since it’s useful for a TC developer, there’s every chance it could be re-purposed as a plugin (I’d probably pass it over to @kazurayam to do that - I’m not an Eclipse/Java guy as we all know )

[kazurayam]

@Russ_Thomas

@Russ_Thomas

I’d probably pass it over to @kazurayam to do that

Do you still want me to do that?

[kazurayam]

One idea:

import some.package.DataURL

WebUI.navigateToUrl(DataURL.get('data:text/html;charset=utf-8,<div>Katalon</div>').toFileURL().toString())

The toFileURL keyword method of DataURL class will do “write the to a temporary file and return an instance of java.net.URL with file: schema which points to the temporary file”.

Are you satisfied if the keyword supports only the MIME TYPE text/html?Do you require other MIME TYPES such as image/png and video/mp4?

[Russ_Thomas]

Does that bump into this ?

@Russ_Thomas

I had to write the string to a file and then navigate to it using a file: schema. That works but it’s a messier way to do such a simple thing. (Plus the files need to be cleaned up at some point.)

[kazurayam]

Yes.
DataURl class would be responsible for creating and cleaning temp files.

[Russ_Thomas]

@kazurayam

Are you satisfied if the keyword supports only the MIME TYPE text/html ?Do you require other MIME TYPES such as image/png and video/mp4 ?

As a test, yes. However, if the package has a class called dataURI, it should support all possible dataURIs - don’t you think?

As an initial step, I suppose you could write:

some.package.dataURI.text_html

and in future add…

some.package.dataURI.text_plain
some.package.dataURI.image_png

etc. Make sense?

[kazurayam]

I have a new finding on Data URI support in Katalon Studio. I made 2 TestCase scripts.

Test Cases/research/1_webui_navigateToUrl_data`

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

WebUI.openBrowser('')
WebUI.navigateToUrl("data:text/html,<h1>Hello, world!</h1>")

The 1st script opened a browser, navigated to the URL:

• file:///Users/<username>/<projectdir>/data:text/html,%3Ch1%3EHello,%20world!%3C/h1%3E

However, the file is not present. The browser showed the following view, which is obviously unsatisfactory:

Test Cases/research/2_webdriver_navigate_data

import org.openqa.selenium.WebDriver

import com.kms.katalon.core.webui.driver.DriverFactory
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

WebUI.openBrowser('')
WebDriver driver = DriverFactory.getWebDriver()
driver.navigate().to("data:text/html,<h1>Hello, world!</h1>")

The 2nd script opened a browser, navigated to the URI:

• data:text/html,<h1>Hello, world!</h1>

The browser showed the following view. my Goodness!

---

This experiment showed me that the Selenium WebDriver API supports data: URI with no problem. On the other hand, Katalon’s WebUI API mis-interprets a data: URI to be a file: URL.

Therefore, I have got 2 points.

1. If you bother yourself to call WebDriver API in your Test Case, you can use data: URI now.
2. How WebUI.openBrowser(String url) and WebUI.navigateTo(String url) make mis-interpretation? I will look at the source code.

I hope that I would be able to modify the implementation of these keywords dynamically by Groovy’s Meta-programming technique to work-around the bug.

[kazurayam]

@ThanhTo
@Russ_Thomas
@duyluong

I looked at the Source of WebUI.navigateToUrl keyword and easily found the reason of the mis-interpretation.

Current code:

    @CompileStatic
    public void navigateToUrl(String rawUrl, FailureHandling flowControl) throws StepFailedException {
        WebUIKeywordMain.runKeyword({
            logger.logDebug(StringConstants.KW_LOG_INFO_CHECKING_URL)
            if (rawUrl == null || rawUrl.isEmpty()) {
                throw new IllegalArgumentException(StringConstants.KW_EXC_URL_CANNOT_BE_NULL_OR_EMPTY)
            }

            URL url = PathUtil.getUrl(rawUrl, "http")
            logger.logDebug(MessageFormat.format(StringConstants.KW_LOG_INFO_NAVIGATING_TO, url.toString()))
            WebDriver webDriver = DriverFactory.getWebDriver()
            webDriver.navigate().to(url.toString())
            logger.logPassed(MessageFormat.format(StringConstants.KW_LOG_PASSED_NAVIGATE_TO, url.toString()))
        }, flowControl, true, MessageFormat.format(StringConstants.KW_MSG_CANNOT_NAVIGATE_TO, rawUrl))
    }

The following statement is interesting.

       URL url = PathUtil.getUrl(rawUrl, "http")

This statement makes things complicated, and it looks unnecessary.

Should rather be:

    @CompileStatic
    public void navigateToUrl(String rawUrl, FailureHandling flowControl) throws StepFailedException {
        WebUIKeywordMain.runKeyword({
            logger.logDebug(StringConstants.KW_LOG_INFO_CHECKING_URL)
            if (rawUrl == null || rawUrl.isEmpty()) {
                throw new IllegalArgumentException(StringConstants.KW_EXC_URL_CANNOT_BE_NULL_OR_EMPTY)
            }

            logger.logDebug(MessageFormat.format(StringConstants.KW_LOG_INFO_NAVIGATING_TO, url.toString()))
            WebDriver webDriver = DriverFactory.getWebDriver()

            webDriver.navigate().to(rawUrl)    // java.net.URL class should NOT be involved!

            logger.logPassed(MessageFormat.format(StringConstants.KW_LOG_PASSED_NAVIGATE_TO, url.toString()))
        }, flowControl, true, MessageFormat.format(StringConstants.KW_MSG_CANNOT_NAVIGATE_TO, rawUrl))
    }

[Russ_Thomas]

[kazurayam]

@Russ_Thomas

I have got a solution for this issue. See the following topic.

Enabling Data URL in Katalon Studio

I have published a GitHub repository. Problem to solve In the Katalon Forum, there are a few posts that ask Katalon Studio to support Data URL. URLs supplied to Katalon APIs MUST support data URIs Katalon does not support Data URLs (rfc2397) I want to write a Test Case script like this: import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI WebUI.openBrowser("data:text/html,<h1>Hello, world!</h1>") Here the code fragment " data:text/html,<h1>Hello, world!</h1> " is ca…