Passing script variables when running a test suite collection

Michiel_van_Erp · October 9, 2020, 12:01pm

I’ve got a test suite collection, that contains multiple test suites. One of the test suites calls a test case with parameters, using Variable Binding. It’s one variable, defined as Script Variable and its type is a List of strings.

Running the test suite works fine, the test case gets called with the parameter, a list of strings.
Running the test suite collection fails when entering the parameterized test case:

Test Cases/Algemeen/Verwijder openstaande workflows FAILED.
Reason:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
General error during semantic analysis: String index out of range: -1

java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1931)
at org.codehaus.groovy.runtime.MetaClassHelper.capitalize(MetaClassHelper.java:464)
at org.codehaus.groovy.control.StaticImportVisitor.getAccessorName(StaticImportVisitor.java:488)
at org.codehaus.groovy.control.StaticImportVisitor.findStaticFieldOrPropAccessorImportFromModule(StaticImportVisitor.java:368)

What’s going on?

Russ_Thomas · October 9, 2020, 12:48pm

Suites – and most certainly Suite Collections – have their challenges. What we need is something like “universal variables” tied to the Katalon execution start point and available until all execution stops. Right now, there is nothing built in to the system that provides that “out of the box”.

However, there are ways of accomplishing something similar using other, manual means. But first…

How are you passing the variables right now? It’s not clear from your description. Can you provide code/screenshots showing the setup?

Aside: I’ll assume for now you’re not using parallel execution in the collection. That would make things much more complicated.

Michiel_van_Erp · October 9, 2020, 1:46pm

Thanks Russ! I don’t get why we would need ‘universal variables’. I know global variables are tied to a test suite and in my case, the test suite has the parameters hardcoded configured for the child test case.

This is the test suite collection, configured to run sequential. I’m aiming at parallel execution in the near future though.

This is the test suite.

The 2nd test case in that test suite has this data binding configured:

More details about the parameter being passed:

The configuration of the parameter in the test case:

If I run the test suite on its own, it works fine. When ran as part of the test suite collection, it fails.

Russ_Thomas · October 9, 2020, 2:09pm

Okay, that’s not what I imagined - thanks for clarifying. Sounds buggy, to me. I’ll move this thread. In the meantime, let’s see what the devs have to say. @ThanhTo @duyluong

Universal Variables:
To ensure that all executions have access to data at the time they need it, some people (including me) resort to outside storage. I keep a json file with start up values stored there. Typically, these are copied to GlobalVariables (if needed) and hence usable anywhere. Something new, “universal variables”, could provide that out of the box. I’ve forgotten the details of the gotcha that this addressed, something to do with a disconnect between the default profile, custom profiles, and suites? It was a scope issue of some kind. I guess I used my solution and never looked back.

anon46315158 · October 10, 2020, 10:58am

@Michiel_van_Erp
I made a simple experiment:

One test case with a variable defined, having a default value. In the testcase i simply print the variable value, see pic:

testcase1599×235 17 KB
Testcase is added to a suite, having a similar binding like you did (script variable of list type):

The suite is added to a collection:

It run’s just fine, output:

2020-10-10 13:33:31.689 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2020-10-10 13:33:31.690 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/New Test Case
2020-10-10 13:33:31.710 INFO  c.k.katalon.core.main.TestCaseExecutor   - variable = [eins, zwei, drei]
2020-10-10 13:33:31.944 DEBUG testcase.New Test Case                   - 1: println(variable)
[eins, zwei, drei]
2020-10-10 13:33:31.949 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/New Test Case

However, side note:
If you would expect the execution to iterate over the list you defined, it won’t happen. The testcase is executed only once, as shown in console output. The variable printed is a list.
If this is what you intend to achieve, all fine up to now (at least for me)

If you need to iterate over the testdata values, you have to create a testdata instead (internal data, excel, whatever) see the next example:

binding:

Now, when i execute the collection, the test case is running three times:

2020-10-10 13:47:23.560 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2020-10-10 13:47:23.560 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/New Test Case
2020-10-10 13:47:23.582 INFO  c.k.katalon.core.main.TestCaseExecutor   - variable = eins
2020-10-10 13:47:23.818 DEBUG testcase.New Test Case                   - 1: println(variable)
eins
2020-10-10 13:47:23.825 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/New Test Case
2020-10-10 13:47:23.847 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2020-10-10 13:47:23.847 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/New Test Case
2020-10-10 13:47:23.860 INFO  c.k.katalon.core.main.TestCaseExecutor   - variable = zwei
2020-10-10 13:47:23.862 DEBUG testcase.New Test Case                   - 1: println(variable)
zwei
2020-10-10 13:47:23.866 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/New Test Case
2020-10-10 13:47:23.886 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2020-10-10 13:47:23.886 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/New Test Case
2020-10-10 13:47:23.897 INFO  c.k.katalon.core.main.TestCaseExecutor   - variable = drei
2020-10-10 13:47:23.900 DEBUG testcase.New Test Case                   - 1: println(variable)
drei
2020-10-10 13:47:23.903 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/New Test Case

Hope it helped (or not, enjoy debugging your project )!

anon46315158 · October 10, 2020, 12:29pm

@Michiel_van_Erp
I noticed that in your setup, the variable at testcase level is an empty list (no default value).
Mine was a string, with a default value.
So i made a new experiment similar with yours, just to be sure:

Test collection is stil running fine:

2020-10-10 15:24:08.547 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2020-10-10 15:24:08.547 INFO  c.k.katalon.core.main.TestCaseExecutor   - START Test Cases/New Test Case
2020-10-10 15:24:08.570 INFO  c.k.katalon.core.main.TestCaseExecutor   - variable = [drei, zwei, eins]
2020-10-10 15:24:08.784 DEBUG testcase.New Test Case                   - 1: println(variable)
[drei, zwei, eins]
2020-10-10 15:24:08.791 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/New Test Case

So, to properly debug it, i think we may need to have a deeper look into your code.
How do you use this variable in your testcase?

Michiel_van_Erp · October 12, 2020, 2:40pm

Thanks @anon46315158 for trying to reproduce this error.
Some strange things happened. The test suite itself now started giving the same error as well.
Looking into the test case code - as you mentioned - and disabling the lines that worked with the test case parameter solved the error.

I initially had a for-loop that apparently later on was auto-optimized to these lines, where workflowsToCancel is the test case parameter:

for (def item : workflowsToCancel) {
workflowsToCancelAsString += “”$item","
}

Refactoring it into something else, made the error go away:

for (String item : workflowsToCancel) {
workflowsToCancelAsString += “"” + item + "", "
}

I recall the upper block had something like ${item} in its assignment, a shorthand Groovy syntax that wasn’t handled well, it seems.

anon46315158 · October 12, 2020, 4:35pm

Strange things may happend if you switch from manual view to script view and back…
So i just made a habit to not touch manual view. Never. It’s evil!

Michiel_van_Erp · October 13, 2020, 1:15pm

@anon46315158 haha, think so? I noiticed switching between script and manual had an effect on the coding layout sometimes, but didn’t know it could mess up my code. I’ll keep my eyes open … thanks again for the help!

Russ_Thomas · October 13, 2020, 2:04pm

It happens, though rarely. And, it is known to mess with string/GString constructs (exactly where your problem occurred). The complaints are on the forum somewhere. There’s a reason I moved this discussion to bug reports.

ThanhTo · October 16, 2020, 4:01am

@Michiel_van_Erp @Russ_Thomas

So is the problem resolved, is this a problem with Katalon ? Sorry, there’s a lot in this so I can’t exactly grasp all in here.

anon46315158 · October 16, 2020, 7:05am

@ThanhTo for the main topic, it is not a katalon issue, for some reason the code of the OP got messed …
could be because of switching from manual to script view but for that behavior for sure there are other open topics (i’ve been hit too few times)

Russ_Thomas · October 16, 2020, 12:03pm

No, but there’s nothing actionable beyond what we already know. Sometimes, some people have an issue switching between Manual/Script view.

Russ_Thomas · October 16, 2020, 12:49pm

@ThanhTo

Can’t believe how long it took to find it…

anon46315158 · October 16, 2020, 1:18pm

usually i saw such behavior when mixing java with groovy code styling, e.g Michiel had such code snippet:

for (def item : workflowsToCancel) {
workflowsToCancelAsString += “”$item","
}

manual view will probably mess it due to multiple possible causes.

i will rewrite this to something like:

workflowsToCancel.each { workflowsToCancelAsString += "${it}, "}

or take the full java approach, to avoid issues with parens.
not sure if quotes are needed inside that string too, that will be another source for headaches.
If quotes are needed inside the resulting string, the easiest solution will be to use escaping.
Even better, the entire loop can be avoided by using .join()

so we cannot say 100 % it is a bug in katalon, but more like an undesired effect, most probably the code rendering engine is confused in certain condition, particularly when coding styles are mixed.

Topic		Replies	Views
[Command Line] pass parameter Global Variable not work in TestSuites Collection Archive katalon-studio , bugs-report	7	3493	January 16, 2022
How to pass variables between test cases in a suite (the "right" way...) Katalon Studio katalon-studio , tips-tricks	25	14097	December 21, 2022
Global Variable Archive	4	814	October 30, 2018
Pass global variable from 1 test suite collection to another Katalon Studio katalon-studio , web-testing	9	4179	February 15, 2020
Pass values from test suite to test case Katalon Studio katalon-studio , api-testing	2	777	February 27, 2020

Passing script variables when running a test suite collection

Related Topics