Command Line Execution Logic

staff-20

#1

I’m looking for clarification on the current logic behind command line execution. Looking at the console log, I see the following snippet:

image

In particular, I’m concerned with the following lines:

Delete folder: bin
Delete folder: Libs

… after which the project is opened (triggering a recompile). Looking back a ways, I believe this approach may have been an attempt to resolve everyone’s favorite bug:

… where I make a suggestion to delete those directories, then reopen your project, allowing the studio to rebuild the workspace. However, I never intended this to be a permanent solution to that problem so long ago, it was simply a quick fix to get people up and running.

There is a major problem with integrating this into console mode execution by default, and I believe it may be the source of several issues that have been brought up:


The problem I see most often is when multiple executions are triggered simultaneously and/or relatively close together. Consider the following scenario:

1.) An execution is triggered, during which the bin and Libs folders are deleted, and the studio then tries to open the project. While the project is opening, and the source code is being recompiled, there is a period of time where these directories (and the binaries they contain) are not available.

2.) Another execution is triggered while this recompilation is going on. Because those directories are in the process of being rebuilt, one of the following has a good chance of happening:

  • An error is thrown during the second job stating that the directories cannot be deleted (either because they don’t exist, or they do exist but are locked by the first job).
  • The timing is such that job 2 is able to delete the directories, but this causes job 1 to throw errors, usually related to missing classes.

This issue can be easily recreated by simply triggering two CLI executions simultaneously.

Again, I’m just looking for some insight on why console mode execution does this. In my humble opinion, deleting these folders then recompiling is not a very good solution, especially when multiple exections are triggered as I’ve shown. This is particularly troublesome with Jenkins integration, where multiple nightly jobs are often triggered together.


#2

Awesome diligence, @Brandon_Hein (and no, I don’t throw “awesome” around lightly).

I don’t know why (but I can guess[1]) the tool doesn’t take a complete snapshot of the source(s) and run them in an effective compile-free sandbox.

[1] Would significantly change the runtime resource use figures.


#3

After further testing, this has the potential of causing a myriad if issues. @devalex88 @ThanhTo please please please review this issue! It’s extremely easy to reproduce…

Steps to reproduce:

1.) Open the CLI
2.) Enter your console mode execution argument. Mine is:

katalon -noSplash -runMode=console -consoleLog -noExit -projectPath=“C:\projects\qa-katalon\master.prj” -retry=0 -testSuitePath=“Test Suites/bluekc/pc2/smoke/smoke” -executionProfile=“BlueKC-1” -browserType=“Chrome”

3.) As closely-together as possible, execute this twice.

Depending on the timing, there are tons of different outcomes. None of them good. Below are just a few that I’ve encountered.

First test

I was just able to reproduce this:

The first execution gets hung up on “Parsing custom keywords…”. The second execution completed successfully.

Second test

The first execution produces the following error log:

log.txt (13.0 KB)

The second execution completes successfully.

Third test

Same result as First test.

Fourth test

The first execution produces errors indicating unable to delete certain files/directories, as mentioned in my OP, then hangs.

log.txt (8.4 KB)

The second execution completes successfully.


#4

Hi @Brandon_Hein

Thank you for bringing up, this completely slipped my mind until now. Our development team is currently looking into it and will let you know !

Cheers !


#5

Thank you @Brandon_Hein,

Sharing the same source folder concurrently between different Katalon Studio processes is not supported at this moment. May I know the reason why you need this feature?


#6

The main reason is stated in the OP:

If multiple clients have nightly builds that either happen concurrently, or relatively closely together, then the test executions for these will almost certainly overlap at some point.


#7

this is particularly lack of knowledge about jenkins, and any CI. assign as many workers/agents to the master and any task will run, independent of each other. katalon is a standalone application


#8

that can happen only because poor planing


#9

Sure, if you have more than one agent machine to work with. What you assume is a lack of knowledge may just be a lack of resources.

Poor planning between the various project teams perhaps.

This isn’t primarily about CI though. The ability to share pre-compiled source code is not an extremely tall order in my mind.


#10

why would somebody need to run two concurent instances, at the same time, on the ‘same’ machine? just for fun? user space isolation has been invented long time ago. in multiple forms. virtualization, containers and so on


#11

I’ll take “just for fun?” for 1000, Alex.