Process files on Linux

Has anyone done processing of files on a Linux server that your Web app is running on?

My application runs on a Linux server. There is a task within the application that reads in a file from one of the Linux directories and processes it into the application. Each time we run the test case I need to copy a fresh file into that Linux directory to be processed (either by copying a new file on the server or download to the server directory from Katalon). In addition, after the file is processed I need to verify the file was moved to an archive directory on the server.

I am able to interact with files in a directory on my Windows laptop in Katalon but not sure if there is a way on Linux. Any help would be greatly appreciated! Thank You.

Whatever APIs you’re using on Windows, you should be able to do the same on Linux. Just make sure the process has the relevant rights to manipulate files.

For finer details, we’ll see if @bionel has any thoughts.

Thank you Russ. To clarify my post further:

I am running my Katalon tests via runtime engine on a windows remote desktop. My application to test runs on a Linux server within the same data center. So while the test is running on windows and interacting with the application UI, it will need to somehow connect to the Linux server to manipulate files.

As I think about this, it may be out of the scope of Katalon. Perhaps maybe a Rest API that will allow Katalon to issue the file copy on the Linux server and another one to ensure the file is moved after it is processed (new to Rest so not sure if that will work but seems plausible).

That sounds like a job for a shell script in Jenkins or similar. How are you starting KRE? (Asking because @bionel is going to ask you).

I am running my scripts via the Jenkins Katalon plugin.

something does not fit here.

I am running my Katalon tests via runtime engine on a windows remote desktop.

So, in the end, how you are running your tests?
On a jenkins windows agent … on a jenkins linux agent or … manualy on a certain windows host through rdp or whatever else?

However, the solution to connect to a certain Linux host an manipulate files there are:

  • ssh - login into the host and manipulate the files straight on it, from a windows or linux machine
  • scp - copy whatever files needed to a linux host, either from a windows machine or a linux machine, over the ssh protocol
  • rsync (only for advanced linux users)

Hello! Thank you for the response.

We run our tests on a Windows 10 server which we connect to via remote desktop. We kick off the tests using Jenkins which checks the project out from Git and then invokes Katalon runtime engine to run them.

As a background, customers send files to our application server nightly and we have a task that runs on a schedule to read in the files and process them. For a regression test I need to get a file into that Linux input directory and then create and run the task to process it.

This test case in our discussion connects to our application via web URL and navigates to the task submission screen and creates a task to read a file from the input directory on our Linux server where the application is running. Prior to submitting this task we need to get a file into that directory on the Linux server.

I know I could first run some other non-Katalon type of job in Jenkins to copy a file there but I would like to know if it is possible to do that within the Katalon test case.

One option would be a process to SCP a file to the Linux server but I am not sure how to do that within Katalon.

A second option (more complicated) would be to copy the file from another directory on the server which would imply the ability to issue a Linux command remotely from Katalon. The second option would be nice as we also need to verify the file was moved from the input directory to an archive directory after the task completes.

As a background, customers send files to our application server nightly and we have a task that runs on a schedule to read in the files and process them. For a regression test I need to get a file into that Linux input directory and then create and run the task to process it.

There may be other context around this that I’m not aware of, but this seems like a really poor implementation to me. Instead of having your customers send files to your app server directly every night, why not have an API that they can call to send the file (maybe this already exists?), or have a shared location for the file which your application can query on a scheduled basis? This is not your problem, obviously it’s on your developers, but this just seemed strange to me…

If your app had an API to call to send the data, then sending the file from Katalon would be trivial. Same story with the shared location.

Anyway, within the constraints that you’ve given us, I think the best answer is to establish an SSH connection from Katalon (we’ll need a third-party library here) and scp the file, as you’ve said. You could get it going fairly easily with Java:

https://www.baeldung.com/java-ssh-connection

Use that as a starting point at least.

You already provided the solution.
Let jenkins do the pre-requisites (connect to the linux machine, put the needed files in place etc)

You don’t need a separate job for this.
Just structure your pipeline / job properly:

  • run a ssh / scp step to put the needed files in place
  • start katalon test
  • (if needed) run a post-step to do eventually cleanups

If you really want to do the file transfer from katalon himself, you can use whatever Java library implementing ssh / scp, but I will avoid such.
It will make your test design too fragile.

see for example the link from @Brandon_Hein and this one for details:
http://www.jcraft.com/jsch/

here is also a scp implementation example using the same class:
https://medium.com/@ldclakmal/scp-with-java-b7b7dbcdbc85

1 Like

Agree with @bionel, if your jenkins build can put the file(s) in place beforehand, that’s actually probably a better solution.