[AKA] Katalon TestOps - Feb 26 to Mar 12, 2024 🪄

Extended

Hey TestOps Crew! :hibiscus:

We are pumped to announce that AKA is now LIVE! :tada: Whether you’re exploring the platform, encountering challenges, or simply looking for tips to enhance your testing workflows – Our Product team members are here to answer all of your questions.

I am Win Le - A part of Katalon’s Product team. Currently, I am working on the Platform module, which is responsible for the end-to-end testing experience, including test organization, planning, execution, and reporting on the Katalon Platform. As a Product Manager, I aim to empathize with our users to the best of my ability, please feel free to ask me anything related to the Katalon Platform.

:new: Guidelines :spiral_notepad:

  • If you have any questions about TestOps, please raise them directly under this thread. Ask one question at a time by replying to this topic, using the Reply button AT THE BOTTOM of this topic.
  • We will provide a weekly Q&A summary of Katalon TestOps discussion in a separate thread. Please read through the existing questions in the summary to see if your question has already been asked by another member, before posting your own.
  • Please don’t reply to anyone else’s post. If you’d like to discuss a related topic in more detail, you can Create a new topic
  • After posting your question, please give me from 1-3 days to answer your questions as I may also be busy with other projects.
  • And lastly, don’t forget to have fun! :wink:

Posts/replies that do not follow these guidelines may be removed to keep the AKA session flowing smoothly. Thanks! :+1:


We are also having a small Kudos giveaway for anyone who spread the word about our Ask Katalon Anything session. Head to our main thread below to learn more :point_down:


Cheers to curiosity and all things testing,
Win Le

2 Likes

Do you also have questions about Katalon Studio and/or Katalon TrueTest™️?

Then head to the two threads below where @xuan.tran and @nghi.hua will be more than happy to answer any questions you may have!

:pushpin: [AKA] Katalon Studio - Feb 26 to Mar 04, 2024 ✨

:pushpin: [AKA] Katalon TrueTest™️ - Feb 26 to Mar 04, 2024 ✨

1 Like

Hi all,

While there will not be anyone from our Cloud Studio team joining AKA this time, we want to let you know that you can still create a topic in the Katalon Cloud Studio and get answers from either @nhi.knguyen or @quyen.huynh (albeit a bit slower).

Thanks, :+1:
Albert

1 Like

Katalon test ops has recently launched a new feature where you can record and playback test cases in the test ops using the Katalon recorder plugin.

However, while recording some test cases, I keep on getting a crash browser when I reach a particular step that I am recording, the step is expected to open a modal and continue my next step there, the page freezes and eventually crashes after a few seconds.

Please see the recorded steps and errors

. Hope you can help me fix this. Thanks
2 Likes

Hi @rlinatoc,
I’m Quyen, PM of Cloud Studio from Katalon (the Record feature that you used) . First of all, thank you for contacting us on the issue. We’ve tried to reproduce it, but look like it’s a private link or internal url that we cannot register an account.

We took a closely look at the inspect that you shared in the video. So far, there is no error nor issue related to the Recorder engine spotted in the inspection. We need to collect more information from the Dev Tools of Katalon Platform Recorder Extension to better troubleshoot the issue. If you don’t mind, please help to reproduce it, and at the frozen step, record what you can find in the Dev Tools for us. It would help a lot in figuring out what is going on there.

To access the Dev Tools, please open the “Manage Extensions” dialog on your browser. Then, under “Katalon Platform Recorder” extension, click on “service worker”, and capture all you see under the Console log.

Looking forward to your reply. Thank you very much for your time.

1 Like

Hi.

A question with regard to the pricing on TestOps. Not so long ago, the pricing changed from “number of tests executed through testOps” to “number of results within testOps”. As you can see from the screenshot on your pricing page, the Monthly test results limit for e.g. 10.000 test results is $1.300/year. These test results include Katalon tests but also imported tests.

image

A couple of points I’d like to make on this:
1° We are grateful to be able to use the Katalon Report Uploader to upload the jUnit test reports from our ReadyAPI tests onto TestOps (sorry, their API testing tool still is more advanced compared to the WS-testing offered within Katalon). This makes TestOps the central place to check on the quality for all our projects, which is a very strong argument in our choice of test automation tooling in favor of Katalon. However, for us the API tests imported form the biggest chunk of test results in TestOps. In some of our projects this is 80% of all the test results. This is normal, since the API tests are always considered as the bigger layer of the “test automation pyramid” (see here your own link The Evolution of the Testing Model and Where It Will Go Next :wink: ). So this change of pricing for us means a very steep increase of price, which (when our licencing needs re-evaluation) I might not be able to defend.

2° To me, the practice of making the customer pay for the number of tests they execute is a bad practice. The goal of test automation is just to have as much automated as possible. If customers need to take into account the number of test results they are allowed to have, Quality Assurance will be compromised. Then things will be pushed which should be avoided, like

  • Let’s disable those tests,
  • Let’s not automate this part,
  • Let’s only run on day X
  • Let’s just do manual testing…

I do understand the hosting and server costs for Katalon increase the more data is stored on testOps. I also understand that Katalon wants to pass on these costs to the customer…

But isn’t there a better way to do this?

For instance:

  • Allow customers to, on a project basis determine which and how long test results are stored. There are always projects which may be less “active”. Perhaps once delivered there might be only sporadic new releases needed. For those projects, we want to keep on running the tests (as external changes could always cause regression), but perhaps we don’t need to have those test results stored for longer than a week. Of those, perhaps we only want to keep on storing the tests runs from the highest (most production-like) environment (see my other question I’ll raise :wink: ). From lower environments we only want to see the results from the day itself.
  • Make not the number of test results be the cost-driver, but the actual size of storage space. There will be much less storage use for jUnit imported tests (without a screenshot). The Katalon tests that fail do have screenshots and those might then also be a motivator to get those fixed (either the application or the test case itself).
1 Like

As I posted here more than a year ago, I’d like to now how Katalon TestOps sees the concept of “Test Environments”.

Use case: I have several projects where we run the Katalon tests on multiple environments, namely TST and ACC. Since we do agile iterative development, the tests on TST could break and often need rework. On ACC we have a production-like application that normally should always be green. It might be, due to integration with external parties, that on ACC a test suddenly starts failing (in other words: the external parties did some changes and these might impact our application in a negative way!).
So it is important for us to be able to quickly report on a state of TST & ACC.

We have the Katalon “profile” concept, where we do make a distinction on the environment (among other things) via the profile name. So we can filter out via ExecutionTestResult.profile:“~acc” all the test results from ACC, but that’s not ideal. Furthermore, we also upload jUnit test reports from ReadyAPI for both environments onto the Katalon TestOps via the Katalon Report Uploader (see Upload JUnit and report files to using command line | Katalon Docs). There we do not have the concept of “profile”, so we cannot filter those out.

I tried to use the labels feature as described in my post above:
So, I created a custom field “Environment” with labels “TST” & “ACC”. I manually linked it to a test Suite Collection result and hoped to be able to have a quick filter. It works… but it’s not ideal. Some shortcomings:

  1. Obviously, goal should be to have this label automatically set to my test runs. Perfect place for me would be as a parameter in both the Katalon Runtime Engine command line (see Command-Line Syntax in Katalon Runtime Engine | Katalon Docs) AND in the Katalon Report Uploader command line.
  2. There is no url adaptation (see screenshot) for this filter, so I cannot store it as a favorite “application X - ACC results” in my browser. Each time, I will have to go for the custom filter and filter out the “TST” or “ACC”. Developers might for instance be only interested in the view for TST, while management wants to quickly see the state of ACC before an approval to go to PRODUCTION. I cannot go and ask them to play with the filter themselves…

Hi @joost.degeyndt, thanks for your dedicated feedback regarding the pricing model. I would like to apologize for any inconvenience and fully understand your concerns here. To be honest, the business is crucial to Katalon’s existence, and the changes in the pricing model have aided us in scaling and developing more capabilities that benefit Katalon users.

Although business is important, the Katalon team always aims for Product-Model Fit, which reflects the balance between pricing models and product growth. Your feedback is very valuable at this point. It’s my pleasure to incorporate your feedback into the further discussion of the Katalon team regarding the pricing model.

1 Like

Hi, Can i pls get any info on the below scenario.
Looking for a way in Katalon to enter data into different test cases but the data is dependent on each case. For example one policy information has multiple location addresses. So we want to enter information test case one after another and enter single policy information and its corresponding multiple locations and then loop to next policy

Looking for a way in Katalon to enter data into different test cases but the data is dependent on each case. For example one policy information has multiple location addresses. So we want to enter information test case after another and enter single policy information and its corresponding multiple locations.

Hi @here, :wave:

We are excited to share with you that AKA has been extended until 2024-03-12T16:59:00Z to give you more time to share your questions and have them answered by our Product team! :tada:

We are also thrilled to welcome Vy Huynh (@vy.huynh) from our TestOps team who will also be joining this thread to answer your questions!

Keep the questions coming! :sunglasses:

2 Likes

Hi @klella,

It seems like your question has more to do with Katalon Studio. In that case, I suggest you to ask in the below thread instead. Thanks.

Hi Katalon team,

May I trigger a TestOps schedule from my CI/CD pipeline? The schedule feature on TestOps is nice, but it does not fit my situation since it only supports triggering schedules by interval. The release can come at any time, and we need to trigger our tests immediately then.

Hopefully @vy.huynh can help me to answer this question. Thanks in advance!

Hi Pedro
You can certainly use Katalon CLI, which is published on Docker Hub at Docker. With Katalon CLI, you can use it within your CI/CD tools, and trigger TestOps schedules from there by schedule IDs or schedule names. For more details, please refer to the following documentation Katalon CLI | Katalon Docs.

1 Like

Hi team,

Is there any way to connect TestOps to my repository without using my PAT? Thanks in advance!

Hello everyone!

I would be interested to know what options are there for Web Image-based Testing and/or PDF content testing and how one can incorporate this with Katalon Studio using TestOps in order to find out and compare potential changes and mismatches between test case runs?

Thanks in advance,
Kind Regards

2 Likes

Hi @here, :wave:,

Our Ask Katalon Anything session has come to an end, and we want to thank everyone for joining. We hope you were able to find the answers you need from our Product team members.

Our Product Managers will be reviewing the unanswered questions and providing the answers in the coming days. Should you have any other Katalon-related questions, feel free to Create a new topic.

We would also like to take this chance to ask for your input and suggestions so that we can make our future AKA sessions better. Feel free to let us know your feedback in the short form below (should not take you more than 5 minutes!):

Thank you once again for your participation and support, and we look forward to seeing you at our future events soon! :sunglasses:

Hi @apolezhako

You can download your repository, zip it and then use the function Upload Script Repository to TestOps.

Hope this helps. Thank you.

1 Like

Hi @qa-dept

You can read more about Katalon TestOps visual testing in Visual Testing overview | Katalon Docs.
Or reading about PDF testing here [KShare] Using Katalon Studio to read PDF files directly on a webpage - Product Insights - Katalon Community

Thank you

1 Like