Hi.
A question with regard to the pricing on TestOps. Not so long ago, the pricing changed from “number of tests executed through testOps” to “number of results within testOps”. As you can see from the screenshot on your pricing page, the Monthly test results limit for e.g. 10.000 test results is $1.300/year. These test results include Katalon tests but also imported tests.
A couple of points I’d like to make on this:
1° We are grateful to be able to use the Katalon Report Uploader to upload the jUnit test reports from our ReadyAPI tests onto TestOps (sorry, their API testing tool still is more advanced compared to the WS-testing offered within Katalon). This makes TestOps the central place to check on the quality for all our projects, which is a very strong argument in our choice of test automation tooling in favor of Katalon. However, for us the API tests imported form the biggest chunk of test results in TestOps. In some of our projects this is 80% of all the test results. This is normal, since the API tests are always considered as the bigger layer of the “test automation pyramid” (see here your own link The Evolution of the Testing Model and Where It Will Go Next 
). So this change of pricing for us means a very steep increase of price, which (when our licencing needs re-evaluation) I might not be able to defend.
2° To me, the practice of making the customer pay for the number of tests they execute is a bad practice. The goal of test automation is just to have as much automated as possible. If customers need to take into account the number of test results they are allowed to have, Quality Assurance will be compromised. Then things will be pushed which should be avoided, like
- Let’s disable those tests,
- Let’s not automate this part,
- Let’s only run on day X
- Let’s just do manual testing…
I do understand the hosting and server costs for Katalon increase the more data is stored on testOps. I also understand that Katalon wants to pass on these costs to the customer…
But isn’t there a better way to do this?
For instance:
- Allow customers to, on a project basis determine which and how long test results are stored. There are always projects which may be less “active”. Perhaps once delivered there might be only sporadic new releases needed. For those projects, we want to keep on running the tests (as external changes could always cause regression), but perhaps we don’t need to have those test results stored for longer than a week. Of those, perhaps we only want to keep on storing the tests runs from the highest (most production-like) environment (see my other question I’ll raise ). From lower environments we only want to see the results from the day itself.
- Make not the number of test results be the cost-driver, but the actual size of storage space. There will be much less storage use for jUnit imported tests (without a screenshot). The Katalon tests that fail do have screenshots and those might then also be a motivator to get those fixed (either the application or the test case itself).