What are the CI/CD tools that you use in your companies to execute UI test runs?
Here’s our situation.
Given: Katalon test collection of all UI scripts written in Groovy.
Needed: CI/CD tool for nightly execution of this collection.
What we’ve tried: 1) running in Jenkins which resulted in a very high number of failed tests.
2) running from the local machine in Katalon - very time-consuming and inconvenient but fewer failed tests - and that is what we’ve been doing for quite some time.
3) running from Azure Pipelines - our build failed due to the fact that Pipelines image does not have enough resources to finish the suite in the time allotted.
Please don’t offer Katalon TestOps as we don’t currently have an option to run from it.
I can not answer to your question. But I have something to tell you.
Most probably you tests failed due to their own defects/fragility/mistakes in your test scripts and/or configuration. I believe the failures are not caused by Jenkins. You should fix the failures by looking into each failures one by one. The reasons of a failure will be in your code, not in Jenkins.
You may try other CI/CD tools but, I suppose, the failures will repeat. You will just waste your time.
As an example we need to increase our test case timeouts when running the same test cases in our Jenkins and Azure pipelines. For example 10 seconds works locally but we need to increase those to 90+ seconds in some cases when running them via the Pipelines. Our Pipelines run on servers that perform many tasks which result in performance challenges, setting longer wait times aids in reducing test case failures. You may also want to check into Katalon’s Smart Wait Function | Katalon Docs
No CI/CD tool will help you here if you don’t allocate proper resources for jobs execution.
(aside of writting proper / robust tests)
I assume, in Jenkins case, you just set-up a master server on whatever machine and that’s all.
This is the worst setup possible.
For Azure Pipelines I suppose you just used some standard images for agents. I don’t know the spec of them, I never used it.
For jenkins you can register dedicated nodes for execution. Never run on master server.
For Azure Pipeline you can use self hosted agents of your own specs.
Where to host them? Only your infrastructure support team can answer, can be AWS VM’s, ESXi VM’s, physical dedicated machines etc.
Ask your DevOps engineers for a solution, if you have such team.
Never used, but if I understand it right, TestOps Cloud is a platform offering already made ‘agents’ (test environments).
Could work, as the doc mention it can be easily integrated with any “orchestrator” (the term used here by Katalon doc may be misleading, therefore my quotes) , but a CI tool seems to be still required to schedule runs etc.
So it depends on what @natallia.dyer is looking for, a complete inhouse solution, just the CI to be inhouse, etc.
(let’s don’t speak right now about the CD part which is a slightly different animal, it is more like a practice than a tool)
Everything came with a cost, even if using free / opensourced tools.
To get stable results, stable environments are needed, so what the team should do now is, define the needs.
After that, proceed to evaluate various platforms and see what suits best.
Check also what you already have, perhaps you can setup own resources with lesser costs and so on.
And by cost, evaluate also human power, you already have team members with certain expertise in such, or you have to learn everything from scratch…
Based on resources available and, ofcourse budget available, you can opt for cloud services (already made solutions, let’s say), pure inhouse, or mixed.
Free is nothing!
… to execute tests in testcloud it is only a matter of running KRE with a certain set of parametters.
So, this indeed should work with any CI tool, the only thing needed is to run a script.
With Jenkins, there is no need in such case for dedicated nodes, as the isolation between tests environments will be provided by the TestClouds platform. The machine where Jenkins controller is hosted should only be able to sustain certain jobs running in parallel, if this is desired, otherwise schedulling should be done carefully.
Note that, any job will still execute Java threads, as the KRE runs on the Jenkins controller or node.
For AzurePipeline, same, there is no longer need for a powerfull VM, any image able to run a script and execute java should work properly, as the ‘execution pressure’ should be transferred to the TestClouds VM’s (however still need some resources available to run the groovy scripts themself)
It came with some price, ofcourse, but can save the team from certain headache, like maintain the infrastructure, setup properly the OS, keep the hosts up-to-date etc.
Worth to try …
Browsterstack can be evaluated also, is pretty similar
Test are pretty stable when they are run from Katalon Studio itself. Much fewer failures than when run via Jenkins. I’m guessing the problem is with the non sufficient resources as follows from @bionel response.