How TrueTest works, how to install TrueTest, and its use cases
1. How does TrueTest know that the changed behavior in the application is correct?
If you are familiar with data analytics tools such as Google Analytics and Fullstory, what TrueTest does is quite similar. It tracks user interactions within the application through user sessions. It then analyzes this data to identify common patterns, representing user behavior over a specific period of time.
2.Is it necessary for TrueTest to operate within a production environment to achieve this? How could one deploy it in a QA or UAT environment to obtain production-equivalent results?
To install TrueTest in an environment, you’ll need to integrate a JavaScript code snippet into your application code and deploy it to the desired environment. If you aim to generate tests based on production user behaviors, you’ll need to deploy the code snippet to the production environment. Alternatively, you can utilize Google Tag Manager to install TrueTest without modifying the code.
Regarding operating in a QA or UAT environment, TrueTest can capture flows in any environment where the JavaScript snippet can be installed. This means it’s possible to record flows in pre-production environments as well. Additionally, you can run tests generated from one environment (e.g., Production) in another environment (e.g., UAT) by modifying the data driving the tests.
3. When a new TrueTest flow is captured again for the same interactions, does this automatically update existing object locators that perhaps need maintenance, or does it create new ones?
It will update the existing test objects.
4. Can TrueTest be used for scenarios like end-of-financial-year activity, which is only conducted for three weeks in the year, but then intensively and is critical?
You can select a specific time frame to generate user journey maps and test cases. If you need to create test cases for certain activities, choose the time frame during which you know these activities will take place.
5. Can we use TrueTest in closed environments without internet access?
If there is no internet access from the application you want TrueTest to capture the user interactions, then you cannot use TrueTest yet, until we release an on-prem version later next year. Please stay tuned for updates from us!
6. What will be the impact on the app’s performance by including this JavaScript?
There shouldn’t be any impact on the application’s performance. We take this issue very seriously; in fact, we have TrueTest installed on our own applications. We always conduct performance testing on various types of applications before every production release. Plus, you can quickly disable TrueTest if you suspect any issues.
7. Suppose one user has already added Java code snippets to a UAT (User Acceptance Testing) application. Can another user also utilize the same app with same code snippets or they need to add it again?
The snippet is connected to your organization, so it only needs to be installed once for a given application. Once a developer adds the JavaScript code snippet to an application, TrueTest will be enabled for all users that are navigating on that application.
8. How can I install TrueTest on my application?
You need to add a Javascript code snippet to the element of the application. More details can be found at: https://docs.katalon.com/docs/proof-of-concept/autonomous-test-generation/set-up-katalon-ai-agent
And…
9. Does the development platform make any difference in the recorded tests?
We don’t expect any issues regarding the development tool or framework you used for your web application.
10. Can we run Soap API test suites in parallel in a Test Collection?
Today TrueTest only generates web tests. However, if you have API tests in Katalon Studio, you should be able to run them alongside TrueTest-generated web tests.
11. How does it work with third-party integrations?
You need to set up TrueTest on the sites you are redirecting to. If that’s not feasible, then TrueTest would stop capturing user interactions upon redirection to the third-party site. When the user is redirected back to your application where TrueTest is enabled, user interactions capturing would start again by TrueTest.
12. What if the project already has many customized keywords and sub-test cases created and frequently used for specific needs? Can TrueTest leverage these?
At this time, TrueTest will generate new test cases based on the user journeys. You can enable TrueTest in the test environment where you run your current Katalon tests so that you can visualize the coverage of your existing Katalon tests against the real user journeys to understand how much of the business-critical scenarios your existing Katalon tests are covering. And then you can ask TrueTest to generate additional tests to bridge any coverage gaps.
13. Can it be used for API automation?
Currently TrueTest is focusing on UI test automation for Web applications. It is our goal to extend it to API test automation next year.
14. Does it support web apps that use HTML Canvas?
TrueTest technology does capture user interactions on Canvas-based HTML applications. However we’re looking for design partner customers to help us polish the processing of Canvas-based events. Let us know if you’d like to join TrueTest’s beta program so we can do this together!
15. Can I use TrueTest for Database testing?
Currently TrueTest is focusing on UI test automation for Web applications. We’re considering future roadmap items.
16. Does TrueTest support testing for web applications with lengthy functional processes, for example: public records, university enrollment management, etc?
Yes. Long workflows/user journeys are very interesting to be visualized in a user journey map so you can truly understand how your users actually navigate through those functional processes. Often our customers find that real users go through those complex workflows in ways they didn’t expect when they designed those functional processes and can make decisions to improve the customer experience based on TrueTest’s user journey map. Those long workflows are very hard to create tests for and having TrueTest generate those tests automatically that truly represent how users navigate those long processes is one of the biggest values TrueTest delivers.
17. Does the coverage check work if we are not currently using Katalon solutions?
Yes. If you run your manual or automated tests in your test environment with TrueTest enabled on your application under test, TrueTest will capture those browser interactions and display the coverage of those tests (manual and automated) against what real users do in production.
18. Does the coverage check work if we are not currently using Katalon solutions?
We are working on the capability of showing coverage for all testing activities, not just Katalon exclusively.
19. Does TrueTest act as a data generator?
TrueTest generates the synthetic data needed for its test case execution.
20. Can the user journey map be used for usability testing?
TrueTest’s user journey map gives you insights into how users flow through the application in production, which helps review test coverage and identify gaps.
21. Does TrueTest support mobile testing?
For now, TrueTest only supports web applications.
22. Our tests are built 99% on custom methods that call the keywords provided by Katalon and other existing custom, does TrueTest leverage existing test cases?
TrueTest does not leverage existing test cases or custom methods.
23. Do we need test suites, or can individual tests run from TestOps™?
Test cases generated by TrueTest are normal Katalon Studio test cases. You can open them in Katalon Studio and execute them individually or as part of test suites. If you want to use the Schedule Test Run feature in TestOps, the test cases must be added to test suites or test suite collections.
[You are here] Questions about how TrueTest works, how to install TrueTest, and TrueTest’s use cases
