[Webinar - Q&A Thread] Introducing TrueTest™: AI-powered Regression Testing

Hi folks, :wave:

Thank you very much for tuning into our webinar where we got to introduce you to TrueTest, and how it would help to boost your testing efficiency, especially in Regression Testing.

For those of you who was not able to join, please follow the link below to watch the recording :point_down:


In this thread, we will be compilating all the questions asked, and our answers to them during the session.

:information_source: This thread will be a bit long, so we have prepared the below anchor links for you:

:pushpin: Questions about how TrueTest works, how to install TrueTest, and TrueTest’s use cases

:pushpin: Questions about Security and Data Safety

:pushpin: Questions about Platform Integration

:pushpin: Questions about Pricings

How TrueTest works, how to install TrueTest, and its use cases


1. How does TrueTest know that the changed behavior in the application is correct?

If you are familiar with data analytics tools such as Google Analytics and Fullstory, what TrueTest does is quite similar. It tracks user interactions within the application through user sessions. It then analyzes this data to identify common patterns, representing user behavior over a specific period of time.


2.Is it necessary for TrueTest to operate within a production environment to achieve this? How could one deploy it in a QA or UAT environment to obtain production-equivalent results?

To install TrueTest in an environment, you’ll need to integrate a JavaScript code snippet into your application code and deploy it to the desired environment. If you aim to generate tests based on production user behaviors, you’ll need to deploy the code snippet to the production environment. Alternatively, you can utilize Google Tag Manager to install TrueTest without modifying the code.

Regarding operating in a QA or UAT environment, TrueTest can capture flows in any environment where the JavaScript snippet can be installed. This means it’s possible to record flows in pre-production environments as well. Additionally, you can run tests generated from one environment (e.g., Production) in another environment (e.g., UAT) by modifying the data driving the tests.


3. When a new TrueTest flow is captured again for the same interactions, does this automatically update existing object locators that perhaps need maintenance, or does it create new ones?

It will update the existing test objects.


4. Can TrueTest be used for scenarios like end-of-financial-year activity, which is only conducted for three weeks in the year, but then intensively and is critical?

You can select a specific time frame to generate user journey maps and test cases. If you need to create test cases for certain activities, choose the time frame during which you know these activities will take place.


5. Can we use TrueTest in closed environments without internet access?

If there is no internet access from the application you want TrueTest to capture the user interactions, then you cannot use TrueTest yet, until we release an on-prem version later next year. Please stay tuned for updates from us!


6. What will be the impact on the app’s performance by including this JavaScript?

There shouldn’t be any impact on the application’s performance. We take this issue very seriously; in fact, we have TrueTest installed on our own applications. We always conduct performance testing on various types of applications before every production release. Plus, you can quickly disable TrueTest if you suspect any issues.


7. Suppose one user has already added Java code snippets to a UAT (User Acceptance Testing) application. Can another user also utilize the same app with same code snippets or they need to add it again?

The snippet is connected to your organization, so it only needs to be installed once for a given application. Once a developer adds the JavaScript code snippet to an application, TrueTest will be enabled for all users that are navigating on that application.


8. How can I install TrueTest on my application?

You need to add a Javascript code snippet to the element of the application. More details can be found at: https://docs.katalon.com/docs/proof-of-concept/autonomous-test-generation/set-up-katalon-ai-agent

And…

https://docs.katalon.com/docs/proof-of-concept/autonomous-test-generation/autonomous-test-generation-with-katalon-truetest


9. Does the development platform make any difference in the recorded tests?

We don’t expect any issues regarding the development tool or framework you used for your web application.


10. Can we run Soap API test suites in parallel in a Test Collection?

Today TrueTest only generates web tests. However, if you have API tests in Katalon Studio, you should be able to run them alongside TrueTest-generated web tests.


11. How does it work with third-party integrations?

You need to set up TrueTest on the sites you are redirecting to. If that’s not feasible, then TrueTest would stop capturing user interactions upon redirection to the third-party site. When the user is redirected back to your application where TrueTest is enabled, user interactions capturing would start again by TrueTest.


12. What if the project already has many customized keywords and sub-test cases created and frequently used for specific needs? Can TrueTest leverage these?

At this time, TrueTest will generate new test cases based on the user journeys. You can enable TrueTest in the test environment where you run your current Katalon tests so that you can visualize the coverage of your existing Katalon tests against the real user journeys to understand how much of the business-critical scenarios your existing Katalon tests are covering. And then you can ask TrueTest to generate additional tests to bridge any coverage gaps.


13. Can it be used for API automation?

Currently TrueTest is focusing on UI test automation for Web applications. It is our goal to extend it to API test automation next year.


14. Does it support web apps that use HTML Canvas?

TrueTest technology does capture user interactions on Canvas-based HTML applications. However we’re looking for design partner customers to help us polish the processing of Canvas-based events. Let us know if you’d like to join TrueTest’s beta program so we can do this together!


15. Can I use TrueTest for Database testing?

Currently TrueTest is focusing on UI test automation for Web applications. We’re considering future roadmap items.


16. Does TrueTest support testing for web applications with lengthy functional processes, for example: public records, university enrollment management, etc?

Yes. Long workflows/user journeys are very interesting to be visualized in a user journey map so you can truly understand how your users actually navigate through those functional processes. Often our customers find that real users go through those complex workflows in ways they didn’t expect when they designed those functional processes and can make decisions to improve the customer experience based on TrueTest’s user journey map. Those long workflows are very hard to create tests for and having TrueTest generate those tests automatically that truly represent how users navigate those long processes is one of the biggest values TrueTest delivers.


17. Does the coverage check work if we are not currently using Katalon solutions?

Yes. If you run your manual or automated tests in your test environment with TrueTest enabled on your application under test, TrueTest will capture those browser interactions and display the coverage of those tests (manual and automated) against what real users do in production.


18. Does the coverage check work if we are not currently using Katalon solutions?

We are working on the capability of showing coverage for all testing activities, not just Katalon exclusively.


19. Does TrueTest act as a data generator?

TrueTest generates the synthetic data needed for its test case execution.


20. Can the user journey map be used for usability testing?

TrueTest’s user journey map gives you insights into how users flow through the application in production, which helps review test coverage and identify gaps.


21. Does TrueTest support mobile testing?

For now, TrueTest only supports web applications.


22. Our tests are built 99% on custom methods that call the keywords provided by Katalon and other existing custom, does TrueTest leverage existing test cases?

TrueTest does not leverage existing test cases or custom methods.


23. Do we need test suites, or can individual tests run from TestOps™?

Test cases generated by TrueTest are normal Katalon Studio test cases. You can open them in Katalon Studio and execute them individually or as part of test suites. If you want to use the Schedule Test Run feature in TestOps, the test cases must be added to test suites or test suite collections.


:pushpin: [You are here] Questions about how TrueTest works, how to install TrueTest, and TrueTest’s use cases
:pushpin: Questions about Security and Data Safety
:pushpin: Questions about Platform Integration
:pushpin: Questions about Pricings

:point_up: Back to top :point_up:

Security and Data Safety

1. Where does TrueTest store captured data?

TrueTests stores data on Katalon AWS.


2. Can TrueTest be deployed in any environment or just in production?

You can deploy TrueTest on any environment as long as that environment can send data to TrueTest and TrueTest can access your test environment to generate test cases.


3. Is GDPR compliance guaranteed?

Please refer to the documentation below regarding our approach towards data privacy:

https://docs.katalon.com/docs/proof-of-concept/autonomous-test-generation/protect-user-data-privacy#ariaid-title1


3. Does TrueTest impact HIPPA regulations?

No. TrueTest removes PII data at the source (i.e. user’s browser), so it never sees or captures PII data. In addition, the Katalon customer can choose what additional data types and UI elements that should not be exposed to TrueTest for capturing.


4. Is it possible to encrypt some personal data provided during behavior tracking?

Personal data will be automatically excluded BEFORE sending the data to TrueTest. You have full control over what you want to send to TrueTest.


5. What about user data (which we don’t have in the test environment)?

Users’ sensitive data will be automatically excluded. You will need to provide login credentials for any authentication required.

Please refer to the documentation below regarding our approach towards data privacy:

https://docs.katalon.com/docs/proof-of-concept/autonomous-test-generation/protect-user-data-privacy#ariaid-title1


6. Can you mask visual checkpoints (date/time stamps, etc)?

You can define which part of the page to ignore by following the guideline below:

https://docs.katalon.com/docs/analyze/analytics/visual-testing/use-testops-visual-testing#ignored-zones


7. Is data obfuscated at the source and not sent to Katalon’s cloud?

We apply a triple-step approach for sensitive data:

  1. Automatically recognized and excluded at source,
  2. Re-scan and delete upon receiving,
  3. Routinely scan using AWS tools.

For types of data to be considered sensitive, please refer to our documentation below:
https://docs.katalon.com/docs/proof-of-concept/autonomous-test-generation/protect-user-data-privacy#ariaid-title1


:pushpin: Questions about how TrueTest works, how to install TrueTest, and TrueTest’s use cases
:pushpin: [You are here] Questions about Security and Data Safety
:pushpin: Questions about Platform Integration
:pushpin: Questions about Pricings

:point_up: Back to top :point_up:

Platform Integration

1. Where can I execute test cases created by TrueTest?

TrueTest generates Katalon Studio test cases. Once you have the test cases, where you want to execute them is up to you, including Katalon Runtime Engine and Katalon TestOps.


2. Is TrueTest integrated with Katalon Studio?

Yes. Tests generated by TrueTest are added to your git repository. So you can open, view and run those tests from Katalon Studio just like you normally would with a Katalon Studio-generated test.


3. Is there Jenkins integration?

Certainly. Katalon support Jenkins integration. Please refer to our documentation below for more information:
https://docs.katalon.com/docs/execute/cicd-integrations/jenkins-integration/use-katalon-plugins-for-jenkins-integration/use-katalon-plugins-for-jenkins-integration-on-windowsmacos#ariaid-title1


4. How does TrueTest interact with customized methods?

TrueTest will not use your existing test cases or custom methods.


5. Does TrueTest make KatalonStudio’s web automation capabilities obsolete?

TrueTest brings a different approach to test automation. It complements Katalon Studio’s.


:pushpin: Questions about how TrueTest works, how to install TrueTest, and TrueTest’s use cases
:pushpin: Questions about Security and Data Safety
:pushpin: [You are here] Questions about Platform Integration
:pushpin: Questions about Pricings

:point_up: Back to top :point_up:

Pricing Questions

1. Is TrueTest included with Katalon’s enterprise tier or is it a separate subscription?

TrueTest is in Beta, so you can just register for the Beta program and get it for free.


2. For those who want to use it, do they also need to have Katalon installed or need a license?

TrueTest will be a separate feature, but you can get free early access today. The real users who interact with the instrumented web application do not need to have Katalon installed or a license.


3. Are there going to be any limitations on how many test cases/test suites TrueTest can generate?

There are no limitations during the Beta period. We are still working on the pricing model when TrueTest goes to GA.


4. Are there any limitations on how many test cases TrueTest can generate?

We are still working on TrueTest’s pricing model, but we expect no limitation for the number of test cases generated.


:pushpin: Questions about how TrueTest works, how to install TrueTest, and TrueTest’s use cases
:pushpin: Questions about Security and Data Safety
:pushpin: Questions about Platform Integration
:pushpin: [You are here] Questions about Pricings

:point_up: Back to top :point_up: