The Need for a Testing Platform for Playwright

:pushpin: For more articles like this, visit our Product Insight corner

:pushpin: If your job involves using automation testing frameworks and tools such as Playwright, then we would love to hear from you. From now till Apr 21, 2023, complete our survey and you will get a chance to win exclusive gifts from us! :gift:


For automation engineers and QA practitioners, Selenium has been the go-to choice for its versatility and vast community support. However, the automation community is now embracing other promising non-Selenium tools, such as Playwright: a powerful, open-source, feature-packed framework built by Microsoft.

Since its introduction in 2020, Playwright has steadily gained popularity. As of March 2023, some interesting statistics from the GitHub repository of Playwright include:

  • 48.3k stars
  • 2.4k forks
  • 382 contributors

So why has Playwright become so popular? How is it catching up with other widely used frameworks like Selenium and Cypress?

This article aims to answer those questions by discussing the capabilities and limitations of Playwright, as well as the potential for future adoption of this framework into the Katalon Platform.

About Playwright

Playwright is an open-source framework designed for reliable end-to-end testing of modern web apps. As a cross-browser, multi-platform tool, it supports all popular rendering engines, including Chromium, Firefox, and WebKit, on Windows, macOS, and Linux.

The Playwright library operates as a unified API, allowing developers to write automation scripts and launch them via the command line interface. It also supports multiple programming languages, including JavaScript, TypeScript, Python, .NET, and Java. This makes Playwright an approachable and accessible framework for new users.

Despite being a relatively new entrant to the automation market, Playwright has gained significant traction. In the 2022 State of JS developer survey, Playwright showed impressive growth in usage and retention rates.

Benefits of Playwright

Let’s dive into the feature set of Playwright.

Compared with other existing solutions, Playwright has successfully accommodated the features of a complete web testing framework. Some of the prominent features include:

  • Quick and easy setup: Playwright requires very minimal installation to get started.
  • Cross-browser support: The framework supports all the core engines of popular browsers: Chromium for Chrome and Edge, WebKit for Safari, and Firefox.
  • Auto-wait: A form of smart wait feature. Playwright waits for dynamically loaded elements before taking the next steps, as opposed to using the flaky fixed wait time.
  • Web-first assertions: Playwright automatically retries checks until a necessary condition is met, before proceeding.
  • Execution trace: The tool can trace test execution and provide necessary insights in the form of videos and screenshots.
  • Browser context: Instead of using multiple browser sessions, Playwright leverages the use of browser context to create multiple browser profiles for parallel testing. This also delivers complete test isolation with minimal overhead.

In addition to the rich feature set, the team at Microsoft has also equipped Playwright with powerful tooling.

Test Generator and Inspector: Playwright comes with the ability to generate tests out-of-the-box, making it a great way to quickly get started with testing. When you launch the Test Generator, Playwright opens two windows: a browser where you can interact with the website under test and the Inspector, which helps you write and debug scripts.

Trace Viewer: Being able to trace test execution and record snapshots means that you can perform time travel debugging with Playwright. The Trace Viewer tool allows you to step back and forth through each action of your tests and visually see what was happening during each action.

Multiple Reporters: Playwright comes with multiple built-in reporters for different needs, such as HTML, JSON, JUnit, and the ability to create custom reporters.

To quickly view a test report, you can serve up an HTML reporter on your local machine and easily walk through the steps of your test. Combining this with the trace file, Playwright gives you powerful options for dynamically viewing reports and, at the same time, debugging your tests.

Playwright limitations

While Playwright is considered on par with top testing frameworks, it does come with limitations, such as:

  • Limited community and documentation support: Although the interest for Playwright is growing, it’s still relatively small. When you hit a roadblock with the tool, it’d be more difficult to find documentation and recipes.

  • Lack of infrastructure for collaboration: Playwright is a feature-packed framework but does not provide the infrastructure to run tests and collaborate with other team members. This can make it challenging to share assets, get updates, manage test reports, especially when dealing with frequent code changes.

The current practice for Playwright users is to work with Visual Studio Code, and synchronize with team members through Git. In a large test project with many contributors, this can be challenging as version control only supports managing source code, not test artifacts. When reviewing changes, testers can be bombarded with irrelevant data and not entirely invested in the testing activities.

To run Playwright tests, users have to prepare test environments and schedule test executions, which can be difficult to keep track of. Also, test reports from these environments need to be centralized. The default Playwright report files can be handy, but they provide little insights for cross-platform testing.

What if the Katalon Platform supports Playwright?

While Playwright is a powerful automation tool, it’s hard to complete the testing pipeline using it alone. In an enterprise setting, the pipeline involves multiple stakeholders, including developers, quality engineers, QA managers, and others who want to audit data and ensure standards are met.

For this reason, test managers and team leads using Playwright often look for a comprehensive testing platform that allows them to complete their pipeline for automation, from test planning, authoring, management, and execution to reporting and analytics. This is where the Katalon comes in.

The addition of Playwright into Katalon, our all-in-one platform, can open up new possibilities for testers. Let’s imagine what the Playwright-Katalon integration would look like.

By setting up a Katalon-Playwright test project, users can create a central container for all testing activities and synchronize it with other team members. The multi-level management implemented in the Katalon Platform, including Account, Organization, Team, and Project with different roles, facilitates the involvement of stakeholders in the enterprise testing pipeline. For example, Billing managers can take care of subscriptions and payment methods, freeing up testers from the hassle of distributing licenses among themselves.

Once the test project is ready and accessible, test engineers can begin the authoring process with Katalon Studio, which provides a feature-rich environment for test automation. With the VS Code extension already being popular among the Playwright community, users of Katalon Studio can enjoy the same benefits and features. Katalon Studio’s integrated features, such as its Object Repository, Record and Playback, and built-in keywords, help users create tests quickly and easily.

In addition to the authoring process, Katalon offers solutions for managing test artifacts, including test cases, test suites, and page objects. These artifacts can also be linked to other management tools like Jira and Xray, providing a streamlined approach to test management. As a framework, Playwright only offers simplistic tagging as a management feature, while Katalon’s custom fields and tags allow for extensive metadata association and enhanced organization of test artifacts.

The platform also provides an extensive set of test environments through Katalon TestCloud. This allows users to configure operating systems and devices for cross-platform testing and execute Playwright tests in various CI environments using Katalon Runtime Engine and Docker.

After tests are executed, the platform’s analytics module provides advanced test reports, which are automatically uploaded to TestOps and analyzed for insights, such as productivity, requirement coverage, platform coverage, and release readiness. This feature helps users to make data-driven decisions based on the results of their test runs.

Currently, the Katalon Platform has already embraced Selenium and Appium for Web and Mobile testing. With the addition of Playwright support, Katalon can become even more versatile and capable of handling a wider range of test scenarios, providing users with a comprehensive and robust testing ecosystem.


:pushpin: For more articles like this, visit our Product Insight corner

4 Likes

Welcome news - I think.

But the article doesn’t make clear what is being offered.

Can a Katalon Studio user develop Playwright (i.e. JavaScript-based) tests directly in Katalon Studio? Or does Katalon platform merely provide a way to hook up Playwright tests into a test pipeline?

This is not my first time talking to you guys about Playwright (or Cypress). I’ll add you to the PM exchange from May 2020.

2 Likes

Well … at least, is considered :slight_smile:

I was hoping for a response to my post above. Since that didn’t happen, I’ll dissect the only paragraph that seems to suggest, rather obliquely, what is on offer:

Now let’s look again with the marketing fluff and any extraneous “noise” removed:

I had thought that the preceeding might answer my question…

But it’s merely talking about setting up a project in TestOps, right? Nowhere in that paragraph does it talk about a Katalon *.prj, where a user might write a JavaScript test case directly in Katalon Studio where the JS code is treated as a first class citizen (e.g syntax coloring, member lookup, etc).

What I think you are offering, though it isn’t made clear, is:

“we have found a way to host/incorporate Playwright tests (written in VS Code) within TestOps.”

What concerns me is the increasing use of “website marketing materials” being used on the forum without the requisite technical background you must know we’ll want to see.

This article is posted under Product Insights but it manages to avoid providing any insights by drawing a veil over a ton of pretty obvious questions.

This is about Katalon Platform.
The Standalone product will never support this :smiley:
You got it?

Hi guys, sorry for the late reply, I’m a member of Katalon’s product team. This article reflects what we are thinking about the integration between Playwright and Katalon Platform. It would be nice if we can have a powerful framework like Playwright in Katalon Platform and users can use Katalon in their test stack. We are still working hard on it and researching many use cases from users to make this initiative reliable.

A testing experience survey is underway, which you can check out here. Any sharing or feedback about Playwright is welcome and very valuable to Katalon team.

But Katalon is not the name of a product. Are you saying, within KP, users can use Katalon Studio to develop Playwright tests?

Done.

1 Like

I understand your excitment, but before moving forward, please you take this survey:

  • what is your actual experience with automated testing and with testing process, generally speaking?
  • provided you answered the above with ‘not null’ : how many frameworks you know and how many from those you actually used?

After that we move forward.

Hi @Russ_Thomas, many thanks for your submission. Honestly, Katalon Studio is a Selenium-based tool, so if we only talk about Katalon Studio, it is hard to support Playwright with Katalon Studio. But it is not impossible, so I cannot make any confirmation now.

Hi @bionel, I’m unsure about your question’s purpose, but I will honestly answer it. I have a bit of experience with popular testing frameworks like Selenium and Cypress. It means I’m still a newbie :)))) Your sharing will help me a lot.

1 Like

Well … I was thinking that it may be like this.
Well, the idea is …
Altough I understand why this idea is verry attracting, before moving forward, few things to be clarified.

Playwright is a tottaly different framework than Selenium, using totally different technologies.
Shortly speaking, is a different animal.
Altough it has a Java wrapper, so, someone may think that bringing it into Katalon is doable, it is not trivial and few things has to be clarified.

  • Will this be supported to actually develop Playwright tescases (write, run, publish reports)?
  • If yes, will be this available also for the Standalone product or it is just for Platform?
  • If not, please define a bit better what is the intended scope of usage, e.g will be a totally new component for Platform? Something else?

I am a bit concerned that, currently, Katalon team is looking for yet another unicorn, in the detriment of fixing the issues with the current products and improving them.

Cheers,
bionel

3 Likes

Forget Katalon Studio (Eclipse + Groovy).

Why not you consider a new GUI product “Katalon for Playwright” on top of pure JavaScript/TypeScript?

However I would not need it. All I need for Playwright is text editor. I don’t need “Manual mode” GUI for Playwright.

2 Likes

That will be a better approach.
And please, forget about GTK. GTK is awfull.

1 Like

Yea, those are the questions that my team is trying to answer. Our goal is still to provide the most comprehensive solutions for users - a full flow to develop Playwright test cases (write, run, publish reports) like you mentioned. Those are still processing, so I will update you when we have new insights about it.

This is an amazing idea. It made me ask a question, is an on-cloud solution better for you, or a local tool is better?

@kazurayam, through your reply, I can understand that authoring is the most important part for you, but how about other parts like executing or reporting being implemented?

1 Like

For the execution part, Playwright can run on any CI tool.
For the reporting part, again, Playwright has pretty decent already built in plugins:

To publish them it will be only a matter of the CI pipeline.
I never understood why publishing should be a part of the testing tool … reports can be uploaded later to any decent TestCase management tool provided such support a decent format, like Junit.

So, here we come into the tricky part.
Some time ago I was playing with testrail-cli tool to study an alternate approach to upload test reports into testrail other than a baked in tool in a certain framework (for internal use, I cannot disclose data about it)

ref: https://support.testrail.com/hc/en-us/articles/7146548750868-TestRail-CLI

Everyting was fine with reports generated by Pytest and Playwright.
Anyway, attempting to upload a Junit report generated by katalon miserabbly failed.

Why?
Because testrail-cli need a testsuiteID in numeric format, if provided.
Most of Junit reports generated by any othe tools don’t provide a suiteID at all.
If I have to upload by execution to an existing suite, I will provide the ID through the tool parametters, otherwise if missing Testrail will generate a new one for me.

But ofcourse, Katalon is special and generates a suiteID as a string :smiley: in addition to the TestSuite name (which is the same string)
Therefore, to be able to publish such into Testrail, I had to manually edit the report and remove the offending tag.

So, I started to question myself first:

  • OK, I understand the need for custom reportings ‘on the fly’ … since our framework also do this somehow … but …
  • Why to break standard reports formatting generated in addition? (which our framework don’t do)

So, I think, Katalon should reconsider their products (Katalon himself and TestOps) regarding standard report formatting first of all.
After that, we can discuss about other integration levels possible with other various tools.

I would be happy if I can use Jupyter Notebook

It is what you call TestOps, isn’t it?

vscode has a plugin for such, as far as i klnow, which should be decent.
i haven’t tried yet, nor Jupiter himself, but it is on my TO LEARN list (because some other hobby i have like data analysis)
:slight_smile:

It was even more amazing ~3 years ago when I discussed it with @ThanhTo and @duyluong. It’s long overdue that JS became a first-class citizen in Katalon products.

Do you mean this?

1 Like

i had in my mind the usage for python, but the example mentioned by you is cute also