Hi all… I’m new here obviously and am looking for ideas/suggestions on getting started. I work in a small non-profit organization where software development is not our prime focus. I’m one of a few people in the entire organization that has software development in my background (for me it’s 30+ years of mostly embedded dev, low-level unit tests, test plans, and so forth).
We currently have two websites that are developed by an outside organization (contract labor) that also manages our websites. These websites are somewhat complex (or in the process of getting more complex) and automated testing is a new arena in our process. Up to this point both websites have had manual testing.
Our dev-staff seems interested in incorporating tests but they’re a pretty small place and we’re just one of their long-time clients. Obviously I’ll need to take the lead on this which is fine.
My ultimate question is : What’s the best way to start testing a live website where testing is being added after the fact? Yes, I’m 100% aware this is NOT a good way to go but it has already been done … so I just need to adapt and start somewhere.
I’m thinking we need a combo of web-testing and probably some low level unit tests that can test functionality for databases, sending emails and that sort of thing probably using PHPUnit as the code is PHP/Laravel based…
My ultimate goal is to have a series of tests of different types that can be automated each time they’re wanting to deploy a new version of the site(s). I want a system setup that can be used for the next handful of years with alterations as needed for new functionality, etc.
It’s fine. Katalon Studio will let you create umpteen test scenarios that cover all possible paths through your web app (end-to-end testing – I like to call them “user stories”).
Unit tests are another thing entirely. Katalon does not offer in-situ unit testing.
You’re describing a setup very similar to mine (company size, complex web app, ever changing). The only thing I would add to what you’ve described is regression testing. For every bug logged against your app, add another test that specifically tests for the bug’s presence. Put the bug ID/number in the title of the test case.
Test every user story (login, account/profile management, app functionality … logout)
Thanks @Russ_Thomas for your reply! Yes, regression testing is a big one that we would certainly want to handle and make sure these tests are even run at the dev level perhaps nightly just to ensure people aren’t checking in things that cause problems elsewhere.
For unit testing, I wasn’t concerned that Katalon doesn’t handle those – they’re too down in the weeds so to speak … I was expecting to handle those separately.
I don’t know what’s the best. Let me explain my way how to use Katalon Studio.
I always develop 2 sets of codes for quality assurance of a web app project. One is the “Unit tests”, another is the “Visual Inspection”. The two approaches complement each other. I use a term “test” as a code that enumerates some specific conditions to verify if the system works as expected. On the other hand, I use a term “inspection” as a code that expects nothing specific to occur but just observe as many page views and as often as possible to find anything unexpected has occured.
You know what “Unit tests” is — by PHPUnit, JUnit, Cucumber — which unit-testing framework you use, it does not matter. “Unit tests” starts with the low level of abstraction of program codes, and goes upward. The web app developers will (must) develop the Unit tests anyway.
I would begin my WebUI automation with “Visual inspection” and later I would add some detailed UI testing if necessary. However, before adding any verification, I always ask to myself: “should I implement this test as a part of the unit-testing driven by JUnit, or as a part of UI-testing driven by Selenium?”. I am inclined to implement the tests as a part of unit-testing by JUnit, not in the ui-test set by Selenium. Therefore my set of Selenium-based UI testing rarely grows large.
I think that the smaller the UI test set is the better because automated UI test requires human resources to maintain. If you develop your UI-testing detailed and large, it gets increasingly likely to break in future. Once broken, you would be possibly tempted to drop it off rather than trying to fix the breakages.
On the other hand, the code to perform “Visual inspection” is usually trivial: open the top page URL, navigate the menu to visit sub-pages and take screenshots; that’s all. My library helps comparing the screenshots and compiling reports. It has smaller risk of breakages. It’s maintenance cost is rather small. But it helps developers by notifying unexpected change (bugs) in page views. That’s enough for me.