Consider two trends that have been going on in software development for a while now:
-
With software systems moving away from monolithic designs to using much more distributed architectures has come a rise in the use of APIs to connect all the different components and layers. As a result, knowing how to test APIs or use APIs for testing has quickly become an indispensable skill for any modern software tester.
-
Market demands require ever shorter software development release cycles, which leads to software development teams using Continuous Integration (CI) and Continuous Delivery (CD) to automate as much of these development cycles as possible.
Combining these two trends means that more and more teams are looking to run API tests as part of a CI/CD pipeline. In this article, we will take a closer look at how to succeed with API testing in a pipeline.
Benefits of API testing in a pipeline
Let’s take a look at the benefits of both API testing and running API tests as part of CI/CD pipelines.
Here are some of the benefits of API testing:
-
Your APIs are the interface of your component, service, or application to “the outside” (another component, another application, or even an end user). Making sure that the interface behaves as expected is a good idea!
-
Testing at the API level, especially API test automation, is much more efficient and much less complex compared to writing UI tests.
-
Tests at the API level have a scope closer to end-user needs compared to unit tests, which makes them valuable in getting information on whether your software is able to meet end-user requirements.
Running API tests as part of a CI/CD pipeline has many benefits, too:
-
It allows you to automatically gather information about the quality of every change, commit and build of your software.
-
It ensures that the tests you’ve spent so much time and effort on are run often, increasing their return on investment.
-
It allows development teams to get faster feedback on their work and ensures that deployment into an environment (which may be production) meets certain quality standards.
To get the most value out of adding your API tests to a CI/CD pipeline, we’ll need to take several things into consideration.
Don’t write API tests where unit tests are sufficient
As we have seen, tests written at the API level have many benefits, but that doesn’t mean all tests should be written at this level. As a rule, you should never write an API test to provide information that can be provided by a (few) unit test(s).
As an example, while it might be useful to write one API-level test to check that a malformatted path parameter value results in a response with an HTTP 400 status code, testing several different values all result in the same behavior (i.e., the value is rejected and throws the exception that will eventually result in the HTTP 400) is best pushed down to the unit test level.
Remember: the smaller your tests, the faster the feedback.
Spinning up API instances in-memory
Where do you want to run your tests at the API level, or in other words, where do you want your tests to perform HTTP calls (or whatever your transport protocol equivalent is)? You can spin up your API in memory as part of the setup stage of your tests. This enables you to run your tests on every machine, without having to wonder whether the API instance you want to run your tests against is running and accessible. Frameworks like Spring Boot for Java make this a breeze.
When spinning up your API is part of your test execution process, your tests can be run over and over again, from any machine with the correct runtime. This makes running tests, continuously and unattendedly, in CI/CD pipelines a lot less frustrating.
Mocking downstream services and components
Of course, your APIs do not live in isolation, meaning they will rely on downstream services and components (databases, for example) to serve their consumers. This introduces additional complexity when running tests, especially if you want to run those tests early, often and unattended. Components might not be available. It might be difficult to set up the right data or to trigger edge case behavior, and so on.
One strategy to deal with this is by mocking the behavior of those downstream services and components. Use mocks that exert the behavior you either expect from the real dependency (useful for happy path testing) or erroneous behavior that is difficult to trigger in real components (useful for exception handling and resiliency testing).
No longer having to rely on other components and their availability is a great way of accelerating the ability to test your API early and often. You should always keep in mind, though, that your mock implementation behavior might drift from the behavior of the actual component it is simulating. And that in the end, it is not a substitute for testing against “the real thing”. With careful use, however, mocks can be of great help in speeding up your testing efforts, especially as part of a CI/CD pipeline.
Adopting contract testing
One way to deal with the challenges of integration and end-to-end testing of highly distributed systems is contract testing. Contract testing is an approach that attempts to break down the integration testing puzzle by focusing on individual consumer-provider integrations rather than systems as a whole.
Contract testing makes the steps in the process asynchronous, meaning that consumer and provider can perform their part of the process as part of their own pipelines, without relying on the other half to be present in the same testing environment at a particular point in time.
It goes beyond the scope of this article to describe how contract testing works in detail. However, the nature of contract testing, together with the fact that it makes integration testing an asynchronous process, is an important enabler in making API-level integration testing part of a CI/CD pipeline.
As we have seen in this article, there is more to making your API tests part of a CI/CD pipeline than simply adding another step to your build. However, to ensure that your distributed systems keep working as expected, it is vital that your API tests are run on every build, and the techniques and suggestions in this article can bring you a long way in succeeding with your API testing efforts in a Continuous Integration and Continuous Delivery setting.
