Drop us a line
CI/CD and want
to be featured
in our blog?
Continuous Testing: The New Frontier of Continuous Integration and Continuous Delivery
The only way to learn how good your product is or what kind of undesired effects there might be is to test it. Nowadays, testing your software is no longer banished to a separate phase and separated from writing code which is exemplified in the term “continuous testing”: using appropriate approaches, techniques and tools in every step of software development from design to deploying to production.
In this article, you’ll learn about continuous testing, its benefits and role in software development, and how it fits into the process of continuous integration and delivery (CI/CD). To learn about the basics of CI/CD, check out “What is Continuous Integration and How to Benefit from It?”.
What is software testing anyway?
By nature, testing is an exploratory process of uncovering risks and unintended effects, and evaluating the quality of the product to provide information for decision makers or any other parties with vested interest in the product. Software quality is ever so difficult to define in a nutshell precisely because “quality is value to some person (who matters)” (Gerald M. Weinberg). Hence, sometimes talking about software testing reminds of the story of blind men and an elephant where each of them had a different understanding of what an elephant is like based on the part of the elephant the interacted with.
One key challenge of testing is understanding who matters most. Which type of users in our user base are the most common? Who is impacted the most if a business or technical risk is realized? Which of the stakeholders have the most at stake? What level of software quality can we afford and what level of quality do users and stakeholders expect? Answers to these questions help you understand what quality means in your context, and how your tests can help evaluate it.
The goal is to question the product in diverse ways which also means testing it in diverse ways by using various approaches and techniques. For instance, to answer questions about the product’s usability you can undertake user testing, monitoring and measuring user behavior, conducting interviews, etc. However, to answer questions about product’s performance and stability, you will need a different set of skills and tools.
While humans are best at evaluating and judging quality, tools are helpful in probing deeper, getting data faster, and extending human capabilities. After all, human eyes can’t see very far but a powerful telescope is a tool that makes discovery possible.
What is continuous testing?
For quite a while, software testing was trapped in a separate phase at the end of the software development lifecycle. With the arrival of agile methodologies, the walls have started to crumble and software testing has shapeshifted and moved towards the center stage.
Where does testing fit in the software development cycle? It fits everywhere! To release high-quality software and stay in competition, you need to test early, test often, automate what is reasonable, and deliver continuously. Continuous integration was an important step that sped up development. Continuous testing is the next frontier to conquer.
Continuous testing is the mindset of adopting tools and practices in every step the software development process that are used to test the product and deliver valuable information and feedback about its quality.
Next we’ll discuss in more detail where and when continuous testing happens in each step.
Continuously test your idea
Testing early means blitzing your ideas and designs before any of them becomes represented in code. Of course, you will continue questioning and testing your idea in the next steps as well. The key is to analyze and investigate business and technical risks early in order to adjust the design and determine additional changes to be made. Delivering value to customers entails both doing the right thing and doing the thing right (which means establishing a lean and value-driven process and implementing continuous testing).
Continuous testing and automation
Test-driven development mandates coming up with tests before writing code to test your implementation quickly. Code reviews are meant to reveal potential risks in the code, and this is the step where you can also question the original plan or design (which means you’re testing it).
Your software quality strategy determines what type of automated tests and where are relevant and valuable to give rapid feedback about technical and business risks. You have to continuously create, run and also evaluate them: are these tests giving me what I need? For example, BDD (Behavior-Driven Development) drives conversations and collaboration around examples of behavior to exemplify the impact of a scenario to be implemented. It’s also possible to encode the behaviors using tools such as Cucumber and Selenium to run these tests at the right time.
There are plenty of opportunities to automate things at which machines are better than humans (while leaving interpretation of test results to humans). Unit tests, integration tests, UI automation, performance, checks for API endpoints… When risks have been identified in earlier steps, you can determine what type of tests should be at your fingertips and run once the code is done. Your CI/CD platform will play a major role in all of this and in supporting getting the builds frequently and running the relevant tests. CI/CD is the bedrock of being able to integrate and test continuously. Continuous testing does not equal continuously executing automated tests in the CI/CD pipeline, but it’s comprised of a plethora of activities ranging from abstract to technical, each of which is designed to improve quality.
Nevercode supports several test frameworks to supercharge your CI/CD process.
Continuous testing, monitoring and analytics
Once the code has been deployed (by the way, have you tested your deployment pipeline and strategy?), it’s time to monitor, log and collect data about the system’s behavior and that of the released feature. Does it perform well? Does it even get used? Can we associate any problems that have cropped up with the new release? Set up your monitoring wisely and you will get the answers.
Analytics about your product will also help you test out your assumptions about the performance, usage, stability, etc of your product. Nowadays, it also makes sense to sometimes just test in production (while mitigating potential risks and having a well-tested roll-back process) because it can accumulate more valuable information about quality and do it faster than testing it “in house” due to scale, insufficient data, etc. Naturally, when you want to test in production, you will want to make your batch size smaller, i.e. release a little bit at the time to avoid issues. This will have an impact on everything you do, starting from design to coding to deploying and monitoring.
If there’s anything we want you to take with you from reading this, it’s the following:
- Software testing is so much broader than simply “validating” the product or writing test automation
- Continuous testing, if done right, offers diverse opportunities to discover essential information about the quality of your product
- CI/CD is a pillar of your continuous testing efforts