Table of contents
- ‘Deployment Pipeline’ & Role Of Automated Testing in CI/CD
- How To Make The Best Out Of Test Automation In CI/CD Pipeline?
- Incremental Changes & Timely Communication
- Identification Of Tests That Can Be Automated
- One-Click Migration
- Exploiting Parallel Testing As A Best Practice For CI/CD Pipeline
- Learnings From Previously Executed Projects
- Why Documentation Is An Absolute Necessity?
- Central Repository For Code Development & Maintenance
- Rollback With Version Control
- Staging Environment Loosely Mimicking The Production Environment
- Involving Relevant Stakeholders In Test Code Development
- Incorporate Feedback For A Robust Automated Testing Strategy In CI/CD Pipeline
- Frequent Code Commits For Micro-Agility In Test Automation For CI/CD Pipeline
- Seamless Coordination! Remember, it is CI + CD!
- Keep It Simple & Organized By Executing The Smaller Test Cases First!
- Transparency Within The Team & Across The Teams
- Choosing The Right Tool for CI/CD Process
- Conclusion
Every software project involves some kind of ‘processes’ & ‘practices’ for successful execution & deployment of the project. As the size & scale of the project increases, the degree of complications also increases in an exponential manner. The leadership team should make every possible effort to develop, test, and release the software in a manner so that the release is done in an incremental manner thereby having minimal (or no) impact on the software already available with the customer.
In this article, I would touch upon the CI/CD best practices with respect to test automation to help you pace up your go-to market launch.
We have already covered CI/CD in one of our previous article on What is Continuous Integrations and Continuous Delivery? Where we discussed what they are about? Tools of trade involved in the CI/ CD pipeline. Every bug that is uncovered during the testing/verification stage goes through the rigorous process of development, integration, testing & closure. In case this activity is done manually, it could cost a significant amount of man-hours and verification effort, hence it is ideal to have ‘test automation’ in CI/CD.
If you’re new to Selenium and wondering what it is then we recommend checking out our guide — What is Selenium?
Need a great solution for cross browser testing on safari browser online? Forget about emulators or simulators — use real online browsers. Try LambdaTest on safari browser.
‘Deployment Pipeline’ & Role Of Automated Testing in CI/CD
An automated system, called as Deployment Pipeline in CI/CD does the job of automating the testing of the incremental builds that are available on the server. Since the entire process is automated, the overall Turn-around time (TAT) would be significantly lesser when compared to manual testing.
In most of the scenarios, it is not possible to completely automate the testing. There could be some test scenarios where there is manual intervention required or scenarios where manual observation is required to decide whether the test has passed or not. Though automation is an integral best practice for CI/CD pipeline, identifying test scenarios that would fetch better results if they are automated is considered a pivotal best practice for CI/CD.
Though there are several benefits of using automation testing, some of the key benefits of using test automation in CI/CD pipeline are below:
Faster bug closure — Issue Detection -> Issue Fix -> Issue Closure.
Effective utilization of overall resources in hand i.e. testers, testing infrastructure, etc.
Ability to execute tests in Parallel.
Consistency in test planning & execution.
Minimal requirement of technical skills required for automated test-case execution.
Test your native, hybrid, and web apps across all legacy and latest mobile operating systems on the most powerful android online emulator.
How To Make The Best Out Of Test Automation In CI/CD Pipeline?
While the scope of test automation in CI/CD pipeline may vary from one project to another, there are certain best practices for CI/CD that can be applied to any project, irrespective of its scale & size.
Incremental Changes & Timely Communication
Developers can follow a big-bang approach to develop a new feature or fix an issue reported by the test team. This means that the developer will use the option where he will push the feature implementation at one shot. Though the implementation job is done, there are issues with this approach. The major drawback is that it becomes extremely difficult to isolate an issue (if there is a problem with the implementation
A wiser approach would be to break down the feature into different sub-features and make use of unique feature flags for each feature. This technique not only helps in isolating potential issues but can be instrumental in making incremental feature-builds whenever the need arises. It also reduces the probability of having integration problems when the final feature (which is a combination of many sub-features) is pushed to the main-line/production branch.
There could be scenarios where there are inter-dependencies between features i.e. feature ‘A’ is dependent on feature ‘B’, in which case either developer has to wait for 100% completion of the feature on which there is a dependency. By making proper use of stubs & feature flags, both the feature developers can avoid the deadlock situation since stubs would be used for implementation which is planned/under progress. Once the required feature is ready, the developer has to get rid of stub-based/dummy implementation which was only a placeholder for implementation of a known interface.
This development activity requires careful planning and timely interaction between the development teams to make the most out of test automation in CI/CD pipeline. Any kind of indiscipline in following this approach could impact your overall test schedule.
Run your Playwright test scripts instantly on 50+ browser and OS combinations using the LambdaTest cloud. Read more.
Identification Of Tests That Can Be Automated
As mentioned earlier, it is unlikely that 100% of the tests can be automated since there would at least be some tests where manual testing would be more effective when compared to automation tests. Since test automation is core to CI/CD pipeline, realizing those test cases which can be automated is a crucial best practice for CI/CD.
Two broad categories of tests that can be automated are:
Tests that are executed on a frequent basis: If such tests are executed ‘manually’ by testers, it may be prone to errors since the productivity may reduce as the testers have to execute the same test multiple times in a day. Take the case of a web product that has to be tested for browser compatibility testing. Some test-cases may involve capturing the screenshot of your web-app while it is being tested on different combinations of browsers/devices/operating systems. Taking a screenshot for each of these combinations could be a tedious task. Hence, automated cross browser testing could free your testers for writing effective test cases.
Tests that require knowledge & have the dependency on specific set of testers: Dependency on resources (developers/testers) that only have the domain knowledge required for test execution can be risky if that resource is not available during a critical phase of the project. As only that tester is aware of the pre-requisites and procedure for testing, his absence can impact the overall project deliverables. Such situations can be avoided by incorporating test automation in CI/CD pipeline, so that project deadlines are not adversely affected
There could be many other scenarios where automation testing can be used and the intent should be to make use of tools & test platforms (which are scalable) that can help you to achieve that audacious goal.
One-Click Migration
The effort to move the code changes to the production environment can be greatly reduced if there is a one-click feature for migrating the code from one application environment to another.
A well-architected CI/CD pipeline should have one-click migration since it reduces the friction (in code migration) between different operations. Opting for good cloud infrastructure that provides this feature and efficient & elegant usage of test automation could optimize the development & operation process. Pushing your development through a single click is great to aim as a best practice for CI/CD.
Exploiting Parallel Testing As A Best Practice For CI/CD Pipeline
Another best practice for CI/CD would be the parallel test execution. Once you have identified the tests that need to be automated, the next step should be to incorporate the factor of ‘parallel execution’ in your test approach. Having automated testing as a best practice for CI/CD pipeline already accelerates the entire process but the results would be much better if it is coupled with Parallel testing. You can have number of automation tests being executed, simultaneously, which can yield results much faster.
You cannot get the best throughput out of parallel testing if the tests are executed on one single machine. Such a scenario would hog the critical resources on the machine e.g. CPU, GPU, etc. which may slow down the execution of the other tests running on the machine. Hence, the infrastructure on which the tests are executed matter a lot. For testing a web-app/website, having an in-house infrastructure for test deployment & execution might not be an ideal solution (in terms of cost & scalability). This is where LambdaTest comes in to help you perform parallel testing with automated testing frameworks such as TestNG, Pytest and so on, over an on-cloud online Selenium Grid consisting more than 3000+ browsers and browser versions.
A comprehensive end to end Testing tutorial that covers what E2E Testing is, its importance, benefits, and how to perform it with real-time examples.
Learnings From Previously Executed Projects
Every software project has different phases namely — project planning, requirement gathering, implementation, testing, product deployment, etc. There is learning involved at every phase and these learnings can be used for better planning & implementation of the next project. Even if there is a difference in the nature of the projects, you can still make use of some best practices from the past projects so that you can accelerate every phase of the current project. Also, to avoid repeating the mistakes that were committed in other projects.
You should make a note about the test automation techniques being used for accelerating the process as a best practice to CI/CD. You should involve your team members i.e. developers, testers to look into a possibility where best practices for test automation can be reused so that you & your team do not have to reinvent the wheel. For example, there could be certain test frameworks that were used by them which helped me achieve better test results in a shorter amount of time. Document these learnings as it may help in devising a sound test management strategy.
Why Documentation Is An Absolute Necessity?
There are cases where project requirements change/evolve along with the course of project execution. On similar lines, the test automation strategy should be devised in such a manner where you plan keeping the short-term goals & long-term goals in account.
Though there may be changes to the test plan or test strategy over a period of time, the team should at least standardize certain processes. Standardizing could mean short-listing a particular test framework, identifying the cloud infrastructure that suits your budget & requirements, building a competent automation team that work on test scripts (in Python/C#/Java/other programming languages). You should also have a backup plan (Plan-B) in case there are any changes in the schedule or overall business within the organization.
It is important to have these things documented so that it can be referred at any point in time. Along with the pointers that have been mentioned above, the document should also include a section which highlights the ‘Risks & Assumptions’ associated in the execution of the test strategy. For example, the test plan/test strategy would have been created with 4 automation engineers as the resource count; but you may need to scale down the team due to some unforeseen circumstances. Hence, create a fool-proof test strategy by looking at all different parameters associated with the project. The document highlighting the test strategy should be a free-flowing document i.e. updated with important timelines & milestones and should be version-controlled so that revisit the document to track the progress.
Central Repository For Code Development & Maintenance
In any project, there would be number of developers in the team who will push their code which could be a feature implementation or a bug-fix. On the same lines, developers would be executing a pull request to get the latest code changes from the server. Maintaining the source code on central repository is of utmost importance & is considered one of the best practices for CI/CD pipeline. So that the developers can keep their changes up-to-date with the latest source code available on the production server.
A revision/version control system is also important to track changes, identify differences, and maintaining an environment that eases the task to keep track of the application builds.
Rollback With Version Control
There are scenarios where the test team could come across some issues which were not observed in the previous software release; a probable reason could be a side-effect of a fix that is pushed in the release software which is under test. Performing RCA(Root Cause Analysis) could sometimes be time consuming, and you can’t afford to lose a valuable feature for a long time in your production environment.
In such cases, the developer who pushed that fix should be able to roll-back his changes so that the release is not stalled and he also gets some more time to re-look at his implementation. Without version control system, such a seamless roll-back is not possible. The roll-back is not limited to source code; it can be extended to documents, presentations, flow diagrams, etc.
Staging Environment Loosely Mimicking The Production Environment
Irrespective of the version-control tool being used for tracking code changes, usage of development & testing environments is common in all of them. The development team can make use of different ‘development branches’ for a different set of customers, but the code would still be pushed to the Production server/Staging Server. For improved effectiveness, the staging environment should ideally mirror the production environment. A common mistake I have observed is related to live traffic. Often Staging environment misses out on live traffic that your production environment has to go through for handling the load. Here are 13 reasons why Staging Environment is failing for your organization.
This approach makes it easy for deploying working code from staging server to production server. In fact, developers should also have the flexibility where they can create & setup new environments on the click of a button. There are tools like Jenkins which can help to achieve the same. Usage of tools common to the development team & DevOps team would be vital in streamlining the best practices of CI/CD process.
A comprehensive end to end Testing tutorial that covers what E2E Testing is, its importance, benefits, and how to perform it with real-time examples.
Involving Relevant Stakeholders In Test Code Development
It becomes extremely critical to involve the right stakeholders in the test automation planning, development & execution. Since developers in the team can add more value to the test code that is implemented by the test engineers, it would be ideal to involve them in the test case implementation. Developers have a better understanding of the architecture, coding techniques, best practices in software development. Experienced QA engineers may have a good understanding of test infrastructure, test frameworks, etc. Hence, developers collaborating with the QA engineers can reduce the effort & time involved in test case development to speed up test automation for CI/CD pipeline.
In crunch situations, the development team can work on the automation code without involving the test team so that there is no delay in the implementation of automation tests. However, it is advisable to involve the relevant stakeholders, especially QA engineers & developers for the test case implementation.
Incorporate Feedback For A Robust Automated Testing Strategy In CI/CD Pipeline
Test strategy document charts a fair plan on how the test automation & other test-related activities would be planned & executed. Along with updating the test strategy (as and when it is required), the feedback loop should also be used for timely updation of code for test automation in CI/CD pipeline.
The feedback can be used to understand the user’s pain-points & to get finer details about the overall usage of the infrastructure that was provided for automation testing. For example, test automation engineers & DevOps team might have some observations about the infrastructure over a brief period of testing. The observations could be related to methodologies used for capturing logs/porting automation test code implemented in Python & Selenium to the Cloud infrastructure used for testing/performance related to the execution of tests, etc.
All this feedback should be captured in a document and relevant feedback should be incorporated as a best practice to CI/CD automation pipeline, so that the test code is in-sync with the current requirements.
Frequent Code Commits For Micro-Agility In Test Automation For CI/CD Pipeline
Developers begin the development based on the requirements mentioned in the specifications. Once the implementation is complete, the developer performs a unit test on the code and fixes the issues encountered during this round of testing. He should be able to fix issues that are independent of integration since the tests are stand-alone tests. Once the local testing is complete, the developer will push the code to a code repository. In most of the development environments, the code is normally pushed to a ‘development branch’ and once that is approved & tested, it can be pushed to the ‘production branch’.
Hence, developers should be encouraged & motivated to push the code changes more frequently so that it becomes easy for keeping a track of the changes. This important best practice for CI/CD should be followed diligently by the developers in the team so that you are able to achieve micro-agility for activities related to development & test automation in CI/CD pipeline.
Seamless Coordination! Remember, it is CI + CD!
CI & CD are two separate processes but they cannot be termed as inseparable. Any delay or lapse in executing the CI process could hamper the output of the CD process. Though automation can be used in improving the efficiency of the CI/CD process, there are many aspects where automation may not be effective.
Communication & Collaboration are key pillars if we refer to best practices for CI/CD process since there are multiple teams — product planning, development team, verification team, DevOps team, etc. that have to work in harmony for the success of the project. Hence, benchmarking your CI/CD process with the leaders who are best in those areas is important. And is considered as a critical best practices for CI/CD pipeline to provide keen inputs and suggestions for optimization.
Also, remember that in any project, there is no activity that can happen in silo since there would always be someone who is dependent on the results of your activity. For this to happen, there needs to be proper communication between stakeholders of the project. For example, your QA engineers might be working on some test case implementation which not be covering all the requirements. If such tests are executed, you may not achieve good test coverage since important scenarios are missed out in the test case/test suite implementation.
Keep It Simple & Organized By Executing The Smaller Test Cases First!
I have witnessed in many projects that when it comes to testing automation for CI/CD pipeline, we often forget to prioritize our test case in a systematic approach. Here is what I mean! As a best practice of CI/CD, it is always recommended to execute your smaller and simple test cases before your long and complex ones.
This way you could be sure of individual test case coverage and performance with respect to functionality testing in an isolated manner. Complex test cases where you design test cases to check the interaction between multiple modules can come later!
Transparency Within The Team & Across The Teams
Many projects have development & QA teams that are located in different regions. Apart from intense coordination that is required to make sure that every team member is in-sync, better transparency can make things even better. This is where CI can be very effective since it brings the much-needed level of transparency within the key stakeholders in the team.
With CI in place, the team gets a much better idea about the overall status of the tests and what can be done to improve the efficiency of the tests. This is how it fuels the usage of collective intelligence for the betterment of the team & project. It is imperative to have everyone working on the same page, especially if you are working remotely. Project management tools could be highly effective as a best practice for CI/CD process. Using project management tools everyone would vary of the activities performed by other team members, also the deadline by which a task has to be completed. Here are top 19 collaboration tools for your software testing team.
Choosing The Right Tool for CI/CD Process
All the above-mentioned tips and actionable insights would help you pace up your test automation using best practices for CI/CD pipeline. Although, in the end, pacing CI/CD pipeline is highly tool dependent. There is a number of tools available for CI/CD, but you should choose the right tool based on your budget, requirements, and experience. Some of the commonly used CI/CD tools are Jenkins, Travis CI, Gitlab, TeamCity, Codeship, Circle CI, etc.
Before zeroing on any particular tool, we highly recommend that you look at the pros & cons of the tool as any change in the CI tool during the course of development could hamper your deliverables & deadlines.
CI/CD Tools To Integrate With Online Selenium Grid at LambdaTest
Talking about CI/CD tools, here is a bonus tip, LambdaTest offers a Selenium grid which is compatible with every CI/CD tool that you may be using, so you could fasten your cross browser testing effort.
In this Appium tutorial, learn about Appium Automation and its benefits for mobile automation testing. Take a look at how Appium works and see how to perform Appium testing of your mobile applications.
Conclusion
Test automation is an integral part of the overall test strategy, but it has to be planned & executed in a careful manner. There are many aspects of test automation (particularly related to infrastructure) where you can smartly use resources in hand without investing in in-house testing infrastructure. Cloud-based testing could simplify your overall test automation strategy due to the reliability & scalability aspects. These enhancements not only motivate the team members, but also let you get the best out of the best practices for CI/CD pipeline.