In order to get a better understanding of the industry’s current trends and test automation tools used among software testing professionals, we conducted a survey among 644 stakeholders that have direct influence on the test automation procedures in their organizations.
Survey Methodology
An online survey was sent to 1445 stakeholders of which 644 have answered.
Data collection dates: 1/10/2015 – 15/1/2016
The group deviation is: Manual QA Engineer, Test Automation Engineer, Software Developer, QA Team Leader, Software Team Leader, Management roles.
Organizations deviation: 1-10, 10-60, 60-100, 100+
What did we learn?
- We discovered that test automation around the world changed dramatically during the past years and keep shaping!
- More organizations use more than a single test automation solution, in some cases 3-4 tools in parallel.
- Open source tools became very popular, leaving 3rd party tools far behind, 88% of “1- 100” employees organizations are using open source tools.
- Test automation became practically a standard used in the majority (92%) of the organizations that were surveyed.
Management Influence
Surprisingly, in many cases the management is the #1 factor when it comes to introducing test automation to organizations with 60+ employees, and is usually highly involved in smaller organizations as well. The survey revealed that the management consider test automation as a key factor with software quality and R&D cost saving.
Test automation tools breakdown for organizations with 1-100 employees
It seems that open source tools are dominating the 1-100 employees’ organizations. As much as 88% are using open source tools with internally developed frameworks, as some of those companies are expected to grow within the next several years. If this trend will continue in the current pace, it’s quite possible that 3rd party tools will disappear completely within the next few years.
Test automation tools breakdown for organizations with 100+ employees
As shown above, the numbers are quite similar when it comes to organization with 100+ employees.
80% of the organizations use open source tools and only 20% use 3rd party tools.
Most organizations use more than one automation tool; in most cases it could reach three or even four. When multiple test automation tools are required, it may indicate on the application’s complexity, variety of supported platforms (iOS, Android, WEB, desktop, etc.) or simply that none of the existing frameworks provide the organization’s needs of an automation framework at the moment.
As expected, the number of internal tools grows when it comes to open source frameworks, and as it was clearly observed in the survey, they almost always come hand in hand.
Another interesting fact is that almost all organizations that use 3rd party tools, utilize open source technology of some kind (TestComplete – Selenium, Microsoft –Selenium, QTP – Appium-Selenium-jUnit, etc.). Since some of the tool’s capabilities are overlapping, it’s possible that the organizations are keeping the 3rd party tools for legacy and switching to open source rapidly.
Deep analysis, open source tools 1-100 employees companies
The deep analysis of open source tools in mid-size and start-up companies shade light on the testing approaches and segmentation by technology:
Web – functional testing 48%, Mobile – applications testing 18% , Unit & API testing – 34%.
Test automation benefit (all groups)
The vast majority of the survey participants stated that test automation is a beneficial tool, while in some cases admitting that they have difficult time providing a clear explanation regarding the ROI of such investment.
