Jim Trentadue, Software Quality Consulting Director, Original Software
The Buy vs. Build dilemma has long afflicted CIO’s within IT organizations. Historically, this has been evident in Enterprise Software Packages and Development space, but is a growing quandary in the Software Quality and Testing arena. The question spans multiple areas of the testing domain. Some large areas to undertake are: QA and Test Management, Manual Testing, Web Service Testing, Test Data Management, and Automated Testing. We will evaluate each area to answer the most important question: At what point is free-ware not free?
QA and Test Management Solutions
Traditionally, this has been dictated by a handful of vendors, most notably Quality Center/ALM. Teams have used these systems as a repository for their manual and automated testing artifacts. Technology limitations and usability have opened the door for additional commercial products. Only a small handful of open source solutions are making a splash currently. High annual maintenance costs for traditional commercial solutions have made QA Leaders look elsewhere.
The primary challenge would be to retrieve all testing artifacts safely into an alternate solution. If that can be accomplished, this has the highest potential to become more diverse with more flexible and fluid solutions. The question from an open source perspective is do I have the adequate personnel to ensure the solutions are maintained properly as the core source of all my QA work. This needs to be the focus of dedicated, full-time associates, not associate.
Manual Test Case Solutions
There are many groups using Excel and Word for maintaining their manual tests. Manual tests still account for approximately 2/3rds of all testing done. The key advantage is that associates do not have to carry a strong technical skill set to maintain testing artifacts. Any word editor will suffice. However, the major limitation and challenge is upkeep of existing artifacts, usually for a large number of assets. Using a commercial solution to convert a manual test case into an automated test case is a strong consideration, if there is a desire to automate more.
Web Service Testing Solutions
This area has grown significantly over the last five years, primarily by Soap UI as the open source tool of choice. The assessment on whether to use an open source vs. commercial solution is to measure the percentage of native web service testing only vs. functional testing. The answer may be surprising as some organizations have this testing competency as high as traditional system testing.
Test Environment Management Solutions
Most QA professionals are not going to write database triggers or stored procedures. However, a large majority are fluent with basic SQL functions, along with joining data tables and data sources. The number of focused Data Warehouse and Business Intelligence testers in the industry also has increased. There are commercial solutions that allow a tester to not rely on a DBA for test environment data, database backups, restores, and rollbacks.
If you have the expertise in house and can be certain the knowledge will be retained, open source is not a bad avenue to go
Some solutions do not require the tester to have SQL knowledge at all and can help ensure the referential integrity of the data mappings. The DBA can focus on the reliability of the production environment. Leveraging a non-commercial solution or more often, a custom solution built in house, will most times require developer or DBA assistance. If your organization has more resources in this area with available time without competing priorities, this may suit you well.
Automated Testing Solutions
This is the main event for commercial vs. open source in the testing discipline, with the headline being Commercial vs. Selenium. Selenium is the market share leader of open source for web testing automation, with Appium leading in the open source area for automation of native mobile applications. Some organizations use a combination of each of the tools listed above.
Earlier, I posed the question–“At what cost is free-ware not free?” Without trying to upset either side, or trying to upset each side equally, I am going to state some truths. I have been around test automation for just under 20 years. Years back, there were a handful of commercial vendors in the market. The onset of many automation vendors has only come within the last ten years.
Most vendors in this space required testers to have a development skill set. From my research, at least 70 percent of testing associates said they did not have a development background. This is based from the organizations I have worked at and polling the audience each time while speaking at testing conferences. On average, there are 3.5 manual testers for every technical tester. Since most commercial tools required programming knowledge, why should that be paid for? Thus, enter Selenium. Look at what the element Selenium is the cure for and you will understand its origins.
Using Selenium as your automation solution comes with one certainty; the tester must be a developer. The Selenium community does not refute this and in fact, welcomes this. Ask yourself three questions before making the decision either way.
• How many functional testers do I have across my enterprise vs. SDET’s?
• How long will it take to build the framework?
• Is there a dedicated team to support, maintain, and grow the automation?
Those that prefer Selenium will tell you this is not a short process. A custom-built framework works while there is a team for support. If not, IT leadership will need to choose between salvaging all the code written or starting anew. There are many consultants, who know Selenium, which comes with a high price tag.
However, there is a growing trend of organizations not being able to retain associates with knowledge of Selenium, despite holding the primary knowledge of the framework developed. In most cases, these new SDET’s (Software Development Engineer in Test), typically do not have the software testing or application SME knowledge as traditional functional testers.
There are commercial solutions available that have brought the essence of test automation back to the forefront, by leveraging record and playback. This enables manual testers to become adept with automation. Skeptics are still unsure if this will work, but there are solutions where it is proven to work. Reviewing the questions above should provide a CIO the understanding of where the testing department is at and which direction they should go. The decision is also predicated on budget, a capital spend on a commercial solution with a framework, or operational spend on associates or contractors building out a framework from the start.
In addition to the creation of automated test scripts, it is imperative that IT leadership keep a close eye on how the existing tests are being maintained. If there is an open source solution, the lines of code are in the thousands typically, thus making retention of the source of knowledge crucial to continued test execution. Many commercial solutions are built on frameworks that allow for easy maintainability from associate to associate. It becomes a situation if you would rather pay for this upfront with capital spending, or incur the yearly operational cost of contractors to maintain these artifacts. No right or wrong answer, it is a financial decision–spend now or spend later. You will spend either way.
The decision to Buy vs. Build is not straightforward. If you have the expertise in house and can be certain the knowledge will be retained, open source is not a bad avenue to go. However, there is no support desk to call, no SLA to be held to. If that is your preference, then commercial might be your route.