Study: Open Source Delivers Superior Quality... Up To A Point

http://readwrite.com/2013/05/07/study-reveals-open-source-software-quality-is-higher-than-proprietary-code-up-to-a-point

For years open source and proprietary software camps have fought over which model produces better software. According to Coverity's annual Scan report, released today, both sides are right. And wrong. Depending on how big the code base is.

Coverity's Scan report has long served as the state of the union for open-source software quality, though Coverity analyzes proprietary software, too. In Coverity's 2012 report, which analyzed over 450,000,000 lines of code, both open-source and proprietary software saw an increase in quality, as measured by average defect density (errors found per 1,000 lines of code tested). According to Coverity, this can be attributed in part to an overall increase in organizations that have implemented formal development testing processes for their software code. 

The most interesting part of the report, however, is its analysis of the impact of project size on code quality.  

Both open source and proprietary software had roughly equivalent average defect density rates: .69 for open source and .68 for proprietary software. Open source projects had the highest quality when there were between 500,000 – 1,000,000 lines of code: 70% fewer defects, yielding a .44 average defect density. Proprietary software?  It had the best quality (or, lowest defect density) in projects over one million lines of code, registering a .33 average defect density in larger projects.

For smaller code bases, then, open source shows dramatically better quality. In larger code bases, open source has more defects, but isn't far off from proprietary software: .75 vs. .66.

While there’s no single factor that can explain this phenomenon, it’s likely due at least in part to the fact that open-source projects are often purpose-specific, and maintained by a core group of committed developers. As the projects grow in size and scope, and more developers come on board, there’s a greater hesitancy to make changes to the core kernel for fear of a ripple effect that could adversely impact the larger project.

Conversely, proprietary software projects usually need to get to a certain point of critical mass – somewhere around one million lines of code, if the results of Coverity’s report are to be trusted – before an organization implements formal development testing processes to ensure quality software code. 

Here’s an infographic that encapsulates the main findings of this year’s Scan report: