There is a fable going around again that speaks of software enterprises unlocking untold value and productivity from establishing an internal open source practice. Depending on the story teller, the size of the riches an enterprise will receive varies from exceptional to truly grandiose. Depending on the story teller, the scope of the open source practice is for large parts of the enterprise to the entire enterprise with push-button promotion of internally developed code to external open source communities. Depending on the story teller, the amount of code produced via internal open source is so large that it can only vaguely be quantified. Depending on the story teller, …

What History Tells Us About Internal Open Source

This repetitious story, originally heard in 2011, that was probably spoken in Silicon Valley even years before, begs for a reality check once again. A good technical mind remains open to possibilities. But it also wants to critically evaluate — to assess the true value of an approach and the essence of execution for the sake of feasibility. And then only after this evaluation to champion adoption at the level warranted. This is how enterprises keep themselves out of trouble, avoiding investments that won’t pan out. Discriminant adoption of software industry trends, discarding those that are misguided, is a source of competitive advantage.

What first triggers asking if internal open source represents the promised land or is just a mirage is that it has been at least 7 years since the story was first told and yet there is no major well-known example of an organization having achieved the envisioned riches.

The mirage metaphor is suitable. On the hunt for an oasis, the determined walker reaches the top of one sand dune after another, each time sighting an oasis of riches just off in the distance. But upon reaching that new location there is, once again, only more sand. Over the years organizations I know have tried to make it to the place where internal open source generates significant value but that day has never come.

Due diligence warrants polling several colleagues throughout the software industry to assess their experience with internal open source. Folks from my professional network at large and reputable companies, whom are all also familiar with numerous other software companies throughout Silicon Valley, were unable to provide a significant internal open source success story. The best that could be said is that internal open source is appreciated by software engineers and can motivate them to contribute more code for the purpose of career advancement.

Certain Immutable Software Development Truths

What secondarily triggers asking if internal open source represents the promised land or is just a mirage is that its vision cavalierly seems to require overcoming certain immutable software development truths:

• Without Encapsulation (information hiding) Engineers are Exposed to Greater Software Complexity and that Slows Down Development. Frederick Brooks told us that the complexity of software is an essential property and David Parnas prescribed the use of information hiding to manage down software developer exposure to that complexity. Internal open source asks developers to expose themselves to the complexity of code they don’t own and that is fundamentally at odds with information hiding. While open source may on occasion empower a software developer to contribute code to that which is not owned, it is not a viable practice to do at scale due to the time cost penalty that comes with exposure to greater complexity.
• Software Development is More Productive when Engineers are Functional Domain Subject Matter Experts. Anyone with a bit of software development experience can readily see that domain experience enables software engineers to develop code more efficiently than non-experts that have to get through a learning curve to ramp-up productivity. Furthermore, software quality and correctness tends to suffer when written by non-experts.
• Software Development is More Productive When Engineers are Technology Subject Matter Experts. A software engineer that already knows the technology stack (libraries, frameworks, APIs, container, etc.) used for implementation will more efficiently write code than an engineer that lacks technology stack experience.
• Some Types of Software are Sensitive to the Introduction of Defects. The risk and cost of introducing defects into a codebase may warrant limiting development to the owning software team. One can generalize this and say there are probably a handful of other good reasons why codebase owners may discourage open source contributions. If one accepts this as reasonable, what follows is that an internal open source effort is (strategically) limited to a subset of the total code written by a company.
• Software Contributions across Teams Requires Coordination. Here again, internal open source contributions are possible but they may not be the most cost effective. Coordination within a team will generally cost less than across teams — the communication costs are going to be higher and there will be costs due to suboptimal alignment of project timelines.

Given these truths and others not mentioned, one can reasonable guess that first preference should always be for the designated software owner to code the features it offers with a development schedule that aligns with the needs of its users. Second preference should be for the user to ask for feature development that is not in the roadmap to be added. In this case, maintaining a close partnership with the provider typically helps with getting priority. As a case in point from external open source, it is common for large commercial companies to form partnerships with open source providers and to greatly prefer having providers do feature work on their behalf, effectively growing some portion of the open source codebase as directed by commercial companies. There are two variations to a third preference whereby a non-owning team coordinates the build out of features on behalf of the codebase owner. For variation A, the non-owning team does the feature work, contributes the code back to the codebase main line, the owning team accepts the contribution and then has the responsibility for its maintenance and operations. For variation B, the non-owning team does the feature work and maintains temporary ownership of the code including the responsibility for maintenance and operations until an agreed upon later moment in time when the rightful owner takes over full responsibility for the code. Variation B covers the frequently occurring case where code innovations are developed by non-owning teams that are the first to need a feature or that tend to be technical leaders. In either variation, the non-owning team development is subject to governance that puts some guardrails in place for making the contribution. The fourth preference is serendipitous feature development by non-owning teams, an inherently uncoordinated way that features get developed that can’t be relied upon but that is nice to reap the benefits of when it happens.

Internal Open Source Not Ready for External Contribution

On the idea that internal open source can readily be contributed to an external open source community, this tends to be easier said than done. A more prudent view acknowledges that if code is developed internally, unless it is targeted for external sharing from the outset, its design and implementation are likely to require modification prior to external release. That’s generally what experience teaches us. Here are a few of the common reasons that code written for internal consumption requires further development before release to an external open source community:

• Internal code is likely designed for a more specific set of use cases than it needs to cover when released externally. Further abstraction and expansion of the design may be needed to meet those additional use cases. Some reasons why code ends-up solving a more narrow set of use cases when developed internally includes not spending the time to collect the full breadth of use cases, no budget to do additional software engineering, and project timelines that cannot accommodate additional scope even when engineering resources are available.
• Internal code is likely to make assumptions about client software that do not hold for software clients written by the general public. Particularly noteworthy are client trust assumptions and guarantees that do not hold when clients come from the public domain.
• Internal code may have deficiencies related to technical debt that are acceptable for internal but not external release.
• Internal code may include some components that would reveal intellectual property that an organization is unwilling to share.
• Internal code may include proprietary dependencies subject to licensing agreements that prevent sharing.

A Realistic Vision for Internal Open Source

Summarizing, the most prudent path forward is to concede that internal open source does not offer the riches envisioned but that a more tempered expectation of it delivering smaller scale value in certain circumstances is realistic. A broad brush estimate might suggest that 80% of features developed come from teams that own the respective functional domains, 10% of features are built by non-owning development teams in coordination with coding owning teams, and 10% of features are built serendipitously through largely uncoordinated contribution that is sometimes spurred by feature needs and other times the result of benevolent community engagement.