Linux has been a phenomenon in embedded device development. It is now the predomi­nant platform for devices, and for good reason — it offers performance, features, and a thriving ecosystem to support advanced embedded devices. An obvious part of the appeal is the perception that Linux is free. It certainly is free to download and use, and doesn’t have any royalty-bearing fees per se. But there are costs — plus time and risk — associated with Linux development. There is a solid business case for considering a commercially sup­ported Linux during the build-versus-buy decision process for any embedded product.

When considering Linux for a device, it’s important to take the long-term view. Embedded devices are often in production and in service for many years, sometimes decades. In the long run, commercial offerings that provide a proven embedded Linux with support and maintenance are cheaper than maintaining a roll-your-own (RYO) Linux solution in-house.

Your own distribution (whether you like it or not). Newcomers to embedded Linux often don’t understand that every RYO Linux platform done in-house is a unique and custom Linux distribution. Embedded Linux requires compiling and linking on a host system for download to a target host. The creation of all of the right puzzle pieces to make the OS run on your embedded target is effectively your own in-house distribution of Linux. For this reason, embedded Linux is different from desktop and server Linux, where one typically uses a well-known distribution already packaged and ready to go. Companies adopting Linux must understand that they are taking on the maintenance and support of a custom Linux distri­bution, which requires expertise and staff.

Yocto Project to the rescue. The very nature of embedded Linux created the need for tools to build and assemble all of the components of a platform. In fact, Wind River® did so for its Linux distribution as well. But with no stan­dardization, each vendor and user made its platform its own way. The Yocto Project (www.yoctoproject.org) was created to standard­ize the platform for embedded Linux, thereby greatly increasing interoperability and creating an ecosystem of hardware and soft­ware vendors around a common set of standards and tools. The ecosystem also provides support and expertise toward emerging standards such as those being created today for the Internet of Things. The Yocto Project is freely available and provides a stan­dardized embedded Linux build system for production use.
Wind River is a founding member and has been a very active lead­ing contributor to the Yocto Project, and continues to be a strong supporter of the initiative. Encouraging and building on standards makes embedded Linux stronger in the long run. Wind River Linux is based on the Yocto Project and has the fastest release cadence in the market.
Although the Yocto Project solves one of the key issues with RYO embedded Linux, it doesn’t diminish the value of a commercially supported and maintained solution. It’s important to consider the real costs of “free” Linux.
 
Table 1: Code base size and estimated development effort for Linux and associated projects.
Component Code Base
(Lines of Code)		 Estimated Effort (Person-Years)
Yocto Project ~283,500 ~72
OpenEmbedded Core ~2,150,000 ~625
YearBitBake ~72,000 ~17
GNU Compiler Collection ~7,000,000 ~2,100
Linux kernel ~17,400,000 ~5,600
GNU C library ~1,210,000 ~340
Totals ~28,000,000 ~8,000


The true cost of “free” Linux. There are many hidden and unknown costs when adopting embedded Linux, in particular for first-time projects. Newcomers are daunted by the sheer size of the Linux code base and associ­ated compiler, libraries, and platform build systems. Table 1 illus­trates the code base and estimated development effort (figures taken from www.openhub.net).
The fact that Linux leverages over 8,000 person-years of develop­ment is part of the appeal for embedded development. However, Linux is large, complicated, and mostly unknown to your develop­ment team. It takes time to learn how to use, build, and run Linux on an embedded platform. It also takes time and effort to continue to maintain Linux — and you must provide support, patches, and security vulnerability management for Linux and your application code. As stated above, each embedded Linux is a unique distribu­tion and comes with the associated costs and risks.

It’s difficult to predict the cost of adopting Linux for development teams, but based on our experience, the investment is consider­able. The kind of investment required for a Linux-based operating system that would be suitable for a device used in telecommunica­tions, for instance, is shown in Table 2.
 
Table 2: Estimated development costs of RYO Linux over time
Activity Costs
Initial release (porting, testing, hardening,
quality assurance)		 $600K
Year 1 maintenance $250K
Year 2 maintenance and onward $500K/year
Five (5) year total costs for RYO Linux ~$3M


The screenshot in Figure 1 of our telecommunications device exam­ple is derived from the Wind River Linux TCO calculator found at [1]. Most Wind River Linux customers who used to roll their own Linux solutions see an average of 90% savings by switching. This data illustrates that providing the equivalent of commercial quality and support for RYO Linux is very expensive. Note that sup­port over time could actually become more expensive as commu­nity support wanes.

 
Figure 1: Sample calculation from the Wind River Linux TCO Calculator.


Linux may be free to download, but it is not free in the medium to long term—and embedded systems are long-term products; con­sideration of the costs in terms of product lifespans is important. Commercial Linux is a high-quality product that has gone through rigorous testing and intellectual property inspection and is main­tained and supported for the long term.

The Return on Investment from commercial embedded Linux. The greatest benefits of commercial Linux come from the medium and long term. Depending on the development team’s level of expertise, the advantage of a commercially supported product may come sooner rather than later. However, let’s assume that the initial investment in time and money for Linux is zero or “free.” Looking at a theoretical investment curve (investment in dollars per three-month period) in Figure 2, we can see that RYO Linux (roll youir own) quickly becomes more expensive and continues to grow over time.

 
Figure 2: A representative graph of relative investment in commercial Linux versus RYO Linux.


This graph is representative and does not use absolute figures. The commercial Linux investment is usually an upfront fee with an ongoing maintenance and support fee over time. RYO Linux gives the perception of no or very low costs upfront, but those costs quickly grow over time. The labor costs for RYO Linux accumulate significantly faster in the long term, even when factoring in dimin­ishing resources committed to the platform.

Another way to look at this is to consider the return on investment: For every dollar spent on commercial Linux, how much does it pay back? Figure 3 shows that the initial investment is quickly recov­ered and that overall, there is a positive and significant return on investment for using commercial Linux in the medium term.
There are more than labor costs to consider with RYO Linux; it’s also important to understand the implications of using Linux in embedded systems and the risks associated with open source licensing, IP and export compliance, security compliance, and so forth.

 
Figure 3: The value of commercial Linux versus RYO Linux (using the scenario from Figure 2, the value is the difference in cost of each approach over time; commercial Linux continues to provide value while the value of a RYO solution diminishes).


Commercial intellectual property review and disclosure 
Linux and its associated tools and libraries comprise a huge code base. Embedded device manufacturers must realize that although Linux is free to use, it is not without licensing implications. Not all open source licenses are the same, but many have redistribution clauses that place requirements on companies building devices. Unlike using Linux on a desktop or server, shipping a product with Linux is redistribution, which opens you up to more licensing requirements.

Embedding Linux is redistribution. Creating your device with a Linux run-time system as part of its software is equivalent to distribution under many of the open source licenses used in Linux, including the GNU Public License (GPL). There are on the order of 20 million lines of code for Linux and associated open source tools — a large code base with a mul­titude of licenses.

With (re)distribution comes the responsibility to make sure your company is complying with the license requirements. For exam­ple, providing free access to the source code for the open source portions of your product, including any tools that might ship with the product. Often there are clauses in the licenses about derivative works that can include kernel modules, modification of existing code in Linux, libraries or tools, statically linking to open source libraries, and other stipulations. It’s critical that embedded device manufacturers catalog the licenses of the software they are using, understand the level of risk associated with the license, and are prepared to fulfill the obligations associated with each of the licenses. Unfortunately, many companies don’t treat this aspect of Linux seriously and open themselves up to needless liability.