Proposal for an Open Source Shared Runtime Layer
| Abstract |
This paper argues that the open source community is currently trying to support dozens of separate, incompatible development platforms, centered around projects such as Perl, Python, Mozilla, OpenOffice, GNOME, and KDE. It describes the consequences of this fragmentation, and proposes an approach to resolving it. Note that this article is dated December 7, 2000 and is somewhat out-of-date, for example doesn't consider the existence of Mono. |
The Problem
(Note that this article is dated December 7, 2000 and is somewhat out-of-date, for example doesn't consider the existence of Mono.)
Fragmentation and duplication of effort are constant problems in the open source world. Recently, the GNOME and KDE projects have been favorite examples of this problem; developers and users alike are confused by the existence of "k" and "g" variants of every application and API. There is a GNOME database access library, a KDE database access library, a GNOME XML parser, a KDE XML parser, a GNOME GUI toolkit, a KDE GUI toolkit.
There's a reason why GNOME and KDE insist on reinventing the wheel. The problem, simply stated, is that modern programmers expect to work at a high level of abstraction. They expect to have a layer that provides portable implementations of features such as threads and basic data structures. They expect an object system. They expect a signal-slot mechanism (events and event handlers). They expect support for integrated development environments. A modern platform provides these kinds of features.
At the moment, all major applications reinvent these features. Mozilla has NSPR and XPCOM. OpenOffice has the UNO component system. GNOME has GLib, GObject, and Bonobo. Qt has QObject and the Q* data structures. High-level languages such as Java, Perl, and Python naturally have object systems, a way to import modules, and are portable across platforms. Each of these projects in effect invents its own modern development platform.
With this in mind, it becomes clear that GNOME and KDE are simply the largest and most end-user-visible examples of open source platform fragmentation to date. They are hardly the first examples, and won't be the last. Every major open source project ends up inventing its own platform, with features such as threads, an object system, portability abstractions, and event loops.
We can't blame component developers for this fragmentation. If you want to write a component, say a database access library, you have two choices:
Present a low-level interface in C or C++, and require each major project to write a component-specific glue layer to export a high-level interface.
Select one of the development platforms and write for that platform. This is by far the most common choice at present.
When writing an application, you have to choose one of the available development platforms, and you're limited to the components written for that platform. For evidence, just have a look at the IDEs currently in development - there's one for Python, one for Mozilla, one for KDE, one for GNOME. An IDE should present all the useful components and libraries available on a platform. The fragmentation of IDEs shows the very real fragmentation of open source into multiple platforms.
The proliferation of platforms creates confusion among new developers who would like to join the open source world. They aren't sure which platform to use (GNOME, KDE, Perl, Python, Zope, Mozilla, Java...); choosing any platform seems to give up important components or features available on some other platform. There's a constant fear of choosing a dead/dying platform or a platform that's missing some crucial feature.
Compare this situation to Microsoft Windows. Despite the fact that Microsoft has dozens of operating system flavors, several versions of their GUI toolkit, and any number of revisions of each SDK, they maintain a unified platform. Historically they've done this with COM; in the future they are doing it with a more ambitious shared language runtime that's part of the ".NET" strategy. Visual Studio supports half a dozen programming languages and a vast array of libraries and components in a single IDE. All these languages and components fit in to the same glue layer, even though many of them are provided by third parties.
Because every part of the Windows operating system can talk to every other part using COM or .NET, any code Microsoft writes acts as an enhancement to their entire platform. If they write an XML parser, that parser is usable from any of the programming languages they support. If they write a spell checker for Word, or a computation engine for Excel, those components are immediately available for anyone to use in their own applications. Many Visual Basic applications are little more than simple aggregations of existing components.
The effect of this is that even though the open source community produces great quantities of high-quality components, and even though open source by its very nature makes writing reusable software easy and fast, many developers find the Microsoft platform a more comfortable programming environment.
Because Microsoft feels it has the client-side well in hand, with .NET it's making a concerted effort to go after server-side developers. Their goal will be to replace platforms such as Java and Perl on the server side. Their strategy is to offer a compellingly good development platform, taking advantage of the high level of integration made possible by COM and the .NET runtime. Their weakness is our strength: open source technology has a much better track record when it comes to widely adopted infrastructure/glue technology. TCP/IP, email, DNS, all the key Internet technologies, were built as open source software.
For open source to win this battle, we need to solve three key problems. These are:
Lack of interoperability. There is no real way to connect components and applications written for different free software platforms. For GNOME and KDE components, ad hoc X Window System hacks are the only shared communication layer. For nongraphical components, no shared communication layer exists. A between-process communication layer such as XMLRPC, CORBA, or SOAP begins to solve this problem; a within-process component system solves it fully, allowing interoperability on a sub-application scale.
Duplication of effort. It should be possible to write a component that's easily usable from any environment, or an IDE that supports development for any environment, without writing special-case low-level bridge code to each environment.
Lack of uniformity. For every environment, developers have to learn new ways of writing components, new ways of deploying components, and new APIs to do basic things such as parse an XML file or access the network. Shared foundation technology would allow developers to more easily move between environments, for example to use the most task-appropriate programming language, or to write a client-side application after learning the server-side APIs. (This bullet point could also be termed "lack of orthogonality." Choosing a programming language should not affect my choice of XML parser, and vice versa. It should be possible to make each choice on its own merits, so that changing one choice doesn't force developers to start from scratch.)
Note that we explicitly do not need to go around killing off diversity. As mentioned above, Microsoft constantly adds SDKs, libraries, programming languages, and operating system versions. Not to mention the availability of third-party solutions. The problem is not diversity and choice. "Fragmentation" doesn't mean lots of choices; it means artificial choices that conflict with one another in arbitrary and inconvenient ways. The problem is the lack of interoperability, lack of code reuse, and lack of uniformity/orthogonality.
The Internet is an interesting analogy. It works as a single system, composed from a rich range of applications, geographic locations, human languages, and computer platforms. The enabling technology was simply an interoperability protocol, TCP/IP.
Open Source Can Fix It
It's a shame that proprietary software is ahead of the open source world in the area of code reuse and component interoperability. Historically, open source developers have been good at creating widely-adopted solutions that promote interoperability and code reuse, while proprietary vendors have found it difficult to solve interoperability problems because of their attachment to their own proprietary technology. Indeed, we might expect to see .NET founder for this very reason, where an open technology could succeed. We simply need to provide the open technology.
The free software world already has quite a few attempts to address the problems of interoperability, code reuse, and uniformity outlined in the Section called The Problem. Unfortunately, none of them look likely to succeed without changes.
The oldest approach is to try to build bridges between different platforms. Some platforms are even designed to facilitate this. For example, one of the primary design goals of the GObject system in GTK+ is to facilitate integration of objects into the native object system of other frameworks, specifically other programming languages. "Language bindings" can be written which allow the use of GTK+ from languages such as Perl and Python. However, even with explicit support from GObject, language bindings require quite a bit of tedious work, and are hard to keep synchronized with the underlying GTK+ library.
This same approach (building special-case, ad hoc bridges) has been used in other cases. For example, the OpenOffice team plans to develop a bridge between their UNO technology and GNOME's Bonobo technology. (Making this useful raises visions of chains of bridges. To use a UNO component from Python, we might bridge from UNO to Bonobo and Bonobo to GObject and GObject to Python.)
Figure 2 shows the "ad hoc bridge" approach. This approach has a number of disadvantages. For one thing, many possible bridges are missing. Also, most of the bridges that do exist are difficult to use and poorly maintained. An application that uses many bridge layers rapidly becomes bloated and has rampant problems such as memory leaks resulting from incompatible strategies for threading and garbage collection.
Figure 2. Current rat's nest of ad hoc bridges between some open source platforms (bold lines exist now, dotted lines are in progress.)
All theory aside, we're surrounded by empirical evidence that ad hoc bridges don't work. Every project is reimplementing their own components from scratch, because they consider it simpler than using an existing component across a bridge. If ad hoc bridges worked, then people would use them constantly. They don't work well enough in most cases, and people don't use them much as a result.
The ideal solution would be a hub-and-spokes approach, shown in Figure 3. The shared runtime and in-process component system mentioned earlier would be the hub of the wheel. In this scheme, applications written for any platform can access components developed using any other platform, as soon as a single bridge from each platform to the shared runtime layer has been implemented. This single bridge is generic code that automatically works for any component.
Figure 3. A shared runtime layer allows any platform to use any other platform via a single, well-maintained, general-purpose bridge (contrast with Figure 2)
Importantly, the hub of the wheel in Figure 3 serves two purposes. First, it connects the many existing platforms. Second, it's a framework in its own right, keeping future application developers from writing their own platform. Newly-written code can use the shared runtime layer directly, avoiding the overhead of a bridge.
Several technologies currently in existence aspire to this hub position. None of these technologies has all the features that the shared glue layer should have; and all of these technologies are "biased" toward one platform or another, presenting political and technical barriers to widespread adoption. the Section called A Practical Solution enumerates the requirements for an open source shared runtime layer; our companion paper goes into more technical detail and analyzes some existing codebases.
This technology is missing simply because no one has done all the necessary work to make it happen:
Many people have not yet realized the importance of a shared runtime layer.
Those that have seen the importance of a shared runtime layer have not designed solutions that can be widely adopted. Instead, they have tended to evolve a solution tied to some particular environment, leaving unaddressed the issue of interoperability, code reuse, and uniformity across multiple open source environments.
Those that have made plans for a shared layer have not followed through with the necessary technical and political work to get the layer implemented and widely adopted.
Open source developers tend to be focused on evolving the particular development environment they're working on, rather than stepping back to consider the state of free software as a whole. Small, strategic investment in a "big picture" vision could make a huge difference.
The missing element is a shared glue layer comparable to COM or the .NET runtime. This glue layer would include a shared runtime (with features such as a portable threads abstraction) and an in-process component system. Figure 4 shows how the free software community could dramatically reduce duplication of effort with this kind of technology.
Figure 4. Shared runtime layer would unify the open source platform (contrast with Figure 1 in the Section called The Problem)
Figure 4 probably understates the benefits; not only could duplicate components be eliminated, duplication of tools such as IDEs could be eliminated. This importance of this glue layer is comparable to TCP/IP; TCP was the shared channel of communication that allowed the Internet to get off the ground. This layer would similarly bring open source software projects together, resulting in dramatically increased efficiency for the open source world considered as a whole.
Consider the potential value:
A shared glue layer between the many open source platforms promises to massively increase open source developer productivity by reducing duplication of effort. GNOME and KDE can focus on making a nice user desktop, instead of reimplementing database access components and the like.
A component developer would finally have a way to write a component for use by all developers, instead of having to choose between Perl, Python, GNOME, etc. This means we can begin to create a single core platform, gradually reducing the fragmentation problem.
Multi-language applications could finally become a reality; the same components would be available from any language, so programmers could always use the best tool for the job.
We could unify client-side and server-side development. The same database or XML component, with the same API, would be present on both client and server. This remains true even if different implementation languages are selected for client and server. At the moment, clients use one of the platforms with a GUI (GNOME or KDE) but these platforms have poor server-side technology, forcing developers to learn two technologies to write client-server applications.
We should begin to replace .NET with an open solution immediately, before it catches on. With .NET, Microsoft threatens to monopolize an important piece of infrastructure. If all applications require .NET to run, then commoditizing the operating system with Linux has little to no value; Microsoft will have moved on to a new and different infrastructure monopoly.
We can offer compelling advantages compared to Microsoft technology; our solution would be cross-platform, would support more programming languages, and would most likely have far more components available for it. The open source community has an enormous capacity for writing small, useful components (as evidenced by their tendency to write them for all these different platforms, from Perl to GNOME). If we can harness that capacity into a single platform instead of spreading it among dozens, we could do much better than the proprietary world.
There is nothing inherent in the nature of open source that's preventing the development and use of this kind of shared glue layer. After all, most of the Internet technologies we use today were developed as open source technologies. TCP/IP, HTTP, SMTP, DNS, and so on are all open standards with wildly popular open source implementations.
Microsoft's marketing message describes Windows as safe, well-integrated, easy-to-use, interoperable technology. Their message on open source is that we have a fragmented set of technologies with no unified vision. If you look at any single open source platform, such as Perl or GNOME, we have integration and vision, but the range and quality of available components trails Microsoft. If you look at all open source technology in the aggregate, we have a huge range of quality components; but we lack the integration and vision. Only a subset of components are available to any given application. This is the issue we have to resolve.
High-level glue technologies such as .NET and SOAP are replacing TCP/IP and the C library as the minimal protocol to do anything interesting with computers. We should not leave this space to the Microsoft monopoly.
A shared in-process component system and runtime is an enabling technology; it allows entire open source projects to work together, much as the Internet allowed individual open source developers to work together. This would make us a unified force, tens of thousands of developers contributing to a single development platform. At that point we'll be prepared to beat anything proprietary technology has to offer.
Judging by the success of the Internet, the open source world is fully capable of solving this problem. We just have to get serious about doing so.
A Practical Solution
Clearly, then, the open source world should have a shared runtime layer to sit at the hub depicted in Figure 3 in the Section called Open Source Can Fix It. Seeing that goal is the first step. The obvious next question: what practical steps can we take to get there?
As a starting point, here are the general properties that a viable, widely-adopted solution would have:
Completely unrestrictive X-style licensing, to ensure everyone is willing to use it.
Designed from the start to be cross-platform, running on Linux, UNIX, and Windows at minimum; again to ensure that everyone is willing to use it.
Developed cooperatively by a small team made up of engineers from several different projects and companies.
Act as both a foundation for new code and a bridge between old code. As shown in Figure 3 in the Section called Open Source Can Fix It, the hub of the wheel interoperates with existing technologies, so that there's a fairly simple upgrade path to the new software. For example, users of the new layer would be able to access XPCOM or Bonobo objects. Moving in the other direction, the new layer would be able to present its components as XPCOM or Bonobo objects.
Simple, small, lightweight, easy-to-learn; it should be as painless as possible to adopt.
Technically compelling features, to make people want to use it.
A shared runtime layer could vary widely in scope. The simplest possible layer would be something like NSPR/XPCOM; an in-process component system with a portable thread library. The most complex possible layer would be something like Microsoft .NET, with a virtual machine, garbage collector, and so on. The optimum solution is likely to be somewhere in between. The final decision on what to implement would have to be made by the implementation team.
An initial guess at what a shared runtime would implement:
A cross-platform threads abstraction
A basic object system (a way to invoke methods on objects)
Objects should support properties/attributes, signals/events, and exceptions
Introspection (dynamic query of interfaces, events, properties, etc. at runtime) - among other things, this is very important for implementing effective bridges to existing platforms
A main loop for optional use by event-driven applications
A means of activating objects (defined way to deploy components and load them into an application)
Memory management for objects, either reference counting or garbage collection; more sophisticated lifecycle management for remote objects (e.g. leases)
The design should plan for some kind of remote invocation, such as IIOP or SOAP, though the initial development effort should focus on in-process components
An argument for implementing a system that goes beyond a simple component layout like COM or XPCOM is that such an advanced system would attract users better than a copy of already-available technology. No existing free software platform has implemented a shared runtime. A shared runtime implementation would offer concrete benefits over existing component systems and therefore encourage migration.
The project is technically feasible, because the problems are well-understood. Component systems, portable thread layers, main loops, remote object invocation, garbage collection, etc. have all been implemented dozens of times. Individual platforms (from Windows to Mozilla to Delphi to GNOME) already implement solutions in this space. There are common practices used in all these designs. So a shared runtime glue layer does not require a new ground-breaking design. Most of the work would be in specifying the details and in implementation.
The implementation could reuse a lot of existing code. For example, the GLib main loop interface is a superset of pretty much any other main loop interface; it could form the basis for the main loop implementation. There are half a dozen portable thread libraries available; any of those could be the basis for our thread implementation. And so on.
A small team on the order of 6 engineers would be enough to start solving the problem. These engineers must be highly skilled and well-respected, and should come from a mix of backgrounds. The team might include developers from projects such as GTK+, Qt, XPCOM/Mozilla, and Python; and developers with expertise in component systems, compilers, and distributed programming.
The diversity of the team members is important for two reasons; first, it makes widespread adoption politically feasible because the technology won't be "biased" toward one platform; and second, it makes widespread adoption technically feasible, because the team members will be aware of the issues that arise when working with the platforms they know well. (On a more practical level, a diverse team also spreads the financial load between sponsoring organizations.)
This team would work full-time on the project for maybe 6-12 months. Their goal would be to design the in-process glue layer, and implement it fully enough to demonstrate it working with perhaps one preexisting component system, one GUI toolkit object system, and one interpreted language.
Once we have an implementation team assembled, they'll have to figure out what scope the project can have without causing important platforms to reject it out of hand. It seems safe to say that at least a simple in-process component system, threading abstraction, and optional main loop could be widely accepted.
Summary of Importance
Attempting to solve this problem is hardly without risk. But weighing the relatively small investment in a programming team proposed in this paper against the potential benefits, the risk seems worth taking. To review the potential benefits:
Reduced duplication of effort; more interchange of useful code between major parts of the open source community.
A single core platform available to all open source developers.
The ability to write applications in a mix of languages, a technique proven useful by countless large apps from Emacs to StarOffice.
Unified client-side and server-side development; the same components available for each.
An open alternative to .NET technology.
A more powerful solution than .NET: open source, cross-platform, more components available.
These benefits do not rely on instant universal adoption. Every time a new project adopts the technology, the pool of shared components is increased and the sphere of interoperability is enlarged. If we can start with only two or three major platforms as early adopters (say for example a GUI toolkit, a programming language, and Mozilla) we could immediately see concrete benefits and build substantial momentum.
Of all the software the open source world could develop at this point in time, a shared runtime layer has the best cost-benefit ratio. The open source world has an enormous capacity for writing small, useful, standalone components. These components are currently grouped into dozens of separate platforms. By providing a simple hub technology, we can pour them all into the same pool. We can refute claims of platform fragmentation while simultaneously creating the most powerful open source platform yet.