The open source model is a close relative to the scientific development model, with code treated as research results, and published for peer review and for the benefit of mankind. Problems are very similar in both disciplines over priority and the correctness of a vision that are as vital for developers of OSS as they are for scientific researchers. I've seen equivalent behavior in the scientific community. Normally such disputes are played out in a more mature manner, although no less vicious. Here is an anonymous example:
"What "normally" happens to an open source development project when people writing code for it don't get along? How do open source authors deal with conflicts with other open source authors? Let me relate the story about my first experience maintaining open source software, as I think it's a pretty good illustration of this. I used to work on a program (for the sake of this discussion, let's call it P, for Program) that I volunteered to take over development work on around two years ago, after the original author produced version 1.0, and had no more time to work on P.
At the time I took over maintenance and development for the package, I had lots of time free at the business where I worked. My boss was understanding enough to allow me to devote a few hours a week of paid time to P, which I believe doesn't happen often for open source authors.
As time passed, our business grew, and I had less and less time to work on P. I managed to release an alpha version with some improvements, but some people weren't pleased with the pace of development. One of them took it upon himself to start work on his own version of P. For brevity, I'll call him Mr. J, for Jerk.
Now, normally I'd cheer this, since it's what open source is all about. However, it seemed what Mr. J really wanted was acclaim. I had released version 2.0 beta of P, based on the 1.0 version from the original author. Mr. J released a version he called P 1.2, also based on the original 1.0 code, with his own modifications. This generated a lot of confusion, at least in my opinion, since he had the same name for his package as mine, and similar version numbers. So, I asked him to please change the name of the package to something else to clear things up.
His response was to publicly declare that he should be named the official maintainer of P, since I was taking too long. I should note that at this point I would have happily turned over development to him, if I'd thought he could do the job well. I didn't think he could (putting it mildly). I'm going to skip over the discussion we had on the subject on the P mailing list (on my mail server) since it includes a lot of childishness on the part of Mr. J. Suffice it to say that things ended up with Mr. J calling me a few names, and vowing to take over P as part of a larger project he was working on.
Mr. J proceeded to take a copy of the names on my mailing list, and create his own list. He declared his version to be the official version of P, and ceased to take part in the original list I had set up. After all the argument on the original list, I was glad he was gone.
That is, until someone posted a question about my version of P to his list (which he had subscribed me to as well).
He took the opportunity to publicly call me a few more names, and make some comments about my lack of progress. I had enough at this point, and mailed him telling him that I didn't want him to mention my name at all, ever again. I was hoping he'd start completely ignoring me, leaving me free to work on P quietly, without his interruptions.
Boy, was I wrong. Mr. J proceeded to mail me back and tell me that he would do whatever he pleased (again, putting it mildly). He also added a text description offering to let me perform an obscene act on him to his .sig file, which he used publicly on mailing lists and whatnot.
Completely appalled at this point, I e-mailed his providers for web space and connectivity, threatening them and him with lawsuits if he didn't remove my name from his postings. This got another nasty response from Mr. J, but eventually did get him to remove my name from his sig file.
This brings me to present time. I now have such bad feelings associated with the whole affair that I don't like to think about P, much less work on it. I've stopped working publicly on it, in fact, and I only do development on in-house versions that will never see the light of day."
The Problem of the "Lowest Hanging Fruit"
"Blessed are those who have no expectations, for they will never be disappointed."
Buddha
It seems that open source projects are more successful in domains that are directly or indirectly interesting to developers themselves. The growth of the Internet is making a larger spectrum of projects available. The initial period of an OSS project - building a more or less complete prototype by a single individual - tends to be more developer-oriented, which means more complex, but no less useful software may be thought of duller and less interesting. Those who can program naturally tend to work on programs they find personally interesting or programs that looks cool (editors, themes in Gnome), as opposed to applications considered dull. Without other incentives other than the joy of hacking and "vanity fair" a lot of worthwhile projects die because the initial author lost interest and nobody pick up the tag.
That tendency is probably positive. It does not make sense to develop OSS software outside a commercial context without fun. Programming without joy is a kind of slavery. OSS - viewed as a kind of research project - should concentrate on more personally interesting aspects of software and leave mundane tools development to commercial entities.
Not all projects are created equal. The most important feedback in a project directly deals with those parts of the code important to the developers, dealing with operating system, GUI and development tools. As for non-development-related software, the incentives can be much less and the quality and amount of feedback could be insufficient to keep a given project afloat.
Fragmentation and NIH syndrome
Let's start by quoting from the recent interview with Dennis M. Ritchie:
"LF: I cannot avoid making a comparison between you and all the people that is currently working on non-profit projects for free, just because they like it - although I am sure they wouldn't refuse money for the work they do for free. Can you see yourself involved in projects like Linux, or similar, if you were not at Bell Labs? How do you see all these people from inside an innovative research lab with many years of experience on your shoulders? Since our magazine is mainly for Linux users we cannot forget to ask you a questions about Linux. First of all, what is your opinion about all the Linux momentum, and the decision of many companies to start developing software for it (Bell Labs, for example: Inferno has its own port to Linux)?
Dennis: Let me put these questions together. I think the Linux phenomenon is quite delightful, because it draws so strongly on the basis that Unix provided. Linux seems to be the among the healthiest of the direct Unix derivatives, though there are also the various BSD systems as well as the more official offerings from the workstation and mainframe manufacturers. I can't help observing, of course, the "free source" Unix-derived world seems to be suffering from exactly the same kind of fragmentation and strife that occurred and is still occurring in the commercial world."
And this observation that "...the "free source" Unix-derived world seems to be suffering from exactly the same kind of fragmentation and strife that occurred and is still occurring in the commercial world" is a symptom of the fundamental problem - "the tragedy of commons". We already have a dozen of slightly incompatible Linux distributions with Debian and Red Hat as two major players with strong supporters and several second level players (Suse, Caldera, Mandrake, Slackware, etc.). I would like to say that in a sense Linux suffers from a multiple personality disorder. That means that an application written for Red Hat Linux might not run under Caldera's version because, for example, of different versions of libraries used.
As Jordan Hubbard put it "... it is a perfectly natural human desire to form clans and proudly wear the clan colors on Robert the Bruce Day, or whatever, diversity and competition being good things which encourage innovation and inspire people to greater heights of productivity. When you add Rampaging Egos to the mix, however, things get messy very quickly as the various clans decide that they want to be *the* clan, the only ones in line when the awards for best clan colors or most unique sporran are being handed out. Before long, rival clans are firing flaming arrows at one another and launching raids into each other's camps, generally making life difficult all around for a lot of folks who would really rather just get on with the business of living."
The NIH problem is connected to this phenomenon. Everything in the software world is artificial. As in an academic community at some point of maturity problems with adding new features or some kinds of modifications or some license limitations suddenly become ideological.In turn the software community splits into "more permissive", "oppressed" and "orthodox" groups. If one of these groups is successful it's only a matter of time before it becomes "orthodox" and a new split eventually occurs.
It's obvious that Linux wants to be distinct from FreeBSD and FreeBSD needs to be different from Linux. These two camps fought for the same ecological niche, over a matter of personal prestige. Being less PC-friendly then Linux and due to legal problems with AT&T, at some point the FreeBSD movement lost momentum and later suffered also from the additional competition from OpenBSD (after the latter emerged as a viable competitor on Intel platform in the internal split NetBSD/OpenBSD). As a result Linux now looks like a king of the hill on PC platform, cannibalizing available developer resources.
But that does not mean that the problem cannot reoccur at some point. For example, if a sufficient part of kernel developers would feel that the Linus management is unsatisfactory and is harming the Red Hat future, Linux can split into "orthodox" Linux and, say, Redux. It would be interesting to see if Linux kernel development splits if, for example, Alan Cox grow dissatisfied with Linus Torvalds' management. A change of name would probably be mandatory as Linus Torvalds is the owner of the Linux trademark.
In fact, a major "religious war" is currently raging over the relative merits of KDE and GNOME. That in itself is nothing amazing. Like scientific communities, the free software movement is constantly driven by factional disputes over both ideological and technological issues. Mostly, they're meaningless to outsiders. But the KDE-GNOME conflict helps illustrate both the essential strength of the Linux community and its potential Achilles' heel. The conflict is ideological because KDE is built around a piece of software that is not protected by the GPL license; it's not fully "free" in the eyes of Linux purists.
Although KDE is today the most fully developed user-friendly Linux desktop, the main Linux distributor Red Hat devoted programming resources and money to GNOME. As Andrew Leonard wrote:
"We are sad about the fact that the GNOME project was founded," says Bernd Johannes Wuebben, a KDE developer. "We feel that what the free software community needs is unity and focus, not competing standards and the repeated hostility that members of the GNOME project and the more radical elements of the GNU [free software] movement have brought forth against KDE ... We at KDE do believe in free software but we believe that radical views ... are utopian and in the end hurt the free software community severely. In our view there is clearly a place for both: commercial as well as free software development."
Larry Augustin believes that GNOME is ultimately the better desktop on pure technological grounds. But he regrets the fact that Red Hat, the market leader, will not ship the most advanced desktop interface for Linux, thus inhibiting Linux's wider market penetration. Red Hat's Bob Young argues that Red Hat is the market leader because the "technical community trusts us," and that trust depends on Red Hat choosing the appropriate technology. He also suggested that the next release of Red Hat might include an optional version of KDE, packaged separately from the core Red Hat distribution.
Linux outsiders may watch this whole spectacle with some perplexity. Microsoft's great advantage is that it offers software developers a single standard to write their code to, and provides users with a guarantee of software compatibility. Linux, on the other hand, has at least five major distributions -- in addition to Red Hat and Caldera, there is also S.u.S.E. (from Germany), Slackware and Debian (a completely noncommercial version). Each distribution differs from all the others, has different setup procedures and requires a different approach when installing new software.
Corporate America, not to mention the individual consumer, shies away from such variety, with its potential for confusion. Yet at the same time, the diversity is a major source of Linux's strength. "It's the beauty of anarchy. It's the beauty of freedom," says Red Hat's Bob Young. "The distributions that do not do a good job keeping up ... will not survive in the long run."
"It's an advantage because you have more choices and competition," says Augustin. "The distribution vendors are fighting to make distributions better, and that competition helps a lot, it really drives them. There is very little proprietary work in a distribution. It's very easy for someone to come out with a free version of Red Hat, so the distribution vendors have to be constantly making themselves better. If they don't, people will migrate to whoever is best technically.""
Bias toward power users
In the absence of customer support and marketing organizations, feedback and thus development direction (especially requests for new features) are dominated by the most technically savvy users. This factor probably makes open source products more attractive to the top tiers of the user community. Unix traditionally is most popular among professional programmers and in academic, educational and research environments that can be classified as advanced users groups.
Critical mass problem - does winning means stagnation?
"Nothing in the world can replace persistence. Talent will not; nothing more common than the unsuccessful man with talent. Genius will not; unrewarded genius is almost a proverb. Education will not; the world is full of educated derelicts. Persistence and Determination are omnipotent."
Calvin Coolidge.
As in corporate software environment, the most viable projects negatively affect competitive OSS projects (Linux vs Hurd) by depleting developer resources (see Halloween I memo):
"As soon as there is no critical mass of developers users project became a history. Now Linux is killing BSD Unix development by absorbing most of its core ideas. This limit competition in a way very similar to PC world ...Possession of the lion's share of the market provides extremely powerful control over the market's evolution."
Large marketshare has risks, such as inhibiting innovation. As Jamie Zawinski put it:
"... Because the company stopped innovating. The company got big, and big companies just aren't creative. There exist counterexamples to this, but in general, great things are accomplished by small groups of people who are driven, who have unity of purpose. The more people involved, the slower and stupider their union is.
And there's another factor involved, which is that you can divide our industry into two kinds of people: those who want to go work for a company to make it successful, and those who want to go work for a successful company. Netscape's early success and rapid growth caused us to stop getting the former and start getting the latter."
Cult of Personality, burnout of the leader
"Never in the field of human conflict was so much owed by so many to so few."
Winston Churchill
Open source may sound democratic, but it isn't. Leaders of the best-known open source development efforts often explicitly stated that they function as dictators. If Linus Torvalds does not like your patch, no matter how technically good it might be that's it. He controls what's in the kernel. Malcolm Maclachlan wrote:
"The ultimate example is Linux itself. Creator Linus Torvalds has final say over all changes to the kernel of the popular open source clone of Unix. Because the Linux development community has grown so large, most software patches are reviewed by many different people before they reach him, Torvalds said.
If he rejects a patch, it can mean a lot of other people threw a lot of effort down the drain, he said. However, it enables him to keep Linux organized without spending all of his time on it, he added.
"My workload is lower because I don't have to see the crazy ideas," Torvalds said. "I see the end point of work done for a few months or even a year by other people.""
If speed of development is an important issue (as is the case with Linux), a single leader must exercise almost absolute power over the code of others (and to less extent over the direction of the project). This situation tend to be more and more of a problem as a project becomes successful.
A determined and dedicated leader is essential for the survival of a project in its early stages. Often the qualities of the leader and the difficulties of the early stages lead to a very high level of centralization. To a certain extent they objectively lead to a "cult of personality" in the best socialist fashion.
But the same qualities of the leader that ensured success of the project in its early stages can and often became a liability at the mature stages, when qualities other than dedication count and the project scales beyond the capacities of a single person to coordinate it. The same qualities that make projects success in their early states lead to their undoing later on. The latest situation with Linux as well as the earlier examples of Emacs and GCC are typical.
I would like to stress that if a project becomes a success, then its success creates a tremendous amount of work for the leader and eventually leads to burnout. Burnout is the price of becoming a media darling.
The pace of work on an OSS project should be determined by the developer himself and here the analogy with academic research is appropriate. I would like to reiterate that "speed kills." Pressing the developer often leads to the opposite results. Linux is not business as usual. There is a strong connection between Linux and FSF communities, there is a common culture of "GNU generation" to it. That culture is in many ways the same as Unix's culture in the early to mid-1980s. Until recently before mass hiring by Red Hat and other Linux distributors, developers worked on Linux for a variety of reasons, but mostly because for fun and a challenge (for many from Minix days). These programmers received recognition, and they have a sense of control and belonging. Take away that fun with a tight schedule and unless they are converted into regular employees or kept by discounted IPO shares many developers will walk away. So the fact they many Linux developers are now hired is to certain extent a very positive development that probably prevented a revolt of the "old guard".
Microsoft as a vital part of the OSS movement: ABM/BTM as a political agenda
If we use the ESR metaphor in a different context, Linux can be looked as a cathedral as it was build to express passion about Unix by a large group that cared about it very much. The ability to attract a geographically distributed community is characteristic of political movements and religions. Although people are physically separated, they all are working toward a common and important goal. That fuels the Linux movement.
Along with a positive agenda, every powerful social movement requires an enemy, a target that can be used as a powerful unifying force. Therefore it is not accidental that a large proportion of the Linux and open source community really hate the 'Evil Empire'. The history of religious and political movements suggests that the OSS is not simply a loose collection of Internet-connected developers and users motivated only by mutual recognition. The OSS really operates on a larger stage as a political movement and as such has its own political agenda.
Probably the most important part of OSS political agenda is its negative aspect, taking the form of a revolt against Microsoft. I call it the ABM/BTM ("anything but Microsoft/Better than Microsoft") agenda. Both statements are simplifications and both presuppose that Microsoft software is not good enough for any task in hand and view Microsoft in black and white in the best of "we against them" war mentality. That's why the OSS movement and Linux have powerful allies in OS/2 and Macintosh communities. Both OS/2 and Macintosh are based on closed proprietary models (actually not much different from the Windows model), but with powerful communities that also has ABM/BTM as their political agendas.
The ABM/BTM agenda, like any political agenda, is simple yet attractive to some. More importantly, it appeals to certain elements of corporate management with support from a wide spectrum of executives at Intel, IBM and almost all major PC vendors. Given the licensing fees paid by Compaq, Gateway and other PC vendors to Microsoft, you begin to understand why there are programmers working on Linux development in these companies with full (and, if necessary, covert) support of management. Politically, some compare Microsoft to IBM in 70's; that attitude fuels reasonable attempts to develop opposition to Microsoft.
Among Internet providers the same agenda is fueled by a fear that Microsoft can usurp the Internet by using well-funded top corporate technocracy in large manufacturing companies. Large and influential Internet providers are resentful about expensive and restrictive licenses; they perceive Microsoft as a threat ("dark force") that tries to slow them down
A cWare White Paper was one of the first documents to describe Microsoft as an important organizing force in the OSS movement:
"In the early days of the Internet, it was IBM and mainframe hegemony. Today it is Microsoft. Just as the German Reformation enfranchised specific groups previously disaffected (specifically, Luther and the German princes), the Internet empowered individuals and groups previously outside the traditionally well funded technocracy that supported and in turn was nurtured by IBM. Linux has been propelled by the same forces. Currently, a major share of commercial software resources is concentrated around Microsoft products like a large low pressure area. However, such a coalescence of power and influence disenfranchises many for whom high cost and restrictive licenses (lack of freedom really) prevent full and easy access to computing resources. So alternative paths are sought. Like the weather, alternatives may appear randomly and then dissipate. Typically, an additional sustaining force, an opposing low pressure area, is required. For Luther this pressure was provided by the German princes, for the early days of the Internet it was provided by ARPA, and for Linux, it has been provided by the Internet community itself. In the case of Linux, the Internet community desperately needed a competent OS platform. AT&T had shut out many Unix users with restrictive licenses and high fees. UC Berkeley had crippled BSD by removing all vendor proprietary code which adapted it to the underlying hardware: you could study it but not run it! Many saw a potential in Andy Tanenbaum's Minix to counterbalance an increasingly unfree Unix. But Minix was incomplete, did not have critical mass and its source distribution became too restrictive. These conditions inspired the community OS effort, initially derived from Minix, which produced Linux. Linux became readily available and increasingly capable. When it aligned with FSF licensing and could support the powerful GNU tools as well as run on a wide range of inexpensive hardware, a truly useful operating system platform was born. The Internet community finally had a way to run a fully networked Unix cheaply and reliably with no strings attached.
Linux appeared almost randomly on the scene but quickly gathered into a well organized storm because it had a powerful force to react against. It also had a sponsor.
Therefore, the Linux "Bazaar" is not simply a loose collection of vendors and other proponents, motivated only by mutual recognition. The "Bazaar" really operates on a larger stage. When forces of the larger stage organize around a dominant restrictive group, a reactionary force is generated in the remaining community. Over time, this reactive force propels various alternatives. If one or more of these alternatives can find support (the Internet community in the case of Linux), then a new "movement" is born which is sustained and even enriched by the powerful forces of the larger stage. Ironically the more dominant Microsoft becomes, the more powerful the reactive forces become, and the more fuel is fed to movements such as Linux. If an unencumbered BSD had been available earlier running on inexpensive Intel hardware, BSD might have become the seed for this storm. But the same drama would have unfolded: thesis and antithesis on a dialectic stage whose imperative will persist until Microsoft runs out of energy or dissipates its focus. Microsoft has only to look over its shoulder at the cycle of hegemony and superannuation revealed by a once almost omnipotent old technocrat: IBM."
Later this ideas were developed in the Halloween papers. In one way, IBM in the past could be considered as a "bazaar" style organization; it was not much of an advantage because this style does not encourage development at great speeds (see Jonathan Eunice's comments).
In a sense, with Linux, Torvalds "switched camps" and start playing the role of a political ABM/BTM-style leader rather than a technical leader in the course of promoting his idea of "world domination". That means that Linux started losing focus and drifted to tactically beneficial but potentially dangerous in a long run ABM/BTM ideology with the underling requirement to compete with Microsoft both on desktop and server, not to an academic community model, with its focus and mantra "chase the dream, not the competition."
This political coloring increase stress and workload (encouraging quicker burnout) on the main developers by demanding unrealistic goals and schedules. It enforces a (autocratic by definition) war camp mentality which logically should be avoided as much as possible.
