In Part 1 of this series we looked at a model of product adoption and market share that had a special and valuable property:  the parameters of the model could be derived from a single survey question, e.g.:

“What is your awareness with the hand cream called Whizzo-Soft?”

A. I have never heard of it.

B. I have heard of it but I have never tried it.

C. I have tried it once.

D. I use it sometimes.

E. I use it regularly.

Given N responses to that survey questions you can derive the factors in the model by simple math:

• Customer Awareness = 1 – A/N
• Customer Motivation = (C + D + E) / (N -A)
• Customer Satisfaction = (D + E)/(N – A – B)
• Market Share = Customer Awareness * Customer Motivation * Customer Satisfaction

So let’s take a look at how this can be used in practice, taking the leading open source office productivity editor, OpenOffice, and the lesser known LibreOffice fork, as examples.

As mentioned in Part 1, the execution of the survey is critical here.  Without a proper, random survey of the market, the results will not be accurate.  In particular a survey of your current users will not work, since one of your goals is to find out what proportion of users are not familiar with your product.

So in this case I used Google’s new Consumer Survey service which uses sampling and post-stratification weighting to match the target population, which in this case was the US internet population.  In other words, the survey is weighted to reflect the population demographics, for age, sex, region of the country, urban versus rural,  income, etc.   I did this survey in a personal capacity for my own interest.  The Standard Disclaimer applies.

They survey question (and responses were):

What is your familiarity with the software application called “OpenOffice”?

• I have never heard of it
• I am aware of it but have never used it
• I have tried it once
• I use it only sometimes
• I use it on a regular basis

With 1502 responses, the results were:

And then with some simple arithmetic we have:

What does that mean?  In plain English:

• Around 1/4 of US internet users have heard of the OpenOffice software application.  That is the brand recognition.
• Of those who have heard of OpenOffice, around 2/3 of them were sufficiently motivated to try the software.
• And of those who tried OpenOffice 69% were sufficiently satisfied with the software that they continue to use it.
• Overall, 1/8 of the surveyed population uses OpenOffice sometimes or regularly.

The absolute numbers are tricky to interpret in isolation.  More interesting is to look at the numbers over time.  The same survey question, with the same methodology was also given last September.  The results and the change are in the following table, with changes having statistical significance (90% confidence level) emphasized in bold.

The Apache OpenOffice project should be gratified that their efforts have paid off, and awareness of the product is increasing, as well as market share.  This goes contrary to some loudly expressed concerns that the OpenOffice brand would languish at Apache.  Clearly this is not so.  The brand is growing, as well as the market share.

Since these factors are multiplicative, an increase in any one of them, or any combination of them, will grow the market share.  But it is probably easiest to grow the factor that is smallest today.  So looking to the future, increasing the awareness of the existence of OpenOffice would give the “biggest bang for the buck”.

For an entirely different view we can look at the same survey question and methodology, administered at the same times, only substituting the product name “LibreOffice” for “OpenOffice”.  Again, statistically significant changes are shown in bold.

The brand recognition is not growing and is stuck at 10%.   The fact that in its third year of product availability the LibreOffice brand recognition has plateaued (if not declined) should be a concern.

But the more interesting thing here is the large increase in users trying LibreOffice (Motivation)  offset by the large decrease in users who continue to use the product (Satisfaction).  What does this mean?  Only the LibreOffice folks can say for certain, but this pattern is exactly what one would expect from a product where marketing has got ahead of quality.  It is like a movie that previews well, but suffers from bad reviews and poor sales after the first weekend.  Product development aims to make products that users want.  And marketing persuades users to try the product.  But where there is a disconnect between the two, where the product is not fulfilling the needs of those to whom it is being marketed, or (the same thing really) the product is being marketed to unsuitable users, this is what you see.

I should note that LibreOffice supporters like to blame their lack of success on not having the OpenOffice brand.  Yes, having a familiar brand is a nice thing to have, but the drop in Satisfaction for those trying LibreOffice is not a brand issue, since it is entirely among those who are already familiar with the LibreOffice brand.  Satisfaction is an attribute of the product, not due to brand.

Also, we can compare the metrics across products.  When we look at the most recent data OpenOffice clearly has an enormous lead in name recognition and market share, but also a large lead in Satisfaction.   69% of those who tried OpenOffice remained users, compared to 60% for those who attempted to use LibreOffice.    Keep your users satisfied and it is hard to go wrong.

Finally, and to reiterate up what I wrote earlier in my Scarcity Fallacy post, when you consider the position of Microsoft Office in this market, both products have a relatively small presence, with ample of room to grow, at Microsoft’s expense.  This is a great area to advance the cause of open source software, in a product category that almost every user needs.  There is no shortage of opportunity here, only a shortage of imagination.  Imagine if we combined the stability/quality and brand recognition of Apache OpenOffice with the enthusiastic marketing team of LibreOffice? (Combine our 50 million downloads with their 50 million press releases)  What if we combined the disciplined development approach of OpenOffice with LibreOffice ‘s talented developers?   Imagine what we could do?

Let’s admit it.  LibreOffice has plateaued.  They have their Linux desktop users, all 3% of the market that runs Linux on the desktop.  This market share was not earned.  These are not users that they won over.  These are users they got via the control their corporate sponsors have over Linux distributions.  They flipped a bit and instantly had that market share.   But their sponsors are Linux vendors that have little motivation to reach beyond that niche market.  (They certainly have little success doing so).   The opportunity for growth is not on the Linux desktop, unless the goal is to merely be a small fish in an even smaller pond.   Of course, LibreOffice could continue, and languish indefinitely as a pet project of a handful of Linux developers.   Or they could work with us at Apache, and satisfy the Linux users, but do so very much more as well.  This would also be a cost savings for LibreOffice’s corporate sponsors, no small factor in a world of declining PC sales.  The choice now, as it always has been, is theirs.

One of the most common user questions I see on the Facebook and Twitter streams for Apache OpenOffice is “Do you have a iPad version?” or “Do you have a tablet version”?   Although there are companies that offer access to OpenOffice via a virtualized remote session, there is no native tablet version of OpenOffice.

I have received questions, behind the scenes, about the feasibility of starting such an effort at Apache.   Of course, creating a tablet version of OpenOffice, a competitive application with a first-class native touch UI,  with platform integration and optimization, is a  non-trivial effort.    My impression is that there are several companies, small and large, that would find this to be an intriguing possibility.  But the task is too large to do it alone.  But with several companies involved,  as a joint effort, in an open source project, then this becomes possible.

Imagine if we had such an open source tablet version of OpenOffice available today.   It would be an app that everyone would want.  If done right the OpenOffice app would be at the top of the charts just as the desktop OpenOffice is one of the leading open source desktop apps.  The app itself would be free, of course.   But it would be an open platform that we could all build upon.

Possible business models might include:

• Cloud services related to documents, range from storage to sharing and collaboration
• Extensions to the app, in-app purchases of additional templates, content, etc.
• Advertising-supported apps.
• From service provider perspective, avoidance of licensing fees for competing commercial office software.
• A “white label” version that can be rebranded per customer

There are good reasons, I think, for doing such work at Apache, including:

• Existing expertise in the OpenOffice product
• Proven community development culture based on The Apache Way
• Permissive, commercially-friendly Apache License, the preferred license for Android userspace
• Strong brand / name recognition

I’d like to have a discussion with those having a serious interest in making a tablet version of OpenOffice.  By serious, I mean those who might be willing to contribute developers to a larger effort, if such an effort were to materialize.  I’m happy to talk one-on-one.  And if there is sufficient serious interest from multiple parties I can broker a meeting of interested parties to discuss further options.

If any of this sounds interesting and you want to register your interest please send me an email at robert_weir@us.ibm.com.

“Essentially, all models are wrong, but some are useful” said G. E.P. Box of Box-Jenkins fame.  Today we’re going to look at a model of market share, and I hope it is a useful model.  One nice property of it is that it is very easy to estimate the parameters of this model.  A single survey question will do.

This model should be intuitive to various groups, especially financial analysts and space enthusiasts.  The former might recall DuPont analysis of Return on Equity, by which ROE is expressed as a product of business ratios such as margin, turnover and leverage.   And space enthusiasts might recall the Drake equation, which estimates the number of intelligent civilizations in the galaxy, also as a product of various factors, such as the rate of star formation, and average number of planets per star.  Both models are useful, not only for estimating the value of interest, but because the factors themselves have interesting interpretations and tell us something about the system being modeled.

In our model we look at a funnel process that describes the actions that must occur for someone to become a regular user of a product.

We start with the universe of potential customers, everyone who might have a need for your product.

Then the person must be aware of your product. They need to know it exists.

(Note there are other models that start even earlier, that the potential customer must first be aware that they have a need.   For example, those selling medicines go through great lengths to convince people that flaky toe skin is something that requires urgent attention.)

Then the person must be convinced to try your product.   Even free products require some incentive for their time and effort. Why try?  Why now?  Is it safe?   Some will be motivated to try and some will not.

Of those who try, some will have a good experience and continue to use your product, and others will have a bad experience, or insufficiently good experience, and will not continue to use your product.

Those at the end of the funnel who continue to use your product, as a fraction of the total potential market at the top of the funnel, that is your market share.

Where it gets more interesting is when you express this as a product of factors, as in:

Market Share = Customer Awareness * Customer Motivation * Customer Satisfaction

Awareness tells you, of those who are in your market, what portion has heard of your product?  This is a measure of the value of your brand, including advertising and word of mouth.

Motivation tells you, of those who have heard of your product, what portion has even tried it?  This is a measure of the timeliness and fit of your product to the market.  Pricing strategy and promotions/incentives also factor in here.

Customer satisfaction tells you, of those who have tried your product, what portion remain customers?  This is a measure of how well your product meets the promises and expectations laid out in earlier stages of the funnel.

As you can see, the first two factors relate mostly to your brand and marketing efforts, while the last factor, Customer satisfaction, relates to your product.   So by identifying these individual factors you can look at the relative power of your brand and your product.  Do you have a great product that no one knows about?  If that were the case the awareness numbers would be low and the satisfaction numbers would be high.  Are you targeting the wrong users in your marketing efforts?   That would show up as high Motivation scores and low Satisfaction scores.  That could also indicate product quality issues.  There are many different combinations, and the values of these factors, and their trend over time, can tell you much.

Now here is where it gets interesting.   You can estimate all of these factors, Customer Awareness, Customer Motivation and Customer Satisfaction, and Market Share as well,  with a survey of a single question, a question that with responses that match the structure of your funnel.

The question to ask is of the form:

“What is your awareness with the hand cream called Whizzo-Soft?”

A. I have never heard of it.

B. I have heard of it but I have never tried it.

C. I have tried it once.

D. I use it sometimes.

E. I use it regularly.

There are variations on the scale to use, sometimes only four choices, sometimes more, depending on whether you want to make distinctions among occasional versus regular customers.

Note that this requires a real, random survey of your target market.  A poll on your website where visitors self-select will obviously bias the results.   For this to work you really need to survey a random sample of your target market.

Once you have done this the math is easy.  If you have N total responses , and the number of responses for the questions are A ,B, C, D and E, then:

• Customer Awareness = 1 – A/N
• Customer Motivation = (C + D + E) / (N -A)
• Customer Satisfaction = (D + E)/(N – A – B)

In Part 2 of this post, I’ll show a worked out example, including interpretation,  using data from a  recent random survey that looked at OpenOffice and LibreOffice awareness and use.

In my last post I showed you one view of the Apache Software Foundation, the relationship of projects as revealed by the overlapping membership of their Project Management Committees.  After I did that post it struck me that I could, with a very small modifications to my script, look at the connections at the individual level instead of at the committee level.  Initially I attempted this with all Committers in the ASF   This resulted in a graph with over 3000 nodes and over 2.6 million edges.   I’m still working on making sense of that graph.  It was very dense and visualizing it as anything other than a giant blob has proven challenging.  So I scaled back the problem slightly and decided to look at the relationship between individual members of the many PMCs, a smaller graph with only 1577 nodes and 22,399 edges.

Here’s what I got:

As before I excluded the Apache Incubator, Labs and Attic, but looked at all other PMC members.  Each PMC member is a dot in this graph, with a line connecting two people who serve together on a PMC.  The layout and colors emphasizes communities of strong interconnection.  An SVG version of the graph is here.

Each PMC is a “clique”, a group that strongly interacts with itself.  But aside from a small number of exceptions, which you can see at the top of the graph, each PMC has one or more members who are also members of other PMCs.    In structural terms they are “between” the two communities and help connect them.  This could mean various things in social terms, from acting as a conduit of information, a broker, or even a gatekeeper.  The person who introduces you to new people at a party serves the same role as the person who tells the prisoner stories of the outside world.  The context is different, of course, but in either case, the structural position is one of importance.

A common way of quantifying the importance of the nodes that connect other nodes, is via a metric called “betweenness centrality“, which you can think of as a measure of how many shortest paths between other nodes pass through that node.  If the shortest path is always going through you, then you have high betweenness and you’re helping connecting the disparate parts of the organization.

Let’s draw the graph again and show each node with a size proportionate to its betweenness.  You can see more clearly now the position of the high betweenness nodes and how they bridge sub-communities.

Now of course, the structural role doesn’t necessarily equate to the actual social role.  Someone could be inactive or lurking in multiple projects and not serve as the conduit of much of anything, though on paper they appear central.   But Apache participants might take a look at this larger version of the chart, where I have labeled the nodes, and see how well it matches reality in many ways.

So, what do we have here?   This is a graph of Apache projects and how they are related, by one definition of “related” in any case.  Click on the image for a larger PNG version, or here if you would like an SVG.

Each labeled circle (node) in the graph represents one project at Apache.  Or to be specific it represents the membership of a single Project Management Committee (PMC),  the leadership committee that each Apache project has.  The size of the node is proportionate to the size of the PMC.    You can see that the largest PMCs are Apache Axis (56 members),  Httpd (55 members), Subversion (42 members), WS (41 members) and Geronimo (also 41 members).

The edges between the PMC nodes represent the ties between the PMCs as revealed by overlapping membership.  So PMCs that have a larger number of members in common have a thicker line connecting them.  I used the Sørensen–Dice coefficient to express the overlap.  This is a simple calculation that looks at the overlap in membership of two sets, scaled by the size of the individual sets.  It varies from 0 to 1,  with 0 meaning no overlap at all and 1 meaning total overlap.    An example:  Look at the bottom of the graph at the thick line connecting Apache Flume and Sqoop.  The Flume PMC has 20 members and the Sqoop PMC has 13.  They have 6 members in common, so the Dice coefficient is (2*6)/(20+13) = 0.36.   The highest weight edge in the graph is that between Apache Httpd and the Apache Portable Runtime (APR), with a coefficient of 0.52.

(Observant Apache participants will note that the chart is missing some PMCs.  I omitted Apache Labs, Incubator and Attic since they are umbrella projects representing parts of a project lifecycle.  They don’t have a specific technical orientation and the commonality in membership would not mean anything.  I left out Comdev as well, for the similar reasons.)

The color for each node was determined by a community-detection algorithm (modularity) which finds projects that have a high degree of interconnection.  This has brought out some of the larger trends within Apache, such as the grouping of cloud-related projects, big data related ones, content management,  enterprise middleware, etc.  What is interesting is that this graph was created without knowing anything at all about the technology within each project.  The graph is based on PMC membership data only.  So individual volunteers, by their choice of what projects they work, is the motive force behind these groupings.

Some other interesting facts:

• The PMCs with connections to the most other PMCs are Commons (34), WS (32), DirectMemory (31), Aries (28) and Geronimo (28).
• If you look at the most connections to other PMCs (subtly different from the above since it is possible to have more than one member in another PMCs) the top projects are: DirectMemory, Karaf, Servicemix, BVal and Geronimo.
• Betweeness centrality looks at the importance of a node with respect to helping connect other nodes.  It looks at the shortest path between all pairs of nodes, and which specific nodes are most often the ones that are passed through on these shortest paths.  If we were looking at a graph of air traffic routes, the hub cities would be the ones with the highest centrality.  If we were looking at how to communicate an idea, influence opinion, or to spread an infectious  disease (all the same thing, really), these central nodes are ones to look at.  The PMCs at Apache with the highest betweeness are: Commons, DirectMemory, WS, Httpd and Portals.

So how did I do this?

The core data I got from scraping this page, which lists all Apache committers.  I did this in Python using BeautifulSoup, building up the PMC membership in a dictionary.  Then Python’s set operations made calculating the Dice coefficient a simple task:

    intersect = SetA.intersection(SetB)

    dice = (2.0*len(intersect)/(len(SetA)+len(SetB)))

The script then wrote out the graph data, include node size and edge weight into a Gexf-format XML file, which I then processed using Gephi.  Here’s the data file I used if you want to play with the data yourself.

In Part II of this series, I’ll take a look at finer-grained data, at the social network graph of Apache Software Foundation participants at the individual level.

“When a dog bites a man, that is not news, because it happens so often. But if a man bites a dog, that is news.”   That, in the words of John B. Bogart of the New York Sun, is a classic rule of press coverage.  The ordinary is not news.  The expected is not news.  The unusual is news.  Of course this can distort our perception of reality, since we’re bombarded by stories of the atypical.

Here is a recent example, of two cities looking at migrations of their desktop office productivity software.

• In September we heard that Leipzig, Germany, population 530,000, decided to drop Microsoft Office and move to OpenOffice for its city council.  Already 3,900 of 4,200 PC workstations have migrated.
• In November we heard that Freiburg, Germany, population 230,000, decided to drop OpenOffice.org 3.2.1 and move to Microsoft Office 2010 for its city council.  2000 desktops are part of this move.

So which is “dog bites man” and which is “man bites dog”?   A look at the press coverage tells us:

• Leipzig OpenOffice coverage == 2 articles
• Freiburg OpenOffice coverage == 29 articles

The larger migration away from Microsoft Office in Leipzig was barely covered in the press.  But the Freiburg story has had enormous press uptake.   By this I take it that moving from Microsoft Office to open source alternatives like OpenOffice is normal, the expected, the non-newsworthy common occurrence.  It is “dog bites man”.  Moving in the opposite direction, from free software to proprietary is newsworthy because it is so rare.  It is “man bites dog”.

This, I think, is encouraging news.  We just need to make sure that within the open source community we continue  to tell the good news, even if the press does not think it is newsworthy.  Yes, we need to understand better why Freiburg failed, and what we can do to improve.  But we also need to put this in perspective.  This perspective includes other migrations like Leipzig, but also the perspective that in the time it took me to write this blog post we have had more downloads of OpenOffice than were lost in Freiburg.

(This post represents my personal opinion only.  The standard disclaimer applies.)

In previous posts I looked at claims made by LibreOffice, in project blog posts and press releases, related to the number of LibreOffice users and the number of active LibreOffice contributors.  I showed that in both cases the claims from LibreOffice were greatly inflated due to various flaws.  For example, they double counted users who upgraded from earlier release of LibreOffice, often several times over.  And they counted as “active contributors” those who registered for a wiki account but never actually contributed anything.  In this blog post we’ll look at the even more egregious ways which the LibreOffice project is overstating the number of developers that are active with the project.

A Quick Quiz

To prepare your frame of mind for what you are about to learn, I encourage you to first take the following quiz.

When asked to report on the population of your home town, what would you report?

A.  The number of people with primary residences in the town.

B. The number of people who have ever lived in the town, even if they no longer live there.

C. The number of people who drive through the town on their way to somewhere else.

D.  All of the above.

If you picked D, you would be an excellent candidate for the LibreOffice marketing department.

With that mental preparation out of the way, let’s continue.

The Claims

• From a recent LibreOffice announcement:  “growing developer base, which has just reached the number of 550 since the launch of the project, making LibreOffice one of the fastest growing free software projects of the decade.”
• Or a couple of weeks ago: “LibreOffice is the result of the combined activity of 540 contributors”
• Quoted on Linux.com:  “our large developer base — over 540 people at the end of September 2012 — is an incredibly efficient self-governing machine”

You can find many variations on this same claim.

All Your Developer Are Belong to Me

With a number this large, it should not be hard to find these 550 developers.  So let’s see if we can track this down.   One place to start is to look at the LibreOffice credits page.  We see there a large table of “Developers committing code since 2010-09-28″.  If we count the names in this table we get 469.  Not quite 550, but pretty close, yes?

But if you look closer at the names in the list, you begin to scratch your head.  There are names here of former Sun/Oracle developers who lost their jobs when Oracle stopped developing the project.  Some commentators, like Mark Shuttleworth, put much of the blame for Oracle divesting from OpenOffice on the “radical faction” that forked to create LibreOffice.   Now aside from costing them their jobs, LibreOffice now insults them by using their names for propaganda purposes to puff up LibreOffice’s developer claims?!

Looking further, I see the names of IBM colleagues who have never participated in the LibreOffice project.  They are active developers on Apache OpenOffice, and former OpenOffice.org developers, but here they are listed in a table of “Developers committing code”.  How curious the ways of LibreLand!

If you scroll down to the bottom of the table you get a clue in the fine print:  “We can not distinguish between commits that were imported from the OOo/AOO code base and those who went directly into the LibreOffice code base.”

Hmmm… so let me get this right.   If you take my code, you say that I committed it to the LibreOffice project.   And if I contributed to the code to OpenOffice.org or Apache OpenOffice, and you take it, you’ll list me in your LibreOffice developers table for a contribution I never made to LibreOffice and put a “joined” date next to my name for an organization I never joined.   Really?

This is an odd way of accounting for developers.    I’m pretty sure that 100% of readers of LibreOffice press releases and 100% of journalists who write articles based on LibreOffice claims would feel somewhat abused by such idiosyncratic definitions.   It is certainly not the most honest and forthright way of stating how many developers LibreOffice has.   One does not expect that OpenOffice.org developers, who were never involved with the LibreOffice project, and may not even know that their code is being used, will be included in the count.

Monotonically Increasing

Aside from counting people who are not actually involved in LibreOffice and never were, the LibreOffice claims are peculiar because of the low threshold for inclusion and the perpetuity of inclusion once added.  When you hear claims of a “developer base” you are lead to think of a body of actual developers actually working at present on the code.  That would be the normal usage.  But in LibreLand it is not done that way.  If you made a single contribution ever (or as we know now from the above, even never) then you are in the “developer base” and will be listed as a LibreOffice developer for all time.

From the perspective of gratitude and acknowledgement, giving credit is fair and generous.  Apache OpenOffice also has a long list of names on its Credits page.   But we don’t tally this retrospective list of past contributors and claim that number as an active community size.  From the perceptive of claiming a community size, this would be deceptive.  That is like calculating the population of a town by listing everyone who ever lived there.

Because of this odd practice, the LibreOffice developer count will never decrease.  It can only go up.  Even — worst case — if an asteroid hits their next hackfest — the numbers would merely be flat.  (So would the developers present)  In any case if you’ve designed a metric that can never decrease, then it should not be newsworthy for you to report that it is increasing.  This is not an accomplishment.  That is just mathematics.

How to Juice the Developer Count

An easy way to increase, for reporting purposes, the number of “developers” a project can claim is to encourage trivial churning of the code base.  For example, translating comments from German to English, removing dead code and other similar tasks can be done without even really knowing C++, or at least not knowing it well.  But it can prompt the temporary or even one-time participation of many “developers”, and in the process increase your developer count.  LibreOffice made a tremendous effort to enable a low threshold for contributions and this effort paid off, at least in developer counts.

As an example of the impact such practices can have, I took a look at the “core” git repository for LibreOffice, and all of the commits since 2010-09-28.  After identifying and collapsing multiple email addresses used by some persons, I ended up with 518 names.  Of those names, 166 , or 1/3 of them, have made only a single commit, and then were never heard of again.  So it is curious to count them as part of LibreOffice’s vaunted “developer base”.  A community is not made up of those who contribute once and then leave.

In fact, once you take out those who never participated in LibreOffice but had their code taken from OpenOffice, you find that almost no one in this “developer base” actually does anything. For example 261 of the “developers” combined (over 1/2 of all of the claimed developers) together did only 1% of total commits.  So there is a long tail of inactive “developers” who are puffing up the LibreOffice claims.

This is a little easier to see with reference to the following chart, which shows the cumulative number of code commits (y-axis) against the cumulative number of developers (x-axis).  It shows, for example, that 10% of the developers, mainly Novell/SUSE and RedHat employees, were responsible for nearly 90% of all of the work.  It also backs up my observation that the vast majority of the claimed “large developer base — over 540 people” and the “incredibly efficient self-governing machine” makes an overall miniscule contribution.  There is nothing wrong with this graph per se,.  Many projects will show some form of this.  But if you make a primary claim on your project’s success as having an independent developer community of 550 people, it is a bit embarrassing that most of them are not actually active, and that many of them never were.

Toward a Better Metric

Part of the confusion here seems to stem from the desire to illustrate two things with one metric:  project capabilities and project diversity.  That is asking too much for one metric.  If you want to look at the capabilities of the project, and do it from the input side, then you need to deal with normalizing differences in skills, experience, time on task, motivation, etc.  This is very difficult in a project where you have a mix of full-time Novell/SUSE employees mixed in with part time and occasional developers.  But of all available options, a raw count of developers is the worst possible metric to pick.  It is meaningless.  Better would be to look at commits, or better line counts, or even better function points, or hours on task, or features, or some measure of output.   No one cares what your input is.  A feature developed by 3000 is not necessarily better than a feature developed by 3.  Results are what counts.

From the diversity standpoint, adding hundreds of names who do nothing is not a way to increase diversity.  There are standard metrics for measuring diversity, inspired by Shannon’s definition of information entropy and commonly used in ecological species surveys.   The Shannon Equitability Index is a scaled value, 0-1, that measures diversity.  A value of 0 would indicate no diversity, that one person did all the work and the other names had zero contribution..  A value of 1 would indicate that the work was evenly done.  In the chart above a value of 1 would have a line at 45-degrees up from lower left to upper right.   If you calculate the Shannon Equitability Index for LibreOffice for all commits since 2010-09-28 you get a value of 0.6413.  It would be interesting to see how this value evolves over time.   Oh, and if you calculate this index for Apache OpenOffice, the value is 0.7268, which is even better, more diverse.

(This post represents my personal opinion only.  The standard disclaimer applies.)

(This post represents my personal opinion only.  The standard disclaimer applies.)

In a previous post I looked at how LibreOffice inflates its user and download stats, claiming to have far more users than it actually has.  Several journalists took these claims at face value and repeated them in their articles, never questioning whether LibreOffice representatives were peddling anything other than the plain, honest truth.  No one seemed to noticed that the claims  did not pass the” sniff test”.  No one investigated more deeply.  Until now.  I hope that after reading these posts that you, gentle reader, will exercise your brain the next time you read a press release or blog post from LibreOffice, and try harder to separate fact from fiction.  It will not be easy.

In this post I’m taking a look at another set of claims, those concerning the size of the LibreOffice community.  I’ll lay out the plain facts and the analysis and invite contradictions or confirmations.  In return I’ll probably get more personal attacks, but that comes with the territory.  The LibreOffice marketing lead has already declared me personally to me their number one enemy.  I’m sure Microsoft is comforted by this thought.

The Claims

LibreOffice, from the start, has made incredible claims as to the size of its volunteer base.  The claims read like something from ancient battle accounts, with men 10-feet tall and armies of 500,000.

Specifically, in a recent blog post, LibreOffice makes the following claims:

• “We are now a family of thousands of contributors around the globe”
• They have “…an even larger number of active volunteers taking care of localizations, quality assurance, community development and marketing at global and local levels”
• And that these additional volunteers are “a community of over 3,000 volunteers from the five continents”

As is common with LibreOffice announcements, these claims are made without definitions, without a stated methodology, without context.  So the innocent reader might read terms like “family of contributors” or “active volunteers” or “community” and think these terms are used in their ordinary sense.  But they are not, as we shall see.

The Mythical Wiki Army of 3000

The key to finding the 3000 contributors claimed by LibreOffice is to note the fine print of their blog post, where they say of the additional volunteers:  “Overall, the number of these people is over 3,000, if we take as a measure those who have contributed to the project wiki.”

Ah, so to the wiki we go now to seek out this mighty army of 3000.

Let’s take a look at their wiki stats then.  I’ll give a screen shot in case this page becomes unavailable:

So as you see, they do indeed have 3,510 “registered users”.  So their blog post was correct, end of story.   They indeed have “a community of over 3,000 volunteers from the five continents”.  Right?

Not so quick.  There is less here than meets the eye.  Far less.   Let’s look at some problems with this figure:

First, the sniff test.  If you had a community of 3500 wiki contributors, would after 2 years your wiki have only 2160 content pages?  Is this the output one would expect from a community that size?  Less than one half-page per contributor per year, from this “larger number of active volunteers” ?  This disconnect between claims and reality should be enough to warrant a closer examination.  This just doesn’t sound credible.

Fortunately the wiki stats allow us to see exactly how many edits each registered user made to the wiki.  Curiously, of this “community of over 3,000 volunteers from the five continent”, 1777 of them (over half) have made zero edits.  Zero, zip, gar nichts, nada, niente, zilch.  This is quite remarkable.  A community of contributors where half have made no contributions?!  Is that what you commonly think of when you read the phrase “active contributor” in a press release?  Evidently, in LibreLand you do.

Further, there are many users with a single edit, accounts like Cashloans121, Fastloans1, Fastloans2, etc.  Interesting names, yes?  Of course, these are the spam accounts, created so that advertising could be added to User or other pages.   It is also common for users to register and to make no other “contribution” than to put their C.V. on their User page.  I won’t embarrass the individuals users who have done this, but I see many examples of this on the LibreOffice wiki, where the only “contribution” from a user is self-promotion.   In total, 583 of the registered users made only a single edit in the past two years.   449 have made only 2 edits.  Sadly, our army of 3000 “active volunteers” is shrinking at a distressing rate.

Spam and other issues are well-known to organizations that use wikis.  You don’t claim that your community consists of all registered users of the wiki.  To make a claim like that in a press release is deceptive.  It is a statistic that has no relation to reality.   If the Apache OpenOffice project did exactly what LibreOffice did, and claimed its community size based on the number of registered wiki users, it could claim it had over 75 thousand contributors!

So what is one to do, with such messy data?  Certainly claiming 3,000 contributors without any caution about the above concerns is not recommended.   Instead, if the phrase “active volunteer” is more than empty syllables you need to apply some reasonable threshold to separate an active member of the community from spam, empty registrations or volunteers that showed up for one day and were never seen again.

One criterion is suggested by MediaWiki itself, in its stats report.  It shows that LibreOffice has 112 “active users”, which it defines as users who have made a contribution in the last month.  Another technique might be to look at users who have made, say some non-trivial contribution, say 10 page edits in the past two years, in which case you will show 343 active contributors.  Another way is to ask how many contributors combined account for 90% of all of the edits.  I prefer that metric, and the answer in this case is 342 active contributors.   But the only way you can claim  “a community of over 3,000 volunteers from the five continents” is to have a disregard for facts, and also a disrespect for your reader.  You burn credibility, one of the most important assets an open source project can have.

However you slice it, LibreOffice is overstating the size of their active contributor community by a factor of 10.

(This post represents my personal opinion only.  The standard disclaimer applies.)

(This post represents my personal opinion only.  The standard disclaimer applies.)

Part II is here and Part III is here.

The Claims

I’ve recently read some implausible claims from the LibreOffice project,  concerning their stats for downloads and users.  (These two different statistics are unfortunately conflated in their publicity campaigns, but more about that later).   Their claims fall apart if given any scrutiny and placed against comparable numbers from Apache OpenOffice.  I think you’ll agree by the time you are done reading my analysis .

If this were merely yet another case of puffery from the LibreOffice marketing department then I might just let it go, as I have with many other similar claims in the past couple of years.  But to the extent that some people seem  take these claims as facts, and are repeating, them, then I hope I will be forgiven for giving truth a chance to be heard.  I’ll  lay out the numbers as I know them and let you be the judge.

First, what do we have on the Apache OpenOffice side?  Most of our downloads are from our download site hosted by SourceForge.  The download stats are public and exposed by SourceForge via their REST API.    We gather these stats with a Python script (also public here) and that data is saved to a data file, which is then plotted on our website.  So everything is open and transparent here.  The downloads are counted by a respected 3rd party and the entire processing of these numbers is open for inspection.    It is all there, day-to-day, including breakdown by country and operating system.  We have nothing to hide.

The LibreOffice numbers, on the other hand, we only know from download claims in press releases, and then only at long intervals.  We have no idea what exactly they are counting.  They have never made the detailed stats public.  This does not mean that the numbers are incorrect of course.  It just means that no one outside of their project’s leadership is able to verify the claims.

Actual Numbers

But taken for what its worth, let’s look the recent LibreOffice claims and compare it to the actual data posted by Apache.

• On Sept 27th, LibreOffice claimed “Downloads since January 25, 2011, the date of the first stable release, have just exceeded 18 million”.
• On that same day, OpenOffice had accumulated 18,207,610 download via SourceForge.  (Per the posted data file, which you can verify against SourceForge’s Stats API if you wish)

So both projects are doing equally well, yes?

Well, no, not at all.  You need to take the time interval into consideration.  The LibreOffice counts were from January, 2011.  The OpenOffice counts were from May, 2012.  So in just a few months OpenOffice was downloaded as many times as LibreOffice was in its first two years.

If we convert to an average daily download rate we see:

• LibreOffice:   18,000,000/611 days = 29, 460 /day
• Apache OpenOffice:  18,207,610/143 days = 127, 326 /day

So the download rate has been 4x greater for Apache OpenOffice, and shows no sign of slowing.

A chart might make this clearer, showing the actually OpenOffice download figures (that is why the line is a little wavy) and the claimed LibreOffice trend.  (Anyone want to guess on what this chart will look like six months from now?)

Downloads versus Users

It is important when looking at download numbers that one does not equate download counts with user counts.  This is especially true when you are dealing with upgrade cycles.  As you probably know, neither OpenOffice nor LibreOffice have an incremental update facility.  If you want to update, say from Apache OpenOffice 3.4.0 to 3.4.1 then you need to download a complete copy of 3.4.1 and install it over your 3.4.0.

This complicates things.  Upgrades tend to inflate the download counts, since an upgraded user is counted twice: once for their original download and a second time for their upgrade.    This makes estimating the number of users from the number of downloads tricky.  So to be fair, when estimating the number of Apache OpenOffice users we must not neglect the impact of having a minor maintenance release on causing two downloads for users who upgraded.

But if this is an impact on OpenOffice, which had only two releases to reach 18,207,610 downloads, then how much greater must be the impact for LibreOffice?  For example, their 3.5.x series had 7 releases to fix critical bugs.  Their download counts included downloads from 3.3.x,  3.4.x  and 3.6.x series as well, each one with its own set of bug fix releases.  One must assume, due to the long duration of this reporting interval (nearly two years) and the instability of the early releases within each series, that LibreOffice users have upgraded numerous times each, causing numerous duplicate download counts, and leading the aggregate download count to reflect several times the number of actual users.

In other words, having a rapid release cycle with no incremental update facility will juice your download numbers since each real user will end up downloading many copies of your product.  Since LibreOffice had a dozen or more releases, and OpenOffice only two, it is logical to conclude that the LibreOffice user numbers are far less than suggested by their download numbers, perhaps lower by a factor of 4 or 5.

Objection: External Sites

I anticipate several objections against the above analysis, so let’s treat those one by one.

First, one might note that LibreOffice has claimed an additional two million downloads from “external sites offering the same package”.  Since these claims are not backed with names or numbers, I cannot say much other than the fact that OpenOffice is downloaded from external websites as well.  But we don’t count those in our main download counts.   But suppose we wanted to, and wanted to do apples-to- apples comparison with LibreOffice, with numbers from a 3rd party neutral source?

Let’s take Download.com, CNET’s software download repository, one of the most popular download sites around, as an example.  Here are the download numbers for the 3-month period from 7/28/12 through 10/28/12:

• OpenOffice: 328,846 downloads
• LibreOffice:  18,008 downloads

In this case the OpenOffice download numbers are greater by a factor of 18x.

So I don’t think the external download sites changes things much.  The numbers are small overall, but per day the OpenOffice numbers are far higher than LibreOffice’s.

Objection: Linux users

On top of the 20 million users LibreOffice claims on Windows and Mac, they also stick a finger in the air and decide they have 30 million Linux users as well.  This leads to extravagant claims like, “As of today, LibreOffice is being used by close to 60 million people”.  They don’t detail how they arrive at this number, but it appears to be the culmination of a series of implausible assumptions:

• Take the highest of the several estimates for the number of Linux desktops
• Assume that everyone is using their Linux desktop for  document editing
• Conveniently ignore AbiWord, KOffice, Gnumeric, Calligra, Google Docs or even MS Office under Wine, and assume that everyone on Linux uses LibreOffice.
• Ignore the many Linux users who are displeased with LibreOffice and who have uninstalled it and replaced it with OpenOffice instead.

They make these assumptions and then claim another 30 million LibreOffice users on top of their inflated claim of 20 million Windows/Mac users.

But this really misses the point.  The trajectory is what matters.  In a long race you bet on the faster horse, not the one who has a small head start.  You can have 100% of the 3% Linux desktop market and even under the rosiest assumptions that is only 3%.    And that number is decreasing, as desktop users move to tablets, where Android is the player and the  Apache License is preferred for userspace code.  And I doubt Google will prefer LibreOffice in this space over their own recent QuickOffice acquisition, which already has an app supporting Android (and iOS).

Another point is that one should not equate users who intentionally download and install a product  with users who have it automatically installed as part of an OS, without their knowledge.  These are not the same thing, and to treat them as such is to confuse a downhill skier with someone who fell down a snowy hill.  The one does something intentionally;  the other has something done to them.

That is not to say that Linux users are not important.  We certainly treat Linux as a first-class platform within the project.  I’d like to see us do the packaging work necessary to make Apache OpenOffice available to users on Linux, via their distros.   Users should have choice, even on Linux.  If you’re interested in helping with this,  send me an email.

Objection: All numbers are incomparable

Another objection is to say that all projects live in a different context, with a different user base and that the numbers can never be compared against each other in a fair way.  All is relative, subjective, and LibreOffice is justified in making any claim it feels like, since it is its own reference and base of comparison.

There are several counters to this objection.  First, when LibreOffice publishes numbers, in press releases and blog posts, it has an obligation not to deceive its readers.  This is basic professional ethics.   When you claim a certain number of users, there should be some solid basis for making that claim, not merely the absence of contradictory information.  In any case, I’ve provided adequate contradictory data in this post.

Another counter is to point out that some comparisons are closest to an apples-to-apples comparison.  For example, the number of Windows downloads directly from a project’s website.  OOr downloads from a neutral 3rd party website like CNet’s Download.com.   Of course we can debate the fine details and nuances to the right of the decimal place.  But that does not provide an excuse for conflating download numbers with user numbers in a  press release.  You may not know everything, but you should know that this is not right.

Summary

Apache OpenOffice makes available detailed download statistics in near-realtime for inspection.   LibreOffice makes download claims in press releases at wide intervals with no supporting data.

If you do an apples-to-apples comparison, of Windows and Mac users, which together constitute 97% of the desktop market,  Apache OpenOffice, although it took a while to make its first release,  3.4.0, has taken off like a rocket, and has eliminated any head-start advantage LibreOffice had, and is racing ahead with 4x the downloads that LibreOffice is reporting. And since the LibreOffice numbers are inflated by duplicate counting of upgrade downloads, OpenOffice is probably already ahead of LibreOffice in users on these platforms by a factor of 10 or more.

Under a series of implausible assumptions, LibreOffice claims an additional 30 million users on Linux.  The actual number is unknown, but likely far less.  But since Linux desktops are only 3% of the desktop market, and that market is shrinking, this is not a realistic growth opportunity for LibreOffice.

(This post represents my personal opinion only.  The standard disclaimer applies.)

From the whispers of ApacheCon, OpenOffice.org may never leave the incubator project. The intention may be to do a thorough code audit and produce one last, clean release that the rival LibreOffice can absorb.

That was what you may have heard 10 months ago,  if you listened to the rumormongers.   Certainly there were a lot of rumors being spread.  (Or should we call it FUD?)  Whatever you call it, the whispers continued, in a negative propaganda campaign that the open source community should be ashamed to be associated with.   Even just a few weeks ago I heard from one LibreOffice lead that he was certain that the Apache OpenOffice podling would never graduate and that we’d fail, give up, shut down the project and give the OpenOffice trademarks to LibreOffice.  I’m sorry to disappoint, but this kind of FUD has an expiration date, and that date is now.

From the whispers of ApacheCon…

Yes, you will hear talk of OpenOffice at ApacheCon next month, a lot of it, but it will be quite in the open, no whispers there.  The Apache OpenOffice Project, no longer a “podling”, (Did I neglect to mention that we graduated from the Apache Incubator in a unanimous ASF Board resolution?) will be running a track dedicated to OpenOffice and related technologies.

And as for a clean release that LibreOffice can “absorb”, they are welcome to it.  In fact they have for several months now been merging (“rebasing” is there preferred term) Apache OpenOffice code into LibreOffice, and I couldn’t be happier about it.    Ironically, after demonizing the permissive Apache License,  it is for this very reason that LibreOffice is doing this “rebasing”, to escape from the constraints of LGPL.   After all the demagoguery, their source files will now carry an Apache License notice.

I need not repeat the long list of other false predictions and rumors: that we would never be able to bring the product’s IP up to Apache standards (we did), that we would not be able to issue security patches for OpenOffice (we did),  that we would never get a release out the door (we did, twice), that we had delayed too long in our release and were thus irrelevant (we had more downloads in 4 months than LibreOffice has had  in 2 years), that we would never contribute developers to the OpenOffice effort (we have), that we would never donate Symphony to Apache (we did), that we would dominate the project (we don’t) or that we would force Symphony to be the new base of OpenOffice (we didn’t), etc.  The FUD went on and on and continues even today, combined with exaggerations of their own modest achievements.

It is probably a vain hope to expect the FUD to stop now that we’ve graduated, though I would be happy to be wrong.  But at the very least I think we’ve established a record of accomplishment that stands in stark contrast to the repeated false predictions of the anti-Apache whisper campaign.  And it is worth noting this,  and preserving some skepticism when hearing further FUD from these same sources.   And this is something worth saying louder than a whisper.

I am not an accountant.   However, as a Graham and Dodd value investor over the years, I’ve picked up some of the fundamental principles.   A key one is the Matching Principle, that revenues and expenses should be booked in a way that clarifies the underlying business performance, rather than based purely on the timing of cash transactions.  In some cases this requires the use of special accounts, for things like depreciation, where the lifetime of a fixed asset (say factory machinery) extends beyond a single revenue cycle.

A similar technique is used when dealing with deferred expenses. For example take the case of a nuclear power plant.  A plant has a useful lifetime, but when that end date arrives there is a clean up cost.   The property is not immediately usable for other purposes, but must undergo an expensive remediation.  From an accounting perspective there is an asset retirement obligation, a form of deferred expense.  This deferred expense is recognized on the books as a liability based on the present value of the expected clean up cost, which is then depreciated.

I think of vendor lock-in in a similar way, but one where the clean up costs are commonly not recognized, either formally or even informally, as a deferred expense.  Instead, the costs are attributed 100% to any alternative vendor.  This leads to absurdities like the claim that moving to OpenOffice from Microsoft Office would be a huge expense.  That is like saying that an old, non-functioning power plant cannot be used for any other purpose because the cleanup costs are too high.    The costs could be real, but we only obscure the economic reality when we recognize vendor lock-in only at exit.  It should have been considered at the entrance, and as a liability against picking a closed solution.

Something to consider.  When you pick:

• A closed solution
• A non-open standard

In those cases you are risking vendor lock-in.  And by doing so you are committing  your company to either a perpetuity of lock-in, or to a future expensive remediation to restore  your options to the point where you can make alternative choices.  The smart way of doing this is to consider, in terms of analysis to aid decision making,  the cost of lock-in up-front, as a deferred expense, a liability.  When this is done you can more accurately make comparisons among available solutions, closed and open, knowing that you are fairly accounting for the true economic costs of each choice.

You may have read, on the Apache OpenOffice blog, news that the project has had 5 million downloads in the first 6 weeks since the release of version 3.4.  And as the above chart shows, the download rate has increased in the past two weeks, as we’ve started to roll out the upgrade notifications to OpenOffice.org 3.3 users.

When I mention the “5 million” achievement, the reaction is generally along the lines of, “That’s excellent !!!  Right?  That is good, isn’t it?”  The fact is the number is large, but without comparison or context, it is hard to gauge.     I think I can provide some comparisons and context to put these numbers in perspective.

First, let’s look at OpenOffice.org 3, and their famous claim of 100 million downloads.  That was in the time period from October 13, 2008 to October 28, 2009, so 380 days.  That averages out to around 260K downloads/day.   I’m not quite sure what they counted as a “download”, whether just full installers, or language packs as well.   And that time period overlaps with several releases (3.0.0, 3.0.1, 3.1.0 and 3.1.1, so there is some double, triple and quadruple counting of users due to upgrades.

For another comparison, let’s take a look at LibreOffice.  They claimed 7.5 million downloads between January 2011 and October 2011.  That averages out to 27K downloads/day.    And again, it is not clear if that counts all downloads, including multiple downloads by the same user as they update from release to release.

So how does Apache OpenOffice 3.4 compare?  Let’s state the numbers as conservatively as we can.  We’ll count only installer downloads, not language packs or SDK’s.  And we’re counting only for a single release, AOO 3.4, so there is no double counting due to upgrades.  Based on these assumptions, our average download rate has been 118K downloads/day.  (But as the chart shows, since we enabled the update notifications, the rate is now more like 170K downloads/day.

So overall I think we’re doing quite well.   There is room for improvement, but it is a good baseline against which we can show progress.  One thing we can do to grow these numbers to increase the native language support, to restore some of the key translations.   If you are interested in volunteering with Apache OpenOffice, you should read this page, and then send an email to our mailing list to introduce yourself and your interest.

A quick side note:  Some readers will observe that Linux users get their software from the distros, not from downloading from a website.  This is a safe assumption for most, but not all Linux users.  But that doesn’t really change the math much.  Assume that none of the LibreOffice downloads are from Linux users.  Assume that the entire 27K/day are entirely Windows and Mac users.   Then, we can do an apples-to-apples comparison to the Apache OpenOffice numbers, where we know that only 3% of the downloads are from Linux users.  So the better comparison would then be to compare  a very conservative 0.97 * 118K = 114 K/day versus LO’s best-case 27K/day for Windows and Mac.    A similar calculation could be done on the legacy OOo 3 figure, with similar results.

Read more in the official announcement.  You can download Apache OpenOffice 3.4 now, from http://download.openoffice.org/    Tell your friends.  And welcome home.

The Importance of Metrics

Revolutionary movements require revolutionary progress.  However, at the start of a Movement, such progress may not be immediately evident to those whose views of progress have been tainted by commercial software, where progress is measured by feature enhancements, quality improvements and user satisfaction.  These are false idols and the shallow view of progress they support are irrelevant for true free software.

Rejecting the repressive capitalist view of progress-as-production and production-as-consumption, and the doctrinaire emphasis on results-oriented metrics, we instead adopt the dialectic of progress-as-being and being-as-becoming, with metrics illustrating not what is produced, but what is willed.  Rather than galley slaves, prodded by whip lashes to “row harder!”, our motto  shall be: “row louder!”

As Mark Twain famously wrote,  “There are three kinds of lies: lies, damned lies, and statistics.”   Our aim in developing metrics is to avoid the taint of this condemnation by avoiding the statistical sciences entirely.

Two metrics that any free software marketeer must be able to produce are:

• A measure of the size of the community, in order to demonstrate the overwhelming strength and vigorous growth of our Movement.
• A measure of the diversity of the community, in order to demonstrate that it is a true People’s Movement with a broad base of support.

Community Size

A metric that can never disappoint is  “cumulative number of developers” on the project.  Calculating this metric is simple:  count the total number of individuals who have contributed to the project since it started.  You are not concerned with the number of contributions, the quality of contributions, whether the person did one patch and then was never seen again, or any other extraneous detail. You’re just counting names.  Any project can use this metric to show month-after-month steady improvements.

The genius of this metric is that it can never decrease. It is mathematically impossible!  Once a name is added to your list, it is never removed.  One a contributor, always a contributor, at least on paper.  Worst case, even if an asteroid should hit your next project conference and kill every one of your developers, you could still report that your numbers were flat.  Unfortunately, your developers would be as well.

Advanced marketeers should also note the following additional techniques, which can be used to generate even more impressive numbers without additional effort:

• Don’t just count those who are writing code.   Almost anyone can be called a “developer” these days.  Translators (“Localization Engineers”), build lab guys (“Configuration Management Engineers”), testers (Software Quality Engineers),  etc.  Include all of their contributions.
• Don’t limit yourself to those who actively contributed code to your project.  You can also include developers who contributed to precursor versions of your code, or who contributed to 3rd party modules. They might not even know that their code is in your product, but that doesn’t matter.  We appreciate their “contribution”  nonetheless and should acknowledge them as an integral part of our thriving community, in spirit at least.
• One you have made the novel step of  counting non-project members as developers, there is nothing in principle against extending this even further to the developers of the tool set and libraries you are using.   Credit them all, living, dead and legendary.  Larry Wall, Dennis Ritchie,  Alan Turing, and St. Hubertus, patron saint of mathematicians, can all be listed as developers on your project.
• With some advance planning your project’s development team can help increase these numbers even more.  Have them think of some simple text manipulation tasks that can be done by volunteers with little knowledge or understanding of the code.  For example, have them do things like reformat code,  spell check comments, alphabetize include directives, ensure that every C/C++ file ends with a newline character, etc.  Although this will not improve the product for your users, it will enable a much larger group of contributors to get involved so we can add their name to our list.  And once on the list, it stays on the list forever.  I am told on reliable authority that such code changes are perfectly safe and cannot have a negative impact on product quality.

But be warned: use of these advanced techniques might open you up to criticism of promoting numbers that are meaningless, that they saying nothing about the actual size or composition of your development community.   Such criticism can easily be dismissed by feigning to be insulted and responding along the lines, “How could you possibly suggest that we not recognize the hard-working {translators | build lab guys | testers | dead mathematicians |  patron saints} who {contributed to|had stuff we took without their knowledge for} our project?”.   This argument will almost win, since no one will want to appear ungenerous, especially when intercessory saints are involved.

Community Diversity

Another metric that you might be asked for is a metric to illustrate how much of the project is dependent on any one company, such as the project’s main sponsor.    A common error is to look at the contributions made to the project and report how much was apportioned to which individuals and companies, based on recognized software metrics, like # of commits, lines of code, function points, etc.  You must not fall into that trap!

A much better metric, if you want to be seen as diverse, is to count the number of developers in each company. Do this without regard to whether the person made a one line change 2 years ago and was never heard of again, or whether the person works 14 hours a day on the project every day. Just look at the number of developers.   If you have already followed the advice outlined above, and increased your “cumulative number of developers” metric, then you are perfectly positioned to also demonstrate your diversity.

For example, suppose you have 400 developers, and 10 of them do 90% of the work, and they are employed by a single company.  Avoid the naive mistake of saying that one company was responsible for 90% of the contributions.  Instead say that the company’s developers (note we’re emphasizing people not work) represent 10/400 or only 4% of the project.   That is the genius of this metric.  You can have almost all the work done by a single company and still claim that your community is diverse.

What is a fork?

A fork is a form of software reuse.  I like your software module.  It meets some or many of my needs, but I need some additional features.

When I want to reuse existing functionality from another software product, I generally have four choices:

1. If your module is nicely designed and extensible, then I might be able to simply use your code as-is and write new code to extend it.
2. I can convince you to modify your module so it meets my needs.
3. I can work with you in your open source project to make the module (“our” module in this case) meet our mutual needs.
4. I can copy the source code of your module and change the code in my copy, and integrate that modified module into my product.

Note that options #1 and #2 are the only options available with most proprietary modules, since these techniques don’t require access to the module’s source code.  Options #3 and #4 are the additional options made possible by open source.  Option #4 is what we mean by “forking” .  Forking is enabled by open source software and is fundamental to open source ecosystems.  It is neither good nor bad.  It is a tool, part social, part technological, for overcoming an inability or unwillingness to collaborate.  The problem is not with forking.  The problem is the conditions that lead to forking.

Why do forks come about and how do they end?

Forks can come about for many reasons, including leadership conflicts, ideological differences and other political issues, as well as differences in vision and technical direction of the project.

Generally, a fork ends when the conditions that necessitated the formation of the fork have been resolved.  At least that is true for rational participants who are merely trying to optimize outcomes.  But intransigent ideological forks can continue indefinitely, and often do.

The technical side of ending a fork is typically a code merge, as different branches of the project are brought back together again.  This can be laborious, but it is a one-time task.

Ending the Symphony Fork

With the move of OpenOffice to Apache, this open source project has made the critical move from a corporate-led open source project under asymmetrical licensing terms, to a community-led open source project under a single permissive license.  This is a tremendous change and one that should lead all forks of OpenOffice,  and all those who wanted to get involved with OpenOffice before but never did,  to reexamine their orientation to the project.

John Maynard Keynes, when criticized for reversing his position in a dispute, famously quipped, “When the facts change, I change my opinion. What do you do, sir?”  The “facts” of OpenOffice have changed, with the move to Apache, and this change of venue has made a huge impact on the Symphony team, which recently announced that it was ending its fork and committing to contribute their code to Apache and to work with that community going forward.

This does not mean that Symphony enhancements are going away.  Far from it.   We’re very proud of the UI work and other innovations in performance, accessibility and interoperability we’ve brought to Symphony and we will be offering the source code of these enhancements to Apache, and if accepted, will work within that project to merge these changes into Apache OpenOffice.  The DNA of Symphony is not going away.  What is going away is Symphony as a fork, as a divided effort.  The Symphony DNA, the cool work the Symphony team has worked so hard on, will live on, in Apache OpenOffice, combined with other ongoing contributions from the community, in a larger, stronger development effort.

Now that the Symphony fork is ending, the obvious question is:    Who will be next?    If we can end a four-year old fork and merge in our work with Apache, then so much easier it should be for forks that have been around for far less time.  “When the facts change, I change my opinion.  What do you do, sir?”

If you are interested in learning more about the Apache OpenOffice project, I recommend browsing the project’s website and blog.  If you want to get involved, you can sign up for the ooo-dev mailing list and post a note to introduce yourself.  As we push closer to our 3.4 release candidate we’re in particular need of volunteers to help us test this release, on Windows, Mac or Linux.  If you are interested in helping with that, be sure to say so in your note.

(This post has also been translated into Serb-Croatian by Anja Skrba.)

The Apache ODF Toolkit 0.5 (incubating) release is now available for download.  Detailed change notes are also posted.  The ODF Toolkit is a Java library for reading, writing and creating ODF documents.  It is entirely in Java and does not require that you install a desktop editor like OpenOffice.  It operates directly on the file format and is suitable for server-side use, for tasks such as document automation, report generation, information extractions, etc.

As mentioned in a previous post, the Java components from the ODF Toolkit Union have moved over to Apache.  Since this open source project was already using the Apache 2.0 license, the work required to achieve our first Apache release was relatively straightforward.    The major task was to take the various components of the Toolkit, which were treated as independent projects at the ODF Toolkit Union, and get them to work better together as a single Toolkit, e.g., build together using the same version of the JDK, package them together into a consolidated release bundle.  Not rocket science,  but it did require some iteration.

We’re starting now to put together a plan for the next release and future releases.  Some of the items under consideration include:

1. Adding document encryption/decryption support
2. Adding digital signature support
3. Update to final published ODF 1.2 schema
4. Update the demo applications
5. Concurrency testing
6. Adding support for ODF 1.2′s RDFa/RDF XML semantic metadata feature
7. Implement ODF 1.2′s OpenFormula spreadsheet formula language
8. Add high-performance event-driven streaming API, for subset of tasks that can be done efficiently that way
9. More cookbook examples
10. More testing and bug fixing

If you are interested in learning more about the ODF Toolkit, you should visit our website.   If you have further questions, we have a users list and a development list that you are welcome to join.

If you know some Java and are interested in ODF, I’d encourage you to take a look at this project and consider participating.  We are a small, international, welcoming group working on this project,  with a strong focus on quality.  Come, take a look.

I had grand plans.  This was supposed to be a cool looking visualization.  Over 2011 I downloaded nearly a terabyte of raw Wikipedia page access stats.  And recently I had a python script running for 3 weeks around the clock crunching the data.   This was going to be cool.  But I ran out of time to complete this project.

So rather than have this data go to waste, I’ll offer it up for whoever wants it.  Maybe you can think of something interesting to do with it?

So what do I have?  I have hour-by-hour data for the top-3 most-requested pages in the English version of Wikipedia.  It is in a CSV file which you can download (wikipedia-hours.zip).  I filtered out any requests to Wikipedia “Special”, “User”, “Template” or similar utility pages.  What is left is just requests to Wikipedia articles.

And for those of you do not do CSV, I’m including here a table showing the data for January.  Some quick observations:

1. Some of the hits are obviously driven by breaking news, such as the shooting of Gabrielle Giffords or the death of Jack LaLanne.
2. Others are driven by the date, especially holidays or other remembrances
3. Some are driven by other media, especially television
4. Some are inexplicable, unless you look at the Google Doodle history page.  For example, Google highlighted Cezanne’s 172nd birthday on their homepage on January 19th.
5. Others defy explanation entirely.  For example, why so much interest in User Datagram Protocol (UDP)?

In any case, take and enjoy.  Let me know if you do anything interesting with it.

To quote the immortal words of Otis B. Driftwood,  “Let joy be unconfined. Let there be dancing in the streets, drinking in the saloons, and necking in the parlor”.

The day has finally arrived.  Open Document Format (ODF) 1.2 has been approved.  It is now an OASIS Standard.

If you are regular reader of this blog, you know all about ODF 1.2, the enhancements we’ve made with OpenFormula, with RDFa/RDF XML semantic metadata, the digital signature support, etc.  I’ve discussed this all before, on this blog and at conferences.

Most likely your office suite already supports ODF 1.2 today.   If not, ask your vendor when they will be adding support for it.

ODF 1.2 is a large standard,  in four volumes, totaling 1217 pages.  It was the work of many hands, over a four year period.   I’d like to acknowledge some of the many who contributed to the success of this standard, with apologies in advance for any inadvertent omissions.

First, credit goes to the editors of the ODF 1.2: Michael Brauer, Dennis Hamilton, Eike Rathke, David A. Wheeler and especially Patrick Durusau.  With a document of this size and complexity, the work of an editor is as much an engineering exercise as technical writing task.   Hats off to our editors for their their accomplishment.

Many companies enabled their experts to participate on the ODF Technical Committee, including prominent technical contributions from Sun/Oracle, IBM, Novell, Microsoft, Nokia and others.

Open source projects should be especially proud of the contributions made to this standard by their project members, especially Gnumeric, KOffice, Calligra Suite, AbiWord, OpenOffic.org, LibreOffice, WebODF, lpOD and the ODF Toolkit.

In some cases,  participation of technical experts was sponsored by organizations such as NLnet and the Friends of Open Document/Open Document Fellowship, and their support is gratefully acknowledged.

A full list of TC members who contributed to ODF 1.2, as copied from Appendix B of the standard:

Chieko Asakawa, Waldo Bastian,  Thorsten Behrens, Nathaniel Borenstein,  Michael Brauer,  Pete Brunet, Manuel Cano,  Suresh Chande,  Robin Cover,  Pierre Ducroquet, Jerome Dumonteil,  Patrick Durusau, Cherie Ekholm,  Ezer Farhi, David Faure, Jean Gouarne,  Andreas J. Guelzow, Bettina Haberer, Dennis E. Hamilton, Bart Hanssens, Donald Harbison, Mingfei Jia,  Bob Jolliffe, Peter Junge, Kazmer Koleszar,  Peter Korn, Jirka Kosek, Robin LaFontaine, Marcus Lange, David LeBlanc,  Fong Lin,  Jun Ma, Yue Ma,  John Madden,  Doug Mahugh,  Ben Martin, James Mason, Tristan Mitchell, Duane Nickull,  Michael Paciello, Ganesh Paramasivam,  Eric Patterson,  David Pawson, Steven Pemberton, Stephen Peront,  Asokan Ramanathan, Eike Rathke,  Florian Reuter,  Janina Sajka, Svante Schubert, Charles Schulz, Richard Schwerdtfeger, Douglas Schepers, Wei Guo Shi,  Michael Stahl,  Yan Shi, Jomar Silva,  Frank Stecher,  Hironobu Takagi, Malte Timmermann,  John Tolbert,  Elias Torres,  Warren Turkal,  Jos van den Oever, Alex Wang, Robert Weir,  Oliver-Rainer Wittmann,  David A. Wheeler, Cheng XiuZhi,  Panrong Yin, Kohei Yoshida,  Helen Yue,  Jin YouBing, Thorsten Zachmann,  Thomas Zander and Pine Zhang.

ODF 1.2 obviously built on the previous work in ODF 1.1 and ODF 1.0, so it is fair to acknowledge as well those who laid the foundation that we built upon:

Daniel Brotsky, Jerome Dumonteil,  Charles Schulz,  Jerry Berrier,  Donglin Wang,  Rui Zhao,  Stephen Noble, John Madden, Chieko Asakawa, Nathaniel Borenstein, Pete Brunet, Yue Ma, Richard Schwerdtfeger, Robert Weir, Zhi Yu Yue, John Barstow, Patrick Durusau, Michael Paciello, Janina Sajka, David Clark, Waldo Bastian, James Mason, David Faure, Jody Goldberg, David Pawson, Michael Brauer, Peter Korn, Lars Oppermann, Eike Rathke, Svante Schubert,  Frank Stecher,  Malte Timmermann, Daniel Bricklin,  Daniel Carrera,  Bruce D’Arcus,  Gary Edwards,  Elmar Geese,  Sam Hiser,  Michael Kleinhenz,  Tomas Mecir,  Thomas Metcalf,  Stefan Nikolaus,  Florian Reuter,  Daniel Vogelheim, David A. Wheeler,  Chris Nokleberg,  Paul Grosso,  Tom Magliery,  Doug Alberg,  Paul Langille, John Chelsom, Monica Martin, Jason Harrop, Uche Ogbuji,  Lauren Wood, Simon Davis,  Mark Heller and Phil Boutros.

The work of the ODF TC was complemented and amplified by a larger community of related committees and organizations, including the OASIS ODF Adoption TC, the OASIS ODF Interoperability and Conformance TC, the ODF Alliance, the OpenDoc Society and OpenForum Europe.

I’d like to also acknowledge the many who who submitted comments during the unprecedented 270 days that ODF 1.2 was under public review, with notable contributions from Regina Henschel, Leonard Mada, Michiel Leenars, Norbert Bollow, Casper Boemann,  Németh Lászl, Søren Roug, and someone known to us only as “Ronnie the Bonnie”.  We also received, via the comment list and via official defect reports, valued contributions from ISO/IEC JTC1/SC34 technical experts, including Alex Brown (UK), Murata Makoto (Japan) and Jesper Lund Stocholm (Denmark).

My personal thanks as well to the OASIS staff, especially Chet, Robin, Mary, Carol, Jane, Laurent and Jamie, for their constant support.

Finally,  a special thanks Michael Brauer, whose diligent efforts and leadership as ODF TC Co-Chair, made ODF 1.0, 1,1 and 1.2 possible, and whose legacy lives on into our work on ODF 1.3.

When work first started on ODF 1.0, back in December, 2002,  the idea of having an open standard for office documents was radical. Every word processor had its own format, and most formats were undocumented or had documentation available only under anti-competitive licenses.  ODF challenged that status quo and shook the palace walls of companies whose business models relied on ensuring that your documents were the source of their vendor lock-in.  In some places the walls crumbled.  Today having an open standard document format is considered to the norm.  We’re all open standards supporters now, at least in words.

So a time to celebrate this important accomplishment.  There will be no OASIS ODF TC next week.  We’ll take the week off from ODF work.  But we’ll be back at it the following Monday.    We need to start preparing a submission to ISO/IEC JTC1, to refresh ISO/IEC 26300:2006.  And we continue work on producing a first draft of ODF 1.3.

Perhaps overlooked in all the excitement generated by the move of OpenOffice.org to Apache was the fact that a parallel move is occurring with the ODF Toolkit.  A few weeks ago we submitted a proposal to Apache to start a new project based on the Java components that were until then hosted by the ODF Toolkit Union.   This was done after consulting with ODF Toolkit community and getting approval from the ODF Toolkit Union’s Steering Committee.  This proposal was recently reviewed, voted on and approved by Apache.   So now we have the Apache ODF Toolkit project in the Apache Incubator.

So what is this project and what is it good for?

This project consists of Java libraries and tools for working with ODF documents.  Not editors, not viewers, not anything with a user interface.  These are not end-user tools.  These are tools for developers who need to write programs that read, write or manipulate ODF documents.  These tools do not require that you have any ODF editor installed.  They operate directly on the files.  So they are ideal for running on a server, for things like report generation, information extraction, document validation, conversion,  etc.   We have a page of demos that gives a good idea of the range of things possible with the ODF Toolkit.

The ODF Toolkit is important because it enables innovation on top of ODF.  By analogy, look at HTML.  At one point, the web consisted mainly of hand-authored documents at a handful of academic and government websites.  If that was all there was to the web, it would not have been very interesting.  What made the web the platform it is today has been the technologies that enable server-side generation of web pages from database queries, or services that analyze web pages and extract and aggregate information.  Google was made possible because HTML was an open standard that could be programmatically understood.  PHP was possible because HTML was an open standard that could be written.

ODF, unlike the previous generation of binary document formats, is also an open standard.  You can read and write ODF documents freely.  But writing the code to understand the nitty-gritty of the ODF format is a considerable task.  The ODF Toolkit makes this easy for Java programmers.  How easy?  Here is a “hello world” text document:

TextDocument doc=TextDocument.newTextDocument();
doc.addParagraph("Hello world!");
doc.save("hello.odt");

Other tasks, like change styles, combining presentations slide decks, searching and replacing text in a document, extracting text from a document are also simple.  More examples that give a flavor of the ODF Toolkit are in the “cookbook“.

But along with the “Simple API” the ODF Toolkit has the ODFDOM layer.  This layer allows you to get to every part of an ODF document, at the finest grain level.    Some tools out there give you only a high level API but then leave you hanging if you want to do something more complicated.  Not so with the ODF Toolkit.  If you want to drill down and adjust the line spacing of a bullet list in a footnote,  then you can do it.

These components enable innovation on top of ODF, innovation that thinks “outside the editors” and “beyond office”.

So how do you get involved?  If you want to help with the project then I invite you to sign up on the project’s development mailing list.  And if you have questions about using the ODF Toolkit,  but don’t want the additional email traffic from the dev list, then you can sign up for the users list.  Of course, I’ve signed up for both lists.  I hope I’ll see you there!

Photo courtesy of Bjorn Lundell

I attended the 6th ODF Plugfest took place in Berlin a few weeks ago, hosted by the German Federal Ministry of the Interior (BMI) and the Ministry of Economics and Technology (BMWi).    It followed the pattern of previous events,  a two-day event,  with the first day dedicated to technical interop activities among implementors, followed by a day of public presentations.   The full agenda is here.

(I’m the jet-lagged figure in the lower left of the above group photo taken on the first day of the Plugfest).

We tested a range of interoperability scenarios, including preservation of RDF metadata,  conditional formatting in spreadsheets and advanced text layout scenarios (“hard cases”).  There were also a variety of presentations, from vendors, academics and government.   I’ve uploaded my presentations if you are interested.    I’ll draw your attention as well to Ross Gardler’s talk on The Apache Way and OpenOffice.org.  He covered a lot of good material.  (A recording of the talk is also online)

Finally, it was also announced that the next ODF Plugfest will occur in the Netherlands, in Gouda, in November.   And another one is anticipated in Brussels in March 2012.