Archives for 2012

(This post represents my personal opinion only.  The standard disclaimer applies.)

In previous posts I looked at claims made by LibreOffice, in project blog posts and press releases, related to the number of LibreOffice users and the number of active LibreOffice contributors.  I showed that in both cases the claims from LibreOffice were greatly inflated due to various flaws.  For example, they double counted users who upgraded from earlier release of LibreOffice, often several times over.  And they counted as “active contributors” those who registered for a wiki account but never actually contributed anything.  In this blog post we’ll look at the even more egregious ways which the LibreOffice project is overstating the number of developers that are active with the project.

A Quick Quiz

To prepare your frame of mind for what you are about to learn, I encourage you to first take the following quiz.

When asked to report on the population of your home town, what would you report?

A.  The number of people with primary residences in the town.

B. The number of people who have ever lived in the town, even if they no longer live there.

C. The number of people who drive through the town on their way to somewhere else.

D.  All of the above.

If you picked D, you would be an excellent candidate for the LibreOffice marketing department.

With that mental preparation out of the way, let’s continue.

The Claims

• From a recent LibreOffice announcement:  “growing developer base, which has just reached the number of 550 since the launch of the project, making LibreOffice one of the fastest growing free software projects of the decade.”
• Or a couple of weeks ago: “LibreOffice is the result of the combined activity of 540 contributors”
• Quoted on Linux.com:  “our large developer base — over 540 people at the end of September 2012 — is an incredibly efficient self-governing machine”

You can find many variations on this same claim.

All Your Developer Are Belong to Me

With a number this large, it should not be hard to find these 550 developers.  So let’s see if we can track this down.   One place to start is to look at the LibreOffice credits page.  We see there a large table of “Developers committing code since 2010-09-28”.  If we count the names in this table we get 469.  Not quite 550, but pretty close, yes?

But if you look closer at the names in the list, you begin to scratch your head.  There are names here of former Sun/Oracle developers who lost their jobs when Oracle stopped developing the project.  Some commentators, like Mark Shuttleworth, put much of the blame for Oracle divesting from OpenOffice on the “radical faction” that forked to create LibreOffice.   Now aside from costing them their jobs, LibreOffice now insults them by using their names for propaganda purposes to puff up LibreOffice’s developer claims?!

Looking further, I see the names of IBM colleagues who have never participated in the LibreOffice project.  They are active developers on Apache OpenOffice, and former OpenOffice.org developers, but here they are listed in a table of “Developers committing code”.  How curious the ways of LibreLand!

If you scroll down to the bottom of the table you get a clue in the fine print:  “We can not distinguish between commits that were imported from the OOo/AOO code base and those who went directly into the LibreOffice code base.”

Hmmm… so let me get this right.   If you take my code, you say that I committed it to the LibreOffice project.   And if I contributed to the code to OpenOffice.org or Apache OpenOffice, and you take it, you’ll list me in your LibreOffice developers table for a contribution I never made to LibreOffice and put a “joined” date next to my name for an organization I never joined.   Really?

This is an odd way of accounting for developers.    I’m pretty sure that 100% of readers of LibreOffice press releases and 100% of journalists who write articles based on LibreOffice claims would feel somewhat abused by such idiosyncratic definitions.   It is certainly not the most honest and forthright way of stating how many developers LibreOffice has.   One does not expect that OpenOffice.org developers, who were never involved with the LibreOffice project, and may not even know that their code is being used, will be included in the count.

Monotonically Increasing

Aside from counting people who are not actually involved in LibreOffice and never were, the LibreOffice claims are peculiar because of the low threshold for inclusion and the perpetuity of inclusion once added.  When you hear claims of a “developer base” you are lead to think of a body of actual developers actually working at present on the code.  That would be the normal usage.  But in LibreLand it is not done that way.  If you made a single contribution ever (or as we know now from the above, even never) then you are in the “developer base” and will be listed as a LibreOffice developer for all time.

From the perspective of gratitude and acknowledgement, giving credit is fair and generous.  Apache OpenOffice also has a long list of names on its Credits page.   But we don’t tally this retrospective list of past contributors and claim that number as an active community size.  From the perceptive of claiming a community size, this would be deceptive.  That is like calculating the population of a town by listing everyone who ever lived there.

Because of this odd practice, the LibreOffice developer count will never decrease.  It can only go up.  Even — worst case — if an asteroid hits their next hackfest — the numbers would merely be flat.  (So would the developers present)  In any case if you’ve designed a metric that can never decrease, then it should not be newsworthy for you to report that it is increasing.  This is not an accomplishment.  That is just mathematics.

How to Juice the Developer Count

An easy way to increase, for reporting purposes, the number of “developers” a project can claim is to encourage trivial churning of the code base.  For example, translating comments from German to English, removing dead code and other similar tasks can be done without even really knowing C++, or at least not knowing it well.  But it can prompt the temporary or even one-time participation of many “developers”, and in the process increase your developer count.  LibreOffice made a tremendous effort to enable a low threshold for contributions and this effort paid off, at least in developer counts.

As an example of the impact such practices can have, I took a look at the “core” git repository for LibreOffice, and all of the commits since 2010-09-28.  After identifying and collapsing multiple email addresses used by some persons, I ended up with 518 names.  Of those names, 166 , or 1/3 of them, have made only a single commit, and then were never heard of again.  So it is curious to count them as part of LibreOffice’s vaunted “developer base”.  A community is not made up of those who contribute once and then leave.

In fact, once you take out those who never participated in LibreOffice but had their code taken from OpenOffice, you find that almost no one in this “developer base” actually does anything. For example 261 of the “developers” combined (over 1/2 of all of the claimed developers) together did only 1% of total commits.  So there is a long tail of inactive “developers” who are puffing up the LibreOffice claims.

This is a little easier to see with reference to the following chart, which shows the cumulative number of code commits (y-axis) against the cumulative number of developers (x-axis).  It shows, for example, that 10% of the developers, mainly Novell/SUSE and RedHat employees, were responsible for nearly 90% of all of the work.  It also backs up my observation that the vast majority of the claimed “large developer base — over 540 people” and the “incredibly efficient self-governing machine” makes an overall miniscule contribution.  There is nothing wrong with this graph per se,.  Many projects will show some form of this.  But if you make a primary claim on your project’s success as having an independent developer community of 550 people, it is a bit embarrassing that most of them are not actually active, and that many of them never were.

Toward a Better Metric

Part of the confusion here seems to stem from the desire to illustrate two things with one metric:  project capabilities and project diversity.  That is asking too much for one metric.  If you want to look at the capabilities of the project, and do it from the input side, then you need to deal with normalizing differences in skills, experience, time on task, motivation, etc.  This is very difficult in a project where you have a mix of full-time Novell/SUSE employees mixed in with part time and occasional developers.  But of all available options, a raw count of developers is the worst possible metric to pick.  It is meaningless.  Better would be to look at commits, or better line counts, or even better function points, or hours on task, or features, or some measure of output.   No one cares what your input is.  A feature developed by 3000 is not necessarily better than a feature developed by 3.  Results are what counts.

From the diversity standpoint, adding hundreds of names who do nothing is not a way to increase diversity.  There are standard metrics for measuring diversity, inspired by Shannon’s definition of information entropy and commonly used in ecological species surveys.   The Shannon Equitability Index is a scaled value, 0-1, that measures diversity.  A value of 0 would indicate no diversity, that one person did all the work and the other names had zero contribution..  A value of 1 would indicate that the work was evenly done.  In the chart above a value of 1 would have a line at 45-degrees up from lower left to upper right.   If you calculate the Shannon Equitability Index for LibreOffice for all commits since 2010-09-28 you get a value of 0.6413.  It would be interesting to see how this value evolves over time.   Oh, and if you calculate this index for Apache OpenOffice, the value is 0.7268, which is even better, more diverse.

(This post represents my personal opinion only.  The standard disclaimer applies.)

(This post represents my personal opinion only.  The standard disclaimer applies.)

In a previous post I looked at how LibreOffice inflates its user and download stats, claiming to have far more users than it actually has.  Several journalists took these claims at face value and repeated them in their articles, never questioning whether LibreOffice representatives were peddling anything other than the plain, honest truth.  No one seemed to noticed that the claims  did not pass the” sniff test”.  No one investigated more deeply.  Until now.  I hope that after reading these posts that you, gentle reader, will exercise your brain the next time you read a press release or blog post from LibreOffice, and try harder to separate fact from fiction.  It will not be easy.

In this post I’m taking a look at another set of claims, those concerning the size of the LibreOffice community.  I’ll lay out the plain facts and the analysis and invite contradictions or confirmations.  In return I’ll probably get more personal attacks, but that comes with the territory.  The LibreOffice marketing lead has already declared me personally to me their number one enemy.  I’m sure Microsoft is comforted by this thought.

The Claims

LibreOffice, from the start, has made incredible claims as to the size of its volunteer base.  The claims read like something from ancient battle accounts, with men 10-feet tall and armies of 500,000.

Specifically, in a recent blog post, LibreOffice makes the following claims:

• “We are now a family of thousands of contributors around the globe”
• They have “…an even larger number of active volunteers taking care of localizations, quality assurance, community development and marketing at global and local levels”
• And that these additional volunteers are “a community of over 3,000 volunteers from the five continents”

As is common with LibreOffice announcements, these claims are made without definitions, without a stated methodology, without context.  So the innocent reader might read terms like “family of contributors” or “active volunteers” or “community” and think these terms are used in their ordinary sense.  But they are not, as we shall see.

The Mythical Wiki Army of 3000

The key to finding the 3000 contributors claimed by LibreOffice is to note the fine print of their blog post, where they say of the additional volunteers:  “Overall, the number of these people is over 3,000, if we take as a measure those who have contributed to the project wiki.”

Ah, so to the wiki we go now to seek out this mighty army of 3000.

Let’s take a look at their wiki stats then.  I’ll give a screen shot in case this page becomes unavailable:

So as you see, they do indeed have 3,510 “registered users”.  So their blog post was correct, end of story.   They indeed have “a community of over 3,000 volunteers from the five continents”.  Right?

Not so quick.  There is less here than meets the eye.  Far less.   Let’s look at some problems with this figure:

First, the sniff test.  If you had a community of 3500 wiki contributors, would after 2 years your wiki have only 2160 content pages?  Is this the output one would expect from a community that size?  Less than one half-page per contributor per year, from this “larger number of active volunteers” ?  This disconnect between claims and reality should be enough to warrant a closer examination.  This just doesn’t sound credible.

Fortunately the wiki stats allow us to see exactly how many edits each registered user made to the wiki.  Curiously, of this “community of over 3,000 volunteers from the five continent”, 1777 of them (over half) have made zero edits.  Zero, zip, gar nichts, nada, niente, zilch.  This is quite remarkable.  A community of contributors where half have made no contributions?!  Is that what you commonly think of when you read the phrase “active contributor” in a press release?  Evidently, in LibreLand you do.

Further, there are many users with a single edit, accounts like Cashloans121, Fastloans1, Fastloans2, etc.  Interesting names, yes?  Of course, these are the spam accounts, created so that advertising could be added to User or other pages.   It is also common for users to register and to make no other “contribution” than to put their C.V. on their User page.  I won’t embarrass the individuals users who have done this, but I see many examples of this on the LibreOffice wiki, where the only “contribution” from a user is self-promotion.   In total, 583 of the registered users made only a single edit in the past two years.   449 have made only 2 edits.  Sadly, our army of 3000 “active volunteers” is shrinking at a distressing rate.

Spam and other issues are well-known to organizations that use wikis.  You don’t claim that your community consists of all registered users of the wiki.  To make a claim like that in a press release is deceptive.  It is a statistic that has no relation to reality.   If the Apache OpenOffice project did exactly what LibreOffice did, and claimed its community size based on the number of registered wiki users, it could claim it had over 75 thousand contributors!

So what is one to do, with such messy data?  Certainly claiming 3,000 contributors without any caution about the above concerns is not recommended.   Instead, if the phrase “active volunteer” is more than empty syllables you need to apply some reasonable threshold to separate an active member of the community from spam, empty registrations or volunteers that showed up for one day and were never seen again.

One criterion is suggested by MediaWiki itself, in its stats report.  It shows that LibreOffice has 112 “active users”, which it defines as users who have made a contribution in the last month.  Another technique might be to look at users who have made, say some non-trivial contribution, say 10 page edits in the past two years, in which case you will show 343 active contributors.  Another way is to ask how many contributors combined account for 90% of all of the edits.  I prefer that metric, and the answer in this case is 342 active contributors.   But the only way you can claim  “a community of over 3,000 volunteers from the five continents” is to have a disregard for facts, and also a disrespect for your reader.  You burn credibility, one of the most important assets an open source project can have.

However you slice it, LibreOffice is overstating the size of their active contributor community by a factor of 10.

(This post represents my personal opinion only.  The standard disclaimer applies.)

(This post represents my personal opinion only.  The standard disclaimer applies.)

Part II is here and Part III is here.

The Claims

I’ve recently read some implausible claims from the LibreOffice project,  concerning their stats for downloads and users.  (These two different statistics are unfortunately conflated in their publicity campaigns, but more about that later).   Their claims fall apart if given any scrutiny and placed against comparable numbers from Apache OpenOffice.  I think you’ll agree by the time you are done reading my analysis .

If this were merely yet another case of puffery from the LibreOffice marketing department then I might just let it go, as I have with many other similar claims in the past couple of years.  But to the extent that some people seem  take these claims as facts, and are repeating, them, then I hope I will be forgiven for giving truth a chance to be heard.  I’ll  lay out the numbers as I know them and let you be the judge.

First, what do we have on the Apache OpenOffice side?  Most of our downloads are from our download site hosted by SourceForge.  The download stats are public and exposed by SourceForge via their REST API.    We gather these stats with a Python script (also public here) and that data is saved to a data file, which is then plotted on our website.  So everything is open and transparent here.  The downloads are counted by a respected 3rd party and the entire processing of these numbers is open for inspection.    It is all there, day-to-day, including breakdown by country and operating system.  We have nothing to hide.

The LibreOffice numbers, on the other hand, we only know from download claims in press releases, and then only at long intervals.  We have no idea what exactly they are counting.  They have never made the detailed stats public.  This does not mean that the numbers are incorrect of course.  It just means that no one outside of their project’s leadership is able to verify the claims.

Actual Numbers

But taken for what its worth, let’s look the recent LibreOffice claims and compare it to the actual data posted by Apache.

• On Sept 27th, LibreOffice claimed “Downloads since January 25, 2011, the date of the first stable release, have just exceeded 18 million”.
• On that same day, OpenOffice had accumulated 18,207,610 download via SourceForge.  (Per the posted data file, which you can verify against SourceForge’s Stats API if you wish)

So both projects are doing equally well, yes?

Well, no, not at all.  You need to take the time interval into consideration.  The LibreOffice counts were from January, 2011.  The OpenOffice counts were from May, 2012.  So in just a few months OpenOffice was downloaded as many times as LibreOffice was in its first two years.

If we convert to an average daily download rate we see:

• LibreOffice:   18,000,000/611 days = 29, 460 /day
• Apache OpenOffice:  18,207,610/143 days = 127, 326 /day

So the download rate has been 4x greater for Apache OpenOffice, and shows no sign of slowing.

A chart might make this clearer, showing the actually OpenOffice download figures (that is why the line is a little wavy) and the claimed LibreOffice trend.  (Anyone want to guess on what this chart will look like six months from now?)

Downloads versus Users

It is important when looking at download numbers that one does not equate download counts with user counts.  This is especially true when you are dealing with upgrade cycles.  As you probably know, neither OpenOffice nor LibreOffice have an incremental update facility.  If you want to update, say from Apache OpenOffice 3.4.0 to 3.4.1 then you need to download a complete copy of 3.4.1 and install it over your 3.4.0.

This complicates things.  Upgrades tend to inflate the download counts, since an upgraded user is counted twice: once for their original download and a second time for their upgrade.    This makes estimating the number of users from the number of downloads tricky.  So to be fair, when estimating the number of Apache OpenOffice users we must not neglect the impact of having a minor maintenance release on causing two downloads for users who upgraded.

But if this is an impact on OpenOffice, which had only two releases to reach 18,207,610 downloads, then how much greater must be the impact for LibreOffice?  For example, their 3.5.x series had 7 releases to fix critical bugs.  Their download counts included downloads from 3.3.x,  3.4.x  and 3.6.x series as well, each one with its own set of bug fix releases.  One must assume, due to the long duration of this reporting interval (nearly two years) and the instability of the early releases within each series, that LibreOffice users have upgraded numerous times each, causing numerous duplicate download counts, and leading the aggregate download count to reflect several times the number of actual users.

In other words, having a rapid release cycle with no incremental update facility will juice your download numbers since each real user will end up downloading many copies of your product.  Since LibreOffice had a dozen or more releases, and OpenOffice only two, it is logical to conclude that the LibreOffice user numbers are far less than suggested by their download numbers, perhaps lower by a factor of 4 or 5.

Objection: External Sites

I anticipate several objections against the above analysis, so let’s treat those one by one.

First, one might note that LibreOffice has claimed an additional two million downloads from “external sites offering the same package”.  Since these claims are not backed with names or numbers, I cannot say much other than the fact that OpenOffice is downloaded from external websites as well.  But we don’t count those in our main download counts.   But suppose we wanted to, and wanted to do apples-to- apples comparison with LibreOffice, with numbers from a 3rd party neutral source?

Let’s take Download.com, CNET’s software download repository, one of the most popular download sites around, as an example.  Here are the download numbers for the 3-month period from 7/28/12 through 10/28/12:

• OpenOffice: 328,846 downloads
• LibreOffice:  18,008 downloads

In this case the OpenOffice download numbers are greater by a factor of 18x.

So I don’t think the external download sites changes things much.  The numbers are small overall, but per day the OpenOffice numbers are far higher than LibreOffice’s.

Objection: Linux users

On top of the 20 million users LibreOffice claims on Windows and Mac, they also stick a finger in the air and decide they have 30 million Linux users as well.  This leads to extravagant claims like, “As of today, LibreOffice is being used by close to 60 million people”.  They don’t detail how they arrive at this number, but it appears to be the culmination of a series of implausible assumptions:

• Take the highest of the several estimates for the number of Linux desktops
• Assume that everyone is using their Linux desktop for  document editing
• Conveniently ignore AbiWord, KOffice, Gnumeric, Calligra, Google Docs or even MS Office under Wine, and assume that everyone on Linux uses LibreOffice.
• Ignore the many Linux users who are displeased with LibreOffice and who have uninstalled it and replaced it with OpenOffice instead.

They make these assumptions and then claim another 30 million LibreOffice users on top of their inflated claim of 20 million Windows/Mac users.

But this really misses the point.  The trajectory is what matters.  In a long race you bet on the faster horse, not the one who has a small head start.  You can have 100% of the 3% Linux desktop market and even under the rosiest assumptions that is only 3%.    And that number is decreasing, as desktop users move to tablets, where Android is the player and the  Apache License is preferred for userspace code.  And I doubt Google will prefer LibreOffice in this space over their own recent QuickOffice acquisition, which already has an app supporting Android (and iOS).

Another point is that one should not equate users who intentionally download and install a product  with users who have it automatically installed as part of an OS, without their knowledge.  These are not the same thing, and to treat them as such is to confuse a downhill skier with someone who fell down a snowy hill.  The one does something intentionally;  the other has something done to them.

That is not to say that Linux users are not important.  We certainly treat Linux as a first-class platform within the project.  I’d like to see us do the packaging work necessary to make Apache OpenOffice available to users on Linux, via their distros.   Users should have choice, even on Linux.  If you’re interested in helping with this,  send me an email.

Objection: All numbers are incomparable

Another objection is to say that all projects live in a different context, with a different user base and that the numbers can never be compared against each other in a fair way.  All is relative, subjective, and LibreOffice is justified in making any claim it feels like, since it is its own reference and base of comparison.

There are several counters to this objection.  First, when LibreOffice publishes numbers, in press releases and blog posts, it has an obligation not to deceive its readers.  This is basic professional ethics.   When you claim a certain number of users, there should be some solid basis for making that claim, not merely the absence of contradictory information.  In any case, I’ve provided adequate contradictory data in this post.

Another counter is to point out that some comparisons are closest to an apples-to-apples comparison.  For example, the number of Windows downloads directly from a project’s website.  OOr downloads from a neutral 3rd party website like CNet’s Download.com.   Of course we can debate the fine details and nuances to the right of the decimal place.  But that does not provide an excuse for conflating download numbers with user numbers in a  press release.  You may not know everything, but you should know that this is not right.

Summary

Apache OpenOffice makes available detailed download statistics in near-realtime for inspection.   LibreOffice makes download claims in press releases at wide intervals with no supporting data.

If you do an apples-to-apples comparison, of Windows and Mac users, which together constitute 97% of the desktop market,  Apache OpenOffice, although it took a while to make its first release,  3.4.0, has taken off like a rocket, and has eliminated any head-start advantage LibreOffice had, and is racing ahead with 4x the downloads that LibreOffice is reporting. And since the LibreOffice numbers are inflated by duplicate counting of upgrade downloads, OpenOffice is probably already ahead of LibreOffice in users on these platforms by a factor of 10 or more.

Under a series of implausible assumptions, LibreOffice claims an additional 30 million users on Linux.  The actual number is unknown, but likely far less.  But since Linux desktops are only 3% of the desktop market, and that market is shrinking, this is not a realistic growth opportunity for LibreOffice.

(This post represents my personal opinion only.  The standard disclaimer applies.)