Standards

How to Write a Standard (If You Must)

2006/12/06 By Rob 6 Comments

Standards are generally a bad idea. They reduce freedom-of-action and limit choice. But sometimes you must have one in order to pacify an anti-business regulator or other socialist-leaning bureaucrat. So what should you do if you find you find yourself in the awkward position of coming up short in the standards department? By all means, create a standard and do it quickly! I offer here some observations on best-practices, and tried-and-true advice on how to make a standard quickly, with little pain or risk.

First some background. Standards writing, as generally practiced, is a multilateral, deliberative process where multiple points of view are considered and discussed, where consensus is reached and then documented. This must be avoided at all costs. The delays introduced by such a consensus process are considerable and the outcome of such a process does not justify that time investment. If you already have a monopoly, why waste time on a consensus? Consider the notable failures of XHTML, XForms, SVG, XSLT, etc. Look at the multiple implementations of these standards, including viral copyleft and IP-deficient products. Do you really want to see this trend continued?

Start with a complete product implementation. This makes the entire process much faster since there is no time wasted discussing such abstract, heady things as interoperability, reuse, generality, elegance, etc. Only Perpetual Adoration of the One True Implementation (the one you already have) will quickly lead to a specification. Avoid consideration of alternatives. Consideration of alternatives is your prime risk factor for introducing delay.

If possible choose an implementation that has layers of complexity from years of undisciplined agglomeration of features. Of course this will lead to a specification of Byzantine complexity and epic length. But since no one will actually read the specification, there is no harm. In fact the length and complexity can bring several benefits:

Any criticism of the specification can automatically be dismissed as nitpicking. For example, if you are presented with a list of 500 faults in a 6,000 pages specification, you can respond, “That is less than 10%. You are just nitpicking. We can fix that in release 1.1”. Or you can even just rely on the familiar justification, “Shipping is a feature”. Any finite list of defects can be made minuscule by a sufficiently large specification.
Further, since review periods at ISO and most other standards bodies are of fixed length, regardless of the length of the specification, a sufficiently large specification will ensure that it receives no, or only cursory review. Divide the length of the specification in pages, by the length of the review period in days. A Freshman English major might be assigned reading of 50 pages per day. Aim to double or triple this reading load. 100+ pages/day is a good rule of thumb to ensure that a volunteer on a standards committee will not be able conduct a thorough review.
The pure size of the specification will imply to some that it is a well-considered, comprehensive and cohesive document. Just like the IRS tax regulations and the federal budget.
In case of emergency the specification can be burned as fuel

Shop around for the best standards development organization (SDO), one that knows how to get the job done quickly. Evaluation criteria include:

A proven ability to approve standards quickly. You are not interested in advancing the state of the art here. You want fast-in-fast-out-stamp-my-ticket processing so you can get on with your business.
A membership model that effectively exclude individual experts and open source projects from the process.
A demonstrated competency in maintaining needed secrecy when developing sensitive standards.
The right to make FastTrack submissions to ISO

Ecma International approved the DVD-RAM, DVD-R, DVD+R, DVD-RW and DVD+RW standards. Although some falsely claim that these overlapping standards have confused consumers, it is clear that having these multiple formats has given manufacturers ample opportunity for upselling multi-format DVD players and burners. With a single format, where is the upsell? Ecma clearly understands the purpose of standards and can be relied upon.

Once you are in an SDO and are ready to create your Technical Committee, be sure to carefully consider the topics of membership and charter. Of course, you’ll want to assemble a team of willing partners. Loyalty can be obtained in many ways. Your consigliari may have some ideas.

You charter is your first line of defense. Since your Technical Committee may contain some technical people, you will want to strictly limit what they discuss. Technical people are dangerous if they are given too much freedom. Who knows what they may do if left on their own? They might even innovate (shudder), and innovation is so disruptive. A well-written Charter can prevent innovation, that unwelcome guest, from ever knocking on your door.

Since its terms restricts the scope of your work, you should ensure that your charter contains any restrictions that you do not want to discuss or defend in the future. Best to get these front loaded into the charter so you can just say, “We have no choice; that is our Charter”. Your goal is to describe the One True Implementation, so a good charter restriction to add is one which will focus the technical committee on that one single task. A roundabout way of doing this is to require that the produced specification must remain 100% compatible with a different specification, one which is your secret. That way you, and only you, can decide whether or not a proposed change is within scope of the charter. This provides a lot of flexibility and avoids unnecessary discussions. “We checked the secret specification and it says that October has 40 days in it. Sorry guys, there is really nothing we can do. The Charter won’t let us.”

A few additional recommendations for the the day-to-day work of describing the One True Implementation:

Observe other successful standards and the process that lead to them. Look at the size of their specifications and how long it took to develop that. Assume that you can safely progress at a rate 10-20x faster. This pace is justified by the superiority of your One True Implementation and your lack of deliberations, discussions or considerations of alternatives.
At all costs avoid reusing any existing standards. Reuse of standards will only lead to generality, interoperability and increased reuse and risk getting you involved in discussions with other standards bodies. This delay must be avoided. The One True Implementation has everything you or anyone else needs to know. It is the true fons et origo of all wisdom and knowledge.
This also implies that you do not engage other standards groups. Assume that your hand-picked technical committee is an expert on everything. In any case, expertise is irrelevant, since you are merely describing the One True Implementation. All the decision making essentially already occurred years ago. Your task is just writing it down.
Secrecy is paramount. If the unwashed masses find out what you are discussing and what issues you face, they might do things like offer you suggestions or alternatives. That is so annoying. So all meetings, minutes, mailing lists, etc., should all be strictly private.

That’s about it. Eveything else is just common sense. Think Speed. Think Heft. Focus on the One True Implementation. I believe the liberal application of the above principles should enable anyone to quickly and painlessly create an International Standard.

When language goes on holiday

2006/10/15 By Rob 4 Comments

This apt phrase is from Wittgenstein, Philosophical Investigations, section 38, “Philosophical problems arise when language goes on holiday”. One cannot be sloppy in language without at the same time being sloppy in thought.

Of course, this thought is not new. In Analects 13:3, Confucius is given a hypothetical question by a disciple: “If the ruler of Wei put the administration of his state in your hands, what would you do first?”. Confucius replied, “There must be a Rectification of Names,” explaining:

If language is not correct, then what is said is not what is meant; if what is said is not what is meant, then what must be done remains undone; if this remains undone, morals and art will deteriorate; if justice goes astray, the people will stand about in helpless confusion. Hence there must be no arbitrariness in what is said. This matters above everything.

In that spirit, let us talk of “choice”, a word loaded with meaning. Choice is good, right? Who would voluntarily give up their god-given right to choose for himself? Reducing choice is immoral. A central role of government is to ensure that we can choose freely. For a market to thrive it must be free of every regulation that reduces our ability to choose. These are all self-evident truths.

Or are they?

Let me set you a problem. I place before you a glass of water. Whether it is half full or half empty I leave to your imagination. What use is this glass of water to you? Certainly you can drink it. Or you could sell it to someone else. Or you could create a derivative option to buy the water, and sell this option to someone else. Or you could pledge the water as collateral for some other purchase. You have several options, several choices. But suppose you are thirsty. Then what do you do with this nice, cold glass of water? If you drink it, then you can no longer sell it, sell options on it, or pledge it. Drinking the water eliminates choice. So better not to drink it. Just let it sit there, on the table. But still you get thirstier and thirstier.

What a cruel dilemma I’ve given you! You cannot drink without reducing your future options, without eliminating choice. Of course, the water slowing gets warmer and evaporates. Even not choosing is itself a choice.

The Moving Finger writes; and, having writ,
Moves on: nor all your Piety nor Wit
Shall lure it back to cancel half a Line,
Nor all your Tears wash out a Word of it.
— Omar Khayyam

How are we to make sense of this paradox? The fact is that every decision, ever choice you make, commits you and eliminates some other choices. We choose because without choosing we cannot claim the value in a single path among alternatives. If you want to quench your thirst then you must drink the water. It is that simple.

So I’ve found it amusing to see how Microsoft and their supporters constantly attack open source and open standards on the grounds that they reduce choice. For example, Microsoft’s lobbying arm, with the Orwellian doublespeak name “The Freedom to Innovate Network” lists this among its policy talking points:

[G]overnments should not freeze innovation by mandating use of specific technology standards

This talking point is picked up and repeated. Open Malaysia picks on a local news article which quoted a Microsoft director speaking on Malaysia’s move toward favoring Free and Open Source Software (FOSS) in government procurements:

My opinion is that it [the policy] limits choice as the country has a software procurement preference policy

The Initiative For Software Choice is the latest face on the hundred-headed hydra spreading FUD around the world. However they have recently had the embarrassment of seeing an example of their handiwork leaked to the press which is worth a read in full.

This in itself is neither new nor news, but it just recently occurred to me that this is all just an abuse of language, with no substance behind it. When one adopts a technology standard one does it with some desired outcome in mind. One chooses this path in order to receive that benefit. Adopting a standard is like drinking a glass of water. You doing it because you are thirsty.

A recent Danish report (the “Rambøll Report”) looked at the significant cost savings of moving the Danish government to OpenOffice/ODF compared to using MS Office with OOXML. Is it wrong to choose a less expensive alternative? Or is it better not to choose at all, and forgo the cost savings?

I think we need to all ask ourselves what we thirst for. Are you suffering from vendor lock-in? Are your documents tied to a single platform and vendor? Are you overpaying for software of which you use only a fraction of the functionality? Are you unable to move to a more robust desktop platform because your application vendor has tied its applications to a single platform? If you are thirsty, I have one word of advice: “Drink”.

A Leap Back

2006/10/12 By Rob

1/23/2007 — A translation of this post, in Spanish has been provided by a reader. You can find it in the Los Trylobytes blog.

I’ve also taken this opportunity to update page and section references to refer to the final approved version of the Ecma Office Open XML specification, as well as providing a link to the final specification.

Early civilizations tried to rationalize the motions of the heavenly bodies. The sun rises and sets and they called that length of time a “day”. The moon changes phases and they called a complete cycle a “month”. And the sun moves through the signs of the zodiac and they called that a “year”. Unfortunately, these various lengths of time are not nice integral multiples of each other. A lunar month is not exactly 30 days. A solar year is not exactly 12 lunar months.

To work around these problems, civil calendars were introduced — some of the world’s first international standards — to provide a common understanding of date reckoning, without which commerce, justice and science would remain stunted.

In 45 B.C., Julius Caesar directed that an extra day be added to February every four years. (Interestingly, this extra day was not a February 29th as we have today in leap years, but by making February 24th last for two days.) This Julian System was in use for a long time, though even it has slight errors. By having a leap year every four years, we had 100 leap years every 400 years. However, to keep the seasons aligned properly with church feasts, etc., (who wants to celebrate Easter in Winter?) it was necessary to have only 97 leap years every 400 years.

So, in 1582 Pope Gregory XIII promulgated a new way of calculating leap years, saying that years divisible by 100 would be leap years only if they were also evenly divisible by 400. So, the year 1600 and 2000 were leap years, but 1700, 1800 and 1900 were not leap years. This Gregorian calendar was initial adopted by Catholic nations like Spain, Italy, France, etc. Protestant nations pretty much had adopted it by 1752, and Orthodox countries later, Russia after their 1918 revolution and Greece in 1923.

So, for most of the world, the Gregorian calendar has been the law for 250-425 years. That’s a well-established standard by anyone’s definition. Who would possibly ignore it or get it wrong at this point?

If you guessed “Microsoft”, you may advance to the head of the class.

Datetimes in Excel are represented as date serial numbers, where dates are counted from an origin, sometimes called an epoch, of January 1st, 1900. The problem is that from the earliest implementations Excel got it wrong. It thinks that 1900 was a leap year, when clearly it isn’t, under Gregorian rules since it is not divisible by 400. This error causes functions like the WEEKDAY() spreadsheet function to return incorrect values in some cases. See the Microsoft support article on this issue.

Now I have no problems with that bug remaining in Excel for backwards compatibility reasons. That’s an issue between Microsoft and their customers and not my concern. However, I am quite distressed to see this bug promoted into a requirement in the Ecma Office Open XML (OOXML) specification. From Section 3.17.41 of SpreadsheetML Reference Material, page 3305 of the OOXML specification (warning 49MB PDF download!) , “Date Representation”:

For legacy reasons, an implementation using the 1900 date base system shall treat 1900 as though it was a leap year. [Note: That is, serial value 59 corresponds to February 28, and serial value 61 corresponds to March 1, the next day, allowing the (nonexistent) date February 29 to have the serial value 60. end note] A consequence of this is that for dates between January 1 and February 28, WEEKDAY shall return a value for the day immediately prior to the correct day, so that the (nonexistent) date February 29 has a day-of-the-week that immediately follows that of February 28, and immediately precedes that of March 1.

So the new OOXML standard now contradicts 400 years of civil calendar practice, encodes nonexistent dates and returns the incorrect value for WEEKDAY()? And this is the mandated normative behavior? Is this some sort of joke?

The “legacy reasons” argument is entirely bogus. Microsoft could have easily have defined the XML format to require correct dates and managed the compatibility issues when loading/saving files in Excel. A file format is not required to be identical to an application’s internal representation.

Here is how I would have done it. Define the OOXML specification to encode dates using serial numbers that respect the Gregorian leap year calculations used by 100% of the nations on the planet. Then, if Microsoft desires to maintain this bug in their product, then have Excel add 1 to every date serial number of 60 or greater when loading, and subtract 1 from every such date when saving an OOXML file. This is not rocket science. In any case, don’t mandate the bug for every other processor of OOXML. And certainly don’t require that every person who wants the correct day of the week in 1900 to perform an extra calculation.

Sure this requires extra code to be added to Excel. Excel has a bug. Of course it will require code to fix a bug. Deal with it. I think the alternative of forcing the rest of the world to a adopt a new calendar system is the ultimate in chutzpah. The burden of a bug should fall on the product that has the bug, not with everyone else in the world.

Further, I’d note that section 3.2.28 (page 2693) defines a workbookPr (Workbook Properties) element with several attributes including the following flag:

date1904 (Date 1904)

Specifies a boolean value that indicates whether the date systems used in the workbook starts in 1904.

A value of on, 1, or true indicates the date system starts in 1904.
A value of off, 0, or false indicates the workbook uses the 1900 date system, where 1/1/1900 is the first day in the system.

The default value for this attribute is false.

What is so special about 1904 you might ask? This is another legacy problem with Excel, that implementations of Excel on the Mac, for reasons unknown to me, had an internal date origin of January 1st, 1904 rather than January 1st, 1900. This is unfortunate for Microsoft’s Mac Business Unit, and has likely been a source of frustration for them, needing to maintain these two date origins in their internal code.

But why is this my problem? Why should a standard XML format care about what Excel does on the Mac? Why should it care about any vendor’s quirks? If RobOffice (a fictional example) wants to internally use a date origin of March 15th, 1903 then that is my business. In my implementation I can do whatever I want. But when it comes to writing a file format standard, then the caprices of my implementation should not become a requirement for all other users of the file format. Further, if I cannot make up my mind and choose a single date origin then my indecisions should not cause other implementations to require extra code because of my indecision.

So there you have it, two ways in which Microsoft has created a needlessly complicated file format, and made your life more difficult if you are trying to work with this format, all to the exclusive advantage of their implementation. I wish I could assure you that this is an isolated example of this approach in OOXML But sadly, it is the rule, not the exception.

Nothing is certain but death and …

2006/09/29 By Rob 2 Comments

October is almost here. The last quarter. This is the time of year I start thinking of next April 15th and what I can do this year to reduce my taxes via by recognizing some capital losses, making charitable contributions, etc. It also makes me think about taxes in general and how much the tax filing process has changed over the years.

I’ve been a taxpayer in Massachusetts since 1988 or so. At first I filled via paper returns. This invariably involved started with a trip to the Post Office or Library to pick through the stacks of forms to find the ones I needed, and then to make a return trip the next day when I found out that I missed a schedule. I would typically get two copies of everything: one to do the draft work in, and one to file. Quantum Electrodynamics, the music of Arnold Schoenberg, James Joyce’s Ulysses — these were all easy and made sense compared to doing taxes.

Sure, I could have just carted my paperwork off to an accountant and let him deal with the mess. But I’m opposed to that in principle. Taxes are paid by everyone and everyone should be able to do their taxes. Something is wrong with the tax system, democracy, or both if a Harvard grad is not able to figure out his own taxes. But sometimes it was a struggle.

Eventually, in the mid 1990’s (my recollection) the Massachusetts Department of Revenue (DOR) started to support online filing. Initially this was via a proprietary, state-issued, Windows-only client that worked with a dial-up modem connection. It was advanced for the time, but still bare bones, and clunky. I actually still did most of the form by hand, and entered in the numbers into the UI for the final calculations and submission.

Things got interesting a few years ago when the DOR stopped issuing their own client. Instead they promulgated standards for on-line tax submission, issued developer guidelines, a compliance test suite and started certifying vendors who wished to write compatible tax filing software. You can see the software guidelines here.

Now, there are some that say that government mandated standards are evil, that they stifle innovation, remove choice, hurt the consumer, etc.

[G]overnments should not freeze innovation by mandating use of specific technology standards
— Policy statement from Microsoft’s “Freedom to Innovate” network

So it is interesting to see what happened in Massachusetts after they issued standards in this area.

Today, Massachusetts residents have a choice of 11 applications for filing their state income taxes, at all price points. These range from free filing for qualified low-income residents, to low-cost web-based applications for simple returns, to rich client-side solutions for more complicated filings. The full list of compliant filing software is here.

So, in this case, the state-mandated standard did not lead to lack of competition or innovation, but rather led to a thriving market of filing solutions, at a variety of price points and feature sets. The tax-paying public benefited with increased choice of products and feature sets. The DOR benefited with returns that could be processed at lower cost and lower error rates. And software vendors benefited by the creation of a new market in tax preparation software.

This lesson is something to bring to mind the next time you hear the contrary FUD about standards and innovation.

Follow the Leader

2006/08/03 By Rob 13 Comments

David Wheeler, the chair of the OASIS ODF Formula Subcommittee has a good status update on our work defining the details of the expression language and supporting functions used in spreadsheet formulas. I’d also like to point out some cool work by Daniel Carrera, who put together some code that post-processes the OpenFormula specification (in ODF format, of course), extracts the details of the embedded test cases, then automatically generates an ODF spreadsheet file which executes the spreadsheet functions and verifies correct results. This resulting spreadsheet allows an implementation to automatically test their compliance to the spec. This gives us a self-testing specification, a great labor savings, as well as a demonstration of the innovative things you can do with ODF. Details are here.

(I note in passing that although the OASIS ODF TC does all of its working documents in ODF format, the Ecma TC45 does none of its working documents in OOXML. They continue to use the old proprietary Microsoft binary formats as their working format on the TC. A suggestion — If they are unable for some reason to use OOXML, then I encourage TC45 to use ISO ODF. They can then download Daniel’s code to help generate test cases from their spreadsheet formula documentation and this, I promise you, will save implementors a lot of time.)

Of course, malcontents will never be pleased by our progress, and will portray this progress as proof that we are yet imperfect, and therefore not useful. The first point is obvious, but the second is dubious.

Stephen McGibbon’s blog entry of a couple weeks ago seems to be the Urquelle of this particular line of reasoning. Here’s one small quote:

I mentioned that in my opinion, Sun were completely aware that ODF wasn’t sufficiently defined to support spreadsheet interoperability as long ago as February 2005 and that the realpolitik inside OASIS was to take advantage of the EU IDA’s request to standardise by rushing to be first despite knowing the ODF specification was deficient in at least this area.

Read the rest of his article and you’ll walk away with two misconceptions:

The lack of a spreadsheet formula definition in a file format documentation is unusual, defective and prevents interoperability
Spreadsheet formulas were left out because ODF standardization was rushed, for political reasons

Let’s take a look at each of these in turn.

First, let’s look at the state of the art in spreadsheet file format documentation over the years, with particular attention to how spreadsheet formulas have been documented. As the following table shows, Excel formulas have never been publicly specified, even though Microsoft has been producing file format documentation for various binary, HTML, XHTML and XML Excel formats for over 9 years. It was only after the ODF TC decided to document our spreadsheet formulas and formed a Subcommittee to do so that Ecma TC45 decided to follow. The FUD followed soon after.

Date	Format version	Formula status
1997	Excel 97 Developers Kit (Microsoft Press, 1997)	not defined
ca 1998	MSDN CD’s in this era had Office file format documentation	not defined
Jan 1999	Office 2000’s XHTML formats for Excel	not defined
May 2001	Office XP’s XMLSS format for spreadsheets	not defined
Nov 2003	Office 2003’s XML Schemas	not defined
Dec 2005	Microsoft submits initial “base document” to Ecma	not defined
January 2006	Ecma TC45’s Working Draft 1.1	not defined
February 2006	The OASIS ODF Formula Subcommittee is formed to add formula definition to the ODF specification
April 2006	Ecma TC45’s Working Draft 1.2	not defined
May 2006	Ecma TC45’s Working Draft 1.3	Mirabile dictu! After 9 years of ignoring it, Microsoft finally decides to start defining their spreadsheet formula language.

So the statement that the lack of a formula language specification is unusual or makes interoperability impossible falls down in the face of 9 years of contrary evidence. Over the years, the industry has managed to have interoperable spreadsheet formulas between different versions of Office as well as between Excel and competing spreadsheets, including 1-2-3, Quattro Pro, OpenOffice, StarOffice, etc., all without ever having a formula specification.

Even though every other spreadsheet file format specification in the past decade failed to document a spreadsheet formula language, the ODF TC knew that we could and should do better. That is why we took the lead and formed a Subcommittee to define, in great detail, with test cases, how spreadsheet formulas, expressions and functions should be interpreted. This is not fixing a problem. This is advancing the state of the art in file format specifications.

They say that imitation is the sincerest form of flattery. If so the ODF community should be blushing with all of this flattery heaped on it. If it wasn’t for the continual market pressure that our innovations bring, Microsoft would never have 1) issued a patent covenant for OOXML, 2)brought OOXML before a standards body, 3) started to document their spreadsheet formula language or 4) started to create an ODF Add-in for Office.

So, then what about the statement that ODF was rushed through the standardization process?

Let’s look at the numbers. Both ODF and OOXML are derived from pre-existing formats . This is not necessarily a bad thing. This is one source of “implementation experience” and this is beneficial to any standard to have this. But only once the “base document” is submitted to a multi-vendor open standards development organization (SDO) does the true work of standardization begin, including deep technical review of the specification to confirm completeness, conciseness, lack of ambiguity, correct use of formal specification language, ensuring platform independence, encourage flexibility and extensibility, etc. So, I’ll start the clock when the base specification is submitted to the SDO, and stop the clock when the SDO approves the standard.

The ODF numbers are clear enough since the 1.0 version is complete. The OOXML numbers require some estimation, since they are not complete, but I’ll justify my estimates this way:

The OOXML Working Draft 1.3 is currently 4,081 pages long. At the SC34 meeting in June we were told by the Ecma Secretary General that more material was coming and that this draft was only 2/3 complete. By my calculations, this gives a final size estimate of around 6,000 pages.
Predicting the completion date is harder. But we do know that Ecma specifications can only be approved twice a year at Ecma General Assembly which are in June and December. If I were Microsoft I’d really really really want OOXML approved in time for the Office 2007 launch, so I’m predicting Ecma approval will be sought at the December Ecma General Assembly.

Of course I could be wrong on either or both of those estimates, but let’s see where the logic takes us. The following table summarizes the time under standardization as well as the rate of standardization (pages/day) for each specification.

Standard	Submitted to SDO	Standard issued	Days elapsed	Standard length	Rate of work
ODF	12 Dec 2002	1 May 2005	867	706 pages	0.8 pages/day
OOXML	15 Dec 2005	31 Dec 2006 (est)	381 (est)	6000 pages (est)	15.6 pages/day (est)

Now I ask you, who is rushing? ODF took 2 ½ years to standardize 700 pages. Microsoft is trying to standardize a 6,000 page behemoth in just 1 year. I think the argument that ODF was rushed through under political pressure just doesn’t stand up to even cursory examination. Honestly, I think this FUD is being spread around as a smoke screen to hide the fact that OOXML is the one that is really being rushed.