Archives for July 2006

Throwing stones at people in glass houses

2006/07/27 By Rob 9 Comments

I work in a house with glass walls. Not literally, of course. The cost to air-condition such a house would be prohibitive. I mean that working on standard in OASIS is a public act, with process transparency and public visibility. The public doesn’t see merely the end-product, or quarterly drafts, they can see (if they are so inclined) every discussion, every disagreement and every decision made by the TC, in near real-time. Our meeting minutes for our TC calls are posted for public inspection. Our mailing list archives, where most of the real work occurs, is there for the public to view. The comments submitted by the public are also available for anyone to read. This information is all archived from when the TC first met back in 2002, all the way to the discussions we’re having today on spreadsheet formula namespaces.

One result of this openness is that it is very easy, trivial even, for our critics to simply read our mailing list, look for a disagreement or discussion of an issue, and repeat our words, usually out of context. Cut & Paste. This is certainly the most efficient way to criticize ODF since it minimizes the amount of thinking required. However, this is a bit tedious, especially when this is applied so asymmetrically, as I shall now explain.

Ecma TC45, the committee producing Office Open XML (OOXML), does not operate in such a transparent manner. They do not have a public mailing list archive. They have not published their meeting minutes. The comments they receive from the public are not open for the public to read. The public has no idea what exactly the TC is working on, what issues they think are critical, whether the TC is in unanimous agreement, whether there is spirited debated or whether Microsoft dominates and determines everything. The fact that they have not yet sent OOXML for an Ecma vote is proof that believe the specification is not yet ready for standardization. But we know no details of what exactly is lacking, what problems are being fixed or, more importantly, what defects are being allowed to remain.

And in this way, the ODF-bashers take advantage of our openness, while holding their deliberations in obscurity. They throw rocks at our glass house while hiding in the shadows.

So this openness at OASIS has an apparent downside. But honestly, I wouldn’t trade it for any alternative. Making a standard, especially one this important, is a privilege, not a right. The public deserves to know how a standard is made, the same way and for the same reasons they deserve to know how legislation is made. I relish this scrutiny because I know it makes us stronger.

Sun’s Simon Phipps has posted his keynote from the recent OSCON conference. The topic was the “Zen of Free” and, among other goodies, Phipps lists 5 requirements for “full support for fully open standards”, of which I quote the 4th, since it states the point better than I have:

…the standard [Phipps here speaking generically and not about any specific standard] should have been created transparently. Just as an open source community looks with concern on a large, monolithic code contribution, so we should be wary of standards created without the opportunity for everyone to participate or, failing that, with a full explanation of every decision that was made in its construction. Without that there’s a chance that it’s designed to mesh with some facility or product that will be used to remove our freedom later.

Another way to attack openness is to do it with legal restrictions. For example, we’re seeing many references to a year-old performance evaluation of an atypical spreadsheet file, and using that to make the ridiculous claim that the ODF format itself is too slow. I’d love to dispute that claim and show it for what it is. I’d love to show that for most common document sizes, ODF documents are actually smaller and faster to load and save than OOXML documents. I’d love to show you all this, but I can’t. Why? Because Microsoft won’t let me. The only implementation of OOXML is the Office 2007 beta, and the End User License Agreement (EULA) has this language:

7. SCOPE OF LICENSE. …You may not disclose the results of any benchmark tests of the software to any third party without Microsoft’s prior written approval

So, our critics can quote benchmark results about ODF running in OpenOffice, but we can’t quote numbers about OOXML running in Office. They can read our mailing lists and quote us discussing ODF issues as we address them, but we cannot even see what they are working on.

What should we make of all this? I suggest that no specification is perfect. That’s why we have version numbers. The question you need to ask yourself is: what leads to a better specification, full and open public discussion and scrutiny? Or something rushed through behind closed doors? You know what the issues with ODF are, and you’ll continue to hear the same small list over and over again. But this is a shrinking list, as the ODF TC experts address these issues. But do you know what the issues with OOXML are, the reasons why Ecma TC45 has not yet put forward their specification as an Ecma standard? What do their experts say when speaking candidly about their specification? The public simply doesn’t not know. Do we assume silence means perfection? I don’t think so.

Add-in finitum

2006/07/26 By Rob 3 Comments

In this post, I will take another look at the Microsoft ODF Add-in debate, suggest some criteria for use in evaluating file format integration, and use those criteria to evaluate both Office 2007’s support for the ODF formats, and OpenOffice’s support for Microsoft’s formats.

My examination of the ODF Add-in for Word (here and here) has sparked a spirited discussion on both sides of the issue. A balanced view is given by Jason Brooks on eWeek.com:

It’s certainly going too far to call Microsoft’s currently lukewarm support for ODF a change of heart, but it is nonetheless an encouraging instance of Microsoft listening to its customers. We call on Microsoft to make way for ODF in the standard supported-file-formats list giving it at least the same stature as the formats of the suite’s once-fierce rivals.

In that spirit I’d like to propose some evaluation criteria for what first-class file format integration looks like, so you’ll recognize it when you see it. I’ll use Word and ODF word processorr documents (.odt file extensions) as an example, but the points apply equally to other formats and other editors:

Is the format support “in the box”, installed and configured when you do a default install?
When you open the default File/Open dialog (the one you get with the Control-O shortcut, the one we all know and have been using for 20 years), do ODF files show up? In other words, does the default file mask include *.odt, or can you at least easily select ODF from the drop-down list of file formats?
If you create a new document in Word, and then type Control-S to save the document, is ODF available on the list of file formats you can save to?
Can an end user or administrator (no coding required) make ODF be the default format for saving documents?
If you open an ODF document, and change it and then Control-S to save it, will this occur?Or will it save to some other format? Or will you get an error?
Is support integrated into the Windows shell via the registry, so you can double-click on an ODF file on the desktop, in a folder or as an email attachment and it will automatically launch in Word? If you are like me, 90% of the time you are not opening a document from within Word, but you are launching Word with a file in this manner

Is there anything here I’m missing?

Let’s compare, using the above criteria, how well Word’s DOC format is treated in OpenOffice 2.0.3, and how ODF is treated in Word 2007 beta 2:

Criterion	DOC Format in OpenOffice	ODF Format in Word
1. Format supported in default install	Yes.	No. Requires a download and install of separate, unsupported Add-in.
2. File Open integration	Yes.	No. ODF is not listed in the default File Open dialog and doing a Control-O will not show ODF documents. However, ODF import is available in a separate menu item elsewhere in the menu system.
3. Save new document integration	Yes.	No. In fact no ODF save ability exists in the current version of the Add-in. There is a place holder for the ODF save operation, though it is on its own menu, and would not be shown when doing a simple Control-S to save a new document.
4. Can be made the default format	Yes.	No. Although other non-Microsoft formats, such as “Plain Text” can be made the default format, ODF cannot.
5. Simple round-tripping	Yes.	No. When an ODF document is loaded, its name is automatically changed and it is made read-only. So loading sampler.odt results in Word having a read-only version of sampler_tmp.docx. Attempting a simple Control-S to save will give an error.
6. Shell integration	Yes.	No.

Once again, the open source community wins. Not only is the fidelity of OpenOffice’s support for MS Office formats higher, but they provide much closer integration of these competing formats.

I’m reading many explanations and excuses for why Office’s support for ODF is the way it is. Here is a selection to ponder:

Brian Jones, a Program Manager for MS Office, said of the Add-in support:

It’s directly exposed in the UI. We’re even going to make it really easy to initially discover the download. We already need to do this for XPS and PDF, so we’ll also do it for ODF. There will be a menu item directly on the file menu that takes to you a site where you can download different interoperability formats (like PDF, XPS, and now ODF).

I guess we differ how we define “directly exposed” and “really easy”. IMHO, the easiest way to make this easy to find would be to install it with Office and put it in the normal place in the menus.

Also, in Office 2007 Beta 2, which I have in front of me, PDF and XPS are in the normal File Save and File Save As menus, not treated separately. I understand that there is a dispute with Adobe and so you are deciding to move the PDF and the competing XPS options out of the main File Save dialog, but that just proves my point. This is clearly a policy decision not a technical restriction. The fact is that PDF and XPS both work fine where they are in the File Save menu in Office 2007 beta 2.

Brian Jones again:

We already have the PDF and XPS support for Office 2007 users that unfortunately had to be separated out of the product and instead offered as a free download.

I agree that this is unfortunate. Full integration should mean it is included in the product and supported in the usual places. I’ll take your world that PDF had to be taken out for non-technical reasons. But what prevents ODF from being integrated at that level?

And Jones on Interoperability by Design:

You see a lot of folks talk about interoperability, but often they just don’t mean the same thing. From our perspective it’s something we want to build directly into the products so that it just works.

That is a reasonable goal. I like it. By this definition, will we ever see a Microsoft-sponsored effort, open source or otherwise, to make ODF so it it is built “directly into the product so that it just works”? As pointed out above, a bunch of open source developers have already given this level of support to Office formats in OpenOffice. Isn’t it embarrassing to be unable to accomplish the same task in Office 2007?

Patrick Schmid is a Microsoft Certified Systems Engineer who has obvious deep knowledge of Office Add-ins and customizations. He writes:

It is possible to write a custom file import/export converter for Word. This would add the file type supported by the converter to the file type lists in the Open and Save As dialogs. Unfortunately, this is not an option for the ODF translator project.

I will accept Schmid’s technical appraisal of Microsoft’s ODF Add-in, since he obviously knows far more about these API’s than I do. However, this limitation appears to me to be a chosen limitation based on how this functionality was designed. Microsoft for years has provided import/export support for other file formats, often competing file formats, and they have always put them in the normal File Open/File Save dialogs. Why the sudden change? You can’t hide behind limitations caused by choices that you choose voluntarily when other less constrained choices were available and in fact more typical.

Further:

The Open & Save As dialogs cannot be customized: There simply is no way for an add-in to add another file format to the file type list in those dialogs. I checked this with the Office beta team just to be sure.

There is no way to do this? When the Open Document Foundation responded to the Commonwealth of Massachusetts’s RFI requesting info about the feasibility of an ODF Plugin, their response included screen shots of what appears to be the File Open/File Save dialogs in MS Office allowing the read and write of ODF files, side by side with the built-in formats like DOC. I hope this will not prove to be another embarrassment, where open source developers accomplish another task which Microsoft says is impossible.

Schmid further writes:

Why should MS waste resources on providing one particular feature for add-in developers that most Office add-in developers (all not affiliated with Microsoft) would put at the very bottom of their wish list?

Keep in mind that a single ODF Add-in could satisfy millions of customers. So we shouldn’t count the value by the number of add-ins which could be written, but by the number of end-users who would use the add-in.

Stay tuned. I don’t think we’ve heard the last of this issue. In particular I’d like to hear more about this Open Document Foundation Plugin, and the closer integration they illustrated in their RFI response.

VML and OOXML: Cum mortuis in lingua mortua

2006/07/24 By Rob 7 Comments

In this post, I will look at the history of Vector Markup Language (VML), how it lost out to the W3C’s SVG back in the 1990’s, but has come back from the dead, showing up in the draft Ecma Office Open XML (OOXML) specification. I offer some opinions on why this is a bad thing.

First, a bit of history. The field is vector graphics, the type of graphics composed of lines and shapes and background fills, the type of graphics that scales nicely to different sizes/resolutions, and different devices, as opposed to raster graphics which is a bunch of pixels such as a GIF or JPEG file. This is a gross oversimplification, but it will suffice.

Vector Markup Language (VML) was an XML vocabulary for vector graphics submitted to the W3C by Microsoft and others back in mid-1998. I will not comment on its quality or merits, but merely note that it was rejected by the W3C in favor of Scalable Vector Graphics (SVG) specification which became a W3C Recommendation (that’s what the W3C calls their standards) in 2001. Since then, SVG 1.0 was upgraded to SVG 1.1. in 2003 and several mobile profiles (SVG Tiny and SVG Basic) were created. SVG has native support in Firefox and Opera, with Plugins available for most other browsers. There is support on mobile phones and PDA’s. A search of Amazon.com shows 19 books dedicated to SVG. The SVGOpen Conference has been going for 5 years strong. This all adds up to SVG being an established, open standard, widely implemented with a thriving implementor/user community and signs of continued innovation. It is a standard with a past, a present and a future.

But what ever happened to VML? VML has been a dead-end, from a standards perspective, for 8 years now, an eternity in web time. I was not able to find any VML books on Amazon. I could not find any VML conferences (unless one counts the Virgina Municipal League’s get-together at Virgina Beach in October). However, there is some lingering VML support in Internet Explorer and Office. Developers still use VML to target those applications, but I wonder, is it done out of preference or out of necessity? Although it is the users who are portrayed as dinosaurs for not upgrading to Office 2003, doesn’t it seem like Microsoft Office and Internet Explorer are the ones in need of an upgrade? They should join the rest of the world and start using SVG rather dragging along a dead spec.

But it is worse than this. I wouldn’t have bothered writing this just to point out something you already know, that Internet Explorer slowly or never adopts relevant web standards. The thing I wish to bring to your attention is that VML, the same VML rejected in 1998, is now being proposed as part of the draft Ecma Office Open XML. Take a look for yourself (warning, 25 MB specification download!).

Section 8.6.2 of the spec (Ecma Office Open XML, Working Draft 1.3) says:

VML specifies the appearance and content of certain shapes in a document. This is used for shapes such as text boxes, as well as shapes which must be stored to maintain compatibility with earlier versions of consumer/producer applications.

How should one parse “earlier versions of consumer/producer applications”? Is this a circuitous way of saying “MS Office and Internet Explorer”?

Now take a look at Chapter 23, VML, pages 3571-3795 (PDF pages 3669-3893). We see here 224 pages of “VML Reference Material”, which appears to be a rehash of the 1999 VML Reference from MSDN, and in this form it hides itself in a 4,081-page OOXML specification, racing through Ecma and then straight into ISO. Is this right? Should a rejected standard from 1998, be fast-tracked to ISO over a successful, widely implemented alternative like SVG?

Why should you care? It is all about reuse.

If a standard reuses an already successful standard, it reuses the collective community wisdom that went into making that standard. It also reuses the considerable editorial effort in writing, editing and reviewing a technical specification. Reusing this effort lets the TC focus their time on on truly innovative aspects of their specification, and leads to a higher quality standard.
When you reuse a standard, you allow implementors to reuse the experience and knowledge they have already developed around that standard. Remember the 19 books dedicated to SVG? There is a lot of SVG knowledge out there. Why waste it?
Reusing an existing standard, especially a popular one like SVG, allows implementors to reuse the various code bases, both commercial and open source that support it. Why reinvent the wheel? Do you really want to rewrite a vector graphics engine? SVG has several good open source libraries including Apache Batik and librsvg.

Use SVG and you get reuse on three fronts. Stick with VML and the only thing that is reused is Microsoft’s legacy code. Using SVG is clearly the better choice.

I suggest that Ecma TC45 investigate this issue and consider moving off of VML and move to SVG, or at least demonstrate why it is impossible to do so. Why does the world need yet another XML vector graphics standard? If there is something missing in SVG (which I doubt) then why not work with the W3C to propose a enhancement for SVG rather than re-proposing the VML standard which was rejected back in 1998?

Again, I make no technical argument why SVG is or isn’t superior to VML. I merely note that SVG has been an adopted W3C standard for 5 years now and should have a presumption of suitability for the task, especially over a specification which were rejected 8 years ago.

Site Updates

2006/07/24 By Rob Leave a Comment

If I had know I was going to be Groklaw’ed, I would have spruced up the place first and prepared for company! Thanks to all for the overwhelming expression of interest in these topics.

This blog is generated by Blogger, and published via ftp to my site. This blog is still quite young and you will continue you see things moving around a bit as I find out what works well.

Recent changes:

I had misconfigured some Blogger settings, causing my feed URL to be incorrect. So, if you subscribed before 7/20 then you’ll want to re-subscribe.
I’m auto-converting Blogger’s default Atom 0.3 feed to Atom 1.0. Thanks to the Sultan of Syndication, Sam Ruby, for help with that.
I’ve added a brief capsule bio linked to from the sidebar. My contact info is there as well.
I’ve also added a blogroll of sites which I find indispensable for keeping up with news related to ODF. I hope you find them as informative as I do.

I plan on writing additional essays related to ISO ODF and the draft Ecma OOXML. Expect to see one or two a week. There are already several bloggers who cover the universe of ODF news/happenings. Rather than repeat their good work, I will try to deal with technical issues in depth, with an emphasis on clearing the air of some of the FUD out there. I’ll also try to keep it fun. If there are any topics you especially would like to hear, please let me know.

A Game of Zendo

2006/07/18 By Rob 11 Comments

It is the type of response that was crafted to end all debate and justify all sins: “Backward compatibility with billions of documents produced over decades”. Variations of this occur everywhere. Rather than cite them all, a simple Google query will bring up a representative sample.

Let’s take a deeper look at this argument.

There is a game called Zendo, where a player, called the “Master”, forms in his mind a secret rule which governs the selection and arrangement of objects (often small colored blocks). Arrangements which conform to the secret rule are said to have “Buddha nature”. The other players take turns selecting and arranging their own blocks to conform to what they think the secret rule is, to which the Master will acknowledge success or failure. The winner is the one who first guesses the secret rule, which might be something “an odd number of blocks, at least one of which must be red”.

Microsoft is playing Zendo with the OOXML specification. The Master has formed a secret rule. He calls it, “backwards compatibility with billions of office documents”. But since the file format documentation for the proprietary legacy binary formats has not been made public, the rule might as well just been called “Buddha nature”. It is just as opaque. We have no way of judging whether any specific requirement of OOXML is there to support backwards compatibility, or whether it is just there for the convenience of the Office development team. Or in fact whether it is there to raise barriers to non-Microsoft implementers. How could we know, since the solitary constraint on the creation of OOXML dependent on information that isn’t public? Does Ecma TC45 itself even have access to the binary format specifications? How are they able to properly judge what is done in the name of compatibility? Do we all just take Microsoft’s word for it?

The key point (in my opinion) is that legacy compatibility may be a constraining factor, but it need not be the sole determining factor. There are many, perhaps an infinite number of possible markups which would be compatible with the legacy formats, meaning the legacy documents can be unambiguously transformed into the new XML format. The constraint should be that they are mappable, not that they must be identical. Among the set of such possible XML formats, some will be elegant, some sloppy, some bloated, some sparse, some which will be easy for others to implement, some designed to minimize conversion work for just one vendor, etc. In other words, this can be done well, or it can be done poorly. The constraint of compatibility does not justify everything. Compatibility is one requirement, but it is not the only requirement.

An example may make things clear. Word has a feature called Art Page Borders. If you are like me, you’ve gone 15 years without seeing or using this feature. But it is there, under the Format/Borders and Shading menu, on the Page Border tab.

The markup needed to define these borders is covered in section 2.18.4 “ST_Border (Border Styles)” of the OOXML specification. Here we see descriptions and images of 200 hundred or so Art Page Borders. The images are heavily weighted to Western European, even Anglo-American celebratory icons, things like gingerbread men for Christmas, pumpkins for Halloween, or images of Cupid for St. Valentine’s day, or globes which are neatly centered on the United States. I think it is a legitimate concern that a document format with such obvious cultural biases is moving forward toward an international standard.

Further, I am concerned that the specification includes what can only be considered a clipart collection. What legal rights does the implementer have to reproduce this clipart? Keep in mind that Microsoft’s “Covenant Not to Sue” covers patents, not copyrights. I haven’t seen anything that would grant implementers of OOXML the rights to reproduce this clipart in their application. Is the specification hard-coded to use clipart which we cannot copy?

All of these problems (spec bloat, cultural bias, non-extensibility, copyright concerns) can be solved by one simple mechanism. Instead of having ST_Border be a fixed enumerated set of values, have it include only a small number of trivial values like the basic line styles, and have everything else (all of the Art Borders) be stored as a separate image file in the document archive.

So, if you load a Word XP document that uses the “candyCorn” Page Border, then when you write it out to OOXML, you would include a single frame of that art in the zip file and have the XML document reference that image for the border, tiling as necessary. This solution has several advantages:

It removes some bloat from the spec. No need to document 100’s of page border clip art
It lowers the barrier to implement. No one is required to implement 100’s of border styles. They are all generated on-the-fly based on images stored in the document.
Copyright concerns are eliminated.
Is an extensible approach. An implementation can include different or additional border styles according to their business and cultural requirements.
It is compatible with legacy documents. Any existing Word binary or XML document can unambiguously be mapped into this scheme

Of course, this approach would require some minimal code changes in Microsoft Word to support this extensible mechanism. But remaining backwards compatible with the Microsoft Word product was never a stated constraint on OOXML. No one ever said that the goal of Ecma OOXML was to reduce the cost for Microsoft to implement it. It is all about the legacy documents, right?

So there it is, one example to illustrate a point that can be repeated over and over again. Among the potential universe of compatible XML formats for Office are those which are flexible, easy to use, easy to implement, as well as those which simply perpetuate the status quo and vendor lock in.