This will be a multi-part post, mixing in a little economics, a little history and a little technology — an intellectual smörgåsbord — attempting to make the argument that a single document format is the inevitable and desired outcome.
In Part I we’ll take a survey of a number of different problem domains, some that resulted in a single standard, some that resulted in multiple standards.
In Part II we’ll try to explain the forces that tend to unify or divide standards and hopefully make sense of what we saw in Part I.
In Part III we’ll look at the document formats in particular, how we got to the present point, and how and why historically there has always been but a single document format.
In Part IV, if needed, we’ll tie it all together and show why there should be, and will be, only a single open digital document format.
Let’s get started!
Standards — in some domains there is a single standard, while in other domains there are multiple standards. What is the logic of this? What domains encourage, or even demand a single standard? And where do multiple standards coexist without problems?
Let’s take a look at some familiar examples and see if we can figure out how this works. We’ll start with some examples where a single standard dominates.
The story of the standard rail gauge is probably familiar to you. At first each rail company laid down their own tracks to their own specifications. In the United States there were different gauges used in the North (5′ 9″) and the South (5′). This was not a major issue so long as rail travel remained local or regional. However, as the reach of commerce increased, the pain of dealing with the “break of gauge” between adjacent gauge systems increased. Passengers and goods needed to be offloaded and transferred to a different train, causing time delays and inefficient utilization of equipment. The decision was made to adopt a Standard Gauge of 5′ 9″ and an ambitious migration project took place on May 31st, 1886, when thousands of workers in the South adjusted the west track and moved it 3″ to the east, lining up with the Northern gauge. Eleven-thousand miles of tracks were converted in thirty-six hours.
It should be noted that this unification was not universally celebrated. In particular, riots occurred at some of the junction points, like Erie, Pennsylvania, where local workers stood to lose the high-paying jobs they had unloading and loading cargo onto new trains. Efficiency is often opposed by those who profited from inefficiency.
Another standard prompted by the railroad was the adoption of standard time. In earlier days each town and city had its own local time, roughly based on solar mean time. When it was noon in Chicago, it was 12:09 in Cincinnati, and 11:50 in St. Louis. The instant of local noon would be communicated to residents by a cannon shot or by dropping a ball from a tower, allowing all to synchronize their clocks. The ball drop could be observed by ships in the harbor by telescope and so was much more accurate than the cannon, since the signal was not delayed by the non-negligible travel time of sound. Some memory of this tradition continues to this day with the New Year’s Eve ball drop in Times Square.
When it took days by coach to travel from Chicago to Cincinnati, it did not matter that your watch was 9-minutes slow. Your watch probably wasn’t accurate enough to tell the difference in any case. When noon came in Cincinnati you would synchronize your watch, knowing that some of the correction was caused by the change in longitude, and some was caused by the imperfections in the watch. But the average person did not care because they did not travel all that much.
However, with the coming of the railroad and then the telegraph, everything changed. People, goods and information could be transferred at far greater speeds. The difference of 9 minutes was now significant.
Initially, each rail company defined its own time, based on the local time of its main office. Timetables would be printed up based on this time. So a large train station, which may serve six different lines, would display six different clocks, all set to different times, some 12 minutes ahead, some 15 minutes behind, etc. At one point, trains in Wisconsin were operating on 38 different times! This was not only an inconvenience to travelers, it was also increasingly a safety concern, since the use of different time systems at the same station increased the chance of collisions.
This was addressed by the adoption of Standard Time in the United Stated on November 18th, 1883, the so-called “Day of Two Noons” . This was the day that the Eastern, Central, Mountain, and Pacific time zones took effect, and on this day every town adjusted its local time to the Standard Time of their new time zone. If you were in the eastern-half of your time zone, then when local noon came you would set your clocks back a specified number of minutes, and would thus observe noon twice. If you were on the western-half of your time zone, you would advance your clocks at local noon a specified number of minutes. The contemporary coverage of this event in The New York Times is worth a read.
Over the years, the every increasing rate of commerce and information flow has lead to greater and greater precision in time-keeping, so that today with atomic clocks and UTC we can now account for the slowing of the Earth’s rotation and the insertion of occasional leap seconds.
The International Civil Aviation Organization (ICAO) is a UN agency that maintains various aeronautical standards, such as airport codes, aircraft codes, etc. They are also responsible for making English the required language for air-to-ground communications. So when an Italian plane, with an Italian crew on an Italian domestic flight contacts the approach tower at an Italian airport, manned by Italian personnel, they will contact the tower in English. Why do you think this is so?
The diameter of beverage cans has but little variation. A can of Coca-Cola and a can of Pepsi will both fit in my car’s cup holder. They also fit fine in the cup holders in my beach chair or rider lawnmower. This works with beer cans as well, with innovative holders such as the novelty beer hat . Vending machines seem to take advantage of this standard as well, since it simplifies their design. The whole beverage can ecosystem works because of standards around beverage can sizes. How is this standard maintained? Was it planned this way?
It is interesting to note that, from the beverage company’s perspective this is non-optimal. A can has minimum surface area for a given volume when it has equal height and diameter. But we never see beverage cans of that shape. Why not?
In the United States, our television signals are encoded in the NTSC system. PAL is used in most of Western Europe and Asia, and SECAM is used in France and Eastern Europe. The United States is moving to a new standard, High Definition, HDTV, by February 17th, 2009. This is the law, as enacted by Congress, that we must move to a new television standard, causing expenses to broadcasters and consumers, as well as generating a lot of revenue for electronic manufacturers. Why did this require a law? If it was good for consumers and for manufacturers, wouldn’t the free market make this move on its own?
The Great Baltimore Fire of 1904 quickly grew beyond the control of local fire companies. As the fire spread to encompass the entire central business district, the unprecedented call went out by telegraph for assistance from fire companies from Washington, DC and Annapolis and as far away as Philadelphia, Atlantic City and New York. But when these companies arrived, with their own equipment, they found that their hose couplings were incompatible. This was a large contributing factor to these fire’s duration and destructive power. Over 1,500 buildings were destroyed over 30 hours. Within a year there was a national standard for fire hoses.
To these can be added the hundreds of standardized items that we work with every day, such as standardized electrical connectors, light bulbs, food nutritional labels, gasoline nozzles, network addresses, batteries, staples, toilet paper holders, telephones networks, remote control infrared signals, envelopes, paper sizes and weights, currency, plumbing fixtures, light switch face plates, radio frequencies and modulations, screws, nails and other fasteners, etc.
Now let’s switch to some examples of domains where multiple standards have flourished.
The textbook example is the safety razor. When the safety razor was invented by Gillette, they were interchangeable, disposable blades made of carbon steel. As such they rusted and needed to be frequently replaced. Wilkinson Sword, later owner of the Schick brand, started making compatible stainless steel blades, which Gillette then copied. So there was a good amount of competition going on.
In the early 1970’s Gillette moved to embed the blades into disposable cartridges which, due to their patent protection, could not be copied by other manufacturers. This lead to our present situation of having multiple, incompatible razor systems. Competition remains fierce, with a battle to see who can put the most blades in a cartridge, from the Gillette Trac II with two blades and the Mach 3 with three blades, to Schick’s Quattro with 4 blades, to Gillette’s Fusion with 5 blades. Any guesses on what is next?
Video game consoles are in a similar position. In fact, they are often called a “razor and razor blade” business, since they sell the consoles at less than cost and later make their profit selling the game cartridges in proprietary formats. There is little interest, and seemingly little demand for a universal game cartridge standard.
Another example is the realm of SLR camera lens mounts. Each camera manufacturer has their own system of incompatible lens mounts. Is one clearly better than another? Have the multiple standards encouraged innovation in the area of lens mounts over the past 40 years? Good question. All I know is I have a bag full of Minolta lenses that I can’t use anymore since I moved to a Pentax camera.
We’ve all seen the many optical storage formats in recent years. Just in the realm of writable DVD disk standards, we’ve seen DVD-R, DVD-RW, DVD+RW and DVD-RAM, many of them in single and double-sided variations.
In the past 5 years we’ve seen perhaps a dozen or more varieties and variations of memory card formats, all of them proprietary and incompatible with each other. It makes the state of optical disk formats seem regular and peaceful in comparison.
To these can be added the hundreds of daily items that have managed to avoid a single standard, such as vacuum cleaner bags, coffee filters, laptop power supplies, cell phone chargers, high definition video disc formats, surround sound audio disc formats, etc.
That is all for Part I. Some questions to ask yourself:
- In the examples given of domains where there is a single standard, most of them did not start off that way. Most started with many competing approaches. What forces led them to a single standard?
- Who won and who lost in moving to a single standard? Who decided to make the move?
- In the cases where there are multiple, incompatible standards, is there a market demand for unified standards? Why or why not?
- If a government decree came down today and mandated a single standard in those areas, what would be gained? What would be lost?
I hope you will continue on with reading Part II.
Interesting. I have two itches in this conversation, and I suspect they will come up anyhow.
1. It would be interesting to talk about information processing standards, because that is significant in the case of document formats. The development of standards for digital codes has a great history, and LBJ’s Federal ASCII mandate had no direct impact (e.g., EBCDIC continued to be the mainframe code of choice), until the microcomputer and then the PC more-or-less settled the matter (so is EBCDIC dead?). Programming languages are also a great example (why isn’t PL/I the only one?).
2. There’s something off in your observation that “historically there has always been but a single document format.” I don’t think that has ever been true, especially around standards for formats and even with the ebb and flow of popular de facto formats. You might argue that the Word .doc is now a de facto standard, but I don’t think even that establishes your thesis.
I look forward to the continuation and the lively discussion that it may bring to the comment threads.
I think taking a narrow technical view of documents, as just being an information processing standards, is to unappreciated the role documents play today. Who creates and consumes documents today? How are they created, exchanged and published? How has this changed in the last 25 years, or even the last 10 years?
This is not longer a concern merely for the guys in the back room with the mainframe. I’ll argue that point a bit more in Part III, including my thoughts on how historically there has always been but a single document format.
Another thing to mention is that organizations tend to use a single standard internally, even when the market around them uses multiple standards. This was alluded to when you wrote about IBM purchasing Lotus and changing their office software standard.
I suspect that this is going to be true as you look at other areas as well. I look forward to the rest of this series.
The case for a single document format…
Isn’t that a bit like trying to break through an open door?
Reducing complexity is the bread and butter of CIOs and enterprise IT architects. They all want fewer suppliers, fewer technologies to integrate and the lower TCO that result from lower complexity. Can you imagine half the market using ODF and the other half using OOXML and everybody being forced to use both standards for document exchanges? Having a single document format is a no brainer. It is the case for having _two_ document standards that is hard to make.
When Microsoft argue there is room for two standards, they actually mean that their format must be elevated to standard status. They don’t care about ODF. They believe the market forces (read their monopoly power) will make OOXML the unique standard that matters practically. What they want in the mean time is to make sure governments that mandate ISO approved standards don’t exclude OOXML and don’t force build an installed base for ODF. It is really that simple.
When Microsoft friends argue that ODF and OOXML are designed to meet different requirements, I always wonder what are the requirements that ODF meets that OOXML doesn’t. Could it be vendor independence?
But Microsoft usually doesn’t discuss the advantages of ODF. They prefer to paint OOXML as the standard that provides interoperability with the installed base, implying that ODF doesn’t. They are saying something every CIO in the world will understand. They are saying if the CIOs want to be compatible with their installed base of documents, applications and workflows, they need OOXML. This is the message Microsoft is sending loud and clear.
Microsoft counts on the CIO’s desire to have only one standard and remain compatible with the installed base to kick ODF out of the market. All this coexistence talk is a diversion to make ODF proponents look like hard line extremists and reduce the opposition to the ISO approval. The real plan is to have only one standard left standing. Everybody knows it and wants it that way, suppliers and customers alike.
There shouldn’t be any discussion on the number of document standards we need. The real question is why this standard should be dictated by a monopoly’s desire to retain control over the market.
Today the answer to this question is one of credibility. As long as CIOs don’t believe ODF proponents can displace Microsoft, this is a self fulfilling prophecy. CIOs will adopt OOXML because they believe everybody else will adopt OOXML, but everybody else adopt OOXML for the same reason. This is the catch-22 ODF must break.
The credibility can change overnight if some influential governments mandate ODF and exclude OOXML. Then application developers must support ODF to enter into this market. ODF will have credibility and the catch-22 is broken.
This is why Microsoft works so hard to get ISO approval. If they can make sure OOXML is not excluded, CIOs will be free to stick to OOXML and ignore ODF.
But this ISO approval is not designed to improve vendor independence. If you believe that, I have a bridge in Brooklyn to sell you. This ISO proposed standard is designed to give Microsoft control over the barriers of entries, making sure they are not excluded by government policies while they retain the ability to exclude everybody else on a legal and technology level.
Queen Elizabeth says
Very interesting! I can’t wait for the next three installments.
However, shouldn’t the broadcast TV standards be under the category of multiple standards? NTSC/PAL/SECAM and their numerous, semi-compatible variants do cause lots of problems (e.g. a film festival I was at where we could not show the feature film!) And HDTV is no panacea, as Europe and the US have mandated–rather stultifyingly–totally incompatible digital TV standards.
Also, Gillette’s proprietary razor attachments are less an example of why multiple standards are bad than the downsides of the patent system. (Do we really need patent protection to devise new razors? Are R&D costs for razors so high that we need to incentivize their development with limited monopoly rights?)
When I speak of a single standard it is with respect to some large population of communicating users, such as a nation. So 240-Volt Electric dryers are the standard in the US, and although it is different in Europe, there are not many people who travel with their clothes driers. And although we have a standard rail gauge in the US, it does not need to be identical to that used in the UK, since not many trains travel from the US to the UK.
Think of Darwinian speciation, where geographical isolation will cause separate adaptions and genetic drift that lead to the creation of new species.
I think video standards are similar. The majority of users in the U.S. will go their entire lives without needed to deal with a video in another format. I wonder if this is different in Europe? For example, how are broadcast signals handled in the region bordering France and Germany?
In any case, nothing is final. If circumstances change, then a domain which once accepted multiple standards without complaints could suddenly find them unacceptable. Note the example of the creation of Standard Time prompted by the increase in rail traffic.
As for razors, I’m not saying that having multiple standards there is bad. I’m just observing that in this domain multiple standards have shown a longevity. In any case, I use an electric razor.
Jason Matusow’s already given the farm away, with some recent statements that what is actually essential – for customers – is working and workable file format translators. I happen to agree with him – it just happens to make the Microsoft insistence on official standardization moot.
For him to make customers’ needs the basic, essential necessity is indeed praiseworthy, considering the customary Microsoft attitude of what’s good for us is good for the customer, even if the customer doesn’t agree, because we know much more than the customer by definition … ;)
So, I declare the “file format war” over. It’s not really a “war” when one party comes up to you and begs, “Can you please say I won this time? Please?”
In any case, the important part is not demolishing Microsoft – that’s Microsoft’s privilege, and it’s one they fully believe in and pay Steve Ballmer handsomely to carry out – the important thing is making sure that Microsoft products are not the automatic purchase decision, because when that happens, Microsoft goes to sleep and we all suffer.
As I understand it, the real problem for MS is that WTA rules require government procurement procedures to include ONLY international standards when available.
The moment ODF got ISO ratification, it became illegal to refer to MS Office compatibility in government procurement.
That is why MS desperately needs ISO ratification of Ecma 376. If it fails, MS will be forced to support ODF as, eg, the EU members will not be allowed to buy or require MS Office anymore.
Wesley Parish said:
“Jason Matusow’s already given the farm away, with some recent statements that what is actually essential – for customers – is working and workable file format translators. I happen to agree with him – it just happens to make the Microsoft insistence on official standardization moot.”
The Wraith says
On the other hand it could easily happen that 80% or more of the world uses OOXML and the rest thing like docbook, latex , PDF and ODF.
I think a document format will not be a very important choice in deciding what Office suite to use. As ISO wants to support standards with market usage it would be weird move letting it pass by.
Also several governments will want to have the MS format ratified by ISO as it gives some kind of control of Microsoft. They could put pressure on ISO which in turn can pressurize Ecma to make changes in the format. This is of particular interest to possibly the EU which has a lot of voting power within ISO.
I think you’re thinking like a scientist or an engineer and not a marketeer. Consider your beverage can example: you say it’s not an optimal shape. Optimal for what? For holding the maximum volume of liquid with minimal materials, or an optimal diameter for holding in a typically sized hand?
Whether a uniform standard or proprietary design wins out over the long term depends on many details and competing interests. I find your blog very interesting, but I don’t think that you’re going to settle this question!
You are correct. There are other concerns other than material efficiency in cans. They need to consider how susceptible the can is to tipping, how strong the can is for stacking during storage, how easy it is to hold, etc. However, even given that, there is benefit to tinkering with the can size. For example, in Canada, Lebatt made a small adjustment to narrow the cans 1/8″ a few years ago and saved 10% on their aluminum consumption.
That leads me to think that the material in the ends of the can is more expensive than the walls of the can. But in any case, my impression is that can companies can only innovate within the narrow range that preserves compatibility with the installed base of cup holders embedded in cars and other places. That was my point.
Ok, I get that (you’ve probably thought about this a lot more than I have). And I suspect you’re right about the ends of the cans, just look at the thickness of the ends vs. sides, which is obviously necessary due to the shape and pressure.
But once again, you bring up engineering style arguments (e.g. saving aluminum costs, even the cupholder is something of a design criteria at this point).
You end your post with a whole laundry list of products that are not standardized, and some questions. Maybe it’s where your articles are headed anyway, but it seems to me that most of these decisions are made by companies trying to make money (and gain market advantage any way possible). Good design is very much an afterthought, and innovation happens by dumb luck (and I don’t mean how MS defines ‘innovation’). Once something approaches infrastructure status, then it’s time to standardize.
Something like razor blades are not an infrastructure item. However, Operating Systems and Word Processors are very much part of the infrastructure of computers. Hopefully we’ll get some sanity in that area in the coming decades.
Well, I’ll go back to lurking.
(sorry, post is way too long)
When I look at your examples, I seems to me that a lot of standardization is customer driven. Customers, which do include retail and producers of other equipment, want standardization. In general, producers do not.
In a demand driven market, we tend to get standardization. In a supply driven market, we don’t.
The razor blade industry tries to keep the market supply-driven by constant change. If they ever slack down, standardization will raise its “ugly” head again.
Given that soft-drink companies don’t want to do the tool-making and distribution, I think they forced standardization onto the can-construction and filling machines and were forced to standardize to some extend by their distribution channel, eg, the vending machines and retail shops. I am not sure what part of the market vending machines take, but that could be an important factor.
This reminds me of a story about the introduction of ATMs in NYC (I believe).
The biggest bank tried to cash in on its market dominance by restricting their ATMs to their customers only. Hoping their larger number of teller machines would draw customers away from the smaller banks, who had less teller machines.
The other banks responded by opening up their teller machines to customers of all compliant banks. In one swoop, the competition had way more teller machines than the biggest bank and now it was losing customers to the smaller banks.
In one go, this seems to me to be the strategy behind ODF. It also illustrates why MS will fight to the end to kill off ODF. Like they did with Open Document.
It also shows a way for ODF to “win” (get accepted). If the world has more developers than MS, the world can construct more ODF applications and services than MS. Given that MS are hampering all efforts from outsiders to build innovative services and applications on top of Office documents, there should be a market here.
One simplistic suggestion from me.
Why are the books in project Gutenberg stored in unprintable formats?
If they were available in ODF, I could buy a portable book reader and have an instant library of classical literature. Or just print and bind them. It cannot be done in PDF because that would prevent text based processing of the books. It cannot be in MS Office format, because each of them have become obsolete in a few years. With Ecma376/OOXML we would have to wait until MS found the time and mitivation to give us the tools. With ODF, the Gutenberg Project could build the transformers themselves easily and we could have printing on demand of all the classics, and more.
“I think taking a narrow technical view of documents, as just being an information processing standards, is to unappreciated the role documents play today.”
I didn’t suggest that. I just noticed that Information Processing Standards also have an important history and, being in the digital domain, along with the kinds of documents and their formats that we are discussing, they are worthy of consideration along with razor blades.
I didn’t mean IT either, but I guess ISO does, huh? Go look at the title page you published.