You said : “Because of this, a physical error of 1-bit introduces more than 1-bit of error in the information content of the document.” I think that is not true, as you perfectly say in the previous sentence. I think you wanted to say :
“Because of this, a physical error of 1-bit introduces a bigger error in the information content of the document in the ZIP than in the uncompressed DOC file.”

Thanks for your analysis which is just perfect.

First, that doesn’t apply in the truncation case because that happens at the zip level and does not affect the XML inside for small truncations (Rob stated this was because the redundant zip toc? is what lies at the end of the file).

Second, for cases where the XML is likely affected, it’s unclear from the data and discussion above (iirc) whether an error for ODF or for OOXML was a fatal error and, if so, whether that was the exact (or part of the) reason the application failed.

If it turns out that the failures were solely because of the XML standard but were otherwise recoverable, then this is something to be considered carefully. However: I’m guessing that if what was lost/corrupted was solely text or various other types of data, then the error might not be defined by the standard as a fatal error. I’m guessing a fatal error would include damage to the information defining the XML structure, and in most of these cases, recovery might not be possible, regardless of what the standard says.

To criticize this aspect of the standard, we should be specific (point to specific mentions of “fatal error”), and then come up with examples that show the fatal error requirement is unwise in such a case (eg, because faithful recovery would have been possible and hence desirable). Not taking these steps leaves us hand waving and possibly dwelling on a non-issue.

IIRC they changed this, it used to be just about any error.

So aren’t all these implementations that try to fuddle through strictly non-conformant?

(just yet another reason xml is a bad choice for just about anything …)

Yes, I think that is correct (“think” only because I can’t be sure without being sure of the algorithm, but I think it’s FIFO-ish replacement more or less), thanks.

>> good.. if a random error makes a valid document invalid.. But we also want applications to be robust to errors

Right.. have errors produce effects that are always identifiable but never catastrophic.

>> maybe one could eliminate the apps from the test altogether and simply test the models in isolation, with simpler command-line apps that merely try to interpret the document

This would complement the fat app approach.

Some types of tests would not be visual and could gain a measurable passing/failing grade with minimal work on focused tools.

Also, attempting pure analysis here or there could in some cases lead to new insight and perhaps even to something approaching an “undeniable proof”.

>> You could even be methodical and introduce a block error starting at every successive offset in the file and in this way map out exactly which parts of the document are vulnerable.

.. a very large number of possibilities, and, if achieved, possibly resulting in data overload.

Perhaps this approach would be successful to stir creative juices and/or to help validate an existing theory.

To you’re other point, I’ve done tests like that before for other formats, but more from the perspective of testing the app. For example, when I worked on the initial port of Xalan XSLT engine to the C++ version I used similar techniques to introduce random perturbations of XSLT scripts to see whether these errors would be handled graciously by the code. That was easy enough, since the engine itself was command-line driven and you could wrap it all in a WIN32 debugger session to catch and recover from all memory faults, etc. Doing the same for a GUI app like MS Office or OpenOffice in theory is possible. You could even be methodical and introduce a block error starting at every successive offset in the file and in this way map out exactly which parts of the document are vulnerable.

In terms of errors introduced by application bugs, I you need to set the criteria carefully. It is a good thing, IMHO, if a random error makes a valid document invalid since that allows errors to be identifies by common tools, e.g., schema validators. But we also want applications to be robust to errors, so users don’t loose their work. Most of the errors I detected were things that should be recoverable by using a robust ZIP library and a robust XML parser. Until the apps handle those more surface issues better, I don’t think testing is going to tell us much about the robustness of the underlying document models. Or, maybe one could eliminate the apps from the test altogether and simply test the models in isolation, with simpler command-line apps that merely try to interpret the document, resolve all references to content and styles, etc., but don’t attempt to render anything?

I am not interested in random disk errors to files. I am thinking that modeling in some way random errors at the XML level (perhaps crudely simulated with random localized errors to XML files) models deviations from “spec” based on bugs. Syntax deviations are like random errors on the file stream. Semantic deviation might function similarly or not depending on the details.

[I used quotes around "bugs" to suggest that certain bugs might not be accidental or, otherwise, might be allowed to exist by a market dominant implementer since these bugs could promote profit margins through their effect of making it more difficult for third parties to reproduce these buggy effects in their competing products.]

It could be useful to identify which format is more amenable to having X changes in semantics/syntax have a more compounding effect more likely to lead to gibberish by third party products not able to reproduce these changes correctly. Intuition led (leads?) me to think that a “delta” representation of data is less robust to changes (“errors”) in the stream, all else being equal. Maybe I need to think about this more, but the idea is not too unlike what you mentioned (or implied) about the problem with “silent” errors going into a document and how these might surface as time goes on, perhaps having a compounding effect as well (through repeated document manipulations, savings, openings, etc).

To be sure I read correctly, this discussion is in reply to Uri but does not correlate to the tests performed, right? In other words, there was no block level testing of XML, right? .. meaning the block damage was done against zipped files only, right? [Testing unzipped ODF/OOXML would depend on how the file system would store related but distinct XML files, and this could vary quite a bit (over time and over different file systems).]

While on this subject…

I would like to see the effects of damages to individual XML files for ODF and for OOXML. Though I am much more familiar with ODF than OOXML, there is much I don’t know about either of these formats. What I am after is to see what percentage of localized errors (like the random block data test) have devastating effects on these two formats.

I wonder because I think I had heard that a greater percentage of OOXML is “delta-based”, meaning that the effect of errors “early” in the stream would compound afterward at a faster rate than “nondelta-based” (but still nested) XML.

Besides the statement that such a result would make about a level of robustness of each format, I’m also thinking that it might be more difficult for third parties to keep up with a proprietary OOXML implementation (eg, a dominating implementation like MSOffice20xx) than it would be a proprietary ODF implementation based on “bugs” found respectively within such closed source implementations. Looked at from a different angle, if the results are as I imagine is the case, then it would be easier to keep a closed-source OOXML implementation from being matched sufficiently well by competing third parties (than would be the case for ODF) through the careful insertion of “bugs” that disagree with the spec (or that leverage ambiguous or missing components of the spec).

Regardless of which is easier to “manipulate”, OOXML or ODF, knowing that answer may help third parties better decide how to address a closed source market leader (in OOXML or in ODF).

In any case, to your question, no I have not entered a defect report on these issues. Since I do not use OOXML myself, the fact that OOXML documents are so easily damaged in irrecoverable ways is not my problem. However, I have posted the test cases and code for reproducing these results,. The vendors are welcome to deal with this bug as they wish. I’d mainly hope that Microsoft would stop making baseless and false robustness claims that are easily refuted. One can hope, at least.

Just a bit of nitpicking.

Does that fall outside of the spec for ODF?

As for XML formats in general, their modularity is in their abstract model. It is not necessarily in the character stream representation. The characters that follow your element X could be content of X, a sibling of X, a child of X or even stuff further up the document tree than X. So any block-level damage could take a whack out of your document tree that is very inelegant and certainly non-localized. Of course, you could, as OOXML does, move different pieces into different XML files. But that won’t have a real impact if, like OOXML, 80% of the stuff still ends up in one place, e..g, in document.xml, as it does.

But it is an interesting question: if you wanted to design an XML format to be resistant to certain kinds of damage, what would be your design points?

@Felix, it is an interesting idea, but I’d still solve it at a level higher than the XML. Why? Because an ODF document can contain other resources, like image files, that are not in XML. But maybe using Error Correcting Codes at the ZIP level would work?

I still believe that users of XML formats (ODF, OOXML, etc) encounter much less severe document corruption, though I don’t have data to back that up. Microsoft could have said: “Remember all those documents our buggy software corrupted? Now we have the same amount of bugs but they are less likely to corrupt your files.” But that’s not good marketing, so they aligned on an alternative story about network and storage failures, which is unfortunately a lie. It’s hardly commendable, but the overall claim of less corrupted documents probably stands.

That’s great for business users with RAID servers, but for everybody else an option which said:

Use Safe File Saving (this will increase the size of your documents)? On/Off

would be pretty useful.

@Uri, this is far more than the Sinofsky quote in a press release. This claim has been repeated over and over again, including the claim that OOXML has magical recovery properties in the face of “storage failures”.

For example this case study claims: “XML-based documents are less likely to lose valuable information if the file suffers a partial corruption through a transfer or storage failure, for example. This makes the new Open XML format more resilient and reliable than the binary format used by previous Microsoft Office releases. Most files can still be opened if a component within the file is damaged.”

Also, the same claim in Microsoft’s technical repository, MSDN.

This is not an off-the-cuff improvisation by a single VP. You will find this mantra repeated dozens of times, in press releases, in white papers, in training materials, in notes to ISO NBs, in Microsoft blogs, etc.

True, this has nothing to do with what Sinofsky said, but he’s a Senior Vice President so you can’t expect him to be in touch with reality. Some manager 4 levels below him was asked to prepare a Powerpoint presentation with talking points about the new format, and the engineers explained the advantage of easier recovery. By the time it got to Sinofsky it was a single context free bullet point that said “improved recovery”, which he used as a theme to wax rhapsodic about without having a clue what that means.

Marketing is saying things that sound good, not correct things. Proving that some marketing claim is incorrect is too easy. It’s much more interesting to check the actual results – what part of actual documents becomes corrupt, and to what degree? I’m guessing that your experiment tells us very little about that, simply because it models a very small part of the prevalent corruption mechanisms.

Nonetheless, the file format should provide these as an option to users.

Rob, can you put something in ODF 1.2 which says “this file is X Bytes, with an additional Y Bytes of data recovery information” and specify that for ODF 1.2 Y will be zero but in subsequent versions it may be greater than 0?

Can you give an approximate size ratio for file in ODF, OOXML and DOC?

Thanks