Speaking as a front-office (dealing room) software developer of (blimey) about 20 years now, I can attest that I have never seen any of the functions mentioned used in anger in an investment banking context. Which appears to be a Good Thing.

As mentioned briefly above, the functions are inadequate in their coverage of day count bases and many other “features” of the real-world financial environment. Because some cases can’t be handled at all, there’s little interest in checking Excel’s accuracy at all in this area. There’s ample money at stake for the banks to invest in making sure they can handle everything, and to be able to extend the libraries as innovation occurs within the markets.

I’d suggest that this is a large part of why the problems have yet to make it into a feature list at MS – the users whose voice would be loudest in complaining aren’t interested.

Frankly, these calculations are far too important to be left to Microsoft, which is a good thing for the Redmond folk, I’d suggest.

Try googling “On the accuracy of statistical procedures in Microsoft Excel” for another area that’s been broken for a long time.

No?

Then all these damn spreadsheets are useful… and I’ll just have to tell my boss that we have to go back to doing financial analysis by hand.

Thanks for dragging me back 30 years. Those bell bottoms are going to look totally hot now.

On the one hand, the ODF TC really would like to finish its work on ODF 1.2, and part of that is completing the OpenFormula work. A key remaining part of OpenFormula is to ensure that our financial functions synch up with how Excel works.

I can appreciate how you strive for accuracy. You really want Excel spreadsheets converted into ODF to return the same numbers which Excel returns. But Excel often gives wrong answers.

If I was writing a new standard, I would want that standard to produce correct answers. It’s ridiculous that leap year bugs for example should be preserved forever.

So why not do this: there should not be a single YEARFRAC function. Rather, there should be, for example, EXCEL98_YEARFRAC, EXCEL2007_YEARFRAC, DIS29500_YEARFRAC and so on. Each function would return exactly the values returned by the YEARFRAC function on the specified platform, bugs and all. You’d also write a ISO26300_YEARFRAC function which fixes all known bugs. Of course, if some bugs are subsequently found in your own function you can add a new ISO26300v2_YEARFRAC function which fixes the previous.

When converting a document from Excel 2007 or any other spreadsheet, function references to YEARFRAC are modified to refer to the platform-specific function name. That will ensure that imported spreadsheet calculations still return the same values.

The user (spreadsheet editor) will be able to make a conscious decision about whether to retain the platform-specific function names (and their bugs) or change to different functions which aren’t buggy.

Definitions of the EXCEL2007_* functions and so on seem to be hard to nail down. These could be documented as extensions to ODF, with the proviso that a document is ODF compliant only if it uses only published extensions.

Publication of the extensions can be via written specification and/or by a reference implementation. A reference implementation would contain code in some well-defined language which could be used as a plugin to any ODF compatible tool. The extension could be named by a URL and the tool would download the URL and compile (or interpret) the code directly.

Yes, it makes a much bigger function list. But the details can be published separately. And you need to be able to preserve the different bugs of Excel 98, Excel 2007 and so on, without crippling people who _don’t_ want to use functions containing those bugs.

Agreed. Exclude the first day and include the last, especially if this is the standard practice. I was arguing against the possibility of having both the first and last day in the range, which as I recall some comments suggested was possible.

re your discussion of start and end – I’d like to point out that number theory implies the reverse of your recommendation – the last day should count – the first should be considered day 0 – not day 1. Here’s why.

1. The definition of ‘last day’ implies that it is part of the range and it is traditional practice to specify COB (Close Of Business) as the time on the last date.

2. Number theory starts counting from 0. This means that the difference between 3 and 5 being 2 (resulting from simple subtraction may seem ambiguous, but consider the border case of the difference between 0 and 3 (3 days delta) – again simple subtraction. By your logic I’d have to subtract 1 from 3 (because 3 is not part of the range), resulting in a difference of 2, then add one because subtraction inherently does not count the ‘first’ day. Finally, the difference between 0 and 0 (same day) is still 0. (Remember – it was the Phoenicians’ invention of 0 that finally made mathematics into a viable tool for counting and accounting work.)

Finally, consider payments on a loan. This is one use of the difference between two dates. Counting the first day as part of the range makes sense as does that last day and everything hangs together (we’re talking delta-days here – not absolute days) provided that you start with day 0 instead of day 1. Paying off a loan on the first day (as day 1) results in unnecesary complications to your equations which also results in more potential for error. Treating the first day as day 0 cleans up a lot of issues and simplifies your logic.

Amazing.

Regarding Mark’s comment and the traditional method:

You wrote “The traditional method, before computers, would be to factor out the number of whole years…”

That definition is based on an approximation similar to

2*(a+b) / (c+d) ≅ a/c + b/d,

which introduces an unnecessary and unjustified error in the calculation, and accounts for the difference between your value for the ISO definition in example 4, and the Excel value.

As you wrote,

YEARFRAC = (interval between start-date and end-date) / length of year.

Those words leave no place for approximating 2(a+b)/(c+d) as a/c + b/d.

Note that (c+d)/2 is the average year length and we are assuming an interval beginning in one year and ending in the next. The three-year case would involve

3(a+e+b)/(c+e+d) ≅ a/c + e/e + b/d,

with (c+e+d)/3 the average year length, and the N-year case would be

N(a+e+f+…+b)/(c+e+f+…+b) ≅ a/c + e/e + f/f + … + b/d.

These approximations might have been used in the old days, but they can’t be described as “actual.”

I gather that you have some influence over what goes into these standards? In that case, I would suggest that the standard ought to use definitions such as that given below for YEARFRAC, instead of words such as

“If the Actual/actual basis is used, the year length used is the average length of the years that the range crosses, regardless of where start-date and end-date fall in their respective years.”

Instead, it should say

“YEARFRAC is the number of days in the interval times the number of years which the interval includes, divided by the total number of days in those years. The earlier of the two dates specifies the first day of the interval, and the later of the two dates specifies the first day not included in the interval.”

Then, it should elaborate:

Let A be the earlier date and B the later date.

Let B−A be the (certainly non-negative) number of days from A to B, zero if A==B.

(As an aside, A is the first day of the interval, and B−1 is the last day (inclusive). The interval begins at the beginning of the first day and ends at the end of the last day. A zero length interval ends the instant before it begins, and a one day interval begins at the beginning and ends at the end of the its single day. How could an interval possibly mean anything else?

Well, we could say the interval ends at the beginning of the first day after the end, i.e., at the beginning of day B, so that a zero length interval ends at the exact instant it begins.

In any case, no reasonable definition would have the interval end at the beginning of the last day. Thus B is obviously the first day not belonging to the interval and no reasonable person could say otherwise. End of aside about interval definitions.)

Let x.y be the year of date x, x.m be the month, and x.d be the day. Let date(x.y,x.m,x.d) be identical to x for all dates x.

Let F be the first day of the first year of the interval, and let L be the last day of the last year included in the interval:

F = date( A.y, 1, 1 ), and

L = date( (B−1).y, 12, 31 ).

Let N be the number of years covered by the interval:

N = L.y + 1 − F.y.

Let D be the total number of days in the years covered by the interval.

D = L + 1 − F.

Obviously, the “average length of
the years the range crosses” is

D / N.

Then clearly the description quoted above (from the “return value” section) cannot mean anything other than

YEARFRAC = N * (B − A) / D.

—-

The first description (from the “description” section) requires us to discard an obviously incorrect interpretation (that all years of the interval should be regarded as having length 366 if the interval crosses February 29). Also, it needs an additional primitive function, F29(X,Y), which is zero if the interval does not include February 29, and one if it does. This very need makes it extremely suspect, and I think the only reasonable interpretation of the standard it to regard that description as a holdover from an early draft which was corrected in one place but not another. However, if you exclude the obviously wrong interpretation, it does describe a well-defined calculation, as follows:

F = date( A.y, 1, 1 )
L = date( (B−1).y, 12, 31 )
N = L.y + 1 − F.y
if N == 1, then
   YEARLEN = 365 + F29(A, B)
else
   AEND = date( A.y, 12, 31 )
   BSTART = date( (B−1).y, 1, 1 )
   YEARLEN = (365 + F29(A, AEND)
     + BSTART − (AEND + 1)
     + 365 + F29(BSTART, B)) / N
endif
YEARFRAC = (B−A) / YEARLEN

What strikes me is that the ambiguity about OOXML Transitional and OOXML Strict is introduced by Doug and Jason themselves. They are probably the most knowledgeable people at Microsoft about this nomenclature, but still Doug simply said “Microsoft would not implement the *final* ISO version of OOXML until Office 14 ships”.

Is this simply a negligence, or is it meant to open the door for a more drastic move by Microsoft at a later date ?
In any case, I read this sentence as meaning that Microsoft will not tweak Office 2007 at all in SP2 to support the changes introduced by the BRM in OOXML Transitional.

Of course, now we have an official appeal against DIS29500 the game becomes much, much longer.

Perhaps they had “wind” of the appeal?

This could drag on for years…

The exact quote is that “IS29500 … is already substantially supported in Office 2007, and the company plans to update that support in the next major version release of the Microsoft Office system, code-named ‘Office 14.’”

All but the last example for YEARFRAC in the IS29500 you have are identical to the ones in ECMA-376. They have not changed.

The final example, illustrating a case with year divisor of 365.33333, is new in IS29500.

All of the examples are reporting what Excel produces. I verified them with Excel 2003 and expect no difference in Excel 2007 and probably not Excel 2000.

My tentative conclusion is that the additional explanation of the parameters and basis values in IS 29500 are simply bogus (except for the easy cases of basis 2 and 3).

Another tentative conclusion is that there is tremendous confusion about which dates from the beginning to the end date are actually *in* the range. This is leading to a lot of mischief, but I’m not sure on who’s part. I notice a couple of places where Wheeler seems to have stumbled into that pitfall, and I’ll clean up my examples and discuss that with him. This bears on only a small part of his analysis in any case and I suspect it has tripped-up others even more.

“Office 2007 Service Pack 2 will add native support for OpenDocument Format (ODF) 1.1, PDF 1.5, PDF/A and XML Paper Specification, an XML-based fixed-document format created by Microsoft.”

and..

“… Mahugh stated that Microsoft would not implement the final ISO version of OOXML until Office 14 ships at an unstated date in the future.”

If true, it is quite an amazing statement and seems to negate all the effort then went into forcing OOXML through ISO…

I’m going to double check this with some of the good examples and edge-case tests that we now have, but I am pretty confident that this is not a problem to be avoided by sticking with pre-2007 versions of Excel. Also, it is not clear whether the non-OpenFormula versions of ODF implementations provide any salvation (say, up through OO.o 2.4 and other products using the same spreadsheet code base).

Nearly nobody uses files to archive a document.
If you do, you have to know that you have to archive the program, you used to produce the document, too.

If the difference was very important, somebody should have realized before, that excel is just estimating a correct result.

Note beside:
I would like to see the sourcecode of the function?
How do they decide when to use which function?

One is archiving. The file that was historically used had incorrect calculations in it. This has to be preserved as is for the sake of fidelity to history. If ODF is used for archiving of current Excel files, then bug preservation becomes a requirement.

This raises the question of which Excel version was used in creating the file. We don’t know yet if the behavior of Excel was the same from version to version. This is not something I would take for granted. Tests are required.

The other is perception. People use Excel and just trust it to be accurate. If the numbers from another software disagree, then a natural reaction might be to assume Excel must be right because this is what people use and have been using for over a decade. Then the incorrect behavior becomes the truth because nobody cares about the actual truth.

Martin raises the possibility of failing SOX audits. But if the auditor uses Excel and believes it to be correct, then using the correct formula is what could land you in trouble. You are safe from a SOX perspective as long as everybody uses Excel.

This means the public relation aspect is essential. There is a need to teach the financial community about the bugs and their consequences. There is a need to teach auditors, CEOs and CFOs that if Excel is part of the financial process, then there is no SOX compliance.

This is not a small point. I know programmers that download in Excel database extracts, process them in VBA and the upload the result in the database. This kind of processing is the major reason for many users to want millions of rows in their spreadsheet. Otherwise they have to break their database extracts in small parts and stitch the result of partial computations afterwards.

Again, I would urge you to allow in the specification a generalized Basis setting for the document as a whole, so that it would be easy to change in a single move.

As far as whether applications need to allow for an “Excel-compatibility” standard, I see that as absolutely vital. Unless someone is an avid reader of “An Antic Disposition,” they are unlikely to understand ambiguities and errors in Excel’s calculation. And if they open a .XLS in OpenOffice.org or Symphony, and see different result calculations than in Excel, the assumption is going to be that the new program is wrong.

I know that sucks, but it’s how the perception will work — guaranteed.

The best thing to do it highlight it in the application. “This spreadsheet uses an ambiguous standard for fractional year calculations, and therefore may misrepresent discount rates. Click here to select from available techniques for calculating fractional years.”