Excel: November 2007 Archives

Weird Excel Calculation Bug

| | Comments (0)

Excel has a "bug" where calculations whose results are near 2^16 (65,536) may not be correctly displayed.

* Microsoft - http://blogs.msdn.com/excel/archive/2007/09/25/calculation-issue-update.aspx
* Chance News 30 - http://chance.dartmouth.edu/chancewiki/index.php/Chance_News_30

What's interesting to me is not the bug, per se, but the fact that calculations on a computer are inherently different from those on a chalkboard. Chance News and the Wolfram Blog both point out that computer calculations are all binary (base 2) at some point, while blackboard calculations are usually base 10. There's a conversion operation somewhere behind the scenes and once you get into decimals and the potential for repeating numbers, it can get tricky.

Another factor is that apparently multiplication is really done via addition of logarithms and lookup tables in order to save processing cycles. In other words, more conversions. This is not how I was taught multiplication in 5th grade.

The bug is pretty rare, so I will still be using Excel. But it's at times like these I do understand "old-school" math teachers wanting to make sure students still know how to do long division by hand. After all, these errors were probably caught by people who realized that the answers from Excel "didn't look right."