Calculator errors: When close isn’t close enough (Part 1)

Far too often, students settle for a numerical approximation of a solution that can be found exactly. To give an extreme example, I have met quite intelligent college students who were convinced that \displaystyle \frac{1}{3} was literally equal to 0.3.

That’s an extreme example of something that nearly all students do — round off a complicated answer to a fixed number of decimal places. In trigonometry, many students will compute \sin \left( \cos^{-1} 0.3 \right) by plugging into a calculator and reporting the first three to six decimal places, like 0.95394. This is especially disappointing when there are accessible techniques for getting the exact answer (in this case, \displaystyle \frac{\sqrt{91}}{10}) without using a calculator at all.



Unfortunately, even maintaining eight, nine, or ten decimal places of accuracy may not be good enough, as errors tend to propagate as a calculation continues. I’m sure every math teacher has an example where the correct answer was exactly $\displaystyle\frac{3}{2}$ but students returned an answer of 1.4927 or 1.5031 because of roundoff errors.

Students may ask, “What’s the big deal if I round off to five decimal places?” Here’s a simple example — which can be quickly demonstrated in a classroom — of how such truncation errors can propagate. I’m going to generate a recursive sequence. I will start with \displaystyle \frac{1}{3}. Then I will alternate multiplying by 1000 and then subtracting 333. More mathematically,

 a_1 = \displaystyle \frac{1}{3}

a_{2n} = 1000 a_{2n-1}

a_{2n+1} = a_{2n} - 333 if n > 0

Here’s what happens exactly:

1000 \times \displaystyle \frac{1}{3} = \displaystyle \frac{1000}{3} = \displaystyle 333\frac{1}{3} = 333.\overline{3}

\displaystyle 333\frac{1}{3} - 333 = \displaystyle \frac{1}{3} = 0.\overline{3}

So, repeating these two steps, the sequence alternates between \displaystyle \frac{1}{3} and \displaystyle 333\frac{1}{3}.

But looks what happens if I calculate the first twelve terms of this sequence on a calculator.


Notice that by the time I reach a_{11}, the terms of the sequence are negative, which is clearly incorrect.

So what happened?

This is a natural by-product of the finite storage of a calculator. The calculator doesn’t store infinitely many digits of $\displaystyle \frac{1}{3}$ in memory because a calculator doesn’t possess an infinite amount of memory. Instead, what gets stored is something like the terminating decimal 0.33333333333333, with about fourteen 3s. (Of course, only the first ten digits are actually displayed.)

So multiplying by 1000 and then subtracting 333 produces a new and different terminating decimal with three less 3s. Do this enough times, and you end up with negative numbers.

One thought on “Calculator errors: When close isn’t close enough (Part 1)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.