Calculator errors: When close isn’t close enough (Part 1)

Far too often, students settle for a numerical approximation of a solution that can be found exactly. To give an extreme example, I have met quite intelligent college students who were convinced that was literally equal to .

That’s an extreme example of something that nearly all students do — round off a complicated answer to a fixed number of decimal places. In trigonometry, many students will compute by plugging into a calculator and reporting the first three to six decimal places, like . This is especially disappointing when there are accessible techniques for getting the exact answer (in this case, ) without using a calculator at all.

Unfortunately, even maintaining eight, nine, or ten decimal places of accuracy may not be good enough, as errors tend to propagate as a calculation continues. I’m sure every math teacher has an example where the correct answer was exactly $\displaystyle\frac{3}{2}$ but students returned an answer of or because of roundoff errors.

Students may ask, “What’s the big deal if I round off to five decimal places?” Here’s a simple example — which can be quickly demonstrated in a classroom — of how such truncation errors can propagate. I’m going to generate a recursive sequence. I will start with . Then I will alternate multiplying by and then subtracting . More mathematically,

if

Here’s what happens exactly:

So, repeating these two steps, the sequence alternates between and .

But looks what happens if I calculate the first twelve terms of this sequence on a calculator.

Notice that by the time I reach , the terms of the sequence are negative, which is clearly incorrect.

So what happened?

This is a natural by-product of the finite storage of a calculator. The calculator doesn’t store infinitely many digits of $\displaystyle \frac{1}{3}$ in memory because a calculator doesn’t possess an infinite amount of memory. Instead, what gets stored is something like the terminating decimal , with about fourteen s. (Of course, only the first ten digits are actually displayed.)

So multiplying by and then subtracting produces a new and different terminating decimal with three less s. Do this enough times, and you end up with negative numbers.