# Calculator errors: When close isn’t close enough (Part 1)

Far too often, students settle for a numerical approximation of a solution that can be found exactly. To give an extreme example, I have met quite intelligent college students who were convinced that $\displaystyle \frac{1}{3}$ was literally equal to $0.3$.

That’s an extreme example of something that nearly all students do — round off a complicated answer to a fixed number of decimal places. In trigonometry, many students will compute $\sin \left( \cos^{-1} 0.3 \right)$ by plugging into a calculator and reporting the first three to six decimal places, like $0.95394$. This is especially disappointing when there are accessible techniques for getting the exact answer (in this case, $\displaystyle \frac{\sqrt{91}}{10}$) without using a calculator at all.  Unfortunately, even maintaining eight, nine, or ten decimal places of accuracy may not be good enough, as errors tend to propagate as a calculation continues. I’m sure every math teacher has an example where the correct answer was exactly $\displaystyle\frac{3}{2}$ but students returned an answer of $1.4927$ or $1.5031$ because of roundoff errors.

Students may ask, “What’s the big deal if I round off to five decimal places?” Here’s a simple example — which can be quickly demonstrated in a classroom — of how such truncation errors can propagate. I’m going to generate a recursive sequence. I will start with $\displaystyle \frac{1}{3}$. Then I will alternate multiplying by $1000$ and then subtracting $333$. More mathematically, $a_1 = \displaystyle \frac{1}{3}$ $a_{2n} = 1000 a_{2n-1}$ $a_{2n+1} = a_{2n} - 333$ if $n > 0$

Here’s what happens exactly: $1000 \times \displaystyle \frac{1}{3} = \displaystyle \frac{1000}{3} = \displaystyle 333\frac{1}{3} = 333.\overline{3}$ $\displaystyle 333\frac{1}{3} - 333 = \displaystyle \frac{1}{3} = 0.\overline{3}$

So, repeating these two steps, the sequence alternates between $\displaystyle \frac{1}{3}$ and $\displaystyle 333\frac{1}{3}$.

But looks what happens if I calculate the first twelve terms of this sequence on a calculator. Notice that by the time I reach $a_{11}$, the terms of the sequence are negative, which is clearly incorrect.

So what happened?

This is a natural by-product of the finite storage of a calculator. The calculator doesn’t store infinitely many digits of $\displaystyle \frac{1}{3}$ in memory because a calculator doesn’t possess an infinite amount of memory. Instead, what gets stored is something like the terminating decimal $0.33333333333333$, with about fourteen $3$s. (Of course, only the first ten digits are actually displayed.)

So multiplying by $1000$ and then subtracting $333$ produces a new and different terminating decimal with three less $3$s. Do this enough times, and you end up with negative numbers.

## One thought on “Calculator errors: When close isn’t close enough (Part 1)”

This site uses Akismet to reduce spam. Learn how your comment data is processed.