Send to a Friend

ScottyMcGeester's avatar

Why does Microsoft Excel sometimes calculate things differently?

Asked by ScottyMcGeester (1897points) July 24th, 2015

I have 9 numbers in excel.
5.2
5.2
5.2
5.2
5.3
5.3
5.2
5.2
5.3

I wanted the average of them so excel did it. But the average came out 5.3. That didn’t sound right to me. There are way more 5.2s than 5.3s. So I used a calculator and the calculator told me 5.23 repeating, which I would round to 5.2. But when I click on the cell in excel and set it to add more significant figures, it gives me 5.26, hence why it rounds to 5.3.

The same kind of thing happens when I want to look up the %CV of certain things. It’s a long story but I basically took the absorbance of water and calculated the average and standard deviation. Both came out to .001.
%CV is the st. deviation divided by the average and multiply that by 100. But excel is telling me the %CV is 86.6 – which makes NO SENSE because when you divide .001 by .001 it obviously gets you 1 and 1 times 100 is 100.

What gives?

Using Fluther

or

Using Email

Separate multiple emails with commas.
We’ll only use these emails for this message.