The lat/lon floating point delusion
31–40 of 154 posts
Re: The lat/lon floating point delusion
#32Earlier quoted context omitted.
Error tends to grow as the calculation proceeds. 6th-decimal "insignificant rounding" done over and over as the calculation proceeds can grow to become arbitrarily large.
Which calculation on lat/lon coordinates would make an atom-scale error grow, say, a million times to millimeter-size?
Re: The lat/lon floating point delusion
#33Does anyone here know what's the actual precision that we can get by the available equipment?
Re: The lat/lon floating point delusion
#34Yes, why don't people spend their time doing the extra work to figure out the exact number of meaningful digits in their measurements when the defaults work just fine for their non-scientific, non-metrological purposes? A mystery for the ages.
Re: The lat/lon floating point delusion
#35Does anyone here know what's the actual precision that we can get by the available equipment?
Re: The lat/lon floating point delusion
#36Did the author think about the implementation? If we assign semantic meaning to the number of decimal points, now we also need to store the number of significant digits side by side with the actual number. We shouldn't round off internally because we might use it in as an intermediate result in a calculation.
Re: The lat/lon floating point delusion
#37On the flip side - who cares? It's just a question of shunting data around - ok someone could add a format specifier somewhere and it would be neater. But there's no particular harm in it either.
1. It implies more precision, which in many cases is a flat out lie. Savvy users will know to truncate, round or something, but others will be lead astray.
2. It takes more space to represent. If the engineers storing double precious floating points had asked themselves if they needed to be sending these out at such a high resolution, they may have been able to cut the storage of these things down 50%.
Though yes, there are probably bigger issues to worry about. Doesn't mean we should outright dismiss these points.
Re: The lat/lon floating point delusion
#38Yes, why don't people spend their time doing the extra work to figure out the exact number of meaningful digits in their measurements when the defaults work just fine for their non-scientific, non-metrological purposes? A mystery for the ages.
I like your argument, but it would be nice to keep things civil.
Re: The lat/lon floating point delusion
#39I'm sure my thermostat internal temperature representation is way more precise than the 0.5 degrees precision it's showing on the display.
Re: The lat/lon floating point delusion
#40Not just lat/lon. I find myself amused when The Washington Post's education columnist Jay Matthews--quite a sharp guy from all I can tell--runs his high school "challenge index" (AP tests taken / size of graduating class) out to six decimal points for small schools. Somebody give that man a slide rule.