Does anyone here know what's the actual precision that we can get by the available equipment?
The lat/lon floating point delusion
21–30 of 154 posts
Re: The lat/lon floating point delusion
#22Hmm. Doesn’t exactly seem like a crisis. So there are some extra digits, that’s ok.
Re: The lat/lon floating point delusion
#23On the flip side - who cares? It's just a question of shunting data around - ok someone could add a format specifier somewhere and it would be neater. But there's no particular harm in it either.
Re: The lat/lon floating point delusion
#24Not just lat/lon. I find myself amused when The Washington Post's education columnist Jay Matthews--quite a sharp guy from all I can tell--runs his high school "challenge index" (AP tests taken / size of graduating class) out to six decimal points for small schools. Somebody give that man a slide rule.
When I see those errors, I often try to work out the exact input figures that led to the stats reported.
At work, I recently saw a survey result from a small customer base reported as "31.3% of respondents" rather than the more communicative "5 out of 16..."
Re: The lat/lon floating point delusion
#25Earlier quoted context omitted.
Error tends to grow as the calculation proceeds. 6th-decimal "insignificant rounding" done over and over as the calculation proceeds can grow to become arbitrarily large.
Which calculation on lat/lon coordinates would make an atom-scale error grow, say, a million times to millimeter-size?
[deleted]
Re: The lat/lon floating point delusion
#26Earlier quoted context omitted.
> We shouldn't round off internally because we might use it in as an intermediate result in a calculation. Only if the precision is meaningful. The author's point is that it isn't for geographical coordinates.
Error tends to grow as the calculation proceeds. 6th-decimal "insignificant rounding" done over and over as the calculation proceeds can grow to become arbitrarily large.
If you're doing calculations with geographical co-ordinates that are so unstable that a nanometre position difference in the original makes an arbitrarily large difference in the result, your results aren't sensible anyway.
Re: The lat/lon floating point delusion
#27I usually go with 6 decimal places. Geojson format advises it also https://tools.ietf.org/html/rfc7946#section-11.2
> For geographic coordinates with units of degrees, 6 decimal places (a default common in, e.g., sprintf) amounts to about 10 centimeters, a precision well within that of current GPS systems. Implementations should consider the cost of using a greater precision than necessary.
If you actually read the RFC, it's giving a common example as an artifact of sprintf, and explicitly stating that implementations should consider how much precision they need...
Re: The lat/lon floating point delusion
#28The author also ignores the fact that the distance associated with a degree of longitude changes dramatically as you go from the equator (69m/111km) to the poles (0) - roughly as a cosine. So the number of decimal places of longitude should shrink as you get further from the equator. https://www.thoughtco.com/degree-of-latitude-and-longitude-d... "At 40 degrees north or south, the distance between a degree of longitu…
Yes, this is what makes it tricky. The units are very strange here. Its not a simple unit like meters or liters. Its not even one of the stranger units like Celsius (which is odd since zero C does not correspond to none of the quantity it measures, which is thermal motion). Its a relative unit, so its hard to judge how many decimals you need to be accurate.
Its almost like if I was to record where an astronomical object was using only time. Assuming you have full knowledge of its past trajectory, depending on how fast that object moves, you would need different precision times for objects traveling at different velocities, to get the same precision in space.
This is why you just give up on the problem and report to a precision that always works, instead of using the standard scientific procedure of significant figures. I guess if GPS measurement devices took all this into account they could report their measurements with the correct number of digits (taking into account measurement error, and the absolute position on Earth). I guess 15 digits is always too many, but whatever.
Re: The lat/lon floating point delusion
#29The author also ignores the fact that the distance associated with a degree of longitude changes dramatically as you go from the equator (69m/111km) to the poles (0) - roughly as a cosine. So the number of decimal places of longitude should shrink as you get further from the equator. https://www.thoughtco.com/degree-of-latitude-and-longitude-d... "At 40 degrees north or south, the distance between a degree of longitu…
Err, no, that fact is in the article.
Re: The lat/lon floating point delusion
#30As far as I know sig figs aren't a primitive type in most languages so the author's point is moot.