Live data from Hacker News

The lat/lon floating point delusion

datafix.com.au

141–150 of 154 posts

Re: The lat/lon floating point delusion

#141
post #100

Earlier quoted context omitted.

You joke but a lot of radar/radio observation is done with ECEF coordinates, which is a (lat, lon, z-axis) triple.

ECEF is a Cartesian (x,y,z) not spherical (𝞺, θ, 𝜑) coordinate system. Units in ECEF are in meters from (0,0,0).

Blah, shows how much I remember about it. We always converted to ENU locally to start with, hah.

Re: The lat/lon floating point delusion

#142
post #121

Earlier quoted context omitted.

Why do bad actors spoof transmissions? Smuggling?

Many reasons, and a significant percentage of all AIS transmissions are dodgy. Smuggling is just one reason. Other common reasons include illegal fishing in protected waters, covert meetings (e.g. a couple yachts coming together in the middle of the ocean), spy ships by state actors trying to look like ordinary commercial traffic, and misrepresenting state of shipping assets for commercial leverage in contract negoti…

I would love to read more about this. Any suggestions?

Re: The lat/lon floating point delusion

#143
post #79

In contrast to his 4-digits-are-fine notions, let me offer a counterexample: http://scissor.com/transient/destination_heatmap/ I am working on a Twitter bot called @sfships, which monitors the comings and goings of large ships in the San Francisco Bay: https://twitter.com/sfships As part of that, I generated the above map from AIS data. [1] It's basically where ships stop. If you zoom in on the San Francisco waterfro…

> If you zoom in on the San Francisco waterfront, you will see a grid of dots.

And yet, some of those dots are significantly further inland than rounding can account for. There's one faint point near 4th and King, for example, and a strong one on Spear between Howard and Mission. All signs point to the source data being imprecise enough that transmitting it at a higher precision wouldn't improve the results.

Re: The lat/lon floating point delusion

#144
I could explain this. Internally all those coordinates are stored as double (not long double). The precision of a double is max 15-17 digits. Usually it's very inprecise though. But not too imprecise, because tiny errors could lead to very costly mistakes. my biggest did cost ~40.000, wrong after the third digit. 30cm. You might remember the financial stories on microdiffs with improper arith. That why one of the most important lessons I told my students was precision and it's limits. Esp. when zooming in.

CAD/GIS system round to the default vsprint %f precision which is 6. They could use a zoom dependent number but nobody would do that, they would laugh at you. It's not a delusion, it's industry practise. If you have better precision, you stick to it.

Re: The lat/lon floating point delusion

#145

Professional geospatial systems frequently use fixed point internally but export floating point for convenience. The Internet treats all geospatial as floating point but positional measurement systems and mapping base layers often come from fixed point data models. Many of the underlying fixed point systems are designed for meter precision, with some newer systems going down to a centimeter. As a practical matter, re…

There are large networks of GPS monitoring sites that regularly report their positions, including movement, down well below centimeters.

The point is about repeatability, not precision. Nominally fixed points on the Earth's surface are constantly in motion relative to GPS and relative to other fixed points. Many of the effects that cause this lack of repeatability can induce centimeters of measured deflection from the mean position over relatively short periods of time. There are models that can be employed to correct some of these effects (e.g. fluctuations in the gravity field, which is measured by satellites) but not all of them. Consequently, the measurement may be precise but there is never a reference frame in which the points are actually "fixed" over time, which is indistinguishable from reduced precision. Local changes in surface geometry are also easily detected with LIDAR.

For surveys that require maximum repeatable precision, they will often use the mean position as measured over e.g. 12 hours. The magnitude of the variance varies quite a bit depending on where you are on Earth.

Re: The lat/lon floating point delusion

#146

Earlier quoted context omitted.

Many reasons, and a significant percentage of all AIS transmissions are dodgy. Smuggling is just one reason. Other common reasons include illegal fishing in protected waters, covert meetings (e.g. a couple yachts coming together in the middle of the ocean), spy ships by state actors trying to look like ordinary commercial traffic, and misrepresenting state of shipping assets for commercial leverage in contract negoti…

I would love to read more about this. Any suggestions?

These are things you learn by doing in-depth analysis of AIS and related data sources. As far as I know, no one writes about it. Physical world data sources are full of interesting patterns and anomalies. Few people ever look because the data sources are challenging to work with and very different from Internet data, which is the only kind of data most people are used to.

Re: The lat/lon floating point delusion

#147

Earlier quoted context omitted.

There are large networks of GPS monitoring sites that regularly report their positions, including movement, down well below centimeters.

The point is about repeatability, not precision. Nominally fixed points on the Earth's surface are constantly in motion relative to GPS and relative to other fixed points. Many of the effects that cause this lack of repeatability can induce centimeters of measured deflection from the mean position over relatively short periods of time. There are models that can be employed to correct some of these effects (e.g. fluct…

I've done a fair amount of work with GPS and precise positioning, and I'm aware of all the dynamics of earth, orbit, and electron path that are involved. I've never before seen a claim of 1cm being some kind of physical limit on position.

There are both short and long-term motions of the earth. All are sufficiently measured and modeled such that repeatable measurements are very much achievable well below 1 cm.

In fact, it is GPS that is used to develop the models of earth movement. So I don't quite get what you are saying.

Re: The lat/lon floating point delusion

#148

Earlier quoted context omitted.

The point is about repeatability, not precision. Nominally fixed points on the Earth's surface are constantly in motion relative to GPS and relative to other fixed points. Many of the effects that cause this lack of repeatability can induce centimeters of measured deflection from the mean position over relatively short periods of time. There are models that can be employed to correct some of these effects (e.g. fluct…

I've done a fair amount of work with GPS and precise positioning, and I'm aware of all the dynamics of earth, orbit, and electron path that are involved. I've never before seen a claim of 1cm being some kind of physical limit on position. There are both short and long-term motions of the earth. All are sufficiently measured and modeled such that repeatable measurements are very much achievable well below 1 cm. In fac…

The centimeter limit is an engineering heuristic used by people combining high-precision GPS, LIDAR, remote sensing, and sometimes other sensors to build high-precision models of the world e.g. maps for autonomous systems. LIDAR alone can routinely detect changes in relative local surface geometry in excess of a centimeter across a day in practice. Autonomous driving typically uses an error bound of around 10 cm on absolute positioning even though the sensors used to determine that positioning often have sub-centimeter precision, for similar reasons. Real-time localization is an interesting problem because measurement with orthogonal sensor modalities can produce conflicting results with differences greater than the nominal precision of those positioning algorithms. Getting the last few centimeters of functional precision out of contradictory measurements is one of the things you need AI for.

I agree that parts of the world have repeatable measurements below 1cm if you can account for the myriad effects that cause displacement, but others do not. The models are also approximate, since some effects require real-time measurement of things we do not have real-time measurements for, or which are not practically available in contexts where local position needs to be determined.

Re: The lat/lon floating point delusion

#149
post #137

Earlier quoted context omitted.

And I'm suggesting that using the built-in floating point representation is a good thing overall, even if it doesn't allow you to specify a precision.

I'm confused by this reply. Like I said "Not at all.". But the point is, if you have some data that you want to communicate, it is also a good idea to store the precision.

It's actually very rare that you need to know the precision of a number, and it's thrown away as soon as you read the value into a standard floating point variable. Keeping the precision is usually more trouble than it's worth.

Re: The lat/lon floating point delusion

#150

Earlier quoted context omitted.

I would love to read more about this. Any suggestions?

These are things you learn by doing in-depth analysis of AIS and related data sources. As far as I know, no one writes about it. Physical world data sources are full of interesting patterns and anomalies. Few people ever look because the data sources are challenging to work with and very different from Internet data, which is the only kind of data most people are used to.

Well at this point I have terabytes of AIS data from the last few years, so if anybody is looking to dig into this, I'm glad to share.
Post reply on HN