Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse geocoding is hard

shkspr.mobi

Post reply on HN
Print view View original

51–60 of 139 posts

Re: Reverse geocoding is hard

#51

Post by tough »

pavel_lishin wrote: post #50
tough wrote: post #43

Earlier quoted context omitted.

something like https://www.pgaudit.org/ ? Basically you keep an history of all changes so you can always roll-back / get that data if needed?

The last time we did this, we basically hand-rolled our own, with a database trigger to insert data into a different table whenever an `UPDATE` statement happened. But this seems like it's probably a better solution.

never had used pgaudit yet to vouch for it but have it on the backburner/log of things to try for such a use case!

I think the real magic is it lleverages the WAL (write ahead logs) from pg engine itself, which you could certainly hook up into too, but im not a db expert here

Re: Reverse geocoding is hard

#52

Post by ryandrake »

sinuhe69 wrote: post #26

Not my area of expertise, but is this not a form of perfectionist problem? I mean, most places have a clear and simple address. For the rest, either a human can solve it, or we can make a few examples and let an AI do the work. We can go back to them later and revise them if we need to. Addresses don't change often, so I think things can stay the same for a long time. Except for emergency dispatch and a few high-prof…

You can call it perfectionism or you can call it "doing it right." I think this gets at a fundamental difference in philosophy among [software] engineers: We have a problem with a lot of edge cases, where a "good enough" solution can be done quickly. What do we do? There's a class of engineers who say 1. Do the "good enough" solution and ignore/error on the edge cases--we'll fix them later somehow (may or may not have an actual plan to do this). And there's a class of engineers who say 2. We cannot solve this problem correctly yet and need more research and better data.

Unfortunately (in my view), group #1 is making all the products and is responsible for the majority of applications of technology that get deployed. Obviously this is the case because they will take on projects that group #2 cannot, and have no compunction against shipping them. And we can see the results with our eyes. Terrible software that constantly underestimates the number and frequency of these "edge cases" and defects. Terrible software that still requires the user to do legwork in many cases because the developers made an incorrect assumption or had bad input data.

AI is making this problem even worse, because now we don't even know what the systems can and cannot do. LLMs nondeterministically fail in ways that sometimes can't even be directly corrected with code, and all engineering can do is stochastically fix defects by "training with better models."

I don't know how we get out of this: Every company is understandably biased towards "doing now" rather than "waiting" to research more and make a better product, and the doers outcompete the researchers.

Re: Reverse geocoding is hard

#53

Post by andrew_eu »

I have a memorable reverse geocoding story.

I was working with a team that was wrapping up a period of many different projects (including a reverse geocoding service) and adopting one major system to design and maintain. The handover was set to be after the new year holidays and the receiving teams had their own exciting rewrites planned. I was on call the last week of the year and got an alert that sales were halted in Taiwan due to some country code issue and our system seemed at fault. The customer facing application used an address to determine all sorts of personalization stuff: what products they're shown, regulatory links, etc. Our system was essentially a wrapper around Google Maps' reverse geocoding API, building in some business logic on top of the results.

That morning, at 3am, the API stopped serving the country code for queries of Kinmen County. It would keep the rest of the address the same, but just omit the country code, totally botching assumptions downstream. Google Maps seemingly realized all of a sudden what strait the island was in, and silently removed what some people dispute.

Everyone else on the team was on holiday and I couldn't feasibly get a review for any major mitigations (e.g. switching to OSM or some other provider). So I drew a simple polygon around the island, wrote a small function to check if the given coordinates were in the polygon, and shipped the hotfix. Happily, the whole reverse geocoding system was scrapped with a replacement by February.

Re: Reverse geocoding is hard

#54

Post by jandrewrogers »

sinuhe69 wrote: post #26

Not my area of expertise, but is this not a form of perfectionist problem? I mean, most places have a clear and simple address. For the rest, either a human can solve it, or we can make a few examples and let an AI do the work. We can go back to them later and revise them if we need to. Addresses don't change often, so I think things can stay the same for a long time. Except for emergency dispatch and a few high-prof…

The update rate for a global map data model, all of which are still woefully incomplete in many contexts, is surprisingly high. The territory underlying the map is a lot less static than people assume. Also, local reality is often much less “regular” than people assume such that a person really can’t figure it out reliably. Currently there are literally thousands of people tasked with incorporating these changes because it has proven to be resistant to automation thus far due to the pervasiveness of edge cases. For your basic global map data model, these are the edge cases that are left after several thousand heuristic and empirically derived rules have been applied.

It is a deeply complex data model that changes millions of times a day in unpredictable ways. Unfortunately, many applications are very sensitive to the local accuracy of the model, which is much higher variance than average accuracy. Only trying to be “good enough” in an 80/20 rule sense is the same as “broken”. The updates are also noisy and often contain errors, so the process has to be resilient to those errors.

The resistance of the problem to automation and the high rate of change has made it extremely expensive to asymptotically converge on model with consistently acceptable accuracy for the vast majority of applications.

Re: Reverse geocoding is hard

#55

Post by jandrewrogers »

TimTheTinker wrote: post #38
jandrewrogers wrote: post #9

Most people don’t have an intuitive sense of just how technically difficult mapping from real geospatial coordinates to feature spaces is. This is a great example of a relatively simple case. You are essentially doing inference on a sparse data model with complex local non-linearities throughout. If you add in dynamic relationships, like things that move in space, it becomes another order of magnitude worse. We frequ…

Just putting this out there. This is one area where Esri's software really shines. They have so many software offerings and so much is said about different things you can do with ArcGIS (and competing systems), but the capability of their projection engine and geocoding systems - the code that lies at its heart - is unmatched, by far, at least as of 5 years ago when I left for a different company. I had long conversa…

Many people don’t appreciate the extent that building robust geospatial systems requires seriously hardcore mathematics and physics skills. All of the mapping companies have really smart PhDs wrangling with these problems. I’ve always enjoyed talking with them about the subtleties of the challenges. There are so many nuances that never occurred to me until they mentioned them.

Re: Reverse geocoding is hard

#56

Post by Robotbeat »

jandrewrogers wrote: post #47
Dachande663 wrote: post #13

Fun fact that was dredged up because the author mentions Australia: GPS points change. Their example coordinates give 6 decimal places, accurate to about 10-15cm. Australia a few years back shifted all locations 1.8m because of continental drift they’re moving north at ~7cm/year). So even storing coordinates as a source of truth can be hazardous. We had to move several thousand points for a client when this happened.

Even accounting for tectonic drift, there is a concept of positioning reproducibility that is separate from precision. In general the precision of the measurements is much higher than the reproducibility of the same measurements. That is, you may be able to measure a fixed point on the Earth using an instrument with 1cm precision at a specific point in time but if you measure that same point every hour for a year wit…

The whole accuracy vs precision thing.

Re: Reverse geocoding is hard

#57

Post by jandrewrogers »

atoav wrote: post #19
Dachande663 wrote: post #13

Fun fact that was dredged up because the author mentions Australia: GPS points change. Their example coordinates give 6 decimal places, accurate to about 10-15cm. Australia a few years back shifted all locations 1.8m because of continental drift they’re moving north at ~7cm/year). So even storing coordinates as a source of truth can be hazardous. We had to move several thousand points for a client when this happened.

In the past year or so I have thought a lot about how to design tables and columns within databases and there is nearly nothing that wouldn't get more robust by adding in a "valid_from" and "valid_till" and make it accept multiple values. Someone's name is Foo ? What if they change it to Bar at some point and you need to access something from before with the old name? If you have only a name field that has a single v…

The tradeoff is that this is very expensive at the scale of large geospatial data models both in terms of performance and storage. In practice, it is much more common to just take regular snapshots of the database. If you want to go back in time, you have to spin-up an old snapshot of the database model.

A less obvious issue is that to make this work well, you need to do time interval intersection searches/joins at scale. There is a dearth of scalable data structures and algorithms for this in databases.

Re: Reverse geocoding is hard

#58

Post by jandrewrogers »

Robotbeat wrote: post #56
jandrewrogers wrote: post #47

Earlier quoted context omitted.

Even accounting for tectonic drift, there is a concept of positioning reproducibility that is separate from precision. In general the precision of the measurements is much higher than the reproducibility of the same measurements. That is, you may be able to measure a fixed point on the Earth using an instrument with 1cm precision at a specific point in time but if you measure that same point every hour for a year wit…

The whole accuracy vs precision thing.

Related but slightly different. The accuracy is real but it is only valid at a point in time. Consequently, you can have both high precision and high accuracy that nonetheless give different measurements depending on when the measurements were made.

In most scientific and engineering domains, a high-precision, high-accuracy measurement is assumed to be reproducible.

Re: Reverse geocoding is hard

#60

Post by meindnoch »

AlotOfReading wrote: post #22
Dachande663 wrote: post #13

Fun fact that was dredged up because the author mentions Australia: GPS points change. Their example coordinates give 6 decimal places, accurate to about 10-15cm. Australia a few years back shifted all locations 1.8m because of continental drift they’re moving north at ~7cm/year). So even storing coordinates as a source of truth can be hazardous. We had to move several thousand points for a client when this happened.

GPS coordinates actually account for the motion of the Earth's tectonic plates. The problem is that it's a highly approximate model that doesn't accurately reflect areas like Australia very well. There's a great visualizer of the coordinate velocity from the Earthscope team: https://www.unavco.org/software/visualization/GPS-Velocity-V...

>GPS coordinates actually account for the motion of the Earth's tectonic plates.

What?

Post reply on HN