Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse geocoding is hard

shkspr.mobi

Post reply on HN
Print view View original

31–40 of 139 posts

Re: Reverse geocoding is hard

#31

Post by curiousObject »

edent wrote: post #30
sinuhe69 wrote: post #26

Not my area of expertise, but is this not a form of perfectionist problem? I mean, most places have a clear and simple address. For the rest, either a human can solve it, or we can make a few examples and let an AI do the work. We can go back to them later and revise them if we need to. Addresses don't change often, so I think things can stay the same for a long time. Except for emergency dispatch and a few high-prof…

I am deeply guilty of being a perfectionist! Ultimately, I just want something which is a nice balance between being useful for a human and not so long that it is overwhelming.

You’re the author?

The final step in the process “Wait for complaints” seems like a smart acceptance of the “perfect is the enemy of good” challenge

Publish and be damned, or as we say now: Move fast and break things

Re: Reverse geocoding is hard

#32

Post by nerdralph »

edent wrote: post #15
andrewaylett wrote: post #5

It's a lot more expensive, but measuring navigation distance rather than straight line distance would avoid the "river" issue. Although depending on the routing engine and dataset it might well introduce more issues where points can be really close on foot but the only known route is a driving route.

If you know of an API which does navigation distance to POI, I'd love to hear about it!

I've used OSRM and Arcgis for addresses in Canada. I think one or both of them have POI support in their APIs. https://route.arcgis.com/arcgis/

Re: Reverse geocoding is hard

#33

Post by blacklight »

As the developer of a GPS tracking app that relies a lot on OpenStreetMap, I've faced many of these problems myself. A couple of learned lessons/insights:

- I avoid relying on any generic location name/description provided by these APIs. Always prefer structured data whenever possible, and build the locality name from those components (bonus points if you let the user specify a custom format).

- Identifying those components itself is tricky. As the author mentioned, there are countries that have states, others that have regions, other that have counties, or districts, or any combination of those. And there are cities that have suburbs, neighbourhoods, municipalities, or any combination. Oh, and let's not even get started with address names - house numbers? extensions? localization variants - e.g. even the same API may sometimes return "Marrakesh" and sometimes "Marrakech"? and how about places like India where nearby amenities are commonly used instead of house numbers? I'm not aware of any public APIs out there that provide these "expected" taxonomies, preferably from lat/long input, but I'd love to be proven wrong. In the absence of that, I would suggest that is better to avoid double-guessing - unless your software is only intended to run in a specific country, or in a limited number of countries and you can afford to hardcode those rules. It's probably a good option to provide a sensible default, and then let the user override it. Oh, and good catch about abbreviations - I'd say to avoid them unless the user explicitly enables them, if you want to avoid the "does everybody know that IL is Illinois?" problem. Just use "Illinois" instead, at least by default.

- Localization of addresses is a tricky problem only on the surface. My proposed approach is that, again, the user is king. Provide English by default (unless you want to launch your software in a specific country), and let the user override the localization. I feel like the Nominatim's API approach is probably the cleanest: honor the `Accept-Language` HTTP header if available, and if not available, fallback to English. And then just expose that a setting to the user.

- Bounding boxes/polygons can help a lot with solving the proximity/perimeter issue. But they aren't always present/sufficiently accurate in OSM data. And their proper usage usually requires the client's code to run some non-trivial lat/long geometry processing code, even to answer trivial questions such as "is this point inside of this enclosed amenity?" Oh, and let's not even get started with the "what's the exact lat/long of this address?" problem. Is it the entrance of the park? The middle of it? I remember that when I worked with the Bing in the API in the past they provided more granular information at the level of rooftop location, entrance location etc.

- Providing localization information for public benches isn't what I'd call an orthodox use-case for geo software, so I'm not entirely sure of how to solve the "why doesn't everything have an address?" problem :)

Re: Reverse geocoding is hard

#34

Post by punnerud »

davidmurdoch wrote: post #23
punnerud wrote: post #17

I created this to solve my own need for reverse geocoding: https://github.com/punnerud/rgcosm (Saving me thousands of $ compared to Google API) Uses OpenStreetmap file, Python and SQLite3. First it finds all addresses using +/- like a square from lat/lon, then calculate distance based on the smaller list (Pythagoras), and pick the closest. It expands until a set maximum if no address is found in the first search.

Just curious if you looked into using S2 cells for this? It's what Pokemon Go uses for its coordinate system. http://s2geometry.io/devguide/s2cell_hierarchy.html

Isn’t the main purpose of S2 to be able to scan from different “directions”? More a purpose of Google Maps when viewing the world as a spherical object compared to Sqlite3 just using a simple B-tree index on lat+lon?

Re: Reverse geocoding is hard

#35

Post by pavel_lishin »

Dachande663 wrote: post #13

Fun fact that was dredged up because the author mentions Australia: GPS points change. Their example coordinates give 6 decimal places, accurate to about 10-15cm. Australia a few years back shifted all locations 1.8m because of continental drift they’re moving north at ~7cm/year). So even storing coordinates as a source of truth can be hazardous. We had to move several thousand points for a client when this happened.

Damn! 7cm per year feels blazing fast when you consider the fact that it's a whole continent.

Re: Reverse geocoding is hard

#36

Post by pavel_lishin »

atoav wrote: post #19
Dachande663 wrote: post #13

Fun fact that was dredged up because the author mentions Australia: GPS points change. Their example coordinates give 6 decimal places, accurate to about 10-15cm. Australia a few years back shifted all locations 1.8m because of continental drift they’re moving north at ~7cm/year). So even storing coordinates as a source of truth can be hazardous. We had to move several thousand points for a client when this happened.

In the past year or so I have thought a lot about how to design tables and columns within databases and there is nearly nothing that wouldn't get more robust by adding in a "valid_from" and "valid_till" and make it accept multiple values. Someone's name is Foo ? What if they change it to Bar at some point and you need to access something from before with the old name? If you have only a name field that has a single v…

If you have an "audit" table, where you write a copy of the data before updating it in the primary table, that's a decision you can make at any point.

Of course, you don't get that historical data, but you do get it going forward from there.

Re: Reverse geocoding is hard

#37

Post by davidmurdoch »

punnerud wrote: post #34
davidmurdoch wrote: post #23

Earlier quoted context omitted.

Just curious if you looked into using S2 cells for this? It's what Pokemon Go uses for its coordinate system. http://s2geometry.io/devguide/s2cell_hierarchy.html

Isn’t the main purpose of S2 to be able to scan from different “directions”? More a purpose of Google Maps when viewing the world as a spherical object compared to Sqlite3 just using a simple B-tree index on lat+lon?

The individual cells being sized and being able to easily to compute neighboring cells seems useful for the described algorithm. I haven't given it much thought on applicability here, but it sounded somewhat similar to a search pattern I once implemented within pgsql to locate items on a map that were within proximity of a given latlong.

Re: Reverse geocoding is hard

#38

Post by TimTheTinker »

jandrewrogers wrote: post #9

Most people don’t have an intuitive sense of just how technically difficult mapping from real geospatial coordinates to feature spaces is. This is a great example of a relatively simple case. You are essentially doing inference on a sparse data model with complex local non-linearities throughout. If you add in dynamic relationships, like things that move in space, it becomes another order of magnitude worse. We frequ…

Just putting this out there. This is one area where Esri's software really shines. They have so many software offerings and so much is said about different things you can do with ArcGIS (and competing systems), but the capability of their projection engine and geocoding systems - the code that lies at its heart - is unmatched, by far, at least as of 5 years ago when I left for a different company.

I had long conversations with Esri's projection engine lead. Really remarkable guy - he's got graduate degrees in geography and math (including a PhD) and he's an excellent C/C++ developer. That kind of expertise trifecta is rare. I'd walk by his office and sometimes see him working out an integral of a massive equation on his whiteboard (not that he didn't also use a CAS). "Oh yeah, I'm adding support for a new projection this week."

Re: Reverse geocoding is hard

#39

Post by nerdralph »

Part of the problem is the different ways addresses are expressed throughout the world. I was born and grew up in Canada, and was confused when I started dealing with companies in China. Instead of street addresses, many are given by province, city, district, sub-district, and a building number.

Another problem is choosing which authority for the "correct" address. I've seen many cases where the official postal address city/town name is different than the 911 database. For example Canada Post will say some street addresses are in Dartmouth, while the official civic address is really Cole Harbour. https://www.canadapost-postescanada.ca/ac/ https://nsgi.novascotia.ca/civic-address-finder/

Even streets can have multiple official names/aliases. People who live on "East Bay Hwy", also live on "Highway 4", which is an alias.

Re: Reverse geocoding is hard

#40

Post by haneefmubarak »

xucheng wrote: post #29
Dachande663 wrote: post #13

Fun fact that was dredged up because the author mentions Australia: GPS points change. Their example coordinates give 6 decimal places, accurate to about 10-15cm. Australia a few years back shifted all locations 1.8m because of continental drift they’re moving north at ~7cm/year). So even storing coordinates as a source of truth can be hazardous. We had to move several thousand points for a client when this happened.

Can this be solved by storing a timestamp of the record along with precise GPS coordinates? Could we then utilize some database to compute the drift from then and now?

I mean, certainly - if you store both GPS time and derived coordinates from the same sampling, then you can always later interpret it as needed - whether relative to legal or geographical boundaries etc as you might want to interpret in the future.
Post reply on HN