Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In Defense of OpenStreetMap's Data Model

stevecoast.substack.com

Post reply on HN
Print view View original

111–120 of 130 posts

Re: In Defense of OpenStreetMap's Data Model

#111

Post by firen777 »

jasonwatkinspdx wrote: post #105
RicoElectrico wrote: post #8

The proposed improvements would obsolete a bunch of problems such as broken polygons [1] which happen regularly. They would also make processing OSM more accessible without needing to randomly seek over GBs of node locations just to assemble geometries which takes a significant runtime percentage of osm2pgsql. For me Steve Coast lost his credibility when he joined the closed and proprietary what3words. [1] https://wi…

What 3 Words makes me so angry. There's around 5.1e14 meters squared on the surface of earth. It takes 34 bits to address this uniquely. If we use one of EFF's dice words style short word lists (6^4 words), we need 5 words to describe any point on earth with 1 meter precision. If we use a projection like say S2 (though plenty of other options exist), these 5 word locators will show strong hierarchical locality. In an…

This comment thread is the first time I hear about w3w. It hurts my brain trying to come up a reasoning how such concept is not some kind of parody one-off project intended to be posted on HN or reddit for the lolz. Instead, it is actually being used by the emergency service?

Trying to google with the query "what3words explained site:reddit.com" gave me this r/911dispatchers post as the first result: [What3Words and why it's trash.](https://www.reddit.com/r/911dispatchers/comments/olcxdv/what...)

(Amusingly, this 10 months old post was last edited 2 days ago.)

Re: In Defense of OpenStreetMap's Data Model

#112

Post by everybodyknows »

Sujan wrote: post #6

I think those are the important bits: > The Engineering Working Group (EWG) of the OSMF has “commissioned” (I think that’s OSMF language for paid) a longstanding proponent of rules and complexity to, uh, investigate how to add rules and complexity to OSM. > [...] > Let us pray that the EWG is just throwing Jochen a bone to go play in the corner and stop annoying the grownups. It's a "response" to https://blog.openstr…

This seems an important bit to me:

> Facebook solved this in a beautifully OSM-like way: daylight. Daylight is a sanitized, consistent and cleaned up map based on OSM

https://daylightmap.org/ https://registry.opendata.aws/daylight-osm/#usageexamples https://gist.github.com/jenningsanderson/3e42a99dcb8f760038a...

Re: In Defense of OpenStreetMap's Data Model

#113

Post by TuringTest »

seoaeu wrote: post #54
TuringTest wrote: post #10

Earlier quoted context omitted.

But you don't need to complicate the storage format to fix a problem like that. You can build validation tools that will check whether the stored data conforms to the correct specified geometry, and only emit valid polygons to later tools in the pipeline when they do. "Be liberal in what you accept and strict in what you send" is still a good principle. The problem with rejecting invalid structures at the data storag…

> "Be liberal in what you accept and strict in what you send" is still a good principle. No, it is a terrible principle which produces brittle software and impossible to implement standards. The problem is that no one actually follows the “be strict in what you send” part, and just goes with whatever cobbled together mess the other existing software seems to accept. Before long, a spec compliant implementation can’t…

> This sounds like another headache. I don’t want my tools silently breaking.

Yet here you are, posting your comment through a web browser on a web page. And the new standard that was intended to make web pages display catastrophic failure and stop processing with each error (XHTML) was never widely adopted. Makes you wonder why? Maybe the nature of an open data platform for human consumption has something inherent to it so that it's better to accept a certain degree of inaccuracy and inconsistencies in its stored data?

Re: In Defense of OpenStreetMap's Data Model

#114

Post by TuringTest »

danShumway wrote: post #73
TuringTest wrote: post #10

Earlier quoted context omitted.

But you don't need to complicate the storage format to fix a problem like that. You can build validation tools that will check whether the stored data conforms to the correct specified geometry, and only emit valid polygons to later tools in the pipeline when they do. "Be liberal in what you accept and strict in what you send" is still a good principle. The problem with rejecting invalid structures at the data storag…

I don't have horribly strong opinions here, but the argument feels circular to me: - The format should be kept simple to encourage more people to build tools on top of it, and users will be more likely to work with it. - We should deal with the emergent complexity of bad validation by making tools more complicated and having them detect errors on their end. If users are going to use a validation tool to work with dat…

> If users are going to use a validation tool to work with data, then they can also use a helper tool to generate data. And if the goal is to make it easier to build on top of data, import it, etc... allowing developers to do less work validating everything makes it easier for them to build things.

That's good thinking for cases where you have a single toolset, in which tools can be kept in sync to collaborate with one another.

But in an open distributed data platform, where several possibly incompatible toolsets will be used, forcing a type of validation on the data itself based on the expectation of one group of tools can make some other applications impossible. In these cases, making the data format simple will make it easier to developers to build new tools, and the difficulties of synchronizing different tools may be dealt with in a different layer.

Re: In Defense of OpenStreetMap's Data Model

#115

Post by TuringTest »

Archelaos wrote: post #62
TuringTest wrote: post #29

Earlier quoted context omitted.

> Saving such temporary state is very rarely needed in OSM and should be never uploaded to the OSM database. Maybe, but you're missing the other use case - that in the future you'll need an extension requiring geometries that are considered invalid by the current set of rules, forcing you to update all tools processing the file format to acommodate the new extension. Keeping storage and validation as two separate ste…

> Maybe, but you're missing the other use case - that in the future you'll need an extension requiring geometries that are considered invalid by the current set of rules, forcing you to update all tools processing the file format to acommodate the new extension. I think the way to go is to define several layers of correctness. A data set might then be partially valid. In such cases a tool might, for example, support…

> I think the way to go is to define several layers of correctness. A data set might then be partially valid

Thanks, that summarizes what I was aiming for. An open platform will be more flexible and and allow for different use cases the fewer assumptions about how it should be used it includes.

Re: In Defense of OpenStreetMap's Data Model

#116

Post by vjk800 »

jasonwatkinspdx wrote: post #105
RicoElectrico wrote: post #8

The proposed improvements would obsolete a bunch of problems such as broken polygons [1] which happen regularly. They would also make processing OSM more accessible without needing to randomly seek over GBs of node locations just to assemble geometries which takes a significant runtime percentage of osm2pgsql. For me Steve Coast lost his credibility when he joined the closed and proprietary what3words. [1] https://wi…

What 3 Words makes me so angry. There's around 5.1e14 meters squared on the surface of earth. It takes 34 bits to address this uniquely. If we use one of EFF's dice words style short word lists (6^4 words), we need 5 words to describe any point on earth with 1 meter precision. If we use a projection like say S2 (though plenty of other options exist), these 5 word locators will show strong hierarchical locality. In an…

> There is no reason a scheme like this can't simply be a standard from some appropriate body, and a few open source reference implementations.

Yet no-one did this and I think that's the point here.

World is full of rent seeking in the form of stuff that is dead simple to do but no-one does without a financial incentive.

In w3w the hard part is not the system itself, but getting people to use it, which must be done because the value of the system comes from the network effect.

Re: In Defense of OpenStreetMap's Data Model

#117

Post by npteljes »

chaps wrote: post #44

Heh, their data model is 99% of the reason why I don't use OSM. It's scattered all over the place with so many tables ! It's such a nice project, but damn is it impossible to work with programmatically, let alone poke around it to discover what's all in there.

Would the Overpass API (have) fit your use case?

Re: In Defense of OpenStreetMap's Data Model

#118

Post by TuringTest »

speed_spread wrote: post #40
TuringTest wrote: post #22

Earlier quoted context omitted.

Why do you need to change the data format to make it faster (at the cost of making it harder to work with to end users)? The data is the same as it was at the beginning, it doesn't justify a technical redesign. Why not just create accelerators based on an intermediat format?

Properly normalized data isn't just faster, it's also easier to work with for the end user. There are much less exceptions, edge cases and snafus to work around and test for. If you're talking about the transition period between formats, well yeah, you're gonna see things breaking. But these were already broken, just not in apparent ways. In the end, everybody wins.

> Properly normalized data isn't just faster, it's also easier to work with for the end user.

Extracting and reusing data, yes. Getting it into the tool in the first place, no way. Tools that won't even allow you to save your data and make it persistent until you conform to every single integrity requirement are a nightmare for end users.

Re: In Defense of OpenStreetMap's Data Model

#119

Post by TuringTest »

seoaeu wrote: post #56
TuringTest wrote: post #7

Earlier quoted context omitted.

> OSM stores maps as graphs, in flat files where each line is either a node, an ordered list of nodes, or metadata. The graph nodes can be arbitrarily ordered in OSM files, which leads to computational complexity when parsing them. This is not a bad thing, since it means that the spec for OSM files can be extremely simple, which makes it easy for people to contribute to OSM. That's actually a sensible design. Treat u…

Wait, the proposed solution to a data format being slow to parse is to work around the bad performance by caching the already parsed representation? That seems like it has a clear flaw if you’re only accessing the data once…

Where's the flaw in that? If you're only accessing the data once, why does it matter how fast or slow it is?

And, you're suggesting that user-facing data should be harder to work with only to make it faster to parse?

Re: In Defense of OpenStreetMap's Data Model

#120

Post by danShumway »

TuringTest wrote: post #114
danShumway wrote: post #73

Earlier quoted context omitted.

I don't have horribly strong opinions here, but the argument feels circular to me: - The format should be kept simple to encourage more people to build tools on top of it, and users will be more likely to work with it. - We should deal with the emergent complexity of bad validation by making tools more complicated and having them detect errors on their end. If users are going to use a validation tool to work with dat…

> If users are going to use a validation tool to work with data, then they can also use a helper tool to generate data. And if the goal is to make it easier to build on top of data, import it, etc... allowing developers to do less work validating everything makes it easier for them to build things. That's good thinking for cases where you have a single toolset, in which tools can be kept in sync to collaborate with o…

> That's good thinking for cases where you have a single toolset, in which tools can be kept in sync to collaborate with one another.

This is interesting. I would actually kind of argue the exact opposite, that more rigorously defined formats are more important the more diverse your toolsets get, and less important the less diverse they are.

The whole point of having a rigorously defined data format that blocks certain validation errors at the data level is that it's easier for diverse toolsets to work with that data, because they don't need to all implement their own validators, and they don't need to worry as much about other tools accidentally sending them malformed/broken data.

> making the data format simple

I think where we might be disagreeing is that I argue more specific data formats that inherently block validation errors are simpler than vague formats where there are restrictions and errors you can make, but those restrictions aren't clearly documented and aren't obvious until after you try to import the data.

I would point to something like the Matrix specification -- they have put comparatively more work into making sure that the Matrix specification (while flexible) is consistent, they don't want clients randomly making a bunch of changes or assumptions about the data format. That's partially inspired by looking back at standards like Jabber and seeing that having a lack of consensus about data formats caused tools to become extremely fragmented and hard to coordinate with each other. See https://news.ycombinator.com/item?id=17064616 for more information on that.

My feeling is that when you introduce validation layers, you have not actually gotten rid of restrictions between user applications, and you have not actually made coordination simpler, because different tools are going to break when they see pieces of data that they consider invalid or that they didn't realize they needed to be able to handle. All that's really happened is that complexity has been moved into the individual applications and that logic has been duplicated across a bunch of different apps.

In contract, when every single tool is speaking the same language and agrees what is and isn't valid data, then it's very fast to build tools that you know will be compatible with everything else in the ecosystem.

Post reply on HN