Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In Defense of OpenStreetMap's Data Model

stevecoast.substack.com

Post reply on HN
Print view View original

21–30 of 130 posts

Re: In Defense of OpenStreetMap's Data Model

#21

Post by TuringTest »

RicoElectrico wrote: post #18
TuringTest wrote: post #10

Earlier quoted context omitted.

But you don't need to complicate the storage format to fix a problem like that. You can build validation tools that will check whether the stored data conforms to the correct specified geometry, and only emit valid polygons to later tools in the pipeline when they do. "Be liberal in what you accept and strict in what you send" is still a good principle. The problem with rejecting invalid structures at the data storag…

The best thing is not to allow invalid geometries to begin with. Any validation would need to be done in an off-line fashion for a number of reasons (such as needing to retrieve any referred OSM elements), and by that time you can't automatically revert offending changes as any revert carries a chance of an object version conflict.

> The best thing is not to allow invalid geometries to begin with.

The best thing for whom? The developer? Certainly not for the end user, who needs to have invalid geometries while the drawing is being made and the data is still incomplete. Having a file format that won't admit that temporary state means that either the user can't save incomplete draft work, or that an entirely different format will be needed to represent such in-process work.

The article is rightfuly critizising that such incomplete way of thinking, that doesn't take into account the full picture nor the systemic effects of a change, is pushed forwards only because they seem "the right thing" from an incomplete understanding of all the concerns and the needs from all stakeholders.

The right technical decision *must* include them to be correct, and the best design might involve a solution other than "update the file format so that it doesn't accept inconsistent geometry (acording to the set of rules that we understand as of today)". But to assess what the right decision is, you need to know how people is using the system in real use-cases beyond classic comp-sci concerns of data storage and model consistency; and to learn those, you need to talk to end users and perform field research to inform your decisions and designs.

Re: In Defense of OpenStreetMap's Data Model

#22

Post by TuringTest »

maxerickson wrote: post #19
TuringTest wrote: post #10

Earlier quoted context omitted.

But you don't need to complicate the storage format to fix a problem like that. You can build validation tools that will check whether the stored data conforms to the correct specified geometry, and only emit valid polygons to later tools in the pipeline when they do. "Be liberal in what you accept and strict in what you send" is still a good principle. The problem with rejecting invalid structures at the data storag…

The "expression" layer of the data model has had 20 years to evolve and has largely been static for a decade. Making everything slower and harder to retain flexibility you don't need isn't a great tradeoff.

Why do you need to change the data format to make it faster (at the cost of making it harder to work with to end users)? The data is the same as it was at the beginning, it doesn't justify a technical redesign. Why not just create accelerators based on an intermediat format?

Re: In Defense of OpenStreetMap's Data Model

#23

Post by TuringTest »

londons_explore wrote: post #16

The exact same argument that praises OSM's super flexible tagged node data model should also praise MS Excel for the number of things that can be achieved in the world of business with just a grid of boxes. Both have been hugely successful, and both have the same pile of downsides.

> Both have been hugely successful, and both have the same pile of downsides.

Exactly. And the solution should not be to throw away spreadsheets completely and turn them into relational databases, but to create new tools to alleviate the downsides and reduce their impact (possible by exporting the spreadsheet information into a relational database, but without taking away the user's option to continue working with it.)

Re: In Defense of OpenStreetMap's Data Model

#24

Post by zigzag312 »

Author states:

>And that’s the point, rules and complexity have completely unknowable downsides. Downsides like the destruction of the whole project. With each rule and added complexity you make the system less human and less fun. You make it a Computer Scientists rube goldberg machine while sterilizing it of all the joy of life.

While too much rules and complexity can certainly be bad, some basic amount of standardization can actually reduce complexity and really doesn't cause a "destruction of the whole project".

As a counterpoint, too much flexibility can also increase complexity. For example, without defined rules, 5.6.2022 can mean 5. June 2022 or 6. May 2022. Nor user, nor parser can know for sure what it means, if standard isn't defined. This kind of flexibility certainly isn't fun.

Example from OSM wiki for "Key:source:date":

> There is no standing recommendation as to the date format to be used. However, the international standard ISO 8601 appears to be followed by 9 of the top 10 values for this tag. The ISO 8601 basic date format is YYYY-MM-DD. https://wiki.openstreetmap.org/wiki/Key:source:date

Just define some essential standards. It won't lead to destruction of the project!

And while you are making breaking changes, please fix the 'way' element. Maps are big. Storing points in ways as 64bit node-ids, while coordinates in nodes are also 64-bit (32bit lon and 32bit lat), just leads to wasted space and wasted processing time. There are billions of these nodes and nearly all of these nodes don't have tags, just coordinates. There is no upside for this level of indirection. And in case tags are needed for a point, this can already be solved with a separate node and a 'relation' element.

OSM data format could certainly be improved and it would benefit end users, as better tools/apps could be made more quickly and easily.

Re: In Defense of OpenStreetMap's Data Model

#25

Post by purple_turtle »

TuringTest wrote: post #21
RicoElectrico wrote: post #18

Earlier quoted context omitted.

The best thing is not to allow invalid geometries to begin with. Any validation would need to be done in an off-line fashion for a number of reasons (such as needing to retrieve any referred OSM elements), and by that time you can't automatically revert offending changes as any revert carries a chance of an object version conflict.

> The best thing is not to allow invalid geometries to begin with. The best thing for whom? The developer? Certainly not for the end user, who needs to have invalid geometries while the drawing is being made and the data is still incomplete . Having a file format that won't admit that temporary state means that either the user can't save incomplete draft work, or that an entirely different format will be needed to re…

> Having a file format that won't admit that temporary state means that either the user can't save incomplete draft work, or that an entirely different format will be needed to represent such in-process work.

Saving such temporary state is very rarely needed in OSM and should be never uploaded to the OSM database.

In addition, in almost all cases it can be simply saved as area of shape that is not yet matching intended one.

Re: In Defense of OpenStreetMap's Data Model

#26

Post by maxerickson »

TuringTest wrote: post #22
maxerickson wrote: post #19

Earlier quoted context omitted.

The "expression" layer of the data model has had 20 years to evolve and has largely been static for a decade. Making everything slower and harder to retain flexibility you don't need isn't a great tradeoff.

Why do you need to change the data format to make it faster (at the cost of making it harder to work with to end users)? The data is the same as it was at the beginning, it doesn't justify a technical redesign. Why not just create accelerators based on an intermediat format?

I guess I don't follow your analysis.

People doing mapping tasks will use an editor and not really see the change.

People consuming the data will also mostly use tools, tools that likely run much faster.

I've written some code to chop up overlapping gis areas into ways and relations (to match the current data model of references to shared nodes). The input to that code is pretty close to the proposed data model, so not going to be more difficult to do that processing (as an example of a task that doesn't just use 3rd party tools).

Re: In Defense of OpenStreetMap's Data Model

#27

Post by matkoniecz »

RicoElectrico wrote: post #14
tinus_hn wrote: post #12

What is stopping users who have a problem with the model from transforming the data into a form that is better for their use case?

It is surprisingly difficult to say which closed ways are areas and which are not. This depends entirely on tags of the way and is only solved by heuristics. https://github.com/tyrasd/osm-polygon-features

In addition, it is common to have objects that are both area and line at once. Or area according to one tool/map/edtor and line according to another.

And many, many multipolygon relations are in inconsistent state and require manual fixup.

Also, complexity of entire area baggage makes explaining things to newbies more complex. You can either try to hide complexity (used by iD in-browser-editor) leaving people hopelessly confused when things are getting complex or present full complexity (JOSM) causing people to be overwhelmed.

See https://wiki.openstreetmap.org/wiki/Area#Tags_implying_area_... for a start of a complexity fractal.

Re: In Defense of OpenStreetMap's Data Model

#28

Post by matkoniecz »

> The harder you make it for them to edit, the less volunteers you’ll get.

And that is why dedicated area type (rather than representing areas with lines or special relations[0]) could help new mappers and new users of data.

There would be very significant transition costs, but maybe it would be overall beneficial.

It is possible to have objects that are both area and line at once. Or area according to one tool/map/edtor and line according to another.

And many multipolygon relations are in inconsistent state and require manual fixup.

Also, complexity of entire area baggage makes explaining things to newbies more complex. You can either try to hide complexity (used by iD in-browser-editor) leaving people hopelessly confused when things are getting complex or present full complexity (JOSM) causing people to be overwhelmed.

See https://wiki.openstreetmap.org/wiki/Area#Tags_implying_area_... for a start of a complexity fractal.

[0] https://wiki.openstreetmap.org/wiki/Area

Re: In Defense of OpenStreetMap's Data Model

#29

Post by TuringTest »

purple_turtle wrote: post #25
TuringTest wrote: post #21

Earlier quoted context omitted.

> The best thing is not to allow invalid geometries to begin with. The best thing for whom? The developer? Certainly not for the end user, who needs to have invalid geometries while the drawing is being made and the data is still incomplete . Having a file format that won't admit that temporary state means that either the user can't save incomplete draft work, or that an entirely different format will be needed to re…

> Having a file format that won't admit that temporary state means that either the user can't save incomplete draft work, or that an entirely different format will be needed to represent such in-process work. Saving such temporary state is very rarely needed in OSM and should be never uploaded to the OSM database. In addition, in almost all cases it can be simply saved as area of shape that is not yet matching intend…

> Saving such temporary state is very rarely needed in OSM and should be never uploaded to the OSM database.

Maybe, but you're missing the other use case - that in the future you'll need an extension requiring geometries that are considered invalid by the current set of rules, forcing you to update all tools processing the file format to acommodate the new extension.

Keeping storage and validation as two separate steps is a more flexible design, preferable on platforms where data is entered by a large number of users in a complex domain that is not easy to model inambiguously.

Think of Wikipedia and what would have happened if its text format had only supported grammatically correct expressions without spelling mistakes, and without letting you save templates with any errors. The project would never have attracted the volume of editors it took to create the initial version with millions of articles, and the product would never have taken off. In an open project with data provided by the general public, keeping user data validation in the same layer as the automatic processing model is a design mistake.

Re: In Defense of OpenStreetMap's Data Model

#30

Post by matkoniecz »

TuringTest wrote: post #10
RicoElectrico wrote: post #8

The proposed improvements would obsolete a bunch of problems such as broken polygons [1] which happen regularly. They would also make processing OSM more accessible without needing to randomly seek over GBs of node locations just to assemble geometries which takes a significant runtime percentage of osm2pgsql. For me Steve Coast lost his credibility when he joined the closed and proprietary what3words. [1] https://wi…

But you don't need to complicate the storage format to fix a problem like that. You can build validation tools that will check whether the stored data conforms to the correct specified geometry, and only emit valid polygons to later tools in the pipeline when they do. "Be liberal in what you accept and strict in what you send" is still a good principle. The problem with rejecting invalid structures at the data storag…

> You can build validation tools that will check whether the stored data conforms to the correct specified geometry, and only emit valid polygons to later tools in the pipeline when they do.

It is not helping at all when the problem is that important areas disappeared.

It is also not helping at all other mappers or confused newbie.

Post reply on HN