Live data from Hacker News

The hunt for the missing data type

hillelwayne.com

131–140 of 259 posts

Re: The hunt for the missing data type

#131

What an awesome article. Kudos to the author! On the core observation "there are too many implementation choices", that is not quite right. True, the author mentions 4, and there are further variations. In practice, a library can: 1. Implement all suitable graph representations. 2. Implement algorithms tailored to the representation(s) that offer the highest performance. 3. Provide transformations from one representa…

Sounds like the makings of a huge library that I’m not sure I’d even use in my work. I use graphs heavily in my work, and my experience matches the people the author interviewed.

We always end up reimplementing graphs because:

- Performance matters, and no off the shelf graph library I’ve seen can take advantage of many of the regularities in our particular data set. (We have an append-only DAG which we can internally run-length encode because almost all nodes just have an edge pointing to the last added item).

- I haven’t seen any generic graph library which supports the specific queries I need to make on my graphs. The big one is a subgraph diffing function.

- Writing something custom just isn’t much work anyway! Graphs are way simpler to reimplement than btrees. You can have a simple graph implementation in tens of lines. Our highly optimised library - with all the supporting algorithms - is still only a few hundred lines of code.

I think it would be handy to have ways to export the data into some standard format. But eh. I think pulling a library in for our use case would add more problems than it would solve.

Re: The hunt for the missing data type

#133

As someone who did a lot of work with graphs, "why don't programming languages have a built-in graph data type?" is a question I’ve been asked a million times. I'm thrilled I'll be able to point folks to a much more in-depth analysis like the one here, instead of saying some variation of "it's really hard to do it well" and having them just take my word for it.

> "it's really hard to do it well" More importantly, there are a lot of tradeoffs. Virtually every language offers a hash map. You can roll your own to outperform in an individual circumstance, the default works pretty well. You can't really do that with a graph. Maybe if you offered a bunch of graph types. --- PS. Bit of trivia: Java's HashMap is a bit different from almost every other language in that it lets you t…

It's also different in that each hash bucket can use a Red-Black tree when there's many entries.

Re: The hunt for the missing data type

#134
post #14

Earlier quoted context omitted.

I used to think that since graphs are such a broad data structure that can be represented in different ways depending on requirements that it just made more sense to implement them at a domain-ish level (the article mentions this in the "There are too many implementation choices" section). Then I saw Petgraph [0] which is the first time I had really looked at a generic graph library. It's very interesting, but I stil…

Why wouldn't an abstract `Graph` type and specific implementations of that work? Like Java has for `Set` and various implementations?

even at that level of abstraction there are many flavors of "graphs". what some people call graph are actually hypergraphs, or attributed graphs.

Re: The hunt for the missing data type

#135

Graphviz has its own foundation graph library, that's not used by any other project. It has its good and bad points. Based on that experience, we had our very own second-system-syndrome experience. We decided our graph library should be modular, type safe, and efficient. (These properties came up in the comments here, too.) This is probably just a variation of "good, fast, cheap - pick any two." By modular, want to w…

[deleted]

Re: The hunt for the missing data type

#137

I've often wondered about a missing application: "Excel for graphs". Just like Excel for tabular data, it would support RAM-sized data (enough to require a computer, but not so much that you need a data center), implement lots of algorithms and visualizations "well enough", and require no programming skill to operate. As the article says, a lot of our real-world problems are graph problems - why are programmers the o…

Yeah, I feel like the article is too quick with its conclusions. Many other problems can be made arbitrary complex and difficult with additional requirements. But there are still data structure and standard libraries to provide good enough experience that fits most use-cases. And if you need something extra spicy you need to build a custom solution.

The article claims that graphs are often just too big, but yeah, if you ask people who are actively working on graph algorithms they might have that sort of experience. But most programmers and users probably only work with really small graphs.

Re: The hunt for the missing data type

#138

Earlier quoted context omitted.

> "it's really hard to do it well" More importantly, there are a lot of tradeoffs. Virtually every language offers a hash map. You can roll your own to outperform in an individual circumstance, the default works pretty well. You can't really do that with a graph. Maybe if you offered a bunch of graph types. --- PS. Bit of trivia: Java's HashMap is a bit different from almost every other language in that it lets you t…

> You can't really do that with a graph. Maybe if you offered a bunch of graph types. And so why isn't this the solution? Most languages support both hash map (fast lookup) and balanced tree (ordered entries) primitives, even though they both implement the "associative map" container type. Can't we have 2, 3, 5, or even 8 different graph types?

> Can't we have 2, 3, 5, or even 8 different graph types?

That’s what graph libraries do, look at petgraph. You’ve got a bunch of graph implementations, and a bunch of algorithms over them.

Re: The hunt for the missing data type

#139
I’ve been building an interactive fiction platform built in C# with a world model contained within a bidirectional graph data structure.

Since IF stories are relatively small in graph terms, it’s a reasonable solution.

Locations and objects are nodes and movement and location of objects are edges. Nodes and edges both can have dynamic properties.

I’ve also noticed the lack of graph structures in programming languages, so this article was very enlightening.

Re: The hunt for the missing data type

#140
post #72

Graph drawing tools are also very underwhelming, they work pretty good for small graphs until you have something like 500 nodes or more, then eventually their output becomes complete incompressible or very difficult to look at it, they miss the ability to automatically organize those graph in hierarchical structures and provide a nice interface to explore them, we are used that everything around us have some kind of…

>Graph drawing tools

It's hard

Graphviz-like generic graph-drawing library. More options, more control.

https://eclipse.dev/elk/

Experiments by the same team responsible for the development of ELK, at Kiel University

https://github.com/kieler/KLighD

Kieler project wiki

https://rtsys.informatik.uni-kiel.de/confluence/display/KIEL...

Constraint-based graph drawing libraries

https://www.adaptagrams.org/

JS implementation

https://ialab.it.monash.edu/webcola/

Some cool stuff:

HOLA: Human-like Orthogonal Network Layout

https://ialab.it.monash.edu/~dwyer/papers/hola2015.pdf

Confluent Graphs demos: makes edges more readable.

https://www.aviz.fr/~bbach/confluentgraphs/

Stress-Minimizing Orthogonal Layout of Data Flow Diagrams with Ports

https://arxiv.org/pdf/1408.4626.pdf

Improved Optimal and Approximate Power Graph Compression for Clearer Visualisation of Dense Graphs

https://arxiv.org/pdf/1311.6996v1.pdf

Post reply on HN