Live data from Hacker News

The hunt for the missing data type

hillelwayne.com

251–259 of 259 posts

Re: The hunt for the missing data type

#251
post #108

> There’s a gap between how often software engineers could use graphs and how little our programming ecosystems support them. Where are all the graph types? They've been there for quite a while :-) https://www.erlang.org/doc/man/digraph.html https://www.erlang.org/doc/man/digraph_utils And if you want to do some set theoretical stuff you're covered as well: https://www.erlang.org/doc/man/sofs.html

Elixir has a pretty nice graph library as well: https://hexdocs.pm/libgraph/api-reference.html

I've used it to do some dependency resolution for operation ordering.

Re: The hunt for the missing data type

#252
post #108

> There’s a gap between how often software engineers could use graphs and how little our programming ecosystems support them. Where are all the graph types? They've been there for quite a while :-) https://www.erlang.org/doc/man/digraph.html https://www.erlang.org/doc/man/digraph_utils And if you want to do some set theoretical stuff you're covered as well: https://www.erlang.org/doc/man/sofs.html

Elixir has a pretty nice graph library as well: https://hexdocs.pm/libgraph/api-reference.html I've used it to do some dependency resolution for operation ordering.

Thanks for sharing. That looks well done. It has some pathfinding algorithms and the reducer is very neat for traversing the graph. I like that it's based on maps, which could make it more performant than the ets based erlang digraph.

Re: The hunt for the missing data type

#253

Earlier quoted context omitted.

Sounds like the makings of a huge library that I’m not sure I’d even use in my work. I use graphs heavily in my work, and my experience matches the people the author interviewed. We always end up reimplementing graphs because: - Performance matters, and no off the shelf graph library I’ve seen can take advantage of many of the regularities in our particular data set. (We have an append-only DAG which we can internall…

What do you mean by “subgraph diffing”? I work with graphs a lot and use SQL almost all the time. Sometimes I need to compute connected components with python.

I have DAG. I can then define a proper subgraph from some set of nodes {A, B, C, ...} such that the subgraph contains all transitive dependencies of that set of nodes. Given two sets of nodes X and Y, I want the set difference between the subgraphs of nodes defined by X and Y (and all of their transitive dependencies). So, what nodes exist in the subgraph of X but not in Y, and vice versa?

Ie, if we have the graph { A -> B, A -> C } then the diff between {A} and {C} is ({}, {C}). And the diff between {B} and {C} is... well, ({B}, {C}).

Re: The hunt for the missing data type

#254
post #230

Earlier quoted context omitted.

I still don't see how this couldn't work with a `Neighbors(v)` function with an unspecified return type. The outputs I gave were examples of how it could be adapted for various use cases, not an exhaustive list of all possibilities; in the example with multiple edges with multiple weights, the output could instead be a relation of (v2, w, c) that indicates that v connects to v2 with weight w with c as the number diff…

In all of your earlier examples (and actually, including the current one too), you're treating vertices as first-class objects, but edges as second-class. There's no way to even identify an edge in your formulation -- only vertices. This is a common oversight in graph structures that ends up being very annoying in many applications. You keep trying to work around it by associating the edge's properties with those of…

I'm not clear what an abstraction that makes edges first class but vertices second class looks like. An edge connects two vertices, so if two edges connect to the same vertex, what does this look like?

Re: The hunt for the missing data type

#255

Earlier quoted context omitted.

In all of your earlier examples (and actually, including the current one too), you're treating vertices as first-class objects, but edges as second-class. There's no way to even identify an edge in your formulation -- only vertices. This is a common oversight in graph structures that ends up being very annoying in many applications. You keep trying to work around it by associating the edge's properties with those of…

I'm not clear what an abstraction that makes edges first class but vertices second class looks like. An edge connects two vertices, so if two edges connect to the same vertex, what does this look like?

The most minimal example of it I can think of would be a little ridiculous, but it could look like this:

  interface Edge { }
  interface Graph {
    List getRoots();  // Returns some ~minimal set of edges with connectivity to all the others
    List getAdjacentEdges(Edge e, boolean tail);
  }
There's no way to directly refer to a vertex here at all (unlike with edges), and yet (since edges have identity) there's enough information to determine the graph structure!

Re: The hunt for the missing data type

#256

Earlier quoted context omitted.

I'm not clear what an abstraction that makes edges first class but vertices second class looks like. An edge connects two vertices, so if two edges connect to the same vertex, what does this look like?

The most minimal example of it I can think of would be a little ridiculous, but it could look like this: interface Edge { } interface Graph { List getRoots(); // Returns some ~minimal set of edges with connectivity to all the others List getAdjacentEdges(Edge e, boolean tail); } There's no way to directly refer to a vertex here at all (unlike with edges), and yet (since edges have identity) there's enough information…

What's an example of an algorithm that could use this sort of interface? All of the algorithms that immediately come to mind are more require more vertex information.

Re: The hunt for the missing data type

#257
post #243

Earlier quoted context omitted.

Btw, solving practical instance of NP problems is often not all that bad in practice. Even solving them to optimality. But you need to move away from writing your own solvers. Instead you use a library that lets you describe your problem, and then throws off-the-shelf solvers at them. See eg https://developers.google.com/optimization That's a good approach even for problems that are in P, because minor changes in the…

Many graph problems also end up having lower bounds, being subject to conjectures like SETH or 3sum SETH has a sub quadratic lower bound and several other graph problems have cubic lower bounds. Many real systems are often saved because it is actually hard to write code that aren't primitive recursive functions. Cycles often are what destroy that, as considering WHILE and GOTO being the difference between primitive a…

> Many graph problems also end up having lower bounds, being subject to conjectures like SETH or 3sum

Those are lower bounds on worst case instances. Not lower bounds on solving typical, practical instances.

> If you consider NP as second-order queries where the second-order quantifiers are only existantials, That will help explain why heuristics (educated guesses) help.

> A graph data type wouldn't have those heuristics.

Sounds like your heuristic for why heuristics help with many NP problems is less than helpful here.

In practice, you can encode many graph problems as eg SAT or integer programming or SMT etc and get good performance.

Even biggish instances of eg the traveling salesman problem are often solved well in practice.

I'm not sure why you bring up primitive recursive functions? Primitive recursion is able to express all of NP (and much more), so it's not much of a constraint in this discussion? (I agree that you have to try hard in practice to go beyond primitive recursion but stay finite.)

Re: The hunt for the missing data type

#258
There's another glaring omission: grammars. Regular expressions are everywhere, languages have literals for them. Context free parsing with grammars? You're gonna have to break out the compiler compilers.

Why there is no parse(grammar, input) function in standard libraries is beyond me. The Earley algorithm seems well suited for it, it can take a grammar as input and and it even work online.

Re: The hunt for the missing data type

#259
post #123

I think this is because a graph is not a data-structure nor a data-type. It is really an abstraction. Fundamentally, all I need to define a graph is a set of vertices v \in V and function Neighbors(v). And that really is all is needed for the most foundational set of graph algorithms. Everything else are case-by-case constraints. Does A->B imply B->A? is the node set partitionable with certain constraints? Are there…

> hypergraph

> I just have a set of vertices

> and a set of sets of vertices

Sounds kind of like a file system to me. Files are the vertices. Directories are the nestable sets of vertices.

Post reply on HN