Graph Mining Library

I’m not saying it’s too much to ask for, but also, when you’re doing distributed in memory graph mining (which means you’ve got an application with a big enough graph that you need to do this, and the technical expertise to need the algorithms in this open source package) maybe it’s expected that you can read the bazel files and bazel docs yourself and figure it out.

Or just write a make file and cut all the bazel build optimization out.

They don’t put instructions on how to start a F1 car inside the cockpit, you don’t hop into a fighter jet and look for the push to start easy button, it’s expected that when you’re at that level you bring expertise.

There are "graph databases" which see graphs as a universal approach to data, see RDF and SPARQL and numerous pretenders. For that matter, think of a C program where the master data structure is a graph of pointers. In a graph like that there is usually a huge number of different edge types such as "is married to", "has yearly average temperature", ...

Then there are "graph algorithms" such as PageRank, graph centrality, and such. In a lot of those cases there is one edge type or a small number of edge cases.

There are some generic algorithms you can apply to graphs with many typerd edges edges such as the magic SPARQL pattern

  ?s1 ?p ?o .
  ?s2 ?p ?o .

which finds ?s1 and ?s2 that share a relationship ?p with some ?o and is the basis for a similarity metric between ?s1 and ?s2. Then there are the cases that you pick out nodes with some specific ?p and apply some graph algorithm to those.

The thing about graphs is, in general, they are amorphous and could have any structure (or lack of structure) at all which can be a disaster from a memory latency perspective. Specific graphs usually do have some structure with some locality. There was a time I was using that magic SPARQL pattern and wrote a program that would have taken 100 years to run and then repacked the data structures and discovered an approximation that let me run the calculation in 20 minutes.

Thus practitioners tend to be skeptical about general purpose graph processing libraries as you may very have a problem that I could code up a special-purpose answer to in less time than you'll spend fighting with the build system for that thing that runs 1000x faster.

----

If you really want to be fashionable though, arXiv today is just crammed with papers about "graph neural networks" that never seem to get hyped elsewhere. YOShInOn has made me a long queue of GNN papers to look at but I've only skimmed a few. A lot of articles say they can be applied to the text analysis problems I do but they don’t seem to really perform better than the system YOShInOn and I use so I haven’t been in a hurry to get into them.

Did you release it without docs so that you could add it to your Perf packet?

Yeah, and somebody who is that smart can probably pack their data structures efficiently and find an approximation to do the job on a macbook pro that people with too many resources need a 1000 machine cluster to do. And get the coding and the computation done in the time that the C++ compiler is still chewing on the headers of the bloated library. (At times I’ve been that guy.)

But seriously, there is such a thing as industrialization. Google is notorious though for hiring 180 IQ people and getting them to perform at a 35 IQ level because there the documentation makes no sense, a procedure which should be done in one step really takes 300, fighting with C++, etc. They can afford to do it because it is just so profitable to help that guy who shows up on every. single. video. roll. who says “you can’t lose weight by exercising”, “you can’t lose weight by cutting carbs” who links to some video that drones on hours and hours and signs you up for some subscription you can never cancel to sell scam supplements.

Shame on them.

BTW, with high-complexity software there is enough question that you got it working right that you expect a significant process of testing that it works for your application. For instance if you got a hydrocode for simulating the explosion of a nuclear weapon you would not take for granted that you had built it properly and were using it properly until you'd done a lot of validation work. A system like that isn't a product unless it comes with that validation suite. The same is true for automated trading software (gonna hook it up straight to the market without validation? hope you have $100M to burn!)

... now there was that time a really famous CS professor e-mailed me a short C program that was said to do something remarkable that crashed before it even got into main() which did teach me a thing about C but that's not what a professional programmer does.

And why would any of those be harmful? Care to elaborate?