41–50 of 56 posts
I used a graph database in https://www.exploravention.com/products/askarch/ because software architects typically need to understand the dependencies of a complex software system before they can suitably lead that technology. A dependency graph is a good data structure to use when reasoning about dependencies and a graph database is a natural choice for capturing dependency graphs. See https://www.infoq.com/articles/…
Start with Postgres and scale later once you have a better idea of your access patterns. You will likely model your graph as entities and recursively walk the graph (most likely through your application). If the goal is to maintain views over graphs and performance/scale matters, consider Feldera. We see folks use it for its ability to incrementally maintain recursive SQL views (disclaimer: I work there).
Graph databases are one of those things that sound neat but you'll be hard pressed to find people using them that don't regret it. I memorably had a job interview which consisted almost entirely of their senior architect going over exactly why he regretted introducing Neo4J several years earlier and how all the work is really about getting away from it. That was just the most extreme example. The truth that people he…
What were some of the pain points mentioned?
Many will go "you need a proper ontology" at which point just use a RDBMS. Ontologies are an absolute tarpit, as the semantic web showed. The graph illusion is similar to that academic delusion "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures." which is one of those quips that makes you appreciate just how theoretical some theoreticians really are.
It's the lack of a fully developed storage engine that avoids vendor lock-in.
Apache GraphAr (incubating) is a step in this direction. But it's an import/export format. Not primary storage.
Unaware of this effort (roots in Chinese graph dbs), I wrote a competing proposal that's more aimed at graphdbs looking to disaggregate compute and storage.
https://adsharma.github.io/beating-the-CAP-theorem-for-graph...
It depends on the shape of your data. In my domain (cloud security), there are many many entities and it's very valuable to map out how they relate to each other.
For example, we often want to answer a question like: “Which publicly exposed EC2 instances are used by IAM roles that have administrative privileges in my AWS account?”
To answer the question, you need to: 1. Join ec2 instances to security groups to IP rules to IP ranges to find network exposure paths to the open internet. 2. Join the instances to their instance profiles, to their roles. 3. Join the IAM roles to their role policies to determine which have admin policies. 4. Chain all of those joins together, possibly with recursive queries if there are indirect relationships (e.g., role assumption chains).
That’s a lot of joins, and the SQL query would get both heavy and hard to maintain.
In graph this query looks something like
match (i:EC2Instance)--(sg:EC2SecurityGroup)--(r:IPPermissionInbound{action:"Allow"})--(rng:IPRange{id:"0.0.0.0/0"}) match (i)--(r:AWSRole)--(p:AWSPolicy)--(stmt:AWSPolicyStatement{effect:"Allow", resource:"*"}) return i.id as instance_id, r.name as role_name
To answer this question what internet open compute instances can act as admins in our environment, we needed to traverse multiple objects, but the shape of the answer is pretty simple: just a list of ids and names.
Graph databases have quirks and add complexity of their own. If your domain isn't this edge heavy, you're probably better off with Postgres, but for our use-case it's been worth the trade-off imo.
2. Writing cypher queries is a job I would never like to have as a human. But LLMs love it, so that an agent can do an ad hoc data science for every single problem. Especially while being aware which criteria were used for graph construction. It is worth ditching things like MCP in favor of tool graph-like solutions. For this purpose I developed my own DSL which only LLM speaks in internally. The effects are mind-blowing.
Earlier quoted context omitted.
What’s so much more compelling about a graph db than a “with recursive” sql call? And even such recursive sql calls should be fairly rare since you would rather cache frequently accessed foreign key relationships anyway.
Among other things, some graph DBs are designed to perform the equivalent of "with recursive", filtering by attributes as you chase links and/or calculating statistics over subgraphs, more efficiently than SQL databases. If you're just pulling up a tree of comments on an article using parent-child relations, SQL will be fine, though for query latency you might be better off with a "flat list" article-comment relation…
If you dive into the query plans of a graph DB you quickly see that there is nothing special about that. In the end it boils down to the same physical joins on node and vertex tables. The only thing graph DBs offer over your typical RDBMS is nicer syntax (while having worse operational maturity), and with the advent of SQL/PGQ event that advandtage is going away.
We validated the obvious thing - the best approach depends on your use case: 1. keyword/grep/regex/bmi25 works best (fastest, cheapest, most accurate) when you know exactly what you're looking for. 2. semantic search works best with unstructured data when you're not exactly sure what you're looking for. 3. text2sql works best when you have a few pre-defined queries with limited joins the agent can use to fetch structured data. 4. knowledge graphs works best when you need to find info across unstructured + structure data that go beyond semantics similarity (i.e. find arxiv reports by x author the discuss novel knowledge graph methods published in the past 3 years but don't mention neo4j).
So - we ended up building a simple add/search api that predicts where the data should come from, and what the user needs this week/today and cache it. It's accurate and it's fast.