Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is a Knowledge Graph?

neo4j.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: What Is a Knowledge Graph?

#41

Post by CharlieDigital »

zxexz wrote: post #40
CharlieDigital wrote: post #9

Earlier quoted context omitted.

During our initial testing, ~1m nodes on a local Docker container with 1G RAM and 1vCPU. But here I mean "performance" in both retrieval time and the overall quality of the fragments retrieved for RAG compared to a `pgvector` only implementation. It is possible to "simulate" these types of graph traversals in pg as well, you'll have to work much harder to get the performance (we tried it first).

Huh. I've had the opposite experience. Neo4j has a pretty nice interface and package overall, but I was not impressed with the performance, and the developer experience was about on-par with Elasticsearch (not comparing the two databases, just the developer resources and communities). For general purpose use I've still not found anything better than Postgres (and yes, knowledge graphs I would consider general purpose…

If your performance is poor, try running your query with `PROFILE {your_query}`. It's very easy to write a query that ends up loading way more nodes than expected. Years ago we had one query that progressively performed worse -- turned out one leg was loading the full node space!

What I have found is that "land and expand" using an index to find the landing spots is key for performance. Reason being once you "land" effectively, "expand" is cheap and fast.

Some of it will also come down to your graph design. If you have a lot of super dense nodes (analogous to a large JOIN), it will create a lot of memory pressure which it does not handle well.

But in a RAG use case, I don't see these as being issues.

Re: What Is a Knowledge Graph?

#42

Post by alexgunnarson »

We employ a knowledge graph at Deft (https://shopdeft.com) to enable searches over ~1M products, amounting to about 1B triples. Because of the complexity of the queries involved, the expressiveness of our data model — supporting n-ary/reified relations, negation, disjunction, linguistic vagueness, etc. — and our real-time latency targets, we built a graph DB engine "from scratch" (certain components are of course from open-source projects). Even RedisGraph wasn't fast enough for the purpose; ours (Deftgraph) is 700x faster on our queries thanks to some SOTA optimizations from various recent papers. You'll notice on our site that the overall search latency is generally acceptable but not great; the vast proportion of that latency comes from 1) LLMs and 2) a less-optimized other graph DB, Datomic, that we still store some of our data in for legacy reasons.

LLMs are great, but knowledge graphs are IMO indispensable to tame their shortcomings.

Re: What Is a Knowledge Graph?

#43

Post by traverseda »

alexgunnarson wrote: post #42

We employ a knowledge graph at Deft ( https://shopdeft.com ) to enable searches over ~1M products, amounting to about 1B triples. Because of the complexity of the queries involved, the expressiveness of our data model — supporting n-ary/reified relations, negation, disjunction, linguistic vagueness, etc. — and our real-time latency targets, we built a graph DB engine "from scratch" (certain components are of course f…

If you have a graph database that is 700x faster on real world use cases than the next nearest competitor, why aren't you selling it? Given the current AI gold rush, it seems like a no brainer to get some VC cash, hire some sales people, and start selling shovels.

Re: What Is a Knowledge Graph?

#44

Post by ic_fly2 »

As someone running neo4j in production I can just warn that the DBs are a pain and need a lot more care and love than Postgres or Oracle DBs. Even much larger instances. Maybe their cloud offerings are better, but they are quite expensive.

Re: What Is a Knowledge Graph?

#45

Post by j-pb »

alexgunnarson wrote: post #42

We employ a knowledge graph at Deft ( https://shopdeft.com ) to enable searches over ~1M products, amounting to about 1B triples. Because of the complexity of the queries involved, the expressiveness of our data model — supporting n-ary/reified relations, negation, disjunction, linguistic vagueness, etc. — and our real-time latency targets, we built a graph DB engine "from scratch" (certain components are of course f…

I'd love to hear about your absolute numbers.

We had a similar problem, Datomic/Datascript not having an open format like RDF, but RDF being clunky and slow, so we build our own open-source solution in Rust (https://github.com/triblespace).

On an M1max we're currently at ~3us per query for a single result (so essentially per query overhead), and have something like 1m QRPS for queries with 3-4 joins.

I'm curious if you've somehow managed to shave off another order of magnitude, as I suspect that most WCO joins will be similarly limited by memory bandwidth. We for example worked out a novel join algorithm family (Atreides Join) and supporting trie based in-memory and succinct zero-copy on-disk data-structures, just to get rid of the query optimiser and its massive constant factor.

Re: What Is a Knowledge Graph?

#46

Post by profsummergig »

Fannon wrote: post #35
profsummergig wrote: post #23

A rant about Chrome Bookmarks Manager (it's on-topic, I promise). A few years ago, a Good Samaritan on HN told me my bookmarks (I had about 10,000 at the time) were my "knowledge graph". I had no idea what that was, but upon researching the concept, I was mind-blown by the simple truth of what I had been told. Since then I became even more rapacious with my bookmarking (and especially editing their "Name" field to ad…

I know its a third party extension, but I created an open-source browser extension that helps me with a lot of that (I also use my bookmarks extensively for this): https://github.com/Fannon/search-bookmarks-history-and-tabs#... If you're afraid that it goes 404: This extension is open-source, very easy to build and use locally and it does not make any external request or relies on external dependencies.

Thanks!

Re: What Is a Knowledge Graph?

#47

Post by zozbot234 »

ic_fly2 wrote: post #44

As someone running neo4j in production I can just warn that the DBs are a pain and need a lot more care and love than Postgres or Oracle DBs. Even much larger instances. Maybe their cloud offerings are better, but they are quite expensive.

Yes, Postgres is actually a great general-purpose graph database (excluding specialized network analysis that's actually pretty niche) if you can deal with the clunky recursive-CTE syntax for graph queries. (The new SQL standard actually comes with an added Property Graph Query/PGQ syntax specifically to make these queries easier to express.)

Re: What Is a Knowledge Graph?

#48

Post by zozbot234 »

lolive wrote: post #38
CharlieDigital wrote: post #29

Earlier quoted context omitted.

What would you recommend as an RDF database to explore?

GraphDB is the one I usually use. It has a web interface that eases the first steps. Virtuoso (especially Virtuoso 7, which is open source) is also an option. [a bit more command line based]. In case you want to have a look a the SPARQL client I maintain, Datao.net, you can go to the website and drop me a mail. [i really need to update the video there as the tool has evolved a lot since that time]

The new kid on the block is very much QLever. Still lacking some features, especially wrt. real time update that make it unsuitable for replacing the Wikidata SPARQL endpoint altogether just yet, but it's clearly getting there.

Re: What Is a Knowledge Graph?

#49

Post by ristos »

I would say that a better alternative to graph dbs would be prolog or datalog, because they're expressive enough to describe hypergraphs.

Prolog is better than datalog in a lot of ways: CLPZ, abduction, homoiconicity, being able to choosing search strategies for different problems, tabling, etc.

There's been some work to integrate prolog with LLMs:

https://swi-prolog.discourse.group/t/llm-swi-prolog-and-larg...

Re: What Is a Knowledge Graph?

#50

Post by cgearhart »

A knowledge graph is really just a projection of structured data from disparate sources into a common schema.

Take a bunch of tables and covert each row into a tuple (rowkey, columnName, value). Now take the union of all the tables.

^ knowledge graph

That’s it…but it’s not very useful yet. It becomes more useful if you apply a shared ontology during the import—ie translate all the columns into the same namespace. Suppose we had a “contacts” table with columns {“first name”, “last name”, …} and a “events” table with columns {“participant given name”, “participant family name”, …} — basically you need to unify the word you use to describe the concept “first name”/“given name”/whatever across all sources.

This can be cool/useful because you now only need one table (of triples) to describe all your structured data, but it’s also a pain because you may need to perform lots of self-joins or recursive queries to recover your data in order to do useful things with it. The final table has a very simple “meta” schema, and you erase the schema from each individual source so you can push the schema into the data.

Post reply on HN