MillenniumDB: Property graph and RDF engine, still in development • Hacker News Board

Re: MillenniumDB: Property graph and RDF engine, still in development

#21

Post by smarx007 » Fri, Jan 31, 2025, 8:21 PM UTC

PaulHoule wrote: post #15

FjordWarden wrote: post #11

Earlier quoted context omitted.

If you expect Jena to be more battle-tested because it is older, forget it, if the process is killed by a unexpected shutdown or some other reason it results in data corruption. At least this was my experience a few years ago. I found graph databases a beguiling idea when I first learned about them, and this is a welcome addition, but I've since temperated my excitement. They are not as flexible and universal a modal…

Jena lets you make little in-memory triple stores that you can use the way people use the list-map-scalar trinity. I've been working on this publication about that (RDF for difficult cases and when ordering counts) for years and it just got published last week https://www.iso.org/standard/76310.html I'll call out my collabortor Liju Fan for being the only person I've met who knew how to do anything interesting with O…

> Updating a SPARQL database is a little bit dangerous because there is no intrinsic idea of what a record is

SPARQL has a notion of a transactional boundary just like SQL has. You can combine multiple SPARQL queries in one transaction, they will all succeed or all fail just like you'd expect.

Re: MillenniumDB: Property graph and RDF engine, still in development

#22

Post by smarx007 » Fri, Jan 31, 2025, 8:34 PM UTC

FjordWarden wrote: post #18

zozbot234 wrote: post #12

Earlier quoted context omitted.

> SPARQL guarantees no such composability. SPARQL has a CONSTRUCT clause which gives you RDF as your query output. Isn't that compositional enough?

Ok, that is true, but how do I tell my graph database that the result of the construct query is some other graph in my DB?

> how do I tell my graph database that the result of the construct query

I am assuming you are asking how to do a CONSTRUCT query that will return you the contents of a given named graph?

https://www.w3.org/TR/sparql11-http-rdf-update/#http-get is a much simpler way to get a graph. As the spec says, it's equivalent to the following query

       CONSTRUCT { ?s ?p ?o } WHERE { GRAPH  { ?s ?p ?o } }

Re: MillenniumDB: Property graph and RDF engine, still in development

#23

Post by PaulHoule » Fri, Jan 31, 2025, 8:39 PM UTC

smarx007 wrote: post #21

PaulHoule wrote: post #15

Earlier quoted context omitted.

Jena lets you make little in-memory triple stores that you can use the way people use the list-map-scalar trinity. I've been working on this publication about that (RDF for difficult cases and when ordering counts) for years and it just got published last week https://www.iso.org/standard/76310.html I'll call out my collabortor Liju Fan for being the only person I've met who knew how to do anything interesting with O…

> Updating a SPARQL database is a little bit dangerous because there is no intrinsic idea of what a record is SPARQL has a notion of a transactional boundary just like SQL has. You can combine multiple SPARQL queries in one transaction, they will all succeed or all fail just like you'd expect.

Sorta kinda.

Your code has to put the right things in a transaction all the time for transactions for transactions to work right. If there is some flow of information like

   application does query -> application thinks -> application does update

you have to wrap the whole sandwich in a transaction, people frequently don't do that. If I'm writing 20 of those for an application I want something that I know is bulletproof.

My experience with SQL is that the average SQL developer doesn't really understand how to do transactions right but their ass gets saved (in a probabilistic sense) by the grouping of updates that is implicit by running an INSERT or an UPDATE against a table.

There's also the fact that a lot of triple stores are seriously half baked research-quality code if that. Many triple stores struggle if you just try to load 100,000 triples sequentially, for an application like my YOShInOn RSS reader which I expect to use every day and not have to patch or maintain anything for 18+ months. (Ok, a 20GB database that needs to be pruned crept up on me gradually, but that's an arangodb problem, I'd expect the average triple to store to have crumbled 17 months ago.)

I'd love to have something that updates like a document-oriented database but lets you run a SPARQL query against the union of all the documents. Database experts though always seem to change the subject when it comes to having a graph algebra that lets you UNION 10 million graphs.

(For that matter, I sure as hell couldn't pitch any kind any kind of "boxes-and-lines" query tool [1] etc. that passed JSON documents/RDF graphs over the lines between the operators to the VCs and private equity people who were buying up query engines circa 2015 because they were hung up on the speed of columnar query engines... Despite the fact that the ones that pass relational rows over the lines require people who really aren't qualified to do so create analysis jobs that look like terrible hairballs because of all the joins they do.)

[1] Alteryx, KNIME

Re: MillenniumDB: Property graph and RDF engine, still in development

#24

Post by smarx007 » Fri, Jan 31, 2025, 8:57 PM UTC

PaulHoule wrote: post #23

smarx007 wrote: post #21

Earlier quoted context omitted.

> Updating a SPARQL database is a little bit dangerous because there is no intrinsic idea of what a record is SPARQL has a notion of a transactional boundary just like SQL has. You can combine multiple SPARQL queries in one transaction, they will all succeed or all fail just like you'd expect.

Sorta kinda. Your code has to put the right things in a transaction all the time for transactions for transactions to work right. If there is some flow of information like application does query -> application thinks -> application does update you have to wrap the whole sandwich in a transaction, people frequently don't do that. If I'm writing 20 of those for an application I want something that I know is bulletproof…

> you have to wrap the whole sandwich in a transaction

True, SPARQL does not allow "opening" transactions such that you can run one query, do some logic, and run another query while doing commit. Which was a pain for me. RDF4J has a non-standard API to do that, I think they are trying to upstream it to SPARQL 1.2.

> There's also the fact that a lot of triple stores are seriously half baked research-quality code if that.

Also true. Although excellent researchers who wrote one of the best reasoners (Pellet) decided to leave academia and make a production grade system. They succeeded with Stardog but you don't want to know how much a license costs.

> couldn't pitch any kind any kind of "boxes-and-lines" query tool [1] etc. that passed JSON documents/RDF graphs

I really enjoy this talk from one of the creators of OWL [1]. There, he makes a point that OWL is unpopular not because it's too complex but because it's not advanced enough to solve real problems people care about (read: ready to pay money for). I think the case you described involves VCs having clarity on how to make money off one thing but not the other. I do think that the Semantic Web 3.0 (if we count Linked Data as a Semantic Web 2.0 aka Semantic Web Lite attempt) will need a better (appealing to business) case than the one presented in the 2001 SciAm paper.

[1]: https://videolectures.net/videos/eswc2011_hendler_work

Re: MillenniumDB: Property graph and RDF engine, still in development

#25

Post by PaulHoule » Fri, Jan 31, 2025, 9:08 PM UTC

smarx007 wrote: post #24

PaulHoule wrote: post #23

Earlier quoted context omitted.

Sorta kinda. Your code has to put the right things in a transaction all the time for transactions for transactions to work right. If there is some flow of information like application does query -> application thinks -> application does update you have to wrap the whole sandwich in a transaction, people frequently don't do that. If I'm writing 20 of those for an application I want something that I know is bulletproof…

> you have to wrap the whole sandwich in a transaction True, SPARQL does not allow "opening" transactions such that you can run one query, do some logic, and run another query while doing commit. Which was a pain for me. RDF4J has a non-standard API to do that, I think they are trying to upstream it to SPARQL 1.2. > There's also the fact that a lot of triple stores are seriously half baked research-quality code if th…

Personally I thought Stardog was trash, but if I'd had different requirements I might be happy with it.

The trouble w/ OWL as I see it (talked about in that TR) is that people don't really want "first order logic", but they want "first order logic + arithmetic" which is a nightmare that Kurt Godel warned you about. (That ISO 20022 which that TR is related to is about the financial domain which is all about arithmetic)

After Doug Lenat's death a lot of stuff came out that revealed the problems w/ Cyc, not least that even if you try to build something that is "knowledge based" it can't practically solve all the problems you want using a SMT-based strategy but you have to build a library of special purpose algorithms for everything you want to do and it turns out to be a godawful mess.

I'm disappointed that the semweb community hasn't made a serious crack at usable and efficient production rules (dealing w/ problems like negation, controlling execution order, RETE execution, retraction) instead we get half-answers like SPIN with fixed-point execution (used an even more half-baked version of that to research that TR, gets you somewhere). Of course, production rules never got standardized in any domain because nobody can agree on the way to address those four issues even though it usually isn't hard to find an answer that's fine for a particular application.

(It's a frequently problem that experts on a technology can get by on half-baked specific answers that would need a general solution if they were going to be useful for a general audience. One reason why parser generators are so bad is that if you understand parser generators enough to write a parser generator you aren't bothered by the terrible developer experience of parser generators.)

Re: MillenniumDB: Property graph and RDF engine, still in development

#26

Post by spothedog1 » Fri, Jan 31, 2025, 10:14 PM UTC

smarx007 wrote: post #9

I think if someone is just trying out RDF, it is better to start with Apache Jena/Fuseki or Eclipse RDF4J. Maybe https://github.com/oxigraph/oxigraph if you like to live dangerously (i.e. to use pre-1.0 DBMSs). Use of other systems involves factoring tradeoffs and considerations that are probably not the best for the newcomers. For example, qLever mentioned here is good in query performance and relative disk use but…

Definitely do not start with Jena/Fuseki, pain in the ass to set up. Start with Oxigraph or rdflib in memory to play around with how to query/interact with the graphs

Re: MillenniumDB: Property graph and RDF engine, still in development

#27

Post by sunshine-o » Fri, Jan 31, 2025, 11:27 PM UTC

I got very interested in RDF about 20-25 years ago.

Obviously it did not really succeeded but it seems some industries invested a lot into the tech and it is still around. Especially since AWS built a service around it.

I am really curious, what are the top use cases for it today?

Re: MillenniumDB: Property graph and RDF engine, still in development

#28

Post by kendallgclark » Fri, Jan 31, 2025, 11:35 PM UTC

PaulHoule wrote: post #25

smarx007 wrote: post #24

Earlier quoted context omitted.

> you have to wrap the whole sandwich in a transaction True, SPARQL does not allow "opening" transactions such that you can run one query, do some logic, and run another query while doing commit. Which was a pain for me. RDF4J has a non-standard API to do that, I think they are trying to upstream it to SPARQL 1.2. > There's also the fact that a lot of triple stores are seriously half baked research-quality code if th…

Personally I thought Stardog was trash, but if I'd had different requirements I might be happy with it. The trouble w/ OWL as I see it (talked about in that TR) is that people don't really want "first order logic", but they want "first order logic + arithmetic" which is a nightmare that Kurt Godel warned you about. (That ISO 20022 which that TR is related to is about the financial domain which is all about arithmetic…

You seem nice.

Re: MillenniumDB: Property graph and RDF engine, still in development

#29

Post by kendallgclark » Fri, Jan 31, 2025, 11:39 PM UTC

smarx007 wrote: post #24

PaulHoule wrote: post #23

Earlier quoted context omitted.

Sorta kinda. Your code has to put the right things in a transaction all the time for transactions for transactions to work right. If there is some flow of information like application does query -> application thinks -> application does update you have to wrap the whole sandwich in a transaction, people frequently don't do that. If I'm writing 20 of those for an application I want something that I know is bulletproof…

> you have to wrap the whole sandwich in a transaction True, SPARQL does not allow "opening" transactions such that you can run one query, do some logic, and run another query while doing commit. Which was a pain for me. RDF4J has a non-standard API to do that, I think they are trying to upstream it to SPARQL 1.2. > There's also the fact that a lot of triple stores are seriously half baked research-quality code if th…

OWL ontologies making a big comeback as part of Knowledge Graph groundings for LLM outputs. And several SPARQL and RDF knowledge graph startups are VC-baked and thriving. The world is a big place.

Re: MillenniumDB: Property graph and RDF engine, still in development

#30

Post by smarx007 » Sat, Feb 01, 2025, 12:06 AM UTC

kendallgclark wrote: post #29

smarx007 wrote: post #24

Earlier quoted context omitted.

> you have to wrap the whole sandwich in a transaction True, SPARQL does not allow "opening" transactions such that you can run one query, do some logic, and run another query while doing commit. Which was a pain for me. RDF4J has a non-standard API to do that, I think they are trying to upstream it to SPARQL 1.2. > There's also the fact that a lot of triple stores are seriously half baked research-quality code if th…

OWL ontologies making a big comeback as part of Knowledge Graph groundings for LLM outputs. And several SPARQL and RDF knowledge graph startups are VC-baked and thriving. The world is a big place.

Well, there is the new use case that appeals to VCs! And I guess it's a good reminder that I should re-subscribe to your blog :)