Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache AGE, a PostgreSQL extension with graph database functionality

github.com

Post reply on HN
Print view View original

61–70 of 77 posts

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#61

Post by bayesian_horse »

stevesimmons wrote: post #56
bayesian_horse wrote: post #53

Always bet on PostgreSQL! I hope AGE matures a bit in the future. There are lots of use cases for Graph Databases. One I'm interested in is bitemporality. It's easy to use ltree or CTE for tree-like structures. But what if you want to move nodes in the graph at certain times? Like a device being scheduled to be in different rooms across time. And also the history of those schedules. In a graph database you can label…

That's a really interesting idea. Can you recommend any good references for bitemporality in graph dbs?

Theres XTDB

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#64

Post by sandGorgon »

ArnoVW wrote: post #34
sandGorgon wrote: post #29

has anyone here worked on graph neural networks ? basically creating embeddings for node based on their edge connectivity (or reachability) and using that for neural networks ? how do you do this at scale ?its generally a NP hard problem, but wondering whether something like AGE helps. not sure how Google, etc or even someone on fraud detection does this at scale

You subsample. One package I used made N 'random walks' for each node. The random walks are written out as 'sentences', where the node id's are words. That results in a huge text file, that you then embed as if it were a normal text. The result is a normal 'word embedding' where the words are in reality the node id's. Works like a charm. Highly scalable. https://github.com/dwslab/jRDF2Vec

really ? so u keep subsampling as the data becomes larger and larger.

instead of ...well...throwing more hardware that seems to be easier and easier these days.

P.S. not trolling. im genuinely wondering if there is a better way to split the problem heuristically

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#66

Post by dfox »

zasdffaa wrote: post #47
Sasasu wrote: post #39

Earlier quoted context omitted.

PG has a special memory manage rule, named MemoryContext. All memory allocated in a context will disappear when it leaves that context. this means that you can safely not free memory, or your memory will be freed in unexpected places. this is a big conflict with the way rust manages memory. write extension in rust won't improve it much. And in PG, there is a special method to create a process, creating threads is not…

> creating threads is not possible because the logging system makes heavy use of setjmp(). Naive question from a non-c user, setjmp/longjmp just manipulate the stack and since each thread has its own execution stack, that should be completely safe ISTM - so why is it unsafe/impossible? I'm missing something.

The general issue is that it is not designed for multi threaded environment and there is bunch of global and frequently accessed state. One could probably comb through the entire codebase and make all that state thread-local, but the benefit probably does not outweight the amount of work.

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#67

Post by AtlasBarfed »

tluyben2 wrote: post #23
faizshah wrote: post #5

Anyone tried this? How does the performance compare to neo4j and RedisGraph? I’m about to give RedisGraph a try and I guess I will try this one a go as well.

Did you try dgraph? For our use cases it won over neo4j. Didn’t try redisgraph.

There is also a graph layer for Cassandra. It was based on Titan ... JanusGraph.

I also played around with a graph-document database hybrid when I had downtime, but never got it close to anything usable.

A json document database with relations between documents is basically a property graph. I've seen a lot of the document databases (rethinkdb, orientdb, elasticsearch, etc) that seem close to realizing this too, but no one has run with it.

Most document databases have some sort of nested "walker" api, and if your json doc has properties that are subdocuments, will walk those. That's basically a graph api.

I wrote it as a "streaming api" so a large document/property graph could be serialized out to the client as the lookup engine walked the graph, and you don't need to fully load a complex set of documents in the query layer memory before sending it out to the client.

But I just didn't have the development horsepower to get to the various query and index capabilities. I think the general distributed design was decent and offered hybrid plain-old-table, document, and graph capabilities all in one. And cassandra, PITA that it is, does linearly scale.

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#68

Post by twaway23 »

I'm considering using a graph database for a SaaS product. If I used Apache AGE, I would probably have a "graph" for each customer to partition the data. Are there any downsides or limitations to having thousands of separate graphs?

From the documentation it seems that each graph will use a separate "namespace" in Postgres. Are there any performance costs of switching namespaces for each query?

Or do you recommend that we use a single graph with a label per customer? This option seems like it could open up some security issues if some queries forget to add this label. By using a separate graph per customer, the query will need to have a valid graph name for a customer to return any data. If it is filtered by a label, you can easily forget to add it and think everything is OK because it actually returns results.

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#69

Post by andrewstuart2 »

akshayshah wrote: post #20
andrewstuart2 wrote: post #11

Earlier quoted context omitted.

Have you tried recursive CTEs with a simple id, parent_id etc schema? These should perform very well if those columns are in an index. Afaik this is pretty much the canonical way to store recursive comment trees. Or any kind of DAG.

As long as comments are a tree, there’s only one path from the root (the post) to an individual comment. How would a recursive CTE perform better than a prefix scan on an indexed string column? Storing a pointer to each node’s parent or using sorted sets seems like it would make the parent poster’s query slower. Those approaches would make it easier to reparent comments, though, and they’d support arbitrarily deep tr…

Especially in postgres, you can use a hash index to get constant time index lookups rather than log(n) with a b-tree index, which is the best you could do with prefix lookups on a column. And you're storing and indexing less data since the tree is inherent in the structure and not expressed as part of the data.

Honestly I'd test both implementations to see which is fastest, but my gut is still that recursive CTEs would be fastest, while also simplest, structure-wise. You'd also still be getting all the benefits you'd expect from database-native functionality, like referential integrity and schema enforcement, etc. Presumably there's structure in these indexed string columns that the database knows nothing about, and thus can't enforce any constraints or optimize outside of plain string prefix lookups.

My experience has been that people don't try recursive CTEs because they don't realize they exist, and so they reach for all sorts of exotic or bespoke implementations of essentially that same concept. So I at least try to make sure people know they're a really solid tool to keep in your tool belt, and one that hasn't let me down yet even in very large data sets.

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#70

Post by fulafel »

nine_k wrote: post #36
fulafel wrote: post #33

Earlier quoted context omitted.

PG ships with Lua support: https://www.postgresql.org/docs/current/external-pl.html (Also Python, Javascript, and Java) I don't know specifics about the API coverage. It seems this extension mostly just implements new SQL visible functions and data types, which should be doable from those languages as well. Composite types might have to be defined as PG records (or json) instead of C level new PG object types.

You can write stored procedures in it. Which is fine. You cannot though write a new storage engine, a new kind of index, or something else that takes an extension.

I think very few PG extension use cases need new storage engines. Looking at https://age.incubator.apache.org/overview/ it says it's using the normal PG storage.

Regarding indexes - (1) you can implement most custom indexe scenarios based on supplied expressions involving custom functions: https://www.postgresql.org/docs/current/indexes-expressional... (2) I suspect for the graph use case you could leverage json indexes: https://www.postgresql.org/docs/current/datatype-json.html#J...

This is not to claim with any high degree of certainty that this could be implemented in PG/Lua, just from my armchair there doesn't seem anything immediately obvious that would prevent it.

For example the querying in AGE is syntactically implemented in the function cypher() that is used in the examples to receiving custom syntax as strings with the "dollar quoting"[1] syntax:

  SELECT *
   FROM cypher('graph_name', $$
     MATCH (n)
     WHERE exists(n.surname)
     RETURN n.first_name, n.last_name $$) as (first_name agtype, last_name agtype);
[1] https://www.postgresql.org/docs/current/sql-syntax-lexical.h...
Post reply on HN