Live data from Hacker News

Apache AGE, a PostgreSQL extension with graph database functionality

github.com

31–40 of 77 posts

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#31
post #30

This is written in C. I wonder how common it is to write PG extensions in safer langugaes and what would be the most suitable. I'm somewhat wary of using nontrivial C extensions, having seen so many of them sometimes seg fault the backend (eg PostGIS). There seem to be PG backend crashes described in this projects issues as well.

There's https://github.com/tcdi/pgx for writing extensions in rust.

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#32
post #30

This is written in C. I wonder how common it is to write PG extensions in safer langugaes and what would be the most suitable. I'm somewhat wary of using nontrivial C extensions, having seen so many of them sometimes seg fault the backend (eg PostGIS). There seem to be PG backend crashes described in this projects issues as well.

Postgres itself is written in C. I suppose its every internal interface is in C. I wonder how many unsafe sections would an extension written in Rust have to have to use these interfaces.

I wish something like Lua + LuaJIT could be used to write such extensions; at least it's memory-safe. OTOH mapping these C interfaces to Lua structures, and making them work with GC may happen to be non-trivial.

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#33
post #32
post #30

This is written in C. I wonder how common it is to write PG extensions in safer langugaes and what would be the most suitable. I'm somewhat wary of using nontrivial C extensions, having seen so many of them sometimes seg fault the backend (eg PostGIS). There seem to be PG backend crashes described in this projects issues as well.

Postgres itself is written in C. I suppose its every internal interface is in C. I wonder how many unsafe sections would an extension written in Rust have to have to use these interfaces. I wish something like Lua + LuaJIT could be used to write such extensions; at least it's memory-safe. OTOH mapping these C interfaces to Lua structures, and making them work with GC may happen to be non-trivial.

PG ships with Lua support: https://www.postgresql.org/docs/current/external-pl.html

(Also Python, Javascript, and Java)

I don't know specifics about the API coverage. It seems this extension mostly just implements new SQL visible functions and data types, which should be doable from those languages as well. Composite types might have to be defined as PG records (or json) instead of C level new PG object types.

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#34

has anyone here worked on graph neural networks ? basically creating embeddings for node based on their edge connectivity (or reachability) and using that for neural networks ? how do you do this at scale ?its generally a NP hard problem, but wondering whether something like AGE helps. not sure how Google, etc or even someone on fraud detection does this at scale

You subsample. One package I used made N 'random walks' for each node. The random walks are written out as 'sentences', where the node id's are words.

That results in a huge text file, that you then embed as if it were a normal text. The result is a normal 'word embedding' where the words are in reality the node id's. Works like a charm. Highly scalable.

https://github.com/dwslab/jRDF2Vec

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#36
post #33
post #32

Earlier quoted context omitted.

Postgres itself is written in C. I suppose its every internal interface is in C. I wonder how many unsafe sections would an extension written in Rust have to have to use these interfaces. I wish something like Lua + LuaJIT could be used to write such extensions; at least it's memory-safe. OTOH mapping these C interfaces to Lua structures, and making them work with GC may happen to be non-trivial.

PG ships with Lua support: https://www.postgresql.org/docs/current/external-pl.html (Also Python, Javascript, and Java) I don't know specifics about the API coverage. It seems this extension mostly just implements new SQL visible functions and data types, which should be doable from those languages as well. Composite types might have to be defined as PG records (or json) instead of C level new PG object types.

You can write stored procedures in it. Which is fine.

You cannot though write a new storage engine, a new kind of index, or something else that takes an extension.

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#38

"Apache AGE is currently being developed for the PostgreSQL 12 release" Well sorry, we have PostgreSQL 15 already.

If you keep reading, the words following your quote make it clear that PG15 should be supported:

"and will support PostgreSQL 13 and all the future releases of PostgreSQL."

Re: Apache AGE, a PostgreSQL extension with graph database functionality

#39
post #30

This is written in C. I wonder how common it is to write PG extensions in safer langugaes and what would be the most suitable. I'm somewhat wary of using nontrivial C extensions, having seen so many of them sometimes seg fault the backend (eg PostGIS). There seem to be PG backend crashes described in this projects issues as well.

PG has a special memory manage rule, named MemoryContext. All memory allocated in a context will disappear when it leaves that context. this means that you can safely not free memory, or your memory will be freed in unexpected places. this is a big conflict with the way rust manages memory. write extension in rust won't improve it much.

And in PG, there is a special method to create a process, creating threads is not possible because the logging system makes heavy use of setjmp().

Post reply on HN