Earlier quoted context omitted.
I've done almost exactly the kind of thing described in article for a couple millions of rows. It broke down when I needed to find "friends of friends" of depth 6. None of the optimizations I could come up with helped. Neither did Apache Age. Maybe I'm just not skilled enough to handle such a workload with Postgres. But Neo4j handled it easily
I'm not sure if you are serious or joking. "6 degrees of separation" is the famous radius for how many steps are needed such that everyone is connected. https://en.wikipedia.org/wiki/Six_degrees_of_separation
Representing Graphs in PostgreSQL
61–70 of 80 posts
Re: Representing Graphs in PostgreSQL
#62Earlier quoted context omitted.
I'm not sure if you are serious or joking. "6 degrees of separation" is the famous radius for how many steps are needed such that everyone is connected. https://en.wikipedia.org/wiki/Six_degrees_of_separation
Except that it wasn't about people at all? I was just referring to a _type_ of query
Graph databases have a very narrow usecase, and it's almost always in relation to people - at least ime.
Though the data type isn't really important for the performance question, the amount of data selected is. So a 6-level depth graph of connections that only connect 2-3 entities would never get into performance issues. You'd be able to go way beyond that too with such a narrow window. (3 entity Connections on 6 level join would come out to I believe ~750 rows)
If you're modeling something like movies instead, with >10 actors per production you're looking at millions of rows.
Re: Representing Graphs in PostgreSQL
#63Earlier quoted context omitted.
I've been running into exactly that problem. Which time series add-on would you recommend looking into?
We ended up using ClickHouse after trying Timescale and InfluxDB. ClickHouse is great but important to spend a day or two understanding the data model to make sure it fits what you are trying to do. I have no affiliation with ClickHouse (or any company mentioned).
Re: Representing Graphs in PostgreSQL
#64The approach I came up with was to instead use MSSQL hierarchy IDs (there's a paper describing them, so they can be ported to any database). This specifically optimizes for ancestor-of/descendant-of queries. The gist of it is:
1. Find all cycles/strong components in the graph. Store those in a separate table, identifying each cycle with an ID. Remove all the nodes in the cycle, and insert a single node with their cycle ID instead (turning the graph into a DAG). The reason we do this is because we will always visit every node in a cycle when doing an ancestor-of/descendant-of query.
+---+
+-------> B +------+
+-+-+ +-^-+ +-v-+
| A | | | C |
+---+ | +-+-+
+-+-+ |
| D
Becomes: +---+ +---+
| A +--> 1 |
+---+ +---+
+---+---+
| 1 | B |
| 1 | C |
| 1 | D |
+---+---+
2. You effectively want to decompose the DAG into a forest of trees. Establish a worklist and place all the nodes into it. The order in this worklist may be something that can affect performance (you want the deepest/largest tree first). Grab nodes out of this worklist and perform a depth-first search, removing nodes from the worklist as you traverse them. These nodes can then be inserted into the hierarchy ID table. You should insert all child nodes of each node, even if it has already been inserted before, but only continue traversing downwards if the node hasn't yet been inserted. +---+
+----> B +----+
+-+-+ +---+ +-v-+ +---+
| A | | D +--> E |
+-+-+ +-^-+ +---+
| +---+ |
+----> C +----+
+---+
Becomes: +---+ +---+ +---+
+---> B +--> D +--> E |
+-+-+ +---+ +---+ +---+
| A |
+-+-+ +---+ +---+
+---> C +--> D |
+---+ +---+
Or, as hierarchy IDs A /1
B /1/1
D /1/1/1
E /1/1/1/1
C /1/2
D /1/2/1
Now, you do still need a recursive CTE - but it's guaranteed to terminate. The "interior" of the CTE would be a range operation over the hierarchy IDs. Basically other.hierarchy >= origin.hierarchy && other.hierarchy This was two orders of magnitude faster than a recursive CTE using strings to prevent infinite recursion.The major disadvantage of this representation is that you can't modify it. It has to be calculated from scratch each time a node in the graph is changed. The dataset I was dealing with was 100,000s of nodes for each customer, took under a second, so that wasn't a problem. You could probably also identify changes that don't require a full rebuild (probably any change that doesn't involve a back edge or cross edge), but I never had to bother so didn't solve it.
Re: Representing Graphs in PostgreSQL
#65Re: Representing Graphs in PostgreSQL
#66Re: Representing Graphs in PostgreSQL
#67Re: Representing Graphs in PostgreSQL
#68It is always good to know, at what point does "Postgres as X" break down. For instance, I know from experience that Postgres as timeseries DB (without add-ons) starts to break down in low billions of rows. It would be great to know that for graph DBs as well. I think a lot of people would prefer just to use Postgres if they can get away with it.
"Postgres as graph DB" starts to break down when you try to do serious network analysis with it, using specialized algorithms that are heavy on math - as opposed to merely using 'graphs' as the foundion of your data model, which is what graph databases mostly get used for. It's more about "what your actual use case is" than "how much data you have".
I presume that for larger real world graph datasets, maybe there's some better algorithms and storage methods. I couldn't figure out neo4j fast enough, and it wasn't clear that I could map all of the stuff like block modeling into it anyways, but it would be very useful for someone to figure out a better production ready storage backend for networkx at least where some of the data could be cached in SQLite3.
Re: Representing Graphs in PostgreSQL
#69Im a big fan of GraphDatabase's since about 10 years. I even wrote my own "in memory graph storage" in golang for a specific use case that none of the big GraphDatabase's could cover at the time. That said - i WISH people would embrase the existing GraphDatabases more and make the hosters support them as standard, rather than abusing existing relational databases for graph purposes. And to make it clear,i'm not talki…
My personal opinion is that nobody should touch OrientDB with a 10 ft pole.
Re: Representing Graphs in PostgreSQL
#70It is always good to know, at what point does "Postgres as X" break down. For instance, I know from experience that Postgres as timeseries DB (without add-ons) starts to break down in low billions of rows. It would be great to know that for graph DBs as well. I think a lot of people would prefer just to use Postgres if they can get away with it.
I've been running into exactly that problem. Which time series add-on would you recommend looking into?