A lot of comments not sure about what Graph DBs are good for: * Flexible knowledge association i.e. Knowledge Graphing * Modeling and querying associations / models with many-steps-removed requirements * Expert Systems / Inference Engines * Lazy traversal for complex job scheduling Graph DBs are not good at being a general purpose 95% of use cases database. Just use Postgres/MySQL if you're not sure. We use Neptune (…
JanusGraph – Distributed, open source, scalable graph database
31–40 of 43 posts
Re: JanusGraph – Distributed, open source, scalable graph database
#32Graph databases have been the great white whale at my org for a number of years. We gave a crack at Janus a while back. It (like a few attempts at Neo4J) failed to deliver on the promise of unlocking queries with more than a hop or two, while dramatically underperforming on those one or two hop queries vs a graph implemented in MySQL.
I also don't see many valid use cases for graph databases.
Re: JanusGraph – Distributed, open source, scalable graph database
#33A lot of comments not sure about what Graph DBs are good for: * Flexible knowledge association i.e. Knowledge Graphing * Modeling and querying associations / models with many-steps-removed requirements * Expert Systems / Inference Engines * Lazy traversal for complex job scheduling Graph DBs are not good at being a general purpose 95% of use cases database. Just use Postgres/MySQL if you're not sure. We use Neptune (…
I have yet to see streaming/event sourcing work well. Every time I’ve seen it used it’s caused more problems than it’s resolved. The main problems, out of order events and/or slow to propagate events.
The hard part is that ES/Streaming systems work best / almost necessitate a clean and clear domain model. A clean and clear domain model requires a lot of discussion and consensus with domain experts and product owners. Buy in to have the kind of discussions needed is the source of the issues I've experienced with these kind of systems. CRUD can paint over a lot of cloudy abstract concepts for better or for worse. These kind of discussions are energy intensive and mentally painful to cast light on the cloudy thoughts.
There's not great streaming/es support on a language level outside of the robust actor model systems (e.g. Erlang/Elixir). There are systems like Akka that simulate that to some extent on runtimes like JVM, but a cooperative scheduler and an actor model don't mix great. For non-actor model aspects I've been seeing more service level dataflow systems like KSQL / MaterializeDB gain traction, but are nevertheless a solution for read-models not application logic.
Re: JanusGraph – Distributed, open source, scalable graph database
#34Earlier quoted context omitted.
I have yet to see streaming/event sourcing work well. Every time I’ve seen it used it’s caused more problems than it’s resolved. The main problems, out of order events and/or slow to propagate events.
From a technical perspective there are plenty of ways to serialize / order some stream of events in a reasonable way. Whether that's implementing your event store on top of a transactionally secure database (e.g. Postgres, not Mongo) or using a higher throughput, less persistently secure solution like Kafka. There's also ways to deal with eventual consistency and distributed transactions (SAGAs) depending on the need…
Re: JanusGraph – Distributed, open source, scalable graph database
#35Earlier quoted context omitted.
From a technical perspective there are plenty of ways to serialize / order some stream of events in a reasonable way. Whether that's implementing your event store on top of a transactionally secure database (e.g. Postgres, not Mongo) or using a higher throughput, less persistently secure solution like Kafka. There's also ways to deal with eventual consistency and distributed transactions (SAGAs) depending on the need…
In short, making streaming architectures work require great central planing, a grand architect in the sky, or they fail miserably. I think that is an argument against streaming architectures ;)
The implementation overhead as far as code-to-write for streaming architectures is comparable to CRUD. But there is less knowledge dispersed about practices on how to do it, so there is the cost of learning. It is more cutting edge after all.
Re: JanusGraph – Distributed, open source, scalable graph database
#36Earlier quoted context omitted.
In short, making streaming architectures work require great central planing, a grand architect in the sky, or they fail miserably. I think that is an argument against streaming architectures ;)
I think that's a bit of a reduction/straw man to say they need a lot of architecture/central-planning. An individual developer can do it so long as they have access to domain experts to ask the right questions. Lack of proper planning will result in any architecture failing miserably - best not to code until we know what to code. The implementation overhead as far as code-to-write for streaming architectures is compa…
Re: JanusGraph – Distributed, open source, scalable graph database
#37Earlier quoted context omitted.
I think that's a bit of a reduction/straw man to say they need a lot of architecture/central-planning. An individual developer can do it so long as they have access to domain experts to ask the right questions. Lack of proper planning will result in any architecture failing miserably - best not to code until we know what to code. The implementation overhead as far as code-to-write for streaming architectures is compa…
It is admittedly an oversimplification for affect, but it’s more or less what you said. Maybe I’ve worked at bigger companies, we have over 1000 devs in my division, and trying to get them all aligned is very very difficult. I think central planning is a good analogy for that. The architecture that requires less central planning/coordination will likely do better in such a dynamic.
Regardless I feel streaming has been around long enough now it should be urgently considered anywhere where the words "data pipeline" and "scale" are thrown around together frequently.
Re: JanusGraph – Distributed, open source, scalable graph database
#38Earlier quoted context omitted.
It is admittedly an oversimplification for affect, but it’s more or less what you said. Maybe I’ve worked at bigger companies, we have over 1000 devs in my division, and trying to get them all aligned is very very difficult. I think central planning is a good analogy for that. The architecture that requires less central planning/coordination will likely do better in such a dynamic.
I expect streaming will be adopted like anything - first in smaller more mobile teams working in greenfield contexts, then later in large organizations when the idea is less novel and lower risk/cost to apply at scale. Regardless I feel streaming has been around long enough now it should be urgently considered anywhere where the words "data pipeline" and "scale" are thrown around together frequently.
Re: JanusGraph – Distributed, open source, scalable graph database
#39(hundreds of thousands of folders)
The idea would be to use a graph DB for a first query to get the file ids in scope (all files inside a given folder and its subfolders) before running the actual SQL query, eg creation_date < foo AND file_id in [array from graph db output]
Re: JanusGraph – Distributed, open source, scalable graph database
#40Earlier quoted context omitted.
There is a lot - customer support (aka customer 360), fraud detection, some maintenance cases, inventory, recommendations, etc. But it’s heavily dependent on requirements - what should be response time. Most of graph databases are good in fast response time, but will do it only for 2-3 hops from known start points. For many other things, graph analytics with Spark or something like could be better
2-3 hops (JOINs) is well within RDBMS territory, no?