Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Drasi: Microsoft's open source data processing platform for event-driven systems

github.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: Drasi: Microsoft's open source data processing platform for event-driven systems

#71

Post by FromOmelas »

UltraSane wrote: post #19
robertlagrant wrote: post #17

Earlier quoted context omitted.

We made a health backend partly using Cypher and the only thing I found was the simple queries looked amazing, but as soon as you need to join non-linearly it started looking a lot like SQL again. And when you're using an ORM it stops mattering. And when you need migrations it gets painful!

"you need to join non-linearly " What does this mean?

presumably it has a semantic model of sorts, defining intrinsic relationships between entities (parent-child, composed-of, sibling-of, and so on)

A bit similar how certain joins in SQL can be very straightforward with the "USING" clause, or when it can rely on extra information such as analytic views to derive materialized views (vendor specific)

Re: Drasi: Microsoft's open source data processing platform for event-driven systems

#72

Post by refset »

UltraSane wrote: post #21
CharlieDigital wrote: post #16

Earlier quoted context omitted.

Didn't look too deeply, but one of the keys with Cypher (at least in the context of graph databases) is that it has a nice way of representing `JOIN` operations as graph traversals. MATCH (p:Person)-[r]-(c:Company) RETURN p.Name, c.Name Where `r` can represent any relationship (AKA `JOIN`) between the two collections `Person` and `Company` such as `WORKS_AT`, `EMPLOYED_BY`, `CONTRACTOR_FOR`, etc. So I'd say that line…

The neat thing about Neo4j is that the [r] isn't a join, it is an actual relationship stored on disk.

Like a many-to-many join table?

Re: Drasi: Microsoft's open source data processing platform for event-driven systems

#74

Post by TeMPOraL »

stefanos82 wrote: post #27

Drasi...React...well played Microsoft, well played :D Assuming they choose this name from the Greek δράση which means action, React of course is the exact opposite to action, thus the React-ion; an action expects a reaction, somewhere somehow!

Here I thought they were accidentally or intentionally referring to:

https://babylon5.fandom.com/wiki/Drazi

But now I noticed the spelling difference :/.

Re: Drasi: Microsoft's open source data processing platform for event-driven systems

#75

Post by agentofreality »

otterley wrote: post #22

Looks very Azure-centric. Both installation guides ( https://drasi.io/how-to-guides/install-sample-applications/b... and https://drasi.io/how-to-guides/install-sample-applications/c... ) require Azure to work. And then there's this: > Installing Drasi in an EKS cluster can be significantly more complex than a standard installation on other platforms. Instead of downloading a CLI binary using the provided installation…

I am the Drasi engineering lead and can assure you that any Azure-centricity is purely one of historical convenience and a lag in getting more of our non-Azure-centric doc, samples, and components published.

The main current dependency is having a K8s cluster.

You can run Drasi for dev/test on k3s(https://drasi.io/how-to-guides/installation/install-on-k3s/) or kind(https://drasi.io/how-to-guides/installation/install-on-kind/) and docker desktop also works but is undocumented.

Cloud based options include AKS (we will release the instructions soon) and EKS as mentioned. When we tested on EKS, we hit some storage class issues and decided to publish this with some work-arounds instead of holding back until we do a proper fix, which we will prioritize if there is demand.

On prem K8s should also work, but we haven't put resources into testing those scenarios. We would love to engage with anybody that would be willing to try this out.

Also, in the future we are thinking about other delivery platforms, not just K8S. You will see in the code that our dependency on k8s is abstracted.

If you have any questions, the Drasi Team is most active over on our discord channel (https://aka.ms/drasidiscord) and we would love to answer your questions and help ypu get started using Drasi.

Re: Drasi: Microsoft's open source data processing platform for event-driven systems

#76

Post by agentofreality »

devjab wrote: post #47
otterley wrote: post #22

Looks very Azure-centric. Both installation guides ( https://drasi.io/how-to-guides/install-sample-applications/b... and https://drasi.io/how-to-guides/install-sample-applications/c... ) require Azure to work. And then there's this: > Installing Drasi in an EKS cluster can be significantly more complex than a standard installation on other platforms. Instead of downloading a CLI binary using the provided installation…

Every bit of Microsoft open source is created at least partly as a sales strategy for Azure. They usually start within the Azure infrastructure because, well, why wouldn’t they? Then eventually they tend make it to where you can use them outside of Azure but they never quite leave the part where they are “better” if you’re an Azure customer. Time will tell if Drasi is going to go the path where it becomes more easily…

As mentioned above, Kubernetes is (intended to be) our only platform dependency right now. Drasi is not yet ready for production use and can be explored using k3s, kind, AKS, and EKS, which we felt provided sufficient initial options for people to choose from.

In the coming weeks we will get more of our Sources and Reactions documented as well as docs on how to create custom Sources and Reactions. In the short term, if people have Sources and Reactions they want so they can integrate with a wider range of up and downstream systems, we would love to help support their efforts in developing these.

The Drasi Team is most active over on discord channel (https://aka.ms/drasidiscord), where we are happy to answer questions and help people get started using Drasi.

Re: Drasi: Microsoft's open source data processing platform for event-driven systems

#77

Post by agentofreality »

jameslevy wrote: post #28
otterley wrote: post #22

Looks very Azure-centric. Both installation guides ( https://drasi.io/how-to-guides/install-sample-applications/b... and https://drasi.io/how-to-guides/install-sample-applications/c... ) require Azure to work. And then there's this: > Installing Drasi in an EKS cluster can be significantly more complex than a standard installation on other platforms. Instead of downloading a CLI binary using the provided installation…

Does it require Azure to work? Or could the Azure steps be relatively easily be swapped out for AWS/GCP/etc?

Drasi does not require Azure to work. You install Drasi on a Kubernetes cluster, configure a Drasi Source to connect to a supported database, write Continuous Queries in Cypher Query Language, and configure a Drasi Reaction to do something with the output of the Continuous Query. For K8S, you can currently use k3s, kind, AKS (docs coming soon), or EKS (requires some work arounds).

Drasi docs are here - https://drasi.io/

The Drasi Team is most active over on discord, where we are happy to answer questions and help get people started using Drasi (https://aka.ms/drasidiscord)

Re: Drasi: Microsoft's open source data processing platform for event-driven systems

#78

Post by agentofreality »

emmanueloga_ wrote: post #69

They don't mention "CDC" (Change Data Capture) directly anywhere, but I think that's what Drasi is? (they call it "Data Change Processing platform"). "Debezium", an alternative CDC system, is mentioned in the documentation and sources [1]. I'm not sure if Drasi uses Debezium, or aims to be compatible with it. Maybe someone here can shine more light on the relationship between these two? -- 1: https://github.com/drasi…

Hi, I am the Drasi engineering manager, maybe I can clarify. Drasi is not intended to be another CDC alternative. It doesn't compete with Debezium, in fact we already have some integration with Debezium and hope to do more in the future.

People often use CDC to replicate, consolidate, filter, and transform data. And sometimes they use it as a source of change events to build components/services that look for specific data changes and do something when they detect those changes. This kind of components/services tend to be more complex to build, operate, and maintain than expected. Especially if they bring together data from multiple sources, have complex criteria, need to react in near real-time, be secure, and resilient. Most people that have had to build these can probably agree that they would like it to be less complex.

Drasi was created so people don't have to build this kind of component/service. They just write a Continuous Query (in Cypher), and then configure it to connect to supported Data Sources and Reactions (which do something when changes are detected).

Drasi manages the connection to the Source systems to get the low-level changes when they occur (sometimes using the Debezium library), maintains a perpetually accurate result set for the Continuous Query, and every time a source change results in a change to the Continuous Query result, Drasi generates a diff and sends it to the set of subscribed Reactions. The Reactions do something with those diffs depending on their purpose e.g. update a database, post an event, send an email, send a text message. All of this with no code in a platform that can scale to support many such queries.

There is more to it, but a good starting point is if you ever think to yourself that you want to query a database and then compare the results to a previous query result, and you want to do this periodically, you might consider Drasi as an alternative.

The Drasi Team is most active over on our discord channel(https://aka.ms/drasidiscord) and we would be happy to answer questions and help you evaluate whether Drasi is something that might be useful to you.

Re: Drasi: Microsoft's open source data processing platform for event-driven systems

#79

Post by agentofreality »

iamstan23 wrote: post #48

Weird thing about this project is that neither the website ( https://drasi.io ) or the repo ( https://github.com/drasi-project/drasi-platform ) mention that it's a Microsoft project. Also the only cloud provider it has installation instructions for is AWS's EKS platform. Yet it has integration instructions for Azure CosmosDb Gremlin API. That one customer out there using EKS and Gremlin on CosmosDb is probably over t…

I am the Drasi engineering manager, and I can assure you that any weirdness you sense is purely accidental.

The project was announced through standard Microsoft channels by Mark Russinovich and is part of the Microsoft Collaboration on GitHub. But we are predominantly an open source project from the Azure Incubations team which has a history of releasing open source projects. So we don't feel the need to constantly remind everybody that we are a Microsoft project and team.

The documentation site is missing some content that wasn't ready in time for the release but it includes AKS install instructions as well as additional Source and Reaction docs. These will be out soon.

If you know that customer that uses EKS and Cosmos Gremlin, please let us know, we would also be over the moon.

In any case, the Drasi Team is most active over on our discord channel(https://aka.ms/drasidiscord) where we would love to answer any questions you have about Drasi and help you get up and running.

Re: Drasi: Microsoft's open source data processing platform for event-driven systems

#80

Post by emmanueloga_ »

agentofreality wrote: post #78
emmanueloga_ wrote: post #69

They don't mention "CDC" (Change Data Capture) directly anywhere, but I think that's what Drasi is? (they call it "Data Change Processing platform"). "Debezium", an alternative CDC system, is mentioned in the documentation and sources [1]. I'm not sure if Drasi uses Debezium, or aims to be compatible with it. Maybe someone here can shine more light on the relationship between these two? -- 1: https://github.com/drasi…

Hi, I am the Drasi engineering manager, maybe I can clarify. Drasi is not intended to be another CDC alternative. It doesn't compete with Debezium, in fact we already have some integration with Debezium and hope to do more in the future. People often use CDC to replicate, consolidate, filter, and transform data. And sometimes they use it as a source of change events to build components/services that look for specific…

Thank you for this answer! It makes more sense now that you don’t call Drasi a CDC, I think it could be useful to clarify the distinction in the docs (perhaps also comparing and contrasting with other systems that appear to be closely related, like streaming databases).
Post reply on HN