71–80 of 103 posts
How does this compare to Snowplow? https://github.com/snowplow/snowplow
I haven't looked into rudder but I'll switch if it offers easier setup and schema.
Interesting, we've built our in-house solution that is also Go based, also writing to a postgres db besides forwarding events, but much simpler, without a UI and comes with backend sdks already. What I found interesting is that you wrote 3k/events per second on a rather beefy 2xlarge machine. Our version is MUCH less demanding, I wonder if there isn't a lot of performance left on the table here. I'll keep this in min…
Currently, transformations are run in nodeJS. So, for every batch of events there is a call into nodeJS from go and that is slow. We do batching/parallel-calls but still.
I think, postgres gets us > 15K/sec throughput.
I tought this was https://www.rudder.io/en/
Interesting, we've built our in-house solution that is also Go based, also writing to a postgres db besides forwarding events, but much simpler, without a UI and comes with backend sdks already. What I found interesting is that you wrote 3k/events per second on a rather beefy 2xlarge machine. Our version is MUCH less demanding, I wonder if there isn't a lot of performance left on the table here. I'll keep this in min…
The bottleneck for us (on that instance) is not postgres but transformations. Transformations are tiny snippets of javascripts which convert the event from Rudder JSON to whatever structure (JSON, keyval etc) that the destinations expect. We also support user defined transformations - functions defined by the user to transform/enhance the event. Currently, transformations are run in nodeJS. So, for every batch of eve…
Segment's pipeline involves JS at some point. We had an issue where our 64 bit integers were down-casted silently. We found out the hard way. We use strings now (perhaps should have used strings right away, I am not necessarily the sharpest tool in the shed).
Earlier quoted context omitted.
The bottleneck for us (on that instance) is not postgres but transformations. Transformations are tiny snippets of javascripts which convert the event from Rudder JSON to whatever structure (JSON, keyval etc) that the destinations expect. We also support user defined transformations - functions defined by the user to transform/enhance the event. Currently, transformations are run in nodeJS. So, for every batch of eve…
What happens if I pass a 64 bit integer, and the Rudder pipeline being in JavaScript silently down-casts it to a 53 bit integer? Segment's pipeline involves JS at some point. We had an issue where our 64 bit integers were down-casted silently. We found out the hard way. We use strings now (perhaps should have used strings right away, I am not necessarily the sharpest tool in the shed).
Earlier quoted context omitted.
You put a spotlight by saying "privacy and security focused alternative to segment" Are there things you believe make segment not privacy and security focused? As a long time user of segment, I find their protocols feature and new data and privacy features world class for this. Having also just left their Synapse conference, privacy and security was the #1 topic of discussion throughout the conference. I would say th…
The author responded to you, but I think this deserves emphasis. True privacy is not about additional features or promises made by a cloud provider. It's not private unless you handle the data yourself, period point blank. This should be obvious.
It is the managed infrastructure. Outage? You can keep sleeping. Google Analytics deprecating their API for a new one? Segment's on it.
The have all your data. They can replay all events from beginning if needed (so they say, never had to ask for this).
How does this compare to Snowplow? https://github.com/snowplow/snowplow
We are aiming at routing the events reliably to destinations, transforming events real-time, storing them into your data warehouse with a dynamic schema and eventually build a data platform with help from the community.
However, I believe it's misleading to call it "open source". The SPPL license is not generally considered to be an open source licence by any meaningful definition, and in particular does not meet the OSI definition and is incompatible with most licences that do.
I understand the need these days to protect against aggressive cloud providers, but there are other ways to achieve that without becoming completely non open source, such as the BSL
(See license.https://opensourceforu.com/2019/06/cockroach-labs-changes-it... )
This looks great, and I look forward to finding time to try it soon. However, I believe it's misleading to call it "open source". The SPPL license is not generally considered to be an open source licence by any meaningful definition, and in particular does not meet the OSI definition and is incompatible with most licences that do. I understand the need these days to protect against aggressive cloud providers, but the…
But at heart we want to build an open-source community while still being a viable business on the likes of MatterMost, Elastic etc.