I want to see this married with the Meltano project from Gitlab - it would create an unprecedented end-to-end data environment. https://meltano.com/
Love the thought. We weren't aware of this project - thanks for the pointer. Will follow-up.
Rudder, an open source Segment alternative
61–70 of 103 posts
Re: Rudder, an open source Segment alternative
#62Is it really so difficult for engineers to create a task to process a Kafka topic? It takes one day to write a program to consume from a topic of events and push to an API like Amplitude, and you have total flexibility in how you push to those integrations. Why would you use postgres for an event professing system? This seems like an inefficient architecture.
Here's a blog from Segment explaining how/why they use MySQL in a similar fashion to how Rudder seems to be using Postgres: https://segment.com/blog/introducing-centrifuge/
Re: Rudder, an open source Segment alternative
#63Is it really so difficult for engineers to create a task to process a Kafka topic? It takes one day to write a program to consume from a topic of events and push to an API like Amplitude, and you have total flexibility in how you push to those integrations. Why would you use postgres for an event professing system? This seems like an inefficient architecture.
How would you get ordering for millions of users? How many Kafka topics would you create? How would you managed failed events, would you reorder whole queue?
I don't think it is inefficient. Segment blog linked below talks about specifics of the problem.
Re: Rudder, an open source Segment alternative
#64One issue I had with Segment, I couldn't run real-time transformation of the event to join data from our data tables. We eventually got over it with AWS Lambda and sending it back to Segment. Segment recently announced functions to help on this, still could not get my hands on it. Any plans on this?
Hey there! I work at Segment, and I'm one of the engineers working on Segment Functions. If you let me know your Segment workspace's name (get in touch through jason.tu@segment.com), I can grant you beta access.
Thanks. Will drop an email.
Re: Rudder, an open source Segment alternative
#65We are in the mobile game industry. Due to the incredible high volume events data per user generates per day, we need to join certain events together to reduce the events amount send to our analytics platform. There is no other segment tool can do this for us. We have been working with Rudder Labs to solve this problem. They have been really helpful and act super fast to our requests and suggestions. With Rudder Labs SDK, we are able to join event data on the platform(which we choose to host internally) + all other segment tool features. Besides that, since we run a freemium game so that most of our users are free users/non-payers, cost is another thing we have to consider, current segment market pricing apparently is way too high for us. Rudder labs solves this problem for us as well. Great deal!
Re: Rudder, an open source Segment alternative
#66We are in the mobile game industry. Due to the incredible high volume events data per user generates per day, we need to join certain events together to reduce the events amount send to our analytics platform. There is no other segment tool can do this for us. We have been working with Rudder Labs to solve this problem. They have been really helpful and act super fast to our requests and suggestions. With Rudder Labs…
Thanks for being our first pilot and for all your patience while we were fixing/improving the product.
Re: Rudder, an open source Segment alternative
#67We are in the mobile game industry. Due to the incredible high volume events data per user generates per day, we need to join certain events together to reduce the events amount send to our analytics platform. There is no other segment tool can do this for us. We have been working with Rudder Labs to solve this problem. They have been really helpful and act super fast to our requests and suggestions. With Rudder Labs…
> Due to the incredible high volume events data per user generates per day, we need to join certain events together to reduce the events amount send to our analytics platform.
Don't gloss over the fact that telemetry over cell networks can be costly for users (more and more plans are unlimited, so this doesn't worry me as much) and draining on batteries. However you do it, data that's not latency-critical should be buffered and batched.
Re: Rudder, an open source Segment alternative
#68We are in the mobile game industry. Due to the incredible high volume events data per user generates per day, we need to join certain events together to reduce the events amount send to our analytics platform. There is no other segment tool can do this for us. We have been working with Rudder Labs to solve this problem. They have been really helpful and act super fast to our requests and suggestions. With Rudder Labs…
> Due to the incredible high volume events data per user generates per day, we need to join certain events together to reduce the events amount send to our analytics platform. Don't gloss over the fact that telemetry over cell networks can be costly for users (more and more plans are unlimited, so this doesn't worry me as much) and draining on batteries. However you do it, data that's not latency-critical should be b…
Yes, the SDK buffers and batches events before sending to Rudder BE. In the BE, the events are combined (value summed up, not batched) before it is sent to the analytics platform. We wrote a blog for this interesting use case
Re: Rudder, an open source Segment alternative
#69How does this compare to Snowplow?
Re: Rudder, an open source Segment alternative
#70Interesting, we've built our in-house solution that is also Go based, also writing to a postgres db besides forwarding events, but much simpler, without a UI and comes with backend sdks already.
What I found interesting is that you wrote 3k/events per second on a rather beefy 2xlarge machine. Our version is MUCH less demanding, I wonder if there isn't a lot of performance left on the table here.
I'll keep this in mind once we've grown out of our solution, though.