Did you not consider snowplow analytics with something like periscope over the top?
How we saved money by replacing Mixpanel with BigQuery and K8S
31–40 of 70 posts
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#32"During July 2017, our new data analytics pipeline was processing about 500 events every second."
Is 500 events per second considered high throughput over here, or do other people feel they really just needed fault tolerance from Google cloud?
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#33This article epitomizes the old adage: software engineers are terrible at estimating. I have absolutely no stake in Mixpanel, and it has its own flaws, but seeing something as misleading as this on the front page of HN means I have to write a clarifying, if not somewhat edifying, comment =/ 1. Right off the bat, there's the cost of building all of this + maintaining it. Something like Mixpanel at scale requires at le…
I agree with all three of your points concerning Mixpanel and what it would take to replace it with something similar. But I think you're being harsh when you say "This article epitomizes the old adage: software engineers are terrible at estimating".
I don't see anywhere that the article says they have replaced all of Mixpanel. It only says that they were able to replicate the functionality their client actually needed. I don't see them making claims that the solution they built is anywhere comparable to the full solution from Mixpanel. Only that it provides all the features their client needed.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#34For me, the issue is not cost. It's the data. You're paying for what's essentially a data warehouse and BI tool... but you have little to no control over it. Sure, it's fully managed, but you want to change X? You want to evolve your schema? You need to change some data in history (I know)? You want to build real-time systems based on these streams of data? You want Mixpanel UI for other datasets in your company?
All of these are issues (that I have faced). From what I've seen, people just keep downloading Mixpanel data and uploading to their DWH, sort of voiding the reason to implement Mixpanel in the first place.
The tool is lovely, their export policy is great, but there's something about actually owning your data.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#35"During July 2017, our new data analytics pipeline was processing about 500 events every second." Is 500 events per second considered high throughput over here, or do other people feel they really just needed fault tolerance from Google cloud?
"500 events per second" is approximately zero. We go to 3rd providers, not for "tolerance" but because getting milk off-the-shelf in the supermarket is easier than feeding and milking your own cow (unless you need raw milk or something).
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#36# Requirements Unless you have restrictions about how your data moves and where it is stored, or need to have a trusted computing base with no externally developed software, or have very strict requirements that no available service implements (unlikely), you are better off just using an open source solution or paying for a service. # Real cost Even if you have to pay for a commercial service, in comparison, the TCO…
To be fair though, it seams to me that they moved from one service that was providing 150% of what they need to another one, Google cloud, that is providing 90% of what they need plus blueprint exemples on how to implement the 10% that are missing.
If I'm not wrong this is totally reasonable, especially if it gives them the flexibility to mine this data the way they want, since it's really close to the core of their business.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#37I dunno. The way I look at it, whatever you could have been working on to advance your company when Mixpanel took care of things, you weren't when you were building this and can't when you have to maintain it.
This was done by a contractor.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#38Earlier quoted context omitted.
The whole thing is a big ball of mud. If you commit to a vendor you risk vendor-lock in. If you commit to hand-roll you risk runaway deadlines, feature creep, and a slew of other threats. Once you pick one or the other and get it to a business functional level you lose all momentum and willpower to pivot to the best solution now that you have a "good enough" approach.
Software is always 'Pay me now, or pay me later'. In the end I believe you usually always end up paying pretty close to the same, it just depends on if you are happy with the solution in 18-36 months or hate it.
The tricky bit is when opportunity costs come in... "Pay me now" means sacrificing clear opportunities in the visible horizon, "pay me later" does the same thing at an less-controlled future point sacrificing presently unknown opportunities.
Personally I much prefer 'pay me now', because 9 times out of 10, IME, some form of technical debt underlies the reasoning about why a team can't do something. 'It always pays to take the pain up front', basically.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#39> Having said that, the event data is crucial and no data loss can be tolerated. May I ask why? Seems like exactly the kind of data where some loss is fine.
Billing, perhaps.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#40Honestly you don't even need some kind of fancy google architecture, query engine, or columnar data store.
I recently implemented 99% of what mixpanel provides using S3 and lambda. I stored the events on S3 (actually get requests to an S3 bucket with logging turned on and what I wanted to track seeialized as JSON in the query string of the request.
From here it's as easy as writing lambda functions to process the logs and output the results in some location where a static web app can visualize them.