Building a self-hosted copy of MixPanel is surprisingly easy. I too did it for a desktop app I'm working on [1]. Took a few days, not more, and is much more flexible. Another advantage is that I retain control of the data.
How we saved money by replacing Mixpanel with BigQuery and K8S
21–30 of 70 posts
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#22Built a similar thing last year -- nginx (openresty) + lua -> NSQ -> python streaming -> BQ Using Metabase for visualization. The other key benefit is you can warehouse the rest of your data in BQ as well -- where it can be easily linked to your non-event data; marketing data from other systems, etc.
Forgot to add -- analytics.js on the frontend...
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#23Just to flesh out the vendor comparison... Five weeks of, say, three engineers with $150k salaries ($200k fully loaded) is $60k. People say that the majority of software is after initial development. Let's say 20% is development [0]. That means this project costs $300k total and lives for 5 years before the team decides to rewrite it. That brings us to $60k / year. This is good savings! My general guidance to folks t…
[deleted]
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#24> Having said that, the event data is crucial and no data loss can be tolerated.
May I ask why? Seems like exactly the kind of data where some loss is fine.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#25This article epitomizes the old adage: software engineers are terrible at estimating. I have absolutely no stake in Mixpanel, and it has its own flaws, but seeing something as misleading as this on the front page of HN means I have to write a clarifying, if not somewhat edifying, comment =/ 1. Right off the bat, there's the cost of building all of this + maintaining it. Something like Mixpanel at scale requires at le…
I think you are not counting: QA, user experience, DevOps, technical writing, security auditing, user training, integration support...
What happens if the system collecting all of your data has vulnerability? suddenly all of your apps and data are owned.
What if it's inefficient? then you have DDoSed yourself.
Non-functional requirements cannot be solved with a functional requirement mindset.
What if you are losing data? what if you aggregate it incorrectly? what if data is ingested in the wrong order? ... you get the idea.
It's not a 3 people problem. When you take on a hard problem like this you have 10 different things to cover and you need to pick all of them.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#26Just to flesh out the vendor comparison... Five weeks of, say, three engineers with $150k salaries ($200k fully loaded) is $60k. People say that the majority of software is after initial development. Let's say 20% is development [0]. That means this project costs $300k total and lives for 5 years before the team decides to rewrite it. That brings us to $60k / year. This is good savings! My general guidance to folks t…
> There's a reason MixPanel (and any other vendor) has so many engineers. You're paying them to have already made the mistakes that you're about to make.
The real reason Mixpanel (et al) have so many engineers is:
A) They need to scale their service to reliably provide to N corporations which is frequently non-trivial.
B) They need to implement the 80% of features that aren't critical path for Company A because some subset of them are for Company B,C,D,E,F,G,H,I,J,K,etc.
C) I've seen software vendors make some pretty terrible and painfully obvious mistakes.
Idk, maybe I'm crazy but I have very little faith in this idea that more engineers = better decisions were made. I've just never found quantity has an impact on quality of the end product, only the speed at which its produced.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#27Does anyone have any experience of negotiating your costs down with Mixpanel? Years ago I remember my CTO ringing up our hosting provider and in the space of a phone call bringing some of our costs down 90%.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#28I do some of this but in a slightly different, maybe simpler way:
1. App runs in App Engine (flex). It includes a simple /collect endpoint for mobile events that just logs to /var/log/app_engine/custom_logs/track.json which automatically gets picked up by Google's Stackdriver. App code can just directly log to that file as well to record server-side events. (no network involved in recording events).
2. Stackdriver stores logs in Cloud Storage and also dispatches a Cloud PubSub push to a configured topic.
3. Another mini App Engine flex app receives the message, does the ETL, and inserts the row into BigQuery.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#29Just to flesh out the vendor comparison... Five weeks of, say, three engineers with $150k salaries ($200k fully loaded) is $60k. People say that the majority of software is after initial development. Let's say 20% is development [0]. That means this project costs $300k total and lives for 5 years before the team decides to rewrite it. That brings us to $60k / year. This is good savings! My general guidance to folks t…
The whole thing is a big ball of mud. If you commit to a vendor you risk vendor-lock in. If you commit to hand-roll you risk runaway deadlines, feature creep, and a slew of other threats. Once you pick one or the other and get it to a business functional level you lose all momentum and willpower to pivot to the best solution now that you have a "good enough" approach.
Sometimes "good enough" is, well, good enough. How accurate good is good enough? 80%? 95%? It's different for every business, and the needs change over time. 60% accurate for an eagerly stage bootstrapped company? Gold! 60% accuracy on a growth-stage business where you should know what to do? Terrible.
Be limber enough to eschew the tools that aren't working. But "good enough" is, well, good enough for most folks. Perfection the enemy of progress and all that.
Re: How we saved money by replacing Mixpanel with BigQuery and K8S
#30I do some of this but in a slightly different, maybe simpler way: 1. App runs in App Engine (flex). It includes a simple /collect endpoint for mobile events that just logs to /var/log/app_engine/custom_logs/track.json which automatically gets picked up by Google's Stackdriver. App code can just directly log to that file as well to record server-side events. (no network involved in recording events). 2. Stackdriver st…
The downside of app engine over GCE/GKE is lack of sustained use discounts