Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we saved money by replacing Mixpanel with BigQuery and K8S

blog.doit-intl.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: How we saved money by replacing Mixpanel with BigQuery and K8S

#51

Post by tzury »

Smart move, sure enough in this case.

See, once you have the data pipeline in place, you can bring front-end/ reporting platform to generate any reports you wish, from free (e.g Google Data Studio) to self-hosted open source based, up to SaaS solutions which available few hundred bucks per month. Yet, you have full control/access to your data. This is the main benefit IMHO.

In our region, Tel Aviv, where Jelly Button and DoIT are located, $240K per year == 2x great s/w engineers annual salary.

Perhaps it is time for MixPanel's product/exec. people to reconsider pricing model at large scale. I assume there was a discussion between the parties before they went on let's build our own.

Re: How we saved money by replacing Mixpanel with BigQuery and K8S

#54

Post by code4tee »

The challenge for offerings like MixPanel is that the market on other scalable as-a-Service offerings (streaming data, big query, et al) has gotten to the point where it’s far more trival for companies to just roll their own. Trying to build all this 3-4 years ago would have been much much harder. MixPanel’s pricing doesn’t reflect this change in their value-add proposition for larger companies.

Just as a company gets big enough to offer a large revenue stream to Mixpanel is the same time when it probably makes sense to ditch Mixpanel and and just run all this in house for collecting, storing and processing massive data streams. I see a lot of this sort of decision making taking place now.

Re: How we saved money by replacing Mixpanel with BigQuery and K8S

#55

Post by boundlessdreamz »

brianwawok wrote: post #48
jshen wrote: post #9

Why not use App Engine instead of Container Engine? You could likely save more that way.

What? Order of pricing cheap to expensive is GKE, GCE or Container Engine, App Engine.

How is GKE cheaper than GCE?

Re: How we saved money by replacing Mixpanel with BigQuery and K8S

#56

Post by a-saleh »

trjordan wrote: post #4

Just to flesh out the vendor comparison... Five weeks of, say, three engineers with $150k salaries ($200k fully loaded) is $60k. People say that the majority of software is after initial development. Let's say 20% is development [0]. That means this project costs $300k total and lives for 5 years before the team decides to rewrite it. That brings us to $60k / year. This is good savings! My general guidance to folks t…

We went for a vendor to replace one of our internal tools. It looked too costly to pay team of 4 engineers a year to support it.

Unfortunately, the part "implement a vendor and see how you like it" went only with the first half. Complaints of developers supposed to be using the tool were brushed aside as a simple resistance of change. In the end it was implemented by decree from higher management.

Only after all the paperwork was signed, we started to see that the vendor doesnt support our usecases, that their hardware is underpowered for the load we were expecting, e.t.c ...

In the end, I have heard at least 20 various senior and beyond senior engineers from various teams complaining that they spend 20 to 50 % of time working around the vendors bugs, around once a month bringing down vendors infrastructure, because our use-case mandated uploading more data than the infra could handle.

I definitely agree that clear requirement list and clearly scoped project and implementing a vendor to see how you like it is usually better than in-house

On the other hand in-house is usually better than implementing a random vendor because management feels like it ;)

Re: How we saved money by replacing Mixpanel with BigQuery and K8S

#57

Post by brianwawok »

boundlessdreamz wrote: post #55
brianwawok wrote: post #48

Earlier quoted context omitted.

What? Order of pricing cheap to expensive is GKE, GCE or Container Engine, App Engine.

How is GKE cheaper than GCE?

I think I messed up my branding.. I think GKE and GCE are now the same thing?

I was thinking GKE = Kubernetes, and GCE = was their "run container on a VM" thing, but I think GKE and GCE are the same thing...

Re: How we saved money by replacing Mixpanel with BigQuery and K8S

#58

Post by pc86 »

jdwyah wrote: post #45

There's one other "big hammer" for saving money at MixPanel: "send fewer events". We saved 60% (which was no small amount of money) by realizing that sending more than 1 of an event per user per hour was: A) Not necessary B) Not even something you could query in MixPanel. more thoughts in: https://blog.ratelim.it/blog/how-to-save-money-on-event-trac...

That's a great domain name and seems like a neat product. One of those things you never think of then when you see it you think "of course that's a thing, it makes perfect sense."

Re: How we saved money by replacing Mixpanel with BigQuery and K8S

#59

Post by corporateslave3 »

code4tee wrote: post #54

The challenge for offerings like MixPanel is that the market on other scalable as-a-Service offerings (streaming data, big query, et al) has gotten to the point where it’s far more trival for companies to just roll their own. Trying to build all this 3-4 years ago would have been much much harder. MixPanel’s pricing doesn’t reflect this change in their value-add proposition for larger companies. Just as a company get…

This exactly. The big data technology is automating a ton of the distributed systems difficulty from what mixpanel is creating. We built n in house version of mixpanel in six months, that is more customized and more expansive.

Re: How we saved money by replacing Mixpanel with BigQuery and K8S

#60

Post by orliesaurus »

geofft wrote: post #5
orliesaurus wrote: post #3

240K - that's huge! I can't believe MixPanel gets that expensive - wow considering their plans are like: * FREE * 999 a year * Contact us That number 3 must be really spooky!

I had the opposite reaction. If you're in a big city in the US, $240K is the cost including overhead of one, maybe two engineers. If running this in-house takes up a total of a full-time engineering load (i.e., if the team maintaining this wants to add headcount to run this and maintain prior commitments), you haven't saved very much and you might be net negative. (The math may be different in different job markets,…

$240K for 1 Engineer? Wow-zie
Post reply on HN