We saved $50k/year with a Go microservice coded in a hackathon

It's like those infamous enterprise benchmarks from yesteryear.

"NYSE moves from Solaris to RHEL and gains a 800% performance benefit".

While I don't doubt a brand new RHEL has more performance optimizations than what is actually a SunOS 5.2, the guys benchmarking should have also said that the original hardware was the equivalent of a PIII and now they're moving to the latest Xeons.

I'm not kidding, I've actually seen a press release like this.

Blogpost author here. Thank you so much for all the attention, comments, upvotes, likes, retweets, etc! I've done a pass over the comments and can't really answer them all but I'd like to clarify a few things:

There seems to be a general opinion trend that the queries generated by the group builder algorithm are very inefficient, that it'd be easy to come up with a solution with much better response times, and that that would be achievable in any reasonable programming language in roughly the same time with similar results.

The language argument will always be controversial and I won't address it here; we have a point of view that is expressed in the Conclusion and on this blogpost: https://movio.co/en/blog/migrate-Scala-to-Go/

I can imagine that seeing a query with JOINs, subqueries, GROUP BYs and UNIONs can raise some eyebrows, but there is some lacking context in that story, and that's on me. Here's some of that context:

* The schema that the group builder algorithm operates on is not uniform in nature or composed of simple yes/no fields; it's an incredibly complex legacy schema that to a large degree wasn't even up to Movio: it's been up to the film industry as a whole, and it has evolved over the years, as is the case everywhere. Note that every different kind of filter translates to a very different kind of query, and we have more than 120 different filters, sometimes with dynamic parameters, and sometimes even bespoke for a particular customer!

* The group builder algorithm predates the team that built this service (myself included), as well as predating the first commercial release of Elasticsearch, MariaDB, mainstream Go success, etc. Nevertheless, it's still very fast and is being used today by ~88% of our customers (i.e. all the non-behemoths). It's been successful for many years, and continues to be, for the most part.

* But I don't like it because it's fast: I like it because it's simple and flexible. It allows our customers to build a really complex (and arbitrary) tree of filters to segment their loyalty member base, and it compiles all of that into one big SQL query, that in most cases is quite performant. That's pretty awesome. But yes; it doesn't scale to several million members.

* Migrating the very engine of the main product of a company is not a decision that is taken lightly. As is the case with every big company I can remember (e.g. Twitter, SoundCloud), behind a big success story there's always a legacy monolith, and our case is no exception. From that standpoint, achieving such breakthrough (i.e. cost reduction + significant response time improvement) within one hackathon day is really not all that common in my experience. Definitely something worth sharing, IMO.

Hopefully that clarifies some of the questions :) Cheers.

> those fools at Google, Facebook, Twitter, Pinterest, Amazon

... have dedicated hundreds of engineers and millions of dollars to nothing more than keeping MySQL up, running, and not crapping the bed every time someone looks at it funny. If you can afford that resource expenditure, by all means go nuts with MySQL. Most companies can't and would be far better served by something which doesn't need that amount of handholding to serve its basic purpose.

I'm sadly not that much of a Vim expert, Hackernews lacks formatting for a strikethrough and I wasn't sure how a ~~/dev/null~~ would be interpreted.

Confession time: in my first job, I build something like this (and it worked pretty well in the sense that it was very flexible), but then I also had to do a 'select' based on iirc 14 of such properties. I don't really recall the exact problem I had at first, but my solution was to create two separate (temporary) tables, select 7 of the properties into one and 7 into the other, run a select on both of those tables, then join the results in code. This ran at and acceptable speed (I must have done something so that adding criteria made the run time increase non-linearly - doing it on 14 was orders of magnitude slower than on 7).

Then years later I ran into the guy who had to do some work on it after I left that company. I must have scarred him pretty badly, because he remembered it enough to bring it up as pretty much the first topic after the obligatory 'hey so what are you up to nowadays'. When I think back about it now, it was a cringey solution - then again, this was at a company where nobody had ever heard of a 'database index' (or if they did, never mentioned or implemented them).