Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We saved $50k/year with a Go microservice coded in a hackathon

movio.co

Post reply on HN
Print view View original

11–20 of 264 posts

Re: We saved $50k/year with a Go microservice coded in a hackathon

#11

Post by wasd »

owenmarshall wrote: post #9
wasd wrote: post #3

Great story, thanks for sharing. I wanted to ask a quick question about something: > Refreshing caches automatically How do people usually handle this? Is this something done on the application layer or database layer? Where is the cache stored?

Caches could be stored in the database in a materialized view, in an external service like memcache or redis, or even in the application itself. Expiry can take a few different forms. Some caches have a defined space and use a replacement scheme like "fill the cache up, then remove the least recently accessed value". Some don't have defined sizes but instead remove entries based on timestamps (cache for n minutes). S…

That's extremely helpful. Thanks.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#12

Post by [deleted] »

orf wrote: post #8

I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…

[deleted]

Re: We saved $50k/year with a Go microservice coded in a hackathon

#13

Post by owenmarshall »

orf wrote: post #8

I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…

> Stuff like this is bread and butter SQL.

Ten or fifteen years ago, sure - a DBA would look at a query plan and figure out how to do it properly. Worse case you'd slap a materialized view in and query that.

But this is 2018! Programmers don't want to treat the database as anything but one big key value store ;)

Re: We saved $50k/year with a Go microservice coded in a hackathon

#14

Post by hippich »

What else it shows - how expensive AWS hardware vs hosting own hardware. I guess you have to consider how often you have to scale, but hetzner offers dedicated servers with 64Gb and NVMe drives starting from 54 euros per month - https://www.hetzner.com/dedicated-rootserver?country=us - compare that to $580 per month these guys were paying for i3.2xlarge instance..

Re: We saved $50k/year with a Go microservice coded in a hackathon

#15

Post by strukturedkaos »

hnov wrote: post #4

Elasticsearch works for this use-case quite well, you'd store a fairly straightforward representation of the MySQL row as a document, query by the fields you're interested in and ask for aggregations on the matching documents. Common bitsets get cached automatically.

This exactly how we implemented the rules engine in Kevy. We construct an Elasticsearch query based on the rules selected in the UI. Then we use the scroll API to retrieve the matching documents.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#16

Post by EpicEng »

hnov wrote: post #4

Elasticsearch works for this use-case quite well, you'd store a fairly straightforward representation of the MySQL row as a document, query by the fields you're interested in and ask for aggregations on the matching documents. Common bitsets get cached automatically.

Or maybe just write a less insane SQL query to begin with?

Re: We saved $50k/year with a Go microservice coded in a hackathon

#17

Post by tytytytytytytyt »

wiradikusuma wrote: post #7

Just to be sure, any language will do right? Because I thought it was about Go vs (put your slow programming language here).

It seems like it from his explanation. He says he'll explain why he thinks only Go could have done it, but then nothing specific to Go really materializes. This is how it seems to usually go with posts like these.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#19

Post by cookiecaper »

owenmarshall wrote: post #13
orf wrote: post #8

I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…

> Stuff like this is bread and butter SQL. Ten or fifteen years ago, sure - a DBA would look at a query plan and figure out how to do it properly. Worse case you'd slap a materialized view in and query that. But this is 2018! Programmers don't want to treat the database as anything but one big key value store ;)

Yeah, sadly, this is not too much of an exaggeration. I've worked on teams that insisted they needed DynamoDB, because, well, Dynamo is for "Big Data", and they certainly wouldn't work somewhere that had "Small Data"! Replace the buzzwords/products as applicable; you could actually probably just scramble them and it'd work just as well, since someone out there thinks "RabbitMQ means Web Scale", etc.

SQL databases are amazing, robust examples of engineering. They are your friends and they're the appropriate choice for the vast majority of software. They are not outmoded or passe. Though I acknowledge there is a separate use case for K-V stores, I almost want to make policy preventing their use just because I know so many developers will abuse them badly and then stare back at you blankly during the semi-annual massive downtime event, muttering something like "Well, it's based on research at Google, so I'm sure there's a way to recover the data..."

Re: We saved $50k/year with a Go microservice coded in a hackathon

#20

Post by ThrustVectoring »

orf wrote: post #8

I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…

The UI they showed in the blog post looks like it has enough data available to generate that kind of query, too. Like, the ands/ors/nots are right there on the page, the filters are already there too getting translated to SQL as well, just mash them together and you get the same "algebra of sets" stuff right in the WHERE clause.

As it stands the SQL query is quite silly. It gets a list of every user ID that is included by each filter and compares which ones are in the filters you want and not the filters you don't want. Much better is to pass the filters into SQL, let it figure out which users match the filters you want and not the filters you don't, and just use that result.

Post reply on HN