We saved $50k/year with a Go microservice coded in a hackathon
1–10 of 264 posts
Re: We saved $50k/year with a Go microservice coded in a hackathon
#2Cool post - Definitely an improvement and a good fit for Go services. I'm curious - did you run performance comparisons on optimizing the SQL itself as compared to adding this additional service?
Maybe I'm crazy, but just looking at that query it seems like there's definitely room for improvement with the SQL alone. Unless the "..." is hiding something I'm missing?
Re: We saved $50k/year with a Go microservice coded in a hackathon
#3Great story, thanks for sharing. I wanted to ask a quick question about something:
> Refreshing caches automatically
How do people usually handle this? Is this something done on the application layer or database layer? Where is the cache stored?
Re: We saved $50k/year with a Go microservice coded in a hackathon
#4Elasticsearch works for this use-case quite well, you'd store a fairly straightforward representation of the MySQL row as a document, query by the fields you're interested in and ask for aggregations on the matching documents. Common bitsets get cached automatically.
Re: We saved $50k/year with a Go microservice coded in a hackathon
#5This seems a lot less about go and much more about changing the approach to the problem.
Re: We saved $50k/year with a Go microservice coded in a hackathon
#6I once built a micro service in Go that saved about as much. Took me a couple of days (meetings and all that) from start to deployed. It still runs on the same cheap aws instance. Go is really good at that sort of thing.
Re: We saved $50k/year with a Go microservice coded in a hackathon
#7Just to be sure, any language will do right? Because I thought it was about Go vs (put your slow programming language here).
Re: We saved $50k/year with a Go microservice coded in a hackathon
#8I don't quite get this. How fast was running this query:
Select loyaltyMemberID
from table
WHERE gender = x
AND (age = y OR censor = z)
Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this.
Obviously I'm armchairing hard here but it smells like over engineering from this post alone. Stuff like this is bread and butter SQL.
Edit: I've just read the query in the post again and I really can't understand why you would write it like that. Am I missing something here?
Seems like a fundamental misunderstanding of SQL rather than a particularly hard problem to solve.
Re: We saved $50k/year with a Go microservice coded in a hackathon
#9Great story, thanks for sharing. I wanted to ask a quick question about something: > Refreshing caches automatically How do people usually handle this? Is this something done on the application layer or database layer? Where is the cache stored?
Caches could be stored in the database in a materialized view, in an external service like memcache or redis, or even in the application itself.
Expiry can take a few different forms. Some caches have a defined space and use a replacement scheme like "fill the cache up, then remove the least recently accessed value". Some don't have defined sizes but instead remove entries based on timestamps (cache for n minutes). Some depend on invalidation messages from the application. It all depends on the applications needs.
The most important thing to remember is that caching means your system becomes inherently a distributed one. State can become split across multiple sources, the cache can return stale data, invalidation might not happen when you expect, ...
That's fine, but you have to program accordingly.
Re: We saved $50k/year with a Go microservice coded in a hackathon
#10I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…
Damn, you're not kidding. I wonder why they needed more than one query here plus UNION is slowwwwwwww. They never mention how frequent this query needs to run either, only the amounts of data involved in some aspects of this table.