[deleted]
We saved $50k/year with a Go microservice coded in a hackathon
31–40 of 264 posts
Re: We saved $50k/year with a Go microservice coded in a hackathon
#32I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…
The user data is most likely in rows instead of columns. Instead of having id, name, age, gender 1213, fake, 60, female they would have property_id, user_id, value 1 (assume age), 1213, 60 2 (gender), 1213, female This gives them the freedom to add more properties to the user without always having to add a column to the users table. When querying the database you'll have to do unions or joins.
Hmm, that does seem probable. In fact that could make the SQL even more efficient as you'd only need a combined index on the 'prop' and 'value' columns, rather than N arbitrary combinations of indexes that may or may not be used.
Edit: Had some bad attempt at writing this query but it's rather late and it made no sense.
Re: We saved $50k/year with a Go microservice coded in a hackathon
#33Earlier quoted context omitted.
The user data is most likely in rows instead of columns. Instead of having id, name, age, gender 1213, fake, 60, female they would have property_id, user_id, value 1 (assume age), 1213, 60 2 (gender), 1213, female This gives them the freedom to add more properties to the user without always having to add a column to the users table. When querying the database you'll have to do unions or joins.
Hmm, that does seem probable. In fact that could make the SQL even more efficient as you'd only need a combined index on the 'prop' and 'value' columns, rather than N arbitrary combinations of indexes that may or may not be used. Edit: Had some bad attempt at writing this query but it's rather late and it made no sense.
You would need to have a new join for each new property
SELECT DISTINCT loyaltyMemberID
from members as m
INNER JOIN properties as p1 on m.id = p1.user_id
INNER JOIN properties as p2 on m.id = p2.user_id
INNER JOIN properties as p3 on m.id = p3.user_id
AND (p1.prop = 'gender' AND p1.value = x)
AND ((p2.prop = 'age' AND p2.value = y) OR (p3.prop = 'censor' AND p3.value = z))Re: We saved $50k/year with a Go microservice coded in a hackathon
#34I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…
>Seems like a fundamental misunderstanding of SQL rather than a particularly hard problem to solve.
Without knowing the rest of their stack, or what their data ingestion looks like, I think your query is oversimplified. If they are doing a union, then it's likely they aren't querying one table, but they are querying multiple tables. The article mentions that individual customers had as many as 500 million rows. Likely each customer has their own set of data they also pipe into the system. Next their own custom query language may support more complex algebra than standard equality.
IMO, the article doesn't sufficiently describe the problem for us to understand why their solution works. To you and I there are 100 other solutions they could have tried that seem simpler than the one they presented.
It's less likely that they overengineered - we are probably just underinformed.
Re: We saved $50k/year with a Go microservice coded in a hackathon
#35what did the Go solution replace? I only read about DB changes, and cannot draw any conclusions.
It replaced a couple sub queries/ common table expression.
Somehow using the same data store makes it a microservice rather than a distributed monolith.
Of course it’s a lot less 2018/webscale to just optimize your DB rather than replace something with Go
Re: We saved $50k/year with a Go microservice coded in a hackathon
#36I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…
The user data is most likely in rows instead of columns. Instead of having id, name, age, gender 1213, fake, 60, female they would have property_id, user_id, value 1 (assume age), 1213, 60 2 (gender), 1213, female This gives them the freedom to add more properties to the user without always having to add a column to the users table. When querying the database you'll have to do unions or joins.
Is `value` a string here?
Re: We saved $50k/year with a Go microservice coded in a hackathon
#37Earlier quoted context omitted.
Yeah, sadly, this is not too much of an exaggeration. I've worked on teams that insisted they needed DynamoDB, because, well, Dynamo is for "Big Data", and they certainly wouldn't work somewhere that had "Small Data"! Replace the buzzwords/products as applicable; you could actually probably just scramble them and it'd work just as well, since someone out there thinks "RabbitMQ means Web Scale", etc. SQL databases are…
I think this is a case for the return of the traditional "sysadmin" as "devops"/"SRE" is now the role of unblocking deploying a solution instead of questioning it's complexity/fitness.
While I agree, I also think that overt gating and approval processes create a high tension dynamic that frequently breaks down, whether it's ops v. dev, security v. dev, or others. It's easy for people to get their pride wounded, and they end up encouraged to find workarounds to the process. The simple answers to this are pretty much imaginary, unfortunately.
Re: We saved $50k/year with a Go microservice coded in a hackathon
#38I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…
Can be tricky to use EAV data models with traditional ORMs.. this type of functionality can often be slow or require plugins, if implemented at all:
https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...
Re: We saved $50k/year with a Go microservice coded in a hackathon
#39Earlier quoted context omitted.
The user data is most likely in rows instead of columns. Instead of having id, name, age, gender 1213, fake, 60, female they would have property_id, user_id, value 1 (assume age), 1213, 60 2 (gender), 1213, female This gives them the freedom to add more properties to the user without always having to add a column to the users table. When querying the database you'll have to do unions or joins.
Is `value` a string here?
Depends
You could have
property_id, user_id, value(string)
1 (assume age), 1213, 60
2 (gender), 1213, female property_id, user_id, value_str, value_int
1 (assume age), 1213, null, 60
2 (gender), 1213, female, nullRe: We saved $50k/year with a Go microservice coded in a hackathon
#40This is cool but it seems strange (to me) that this was a “Hackathon” project as opposed to just a stand-alone problem to be addressed as a normal course of doing business. It doesn’t make the solution less cool. It just seems like a strange distinction on what a Hackathon is.