Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We saved $50k/year with a Go microservice coded in a hackathon

movio.co

Post reply on HN
Print view View original

61–70 of 264 posts

Re: We saved $50k/year with a Go microservice coded in a hackathon

#61

Post by jehlakj »

owenmarshall wrote: post #13
orf wrote: post #8

I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…

> Stuff like this is bread and butter SQL. Ten or fifteen years ago, sure - a DBA would look at a query plan and figure out how to do it properly. Worse case you'd slap a materialized view in and query that. But this is 2018! Programmers don't want to treat the database as anything but one big key value store ;)

You can thank full stack developers for that

Re: We saved $50k/year with a Go microservice coded in a hackathon

#62

Post by badminton1 »

Unless you can directly see how a query can be optimized, first thing you do is get the execution plan (e.g: EXPLAIN query).

The execution plan will tell you how expensive is each bit of your query and help you adjust it.

From there, if things are not getting better, you have a lot of alternatives:

- Consider creating an index

- If the value doesn't change often, consider writing it into another table or caching it.

- Replication, partitioning, sharding, changing the schema.

- Reconsider the requirement being implemented in order to have a more scoped query or to perform the query less often.

Then... OLAP is not OLTP. If you can, do reporting in another database.

Finally, creating your own project in the end may not save you $50,000. How about maintenance? tooling built around it? integration costs? documentation? usability? new hires having to learn about it? You can hire people that already know SQL without having to incur that cost yourself. All the tooling is built, battle-tested and readily available. Plus, skills related to internal tools are harder to trade in the market because they're harder to verify and less transferable.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#63

Post by chiefalchemist »

jehlakj wrote: post #61
owenmarshall wrote: post #13

Earlier quoted context omitted.

> Stuff like this is bread and butter SQL. Ten or fifteen years ago, sure - a DBA would look at a query plan and figure out how to do it properly. Worse case you'd slap a materialized view in and query that. But this is 2018! Programmers don't want to treat the database as anything but one big key value store ;)

You can thank full stack developers for that

Them, or the manager that hired them? If no one up the chain brings on a DBA what are they supposed to do? I hear ya. But this is as much a symptom of naive (and budget stretched) leadership as it is of the hands on deck.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#64

Post by rbranson »

toddBarkus wrote: post #26
cookiecaper wrote: post #19

Earlier quoted context omitted.

Yeah, sadly, this is not too much of an exaggeration. I've worked on teams that insisted they needed DynamoDB, because, well, Dynamo is for "Big Data", and they certainly wouldn't work somewhere that had "Small Data"! Replace the buzzwords/products as applicable; you could actually probably just scramble them and it'd work just as well, since someone out there thinks "RabbitMQ means Web Scale", etc. SQL databases are…

I think this is a case for the return of the traditional "sysadmin" as "devops"/"SRE" is now the role of unblocking deploying a solution instead of questioning it's complexity/fitness.

If your “SRE” team is only “unblocking deploying a solution” then I’m sad to say they are an operations team who has rebranded themselves to appear more relevant.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#65

Post by rbranson »

hippich wrote: post #14

What else it shows - how expensive AWS hardware vs hosting own hardware. I guess you have to consider how often you have to scale, but hetzner offers dedicated servers with 64Gb and NVMe drives starting from 54 euros per month - https://www.hetzner.com/dedicated-rootserver?country=us - compare that to $580 per month these guys were paying for i3.2xlarge instance..

... and then you have to clone all the AWS services yourself. If you just need a few boxes, there are definitely cheaper options than AWS.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#66

Post by neuromantik8086 »

mikeryan wrote: post #40

This is cool but it seems strange (to me) that this was a “Hackathon” project as opposed to just a stand-alone problem to be addressed as a normal course of doing business. It doesn’t make the solution less cool. It just seems like a strange distinction on what a Hackathon is.

A hackathon is something that used to be a cool party for geeks (i.e., a Mathletics competition or an ACM programming contest) until the corporate overlords bastardized it and converted a good thing into unpaid overtime with free beer.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#67

Post by hueving »

nemothekid wrote: post #34
orf wrote: post #8

I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…

> Seems like a fundamental misunderstanding of SQL rather than a particularly hard problem to solve. Without knowing the rest of their stack, or what their data ingestion looks like, I think your query is oversimplified. If they are doing a union, then it's likely they aren't querying one table, but they are querying multiple tables. The article mentions that individual customers had as many as 500 million rows. Like…

>It's less likely that they overengineered - we are probably just underinformed.

Based on 15 years in software companies in the valley it's much less likely that this isn't over-engineered. Nearly every decision I've seen chasing technology hype has been based on ignorance of existing solutions.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#68

Post by flukus »

jehlakj wrote: post #61
owenmarshall wrote: post #13

Earlier quoted context omitted.

> Stuff like this is bread and butter SQL. Ten or fifteen years ago, sure - a DBA would look at a query plan and figure out how to do it properly. Worse case you'd slap a materialized view in and query that. But this is 2018! Programmers don't want to treat the database as anything but one big key value store ;)

You can thank full stack developers for that

If they don't know how to look at a query plan and craft a solution can you really call them full stack?

Re: We saved $50k/year with a Go microservice coded in a hackathon

#69

Post by nstart »

any626 wrote: post #29
orf wrote: post #8

I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…

The user data is most likely in rows instead of columns. Instead of having id, name, age, gender 1213, fake, 60, female they would have property_id, user_id, value 1 (assume age), 1213, 60 2 (gender), 1213, female This gives them the freedom to add more properties to the user without always having to add a column to the users table. When querying the database you'll have to do unions or joins.

Oh gosh this pattern. The first time I encountered it was in my first job where we used Magento. Super flexible. Also super slow. Does anyone have any advice how to make a db design like this work faster? Generally I thought when data is arranged like this it might be a prime candidate for document based storage. But I'm no dba so I have no idea if that would be correct.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#70

Post by pandapunchpower »

EGreg wrote: post #52
cookiecaper wrote: post #19

Earlier quoted context omitted.

Yeah, sadly, this is not too much of an exaggeration. I've worked on teams that insisted they needed DynamoDB, because, well, Dynamo is for "Big Data", and they certainly wouldn't work somewhere that had "Small Data"! Replace the buzzwords/products as applicable; you could actually probably just scramble them and it'd work just as well, since someone out there thinks "RabbitMQ means Web Scale", etc. SQL databases are…

Question, though... If your columns have types, how can you encrypt them using custom keys for each one? Is it possible? I want the keys to reside on the client and search for encrypted data. Basically single row lookups at a time.

The column type would have to be the type of the encrypted value. The type of the unencrypted data could not be enforced by the DB and you would have to rely on code doing the correct thing.

I am however extremely wary of doing it that way. I don't know your requirements of course.

Post reply on HN