Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We saved $50k/year with a Go microservice coded in a hackathon

movio.co

Post reply on HN
Print view View original

201–210 of 264 posts

Re: We saved $50k/year with a Go microservice coded in a hackathon

#201

Post by justherefortart »

rbranson wrote: post #65
hippich wrote: post #14

What else it shows - how expensive AWS hardware vs hosting own hardware. I guess you have to consider how often you have to scale, but hetzner offers dedicated servers with 64Gb and NVMe drives starting from 54 euros per month - https://www.hetzner.com/dedicated-rootserver?country=us - compare that to $580 per month these guys were paying for i3.2xlarge instance..

... and then you have to clone all the AWS services yourself. If you just need a few boxes, there are definitely cheaper options than AWS.

And now you have a "nexus" in California and they think you should pay their state taxes.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#202

Post by jmts »

Regardsyjc wrote: post #47
owenmarshall wrote: post #13

Earlier quoted context omitted.

> Stuff like this is bread and butter SQL. Ten or fifteen years ago, sure - a DBA would look at a query plan and figure out how to do it properly. Worse case you'd slap a materialized view in and query that. But this is 2018! Programmers don't want to treat the database as anything but one big key value store ;)

Do you have any recommendations for resources to learn best database practices? I'm currently designing my first database and I'm not sure what information is worth storing (like calculations) and how to choose which data to group in tables.

I'm going to go out on a limb and suggest you start writing the application without a database first.

My early education on databases always seemed to follow a "how do we make a database do this?" rationale rather than "what data do we need to store to support these features?", which I think leads to a software design that is too strongly coupled with the database. Software modules end up dependent on database features, or table structure, and refactoring or switching data stores becomes more costly.

Instead, start with a simple in-memory data store - a list of objects with some interface for accessing them, will probably your starting point. Add some basic serialisation/deserialisation features (CSV, JSON, etc) when you get past initial testing and require some persistent data. Then, once you have your API in place and your software design is stabilising, you should be able to map that data to a database fairly easily:

* The primary structure maps to your main table

* Child structures become additional tables, with foreign keys

* Data used to lookup records can be indexed for better performance

Beyond that, you should profile/benchmark your application to find what needs to be optimised, and then investigate whether your software design or your data store should be doing the optimisation.

Let your software's features influence the design of your database. Don't let the database's features influence the design of your software.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#203

Post by tzahola »

So, instead of fixing your messed up data model, you wrote a service (sorry, microservice) which tries to keep your main DB in sync with a columnar cache, so you can keep using that awful query with multiple UNIONs. Throw in some "Go" and boy, you have your Medium post going!

Re: We saved $50k/year with a Go microservice coded in a hackathon

#204

Post by 83457 »

TimJYoung wrote: post #59
any626 wrote: post #33

Earlier quoted context omitted.

You would need to have a new join for each new property SELECT DISTINCT loyaltyMemberID from members as m INNER JOIN properties as p1 on m.id = p1.user_id INNER JOIN properties as p2 on m.id = p2.user_id INNER JOIN properties as p3 on m.id = p3.user_id AND (p1.prop = 'gender' AND p1.value = x) AND ((p2.prop = 'age' AND p2.value = y) OR (p3.prop = 'censor' AND p3.value = z))

There's no need for the extra joins, you can just do the one join and then filter everything in the WHERE clause: SELECT DISTINCT loyaltyMemberID from members as m INNER JOIN properties as p on m.id=p.user_id WHERE (p.prop='name' AND p.value = value) AND ...etc.

But how would you do exclusions with your approach?

Re: We saved $50k/year with a Go microservice coded in a hackathon

#205

Post by smoe »

attaboyjon wrote: post #78

I've come to the conclusion that the problem in tech is that all the people doing the work are in their early twenties and have no idea what they are doing. Once they get some experience they are quickly promoted to the CTO position. Rinse and repeat. What we have here is a classic dbms problem and no one at Movio seems to know how to deal with that. Instead of migrating from Mysql to something serious (Postgres) the…

I think it is not necessarily the age but the mindset of focusing on solutions instead of understanding the problem first.

It often goes like this: Oh snap, we encountered a problem! Lets find a tool, framework, language that promises to solve a similar sounding problem. Now we have a problem with a layer of abstraction on top. Soon to be two problems. Lets find a tool, framework, language to solve both of them ...

It is a spaghetti to the wall approach, where you just throw a bunch of things at your problem hoping that something sticks. And who cares how long it will stick.

Secondly as a developer I think in start-ups dedicated db experts are way underrated. Sure your fullstack devs can cobble together some tables, changing them 15 times a day to accommodate business requests and slap indexes on everything that gets slow. That is also the way to get into trouble once you scale, and instead of reflecting why this is, people reach for the bowl of pasta.

I was no different, when just starting out. I thought my biggest strength was, how quickly I can come up with easy "solutions" for any problem the company had. Took me years to realize how silly of an approach this is.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#206

Post by collyw »

attaboyjon wrote: post #78

I've come to the conclusion that the problem in tech is that all the people doing the work are in their early twenties and have no idea what they are doing. Once they get some experience they are quickly promoted to the CTO position. Rinse and repeat. What we have here is a classic dbms problem and no one at Movio seems to know how to deal with that. Instead of migrating from Mysql to something serious (Postgres) the…

Agreed.

I have 15 years of experience and can built a decent clean system using "boring" technologies. But all the decent paid work where I live is maintaining big balls of mud with tech that was obviously peak hype when it was chosen, and nothing done according to best practices because of that would require sticking with a tech and learning it properly. Its quite frustrating.

Then we have the interview process where people expect me to give up my weekend for their coding test and can't even be bothered to give you feedback afterwards. Or some ridiculous algorithmic nonsense that has no relevance to the job. Getting bored of it all.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#207

Post by _pmf_ »

indigochill wrote: post #125
attaboyjon wrote: post #78

I've come to the conclusion that the problem in tech is that all the people doing the work are in their early twenties and have no idea what they are doing. Once they get some experience they are quickly promoted to the CTO position. Rinse and repeat. What we have here is a classic dbms problem and no one at Movio seems to know how to deal with that. Instead of migrating from Mysql to something serious (Postgres) the…

Well, a problem in tech certainly. An alternative possibility (and another problem in tech) could be some mid-level developer could have figured out the problem at the start but because of artificial time pressure to deliver they didn't have the time to, so went with the first bad idea that popped into their head without taking the necessary time to evaluate it. The fact this had to happen in a hackathon suggests a t…

Typical prototype-gone-live scenario. A.k.a. 90 per cent of projects where prototypes are used at any stage ...

Re: We saved $50k/year with a Go microservice coded in a hackathon

#208

Post by icedchai »

tzahola wrote: post #203

So, instead of fixing your messed up data model, you wrote a service (sorry, micro service) which tries to keep your main DB in sync with a columnar cache, so you can keep using that awful query with multiple UNIONs. Throw in some "Go" and boy, you have your Medium post going!

If they said "we added a couple of indexes to some database tables for 10x improvement", it wouldn't be too exciting, right? ;)

Re: We saved $50k/year with a Go microservice coded in a hackathon

#209

Post by collyw »

jehlakj wrote: post #61
owenmarshall wrote: post #13

Earlier quoted context omitted.

> Stuff like this is bread and butter SQL. Ten or fifteen years ago, sure - a DBA would look at a query plan and figure out how to do it properly. Worse case you'd slap a materialized view in and query that. But this is 2018! Programmers don't want to treat the database as anything but one big key value store ;)

You can thank full stack developers for that

I call myself full stack and I can tune a database. decent database design comes first though.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#210

Post by Thaxll »

dijit wrote: post #122
peburrows wrote: post #89

Earlier quoted context omitted.

Overall, I agree with the sentiment of your comment, but the assertion that MySQL isn’t a serious database is just flat out wrong.

I disagree with your assertion that MySQL /is/ a serious database. The questions I usually ask myself when evaluating database solutions is: * Does it accept invalid data? * Does it change data on error? * Does the query planner change drastically between minor versions? * How strong is transaction isolation? can I create constraints, columns or tables in a transaction? * Does it scale vertically above 40~ CPU thread…

All this list is fud. You should provide examples to those claims because right now you sound like someone that never use MySQL.
Post reply on HN