Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We saved $50k/year with a Go microservice coded in a hackathon

movio.co

Post reply on HN
Print view View original

151–160 of 264 posts

Re: We saved $50k/year with a Go microservice coded in a hackathon

#151

Post by stu432 »

dbrgn wrote: post #117
DCoder wrote: post #97

Earlier quoted context omitted.

This is a pretty popular pattern known as Entity-Attribute-Value [0]. It's used by many products where a) data model needs to be very flexible and allow new attributes without schema changes, or b) a typical entity has a large number of possible attributes that may or may not be set for all entities ("sparse" attributes). WordPress uses this to store post metadata, Magento uses this to store product attributes and mo…

Sounds like a great case for Postgres hstore (like OpenStreetMap does it)?

I mean, not really, theres no magic in Postgres' implementation of the pattern. They even spell it out for you:

"can be useful in various scenarios, such as rows with many attributes that are rarely examined"

They are querying these quite heavily, they aren't just random attributes they need to retrieve.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#152

Post by btb »

mikeryan wrote: post #40

This is cool but it seems strange (to me) that this was a “Hackathon” project as opposed to just a stand-alone problem to be addressed as a normal course of doing business. It doesn’t make the solution less cool. It just seems like a strange distinction on what a Hackathon is.

My guess would be a dev that knew that something was wrong with the performance of their old system, but for whatever reason it wasnt something that anyone higher up prioritized. If thats the case, I can see why one might come up with an idea of trying to solve it outside the bounds of the normal production/SQL environment. At least in my company, the only time we devs ever get to specifically look at the performance of our website is when we have "hackathon" days where we choose our own projects. I often times feel like my regular time would be much better spent trying to optimize our 2 second+ initial pageload times, instead of all the other small tasks/tweaks/bugfixes that gets sent my way. But performance is something very few people higher up seems to care about. Or maybe its a case of users and managers becoming so accustomed to something being slow they dont notice anymore.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#153

Post by ptman »

zaarn wrote: post #132
icebraining wrote: post #94

Earlier quoted context omitted.

The question is whether it can be stored like this while allowing for fast queries. For example, unless it changed recently, Postgres doesn't calculate statistics to help the query planner on jsonb fields.

JSONB columns should behave just like any datatype with a GIN index in the recent releases, to my knowledge. Still, a JSON column will arguably be faster than a pure KV table since you can more efficiently query it, especially any non-JSON columns.

IIRC JSONB still has problem with index statistics

So values in JSONB columns can be indexed nicely, but the statistics can be much worse than for non-JSONB columns, which can lead the query planner astray.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#154

Post by BartBoch »

dijit wrote: post #122
peburrows wrote: post #89

Earlier quoted context omitted.

Overall, I agree with the sentiment of your comment, but the assertion that MySQL isn’t a serious database is just flat out wrong.

I disagree with your assertion that MySQL /is/ a serious database. The questions I usually ask myself when evaluating database solutions is: * Does it accept invalid data? * Does it change data on error? * Does the query planner change drastically between minor versions? * How strong is transaction isolation? can I create constraints, columns or tables in a transaction? * Does it scale vertically above 40~ CPU thread…

I think a lot of what have you written can be solved software-side. Good Database should be no excuse for bad code.

I do not think MySQL is a technological debt as in 80% startups moving to the different solution is cheap and non-problematic. The LAMP is good enough and quickest/cheapest for the majority of tech companies.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#155

Post by est »

orf wrote: post #8

I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…

I am building a rule engine quite similar to this. An AST parser will run all python DSLs and generated list of tables for INNER JOIN, then SELECT all the tables data out with filters in one pass, then run all results through the Python code.

It's quite fun.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#156

Post by poooogles »

jjeaff wrote: post #146
dijit wrote: post #122

Earlier quoted context omitted.

I disagree with your assertion that MySQL /is/ a serious database. The questions I usually ask myself when evaluating database solutions is: * Does it accept invalid data? * Does it change data on error? * Does the query planner change drastically between minor versions? * How strong is transaction isolation? can I create constraints, columns or tables in a transaction? * Does it scale vertically above 40~ CPU thread…

Considering many, many of the world's largest tech companies use MySQL or MySQL compatible databases, it's rather absurd to say that MySQL isn't a serious database. Regardless of whether it matches yours or someone else's personal list of capabilities.

One true Scotsman fallacy at work is what it is.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#157

Post by wimdetroyer »

attaboyjon wrote: post #78

I've come to the conclusion that the problem in tech is that all the people doing the work are in their early twenties and have no idea what they are doing. Once they get some experience they are quickly promoted to the CTO position. Rinse and repeat. What we have here is a classic dbms problem and no one at Movio seems to know how to deal with that. Instead of migrating from Mysql to something serious (Postgres) the…

What's wrong with mySQL?

Re: We saved $50k/year with a Go microservice coded in a hackathon

#158

Post by mjburgess »

silveroriole wrote: post #114

Gotta agree with others and say that they’re clearly skimming over the facts that: - they didn’t have the expertise to actually fix the SQL. That query smells bad. The data model smells bad. For some reason HN is always superstitiously afraid of letting developers touch the database, but if you don’t let devs touch the database enough you end up with this sort of thing; or that crap data model with properties in rows…

There's no CTO on https://movio.co/en/company/

Seems like a red flag.

I'm all for companies releasing technical blog posts, but there's some really strange framing here.

This is actually a story about how decisions get made, and how better ones can be made. Reading a company's mea cupla tells you they are well-informed and well-intentioned.

This is not a story which presumes good decisions were made and "the (tiny, startup) database company went bust". That's their framing. Yikes.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#159

Post by majewsky »

zaarn wrote: post #127
matte_black wrote: post #83

Earlier quoted context omitted.

If that’s where they bottom out they should be called “mid stack” rather than full stack. If you are claiming to be full stack you better be prepared to go all the way.

I would describe myself full stack under that definition. Most "full stack" people I know that sit in my Uni courses have mostly learned Java EE + Oracle DB or Javascript + /dev/null^w^w MongoDB. Most of them would probably not be able to construct a relational database or libc from scratch. Granted, such knowledge isn't immediately useful since it's something I or anyone is likely to do but it grants insight into sy…

> /dev/null^w^w MongoDB

Nitpick: That's actually either three ^w or just one ^w, depending on how your WORDCHARS is set up. :)

Re: We saved $50k/year with a Go microservice coded in a hackathon

#160

Post by chiefalchemist »

olalonde wrote: post #136
chiefalchemist wrote: post #63

Earlier quoted context omitted.

Them, or the manager that hired them? If no one up the chain brings on a DBA what are they supposed to do? I hear ya. But this is as much a symptom of naive (and budget stretched) leadership as it is of the hands on deck.

Your solution costs 150K$/year while theirs took a weekend and saves 50K$/year...

Allow me to clarify.

1) It took a weekend to complete. The friction was building for far longer. There's a cost to that, esp if it effects customer satisfaction and retention. They didn't refactor for fun, did they :) How many dev teams aren't so lucky? Is this article a no choice outlier, or a best practice?

2) My comment wasn't directed at the article but on another comment that blamed the developers. These problems should be owned by ownership / leadership / management more and engineers less.

3) That said, hire a DBA? I don't think that's necessary.

Post reply on HN